Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

PE encryptor/decryptor

  1. Jul 11, 2008 #1
    I am working on a PE encryptor / decryptor which doesn't use a static key. The encryptor generates a random number (1-255) to use as its key. Knowing that the first character in all PE files (excluding VxDs and 16-bit DLLs) is "M" (starting the DOS MZ header), the decryptor will keep trying different keys until the first byte of the PE is "M". Once we find the key, we decrypt the rest of the file. What I have written looks like this:

    Code (Text):
        lea esi, offset FILE_CONTENTS   ; ESI = contents of encrypted file
        xor edi, edi            ; EDI = 0 (encryption key)

        inc [edi]               ; EDI++
        xor byte ptr [esi], [edi]       ; FILE_CONTENTS[ESI] = XOR FILE_CONTENTS[ESI], EDI
        cmp byte ptr [esi], "M"     ; did we find the key?
        jne find_key            ; if not, check again

        mov ecx, FILE_SIZE - 1      ; ECX = FILE_SIZE - 1 (we already decrypted the first byte)

        xor byte ptr [esi], [edi]       ; encrypt byte
        inc esi             ; get next byte
        loop    DECRYPT             ; loop until ECX = 0
    Unfortunately, this doesn't work. I get the error "Operand types do not match" in both of the XOR lines. Does anyone have any input or ideas on how to solve this problem?
  2. jcsd
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook

Can you offer guidance or do you also need help?
Draft saved Draft deleted