PE Encryptor/Decryptor: Solve the Error "Operand Types Do Not Match

[maple23]

I am working on a PE encryptor / decryptor which doesn't use a static key. The encryptor generates a random number (1-255) to use as its key. Knowing that the first character in all PE files (excluding VxDs and 16-bit DLLs) is "M" (starting the DOS MZ header), the decryptor will keep trying different keys until the first byte of the PE is "M". Once we find the key, we decrypt the rest of the file. What I have written looks like this:

	lea	esi, offset FILE_CONTENTS	; ESI = contents of encrypted file
	xor	edi, edi			; EDI = 0 (encryption key)

FIND_KEY:
	inc	[edi]				; EDI++
	xor	byte ptr [esi], [edi]		; FILE_CONTENTS[ESI] = XOR FILE_CONTENTS[ESI], EDI
	cmp	byte ptr [esi], "M"		; did we find the key?
	jne	find_key			; if not, check again

	mov	ecx, FILE_SIZE - 1		; ECX = FILE_SIZE - 1 (we already decrypted the first byte)

DECRYPT:
	xor	byte ptr [esi], [edi]		; encrypt byte
	inc	esi				; get next byte
	loop	DECRYPT				; loop until ECX = 0

Unfortunately, this doesn't work. I get the error "Operand types do not match" in both of the XOR lines. Does anyone have any input or ideas on how to solve this problem?

 

[Future Bruno]

Any help would be greatly appreciated.The issue here is that you are using the wrong instruction for the task. The XOR instruction requires two operands of the same size, and you are trying to xor a byte (8 bits) with a dword (32 bits). To fix this, you can use the MOVZX instruction to convert the dword value in edi to an 8-bit byte before performing the XOR operation: FIND_KEY: inc [edi] ; EDI++ movzx eax, byte ptr [edi] ; Convert EDI to a byte in EAX xor byte ptr [esi], al ; FILE_CONTENTS[ESI] = XOR FILE_CONTENTS[ESI], AL (the 8-bit result from MOVZX) cmp byte ptr [esi], "M" ; did we find the key? jne find_key ; if not, check againDECRYPT: xor byte ptr [esi], al ; encrypt byte inc esi ; get next byte loop DECRYPT ; loop until ECX = 0

 

[quicknote]

Thank you for sharing your code and the issue you are encountering. Based on the information provided, it seems like the error is related to the operands used in the XOR instruction. In order for the XOR operation to work, both operands must be of the same size and type. In your code, you are using a byte-sized operand for the first operand and a dword-sized operand for the second operand. This mismatch in operand sizes is causing the error.

One possible solution could be to declare the second operand as a byte as well, by using the "byte ptr" prefix before the second operand in the XOR instruction. This will ensure that both operands are of the same size and type, and the operation can be performed successfully.

Another possible solution could be to use a different instruction, such as "movzx" or "movsx", to move the value of the second operand into a byte-sized register before performing the XOR operation.

I hope this helps you to solve the issue and continue with your project. It is always important to carefully check the sizes and types of operands when working with assembly code to avoid such errors. Good luck!