I'm currently trying to reverse engineer the decryption algorithm for an old online game, using a chat message packet, as it contains text which is easily recognizable.

I used a packet sniffer to get the received packet (which is encrypted) from the server:

5e 00 09 32 3c c1 6e b5 ae be 90 4a 70 8f b7 3d
3a 81 c5 3a f9 11 a6 06 36 3d f3 68 a3 dd 72 a3
ff e1 c3 e9 12 28 d7 c5 a0 f1 ce 38 35 59 19 b6
85 90 76 23 42 af 50 6f 66 52 ec ad f9 da 61 3f
5d 09 ee 8d dd 9e ff ee 7d 27 0f 5c 1e df ba 30
de d0 c8 8c 1b 93 1d 53 66 13 98 ff 29 db

The first 4 bytes are known and unencrypted:

5E 00 is the size, here 94.
09 32 is the OPCODE as a big endian: here 2354.
What follow is the encrypted payload.

Knowing the encrypted packet, I used Cheat Engine to set a breakpoint right after the winsocket receive function and then search the buffer where the packet is stored.

**Important to note:** Apparently, the buffer which stores the received (encrypted) packet will also store the decrypted one.

Checking what writes to this address results in exact one instruction.
4ssuh.png

As the buffer which contains the encrypted packet also contains the decrypted one, I can say for sure that this has to be the function which decrypts the value.

The decrypted packet looks the following (last bits contain the chat message, somewhere in between the username is stored):

5e 00 09 32 00 00 00 00 4d 00 6e 60 00 00 00 00
06 0a 2c 00 00 00 74 00 65 00 73 00 74 00 63 00
68 00 61 00 72 00 31 00 00 00 91 12 ad 14 5e 75
00 00 cf 00 00 00 00 00 28 cd b8 69 fc f7 91 12
aa f1 07 4d 00 00 cf 00 00 00 00 00 10 74 65 73
74 5f 6d 65 73 73 61 67 65 5f 31 32 33 00

I then used Ghidra to inspect the specific instructions which handle the decryption part - this is where I am currently stuck and don't know how to proceed, as I don't have a deep understanding of this topic.


Code:
undefined4 decrypt_package(int type,int param_2,int param_3,int param_4,int param_5)
{
undefined4 success;
int counter;

if ((param_3 == 0) || (param_4 == 0)) {
    s = 0;
}

else {
    // Old encryption function which only used byte ^ 255
    // I assume it's still in here, because the code wasn't cleaned up.
    if (type == 0) {
        counter = 0;
        while (counter < param_5) {
        *(byte *)(param_3 + counter) = *(byte *)(param_4 + counter) ^ 0xff;
        counter = counter + 1;
    }
}
else { // Don't know when this is used - at least not for decryption
if (type == 1) {
    param_2 = 0x48473c;
    counter = 0;
    while (counter < param_5) {
        *(byte *)(param_3 + counter) = *(byte *)(param_4 + counter) ^ (byte)((uint)param_2 >> 8);
        param_2 = ((uint)*(byte *)(param_3 + counter) + param_2) * 0x2ba339 + 0x2cad2b5;
        counter = counter + 1;
    }
}

else {
// This block of code is called for the decryption part.
if (type == 2) {
    counter = 0;
    while (counter < param_5) {
        *(byte *)(param_3 + counter) = *(byte *)(param_4 + counter) ^ (byte)((uint)param_2 >> 8);
        param_2 = ((uint)*(byte *)(param_3 + counter) + param_2) * 0x8e9a99 + 0x685b24;
        counter = counter + 1;
    }
}
}
}
success = 1;
}
return success;
}
(Sorry for the weird formatting, but I wasn't able to copy it with the right indent)

(Note for the type == 0: When the game first released, it only used a simple XOR with 255 as an encryption, this was apparently later discontinued)

From my understanding, the code does the following (assuming, param_2 is the buffer for the packet?):

1. Loop over the buffer, as long as counter < param_5 (which I assume is the length of the payload to decrypt?)
2. XOR the byte with a value (first byte that needs to be decrypted would be at position 5, so I assume that param_3 would be a kind of offset to skip the first 4 bytes.)
3. Return a bool (don't know why it is declared as undefined) whether the decryption worked or not.

Below I also posted the assembly code for this block of code (starting at the counter = 0 in the while loop for type == 2)
uCtlF.png

----------

As said in the beginning, I am not experienced when it comes to this kind of things.

I therefore would like some additional information on how I would continue from here on, so that I could implement the decryption in my programming language of choice - what exactly is the code block above doing?

I hope I included all information needed.

Thanks for reading (and helping)!