19-06-15, 11:55 | #1 |
Member
Joined: Sep 2006
Posts: 24
|
Does anyone have the source code for TRL extaction tools?
I'm want to compare it with my Soul Reaver2 mesh extractor to help me fix some bugs, since the games use the same engine.
Last edited by CodeReaver; 19-06-15 at 12:18. |
19-06-15, 20:28 | #2 |
Member
Joined: Dec 2010
Posts: 2,773
|
I don't think the TRL available mesh tools extracted everything. The SR tools in general are more advanced. The mesh formats should be drastically different since Legend is based off Defiance engine.
Regards. |
20-06-15, 19:47 | #3 |
Member
Joined: Sep 2006
Posts: 24
|
Well, Defiance is based on the Soul Reaver 2, so technically SR2 is Legend's grandparent. It still inclues many of the same functons if you compare them in PS2DIS. SR2 uses drm, vrm, smf, snf and raw files, with most of the formats being exactly the same. The drm files are slightly different, but I'm hoping Legend uses the same method of packing the floating point texture coordinates into two bytes, because something is going wrong in the code I'm using. I've no idea how it works in Defiance as I'm trying to work my way up through the iterations of the engine.
Last edited by CodeReaver; 20-06-15 at 19:48. |
20-06-15, 23:38 | #4 | |
Member
Joined: Dec 2010
Posts: 2,773
|
Quote:
Regards. |
|
22-06-15, 13:05 | #5 |
Member
Joined: Sep 2006
Posts: 24
|
On Soul Reaver 1 (Dreamcast version), Soul Reaver 2 (PC *and* PS2) and possibly Defiance, the UVs are packed into 4 bytes using SIMD instructions. We have a working unpack function for the SR1 Dreamcast version, but on SR2 PC the code is a little messier with stages spread across a few different functions, so it's harder to debug and confirm that the same formula works. I found a .NET reflector that I tried on the Tomb Raider: Underworld extractor and all that seems to do is multiply by 2048.0f to get the extracted value. Doing that doesn't help much with SR2 and the Dreamcast formula is the only thing that gives a close approximation even though there's a lot that it gets wrong.
I can tell you that there's little difference in the file format across platforms for SR1, SR2 and I think Defiance as well. The textures on the other hand are quite different, although off topic, you might be interested to know that the format changed very little from the SR1 Proto1 demo to the retail version. Last edited by CodeReaver; 22-06-15 at 13:10. |
22-06-15, 17:57 | #6 | |
Member
Joined: Dec 2010
Posts: 2,773
|
Quote:
This is very interesting! Maybe you could look into decompiling the actual code itself, it's useful and normally effective for PS1/PS2 games! I did manage to decompile the code used for hashing the file names in Soul Reaver 1 and wrote a bigfile unpacker (but there are already existing ones). I will have to note that the hashing "algorithm" itself is disgraceful, they didn't get very creative hence why earlier builds of the game I played crash due to duplicate hash names. It's fairly simple but it does work fine for the final version even though it wasn't adjusted that much. I've never heard of UVs being packed into 4 byte SIMD instructions. It actually sounds similar to VIF tagging used across various PS2 games (including TRL I believe). Is this what you meant? Usually VIFTags need to be decoded, consisting of 4 bytes which reveal the data type for the buffer and length. If it is that then you could take a look at Noesis since it's perfectly supported but I don't believe it will solve the issue with the UV data itself needing to be converted. That's right about Tomb Raider: Underworld, all games after this also require UVs to be scaled up by 2048f. What do the UVs look like when you read them? Do they look scaled? It could also be a signed/unsigned issue. If you can load the UV data and show me how they look I might have something to suggest. I suspect that the textures are probably 4-bit or 8-bit TIM/TIM2 textures. I did take a look at "Proto1" and the VRM files just screamed TIM at me. I recall the bigfile format being slightly different from the retail release, can't remember if I wrote an unpacker or anything but there's probably one existing already. Regards. |
|
22-06-15, 22:01 | #7 |
Member
Joined: Sep 2006
Posts: 24
|
Regarding the decompiler, you're right, that might at least give me an idea at what point the temporary buffer is allocated. The data is modified, then shoved into a buffer for the second pass, but the debugger isn't good at keeping track of memory allocations and can't seem to break on read by an SIMD instruction. When I said about SIMD before, I didn't mean the UVs were stores as SIMD instructions, I mean they are read by those instructions like so:
punpcklwd mm2, mm2 // Turns 0000LLRR into LLLLRRRR psrld mm2,10 // Turns LLLLRRRR into 00LL00RR by shifting both WORDs right by 16 The VIFTag stuff you mentioned sounded interesting, however it doesn't sound like it would match the { fX, fY, fZ, uColour, usU, usV } structure I'm seeing (float, float, float, unsigned int, unsigned short, unsigned short). Regarding the hash algorithms, does that mean you have formulas for the SR1 Betas? I think I know where the code is, but IIRC, it was a very long function and I never had the time. I have the formulas for Retail SR1, Retail SR2, European SR2 Demo, US SR2 Demo and Defiance if you need any of those. If you'd be willing to share any of the ones you have, hash algrorithms for TRL, TRA and TRU and and other versions Soul Reaver could be really helpful for upgrading the unpacker I've been using (it's meant to be multi-game). The SR1 PS1 texture format isn't tim, although there might have been a couple of those in there. It's an indexed format where the colour table is stored with the mesh data. EDIT: I don't really have anywhere to store screenshots at the moment. I'll try to figure something out. Last edited by CodeReaver; 22-06-15 at 22:09. |
22-06-15, 22:35 | #8 | |
Member
Joined: Dec 2010
Posts: 2,773
|
Quote:
I tried to find code for SR's hashing algorithm from when I wanted to unpack the archives for a beta version but unfortunately couldn't find anything out in public Had to spend several minutes decompiling mips assembly but it was fun nevertheless. Surprisingly, the algorithm is very similar to the final release from a version i looked at. The only difference is no if statement is required which checks if the current char in the for loop iterating through the string is equal to the first char in the string. Code:
if (c != str[0]) { //Calc hash } Here is the code for the hashing algorithm used by all Crystal Tomb Raider games for every single platform Regards. Last edited by Gh0stBlade; 22-06-15 at 22:38. |
|
23-06-15, 14:04 | #9 |
Member
Joined: Sep 2006
Posts: 24
|
I can show you samples of the data before, but not after. Like I said, it's a two pass algorithm. It does the first stage, then buffers it and then does more with the buffered data at some much later time. Cheat Engine can't seem to detect what reads the buffered data back in. Actually I thing CE might be misreading a few instructions. I can show you the algorithm that worked on SR1 and the data it outputs. It does get pretty close, but just the odd polygon here and there seems to go wrong.
Thanks for the TRL hash algorithm. That will come in handy. Do you want the Soul Reaver hash algorithms I mentioned? Is there somewhere I can send them? You say you have a MIPS decompiler. Do you know where to find a Dreamcast one? Or even a dissembler like PS2DIS? |
23-06-15, 15:47 | #10 | |
Member
Joined: Dec 2010
Posts: 2,773
|
Quote:
Cheat Engine is a bad tool for "modifying" PS1 games, especially when wanting to set breakpoints etc. Usually it refuses to break or gives misleading information as it wasn't intended for debugging emulators. I believe that's what you're experiencing. I recall someone creating a debugger that can attach to emulators whilst displaying the mips asm and furthermore allowing breakpoints to be set (I think that was for PS2 only and cannot remember the name, was written by LXShadow). Normally, no$psx is what I use for PS1 games. It's an emulator with a built in debugger, it's usually enough for these sort of tasks but I don't like the interface. I'd like to see the SR hashing algorithms, I actually thought they'd be the same as Tomb Raider Legend but I guess I'm wrong. You could probably drop the code on something like pastebin. You could use IDAPro on Dreamcast games, I don't have a MIPS decompiler, i just used a disassembler and decompiled it myself. The decompiled code is as follows (Ignore the ugliness, never bothered to clean it up): Code:
unsigned int BigFileV2_GetHash(string str) { unsigned int c = 0, cipher = 0, r21 = 0, sum = 0; int extID = BigFileV2_GetExtensionID(str); //If a known extension is detected we need to cut it off! if (BigFileV2_IsValidExtension(str)) { str = GetFileNameWithoutExtension(str); } for (int i = (str.size() - 1); i >= 0; --i) { c = str[i]; //Final Version only! if (c != str[0]) { if ((c - 0x61) < 0x1A) c = (c & 0xDF); sum ^= ((c - 0x1A) & 0xFF) * cipher; r21 += ((c - 0x1A) & 0xFF); cipher++; } } return ((cipher << 0x1B) | (r21 << 0xF) | (sum << 0x3) + extID); } |
|
Thread Tools | |
|
|