Tomb Raider Forums  

Go Back   Tomb Raider Forums > Tomb Raider Level Editor and Modding > Tomb Raider Modding

Reply
 
Thread Tools
Old 19-06-15, 11:55   #1
CodeReaver
Member
 
Joined: Sep 2006
Posts: 24
Default Does anyone have the source code for TRL extaction tools?

I'm want to compare it with my Soul Reaver2 mesh extractor to help me fix some bugs, since the games use the same engine.

Last edited by CodeReaver; 19-06-15 at 12:18.
CodeReaver is offline   Reply With Quote
Old 19-06-15, 20:28   #2
Gh0stBlade
Member
 
Gh0stBlade's Avatar
 
Joined: Dec 2010
Posts: 2,773
Default

I don't think the TRL available mesh tools extracted everything. The SR tools in general are more advanced. The mesh formats should be drastically different since Legend is based off Defiance engine.

Regards.
Gh0stBlade is offline   Reply With Quote
Old 20-06-15, 19:47   #3
CodeReaver
Member
 
Joined: Sep 2006
Posts: 24
Default

Quote:
Originally Posted by Gh0stBlade View Post
I don't think the TRL available mesh tools extracted everything. The SR tools in general are more advanced. The mesh formats should be drastically different since Legend is based off Defiance engine.

Regards.
Well, Defiance is based on the Soul Reaver 2, so technically SR2 is Legend's grandparent. It still inclues many of the same functons if you compare them in PS2DIS. SR2 uses drm, vrm, smf, snf and raw files, with most of the formats being exactly the same. The drm files are slightly different, but I'm hoping Legend uses the same method of packing the floating point texture coordinates into two bytes, because something is going wrong in the code I'm using. I've no idea how it works in Defiance as I'm trying to work my way up through the iterations of the engine.

Last edited by CodeReaver; 20-06-15 at 19:48.
CodeReaver is offline   Reply With Quote
Old 20-06-15, 23:38   #4
Gh0stBlade
Member
 
Gh0stBlade's Avatar
 
Joined: Dec 2010
Posts: 2,773
Default

Quote:
Originally Posted by CodeReaver View Post
Well, Defiance is based on the Soul Reaver 2, so technically SR2 is Legend's grandparent. It still inclues many of the same functons if you compare them in PS2DIS. SR2 uses drm, vrm, smf, snf and raw files, with most of the formats being exactly the same. The drm files are slightly different, but I'm hoping Legend uses the same method of packing the floating point texture coordinates into two bytes, because something is going wrong in the code I'm using. I've no idea how it works in Defiance as I'm trying to work my way up through the iterations of the engine.
Yep, there were also quite a lot of changes between Defiance and Legend but the main structures are all basically the same. I didn't get chance to pull out the PS2 meshes, nobody has ever done that yet. The PC UVs are just float but to my knowledge when I looked the PS2 formats for the TRL comparing to PC showed they're drastically from each other so it might not be float for PS2 meshes.

Regards.
Gh0stBlade is offline   Reply With Quote
Old 22-06-15, 13:05   #5
CodeReaver
Member
 
Joined: Sep 2006
Posts: 24
Default

On Soul Reaver 1 (Dreamcast version), Soul Reaver 2 (PC *and* PS2) and possibly Defiance, the UVs are packed into 4 bytes using SIMD instructions. We have a working unpack function for the SR1 Dreamcast version, but on SR2 PC the code is a little messier with stages spread across a few different functions, so it's harder to debug and confirm that the same formula works. I found a .NET reflector that I tried on the Tomb Raider: Underworld extractor and all that seems to do is multiply by 2048.0f to get the extracted value. Doing that doesn't help much with SR2 and the Dreamcast formula is the only thing that gives a close approximation even though there's a lot that it gets wrong.

I can tell you that there's little difference in the file format across platforms for SR1, SR2 and I think Defiance as well. The textures on the other hand are quite different, although off topic, you might be interested to know that the format changed very little from the SR1 Proto1 demo to the retail version.

Last edited by CodeReaver; 22-06-15 at 13:10.
CodeReaver is offline   Reply With Quote
Old 22-06-15, 17:57   #6
Gh0stBlade
Member
 
Gh0stBlade's Avatar
 
Joined: Dec 2010
Posts: 2,773
Default

Quote:
Originally Posted by CodeReaver View Post
On Soul Reaver 1 (Dreamcast version), Soul Reaver 2 (PC *and* PS2) and possibly Defiance, the UVs are packed into 4 bytes using SIMD instructions. We have a working unpack function for the SR1 Dreamcast version, but on SR2 PC the code is a little messier with stages spread across a few different functions, so it's harder to debug and confirm that the same formula works. I found a .NET reflector that I tried on the Tomb Raider: Underworld extractor and all that seems to do is multiply by 2048.0f to get the extracted value. Doing that doesn't help much with SR2 and the Dreamcast formula is the only thing that gives a close approximation even though there's a lot that it gets wrong.

I can tell you that there's little difference in the file format across platforms for SR1, SR2 and I think Defiance as well. The textures on the other hand are quite different, although off topic, you might be interested to know that the format changed very little from the SR1 Proto1 demo to the retail version.

This is very interesting! Maybe you could look into decompiling the actual code itself, it's useful and normally effective for PS1/PS2 games! I did manage to decompile the code used for hashing the file names in Soul Reaver 1 and wrote a bigfile unpacker (but there are already existing ones). I will have to note that the hashing "algorithm" itself is disgraceful, they didn't get very creative hence why earlier builds of the game I played crash due to duplicate hash names. It's fairly simple but it does work fine for the final version even though it wasn't adjusted that much.

I've never heard of UVs being packed into 4 byte SIMD instructions. It actually sounds similar to VIF tagging used across various PS2 games (including TRL I believe). Is this what you meant? Usually VIFTags need to be decoded, consisting of 4 bytes which reveal the data type for the buffer and length. If it is that then you could take a look at Noesis since it's perfectly supported but I don't believe it will solve the issue with the UV data itself needing to be converted.

That's right about Tomb Raider: Underworld, all games after this also require UVs to be scaled up by 2048f.

What do the UVs look like when you read them? Do they look scaled? It could also be a signed/unsigned issue. If you can load the UV data and show me how they look I might have something to suggest.

I suspect that the textures are probably 4-bit or 8-bit TIM/TIM2 textures. I did take a look at "Proto1" and the VRM files just screamed TIM at me. I recall the bigfile format being slightly different from the retail release, can't remember if I wrote an unpacker or anything but there's probably one existing already.

Regards.
Gh0stBlade is offline   Reply With Quote
Old 22-06-15, 22:01   #7
CodeReaver
Member
 
Joined: Sep 2006
Posts: 24
Default

Regarding the decompiler, you're right, that might at least give me an idea at what point the temporary buffer is allocated. The data is modified, then shoved into a buffer for the second pass, but the debugger isn't good at keeping track of memory allocations and can't seem to break on read by an SIMD instruction. When I said about SIMD before, I didn't mean the UVs were stores as SIMD instructions, I mean they are read by those instructions like so:
punpcklwd mm2, mm2 // Turns 0000LLRR into LLLLRRRR
psrld mm2,10 // Turns LLLLRRRR into 00LL00RR by shifting both WORDs right by 16

The VIFTag stuff you mentioned sounded interesting, however it doesn't sound like it would match the { fX, fY, fZ, uColour, usU, usV } structure I'm seeing (float, float, float, unsigned int, unsigned short, unsigned short).

Regarding the hash algorithms, does that mean you have formulas for the SR1 Betas? I think I know where the code is, but IIRC, it was a very long function and I never had the time. I have the formulas for Retail SR1, Retail SR2, European SR2 Demo, US SR2 Demo and Defiance if you need any of those. If you'd be willing to share any of the ones you have, hash algrorithms for TRL, TRA and TRU and and other versions Soul Reaver could be really helpful for upgrading the unpacker I've been using (it's meant to be multi-game).

The SR1 PS1 texture format isn't tim, although there might have been a couple of those in there. It's an indexed format where the colour table is stored with the mesh data.

EDIT: I don't really have anywhere to store screenshots at the moment. I'll try to figure something out.

Last edited by CodeReaver; 22-06-15 at 22:09.
CodeReaver is offline   Reply With Quote
Old 22-06-15, 22:35   #8
Gh0stBlade
Member
 
Gh0stBlade's Avatar
 
Joined: Dec 2010
Posts: 2,773
Default

Quote:
Originally Posted by CodeReaver View Post
Regarding the decompiler, you're right, that might at least give me an idea at what point the temporary buffer is allocated. The data is modified, then shoved into a buffer for the second pass, but the debugger isn't good at keeping track of memory allocations and can't seem to break on read by an SIMD instruction. When I said about SIMD before, I didn't mean the UVs were stores as SIMD instructions, I mean they are read by those instructions like so:
punpcklwd mm2, mm2 // Turns 0000LLRR into LLLLRRRR
psrld mm2,10 // Turns LLLLRRRR into 00LL00RR by shifting both WORDs right by 16

The VIFTag stuff you mentioned sounded interesting, however it doesn't sound like it would match the { fX, fY, fZ, uColour, usU, usV } structure I'm seeing (float, float, float, unsigned int, unsigned short, unsigned short).

Regarding the hash algorithms, does that mean you have formulas for the SR1 Betas? I think I know where the code is, but IIRC, it was a very long function and I never had the time. I have the formulas for Retail SR1, Retail SR2, European SR2 Demo, US SR2 Demo and Defiance if you need any of those. If you'd be willing to share any of the ones you have, hash algrorithms for TRL, TRA and TRU and and other versions Soul Reaver could be really helpful for upgrading the unpacker I've been using (it's meant to be multi-game).

The SR1 PS1 texture format isn't tim, although there might have been a couple of those in there. It's an indexed format where the colour table is stored with the mesh data.

EDIT: I don't really have anywhere to store screenshots at the moment. I'll try to figure something out.
Hmm, it should be possible to break near the function that is modifying the Uv data not sure why you can't. But then again, it's very tricky to track such things. Do you have samples of the data before/after it is converted by any chance? It's still useful.

I tried to find code for SR's hashing algorithm from when I wanted to unpack the archives for a beta version but unfortunately couldn't find anything out in public Had to spend several minutes decompiling mips assembly but it was fun nevertheless. Surprisingly, the algorithm is very similar to the final release from a version i looked at. The only difference is no if statement is required which checks if the current char in the for loop iterating through the string is equal to the first char in the string.

Code:
if (c != str[0])
{
  //Calc hash
}
The code is pretty long so overall so since you already have code for it I'm sure you'll understand what I'm referring to. If you don't I can explain it no problem

Here is the code for the hashing algorithm used by all Crystal Tomb Raider games for every single platform

Regards.

Last edited by Gh0stBlade; 22-06-15 at 22:38.
Gh0stBlade is offline   Reply With Quote
Old 23-06-15, 14:04   #9
CodeReaver
Member
 
Joined: Sep 2006
Posts: 24
Default

I can show you samples of the data before, but not after. Like I said, it's a two pass algorithm. It does the first stage, then buffers it and then does more with the buffered data at some much later time. Cheat Engine can't seem to detect what reads the buffered data back in. Actually I thing CE might be misreading a few instructions. I can show you the algorithm that worked on SR1 and the data it outputs. It does get pretty close, but just the odd polygon here and there seems to go wrong.

Thanks for the TRL hash algorithm. That will come in handy. Do you want the Soul Reaver hash algorithms I mentioned? Is there somewhere I can send them?

You say you have a MIPS decompiler. Do you know where to find a Dreamcast one? Or even a dissembler like PS2DIS?
CodeReaver is offline   Reply With Quote
Old 23-06-15, 15:47   #10
Gh0stBlade
Member
 
Gh0stBlade's Avatar
 
Joined: Dec 2010
Posts: 2,773
Default

Quote:
Originally Posted by CodeReaver View Post
I can show you samples of the data before, but not after. Like I said, it's a two pass algorithm. It does the first stage, then buffers it and then does more with the buffered data at some much later time. Cheat Engine can't seem to detect what reads the buffered data back in. Actually I thing CE might be misreading a few instructions. I can show you the algorithm that worked on SR1 and the data it outputs. It does get pretty close, but just the odd polygon here and there seems to go wrong.

Thanks for the TRL hash algorithm. That will come in handy. Do you want the Soul Reaver hash algorithms I mentioned? Is there somewhere I can send them?

You say you have a MIPS decompiler. Do you know where to find a Dreamcast one? Or even a dissembler like PS2DIS?
I am interested if you can somehow post the data before it's converted and the code which partially works!

Cheat Engine is a bad tool for "modifying" PS1 games, especially when wanting to set breakpoints etc. Usually it refuses to break or gives misleading information as it wasn't intended for debugging emulators. I believe that's what you're experiencing.

I recall someone creating a debugger that can attach to emulators whilst displaying the mips asm and furthermore allowing breakpoints to be set (I think that was for PS2 only and cannot remember the name, was written by LXShadow). Normally, no$psx is what I use for PS1 games. It's an emulator with a built in debugger, it's usually enough for these sort of tasks but I don't like the interface.

I'd like to see the SR hashing algorithms, I actually thought they'd be the same as Tomb Raider Legend but I guess I'm wrong. You could probably drop the code on something like pastebin.

You could use IDAPro on Dreamcast games, I don't have a MIPS decompiler, i just used a disassembler and decompiled it myself.

The decompiled code is as follows (Ignore the ugliness, never bothered to clean it up):

Code:
unsigned int BigFileV2_GetHash(string str)
{
	unsigned int c = 0, cipher = 0, r21 = 0, sum = 0;
	int extID = BigFileV2_GetExtensionID(str);

	//If a known extension is detected we need to cut it off!
	if (BigFileV2_IsValidExtension(str))
	{
		str = GetFileNameWithoutExtension(str);
	}

	for (int i = (str.size() - 1); i >= 0; --i)
	{
		c = str[i];

		//Final Version only!
		if (c != str[0])
		{
			if ((c - 0x61) < 0x1A)
				c = (c & 0xDF);

			sum ^= ((c - 0x1A) & 0xFF) * cipher;
			r21 += ((c - 0x1A) & 0xFF);
			cipher++;
		}
	}

	return ((cipher << 0x1B) | (r21 << 0xF) | (sum << 0x3) + extID);
}
Regards.
Gh0stBlade is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



All times are GMT. The time now is 09:28.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.
Tomb Raider Forums is not owned or operated by CDE Entertainment Ltd.
Lara Croft and Tomb Raider are trademarks of CDE Entertainment Ltd.