Tomb Raider 5 Decompilation Project

zdimension · 17-09-17, 21:58

Tomb Raider 5 Decompilation Project

TL;DR AT THE END

The goal of this project is to decompile entirely Tomb Raider 5 to C/C++ code. There are many reasons for that; first, having the source code (or a functionally equivalent code) of the game allows modding it much more extensively than what the existing patches (TREP, TRNG, etc.) can do, because where those patches will inject code, and patch parts of the memory using hacks and workarounds, here, we can simply modify the code and recompile. Second, it is almost certain that Square Enix / Crystal Dynamics are not in possession of the code. We know that the source code of TR1 and TR2 (used to make the mobile versions) have been given to SE by former Core Design employees, but we are almost sure that the source code of the other games have been lost. This project allows us to preserve a part of history that would be lost otherwise.

How?

A lot of information required to decompile the game comes from the symbol files contained in the leaked TR4-5 PSX SDK. A symbol file is a file contains a list of memory address, and the name of "what is at that address". When you load the executable binary file of the game, you only get the raw binary code, without any function or variable name. A symbol file allows us to put back the original function and variable names, hence making the reverse engineering process much easier.
The symbol files contained in the leaked SDK are of a special format that contains much more information than simply the names, and thanks to Steffen Ohrendorf (OpenTomb & EdisonEngine developer) we have been able to get even more than names: function definitions, variable types and even the original file names.
Thus, our current source tree is very similar (for the PSX version, it's identical) to the original one.

Source tree

The repository is divided in multiple folders (exactly like the original source code):

GAME = main game code, that is common between platforms
SPEC_* = platform-specific code
- SPEC_PC = PC version
- SPEC_PSX = PSX version (requires the Psy-Q SDK and toolchain)
- SPEC_PSXPC = PSX version that runs on PC (used for debugging purposes)
- (maybe one day) SPEC_CE = Dreamcast

Current state of the project

As of today (2017-09-17), all three trees compile without errors. The PSX and PSXPC run up to the main game loop, where they just loop and do nothing.
The PSX version can currently go up to the first loading screen (with the spinning CD icon), but a lot of work still needs to be done since on PSX, all the loading code is located in a separate file that is dynamically injected at runtime.
The PC version can currently go up to the main game thread and the level loading part (title level) but the level loading code is not finished yet, so the game crashes a little bit, unless you comment out the main thread code, then the game just loops indefinitely like its PSX counterpart.

Who?

Gh0stBlade: main developer, now works mainly on PSX version
zdimension: main developer of PC version

Legal stuff

All code here comes from decompilation of publicly available (this includes the leaked SDK) binaries and ressources of Tomb Raider 5. Additional informations about the game's internal mechanisms come from the TRosettaStone 3 project. Square Enix plz don't sue us. We're basically retrieveing the game's source for free whereas your developers would want money.

TL;DR

We're decompiling TR5 so that you can mod it more easily (and that we, developers, can understand it better and make better tools).
It is not finished at all.
There is still a lot of work to do.
It may take months, it may take years.

Want to help?

Currently we're searching for people that know C and C++, and that can use IDA Pro or similar reverse engineering tools. Knowledge of x86 and/or MIPS assembly is a plus.

Shut up and take my money!

GitHub:

Main repository: https://github.com/TOMB5/TOMB5

Binaries:
None

Screenshots:
click here

HD86 · 18-09-17, 00:18

I am going to say my opinion and I know many people will not agree. If you want to mod the game extensively, you should just build a new game or play one of the recent Tomb Raider releases.

I personally like the classic Tomb Raider game as it is and I don't want to see it modified extensively, because that will most likely ruin it. It will become a different game. I can't imagine that you are going to be modest in modifying it after you manage to decompile it.

Boobandie · 18-09-17, 01:44

Why rag on the project? No matter what we do you can play Chronicles as is. This is more for the benefit of fan games and a historical record than it is for just modifying TR5. Though once it is decompiled it could lead to mobile and IOS ports of TR5, or a linux version/better functionality on new operating systems.

Personally I can't wait to see where this goes, the TR5 engine is a lot better than TR4 and I've always been annoyed that it wasn't the version included as the Level Editor.

A_De · 18-09-17, 06:01

Quote:

Originally Posted by HD86

I am going to say my opinion and I know many people will not agree. If you want to mod the game extensively, you should just build a new game or play one of the recent Tomb Raider releases.

I personally like the classic Tomb Raider game as it is and I don't want to see it modified extensively, because that will most likely ruin it. It will become a different game. I can't imagine that you are going to be modest in modifying it after you manage to decompile it.

You were right

Just imagine now, how much the game will be ruined by TRNG's plugins, for example

To be serious: my best wishes to the decompiling team. I really want to see the "different game" some day

Caesum · 18-09-17, 07:41

Quote:

Originally Posted by HD86

If you want to mod the game extensively, you should just build a new game or play one of the recent Tomb Raider releases.

That is kinda silly tbh. Using that logic we should suggest barracuda415 to stop and remove his TombATI project, because if one wants to mod the game he can just play TR Anniversary, no? I mean it doesn't even make sense.

Recompiling an engine gives a variety of new possibilities. It's not only about modding the existing game, it's making it possible to create an entire new one based on the source of TR5, or modifying TRLE in order to give it the features of TR5 engine. TR5 has been heavily modified, especially graphically. The main difference is that integers were replaced by floating points, making the entire game less shaky and much better looking on PC version.

Even your beloved TR5 (and mine btw) can benefit greatly if we can manage to reclaim the source code. TR5 is a Windows 98 game which doesn't work well on modern hardware. Heck, at this point TR1 with TombATI is more compatible than TR2-5. The source gives the possibility to modify the executable in order to make it work on modern PCs. No dirty hacks needed, no wrappers, no emulation. Modifying the game can also make it possible to restore PSX-exclusive effects like echo, footprints, dynamic Dreamcast shadows, special effects. It can also allow rewriting the rendering engine to something more modern than DirectX 5 and thus more compatible.

Also, understanding the engine can make it possible to modify enough to be able to make new adventures in it. Maybe even one would be able to replace the old and quirky TRLE engine with it. The source makes it possible to add new features without dirty hacks that do more harm than good (TRNG). We already have the official Room Editor for TR5 along with several tools and PSX devkit.

At last, decompiling the source will give new information for programmers to improve and extend their custom engines, eg. OpenTOMB and OpenLARA.

Dustie · 18-09-17, 07:43

Quote:

Originally Posted by HD86

I am going to say my opinion and I know many people will not agree. If you want to mod the game extensively, you should just build a new game or play one of the recent Tomb Raider releases.

I personally like the classic Tomb Raider game as it is and I don't want to see it modified extensively, because that will most likely ruin it. It will become a different game. I can't imagine that you are going to be modest in modifying it after you manage to decompile it.

You are right about building a new game, some of the builders would definitely love to see a completely new engine at their disposal and with a decompiled code the path to making it happen is pretty much open. A lot the ideas we have oftentimes make us conclude that we might as well make ourselves a new game if we want to make them happen.

But I think the core (sic!) purpose here is to be able to patch certain things up, increase compatibility with newer systems and possibly introduce certain graphical tweaks and extensions, which would add up to the classic experience, not ruin it. I'm convinced such efforts come from passion to the classic Tomb Raider, not the hate for it.

Lwmte · 18-09-17, 10:14

I can only welcome and applause for this effort. And I can't understand people who possibly say that they are happy with current state of the game, and this is why:

1. As already said by Caesum, TR5 is still Win9x application. It's only a matter of time before we can't use it with modern hardware. What's the point of having broken binary which only works through virtual machine or with quirky workarounds like compatibility modes or 16-bit color depth (say hi to winroomedit)?

2. TR5 is the less known engine in old-school TR engine family. File formats are less known, internal processing is less known, lots of stuff altered, replaced, lots of workarounds were written, and so on. Decompiling it will clarify many things that are currently vaguely known.

3. As we already have TR1 and TR2 decompilation projects in progress, having TR5 decompilation is like pinning down whole engine family from both sides - that is, if you have decompiled TR1 or TR2 and decompiled TR5, it means that you (sort of) decompiled TR3 and TR4 as well

Also, as said by Caesum, it will be a great benefit for developing OT/OL engines.

4. Don't forget about some stupid TR5 bugs, especially in final section. You will finally get the chance to fix them.

All in all, decompiling original engine always foreshadow great changes. Updated compatibility, graphics, ability to fork "vanilla" engine into new one, etc. etc. But decompiling is the hardest thing, it's really easier to write engine from scratch than decompiling existing one. Fortunately, from what I see, progress is quite fast. So my best wishes to the team!

HD86 · 18-09-17, 14:16

The current TR game is what makes this community exists. If the source code becomes available, we will have a plethora of new TR games as many people will try to build their own versions of the game. The community will splinter because many will be playing different games.

Anyway, I am of course not against upgrading the game so that it remains compatible with new systems. Technical improvements are welcome as long as they don't change the gameplay.

Zebra · 18-09-17, 14:19

^How is that differernt to what's been happening with the official games for years? People have long been discussing whether LAU or the reboot games (or AoD, for that matter) can be called Tomb Raider so that's really nothing new and I don't think it's much of a problem. People will just keep using their own personal definitions of what is and isn't Tomb Raider and move on with their lives

.

This project is awesome. I hope you'll be successful in recreating the source code

!

DunHill · 18-09-17, 16:28

This project is amazing, I agree with many people here. It could replace the TR4 engine with many new features.
I'm so delighted to see so many people working on the modding community. With the new engines, the new Tomb Editor... All of this will bring so much more possibilities to the TRLE world

Quote:

Originally Posted by HD86

The current TR game is what makes this community exists. If the source code becomes available, we will have a plethora of new TR games as many people will try to build their own versions of the game. The community will splinter because many will be playing different games.

Anyway, I am of course not against upgrading the game so that it remains compatible with new systems. Technical improvements are welcome as long as they don't change the gameplay.

Tomb Raider 5 was built on the same old grid system, it wouldn't make sense to change the gameplay to something modern

I think that the main goal here is to keep the same game, just more compatible with modern hardware

Though, If the code becomes available and we are able to add graphical features and gameplay changes, I think we could find a balance between old and modern. Let me explain myself :
We have to keep the easy way of building thanks to the grid system, for people who don't want to learn c++ or game design to build on newer engines such as Unreal engine. Plus, it will be even easier for people to edit wads and objects with the new editor.
BUT we can find a way to make the gameplay more fluid while keeping it adapted to the grid system.
And Core Design had the right idea when building Anniversary Edition : they imported TR1 maps, modified them graphically, but kept the grid-based system for the jumps and so on while having way more responsive controls.

17-09-17, 21:58	#1
zdimension Hobbyist Join Date: Feb 2016 Location: Somewhere in France Posts: 68	Tomb Raider 5 Decompilation Project Tomb Raider 5 Decompilation Project TL;DR AT THE END The goal of this project is to decompile entirely Tomb Raider 5 to C/C++ code. There are many reasons for that; first, having the source code (or a functionally equivalent code) of the game allows modding it much more extensively than what the existing patches (TREP, TRNG, etc.) can do, because where those patches will inject code, and patch parts of the memory using hacks and workarounds, here, we can simply modify the code and recompile. Second, it is almost certain that Square Enix / Crystal Dynamics are not in possession of the code. We know that the source code of TR1 and TR2 (used to make the mobile versions) have been given to SE by former Core Design employees, but we are almost sure that the source code of the other games have been lost. This project allows us to preserve a part of history that would be lost otherwise. How? A lot of information required to decompile the game comes from the symbol files contained in the leaked TR4-5 PSX SDK. A symbol file is a file contains a list of memory address, and the name of "what is at that address". When you load the executable binary file of the game, you only get the raw binary code, without any function or variable name. A symbol file allows us to put back the original function and variable names, hence making the reverse engineering process much easier. The symbol files contained in the leaked SDK are of a special format that contains much more information than simply the names, and thanks to Steffen Ohrendorf (OpenTomb & EdisonEngine developer) we have been able to get even more than names: function definitions, variable types and even the original file names. Thus, our current source tree is very similar (for the PSX version, it's identical) to the original one. Source tree The repository is divided in multiple folders (exactly like the original source code): GAME = main game code, that is common between platforms SPEC_* = platform-specific code SPEC_PC = PC version SPEC_PSX = PSX version (requires the Psy-Q SDK and toolchain) SPEC_PSXPC = PSX version that runs on PC (used for debugging purposes) (maybe one day) SPEC_CE* = Dreamcast* Current state of the project As of today (2017-09-17), all three trees compile without errors. The PSX and PSXPC run up to the main game loop, where they just loop and do nothing. The PSX version can currently go up to the first loading screen (with the spinning CD icon), but a lot of work still needs to be done since on PSX, all the loading code is located in a separate file that is dynamically injected at runtime. The PC version can currently go up to the main game thread and the level loading part (title level) but the level loading code is not finished yet, so the game crashes a little bit, unless you comment out the main thread code, then the game just loops indefinitely like its PSX counterpart. Who? Gh0stBlade: main developer, now works mainly on PSX version zdimension: main developer of PC version Legal stuff All code here comes from decompilation of publicly available (this includes the leaked SDK) binaries and ressources of Tomb Raider 5. Additional informations about the game's internal mechanisms come from the TRosettaStone 3 project. Square Enix plz don't sue us. We're basically retrieveing the game's source for free whereas your developers would want money. TL;DR We're decompiling TR5 so that you can mod it more easily (and that we, developers, can understand it better and make better tools). It is not finished at all. There is still a lot of work to do. It may take months, it may take years. Want to help? Currently we're searching for people that know C and C++, and that can use IDA Pro or similar reverse engineering tools. Knowledge of x86 and/or MIPS assembly is a plus. Shut up and take my money! GitHub: Main repository: https://github.com/TOMB5/TOMB5 Binaries: None Screenshots: click here Last edited by zdimension; 20-02-19 at 17:28.

18-09-17, 14:16	#8
HD86 Historian Join Date: Apr 2014 Location: US Posts: 260	The current TR game is what makes this community exists. If the source code becomes available, we will have a plethora of new TR games as many people will try to build their own versions of the game. The community will splinter because many will be playing different games. Anyway, I am of course not against upgrading the game so that it remains compatible with new systems. Technical improvements are welcome as long as they don't change the gameplay. Last edited by HD86; 18-09-17 at 15:09.

18-09-17, 14:19	#9
Zebra Legend Join Date: Jun 2007 Location: Germany Posts: 26,649	^How is that differernt to what's been happening with the official games for years? People have long been discussing whether LAU or the reboot games (or AoD, for that matter) can be called Tomb Raider so that's really nothing new and I don't think it's much of a problem. People will just keep using their own personal definitions of what is and isn't Tomb Raider and move on with their lives . This project is awesome. I hope you'll be successful in recreating the source code ! __________________ Wozu zuhören, wenn man reden kann?

18-09-17, 00:18	#2
HD86 Historian Join Date: Apr 2014 Location: US Posts: 260	I am going to say my opinion and I know many people will not agree. If you want to mod the game extensively, you should just build a new game or play one of the recent Tomb Raider releases. I personally like the classic Tomb Raider game as it is and I don't want to see it modified extensively, because that will most likely ruin it. It will become a different game. I can't imagine that you are going to be modest in modifying it after you manage to decompile it.

18-09-17, 01:44	#3
Boobandie Professor Join Date: Jul 2012 Posts: 4,268	Why rag on the project? No matter what we do you can play Chronicles as is. This is more for the benefit of fan games and a historical record than it is for just modifying TR5. Though once it is decompiled it could lead to mobile and IOS ports of TR5, or a linux version/better functionality on new operating systems. Personally I can't wait to see where this goes, the TR5 engine is a lot better than TR4 and I've always been annoyed that it wasn't the version included as the Level Editor.

18-09-17, 10:14	#7
Lwmte Archaeologist Join Date: Aug 2010 Posts: 1,799	I can only welcome and applause for this effort. And I can't understand people who possibly say that they are happy with current state of the game, and this is why: 1. As already said by Caesum, TR5 is still Win9x application. It's only a matter of time before we can't use it with modern hardware. What's the point of having broken binary which only works through virtual machine or with quirky workarounds like compatibility modes or 16-bit color depth (say hi to winroomedit)? 2. TR5 is the less known engine in old-school TR engine family. File formats are less known, internal processing is less known, lots of stuff altered, replaced, lots of workarounds were written, and so on. Decompiling it will clarify many things that are currently vaguely known. 3. As we already have TR1 and TR2 decompilation projects in progress, having TR5 decompilation is like pinning down whole engine family from both sides - that is, if you have decompiled TR1 or TR2 and decompiled TR5, it means that you (sort of) decompiled TR3 and TR4 as well Also, as said by Caesum, it will be a great benefit for developing OT/OL engines. 4. Don't forget about some stupid TR5 bugs, especially in final section. You will finally get the chance to fix them. All in all, decompiling original engine always foreshadow great changes. Updated compatibility, graphics, ability to fork "vanilla" engine into new one, etc. etc. But decompiling is the hardest thing, it's really easier to write engine from scratch than decompiling existing one. Fortunately, from what I see, progress is quite fast. So my best wishes to the team!