www.tombraiderforums.com

www.tombraiderforums.com (https://www.tombraiderforums.com/index.php)
-   Software Development (https://www.tombraiderforums.com/forumdisplay.php?f=55)
-   -   Tomb Raider 5 Decompilation Project (https://www.tombraiderforums.com/showthread.php?t=218674)

zdimension 17-09-17 21:58

Tomb Raider 5 Decompilation Project
 
Tomb Raider 5 Decompilation Project

TL;DR AT THE END

The goal of this project is to decompile entirely Tomb Raider 5 to C/C++ code. There are many reasons for that; first, having the source code (or a functionally equivalent code) of the game allows modding it much more extensively than what the existing patches (TREP, TRNG, etc.) can do, because where those patches will inject code, and patch parts of the memory using hacks and workarounds, here, we can simply modify the code and recompile. Second, it is almost certain that Square Enix / Crystal Dynamics are not in possession of the code. We know that the source code of TR1 and TR2 (used to make the mobile versions) have been given to SE by former Core Design employees, but we are almost sure that the source code of the other games have been lost. This project allows us to preserve a part of history that would be lost otherwise.

How?

A lot of information required to decompile the game comes from the symbol files contained in the leaked TR4-5 PSX SDK. A symbol file is a file contains a list of memory address, and the name of "what is at that address". When you load the executable binary file of the game, you only get the raw binary code, without any function or variable name. A symbol file allows us to put back the original function and variable names, hence making the reverse engineering process much easier.
The symbol files contained in the leaked SDK are of a special format that contains much more information than simply the names, and thanks to Steffen Ohrendorf (OpenTomb & EdisonEngine developer) we have been able to get even more than names: function definitions, variable types and even the original file names.
Thus, our current source tree is very similar (for the PSX version, it's identical) to the original one.

Source tree

The repository is divided in multiple folders (exactly like the original source code):
  • GAME = main game code, that is common between platforms
  • SPEC_* = platform-specific code
    • SPEC_PC = PC version
    • SPEC_PSX = PSX version (requires the Psy-Q SDK and toolchain)
    • SPEC_PSXPC = PSX version that runs on PC (used for debugging purposes)
    • (maybe one day) SPEC_CE = Dreamcast

Current state of the project

As of today (2017-09-17), all three trees compile without errors. The PSX and PSXPC run up to the main game loop, where they just loop and do nothing.
The PSX version can currently go up to the first loading screen (with the spinning CD icon), but a lot of work still needs to be done since on PSX, all the loading code is located in a separate file that is dynamically injected at runtime.
The PC version can currently go up to the main game thread and the level loading part (title level) but the level loading code is not finished yet, so the game crashes a little bit, unless you comment out the main thread code, then the game just loops indefinitely like its PSX counterpart.

Who?
  • Gh0stBlade: main developer, now works mainly on PSX version
  • zdimension: main developer of PC version

Legal stuff

All code here comes from decompilation of publicly available (this includes the leaked SDK) binaries and ressources of Tomb Raider 5. Additional informations about the game's internal mechanisms come from the TRosettaStone 3 project. Square Enix plz don't sue us. We're basically retrieveing the game's source for free whereas your developers would want money.

TL;DR

We're decompiling TR5 so that you can mod it more easily (and that we, developers, can understand it better and make better tools).
It is not finished at all.
There is still a lot of work to do.
It may take months, it may take years.

Want to help?

Currently we're searching for people that know C and C++, and that can use IDA Pro or similar reverse engineering tools. Knowledge of x86 and/or MIPS assembly is a plus.

Shut up and take my money!

GitHub:
Binaries:
None

Screenshots:
click here

HD86 18-09-17 00:18

I am going to say my opinion and I know many people will not agree. If you want to mod the game extensively, you should just build a new game or play one of the recent Tomb Raider releases.

I personally like the classic Tomb Raider game as it is and I don't want to see it modified extensively, because that will most likely ruin it. It will become a different game. I can't imagine that you are going to be modest in modifying it after you manage to decompile it.

Boobandie 18-09-17 01:44

Why rag on the project? No matter what we do you can play Chronicles as is. This is more for the benefit of fan games and a historical record than it is for just modifying TR5. Though once it is decompiled it could lead to mobile and IOS ports of TR5, or a linux version/better functionality on new operating systems.

Personally I can't wait to see where this goes, the TR5 engine is a lot better than TR4 and I've always been annoyed that it wasn't the version included as the Level Editor.

A_De 18-09-17 06:01

Quote:

Originally Posted by HD86 (Post 7779869)
I am going to say my opinion and I know many people will not agree. If you want to mod the game extensively, you should just build a new game or play one of the recent Tomb Raider releases.

I personally like the classic Tomb Raider game as it is and I don't want to see it modified extensively, because that will most likely ruin it. It will become a different game. I can't imagine that you are going to be modest in modifying it after you manage to decompile it.

You were right :D
Just imagine now, how much the game will be ruined by TRNG's plugins, for example :vlol:

To be serious: my best wishes to the decompiling team. I really want to see the "different game" some day :)

Caesum 18-09-17 07:41

Quote:

Originally Posted by HD86 (Post 7779869)
If you want to mod the game extensively, you should just build a new game or play one of the recent Tomb Raider releases.

That is kinda silly tbh. Using that logic we should suggest barracuda415 to stop and remove his TombATI project, because if one wants to mod the game he can just play TR Anniversary, no? I mean it doesn't even make sense.

Recompiling an engine gives a variety of new possibilities. It's not only about modding the existing game, it's making it possible to create an entire new one based on the source of TR5, or modifying TRLE in order to give it the features of TR5 engine. TR5 has been heavily modified, especially graphically. The main difference is that integers were replaced by floating points, making the entire game less shaky and much better looking on PC version.

Even your beloved TR5 (and mine btw) can benefit greatly if we can manage to reclaim the source code. TR5 is a Windows 98 game which doesn't work well on modern hardware. Heck, at this point TR1 with TombATI is more compatible than TR2-5. The source gives the possibility to modify the executable in order to make it work on modern PCs. No dirty hacks needed, no wrappers, no emulation. Modifying the game can also make it possible to restore PSX-exclusive effects like echo, footprints, dynamic Dreamcast shadows, special effects. It can also allow rewriting the rendering engine to something more modern than DirectX 5 and thus more compatible.

Also, understanding the engine can make it possible to modify enough to be able to make new adventures in it. Maybe even one would be able to replace the old and quirky TRLE engine with it. The source makes it possible to add new features without dirty hacks that do more harm than good (TRNG). We already have the official Room Editor for TR5 along with several tools and PSX devkit.

At last, decompiling the source will give new information for programmers to improve and extend their custom engines, eg. OpenTOMB and OpenLARA.

Dustie 18-09-17 07:43

Quote:

Originally Posted by HD86 (Post 7779869)
I am going to say my opinion and I know many people will not agree. If you want to mod the game extensively, you should just build a new game or play one of the recent Tomb Raider releases.

I personally like the classic Tomb Raider game as it is and I don't want to see it modified extensively, because that will most likely ruin it. It will become a different game. I can't imagine that you are going to be modest in modifying it after you manage to decompile it.

You are right about building a new game, some of the builders would definitely love to see a completely new engine at their disposal and with a decompiled code the path to making it happen is pretty much open. A lot the ideas we have oftentimes make us conclude that we might as well make ourselves a new game if we want to make them happen.

But I think the core (sic!) purpose here is to be able to patch certain things up, increase compatibility with newer systems and possibly introduce certain graphical tweaks and extensions, which would add up to the classic experience, not ruin it. I'm convinced such efforts come from passion to the classic Tomb Raider, not the hate for it.

Lwmte 18-09-17 10:14

I can only welcome and applause for this effort. And I can't understand people who possibly say that they are happy with current state of the game, and this is why:

1. As already said by Caesum, TR5 is still Win9x application. It's only a matter of time before we can't use it with modern hardware. What's the point of having broken binary which only works through virtual machine or with quirky workarounds like compatibility modes or 16-bit color depth (say hi to winroomedit)?

2. TR5 is the less known engine in old-school TR engine family. File formats are less known, internal processing is less known, lots of stuff altered, replaced, lots of workarounds were written, and so on. Decompiling it will clarify many things that are currently vaguely known.

3. As we already have TR1 and TR2 decompilation projects in progress, having TR5 decompilation is like pinning down whole engine family from both sides - that is, if you have decompiled TR1 or TR2 and decompiled TR5, it means that you (sort of) decompiled TR3 and TR4 as well :D Also, as said by Caesum, it will be a great benefit for developing OT/OL engines.

4. Don't forget about some stupid TR5 bugs, especially in final section. You will finally get the chance to fix them.

All in all, decompiling original engine always foreshadow great changes. Updated compatibility, graphics, ability to fork "vanilla" engine into new one, etc. etc. But decompiling is the hardest thing, it's really easier to write engine from scratch than decompiling existing one. Fortunately, from what I see, progress is quite fast. So my best wishes to the team! :tmb:

HD86 18-09-17 14:16

The current TR game is what makes this community exists. If the source code becomes available, we will have a plethora of new TR games as many people will try to build their own versions of the game. The community will splinter because many will be playing different games.

Anyway, I am of course not against upgrading the game so that it remains compatible with new systems. Technical improvements are welcome as long as they don't change the gameplay.

Zebra 18-09-17 14:19

^How is that differernt to what's been happening with the official games for years? People have long been discussing whether LAU or the reboot games (or AoD, for that matter) can be called Tomb Raider so that's really nothing new and I don't think it's much of a problem. People will just keep using their own personal definitions of what is and isn't Tomb Raider and move on with their lives :p.

This project is awesome. I hope you'll be successful in recreating the source code :tmb:!

DunHill 18-09-17 16:28

This project is amazing, I agree with many people here. It could replace the TR4 engine with many new features.
I'm so delighted to see so many people working on the modding community. With the new engines, the new Tomb Editor... All of this will bring so much more possibilities to the TRLE world :cln:

Quote:

Originally Posted by HD86 (Post 7779944)
The current TR game is what makes this community exists. If the source code becomes available, we will have a plethora of new TR games as many people will try to build their own versions of the game. The community will splinter because many will be playing different games.

Anyway, I am of course not against upgrading the game so that it remains compatible with new systems. Technical improvements are welcome as long as they don't change the gameplay.

Tomb Raider 5 was built on the same old grid system, it wouldn't make sense to change the gameplay to something modern :confused:
I think that the main goal here is to keep the same game, just more compatible with modern hardware :p
Though, If the code becomes available and we are able to add graphical features and gameplay changes, I think we could find a balance between old and modern. Let me explain myself :
We have to keep the easy way of building thanks to the grid system, for people who don't want to learn c++ or game design to build on newer engines such as Unreal engine. Plus, it will be even easier for people to edit wads and objects with the new editor.
BUT we can find a way to make the gameplay more fluid while keeping it adapted to the grid system.
And Core Design had the right idea when building Anniversary Edition : they imported TR1 maps, modified them graphically, but kept the grid-based system for the jumps and so on while having way more responsive controls. :D

Alex Fly 18-09-17 18:50

Good continuation with your project! I'll be keeping an eye on this one, that's for sure. :tmb:

zdimension 18-09-17 20:12

Added screenshots to main post.

Caesum 18-09-17 21:22

Quote:

Originally Posted by HD86 (Post 7779944)
The current TR game is what makes this community exists. If the source code becomes available, we will have a plethora of new TR games as many people will try to build their own versions of the game. The community will splinter because many will be playing different games.

Hacks, wrappers and decompilers have been there at least since the year 2000. They helped greatly in fixing bugs of the original games and improving/enhancing what we have now.

Infact, the modding community is THE reason why the classics are so active 20 years after the first TR release. Look at the games that don't have any modding community. Most of them die after two/three years when the next installment is released. Now look at the games that can produce mods. They are being constantly improved by people, they can get new stories, new graphics, better optimization et cetera. Gothic 1 & 2 received unofficial DX11 update which enhanced gameplay with new graphics effects, LOD et cetera. Service Pack fixed the bugs of the original game and allowed it to work on modern hardware while maintaining the vanilla look and feel of the game.

Tomb Raider Level Editor has greatly lengthened the life span of the original series, since it gives the possibility to play a game that can be more classic than anything we received post-TRC.

Getting the source code can only benefit the game. This will allow it to be fixed, patched, improved and enhanced. If anything, only the modding community will be divided, but frankly, it already is. There are people who use Room Editor, who use TRNG, or Dxtre3D. Finally there are guys who work on their own modifications to the engine or completely new ones.

Also, no matter how much one can modify the original TR2 engine or levels, your CD disc stays the same and so does the game within. The idea of "splitting the community" is an illusion. Nothing like that has happened over the years even though there's been plenty of occasions to do so. The legacy/vanilla Tomb Raider is in no danger, but it can only change for the better. And optionally, not forced.

zdimension 19-09-17 19:20

https://i.imgur.com/TT40lfw.png

Nightshades 21-09-17 15:26

Hey it's been a while, nice to see the progress, you guys really love tomb raider ! i'll take a look, thanks.

Gancian 21-09-17 18:17

Just wanted you to know that I love and support all of this work. Thanks for your dedication <3

Alain19 01-10-17 00:08

Great work!
 
I just discovered what you have done ( when I was doing something else) and I am flabbergasted by the amount of work done ( including some for the PC!:D) and I will follow your progress closely! In the meantime I will try to follow decompliling a little more so later I could be able to help.;)
congrats!:hug:

DunHill 23-01-18 16:53

I was looking a bit at he github of this project, I could see some updates from time to time and was wondering how well this was going ? :)

Gh0stBlade 26-01-19 01:09

Progress on PSXPC_N (PSX emulated on PC) :cln:

Video

Woops 26-01-19 09:40

Nice work :D

Nicklander 26-01-19 11:32

What’s the difference between this project and Monty’s TR5Main ?
Couldn’t you simply help Monty ?

Gh0stBlade 26-01-19 13:07

Quote:

Originally Posted by Nicklander (Post 8034356)
What’s the difference between this project and Monty’s TR5Main ?
Couldn’t you simply help Monty instead of working on yet another engine ?

There are many differences between tomb5 main and tomb5 decompile. Our project is not using injections into the main game binary like tr5main does. Tr5main is more focused on renderer enhancements for now.

This isn't another engine. It's the same original game engine Core Design used with zero modifications for now.

I'm not going to scrap my project after working almost 3 years on it for the lights of something else.

Nice try.

Dustie 26-01-19 17:29

Decompiling Tomb Raider 5 will help in developing TR5Main, and in the future should allow for making a standalone engine with the enhancements of TR5Main.

Nicklander 26-01-19 17:32

Quote:

Originally Posted by Gh0stBlade (Post 8034368)
There are many differences between tomb5 main and tomb5 decompile. Our project is not using injections into the main game binary like tr5main does. Tr5main is more focused on renderer enhancements for now.

This isn't another engine. It's the same original game engine Core Design used with zero modifications for now.

I'm not going to scrap my project after working almost 3 years on it for the lights of something else.

Nice try.

Yoyoyo chill. I didn't want to be rude. I apologize. :hug:

Ok so as far as I understood it, the most relevant ones are:
- TR5Main is a hack similar to TRNG but focussing on its TombEditor (will eventually become an engine some day)
- TR5 Decompilation Project tries to obtain the original TR5 engine code or at least smth very close
- OpenLara is a (completely?) new engine
- OpenTomb/TRE is also a (completely?) new engine, but kinda dead (?)

It's awesome that you are so dedicated to this project, but you need to understand that I and many other users on this forum are confused by the amount of engine-related projects going on here. This is mainly due to a lack of explaination and transparency, I really don't feel like going thru hundrets of pages just to understand what it's actually about and what progress has been made so far.

Also it's quite frustrating if you ask me, I'm following some of the development projects for years now and it seems like the teams working on them are splitting more and more. From the perspective of a level builder there's only focus on one point: An open-source engine that allows modifications and the implementation of new features. That's basically the end goal of all the 5 projects listed above. I understand that you all have different approaches to reach that goal but you may agree that it's not very effective working in such small groups. Things could be moving way faster if all the effort was put in a single project. There are countless examples of abandoned projects in our community, so far only the hard work put in Monty's Editor has actually paid off. Trust me, nobody here wants even more abandoned projects. :(

Anyways, it's really cool that you are so dedicated to the TR community. I just wanted to express my frustration about this whole situation and hope that we'll see results some day.

EDIT: Ok, I might have got this a bit wrong. It's nice to see that there's a Discord server uniting all of those projects. :)

Gh0stBlade 26-01-19 18:29

Quote:

Originally Posted by Nicklander (Post 8034424)
Yoyoyo chill. I didn't want to be rude. I apologize. :hug:

Ok so as far as I understood it, the most relevant ones are:
- TR5Main is a hack similar to TRNG but focussing on its TombEditor (will eventually become an engine some day)
- TR5 Decompilation Project tries to obtain the original TR5 engine code or at least smth very close
- OpenLara is a (completely?) new engine
- OpenTomb/TRE is also a (completely?) new engine, but kinda dead (?)

It's awesome that you are so dedicated to this project, but you need to understand that I and many other users on this forum are confused by the amount of engine-related projects going on here. This is mainly due to a lack of explaination and transparency, I really don't feel like going thru hundrets of pages just to understand what it's actually about and what progress has been made so far.

Also it's quite frustrating if you ask me, I'm following some of the development projects for years now and it seems like the teams working on them are splitting more and more. From the perspective of a level builder there's only focus on one point: An open-source engine that allows modifications and the implementation of new features. That's basically the end goal of all the 5 projects listed above. I understand that you all have different approaches to reach that goal but you may agree that it's not very effective working in such small groups. Things could be moving way faster if all the effort was put in a single project. There are countless examples of abandoned projects in our community, so far only the hard work put in Monty's Editor has actually paid off. Trust me, nobody here wants even more abandoned projects. :(

Anyways, it's really cool that you are so dedicated to the TR community. I just wanted to express my frustration about this whole situation and hope that we'll see results some day.

EDIT: Ok, I might have got this a bit wrong. It's nice to see that there's a Discord server uniting all of those projects. :)

I think the bigger problem is, the Tomb Raider community lacks programmers so it's hard to form a solid team of programmers in the first place. Unifying everyone into one single team won't work, we tried that with OpenTomb and everyone eventually got bored, upset and moved onto their own projects. One person might like one programming language, style or library over another and it's just argument after argument and everyone leaves.

I'm not saying it's better to work alone, but sometimes it avoids a big mess and fall outs. I would personally love to see more programmers and reverse engineers working on my project but sadly nobody but zdimension joined since the first commit in July 2017. We tried to advertise on other websites but it seems nobody is interested or people generally lack the skill required to work on this project.

Cheers.

Gh0stBlade 26-01-19 18:30

Quote:

Originally Posted by Gh0stBlade (Post 8034438)
I think the bigger problem is, the Tomb Raider community lacks programmers so it's hard to form a solid team of programmers in the first place. Unifying everyone into one single team won't work, we tried that with OpenTomb and everyone eventually got bored, upset and moved onto their own projects. One person might like one programming language, style or library over another and it's just argument after argument and everyone leaves.

I'm not saying it's better to work alone, but sometimes it avoids a big mess and fall outs. I would personally love to see more programmers and reverse engineers working on my project but sadly nobody but zdimension joined since the first commit in July 2017. We tried to advertise on other websites but it seems nobody is interested or people generally lack the skill required to work on this project.

Cheers.

Quote:

Originally Posted by Dustie (Post 8034421)
Decompiling Tomb Raider 5 will help in developing TR5Main, and in the future should allow for making a standalone engine with the enhancements of TR5Main.

Correct, both projects are sort of interchangeable. Only problem with TOMB5Main is that it doesn't follow the same file structure as the original game if i recall correctly so it's difficult to find certain pieces of code.

This project also runs on PCPSX, PSX and PC.

TheBlueFox 29-01-19 04:07

Quote:

Originally Posted by Gh0stBlade (Post 8034438)
I think the bigger problem is, the Tomb Raider community lacks programmers so it's hard to form a solid team of programmers in the first place. Unifying everyone into one single team won't work, we tried that with OpenTomb and everyone eventually got bored, upset and moved onto their own projects. One person might like one programming language, style or library over another and it's just argument after argument and everyone leaves.

I'm not saying it's better to work alone, but sometimes it avoids a big mess and fall outs. I would personally love to see more programmers and reverse engineers working on my project but sadly nobody but zdimension joined since the first commit in July 2017. We tried to advertise on other websites but it seems nobody is interested or people generally lack the skill required to work on this project.

Cheers.

It's a shame so few people working on a project that could greatly help the Tomb Raider community, but at the same time, understandable.
Wish you guys the best of luck with the project.

MontyTRC 02-02-19 18:58

TR5Main goes beyond TRNG because I am replacing all the API-dependent code with new code using new APIs. No more DirectSound, DirectX 6, etc but new and modern APIs.
99% of the issues with new Windows version is related to APIs.

Gh0stBlade 13-02-19 17:02

Latest progress is font rendering and title menu code added. :cln:
Video

Gh0stBlade 16-02-19 18:45

Latest build:
Video

Progress with the title screen menus, many bugs fixed also.

We are also looking for dedicated Game Testers. Let me know if you want to join the team!

laralives 16-02-19 21:07

Love seeing progress! :tmb:

Alex Fly 17-02-19 14:46

I agree, it's impressive to follow your progress. :tmb:

Gh0stBlade 18-02-19 19:25

Come on, we need everyone and their mothers to make this project possible!

We're still looking for QA Testers (we have none). :)

DunHill 19-02-19 07:01

That's some really impressive work here :tmb: It's amazing to see the progress

While I would love to help on such projects, I have only gained a bit of experience in Java and C programming during my 2 years IT course, probably not enough to do good here :(

Gh0stBlade 21-02-19 18:33

Video

PSX version progress.
Sound effects work too!

Still looking for dedicated QA testers, PM me if you're genuinely interested in joining the team as a tester.

FantaseaII 21-02-19 19:14

Nice work !

Gh0stBlade 15-04-19 22:37

Bump. Final call... still looking for testers.

VictorXD 16-04-19 03:01

I wish I could commit to it, but I don't know if I'll have the spare time :o What exactly are you looking for in testing??

Dan-d 16-04-19 19:10

Quote:

Originally Posted by Gh0stBlade (Post 8074547)
Bump. Final call... still looking for testers.

What has to be tested exactly Gh0stBlade?

Gh0stBlade 16-04-19 21:05

Quote:

Originally Posted by Dan-d (Post 8075043)
What has to be tested exactly Gh0stBlade?

Currently, title screen menu must be tested for bugs.


All times are GMT. The time now is 01:28.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.