The tomb5 project

[PedroTheGamer]

wow theses edits are really suberb! now real chronicles are being released by parts

[Sardoc]

Originally Posted by Woops

• Red Alert savegame bugs are officially history

Wow, I'm really impressed! Will download this asap.

So, what was wrong there? Some variable not getting saved?

[Jathom95]

^ Yeah, after all of these years I'm curious about what the mess behind Red Alert actually was. Care to enlighten us Troye lol?

[Woops]

I wish the reason is a bit easier to explain but I will try my best

so, a little bit of background
- Flipmaps have numbers to them, there is, for example, flipmap 0, flipmap 1, and so on, TRLE builders are familiar with this.
-Internally, the game keeps an array of numbers (like, literally a list of numbers) that flag whether a flipmap is on or not
1 if it's on, 0 if it's not on. It looks something like this
1 0 1 1 0 ....
^ in this example, flipmap 0 is active, flipmap 1 is not, flipmaps 2 and 3 are active, 4 is not, and so on!
in TR4, this array is only able to store 10 numbers, making the available flipmaps in tr4 from 0 to 9!
In TR5, it seems they decided to increase this from 10 to 255! (and this is what indirectly caused the problem)
The system works perfectly fine even with 255 flips available, the problem lies within the saving system.
In memory, each of these numbers in the array occupied 4 bytes of memory, which is okay for runtime memory on PC. (PSX has 10!)

So, let's talk about the saving process
as it is with everything TR, saving memory was Core Design's top priority, which is understandable, and they have very smart ways of doing so!
especially savegames, saving the entire game memory would cost a lot, especially for PS1 saves with more limited space on memory cards, as a result, saves format is very compact, and only saves what is absolutely necessary.

So, it's time to save the flipmaps array. instead of saving the entire array and use 255*4 bytes for it, they instead saved it in a 2 byte memory space, using bits. Every byte contains 8 bits, bits can either be on or off, 1 or 0. which is more than enough for our purpose.
So, using 2 bytes to save flipmaps: we get 16 bits to save the status of flipmaps in, 1 bit each.

The code to do so looks like this:

for (int i = 0; i < 255; i++)
	{
		if (flip_stats[i])
			word |= (1 << i);
	}

so, let's break this down!
flip_stats is the array of flipmaps we talked about^
word is the 2byte space of memory we are saving the array in.
i is a simple counter.

this is a for loop, this is how they work:
First thing, the "for" line, has 3 parts, separated with a semicolon ;

part 1 is only executed ONCE at the beginning of the loop. for us, it sets i to 0

Second part is the condition the loop will keep running under, once the condition is not true, the loop stops going and code execution continues.

Third and last part is what to do after the body of the loop has been executed, and we need to execute it again.

so, let's take this loop for example

first we set i to 0,
then we go and execute the body

if (flip_stats[i])
			word |= (1 << i);

then we do i++, this just adds 1 to i.
So, first executed body would look like this:

if (flip_stats[0])
			word |= (1 << 0);

then we add 1 to i

if (flip_stats[1])
			word |= (1 << 1);

and once again

if (flip_stats[2])
			word |= (1 << 2);

All the way until we get to 254, since it is the last number that is less the 255.

now, let's break down the body!

simply put into English: If flip_stats[i] is not 0, then set the corresponding bit to 1.

so, let's say the first 5 flip_stats looks like this:
1 0 1 1 0

then word, the 2byte space we are using, will look something like this:
10110 00000000000
Notice, 16 bits total, and 5 first bits are set correctly! (remember, TR4 still had 5 more bits to go, 10 total, TR5 had 250 more bits to go, for a total of 255. You can start to see where the problem is :P)

Anyway, we access bits using something called bitshifts.
Bitshifts do exactly what they suggest: they shift bits left and right a certain amount of times
bitshifts are the << or the >> symbols. << are left shifts, >> are right shifts.
and since bits in computers are represented by powers of 2, we are able to represent bitshifts using simple math as follows:

for left shifts:
number << other_num = number * (2^other_num)

and right shifts:
number >> other_num = number / (2^other_num)

Back to TR code: notice how the number we shift is always 1, and it's a left shift. 1 is used so we can determine which bit we want to write to.
Let' see it in practice:
1 << 0 = 1: one is simple

1

in binary, which is the first bit
1 << 1 = 2: two is

10

in binary, which is the second bit
1 << 2 = 4: four is

100

in binary, which is the third bit
1 << 3 = 8, etc.
you see how it goes over each bit, as we increase i, our counter.

Now, where the problem is. you may wonder, since we are limited to 2 bytes, 16 bits, what happens when i is more than 15? the answer is: nothing. for example when trying to write to bit 25, it is out of our memory limit, so it is ignored.
Now here is the bug: when shifting 1byte, 2byte and 4byte spaces, bitshifts are masked to 5 bits, meaning the number of bits it will attempt to change is 5^2, which is 32, (from bit 0... all the way to bit 31, in 4byte spaces) and after that, it will return and write to bit 0!

so
Let's go back and see how our loop behaves in this case
from i 0 to i 15, everything is fine. it writes the first 16 bits, from bit 0 to bit 15
Now 16, it tries to write to bit 16, but it is out of range, so doesn't do so.
all the way to 31, it tries to write to bit 31, out of range, so doesn't do so.
Then we get to bit 32, the CPU switches back to write to bit 0, and so on. Now you may ask, since writing is blocked by the if line, if the flipmap is active: so did Core have flipmaps > #15 activated in Red Alert? The answer is no, they did not.

the first time around, flipmaps are saved properly.
Now when it comes to reloading, the game writes to flip_stats directly, in a similar for loop.
it reads each bit in the 2byte space, then writes it to flip_stats as is.
As is with saving, the first 15 bits and reloaded fine, then from 16 to 31 they are ignored. Then when we get to 32, same with writing, CPU goes back to reading the first bit of our 2byte space, but writes it to flip# 32. so if, for example, our 2byte space looked like
10110 00000000000
then, when reloading, flip_stats will look like:

10110000000000000000000000000001011000000000000000 0000000000001011
and so on, you can see higher numbers of flip_stats are getting corrupt.

Now go back to saving again, our 2byte space itself will corrupt, then reload again, and the flip_stats array will be corrupt, and so on.

that is it
Remember this only happens because Core tried to overflow their 2byte memory space with 255 flipmaps in TR4 it was just 10 fllipmaps, so this bug does not happen.
To fix it, I simply limited saving and reloading flips to 16, so game now only saves and reloads 16 flips.

Sorry the explanation is very lengthy, I tried to be as elaborate and explained step by step for people who don't necessarily know programming.

TL;DR: Core tried to save 255 flipmaps in memory space that only fits 16, which causes corruption
moral of the story: don't try to stuff 255 bits into 16, you'll run into unforeseen consequences

(edit: wow this took me almost an hour to write )

[ANoDE]

Thanks for the very detailed explanation!

I wonder why they prepared the game code to support this many flipmaps, when they're only using 16 anyway.

[LateRaider]

This is the easiest to use TR mod I have ever seen. You're a deity, Troye.

[Sardoc]

Oh I see! Ok, it makes sense. So, even with a huge limit like that, they de facto never used more than 10 flip maps? Or did they reach the 16 after all?

I wonder to what extent the 255 limit was intentional.

Seems like either it was intentional from early on, just not tested enough (or at all), or maybe it was some kind of last-minute change (and as a consequence, again not tested enough or at all).

I'm leaning towards last-minute change... I mean, for a game that no one wanted to develop, Chronicles wasn't THAT buggy, with the exception of Red Alert of course.

[Woops]

To me it looks like an untested last minute change.
This is all speculation: but, keeping in mind that VCI level files are called "richx.trc", it suggests that Richard Flower built the levels, which, Richard was also a programmer and working on the TR engine
source

so, it might have been him, but doesn't explain why PSX is not touched, and still processes 10 flips normally. who knows!

[.snake.]

These changes are absolutely phenomenal. Thank you so much for making such a wonderful and user-friendly mod and fix to Chronicles.

Originally Posted by Woops

To me it looks like an untested last minute change.
This is all speculation: but, keeping in mind that VCI level files are called "richx.trc", it suggests that Richard Flower built the levels, which, Richard was also a programmer and working on the TR engine
source

so, it might have been him, but doesn't explain why PSX is not touched, and still processes 10 flips normally. who knows!

It's actually Richard Morton who built the VCI levels though.

But I guess Richard Flower helped with programming in all of the game.

[SrDoggo]

Originally Posted by Woops

• Red Alert savegame bugs are officially history

Hi, I made this account just to thank you and your team for all of this. I played Chronicles like 3 months ago and so, and I must say it's one of my favorites, but I had to drop it in Red Alert because the game kept corrupting my saves.

Now Chronicles with all of this improvements and fixes will feel like a more complete experience.

So thank you very much for everything!