Get ready for a TRL / TRA level editor!

[deactivated]

Originally Posted by Michiel

That is completely off-topic, I think we have private messages for this.



Are you sure about this? I made a small tool to quickly test all files and no file was found that was larger then the bigfile it was in. The last file in bigfile.001 correctly ends within bigfile.001. I don't think there is such thing as files being in 2 bigfiles at once.

The test tool I used:


All 3 extracted seem to have the same structure when you view them in a hex editor, so it seems they are all the correct files that have been extracted.

WOW

Where can I get this 'BigTool ?
Well, I'm as sure as can be. Unless the size thing is misdirection.

But then, the end of bigfile.001 does not have a buffer and there is that chunk of stray data at the start of bigfile.002 that does have an ending buffer, so....and the next offset is correct.

I would have to examine the whole set in detail. If there are other cross-overs, there might be a reason for it.

I think these unidentified mega chunks are level data, one per. And they seem to include the Lara meshes (maybe) which might be why the bikini does not work in any level but the manor as it might be included only in the manor.
Seems wastefull to me...

(Idea: look for the bikini mesh and identify the manor level...)

You may have extracted sub-data blocks from those mega-chunks and they might not cross the file boundaries.

Where did you get those filenames (in the pic) ? Do you have a full list yet ?

Don't forget to have fun....

[Michiel]

There is a list of filenames somewhere I think. But I just guessed most of them. I knew the filename of winston, so I tried out the rest. The tool works in TRA demo as well, I managed to correctly guess some filenames. I tried lara.drm, trex.drm, raptor.drm and bear.drm and they were found inside the bigfiles. Since they simply use pc-w\(charactername).drm it's easy to find these files, even without the list.

About the drm files: I don't have the full format yet, but the beginning looks like this:

int unknown; // always 0x0E it seems
int numdata;

After this a 20-byte structure is found, the numdata value tells how many of those structures there are. After these structures, there is some unknown data, after that mostly there are textures. But I haven't looked at that data yet.

[deactivated]

Originally Posted by Michiel

There is a list of filenames somewhere I think. But I just guessed most of them.

Indeed.
I wonder how GodModder is doing on this...

About the drm files: I don't have the full format yet...[/QUOTE]


A reverse way of detecting which fileformat it is would be to go to http://www.wotsit.org/default.asp
and download all 3D format and try to match the values to the structure described. Flatfoot all the way through...

[deactivated]

I think that the typeless data blocks do make up one mega block of data that has to be extracted to one file.

However, in the header entries, some of those data blocks have a specified lenght that is smaller than the fully filled block.
So what I think is that the space beyond the lenght is garbage, just some repeated data that looks legit.
But to export as one file, all legit data have to be copied one after the other into that exported file.

Now which point of view is correct ?
I guess I'll have to do both and analyse.

[deactivated]

Big file format: I just found out that the data block labeled 'ENIC" are not listed in the bigfile.000 header.

Only the WAR and the undefined-type blocks are.

Maybe they are listed somewhere else (maybe in the undefined-type data blocks ?) called when a trigger in the level calls for them to be loaded.
If that is the case, finding them will give us both the trigger number and the place where it is called for, plus by looking at the dialogs texts (for the bottom captions) we could identify which undefined-type data block is for which level.

[godmodder]

Hello,

glad to see you guys working on identifying the file format too. I've tried reversing the .drm format, but haven't made any progress on it anymore: guess it's too hard for me.

I've read through all the posts here and I have a few comments:

- the left shift by 11 is the same as multiplying by 2048, so yes this is indeed correct for calculating the offset of the files in the bigfile archive
- there are no cross-over files in the bigfile archives. I can correctly find every file (.drm, .mul,...) with the corresponding offset. I guess the theory about the bigfiles acting like a FAT (or directory table) might be correct in that after the offset+length there is some garbage left to compensate for varying filesizes during development.
- The .drm in TRA is indeed identified by 0xOE, I've tracked this with ollydbg and it checks this everytime a model is loaded.
- The second DWORD of a .drm file is the number of files that are contained within the .drm file (textures, models, unknowns,...)
- As for the rest of the data, I honestly can't figure it out. I've sitten hours in front of my debugger looking how the data gets used, but I haven't made a breakthrough. The only data I can really identify is raw texture data with a PCDXT fileheader.
- As for the filenames, remember that I've written a filename->hashcode converter so you can check if a filename is valid. Unfortunately a hash algorithm doesn't work the other way around, but you can find all the names of the objects in the game in the file objectlist.txt. (strangely enough there's also an objectlist.dat !!!)

So to summarize: I think we've pretty much figured out the bigfile format, but deciphering the .drm format is going to be the most important to write an editor.

I hope you guys are better reversers than I am... But I'll keep trying
Jeroen

[deactivated]

Originally Posted by godmodder

Hello,

glad to see you guys working on identifying the file format too. I've tried reversing the .drm format, but haven't made any progress on it anymore: guess it's too hard for me.

Originally Posted by godmodder

I've read through all the posts here and I have a few comments:

- the left shift by 11 is the same as multiplying by 2048, so yes this is indeed correct for calculating the offset of the files in the bigfile archive

Noted.

Originally Posted by godmodder

- there are no cross-over files in the bigfile archives. I can correctly find every file (.drm, .mul,...) with the corresponding offset. I guess the theory about the bigfiles acting like a FAT (or directory table) might be correct in that after the offset+length there is some garbage left to compensate for varying filesizes during development.

Why did they not clean up ?
Anybody tried rewriting the bigfiles without that junk to see if they would still run ?

Originally Posted by godmodder

- The .drm in TRA is indeed identified by 0xOE, I've tracked this with ollydbg and it checks this everytime a model is loaded.
- The second DWORD of a .drm file is the number of files that are contained within the .drm file (textures, models, unknowns,...)
- As for the rest of the data, I honestly can't figure it out. I've sitten hours in front of my debugger looking how the data gets used, but I haven't made a breakthrough. The only data I can really identify is raw texture data with a PCDXT fileheader.
- As for the filenames, remember that I've written a filename->hashcode converter so you can check if a filename is valid. Unfortunately a hash algorithm doesn't work the other way around, but you can find all the names of the objects in the game in the file objectlist.txt. (strangely enough there's also an objectlist.dat !!!)

Are those the same as contained in the trltools from darkfadder?
If not could you pretty please send them to me at bitshifter@sympatico.ca ?

Originally Posted by godmodder

So to summarize: I think we've pretty much figured out the bigfile format, but deciphering the .drm format is going to be the most important to write an editor.

Is there an online document summarizing all this information ?

What about the CINE format ?

[Michiel]

Originally Posted by deactivated

Big file format: I just found out that the data block labeled 'ENIC" are not listed in the bigfile.000 header.

Only the WAR and the undefined-type blocks are.

I think you didn't really understand what the bigfiles are. It's not a group of data blocks, it's more like a harddrive that stores files, using a hash table for quick searching. The file table is sorted using the hash key, the file with lowest hash code comes first. This way you can do a quicksearch for the file you want, which speeds up things when you have many files.

But what it actually contains are smaller files itself. There is no reason for those data blocks you find to be at the start of a file format. Many files have some kind of label at the first 4 bytes, but this is not needed at all. So it's possible that a file starts with data directly. And inside those files there might be internal pointers and different data blocks as well. That ENIC block is most proberly something that is inside another file, the bigfile only has pointers to the files, not to subblocks inside them.

[deactivated]

Originally Posted by Michiel

I think you didn't really understand what the bigfiles are. It's not a group of data blocks, it's more like a harddrive that stores files, using a hash table for quick searching. The file table is sorted using the hash key, the file with lowest hash code comes first. This way you can do a quicksearch for the file you want, which speeds up things when you have many files.

I sorted mine using the table that follows the hash table, sorted on the offsets, so as to go from one block/file to the next.
I wanted to spot holes and any TR4 style order there might be.

If it is sorted by hash, is the order that different ?

Originally Posted by Michiel

But what it actually contains are smaller files itself. There is no reason for those data blocks you find to be at the start of a file format. Many files have some kind of label at the first 4 bytes, but this is not needed at all. So it's possible that a file starts with data directly. And inside those files there might be internal pointers and different data blocks as well. That ENIC block is most proberly something that is inside another file, the bigfile only has pointers to the files, not to subblocks inside them.

That, I had gotten.

Bit it did strike me as kinda funny that WAR files would be listed and not ENIC.
In an earlier post, I did say that ENIC files must be called from inside the untyped blocks/files as the player tripped triggers. It was just unexpected.

The TR4 format was truly hierarchical in structure and could be figured by increment; this one is so much more engine dependent that we'll have to figure it all in in one shot.

BTW, I think ENIC blocks/files are like mini-levels with meshes but their animations are played out by the engine instead of by the player.
One reason for that is when the mesh texture is changed to something custom, the change carries through that cinematic.
That would mean, in its structure, one is likely to find repeat of some part of the level terrain meshes it is triggered from.

[aktrekker]

deactivated: If you remove that "junk" from the files, the 2048 alignment will be off. That alignment makes the files load much faster, and there may be engine code that depends on it.

As for the .drm format, since I don't have the game I can't help alot. But if it is an object file, you will probably have to look for the following information (you may be able to recognize patterns)

Array of vertices
Array of polygons as indices into the vertex array
Array of UV coordinates (may be contained in vertex array)
Array of animations (should be able to find some sort of pattern)

Note that there may only be a single mesh and no joints. Very unlike TR4 models.

Finding the vertices can be easy depending on the object you are examining. For example, if it is an actor type model (Lara, friend, enemy, etc), then the x and z coordinates will be fairly small, and y coordinates will be larger. Values may be floating point, but those are still pretty easy to recognize - the last byte of every 4 will probably be in the range 0x38 - 0x48.

The polygons will probably follow the vertices, but no guarantees. It should be pretty easy to find. Once you know how many vertices there are, the values in the polygon array will never be larger than that.

UV coordinates will also be floating point, and always in the range 0-1.


Since you are using a debugger, catching the calls to DirectX will allow you to get the exact vertex format being used. This will show what other information might be stored in the file.