Tomb Raider Forums  

Go Back   Tomb Raider Forums > Tomb Raider Series > Tomb Raider

Reply
 
Thread Tools
Old 24-12-22, 19:24   #1
Famicom
Member
 
Famicom's Avatar
 
Joined: Mar 2009
Posts: 466
Default Yet another set of upscaled FMV videos

Hey y'all.

There's already at least a couple of FMV upscaling projects that I know of. They've achieved pretty good results given the source material's resolution and framerate, but I always felt there were a few aspects that could be improved upon, such as:

- Extremely low detail, presence of halos and a bit over the top noise reduction.
- Picture a bit too punchy, contrasty and saturated for my taste.
- Aspect ratio (ah, the aspect ratio!). Source material is 320x208 (black margins on top and bottom included), which is roughly 4:2.6. Back then most TVs were still 4:3 so I assume both the picture got a bit stretched vertically to fill the 4:3 frame and for this reason FMVs don't look as flattened on consoles. Be that as it may, I can't remember anyone addressing this issue... ever. So I had to do something about it. Here's a couple of examples that prove aspect ratio's wrong in case someone's raised a sceptical eyebrow (please don't mind color depth and banding. It's just the gif files):




First off, download link is at the very bottom.

Now, I'm well aware of the usual issues and artifacts provoked by AI upscaling and frame interpolation. Even though I've worked hard to keep them to a minimum, a few of them still remain being their fixing beyond the amount of time I can devote to this matter at the moment, and/or beyond my technical knowledge and/or skill to fix them (E.G. Natla video stream on the laptop in the CAFE sequence, among others). Also, I've been very careful so as not to wipe out any remains of fine detail from the FMVs. The same applies to dynamic range. For this reason, I've kept noise reduction to a minimum and haven't fiddled with brightness, contrast, saturation etc. in any way to preserve as much picture information as possible. For the most part I think it works ok, with a few exceptions (E.G. the CANYON sequence may look a bit rough around the edges. I might just upscale it again with a little more aggresive noise reduction, and see if I can do something with some uncanny faces while I'm at it).

Text and fonts: I've combined 3 different fonts to recreate as faithfully as possible both titles from the CAFE sequence (LOS ALAMOS.../IMPERIAL HOTEL...). Every single letter has been meticulously and individually put into place and a tad of shadow added to mask as much of the original titles as possible. Fade in and out effect has also been carefully recreated to both keep faithful to the original and cover the original text. The result may not be perfect, but it's certainly the best I can do. In general, all the AI upscaling models I've tried are not particularly kind to any kind of texts. Magazine covers are no exception, so there's always some manual work to do after the upscaling to help it look better. Not perfect, though:


Also, to preserve as much quality as possible up until the final stage of the process, I've worked with png sequences and wave files all the way through. Audio remains intact in all the sequences but one: I've lengthen the sound in the "Escape video compression systems" video by adding some reverb so that it has time to fade out properly to avoid the abrupt cut from the original file.

Apart from all that, a big amount of Photoshop time has been dedicated to fix the warping artifacts introduced by the frame interpolation method (more on that later):


Now for the specifics of the process. This is not meant to be a comprehensive guide to explain every step of the process nor does it necessarily convey the best way of doing things, but I hope it may come in handy to those eager to try their luck and build upon what's been done so far:

1.- Download jPSXdec from https://github.com/m35/jpsxdec. Launch the app, load PSX's TR1 bin file. If you have a multi bin image, you may need an app to convert it back to a single bin+cue image. CDmage worked for me (https://www.videohelp.com/software/CDMage). Now browse the CD content until you find the FMV files and extract the one you want to a png sequence (320x208)
+ wav file (37800 Hz).

2.- Crop the black margins out of every png file. The resulting files will be 320x120. Create a Photoshop action to achieve this with no sweat. This is valid for all FMV sequences minus CORE logo and ESCAPE Video compression systems.

3.- Load your png sequence into Topaz Video AI (or any other upscaling software you consider best). Upscale height to 1080 keeping the aspect ratio. I recommend you use Proteus Fine-Tune/Enhance model. You might as well fine tune individual parameters from scratch or use automatic values as starting point ("Relative to Auto"). Export to png sequence. That'll give you 2880x1080 per frame.

4.- Back in Photoshop, squeeze every frame horizontally from 2880 to 2376 pixels. You might as well automate the process by creating a new Photoshop action. You'll end up with 2376x1080 files.

5.- Time to download FlowFrames. This software is some kind of GUI on steroids for Rife, DAIN and other AI interpolation models. Grab it here: https://nmkd.itch.io/flowframes. Install it, launch it and let it download the files it needs. Then, load your png sequence, specify the original framerate (15fps) and your target frametrate (30fps). DAIN has been the king up until recently, but not anymore (it takes ages, literally ages, to finish the job, it is extremely slow and the results, as far as I know, are barely any better than what Rife achieves); the best AI model as of today is RIFE. The lastest ree version of FlowFrames (v1.36) includes Rife 4.0. Select Rife-CUDA (if your vga supports it) + Rife 4.0. Go to output mode and select Image Sequence - PNG. Enter a low value on Fix Scenes Change - Sensitivity. A value of 10 is all right, but an even lower value will save you time later.when trying fo fix some warping artifacts in Photoshop. Fire up the Interpolate! button. A few minutes should be enough to finish the process thanks to Cuda.

6.- Inspect your resulting png sequence in search for interpolation glitches and artifacts. Depending on how the camera moves, their fixing in Photoshop will go from easy peasy to OMFGWTF. The previous and next frame will usually provide with good chuncks of image that can be copy-pasted, resized, perspective-cloned, rotoscoped... into your faulty frame. Also, Rife's not particularly good at guessing scene changes. Most of the time it the interpolated frame it will create between two different scenes will be an ugly piece of crap. In that case all you have to do is ditch that frame and replace it with your own: just take the previous (A) and following (B) frames and mix them in Photoshop. Put Frame A in the bottom layer and stack frame B on top, in another layer. Reduce B's layer oppacity to 50% and export the resulting image back to png. Make sure you name it like the interpolated frame you want to replace.

7.- Import your interpolated png sequence and wav file to your video editor (Premiere, Vegas...). Make sure you specify your png sequence's framerate (30) before dragging and dropping it into the project's timeline. When exporting to a video file, we'll make sure we'll select a bitrate low enough for Tomb1Main not to choke with the video file. This is a known bug (https://github.com/rr-/Tomb1Main/issues/262). If you want the final video file to have a FullHD effective resolution, you can set 1920 as horizontal resolution and let your editor calculate the vertical res. for you while keeping the aspect ratio. The settings that worked best for me in Premiere, are as follows:

* Video output: H264 MP4 1920x872, 30fps, VBR 2 pass, target 6,00 Mbps, Max 8,00 Mbps. Maximum Render Quality + Render at Maximum Depth should be selected.
* Audio output: AAC 320kbps 48kHz Stereo.

8.- Sometimes the workflow needs to be a bit different. Take for instance CAFE fmv: as I mentioned, some brand new titles had to be superimposed on the picture. Now, you could either interpolate first in FlowFrames and then add the titles in Premiere, or do it the other way around. I chose the latter, that is, add the titles in Premiere, export to png sequence, import in FlowFrames, interpolate, export back to png, import into Premiere (remember to change framerate to 30fps), add sound and finally export to video with the settings I mentioned before. The reason to do it this way being interpolation can be extremely glitchy when there's text involved, so the clearer, sharper and more defined the text, the better results you'll obtain. If you interpolate the pictures with the original blurry text, the result will be a wobbly mess that will produce all sorts of artifacts and garbage way beyond the limits of where the text is placed, so you'll have a hard time masking them with your new text later, unless you spend tons of time removing them in Photoshop first. If you want to save a bit of time, you can export directly to a 15fps video file when adding the titles in Premiere (in that case, remember to add the sound too), import the video file into FlowFrames, interpolate to 30fps and export it back to H264 MP4 with quality level 19 (default is 20). The lower the value, the higher resulting bitrate, but I experienced issues in Tomb1Main with compression levels lower than 19.

9.- In some special cases, like in the VISION sequence, there's a burst of 1 frame images (secs 0:04 to 0:06) that don't get on well with interpolation and framerate changes at all. In that case all I could do was duplicate frames so the original artistic vision (no pun intended) was preserved. Therefore those couple of seconds look like a 15 fps video even though the whole video's at 30fps, and that's because there are effectively 15 unique frames every second.








Some final considerations:

YouTube compression has completely butchered my videos. Rest assured the video files you're about to download don't have those huge blocky compression artifacts, even though they don't have the highest bitrate possible for the reason I mentioned before.

Remember: video file extensions supported by Tomb1Main are: ".mp4", ".mkv", "mpeg", ".avi", ".webm", ".rpl" and ".FMV". Once you have your brand new MP4 files in the FMV folder, you don't need to keep the original .rpl video files around anymore.

I find upscaling to 4K in this particular case a bit of a waste of computing resources and disk space. Change my mind.

This is not a competition for the best FMV upscaling. I've taken a different approach so as to achieve a -in my opinion- better result than what was available so far (that I know of), but that doesn't necessarily mean that you have to share my approach. I respect and admire the guys that invested their time and efforts to improve these videos and make the most of them and I hope someone will improve upon their work or mine and make them even better. As upscaling and frame interpolation tools evolve and improve, and they do it fast, it's just a matter of time.

Feel free to share your thoughts. Help me improve the results. All constructive feedback is very welcome.

Finally, a big thank you to LeonDeka for being and inspiration and to all the crew behind Tomb1Main [<- the best thing to ever happen to Tomb Raider 1 IMO]

MERRY CHRISTMAS!!!
It's dangerous to go alone! Take THIS.

UPDATE: Added a version of CAFE fmv with scentil fonts for the titles. Now the font (or combination of a bunch of fonts I should say) resembles more to the original. Text shadow has also been reduced/adjusted. I recommend using this new version (remember to rename it from "CAFE (stencil font).mp4" to "CAFE.mp4"). Have fun!

Last edited by Famicom; 24-12-23 at 11:25.
Famicom is offline   Reply With Quote
Old 24-12-22, 20:07   #2
Kirishima
Member
 
Joined: Oct 2011
Posts: 757
Default

Quote:
Originally Posted by Famicom View Post
- Aspect ratio (ah, the aspect ratio!). Source material is 320x208 (black margins on top and bottom included), which is roughly 4:2.6. Back then most TVs were still 4:3 so I assume both the picture got a bit stretched vertically to fill the 4:3 frame and for this reason FMVs don't look as flattened on consoles.
While that may be the case, overscan is also a possible reason as to why fmv's didn't look as flat... at least for NTSC monitors. I'm not sure about PAL.

As for the videos themselves, I kinda want to see them in plain old 15fps because I'm not really a fan of upscaled framerates, but that's just me.

Last edited by Kirishima; 24-12-22 at 20:37.
Kirishima is offline   Reply With Quote
Old 27-12-22, 16:19   #3
byblo
Member
 
byblo's Avatar
 
Joined: Apr 2006
Posts: 62
Default

Seems a lot of work involved here, and the respected aspect ratio plus the doubled frames per seconds makes it amazing


But some notes:

- Why is nobody respecting the original fonts? Why not using those "Army" font style instead? (cafe.mp4) (and if anyone needs it, I think I still have them saved somewhere on my computer)

- Are you sure about the current limited bandwidth allowed by Tomb1Main? I have some 12Mb/s mp4 videos working well here. Your videos seems to be limited from 3 to 9Mb/s.

Maybe it would worth to try to increase a bit the final bandwitdh, since the current result showing some compression blocks from here and there, specially on darker scenes like the cafe.

Thank you for the beautiful gift


Edit:

In case, I think I found pretty similar fonts matching the cafe.rpl

These two seems the closests ones:
https://www.1001freefonts.com/stardos-stencil.font (notice the E)
https://www.1001freefonts.com/le-architect.font (notice the A)

Two others, just in case:
https://www.1001freefonts.com/chaplone.font (too thin maybe)
https://www.1001freefonts.com/lintsec.font (too fat maybe)

Last edited by byblo; 27-12-22 at 16:53.
byblo is offline   Reply With Quote
Old 27-12-22, 17:20   #4
Famicom
Member
 
Famicom's Avatar
 
Joined: Mar 2009
Posts: 466
Default

Quote:
Originally Posted by byblo View Post
Why is nobody respecting the original fonts? Why not using those "Army" font style instead? (cafe.mp4) (and if anyone needs it, I think I still have them saved somewhere on my computer)
I went literally through a few hundreds of fonts: used the app WhatTheFont and checked manually in a few websites and ended up combining 3 different fonts since I couldn't find any font, not even the "military" ones, that were 100% true to the original. I'd be glad if you could pass me the font you're referring to and see if it gets any closer. In that case, I'll upload a new version of CAFE.mp4 with said font

Quote:
Originally Posted by byblo View Post
Are you sure about the current limited bandwidth allowed by Tomb1Main? I have some 12Mb/s mp4 videos working well here. Your videos seems to be limited from 3 to 9Mb/s.

Maybe it would worth to try to increase a bit the final bandwitdh, since the current result showing some compression blocks from here and there, specially on darker scenes like the cafe.
I tried with more bandwith and they worked fine... for the most part. They always lagged at some point (for instance, CAFE.mp4 plays perfectly fine until the moment when the camera travels through the cloud of dust and gets into the crater. No matter how many times or in what format I re-rendered the video, image always froze there for a moment, then skipped a few frames to catch up with the sound that kept on playing). I can upload some heavier versiones so you can take a look or run a few tests yourself. That'll have to wait a couple days, though.

Quote:
Originally Posted by byblo View Post
Thank you for the beautiful gift
My pleasure. Thank you for stopping by and share your thoughts.

Last edited by Famicom; 27-12-22 at 23:40.
Famicom is offline   Reply With Quote
Old 27-12-22, 17:39   #5
byblo
Member
 
byblo's Avatar
 
Joined: Apr 2006
Posts: 62
Default

Quote:
Originally Posted by Famicom View Post
I'd be glad if you could pass me the font you're referring to and see if it gets any closer. In that case, I'll upload a new version of CAFE.mp4 with said font
I have edited my previous post, where you can find some fonts links


Quote:
Originally Posted by Famicom View Post
They always lagged at some point (for instance, CAFE.mp4 plays perfectly fine until the moment when the camera travels through the could of dust and gets into the crater. No matter how many times or in what format I re-rendered the video, image always froze there for a moment, then skipped a few frames to catch up with the sound that kept on playing). I can upload some heavier versiones so you can take a look or run a few tests yourself.
Maybe you can upload just one, one that is giving you problem, so I can test on my pc and to give you some feedback?
byblo is offline   Reply With Quote
Old 27-12-22, 17:51   #6
Famicom
Member
 
Famicom's Avatar
 
Joined: Mar 2009
Posts: 466
Default

Quote:
Originally Posted by byblo View Post
I have edited my previous post, where you can find some fonts links
Thanks!! Will take a closer look as soon as I reach my desktop.


Quote:
Originally Posted by byblo View Post
Maybe you can upload just one, one that is giving you problem, so I can test on my pc and to give you some feedback?
Sounds good to me. I'll keep you posted.
Famicom is offline   Reply With Quote
Old 27-12-22, 21:54   #7
Tremolo
Member
 
Tremolo's Avatar
 
Joined: Nov 2016
Posts: 499
Default

Wow. This is wonderful. Are you planning to launch these for Tomb Raider 2 too?
Tremolo is offline   Reply With Quote
Old 27-12-22, 23:38   #8
Famicom
Member
 
Famicom's Avatar
 
Joined: Mar 2009
Posts: 466
Default

Quote:
Originally Posted by Tremolo View Post
Wow. This is wonderful. Are you planning to launch these for Tomb Raider 2 too?
Glad you like'em. Sure I'll give it a spin as soon as I'm done with TR1
Famicom is offline   Reply With Quote
Old 28-12-22, 15:03   #9
klona
Golden
 
klona's Avatar
 
Joined: Oct 2009
Posts: 11,535
Default

These upscales are amazing. It's clear you put a lot of work into them!
klona is online now   Reply With Quote
Old 29-12-22, 11:42   #10
Heartache
Member
 
Heartache's Avatar
 
Joined: Dec 2008
Posts: 3,424
Default

amazing job, thank you so much
Heartache is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



All times are GMT. The time now is 15:12.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.
Tomb Raider Forums is not owned or operated by CDE Entertainment Ltd.
Lara Croft and Tomb Raider are trademarks of CDE Entertainment Ltd.