Thief 1/2 & SShock 2: DDFix and Enhanced Resolution Patch - discussion

Timeslip on 1/4/2008 at 18:36

Quote Posted by Hiatus

you mean offset 80 (right after all these 00 00's)?

could you explain what changing these bytes do?

0x80 == 128, so either will do. :p

It's just a way for ddfix to tell that the texture needs to be overridden. The first 128 bytes are the file header, and overwriting them will cause thief to spot something is wrong, so ddfix overwrites the texture data instead. It's far more complicated than it needs to be now, because originally I was only looking at the texture after it had been converted to 16 bit, so a 1 in the middle of the texture data could become anything, depending on what was in the palette. Now that I'm getting it on file load, I could just stick some ascii text into the file. It's easier to leave it as it is than to change it now though.

Actually, for it to work properly, you'll need to change the rest of the bytes down to 769 (or 0x301, if you prefer :p) from the end of the file to something less than 0xc0 as well. (Otherwise the compression used will mean that the file no longer contains the correct number of pixels.) The last 769 bytes are the palette, which breaks the texture if it isn't there even though thief ignores it once it's loaded. The GUI handles that all automatically, so you might as well use that instead now that you've installed it.

Hiatus on 1/4/2008 at 19:29

you mean I change .pcx in 2 places:

1. at the start of the file, starting from offset 80 (byte 128): 51 bytes+file_name_string(no path&ext)+1byte (60 bytes in total if texture file name is 8 chars - so now we're at 188bytes/offset 0xBC), and it can't exceed offset 300.

2. at the end of .pcx is some palette byte pool which has to be changed as well (all bytes must be < C0 there) so if something is i.e. F0 it gets changed to BF (or less), if i.e. B9 then left alone? Or is it the space at the beginning of file, (in my example) between offsets BC and 300 (bytes 188 - 769)?

Did I get it right (probably not - don't know .pcx file structure at all)? Could you up some example hi-res(?) .tga (to put in \res\ddfix\, and .pcx before AND after the patching so I can compare them and see by example what inside should be changed to what)?

Timeslip on 1/4/2008 at 20:05

Quote Posted by Hiatus

EDIT: I don't get it: in 'new file path' I enter blabla, GUI says: <thief2_full_path>\res\ddfix\blabla.tga does not exist. Continue anyway? why .tga? Then I point it to some .pcx, it patches it (says done), and pcx gets destroyed (contents). What's going on? What .tga should I have - is it hi-res replacement for patched .pcx?

Yes, the .tga is the replacement for the patched .pcx. I only used .tga because it's an easy format to read. I may add support for dds later, so that you can get customized mipmaps.

Thief needs to see a valid 8 bit .pcx or it'll crash. (Or gif/bmp/tga, but ddfix doesn't intercept those file reads yet so you can only replace pcx's.) The new texture therefore needs to be someplace else where thief can't see it. Thief loads the .pcx from \res\fam like normal and can't see anything wrong, but then when it tires to load it into video memory I switch it out with the replacement texture.

The data in the pcx is replaced with the info to tell ddfix where the replacement texture is. The GUI was supposed to make backups, but that seems to be broken. :( (I knew there was a reason I wrote BETA all over it...) If you run thief without ddfix or if ddfix can't find the replacement texture, thief will try to render the modified pcx which is usually just a solid block of colour with some random gunk along the top line.

Quote Posted by Hiatus

Btw, terrain/world texture .pcx's are stored in fam.crf (renamed .zip) in \res, not standalone files AFAIK.

I know. The gui can't edit files inside the .crfs, so you'll need to extract them while you're editing. (to res\fam\, like normal.) Remember that if thief see's a fam folder it'll ignore fam.crf even if the folder has files missing, so you have to extract the whole thing even if you only want to modify one or two.

You can pack them back into the archive when you're done if you want.

Quote Posted by Hiatus

you mean I change .pcx in 2 places:

1. at the start of the file, starting from offset 80: 51 bytes+file_name_string(no path&ext)+1byte, and it can't exceed offset 300.

2. at the end of .pcx is some palette byte pool which has to be changed as well (all bytes must be < C0 there) so if something is i.e. F0 it gets changed to BF (or less), if i.e. B9 then left alone?

Did I get it right (probably not - don't know .pcx file structure at all)? Could you up some example hi-res(?) .tga (to put in \res\ddfix\, and .pcx before AND after the patching so I can compare them and see by example what inside should be changed to what)?

Not quite. The pcx file format consists of 128 bytes of header, followed by a variable number of bytes of texture data, followed by 769 bytes of optional palette. You need to edit the texture data without touching the header or palette. You don't need to modify the whole of the texture data, just overwrite the start of it and then make sure that it still contains the correct number of pixels. Changing all bytes in the section to be less than 0xC0 is equivelent to not using compression, which makes it easier to keep track of how many more pixels you need. (In case it's not clear, the data section is probably going to change length.)

The data section is compressed, so even though you can tell how many pixels are in the final image you don't know the size of the section in the file. The compressed size should be stored in the header really, but it's a bit of a dodgy file format design. The palette is 768 bytes long, preceded by a 0xC, so that an app can easily check if a palette might be present by seeking back 769 bytes from the end of the file; obviously you'd have to decompress the whole file to be 100% sure that the palette exists and that that byte wasn't just 0xC by chance. (Again, whether or not the palette is present and its offset into the file should really be stored in the header...)

Check (http://www.fileformat.info/format/pcx/) here for the full format specs. Thief uses version 5.

Anyway, the whole point of adding the ability to edit the pcx's to the GUI was so that you didn't have to worry about exactly what changes were being made. :p There's some tga's in \res\fam\skyhw\ that you can use for testing purposes. Just make you own backup of a pcx and then modify it with the gui to see the difference. (You should see that the start and the end of the files are identicle, with the middle section changed.)

Hiatus on 1/4/2008 at 20:29

I had already borrowed some .tga from elsewhere and I got it to work I think (I compare .pcx file content before and after patching and I can see these changes starting from offset 80. Thank you for thorough explanation of how it works/pcx file structure etc, anyway I'm glad GUI does the patching automatically so I don't have to delve into too much :).

ps should the patched .pcx be/look corrupt (beginning from) near file start when I try to open it in file viewer - I mean is it normal (after all (compressed) data section gets patched)?

===

now it's time to take some .pcx texture from known (to me) location/level, get higher res .tga replacement (put it in res\ddfix), patch .pcx, pack it back to fam.crf, load T2 and that level/location and see for myself what's going to happen, I guess ;).

Timeslip on 1/4/2008 at 20:36

Quote Posted by Hiatus

ps should the patched .pcx be/look corrupt (beginning from) near file start when I try to open it in file viewer - I mean is it normal (after all (compressed) data section gets patched)?

If you look at a modified pcx in a file viewer, it should look something like (http://img182.imageshack.us/img182/2289/66327432dk5.png) this. The colours will vary depending on the texture family, but the general picture should be the same. (A solid block of colour with some noise in the top line or two.) It's still a perfectly valid pcx.

Hiatus on 1/4/2008 at 21:07

yes, mine (a few I tried) look similar (no wonder, data section is same value repeated through whole (non-overwritten) section).

ps would you recommend any characteristic/suitable/easy to find world texture to replace (preferably some grass/ground texture from beginning of some level)? Preferably sth 64x64/128x128 (I would replace it with 256x256 .tga I already got - would it be ok for testing or is it too small and even vanilla T2 could load it?) .All I find and could locate within the game is too big :/. OR do you have some 512x512 .tga you could up for me to test?

btw: just noticed that ddfixGUI.dll contains sth like that: "Direct3DCreate9 d3d9.dll" - I suppose it's a remnant from ddfix 1.0/1.1 - am I right?

sNeaksieGarrett on 1/4/2008 at 22:01

Quote Posted by TheNightTerror

Last time I checked 1280x800 was a widescreen resolution? The Dark engine doesn't support widescreen resolutions.

I did not know that, sorry.:erg:
I probably should have figured that, but I didn't know. I just checked desktop display and it shows a little pic view of what 1280 x 800 looks like, and now I can tell.

Quote Posted by Timeslip

1.2.7 is up. Hopefully it'll fix the crash on exit problem. The GUI also recognises thief.exe now, so there's no need to rename it when playing thief 1.

Wow, another version!:D It appears Timeslip has not abandoned us.:)

Quote Posted by Dan Knott

Hmm 1280*1024 works fine for me and my computer is quite old - what kind of lag do you mean? Weird.

How old is "old"? Also, I mean a sort of stuttering type lag. Like, iirc, it isn't really low FPS, it's more of a slight lagging, like stutter?

inselaffe on 2/4/2008 at 01:35

Quote Posted by sNeaksieGarrett

How old is "old"? Also, I mean a sort of stuttering type lag. Like, iirc, it isn't really low FPS, it's more of a slight lagging, like stutter?

Hmm well yer, i just checked with fraps and mine stays above 45 - generally around 50 or 60 (i think thief is capped at 60 anyway).

Mine is just a radeon 9600 pro (though the 256Mb version if that makes any difference) and 1Gb of ram and amd 64 3200+ but on normal windows xp home.

sNeaksieGarrett on 2/4/2008 at 03:46

You got me beat dude, I have a crappier computer than you.:o :laff:
At least mine is a little bit weaker than yours.
Actually, thief is not capped, unless you have V-Sync on.

Timeslip on 2/4/2008 at 05:34

Quote Posted by Hiatus

ps would you recommend any characteristic/suitable/easy to find world texture to replace (preferably some grass/ground texture from beginning of some level)? Preferably sth 64x64/128x128 (I would replace it with 256x256 .tga I already got - would it be ok for testing or is it too small and even vanilla T2 could load it?) .All I find and could locate within the game is too big :/. OR do you have some 512x512 .tga you could up for me to test?

I've been replacing \core_2\patio.pcx, which is used on the ground in the first mission. There's no requirement for the new texture to be bigger than 256x256; you can use whatever size you like. As long as the texture is in \res\ddfix, you can be certain that it's ddfix loading it and not thief.

Thief 2 can load a 256x256 tga, but since it downconverts it to a4r4g4b4 it's still better to load it with ddfix instead: Just compare my (http://img358.imageshack.us/img358/3102/83184637hg2.png) first screenshot with what happens if I let (http://img181.imageshack.us/img181/444/75693307lm3.png) thief load the tga. (Actually, if I let thief load the tga it crashes. Does it not support them for landscape textures? Anyway, that's an a4r4g4b4 texture, so that's what it would look like if it was thief rendering it.)

Quote Posted by Hiatus

btw: just noticed that ddfixGUI.dll contains sth like that: "Direct3DCreate9 d3d9.dll" - I suppose it's a remnant from ddfix 1.0/1.1 - am I right?

Yes. 1.0/1.1 used it for checking your graphics card had dx9 support, and getting the supported resolutions, AA and AF. 1.2 still uses it to get the resolution list. I didn't bother changing it down from dx9 because the list of supported resolutions is going to be the same either way.

Edit: I've updated the beta.

Code:

Fixed GUI not creating backups
The GUI can now batch modify a whole pcx family
Fixed an issue with tga's containing an Image ID field getting their colour channels flipped
Textures are now correctly freed from system memory at the end of each level
File paths for replacement textures can now contain characters other than a-z, 0-9 and '\'.

The textures now behave in exactly the same way as the default thief textures: i.e. At the start of a mission all textures used in that level are loaded into system memory, they are freed when the level ends or when you reload, and they only exist in video memory if there's something using them visable on screen.

You'll need to remodify any already modified pcx's; I've switched to using ascii text now that I'm not looking at the 16 bit textures any more. I was thinking of dropping the texture path length limit down from 128 to 40 characters to let you modify 8x8 textures, but since there aren't many textures that small there doesn't seem to be much point. :confused:

Edit2: Think it's worth moving this 32 bit texture discussion to the editors guild forum? It's not really related to troubleshooting any more, and more people who might want to make use of it would see it over there.