Shrinking IBT files for distribution

Shadowspawn on 5/4/2006 at 20:01

I would expect to have something to field test in the next 2 weeks.

I can create the IBT file during decompression, and delete the CBT. Does that work for you?

I would expect to have the CBT file in the same location as the IBTs.

One of my goals is to have the decompression be flexible enough to allow better compression in the future and still have it work. And be backwards compatible. I've had great luck with compressing 'Crystal Clear Conscience', but not so great with 'Krellek's Labyrinth', probably because Labyrinth has so much custom content. I'll post those results tonight. I want to try a few more optimizations and see if I can crunch Labyrinth down a bit more.

potterr on 5/4/2006 at 20:14

Quote Posted by Shadowspawn

I would expect to have something to field test in the next 2 weeks.

Cool, I will be able to do a fair bit of testing with it then.

Quote Posted by Shadowspawn

I can create the IBT file during decompression, and delete the CBT. Does that work for you?

Deleting the CBT afterwards would be best for me as it just means a slight change to the current_install.txt file and a bit of space saved on the drive. I'm not fussed either way is fine for me although I would guess people would prefer the space saving.

Quote Posted by Shadowspawn

I would expect to have the CBT file in the same location as the IBTs.

Good thats where I have them set up for.

Quote Posted by Shadowspawn

One of my goals is to have the decompression be flexible enough to allow better compression in the future and still have it work. And be backwards compatible. I've had great luck with compressing 'Crystal Clear Conscience', but not so great with 'Krellek's Labyrinth', probably because Labyrinth has so much custom content. I'll post those results tonight. I want to try a few more optimizations and see if I can crunch Labyrinth down a bit more.

Good stuff, one other thought I had was, are CBT files used in any way in T1/2? the reason being is that I need to be sure that:
As GL uses the fact that an IBT file exists in the zip it determines it is a T3 FM, now as the IBT files won't exist here but CBT ones will, GL will determine that both IBT and CBT files indicate its a T3 FM. However I wouldn't want GL to determine that a T2 FM was a T3 FM if it used a CBT file (albeit a different file type but same extention).
Not a big problem, I just need to do some extra checks in the code.

EDIT: one other thing, can the DLL have these extras:
1) A version check call (so GL can determine against the readme or some other file in the zip if the re-packager DLL is the right one and is going to work, i.e. not an older DLL on a FM that needs a newer version).
2) Possibly a progress report event, not essential but useful if the process takes a long time to do.
:thumb:

Shadowspawn on 6/4/2006 at 01:03

Some results of testing compression on other large IBT files. (Of course, this assumes the method I'm using will work for decompression).

Crypt.ibt = 31,823,872 bytes
Crypt.zip = 22,237,075 bytes
CryptC.zip = 3,188,444 bytes

CrystalClear.ibt = 44,021,760 bytes
CrystalClear.zip = 31,487,504 bytes
CrystalClearC.zip = 3,999,084 bytes

CrystalClear2.ibt = 16,500,736 bytes
CrystalClear2.zip = 11,779,242 bytes
CrystalClear2C.zip = 991,385 bytes

CrystalClear3.ibt = 25,819,136 bytes
CrystalClear3.zip = 19,341,495 bytes
CrystalClear3C.zip = 2,822,411 bytes

Labyrinth.ibt = 127,557,632 bytes
Labyrinth.zip = 77,696,754 bytes
LabyrinthC.zip = 49,498,502 bytes

As you can see, Labyrinth didn't compress so well, but as far I as I tell there is a lot of custom content. So that would be expected. But the other missions compressed very well.

At this point, I'll consider this a reasonable performance level and move on to coding the decompressor. Wish me luck! :thumb:

Oh yeah! CBT files are not used in T1/2, or even SS2 for that matter. So we should be good for GL.

New Horizon on 6/4/2006 at 01:24

Smashing job sir! Simply smashing! :)

ascottk on 6/4/2006 at 01:34

Quote Posted by Shadowspawn

As you can see, Labyrinth didn't compress so well, but as far I as I tell there is a lot of custom content. So that would be expected. But the other missions compressed very well.

Yup, Labyrinth is a beast :cheeky: There's a lot of custom textures and smeshes in that thing. At least 70 megs have been trimmed off! That should make epithumia happy :thumb:

242 on 6/4/2006 at 09:36

Quote Posted by Shadowspawn

Man, it's so cool :thumb:

d'Spair on 6/4/2006 at 14:08

Looking forward to playing with the compress tools! If everything works fine, I guess we will see a new birth of T3 FMs.

STiFU on 7/4/2006 at 12:08

At least the t3 FMs will gain more popularity... I am totally impressed by the work you and all the other guys here do... Keep it up!! Can't wait to hear more about it!!

bloodangel1977 on 7/4/2006 at 12:33

Any chance you going to release the source for this Shadowspawn?
Wouldnt mind myself seeing it. I understand if you dont want to.

Anyway I was thinking about this last night. This contains textures and models right? Wouldnt you get better compression if you extracted out the two and stored them in two or three files?

New Horizon on 7/4/2006 at 12:55

Quote Posted by bloodangel1977

Anyway I was thinking about this last night. This contains textures and models right? Wouldnt you get better compression if you extracted out the two and stored them in two or three files?

From my understanding, all the Vanilla T3 textures and models are stripped out. Only custom content, that can't be reinflated with the vanilla t3 .ibt files, are left inside the newly compressed files.