nhivanye on 14/3/2006 at 09:55
Shadowspawn,
I'll start looking at matlib stuff. Unfortunately I don't think I'll finish anything before Oblivion arrives next week :D which will absorb a lot of my spare time for a week or two at least :eek:
STiFU on 14/3/2006 at 13:42
Hehe... It will probably the same here dude!! ;) But cool that we have another guy here looking at the filetypes!!
Shadowspawn on 20/3/2006 at 02:16
I managed a few more hours of work on this, and there's good news and bad news.
1.) Good news! Lots and Lots of the matching data type / size entries are EXACTLY the same, and can just be read from the other stock .IBT files. So we're good there.
2.) Other entries, although the same size, have small (1-4) byte differences. This makes it a tiny bit harder.
My current thought is to zero out the data portion of matching entries and fill them in from the stock IBT. I'm toying with the idea of zeroing out all but the differences, and not overwriting them during the stock refill. (there's problems, what if the correct value is 0, I don't want to overwrite it).
I figure ZIP will crunch down any region of 0s to almost nothing, and it makes inflating the CBT that much easier. I'm still a few hours / days away from a test version - stay tuned.
Ziemanskye on 20/3/2006 at 06:59
However long it takes, you're doing us a great service.
Crispy on 21/3/2006 at 06:00
Awesome.
Quote Posted by Shadowspawn
I'm toying with the idea of zeroing out all but the differences, and not overwriting them during the stock refill. (there's problems, what if the correct value is 0, I don't want to overwrite it).
You could keep a list of locations in your CBT (I'm assuming that stands for Compressed IBT) where the correct value is 0 but your program will overwrite it with a wrong value from the stock IBT. So then you inflate the CBT using the method you mentioned, and then you go through your zeroes-list and write 0 to each location. Obviously there are some optimisations to be made to this algorithm (like using the zeroes-list
while inflating the CBT instead of afterwards, to avoid massive seek times when you're writing to widely-spaced locations in the newly-inflated IBT).
Might be a little inefficient, but I can't think of anything better off the top of my head.
Oh, and 0 is probably a poor choice for what I'm describing, since there are likely to be many 0 bytes in an IBT. :) Perhaps 254 or 255 or so would be better.
Shadowspawn on 21/3/2006 at 13:45
Actually, I came up with an easy fix.
Since the differences are very small, I'll just subtract one from the other and store the differences. That way it'll be almost pure 0s, and ZIP will compress the hell out of it. When I re-inflate the file, I'll just add the stock item to the existing data. Should work fine.
Now, for a few more experiments....:thumb:
Crispy on 22/3/2006 at 03:27
Quote Posted by Shadowspawn
Since the differences are very small, I'll just subtract one from the other and store the differences. That way it'll be almost pure 0s, and ZIP will compress the hell out of it. When I re-inflate the file, I'll just add the stock item to the existing data. Should work fine.
Ah, good idea. I never would have thought of that. :)
Shadowspawn on 22/3/2006 at 03:48
Hopefully I'll have the compressor finished this week, I'll have to wait to try the inflator. At least we can see how much it's likely to save us.
I'll be on vacation next week in Aruba - a much needed break. I'll finish the inflator when I get back. I think this will work out pretty well, the bulk of the compression will be the textures, which won't compress well, and remain constant (more or less) from mission to mission.
Hey, any taffers live in Aruba? I'm buying the first round of drinks!
Shadowspawn on 4/4/2006 at 17:52
I've just finished debugging the compressor and have some preliminary results to share. I need to run more tests, but the first round looks good.
Evicted.ibt = 86,534,144 bytes
Evicted.zip = 58,495,070 bytes - This is just IBT compressed via WinZip
EvictedC.zip = 11,358,987 bytes - This is the compressed IBT run through WinZip
It's almost 8 to 1 vs the orginal, and a 5 to 1 compared to the raw IBT zipped. I need to run some more large IBT files through it and I'll post the results.
Now, if I can just properly re-inflate the file...Still working on it...