FenPhoenix on 11/3/2020 at 09:44
Unfortunately yes, it takes forever to scan 7z files. :(
(tech talk ahead)
It's because AL's scanner needs to have random access to the archive, but 7z archives don't allow that (unless you specifically set the archive to "non-solid" when you create it, which most people don't, because it's a non-default option and selecting it would probably negate much of 7z's compressed size advantage). Every time you want to read a file in the .7z archive, you have to decompress through the whole archive until you find it, wasting a bunch of time decompressing data that you just throw away. If a readme file or missflag.str (a very important file for scanning) happens to be near the end of a 500MB archive, then the scanner would have to churn through close to 500MB of compressed data just to find the file that tells it what other files it needs, and then it would have to churn through that same data again looking for them. It's relatively likely that doing this would take more time than just extracting the entire archive to a temp folder once and then reading the files in a random-access fashion from there, so that's what the scanner does. Wish I could do better, but there you are. Nature of the format.
thieff on 11/3/2020 at 15:13
Quote Posted by FenPhoenix
Unfortunately yes, it takes forever to scan 7z files. :(
(tech talk ahead)
It's because AL's scanner needs to have random access to the archive, but 7z archives don't allow that (unless you specifically set the archive to "non-solid" when you create it, which most people don't, because it's a non-default option and selecting it would probably negate much of 7z's compressed size advantage). Every time you want to read a file in the .7z archive, you have to decompress through the whole archive until you find it, wasting a bunch of time decompressing data that you just throw away. If a readme file or missflag.str (a very important file for scanning) happens to be near the end of a 500MB archive, then the scanner would have to churn through close to 500MB of compressed data just to find the file that tells it what other files it needs, and then it would have to churn through that same data again looking for them. It's relatively likely that doing this would take more time than just extracting the entire archive to a temp folder once and then reading the files in a random-access fashion from there, so that's what the scanner does. Wish I could do better, but there you are. Nature of the format.
thanks for the explanation. I expected as much.
in short it's pointless to switch to 7z. better stick with ZIP. it works. no reason to "fix" that.
it's not a big deal.
but CSV export would be helpful. but don't sweat it too.
I remember having a program that could copy any type of standard WinAPI text field, listbox, memo or a string grid from any running windows application to a clipboard. I could try that instead, and if I put all the fm archives in a single folder there's no need to have additional path column.
I just need to recall the name of the program.
cheers.
R Soul on 11/3/2020 at 21:32
A solid .7 file can allow quick access if the FM author makes first the file in two stages:
1. Generate a new .7z file for everything except the readme, missflag.str, and the optional file fm.ini
2. Addsthe remaining files to the existing .7z file.
That creates a file with two solid blocks.
[It may be the other way round, but I tried this, one way or the other, a while ago and performance was good]
However, it depends on the FM author remembering to do it. Also, I think AnglelLoader may scan the .mis file to work out what game the FM is for, so that's another thing for the author to add to the second stage, but that would make the second block larger, slowing things down a bit.
As for the CSV file, have you had a look in AngelLoader's .ini file? You might be able to generate something from that data.
FenPhoenix on 11/3/2020 at 23:35
Yeah, I could probably do more thorough testing to see if I can be faster in some cases, even if not all.
IIRC the files AngelLoader needs to read the contents of are:
-missflag.str (to find out which .mis files are actually used - a lot of FMs have "dummy" .mis files that aren't in the right format or are just garbage data)
-newgame.str (it can find potential titles here)
-title.str / titles.str (ditto)
-all .rtf, .txt, .wri, .glml files in the base dir (or FanMissionExtras / Fan Mission Extras for T3) - the readmes, which it can scan for all kinds of data
-fm.ini
-fminfo.xml
-mod.ini (common in SS2 FMs)
-the smallest used .mis file ("used" meaning specified in missflag.str)
-the smallest .gam file if any are present
It needs to scan the above two files to determine game type. It does a quick check of the .mis file first because if it finds a string at a certain location very early in the file, it knows it's Thief 2 and can early-out right there. Otherwise it has to scan the whole .mis file, but it will scan the .gam file if there is one because those are much smaller and can still be used for detection.
What I could do is to only extract the files I actually might need to read. That way, even though I'll most likely still end up extracting more files than I need, at least it won't necessarily be the entire archive. How much time that would save I don't know; .mis files tend to be among the largest files in an FM and I would still need to extract all of them. But I guess for an FM like The Rebellion of the Builder 2 where it has a huge amount of its size in custom resources, it would save a substantial amount of time, that is, assuming files happened to be arranged in an opportune order...
thieff on 12/3/2020 at 20:06
Quote Posted by R Soul
As for the CSV file, have you had a look in AngelLoader's .ini file? You might be able to generate something from that data.
yes, I've checked it. all data I need is there. thanks.
FenPhoenix on 12/3/2020 at 20:21
Ratings are in the Rating=(some number) lines. But I should note that, for efficiency reasons, any field which has a blank or default value is not written out to the FMData.ini file. So if an FM has a rating of 0-10, say 5 for example, it will say Rating=5, but if the rating is -1 (unrated) it won't have a Rating= line at all.
FenPhoenix on 14/3/2020 at 02:10
Thanks :)
FenPhoenix on 31/3/2020 at 23:09
(
http://fenphoenix.com/apps/AngelLoader/AngelLoader_v1.4.1.zip)
AngelLoader v1.4.1 is out.
Minor tighten-up and bugfix release.
Changelog:-Readme files are now run through a character encoding detector before being loaded, so broken characters (eg. "procuré" instead of "procuré") should never - or at least extremely rarely - appear now.
-Fixed-width font is now the default for plaintext readmes, if it isn't already set.
-startmis.sav is now excluded from differential ("All changed files") backups, matching FMSel's behavior.
-Both '\' and '/' path separators are now properly handled everywhere.
Fixes:
-Fixed: FM added dates wouldn't be cached in the data file.
-Fixed: Author filter didn't take highlighted recent FMs into account.
-Fixed: Left and right arrow images on buttons were being drawn slightly incorrectly.
---
Looking towards the future, here are some major things on my wish list (but not implemented yet):
-Ability to select multiple FMs at once, so you can perform one operation on many (eg. setting "Finished on Expert" to multiple at once, or even install/uninstall multiple at once).
-Ability to delete selected FM(s) from disk (this has been requested a couple times).
-Better support for running on Linux. It will still require Wine/Mono/whatever-it-is-people-use for the foreseable future, but performing some testing and tweaking to make sure it can run as best as it can is something I'd like to do.
-Optional auto-update functionality, so people don't have to keep coming back here and doing the whole manual rigamarole. I'll make it optional and as unintrusive as possible. The main roadblock is just switching fenphoenix.com over to https for future-proofing, so I can have a permanent auto-update URL that all AngelLoader versions can check.