KCTools [v1.3.3] - Fast external replacement for DromEd lighting and other tools

HavvicGames on 23/2/2025 at 18:50

KCTools is a collection of tools for working with NewDark fan missions (1.27 and 1.28 tested, equivalent SS2 versions should also be supported, theoretically functional back to 1.19). Currently there are two tools, a lightmapping tool, and a .BIN model export tool.

Downloads:
(https://github.com/JarrodDoyle/KCTools/releases/download/v1.3.3/kctools_1.3.3_dromed_win_x64.zip) KCTools DromEd v1.3.3
(https://github.com/JarrodDoyle/KCTools/releases/download/v1.3.3/kctools_1.3.3_win_x64.zip) KCTools Full v1.3.3 (Windows)
(https://github.com/JarrodDoyle/KCTools/releases/download/v1.3.3/kctools_1.3.3_linux_x64.zip) KCTools Full v1.3.3 (Linux)
Additionally you can find the source code (https://github.com/JarrodDoyle/KCTools) here, and all previous versions and changelogs (https://github.com/JarrodDoyle/KCTools/releases) here.

The DromEd release is intended to be used directly within DromEd and now supports all NewDark versions, though 1.27+ is most tested. The full release includes an additional lighting GUI app and will likely be removed in later versions now the DromEd integration is improved.

Lighting:

* Replicates all existing DromEd lighting functionality
* Massively improves lighting times
* Better light culling, improving performance in large-scale city maps and eliminating object lighting errors
* Provides additional warnings to mappers highlighting mis-configured light settings
* Generally produces more accurate shadows
* Direct integration with DromEd in 1.19 and above

The lighting tool is intended to replace DromEd's lighting step (especially in later stages of mission development where lighting becomes very slow). DromEd's built-in lighting is single-threaded, has a number of minor bugs and scales extremely poorly with lightmap scale and complex large open spaces (e.g. modern city missions). KCTools lightmapper takes advantage of modern CPUs with a multi-threaded approach and improves lighting times, reduces lighting artifacts, and has the potential to improve in-game performance in some complex scenes.

Inline Image:

https://i.imgur.com/X02RZQS.png

The primary way to use the lighting tool is with NewDark and the DromEd version of KCTools. A temporary editor-time OSM is used to launch KCTools externally and functionally near seamlessly replaces DromEd's lighting stage. Assuming the DromEd basic toolkit is being used (and files haven't been moved around too much), simply extract the contents of the DromEd release into the root of your Thief/SS2 installation. The KCTools path can be customised from the default by setting "kctools_path your/path" in an appropriate Dark engine config file.

If you don't want or need direct DromEd integration, use the Full release. Lighting can be ran either via command line (run "kctools.exe light --help" in the command line for instructions) or via the GUI executable.

Inline Image:

https://i.imgur.com/692IgFy.png

Model Export:

* Export specific models, or every model as a binary glTF
* Embeds and correctly palettises required model textures
* Respects object hierarchy for easy use in custom animations

Available on the command line, run "kctools.exe model export --help" for instructions. Currently this is a pretty simple tool, but is likely to be expanded in the future.

Changelog:

Code:

v1.3.3:
- Lighting
- Added `Portalize.cmd` (missed this previously because I only ever optimize)
- Fixed crash that could occur when traversing the BSP tree and encountering nodes that reference non-cell planes
- General
- Added logging the current version at beginning of logs
- Added logging of any exceptions (crashes) to make debugging simpler
- Replaced `Magick.NET` with simpler wrapper project. Anyone updating from a previous version should remove `Magick.Native-Q8-x64.dll` from the kctools folder as it is no longer needed.

v1.3.2:
- Lighting
- Fixed cell AnimLight bitmask corruption that could occur when >32 AnimLights see the same cell
- General
- Added loading resources from an implicit `./` path
- Fixed order-dependant incorrect values when reading variables that have a shared prefix (e.g. `game dark` and `game_hardware 1`)

v1.3.1:
- Lighting
- Fixed "Blocks Vision" doors not correctly blocking light in rare cases where they're in a split cell
- Added BSP based Light/Door cell detection (technically faster, but we're talking 0.005s sort of thing)

v1.3.0:
- Lighting
- Added direct DromEd integration support for NewDark versions prior to 1.28 (Should function back to any mission portalised in 1.19)
- Added integration of log into DromEd MONO
- Added warning when AnimCellMap limit is reached
- Added warning for out-of-range Hue and Saturation on lights
- Fixed incorrect campaign detection with the DromEd integrated setup when multiple DromEd instances are open
- Fixed engine AnimCellMap limit being ignored
- Fixed crash when too many AnimLights attempted to light a cell polygon
- Fixed absolute paths in install configs being incorrectly joined as a relative path. Note this only works on windows/through WINE because `C:/blah/...` is not a valid unix path
- Fixed user placed Blockable brushes incorrectly blocking light
- Model Export
- Added support for including the `.bin` model extension with the `--model-name` flag
- Fixed incorrect transparency on some exported `gif`/`pcx` textures
- General
- Updated all projects to .NET 10
- Added support to `KeepersCompound.Formats` for saving modified models
- Added early out when the requested campaign can't be found
- Fixed a crash when the expected FM directory (e.g. "./FMs") doesn't exist
- Fixed a crash when specified install directory didn't contain `cam.cfg` or `cam_ext.cfg`

v1.2.3:
- Lighting
- Fix door cells not blocking light immediately after portalisation
- Fix roombrush EAX types being dropped when saving
- Fix potential issue when writing rendering parameters (in practice I don't think anything was wrong, but just to be safe)
- Revert part of 1.2.0s cell visibility calculation changes that was causing cells viewed through a small gap to not be lit

v1.2.2:
- Lighting
- Fix global attenuation and saturation (`lm32_params`) being applied to 16-bit lightmaps when they shouldn't be
- Fix incorrect saturation when non-default `lm32_params` are used
- Fix accidental modification of LightColor Saturation values when non-default `lm32_params` are used (oops)
- Fix issue with GUI dropdown menus not reopening when re-clicking a keyboard focused field
- Add reload resources button to GUI
- Model Export
- Fix inaccurate help screen text for `model export --output-directory`

v1.2.1:
- Lighting
- Update helper scripts to support SS2
- Model Export
- Fixed DDS textures not always being correctly applied

v1.2.0:
- Lighting
- Fixed water polygons not being lit when a cell has no solid polygons
- Fixed an infinite loop when a mission contained a model with self-referential sub-objects
- Improved raytrace scene construction time
- Reduced memory allocations during cell visibility calculation
- Reduced cell visibility calculation time by up to 1.5x (mostly on large open maps with high light counts)
- Reduced cell visibility false-positives
- Removed `--simpleVis` flag
- Model Export
- Fixed a crash when a model has self-referential sub-objects
- Fixed a crash when exporting models using PNG or DDS textures from outside a zip file
- Added vertex normals on models that use them
- Added PCX, TGA, and BMP texture support (stock SS2 models can now be exported with textures)
- Replaced custom image decoders and ImageSharp library with Magick.NET
- General
- Fixed temporary extracted files (`obj.crf` and `fam.crf`) not being cleaned up on Windows systems. Relevant files are now read directly from their archives and cached for repeated use.
- Rewrote model loading and split it into a separate standalone Formats library (release Soon™)
- Various other refactors, internal changes, and chores :)

v1.1.3:
- Fixed incorrect campaign determination when "return to fm selector" is enabled

v1.1.2:
- Added additional logging around resource path gathering and campaign detection timings
- Fixed incorrectly applying zeroed out transforms to some static model sub objects
- Fixed applying rotation to translating model joints, they now slide as expected

v1.1.1:
- Hopefully fixed the integrations with DromEd!
- Added pre-modified version of `Optimize.cmd` to save users the effort
- Fixed `DoLight.cmd` issue with saving when FM directory has a . in it's name
- Added `--auto-campaign` flag to KCTools to determine campaign/FM from `DromEd.log`
- Improved camera restoration in `DoLight.cmd` (rotation is now restored, and issues with deleting the incorrect brush should be fixed)
- Added better logging when `install.cfg` doesn't contain any valid `resname_base` paths
- Fixed crash when `--install-path` is invalid

v1.1.0:
- Add GUI frontend
- Add `.bin` model export command
- Modified light command to have more unified arguments with model export and give space for future tools
- Add `--quiet` flag for all commands
- Add lighting `--inspect` mode/flag
- Add experimental support for System Shock 2 (executes fine, but I don't have a good SS2 setup to actually check results)
- Add DromEd 1.28 support scripts

v1.0.1:
- Fix fast light vis not including the cell containing the light
- Fix sunlight bleed
- Fix light vis erroneously discarding some cells

v1.0.0:
- Initial release

Contact:
I don't use TTLG much, so I'll probably be slow to any responses. If you need help or something seems broken the best place to contact me is the (https://www.ttlg.com/forums/showthread.php?t=150598) DromEd discord server, my username there is Jayrude.

Skol on 24/2/2025 at 15:19

And by "improves lighting time", he means from 1 hour to 15 seconds.

PinkDot on 24/2/2025 at 22:00

Congrats! Amazing job and important milestone for the community!

ZylonBane on 18/3/2025 at 07:28

So, any plans to make an SS2 compatible version of this?

HavvicGames on 18/3/2025 at 21:01

I don't have a ShockEd setup to test with, but it may already work. If you try it/have already tried it and encounter issues could you share them?

If it's incompatible then there's three areas where I could imagine the incompatibility existing:
1. Configs. SS2 might be using some config stuff I don't know about and the game resources get loaded incorrectly
2. Properties. I know SS2 has some differing object properties, but I haven't looked into whether any of them would affect lighting
3. File formats. If there's any major differences to the parts of the various file formats I read/modify then parsing them will fail

Depending on how much work resolving the problems is (assuming it is incompatible) I can look into adding SS2 support soon.

ZylonBane on 19/3/2025 at 05:27

I tried it, but it complained that it couldn't find thief.exe.

As for the other considerations, I'd imagine that NewDark unified lighting between the two engines quite a bit.

Kubrick on 23/3/2025 at 22:28

As a total newbie, I'd really benefit from a small tutorial video on how to use. I tried it when it was first released but I shamelessly failed to use it properly the few times I tried.

HavvicGames on 1/6/2025 at 09:03

New version (v1.1.0) and a rename due to adding another tool :cool:

Changes:

* Added GUI lighting frontend
* Added ".bin" model export command
* Modified light command to have more unified arguments with model export and give space for future tools
* Added "--quiet" flag for all commands
* Added lighting "--inspect" mode/flag to quickly preview lighting configuration issues without actually performing lighting
* Added experimental support for System Shock 2 (executes fine, but I don't have a good SS2 setup to actually check the result)
* Added DromEd 1.28 support scripts to help setup running directly through DromEd

HavvicGames on 2/6/2025 at 19:59

There are unfortunately issues with the dromed integration scripts. They work on linux via WINE (after fixing some filenames), but on windows they're a no-go. Working on some alternatives that will hopefully be ready soon.

HavvicGames on 4/6/2025 at 19:29

New version v1.1.2 (v1.1.1 didn't get posted here).

Changes since v1.1.0:
- Hopefully fixed the integrations with DromEd!
- Added pre-modified version of "Optimize.cmd" to save user effort
- Fixed "DoLight.cmd" issue with saving when the FM directory has a "." in it's name
- Added "--auto-campaign" flag to KCTools to determine campaign/FM from "DromEd.log"
- Improved camera restoration method when reloading the mission in "DoLight.cmd" (rotation is now restored, and some issues with deleting the incorrect brush should be fixed)
- Added better logging when "install.cfg" doesn't contain any valid "resname_base" paths
- Added additional logging around resource path gathering and automatic campaign direction timings
- Fixed a crash when "--install-path" is invalid
- Fixed incorrectly applying zeroed out transforms to some model sub-objects
- Fixed applying rotation to translating model joints, they now slide as expected.

Update: Quick hotfix
- Fixed incorrect auto-campaign when using "return to FM selector"