vfig on 19/12/2023 at 22:57

---

This is a repost from my now-defunct blog, so that the information is still available:

While working on my upcoming Thief mission, I noticed an intriguing property in Dromed: Renderer > Face Textures
https://i.imgur.com/VKxCDU6.png

Smiling, wincing, and stunned faces? I had my surprised face on at any rate. Thief never changed NPC’s faces, did it? If I added this property to an object, the game crashed. Ah, obviously this is dead code relating to a feature cut long before the game shipped.

Things got more interesting though: when I added this property to an object, it also dropped in this related state property: Renderer > Face State
https://i.imgur.com/eZxjO44.png

What is this “talking” bit about? Well this made me remember an oddity in the list of Dromed commands that I’d half noticed earlier, but not really paid attention to:

…textures to match speech samples. Does this mean Thief at one point had–not just NPC faces changing to match their emotional state, but also animated mouths when they talked? That’d be a big deal. Valve had been making a big deal of their feature in Half-Life that animated an NPC’s mouth in real time to match their speech. Huh. I did a little more poking around in Dromed and found another property that seemed related: Sound > Face Motions
https://i.imgur.com/aL5rnAD.png

I was well and truly interested now, and decided to do some digging. First point of call was the Thief 2 source code, and it didn’t take long to find the corresponding code:
https://i.imgur.com/28VCvIT.png

Having poked through the face code and the rendering code a little, it does look as though all the code for this feature remained in the game. Having read the code a little more carefully, and done some tests in Dromed, here’s what I’ve found:

The face_process command in Dromed will generate a file facepos.str with animation frames for all speech samples whose schemas have the Sound > Face Motions property turned on. Here’s a small sample I generated with it, from a hammerite muttering:



Each of these strings is a sequence of frames for a 16-frame-per-second facial animation for the sample in question. Each digit relates to one of the following mouth positions:


I wanted to see how well these generated frames corresponded to the speech, so here’s hm1a0mu3 alongside a set of simple faces that I drew and animated with a bit of javascript:
[video=youtube;7BqYCRsuXPY]https://www.youtube.com/watch?v=7BqYCRsuXPY[/video]

I don’t know about you, but having made that animation, I was now really keen to see if I could get this working in the game somehow. Now, I haven’t got it working at all yet, but I have got as far as having it not crash when I use the Renderer > Face Textures property. Here’s how it works:

Each of the Neutral/Smile/Wince/Surprise/Stunned textures should be the prefix of a set of textures in the mesh/txt and mesh/txt16 folders for each of these visages (to use the term from the source code). Let’s say I’d put in guardneut for the neutral visage. Now if this visage won’t support mouth animations, then I only need to create the guardneut1.gif texture for this visage. If it does support animations, then I also need guardneut2.gif and so on up to guardneut6.gif. Here the digits 1-6 correspond to neutral, small oh, smalle ee, big oh, big ee, and shout, respectively. Note that while the frames in facepos.str start at zero, the texture names start at one, presumably to make them more artist-friendly. Strange that the programmers didn’t also do the same for the page numbering in book files:
https://i.imgur.com/QlBziIB.png

But anyway, once I had the numbered textures in place, and set the visage names to the appropriate prefixes, the game no longer crashed when the guard starting talking. It didn’t swap out the texture at all, but it was clearly doing something.

But that’s as far as I’ve got just now. I haven’t got a firm handle on what the “texture to replace” field should contain, and how it corresponds to the texture name baked into the model file. I think I’m going to need to use a debugger for the next bit, to break when it’s checking the materials to replace so I can really see what’s going on. So expect an update sometime after I do that.

vfig on 19/12/2023 at 22:58

---

This is a repost from my now-defunct blog, so that the information is still available:

Nothing is ever that easy, is it?

Now that it was clear that the feature was only in Thief 2, rather than the demo being some random guard humming and mumbling to himself, instead there was an obvious conversation to use to show off the feature: the archers of Lady Van Vernon and Master Willey hurling insults at each other across the rooftops, from Life of the Party. So I immediately set work on that.

First I added a second archer to the test map, this time the one with the purple shirt. And quickly hacked a copy of his textures with the same sketched mouths as the first (red-shirted) archer. Loaded up the map and checked that he, too, moved his mouth when coughing and muttering. Good.

Identifying the speech schemas for the conversation in question just needed a quick look at CONV.SCH:



Then it was back to Dromed and find_obj sg11101A and so on to select the schemas one by one in the Object Hierarchy, and add the Sound > Motion property to each. I don’t know if there’s a better process to do this en masse than manually, but for just eight schemas the manual steps are fine. And so I regenerated facepos.str.

Now conversations in Thief use the so-called pseudoscripts provided by the AI > Conversation property. A pseudoscript is basically a bunch of actions that can be performed: play a sound schema or a motion schema, turn to face a particular direction, move to someplace, add or remove a metaproperty (for behavioural changes), and so on. Due to the high barrier to writing custom .osm scripts for the game (the script language is basically a custom C++ preprocessor), many fan mission designers–I haven’t looked to see if the same is true of the official missions!–also used this pseudoscript system for other scripted behaviours. So anyway, I needed a conversation setup.

So I opened up the Life of the Party mission and found the object there that drove the archers’ conversation. With my usual hacky method for copying a few things between mission files in the editor, I went through each step of the conversation and screenshotted the edit dialog, pasting them into the copy of Photoshop I always have running. Then I can just refer to the screenshots when copying the stuff into my own mission. There’s probably a better way of doing this, but–knowing Dromed–doubtful that there’s a much better way. And it turned out to be quite a good thing that I copied it this way.

So anyway, I went back to my mission and created a conversation, linked it to my two guards, then went step by step through the eight pages and filled out the speech actions in each (not bothering with the motions and behavioural bits just yet). Added a button to trigger it, hit Alt+G to get into the game, and pushed the button.
https://i.imgur.com/Az6fNoA.png

https://i.imgur.com/9j009Pw.png

Looking good. And of course I’d forgotten to save the changes to the test map.

Taking a look at the source code again, I found that the speech system helpfully offers a callback interface for other parts of the game to get notified when a line of dialog plays; but it’s only used in two places: the face animation uses this to know when to start and stop animating; and the conversation system uses it to know when a line has finished, so that it can play the next step of the pseudoscript. Shouldn’t be a problem though, cause these kinds of callback systems are always set up to handle multiple callbacks, and … oh.



Ah. So the speech system originally handled up to four different callbacks, but the conversation system didn’t manage its own memory properly and so someone hacked the speech system to only support one at a time. The assertion error was a check that the array of callbacks wasn’t already full: and since both the face animations and the conversation were trying to register callbacks, the first one in filled up the array. This was not going to be easy to work around.

Anyway, now that I knew the source of the assertion failure and why it was happening, I decided to run the test map again and ignore the failure this time. Here be dragons, but I figured they were reasonably polite dragons that wouldn’t cause much harm. And so once again I pressed the button and got the error, but this time chose Cancel. The game continued, and the first guard proceeded to spout his line of dialog, and stopped. And then the other guard just stared mutely at him. Like he’d forgotten his lines. Anyway, this is where it really clicked with me that the conversation system needed the callbacks installed to work properly, otherwise it just never moves on from the first step.

Because this limit is baked into the code and the statically allocated array of callbacks, it’s not really changeable. And who knows if the bug described still exists even if it was changed? (It probably does.) So I had to come up with a different way of handling this.

And that way was: eight different conversations! One for each line. So thankfully I still had all my screenshots of the conversation setup, so it was straightforward to refer to them again when setting up the eight conversations. But a new problem now: one button wasn’t going to be enough to set off the conversations properly, so it was time for more hacks!

I wrote a Squirrel script for the button that would turn on the first conversation, wait for a time, then turn on the second conversation, wait again, and so on. I opened up each of the sound files for the conversation in Audacity to get the length, and set the wait times accordingly. And it worked! It’s ugly as hell, and painful, and not at all robust enough to use in anything that’s going to ship, but it worked. The first guard finished his line, and a fraction of a second later the second conversation was triggered and the second guard remembered to say his line, and so on. At last time for the final spit and polish, I thought.

I googled for references for mouth positions for different sounds, and found a ton of cartoon ones, and a few very bland and expressionless photographic ones, and then (https://miko-noire.deviantart.com/art/Mouth-References-436290183) this one by Miko-Noire which–although it’s not really for different sounds–had exactly the amount of liveliness I was looking for.
https://i.imgur.com/koAZKqW.png

So I slapped those into the guard textures, and with a bit of slapdash colour alteration, they fit both the light-skinned and dark-skinned guards pretty well. A proper art job would have given them different mouths, of course, and probably also adjusted the cheeks to match. But I’m no artist, and this was already looking much better than my temporary version.
https://i.imgur.com/assMtpw.png

There was one final bug waiting for me. You see, the face animation system will animate the face for a speech sample even if it doesn’t have an entry in facepos.str. It just plays a predefined, simplistic animation in a loop:


And this had probably been coming up before, but now that the faces looked better and I was paying attention to how they looked, I saw this animation coming up. It was now quite noticeable, because it doesn’t have the neutral “mouth closed” frame, nor either of the two frames where my new images had the mouth wide open. So now I could see that randomly, one or the other of the guards would just flap their lips continually without ever really opening or closing their mouth: the telltale sign that this animation was being used.

Well, I spent a lot of time trying to figure this one out. I thought maybe there was a race condition, and so the strings file needed to be preloaded (nope). I thought perhaps the speech samples needed to be preloaded (nope). I thought maybe there sometimes wasn’t enough time at the end of one line for the sample to actually finish, so a bit more delay between lines might help (nope). I thought maybe the conversation callbacks were the issue, and that triggering the guards’ lines through a horrible hack involving AIWatchObj links and teleporting markers around–but that didn’t even work, let alone fix the problem. In the end, I just decided to live with it. In the worst case, I could record multiple takes, and just splice in each guard’s line when it actually worked. It’d be a pain, but it’d work.

So finally the last thing I did was throw in a couple of camera markers, and update the script to attach the player’s camera to each in turn, so that each guard would get a closeup. Oh, and put into the conversation the motions and behaviour changes needed to make the characters move and then start shooting at each other at the end.

And then when I was finally read to record, all eight lines played perfectly, with the correct mouth animations, all on the first take. So maybe there is a Master Builder after all! so here, finally is the result:
[video=youtube;pIAyFXP-6NI]https://www.youtube.com/watch?v=pIAyFXP-6NI[/video]

If you want to muck about with this yourself, you can (https://github.com/vfig/thiefy-stuff/raw/master/T2%20Face%20Animation%20Demo/t2_face_animation_demo.zip) download the demo mission from Github. You’ll need Thief 2 NewDark 1.25 to open it.

Update 2: NewDark 1.26 has just been released, with this gem in the patch notes:


This means using Face Textures should no longer break Conversations, and vice-versa. One could actually ship a mission using this feature now!