The AI Tools Used to Create a Surreal FMV Game
From movie trailers to Notebooklm funkery, to questionable deep dream experiments back in 2015, I just love fucking with machine learning.
For this project, I wanted to see if it was possible to create an FMV game that flowed well [i.e. smooth enough transitions that my OCD approved].
Before we dive in, let’s dish out what the AI tools were used for:
- Runway - Generative videos
- Minimax — Generative videos
- Firefly — Generative images for buttons
- ElevenLabs — Generative sounds for buttons, endings & gameovers
- LANDR — Music mastering for 2 songs
- ChatGPT — Ending story, some video prompts
These were used but their creation wasn’t:
- Topaz Video AI — Upscaling, stabilization & interpolation [@60]
- Suno — Theme song
Runway & Minimax
For LEAVE, the majority of the videos were made using Runway [Gen-3], as both its speed and keyframe to keyframe feature were super useful.
On the GIF below [played at -200%] you can see it starts with guy to aliens, and on the other path, guy to table.
It’s pretty impressive when both keyframes don’t have a lot in common:
Sometimes the extend feature was used too, but it was harder to make drastic changes; it was also used with no prompt, which made great stuff.
One of the prompts for extend:
The building opens into a portal, to another planet. Maybe Venus?
One of the prompts for one section:
The upside-down White House is disorienting, with everything flipped. Mirrors reflect the warped space, creating endless, twisted hallways. Chandeliers sway slightly, casting eerie light across cracked floors. Colors shift between sepia tones and deep shadows, while distant whispers echo through the halls. Some areas seem to breathe, with walls subtly expanding and contracting. Occasionally, paintings or objects glitch, momentarily distorting as if caught between dimensions, adding to the unsettling atmosphere.
Originally the titles were made using Adobe Express, but the animations were too.. normal for what I needed!
And then I tried making the LEAVE title in Runway, which took four different prompts to get the style.
The winning prompt for the LEAVE title:
Logo for a TV show. It’s just called LEAVE and the letters are made out of different parts of a building. It’s a bit surreal. The background under the text is just green, so that it can easily be keyed. The letters wiggle like a storm is going through them.
Minimax’s most helpful feature making this game has been not giving a shit about IPs, allowing Mario and Will Smith to gracefully appear.
This can be a whole other post about what should be allowed to be made, but I mean, you could just make your own LoRA for anyone you want.
Firefly
With local stable diffusion set up locally already, why not use that for the buttons? Well…. I like trying new stuff out and wanted to give Firefly a try!
After using it, I liked the consistency, but it’s definitely more limited than using a crazy ComfyUI workflow.
Frames from each section’s video were used as a reference.
Composition helped make buttons like restart and start; also when I needed variations of a similar button.
I would love to post an example prompt of one of the buttons, but Firefly doesn’t have a history knowledge [at least when I used it].
ElevenLabs
They have awesome audio tools, but the one I used was text to sound effect. 192 different sounds were made [not all were used though].
Check out the sounds in the video above! In order:
- droplet with dreamy reverb
- man walking, and then the sound of spaghetti getting thrown and eaten
- balloon popping, reverb, dreamy
- window closing dreamy reverb echo in a cave
Honestly, I really love these sounds and will be using them for future projects — strange how they can make me so happy!
LANDR
Out of all songs, only two songs got mastered by LANDR. These two tracks were recorded off a PS3, using a game called Sound Shapes.
ChatGPT
Each of the 5 endings were generated with Chat. I won’t show them here because, like, that’s spoiling it. But here is at least a redacted prompt:
this time instead of !!!!, the ending is with the !!!! !!!! [don’t mention her name]. there are !!!! everywhere, and something the main character did, triggered !!!! !!!! !!!! !!!! !!!! which !!!! !!!! !!!! !!!! !!!! [no idea how many, but a lot] — why is this a good thing? well, now she can !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! [which was the game].. do something with that.
Sometimes, using it to expand on basic ideas or describe an image in more detail can result in better video prompts.
The unused
Like with all experiments, some just don’t work out, either because of what it generated [inconsistent style, bad quality, etc.] or the style just didn’t fit.
Topaz’s Video AI was used for upscaling [1280x768 to 2560x1536], enhancing [Theia & Proteus], interpolation [60fps] and stabilization.
The upscaled videos were not worth it as it required a higher bitrate to take advantage of, which then wouldn’t work with itch.io’s 512 MB HTML 5 limit.
I rock 240Hz monitors and have created 300fps fractal videos before, but for the LEAVE clips, upping the framerate made me a bit nauseous. Stabilization was added but it didn’t help. Honestly 25fps looks the best, as it gives a more movie feel to it anyway.
Suno was used for the main ending theme song, but it just didn’t work as it was so different than the songs I made, so it just wasn’t used.
Ending notes
AI is here to stay, and while its future can feel uncertain, it’s also a tool that’s redefining creativity. Rather than resist it, I see value in exploring what it can do — pushing boundaries and blending the strengths of human intuition with machine-generated precision. For now, it’s all about curiosity and seeing where this collaboration leads.
And yes, that last paragraph was ChatGPT; little dramatic no?!
If you still haven’t checked out LEAVE yet, it’s here waiting for you to leave.
Thanks for reading / skimming / playing! 😸👍
