The AI Tools Used to Create a Surreal FMV Game

sorryaboutyourcats
6 min read4 days ago

--

video & text made in runway

From movie trailers to Notebooklm funkery, to questionable deep dream experiments back in 2015, I just love fucking with machine learning.

For this project, I wanted to see if it was possible to create an FMV game that flowed well [i.e. smooth enough transitions that my OCD approved].

Check out LEAVE here.

Before we dive in, let’s dish out what the AI tools were used for:

  • Runway - Generative videos
  • Minimax — Generative videos
  • Firefly — Generative images for buttons
  • ElevenLabs — Generative sounds for buttons, endings & gameovers
  • LANDR — Music mastering for 2 songs
  • ChatGPT — Ending story, some video prompts

These were used but their creation wasn’t:

flowchart of LEAVE without spoiling too much

Runway & Minimax

For LEAVE, the majority of the videos were made using Runway [Gen-3], as both its speed and keyframe to keyframe feature were super useful.

On the GIF below [played at -200%] you can see it starts with guy to aliens, and on the other path, guy to table.

example of two paths in LEAVE that use Runway’s keyframe to keyframe feature

It’s pretty impressive when both keyframes don’t have a lot in common:

guy keyframe
aliens keyframe
table keyframe

Sometimes the extend feature was used too, but it was harder to make drastic changes; it was also used with no prompt, which made great stuff.

One of the prompts for extend:
The building opens into a portal, to another planet. Maybe Venus?

One of the prompts for one section:
The upside-down White House is disorienting, with everything flipped. Mirrors reflect the warped space, creating endless, twisted hallways. Chandeliers sway slightly, casting eerie light across cracked floors. Colors shift between sepia tones and deep shadows, while distant whispers echo through the halls. Some areas seem to breathe, with walls subtly expanding and contracting. Occasionally, paintings or objects glitch, momentarily distorting as if caught between dimensions, adding to the unsettling atmosphere.

Originally the titles were made using Adobe Express, but the animations were too.. normal for what I needed!

runway is great for titles

And then I tried making the LEAVE title in Runway, which took four different prompts to get the style.

intro to the game LEAVE

The winning prompt for the LEAVE title:

Logo for a TV show. It’s just called LEAVE and the letters are made out of different parts of a building. It’s a bit surreal. The background under the text is just green, so that it can easily be keyed. The letters wiggle like a storm is going through them.

oh hello not yoshi

Minimax’s most helpful feature making this game has been not giving a shit about IPs, allowing Mario and Will Smith to gracefully appear.

This can be a whole other post about what should be allowed to be made, but I mean, you could just make your own LoRA for anyone you want.

10 of the ~70 buttons for LEAVE made in Adobe Firefly

Firefly

With local stable diffusion set up locally already, why not use that for the buttons? Well…. I like trying new stuff out and wanted to give Firefly a try!

After using it, I liked the consistency, but it’s definitely more limited than using a crazy ComfyUI workflow.

Frames from each section’s video were used as a reference.

Composition helped make buttons like restart and start; also when I needed variations of a similar button.

I would love to post an example prompt of one of the buttons, but Firefly doesn’t have a history knowledge [at least when I used it].

ElevenLabs

They have awesome audio tools, but the one I used was text to sound effect. 192 different sounds were made [not all were used though].

Check out the sounds in the video above! In order:

  1. droplet with dreamy reverb
  2. man walking, and then the sound of spaghetti getting thrown and eaten
  3. balloon popping, reverb, dreamy
  4. window closing dreamy reverb echo in a cave

Honestly, I really love these sounds and will be using them for future projects — strange how they can make me so happy!

LANDR

Out of all songs, only two songs got mastered by LANDR. These two tracks were recorded off a PS3, using a game called Sound Shapes.

top was mastered by LANDR, see the difference?

ChatGPT

Each of the 5 endings were generated with Chat. I won’t show them here because, like, that’s spoiling it. But here is at least a redacted prompt:

this time instead of !!!!, the ending is with the !!!! !!!! [don’t mention her name]. there are !!!! everywhere, and something the main character did, triggered !!!! !!!! !!!! !!!! !!!! which !!!! !!!! !!!! !!!! !!!! [no idea how many, but a lot] — why is this a good thing? well, now she can !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! !!!! [which was the game].. do something with that.

Sometimes, using it to expand on basic ideas or describe an image in more detail can result in better video prompts.

wonder which one is related to LEAVE…

The unused

Like with all experiments, some just don’t work out, either because of what it generated [inconsistent style, bad quality, etc.] or the style just didn’t fit.

Topaz’s Video AI was used for upscaling [1280x768 to 2560x1536], enhancing [Theia & Proteus], interpolation [60fps] and stabilization.

The upscaled videos were not worth it as it required a higher bitrate to take advantage of, which then wouldn’t work with itch.io’s 512 MB HTML 5 limit.

I rock 240Hz monitors and have created 300fps fractal videos before, but for the LEAVE clips, upping the framerate made me a bit nauseous. Stabilization was added but it didn’t help. Honestly 25fps looks the best, as it gives a more movie feel to it anyway.

Topaz’s Video AI dialog, usually producing great results

Suno was used for the main ending theme song, but it just didn’t work as it was so different than the songs I made, so it just wasn’t used.

lyrics to an unused ending song

Ending notes

AI is here to stay, and while its future can feel uncertain, it’s also a tool that’s redefining creativity. Rather than resist it, I see value in exploring what it can do — pushing boundaries and blending the strengths of human intuition with machine-generated precision. For now, it’s all about curiosity and seeing where this collaboration leads.

And yes, that last paragraph was ChatGPT; little dramatic no?!
If you still haven’t checked out LEAVE yet, it’s here waiting for you to leave.

Thanks for reading / skimming / playing! 😸👍

--

--

sorryaboutyourcats
sorryaboutyourcats

No responses yet