AI sound engineer

Problem

In a live set that runs as one continuous space, the mix never stands still. Generative parts add and remove layers, agents open filters over several minutes, and the density of the whole thing keeps changing. Every one of those moves changes levels. Left alone, a long build does what long builds tend to do: it gets louder, the low end piles up, the limiter on the master starts working harder than anyone asked it to, and by the peak there's no headroom left for the moment that was supposed to be the peak.

A human engineer at the desk fixes this all night by ear. At a set like this there isn't always one, and even when there is, the engineer doesn't know that the tension is supposed to keep rising for another six minutes.

Hypothesis

A feedback loop can do the engineer's routine part: listen to the output, compare it with a target, adjust gain, EQ and compression, listen again. If that loop also receives the musical state from the rest of the system (where the energy is meant to go, how long the build should last), it can keep the level roughly where it should be and leave room for intensity to come from arrangement and timbre instead of volume.

The part I'm least sure of is the second half. Keeping levels sane is a solved problem in principle. Knowing that a quieter moment is intentional and shouldn't be “corrected” is not.

Setup

The starting point is the system I describe in the book chapter about sound engineers: a model that listens to the room through a measurement microphone, compares what it hears with a target, and drives a digital mixer over the network the way a tablet would. It was built for a band's live sound. For Nobody Quits it's being adapted to a different job:

• inputs: the stereo output and the main stems from the DAW, plus the room mic when there is a room;
• context: the musical state stream from the live system (tempo, scene, energy, intended direction);
• outputs: channel gain, EQ, bus compression and send levels; the master limiter is left alone and only watched;
• a hard override: I can freeze any channel or the whole loop with one control, and the loop is not allowed to touch anything I've frozen.

Targets are defined per section of the set rather than as one fixed curve, so that a breakdown is allowed to be quiet.

Result

Provisional. The loop is in testing on the band setup it was built for; runs on a Nobody Quits set are still ahead.

• Keeping levels steady while layers come and go: TBD for this material.
• Headroom at the peak of long builds compared with an unassisted run: TBD, needs a like-for-like comparison.
• Whether the loop respects intentional quiet sections when it has the musical state: TBD. This is the main question.
• How often I had to use the override: TBD, will be logged.

No numbers go here until they come from a recorded run.

What changed

The first thing that changed is how I think about the target. I started with “one good-sounding mix”, and it became clear quickly that a set needs a sequence of targets that follow the musical state, or the loop will iron out the very dynamics I'm trying to build.

The second is the override. It started as a safety feature. It's turning into an instrument: freezing the loop on purpose, for a stretch where I want the mix to stay rough, is a musical decision like any other.