Machine listening, streamed to the portal

Problem

When a set is streamed, the website around it usually knows nothing about the music. There's a player, a chat, maybe a countdown. The page looks the same during a twenty-minute build as it does during a breakdown, which is a strange thing to accept for music whose whole point is that it moves.

I wanted this site to know what the set is doing while it's doing it, and to show it as a view that changes with the music because it's actually being told what the music is. An engineering dashboard with meters and graphs would answer a different question.

Hypothesis

The live set already contains most of what the site needs to know. Ableton has the tempo and the current scene, and the audio itself carries the energy and, roughly, the key. If a device inside the set reads that state and sends it out a few times a second, the portal can draw from it directly, and visitors will feel the connection without anyone having to explain it.

A second, quieter hypothesis: a small number of well-chosen values is enough. Tempo, scene, energy and key should carry more of the feeling than a full spectrum would, because they're closer to how people actually describe a set.

Setup

• A Max for Live device sits on the master track of the live set. It reads tempo and the active scene from Live, measures RMS energy on the output, and estimates the key.
• The values go through node.script inside the device to a WebSocket connection to the portal server. The connection is authenticated with a token, so only the set can write to it.
• The server rebroadcasts the state to everyone who has the page open.
• On the site, the state drives the SYSTEM ONLINE view: the terrain, the light and the motion respond to the incoming values. When no set is running, the view says so and falls back to its quiet mode.

Deliberately left out for now: per-track data, raw audio, anything that would let someone reconstruct the set from the stream.

Result

Provisional.

• The pipe from device to server to page: in progress; end-to-end status TBD.
• Latency between a change in the set and a change on the page: TBD, to be measured once the pipe runs end to end. What matters is whether it feels simultaneous, not the number itself.
• Key detection on dense, filtered material: TBD. I expect it to be the least reliable of the four values and plan to treat it as a hint.
• Whether visitors read the view as “the music is here” without explanation: TBD, can only be learned from a real session.

What changed

The main change so far is in scope. It would be easy to stream much more, every track's level and every parameter move. Keeping it to four values makes the device simpler and the view clearer, and it matches the idea behind the whole project, that a set is one continuous state rather than a pile of separate events.

The other change is the fallback. The page needs to look intentional when the system is offline, which is most of the time. SYSTEM ONLINE only means something if there's a believable version of the page that isn't.