notes into fields

An audio-reactive visual guesses at music from a waveform. drift doesn't guess. Before a frame is drawn, every note of the piece is timed to the actual recording: when it starts, its pitch, how long it lasts and how hard it is played, filed under the player who plays it.

That is what lets a picture tell a cello from a viola, or know that a phrase is about to end. The picture reads the score at the clock of the recording, one frame at a time.

Sync is gated with a control clip built to be unambiguous: the note stream and the audio line up at 0 frames of lag. On Mozart's serenade, all 17 twenty-second windows matched at 0 ms.

A song is cut into sections where the music turns. Boundaries come from musicians' own notes first (the fugue subject entering, the recapitulation), then a tradition's grammar (alap, gongan), and only then audio novelty to fill long gaps. A song gets as many sections as its arcs ask for: three for a 3½-minute concerto movement, five for an 8½-minute toccata and fugue.

Sections alternate between one system alone and a layered set, and the densest stretch is always layered. A layered set is a memory ground with present layers over it, turned apart in hue.

Nothing repeats inside a song unless the music does. A return copies the section it returns to, folded through a mirror. When no unused system can fill a section, the strongest one already used comes back folded and re-hued, spread across systems so no single one returns twice.

How layers meet depends on the ground. difference subtracts: over a sparse ground it draws, over a bright one it cancels. A bright ground takes screen on both layers above it.

The first full set of videos had empty screens. Chopin went black from 1:20, the gamelan from 2:00. The director had given those sections to cells from a fallback list, and on a soft nocturne or a 1931 field recording, cells never seeds.

Now, before a system gets a section, the engine runs it on that section's own audio, entering at the section start exactly as the video will, and measures how much of the frame is lit. A system carrying a section alone, or a layered ground, needs 20%. A layer on top needs 6%.

The first version of the test lied. It ran at thumbnail size, where the grid is only 200 cells and cells looked 21% lit; at the 4K grid it was under 1%. The test now runs at the smallest size that uses the same 1000-cell grid as 4K.

Every layer is its own copy of the engine on its own worker thread, with its own state, stepped against one clock: seconds into the recording. On a ten-second Vivaldi clip, worker layers ran at 25.1 frames per second against 12.55 in the old one-thread path, with pixel-identical frames.

The video opens on the title card, in the style of a Bauhaus poster. Every change slides right to left, the card's exit included; the only fade is to black at the end. Layers for an upcoming section run six seconds early, unseen, so a field is already alive when it slides in.

The camera moves the field, not the image. Warping the simulation lets the chemistry regenerate detail, and a slower view scale runs from 1× to 2× and wraps, invisible because the field is a seamless tile. One cycle takes about 36 seconds; a loud onset only nudges it.

Screen recording is a race between simulation, encoder and compositor, and at 4K the encoder wins by dropping frames. drift renders offline: the clock steps by hand, each finished frame goes to a hardware H.264 encoder with an explicit timestamp, and the frame count is checked against the edit before audio is added.

A 4K render holds a lot of memory for a long time on a 16 GB machine. Renders run one at a time, and a guard stops one only when the kernel reports critical memory pressure for 30 seconds running.

Then the finished video is sampled, every 2 seconds for anything doubtful, and each section's lit share is read off real frames. That pass caught what the test missed: Beethoven's coda at 3 to 15% lit (difference cancelling a bright ground, now 61 to 100%), and Debussy twice.