PLAI by ear

You hear something you want to play, and the gap between hearing it and reading it is where most people stop. PLAI closes that gap in one step: upload an audio clip, get back piano sheet music you can actually read off. Built over a weekend at the UC Berkeley AI Hackathon in 2025 with Litong Deng and Bella.

The architecture is the part I actually worked on, and the whole bet was to not build the obvious thing. Audio transcription normally means a signal-processing stack — onset detection, pitch tracking, quantization — and there is no version of that you get working well in a weekend. So we skipped it. Hand the raw audio file straight to a multimodal model, ask for a note sequence, and hand that sequence to a real notation engine. Gemini 2.5 Pro does the listening; VexFlow, pulled in from a CDN at runtime, does the engraving. Two well-chosen components where a pipeline used to be.

Worth being straight about how it got built. The three of us worked around a single laptop prompting Cursor, so almost none of the commits are in my name — my contribution was arguing out the architecture, deciding where the model boundary should sit, and running the demo. That's a real part of building something and it's also not the same as having written the files, so I'd rather say it than let a commit graph imply otherwise.

The hard part was never getting notes out of the model. It was getting those notes into notation that looked right — a sequence that's numerically correct and visually wrong is still useless to someone trying to play it. Model choice turned out to be a taste question as much as an accuracy one, and the route still falls back from Gemini 2.5 Pro through 1.5 Pro to 1.5 Flash if a model is unavailable. It handles short clips, piano only, one clef. Everything past that — full songs, other instruments, real time signatures — is the list we ran out of weekend on.