rythme

What it is

rythme is a guitar game in the style of Guitar Hero that can turn any song into a chart to play. Give it an MP3, or paste a YouTube link, and it separates the guitar from the rest of the band, writes every note of the guitar part with a neural network trained on real charts, and builds the full song to play, at four difficulties.

You play on the keyboard or an Xbox-style guitar controller: hold the frets, strum on the beat, and keep a streak going to raise the multiplier up to 4x. Notes with a white centre are hammer-ons and pull-offs, played by fretting alone. A YouTube song plays through the official video, ads included, while the notes scroll over it.

Why I made it

I have played Guitar Hero for a long time, and I always wanted to play more songs than the ones in the games and the custom charts other players make. Charting a song by hand takes hours, so I made a game that does it by itself: any song I like, from a file or a YouTube link, becomes something I can play.

How it's built

The game is C++17 on SDL3 and it is cross platform: the same code builds for the desktop and for the web (Emscripten). The charts come from Python tools built on open-source libraries.

• Separating the guitar: Demucs splits the guitar from the band, so the model listens to the guitar part alone.
• Writing the notes: a PyTorch network, a CNN with a bidirectional GRU over a constant-Q spectrogram of the guitar, predicts every 11.6 ms, for each difficulty, whether a note starts there, on which of the five frets and whether it is held.
• Trained on real charts: the official Guitar Hero charts that ship with Frets on Fire (191 songs), learned from the same imperfect Demucs stems it sees on a new song. On 16 songs it never trained on, Expert note timing reaches an F1 score of 0.86 against the human charts.
• On the beat: librosa tracks the beats and the notes snap to them. A "fret direction" output (down, same, up) places the frets of a whole passage at once, which raised how often fret moves match the real charts from about 59% to about 70%.
• YouTube without downloading: the video plays in the official player while the page listens to it once; the audio is charted and thrown away, and only the chart is kept, so the next time the song starts right away. The game follows the player's clock from C++ and waits during ads or buffering.
• Feel: the hit window follows Frets on Fire, about 156 ms each side at 120 BPM, and a strum still counts if the fret arrives within 100 ms after it.

The public page here has one song charted from YouTube; importing your own songs runs on your computer, since Demucs and the network need Python.

Timeline

Built in October and November 2026.

Technologies

• C++17
• SDL3
• WebAssembly
• YouTube IFrame API
• PyTorch (charts)