Erik Michaelson

← Experiments

Bookshelf(experiment)

Synchronized audiobook player × ebook reader — same place in both.

Loading…

Player above is the sync stack on a rights-clean seed (short script + two AI narrations, MMS-aligned). Switch voices; the sentence address stays put.

Syncs an audiobook to a real EPUB with its formatting, so text scrolls and highlights sentence-by-sentence with narration. Every sentence has two coordinates: a text address (book, chapter, paragraph, sentence) stable across editions, and a time span on one audio file. Matching produces the second from the first. Scroll-sync, deep links, multi-edition place-preserving switch, playlists — all address lookups once that table exists.

Abridgments cut text; narrators ad-lib; rips have trims. Matching is a full outer join:

The join

Text with no audio partner → unvoiced. Audio with no text partner → unmatchedAudio (narrator intros, credits). Both are first-class rows.

FIG. 01

The join, on two lanes

TEXTAUDIO(5,1,1)(5,1,2)(5,1,3)(5,1,4)(5,2,1)(5,2,2)(5,2,3)unvoiced — cut in this editionunmatchedAudio —narrator intro, no sentence00:00:4100:01:1200:02:03matched rows carry (addr) ↔ [t₀, t₁)
The outer join. Sentence (5,1,3) exists in the text but is never narrated; the first 40 seconds of audio exist but match no sentence. Tolerance, sparse, and structural align regimes decide how aggressively the join insists on partners.

Placement

TorchAudio MMS forced alignment walks the book in 300-word batches with a 60-word lookahead, anchored at the end of the last mapped sentence. An EMA of frames_per_char predicts next-span length; a monotonic clamp forbids starting before the anchor. Word timings cut sentence spans, then discard.

FIG. 02

Placement: anchored windows, monotonic clamp

AUDIO — chapter filealready mappedanchor = end of last mapped sentence300-word batch+60 lookaheadcandidate wants to start here…clamped ≥ anchorexpected span ≈ chars × frames_per_char (EMA)← predicted duration seeds the search
Forward-only placement. The clamp keeps one bad placement from dragging later rows backwards — errors stay local.

Drift and repair

Every alignment embeds a content-hash of the text it aligned against. align qa checks monotonicity, duration sanity, and rate residuals. align doctor probes chapters with distinctive sentences to localize failure.

Doctor originally reused CTC inside an unanchored wide window. Viterbi spreads a short target across lookalike fragments and reports a great score for doing so — a 38-character sentence “placed” across 1,537 seconds at score 0.95.

Fix (scan_range): slide subwindows of ~2.5× expected duration D at 0.5×D stride; reject placements longer than 3×D; keep best valid score. Escalation widens the scanned range; the model still sees only the subwindow.

FIG. 03

Where it lies: CTC smear

A — UNANCHORED WIDE WINDOWtarget smeared across 1,537 sscore 0.95 — confident and wrongactual sentence: ~8 s (38 chars)B — SCAN_RANGE: SLIDING SUBWINDOWS7.9 swindow ≈ 2.5×Dstride ≈ 0.5×Dbest valid score wins · placements > 3×D rejectedescalation widens the scanned range —the model never sees a wider windowverdict is interval overlap:green = IoU ≥ 0.3, or both ends within 1.5 s43 tests incl. anti-smear over 20-minute ranges keep this honest
Same model, honest geometry. The pathology wasn’t a scoring bug — _align_window verified correct in tight windows. It was geometry: give Viterbi room to smear and it will.

Diagnosis

QA after every alignment. When residuals look wrong, doctor drops distinctive-sentence probes (corpus IDF) across the chapter, places each independently against cached emissions, and reads Δ = found − expected: constant shift, growing rate drift, missing audio, wrong edition.

FIG. 04

Diagnosis: what probe deltas look like

CONSTANT Δ → SHIFT / WRONG TIMELINEexpected ▮ vs found ▮ — every probe off by the same amountb5 ch1: Δ ≈ −1,043 s, all 16 probesGROWING Δ → RATE DRIFT / EDITION MISMATCHΔ grows along the chapter — squeeze/stretch rate residuals in QANO PLACEMENT → MISSING AUDIOno probe landscoarse scan brackets the region, boundary bisection tightens itoutput: a classified region worklist → targeted re-align
Probe patterns. The book-5 incident read as strip one: a healthy re-align plus an all-probes-Missing verdict at constant Δ exposed the smear bug in the probes themselves — the committed rows and the doctor were measuring different truths.

Sidecars

heal re-places damaged regions and cascade-nudges neighbors rather than re-running full-chapter alignment. Alignments are content-addressed sidecars — coordinates only. Audio and book text stay with the user.