Bookshelf

Bookshelf syncs an audiobook to a real ebook — not a generated transcript, the actual EPUB with its actual formatting — so the text scrolls and highlights sentence-by-sentence as the narration plays. The reader syncs because every sentence has two coordinates: a text address — (book, chapter, paragraph, sentence), stable across editions and features — and a time span on one specific audio file. Matching is the act of producing the second coordinate from the first. Everything downstream — scroll-sync, deep links, multi-edition switching that preserves your place, playlists that reorder a book’s sections — is just address lookups once that table exists.

The trap is assuming the two sides cover each other. They don’t: abridged editions cut text, narrators ad-lib intros and credits, rips have trims and gaps. So the matching model is a full outer join, not a mapping.

The join

Text rows with no audio partner are unvoiced — real sentences this edition never narrates. Audio spans with no text partner are unmatchedAudio — sound with no sentence, like a narrator’s opening. Both are first-class rows, not errors.

FIG. 01

The join, on two lanes

Placement

The aligner (TorchAudio MMS forced alignment) never searches the whole file per sentence. It walks the book in 300-word batches with a 60-word lookahead, anchored at the end of the last mapped sentence. An EMA of frames_per_char predicts how long the next span should be, and a monotonic clamp forbids any placement from starting before the anchor — narration only moves forward. Word timings are used to cut sentence spans, then discarded.

FIG. 02

Placement: anchored windows, monotonic clamp

Where it lies

The hard-won engineering lives in trust and repair. Alignment drift used to be findable only by listening, so I built a QA layer: every alignment embeds a content-hash fingerprint of the text it was aligned against, align qa checks monotonicity, duration sanity, and rate residuals automatically after every run, and align doctor probes chapters with distinctive sentences to localize where an alignment went wrong.

Building doctor surfaced a genuinely fun bug. The probe machinery originally reused the same CTC trick — alignment inside a search window — but unanchored, over windows far wider than the target. That’s when CTC stops telling the truth: Viterbi is free to spread a short target across every lookalike fragment in range, and it will happily report a great score for doing so. We caught a 38-character sentence “placed” across 1,537 seconds at score 0.95. The score was real; the placement was meaningless.

The fix (scan_range) never lets the model see a wide window at once: slide subwindows of about 2.5× the expected duration D across the range at a 0.5×D stride, score the target inside each, reject any placement longer than 3×D, and keep the best valid score. Escalation widens only the range being scanned — never the window the model sees.

FIG. 03

Where it lies: CTC smear

Diagnosis

QA runs after every alignment (monotonicity, durations, gaps, rate residuals, confidence-dip runs, coverage). When residuals look wrong, the doctor drops distinctive-sentence probes — picked by corpus IDF — across the chapter, places each independently against cached emissions, and reads the pattern of Δ = found − expected. The pattern is the diagnosis: constant shift, growing rate drift, missing audio, wrong edition.

FIG. 04

Diagnosis: what probe deltas look like

Where this goes

With scan_range proven, heal sits directly on top of it: take the doctor’s classified worklist, re-place only the damaged regions, and cascade-nudge neighbors instead of re-running 20 minutes of CPU-bound alignment per chapter. And because every row is addressed, the whole matching layer generalizes past this project — the same outer-join table is what a shareable sidecar registry distributes: coordinates, never files.

The copyright posture shapes the whole architecture: no audio and no book text is ever distributed. Alignments are content-addressed sidecar files — coordinates, not content — which points at the productizable version: a shareable registry of sentence-to-timestamp maps where everyone supplies their own files.

What’s real

• Netflix-style library home + full-bleed reader chrome
• Audiobook sync with chapter jumps and media-key support
• Position restore via localStorage (including Xbox Edge)
• Alignment editor for when the timestamps drift

Roadmap

• Corpus build + reader JSON
• Forced alignment seed path
• Synced player/reader shell
• Progressive / on-device alignment
• Import UX for arbitrary epub + audiobook pairs
• Public-domain demo (Twelfth Night)