Giving every word of a story a timestamp

Manarway is a language-learning platform I work on at Akera. Its students are learning English, and besides the exercises, lessons contain stories and units contain paragraphs to listen to. Both come with narration. While it plays, the sentence being read is highlighted with its Arabic translation above it, and hovering a word shows that word's translation.

The narration comes from ElevenLabs text-to-speech. Most of the path from a teacher's edit to that highlighted sentence is code I wrote, and this post follows it.

Generating on save#

The first version ran in the browser. The teacher app called an audio endpoint, got back the recording and its word timings, built the transcript itself and then saved. In March I moved that work to the server, so the services that save lesson stories and unit paragraphs now regenerate the narration whenever a save changes the text. This is the lesson side:

if (!skip_audio && stepData.type === 'story' && stepData.story?.text) {
  const currentText = step.story?.text || '';
  const newText = stepData.story.text;

  if (newText !== currentText) {
    try {
      const audioResult = await this.elevenLabsService.generateAudioWithTranslation(newText);
      if (audioResult) {
        const paragraphs = newText.split('\n\n').filter((p) => p.trim());
        const existingTranslations = stepData.story.translation
          ? stepData.story.translation.split('\n\n')
          : undefined;

        stepData.story.audio_url = audioResult.audio_url;
        stepData.story.transcript = await buildTranscriptFromAudio(
          paragraphs,
          audioResult,
          existingTranslations,
        );
      }
    } catch (error) {
      Logger.warn(`Audio generation failed for step ${stepId}: ${error}`);
    }
  }
}

Nothing is generated unless the new text differs from what's stored. Paragraphs are separated by blank lines, and the teacher's Arabic translation is split the same way, so each paragraph keeps the translation the teacher wrote for it. If generation fails, the error is logged and the save goes ahead without new audio.

Each regeneration uses ElevenLabs character quota, so the teacher decides whether it happens. When a save changes the text, the teacher app opens a dialog with "Save Without Audio" next to "Save & Regenerate Audio". The first sends skip_audio: true, and the server leaves the existing recording and transcript as they are. If the text is unchanged, the app saves without asking and without generating anything.

Timings from ElevenLabs#

The request goes to ElevenLabs' with-timestamps endpoint with the eleven_multilingual_v2 model and mp3_44100_128 output. Along with the audio, as base64, it returns an alignment: every character of the input with a start and end time in seconds. A transcript needs words, so the first step folds characters into words:

function parseAlignment(alignment: {
  characters: string[];
  character_start_times_seconds: number[];
  character_end_times_seconds: number[];
}): Array<{ word: string; start: number; end: number }> {
  const words: Array<{ word: string; start: number; end: number }> = [];
  let currentWord = '';
  let wordStart = -1;

  alignment.characters.forEach((char: string, index: number) => {
    const isSpace = char === ' ' || char === '\n';
    const startTime = alignment.character_start_times_seconds[index] ?? 0;
    const endTime = alignment.character_end_times_seconds[index] ?? 0;

    if (wordStart === -1 && !isSpace) {
      wordStart = startTime;
    }

    if (!isSpace) {
      currentWord += char;
    }

    if ((isSpace || index === alignment.characters.length - 1) && currentWord) {
      words.push({ word: currentWord, start: wordStart, end: endTime });
      currentWord = '';
      wordStart = -1;
    }
  });

  return words;
}

A space or a newline closes the current word. A word starts when its first character starts and ends when its last character ends. The last word in the text is closed by the index check, since no space follows it. Punctuation isn't stripped, so said, is one word, comma included. That keeps this list in line with the next step, which splits the original text on whitespace too.

This parsing is older than the service. I first wrote it inside the original audio endpoint, and a teammate later reworked the request code around it, adding clearer error messages and a helper for the API key.

Speaker names#

Some texts have lines that begin with a speaker's name and a colon. The narrator shouldn't read the name aloud, but the student should still see it. Text with two or more speakers counts as a dialogue and takes its own path, covered next. For everything else one request is enough, and stripSpeakerNames removes any prefixes from the text sent to ElevenLabs while recording the positions of the words it removed:

if (isDialog(text)) {
  return this.generateDialogAudio(text, voiceId, secondVoiceId);
}

const { strippedText, speakerWords } = stripSpeakerNames(text);
const voice = voiceId ?? MALE_VOICES[0]!;

const { audioBase64, ttsWords } = await this.callTTS(strippedText, voice);

const originalWords = text.split(/\s+/).filter((w) => w.length > 0);
const uniqueWords = [...new Set(originalWords)];
let translations: Record<string, string> = {};
try {
  translations = await translateWords(uniqueWords);
} catch {
  Logger.warn('Word translation failed, continuing without translations');
}

let ttsIndex = 0;
const words: AudioWord[] = originalWords.map((word, i) => {
  const translation = translations[word] ?? '';
  if (speakerWords.has(i)) return { word, translation };
  const ttsWord = ttsWords[ttsIndex++];
  return { word, start: ttsWord?.start ?? 0, end: ttsWord?.end ?? 0, translation };
});

return {
  audio_url: `data:audio/mpeg;base64,${audioBase64}`,
  words,
  duration: ttsWords[ttsWords.length - 1]?.end ?? 0,
};

Every word of the original text becomes a transcript word. A speaker's name gets a translation but no timing, and any other word takes the next timed word from ElevenLabs. The match is purely by position. No strings are compared, so it holds only while both sides split the text the same way.

Two voices for a dialogue#

isDialog counts lines that start with a speaker prefix: a capitalised word of letters followed by a colon. With at least two of those lines and at least two different names, the text is read with a voice per speaker.

Choosing voices starts with a guess. The names go to a language model (Gemini 2.5 Flash through OpenRouter), which is asked to return a JSON object mapping each name to "male" or "female". A teammate later moved that call onto a shared Mastra agent, with the same prompt and the same parsing of the reply. Speakers then take turns through a list of two voices per gender. If the call fails, the map comes back empty and every speaker falls back to a male voice.

Each line is then its own request:

let runningOffset = 0;
const allWords: AudioWord[] = [];
const audioBuffers: Buffer[] = [];
let currentSpeaker: string | null = null;

for (const line of lines) {
  const match = line.match(/^([A-Z][a-zA-Z]*):\s*/);
  const speaker = match?.[1];
  if (speaker) currentSpeaker = speaker;
  const voice = currentSpeaker
    ? (speakerVoices[currentSpeaker] ?? MALE_VOICES[0]!)
    : MALE_VOICES[0]!;
  const contentText = match ? line.slice(match[0].length) : line;
  const lineWordTexts = line.split(/\s+/).filter((w) => w.length > 0);

  const { audioBase64, ttsWords } = await this.callTTS(contentText, voice);
  audioBuffers.push(Buffer.from(audioBase64, 'base64'));

  let ttsIdx = 0;
  const lineWords = lineWordTexts.map((wordText, wi) => {
    if (match && wi === 0) return { word: wordText, translation: '' };
    const ttsWord = ttsWords[ttsIdx++];
    return {
      word: wordText,
      translation: '',
      ...(ttsWord?.start != null && { start: ttsWord.start + runningOffset }),
      ...(ttsWord?.end != null && { end: ttsWord.end + runningOffset }),
    };
  });

  allWords.push(...lineWords);
  runningOffset += ttsWords[ttsWords.length - 1]?.end ?? 0;
}

Stitching the clips#

The requests run one after another, and each line's MP3 goes into audioBuffers. At the end, Buffer.concat joins them into one recording. ElevenLabs times every clip from zero, so each line's words are shifted by runningOffset. When a line starts with a speaker's name, that first word wasn't spoken, so it gets no timing.

The offset is an approximation. It grows by the end of each line's last word rather than by the length of the clip, so if a clip runs on past its last word, that tail isn't counted and the lines after it are highlighted slightly early. Over a long dialogue the error adds up. The pipeline's TTS service, which a teammate wrote, has a getMp3DurationSeconds function that reads an MP3's real length from its frame headers, and using it here would keep the offsets in step with the audio.

From words to a transcript#

buildTranscriptFromAudio turns each paragraph into a transcript segment:

let wordIndex = 0;

const segments: KyselyTranscript[] = await Promise.all(
  paragraphs.map(async (text, i) => {
    const paragraphWordTexts = text.split(/\s+/).filter((w) => w.length > 0);

    const words = paragraphWordTexts.map((wordText) => {
      const audioWord = audioResult.words[wordIndex];
      wordIndex++;

      return {
        text: wordText,
        translation: audioWord?.translation || '',
        ...(audioWord?.start != null && { start: audioWord.start }),
        ...(audioWord?.end != null && { end: audioWord.end }),
      };
    });

    const timedWords = words.filter((w) => w.start != null);
    const segmentStart = timedWords[0]?.start ?? 0;
    const segmentEnd = timedWords[timedWords.length - 1]?.end ?? segmentStart;
    // …

A single wordIndex walks the word list across all the paragraphs. That's safe inside Promise.all because each callback finishes its counting before its first await, so the paragraphs claim their words in order and only the sentence translations run concurrently. A segment starts at its first timed word and ends at its last, which leaves speaker names out. The sentence translation is the teacher's when there is one, and Google Translate's otherwise.

Translating word by word#

Word translations come from translateWords, which I first wrote for the seeder that migrates content from a legacy database. It translates from English to Arabic with the translate-google package. Punctuation is stripped before translating and put back afterwards. Results are cached by lower-cased word, and unique words go out in batches of 50 with a 500 ms pause between them. A failed batch is retried with backoff, and a translation identical to its source word counts as a failure.

Contractions such as "mother's" use the translation of their base word. Words still missing at the end get one more attempt, wrapped in the phrase The word "${cleanWord}" in English, with the translation read back from between the quotes. I added that retry for short words that came back untranslated on their own.

Stored as rows#

Transcripts live in their own tables. In February I moved lesson steps out of JSON columns and into tables, story_transcripts and transcript_words among them. In March the paragraph transcripts moved in as well, the table was renamed transcripts, and transcript_words gained nullable start and end columns for the timings. A save that carries a transcript deletes the story's rows and writes them again. Reads load the segments and words for a batch of stories together and group them with Object.groupBy, instead of querying one story at a time.

Following along in the browser#

The student page wraps the recording in new Audio(audioUrl) and copies its currentTime into React state on every timeupdate event. A data URL plays like any other, so the page never needs to know where the audio came from. TranscriptRenderer gets the segments and the current time, and a segment is active while the time sits between its start and end. The active sentence turns yellow and its Arabic translation opens in a tooltip above it. Hovering a word shows that word's translation. The renderer still accepts the en and ar keys that words had when transcripts were JSON.

Word-by-word highlighting is still there, behind a highlightMode prop. When I added the prop in March, sentence mode became the default. Before that, the renderer lit up individual words whenever it had timings. Nothing passes 'word' today, so in practice the word timings decide where each sentence starts and ends.

The active sentence scrolls itself to the middle of the screen, once per sentence. That used to fight a student scrolling back to reread something, so a wheel or touch movement now pauses the auto-scroll for three seconds:

const handleUserScroll = useCallback(() => {
  isUserScrollingRef.current = true;
  clearTimeout(scrollTimeoutRef.current);
  scrollTimeoutRef.current = setTimeout(() => {
    isUserScrollingRef.current = false;
  }, 3000);
}, []);

useEffect(() => {
  window.addEventListener('wheel', handleUserScroll, { passive: true });
  window.addEventListener('touchmove', handleUserScroll, { passive: true });
  return () => {
    window.removeEventListener('wheel', handleUserScroll);
    window.removeEventListener('touchmove', handleUserScroll);
    clearTimeout(scrollTimeoutRef.current);
  };
}, [handleUserScroll]);

The paragraph's ref callback does the rest. It scrolls only for an active sentence that isn't the one it scrolled to last, and only if the student hasn't scrolled in the past three seconds.

Paragraph lessons also pass preventCopy, which blocks copying and the context menu on the text. Those paragraphs are followed by a self-test in which the student types the paragraph back. A teammate later added a setting that keeps the current sentence highlighted while the audio is paused.

What I'd change#

The editor path keeps the recording in the database. generateAudioWithTranslation returns a data:audio/mpeg;base64, URL, and the service writes it straight into stories.audio_url, a text column. It goes back to the first version, whose endpoint handed the teacher app the same kind of URL. It's the simplest option: there's no upload step, and the audio is saved in the same request as its transcript.

The cost is size. At 128 kbps a second of speech is 16 KB of MP3, and base64 adds a third, so a minute of narration is over a megabyte of text in one row. It comes back with every query that selects the whole story and travels inside the step's JSON whenever a student opens the lesson. The teacher app also sends it back to the server with every save of a lesson story. The content pipeline, which teammates built, already uploads its narration to storage and keeps only the file URL. I'd send the editor's audio through the same upload.

The other change is to "Save Without Audio". It updates the story's text but leaves the transcript rows as they were, and the lesson views render from the transcript whenever there is one. So students keep reading the old wording, and hearing the old recording, until someone regenerates. I'd rebuild the transcript from the new text on that path too, without timings, so the words are right and the highlighting waits for the next recording.

NextPricing a shipping container