Why YouTube summarisers miss code and on-screen text

A lot of "YouTube to notes" tools work the same way: fetch the video's captions, give the text to a language model, return a summary. That works well for a talk where people just speak. It breaks on the videos people most want notes from: coding tutorials, whiteboard lectures and anything where the important part is on the screen.

I build Unwatch, and I ran into this while testing how to make lessons cheaper. Here is what I measured, and a two-minute way to check any tool yourself.

What a transcript leaves out

Captions contain what was said. In a tutorial, the most useful things are often shown, not said:

• the exact URL, endpoint or file path typed into the editor
• the code itself, which the speaker reads out loosely ("so we add the handler here")
• terminal commands and their output
• slide text, diagram labels and whiteboard writing

A model working from captions never sees any of it. It can only guess from the words around it.

The test

I used three clips and compared lessons made from captions only against lessons made by a model that watches the video (it reads the screen as well as the audio). The table below has the two clips where I measured both; on the third I only have the example further down.

1. A 15-minute code-heavy API tutorial (Postman, request URLs, JSON).
2. "React in 100 Seconds", a fast-cut video with code on screen for a few seconds at a time.
3. A 10-minute section of a lecture where the teacher writes on a board and uses a tokenizer tool.

For each clip I listed the "screen facts": things that appear only on screen, such as an exact URL or the output of a calculation. Then I counted how many each lesson got right. The reference was a lesson made by a full watch of the video.

Two examples

The wrong URL. In the API tutorial the speaker says "v1 completions" out loud. The URL on screen is https://api.openai.com/v1/response. Every lesson made from captions wrote the spoken version. Every lesson made by watching the video got the URL from the screen right. If you copied the caption-based version into your code, you would be calling the wrong endpoint.

The word the board corrected. In the whiteboard clip the captions said "invisible". The teacher had written "indivisible" on the board. A model that looked at those seconds fixed it.

One more example, against my own work: an older run of my watching pipeline misread a long multiplication result shown on screen. Newer runs read it correctly. A model that watches the video isn't automatically accurate; it just has the information.

Why the answers look plausible

The caption-based lessons weren't obviously broken. They read fine. They had sensible-looking code or URLs, written the way the speaker said them. That's the problem: a guess about code looks just like the real thing. You find out when it fails to run.

Check any tool in two minutes

1. Pick a video with code where a URL, flag or version appears only on screen.
2. Find that exact string in the video and write it down.
3. Paste the video into the tool and search the result for it.
4. Check whether the tool gives timestamps, so you can verify a line in seconds, and whether it says when something was guessed from speech.

I haven't tested the named tools in this space. I tested a method (captions into a model) against another (watching the video). If a tool fails your check, that is the likely reason.

Limits of my test

• Three clips, one run per setup. Language model output varies, so small differences mean little. The 0 of 6 versus 5 to 6 of 6 gap is large enough that I'm confident about the direction.
• The reference lesson was made by a watching model, so a screen fact the model also missed would not be counted.
• Captions are the cheap option (in my tests the caption-based lesson was cheaper to make than the watching one), which is why many tools use them. For a talk with no code, they are often fine.

What I did about it

Unwatch makes a Full lesson by default: the model watches the video and writes in the video's order, copying code and on-screen text as shown. There is also a Transcript only option that uses half your video time, for talks where nothing important is on screen. It has the weakness described above, so use it for talks, not for coding tutorials.

You can see the result on a real tutorial: the web scraping lesson has the code from the video, with timestamps that link back to the exact moment so you can check any line.