Step 3c: Parse json3 into readable timestamped segments

python3 -c "
import json

with open('$WORK_DIR/transcript.json3') as f:
    data = json.load(f)

for event in data.get('events', []):
    segs = event.get('segs', [])
    if not segs:
        continue
    start_ms = event.get('tStartMs', 0)
    duration_ms = event.get('dDurationMs', 0)
    text = ''.join(s.get('utf8', '') for s in segs).strip()
    if not text or text == '
':
        continue
    s = start_ms / 1000
    e = (start_ms + duration_ms) / 1000
    print(f'[{int(s//60):02d}:{int(s%60):02d} - {int(e//60):02d}:{int(e%60):02d}] {text}')
" > "$WORK_DIR/transcript.txt"

Read $WORK_DIR/transcript.txt to get the full transcript with timestamps.

Fallback: No transcript available

If no subtitles exist at all, inform the user and proceed with visual-only analysis.

Step 4: Download Video and Extract Frames

Step 4a: Download video (720p is sufficient for frame analysis)

yt-dlp -f "bestvideo[height<=720]+bestaudio/best[height<=720]" r
       -o "$WORK_DIR/video.mp4" "$VIDEO_URL"

Step 4b: Get exact duration

DURATION=$(ffprobe -v quiet -show_entries format=duration -of csv=p=0 "$WORK_DIR/video.mp4")

Step 4c: Extract frames using adaptive interval strategy

Choose interval based on video length:

# Example for a 5-20 minute video (interval=20):
ffmpeg -i "$WORK_DIR/video.mp4" -vf "fps=1/20" -q:v 3 "$WORK_DIR/frames/frame_%04d.jpg" 2>&1

For scene-change-detection (software HowTos, UI demos):

ffmpeg -i "$WORK_DIR/video.mp4" r
       -vf "select='gt(scene,0.3)',showinfo" r
       -vsync vfr -q:v 3 "$WORK_DIR/frames/scene_%04d.jpg" 2>&1

Step 4d: Calculate timestamps for each frame

For fixed-interval extraction: frame N has timestamp (N-1) * interval seconds.

frame_0001.jpg -> 0:00
frame_0002.jpg -> 0:20
frame_0003.jpg -> 0:40
...

Step 5: Synchronize Frames with Transcript

For each extracted frame:

1. Calculate the frame's timestamp in seconds
2. Find the transcript segment(s) covering that timestamp
3. Create a synchronized pair: {timestamp, transcript_text, frame_path}

This is done mentally or via a simple lookup — no external script needed.

Step 6: Multimodal Analysis

Step 6a: Read and analyze each frame

Use the Read tool (or view tool) to look at each frame image. For each frame, consider:

• UI elements: Buttons, menus, dialogs, settings panels visible
• Text on screen: Code, labels, error messages, URLs, terminal output
• Diagrams/graphics: Charts, flow diagrams, architecture drawings
• Physical actions: Hand positions, tool usage (for physical HowTos)
• Changes: What changed compared to the previous frame?

Step 6b: Synthesize both channels

For each key moment, combine audio and visual:

Segment [TIMESTAMP]:
  SAID: "Click the blue button in the top right"
  SHOWN: Settings page screenshot, blue "Save" button highlighted
         in top-right corner, cursor pointing at it
  SYNTHESIS: -> On the Settings page, click the blue "Save" button
               in the top-right corner

Step 6c: Identify visual-only information

Flag moments where the visual channel provides information NOT present in audio:

• Specific button names, menu paths, exact UI locations
• Code that is shown but not read aloud
• Error messages visible on screen
• Before/after comparisons

Output Formats

Generate the appropriate format based on the user's request:

Format A: Step-by-Step Guide (most common)

# [Video Title] — Guide

## Step 1: [Action] (00:15)
[Description based on transcript + frame analysis]
> Visual: [What the screen/image shows at this point]

## Step 2: [Action] (00:42)
[...]

Format B: Comprehensive Summary with Visual Anchors

# [Video Title] — Summary

## Overview
[2-3 sentence summary of the entire video]

## Key Sections

### [Section Name] (00:00 - 02:30)
[Summary of this section]
- Key visual: [Description of what's shown]
- Key quote: "[Important spoken content]"

### [Section Name] (02:30 - 05:00)
[...]

## Key Takeaways
- [Takeaway 1]
- [Takeaway 2]

Format C: Technical Detail Analysis

Separate analysis of both channels plus discrepancy detection:

# [Video Title] — Technical Analysis

## Audio Channel Analysis
[What was said, key points, structure]

## Visual Channel Analysis
[What was shown, UI flows, code, diagrams]

## Channel Synchronization
[Where audio and visual complement each other]

## Visual-Only Information
[Important details only visible in frames, not mentioned in speech]

Error Handling & Edge Cases

Cleanup

After analysis is complete, remove temporary files:

rm -rf "$WORK_DIR"

Tips for Best Results

• Software HowTos: Use scene-change detection — UI transitions create clear visual breaks
• Physical HowTos: Use tighter frame intervals (10-15s) — movements are subtler
• Read the transcript first: Identify "interesting timestamps" before extracting frames. Look for phrases like "as you can see here", "let me show you", "on the screen" — these signal important visual moments
• Context-aware frame analysis: When analyzing a frame, always provide the transcript context. The speaker often explains what's about to be shown
• Batch frame reading: Read frames in batches of 8-10 to maintain context across sequential frames and detect visual changes
• Always extract both channels in parallel: Start the video download while processing the transcript to save time