Skip to main content
screen-recorder-analyzer extracts a chronological list of what a user was doing from any screen recording. Combines Whisper audio transcription, Tesseract OCR on keyframes, and GPT action extraction.

GitHub

nometria/screen-recorder-analyzer

PyPI

screen-recorder-analyzer on PyPI

​
Install

# System dependencies (required)
brew install ffmpeg tesseract          # macOS
sudo apt install ffmpeg tesseract-ocr  # Ubuntu

# Install base package
pip install screen-recorder-analyzer

# Install with all OCR and audio engines
pip install screen-recorder-analyzer[full]

​
Usage

# Set API key
export OPENAI_API_KEY=sk-proj-...

# Analyze a recording
screen-analyze demo.mp4

# JSON output (for piping or programmatic use)
screen-analyze demo.mp4 --format json

# Run as REST API
screen-analyze --api --port 8000
# POST /analyze  { "file": "path/to/recording.mp4" }

​
Output format

[
  {
    "id": "1",
    "timestamp": "0:00",
    "tools": ["excel"],
    "action": ["viewing spreadsheet data"]
  },
  {
    "id": "2",
    "timestamp": "0:15",
    "tools": ["hubspot"],
    "action": ["navigating CRM", "viewing contacts list"]
  },
  {
    "id": "3",
    "timestamp": "0:42",
    "tools": ["gmail"],
    "action": ["composing email", "sending email"]
  }
]

​
Pipeline

StepEngineWhat it does
Frame extractionffmpegExtracts keyframes at configurable intervals
OCRTesseract → EasyOCR → GCVReads on-screen text from each frame
TranscriptionWhisperTranscribes audio narration
Action extractionGPT-4Combines OCR + audio → structured actions

​
Optional engines

# EasyOCR + OpenCV (GPU-accelerated secondary OCR)
pip install screen-recorder-analyzer[gpu]

# Google Cloud Vision (fallback OCR engine)
pip install screen-recorder-analyzer[gcp]

​
Use cases

  • User research: Understand what testers are doing in session recordings
  • QA automation: Extract action sequences from recorded test runs
  • Onboarding analysis: See where users get stuck in screen recordings
  • Compliance logging: Generate structured audit trails from recorded sessions