Audio/Video Processing Skill

You are an expert audio/video engineer with deep mastery of ffmpeg and ffprobe. You produce correct, efficient, copy-safe shell commands with clear explanations.

This skill covers a wide range of audio/video tasks organized by sections below. To know only the features offered by this skill (without knowing the specific commands), read the features.md which is also organized by sections. features.md also mentions several use-cases for which different sections of this skill can be used.

STEP 0 — Tool Availability Check (ALWAYS RUN FIRST)

Before generating any commands, verify the required tools are installed. Run or instruct the agent to run:

CODEBLOCK0

Windows: Run these in PowerShell. head is not available — use ffmpeg -version and read the first line manually, or pipe to Select-Object -First 1.

If ffmpeg/ffprobe is NOT installed:

Stop immediately and provide platform-specific installation instructions:

macOS

Linux (Ubuntu/Debian)

Linux (RHEL/Fedora/CentOS)

Windows

Docker (any platform)

Do not proceed with any other steps until ffmpeg is confirmed installed.

STEP 1 — Media Analysis (Always Probe Before Processing)

Always inspect input files before building commands. Load references/ffprobe-analysis.md for full probe patterns.

Quick probe

Key facts to extract before any operation:

• - Video: codec, resolution (WxH), fps, bitrate, pixel format, color space
• Audio: codec, sample rate, channels, bitrate, layout (stereo/5.1/etc.)
• Container: format, duration, total size, nb_streams
• Subtitles/attachments: subtitle streams, embedded fonts

STEP 2 — Command Construction Rules

Load references/ffmpeg-flags.md for the complete flag reference.

Universal Safety Rules

1. 1. Never overwrite without explicit -y flag — use -n to skip existing files by default
2. Always specify -map for multi-stream operations — prevents silent stream loss
3. Prefer -c copy (stream copy) when no re-encoding is needed — faster, lossless
4. Use -crf for quality-based encoding, -b:v for bitrate-based — never both together
5. Validate output: run ffprobe on output to confirm expected streams
6. Use -progress pipe:1 for scriptable progress tracking
7. Set -loglevel warning in production scripts to suppress info spam
8. Always use -movflags +faststart for web MP4s (moves moov atom to front)
9. Quote all file paths — spaces and special characters will break commands
10. Use -ss before -i for fast seek (keyframe seek), after -i for accurate seek

Command Template

ffmpeg \
  [global options] \
  [input options] -i "input_file" \
  [output options] \
  "output_file"

SECTION A — Format Conversion & Transcoding

Load references/codecs-containers.md for codec compatibility matrix.

A1. Video Transcoding

MP4 → High-quality H.264 (universal compatibility)

ffmpeg -i "input.mov" \
  -c:v libx264 -crf 23 -preset slow \
  -c:a aac -b:a 192k \
  -movflags +faststart \
  "output.mp4"

INLINECODE20: lower = better quality/larger file. preset: ultrafast→veryslow (speed vs compression).

H.265/HEVC (better compression, modern devices)

ffmpeg -i "input.mp4" \
  -c:v libx265 -crf 28 -preset slow \
  -c:a aac -b:a 128k \
  -tag:v hvc1 \
  "output_h265.mp4"

INLINECODE22 required for Apple device compatibility.

AV1 (best compression, slowest encode)

VP9 / WebM (web-optimized, royalty-free)

Hardware-accelerated encoding (macOS VideoToolbox)

Hardware-accelerated encoding (NVIDIA NVENC)

Hardware-accelerated encoding (Intel QSV)

A2. Container Remuxing (No Re-encoding)

A3. Image Sequence → Video

A4. Video → Image Sequence

# Extract all frames
ffmpeg -i "input.mp4" "frames/frame_%04d.png"

# Extract at specific fps (1 frame per second)
ffmpeg -i "input.mp4" -vf fps=1 "frames/frame_%04d.jpg"

# Extract keyframes only
# Note: -vsync vfr is deprecated since ffmpeg 5.1; use -fps_mode vfr on newer builds
ffmpeg -i "input.mp4" -vf "select=eq(pict_type\,I)" -fps_mode vfr "keyframe_%04d.jpg"

SECTION B — Audio Processing

B1. Audio Extraction

B2. Audio Conversion

B3. Audio Normalization (EBU R128 Loudness)

B4. Audio Filters

B5. Advanced Audio Processing

# Trim silence (remove leading/trailing silence)
ffmpeg -i "input.wav" -af "silenceremove=start_periods=1:start_silence=0.1:start_threshold=0.01:stop_periods=-1:stop_silence=0.1:stop_threshold=0.01" "output.wav"

# Speed change (pitch preserved using atempo, supports 0.5–2.0 range)
ffmpeg -i "input.wav" -af "atempo=1.5" "output.wav"

# Pitch shift (without tempo change, shift up by 2 semitones)
ffmpeg -i "input.wav" -af "asetrate=44100*1.122,aresample=44100,atempo=0.891" "output.wav"

# Generate waveform data (stats/loudness info)
ffprobe -v quiet -of json -show_streams "input.wav"
ffmpeg -i "input.wav" -af "astats" -f null - 2>&1

# Generate spectrogram PNG
ffmpeg -i "input.wav" -lavfi "showspectrumpic=s=1024x512:mode=combined" -update 1 "spectrogram.png"

SECTION C — Video Editing

C1. Trimming & Cutting

C2. Concatenation

C3. Scaling & Resolution

C4. Frame Rate Conversion

C5. Rotation & Flipping

C6. Cropping

C7. Overlays & Watermarks

C8. Color Correction & Grading

# Brightness, contrast, saturation
ffmpeg -i "input.mp4" \
  -vf "eq=brightness=0.06:contrast=1.2:saturation=1.5:gamma=1.0" \
  -c:a copy "output_color.mp4"

# LUT (Look-Up Table) color grading
ffmpeg -i "input.mp4" \
  -vf "lut3d=file='film_emulation.cube'" \
  -c:a copy "output_graded.mp4"

# Curves adjustment (S-curve for punch)
ffmpeg -i "input.mp4" \
  -vf "curves=r='0/0 0.5/0.6 1/1':g='0/0 0.5/0.52 1/1':b='0/0 0.5/0.44 1/1'" \
  -c:a copy "output_curves.mp4"

# Hue/saturation shift
ffmpeg -i "input.mp4" \
  -vf "hue=h=30:s=1.2" \
  -c:a copy "output_hue.mp4"

SECTION D — Subtitles & Captions

D1. Subtitle Operations

# Burn subtitles into video (hard subtitles)
ffmpeg -i "input.mp4" -vf "subtitles=subs.srt" -c:a copy "output_burned.mp4"

# Add SRT as soft subtitle track (selectable, not burned)
ffmpeg -i "input.mp4" -i "subs.srt" \
  -c copy -c:s mov_text \
  -metadata:s:s:0 language=eng \
  "output_soft_subs.mp4"

# Add ASS/SSA subtitles to MKV
ffmpeg -i "input.mkv" -i "subs.ass" \
  -c copy -c:s copy \
  "output.mkv"

# Extract subtitle track
ffmpeg -i "input.mkv" -map 0:s:0 "subtitles.srt"

# Convert SRT → ASS
ffmpeg -i "subs.srt" "subs.ass"

SECTION E — Thumbnails & Screenshots

E1. Thumbnail Generation

# Single thumbnail at specific timestamp
ffmpeg -i "input.mp4" -ss 00:00:05 -vframes 1 "thumbnail.jpg"

# High-quality thumbnail (PNG)
ffmpeg -i "input.mp4" -ss 00:00:05 -vframes 1 -q:v 2 "thumbnail.png"

# Thumbnail at best frame (highest quality scene)
ffmpeg -i "input.mp4" -vf "thumbnail,scale=640:-1" -vframes 1 "best_thumb.jpg"

# Sprite sheet / contact sheet (multiple thumbnails in a grid)
ffmpeg -i "input.mp4" \
  -vf "fps=1/10,scale=160:90,tile=10x10" \
  -vframes 1 \
  "sprite_sheet.jpg"

# Thumbnail every 10 seconds
ffmpeg -i "input.mp4" -vf "fps=1/10,scale=320:-1" "thumbs/thumb_%04d.jpg"

SECTION F — Streaming & Adaptive Bitrate

F1. HLS (HTTP Live Streaming)

F2. DASH (Dynamic Adaptive Streaming over HTTP)

F3. RTMP Live Streaming

# Stream to RTMP endpoint (YouTube, Twitch, etc.)
ffmpeg -re -i "input.mp4" \
  -c:v libx264 -preset veryfast -b:v 4500k -maxrate 4500k -bufsize 9000k \
  -pix_fmt yuv420p -g 60 \
  -c:a aac -b:a 160k -ar 44100 \
  -f flv "rtmp://live.twitch.tv/app/YOUR_STREAM_KEY"

# Screen capture + stream (macOS)
ffmpeg \
  -f avfoundation -framerate 30 -i "1:0" \
  -c:v libx264 -preset veryfast -b:v 4500k \
  -c:a aac -b:a 160k \
  -f flv "rtmp://YOUR_RTMP_ENDPOINT"

SECTION G — Screen & Webcam Capture

G1. Screen Recording

G2. Webcam Capture

# macOS
ffmpeg -f avfoundation -framerate 30 -video_size 1280x720 -i "0" \
  -c:v libx264 -crf 23 "webcam.mp4"

# Linux (v4l2)
ffmpeg -f v4l2 -framerate 30 -video_size 1280x720 -i /dev/video0 \
  -c:v libx264 -crf 23 "webcam.mp4"

# Windows (dshow)
ffmpeg -f dshow -i video="Integrated Camera" \
  -c:v libx264 -crf 23 "webcam.mp4"

SECTION H — GIF & Animated Images

H1. Video → GIF (High Quality)

H2. GIF → Video

H3. WebP Animation

ffmpeg -i "input.mp4" -vf "fps=24,scale=480:-1:flags=lanczos" \
  -loop 0 -preset default -an -fps_mode passthrough \
  "output.webp"
# Note: -vsync 0 is deprecated since ffmpeg 5.1; use -fps_mode passthrough

SECTION I — Batch Processing & Scripting

I1. Batch Convert (Shell)

I2. Progress Monitoring

I3. Two-Pass Encoding (Precise Bitrate Control)

# Pass 1 (analysis only)
ffmpeg -y -i "input.mp4" \
  -c:v libx264 -b:v 2M -pass 1 -an -f null -

# Pass 2 (encode with target bitrate)
ffmpeg -i "input.mp4" \
  -c:v libx264 -b:v 2M -pass 2 \
  -c:a aac -b:a 192k \
  "output_2pass.mp4"

SECTION J — Advanced Filtergraphs

J1. Complex Filter Examples

J2. Audio/Video Sync Repair

# Fix audio delay (audio is 500ms late)
ffmpeg -i "input.mp4" -itsoffset -0.5 -i "input.mp4" \
  -map 1:v -map 0:a -c copy "output_synced.mp4"

# Add audio delay (audio is 500ms early)
ffmpeg -i "input.mp4" -itsoffset 0.5 -i "input.mp4" \
  -map 0:v -map 1:a -c copy "output_synced.mp4"

SECTION K — Quality Analysis & Verification

K1. Quality Metrics

K2. Output Validation Checklist

After every encode, verify:

CODEBLOCK47

SECTION L — Platform-Specific Presets

L1. Web (HTML5 video)

L2. YouTube Upload

L3. Instagram / TikTok (Vertical 9:16)

L4. Apple ProRes (Post-production)

L5. Discord / Messaging (8MB limit)

# Calculate target bitrate for 8MB / duration
DURATION=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "input.mp4")
TARGET_KBPS=$(python3 -c "print(int(8*1024*8 / $DURATION - 128))")
ffmpeg -i "input.mp4" \
  -c:v libx264 -b:v "${TARGET_KBPS}k" -pass 1 -an -f null - && \
ffmpeg -i "input.mp4" \
  -c:v libx264 -b:v "${TARGET_KBPS}k" -pass 2 \
  -c:a aac -b:a 128k \
  "discord_clip.mp4"

SECTION M — Video Stabilization

Stabilize shaky footage from handheld cameras, drones, or action cameras. Uses a two-pass approach: pass 1 analyzes motion, pass 2 applies correction.

Check libvidstab availability

CODEBLOCK53

If not available: Install a full-featured ffmpeg build.

• - macOS: brew install ffmpeg-full or INLINECODE24
• Linux: sudo apt install ffmpeg (Ubuntu 20.04+ includes libvidstab)
• Windows: Download from https://www.gyan.dev/ffmpeg/builds/ (full build includes libvidstab)

Pass 1 — Detect motion (analyze shakiness)

Pass 2 — Apply stabilization

One-liner (both passes)

Windows (PowerShell)

Parameter tuning

Goal	Settings
Mild stabilization	INLINECODE26
Heavy stabilization

shakiness=8:smoothing=50:zoom=5 |

SECTION N — 360° / VR Video

Handle 360° footage from cameras like GoPro Max, Insta360, Ricoh Theta. Convert between projections and inject spherical metadata so platforms (YouTube VR, Facebook 360) recognize the video correctly.

Check v360 filter availability

Convert equirectangular → cubemap (3x2 layout)

Convert cubemap → equirectangular

Inject spherical metadata (YouTube VR / Facebook 360)

Note: For full YouTube VR compliance, use Google's spatial-media tool after encoding to inject the proper XMP metadata atom. FFmpeg metadata injection is a best-effort fallback.

Reframe / extract a flat view from 360° video

macOS / Linux / Windows

SECTION O — HDR / Color Science

Handle HDR10 content from modern phones (iPhone, Android) and cameras. Convert HDR to SDR for web delivery, or tag files with correct color metadata.

Check color filters availability

If zscale is missing: Install with --enable-libzimg. macOS: brew install ffmpeg-full. Linux: sudo apt install ffmpeg (20.04+). Windows: use full build from gyan.dev.

HDR10 → SDR tone mapping (real HDR source required)

Colorspace conversion (SDR bt2020 → bt709)

Encode HDR10 output (for archival/editing)

Tag existing file with correct color metadata (no re-encode)

Tone mapping algorithm comparison

Algorithm	Description	Best for
INLINECODE34	Filmic, preserves highlights	General use
INLINECODE35

Simple, global | Fast previews |

Windows / Linux / macOS

SECTION P — Advanced Streaming

SRT (Secure Reliable Transport) — low-latency streaming

SRT handles packet loss and network jitter, making it ideal for unstable connections (cellular, satellite, long-distance contribution).

CODEBLOCK68

Windows: Same commands work in PowerShell. SRT is included in full ffmpeg builds from gyan.dev.

Multi-endpoint restreaming (tee muxer)

Stream to YouTube, Twitch, and Facebook simultaneously from one ffmpeg process:

CODEBLOCK69

Rolling window DVR recording

Continuous loop recording — keeps only the last N segments (useful for security cameras, broadcast monitoring):

CODEBLOCK70

SECTION Q — Repair & Recovery

Recover from corrupt or truncated files

VFR → CFR conversion (fix audio sync drift)

Variable frame rate (VFR) footage — common from phones and screen recorders — causes audio sync drift in editing software. Convert to constant frame rate (CFR):

CODEBLOCK72

Fix audio/video sync offset

Rebuild broken index / moov atom

Windows / Linux / macOS