TakeCheck - video acceptance

takecheck

pytest for AI-generated video. Turn “is this take usable?” into an executable spec and a CI gate.

Status: v0.1.0 — published on PyPI. Spec compliance plus frozen / duplicate / blank / flicker / motion / subtitle residue / audio, with a self-contained HTML report.

Why

A 20-minute render should not be judged by scrubbing frames by hand. takecheck reads a takes.yml spec, compares the delivered file against it, and returns pass / fail + the reason — in seconds, on CPU, while the GPU is still rendering.

Generation pipelines fail in ways a thumbnail cannot show: a shot freezes for 300 ms, the camera starts flickering, the reference clip’s burned-in subtitles survive the redraw, the audio track silently disappears, the last frame of segment 3 does not match the first frame of segment 4. Each is cheap to detect and expensive to discover after a 20-minute render — or after stitching the whole film.

Existing tools do not cover this layer: VMAF/DOVER/DVQA score perceptual quality against a reference or a learned aesthetic prior, VideoGen-Eval is an LLM-API research harness, and ComfyUI QC nodes check tensors inside a graph instead of the delivered file.

Install

pip install takecheck

Requires ffmpeg / ffprobe on PATH. No PyTorch, no GPU — numpy is the only numeric dependency.

Quick start

takes.yml:

name: h3-ep12-shot03
expect:
  duration: 5.0
  fps: 24
  frames: 121
  resolution: 1280x720
  audio: required
  subtitles: forbidden
  frozen_max_ms: 200
  black_max_frames: 0
  flicker_max: 0.35
  motion_min: 0.002

takecheck check out/shot03.mp4 --spec takes.yml --json report.json --html report.html

FAIL  h3-ep12-shot03  (shot03.mp4)
  PASS  spec      1280x720 | 24fps | 121f | 5.04s
  FAIL  frozen    1 frozen stretch(es) over 200ms: 0.00-0.83s (longest 833ms)
  PASS  duplicate 0/120 frames replay an earlier frame (0.0%)
  PASS  blank     0 blank frame(s), luma 12-241
  PASS  flicker   worst 5-frame luma swing 9/255 (3.5% <= 35%)
  PASS  motion    mean frame-to-frame motion 0.0412 >= 0.0020
  PASS  subtitle  no persistent text band (row contrast 1.28 < 2 on 0% of frames)
  PASS  audio     peak -3.2 dB, -18.4 LUFS, 0% silent

Add --html report.html for a self-contained report: a contact sheet with failing moments outlined in red, every measured value, and the raw JSON inline. No external assets, so it can be emailed or attached to a PR as a single file.

Exit codes: 0 pass, 1 fail, 2 bad spec / unreadable file.

What the checks measure

Every check reports the numbers behind its verdict in --json and --html, not just PASS/FAIL.

| Check | Signal | Fails when | |---|---|---| | frozen | pHash Hamming distance between consecutive frames | ≤ 3 for longer than frozen_max_ms | | duplicate | repeated frame hashes, ignoring neighbours, blanks and freezes | more than 1% of frames replay an earlier one | | blank | mean luma per frame | below 16 or above 240, more than black_max_frames | | flicker | peak-to-peak luma inside a 5-frame window | above flicker_max (fraction of 0–255) | | motion | mean absolute frame-to-frame difference | below motion_min | | subtitle | busiest row's horizontal activity ÷ that band's median row | above 2.0 on ≥ 30% of frames, when subtitles: forbidden | | audio | peak dBFS, EBU R128 loudness, silent fraction | peak ≥ −0.1 dBFS, or more than 60% silent |

Three design choices worth knowing:

• One defect, one check. A cut to black is reported by blank only; duplicate skips blank frames, and flicker skips windows containing them. A freeze is reported by frozen, and duplicate ignores frames a freeze already explains. A missing audio track is reported by the spec check, so audio stays quiet about it instead of failing twice for one problem.
• motion is exempt on purpose. It is a whole-clip aggregate ("does this clip move at all"), not an event with a timestamp, so it may legitimately co-fire with a local check: a fully black clip really is both blank and motionless.
• Thresholds are calibrated, not guessed. motion_min defaults to 0.002 because a normally moving clip measures ≈ 0.009 and a still render ≈ 0.000 on the bundled fixtures. Real footage differs — set the values in takes.yml for your own pipeline.
• Deliberately slow shots read as still. A clip slowed down 8x measures mean_motion ≈ 0.0010 and will fail the default motion_min of 0.002. If your pipeline uses slow motion or static compositions on purpose, lower motion_min (or set it to 0) in takes.yml.

Known limitations

• The subtitle check has only been validated on synthetic footage. Its fixtures use a hard white bar on black. Real subtitle ghosts are semi-transparent and blurred, and a scene that legitimately contains text in the lower third will trip the check. Treat the default threshold as a starting point, not a calibrated value.
• Loudness is reported, not enforced. Only clipping and excessive silence fail the audio check; a quiet or hot mix is recorded under measured for you to judge.
• Long clips are untested. Decoding is streamed, so memory should stay flat, but a 20-minute render has not been benchmarked.
• Grading cannot judge content. Pose, framing, expression and prop replacements are semantic; a high score only means the picture changed substantially relative to its source.

Replacement grading (source comparison)

The eight mechanical checks answer "is this file broken?". They cannot answer "did the replacement happen?" — a take that was never replaced is a perfectly healthy video, because it is the source clip. grade and batch compare the output with its source and turn the difference into a replacement-completeness score (0–100) and a band. Neither is a quality verdict; the bands name the next action, and the final call stays with a person.

# one clip
takecheck grade out/shot03.mp4 --source source/shot03.mp4 --spec profiles/captain-america.yml

# a whole directory, grouped by band
takecheck batch 输出视频/ --source-dir 原料/参考视频/ \
    --spec profiles/captain-america.yml --json grades.json

Signals measured against the source (decoded to 160x90, one ffmpeg pass each):

| Signal | What it catches | |---|---| | zero_frame_ratio | frames byte-for-byte identical to the source — the strongest "never replaced" evidence | | mean_diff / changed_ratio | how much of the picture changed | | motion_corr / motion_ratio | whether the action survived or was flattened to a still | | edge_corr | whether the outline is locked to the source | | frame_count_delta | a generator inventing frames |

Bands are configurable in the spec's grade block. Defaults:

| Score | Band | Meaning | |---|---|---| | 0–29 | 直通 | identical to source; auto-reject, no human review needed | | 30–49 | 重跑 | clearly not done; rework | | 50–69 | 待看 | uncertain; inspect frame by frame | | 70–100 | 候选 | worth a human look; the tool does not declare it good |

Calibration (2026-11-06, 75 labelled real takes: 23 accepted / 52 rejected):

| Band | Takes | Accepted | Rejected | |---|---|---|---| | 直通 | 21 | 0 | 21 | | 重跑 | 6 | 1 | 5 | | 待看 | 11 | 2 | 9 | | 候选 | 37 | 20 | 17 |

The 0–29 band is pure: not one accepted take was auto-rejected. But the 70+ band still contains 17 rejected takes — a high score means "look at this", never "ship this". Rejections about pose, framing, expression or a missing prop cannot be settled by pixel statistics; those still need a human to view every frame.

Semantic observation (experimental)

Pixel statistics cannot tell whether the right thing changed. On the 75-take calibration, every numeric signal overlaps between accepted takes and takes rejected for pose / framing / prop errors — which is why the 70+ band is only ~54% pure. No weight tuning fixes that; the missing dimension is semantic observation.

src/takecheck/observation.py defines the contract for a pluggable observer (a local VLM, a hosted agent, or a human). Hard rule learned from real misjudgements: an observer reports facts, never verdicts. "The pot is upside down" is a fact; whether that violates the requirement is decided by the requirement, not by the model. SemanticRule maps observations to concerns against a stated requirement — it still does not declare a take good.

Feasibility experiment (2026-11-06): a local Qwen3.8-27B VL read source/output side-by-side clips. It described pose, orientation, framing and extra objects, but reported the same prop orientation for a rejected take and an unlabelled one — so its output is evidence, not judgement. The first run was invalid because a forced ScaleTo distorted the widescreen pair; it was redone without scaling. See scripts/research/README.md. Known blind spot: the score averages over the whole clip, so a long take where only a short section was replaced is diluted. On the 239-take project run, three 122-frame clips that correctly replaced only their first ~20 frames scored 29/100 and landed in 直通. Use grade on a cut-down section, or read the per-clip zero_frame_ratio, before trusting a low score on a long merged take.

Frame-to-frame coherence (observation, not a score)

Every frame can differ strongly from the source and still be unusable if the frames were generated independently and stitched — the result jitters when played. coherence measures that, as a separate observation:

takecheck coherence out.mp4 --source source.mp4 [--json coh.json] [--all]

It reports the mean frame-to-frame jump of the output relative to the source (jump_ratio). Absolute values depend on shot content, so only the ratio is meaningful. Calibration on real takes (2026-11-07):

| Take | jump_ratio | |---|---| | continuously rendered A/B versions | 1.117 / 1.116 | | B2 probe clip | 1.284 | | older 段19 take | 1.176 | | frame-by-frame stitched (unusable) | 1.651 |

Continuously rendered takes in this calibration all fall in 1.10–1.32x; the stitched clip is the only one above 1.5x. The range is a summary of the rows above, not a claim about unlisted samples.

A ratio above 1.5 prints a warning to play the clip. This never changes the score or the band — it is an observation layered beside them, so a high 候选 score can no longer be mistaken for "playable". grade also prints the coherence line and includes it in its JSON output.

The signal is deliberately kept out of CompareMetrics / _score_parts; a unit test guards that separation. Windows GBK consoles cannot encode decorative characters, so warnings use plain ASCII-safe text. Blind spot confirmed by a real failure (2026-11-07): a run that generated each frame separately and stitched them was graded 94/100 (候选) — the highest of three candidates — while the two continuously-rendered versions scored 71 and 72. The stitched clip is unusable (frame-to-frame jitter, motion_ratio 0.10), but every frame differs strongly from the source, which is exactly what the score rewards. The score measures "how much changed", not "whether the frames are coherent". Never read a high 候选 score as a substitute for playing the clip. The same run also showed A/B prompt-timing variants differing by only 1 point, independently reproducing the human observation that both versions look about the same. 239-take project run (2026-11-06): 直通 35 / 重跑 31 / 待看 39 / 候选 133, one clip unscored because its filename had no matching source segment number. No clip labelled 合格 or 定稿 appeared in 直通 (zero false rejects).

inspect prints the per-frame table for the older appearance-contract experiment. That colour-ratio approach was superseded: on real footage 12/23 accepted takes and 40/52 rejected takes both had almost no target colour, so the signal could not separate an empty shot from a failed replacement. It is kept for reference, not as the recommended judgement path.

Roadmap

• [x] v0.0.1 — spec format, ffprobe compliance check, CLI, takecheck selftest
• [x] v0.1.0 — frozen / duplicate / blank / flicker / motion / subtitle / audio, HTML report, on PyPI
• [x] ComfyUI node — drop it after a sampler, branch on pass
• [x] v0.2.0 — source-comparison grading, calibrated on 75 labelled real takes
• [ ] head/tail continuity
• [ ] v0.3.0 — pipeline lint (custom-node provenance, model hash pinning)
• [ ] v0.4.0 — pluggable VLM judge

ComfyUI

comfyui/ holds a node that runs these checks on the IMAGE batch you just generated. It reuses this package — no check logic is duplicated — and runs on the CPU while your GPU keeps sampling.

cmd /c mklink /J "<ComfyUI>\custom_nodes\ComfyUI-TakeCheck" "<repo>\comfyui"
"<ComfyUI>/../python/python.exe" -m pip install takecheck

Restart ComfyUI; the node appears under video/validation → TakeCheck. Outputs are pass (BOOLEAN), verdict, report (path) and detail (the failing checks in one line). See comfyui/README.md for the spec options, and run python comfyui/TakeCheck.py to self-test without ComfyUI.

Two things to know about the node:

• flicker is less sensitive there than in the CLI. ComfyUI hands over IMAGE as clamped RGB, so a take whose brightness was pushed past 0/255 in YUV arrives with the swing already squashed. A real exposure flicker is still caught; run the CLI on the saved file if you depend on this check.
• A malformed spec raises an error rather than returning "fail", so you can tell a broken config from a bad take.

Test media

testmedia/synthetic/ holds seven tiny clips that each reproduce exactly one defect — frozen frames, duplicate frames, blank frames, flicker, missing audio, burned-in subtitles — plus good.mp4 as the control that must always pass. Rebuild them with:

python scripts/make_testmedia.py

The script also verifies each clip against its expected profile (duration, frame count, audio) and runs in CI, so the fixtures cannot silently rot. A second script asserts the verdict for each one:

python scripts/check_synthetic.py

It fails if good.mp4 trips any check, or if a defect clip stops being caught — so a threshold change cannot quietly break detection.

Real footage never goes in the repository: testmedia/real/ is git-ignored.

Development

pip install -e ".[dev]"
pytest -q
takecheck selftest            # builds synthetic clips in a temp dir; needs ffmpeg
python scripts/check_synthetic.py

Note for contributors on Windows: drawtext needs an explicit fontfile=, otherwise ffmpeg aborts with a Fontconfig error instead of drawing anything. The test helpers search a list of common font paths for this reason.

中文简介

takecheck 是给 AI 生成视频用的验收层：用 takes.yml 声明一条片子该满足什么（时长、帧率、 无静帧、无字幕残影、音轨完整），一条命令返回通过与否和原因，可直接当 CI 门禁。 全程 CPU、秒级，能与渲染并行，不占显存。

License

MIT.