Verification

Adversarial review — ComfyUI-Cinematographer

Reviewer: adversary (task-3) · Date: 2026-10-29 · Deadline: 2026-10-30 09:00 PT Stance: falsify, not confirm. A review that finds nothing is a failed review. Write scope: this file only. Nothing under app/ or any top-level .md was modified. All mutation testing was done on a throwaway copy at /tmp/adv-cine; the working tree was verified unchanged afterwards.

0. Bottom line, up front

Not safe to submit as-is. One CRITICAL defect breaks the entry's headline deliverable: installed exactly as app/README.md documents, the ComfyUI nodes do not load at all. A judge who follows the README gets an empty category, not six nodes.

The verification story is much stronger than the average competition entry — the differential check, the parity check and the workflow validator are all real, all re-runnable, and all caught most of what I threw at them. But the two headline numbers ("289 differential checks", "69 unit tests, every expected value hand-computed") are both blind to the same class of defect: 12 engine constants can be changed to wrong values and every check in the repository still passes. And three claims stated as fact are contradicted by a runnable check.

Fix the install path, the dead command in verification/README.md, and the SamplerCustomAdvanced slot count before submitting. Everything else is survivable.

1. Verdicts

2. How to reproduce everything in this document

cd /Users/nathaniel/companies/magieva/GitHub/higgsfield-comfyui-os
PY=/tmp/cine-venv/bin/python          # pytest is not installed for system python3

# A — the differential check
PYTHONDONTWRITEBYTECODE=1 $PY verification/differential_check.py verification/go-reference-dump.json

# B — the unit suite
cd app && PYTHONDONTWRITEBYTECODE=1 $PY -m pytest tests/ -q -p no:cacheprovider && cd ..

# D — panel parity, panel logic, panel in a real browser
PYTHONDONTWRITEBYTECODE=1 $PY app/web/parity_check.py
node app/web/tools/smoke_test.mjs
(cd app/web && python3 -m http.server 8802 &) ; sleep 1
node app/web/tools/browser_check.mjs http://127.0.0.1:8802/ /tmp/panel-adv.png

# E — workflow structure
PYTHONDONTWRITEBYTECODE=1 $PY app/workflows/validate_graphs.py

# G — the install path, simulated
mkdir -p /tmp/fakecomfy/custom_nodes
cp -R . /tmp/fakecomfy/custom_nodes/ComfyUI-Cinematographer
ls /tmp/fakecomfy/custom_nodes/ComfyUI-Cinematographer/__init__.py   # -> No such file

Method note — a trap that invalidates naive mutation testing here. Stale __pycache__ silently masks mutations: my first mutation run reported 7 false "MISSED" results because app/cinematographer/__pycache__ was being reused. Every mutation result below was produced with PYTHONDONTWRITEBYTECODE=1 and __pycache__ deleted between runs. If you re-run any mutation test in this document without doing both, you will get wrong answers.

3. Claim A — the 289-check differential result

Baseline reproduced exactly.

$ PYTHONDONTWRITEBYTECODE=1 /tmp/cine-venv/bin/python verification/differential_check.py verification/go-reference-dump.json
differential checks run: 289
ALL FIXTURES AGREE with the original Go implementation.
EXIT=0

I instrumented the counter to decompose 289. It is not 289 field comparisons — it is 289 heterogeneous units:

That is a fair count of rule-slots checked; it is not a count of assertions. The README's §1 "What is compared" table is honest about the surfaces. The headline is not.

Falsification attempt 1 — can the check fail at all?

Yes. I copied the tree to /tmp/adv-cine and injected 30 defects one at a time. 19 of 30 were caught, with large failure counts:

This is a genuinely strong check on value fidelity. It is not a rubber stamp.

Falsification attempt 2 — what does it miss? (SUCCEEDED)

11 of 30 injected defects were MISSED, and they are all the same kind: a threshold or a boundary constant moved slightly, where no fixture sits near the boundary.

Root cause, measured. The fixture set has near-zero boundary diversity. For the 9 framing fixtures, in_frame has exactly one distinct value (True) and level has exactly one distinct value (0 / True). No fixture exercises a boundary, so moving a boundary is invisible. The check is strong on arithmetic and blind to thresholds.

This matters because thresholds are where the cinematography actually lives. 0.4 <= frac <= 0.75 is the entire medium-shot rule; 0.12 is the entire rule-of-thirds tolerance. Both can be wrong and the entry's strongest claim still prints 289 / ALL FIXTURES AGREE.

Falsification attempt 3 — does it compare everything it says it compares? (SUCCEEDED)

It does not. Three surfaces are compared only partially, and the skipped fields really do differ:

SceneFramingExtras — differential_check.py:233-246 compares rule, pass, and (except for size_importance) value. It never compares target, weight, or correction.

Go     extras_big[0].correction = "a prop out-sizes the key subject — resize/reposition so the subject dominates"
Python extras_big[0].correction = "a prop out-sizes the key subject - resize/reposition so the subject dominates"

Em dash vs hyphen. app/cinematographer/scene.py:203 uses -; 01-SPEC.md:79 and the Go source both use —. The framing rules get this right (geometry.py:368-371 uses \u2014); the scene rules were rewritten to ASCII and nothing notices.

EvaluateContinuity — differential_check.py:203-210 compares rule, pass, shot_a, shot_b. It never compares detail.

Go     30_degree: "camera angle changed 3° (need ≥30° to avoid a jump cut)"
Python 30_degree: "camera angle changed 3 deg (need >=30 deg to avoid a jump cut)"

Go     180_axis:  "axis crossed — the subject's screen direction flips between the cuts"
Python 180_axis:  "axis crossed - the subject's screen direction flips between the cuts"

Three of the four continuity_go_test details differ. app/cinematographer/continuity.py:121,140-141.

size_importance — differential_check.py:243-244 counts the slot in the 289, then skips the value. So 2 of the 289 headline "checks" compare pass and nothing else. The skip is disclosed in §1 of verification/README.md; the counting of a skipped slot as a check is not.

Consequence: verification/README.md:12-13 claims the port "reproduces the original Go cinematography implementation value-for-value". For the numeric values, that is true and well-evidenced. For the report's user-facing strings — which are what the panel and the ComfyUI nodes actually display — it is false for 2 of the 14 rules and for both continuity rules.

Falsification attempt 4 — is eyeline verified at all? (SUCCEEDED — it is not)

eyeline (rule 12 of 14) is emitted only when len(scene.characters) >= 2 (app/cinematographer/scene.py:231). Both differential extras fixtures have exactly one character, so eyeline is never produced on the Go side and never compared:

extras_big -> [('size_importance', False, 10.125), ('foreground_occlusion', True, 0)]
extras_ok  -> [('size_importance', True, 0),      ('foreground_occlusion', True, 0)]

verification/README.md §4 lists five UNVERIFIED items. eyeline is not among them. One of the 14 rules the entry counts as verified has no differential evidence and no disclosure.

4. Claim B — 69 unit tests, hand-computed expected values

Baseline reproduced.

$ cd app && PYTHONDONTWRITEBYTECODE=1 /tmp/cine-venv/bin/python -m pytest tests/ -q -p no:cacheprovider
69 passed in 0.14s

Are the values first-principles or snapshots?

First-principles. I re-derived the reference-frame numbers by hand from 01-SPEC.md §4.3:

vfov = 2*atan((36/(16/9))/(2*35)) = 0.563154 rad
f    = 1/tan(vfov/2)              = 3.45679012345679
top  = (0, 1.8, 0) -> d=(0, 0.144, -3) -> ndcY = (0.144/3)*f = 0.048*f = 0.1659259
headroom = 1 - 0.1659259 = 0.8340741
base = (0, 0, 0)   -> d=(0,-1.656, -3) -> ndcY = -0.552*f = -1.9081481
shot_size = (0.1659259 - (-1.9081481))/2 = 1.0370370

test_framing.py:38-39 asserts 0.834 and 1.037 at abs=1e-3. The arithmetic is sound and the docstring derivation is correct. This claim is honest about provenance.

But does the suite constrain the engine? (Falsification SUCCEEDED)

11 of 24 injected defects survive all 69 tests. The suite is a value test, not a contract test — it pins a handful of reference frames and leaves the rest of the parameter space open. See the table in §3: the pytest column shows the misses.

Two of the misses are notable because the tests look like they should catch them:

• test_level_rule_honours_an_intended_dutch uses roll=8, dutch=0, which fails under both < 1 and < 2. It cannot distinguish the two tolerances.
• test_thirds_rule_passes_when_the_subject_sits_on_a_third places the subject exactly on -1/3, which passes under both 1/3 and 1/4 third-lines.

A hand-computed constant that is wrong

app/tests/test_framing.py:13:   F = 3.4567701  # 1/tan(vfov/2) for the reference camera
verification/README.md:109:     f = (2*focal)/(sensor/aspect) = 70/20.25 = 3.45679012345679

The test's own constant is wrong in the 5th decimal (correct: 3.4567901). It is masked by the abs=1e-3 tolerance, so it changes no result — but it is a miscomputed "hand-computed" value in the file that claim B rests on, and it contradicts the verification README's own arithmetic.

5. Claim C — solver convergence honesty

Verdict: CONFIRMED. I could not falsify it.

I swept 1800 finite solver calls — heights [0.0, -1.0, 0.01, 0.1, 0.5, 1.0, 1.8, 2.5, 10.0, 1e6], all yaws, 10 intents, 3 shot types — and checked five invariants:

The differential check independently asserts converged == report.overall_pass for all 18 Go fixtures (differential_check.py:180-181), and test_converged_flag_agrees_with_the_report covers 9 more combinations. This claim is well-evidenced.

Caveats worth knowing (not defects)

1. converged=True does not mean the frame is good. 24 converged frames still have a failing non-weighted rule. Example: (height=1.8, 'medium', center=True) converges in 1 iteration with headroom=0.912 (band [0.10, 0.45], weight 0.7) and the subject's feet projecting to ndcY = -1.012 — off-frame. This is correct per spec: in_frame checks the eye anchor only, and framing.py:102-103 documents that a close-up crops the body. But the panel's success text (app/web/app.js:676-678) says "the frame now passes every rule with weight ≥ 0.8", which is literally true and rhetorically stronger than the reality. A judge who probes it will find a "converged" frame with a badly wrong headroom.
2. The Go failure mode the entry sells is not demonstrated by any fixture. All 18 Go autoframe fixtures end on a passing frame (go_pass = True for 18/18). So "the Go original returns the 24th iterate unconditionally and nothing guarantees overall_pass" is a true statement about the Go code, but the entry has no captured case of it actually failing. It is a code-reading claim presented next to measured ones.
3. The early exit does engage on every fixture — Python uses 1–3 iterations where Go uses 24, 18/18. That half of the claim is measured.
4. NaN input is unguarded. auto_frame_camera with height=float('nan') raises ValueError: cannot convert float NaN to integer from round3. Not a realistic input; noted for completeness.

6. Claim D — panel parity

Verdict: CONFIRMED. And it is not circular.

$ PYTHONDONTWRITEBYTECODE=1 /tmp/cine-venv/bin/python app/web/parity_check.py
  engine hash   : 24621deb23c66c8f…  (matches live engine)
  [1/3] JS vs frozen fixtures      12/12, 4/4, 7/7, 5/5 OK
  [2/3] Python engine vs fixtures  12/12, 4/4, 7/7, 5/5 OK
  [3/3] Python engine vs JS (live) 12/12, 4/4, 7/7, 5/5 OK
RESULT: PASS — 28 fixtures, 0 mismatches
EXIT=0

The circularity question, answered. It would be circular if all three legs compared the Python engine to itself. They do not:

• Leg [1] compares the JS mirror to frozen fixtures.
• Leg [2] compares the Python engine to the same fixtures.
• Leg [3] runs Python and JS live against each other, field by field.

Legs [1] and [2] share a fixture file, but leg [3] does not go through the fixtures at all. The fixtures are generated by tools/gen_fixtures.py from the Python engine, so a systematic error present in both engines would be frozen in — but a JS-only or Python-only drift cannot hide. I confirmed this empirically: JS mutations are caught (below). The check is a genuine two-engine equivalence proof, not a self-comparison.

JS mutation testing (on /tmp/adv-cine, __pycache__ cleared):

The parity check has the same boundary-blindness as the differential check, for the same reason: the fixtures do not sit near any threshold.

Panel end-to-end, reproduced:

$ node app/web/tools/smoke_test.mjs
PASS — 38/38 checks passed
EXIT=0

$ node app/web/tools/browser_check.mjs http://127.0.0.1:8802/ /tmp/panel-adv.png
PASS — 18/18 browser checks passed
EXIT=0

The browser check really does drive headless Chrome, click Auto-Frame, and sample WebGL pixels. This is the best-verified surface in the entry.

7. Claim E — the workflow graphs

Verdict: PARTIALLY CONFIRMED.

The validator is real (falsification FAILED — it caught everything)

$ PYTHONDONTWRITEBYTECODE=1 /tmp/cine-venv/bin/python app/workflows/validate_graphs.py
PASS  reshoot_ltx_crossview.json  (35 nodes, 49 links)
PASS  reshoot_wan22_fun_control.json  (25 nodes, 33 links)
All structural checks passed. This does NOT mean the graphs run: they have never been executed.
EXIT=0

I injected 6 structural defects into a copy and all 6 were caught:

The class types exist and the slot counts are (almost) right

I downloaded the real upstream sources and parsed every class_type in the table with ast (handling both RETURN_TYPES = (...) and the new io.Schema(outputs=[...]) form):

32 of 33 slot counts match upstream exactly. All 33 class types exist. CrossViewWarp (RETURN_TYPES = ("IMAGE", "IMAGE")), LTXICLoRALoaderModelOnly (2), LTXAddVideoICLoRAGuide (3), GetVideoComponents (5), LTXVCropGuides (3) — all correct.

One slot count is wrong, and it falsifies a provenance claim

app/workflows/validate_graphs.py:72:    "SamplerCustomAdvanced": 1,

comfy_extras/nodes_custom_sampler.py:1033-1035:
            outputs=[
                io.Latent.Output(display_name="output"),
                io.Latent.Output(display_name="denoised_output"),
            ],

Upstream declares 2 outputs; the table says 1. The graph only consumes slot 0, so the validator passes and the graph is unaffected — but validate_graphs.py:23-25 says:

"Each count was read from the node source named in the graph's own _README PROVENANCE block, not guessed. If a node pack changes its outputs, this validator is what will notice."

Both graphs' _README repeat the claim ("a table of output-slot counts read from the node sources"). For SamplerCustomAdvanced the count is wrong, and the validator would not notice if that node's slot 1 changed. This is a small defect with an outsized cost: the entry's whole argument is that its provenance claims are checkable, and this one is falsifiable in a single grep.

The graphs are honestly marked

Both _README blocks open with STATUS: NOT VERIFIED. This graph has never been executed. and close with MEASURED: nothing. No timing, no VRAM figure, no output. This is exactly right, and it is the strongest honesty signal in the entry.

8. Claim F — licence-clean, no trademark

Verdict: CONFIRMED.

The only external fetch in the whole entry is three.js from unpkg/jsDelivr in the panel (app/web/app.js:348-349), which app/web/README.md:124-126 discloses and which degrades to a note if the CDN is unreachable. Not a closed API, not a model dependency.

The licence table (LTX 2.3 community, CrossView-Warp Apache-2.0, MoGe-2 open, Wan 2.2 Apache-2.0) is consistent with what the graphs actually load. I could not independently verify the licence terms themselves (see §11).

9. Claim G — README setup works from a clean clone

Verdict: REFUTED. This is the CRITICAL defect.

The pytest path works

$ cd app && PYTHONDONTWRITEBYTECODE=1 /tmp/cine-venv/bin/python -m pytest tests/ -q
69 passed in 0.14s
EXIT=0

The ComfyUI install path does not

app/README.md:33-41 says:

cd ComfyUI/custom_nodes
git clone <this-repo> ComfyUI-Cinematographer
# no dependencies — the engine is pure stdlib

Restart ComfyUI. The nodes appear under the Cinematographer category.

Simulated exactly:

$ ls -a /tmp/fakecomfy/custom_nodes/ComfyUI-Cinematographer
.  ..  00-BREAKDOWN.md  01-SPEC.md  02-DEMO-SCRIPT.md  03-SUBMISSION.md
analysis  app  raw-articles  verification

$ test -f .../ComfyUI-Cinematographer/__init__.py && echo YES || echo NO
NO

ComfyUI's loader requires <pack_root>/__init__.py. From nodes.py:

# nodes.py:2261-2276
elif os.path.isdir(module_path):
    sys_module_name = module_path.replace(".", "_x_")
...
    module_spec = spec_from_file_location(sys_module_name, os.path.join(module_path, "__init__.py"))
    module_dir = module_path
module = importlib.util.module_from_spec(module_spec)
module_spec.loader.exec_module(module)

governance.pack_module_spec (app/governance.py:123-127) is a thin wrapper over spec_from_file_location — it adds no fallback for a missing entry file. And init_external_custom_nodes (nodes.py:2369) iterates os.listdir(custom_node_path) — top level only, no recursion. Reproducing the loader's directory branch against the simulated clone:

entry file: .../ComfyUI-Cinematographer/__init__.py exists: False
LOAD FAILED: FileNotFoundError [Errno 2] No such file or directory:
             '.../ComfyUI-Cinematographer/__init__.py'

The NODE_CLASS_MAPPINGS export lives at app/cinematographer/__init__.py, two levels below the pack root. There is no __init__.py at the repo root and no app/__init__.py. There is also no pyproject.toml at the pack root, so ComfyUI's config_parser.extract_node_configuration finds nothing either (the [tool.comfy] block is at app/pyproject.toml).

Impact: acceptance criterion 8 ("README setup works from a clean clone") fails, and the entry's primary artefact — a ComfyUI custom node pack — is not installable by the documented route. The panel and the tests work, so a judge who only runs those will not notice.

Secondary, same section: app/README.md:47 documents python -m pytest tests/ -q but never says to install pytest. requirements.txt lists it only as a commented-out optional, and pyproject.toml puts it behind [project.optional-dependencies] dev. On a clean machine with system python3:

$ python3 -m pytest tests/ -q
No module named pytest
EXIT=1

Tertiary: acceptance criterion 1 (python -c "import cinematographer") works from app/ but not from the repo root, which is where the README's install section leaves the reader.

10. Claim H — honesty about measured vs. guessed

This is the entry's differentiator, so it gets the harshest treatment. Verdict: PARTIALLY CONFIRMED.

What is genuinely excellent

• Both workflow _README blocks: STATUS: NOT VERIFIED, MEASURED: nothing, an explicit VERIFIED/NOT-VERIFIED split, and 10 enumerated deliberate deltas from the upstream template.
• app/web/README.md:110-120 — a "Mocked, stubbed, or simply not wired" table that says the panel "generates nothing", the subject is synthetic, the viewport is an illustration, and the continuity strip's footage is not real.
• app/README.md:126-135 — an "Honest limitations" section that names the size_importance approximation and the unverified graphs.
• 03-SUBMISSION.md:133-145 — an explicit ledger of claims that may and may not be made, including "38 cinematography rules" ⛔ false and "Beats Higgsfield's Cinema Studio" ⛔ unfalsifiable.
• framing.py:102-103 documents that in_frame checks the face, not the feet.

This is well above the norm. The defects below are therefore more damaging, not less — they are inconsistencies in a document set that otherwise earns its credibility.

Claims stated as fact that a runnable check contradicts or does not back

Spec-vs-code divergence found while auditing

01-SPEC.md:229 says yellow is 45° < |azimuth| ≤ 60°, **or |elevation| > 30°**, and 01-SPEC.md:230 says red is "beyond both". app/cinematographer/reliability.py:75-102 requires both axes to be out for red only when azimuth exceeds 60; an elevation-only violation falls through to red:

The code's behaviour is arguably the more useful one (a 40° elevation is outside the trained envelope), but the spec and the code disagree, and app/web/README.md:118 describes the constants as "taken from the spec".

11. Defect register

Severity: CRITICAL = blocks the submission or breaks the primary artefact · MAJOR = a judge can falsify a headline claim · MINOR = worth fixing, not fatal.

12. Falsification attempts — full log

Note on discarded findings. My first workflow-input audit reported ~15 "UNDECLARED inputs" (clip, text, device, codec, audio_vae, and the CrossViewWarp optional block). These were artifacts of a crude regex extractor that ignored "optional" blocks and old-style INPUT_TYPES. I re-verified each against the real sources: every one is declared. No input-name defect exists. They are recorded here only so a later reader does not rediscover the false positives and report them.

13. What I could not verify

• No live ComfyUI. I could not confirm the graphs run, and I could not confirm that a fixed install layout would load — I verified the failure of the documented layout against ComfyUI's loader source, not against a running instance.
• No Magieva Go checkout. I could not re-run go-harness/zz-cine-dump.go. I verified the Python side against the frozen go-reference-dump.json only. The dump's own provenance is unverifiable from here — if it were fabricated or stale, none of my claim-A work would detect it. This is the single largest hole in the verification chain, and it is inherent to the setup.
• No render. LTX 2.3 + CrossView-Warp + MoGe-2 output quality is unverified, as the entry says.
• The CrossView-Warp ±45° trained range. Taken from the spec and the node pack's docs; not measured. app/web/README.md:118 discloses this.
• Licence terms. I checked which models the graphs load and that no closed API is in the critical path. I did not read the LTX-2.x Community licence, the Wan 2.2 licence, or the MoGe-2 licence to confirm the revenue caps and territory exclusions quoted in the README.
• The competition rubric. Unpublished; nothing can be verified against it.
• Browser coverage. I ran the browser check under Chrome at one viewport against a local python3 -m http.server. I did not test the three.js CDN-offline fallback path, Firefox/Safari, or a mobile viewport.
• MoGeInference and other node input names beyond the flagged set. I verified slot counts for all 33 class types and input names for the nodes that were flagged. I did not exhaustively re-derive every input name for every node from source.
• AutoFrameGroup. Not ported; the entry says so (verification/README.md:66).

14. What I would fix, in order

1. app/README.md install path (CRITICAL). Either add a repo-root __init__.py that re-exports app.cinematographer's mappings, or move cinematographer/ to the repo root, or change the instructions to clone into custom_nodes/ComfyUI-Cinematographer and then symlink app/cinematographer to the pack root. Also move pyproject.toml to the pack root so [tool.comfy] is found. Then verify by actually loading it in ComfyUI.
2. verification/README.md:122 — replace the dead command with python app/web/parity_check.py, and move the row out of UNVERIFIED.
3. validate_graphs.py:72 — "SamplerCustomAdvanced": 2.
4. verification/README.md:12-13 — change "value-for-value" to name the compared surfaces explicitly, and either fix the scene/continuity strings to match Go or disclose the divergence.
5. Add boundary fixtures. One fixture per threshold, sitting exactly on the boundary. This closes the 12-constant hole in both the differential check and the parity check, and it is the highest-value verification work available.
6. Add a two-character scene fixture so eyeline is differentially verified, or list it as unverified.
7. app/tests/test_framing.py:13 — F = 3.45679012345679.
8. reliability.py vs 01-SPEC.md:229 — pick one behaviour and make the other match.
9. 02-DEMO-SCRIPT.md:79-80, 95 — record the actual panel states, or state the controls.

15. Bottom line

Do not submit as-is. Fix defect #1 first — without it the entry's primary artefact does not install, and that is the failure a judge is most likely to hit first and remember.

After that, this is a strong entry with an unusually honest verification story. The differential check, the parity check and the workflow validator are all real and all re-runnable; I attacked each of them and they held up far better than I expected. The solver's convergence claim survived 1800 adversarial calls without a single violation, and the licence/trademark position is clean.

The residual risk is concentrated in one place: the entry's two headline numbers both measure value fidelity and neither measures threshold fidelity. Twelve constants — including the entire medium-shot size band and the entire rule-of-thirds tolerance — can be wrong while 289 / ALL FIXTURES AGREE and 69 passed both print green. A judge who mutates one constant will find that, and it will look worse than it is, because the entry has spent so much credibility claiming otherwise. Adding boundary fixtures is a two-hour fix that removes the whole class.

The three claims I would least like a judge to find first, in order:

1. verification/README.md:122 — a verification README that tells you to run a script that does not exist.
2. validate_graphs.py:72 — a "read from the node sources, not guessed" table with one guess in it.
3. app/README.md:41 — "Restart ComfyUI. The nodes appear under the Cinematographer category." They do not.