When Guard confirms a regression, the failing trace opens in Trace for forensic replay. Debugger cites events on the five-ribbon timeline; confirmed fixes hand off to Mender, which drafts a fix pull request with regression tests. A human always approves the merge.
This page documents the failure → trace → promote → Mender PR workflow — Trace's contribution to Molar's closed-loop QA story.
The closed loop
Guard check fails
│
▼
Trace captured (auto-pin)
│
▼
Engineer opens trace detail
│
├──▶ Debugger: "why did step N fail?"
│
├──▶ Layer 2 replay: validate hypothesis
│
▼
Promote to fix (confirmed diff)
│
▼
Mender drafts fix PR + regression test
│
▼
Human reviews and merges
│
▼
Guard re-validates on PR → trace proves green
Tag line across the portfolio: failure → trace → mender.
Step 1 — Guard failure produces a trace
Every Guard check — PR gate, production monitor, shadow-prod diff, manual run — will capture a trace via @molar/trace-capture when Guard integration ships.
| Guard event | Trace behavior |
|---|---|
| Check passes | Trace status=passed, normal retention |
| Mender attempt exists | mender_summary written to summary.json |
Deep links:
| From | To |
|---|---|
| GitHub check annotation | trace.molar.it/traces/{id}?step={N} |
| Guard check detail | Open trace button |
| Slack alert | Permalink /r/{shortId} |
Configure: Guard project settings → Trace capture (default on). See Integrations.
Step 2 — Forensic replay in Trace
Open the trace from the Trace list or the GitHub annotation.
What to review first
| Surface | Look for |
|---|---|
| Overview tab | Failure signature, cluster count, Mender pre-classification |
| Five ribbons | Align network 404 with console error at same timestamp |
| Clones tab | Stripe/email state at failure step — webhook URL, amounts |
| Debugger tab | Auto-generated forensic summary |
Example failure (demo trace):
- Signature:
webhook-404 - Step 7: webhook POST returned 404
- Clone state:
amt=9000instead ofamt=900— coupon multiplier inverted - Mender pre-classification:
product_bugat 87% confidence
Full viewer guide: Trace detail.
Step 3 — Debugger confirms root cause
Ask Debugger targeted questions before promoting:
"Why did step 7 fail?" "What changed in the Stripe clone state between step 6 and 7?" "Compare network calls to the last green run on main."
Debugger cites (seq:N) references — click to jump scrubber. It reads Mender's existing classification via summarize_mender_analysis() without re-running Mender.
When hypothesis needs validation:
"Replay from step 7 with webhook URL
/api/webhooks/stripe"
Debugger calls run_layer2_replay with user confirmation. See Debugger & replay.
Step 4 — Layer 2 validates the fix
Layer 2 replay produces a child trace with the edited source. The diff view shows step status deltas today.
Target dimensions:
| Dimension | What changed |
|---|---|
| Network | 404 → 200 on webhook POST |
| Clone state | Correct subscription amount |
| Steps | Step 7+ pass in replay |
| Console | Error cleared |
Review the Patch tab for the unified diff Mender will receive.
Only promote when:
- Replay
status=succeeded - Side-by-side diff shows the expected fix
- No unexpected regressions in network or clone diffs
Step 5 — Promote to fix
The Promote to fix button appears in:
| Location | When visible |
|---|---|
| Overview tab → Mender card | Mender classification exists |
| Layer 2 diff view footer | Replay succeeded with edits |
| Debugger chat | After confirmed Layer 2 success (tool suggestion) |
Click Promote to fix PR → confirmation modal:
| Field | Content |
|---|---|
| Source trace | UUID + short_id |
| Replay trace | Child trace (if Layer 2 ran) |
| Edits payload | { scenario_diff, source_diffs[{path, diff}] } |
| Classification | From Mender or Debugger |
| Target branch | PR branch from Guard source metadata |
Submit → POST /v1/traces/{id}/promote-fix creates a mender_attempts row and enqueues Mender when guard_api_url is configured. Without Guard API wiring, the endpoint returns a draft payload for review (.
RBAC
| Role | Promote |
|---|---|
| Owner | ✓ |
| Member | ✓ |
| Viewer | ✗ |
Step 6 — Mender drafts the fix PR
Mender (Guard's auto-fix subsystem) receives:
- Trace NDJSON slices for failure context
- Layer 2 edit payload (scenario + source diffs)
- Debugger analysis markdown
- Classification + confidence score
Mender pipeline:
- Analyze — confirm classification against trace evidence
- Patch — apply source diffs to target branch
- Test — add/update regression test from scenario file
- PR — open draft pull request via GitHub App
- Re-validate — Guard runs on Mender PR → new trace proves green
| Mender field | Trace source |
|---|---|
classification | Overview Mender card |
confidence | mender_attempts.confidence |
analysis_markdown | Debugger + Mender summary |
patch JSONB | Layer 2 edits payload |
pr_url | Outbound link in Overview |
outcome | suggested → applied → rejected |
Track status from Trace Overview → Open Mender attempt or Guard → Mender tab.
Full Mender docs: Guard Mender auto-fix.
Step 7 — Human review and merge
Mender never auto-merges. The PR draft includes:
- Fix commit with source changes
- Regression test derived from the same
.molar/scenarios/{id}.molar.mdfile - PR description with trace permalink and failure summary
- Guard check status on the Mender PR
Review checklist:
- Fix matches Layer 2 diff you validated in Trace
- Regression test covers the failure signature
- No unrelated file changes
- Guard check green on Mender PR (new trace linked)
After merge, the original failure signature cluster should stop growing. Verify in Trace list → Failed · 24h view.
Mender card in Trace Overview
When Guard has already invoked Mender (pre-promotion classification):
┌─────────────────────────────────────────────────┐
│ MENDER ANALYSIS 87% · product_bug │
│ Webhook URL mismatch — route renamed in PR #4521 │
│ without updating Stripe clone config. │
│ [Open Mender attempt] [Promote to fix PR] │
└─────────────────────────────────────────────────┘
| Outcome badge | Meaning |
|---|---|
suggested | Mender analyzed; no PR yet |
applied | PR opened |
rejected | Human dismissed |
Promote overrides suggested → applied when you confirm the fix.
End-to-end example
Scenario: checkout-stripe fails on PR #4521 with webhook-404 signature.
- Guard check fails → trace
xY9zQ2mNpauto-pinned - Engineer opens trace → step 7 selected, NETWORK ribbon shows red tick
- Debugger: "webhook 404 — URL
/api/stripe/webhookvs handler at/api/webhooks/stripe" - Layer 2 replay from step 7 with config fix → child trace passes
- Diff view: webhook 404 → 200, amount corrected in diff view
- Promote to fix → Mender opens PR #4522
- Engineer reviews PR, merges
- Guard re-runs on main → green trace, cluster count drops
What Trace does NOT do
| Action | Owner |
|---|---|
| Open pull requests | Mender |
| Modify source files directly | Mender (via PR) |
| Auto-merge | Never — human required |
| Re-run Guard checks | Guard |
| Author scenarios | Cartographer |
Debugger may suggest promoting to Mender but cannot invoke it without explicit user action on the Promote button.
Configuration
| Setting | Location | Purpose |
|---|---|---|
| Mender enabled | Guard project settings | Allow auto-fix pipeline |
| GitHub App installed | Org integrations | PR creation + source reads |
| Trace capture on | Guard project settings | Ensure failure artifacts exist |
| Promote RBAC | Trace team settings | Restrict to owners if needed |
Related pages
| Page | When |
|---|---|
| Guard Mender auto-fix | Mender configuration and PR format |
| Debugger & replay | Layer 2 before promoting |
| Trace detail | Overview Mender card |
| Integrations | Guard → Trace wiring |