# Agent QC full documentation for LLMs

Source root: https://limecloud.github.io/agentqc

Agent QC is a portable standard for evidence-driven quality control of Agent projects.


# What is Agent QC?

Source: https://limecloud.github.io/agentqc/en/what-is-agent-qc

# What is Agent QC?

Agent QC is a portable standard for proving that Agent projects work.

Agent software has failure modes that ordinary app testing often misses: tool calls can drift from declarations, permission gates can be bypassed, model streams can produce malformed events, background tasks can get stuck, live providers can silently change behavior, and UI surfaces can make runtime facts look successful when they are not.

Agent QC gives these risks a shared vocabulary:

1. classify the Agent project profile;
2. identify touched interaction surfaces;
3. choose gates that match the profile, surface, and risk;
4. write behavior-level cases;
5. collect inspectable evidence;
6. judge pass/fail with explicit verdicts;
7. report remaining risk, blockers, exhausted attempts, reviews, and waivers.


## Why Agent projects need classification

A Rust runtime agent does not need the same gates as a Telegram gateway, a TUI, a browser automation harness, or a VitePress standards site. Agent QC therefore classifies first, then chooses gates.

Examples:

- Codex-style runtime: sandbox, apply-patch, MCP, app-server protocol, CLI e2e, TUI snapshots, cross-platform release.
- Claude Code-style TUI runtime: Ink rendering, remote permissions, WebSocket/control streams, SDK adapters, plugin/skill reload visibility.
- OpenClaw-style gateway: channel contracts, secrets, provider live lanes, QA Lab reports, WebUI/browser evidence, Docker/install smoke.
- Hermes-style background agent: pytest markers, cron, gateway, browser safety, concurrency stress, Docker smoke, credential isolation.
- Desktop GUI agent: native bridge contracts, workspace/session readiness, GUI smoke, browser evidence, release checks.

## Runtime-backed surfaces

Agent QC adopts a key Agent UI rule: visible surfaces are projections, not truth owners.

A surface pass should connect:

```text
entrypoint -> user action -> visible frame -> runtime/protocol fact -> evidence ref -> cleanup
```

A screenshot without runtime backing is visual smoke. A runtime log without visible frame is not a surface test.

## What counts as evidence

Evidence can be a test report, command log, CI URL, qcloop attempt, verifier round, Playwright trace, screenshot, terminal snapshot, model/tool transcript, protocol transcript, browser console/network log, package manifest, Docker smoke output, eval rubric, judge output, or human review record.

A model's final prose is never enough by itself.


# Specification

Source: https://limecloud.github.io/agentqc/en/specification

# Specification

Agent QC v0.5.0 is a portable draft standard for evidence-driven quality control of Agent projects.

An Agent project can be a runtime CLI, SDK, tool server, MCP/ACP gateway, multi-channel bot, GUI/TUI/desktop client, skills or plugin ecosystem, background scheduler, distribution package, or evaluation suite. Agent QC does not assume one product shape. It starts by classifying the project profile and then selects gates that match its risk.

## Scope

Agent QC standardizes:

1. Project profiles for Agent systems.
2. Test plan, case, gate, run, evidence, verdict, and report objects.
3. Gate taxonomy from static checks to live provider and release smoke.
4. Evidence-backed pass/fail semantics.
5. qcloop-compatible batch QC for repeated independent cases.
6. Benchmark and hill-climbing evidence for runtime, prompt, tool, and context improvement.
7. Case-study mapping for representative runtime, TUI, gateway, scheduler, UI, skills, release, and eval projects.

Agent QC does not standardize any single programming language, CI vendor, test framework, browser driver, model protocol, storage backend, or UI skin.


## Document set

The latest standard is split by use:

| Page | Purpose |
| --- | --- |
| [Quickstart](./authoring/quickstart) | fastest path to a QC plan |
| [Best practices](./authoring/best-practices) | authoring rules and anti-patterns |
| [Test techniques and compositions](./authoring/test-techniques-and-compositions) | snapshots, smoke tests, black-box, white-box, runtime/UI/skills testing, and advanced evidence braids |
| [Benchmark and hill climbing](./authoring/benchmark-and-hill-climbing) | frozen tasks, trials, rewards, and trajectories for proving Lime improves |
| [Project classification](./authoring/project-classification) | profile taxonomy and mixed-profile rules |
| [Gate matrix](./authoring/gate-matrix) | profile/surface/risk to gate mapping |
| [Interaction surface testing](./authoring/interaction-surface-testing) | CLI/TUI/WebUI/desktop/browser/channel/eval UI evidence |
| [Evidence contract](./contracts/evidence-contract) | portable evidence, verdict, waiver fields |
| [Performance and reliability metrics](./contracts/performance-and-reliability-metrics) | timing, flake, cleanup, scheduler, release metrics |
| [Flow and taxonomy](./reference/flow-and-taxonomy) | complete lifecycle and taxonomy reference |
| [Star project testing systems](./reference/star-project-testing-systems) | representative Agent project testing-system case studies |

## Project profiles

A `qc_plan.project_profiles` array declares which project shapes apply.

| Profile | Typical risks | Example gates |
| --- | --- | --- |
| `agent-runtime-cli` | tool execution, sandboxing, permission, streams, resume, subprocess cleanup | unit, protocol, fake model server, CLI e2e, sandbox tests |
| `agent-sdk-api` | public API compatibility, generated contracts, fake server behavior, async cancellation | signature tests, generated contract diff, fake server integration |
| `agent-tool-mcp-gateway` | tool declaration drift, stdio/http transport, recovery, resource access, audit refs | protocol conformance, mock server, transport recovery, contract tests |
| `multi-channel-agent-gateway` | channel adapters, auth, secrets, webhook verification, provider drift, media routing | channel contract tests, secret isolation, live opt-in, docker smoke |
| `agent-ui-tui-desktop` | rendering, terminal/browser state, user controls, screenshots, accessibility, bridge readiness | UI unit, snapshot, Playwright, terminal fixtures, GUI smoke |
| `agent-skills-plugins` | manifest shape, loader, package boundary, trust, marketplace or registry drift | schema, discovery, package export, fixture install, security scan |
| `background-agent-scheduler` | cron, queues, leases, retries, concurrency, idempotency, stuck-loop recovery | deterministic scheduler tests, race tests, stress tests, checkpoint/reclaim |
| `agent-distribution-release` | install, package contents, Docker, cross-platform, lockfiles, supply-chain | install smoke, package dry run, Docker smoke, OS matrix, lock checks |
| `agent-evals-quality` | model behavior regressions, prompt drift, rubric quality, answer grounding | eval suite, baseline comparison, LLM/human judge, qcloop batch |

A project MAY combine profiles. For example OpenClaw combines channel gateway, tool gateway, distribution, live provider, and plugin profiles.

## Interaction surfaces

A project profile says what the project owns. An interaction surface says where users or operators observe the Agent. `qc_case.surface` is optional in the JSON schema but SHOULD be present for user-visible gates.

| Surface | Applies to | Extra evidence required |
| --- | --- | --- |
| `cli-stream` | command output, JSONL/NDJSON, stdout/stderr | exit status, stdout/stderr transcript, structured event sample |
| `tui` | terminal UI, Ink, ratatui, curses | terminal snapshot, viewport size, key sequence, runtime transcript |
| `webui` | browser dashboard, extension UI, admin/QA console | screenshot/trace, console log, route state, browser-only assertion |
| `desktop-gui` | Tauri, Electron, native shell | shell start evidence, bridge health, workspace/session readiness, OS note |
| `browser-automation` | CDP, Playwright, browser-use, remote browser providers | screenshot, DOM/a11y snapshot, console/network log, cleanup evidence |
| `channel-ui` | mobile, QR, chat apps, webhook surfaces | channel transcript, media fixture, auth/webhook replay, device/emulator log |
| `eval-ui` | QA dashboards and semantic evaluation reports | rubric, judge output, baseline delta, reviewer note |

A `ui-interaction` gate SHOULD name one of these surfaces. A pass without surface-specific evidence is incomplete. Surface proof SHOULD connect entrypoint, user action, visible frame, runtime backing, and cleanup evidence.

## Core objects

| Object | Purpose |
| --- | --- |
| `qc_plan` | A test plan for one change, release, investigation, or regression sweep. |
| `qc_case` | One behavior-level item with steps, expected result, required gates, and evidence. |
| `qc_gate` | A validation boundary such as static, unit, contract, integration, e2e, live, stress, release, or review. |
| `qc_run` | One execution attempt with command, executor, environment, output refs, duration, and result. |
| `qc_evidence` | A reference to logs, reports, traces, screenshots, fixtures, qcloop attempts, CI runs, or review notes. |
| `qc_verdict` | A judgment over evidence: passed, failed, blocked, exhausted, waived, or needs-review. |
| `qc_report` | The aggregate result, remaining risk, waivers, and next action. |

## Gate families

Gate families define the quality boundary. They are implemented with concrete techniques such as static checks, white-box unit tests, property/fuzz tests, golden transcripts, snapshots, contract tests, fake integrations, black-box smoke, runtime E2E, surface E2E, replay/regression, stress/chaos, security/adversarial tests, semantic evals, benchmark evals, and release install smoke.

| Family | Purpose | Evidence examples |
| --- | --- | --- |
| `static` | format, lint, type, dependency and policy hygiene | command logs, SARIF, lockfile check output |
| `unit` | deterministic local behavior | test report, coverage, fixture output |
| `property-fuzz` | invariants and generated input | seed, corpus, failing case artifact |
| `contract-protocol` | schemas, APIs, generated clients, command/tool surfaces | contract report, schema diff, mock transcript |
| `fake-integration` | integration against fake servers or local adapters | fake server log, request/response transcript |
| `runtime-e2e` | real CLI/runtime/task flow without external provider risk | CLI transcript, process cleanup evidence, state snapshot |
| `ui-interaction` | GUI/TUI/browser/terminal behavior | screenshot, trace, video, accessibility report |
| `live-provider` | opt-in real provider or network path | redacted transcript, credentials-scope note, cost/budget |
| `stress-concurrency` | races, leases, retries, long-running loops | stress report, worker timeline, seed, benchmark |
| `distribution-release` | install, package, Docker, cross-platform release readiness | tarball manifest, Docker smoke, OS matrix, release check |
| `semantic-eval` | model output quality, grounding, policy, user intent | eval result, rubric, judge output, baseline delta |
| `benchmark-eval` | runtime/prompt/tool/context changes outperforming a baseline | dataset/task version, trial trajectory, reward details, pass@k or delta |
| `review` | human or LLM review | reviewer decision, rubric, evidence refs |

## Status values

`qc_case.status`, `qc_gate.status`, and `qc_report.status` use:

- `planned`
- `running`
- `passed`
- `failed`
- `blocked`
- `exhausted`
- `waived`
- `skipped`
- `needs-review`

A waived gate MUST include `waiver.reason`, `waiver.approver`, and `waiver.expires` when the project has a waiver process.

## Evidence rules

A `passed` verdict MUST include evidence. A `failed` verdict MUST include the smallest actionable failure. A `blocked` verdict MUST identify the missing environment fact. An `exhausted` verdict MUST preserve attempts and verifier feedback.

Self-report is not evidence. The sentence "the agent checked it" is only valid when it links to command output, test report, transcript, trace, screenshot, or review record.

## qcloop mapping

A `qc_case` can become a qcloop `item_value`. qcloop `attempt` maps to `qc_run`; qcloop `qc_round` maps to `qc_verdict`; qcloop `exhausted` maps to Agent QC `exhausted`, not generic failure.

Use qcloop when cases are repeated, independent, and verifier-friendly. Do not use qcloop to replace required project gates or to hide live-provider risk.


# Quickstart

Source: https://limecloud.github.io/agentqc/en/authoring/quickstart

# Quickstart

Use this flow when an agent, maintainer, CI job, or qcloop run needs to test an Agent project. The flow is portable across runtime, UI, gateway, scheduler, skill, eval, and release product shapes.

## 1. Define the QC scope

Write one sentence that names what is being judged.

Examples:

- "Release v1.4.0 of a Codex-like runtime CLI."
- "A Claude Code-like TUI remote-permission change."
- "An OpenClaw-like channel gateway auth/media change."
- "A Hermes-like scheduler restart fix."
- "A desktop GUI native-bridge change."

The scope decides which claims the report may make. Do not let a broad report imply untested surfaces.

## 2. Classify the project profile

Pick one or more profiles:

- `agent-runtime-cli`
- `agent-sdk-api`
- `agent-tool-mcp-gateway`
- `multi-channel-agent-gateway`
- `agent-ui-tui-desktop`
- `agent-skills-plugins`
- `background-agent-scheduler`
- `agent-distribution-release`
- `agent-evals-quality`

Classify by owned risk, not language. A Python project can own browser automation; a Rust project can own a TUI; a docs site can own a distribution/release surface.

## 3. Identify touched surfaces

Name where a user or operator observes the behavior:

- `cli-stream`
- `tui`
- `webui`
- `desktop-gui`
- `browser-automation`
- `channel-ui`
- `eval-ui`

If behavior is visible, include `qc_case.surface`. A UI pass without surface evidence is incomplete.

## 4. Assign fact owners

For each case, decide which system owns the fact:

| Fact | Owner example | Evidence |
| --- | --- | --- |
| runtime accepted work | agent runtime | stream or session transcript |
| tool call succeeded/failed | tool runtime or protocol adapter | tool id, progress, result/error |
| approval was resolved | policy/runtime action API | request id and response transcript |
| UI rendered a state | UI projection | screenshot, trace, terminal snapshot |
| artifact was created | artifact/release service | manifest, file, version, export ref |
| verdict passed | Agent QC report | linked evidence refs |

This prevents the common Agent UI failure where visible text is treated as runtime truth.

## 5. Select gate lanes

Use the [gate matrix](./gate-matrix). At minimum, separate these lanes:

1. deterministic local gates: `static`, `unit`, `contract-protocol`, `fake-integration`;
2. runtime gates: `runtime-e2e`, `stress-concurrency` when needed;
3. surface gates: `ui-interaction` with a named surface;
4. live gates: `live-provider` only with explicit opt-in;
5. release gates: `distribution-release` when anything is shipped;
6. semantic gates: `semantic-eval` and `review` when quality judgment matters.
7. benchmark gates: `benchmark-eval` when claiming a runtime, prompt, tool, context, or routing change improves Lime against a frozen baseline.

## 6. Write behavior-first cases

Each `qc_case` should state:

- behavior to prove;
- profile and surface;
- exact steps or commands;
- expected result;
- required gates;
- required evidence;
- status mapping for fail, blocked, exhausted, waived, and needs-review.

Avoid cases like "component exists". Prefer cases like "user denies a tool call; runtime records denial; TUI removes pending approval; no side effect occurs".

## 7. Define evidence policy

Before running, decide what counts as proof.

| Claim | Evidence |
| --- | --- |
| CLI stream works | command, exit status, stdout/stderr transcript, structured event sample |
| TUI works | viewport, key sequence, terminal snapshot, runtime transcript |
| WebUI works | Playwright/browser trace, screenshot, console/network log, route assertion |
| Desktop GUI works | shell start, bridge health, workspace readiness, screenshot, OS note |
| Browser automation works | DOM/a11y snapshot, screenshot, console/network, cleanup proof |
| Channel works | webhook replay, media fixture, auth proof, redacted channel transcript |
| Scheduler works | deterministic time/env, checkpoint, lease/reclaim, duplicate-work proof |
| Release works | package manifest, clean install, Docker/OS matrix, version output |
| Eval works | rubric, judge output, baseline delta, reviewer note |

See [Evidence contract](../contracts/evidence-contract).

## 8. Use qcloop for repeatable independent checks

Use qcloop for many similar cases: files, channels, providers, commands, prompts, packages, or regression examples.

Do not use qcloop to hide missing project gates. A qcloop batch may produce verdicts, but bridge health, GUI smoke, package install smoke, and live-provider opt-in still need their own evidence.

## 9. Run gates from cheap to risky

A practical order is:

1. `static` and schema checks;
2. targeted unit or contract tests;
3. fake integration and runtime e2e;
4. surface smoke or Playwright/TUI evidence;
5. stress/concurrency where relevant;
6. live provider or channel tests only with opt-in;
7. release/install/Docker checks;
8. semantic eval and review.

Stop early only when the failure makes downstream evidence meaningless. Otherwise record partial evidence and mark remaining gates accurately.

## 10. Report verdicts and limits

A complete report includes:

- scope;
- profiles and surfaces;
- gates required and gates executed;
- evidence refs;
- verdicts by case;
- blockers, exhausted attempts, waivers, and needs-review items;
- remaining risk;
- next action.

A report is not complete when it says only "tests passed" or "the agent checked it".

## Minimal plan skeleton

```json
{
  "schema_version": "0.4.0",
  "project": "example-agent",
  "project_profiles": ["agent-runtime-cli", "agent-ui-tui-desktop"],
  "required_gates": ["static", "contract-protocol", "runtime-e2e", "ui-interaction"],
  "cases": [
    {
      "id": "permission-denial-tui",
      "project_profile": "agent-ui-tui-desktop",
      "surface": "tui",
      "target": "remote permission prompt",
      "required_gates": ["contract-protocol", "runtime-e2e", "ui-interaction"],
      "steps": ["start fake runtime", "trigger high-risk tool", "deny approval"],
      "expected": ["runtime records denial", "TUI removes pending prompt", "no side effect occurs"],
      "required_evidence": ["protocol transcript", "terminal snapshot", "runtime transcript"]
    }
  ]
}
```


# Best practices

Source: https://limecloud.github.io/agentqc/en/authoring/best-practices

# Best practices

Use this page as the authoring checklist for Agent QC plans. It adapts the runtime-first style of Agent UI, the progressive-disclosure style of Agent Skills, and testing patterns observed across runtime CLIs, TUI agents, multi-channel gateways, background/browser agents, desktop clients, and eval systems.

Agent QC is a standard protocol, not a single-product checklist.

## Start from owned risk

Classify what the project owns before choosing commands.

Good wording:

> This change touches a runtime permission boundary and a TUI approval surface. The plan requires `contract-protocol`, `runtime-e2e`, and `ui-interaction` evidence.

Bad wording:

> This is a TypeScript project, so Vitest is enough.

Owned risk usually falls into one or more of these lanes:

| Risk owner | Typical proof |
| --- | --- |
| Runtime | command transcript, stream events, cleanup, state snapshot |
| Protocol or SDK | schema diff, fake server transcript, generated client check |
| Tool or MCP gateway | declaration, permission, progress, result, recovery transcript |
| UI/TUI/WebUI/desktop | snapshot, screenshot, trace, accessibility output, console log |
| Browser automation | DOM/a11y snapshot, screenshot, console/network log, cleanup proof |
| Channel gateway | webhook replay, media fixture, identity/auth proof, redaction |
| Scheduler | deterministic clock, lease/checkpoint, restart/reclaim, duplicate-work guard |
| Release | package manifest, clean install, Docker smoke, OS matrix, lock/security |
| Eval quality | rubric, baseline delta, judge output, reviewer note |

## Keep verdict facts owned by evidence

A QC report MUST NOT infer pass/fail from final prose. The verifier may summarize, but the verdict belongs to evidence.

Every pass needs:

- behavior statement;
- gate family;
- command or interaction steps;
- expected result;
- evidence refs;
- verdict status;
- remaining risk or waiver if incomplete.

If the only proof is "the agent says it passed", the status is `needs-review` or `blocked`, not `passed`.

## Separate deterministic, live, and release gates

Do not hide expensive or flaky risk inside ordinary unit tests.

| Lane | Runs by default | Typical evidence | Common anti-pattern |
| --- | --- | --- | --- |
| Deterministic | yes | lint/type/unit/contract/fake-server logs | using live keys in unit tests |
| Runtime | usually | CLI/task/session transcript and cleanup proof | judging runtime from component tests only |
| Surface | when user-visible | TUI snapshot, Playwright trace, screenshot, console log | screenshot without runtime transcript |
| Live provider | explicit opt-in | redacted request/response, budget, credential scope | live call hidden behind `npm test` |
| Release | before shipping | package/Docker/install/OS matrix output | source tests only |
| Review/eval | when semantic quality matters | rubric, judge output, examples, reviewer | pass/fail without baseline |

OpenClaw is a useful example of explicit live and Docker lanes. Hermes is a useful example of blanking provider credentials in normal tests. Codex is a useful example of fake servers and fixtures before live/provider claims.

## Map surfaces to Agent UI facts

Agent UI's most important lesson for Agent QC is runtime-backed projection. A visible surface is not enough; the visible state must link to the owning runtime fact.

| Agent UI surface | Agent QC case focus |
| --- | --- |
| Composer | submit, queue, steer, interrupt, attachments, context chips |
| Message parts | final text separated from reasoning, tools, diagnostics, artifacts |
| Runtime status | first status before text, blocked/retrying/failed/done states |
| Tool UI | tool id, safe args summary, progress, result, error, offload ref |
| Human-in-the-loop | approval/input id, scope, decision, runtime confirmation |
| Task capsule | queued/background/subagent status, ownership, failure, retry |
| Artifact workspace | artifact id, preview, version, diff, export, save failure |
| Timeline/evidence | trace, replay, verification, review, audit refs |
| Session/tabs | old-session restore, stale/hydrating state, unread/running state |
| Team workbench | coordinator, worker, handoff, review, remote/background teammate |

QC rule: a user-visible pass SHOULD connect entrypoint, user action, visible frame, runtime event, evidence ref, and cleanup.

## Treat missing facts honestly

Use explicit statuses instead of guessing:

- `blocked` when the environment, credentials, fixture, or binary is missing;
- `exhausted` when attempts or budget are consumed without proof;
- `needs-review` when evidence exists but the judgment is semantic, safety-sensitive, or disputed;
- `waived` only when an accountable owner accepts a gap with reason and expiry.

Do not show `passed` because a UI looked healthy if bridge/runtime evidence is missing. This applies to all Agent UI/TUI/WebUI projects.

## Prefer behavior-level scenarios

A QC case should read like a reproducible user or operator flow, not like a file inventory.

Good case:

> User denies a high-risk tool call; the runtime records denial, the TUI removes the pending approval, and no side effect occurs.

Weak case:

> Approval component exists.

Behavior-level cases should cover:

- happy path;
- denied or failed path;
- cancellation/interruption;
- reconnect/retry/recovery;
- stale or missing facts;
- old-session or resumed state;
- platform and viewport differences when relevant.

## Make qcloop narrow and inspectable

qcloop is best for repeated independent checks: many files, many channels, many providers, many command variants, or many prompt/eval items.

Use qcloop when each item can be judged from its own output and evidence refs. Do not use qcloop to replace required project gates such as bridge health, package install smoke, Playwright trace collection, or live-provider opt-in policy.

A good qcloop item includes:

- project profile;
- touched surface;
- gate family;
- exact input or command;
- expected behavior;
- evidence policy;
- verifier rubric;
- status mapping for pass/fail/blocked/exhausted/waived.

## Preserve source traceability

When a standard page changes, update [Source index](../reference/source-index) or a case-study page. Local repos are examples, not requirements.

Traceability levels:

| Level | Use |
| --- | --- |
| Public specification | Agent Skills, Playwright, Vitest, pytest, protocol docs |
| Local case study | Codex, Claude Code local snapshot, OpenClaw, Hermes, desktop GUI and release examples |
| Project-specific rule | a concrete product's scripts, CI, workflow, or AGENTS file |
| Evidence artifact | command output, trace, screenshot, transcript, report |

## Write for progressive disclosure

Follow the Agent Skills style: a short entry page, tables for fields and constraints, minimal examples, and deeper reference pages.

For Agent QC pages:

- quickstart pages choose the path;
- authoring pages explain how to write plans and evidence;
- contract pages define portable fields and verdict constraints;
- reference pages hold taxonomy and project research;
- example pages show real plan shapes.

## Avoid single-framework lock-in

Agent QC may mention Playwright, Vitest, pytest, cargo nextest, Bazel, Docker, qcloop, and VitePress examples, but the standard requirement is the evidence shape.

Good guidance:

> Browser UI gates should retain a trace/screenshot on failure and record console or network evidence.

Bad guidance:

> Every Agent project must use Playwright with the same config.

## Review checklist

Before publishing a QC plan or report, verify:

| Question | Required answer |
| --- | --- |
| Is this standard tied to one product? | No; profiles apply to all Agent project types. |
| Are project profiles declared? | One or more profiles are named. |
| Are touched surfaces named? | User-visible cases include `qc_case.surface`. |
| Are gates separated? | Deterministic, runtime, surface, live, release, and eval lanes are distinct. |
| Is evidence inspectable? | Each pass/fail links to logs, reports, traces, transcripts, screenshots, or review refs. |
| Are limitations explicit? | Missing metadata, blocked credentials, or local-only assumptions are recorded. |
| Are waivers accountable? | Waiver owner, reason, scope, and expiry are present. |
| Can qcloop repeat it? | Repeated cases have stable item values and verifier rubrics. |


# Test techniques and compositions

Source: https://limecloud.github.io/agentqc/en/authoring/test-techniques-and-compositions

# Test techniques and compositions

Agent QC gate families describe **why** a boundary must be checked. Test techniques describe **how** evidence is produced. Strong Agent QC plans combine techniques instead of relying on one broad command.

Use this page when a plan says only "run tests" or "do UI smoke" and you need a richer, inspectable strategy for Agent runtime, Agent UI, skills/plugins, browser automation, channel gateways, or release packages.

## Evidence braid rule

A high-confidence Agent test usually braids five strands:

```text
white-box invariant -> protocol/contract -> black-box run -> surface artifact -> cleanup/review
```

Not every case needs all five, but every pass must state which strands are present and which claims remain unproven.

## Technique taxonomy

| Technique | What it proves | Required evidence | What it does not prove alone |
| --- | --- | --- | --- |
| Static/policy check | Formatting, types, import boundaries, generated drift, forbidden APIs | command log, SARIF or lint report, tool version | runtime behavior or UX |
| White-box unit test | Reducers, parsers, serializers, permission decisions, state machines | test report, fixture ids, assertion diffs | packaged app or user-visible behavior |
| Property/fuzz/metamorphic test | Invariants over large or generated input sets | seed, corpus, minimized failure, invariant text | exact user flow |
| Golden transcript | Stable CLI/runtime/protocol/event output shape | transcript file, update diff, dynamic-field normalization | visual layout or live provider quality |
| Snapshot test | Stable rendered output or serialized object | snapshot diff, viewport/device, update review | correctness of the source runtime fact |
| Contract/protocol test | Schema, tool declarations, SDK/API, manifest, transport behavior | schema diff, fake server transcript, generated artifact check | actual live provider behavior |
| Fake integration | Adapter/runtime behavior against a controlled local service | fake server log, request/response refs, fixture version | real provider drift |
| Black-box smoke | Minimal delivered behavior through public entrypoint | command/browser/app/channel log, exit status, screenshot when visible | deep edge cases |
| Runtime E2E | Agent loop, tools, permissions, resume, cleanup | runtime transcript, state snapshot, side-effect proof | UI projection unless linked |
| Surface E2E | User/operator can see and control the behavior | screenshot/trace/terminal frame, key/click/message sequence | underlying runtime truth unless linked |
| Replay/regression | Past failure remains fixed | replay fixture, old bug id, expected failure mode | new unknown failures |
| Stress/concurrency/chaos | Race, lease, retry, cancellation, long-running resilience | worker timeline, seed/config, duration, cleanup | semantic answer quality |
| Security/adversarial | Permission, prompt injection, path, SSRF, secret, policy boundaries | attack fixture, denial transcript, side-effect check | happy path usability |
| Semantic eval | Output quality, grounding, tool choice, policy adherence | dataset, rubric, model/judge, baseline delta | deterministic code correctness |
| Benchmark eval | Runtime/prompt/tool/context candidate improvement | frozen dataset, trial trajectories, reward details, baseline/candidate delta | release safety or P0 QC pass |
| Release/install smoke | Shipped artifact can install and run outside the source tree | package manifest, clean install, Docker/OS log, version output | source tree test coverage |

## Black-box, white-box, and gray-box

| Mode | Agent QC use | Best targets | Evidence pattern |
| --- | --- | --- | --- |
| White-box | Prove internal invariants before a user flow exists | event reducers, permission policy, tool args sanitizer, stream parser, scheduler lease | unit/property report plus fixture ids |
| Black-box | Prove delivered behavior through public entrypoint | CLI command, SDK call, TUI flow, WebUI route, desktop shell, webhook, package install | command or interaction transcript plus exit/status and artifacts |
| Gray-box | Combine public behavior with internal instrumentation | runtime UI, browser agent, channel gateway, background scheduler | black-box run plus protocol/runtime transcript and state snapshot |

Agent projects need gray-box testing more often than ordinary apps because the visible output can be plausible while the runtime state is wrong.

## Snapshot standards

Snapshots are useful only when they are scoped and reviewable.

| Snapshot kind | Use it for | Must include |
| --- | --- | --- |
| Text/golden transcript | CLI output, JSONL/NDJSON stream, model event normalization | stable fixture, exit status, dynamic id redaction |
| Terminal snapshot | TUI frame, approval overlay, footer/status row, composer | terminal size, key sequence, ANSI/Unicode policy |
| DOM/ARIA snapshot | WebUI accessibility tree, browser-mode component state | route, viewport/device, locator or role assertion |
| Screenshot/video | GUI/desktop/browser/channel report surface | action sequence, OS/browser/device, console/network note |
| Protocol/schema snapshot | generated schema, SDK wire contract, MCP/tool declaration | generator command, diff, compatibility note |
| Runtime state snapshot | session/thread/turn/tool/artifact/scheduler state | correlation ids, timestamp policy, cleanup note |
| Package manifest snapshot | tarball/image/install contents | version, platform, file allow/deny policy |

Snapshot rules:

- Normalize timestamps, random ids, temp paths, and provider-specific text before snapshotting.
- Review snapshot updates as product changes, not as mechanical noise.
- Pair UI snapshots with runtime/protocol transcripts when the claim is more than visual layout.
- Pair protocol snapshots with fake integration when the claim is more than schema shape.
- Keep one focused snapshot per behavior; avoid giant snapshots that hide meaningful diffs.

Codex-style TUI testing shows the value of terminal snapshots for approval overlays, footer modes, picker widths, request forms, narrow terminal heights, and diff/code blocks. Hermes-style TUI testing adds terminal mechanics such as OSC52, virtual history, Unicode, streaming markdown, queue state, and session lifecycle. Claude Code-style local source inspection shows that Ink TUI, remote permission, WebSocket control, and SDK stream adapters need snapshot plus control transcript, not snapshot alone.

## Smoke test ladder

Smoke tests are fast confidence checks. They do not replace runtime, contract, or surface evidence.

| Smoke level | Purpose | Examples | Exit rule |
| --- | --- | --- | --- |
| Import/build smoke | Prove package imports or builds | `cargo test -p crate`, `vitest run`, `python -m package --help` | fail fast on syntax/link/import break |
| Runtime smoke | Prove the agent loop starts with fake/local provider | `agent exec "hello"`, fake tool call, MCP list tools | transcript shows terminal status and cleanup |
| Surface smoke | Prove visible shell can open and reflect runtime state | TUI first frame, WebUI route, desktop bridge health, channel webhook replay | surface artifact plus runtime backing |
| Release smoke | Prove artifact works outside source tree | clean install, Docker start, package help/version | install log and manifest match release |
| Canary/live smoke | Prove real provider/channel still works | opt-in provider call, live channel ping, model profile probe | redacted transcript, budget, credential scope |

Use smoke for broad detection and then use targeted tests for diagnosis.

## Testing Agent runtime

Runtime tests should treat the agent as a state machine, not as a text generator.

Minimum runtime invariants:

| Runtime area | Required cases | Evidence |
| --- | --- | --- |
| Turn lifecycle | accepted, queued, running, completed, failed, cancelled | event transcript, terminal status, exit code |
| Stream shape | partial text, reasoning/tool events, final text, terminal marker | JSONL/SSE fixture, parser report, golden transcript |
| Tool execution | declaration, argument validation, progress, result, error | tool id correlation, fake tool transcript, side-effect check |
| Permission/HITL | allow, deny, edit/input, timeout, cancel, reconnect | approval request/response transcript, surface frame |
| Files/processes | cwd, sandbox, patch/write, subprocess tree, cleanup | command log, path fixture, orphan-process proof |
| Resume/persistence | old session, crash/restart, checkpoint, artifact refs | state snapshot, replay transcript, cleanup note |
| Scheduler/parallelism | lease, retry, fanout/fanin, duplicate-work prevention | deterministic clock, worker timeline, stress/chaos result |
| Credential/provider scope | fake by default, live opt-in, redaction, budget | env scope, redacted request/response, waiver if missing |

Runtime anti-patterns:

- asserting only final assistant text;
- hiding provider calls inside default unit tests;
- testing tool declaration but not invocation and failure;
- testing success but not deny/cancel/abort/resume;
- omitting cleanup proof for subprocesses, browsers, workers, or temp state.

## Testing Agent UI

Agent UI tests must prove that visible surfaces are runtime-backed projections.

| UI area | What to test | Strong evidence |
| --- | --- | --- |
| Composer/input | submit, queued input, steer-current, attachments, paste, slash commands | key/click sequence, runtime input id, snapshot |
| Status | first status before text, retrying, blocked, failed, done | runtime event order, UI frame, timing metric |
| Tool cards | safe arg summary, progress, result, error, offload refs | tool id correlation, screenshot/terminal snapshot, transcript |
| Approval/HITL | pending, allow, deny, edit, timeout, cancellation | action request/response transcript, keyboard/a11y proof |
| Artifacts | create, diff, preview, export, failed save | artifact id/path, UI snapshot, export log |
| Evidence/replay | trace links, report export, old-session hydration | evidence ids, report screenshot, hydration log |
| Team/background | queued worker, running worker, failed/retried worker, handoff | delegation graph, task card snapshot, worker transcript |
| Empty/stale states | missing facts, bridge unavailable, reconnecting, blocked | safe fallback frame, console/network log, runtime state ref |

Surface-specific upgrades:

- TUI: multi-viewport, ANSI/Unicode width, Ctrl-C vs Esc semantics, resize, clipboard/OSC52 if supported.
- WebUI: browser trace, DOM/ARIA snapshot, console/network, reload/resume, keyboard/a11y.
- Desktop GUI: app shell start, bridge health, workspace readiness, native command contract, OS note.
- Browser automation: screenshot plus DOM/a11y, console/network, unsafe navigation/SSRF fixtures, orphan cleanup.
- Channel/mobile: webhook replay, media fixture, auth proof, redacted transcript, device/emulator logs.

## Testing skills and plugins

Agent Skills-style systems need their own lifecycle tests. The standard lesson is progressive disclosure: a skill is a small package with metadata, instructions, optional scripts/assets, and evaluation evidence. Testing should follow that shape.

| Skill/plugin phase | Tests | Evidence |
| --- | --- | --- |
| Manifest/frontmatter | required fields, name/description, when-to-use, paths/hooks if supported | schema report, parse failure fixtures |
| Discovery/loading | user/project/bundled precedence, symlink canonicalization, duplicate names, disabled settings | loader transcript, fixture directory tree |
| Context budget | frontmatter-only routing, lazy loading, token/size limits | token estimate, selected skill list, rejection evidence |
| Scripts/assets | script existence, executable bit, relative path resolution, clean temp dir, no raw secrets | dry-run log, sandbox/env scope, asset manifest |
| Trust boundary | local vs managed vs remote/MCP skill policy, path traversal, hook restrictions | policy test, denial transcript, audit note |
| Runtime effect | skill changes allowed tools/prompts only through owning API | runtime event, tool declaration diff, UI status |
| Evaluation | clean-context task, assertion grading, transcript, human feedback loop | eval rubric, attempt transcripts, verifier output |
| Packaging/release | package contents, install fixture, marketplace/registry metadata | manifest snapshot, install smoke, version check |

Claude Code local source exposes useful loader concerns: `SKILL.md` directory format, frontmatter parsing, hooks validation, path frontmatter, symlink canonicalization, token estimation, duplicate detection, and remote MCP skills as untrusted. Agent QC generalizes those as skill/plugin gates; it does not require Claude Code's exact implementation.

## Advanced composition recipes

### Runtime + UI evidence braid

Use when a runtime fact is visible in TUI/WebUI/desktop GUI.

```text
contract-protocol
  -> fake runtime transcript
  -> black-box user action
  -> surface snapshot/trace
  -> state snapshot + cleanup
```

Example claims: approval overlay, tool card progress, bridge health, queued worker state.

### TUI approval braid

```text
white-box permission resolver
  -> protocol action_request fixture
  -> pseudo-terminal key sequence
  -> terminal snapshots for pending/allow/deny/cancel
  -> side-effect denial check
  -> subprocess cleanup
```

Add multi-viewport, Unicode/ANSI, Ctrl-C/Esc, and reconnect variants when the TUI is core product surface.

### Provider adapter ladder

```text
normalizer unit tests
  -> contract/schema snapshot
  -> fake provider replay
  -> runtime E2E with fake provider
  -> opt-in live canary
  -> semantic eval and reviewer note
```

Use this for LLM providers, browser providers, search providers, channel providers, or gateway backends.

### Browser agent safety braid

```text
URL/path policy unit tests
  -> SSRF/file/credential attack fixtures
  -> Playwright/browser trace with DOM+a11y snapshot
  -> console/network log inspection
  -> orphan browser/tab cleanup proof
```

A screenshot-only pass is insufficient for browser automation.

### Channel gateway braid

```text
auth verifier unit test
  -> webhook replay before body parsing
  -> media fixture and redaction check
  -> fake channel send transcript
  -> optional live channel canary
  -> report redaction review
```

Use separate gates for channel contract, media handling, live transport, and semantic model quality.

### Scheduler/recovery braid

```text
deterministic clock unit test
  -> lease/checkpoint fake integration
  -> crash/restart replay
  -> concurrency stress or chaos kill
  -> duplicate-work oracle
  -> cleanup and ownership report
```

This is mandatory for background agents, multi-agent workers, and long-running jobs.

### Skill/plugin lifecycle braid

```text
manifest schema
  -> discovery/precedence fixture
  -> script/asset dry run in clean temp dir
  -> trust boundary denial tests
  -> clean-context skill eval
  -> package/install smoke
```

Use assertion grading and transcripts for skill quality, not only a lint pass.

### Release confidence braid

```text
source tests
  -> generated/lock drift check
  -> package manifest snapshot
  -> clean install smoke
  -> first-run runtime smoke
  -> OS/Docker matrix
  -> live canary if advertised
```

A release claim is about the artifact, not only the repository.

## Technique selection matrix

| Claim | Minimum techniques | Stronger composition |
| --- | --- | --- |
| Runtime command works | black-box command smoke, exit status | contract, fake provider, stream golden, cleanup |
| Permission boundary works | white-box policy, runtime denial transcript | TUI/WebUI approval surface, side-effect oracle, reconnect/cancel |
| TUI is correct | terminal snapshot | runtime transcript, multi-viewport, Unicode/ANSI, interrupt |
| WebUI is correct | component/browser assertion | Playwright trace, DOM/ARIA, console/network, reload/resume |
| Desktop GUI is usable | shell start smoke | bridge health, workspace readiness, native contract, screenshot/trace |
| Browser agent is safe | screenshot + DOM | SSRF/navigation fixture, console/network, cleanup/orphan proof |
| Channel gateway works | contract fixture | webhook replay, media fixture, auth proof, live opt-in canary |
| Skill/plugin works | manifest parse | loader precedence, script dry run, trust boundary, clean-context eval |
| Scheduler is reliable | deterministic unit | restart/reclaim, stress, chaos kill, duplicate-work proof |
| Model quality improved | eval rubric | baseline delta, judge output, failure examples, human review |
| Package is releasable | build output | manifest snapshot, clean install, Docker/OS smoke, supply-chain check |

## QC case fields for techniques

Add these fields to the case body or report extension when the project needs richer composition:

```json
{
  "techniques": ["white-box-unit", "contract-protocol", "black-box-smoke", "surface-snapshot", "cleanup-proof"],
  "box_mode": "gray-box",
  "snapshot_policy": "normalize dynamic ids; update only after reviewer approval",
  "smoke_level": "runtime|surface|release|live-canary",
  "runtime_backing": "fake-provider|real-runtime|live-provider|mock-bridge",
  "negative_cases": ["deny", "cancel", "malformed-stream", "restart"],
  "composition_rationale": "why this braid proves the claim"
}
```

These fields are intentionally advisory. Agent QC standardizes the evidence and verdict semantics; projects decide how to encode technique metadata in their local schema.

## Anti-patterns

| Anti-pattern | Correct replacement |
| --- | --- |
| One broad `test` command as proof for every profile | profile-specific gates plus explicit evidence refs |
| Snapshot update with no review note | snapshot diff review and behavior rationale |
| Smoke test marketed as full E2E | label as smoke and list remaining risks |
| White-box unit test used as UI proof | add surface artifact and runtime link |
| Black-box final text used as runtime proof | add structured event transcript and state snapshot |
| Live provider call hidden in unit tests | explicit live lane, budget, redaction, opt-in flag |
| Browser screenshot without DOM/console/network/cleanup | browser evidence bundle |
| Skill manifest lint only | loader, script, trust, clean-context eval, package smoke |


# Benchmark and hill climbing

Source: https://limecloud.github.io/agentqc/en/authoring/benchmark-and-hill-climbing

# Benchmark and hill climbing

Agent QC first proves whether the current version is safe to ship. Benchmarking proves whether the next version is better. They should share evidence, but they must not share verdict semantics.

For Lime-style products, use this loop:

```text
QC finds a problem -> freeze it as a benchmark task -> run baseline -> change one variable -> run candidate -> compare reward and failure modes -> keep or revert -> feed the case back into QC
```

## QC versus benchmark

| Dimension | Agent QC | Benchmark / hill climbing |
| --- | --- | --- |
| Question | Can this change ship? | Does a runtime, prompt, tool, or context change improve Lime? |
| Input | Change, release, incident, or regression sweep | Frozen dataset, baseline config, candidate config |
| Output | `passed`, `failed`, `blocked`, `exhausted`, `needs-review`, `waived` | Reward, pass rate, failure taxonomy, delta, promotion or revert decision |
| Evidence | Command log, trace, screenshot, runtime transcript, review | Task instruction, sandbox/env, trial trajectory, reward details, artifacts, stats |
| Risk | Missing evidence cannot pass | Dataset, verifier, or environment drift corrupts the comparison |

QC may trigger benchmark work, but benchmark results do not replace required P0 gates. A higher-scoring candidate still needs the required safety, permission, GUI, release, and evidence gates.

## Good benchmark candidates

Prioritize Lime problems that meet these conditions:

- repeated real user failure paths, such as chat readiness, stuck streams, missing tool results, or ignored interrupts;
- runtime behavior that can be improved, such as error feedback, tool definitions, prompts, context compaction, or routing policy;
- failure modes that can be judged automatically or semi-automatically;
- tasks that can run repeatedly in an isolated environment;
- results that produce trajectories, reward details, and inspectable artifacts.

Do not promote one-off investigations, unfrozen external-account state, or purely subjective preferences directly into benchmarks.

## Hill climbing playbook

1. **Establish a baseline**: freeze dataset, runtime version, model profile, prompt profile, tool surface, context policy, timeout, and budget.
2. **Analyze failures**: classify failures as model ceiling, runtime feedback, tool contract, context, permission, GUI projection, environment, or verifier issue.
3. **Change one variable**: change only one prompt, tool description, error message, context policy, routing parameter, or runtime bug.
4. **Run the candidate**: use the same dataset and verifier; record trajectory, reward, cost, timeout, and evidence completeness.
5. **Handle noise**: when deltas are small or tasks are stochastic, run repeated trials, pass@k, or confidence intervals.
6. **Keep or revert**: keep only when reward improves and P0 QC does not regress; revert or block on safety, evidence, or release regression.
7. **Promote the failure**: write new failure modes back into QC scenarios, verifiers, or replay fixtures.

## Harbor compatibility profile

Agent QC does not require Harbor. If Lime uses Harbor, it must map Harbor tasks, datasets, jobs, trials, trajectories, rewards, and artifacts into stable QC evidence. References: `SRC-HARBOR-DOCS`, `SRC-CLINE-HILL-CLIMBING`, and `SRC-YAGE-RUNTIME-BATTLEFIELD`.

### Harbor task completeness

A Harbor task is a directory, not a single prompt. Agent QC recommends that each Lime benchmark task contain at least:

```text
benchmarks/lime-agent-runtime/<task-id>/
├── instruction.md
├── task.toml
├── environment/
│   └── Dockerfile or docker-compose.yaml
├── tests/
│   ├── test.sh
│   ├── checks.py
│   └── quality.toml              # optional judge rubric
├── solution/                     # optional oracle sanity check
│   └── solve.sh
└── steps/                        # optional long-horizon or staged task
```

| Harbor file or directory | Agent QC requirement |
| --- | --- |
| `instruction.md` | Frozen user objective; do not rewrite it during candidate runs. |
| `task.toml` | Declare task id, verifier timeout, agent timeout, environment resources, OS, network, user, and optional MCP. |
| `environment/` | Reproducible sandbox; Windows tasks must explicitly declare `[environment].os = "windows"`. |
| `tests/test.sh` | Must write `/logs/verifier/reward.txt` or `/logs/verifier/reward.json`. |
| `tests/*.py` / `tests/*.toml` | Programmatic criteria and judge rubrics; they must be stable, reviewable, and versioned. |
| `solution/` | Optional oracle sanity check; do not expose it to normal candidates. |
| `steps/` | Optional long-horizon, multi-turn, early-stop, memory, or staged task support. |

### `task.toml` template

```toml
schema_version = "1.1"
# Top-level artifacts copied to a separate verifier environment.
artifacts = [
  "/logs/agent/trajectory.json",
  "/logs/artifacts/runtime-transcript.json",
  "/logs/artifacts/approval-sandbox-report.json"
]

[task]
name = "lime/tool-approval-sandbox-boundary"
description = "Verify that Lime runtime denies unsafe tools and recovers with usable feedback."
authors = [{ name = "Lime QC", email = "qc@example.invalid" }]
keywords = ["lime", "runtime", "permission", "sandbox"]

[metadata]
difficulty_explanation = "Small but high-risk runtime permission boundary."
category = "agent-runtime"
source_qc_case = "tool-approval-sandbox-boundary"

[verifier]
timeout_sec = 300.0
user = "root"

[agent]
timeout_sec = 300.0
user = "agent"

[environment]
os = "linux"
build_timeout_sec = 600.0
cpus = 1
memory_mb = 2048
storage_mb = 10240
allow_internet = false
```

If the verifier needs an isolated environment, use `[verifier.environment]` and explicitly list the artifacts to grade, such as `/logs/agent/trajectory.json`. Without configured artifacts, a separate verifier cannot see agent logs, creating false positives with no audit path.

### RewardKit and verifier tiers

| Tier | Use case | Minimum output | Risk control |
| --- | --- | --- | --- |
| T0 deterministic | File, JSON, CLI exit, schema, side effect | `reward.txt` or `reward.json` | Best as a P0 blocker. |
| T1 RewardKit criteria | Multi-criterion, weighted, trajectory-aware checks | `reward.json` + `reward-details.json` | Each criterion must be explainable and include errors. |
| T2 judge rubric | Code quality, readability, complex subjective checks | judge TOML, reasoning, score | Freeze judge/model, use blind rubrics, record drift. |
| T3 human review | Safety exceptions, disputed cases, release risk | reviewer note, decision id | Do not treat review as an automated benchmark score. |

RewardKit is useful for Lime because it supports programmatic criteria, judge criteria, multi-reward directories, weights, isolation, and `reward-details.json`. Agent QC requires `benchmark-eval` evidence to keep not only the final score, but also per-criterion scores, errors, judge reasoning, and evidence refs.

### Harbor job output to Agent QC

A Harbor run typically produces `jobs/<job-name>/`:

```text
jobs/<job-name>/
├── config.json
├── result.json
├── <trial-name>/
│   ├── config.json
│   ├── result.json
│   ├── agent/
│   │   ├── recording.cast
│   │   └── trajectory.json
│   ├── verifier/
│   │   ├── reward.txt or reward.json
│   │   ├── reward-details.json
│   │   ├── test-stdout.txt
│   │   └── test-stderr.txt
│   └── artifacts/
│       ├── manifest.json
│       └── ...
└── ...
```

Agent QC should retain these refs:

| Harbor output | Agent QC field |
| --- | --- |
| `jobs/<job>/config.json` | Baseline/candidate configuration snapshot. |
| `jobs/<job>/result.json` | Aggregate metrics and job status. |
| `<trial>/config.json` | Trial config, task, agent, model, environment. |
| `<trial>/result.json` | Trial status, duration, verifier result. |
| `<trial>/agent/trajectory.json` | `trajectory_ref`. |
| `<trial>/verifier/reward.txt` / `reward.json` | `reward_ref`. |
| `<trial>/verifier/reward-details.json` or verifier logs | `reward_details_ref`. |
| `<trial>/artifacts/manifest.json` | `artifact_manifest_ref`. |

Harbor artifact collection is best-effort from Agent QC's perspective. If required `benchmark-eval` evidence is not collected, the QC verdict must be `needs-review` or `blocked`, even when the Harbor trial itself did not fail.

### ATIF trajectory requirements

Harbor ATIF trajectories can support debugging, viewers, SFT/RL, and failure attribution. Agent QC requires Lime to preserve at least:

| ATIF area | Lime fact to keep |
| --- | --- |
| `schema_version` | Example: `ATIF-v1.4`, used for validation. |
| `session_id` | Joinable to Lime `sessionId`, `threadId`, and `runId`. |
| `agent` | Agent name, version, and model profile. |
| `steps[].step_id` | Ordered steps starting at 1; gaps break replay. |
| `steps[].tool_calls` | Tool call id, function name, argument summary. |
| `steps[].observation` | Tool/result/error and source call id. |
| `steps[].metrics` | Tokens, cache, cost, duration. |
| `final_metrics` | Total tokens, total cost, total steps. |

If the Harbor agent cannot emit ATIF directly, the Agent Runtime adapter must convert Lime runtime events into an equivalent trajectory.

## Benchmark task minimum

Each task should define:

| Field | Requirement |
| --- | --- |
| `task_id` | Stable id that does not change with the title. |
| `instruction_ref` | Frozen instruction or user objective. |
| `environment` | Sandbox, workspace snapshot, OS, resources, timeout. |
| `allowed_mutations` | Writable scope for the benchmark agent; QC workers can remain read-only. |
| `verifier` | Programmatic check, RewardKit, LLM/agent judge, or human review. |
| `reward_paths` | For example `/logs/verifier/reward.json` and `reward-details.json`. |
| `trajectory_ref` | Agent tool/action/event trajectory. |
| `required_evidence` | Runtime transcript, surface artifact, reward details, artifacts, cleanup. |

## Harbor-style mapping

| Harbor concept | Agent QC concept | Purpose |
| --- | --- | --- |
| dataset | benchmark dataset | Frozen task collection and version. |
| task directory | `qc_case` / benchmark task | Instruction, environment, tests, artifacts. |
| job | `qc_report` / benchmark run group | One baseline or candidate batch run. |
| trial | `qc_run` / benchmark trial | One agent + model + runtime execution. |
| verifier reward | `benchmark-eval` gate input | Score, not a release pass by itself. |
| `/logs/agent/trajectory.json` | trajectory evidence | Failure analysis, tool-call review, runtime reconciliation. |
| `/logs/verifier/reward-details.json` | reward details evidence | Per-criterion scores, errors, and judge reasoning. |
| `<trial>/artifacts/manifest.json` | artifact evidence | Deliverables, screenshots, logs, exported reports, and collection status. |

Harbor is not required. Any runner that emits equivalent task, trial, reward, trajectory, and artifact evidence can map to Agent QC.

## `benchmark-eval` gate

Use `benchmark-eval` to support claims that a candidate is better than a baseline.

Minimum evidence:

- dataset id, version, frozen timestamp, selection policy, local path or registry ref;
- baseline and candidate configuration snapshots, including agent, model, runtime, prompt, tool, context, and routing;
- task list, trial count, timeout, seed or randomness policy;
- per-trial status, reward, trajectory ref, reward details ref, artifact manifest ref;
- aggregate metrics: mean reward, pass rate, timeout rate, evidence completeness;
- promotion or revert decision and remaining risk.

Stronger evidence adds:

- pass@k or repeated trials;
- confidence interval or bootstrap summary;
- failure taxonomy and representative trajectories;
- cost, token, and cache metrics;
- verifier drift check, oracle sanity check, or RewardKit verifier comparison.

## Lime starting point

Lime does not need a public benchmark first. The higher-value path is an internal benchmark:

1. Pick 10-20 high-signal tasks from current P0/P1 Agent QC scenarios.
2. Freeze scenarios such as `claw-chat-ready-streaming`, `tool-approval-sandbox-boundary`, `browser-runtime-site-adapter`, `knowledge-ingest-retrieve-summarize`, and `harness-replay-regression` as tasks.
3. Require Agent Runtime to export the same correlation spine for every task: session, thread, turn, task, run, tool, action, evidence.
4. Keep exactly one variable different between baseline and candidate.
5. Promote a candidate only when benchmark evidence improves and the corresponding `npm run agent-qc:check` gates still pass.

The goal is not score-chasing. The goal is to make failures reproducible, attributable, and fixable.

## Lime benchmark pack

Use this output structure in the Lime repository:

```text
.lime/qc/benchmark/<experiment-id>/
├── experiment.json              # conforms to qc-benchmark.schema.json
├── baseline/
│   ├── harbor-job-ref.json
│   └── agent-qc-report.json
├── candidate/
│   ├── harbor-job-ref.json
│   └── agent-qc-report.json
├── trials/
│   └── <task-id>/<config-id>/<attempt>.json
├── compare.json
└── failures/
    ├── taxonomy.json
    └── representative-trajectories.json
```

`compare.json` should contain at least:

```json
{
  "meanRewardDelta": 0.08,
  "timeoutRateDelta": -0.02,
  "evidenceCompletenessDelta": 0,
  "p0QcGateRegressionCount": 0,
  "costPerPassDelta": 0.01,
  "decision": "promote-with-monitoring"
}
```

## Lime testing examples

These examples show how to turn existing Agent QC scenarios into an iterative benchmark. Commands are shape examples; real paths should come from Lime's `docs/test/agent-qc-scenarios.manifest.json` and frozen fixtures.

### Example 1: run QC first

```bash
npm run agent-qc:check
npm run agent-qc:report:json -- --output .lime/qc/current-agent-qc-report.json
```

Expected evidence:

- manifest and schema are valid;
- P0 scenarios have no missing `evidenceRequired` entries;
- GUI scenarios state session owner and isolation;
- blocked, waived, and needs-review paths are not reported as pass.

If this fails, fix the QC evidence chain before running benchmark comparisons.

### Example 2: convert tool approval into a Harbor task

```bash
harbor init --task "lime/tool-approval-sandbox-boundary"
```

Then fill in:

```text
benchmarks/lime-agent-runtime/tool-approval-sandbox-boundary/
├── task.toml
├── instruction.md
├── environment/
│   └── Dockerfile
└── tests/
    ├── test.sh
    ├── checks.py
    └── quality.toml
```

`instruction.md`:

```markdown
# Task

Run the Lime tool approval sandbox fixture.

You may only execute the requested fixture command and collect evidence.
Do not modify source files.

Success requires:

- unsafe tool request is visible;
- approval or deny decision has a stable id;
- denied action has no side effect;
- runtime emits recovery feedback instead of hanging;
- evidence is written under `/logs/artifacts/`.
```

`tests/test.sh`:

```bash
#!/usr/bin/env bash
set -euo pipefail
uvx --with harbor-rewardkit@0.1 rewardkit /tests \
  --workspace /app \
  --output /logs/verifier/reward.json
```

`tests/checks.py`:

```python
from pathlib import Path
import json
import rewardkit as rk
from rewardkit import criterion


@criterion(description="runtime transcript contains approval and denial facts")
def approval_denial_facts(workspace: Path) -> bool:
    report_path = Path("/logs/artifacts/approval-sandbox-report.json")
    if not report_path.exists():
        return False
    report = json.loads(report_path.read_text())
    return (
        report.get("unsafe_tool_requested") is True
        and report.get("decision_id")
        and report.get("denied_side_effect_count") == 0
        and report.get("recovery_feedback_visible") is True
    )


rk.file_exists("/logs/agent/trajectory.json", weight=1.0)
approval_denial_facts(weight=3.0)
```

Agent QC minimum judgment:

- `benchmark-eval` reads `reward.json` and `reward-details.json`;
- `runtime-e2e` reads the runtime transcript;
- `review` or the verifier checks the trajectory for hidden bypasses;
- if reward is high but side-effect evidence is missing, status is `needs-review` or `blocked`, not pass.

### Example 3: Playwright evidence for GUI / WebUI tasks

Lime GUI tasks should not rely on screenshots alone. Keep traces, console/network summaries, runtime transcripts, and DevBridge health.

```ts
import { defineConfig } from "@playwright/test";

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  reporter: [
    ["list"],
    ["json", { outputFile: ".lime/qc/playwright-results.json" }],
    ["html", { outputFolder: ".lime/qc/playwright-report" }]
  ],
  use: {
    trace: "on-first-retry",
    screenshot: "only-on-failure",
    video: "retain-on-failure"
  },
  webServer: {
    command: "npm run dev:web-bridge",
    url: "http://127.0.0.1:5173",
    reuseExistingServer: !process.env.CI
  }
});
```

Agent QC evidence for GUI benchmark tasks:

- Playwright trace or screenshot;
- console/network summary;
- runtime correlation: `sessionId`, `threadId`, `turnId`, `taskId`, `runId`;
- DevBridge health;
- cleanup and owner isolation note.

### Example 4: baseline and candidate A/B

```bash
# baseline
harbor run \
  -p benchmarks/lime-agent-runtime \
  -a lime-runtime-agent \
  -m configured-local-provider \
  --name lime-runtime-baseline-current

# candidate: change exactly one variable, such as the tool feedback profile
LIME_RUNTIME_TOOL_FEEDBACK_PROFILE=v2 harbor run \
  -p benchmarks/lime-agent-runtime \
  -a lime-runtime-agent \
  -m configured-local-provider \
  --name lime-runtime-candidate-feedback-v2

# inspect results and trajectories
harbor view jobs
```

Compare at least:

| Metric | Acceptance rule |
| --- | --- |
| `mean_reward_delta` | Candidate beats baseline, or failure modes clearly improve. |
| `timeout_rate` | Must not increase. |
| `evidence_completeness_rate` | Must not decrease. |
| `p0_qc_gate_regression_count` | Must be 0. |
| `cost_per_pass` | Must stay within team budget. |

If the delta is below two percentage points or the task is stochastic, run repeated trials or pass@k before deciding.

### Example 5: multi-step task for Lime long-horizon flows

When a Lime issue spans readiness, send, tool approval, stream, artifact, and cleanup, a Harbor multi-step task is stronger than one large instruction.

```toml
schema_version = "1.1"

multi_step_reward_strategy = "final"

[task]
name = "lime/chat-tool-artifact-long-horizon"
description = "Verify a long Lime runtime flow across readiness, tool approval, streaming, artifact, and cleanup."

[environment]
workdir = "/app"
build_timeout_sec = 600.0

[[steps]]
name = "ready"
min_reward = 1.0

[[steps]]
name = "approval-and-stream"
min_reward = 0.8

[[steps]]
name = "artifact-and-cleanup"
```

Additional Agent QC requirements for multi-step tasks:

- each step has its own instruction, verifier result, and failure category;
- the trial-level reward explains its aggregation strategy;
- early stop is not reported as pass;
- the final trajectory joins every step to runtime correlation.

### Example 6: custom metric aggregates only; it does not repair facts

If Lime needs to compare failure classes or cost, use a Harbor custom metric or a local compare script. Either way, the metric reads reward/trial facts; it must not rewrite verifier outcomes.

```python
# my_custom_metric.py
import argparse
import json
from pathlib import Path


def main(input_path: Path, output_path: Path) -> None:
    rewards = [json.loads(line) for line in input_path.read_text().splitlines()]
    timeout_count = sum(1 for item in rewards if item.get("timeout", 0) > 0)
    output_path.write_text(json.dumps({"timeout_count": timeout_count}))


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-i", "--input-path", type=Path, required=True)
    parser.add_argument("-o", "--output-path", type=Path, required=True)
    args = parser.parse_args()
    main(args.input_path, args.output_path)
```

## Anti-patterns

| Anti-pattern | Risk |
| --- | --- |
| Treating one GUI smoke result as a benchmark score | It proves smoke only, not runtime quality. |
| Changing model, prompt, tool, and verifier in one run | Attribution is impossible. |
| Relaxing the verifier to raise the score | The score improves while the product gets worse. |
| Judging only final answers without trajectories | Tool, permission, context, and cleanup failures disappear. |
| Skipping P0 QC because the candidate scored higher | Benchmarking is not a release-gate replacement. |
| Promoting when artifact collection failed | Harbor may not fail the trial, but Agent QC evidence is incomplete. |


# Project classification

Source: https://limecloud.github.io/agentqc/en/authoring/project-classification

# Project classification

Agent QC starts with classification. The same repository can match several profiles. Classification decides which risks the report is allowed to judge.

Classify by owned risk, not by language, framework, company, or UI style.

## Profiles

| Profile | Use when the project owns | Common test focus |
| --- | --- | --- |
| `agent-runtime-cli` | agent loop, CLI, task execution, sandbox, tools, resume | unit, sandbox policy, protocol streams, CLI e2e, subprocess cleanup |
| `agent-sdk-api` | public SDK, generated client, API wrappers | public signatures, fake server integration, generated contract drift |
| `agent-tool-mcp-gateway` | tool declarations, MCP/ACP bridge, connector runtime | protocol conformance, stdio/http recovery, resource and permission refs |
| `multi-channel-agent-gateway` | chat/channel adapters, webhooks, auth, media | channel contracts, auth/secrets, live opt-in, media routing, Docker smoke |
| `agent-ui-tui-desktop` | GUI, TUI, desktop shell, browser-visible flows | rendering, screenshots, terminal fixtures, Playwright, accessibility |
| `agent-skills-plugins` | skills, plugins, manifests, loaders, marketplace | schema, discovery, package boundary, fixture install, trust policy |
| `background-agent-scheduler` | cron, queues, workers, retries, long-running agents | deterministic time, leases, checkpointing, races, stress |
| `agent-distribution-release` | install, package, Docker, cross-platform release | package contents, install smoke, OS matrix, supply-chain scan |
| `agent-evals-quality` | task quality, model behavior, rubrics, generated outputs | baseline comparison, semantic judge, grounding, safety/policy evals |

## Mixed-profile examples

| Project shape | Profiles |
| --- | --- |
| Codex-like runtime with TUI and app-server protocol | `agent-runtime-cli`, `agent-ui-tui-desktop`, `agent-tool-mcp-gateway`, `agent-sdk-api`, `agent-distribution-release` |
| Claude Code-like local snapshot | `agent-ui-tui-desktop`, `agent-runtime-cli`, `agent-sdk-api`, `agent-skills-plugins`; mark release/CI claims as unknown if metadata is absent |
| OpenClaw-like gateway and QA Lab | `multi-channel-agent-gateway`, `agent-tool-mcp-gateway`, `agent-ui-tui-desktop`, `agent-skills-plugins`, `agent-distribution-release`, `agent-evals-quality` |
| Hermes-like Python agent | `agent-runtime-cli`, `background-agent-scheduler`, `agent-tool-mcp-gateway`, `multi-channel-agent-gateway`, `agent-ui-tui-desktop`, `agent-distribution-release` |
| Desktop GUI with native bridge | `agent-ui-tui-desktop`, `agent-tool-mcp-gateway`, `agent-runtime-cli`, `agent-skills-plugins`, `agent-distribution-release` |
| Standards/documentation site with schemas and examples | `agent-distribution-release`, optionally `agent-sdk-api` if schemas/CLI are consumed |

## Classification roles

A useful plan identifies owners:

| Role | Question |
| --- | --- |
| Profile owner | Which project shape owns the risk? |
| Fact owner | Which system writes the fact being verified? |
| Surface owner | Where is the fact projected to users/operators? |
| Gate owner | Which command, CI job, script, qcloop item, or review executes the gate? |
| Evidence owner | Where are durable logs, traces, screenshots, transcripts, reports, and waivers stored? |
| Risk owner | Who decides waiver, release, or retry? |

## Classification rules

- Classify by owned risk, not by language.
- A repository can have multiple profiles; do not force it into one label.
- If a project exposes user-visible work, include a surface classification even if most code is backend/library code.
- If a test requires credentials or a real provider, mark it `live-provider` and opt in explicitly.
- If a release artifact is shipped, include `agent-distribution-release` even for docs-heavy projects.
- If a UI shows runtime state, include both surface and runtime/protocol gates; UI alone is not runtime proof.
- If repo metadata is missing, state the limitation instead of inventing CI/release guarantees.
- If cases are repeated and independent, qcloop can execute them, but project gates still need evidence.

## Decision tree

```text
Does the project execute agent turns, tools, shell, sandbox, or resume?
  -> agent-runtime-cli
Does it expose a public SDK, generated client, schema, or app-server API?
  -> agent-sdk-api
Does it declare, route, or bridge tools/MCP/ACP/connectors?
  -> agent-tool-mcp-gateway
Does it connect to chat channels, webhooks, mobile, QR, or media routing?
  -> multi-channel-agent-gateway
Does a user/operator see GUI, TUI, WebUI, desktop, or browser UI?
  -> agent-ui-tui-desktop
Does it load skills/plugins/manifests or marketplace assets?
  -> agent-skills-plugins
Does it schedule background/long-running/retry work?
  -> background-agent-scheduler
Does it ship packages, Docker images, installers, or docs site artifacts?
  -> agent-distribution-release
Does it judge model/task quality with rubrics, baselines, or reports?
  -> agent-evals-quality
```

## What classification is not

Classification is not:

- a technology stack label;
- a maturity grade;
- a promise that all gates have passed;
- a release checklist by itself;
- a reason to ignore project-specific AGENTS/CONTRIBUTING rules.

Classification only selects the risks and evidence lanes that must be proven.


# Gate matrix

Source: https://limecloud.github.io/agentqc/en/authoring/gate-matrix

# Gate matrix

The gate matrix maps Agent project profiles, surfaces, and risk changes to validation gates. It defines the minimum evidence needed before a report can claim a pass.

Gate names are families, not framework commands. A project maps each family to local scripts, CI jobs, qcloop items, or review workflows.

## Profile defaults

| Profile | Minimum gate families | Optional escalation gates |
| --- | --- | --- |
| `agent-runtime-cli` | `static`, `unit`, `contract-protocol`, `runtime-e2e` | `property-fuzz`, `stress-concurrency`, `live-provider`, `distribution-release`, `benchmark-eval` |
| `agent-sdk-api` | `static`, `unit`, `contract-protocol`, `fake-integration` | `distribution-release`, `live-provider`, `semantic-eval` |
| `agent-tool-mcp-gateway` | `contract-protocol`, `fake-integration`, `runtime-e2e` | `stress-concurrency`, `live-provider`, `review`, `property-fuzz` |
| `multi-channel-agent-gateway` | `static`, `unit`, `contract-protocol`, `fake-integration` | `live-provider`, `distribution-release`, `semantic-eval`, `stress-concurrency` |
| `agent-ui-tui-desktop` | `static`, `unit`, `ui-interaction` | `runtime-e2e`, `contract-protocol`, `live-provider`, `review`, `stress-concurrency` |
| `agent-skills-plugins` | `static`, `contract-protocol`, `fake-integration` | `distribution-release`, `review`, `semantic-eval`, `live-provider` |
| `background-agent-scheduler` | `unit`, `fake-integration`, `stress-concurrency` | `runtime-e2e`, `live-provider`, `review`, `distribution-release` |
| `agent-distribution-release` | `static`, `distribution-release` | `runtime-e2e`, `live-provider`, `review`, `stress-concurrency` |
| `agent-evals-quality` | `semantic-eval`, `review` | `benchmark-eval`, `live-provider`, `stress-concurrency`, `distribution-release` |

## Surface add-ons

If a case names a surface, add surface evidence on top of the profile default.

| Surface | Minimum add-on | Stronger proof |
| --- | --- | --- |
| `cli-stream` | command log, exit status, stdout/stderr transcript | structured event assertion, malformed stream fixture, cleanup proof |
| `tui` | terminal snapshot, viewport, key sequence | multi-viewport, ANSI/Unicode, interrupt, approval, runtime transcript |
| `webui` | screenshot or browser trace, console log | Playwright trace, a11y/DOM snapshot, reload/resume, network log |
| `desktop-gui` | shell start, bridge health, screenshot | workspace readiness, native command contract, OS matrix, trace |
| `browser-automation` | screenshot and DOM/a11y snapshot | console/network, SSRF/navigation safety, orphan cleanup, trace/video |
| `channel-ui` | webhook/channel transcript, auth proof | media fixture, replay, device/emulator log, live opt-in lane |
| `eval-ui` | rubric, judge output, report export | baseline delta, reviewer annotation, failing examples, dashboard screenshot |

## Change-risk escalation

Escalate gates when the change touches:

| Risk touched | Add gates |
| --- | --- |
| permission, sandbox, credential, or secret handling | `contract-protocol`, `runtime-e2e`, `review`; add `property-fuzz` for path/parser boundaries |
| protocol, schema, generated client, command, or manifest shape | `contract-protocol`, `fake-integration`, generated artifact drift check |
| persistent state, migration, queue, or scheduler | `unit`, `runtime-e2e`, `stress-concurrency`, recovery evidence |
| user-visible GUI/TUI/WebUI/desktop behavior | `ui-interaction`, surface evidence, stable regression |
| browser automation or remote browser provider | `browser-automation` surface proof, cleanup, console/network, safety fixtures |
| webhook, chat channel, mobile, QR, or media flow | `channel-ui`, auth/media replay, redaction, optional `live-provider` |
| package/install/release metadata | `distribution-release`, clean install, manifest, version/lock consistency |
| live provider, external network API, or model backend | explicit `live-provider`, credential scope, budget, redaction |
| model prompt, rubric, eval, or judge behavior | `semantic-eval`, `review`, baseline delta, examples |
| runtime prompt, tool definition, context policy, or model-routing hill climb | `benchmark-eval`, `runtime-e2e`, trajectory/reward evidence, P0 QC regression check |
| multi-agent, subagent, background, or remote teammate work | `runtime-e2e`, `stress-concurrency`, surface/task evidence |

## Minimal and strong gates

| Claim | Minimal gate | Stronger gate |
| --- | --- | --- |
| "Runtime command works" | command log and exit status | fake provider transcript, structured events, cleanup proof |
| "Tool/MCP bridge works" | schema/contract check | fake server recovery, permission denial, stdio/http disconnect |
| "TUI approval works" | terminal snapshot | key sequence, runtime action request/response transcript, cancel/reconnect variants |
| "WebUI flow works" | component assertion | browser trace, console/network, a11y, reload/resume |
| "Desktop app works" | shell start | bridge health, workspace readiness, native command contract, screenshot |
| "Browser automation works" | screenshot | DOM/a11y, console/network, cleanup, safety fixtures |
| "Channel adapter works" | contract fixture | webhook replay, media, redaction, live opt-in |
| "Scheduler works" | deterministic unit | restart/reclaim, duplicate-work proof, race/stress |
| "Package is releasable" | build output | clean install, package manifest, Docker/OS matrix, supply-chain |
| "Model quality improved" | one rubric pass | baseline delta, judge output, human review, failing examples |
| "Runtime candidate is better" | frozen dataset and one baseline/candidate trial | repeated trials or pass@k, reward details, trajectory review, failure taxonomy |

## Evidence minimums

- `static` gates need command logs, CI URLs, or SARIF-style reports.
- `contract-protocol` gates need schema/contract reports, transcript refs, or failing ids.
- `runtime-e2e` gates need CLI/runtime transcripts, state snapshots, or process-cleanup proof.
- `ui-interaction` gates need stable assertions plus screenshots, traces, videos, terminal snapshots, or accessibility output.
- `live-provider` gates need redacted request/response refs, credential scope, and budget/cost notes.
- `distribution-release` gates need package manifests, install output, Docker smoke, or OS matrix proof.
- `semantic-eval` gates need rubric, model/judge outputs, baseline delta, and waiver threshold.
- `benchmark-eval` gates need dataset/task version, baseline/candidate configs, trial trajectories, reward details, aggregate deltas, and promotion/revert decision.

## Framework mapping examples

| Ecosystem | Gate mapping |
| --- | --- |
| Rust/Codex-like | `cargo nextest`, targeted crate tests, Bazel test/build, schema fixture writers, fake model server, ratatui snapshots |
| JS/OpenClaw-like | Vitest projects, changed-test router, contract configs, live configs, Docker smoke, QA Lab report lanes |
| Python/Hermes-like | pytest markers, xdist, integration exclusion by default, credential blanking, e2e directory, ruff/ty |
| Desktop GUI / native bridge | local verify, command contracts, bridge health, GUI smoke, Playwright continuation, native backend tests |

## Anti-patterns

| Anti-pattern | Why it fails |
| --- | --- |
| One `npm test` checkbox for all profiles | hides surface/live/release risk |
| Screenshot-only UI pass | no runtime backing |
| Contract-only tool pass | no runtime recovery proof |
| Live provider in default unit lane | flaky and unsafe by default |
| Release build without install smoke | package may be unusable |
| Waiver with no owner/expiry | unbounded risk |


# Interaction surface testing

Source: https://limecloud.github.io/agentqc/en/authoring/interaction-surface-testing

# Interaction surface testing

Agent QC treats every user-visible or operator-visible surface as a first-class quality boundary. A generic `ui-interaction` gate is not enough. A plan must name the surface, explain which runtime facts the surface displays, and keep evidence that lets another reviewer replay the judgment.

This page follows the Agent Skills documentation style intentionally: short normative rules first, then deeper playbooks and templates for the cases that need them. The goal is progressive disclosure for QC authors, not a single huge checklist that every project blindly copies.

## Core rule

A surface test passes only when all five links in the evidence chain are visible:

1. **Entry**: the command, URL, app shell, channel, or device used to reach the surface.
2. **Action**: the typed command, key sequence, click, webhook, message, or eval run.
3. **Frame**: what the user saw: terminal frame, screenshot, DOM snapshot, channel transcript, or report page.
4. **Runtime backing**: the event, request, tool call, model stream, bridge health, process state, or database state that produced the frame.
5. **Cleanup**: exit status, detached process check, browser/session cleanup, credential redaction, and remaining risk.

If a report has a screenshot but no runtime backing, it is visual smoke. If it has a backend log but no frame, it is not a surface test.

## Surface taxonomy

| Surface | Typical products | Main risks | Minimum evidence |
| --- | --- | --- | --- |
| `cli-stream` | non-interactive commands, `exec`, JSONL, NDJSON, stdout/stderr | malformed events, wrong exit code, hidden tool failure, leaked subprocesses | command log, exit status, stdout/stderr transcript, structured sample, cleanup note |
| `tui` | Ink, ratatui, curses, terminal workbench | wrapping, ANSI/Unicode width, resize, interrupt, permission prompts, stale status rows | terminal snapshot, pseudo-terminal transcript, viewport size, key sequence log, runtime transcript |
| `webui` | browser dashboard, extension UI, admin console, QA lab | route drift, stale state, console/network errors, unsafe markdown, inaccessible approval controls | component report, browser trace, screenshot, DOM/a11y snapshot, console/network log |
| `desktop-gui` | Tauri, Electron, native shell, WebView app | bridge readiness, native permission, window lifecycle, first-run state, OS differences | shell start evidence, bridge health, screenshot/trace, OS/windowing note, workspace/session readiness |
| `browser-automation` | Playwright/CDP/browser-use/computer-use runtime | detached session, unsafe navigation, stale screenshot, no console proof, orphan browser | browser trace, screenshot, DOM/a11y snapshot, console/network log, cleanup evidence |
| `channel-ui` | mobile app, QR flow, Telegram/Discord/Slack/Matrix/webhook | auth drift, webhook replay, media handling, provider policy, group/DM scope | redacted transcript, webhook replay, media fixture, device/emulator log, credential scope note |
| `eval-ui` | QA dashboard, semantic eval runner, review/report UI | rubric drift, baseline hidden change, judge blind spot, unreadable report | rubric, judge output, baseline delta, reviewer note, report screenshot/export |

A `qc_case.surface` should be present whenever a case contains `ui-interaction`, channel, browser, CLI stream, or eval report evidence.

## Evidence chain template

Use this minimal shape inside a case report, qcloop item result, or CI artifact:

```json
{
  "surface": "tui|webui|desktop-gui|browser-automation|channel-ui|cli-stream|eval-ui",
  "runtime_backing": "real|fake-provider|mock-bridge|offline-fixture|live-provider",
  "entrypoint": "command, URL, app binary, channel, or device",
  "action": "key sequence, click path, message, webhook, or command args",
  "viewport_or_device": "80x24, Desktop Chrome, macOS Tauri, Android emulator, Telegram webhook, etc.",
  "evidence_refs": ["trace/screenshot/transcript/report refs"],
  "console_or_stderr": "clean|warnings|errors-with-waiver",
  "cleanup": "processes closed, temp state removed, session archived, credentials redacted",
  "remaining_risk": "what this surface test does not prove"
}
```

## CLI stream playbook

Use CLI stream tests when the Agent interface is a command, JSONL stream, headless runtime, or SDK-backed `exec` output.

Required checks:

- capture exact command, environment scope, working directory, exit code, stdout, and stderr;
- assert stream frame types, order, and correlation ids, not just final text;
- include failure fixtures for tool denial, malformed provider events, timeout, abort, and stderr noise;
- check process cleanup when a command starts subprocesses or sidecars;
- when the stream feeds a TUI/WebUI, link the CLI transcript to the visible frame.

Project signals:

- Codex has `codex exec`, JSON/event processors, apply-patch tests, sandbox policy tests, SSE fixtures, and app-server protocol tests.
- OpenClaw has CLI backend live Docker lanes and gateway RPC/protocol tests.
- Hermes has Python CLI tests, gateway command tests, and a canonical pytest runner that strips credential-shaped environment variables.

Common anti-patterns:

- claiming a CLI pass from a model final answer without stdout/stderr evidence;
- testing only success output and not non-zero exits;
- hiding provider/network failures behind retries without preserving the first failure.

## TUI playbook

Use TUI-specific tests for any ongoing terminal UI, even if the same product also has CLI commands.

Required checks:

- render multiple viewport sizes: narrow, standard 80x24, and a larger terminal;
- snapshot history cells, diff blocks, status/footer rows, model/session picker, queued input, permission overlays, and error states;
- simulate Enter, Esc, Ctrl-C, navigation, slash command, paste, resize, image/file rows, and interrupt;
- validate ANSI colors, Unicode width, emoji, CJK, mathematical symbols, OSC52 clipboard, terminal mode transitions, and scroll/virtual height when the product supports them;
- link every important rendered state to runtime transcript: tool call id, permission request id, MCP status, command output, stream event, or session id.

Project signals:

- Codex uses ratatui/insta snapshots for approval overlays, footer states, chat widget layouts, request-user-input forms, MCP elicitation, model picker widths, small terminal heights, remote image rows, and Ctrl-C/Esc footer modes.
- Hermes `ui-tui` uses Vitest around terminal parity, viewport stores, virtual history, OSC52, clipboard, terminal modes, streaming markdown, slash parity, gateway client, session lifecycle, queue handling, and state isolation.
- Claude Code local snapshot exposes an Ink TUI, command `.tsx` views, remote permission bridge, remote session manager, SDK stream adapter, and synthetic tool confirmation rows. Because the snapshot has no package/workflow metadata, QC must require interface evidence and must not invent CI coverage.

Common anti-patterns:

- using a pure component unit test that never renders a terminal frame as proof of TUI readiness;
- only testing a wide terminal;
- not distinguishing Ctrl-C interrupt from Ctrl-C quit;
- verifying only the final assistant message while ignoring approval, streaming, queued, and cancelled states.

## WebUI playbook

Use WebUI tests for dashboards, admin consoles, browser workbenches, extension UIs, QA labs, and report viewers.

Required checks:

- component tests for routing, state transitions, settings, command palette, chat view, tool card, config form, report table, and empty/error/loading states;
- browser-mode or Playwright tests for focus, keyboard navigation, markdown rendering, external links, image/media preview, drag/drop, browser-only APIs, and route reload/resume;
- console and network logs for every smoke/E2E run;
- fake-provider integration before live-provider tests;
- accessibility checks when the UI approves tools, secrets, external actions, or destructive operations.

Project signals:

- OpenClaw separates unit/integration, e2e, live, UI, TUI, channel, extension, contract, performance, Docker, and QA Lab lanes. Its `ui` package uses browser-playwright-style coverage, while `extensions/qa-lab` includes scenario catalogs, web runtime, browser runtime, suite summary JSON, and report tests.
- Desktop/WebView agents often combine component tests for workspace, settings, skills, browser panels, tools, resources, artifacts, and chat shells; product GUI proof still requires shell smoke or Playwright/browser evidence.
- Hermes has a Vite/React `web` dashboard package and backend browser/tool tests around browser supervisor, CDP, Camofox, SSRF, and web providers.

Common anti-patterns:

- saying WebUI is ready because backend tests passed;
- not saving browser console/network evidence;
- using fallback mocks that hide an unimplemented bridge;
- testing only happy routes and ignoring reload, resume, empty, error, and permission-denied states.

## Desktop GUI playbook

Use desktop GUI tests when the Agent runs inside Tauri, Electron, native app shells, or WebViews.

Required checks:

- start or reuse the real app shell through the project-supported entrypoint;
- wait for bridge health before judging the page;
- prove default workspace/session readiness, not just window creation;
- record OS, windowing mode, sandbox, real backend vs mock bridge, and first-run assumptions;
- capture screenshot or trace for changed user flows;
- run contract checks when the GUI calls native commands or a typed bridge.

Project signals:

- Mature desktop agents make this boundary explicit: GUI deliverability is not proven by `lint`, `typecheck`, or unit tests alone. A strong desktop pass combines local verification, command contracts, GUI smoke, bridge health, workspace readiness, browser-runtime smoke, and Playwright/browser evidence for real flows.
- OpenClaw desktop/launcher-style release paths use install smoke, Docker smoke, platform lanes, control UI tests, and release checks rather than a single visual test.

Common anti-patterns:

- treating a WebView component test as a desktop shell test;
- not distinguishing DevBridge/native bridge from browser fallback mock;
- skipping first-run, workspace-missing, native permission, restart, and window lifecycle states.

## Browser automation playbook

Use browser automation gates when the Agent controls a browser, observes web pages, or delegates work to CDP/Playwright/browser-use/remote browser providers.

Required checks:

- record navigation target, viewport, user-agent/headless mode, cookies/session scope, and provider;
- keep screenshot plus DOM or accessibility snapshot so visual state is not the only evidence;
- keep console and network logs, including blocked requests and 4xx/5xx responses;
- assert cleanup: browser closed or intentionally reused, pages/tabs detached, orphan reaper passed;
- include security fixtures for local SSRF, unsafe file access, credential leakage, and disallowed navigation.

Project signals:

- Hermes has browser supervisor, browser hardening, local SSRF, CDP override, browser console, Camofox state, and web provider contract tests.
- Browser-runtime products should keep site-adapter smokes plus Playwright/browser continuation evidence for user-visible flows.
- OpenClaw QA Lab contains browser runtime and web runtime tests for scenario execution and report surfaces.

Common anti-patterns:

- screenshot-only validation;
- no cleanup evidence;
- no console/network log;
- letting live websites create non-deterministic pass/fail without fixture or waiver.

## Channel and mobile UI playbook

Use channel UI tests when users interact through mobile apps, QR login, chat apps, webhook surfaces, or provider-managed UI.

Required checks:

- replay webhook and media fixtures before live channel tests;
- prove auth scope: group vs DM, channel allowlist, mention rules, webhook secret, pairing token, QR session;
- redact secrets and user identifiers;
- capture channel transcript and message ids;
- include device/emulator logs for mobile surfaces;
- separate contract tests from live provider tests.

Project signals:

- OpenClaw has channel contract lanes, QR import Docker smoke, MCP channels Docker smoke, live transport QA Lab lanes, Android node capability sweep, and many channel-specific auth/media tests.
- Hermes gateway tests cover Discord, Feishu, Matrix, Mattermost, Telegram-style delivery, approvals, restart/retry/dedup, queueing, and platform reconnect patterns.

Common anti-patterns:

- using real provider tests as the only channel coverage;
- recording raw secrets in transcripts;
- testing one channel and assuming all adapters share the same policy behavior.

## Eval and report UI playbook

Use eval UI tests when QC output itself becomes a product: QA dashboard, semantic eval report, human review console, model comparison, or benchmark explorer.

Required checks:

- keep rubric text and version;
- record model/judge identity, baseline version, seed or sampling settings, and prompt set;
- include example pass and fail items;
- assert report rendering, export shape, filter/sort, and reviewer annotations;
- require human review notes when rubric coverage is incomplete.

Project signals:

- OpenClaw QA Lab has scenario catalog, self-check, multipass, suite summary JSON, character/discovery/model-switch evals, reports, live transport lanes, and web runtime.
- Agent Skills recommends eval runs with clean context, assertion grading, human feedback, execution transcripts, and iterative improvement; Agent QC adopts the same evidence-first loop for project QC.

Common anti-patterns:

- unversioned rubric;
- no baseline delta;
- LLM judge output without evidence refs;
- polished dashboard that hides failing cases or waivers.

## Minimum surface suite by project shape

| Project shape | Must-have surface gates | Often-needed companion gates |
| --- | --- | --- |
| Runtime CLI like Codex | `cli-stream`, `tui` if interactive | `contract-protocol`, `runtime-e2e`, sandbox, SDK/API |
| TUI runtime like Claude Code snapshot | `tui`, `cli-stream` for stream adapters | remote permission contract, reconnect/cancel, SDK stream |
| Multi-channel gateway like OpenClaw | `channel-ui`, `webui`, `cli-stream` | channel contracts, live provider, Docker/install, secret redaction |
| Background agent like Hermes | `cli-stream`, `tui`, `browser-automation`, channel UI if gateway-enabled | scheduler stress, checkpoint/recovery, browser safety, Docker smoke |
| Desktop GUI with native bridge | `desktop-gui`, `webui` for WebView panels, `browser-automation` if browser runtime exists | bridge contract, native command registration, workspace readiness, Playwright trace |
| Eval suite / QA lab | `eval-ui`, `webui` if dashboard exists | semantic eval, reviewer loop, baseline diff, export schema |

## Waiver rules

A surface gate may be waived only when the report states:

- exact missing capability, such as "no Windows runner" or "no real Telegram credential";
- risk accepted by the waiver;
- expiration or recheck trigger;
- substitute evidence, if any;
- why the release/change can still proceed.

A waiver cannot turn a screenshot-only test into a runtime-backed surface pass. It only documents accepted residual risk.

## Agent UI mapping checklist

Agent UI contributes a reusable projection checklist. For every Agent QC `ui-interaction` case, ask which Agent UI surface the project is proving and which runtime fact backs it.

| Agent UI surface | QC assertion |
| --- | --- |
| Composer | submit/queue/steer/interrupt controls call the owning runtime API and show pending/failure state. |
| Message parts | final answer text is separate from reasoning, tool progress, diagnostics, artifact refs, and evidence refs. |
| Runtime status | first status, blocked, retrying, failed, cancelled, and done states come from runtime events. |
| Tool UI | tool start, safe args summary, progress, output ref, and error all preserve tool call id. |
| Human-in-the-loop | approval/input request includes id, scope, consequence, response, and runtime confirmation. |
| Task capsule | queued/background/subagent/team work has stable task/agent ids and visible ownership. |
| Artifact workspace | artifact preview/edit/export uses artifact facts, not copied assistant prose. |
| Timeline/evidence | trace, replay, verification, review, and audit refs are durable and inspectable. |
| Session/tabs | old-session hydration shows shell and recent state without guessing missing details. |
| Team workbench | coordinator, worker, remote, background, handoff, and review states are not flattened into one assistant. |

This checklist is not a visual design requirement. It is a fact-ownership requirement.


# qcloop integration

Source: https://limecloud.github.io/agentqc/en/authoring/qcloop-integration

# qcloop integration

qcloop is a batch execution loop for repeated independent QC cases. Agent QC defines how to prepare items and how verifier output maps to evidence-backed verdicts.

## When to use qcloop

Use qcloop when:

- many cases share one worker/verifier template;
- each item can be judged independently;
- missed items are a real risk;
- repair/retry should be bounded;
- attempts, verifier rounds, and feedback need to be auditable.

Do not use qcloop for one open-ended investigation, for hidden shared-state tests, or to bypass a required CI/project gate.

## Generic item shape

```json
{
  "id": "tool-permission-deny-write-outside-workspace",
  "project_profile": "agent-runtime-cli",
  "target": "sandbox permission boundary",
  "steps": ["Run the deny-write fixture", "Capture tool result and exit status"],
  "expected": ["Write is blocked", "Error is surfaced without leaking secrets"],
  "required_evidence": ["command_log", "tool_transcript"],
  "risk": "permission bypass"
}
```

## Verifier output

Verifier output SHOULD be strict JSON:

```json
{
  "pass": false,
  "status": "failed",
  "severity": "high",
  "feedback": "The transcript shows the write succeeded outside the workspace.",
  "evidence_refs": ["qcloop://jobs/job-123/items/deny-write/attempts/1"],
  "remaining_risk": "Sandbox policy may not be enforced for this path."
}
```

## Mapping qcloop states

| qcloop state | Agent QC meaning |
| --- | --- |
| `success` | Item has a passing verifier verdict. |
| `failed` | Worker or repair execution failed. |
| `exhausted` | Budget or `max_qc_rounds` reached without proof. |
| `pending` / `running` | Plan is incomplete. |


# Evidence-driven verdicts

Source: https://limecloud.github.io/agentqc/en/authoring/evidence-driven-verdicts

# Evidence-driven verdicts

A verdict is a claim about observed evidence. The model's final prose is not enough.

Use this page for authoring guidance and the [Evidence contract](../contracts/evidence-contract) for portable fields.

## Verdict statuses

| Status | Meaning | Required proof |
| --- | --- | --- |
| `passed` | Evidence proves all required expectations. | evidence refs tied to case and gate |
| `failed` | Evidence disproves an expectation or a gate exits non-zero. | smallest actionable failure and evidence |
| `blocked` | Environment, credentials, dependency, fixture, binary, or access prevents judgment. | blocker, owner, and retry condition |
| `exhausted` | Attempts or budget were consumed without proof. | attempts, budget, verifier feedback, remaining uncertainty |
| `waived` | A responsible actor accepted a known gap. | approver, reason, scope, expiry |
| `needs-review` | Evidence exists but semantic, safety, UX, or policy review remains. | reviewer or queue and evidence refs |
| `skipped` | Gate is intentionally not applicable to the current scope. | reason and scope |

## Verdict strength ladder

| Strength | Example | When to use |
| --- | --- | --- |
| Weak observation | screenshot only | visual smoke, not runtime-backed pass |
| Deterministic proof | unit/contract/fake server report | local correctness without live risk |
| Runtime proof | CLI/session/tool transcript | agent loop and side effects are involved |
| Surface proof | trace/screenshot/terminal snapshot with runtime link | user/operator sees the behavior |
| Live proof | redacted provider/channel transcript | real network/model/channel is part of the claim |
| Release proof | clean install/package/Docker/OS matrix | artifact is shipped |
| Semantic proof | rubric, baseline, judge, reviewer | output quality is the claim |

Strong reports combine the levels required by the risk. They do not always need every level.

## Good evidence

Good evidence is inspectable and scoped:

- command log or CI job URL;
- JUnit/JSON/HTML/coverage report;
- protocol transcript or mock server request log;
- runtime transcript, event stream, or session state snapshot;
- Playwright trace, screenshot, video, DOM/a11y snapshot, or terminal snapshot;
- browser console/network log;
- qcloop attempt and QC round refs;
- package manifest, tarball listing, Docker smoke output, OS matrix;
- model output plus rubric and judge verdict;
- human review note with reviewer and scope.

## Bad evidence

Bad evidence is unverifiable or overclaims:

- "looks good";
- "the tests passed" without command, CI ref, or report path;
- hidden local state with no path or transcript;
- screenshot without runtime backing for a runtime claim;
- live provider claim with no redacted request/response or budget note;
- TUI snapshot with no viewport/key sequence;
- browser screenshot with no console/network or cleanup note;
- qcloop summary without attempts and verifier feedback;
- waiver without owner or expiry.

## Status selection guide

| Situation | Status |
| --- | --- |
| Required evidence exists and expectations are proven | `passed` |
| Command exits non-zero and failure matches changed risk | `failed` |
| Test cannot start because credential/binary/fixture is absent | `blocked` |
| Repeated qcloop attempts cannot produce proof within budget | `exhausted` |
| The product owner accepts missing Windows smoke until a date/version | `waived` |
| Eval output exists but rubric is ambiguous or safety review remains | `needs-review` |
| Mobile channel not touched by current change and not in scope | `skipped` |

## Waiver rules

A waiver must include:

| Field | Meaning |
| --- | --- |
| `approver` | accountable person/team/policy owner |
| `reason` | why risk is accepted for this scope |
| `scope` | exact case, gate, platform, provider, or release range |
| `expires` | date, version, or event requiring recheck |
| `replacement_evidence` | weaker proof still available, if any |
| `follow_up` | issue, task, or next QC case |

A waiver never converts missing evidence into a pass. It records accepted residual risk.

## Failure writing

A useful `failed` verdict answers:

1. What expectation was disproven?
2. What is the smallest command, case, selector, event id, or fixture that reproduces it?
3. Which evidence proves the failure?
4. Which claims are still valid despite the failure?
5. What should be fixed or rerun next?

Avoid broad failures like "GUI broken". Prefer "desktop-gui case `bridge-health-workspace-ready` failed: bridge health timed out after 120s; screenshot shows fallback mock banner; command contract check passed."

## Blocked vs exhausted

Use `blocked` when the run cannot meaningfully start or judge because a prerequisite is absent.

Use `exhausted` when the system tried within declared attempts/budget but still cannot prove the claim.

Examples:

| Case | Status |
| --- | --- |
| No Telegram token available for live channel test | `blocked` |
| qcloop ran 5 attempts and verifier still cannot find required evidence | `exhausted` |
| Playwright browser binary missing | `blocked` |
| flaky browser test retried according to policy and still no stable trace | `exhausted` |

## Review before pass

Use `needs-review` for:

- semantic evals where rubric coverage is incomplete;
- safety or policy-sensitive output;
- UX judgment from screenshots or recordings;
- generated content quality;
- suspicious live-provider drift;
- evidence that conflicts across gates.

A reviewer may change `needs-review` to `passed`, `failed`, or `waived`, but must cite evidence.

## Report checklist

Before finalizing a report:

- every required gate has a status;
- every `passed` and `failed` status cites evidence;
- every surface claim links visible state to runtime/protocol evidence;
- every live-provider claim has redaction and budget notes;
- every waiver has owner and expiry;
- every blocked/exhausted item has next action;
- remaining risk is written in plain language.


# Acceptance scenarios

Source: https://limecloud.github.io/agentqc/en/authoring/acceptance-scenarios

# Acceptance scenarios

Agent QC validates behavior and evidence, not repository shape alone. Use these scenarios for manual QA, automated tests, qcloop batches, CI gates, or release review.

A scenario passes only when evidence proves the behavior. A scenario with missing evidence is `blocked`, `exhausted`, `waived`, or `needs-review`, not passed.

## 1. Runtime CLI permission boundary

1. User or test triggers an unsafe tool/command action.
2. Runtime emits a permission or policy decision with stable id.
3. The action is denied or requires approval.
4. No unauthorized side effect occurs.
5. CLI/TUI/WebUI shows a controlled error or pending approval.

Pass condition: denied action is visible, correlated, and side-effect-free.

Evidence: command transcript, policy event, side-effect check, surface artifact when visible.

## 2. Tool or MCP transport recovery

1. A stdio/http/WebSocket tool server disconnects or returns an error.
2. Runtime surfaces failure and recovery or terminal failure.
3. Tool state does not corrupt the next call.
4. UI/TUI shows failure outside final answer text.

Pass condition: recovery and failure are inspectable and do not invent success.

Evidence: protocol transcript, retry log, tool id correlation, surface frame.

## 3. SDK/API contract drift

1. Public SDK or generated client changes shape.
2. Schema/generation check runs.
3. Fake server or fixture verifies the new contract.
4. Old incompatible behavior is either migrated or explicitly versioned.

Pass condition: contract drift is reviewed before runtime or UI claims.

Evidence: schema diff, generated artifact check, fake server transcript.

## 4. CLI stream final reconciliation

1. Runtime streams partial text/tool events.
2. Runtime emits final message or terminal status.
3. CLI output or consumer reconciles final content without duplication.
4. Exit code matches terminal status.

Pass condition: no duplicate final text, hidden tool failure, or wrong exit status.

Evidence: stdout/stderr transcript, structured event sample, exit code.

## 5. TUI first status and interrupt

1. User submits a prompt.
2. Listener binds before submit or before the first runtime event.
3. Runtime status appears before first answer text when accepted.
4. Interrupt/cancel is available when supported.
5. Interrupt stops the run without orphan subprocesses.

Pass condition: the user can tell the agent is alive and can stop it safely.

Evidence: pseudo-terminal transcript, terminal snapshot, runtime transcript, cleanup proof.

## 6. TUI tool and permission overlay

1. Runtime emits tool start with stable tool id.
2. TUI shows safe input summary and progress.
3. Runtime emits action request for a high-risk operation.
4. User approves, rejects, edits, or answers.
5. TUI marks resolved only after runtime confirmation.

Pass condition: tool progress and approval state are visible, correlated, and auditable.

Evidence: terminal snapshot, key sequence, action request/response transcript.

## 7. WebUI reload and stale state

1. User opens a running or recently completed session.
2. WebUI renders route shell and current status.
3. Page reload or route revisit does not fabricate success.
4. Missing facts render as `unknown`, `unavailable`, `stale`, or `blocked`.

Pass condition: reload/resume preserves runtime truth and safe fallback states.

Evidence: browser trace, screenshot, console/network log, runtime state ref.

## 8. Desktop GUI bridge readiness

1. App shell starts or is reused through the supported entrypoint.
2. Bridge health is checked before judging the page.
3. Default workspace/session readiness is proven.
4. A user-visible flow runs with screenshot/trace.
5. Native command contracts are synchronized when touched.

Pass condition: desktop readiness is proven beyond component tests.

Evidence: shell log, bridge health, workspace readiness, screenshot/trace, OS note.

## 9. Browser automation safety and cleanup

1. Agent opens or controls a browser session.
2. Test records URL, viewport, provider, and session scope.
3. DOM/a11y and screenshot evidence prove the observed state.
4. Console/network logs are inspected.
5. Browser/tabs/processes are closed or intentionally reused.

Pass condition: observation, safety, and cleanup are all proven.

Evidence: screenshot, DOM/a11y, console/network, cleanup/orphan proof.

## 10. Channel gateway auth and media

1. Channel adapter receives a webhook/message with auth context and media.
2. Gateway verifies identity before parsing user content.
3. Media is stored or rejected by policy.
4. Response transcript is redacted and traceable.
5. Live channel path is opt-in if used.

Pass condition: identity, media, and response behavior are proven without leaking secrets.

Evidence: webhook replay, media fixture, redacted transcript, auth decision.

## 11. Queue and steer distinction

1. A run is active.
2. User sends another prompt or control action.
3. System distinguishes queue-next from steer-current.
4. Runtime emits stable queued/steer ids.
5. Surface shows pending state and final resolution.

Pass condition: users can distinguish "run later" from "change current run".

Evidence: runtime events, UI/TUI snapshot, queue state transcript.

## 12. Artifact handoff and evidence export

1. Runtime creates or updates an artifact.
2. UI/CLI links compact artifact reference.
3. Artifact details open through artifact service or durable path.
4. Evidence export creates durable refs.
5. Report links artifact/evidence ids to the producing case.

Pass condition: deliverables and evidence leave the chat body and become traceable artifacts.

Evidence: artifact path/id, export log, screenshot/report link.

## 13. Old-session recovery

1. User opens old session/task/thread.
2. Shell or summary appears without full history blocking first paint.
3. Recent messages/status hydrate before heavy details.
4. Tool output, artifacts, and evidence load on demand.
5. Stale or missing facts remain explicit.

Pass condition: old sessions are usable and do not guess missing truth.

Evidence: timing metrics, screenshot, hydration log, cursor/page refs.

## 14. Background scheduler restart

1. Scheduled/background task starts and writes checkpoint or lease.
2. Owner is interrupted or process restarts.
3. New owner reclaims or resumes according to policy.
4. Duplicate and lost work are prevented.
5. Final state includes cleanup and ownership evidence.

Pass condition: restart does not duplicate, lose, or hide work.

Evidence: deterministic clock/env, checkpoint, lease timeline, worker logs.

## 15. Parallel worker fanout/fanin

1. Coordinator starts multiple independent workers/subagents/tasks.
2. Each worker has stable id, role, parent, and status.
3. Partial success, failure, retry, and wait states remain visible.
4. Final synthesis links worker results without rewriting authorship.

Pass condition: parallel work is visible, resumable, and auditable.

Evidence: delegation graph, worker transcripts, final evidence refs.

## 16. Remote agent or teammate handoff

1. Runtime connects to remote agent or hands work to another teammate.
2. UI/TUI shows remote task id, owner, reason, auth/input needs, and status.
3. Input/auth required states are promoted to user controls.
4. Idle/transient state is not treated as completion.

Pass condition: remote ownership and completion truth are preserved.

Evidence: remote protocol transcript, task card snapshot, handoff log.

## 17. Eval regression and report UI

1. Prompt/eval suite runs against current behavior and baseline.
2. Rubric and judge/model settings are recorded.
3. Report shows pass/fail examples and baseline delta.
4. Reviewer can inspect raw outputs and waivers.

Pass condition: semantic quality claim is backed by comparable evidence.

Evidence: dataset/rubric, judge output, baseline delta, report screenshot/export.

## 18. Distribution install smoke

1. Release package/image is built.
2. Clean environment installs or starts it.
3. Version/help/minimal runtime command works.
4. Package contents match manifest.
5. Platform-specific limitations are recorded.

Pass condition: shipped artifact is usable outside the source tree.

Evidence: package manifest, install log, Docker/OS matrix, version output.

## 19. Live provider opt-in

1. Case declares live provider/channel/model requirement.
2. Credentials are scoped and redacted.
3. Budget/timeout is recorded.
4. Request/response or provider transcript is stored safely.
5. Failure is not retried into invisibility.

Pass condition: live behavior is proven without contaminating deterministic lanes.

Evidence: opt-in flag, redacted transcript, budget note, provider id.

## 20. qcloop repeated QC

1. Plan creates independent qcloop items.
2. Each item includes profile, surface, gates, expected result, and evidence policy.
3. Attempts and verifier rounds are preserved.
4. Exhausted items remain `exhausted`, not generic failed.
5. Aggregate report states remaining risk.

Pass condition: repetition improves coverage without hiding required project gates.

Evidence: qcloop job id, item values, attempts, verifier feedback, verdict refs.

## 21. Waiver and blocked path

1. Required gate cannot run or is intentionally deferred.
2. Report records missing fact, owner, scope, and risk.
3. Waiver includes approver, reason, expiry, and follow-up.
4. Release or next action does not call the waived gate passed.

Pass condition: incomplete proof is visible and accountable.

Evidence: waiver object, blocker note, replacement evidence, follow-up link.

## 22. Benchmark hill climbing

1. A frozen dataset and task set are selected from real Lime failures or high-risk flows.
2. Baseline and candidate configs differ by exactly one variable.
3. Each trial stores trajectory, runtime transcript, reward details, artifacts, timeout/cost metrics, and cleanup evidence.
4. Aggregate comparison reports reward delta, timeout rate, evidence completeness, and P0 QC regression count.
5. Candidate promotion is blocked if required QC gates regress, even when reward improves.

Pass condition: the improvement claim is reproducible, attributable, and still compatible with required Agent QC gates.

Evidence: dataset/task version, baseline/candidate configs, trial trajectories, reward details, comparison summary, Agent QC report refs.

## Scenario selection guide

| Project shape | Must include |
| --- | --- |
| Codex-like runtime CLI | scenarios 1, 2, 4, 5, 18 |
| Claude Code-like TUI runtime | scenarios 5, 6, 11, 16, 21 |
| OpenClaw-like channel/WebUI gateway | scenarios 7, 9, 10, 17, 18, 19 |
| Hermes-like background/browser agent | scenarios 9, 14, 15, 18, 19 |
| Desktop GUI / native bridge | scenarios 7, 8, 9, 12, 21 |
| Eval/QA lab | scenarios 17, 20, 21, 22 |
| Lime internal benchmark | scenarios 1, 5, 7, 9, 17, 22 |


# Evidence contract

Source: https://limecloud.github.io/agentqc/en/contracts/evidence-contract

# Evidence contract

A verdict is only as strong as the evidence it references. This contract defines the minimum portable fields for evidence-backed Agent QC reports.

## Evidence reference

| Field | Required | Description |
| --- | --- | --- |
| `id` | Yes | Stable evidence id inside the report. |
| `kind` | Yes | Evidence kind such as `command-log`, `test-report`, `protocol-transcript`, `surface-artifact`, `release-artifact`, `eval-artifact`, `benchmark-artifact`, `trajectory`, `reward-details`, `review-note`, or `qcloop-run`. |
| `source` | Yes | Local path, artifact URL, CI URL, qcloop id, or evidence service id. |
| `scope` | Yes | Case id, gate id, command, surface, profile, or release target covered. |
| `created_at` | Recommended | Timestamp or run id. |
| `environment` | Recommended | OS, runtime, browser, terminal size, provider mode, CI job, or Docker image. |
| `redaction` | Conditional | Required when credentials, user data, provider requests, or channel transcripts are involved. |
| `summary` | Recommended | Short human-readable result. |
| `raw_ref` | Optional | Safe raw payload ref. Do not inline secret-bearing payloads. |

## Verdict object

| Field | Required | Description |
| --- | --- | --- |
| `status` | Yes | `passed`, `failed`, `blocked`, `exhausted`, `waived`, `needs-review`, or `skipped`. |
| `case_id` | Yes | Case being judged. |
| `gate_family` | Yes | Gate family being judged. |
| `evidence_refs` | Yes except `skipped` | Evidence ids supporting the claim. |
| `expectations_met` | Recommended | Explicit expectation ids or text snippets proven by evidence. |
| `failure` | Required for `failed` | Smallest actionable failure, not a broad complaint. |
| `blocker` | Required for `blocked` | Missing environment fact and owner. |
| `attempts` | Required for `exhausted` | Attempt refs, budget, and remaining uncertainty. |
| `waiver` | Required for `waived` | Approver, reason, scope, expiry. |
| `review` | Required for `needs-review` | Reviewer, queue, or reason semantic review remains. |

## Evidence minimum by gate

| Gate | Minimum evidence |
| --- | --- |
| `static` | command/CI log, tool version, failing ids or success summary |
| `unit` | test report or command log with suite and failure ids |
| `property-fuzz` | seed/corpus, invariant, failing minimized case if any |
| `contract-protocol` | schema diff, generated artifact check, fake server or protocol transcript |
| `fake-integration` | fake server log and request/response refs |
| `runtime-e2e` | runtime transcript, state snapshot, process cleanup or retry proof |
| `ui-interaction` | surface artifact plus runtime/protocol link |
| `live-provider` | opt-in flag, redacted request/response, credential scope, cost/budget note |
| `stress-concurrency` | worker timeline, seed/config, duration, race/retry result |
| `distribution-release` | package manifest, clean install, Docker/OS matrix, version output |
| `semantic-eval` | dataset/rubric, model/judge info, baseline delta, threshold |
| `benchmark-eval` | dataset/task version, baseline and candidate configs, trial trajectory, reward details, aggregate delta, promotion/revert decision |
| `review` | reviewer identity, scope, evidence refs, decision |

## Surface evidence add-ons

| Surface | Add-on evidence |
| --- | --- |
| `cli-stream` | stdout/stderr transcript, exit code, structured event sample |
| `tui` | terminal size, key sequence, terminal snapshot, linked runtime transcript |
| `webui` | Playwright or browser trace/screenshot, console output, route/state assertion |
| `desktop-gui` | shell start log, bridge health, workspace readiness, screenshot, OS note |
| `browser-automation` | DOM/a11y snapshot, console/network, screenshot, cleanup/orphan-process proof |
| `channel-ui` | webhook replay, channel transcript, media fixture, identity/auth proof |
| `eval-ui` | report screenshot/export, rubric, judge output, reviewer note |

## Waiver contract

A waiver is not a pass. It is a time-bounded risk decision.

| Field | Required | Description |
| --- | --- | --- |
| `approver` | Yes | Person, team, or policy owner accepting the risk. |
| `reason` | Yes | Why this gate is not required for this scope. |
| `scope` | Yes | Case, gate, platform, provider, or release range. |
| `expires` | Yes | Date, version, or condition that invalidates the waiver. |
| `replacement_evidence` | Recommended | Lower-strength evidence that still exists. |
| `follow_up` | Recommended | Issue, task, or next QC case. |

## Anti-patterns

| Anti-pattern | Correct status |
| --- | --- |
| "Looks good" with no artifact | `needs-review` or `blocked` |
| Screenshot without command/runtime evidence | partial `ui-interaction`, not full pass |
| Live provider output with no redaction/budget note | `needs-review` |
| Unit tests only for desktop bridge behavior | `blocked` for GUI/surface claim |
| qcloop exhausted but reported as failed without attempts | `exhausted` |
| Waiver without owner or expiry | invalid waiver |

## Report closeout checklist

A report is ready to publish when:

- every required gate has a verdict;
- every `passed` or `failed` verdict has evidence refs;
- every `blocked`, `exhausted`, `waived`, or `needs-review` status explains why it is not a pass;
- live-provider evidence is redacted and budgeted;
- surface evidence links visible behavior to runtime or protocol facts;
- benchmark evidence includes frozen dataset, trial trajectories, reward details, and baseline/candidate configs when improvement is claimed;
- remaining risk and next action are explicit.


# Performance and reliability metrics

Source: https://limecloud.github.io/agentqc/en/contracts/performance-and-reliability-metrics

# Performance and reliability metrics

Agent projects often fail by appearing alive while queues, streams, tools, browsers, or background workers are stuck. QC evidence should capture enough timing and reliability data to explain perceived slowness and flaky behavior.

Agent QC does not mandate universal thresholds. Each project should define thresholds by product profile and risk.

## Runtime responsiveness

| Metric | Meaning | Applies to |
| --- | --- | --- |
| `submit_to_accept_ms` | user action to runtime acceptance | CLI, TUI, GUI, WebUI |
| `first_status_ms` | first user-visible runtime status | Agent UI/TUI/desktop |
| `first_text_delta_ms` | first model/user-facing text delta | streams and chat UIs |
| `first_tool_event_ms` | first tool start/progress event | tool/runtime gates |
| `interrupt_ack_ms` | interrupt/cancel request to runtime acknowledgement | CLI/TUI/GUI |
| `resume_ready_ms` | old session or task resume to usable state | sessions, schedulers |

These metrics are inspired by Agent UI's separation of listener binding, runtime acceptance, first status, first text, and paint timing.

## Stream and projection health

| Metric | Meaning | Evidence |
| --- | --- | --- |
| `event_sequence_gap_count` | missing or out-of-order runtime events | protocol transcript |
| `delta_backlog_depth` | queued unrendered text/tool deltas | UI diagnostics |
| `oldest_unrendered_delta_ms` | oldest pending delta age | UI diagnostics |
| `final_reconciliation_duplicates` | duplicated streamed/final text count | transcript + surface artifact |
| `stale_success_count` | UI claimed success before runtime confirmation | runtime/UI comparison |
| `missing_fact_fallback_count` | `unknown`, `unavailable`, `stale`, or `blocked` projections | UI snapshot/report |

## Tool and permission reliability

| Metric | Meaning | Evidence |
| --- | --- | --- |
| `tool_start_to_result_ms` | tool duration by tool id | tool transcript |
| `tool_error_recovery_count` | retry/recovery attempts after tool failure | runtime transcript |
| `approval_pending_ms` | time in human-in-the-loop state | action transcript |
| `approval_correlation_failures` | request/response id mismatches | protocol test |
| `denied_side_effect_count` | denied action still caused side effect | sandbox/process evidence |
| `orphan_process_count` | subprocess/browser workers left behind | cleanup evidence |

## Browser, WebUI, TUI, and desktop reliability

| Surface | Metrics |
| --- | --- |
| `webui` | page load, first status paint, console error count, failed network count, trace size |
| `desktop-gui` | shell start, bridge health time, workspace readiness, native command timeout, mock fallback count |
| `tui` | first frame, redraw latency, viewport reflow failures, key handling failures, Unicode/ANSI rendering failures |
| `browser-automation` | navigation time, DOM ready, console/network errors, screenshot/trace success, cleanup/orphan count |
| `channel-ui` | webhook verification time, dedup count, media processing time, retry count, delivery ack time |

Playwright-style projects should retain trace/screenshot/video on failure and record browser project/device when relevant. Vitest browser-mode or component tests can prove component behavior, but browser-only APIs need browser evidence.

## Scheduler and background reliability

| Metric | Meaning |
| --- | --- |
| `lease_reclaim_ms` | time to reclaim work after interrupted owner |
| `checkpoint_age_ms` | age of last durable checkpoint |
| `duplicate_job_count` | duplicate execution for same job id |
| `lost_job_count` | scheduled jobs not executed by deadline |
| `retry_attempt_count` | attempts per task before success/failure/exhaustion |
| `worker_shutdown_ms` | graceful worker termination time |
| `queue_depth` | pending work by queue or priority |

Hermes-style projects should pin deterministic clock/env for normal tests and reserve live provider/channel checks for explicit opt-in lanes.

## Release and distribution reliability

| Metric | Meaning |
| --- | --- |
| `clean_install_ms` | fresh install duration |
| `package_size_bytes` | package or image size |
| `manifest_missing_count` | expected files absent from package |
| `version_mismatch_count` | package/app/Cargo/Tauri/version drift |
| `docker_smoke_ms` | Docker smoke duration |
| `platform_failure_count` | OS matrix failures |
| `lock_drift_count` | lockfile or generated artifact drift |

Codex-style projects may use Bazel/nextest/release binaries. OpenClaw-style projects may use Docker/install smoke and plugin release checks. Agent QC only requires the evidence shape.


## Benchmark and hill-climbing metrics

| Metric | Meaning | Evidence |
| --- | --- | --- |
| `mean_reward` | Average verifier reward across tasks or trials | reward.json aggregate |
| `pass_rate` | Fraction of trials with passing reward/status | trial table |
| `pass_at_k` | Whether at least one of k attempts passes | repeated trial set |
| `mean_reward_delta` | Candidate minus baseline reward | baseline/candidate summary |
| `timeout_rate` | Fraction of trials ending in timeout | trial status and duration |
| `verifier_error_rate` | Verifier failures independent of agent behavior | verifier logs |
| `evidence_completeness_rate` | Trials with required trajectory/reward/artifact refs | evidence report |
| `cost_per_pass` | Model/provider/runtime cost divided by passes | cost and reward summary |
| `tokens_per_trial` | Input/output/cache tokens per trial | runtime/provider telemetry |
| `p0_qc_gate_regression_count` | Required QC gates that regressed while benchmark improved | Agent QC report |

A candidate can improve product quality only when benchmark metrics and required QC gates agree. A higher `mean_reward` with lower evidence completeness is not a clean win.

## Suggested threshold policy

A QC plan SHOULD define:

| Threshold | Example |
| --- | --- |
| Local default | deterministic gates must pass with no live credentials |
| Surface smoke | first status or bridge health must appear within product-specific timeout |
| Flake budget | retry count and rerun policy for known flaky lanes |
| Live budget | provider/channel cost, credential scope, and timeout |
| Release budget | install time, package size, OS matrix, Docker smoke timeout |
| Waiver expiry | date/version when missing metric must be rechecked |

## Evidence guidance

When a performance or reliability gate fails, preserve:

- the command or interaction that started the run;
- timestamps and environment;
- trace/screenshot/transcript around the slow or flaky segment;
- retry and cleanup outcome;
- whether the failure blocks release, needs review, or can be waived.


# Agent project patterns

Source: https://limecloud.github.io/agentqc/en/reference/agent-project-patterns

# Agent project patterns

These patterns come from local repository inspection and public docs. They are examples, not normative requirements.

For the expanded system-by-system walkthrough, see [Star project testing systems](./star-project-testing-systems).

## Codex-style runtime CLI

Local source: `/Users/coso/Documents/dev/rust/codex`.

Observed testing shape:

- Rust workspace uses `cargo nextest run --no-fail-fast` for routine local runs.
- CI includes `cargo fmt`, `cargo clippy`, `cargo test`, `cargo nextest`, `cargo-deny`, `cargo shear`, Bazel test matrices, and SDK jobs.
- Tests cover sandboxing, apply-patch behavior, MCP server/client behavior, protocol events, CLI execution, app server, streamable HTTP recovery, and SDK public APIs.
- Fixtures include fake model servers, SSE fixtures, test stdio/http servers, expected patch outputs, terminal snapshots, app-server protocol fixtures, and SDK stream tests.

Surface coverage: `cli-stream`, `tui`, protocol/runtime transcripts, and release artifacts.

Agent QC profile mapping: `agent-runtime-cli`, `agent-tool-mcp-gateway`, `agent-sdk-api`, `agent-ui-tui-desktop`, `agent-distribution-release`.

## OpenClaw-style multi-channel gateway

Local source: `/Users/coso/Documents/dev/js/openclaw` and public OpenClaw docs.

Observed testing shape:

- Many Vitest lanes: unit, gateway, channels, contracts, e2e, live, Docker, install smoke, performance, startup, platform-specific lanes.
- CI preflight computes changed scopes and routes jobs instead of running one fixed command.
- Tests emphasize channel contracts, secrets, provider surfaces, media handling, plugin boundaries, Docker/install smoke, and live opt-in providers.
- Release workflows include npm, Docker, plugin, install-smoke, and platform-specific checks.

Surface coverage: `channel-ui`, `webui`, `browser-automation`, `eval-ui`, and release smoke.

Agent QC profile mapping: `multi-channel-agent-gateway`, `agent-tool-mcp-gateway`, `agent-skills-plugins`, `agent-ui-tui-desktop`, `agent-distribution-release`, `agent-evals-quality`.

## Claude Code-style TUI/runtime snapshot

Local source: `/Users/coso/Documents/dev/js/claudecode`.

Observed testing shape:

- The local snapshot is incomplete: no `package.json` or GitHub workflows were available.
- Source surfaces still show TUI/runtime risks: Ink rendering, shell interaction, remote session permission requests, WebSocket/SSE/HTTP transports, tool-use messages, SDK stream adapters, and injected query dependencies.
- Agent QC should treat this as interface-surface evidence only, not as a conclusion about the upstream project's test strategy.

Surface coverage: `tui`, `cli-stream`, remote permission protocol, SDK stream adapter, and skill/plugin reload visibility.

Agent QC profile mapping: `agent-runtime-cli`, `agent-ui-tui-desktop`, `agent-sdk-api`, `agent-tool-mcp-gateway`, `agent-skills-plugins`.

## Hermes-style background agent

Local source: `/Users/coso/Documents/dev/python/hermes-agent` and public Hermes docs.

Observed testing shape:

- Python uses pytest with markers, xdist, ignored integration/e2e paths, and a canonical `scripts/run_tests.sh` that normalizes env, workers, timezone, locale, hash seed, and credentials.
- Tests include cron, gateway, plugins, memory providers, stress/concurrency, checkpointing, evidence store, subprocess e2e, and TUI Vitest tests.
- CI includes pytest, separate e2e, ruff/ty lint, uv lock checks, OSV scanner, Docker build and smoke, docs-site checks, and skills index jobs. The repo also has `ui-tui` Vitest tests and a Vite/React `web` dashboard package.

Surface coverage: `cli-stream`, `tui`, `browser-automation`, `channel-ui`, and scheduler evidence.

Agent QC profile mapping: `agent-runtime-cli`, `background-agent-scheduler`, `multi-channel-agent-gateway`, `agent-ui-tui-desktop`, `agent-tool-mcp-gateway`, `agent-distribution-release`, `agent-skills-plugins`.

## Cross-case lessons

1. Use project profiles before gates.
2. Keep fake/local integration separate from live providers.
3. Treat install/package/Docker smoke as first-class release gates.
4. Preserve protocol transcripts and UI traces as evidence.
5. Use qcloop for repeated cases, not as a substitute for framework-native gates.
6. Normalize environment for agent tests: credentials, time, locale, working directory, concurrency, and sandbox.


# Flow and taxonomy

Source: https://limecloud.github.io/agentqc/en/reference/flow-and-taxonomy

# Flow and taxonomy

This page is the complete Agent QC lifecycle and taxonomy reference. It mirrors the specification style used by Agent UI: explicit dimensions, fields, constraints, lifecycle stages, and validation cases.

## Core contract

Agent QC is an evidence protocol for Agent project quality. A compatible QC plan classifies owned risk, selects gates, executes checks, stores evidence, and emits verdicts without turning model prose into proof.

Compatible QC reports MUST:

- classify one or more project profiles;
- name touched interaction surfaces when user-visible behavior is involved;
- map each required gate to concrete local commands, CI jobs, qcloop items, or review steps;
- preserve inspectable evidence refs for every pass/fail/blocked/exhausted/waived verdict;
- separate deterministic, runtime, surface, live-provider, release, semantic-eval, and benchmark-improvement claims;
- state limitations and waivers explicitly.

Compatible QC reports MUST NOT:

- treat a final assistant answer as evidence without a linked artifact;
- infer runtime success from UI text alone;
- hide live-provider calls inside default deterministic tests;
- collapse screenshots, traces, terminal snapshots, and protocol transcripts into one vague "UI checked" claim;
- call a gate passed when required evidence is missing.

## Lifecycle overview

```text
change or release scope
  -> classify profiles
  -> identify touched surfaces
  -> assign fact owners and risk owners
  -> select gate lanes
  -> write behavior-level cases
  -> execute deterministic gates
  -> execute runtime and surface gates
  -> opt into live/release/eval gates when required
  -> collect evidence refs
  -> issue verdicts
  -> publish report, waivers, and next action
```

The flow applies to CLI agents, SDKs, MCP/tool gateways, channel bots, TUI/GUI/WebUI products, browser automation systems, schedulers, skills/plugins, distribution packages, and eval suites.

## Taxonomy dimensions

### Project profile

Profiles describe owned project shape.

| Profile | Owns | Default risks |
| --- | --- | --- |
| `agent-runtime-cli` | agent loop, CLI, task execution, sandbox, tools, resume | stream drift, permissions, subprocess cleanup, resume consistency |
| `agent-sdk-api` | public SDK, generated client, API wrappers | signature drift, async cancellation, fake-server behavior |
| `agent-tool-mcp-gateway` | tool declarations, MCP/ACP bridge, connector runtime | protocol conformance, stdio/http recovery, resource permission |
| `multi-channel-agent-gateway` | chat/channel adapters, webhooks, auth, media | identity, webhook verification, media routing, secret redaction |
| `agent-ui-tui-desktop` | GUI, TUI, desktop shell, browser-visible flows | projection drift, stale success, bridge readiness, screenshots/traces |
| `agent-skills-plugins` | skills, plugins, manifests, loaders, marketplace | manifest drift, package boundary, trust policy, fixture install |
| `background-agent-scheduler` | cron, queues, workers, retries, long-running agents | duplicate work, lost checkpoints, race, stuck loop |
| `agent-distribution-release` | package, Docker, installers, cross-platform release | missing files, broken clean install, lock drift, supply chain |
| `agent-evals-quality` | task quality, model behavior, rubrics, generated outputs | prompt drift, judge instability, baseline regression, grounding gap |

### Interaction surface

Surfaces describe where the behavior is observed.

| Surface | Use when | Required evidence |
| --- | --- | --- |
| `cli-stream` | stdout/stderr, JSONL/NDJSON, command UI | command, exit status, transcript, structured sample |
| `tui` | terminal UI, Ink, ratatui, curses | viewport, key sequence, terminal snapshot, runtime transcript |
| `webui` | browser dashboard, extension UI, QA/admin console | screenshot/trace, console log, route/state assertion |
| `desktop-gui` | Tauri, Electron, native shell | shell start, bridge health, workspace/session readiness, OS note |
| `browser-automation` | CDP, Playwright, browser-use, remote browser | DOM/a11y, screenshot, console/network, cleanup proof |
| `channel-ui` | chat app, QR, mobile, webhook-visible flows | channel transcript, media fixture, auth/webhook replay, redaction |
| `eval-ui` | QA dashboards and eval reports | rubric, judge output, baseline delta, reviewer note |

### Gate family

Gate families describe validation style, not framework names.

| Family | Default use | Escalate when |
| --- | --- | --- |
| `static` | format, lint, type, schema, dependency hygiene | generated files or policy boundaries change |
| `unit` | deterministic local behavior | algorithms, parsers, reducers, adapters change |
| `property-fuzz` | invariants and generated input | parser, sandbox, path, protocol, serializer risk is high |
| `contract-protocol` | schema/API/command/tool surfaces | any wire shape, manifest, command, or SDK shape changes |
| `fake-integration` | local fake server or adapter flow | external API behavior is simulated |
| `runtime-e2e` | real CLI/task/session without live provider risk | loop, tool, permission, resume, subprocess flow changes |
| `ui-interaction` | GUI/TUI/WebUI/browser/channel visible behavior | users or operators observe the changed behavior |
| `live-provider` | opt-in real network/model/channel path | provider/channel behavior is part of the claim |
| `stress-concurrency` | races, queue, leases, retries, long runs | scheduler, parallel agents, workers, or locks change |
| `distribution-release` | package/install/Docker/OS matrix | anything shipped outside source changes |
| `semantic-eval` | task quality, prompt, rubric, judge | model behavior or output quality is the product |
| `benchmark-eval` | frozen tasks, trials, rewards, trajectories | claiming a candidate runtime/prompt/tool/context profile is better |
| `review` | human/LLM review | safety, policy, UX, or semantic judgment is required |

### Evidence kind

| Kind | Examples | Must include |
| --- | --- | --- |
| `command-log` | shell output, CI step, cargo/npm/pytest/vitest output | command, exit status, environment note |
| `test-report` | JUnit, JSON, coverage, HTML report | suite id, failing ids, artifact path or URL |
| `protocol-transcript` | fake server, MCP/ACP, WebSocket, HTTP transcript | request/response refs, redaction note |
| `runtime-transcript` | CLI JSONL, TUI-linked events, session state | run/session ids, event order, cleanup |
| `surface-artifact` | screenshot, video, Playwright trace, terminal snapshot | viewport/device/OS, action sequence |
| `browser-diagnostic` | console, network, DOM/a11y snapshot | route, selector or accessibility assertion |
| `release-artifact` | package manifest, tarball list, Docker smoke | version, platform, install command |
| `eval-artifact` | rubric, judge output, baseline diff | dataset, model/judge, threshold |
| `benchmark-artifact` | reward.json, reward-details, comparison summary | dataset/task version, config ids, trial ids |
| `trajectory` | agent tool/action/event trace | task id, trial id, runtime/model config, redaction |
| `review-note` | human or LLM review | reviewer, scope, evidence refs, decision |
| `qcloop-run` | attempt and QC round refs | item value, attempt id, verifier feedback |

### Verdict status

| Status | Meaning | Required fields |
| --- | --- | --- |
| `passed` | evidence proves all required expectations | evidence refs and scope |
| `failed` | evidence disproves an expectation or a gate failed | smallest actionable failure and evidence |
| `blocked` | missing environment, credential, dependency, fixture, or binary prevents judgment | blocker and owner |
| `exhausted` | attempts or budget ended without proof | attempt refs and remaining uncertainty |
| `waived` | accountable owner accepted known gap | approver, reason, scope, expiry |
| `needs-review` | evidence exists but judgment still needs semantic/safety review | reviewer or review queue |
| `skipped` | intentionally not applicable for this scope | reason and scope |

## Fact owners

Agent QC should name who owns each fact instead of treating the report as the owner of everything.

| Owner | Owns | QC responsibility |
| --- | --- | --- |
| Runtime | task/session/tool/permission state | capture transcript and state refs |
| Protocol/SDK | schemas, generated clients, adapters | capture contract diff and fake transcript |
| UI projection | visible rendering and user controls | capture surface artifact and runtime linkage |
| Evidence service | durable traces, replay, reviews | link evidence ids and export jobs |
| Policy/security | approvals, waivers, credentials, retention | record risk decision and scope |
| Artifact/release | deliverables, package contents, versions | capture manifest and install proof |
| Scheduler | leases, checkpoints, retries, workers | capture timeline and duplicate-work proof |
| Eval system | rubrics, judge outputs, baselines | capture dataset, threshold, and deltas |
| Benchmark runner | frozen tasks, trials, trajectories, rewards | capture config snapshots, reward details, and comparison decisions |

## Standard case envelope

A portable `qc_case` should carry these fields even when the JSON schema allows extension.

| Field | Required | Purpose |
| --- | --- | --- |
| `id` | yes | stable case id |
| `project_profile` | yes | one profile from the taxonomy |
| `surface` | recommended for visible cases | observation surface |
| `target` | yes | file, command, package, flow, API, or release target |
| `risk_owner` | recommended | runtime, protocol, UI, scheduler, release, eval, policy |
| `required_gates` | yes | gate families to satisfy |
| `steps` | yes | reproducible commands or interactions |
| `expected` | yes | behavior-level expectations |
| `required_evidence` | yes | artifacts needed for verdict |
| `live_policy` | conditional | opt-in, credential scope, redaction, budget |
| `waiver_policy` | conditional | owner, reason, expiry rules |
| `verdict` | after run | status and evidence refs |

## Standard report envelope

A portable QC report should answer:

| Field | Question |
| --- | --- |
| Scope | What change, release, or regression sweep is being judged? |
| Profiles | Which project profiles apply? |
| Surfaces | Which user/operator surfaces were touched? |
| Required gates | Which gates were required and why? |
| Executed gates | Which commands, CI jobs, qcloop runs, or reviews ran? |
| Evidence refs | Where are logs, traces, screenshots, transcripts, reports, and reviews? |
| Verdicts | Which cases passed, failed, blocked, exhausted, waived, or need review? |
| Remaining risk | What still should not be claimed? |
| Next action | Fix, rerun, review, release, or waive? |

## Validation cases for the standard itself

A project can claim Agent QC compatibility only if these cases are representable:

1. Codex-like runtime permission denial with CLI transcript, protocol event, and TUI row.
2. Claude Code-like remote permission request with WebSocket/control transcript and TUI prompt.
3. OpenClaw-like channel webhook replay with media fixture and redacted credential policy.
4. Hermes-like scheduler restart with deterministic time, checkpoint, and duplicate-work proof.
5. Desktop GUI native-bridge change with bridge health, workspace readiness, screenshot, and command-contract proof.
6. Browser automation flow with DOM/a11y, screenshot, console/network, and cleanup evidence.
7. Release smoke with package manifest, clean install, and platform note.
8. Semantic eval regression with rubric, judge output, baseline delta, and reviewer note.


# Star project testing systems

Source: https://limecloud.github.io/agentqc/en/reference/star-project-testing-systems

# Star project testing systems

This reference explains how several strong Agent projects organize testing. Agent QC does not copy their commands as a universal recipe. It extracts reusable test architecture: how each project separates deterministic tests from live provider risk, how UI/TUI/WebUI evidence is captured, and how runtime/protocol facts are connected to visible surfaces.

## How to read this page

- Treat each local repository as a case study, not as a normative dependency.
- Copy the testing shape, not the exact stack.
- Keep limitations explicit. The Claude Code local snapshot has useful interface code but no local `package.json` or workflow metadata, so this page does not claim upstream CI behavior for that snapshot.
- When a project has UI, require both surface proof and runtime proof.


## Agent UI and Agent Skills lessons applied here

This page treats Agent UI as a primary reference for surface testing. The reusable lessons are:

- UI/TUI/WebUI/desktop states must be runtime-backed projections, not independent truth.
- Final answer text must stay separate from reasoning, tool progress, approvals, artifacts, evidence, diagnostics, and team events.
- Missing runtime facts must render as `unknown`, `unavailable`, `stale`, or `blocked`, not guessed success.
- Controlled writes such as approval, interrupt, queue, steer, artifact edit, evidence export, review, or replay must go through the owning API.
- Old sessions and long-running work need progressive hydration and surface-specific evidence.
- Metrics such as first status, first text, bridge readiness, queue wait, trace size, and cleanup time are part of QC evidence.

Agent Skills contributes the authoring style: short entrypoints, frontmatter, field tables, minimal examples, progressive disclosure, eval loops, assertion grading, and transcripts. Agent QC uses that style for quality plans rather than skill packages.

## Framework documentation lessons

Official framework docs are used as examples of evidence shape, not as mandatory tool choices:

| Framework | Reusable QC lesson |
| --- | --- |
| Playwright | Projects/devices, `webServer`, retries, reporters, trace/screenshot/video policies, and test isolation are portable browser-evidence concepts. |
| Vitest | `run`, projects/workspaces, JSON/JUnit reporters, coverage, snapshots, and browser mode map JS projects into deterministic and browser lanes. |
| pytest | markers and `-m` selection, skip/xfail, parametrization, xdist, and JUnit-style reports help Python projects separate deterministic, integration, e2e, and live suites. |
| cargo nextest/Bazel | fast Rust workspace runs, no-fail-fast behavior, release binary builds, and generated schema checks show how runtime projects layer local and release evidence. |

## Cross-project surface map

| Project | Runtime CLI / stream | TUI | WebUI | Desktop GUI | Browser automation | Channel/mobile | Eval/report UI | Release/distribution |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Codex | strong: `codex exec`, JSON/event processors, SSE fixtures | strong: ratatui/insta snapshots | indirect: app-server/client protocol surfaces | no desktop shell in inspected repo | limited through app/server tooling, not primary | no | review/protocol artifacts | strong: Bazel, release binaries, npm packages |
| Claude Code local snapshot | visible SDK stream adapters and commands | visible Ink surface and command views | not enough metadata to claim | no | remote bridge/control surfaces | no | no | not enough metadata to claim |
| OpenClaw | strong gateway/CLI/router tests | dedicated `tui` command and TUI lanes | strong control UI and QA Lab web runtime | platform release paths, mac/mobile scripts | QA Lab browser runtime, Docker/browser lanes | strong channel contracts, QR, Android/iOS, live transports | strong QA Lab scenarios/reports | strong Docker/install/release checks |
| Hermes Agent | strong Python CLI/gateway tests | strong `ui-tui` Vitest package | Vite/React dashboard package | no native shell in inspected repo | strong browser supervisor/CDP/Camofox/SSRF tests | strong gateway/channel tests | release notes and web/dashboard surfaces | Docker, uv lock, OSV, package checks |

Agent QC conclusion: a project can be "well tested" in one surface and still under-tested in another. Do not collapse all UI proof into one boolean.

## Shared test architecture pattern

Across the projects, the useful pattern is:

1. **Local deterministic lane**: format, lint, typecheck, unit, contract, fake integration.
2. **Runtime lane**: real CLI/task/session flow with fake provider or local server.
3. **Surface lane**: TUI/WebUI/GUI/browser/channel evidence with screenshots, snapshots, traces, or transcripts.
4. **Live lane**: opt-in real provider/channel/model tests with redaction and budget.
5. **Distribution lane**: install, Docker/package, cross-platform, release manifest, lock/supply-chain checks.
6. **Review/eval lane**: semantic quality, rubric, baseline diff, human or LLM review.

Agent QC plans should name which lanes apply and which lanes are intentionally out of scope.

## Codex: runtime CLI plus TUI plus protocol stack

Local source: `/Users/coso/Documents/dev/rust/codex`.

### Product shape

Codex combines several Agent product shapes:

- Rust runtime CLI and task loop.
- `codex exec` and structured stream outputs.
- TUI implemented with ratatui and snapshot tests.
- MCP/tool gateway, app server, app-server protocol, SDKs, release packaging, and sandbox layers.
- Cross-platform sandbox and process execution policies.

### How tests are organized

| Layer | Concrete signals | Agent QC interpretation |
| --- | --- | --- |
| Repository policy | root `AGENTS.md` tells contributors to run targeted crate tests first, then full `cargo test`/`just test` when common/core/protocol changed | targeted verification before broad sweeps |
| Local Rust lane | `just test` runs `cargo nextest run --no-fail-fast`; `cargo test -p <crate>` for focused work | deterministic `unit` and `runtime-e2e` evidence |
| Bazel lane | `bazel test //... --keep_going`, Bazel clippy, module lock checks, release binary builds | cross-toolchain parity and release confidence |
| Supply/policy lane | `cargo-deny`, codespell, clippy, argument-comment lint, lock checks | `static` and `distribution-release` hygiene |
| Sandbox/process lane | `exec_policy_tests`, Windows sandbox tests, sandbox tag tests, Landlock/bwrap/seatbelt-related tests | permission boundary and platform-specific runtime gates |
| Tool/protocol lane | MCP fixtures, app-server v2 protocol tests, schema fixture regeneration, dynamic tools, request permission tests | `contract-protocol` and fake integration |
| Stream lane | SSE end-to-end tests, fake response helpers, stream event utilities, JSON/event processor tests | stream shape evidence before semantic claims |
| TUI lane | ratatui/insta snapshots across chat widget, bottom pane, approval overlay, footer, request-user-input, MCP elicitation | TUI `ui-interaction` evidence |
| SDK/API lane | TypeScript SDK event/thread APIs and app-server client surfaces | `agent-sdk-api` contract evidence |
| Release lane | Bazel release binaries, npm/native package build scripts, Windows/zsh release workflows | `distribution-release` gate |

### TUI details worth standardizing

Codex demonstrates that TUI testing needs more than a screenshot:

- terminal-width and height variants: narrow, standard, and large terminals;
- approval overlays for exec, patch, network, cross-thread, and additional permissions;
- footer states: idle, running, Ctrl-C quit, Ctrl-C interrupt, Esc hint, queue hint, mode indicator, context/token status;
- request-user-input forms: options, freeform, multi-question, tight height, hidden options, long option text;
- model/session pickers: model migration prompt, fixed/auto column widths, narrow rows, scroll states;
- composer edge cases: paste, backspace after paste, slash popup, mention popup, plugin popup, remote image rows, shell-command mode;
- history/chat frames: diff syntax, code blocks, completed hook output, pending input, stream deltas, compact/resume/fork shapes;
- MCP and app-server states: MCP startup failures, elicitation forms, app-server collaboration and guardian review states;
- platform-specific snapshots such as Windows approval popup variants.

Agent QC rule: a TUI pass should cite terminal snapshots **and** runtime transcripts. Snapshot-only proof shows rendering; transcript-linked proof shows the rendering came from the correct Agent event.

### Runtime details worth standardizing

Codex separates deterministic runtime tests from live/provider risk:

- fake model server and SSE fixtures test stream shape without burning provider budget;
- app-server protocol tests assert wire shape independently from the TUI;
- apply-patch tests cover CLI and tool surfaces;
- exec/unified process tests preserve command output, cleanup, and failure semantics;
- sandbox tests assert denied actions and platform policy transforms;
- schema fixture writers make protocol drift reviewable.

Agent QC rule: CLI/runtime projects need `contract-protocol` and `runtime-e2e` gates before `semantic-eval` can be trusted.

### What to copy into Agent QC plans

For a Codex-like project, include cases such as:

- denied unsafe command produces a visible controlled error and non-success runtime event;
- apply-patch success/failure has stable CLI transcript and patch result;
- MCP tool declaration round-trips through config, server fixture, runtime event, and TUI row;
- Ctrl-C interrupts a running turn without leaving orphan subprocesses;
- app-server protocol schema diff is reviewed when command shape changes;
- release package contains expected native binaries and platform helpers.

## Claude Code local snapshot: TUI/runtime surface under incomplete repo metadata

Local source: `/Users/coso/Documents/dev/js/claudecode`.

### Source limitation

The inspected local snapshot contains source files under `src/` and `vendor/`, but no local `package.json`, lockfile, or GitHub workflow metadata. Agent QC must therefore avoid claiming upstream CI, test commands, package coverage, or release guarantees from this snapshot. The useful signal is interface-surface shape.

### Product surfaces visible in the snapshot

| Surface | Local indicators | Agent QC gate |
| --- | --- | --- |
| Ink TUI | `src/ink.ts`, `.tsx` command views, terminal focus/input/selection hooks, task views | TUI `ui-interaction` |
| Command palette | many `src/commands/**` handlers and renderable command views | command routing and TUI state snapshots |
| Remote session bridge | `src/remote/RemoteSessionManager.ts`, `src/remote/SessionsWebSocket.ts`, server direct connect manager | `contract-protocol`, `runtime-e2e` |
| Permission flow | `remotePermissionBridge.ts`, control schemas with `can_use_tool`, synthetic assistant/tool confirmation flow | high-risk TUI + protocol evidence |
| SDK stream | `src/remote/sdkMessageAdapter.ts`, `src/entrypoints/agentSdkTypes.ts`, stream/control schemas | `agent-sdk-api` stream contract |
| Skills/plugins | SDK schemas include skills/plugins; output style/plugin loading code is visible | `agent-skills-plugins` |

### What the standard should require

A Claude Code-style TUI runtime should prove:

- success, empty, error, cancelled, reconnecting, disconnected, and remote states render distinctly;
- command views route to the same state transitions as slash commands or command palette entries;
- permission prompts show tool name, request id, proposed input, permission suggestions, and deny/allow outcome;
- remote permission responses preserve request correlation and behavior (`allow`, `deny`, or project-specific modes);
- server-side cancellation removes or marks the pending prompt instead of leaving stale approvals visible;
- reconnect/interrupt cannot leave the TUI showing stale success;
- SDK stream adapters preserve event type, session id, tool-use id, and partial/final message semantics;
- tool results from a remote server render as tool results, not as prompt echoes;
- plugin/skill reload events cannot silently change allowed tools without a visible status or audit event.

### Evidence recipe

A minimal QC case should collect:

1. pseudo-terminal transcript of a remote permission request;
2. TUI snapshot showing the synthetic confirmation row;
3. WebSocket/control transcript for `can_use_tool` request and response;
4. SDK stream fixture proving event conversion;
5. negative case for cancellation or disconnect.

Agent QC rule: when repo metadata is incomplete, write the limitation into `evidence_policy` and require interface-level evidence instead of inventing a CI story.

## OpenClaw: multi-channel gateway plus WebUI plus QA Lab

Local source: `/Users/coso/Documents/dev/js/openclaw`.

### Product shape

OpenClaw is a dense Agent system:

- multi-channel gateway for provider/channel integrations;
- plugin ecosystem and plugin SDK;
- CLI, gateway, TUI command, control WebUI, Android/iOS/macOS platform paths;
- QA Lab extension with web runtime, browser runtime, scenario runner, live transport tests, and reports;
- Docker/install/release smoke paths;
- live provider lanes for models, gateways, and CLI backends.

### How tests are organized

OpenClaw's `package.json` exposes many lanes. The important Agent QC pattern is the separation, not the number of commands.

| Layer | Concrete signals | Agent QC interpretation |
| --- | --- | --- |
| Test router | `node scripts/test-projects.mjs`, `test:changed`, `test:max`, serial/max-worker variants | changed-scope and profile-aware gate selection |
| Static/policy | `check`, `lint`, import-cycle checks, LOC checks, host env policy, webhook/auth boundary lints | `static` plus security policy |
| Unit/gateway lanes | `test:unit`, `test:gateway`, gateway client/server/method configs | deterministic runtime and gateway behavior |
| Contract lanes | `test:contracts:channels`, `test:contracts:plugins`, plugin SDK export/API checks, protocol generation checks | `contract-protocol` for channel/plugin/runtime boundaries |
| WebUI lane | `test:ui`, `ui` package tests, browser-playwright-style UI config tests | `webui` under `ui-interaction` |
| TUI/platform lanes | `tui`, TUI scripts, `test:windows:ci`, `test:macos:ci`, Android/iOS unit/integration scripts | surface-specific UI/platform proof |
| QA Lab lane | `extensions/qa-lab` scenario catalog, web runtime, browser runtime, reports, live transports, suite summary JSON | `agent-evals-quality`, `eval-ui`, `webui`, `browser-automation` |
| Channel lanes | channel configs for Telegram/Matrix/Discord/Feishu/Zalo/etc., webhook/media/auth tests | `multi-channel-agent-gateway` |
| Live provider lanes | `test:live:*`, live model profiles, live gateway Docker lanes, live CLI backend lanes for Claude/Codex/Gemini-style backends | explicit `live-provider` with opt-in |
| Docker/install lanes | install smoke, OpenWebUI Docker, MCP channels Docker, QR import, plugins Docker, gateway network Docker | `distribution-release` and runtime smoke |
| Release lanes | `release:check`, npm checks, plugin release checks, version sync | release readiness and package boundary proof |
| Performance lanes | startup bench, import duration, perf budget, memory checks | performance risk gates |

### WebUI details worth standardizing

OpenClaw shows that WebUI proof should be layered:

- component/state tests for navigation, chat normalization, settings, controller panels, usage panels, tool cards, and config surfaces;
- browser-only tests for focus, markdown, sidebar status, external links, image opening, and browser APIs;
- QA Lab web runtime tests for scenario execution and report rendering;
- Docker-hosted OpenWebUI smoke to prove integration in a clean environment;
- console/network evidence whenever browser behavior is under test.

Agent QC rule: when behavior depends on DOM, focus, browser APIs, markdown sanitization, navigation, or report rendering, `webui` evidence must include browser-level artifacts, not just jsdom/component tests.

### Channel/provider details worth standardizing

OpenClaw makes four separations that Agent QC should require:

- channel contract tests are not live channel tests;
- fake provider integration is not live provider coverage;
- media/webhook/auth replay is separate from model semantic quality;
- plugin boundary tests are separate from runtime gateway tests.

Examples of useful case shapes:

- secret refs are redacted and inactive channel credentials cannot be used;
- QR import creates a scoped session and can be replayed in Docker smoke;
- webhook body verification happens before parsing user content;
- media attachments preserve type/size limits and redaction;
- live transport credentials are leased, timed out, and redacted in reports;
- control WebUI shows actual gateway status, not cached healthy state.

Agent QC rule: `multi-channel-agent-gateway` projects should never hide live-provider assumptions inside ordinary unit tests.

## Hermes Agent: Python agent plus TUI plus browser/web tools plus scheduler

Local source: `/Users/coso/Documents/dev/python/hermes-agent`.

### Product shape

Hermes combines:

- Python Agent runtime, CLI, toolsets, gateway, and ACP/MCP adapters;
- pytest-based backend tests;
- browser, web provider, CDP, Camofox, Browserbase-style provider, and SSRF hardening tests;
- cron/background scheduler, checkpointing, approval, restart/retry, and concurrency surfaces;
- `ui-tui` Ink/React TUI package with Vitest tests;
- `web` Vite/React dashboard package;
- Docker image, uv lock, OSV/security, and release checks.

### How tests are organized

| Layer | Concrete signals | Agent QC interpretation |
| --- | --- | --- |
| Canonical runner | `scripts/run_tests.sh` pins `-n 4`, `TZ=UTC`, `LANG=C.UTF-8`, `PYTHONHASHSEED=0`, activates venv, blanks credential env vars, excludes integration/e2e by default | reproducible local evidence and credential hygiene |
| Pytest backend | `tests/` with gateway, cron, CLI, ACP, browser/tool, security, restart, retry, queue, platform tests | deterministic `unit`, `fake-integration`, `runtime-e2e` |
| Tool safety | write deny, file guards, symlink confusion, URL safety, yolo/approval modes, env passthrough | permission/sandbox gates |
| Browser/web | browser supervisor, browser hardening, CDP, local SSRF, Camofox state, web providers | `browser-automation` gates |
| Gateway/channel | Discord, Feishu, Matrix, Mattermost, Google Chat, QQBot, delivery, media, reconnect, dedup, pairing, roles/DM scope | `channel-ui` and gateway contracts |
| MCP/OAuth/ACP | MCP e2e, OAuth metadata, SSE transport, reconnect, circuit breaker, tool 401 handling, ACP approval isolation | `contract-protocol` and recovery |
| Scheduler | cron jobs, cron prompt injection, inactivity timeout, workdir, scheduler MCP init, checkpoint/session cleanup | `background-agent-scheduler` |
| TUI | `ui-tui` Vitest: terminal parity, viewport, virtual history, slash parity, streaming markdown, OSC52, clipboard, terminal modes | TUI `ui-interaction` |
| Web dashboard | `web` package uses Vite/React build and lint scripts | `webui` when dashboard behavior changes |
| Distribution | Dockerfile builds browser dashboard/TUI assets; uv lock; OSV/security notes | `distribution-release` and supply chain |

### TUI/terminal details worth standardizing

Hermes TUI tests cover practical terminal mechanics:

- text wrapping, virtual history heights, scroll, viewport stores, precision wheel;
- terminal modes, truecolor, OSC52 clipboard, emoji, math Unicode, syntax/markdown;
- slash command parity, gateway events, session lifecycle, queue handling, turn store, state isolation;
- streaming markdown, reasoning/details rendering, subagent tree, status ticker;
- text input navigation, pass-through, wrapping, completion, composer state.

Agent QC rule: TUI testing should include terminal input/output mechanics, not only component snapshots.

### Browser and web details worth standardizing

Hermes browser/tool tests map directly to Agent QC:

- browser supervisor health and orphan reaper;
- browser hardening and local SSRF protections;
- CDP override, browser console, and local provider behavior;
- Camofox persistence/state isolation;
- Brave/DDGS/SearXNG/Tavily-like web provider contracts;
- CLI browser connect and gateway browser-related command tests.

Agent QC rule: browser automation gates must include safety and cleanup evidence, not only screenshots.

### Scheduler/channel details worth standardizing

Hermes shows why background agents need their own gate family:

- cron prompt injection must be scanned after skills/context are assembled, not only at user input;
- scheduler restart must not duplicate work or lose checkpoints;
- inactivity timeout should track real tool activity, not wall-clock time alone;
- gateway restart/retry/dedup tests should preserve message ids and delivery state;
- credential-shaped environment variables must be blanked or scoped in tests.

Agent QC rule: a background scheduler pass should include deterministic clock/env settings, checkpoint evidence, and cleanup evidence.

## Cross-project extraction

Agent QC generalizes these projects into ten reusable rules:

1. Start from owned risk, not from language or framework.
2. Split UI surface proof from runtime/protocol proof, then connect them with evidence refs.
3. Keep fake integration, live provider, and release smoke as separate gates.
4. For TUI/WebUI/GUI, preserve surface artifacts: snapshots, traces, screenshots, console logs, terminal transcripts.
5. For browser automation, require DOM/a11y plus console/network plus cleanup evidence.
6. For channel/mobile, separate webhook/media/auth replay from live provider tests.
7. For background agents, pin deterministic time/env/worker settings and preserve checkpoint evidence.
8. For SDK/protocol surfaces, use generated schema diffs and fake servers before live runs.
9. For release claims, test package contents and installation paths, not just source tests.
10. For incomplete local snapshots, record what was inspected and what cannot be inferred.

## Recommended Agent QC case mix

| Risk | Minimum case | Stronger case |
| --- | --- | --- |
| Permission prompt | snapshot/frame shows prompt, transcript shows request id | allow/deny/cancel/reconnect variants with protocol transcript |
| Tool stream | fake provider stream parses and renders | malformed stream, tool error, partial/final event, retry, abort |
| TUI rendering | one stable snapshot | multi-viewport, key sequence, Unicode/ANSI, runtime-linked transcript |
| WebUI control | component state test | browser trace, console/network, keyboard/a11y, reload/resume |
| Desktop bridge | shell starts | bridge health, workspace readiness, native command contract, screenshot |
| Browser control | screenshot | DOM/a11y, console/network, cleanup, SSRF/navigation safety |
| Channel adapter | contract fixture | webhook replay, media fixture, redacted transcript, live opt-in lane |
| Scheduler | deterministic unit | restart/reclaim, concurrency, checkpoint, duplicate-work prevention |
| Eval report | rubric exists | baseline delta, judge output, failing examples, reviewer note |
| Release | build succeeds | package manifest, install smoke, Docker/platform matrix, lock/security check |


# Source index

Source: https://limecloud.github.io/agentqc/en/reference/source-index

# Source index

Agent QC v0.5.0 is derived from local project inspection plus public documentation. Local repositories are case studies, not normative dependencies.

Last reviewed: 2026-05-17.

## Citation format

Use source ids in design notes or changelogs:

```text
[SRC-AGENTUI-BEST-PRACTICES] -> surface pass must link visible projection to runtime facts.
```

## Local standards repositories

| Source id | Source | Evidence used | Agent QC requirements informed |
| --- | --- | --- | --- |
| `SRC-AGENTUI-BEST-PRACTICES` | `/Users/coso/Documents/dev/ai/limecloud/agentui/docs/en/authoring/best-practices.md` | runtime-owned facts, event classes, stable ids, fallback states, controlled writes, old-session design, latency metrics | Agent QC requires surface evidence to link visible frames to runtime/protocol facts and avoid UI-owned verdicts. |
| `SRC-AGENTUI-ACCEPTANCE` | `/Users/coso/Documents/dev/ai/limecloud/agentui/docs/en/authoring/acceptance-scenarios.md` | send/status, tool, HITL, queue/steer, artifact, evidence, old-session, team/parallel/remote/background scenarios | Agent QC acceptance scenarios cover runtime, TUI, WebUI, team, remote, and eval flows. |
| `SRC-AGENTUI-FLOW` | `/Users/coso/Documents/dev/ai/limecloud/agentui/docs/en/reference/flow-and-taxonomy.md` | lifecycle, event envelope, fact owners, scopes, phases, surfaces, controls, team taxonomy | Agent QC flow/taxonomy mirrors explicit dimensions and fact-owner separation. |
| `SRC-AGENTUI-CONTRACTS` | `/Users/coso/Documents/dev/ai/limecloud/agentui/docs/en/contracts/*.md` | backend coordination, runtime event projection, performance metrics | Agent QC adds evidence, performance, and reliability contracts for UI/TUI/desktop/browser gates. |
| `SRC-AGENTKNOWLEDGE-SPEC` | `/Users/coso/Documents/dev/ai/limecloud/agentknowledge/docs/en/specification.md` | directory-as-standard, progressive disclosure, source maps, compile/eval evidence, knowledge-as-data boundary | Agent QC keeps Knowledge as requirements/context input, not proof, and preserves source traceability. |

## Local project case studies

| Source id | Source | Use |
| --- | --- | --- |
| `SRC-CODEX-LOCAL` | `/Users/coso/Documents/dev/rust/codex` | Runtime CLI, Rust, Bazel, cargo nextest, SDK, MCP, app-server protocol, sandbox, process cleanup, TUI snapshots, schema fixtures, release patterns. |
| `SRC-CLAUDECODE-LOCAL` | `/Users/coso/Documents/dev/js/claudecode` | Partial local source snapshot for Ink TUI, remote bridge, WebSocket control, permission flow, SDK stream adapter, commands, task/team surfaces; not enough metadata for CI/release claims. |
| `SRC-OPENCLAW-LOCAL` | `/Users/coso/Documents/dev/js/openclaw` | Multi-channel gateway, Vitest lane routing, UI browser-mode tests, QA Lab, live provider opt-in, Docker/install smoke, plugin/secret/channel contracts, mobile/platform scripts. |
| `SRC-HERMES-LOCAL` | `/Users/coso/Documents/dev/python/hermes-agent` | Python pytest, markers, xdist, integration/e2e separation, credential blanking, cron/scheduler, browser safety, gateway/channel tests, TUI Vitest, Docker/uv/OSV. |

## External public sources

| Source id | Source | Evidence used | Agent QC requirements informed |
| --- | --- | --- | --- |
| `SRC-AGENTSKILLS-SPEC` | `https://agentskills.io/specification` | Markdown/frontmatter style, directory-as-package, progressive disclosure, fields/constraints/examples. | Agent QC docs use concise entry pages, tables, examples, and deeper reference pages. |
| `SRC-AGENTSKILLS-EVAL` | `https://agentskills.io/skill-creation/evaluating-skills` | Eval-driven iteration, clean-context runs, assertion grading, execution transcripts, human feedback. | qcloop and eval gates require attempts, verifier feedback, rubrics, and evidence refs. |
| `SRC-HARBOR-DOCS` | `https://www.harborframework.com/docs`, `/docs/tasks`, `/docs/run-jobs/run-evals`, `/docs/run-jobs/results-and-artifacts`, `/docs/rewardkit`, `/docs/agents/trajectory-format`, `/docs/metrics` | `harbor init --task`, task directories, `task.toml` schema, separate verifier transfer rules, `/logs/verifier/reward.txt|json`, RewardKit criteria/judges, job/trial layout, ATIF trajectories, custom metrics. | Benchmark and hill-climbing gates require frozen tasks, trial trajectories, reward details, artifact refs, verifier isolation, artifact collection status, and dataset-level metrics. |
| `SRC-CLINE-HILL-CLIMBING` | `https://cline.bot/blog/a-practical-guide-to-hill-climbing` | baseline runs, failure analysis, one-variable A/B changes, repeated runs/pass@k for noise, Harbor execution. | Agent QC adds `benchmark-eval` and the hill-climbing authoring loop for improving Lime without conflating benchmark scores with release gates. |
| `SRC-YAGE-RUNTIME-BATTLEFIELD` | `https://yage.ai/share/agent-runtime-battlefield-20260516.html` | runtime/harness can materially change benchmark outcomes for the same model; builders should A/B on their own repos. | Agent QC treats runtime/prompt/tool/context profiles as benchmark variables and requires project-local tasks for Lime improvement. |
| `SRC-PLAYWRIGHT-CONFIG` | `https://playwright.dev/docs/test-configuration` and Context7 `/microsoft/playwright.dev` | projects, webServer, retries, reporters, trace, screenshot, video, test isolation. | WebUI/browser/desktop gates require trace/screenshot/video policy, browser project/device, console/network, and server startup evidence when relevant. |
| `SRC-VITEST-DOCS` | `https://vitest.dev/guide/cli.html` and Context7 `/vitest-dev/vitest` | CLI run/watch, projects, reporter JSON/JUnit, coverage, browser mode, snapshots. | JS projects map Vitest suites to deterministic, browser, contract, and report evidence lanes. |
| `SRC-PYTEST-MARKERS` | `https://docs.pytest.org/en/stable/example/markers.html` and Context7 `/pytest-dev/pytest` | markers, `-m` selection, skip/xfail, parametrization, test routing. | Python projects separate deterministic, integration, e2e, live, and slow suites with explicit selection and evidence. |
| `SRC-MCP-TOOLS` | `https://modelcontextprotocol.io/specification/2025-11-25/server/tools` | tool declaration/protocol boundary. | Tool/MCP gateway gates require declaration and invocation evidence, not only final text. |
| `SRC-CODEX-ACTIONS` | `https://github.com/openai/codex/actions` | public workflow signal. | Used only as external context; local repo inspection remains the case-study detail. |
| `SRC-HERMES-GITHUB` | `https://github.com/NousResearch/hermes-agent` | public project context. | Used only for public project identity; local repo inspection supplies testing details. |

## Requirement traceability

| Requirement area | Primary sources |
| --- | --- |
| Surface evidence must link visible frame to runtime facts | `SRC-AGENTUI-BEST-PRACTICES`, `SRC-AGENTUI-FLOW`, `SRC-CODEX-LOCAL`, `SRC-OPENCLAW-LOCAL`, `SRC-HERMES-LOCAL` |
| Expanded acceptance scenarios | `SRC-AGENTUI-ACCEPTANCE`, `SRC-CODEX-LOCAL`, `SRC-OPENCLAW-LOCAL`, `SRC-HERMES-LOCAL` |
| TUI evidence | `SRC-CODEX-LOCAL`, `SRC-CLAUDECODE-LOCAL`, `SRC-HERMES-LOCAL` |
| WebUI/browser evidence | `SRC-PLAYWRIGHT-CONFIG`, `SRC-OPENCLAW-LOCAL`, `SRC-AGENTUI-BEST-PRACTICES` |
| Python suite routing | `SRC-PYTEST-MARKERS`, `SRC-HERMES-LOCAL` |
| Live provider separation | `SRC-OPENCLAW-LOCAL`, `SRC-HERMES-LOCAL` |
| Scheduler/background gates | `SRC-HERMES-LOCAL`, `SRC-AGENTUI-ACCEPTANCE` |
| Release/distribution gates | `SRC-CODEX-LOCAL`, `SRC-OPENCLAW-LOCAL`, `SRC-HERMES-LOCAL` |
| Progressive documentation style | `SRC-AGENTSKILLS-SPEC`, `SRC-AGENTKNOWLEDGE-SPEC`, `SRC-AGENTUI-BEST-PRACTICES` |
| qcloop/eval evidence loop | `SRC-AGENTSKILLS-EVAL`, `SRC-OPENCLAW-LOCAL` |
| Benchmark/hill-climbing loop | `SRC-HARBOR-DOCS`, `SRC-CLINE-HILL-CLIMBING`, `SRC-YAGE-RUNTIME-BATTLEFIELD`, `SRC-PLAYWRIGHT-CONFIG` |
| Harbor-compatible benchmark pack | `SRC-HARBOR-DOCS`, `SRC-CLINE-HILL-CLIMBING` |


# Agent standards ecosystem

Source: https://limecloud.github.io/agentqc/en/reference/agent-ecosystem

# Agent standards ecosystem

Agent QC owns the quality-control contract for Agent projects. It links to adjacent standards through refs instead of owning their facts.

Agent QC should answer: **does evidence prove this Agent project quality claim?** It should not become the runtime, UI, policy, artifact, tool, or knowledge standard itself.

## Boundary map

| Standard | Role | Relationship to Agent QC |
| --- | --- | --- |
| Agent Knowledge | Source-grounded knowledge packs. | Supplies trusted requirements, docs, domain facts, source maps, and grounding expectations for tests. |
| Agent UI | User-visible interaction surfaces. | Supplies UI/TUI/desktop/WebUI acceptance expectations and runtime-backed projection rules. |
| Agent Runtime | Execution facts, controls, tasks, tools, streams, and recovery. | Supplies run/task/session state for runtime gates. |
| Agent Evidence | Evidence, verification, review, replay, and export. | Owns durable evidence records referenced by QC verdicts. |
| Agent Policy | Permissions, approvals, risk, retention, waivers. | Defines whether high-risk test actions may run or be waived. |
| Agent Artifact | Durable deliverables and handoff packages. | Stores reports, screenshots, traces, logs, package manifests, and generated outputs. |
| Agent Tool | Tool declarations, calls, progress, results, permissions, and audit refs. | Supplies tool invocation facts for tool, MCP, ACP, and connector gates. |
| Agent Context | Context selection, budgets, injection, missing facts, compaction. | Explains what context worker/verifier/judge agents received during QC. |
| Agent QC | Plans, profiles, gates, evidence, verdicts, waivers, and reports. | Owns whether testing evidence proves a project quality claim. |

## Interop principles

1. **QC references facts; it does not own them.** A GUI pass references Agent UI projection and runtime events; it does not define the UI protocol.
2. **Evidence refs are durable.** The QC report should link to Agent Evidence or artifact refs rather than paste secret-bearing logs.
3. **Policy controls live outside QC.** QC may require approval, but Agent Policy owns whether a dangerous action is allowed or waived.
4. **Knowledge is input, not proof.** Requirements from Agent Knowledge guide tests; passing evidence still comes from execution, traces, or review.
5. **Artifacts are outputs, not verdicts.** A package, screenshot, or report is evidence only when connected to an expectation.
6. **Context is part of reviewability.** qcloop/verifier/model-judge results need context/budget/source refs when context affects the outcome.

## Example: UI/TUI/Desktop case

```text
Agent Runtime emits run/tool/action facts
  -> Agent UI projects them into composer/status/tool/HITL surfaces
  -> Agent Evidence stores trace/screenshot/transcript refs
  -> Agent QC links refs to gate verdicts
  -> Agent Policy records waiver or approval when needed
```

QC failure example: a screenshot shows "done" but no runtime event confirms completion. Agent UI projection may be visually correct for a mock state, but Agent QC must mark the runtime-backed claim `blocked` or `needs-review`.

## Example: Knowledge-driven eval case

```text
Agent Knowledge supplies source-grounded requirements
  -> Agent Context selects test context and budgets
  -> qcloop or eval runner executes cases
  -> Agent Evidence stores attempts, verifier feedback, judge output
  -> Agent QC emits verdicts and remaining risk
```

QC failure example: the model answer cites a source but the source map is missing. The semantic output may look plausible, but the grounded-quality claim is incomplete.

## What Agent QC deliberately does not standardize

- visual theme, component library, typography, or animation;
- a single test framework or CI provider;
- a single runtime event protocol;
- model provider choice;
- storage backend for evidence;
- product-specific release policy;
- exact qcloop implementation.

Agent QC standardizes the evidence shape and verdict semantics that let these systems interoperate.


# Glossary

Source: https://limecloud.github.io/agentqc/en/reference/glossary

# Glossary

| Term | Meaning |
| --- | --- |
| Project profile | Agent project shape that determines likely risks and gates. |
| Gate | Required validation boundary for a profile or change risk. |
| Evidence | Inspectable record that supports or disproves a verdict. |
| Verdict | Evidence-backed passed/failed/blocked/exhausted/waived/needs-review judgment. |
| qcloop | Batch execution loop for worker/verifier/repair tasks. |
| Attempt | qcloop worker or repair execution for one item. |
| QC round | qcloop verifier execution for one item. |
| Exhausted | Attempts or budget ended without proof. |
| Waiver | Explicit accepted gap with reason, owner, and expiry. |
| Live provider | Real external model, API, channel, or network dependency. |
| Semantic eval | Test that judges model/task output against a rubric or baseline. |
| Remaining risk | What current evidence still does not prove. |


# Codex runtime CLI example

Source: https://limecloud.github.io/agentqc/en/examples/codex-runtime-cli

# Codex runtime CLI example

Profiles:

- `agent-runtime-cli`
- `agent-tool-mcp-gateway`
- `agent-sdk-api`
- `agent-ui-tui-desktop`
- `agent-distribution-release`

Typical gates:

- static: `cargo fmt`, `cargo clippy`, dependency/policy checks, Bazel parity.
- unit/runtime: targeted crate tests and `cargo nextest run --no-fail-fast`.
- contract/protocol: MCP fixtures, SSE fixtures, app-server protocol schemas, SDK/API tests.
- ui-interaction: terminal snapshots for approval overlays, footer/status, diff/history, request-user-input, resume/model picker.
- runtime-e2e: CLI exec/apply-patch/resume/sandbox/process cleanup suites.
- distribution-release: Bazel release matrix, native package contents, npm/package staging.

Public QC plan JSON:

```json
{
  "schema_version": "0.4.0",
  "id": "codex-runtime-tui-protocol-qc",
  "target_project": "codex",
  "project_profiles": [
    "agent-runtime-cli",
    "agent-tool-mcp-gateway",
    "agent-sdk-api",
    "agent-ui-tui-desktop",
    "agent-distribution-release"
  ],
  "risk_level": "high",
  "risk_domains": [
    "sandbox",
    "tool-execution",
    "protocol",
    "tui-rendering",
    "release-package"
  ],
  "required_gates": [
    "static",
    "unit",
    "contract-protocol",
    "runtime-e2e",
    "ui-interaction",
    "distribution-release"
  ],
  "cases": [
    {
      "id": "deny-unsafe-tool-action",
      "name": "Deny unsafe tool action",
      "project_profile": "agent-runtime-cli",
      "target": "sandbox policy",
      "steps": [
        "Run the sandbox deny fixture",
        "Capture CLI transcript, structured event sample, and exit status"
      ],
      "expected": [
        "Unsafe action is denied",
        "Transcript includes a controlled error and no orphan subprocess remains"
      ],
      "risk": "permission bypass",
      "required_gates": [
        "unit",
        "runtime-e2e"
      ],
      "required_evidence": [
        "command_log",
        "cli_transcript",
        "process_cleanup_note"
      ],
      "status": "planned",
      "surface": "cli-stream"
    },
    {
      "id": "approval-overlay-renders-request-id",
      "name": "Approval overlay renders request id and action",
      "project_profile": "agent-ui-tui-desktop",
      "target": "ratatui approval overlay",
      "steps": [
        "Trigger an exec or patch approval fixture",
        "Render the TUI at narrow and standard viewport sizes",
        "Capture runtime approval event and terminal snapshots"
      ],
      "expected": [
        "Overlay shows command or patch intent, permission reason, and request correlation",
        "Ctrl-C/Esc footer state is correct while approval is pending",
        "Allow or deny result is reflected in the runtime transcript"
      ],
      "risk": "invisible or stale permission prompt",
      "required_gates": [
        "contract-protocol",
        "ui-interaction"
      ],
      "required_evidence": [
        "terminal_snapshot",
        "viewport_matrix",
        "runtime_event_transcript"
      ],
      "status": "planned",
      "surface": "tui"
    },
    {
      "id": "mcp-tool-event-round-trip",
      "name": "MCP tool event round-trips through protocol and UI",
      "project_profile": "agent-tool-mcp-gateway",
      "target": "MCP tool declaration and stream event",
      "steps": [
        "Mount MCP client/server fixture",
        "Run a fake model stream that calls the tool",
        "Inspect app-server or CLI protocol transcript"
      ],
      "expected": [
        "Tool name, schema, permission boundary, and output shape remain stable",
        "Runtime event preserves tool call id and structured output"
      ],
      "risk": "tool declaration drift or stream event mismatch",
      "required_gates": [
        "contract-protocol",
        "fake-integration"
      ],
      "required_evidence": [
        "schema_diff_or_fixture",
        "fake_server_log",
        "stream_transcript"
      ],
      "status": "planned",
      "surface": "cli-stream"
    },
    {
      "id": "release-binary-package-contents",
      "name": "Release package contains expected platform helpers",
      "project_profile": "agent-distribution-release",
      "target": "native CLI package",
      "steps": [
        "Build or dry-run the release package",
        "Inspect native binaries and helper scripts",
        "Compare package manifest with expected platform matrix"
      ],
      "expected": [
        "Package contains the CLI binary and required platform helpers",
        "Manifest does not include unexpected generated or secret files"
      ],
      "risk": "broken or unsafe distribution artifact",
      "required_gates": [
        "distribution-release"
      ],
      "required_evidence": [
        "package_manifest",
        "build_log",
        "platform_matrix_note"
      ],
      "status": "planned",
      "surface": "cli-stream"
    }
  ],
  "evidence_policy": "Every pass must link to command, fixture, transcript, snapshot, schema diff, package manifest, or CI evidence. TUI proof must include both terminal snapshots and runtime transcripts."
}
```


# Claude Code TUI runtime snapshot example

Source: https://limecloud.github.io/agentqc/en/examples/claudecode-tui-runtime

# Claude Code TUI runtime snapshot example

This example is based on the local snapshot at `/Users/coso/Documents/dev/js/claudecode`. The snapshot does not include package or workflow metadata, so the example only claims interface-surface observations.

Profiles:

- `agent-runtime-cli`
- `agent-ui-tui-desktop`
- `agent-sdk-api`
- `agent-tool-mcp-gateway`
- `agent-skills-plugins`

Typical gates:

- contract/protocol: remote session manager, WebSocket/control schemas, `can_use_tool` permission flow.
- ui-interaction: Ink TUI snapshots for command views, synthetic permission rows, reconnect/cancel states.
- runtime-e2e: remote cancellation, interrupt, reconnect, and stale-state prevention.
- sdk-api: stream adapter fixtures preserving event types and tool-use ids.
- review: explicit limitation note because CI/test runner metadata is not available in the local snapshot.

Public QC plan JSON:

```json
{
  "schema_version": "0.4.0",
  "id": "claudecode-tui-permission-qc",
  "target_project": "claudecode-local-snapshot",
  "project_profiles": [
    "agent-runtime-cli",
    "agent-ui-tui-desktop",
    "agent-sdk-api",
    "agent-tool-mcp-gateway",
    "agent-skills-plugins"
  ],
  "risk_level": "high",
  "risk_domains": [
    "tui",
    "remote-permission",
    "tool-streaming",
    "session-resume",
    "plugin-skill-reload"
  ],
  "required_gates": [
    "static",
    "contract-protocol",
    "runtime-e2e",
    "ui-interaction"
  ],
  "cases": [
    {
      "id": "remote-permission-request-renders-and-resolves",
      "name": "Remote permission request renders and resolves in TUI",
      "project_profile": "agent-ui-tui-desktop",
      "target": "remote permission bridge and Ink TUI",
      "steps": [
        "Inject a remote can_use_tool control request",
        "Render the synthetic tool confirmation row in the TUI",
        "Approve or deny and capture the outbound permission response"
      ],
      "expected": [
        "Permission prompt identifies the remote tool and request id",
        "TUI does not require a local Tool object to render the request",
        "Permission response preserves behavior and request correlation"
      ],
      "risk": "remote tool permission mismatch or invisible prompt",
      "required_gates": [
        "contract-protocol",
        "ui-interaction"
      ],
      "required_evidence": [
        "tui_snapshot",
        "control_request_transcript",
        "permission_response_log"
      ],
      "status": "planned",
      "surface": "tui"
    },
    {
      "id": "sdk-stream-adapter-preserves-tool-use-id",
      "name": "SDK stream adapter preserves tool-use ids",
      "project_profile": "agent-sdk-api",
      "target": "SDK message adapter",
      "steps": [
        "Feed SDK partial assistant messages and tool results into the adapter",
        "Capture converted stream events",
        "Compare event type and tool-use id mapping"
      ],
      "expected": [
        "Partial and final events keep their event type",
        "Tool result keeps the original tool-use id",
        "Malformed input produces a controlled error or ignored event"
      ],
      "risk": "remote stream rendered as wrong tool/result state",
      "required_gates": [
        "contract-protocol",
        "unit"
      ],
      "required_evidence": [
        "stream_fixture",
        "adapter_output_snapshot",
        "negative_case_log"
      ],
      "status": "planned",
      "surface": "cli-stream"
    },
    {
      "id": "remote-cancel-clears-pending-prompt",
      "name": "Remote cancel clears pending permission prompt",
      "project_profile": "agent-ui-tui-desktop",
      "target": "remote session manager cancellation path",
      "steps": [
        "Open a pending remote permission request",
        "Inject a server-side control cancel event",
        "Capture TUI frame and pending request store"
      ],
      "expected": [
        "Pending approval is removed or clearly marked cancelled",
        "No stale allow button remains visible",
        "Runtime transcript records the cancelled request id"
      ],
      "risk": "stale approval after remote cancellation",
      "required_gates": [
        "runtime-e2e",
        "ui-interaction"
      ],
      "required_evidence": [
        "tui_snapshot",
        "control_cancel_transcript",
        "state_store_snapshot"
      ],
      "status": "planned",
      "surface": "tui"
    },
    {
      "id": "plugin-skill-reload-is-visible",
      "name": "Plugin or skill reload is visible and auditable",
      "project_profile": "agent-skills-plugins",
      "target": "skill/plugin reload surface",
      "steps": [
        "Trigger a reload or changed allowed-tools fixture",
        "Capture command/TUI status and runtime audit event",
        "Verify allowed tools changed only through the declared path"
      ],
      "expected": [
        "Reload result is visible to the operator",
        "Allowed tool changes are auditable",
        "No silent privilege expansion occurs"
      ],
      "risk": "silent tool permission drift through plugin or skill reload",
      "required_gates": [
        "contract-protocol",
        "review"
      ],
      "required_evidence": [
        "reload_transcript",
        "allowed_tools_diff",
        "review_note"
      ],
      "status": "planned",
      "surface": "tui"
    }
  ],
  "evidence_policy": "Because the local snapshot has no package or workflow metadata, do not claim upstream CI coverage; require interface-level evidence only."
}
```


# OpenClaw channel gateway example

Source: https://limecloud.github.io/agentqc/en/examples/openclaw-channel-gateway

# OpenClaw channel gateway example

Profiles:

- `multi-channel-agent-gateway`
- `agent-tool-mcp-gateway`
- `agent-skills-plugins`
- `agent-ui-tui-desktop`
- `agent-evals-quality`
- `agent-distribution-release`

Typical gates:

- unit/contract: channel contracts, plugin contracts, secret refs, provider surfaces, media policies.
- fake-integration: gateway with fake provider, webhook replay, QR/import fixture, Docker smoke.
- ui-interaction: control WebUI browser trace, QA Lab report UI, channel transcript, browser runtime.
- live-provider: explicit opt-in model/channel/CLI backend lanes with redaction and budget.
- distribution-release: Docker/OpenWebUI/MCP channel/plugin install smoke and release checks.

Public QC plan JSON:

```json
{
  "schema_version": "0.4.0",
  "id": "openclaw-channel-webui-qalab-qc",
  "target_project": "openclaw",
  "project_profiles": [
    "multi-channel-agent-gateway",
    "agent-tool-mcp-gateway",
    "agent-skills-plugins",
    "agent-ui-tui-desktop",
    "agent-evals-quality",
    "agent-distribution-release"
  ],
  "risk_level": "high",
  "risk_domains": [
    "channel-auth",
    "secrets",
    "media-routing",
    "webui-state",
    "live-provider"
  ],
  "required_gates": [
    "static",
    "contract-protocol",
    "fake-integration",
    "ui-interaction",
    "distribution-release"
  ],
  "cases": [
    {
      "id": "telegram-secret-ref-isolation",
      "name": "Telegram secret ref isolation",
      "project_profile": "multi-channel-agent-gateway",
      "target": "channel secret runtime",
      "steps": [
        "Run channel contract tests",
        "Inspect redacted secret ref transcript",
        "Replay inactive-channel credential fixture"
      ],
      "expected": [
        "Secrets are referenced, not leaked",
        "Inactive channels cannot use active credentials",
        "Webhook/auth checks happen before content dispatch"
      ],
      "risk": "credential leakage or cross-channel auth drift",
      "required_gates": [
        "contract-protocol",
        "fake-integration"
      ],
      "required_evidence": [
        "test_report",
        "redacted_transcript",
        "webhook_replay_log"
      ],
      "status": "planned",
      "surface": "channel-ui"
    },
    {
      "id": "control-webui-shows-real-gateway-status",
      "name": "Control WebUI shows real gateway status",
      "project_profile": "agent-ui-tui-desktop",
      "target": "control WebUI gateway status panel",
      "steps": [
        "Start the gateway with a fake provider",
        "Open the control UI in browser mode",
        "Capture status panel, console log, and network log"
      ],
      "expected": [
        "Status panel matches gateway health response",
        "No console error is hidden behind a healthy UI state",
        "Reload or reconnect does not show stale healthy status"
      ],
      "risk": "WebUI reports cached or fake healthy state",
      "required_gates": [
        "ui-interaction",
        "fake-integration"
      ],
      "required_evidence": [
        "browser_trace",
        "screenshot",
        "console_network_log"
      ],
      "status": "planned",
      "surface": "webui"
    },
    {
      "id": "qa-lab-report-preserves-failures",
      "name": "QA Lab report preserves failing scenario evidence",
      "project_profile": "agent-evals-quality",
      "target": "QA Lab scenario runner and report UI",
      "steps": [
        "Run a mixed pass/fail scenario suite",
        "Open or export the report",
        "Inspect suite summary JSON and visible report rows"
      ],
      "expected": [
        "Failing cases remain visible in report and export",
        "Rubric, model profile, and evidence refs are preserved",
        "Summary counts match scenario-level results"
      ],
      "risk": "eval dashboard hides failures or rubric drift",
      "required_gates": [
        "semantic-eval",
        "ui-interaction",
        "review"
      ],
      "required_evidence": [
        "suite_summary_json",
        "report_screenshot",
        "rubric_and_review_note"
      ],
      "status": "planned",
      "surface": "eval-ui"
    },
    {
      "id": "openwebui-docker-smoke",
      "name": "OpenWebUI Docker smoke proves clean integration",
      "project_profile": "agent-distribution-release",
      "target": "Docker-hosted OpenWebUI integration",
      "steps": [
        "Run Docker smoke with clean state",
        "Open WebUI route and send a fixture request",
        "Capture container logs and browser evidence"
      ],
      "expected": [
        "Containers start without missing runtime assets",
        "WebUI request reaches the gateway through the expected route",
        "Cleanup removes containers or records intentional reuse"
      ],
      "risk": "release works only in dev checkout",
      "required_gates": [
        "distribution-release",
        "ui-interaction"
      ],
      "required_evidence": [
        "docker_smoke_log",
        "browser_trace",
        "cleanup_log"
      ],
      "status": "planned",
      "surface": "browser-automation"
    }
  ],
  "evidence_policy": "Live provider gates must be explicitly opted in, budgeted, and redacted. Channel contracts, WebUI browser proof, QA Lab reports, and Docker smoke evidence are separate gates."
}
```


# Hermes background agent example

Source: https://limecloud.github.io/agentqc/en/examples/hermes-background-agent

# Hermes background agent example

Profiles:

- `agent-runtime-cli`
- `background-agent-scheduler`
- `multi-channel-agent-gateway`
- `agent-ui-tui-desktop`
- `agent-tool-mcp-gateway`
- `agent-distribution-release`

Typical gates:

- deterministic pytest: `scripts/run_tests.sh` pins workers, timezone, locale, hash seed, and credential env cleanup.
- scheduler/concurrency: cron restart, checkpoint, inactivity timeout, duplicate-work prevention.
- browser-automation: browser supervisor, CDP, Camofox, SSRF, web provider contracts, cleanup evidence.
- TUI: `ui-tui` Vitest coverage for terminal parity, viewport, OSC52, streaming markdown, slash parity.
- channel/gateway: restart/retry/dedup, approval, delivery, media, reconnect, redacted transcripts.

Public QC plan JSON:

```json
{
  "schema_version": "0.4.0",
  "id": "hermes-scheduler-tui-browser-qc",
  "target_project": "hermes-agent",
  "project_profiles": [
    "agent-runtime-cli",
    "background-agent-scheduler",
    "multi-channel-agent-gateway",
    "agent-ui-tui-desktop",
    "agent-tool-mcp-gateway",
    "agent-distribution-release"
  ],
  "risk_level": "high",
  "risk_domains": [
    "cron",
    "checkpoint",
    "credential-isolation",
    "browser-safety",
    "tui-rendering"
  ],
  "required_gates": [
    "unit",
    "fake-integration",
    "runtime-e2e",
    "ui-interaction",
    "stress-concurrency"
  ],
  "cases": [
    {
      "id": "cron-restart-does-not-duplicate-work",
      "name": "Cron restart does not duplicate work",
      "project_profile": "background-agent-scheduler",
      "target": "cron scheduler recovery",
      "steps": [
        "Run scheduler restart test with deterministic clock/env",
        "Inspect checkpoint and evidence store",
        "Verify cleanup of ephemeral agent/session resources"
      ],
      "expected": [
        "No duplicate work item",
        "Recovered run owns final state",
        "Credential-shaped environment variables are blanked or scoped"
      ],
      "risk": "duplicate or lost background work",
      "required_gates": [
        "unit",
        "stress-concurrency"
      ],
      "required_evidence": [
        "pytest_report",
        "checkpoint_log",
        "env_scope_note"
      ],
      "status": "planned",
      "surface": "cli-stream"
    },
    {
      "id": "tui-streaming-markdown-parity",
      "name": "TUI streaming markdown and terminal parity",
      "project_profile": "agent-ui-tui-desktop",
      "target": "ui-tui package",
      "steps": [
        "Run terminal parity and streaming markdown fixtures",
        "Capture viewport, virtual history, and OSC52/clipboard assertions",
        "Compare slash command dispatch with gateway events"
      ],
      "expected": [
        "Streaming markdown renders without corrupting terminal width",
        "OSC52/clipboard and terminal mode behavior match expectations",
        "Slash command and gateway events produce consistent visible state"
      ],
      "risk": "terminal UI corrupts stream or command state",
      "required_gates": [
        "unit",
        "ui-interaction"
      ],
      "required_evidence": [
        "vitest_report",
        "terminal_snapshot_or_render_log",
        "viewport_matrix"
      ],
      "status": "planned",
      "surface": "tui"
    },
    {
      "id": "browser-local-ssrf-denied",
      "name": "Browser automation denies local SSRF",
      "project_profile": "agent-tool-mcp-gateway",
      "target": "browser/web tool safety",
      "steps": [
        "Run browser hardening or local SSRF fixture",
        "Capture browser supervisor state and console/network output",
        "Assert browser cleanup or orphan reaper result"
      ],
      "expected": [
        "Unsafe local target is denied or safely blocked",
        "Console/network evidence shows the blocked request",
        "Browser session cleanup is recorded"
      ],
      "risk": "browser tool can access forbidden local resources",
      "required_gates": [
        "fake-integration",
        "runtime-e2e"
      ],
      "required_evidence": [
        "browser_trace",
        "console_network_log",
        "cleanup_log"
      ],
      "status": "planned",
      "surface": "browser-automation"
    },
    {
      "id": "gateway-restart-deduplicates-delivery",
      "name": "Gateway restart deduplicates channel delivery",
      "project_profile": "multi-channel-agent-gateway",
      "target": "gateway restart/retry path",
      "steps": [
        "Replay a channel delivery fixture across restart",
        "Inspect message ids, retry state, and redacted transcript",
        "Verify final user-visible message count"
      ],
      "expected": [
        "Message is not delivered twice",
        "Retry/restart state is auditable",
        "Transcript redacts credentials and user-sensitive ids"
      ],
      "risk": "duplicate or leaked channel messages after restart",
      "required_gates": [
        "fake-integration",
        "runtime-e2e"
      ],
      "required_evidence": [
        "channel_transcript",
        "restart_log",
        "dedup_state_snapshot"
      ],
      "status": "planned",
      "surface": "channel-ui"
    }
  ],
  "evidence_policy": "Credential env vars must be blanked or scoped in test evidence. Scheduler, browser, TUI, and channel proofs must keep separate evidence refs."
}
```


# qcloop batch example

Source: https://limecloud.github.io/agentqc/en/examples/qcloop-batch

# qcloop batch example

Agent QC maps repeated independent cases to qcloop items.

## Job request

```json
{
  "name": "agentqc-runtime-permission-regression",
  "prompt_template": "You are testing an Agent project according to Agent QC. Parse this item JSON: {{item}}. Run only the requested steps. Collect evidence refs. Report status, commands, key output, blockers, and evidence refs. Do not claim pass without evidence.",
  "verifier_prompt_template": "Review this Agent QC item. Item: {{item}} Worker output: {{output}} Return strict JSON only: {\"pass\": true|false, \"status\": \"passed|failed|blocked|exhausted|needs-review\", \"severity\": \"none|low|medium|high|critical\", \"feedback\": \"specific reason\", \"evidence_refs\": [\"ref...\"], \"remaining_risk\": \"...\"}. Pass only when every expected behavior has evidence.",
  "max_qc_rounds": 2,
  "token_budget_per_item": 0,
  "execution_mode": "standard",
  "executor_provider": "codex",
  "items": [
    "{\"id\":\"deny-unsafe-tool-action\",\"name\":\"Deny unsafe tool action\",\"project_profile\":\"agent-runtime-cli\",\"target\":\"sandbox policy\",\"steps\":[\"Run the sandbox deny fixture\",\"Capture CLI transcript and exit status\"],\"expected\":[\"Unsafe action is denied\",\"Transcript includes controlled error\"],\"risk\":\"permission bypass\",\"required_gates\":[\"unit\",\"runtime-e2e\"],\"required_evidence\":[\"command_log\",\"cli_transcript\"],\"status\":\"planned\"}"
  ]
}
```

## Verifier output

```json
{
  "pass": false,
  "status": "failed",
  "severity": "high",
  "feedback": "The transcript shows the unsafe write succeeded outside the workspace.",
  "evidence_refs": [
    "qcloop://jobs/job-123/items/deny-unsafe-tool-action/attempts/1"
  ],
  "remaining_risk": "Sandbox policy may not be enforced for this path."
}
```