core/.trellis/spec/backend/quality-guidelines.md

# Quality Guidelines

Workspace-owned Rust forbids unsafe code. Decimal128 and insertion-ordered Map reuse the pinned `dec` and `indexmap` dependencies in `Cargo.lock`; do not reimplement these domains. Keep source loading, parsing, semantic checking, native emission, runtime and CLI separate. Parser success is not semantic success.

Root dev/test profiles keep `debug = 0` and `incremental = false` so independent focused test binaries do not recreate multi-gigabyte debuginfo and incremental object trees. Release keeps fat LTO and one codegen unit. Native runtime feature combinations share `target/native-runtime/cargo`; exact per-build inputs live under `target/native-runtime/inputs`. Do not restore one complete Cargo target per feature combination.

Native ABI release archives use the root `runtime-pack` profile and one shared `target/native-runtime-abi` Cargo target for all four driver combinations. Save each completed archive before the next feature build. Preserve link symbols, keep `embedded` disabled, and distinguish deterministic pack assembly from cross-host source reproduction; the contracts and assertion points live in toolchain-and-library.md.

Tests exercise source contracts and observable AST structure, not private helper implementations. Include the shared `test/dever-tests/fixtures/core/complete_package.dever` fixture, precedence and associativity, Unicode and line positions, lexical failures, excluded syntax, recovery, structural limits and directory loading. Filesystem tests create and remove only their own temporary directories; source loading never follows symbolic links.

Explicit LLVM Worker acceptance consumes the prepared immutable `target/native-release-inputs/component-fixture`, with recorded source and binary digests. Cargo's `target/debug/component-fixture` is a mutable build output: feature unification can change its bytes while running the workspace matrix. Ordinary Worker tests use their Cargo-provided executable; never bypass the CI input-change gate to treat a mutable output as a fixed author input.

Explicit single-libtest acceptance uses `run_command(..., single_harness=True)` and requires the final parent summary to contain exactly one passed test, no failures and no ignored tests. Nested probe summaries must not be added to that parent count; ordinary Cargo multi-harness commands retain aggregate counting. A zero-test parent, failed parent or missing summary is not a pass even if a child probe succeeded. `test/ci/test_gate.py` owns these rejection cases.

The shared `test/native-runtime-abi/managed-driver.c` runs one warmup and 64 repeated entries on joined host threads, then checks strict allocation balance after each join/TLS destruction. Rust's bounded-channel producer may first allocate its 48-byte thread-local Context only after warmup when the queue fills; measuring before the host thread exits misreports that cache as a leak. Preserve process initialization before any thread, original result checks and strict zero allocation delta; do not add retries, random warmups or an allowed-leak threshold. This measures entry-plus-host-thread cleanup, not unbounded TLS growth on a long-lived embedding thread. `test/ci/test_driver.py` compiles the shared driver with the explicit existing author C tool and verifies late TLS cleanup succeeds while intentional per-entry leaks and changed results fail. Suite probes include the same driver instead of copying its ledger.

Markdown source uses pinned pulldown-cmark in the compiler only. Preserve the `.dever` path and existing semantic/contract checks; shared Markdown extraction/formatting rules and the focused `markdown_source` target are specified in [Toolchain And Official Library Contracts](toolchain-and-library.md).

Focused checks for the current milestone:

```bash
cargo fmt --all --check
cargo clippy --offline -p dever-core -p dever-tests --test syntax_and_format -- -D warnings
cargo test --offline -p dever-tests --test syntax_and_format
git diff --check
```

For core language changes, select the affected targets and announce bounded native/resource checks before running:

```bash
cargo test --offline -p dever-tests --test hello_native
cargo test --offline -p dever-tests --test core_semantics
cargo test --offline -p dever-tests --test core_native
cargo test --offline -p dever-tests --test runtime_foundations
cargo test --offline -p dever-tests --test syntax_and_format
cargo clippy --offline -p dever-core -p dever-runtime -p dever-tests --test hello_native -- -D warnings
cargo clippy --offline -p dever-cli --bin dever -- -D warnings
```

The hello target retains stdout, value mutation, package boundaries, missing toolchain and artifact cleanup regressions. Core targets cover the shared complete-package fixture, static rejection, clause permutation, concrete native output, COW ordering, nullable flattening and source-located faults. Foundation-extension cases additionally cover contextual handler syntax, forwarding/specialization recursion, value-only clause domains, List/Bytes/Stream traversal, early stop/resume and one terminal stream failure. Resource tests create only their own temporary files and loopback sockets on port 0 with bounded reads/timeouts. CLI changes also require the actual check/run/build command. These are focused compiler checks, not an integration environment or workspace-wide test run.

Parentheses must not affect context inference: use the shared Group-unwrapping helper for nulls, empty literals and named handlers. Handler signature types must reuse ordinary boundary validation, including generic arity and nested Map-key rules. Stream adapters must preserve `Pull::Last` rather than turning it into an ordinary item. Never retype an existing Decimal/Int local as Float, even after constant propagation. Float zero patterns match both signs, so substituting the pattern constant for the actual input changes division semantics. Preserve source field and Map pair evaluation order in generated code; declaration order controls layout/display only.

Use an explicit installed toolchain or a command-scoped PATH if Cargo is not discoverable. Do not change global shell configuration or replace the existing `dever` command.

Do not run workspace-wide tests, npm builds, service checks, native compilation benchmarks or integration environments by default. Focused syntax tests do not establish type correctness, formatting round trips or runtime equivalence. Record those later-phase checks as not run.

Explicit Linux no-host release checks share `native_release::acceptance::isolated_os_command`: a real mount-namespace root with only declared OS files, proc and synthetic devices. Plain chroot blocks the nested user namespace required by Worker sandboxing. The explicitly built static `native-acceptance-init` replaces bwrap's masked proc with a complete private proc inside the already pivoted outer namespace; otherwise the kernel rejects the nested proc mount. Preserve the outer fixture's original capabilities and the product's inner capability removal; do not substitute a host proc bind. Verify the final standalone executable after removing source and machine stores; an earlier pre-sandbox artifact or managed-only pass is insufficient.

Installed daemon and launcher checks must share that owned PID/user/mount namespace so kernel peer credentials remain visible. Use the owned bwrap PID report, verify its live process and root inode, then enter it explicitly for clients; never relax production peer authentication to accommodate sibling PID namespaces. Pin the namespace init with a pidfd and wait for namespace exit and daemon-lock release before starting the next daemon. Run real distinct-UID checks separately because a single-UID test user namespace cannot represent multiple host identities.

Toolchain/library changes select `formatter`, `standard_library`, `native_ownership`, `concurrency`, `json_library` or `http_library`. The optional `reference` feature enables only `differential`; production CLI feature resolution must exclude it. HTTP tests independently assert wire bytes and malformed framing, with bounded loopback peers. Formatter changes also exercise the actual `fmt`/`--check` CLI and failure-before-rewrite behavior.

Configured concurrency and reusable TLS clients select `async_composition`, `pooled_network` and `pooled_library`. Assert task cleanup even before first poll, complete-body connection reuse versus abandoned-body close, bounded physical connections including idle entries, stream deadlines, verified CA/name failures, WSS and graceful shutdown. Check real native source conversion alongside runtime peers; include compile rejection for mismatched context/output signatures and unhandled stream failures. Certificates and test private keys belong under root `test/`; peers use only owned loopback ports.

Runtime root entries pin the caller Future once before forwarding it through cleanup adapters and task-local wrappers. Job execution scopes also pin before constructing their async wrapper; identity initialization and task-local entry remain lazy. Preserve borrowed and non-Send roots, concrete errors, component-close/child-drain/cleanup order, and nested-entry rejection. Exercise `runtime_entry_cleanup` and all-feature `durable_jobs`, including signal/transaction shutdown: default-feature success alone misses the larger combined-driver Future layout. Do not increase thread stacks to mask wrapper-induced stack overflow.

Standalone C fixture drivers must acquire the existing process initialization before starting kernel/runtime threads, just like generated process main. The shared `managed-driver.c` also owns initialization for the indexed suite driver. Callable roots deliberately preserve an embedding host's signal policy; do not modify runtime network leaves or test subprocess inheritance to hide a missing driver initialization. Keep the direct SIGPIPE ABI assertion and real TLS upload/download allocation-ledger regression.

The ignored `performance` target runs only for an explicitly announced bounded performance check. Match Decimal initial representation as well as numeric value; prepare parsing outside the timed region. Five native computation kernels cover Int/Float/List/Decimal/Map with exact results and a 1.15 baseline ratio budget. Alternate a fixed 25 independent processes per side, discard each process's warmup, retain every process's nine raw samples, then compare the medians of all 225 samples per side. The same ELF has exhibited clustered timing modes even with CPU affinity; five process groups can accidentally favor one side. More process groups improve coverage without proving host noise is eliminated. Never selectively omit groups or add samples until a failing ratio passes. Production and baseline share `native::OPTIMIZATION_ARGS`, including one codegen unit so generic collection inlining does not depend on generated program size. Record compilation separately and report CPU/environment without inferring HTTP throughput or whole-language performance.

## Standalone Runtime Benchmarks

### 1. Scope / Trigger

Changes to root `test/performance/` and the `network_bench`, `async_bench`, `live_bench` Rust examples. These explicit opt-in benchmarks measure owned binaries/loopback peers, never existing services or default test execution. Operational commands and limitations live in `test/performance/README.md`.

### 2. Signatures

`python3 test/performance/run.py build --output <new-dir>` compiles fixtures and records binary fingerprints/configuration. `--selection protocols` selects runtime/HTTP/live and three explicitly prepared Rust peers; `profiles` selects four runtime profiles; `cms-profiles` also builds both CMS sources. Every Dever fixture uses the public LLVM project build and CMD dispatch, never the removed native Rust fixture builder. `run --artifacts <dir> --output <new-dir> --suite memory|async|http|live|http2|http2-recovery|all` executes only prepared binaries. The HTTP peer CLI is `network_bench load h1|h2 URL PATH RATE SECONDS PHYSICAL_CONNECTIONS STREAMS_PER_CONNECTION TIMEOUT_MS REUSE CA`; H1 requires one stream per connection, while H2 requires connection reuse. Service modes are runtime-http/runtime-https/hyper-http/hyper-https plus runtime-http2/runtime-https2/hyper-http2/hyper-https2. Async comparison uses `async_bench root|worker WORKERS TASK_CAPACITY ITERATIONS`. Long connections use `live_bench PROTOCOL URL CONNECTIONS CYCLES HOLD_MS TIMEOUT_MS`, selected by `--connection-tiers`, `--cycles` and `--duration` in the runner.

For `run`, `--server-cpu <id>` optionally binds the server to one CPU; `--client-cpus <id,id,...>` optionally binds HTTP/1, HTTP/2 and live clients to a CPU set, and `--client-workers <positive-int>` defaults to 1. `Process(cpus=...)` changes only its owned child before exec. An omitted client set inherits the runner's allowed CPUs. The report records `arguments.client_cpus`, `arguments.client_workers` and the effective `environment.client_cpus`/`client_workers`.

### 3. Contracts

Servers publish `READY|port` after preparing shared payloads. JSON counters satisfy `scheduled=sent+dropped`, `sent=completed=succeeded+errors`, and `dropped=dropped_capacity+dispatch_expired`. `latency_ms` measures planned-to-successful-completion; `request_latency_ms` measures dispatched-to-successful-completion; dispatch lag includes completed failures. Empty histograms are null and success histograms have exactly succeeded samples. Record raw rounds, successful throughput, load-generator CPU/lag, config and limits; never average p99 values into a global p99. HTTP/1, HTTP/2 and live clients share explicit `--client-workers`/`--client-cpus` configuration; record both in the report. Python stages each peer beside an owned config/setting.json with a benchmark object; peers read it relative to their executable and never use application environment variables. The manifest records setting-json-v1 and build checks the peer's check-config output before accepting it. Server affinity remains one CPU, distinct from the specified client CPU set.

HTTP/2 records `physical_connections`, `streams_per_connection`, their `request_slots` product and `successful_handshakes`. Preconnect the exact physical connection count before timing, share cloned senders across stream slots, and do not retry, replace or expand connections during measurement. H2 cleartext is prior knowledge; TLS advertises and requires only ALPN `h2`. Runtime and raw Hyper baselines use the same stream/window/header/frame/send-buffer limits. After each warmup, case or recovery cycle, close clients, wait the fixed settle interval and require service FDs to return to the READY baseline; preserve the RSS sequence without treating allocator retention alone as a leak. Any HTTP/2 request error or cgroup OOM event fails the run after completed case data is persisted; capacity drops remain saturation evidence rather than a pass/fail threshold.

Manifest `application_configuration=llvm-cmd-v1` is mandatory alongside `setting-json-v1`; reject older application artifacts before launch. Each runtime/http/live project contains its group's CMD declarations, so per-command aliases share the same multi-command binary and recorded identity. `fixture_command` derives the qualified CMD from the same `ENTRIES` used for building. Live settings belong beside each owned executable; ORM keeps its separate `benchmark.command.run` entry. LLVM O2/LLD and the exact prebuilt runtime archives determine Dever output. Rust peers are separately prepared with Cargo release, copied and fingerprinted. Do not claim identical optimizers or scheduling: Dever uses production defaults, while explicit workers/capacity affect Rust peers. Both retain checked Int arithmetic and structured run/wait; root/worker modes differ by an untimed outer structured task. Sampled /proc context-switch totals count currently alive threads and can decrease as threads exit; use perf for complete process/thread totals.

Live clients flush zero-based `PHASE|cycle|connecting/connected/closing/disconnected` in order. Connected holds perform validated interactions; disconnected holds last 300 ms before the next connecting marker. Compare progress before and after sampling; discard ambiguous transition observations from phase reports. Success requires attempted=connected=disconnected=connections*cycles, errors=0 and at least one validated message per connection (SSE also requires a heartbeat during hold). Counters alone do not prove server cleanup: preserve per-phase FD/RSS and their post-disconnect deltas. TCP/WS/SSE payload validation, finite limits and driver ownership apply on errors as well as success.

CMS concurrent reads close the control client before the baseline. `Process.descriptors()` records FD-to-target maps; `validate_read_descriptor_recovery(load)` is shared by the runner and CI. Its input includes `descriptors_before`, `descriptors_after`, and fixed `pooled_descriptor_limits`: each existing SQLite database/main WAL target is bounded by configured `max_connections`, SHM by one. All remaining descriptor targets and multiplicities must match (`Counter(non_pooled_before) == Counter(non_pooled_after)`). Preserve total FD change as observation. Persist each case's `report.json`, including partial completed data, raw samples, final budget and errors; aggregate failures as well as successes.

### 4. Validation & Error Matrix

Wrong status/type/body -> counted error, no successful latency sample. Exhausted load capacity -> dropped_capacity. Missed measurement window -> dispatch_expired; one late timer wake is not itself a drop. Incomplete samples/checksum mismatch -> runner failure. Missing cgroup permission/controller -> explicit failure and owned-group cleanup, never unlimited fallback. Interrupted run -> terminate/reap owned processes and remove owned empty groups, preserving completed case records.

An empty, duplicate, negative or non-integer `--client-cpus` set fails at argument parsing; an unavailable CPU or overlap with the specified server CPU fails before any output directory or process is created. A nonpositive `--client-workers` value fails at argument parsing. Do not claim independent client/server cores when affinity was omitted.

CMS extra/replaced/duplicated sockets, unexpected files or SQLite handles over the configured bounds fail even if total FD count is unchanged. Record completed publication fields and baseline/limits before starting the read load, so an early read or sampling error cannot erase existing evidence.

Process memory may disappear before waitpid reports exit: skip that unavailable observation and let the process exit check own failure reporting. Every owned server context rejects cgroup OOM after teardown without masking an earlier failure. HTTP warmup and measurement require at least one success and zero request errors; capacity drops remain explicit measurements. Live missing/out-of-order phases or incomplete connection cleanup fail the run. A valid partial SSE frame at the intentional measurement deadline is not a malformed peer frame; never require TCP chunk boundaries to coincide with events.

### 5. Good / Base / Bad Cases

Base: single-worker runtime, 32 server connections, 16 client slots. Good: each implementation prepares and shares the 64 KiB body once, validates exact response bytes and waits the entire measurement window. Bad: only one side reallocates the body per request, failed responses improve p99, or sub-window completion inflates QPS.

CPU placement good: `--server-cpu 0 --client-cpus 1,2,3,4 --client-workers 4` on a host allowing CPUs 0–4. Base: omit client flags for one worker inheriting allowed CPUs, with no isolation claim. Bad: repeat CPU 1 or place CPU 0 in the client set, or infer multi-core use solely from a requested worker count.

CMS FD base: one main/WAL/SHM handle per database. Good: a configured four-connection pool expands to four main and WAL handles, one SHM, and all non-database descriptors recover. Bad: a fifth database handle, an extra socket or a replacement socket masked by a closed control connection.

### 6. Required Tests

Run `python3 -m unittest discover -s test/performance -p test_runner.py -v` for counting/warmup/phase ordering/process teardown, HTTP/2 shape/summary validation, isolated JSON staging and missing-controller cleanup. Assert staged public projects declare the CMD used by every runner call and reject an obsolete application manifest. Configure test binaries in root config/setting.json at performance.network_peer and performance.live_peer. `python3 test/performance/test_network_bench.py -v` uses independent owned Python H1 peers and owned h2c/TLS services for exact-body success, one-connection multi-stream behavior and deliberate wrong-body failure. `python3 test/performance/test_live_bench.py -v` verifies TCP reconnect and rejects wrong TCP/SSE bodies or an SSE hold without heartbeats. Both also provide a no-network check-config test. Announce owned loopback checks before running them; select bounded protocol validation after driver changes, never the full suite by default. Network examples build offline with the workspace release profile; Dever archive/source provenance is recorded separately from Rust reference builds.

After changing CPU placement, assert the child reports the requested affinity without changing the parent, reject empty/duplicate/out-of-range/overlapping IDs, and verify serialized report workers/CPU sets. Run a bounded h2c/TLS+h2 smoke and minimal HTTP/1/live smoke against fingerprinted artifacts. To establish that the client actually exceeded one core, inspect `load.client_resources.cpu_cores_sampled` in high-rate raw cases, alongside service CPU and slot/dispatch drops.

CMS recovery changes require `test_runner.py` and `test/ci/test_gate.py`: bounded pool growth passes, socket replacement/duplication and excessive pool handles fail, and read/recovery failures preserve case and aggregate evidence after cleanup. A real read check longer than the default 30-second HTTP idle timeout exercises the closed-control baseline; it does not replace the 1800-second soak.

### 7. Wrong vs Correct

Wrong: call warmed cgroup current the application's cold-start RAM; subtract two p99 values to infer request latency; equate a completed report with zero errors/OOM. Correct: report RSS/PSS alongside charged cgroup current/peak and OOM/throttling counters, distinguish warmed file pages and pre-exec Python peaks, record request latency directly, and inspect all failure counters before drawing conclusions. No global cache drops. Raw Hyper is a normal-GET baseline, not an equivalent implementation of production slow-peer/fault/structured-cleanup semantics.

Wrong: require CMS total FD delta zero across idle control-client expiry and lazy SQLite pool expansion. Correct: close that client first, preserve exact non-pooled descriptor identity/count and independently enforce the database pool's descriptor bounds.

Wrong: equate a 5-second saturation platform or a complete report with a sustainable 60-second rate. Correct: pick a rate below the observed platform, require at least 99% planned-request success over 60 seconds in both transport modes and memory budgets, and preserve nonpassing calibration data separately.

For the 2026-09-18 HTTP/2 ceiling check, `--client-workers 4 --client-cpus 1,2,3,4` removed the former single-core client constraint while the Dever server stayed on CPU 0. A 16x16 `/plain` scan at 30k/45k/60k/90k observed roughly 36k–40k successful req/s at the upper rates, with the service near one CPU and request-slot drops dominant; this is not an isolated server-only ceiling. Short scans cannot select a reliable steady rate by themselves: 28k failed the 99% planned-request success gate over 60 seconds, while 20k passed for both h2c and TLS+h2 at 128 and 64 MiB. `test/performance/README.md` owns exact commands, counts, RSS/latency, failure and calibration reports, and the limits of the 100-cycle recovery evidence. Do not extrapolate loopback or warm-cache results to production.