feat: publish Dever core with project scaffolding and versioned AI skills
This commit is contained in:
Antigravity committed
1 parent
70fa73cdc2
commit
ac458ffa7f
960 files changed
+221319
-1163
No files matched your search
@@ -0,0 +1,349 @@
|
||||
# 独立二进制性能基准
|
||||
|
||||
这里测常驻内存、异步调度、HTTP/1.1、HTTP/2 和 TCP/WebSocket/SSE 长连接；不改运行时。构建单独执行，运行阶段只启动已编译产物。全部程序绑定自己创建的 loopback 随机端口，不接触已有服务，不进入默认 `cargo test`。
|
||||
|
||||
## 构建与短时验证
|
||||
|
||||
在仓库根目录运行，使用已经安装的 Rust 工具链与锁定的离线依赖：
|
||||
|
||||
```bash
|
||||
cargo build --offline -p dever-cli --bin dever
|
||||
cargo build --offline --locked --release --target-dir target/native-runtime -p dever-tests --example network_bench --example live_bench --example async_bench
|
||||
python3 test/performance/run.py build --output target/performance/build-current --workers 1 --connections 512
|
||||
python3 -m unittest discover -s test/performance -p test_runner.py -v
|
||||
python3 test/performance/test_peer_settings.py -v
|
||||
python3 test/performance/test_network_bench.py -v
|
||||
python3 test/performance/test_live_bench.py -v
|
||||
python3 test/performance/run.py run --artifacts target/performance/build-current --output target/performance/smoke-current --suite all --duration 0.5 --warmup 0.2 --rates 100 --paths /plain --repeats 1
|
||||
```
|
||||
|
||||
只复验当前 Dever/Markdown 双源码 CMS 和 base/SQLite/PostgreSQL/both 四个 profile 时，可使用
|
||||
`python3 test/performance/run.py build --selection cms-profiles --output target/performance/cms-build-current`。
|
||||
该选择不构建无关的 HTTP、长连接、ORM 场景和 peer；生成的产物仍可用
|
||||
`run --suite cms` 分别测量 `cms`（Dever）和 `cms-md`（Markdown）的真实登录、文章创建/替换、发布和前台查询；旧 `READY|3` 计数 fixture 已删除。仅做四 profile 构建可选 `--selection profiles`。PostgreSQL profile 使用不连接数据库的编译专用声明，不需要真实 URL；真实 PostgreSQL ORM/压力测试仍必须显式配置根 `config/setting.json` 的 `database.postgres_test`。
|
||||
|
||||
仅复验 runtime/HTTP/TLS/HTTP2/TCP/WebSocket/SSE 时使用 `--selection protocols`，不构建 ORM/CMS 或重复四 profile；三种 Rust peer 可用 `--network-bench`、`--live-bench`、`--async-bench` 指定明确准备的路径。
|
||||
|
||||
输出目录必须是新路径，保留先前报告。当前 Dever 产物统一经公开 `dever build` 使用 LLVM O2、LLD 和明确准备的 runtime archive；构建日志及二进制 SHA-256、大小、耗时保存在 `manifest.json`。runtime/http/live 每组构建一个包含该组 CMD 的应用，各测量通过不同命令启动，大小不能解读为仅一个函数的程序。`profile-report.json` 仍使用同一份最小 CMD 源码生成 base、SQLite、PostgreSQL 和 both 产物，记录 bridge 的锁定依赖。作者必须准备匹配源码的四个 archive；编译器自身的 debug/release 不决定 archive 优化级别。Rust 对照程序由上面的 Cargo release 命令另行准备并记录二进制摘要。Dever 使用生产默认调度，`--workers` 和 `task_capacity` 只配置 Rust 对照，不声称两者调度配置相同；可用专属 CPU affinity 统一核数。运行器要求 `application_configuration=llvm-cmd-v1`，拒绝旧 Rust 后端产物冒充当前 LLVM 测量。测量不调用 Cargo。
|
||||
|
||||
双源码 CMS 也有独立入口，不构建网络 peer 和无关场景：
|
||||
|
||||
```text
|
||||
python3 -B test/performance/cms.py build --output target/performance/cms-publish-build
|
||||
python3 -B test/performance/cms.py run --manifest target/performance/cms-publish-build/manifest.json --output target/performance/cms-publish-128 --cgroup-parent /sys/fs/cgroup --memory-mib 128
|
||||
python3 -B test/performance/cms.py run --manifest target/performance/cms-publish-build/manifest.json --output target/performance/cms-publish-64 --cgroup-parent /sys/fs/cgroup --memory-mib 64
|
||||
python3 -B -m unittest discover -s test/performance -p test_cms.py -v
|
||||
```
|
||||
|
||||
运行使用仅含二进制和私有 `config/setting.json` 的副本，不保留 module。先在自有 SQLite 平台库 bootstrap、迁移租户库并给 admin/front 分别建立 Owner，再启动随机 loopback 端口。服务启动与登录/发布/查询在专属 cgroup 内，bootstrap 和部署命令不在该预算内。保留 Cookie/Origin、真实身份和租户验证，不伪造登录身份。逐篇创建、替换、发布并核对全部前台结果；重复发布必须返回 409，两套源码结果必须一致。默认 16 篇顺序业务基线报告启动时间、每种请求样本数/中位/最大延迟、RSS/PSS 与 cgroup peak/OOM，不称为并发饱和吞吐或 p99。测试手动传送服务返回的 Secure Cookie，只验证 owned HTTP 协议，不代表浏览器 HTTPS 验收。日志阈值固定 error，出现服务错误、超时、OOM 或清理失败均失败。
|
||||
|
||||
追加 `--concurrency 8 --read-seconds 10` 会在发布后运行八个独立连接的已登录前台列表查询，每次都核对完整业务结果。报告另列 `concurrent_reads` 的实际成功 QPS、p50/p95/p99、测量期资源趋势和断连 FD 回落；这是闭环读负载，不能解释为写入吞吐或预设请求到达率。并发数最多64、时长最多3600秒，同一服务进程内支持30分钟持续读取；客户端使用固定2048桶的对数延迟直方图，分位数报告包含请求及结果核对的桶上界，避免随请求数保存无限样本。首次失败停止其他客户端，所有连接按原请求期限关闭；不记录 Cookie 或密码。完整Linux门及64/128MiB长测入口见 [test/ci/README.md](../ci/README.md)。
|
||||
|
||||
真实 PostgreSQL 的 Owner HTTP/双物理租户验收另由共享 Rust fixture 管理数据库生命周期：
|
||||
|
||||
```text
|
||||
cargo test --offline --locked -p dever-tests --features api,postgres --test postgres_api postgres_cms_http_isolates_two_tenant_databases -- --ignored --exact --nocapture
|
||||
```
|
||||
|
||||
根 `database.postgres_test` 必须指向专属测试实例，URL 数据库名包含 `{case}`，账号允许创建/删除测试库；默认不运行、不探测外部服务。fixture 使用随机控制库及租户前缀，只清理本轮拥有的库，失败也清理，清理失败不能记录 passed。`cms.py postgres-case` 始终先复制传入项目到自有临时目录，端口/密钥/配置/产物仅写副本，输出目录保留日志和不含凭据的结果。Rust 目标额外注入仅在 `test/` 的普通成员 CMD，并显式启用 `--rbac-fixture`：覆盖 Owner、两个租户各两篇发布、相同 slug 独立数据、跨站 Cookie/跨租户登录拒绝，以及非 Owner 多角色、跨站同名角色和原会话撤销立即失效。单独调用不启用该测试 fixture 的工具仍仅证明 Owner 合同。权限键分别读取实际站点目录，GET/DELETE 参数统一走 query，生产 CMS 和 Owner 保留角色规则不改。
|
||||
|
||||
runtime/HTTP/live 夹具已使用当前 App→Port→Adapter 结构，CMD 根有真实 Bool 输出；live 的 typed Adapter setting 在每个产物的私有配置中独立绑定。ORM seed 是测量前的 CMD 部署步骤，count/cancel 是普通 API，取消测试使用单连接池并核对计数和连接归还，不能把 handler 超时冒充客户端取消。
|
||||
|
||||
2026-10-02 LLVM 基线使用 `protocols-llvm-10-02`、`cms-llvm-10-02` 及 `profiles-llvm-10-02` 的准确产物摘要，私有作者编译器来源记录在 `target/closure-current-compiler/provenance.json`。四个最小 CMD profile 大小分别为 base 2,083,880、SQLite 4,139,976、PostgreSQL 2,265,224、both 4,238,568 字节。Dever/Markdown CMS 分别 8,323,880 / 8,326,120 字节。
|
||||
|
||||
| 当前测量 | 条件 | 结果与报告目录（均位于 target/performance） |
|
||||
| --- | --- | --- |
|
||||
| 双 CMS 发布 | 64/128 MiB、每组顺序16篇 | 零错误/OOM，启动44.6–51.9ms，RSS峰值28.20–28.33MiB；`cms-llvm-{64,128}-10-02` |
|
||||
| 双 CMS 登录后列表读取 | 64/128 MiB、8并发、10秒、不绑核 | 1210.6–1322.2 QPS，p99 11.48–13.59ms，RSS峰值30.81–31.04MiB，零错误/OOM；`cms-read-llvm-{64,128}-10-02` |
|
||||
| HTTP/2 明文及 TLS 稳态 | 64/128 MiB、单核、4连接×16流、目标10k、60秒 | 成功/计划99.718–99.794%，9971.6–9979.2 QPS，p99 2.912–3.424ms，RSS峰值4.88–5.72MiB，零错误/OOM/断开后FD差；`http2-steady-llvm-{64,128}-10-02` |
|
||||
| HTTP/2 连接恢复 | 64 MiB、单核、两种传输各100轮 | 合计19,887个完整成功请求，200次断开后FD差均0，零错误/OOM；`http2-recovery-llvm-64-10-02-v2` |
|
||||
| TCP/WS/SSE 连接恢复 | 64/128 MiB、单核、32/128连接、各10轮 | 12个case、120次断开后FD差均0，完整消息校验通过，零错误/OOM；`live-llvm-{64,128}-10-02` |
|
||||
|
||||
这些是本机 loopback 与热文件缓存的有限测量。CMS使用闭环客户端，HTTP/2使用固定速率且保留少量未派发计数，两者不能直接比较。15k TLS短时校准未达到99%计划成功门，所以当前一分钟基线选择10k，不把下文历史20k/饱和结果当作本次结果。首次恢复因运行器重复复制peer耗尽磁盘的失败目录保留，最终恢复证据仅用v2。历史 `cms-publish-*-09-30-*`、`orm-runtime-64-09-30-v1` 与 `live-tcp-64-09-30-v1` 不代表本次重跑。
|
||||
|
||||
共享编译的本机基线记录于 `target/closure-shared-compile-measurement.json`：私有debug核心、热OS缓存下，首次请求7.318秒、命中5.564秒；两个同步客户端提交同一份新源码分别7.463/7.838秒且产物一致。时间包含签名输入复核及IPC，不能当纯编译耗时或正式优化版CLI成绩。每个产物4,239,368字节，两条缓存合计8,478,736字节，校验无损坏；跨真实UID共享另有独立定向验收。
|
||||
|
||||
产物目录含二进制、测试证书、构建配置，可整体复制到兼容的 Linux 机器；运行器仅依赖 Python 3.11+ 标准库和 `/proc`。TLS 证书来自 `test/dever-tests/fixtures/tls/`，仅用于测试，执行时校验证书链和 localhost 名称。测试证书过期后需更新 fixture，不得关闭验证。
|
||||
|
||||
两个独立 peer 测试从仓库 `config/setting.json` 的 `performance` 读取 `network_peer`、`live_peer`，例如：
|
||||
|
||||
```json
|
||||
{"performance":{"network_peer":"target/performance/build-current/network_bench","live_peer":"target/performance/build-current/live_bench"}}
|
||||
```
|
||||
|
||||
把该 `performance` 对象合并进已有设置，保留原有数据库配置。基准进程配置不读取环境变量。运行器在同文件系统硬链接已验证的只读 peer 二进制，仅跨文件系统时复制；每轮仍在其旁边以 `0600` 写入独立 `config/setting.json` 的 `benchmark` 对象。测量期间不得修改输入二进制。字段为 `workers`、`connections`、`task_capacity`、`timeout_ms`、`lifetime_ms`、HTTP/2 参数和 TLS 证书路径。CLI 负载参数仍显式传递。缺少配置、非法类型或越界值会报错；构建器通过不启动网络的 `check-config` 检查 peer 的 JSON 配置协议。旧协议的 manifest 不再接受，应重新构建产物，旧报告仍保留历史意义。
|
||||
|
||||
## 场景与统计口径
|
||||
|
||||
| 组 | 场景 | 计时/验证 |
|
||||
| --- | --- | --- |
|
||||
| memory | 同步空闲、异步空闲、256 个等待 Channel 的任务、HTTP/HTTPS 监听 | READY 后采样 RSS/PSS、VmHWM、线程/FD；parked 用取消场景的等待阶段，不另建实现 |
|
||||
| async | 16384 次 run/wait、16384 次有界 Channel 发送与消费、取消并排空 256 个已启动任务 | 源码内单调时钟；计算结果/数量必须正确，打印在计时外；前两项每进程 10 轮、去掉首轮，取消保留完整排空耗时 |
|
||||
| http | HTTP/HTTPS × Dever/Rust-runtime/原生 Hyper × 三种响应 | 固定速率、有限并发；状态码、Content-Type 和完整正文必须匹配；连接复用与新建连接可分别运行 |
|
||||
| http2 | h2c prior knowledge/TLS ALPN × Dever/Rust-runtime/原生 Hyper × 物理连接/流拓扑 | 计时前建立固定物理连接；校验握手数、状态码、Content-Type、完整正文、吞吐、延迟和资源 |
|
||||
| http2-recovery | 同一服务进程内重复建立、使用、关闭 HTTP/2 连接 | 每轮校验请求总账和握手数，断开后 FD 必须回到 READY 基线，保留 RSS 序列 |
|
||||
| live | TCP/WebSocket/SSE × 连接档位 × 多轮连接/保持/断开 | TCP/WS 校验带轮次和连接标识的回显；SSE 校验初始事件与心跳；对比各轮断开后的 FD/RSS |
|
||||
| orm | SQLite/PostgreSQL 的 idle、CRUD、list/cursor/stream、pool saturation、transaction cancel、HTTP+JSON+DB | 每个 case 使用独立 bundle 和数据库；校验精确结果，再报告数据库操作延迟、吞吐、资源和连接回收 |
|
||||
| cms | `examples/cms/dever` 和 `examples/cms/md` 的 Model、Seed、用户注册和发布事务 | 相同入口、单独 SQLite bundle，业务初始化与事务完成后分别采样 RSS/PSS/VmHWM 和可用时的 cgroup peak |
|
||||
|
||||
## ORM 基准
|
||||
|
||||
所有 Dever 产物在 `build` 阶段通过公开 `dever` 项目路径生成；无数据库声明的 runtime/http/live 使用 base archive。`run` 阶段只复制和执行已有二进制。每个数据库 case 都创建独立 bundle；SQLite 使用该 bundle 自己的 `data/db/orm.db`，重复轮次不会复用数据库状态。
|
||||
|
||||
构建器把测试源码暂存到 `module/benchmark/<domain>/app.dever`，共享能力位于 `benchmark/bench`、`benchmark/config`，由 `api.dever` 声明 CMD。fixture 不生成 main/package/exposes，也不向 `dever build` 传任意 entry。
|
||||
|
||||
CMS 从原始双源码项目构建，通过真实登录/租户/发布 HTTP 流程测量，不注入新的业务入口。普通 `build` 只构建 Dever CMS，`cms-profiles` 构建双源码。下方历史 `READY|3` 报告不代表当前实现。
|
||||
|
||||
2026-09-18 的当前源码专项报告保存在 `target/performance/cms-dual-build-09-18-v2/{manifest.json,profile-report.json}` 和 `target/performance/cms-dual-rss-09-18/report.json`。base/SQLite/PostgreSQL/both 原生程序分别为 411,352 / 2,558,200 / 2,708,112 / 4,672,888 字节。CMS Dever/Markdown 产物分别为 3,151,352 / 3,151,416 字节；各运行 3 次，READY 全部为 3，完成时间中位数分别为 19.87 / 19.69 ms，空闲 RSS 中位数同为 5.37 MiB，PSS 分别为 3.28 / 3.33 MiB，VmHWM 中位数同为 5.37 MiB。采用本机热缓存、独立 SQLite bundle、每次 READY 后 2 秒采样；无吞吐负载，不能把启动时间或空闲内存解释为业务吞吐/峰值容量。当前 cgroup 父目录不可写，未施加 64/128 MiB 硬限额，也未记录 cgroup peak。
|
||||
|
||||
默认只构建和运行 SQLite。真实 PostgreSQL 统一读取仓库根 `config/setting.json` 的 `database.postgres_test`，不接受环境变量或命令行连接覆盖；该本地文件已被 Git 忽略。其 `url` 必须在数据库名中包含唯一 `{case}`，例如 `postgres://bench:secret@127.0.0.1/dever_{case}`，查询参数或用户名中的占位符不算隔离。运行器按“输出目录短哈希 + driver + entry + repeat”展开 case，因此不同报告不会复用数据库；构建器和运行器以 create-new 和 `0600` 权限原子写入 bundle 的 `config/setting.json`，构建结束会删除含连接信息的临时配置，URL 不写入 manifest 或 report。运行器不会创建或清理 PostgreSQL 数据库，因此模板必须指向预先准备的专用测试数据库，不能指向已有业务库。
|
||||
|
||||
```json
|
||||
{
|
||||
"database": {
|
||||
"default": {
|
||||
"type": "sqlite",
|
||||
"path": "data/db/test.db"
|
||||
},
|
||||
"postgres_test": {
|
||||
"type": "postgres",
|
||||
"url": "postgres://bench:secret@127.0.0.1/dever_{case}",
|
||||
"tls": "disabled",
|
||||
"min_connections": 0,
|
||||
"max_connections": 4,
|
||||
"max_page_size": 100,
|
||||
"wait_timeout_ms": 5000,
|
||||
"io_timeout_ms": 30000
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
python3 test/performance/run.py run \
|
||||
--artifacts target/performance/build-current \
|
||||
--output target/performance/orm-smoke \
|
||||
--suite orm --duration 1 --warmup 0.2 --repeats 1
|
||||
|
||||
python3 test/performance/run.py run \
|
||||
--artifacts target/performance/build-current \
|
||||
--output target/performance/orm-postgres-smoke \
|
||||
--suite orm --duration 1 --warmup 0.2 --repeats 1
|
||||
```
|
||||
|
||||
ORM 报告包含 binary size、RSS/PSS/VmHWM、cgroup current/peak/OOM、线程/FD、吞吐和 p50/p95/p99。当前生产 runtime 没有低成本的 pool wait、SQL 数和数据库连接数观测点，报告将这些字段明确标为 `unavailable`，同时记录配置的 pool capacity 和进程 FD，不用推算值冒充真实指标。transaction cancel 先验证单连接 pool 可用，再由长事务占满 pool、在响应完成前断开客户端，并要求连接在 HTTP handler timeout 前归还、计数仍为 0；这样不会把 handler timeout 回滚误报为客户端取消，也不会绕过语言对 transaction 内并发任务的静态限制。
|
||||
|
||||
run/wait 同时跑 Dever 与 Rust-runtime 的 root/worker 两种位置。root 在 `run_entry_with` 入口执行；worker 在一个计时外创建的结构化外层任务内执行。二者内部都用原有 `task::run/wait`、Bytes 输入和 checked Int 加法，保留子任务额度、取消和清理；没有以裸 Tokio task 代替语言任务。这个实验不修改默认调度。Channel/取消暂时仍是 Dever 单方基线。`/proc` 上下文切换计数汇总采样时仍存活的线程，线程退出会使累计值减少；总量分析用独立 `perf stat`，不能将采样差值当作精确任务计数。
|
||||
|
||||
live 用 `--connection-tiers 32,128,512 --cycles 3 --duration 2` 配置，每个档位独立启动服务，每轮保持约 2 秒，再关闭全部客户端并等待 300 ms。Rust 对端发布零基轮次的 `connecting/connected/closing/disconnected` 四阶段；Python 只用 connected/disconnected 保持窗口比较资源，记录最后一次断开样本和相对基线的 FD/RSS 差值。采样前后阶段变化的过渡样本不归属任何阶段，也不写入阶段报告。正文不匹配、阶段缺失、SSE 保持期无心跳或未回收全部客户端都失败；RSS 保留可能来自分配器缓存，不能单凭不回落断言泄漏。默认只测 32 条，档位不能超过构建时的 connections；`--duration` 为 0.3–60 秒，采样间隔最多 100 ms。live 的 SSE/WS 是明文 HTTP/1 场景，不包含 WSS 或 HTTP/2。
|
||||
|
||||
HTTP/2 用 `物理连接数 x 每连接并发流数` 描述拓扑。当前固定每连接 16 流、单流接收窗口 65535 字节、连接接收窗口 262144 字节、frame/send-buffer 上限 16384 字节。客户端在计时前完成精确数量的 h2c 或 TLS+h2 握手，每个连接的 `SendRequest` clone 由其 16 个请求槽共享；测量期间不重试、不替换连接、不自动扩容。`request_slots` 必须等于两个维度的乘积，`successful_handshakes` 必须等于物理连接数。断开后等待 300 ms，再以 READY 基线核对 FD。
|
||||
|
||||
三个路径为 `/plain`（`hello`）、`/json`（预制 `{"ok":true}`）、`/bytes`（65536 个 `x`）。JSON 路径测 JSON 响应传输，不代表动态 JSON 解析/序列化性能。`/bytes` 跨越 HTTP/2 默认单流接收窗口，用于验证流控下的大正文传输。大正文在服务启动时准备，各服务均无业务日志。
|
||||
|
||||
网络记录 `scheduled/sent/completed/succeeded/errors/dropped`；`dropped_capacity` 表示无可用请求槽，`dispatch_expired` 表示负载发生器未能在窗口内发出。`latency_ms` 从**计划发送时间**算到成功完成，包含排队/调度；`request_latency_ms` 单独测实际发出到成功完成，`dispatch_lag_ms` 记录调度延迟。不能用两个 p99 相减估算请求耗时。没有成功样本时成功延迟为 null，失败不会稀释成功分位数。Tokio 定时器会让高频请求成批到期，这个发生器用于可复现的有界负载和相对比较，不适合独立证明微秒级服务延迟或极限吞吐。检查负载端 CPU、lag、丢弃，再解释服务结果。
|
||||
|
||||
延迟使用有界直方图：1 ms 内按微秒分桶，之后每个二倍区间分 64 桶，分位数取桶上界。每轮保留原始计数和资源采样，不把多个 p99 再平均成整体 p99。异步的 `p95_ns_per_operation_sample` 是批次平均耗时的分位数，**不是逐任务 p95**。`ready_seconds` 是父进程启动到观察 READY 的耗时，包含创建进程和最多约 5 ms 的观察误差。
|
||||
|
||||
## 小机器运行预算
|
||||
|
||||
建议先测 1 核/128 MiB，再试 64 MiB 和 2 核/256 MiB。这里是应用预算，不包含操作系统自身内存。`--server-cpu` 固定服务端 CPU，`--client-cpus` 指定逗号分隔的客户端 CPU 集合；`--client-workers` 指定 HTTP/1、HTTP/2 和 live 负载的 worker 数。客户端 CPU 不能包含服务端 CPU；不指定客户端集合时继承当前允许的 CPU。`--cpu-quota` 是计算份额上限，不能代替绑核。
|
||||
|
||||
```bash
|
||||
python3 test/performance/run.py run \
|
||||
--artifacts target/performance/build-current \
|
||||
--output target/performance/budget-128m-1 \
|
||||
--suite all --duration 10 --warmup 2 --repeats 3 \
|
||||
--rates 100,1000 --paths /plain,/json,/bytes \
|
||||
--concurrency 16 --connection-mode both \
|
||||
--http2-connections 1,4 --recovery-connections 1 \
|
||||
--connection-tiers 32,128,512 --cycles 3 \
|
||||
--server-cpu 0 --client-cpus 1 --client-workers 1 \
|
||||
--cgroup-parent /sys/fs/cgroup/system.slice \
|
||||
--memory-mib 128 --cpu-quota 1
|
||||
```
|
||||
|
||||
上例 cgroup 路径只适用于有相应权限且父组已启用 cpu/memory 控制器的机器。工具为每个被测进程创建新的 `dever-perf-*` 子组，仅把自己的子进程放进去；不会修改父组、其他服务或全局 swap。指定内存限额时把该子组 swap 限额设为零。没有权限会明确失败，不回退成“无限额通过”。负载端和 Python 采样进程留在预算之外。
|
||||
|
||||
`memory.current/peak` 包含该组被计费的用户内存、内核/网络资源等；不能与进程 RSS 混用。报告同时保留两者、OOM 事件和 CPU 节流统计。进程由 Python 子进程 exec 启动，cgroup 峰值也可能包含 exec 前短暂的 Python 启动页，空闲基线应看 READY 后的 current/RSS/PSS。50 ms 默认采样可能漏掉短暂峰值，VmHWM 和 cgroup peak 补充整个生命周期峰值。
|
||||
|
||||
构建和指纹读取会预热文件缓存，已缓存的可执行文件/共享库页可能已计费到父组或其他组，因此本工具的 cgroup current 可以小于进程 RSS。它不是冷启动总 RAM 成本；三者应交叉观察。本阶段测热文件缓存下的新进程，不操作系统 `drop_caches`，也不据此宣称整台机器最低内存。
|
||||
|
||||
运行器在结束/异常/中断时终止并回收自己创建的进程，删除自己的空 cgroup。memory/http/http2/live 服务最终由运行器结束进程，**不把这个过程当作语言优雅退出验证**；http2-recovery 和 live 单独验证连接多轮断开后的回收。小时级 soak、故障网络和真实小机器矩阵需另行执行。
|
||||
|
||||
每轮 Rust 负载程序使用同一已验摘要的不可变二进制硬链接，各自保留独立 `config/setting.json` 和日志；跨文件系统才复制。测量期间不得改写输入二进制。这样 100 轮连接恢复不会累积 100 份相同程序，也不通过符号链接让配置解析回到源程序目录。
|
||||
|
||||
## 对照边界与报告
|
||||
|
||||
- Rust-runtime 服务使用与 Dever 相同的 HTTP/TLS/task runtime，帮助定位源代码桥接开销。
|
||||
- 原生 Hyper 服务匹配本测试正常 GET 的响应、连接数、头/正文上限和 TLS 配置；没有复刻 runtime 的完整 WriteTimeout、程序故障传播和结构化 Scope。不能由这个对照推断慢读/慢上传、异常报文的能力等价。
|
||||
- 每轮倒转实现顺序，减轻固定执行顺序造成的偏置。构建配置、CPU 型号、CPU 亲和性、父 cgroup、二进制指纹、原始日志、资源采样、负载端占用写入报告。
|
||||
- `cases.jsonl` 在每个场景完成后追加；`report.json` 即使中途失败也保留已完成结果。`status=complete` 表示测量完整，不表示没有请求错误或已经达到性能目标。先查看 errors/dropped/OOM，再查看延迟和内存。
|
||||
- 不预设绝对 QPS 或最低内存。已有计算热路径的 1.15 对照门槛仍在 `test/dever-tests/tests/performance.rs`，不推广成 HTTP 或整个语言的性能承诺。
|
||||
|
||||
## 首轮结果：2026-09-12
|
||||
|
||||
环境为 Linux x86_64、Intel Xeon Processor (Skylake, IBRS) 虚拟 CPU。服务绑定 CPU 0，负载端绑定 CPU 1；服务 runtime 1 worker、2 blocking slots、272 task slots、32 connections，负载端 16 个请求槽。主测每场景预热 0.5 秒、测量 2 秒，重复 3 个独立进程，目标速率 1000 请求/秒。不是持续压力或吞吐上限测量。
|
||||
|
||||
| Dever 空闲场景 | 三轮 RSS 中位值 |
|
||||
| --- | ---: |
|
||||
| 同步程序 | 1.90 MiB |
|
||||
| 异步运行时 | 2.55 MiB |
|
||||
| 256 个已启动、等待 Channel 的任务 | 3.38 MiB |
|
||||
| HTTP 监听 | 3.45 MiB |
|
||||
| HTTPS 监听 | 4.37 MiB |
|
||||
|
||||
以上为 128 MiB 专属 cgroup 下的 READY 后采样。另用 64 MiB 限额重复 3 轮空闲场景，全部完成、无 OOM。READY 的一次输出会启动异步程序的 blocking worker，异步场景观察到 3 个线程；这是该 fixture 的基线，不是“完全不打印”的最小运行时。
|
||||
|
||||
异步三轮中位值：完整 `run/wait` 一轮约 15.90 µs，Channel 生产与消费平均每项约 0.89 µs；停止并排空 256 个已启动等待任务约 0.61–0.67 ms。前两项每进程校验 10×16384 项，去掉首批预热；包含结构化任务语义，不能把 run/wait 数字称为纯任务分配耗时。尚未提供这几组异步操作的同语义 Rust 对照。
|
||||
|
||||
HTTP/HTTPS 三实现、三路径共 54 组正式测量，108000 个计划请求中 107962 个发出且成功，0 请求错误。38 个未发出：32 个超过发送窗口、6 个无空闲并发槽；预热另有 43 个未发出，原始记录保留。各组成功速率约 999 请求/秒，受目标速率限制，不能宣称最大吞吐为 1000。
|
||||
|
||||
| Dever 路径 | HTTP 每轮实际请求 p99 范围 | HTTPS 每轮实际请求 p99 范围 |
|
||||
| --- | ---: | ---: |
|
||||
| /plain | 0.243–1.248 ms | 0.580–0.909 ms |
|
||||
| /json | 0.280–0.342 ms | 0.897–2.752 ms |
|
||||
| /bytes | 0.392–0.500 ms | 0.839–2.400 ms |
|
||||
|
||||
表中是 dispatched→completed，每列列出三个独立 p99 的范围，没有把它们合并成全局 p99；计划发送延迟和负载端 lag 另在 JSON 中。短时虚拟机抖动明显，这些结果不用于宣称哪种实现更快。Dever 在这组负载的 RSS 中位值约 HTTP 3.9 MiB、HTTPS 4.9–5.1 MiB。
|
||||
|
||||
另以每次新建连接、100 请求/秒测试 HTTP/HTTPS 全路径三实现，共 18 组、900 个正式请求，全部成功且无丢弃。主测 78 组（15 内存、9 异步、54 网络）、新连接 18 组、64 MiB 内存 15 组均无 OOM。
|
||||
|
||||
原始结果及对应产物保存在仓库 `target/performance/`：
|
||||
|
||||
- `baseline-128m/report.json`：最终三轮基线，对应 `build-2/manifest.json`。
|
||||
- `connections-new-128m/report.json`：新连接模式，对应 build-2。
|
||||
- `memory-64m/report.json`：64 MiB 空闲基线，对应 build-1；该组使用的 Dever 二进制与 build-2 一致。
|
||||
- `smoke-1/report.json`：早期连通检查，保留当时指纹，不用于最终网络比较。
|
||||
|
||||
以上首轮结果保留当时的测试范围；后续测量见下节。
|
||||
|
||||
## 第二轮：高负载与调度定位
|
||||
|
||||
`http-load-p2-final/report.json` 复用 build-2 产物，1 worker、32 服务连接和 32 客户端请求槽、服务 1 CPU/128 MiB。仅 `/plain`，预热 0.5 秒、测量 2 秒；HTTP/HTTPS、三个实现、5000/15000/30000 请求每秒，均重复三轮，共 54 组。
|
||||
|
||||
正式窗口共计划 1800000 个请求，1500830 个实际发出且成功，错误 0。未发出 299170 个：容量丢弃 298476、发送窗口过期 694；所有预算组无 OOM。下表是三个独立进程的成功 QPS 中位值，不是最大吞吐。
|
||||
|
||||
| 目标请求/秒 | Dever HTTP | Rust-runtime HTTP | Hyper HTTP | Dever HTTPS | Rust-runtime HTTPS | Hyper HTTPS |
|
||||
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| 5000 | 4993 | 4994 | 4993 | 4970 | 4972 | 4970 |
|
||||
| 15000 | 14904 | 14897 | 14922 | 14839 | 14756 | 14726 |
|
||||
| 30000 | 21866 | 22187 | 22324 | 21217 | 21320 | 22236 |
|
||||
|
||||
最高输入档位 Dever 服务 CPU 中位值为 HTTP 0.53 核、HTTPS 0.62 核，客户端为 0.77/0.86 核；Dever RSS 约 4.12/5.32 MiB。32 请求槽与定时器成批发送已经影响结果，不能据此断言服务端上限是每秒两万。提高并发槽、使用独立负载机器应作为下一次吞吐上限测量的条件。
|
||||
|
||||
`http-load-p2/report.json` 保留初次中断：Linux 退出时 `/proc` 内存字段先于 waitpid 消失，导致采样工具失败。修复后才完整重跑；不能把初次中断算成语言服务失败，也不将两次样本混合。
|
||||
|
||||
`async-p2/report.json` 使用 build-p2、1 worker/1 CPU/128 MiB、1040 task slots、每批 16384 次操作，重复 3 个进程；每进程十批、去掉首批。共 18 组，校验全部通过且无 OOM。下表为进程内中位数再取三进程中位数：
|
||||
|
||||
| run/wait 位置 | Dever | 同 runtime、同优化参数的 Rust |
|
||||
| --- | ---: | ---: |
|
||||
| 入口 root | 15.50 µs/次 | 15.59 µs/次 |
|
||||
| 外层结构化任务 worker | 2.003 µs/次 | 1.984 µs/次 |
|
||||
|
||||
这个串行微任务场景中，调度位置使 Dever 平均单次耗时降低约 87%，约 7.7 倍速度；不推广为所有业务程序加速。两种语言路径数值接近。源码路径显示 `run_entry_with` 用入口线程 block_on，run body 和 supervisor 在 Tokio worker 执行；单核下逐项等待造成频繁 OS 线程切换。`profile-p2/final-runtime-*.stat` 是独立 perf stat 的整体软件计数，对应 build-p2/async_bench；未加 cgroup、固定 CPU 0。虚拟机不支持 cycles/instructions，未报告硬件 IPC 或采样火焰图。
|
||||
|
||||
同组 Channel 约 0.912 µs/项；256 个任务停止并排空三次共 0.688/0.916/1.045 ms。这里只新增对照和证据，生产入口调度尚未优化。后续优化入口调度时需确认入口 future 的 Send 条件，根任务不应挤占子任务配额，并验证入口故障、子任务取消和 blocking 清理；不应直接照搬实验外层 run 或删除结构化任务语义。
|
||||
|
||||
## 第二轮：长连接容量与回收
|
||||
|
||||
`live-p2-final/report.json` 使用 build-p2-final、1 worker/1 CPU/128 MiB、512 connection slots、1040 task slots；32/128/512 条连接各重复三进程，每进程三轮连接/保持 2 秒/断开。TCP/WS 每约 100 ms 校验一次小消息回显，SSE 校验初始事件和 100 ms 心跳。
|
||||
|
||||
27 组共建立并关闭 18144 条连接，校验 377061 条消息，无协议错误和 OOM。81 次断开后的服务 FD 均回到启动基线。下面是第三轮保持窗口 RSS 的三个独立进程中位值，包含分配器在此前两轮保留的内存：
|
||||
|
||||
| 同时连接数 | TCP RSS | WebSocket RSS | SSE RSS |
|
||||
| ---: | ---: | ---: | ---: |
|
||||
| 32 | 3.99 MiB | 5.11 MiB | 5.03 MiB |
|
||||
| 128 | 4.90 MiB | 9.44 MiB | 9.16 MiB |
|
||||
| 512 | 8.70 MiB | 17.00 MiB | 24.97 MiB |
|
||||
|
||||
512 条连接保持时服务 CPU 约 TCP 0.15–0.16 核、WS 0.18–0.20 核、SSE 0.12–0.14 核。这里测的是持续小消息和心跳连接，不是大消息吞吐或完全不读的客户端。
|
||||
|
||||
初次 `live-p2/report.json` 也完成了全部消息校验，但有一条下一轮的采样被误归属上轮 disconnected，产生假的 FD +15。最终报告使用采样前后阶段一致性判断，完整重跑；资源回收结论只引用 final 报告。
|
||||
|
||||
随后 `live-cycles-p2/report.json` 对每种协议保持 512 条连接，十轮重连、每轮保持 3 秒，单场景约 34 秒。共 15360 次连接、471207 条校验消息，错误/OOM 为零，30 次 FD 回落均为零差值。TCP 断开后的 RSS 最后稳定在约 8.6–8.8 MiB，WS 约 17 MiB；SSE 前三轮 21.33/24.48/25.12 MiB，后七轮为 25.24–25.36 MiB，未观察到按轮持续增长。这是短时重复回收证据，不是小时级泄漏或生产网络稳定性证明。
|
||||
|
||||
复现本轮长连接两组命令（输出目录另取新名称）：
|
||||
|
||||
```bash
|
||||
python3 test/performance/run.py run --artifacts target/performance/build-p2-final --output target/performance/live-repeat --suite live --duration 2 --connection-tiers 32,128,512 --cycles 3 --repeats 3 --server-cpu 0 --client-cpus 1 --client-workers 1 --cgroup-parent /sys/fs/cgroup/system.slice --memory-mib 128 --cpu-quota 1
|
||||
python3 test/performance/run.py run --artifacts target/performance/build-p2-final --output target/performance/live-more-cycles --suite live --duration 3 --connection-tiers 512 --cycles 10 --repeats 1 --server-cpu 0 --client-cpus 1 --client-workers 1 --cgroup-parent /sys/fs/cgroup/system.slice --memory-mib 128 --cpu-quota 1
|
||||
```
|
||||
|
||||
本轮 12 个运行器检查、4 个独立 TCP/SSE 对端检查通过，11 个 Dever 入口及两个新 Rust example 离线构建通过。Rustfmt/Clippy 未安装，未运行；Dever 格式、Python 解析、源码空白与最终范围检查通过。生产编译器/运行时没有修改，未运行全量测试。
|
||||
|
||||
## 第三轮：HTTP/2 吞吐、流控与连接回收
|
||||
|
||||
2026-09-13 使用 `build-http2-final` 产物完成正式矩阵。环境仍为 Linux x86_64、Intel Xeon Processor (Skylake, IBRS) 虚拟 CPU；服务绑定 CPU 0，负载端绑定 CPU 1。服务使用 1 worker、128 MiB cgroup、1 CPU quota、swap 0；负载端不计入服务预算。每个 case 预热 0.5 秒、测量 2 秒、启动独立服务进程，三实现与两种传输的执行顺序逐轮倒转。
|
||||
|
||||
`/plain` 共 108 个 case、3600000 个计划请求，其中 2729242 个发出且完整成功，错误 0。870758 个未发出请求中，869368 个受请求槽限制，1390 个超过调度窗口；全部计数守恒。108 次正式负载和 108 次预热的握手数均与物理连接数一致，断开后的 FD 均回到 READY 基线，无 OOM。下表是 30000 请求/秒输入档位的三轮中位数；p99 是实际发出到完整响应的请求延迟。
|
||||
|
||||
| 实现 | 传输 | 1x16 成功 QPS | 4x16 成功 QPS | 4x16 请求 p99 | 4x16 服务 RSS |
|
||||
| --- | --- | ---: | ---: | ---: | ---: |
|
||||
| Dever | h2c | 10578 | 29575 | 2.208 ms | 4.60 MiB |
|
||||
| Rust-runtime | h2c | 10804 | 28996 | 3.392 ms | 5.00 MiB |
|
||||
| Hyper | h2c | 11956 | 29108 | 1.984 ms | 4.59 MiB |
|
||||
| Dever | TLS+h2 | 9957 | 29814 | 1.840 ms | 5.59 MiB |
|
||||
| Rust-runtime | TLS+h2 | 10122 | 28681 | 2.880 ms | 6.07 MiB |
|
||||
| Hyper | TLS+h2 | 11555 | 28719 | 3.200 ms | 5.68 MiB |
|
||||
|
||||
Dever 4x16 在 5000、15000、30000 三档的 h2c 成功 QPS 中位值分别为 4995、14983、29575，TLS+h2 分别为 4994、14977、29814。1x16 在高输入下只有约 10k–10.6k 成功 QPS，而 4x16 接近 30k，说明这次单连接结果受固定 16 个请求槽明显约束；不能把任一数字解释成服务端极限。最高档 4x16 的 Dever 服务 CPU 中位值为 h2c 0.89 核、TLS+h2 0.93 核，负载端也约 0.9 核，进一步限制了极限吞吐结论。三种实现的短时差异不具备跨机器性能承诺意义。
|
||||
|
||||
`/bytes` 共 108 个 case、115200 个计划请求，115180 个完整成功、错误 0、丢弃 20、无 OOM，握手和 FD 检查全部通过。三种实现、两种传输、两种拓扑在 100/500/1000 请求每秒档均维持目标速率；这相当于单 case 最高约 62.5 MiB/s 的响应正文。Dever 的 RSS 中位值为 h2c 4.04–4.23 MiB、TLS+h2 5.05–5.30 MiB。短 case 的 p99 有明显虚拟机抖动，原始三轮数据比单个汇总值更适合定位异常。
|
||||
|
||||
`http2-recovery-128m` 对三种实现、h2c/TLS 各在同一服务进程中运行 10 轮 1x16 连接周期，共 60 轮、30000 个计划请求；29984 个完整成功、错误 0、丢弃 16。每轮握手数正确，60 次断开后的 FD 差值均为 0，无 OOM。六组最后一轮相对第一轮的断开后 RSS 变化为 8–100 KiB，短时序列未观察到按轮持续增长；这不是小时级泄漏证明。
|
||||
|
||||
另在 64 MiB、1 CPU、swap 0 下运行 Dever 1x16、5000 请求每秒的 h2c/TLS 代表场景。20000 个计划请求中 19991 个完整成功、错误 0、丢弃 9、无 OOM；h2c/TLS 成功 QPS 为 4997/4996，RSS 为 4.04/5.17 MiB，PSS 为 2.00/3.10 MiB。cgroup peak 只有 1.30/1.35 MiB，是热文件页可能已计费给其它组所致，不能代替进程 RSS/PSS 或冷启动总内存。
|
||||
|
||||
正式结果和构建清单位于：
|
||||
|
||||
- `build-http2-final/manifest.json`：16 个二进制的指纹、大小、构建配置和固定 HTTP/2 limits。
|
||||
- `http2-plain-128m/report.json`：`/plain` 128 MiB 正式矩阵。
|
||||
- `http2-bytes-128m/report.json`：`/bytes` 128 MiB 正式矩阵。
|
||||
- `http2-recovery-128m/report.json`：128 MiB 十轮连接恢复矩阵。
|
||||
- `http2-active-64m/report.json`：64 MiB Dever 活跃代表场景。
|
||||
|
||||
历史 `build-http2-final` 产物已不在当前工作树；以下是当时使用的命令记录，精确复现历史二进制须先恢复匹配的旧产物。本机当前可直接运行的命令见下方第四轮，复现时必须换新输出目录：
|
||||
|
||||
```bash
|
||||
python3 test/performance/run.py run --artifacts target/performance/build-http2-final --output target/performance/http2-plain-repeat --suite http2 --duration 2 --warmup 0.5 --rates 5000,15000,30000 --paths /plain --http2-connections 1,4 --repeats 3 --server-cpu 0 --client-cpus 1 --client-workers 1 --cgroup-parent /sys/fs/cgroup/system.slice --memory-mib 128 --cpu-quota 1
|
||||
python3 test/performance/run.py run --artifacts target/performance/build-http2-final --output target/performance/http2-bytes-repeat --suite http2 --duration 2 --warmup 0.5 --rates 100,500,1000 --paths /bytes --http2-connections 1,4 --repeats 3 --server-cpu 0 --client-cpus 1 --client-workers 1 --cgroup-parent /sys/fs/cgroup/system.slice --memory-mib 128 --cpu-quota 1
|
||||
python3 test/performance/run.py run --artifacts target/performance/build-http2-final --output target/performance/http2-recovery-repeat --suite http2-recovery --duration 0.5 --warmup 0.1 --cycles 10 --recovery-rate 1000 --recovery-path /plain --recovery-connections 1 --repeats 1 --server-cpu 0 --client-cpus 1 --client-workers 1 --cgroup-parent /sys/fs/cgroup/system.slice --memory-mib 128 --cpu-quota 1
|
||||
```
|
||||
|
||||
N5 的接口与约束见 [LANGUAGE.md](../../LANGUAGE.md)。这轮只建立 loopback、热文件缓存、短时固定速率下的相对证据，不包含跨主机网络、故障注入、小时级 soak 或多平台矩阵。HTTP/2 不保证所有负载都比 HTTP/1.1 快；普通 WebSocket 继续走 HTTP/1，RFC 8441 扩展 CONNECT 尚未实现。
|
||||
|
||||
## 第四轮：多核负载端、HTTP/2 饱和与稳态
|
||||
|
||||
2026-09-18 使用 `typed-orm-build-v2/manifest.json` 中校验过指纹的现存 HTTP/2 二进制和测试证书；原 `build-http2-final` 不在当前工作树，未重建产物。服务端绑定 CPU 0、1 worker、1 CPU quota、无 swap；负载端为 CPU 1–4、4 workers，位于服务 cgroup 外。机器允许 CPU 0–4。服务预算为 128 MiB，稳态另测 64 MiB。所有场景使用 `/plain`、16 条物理连接、每连接 16 流，只有自有 loopback 端口；这是同一虚拟机的热缓存观测，不是跨机器或生产容量。
|
||||
|
||||
128 MiB 扫描覆盖 h2c/TLS+h2 的 30k/45k/60k/90k 输入，每档预热 1 秒、计时 5 秒、2 个重复。16/16 case 计数守恒，请求错误、OOM 和断开后的 FD 差均为 0；每次握手数为 16。下表为成功 QPS 的两轮中位值：
|
||||
|
||||
| 传输 | 30k | 45k | 60k | 90k |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| h2c | 29,899 | 36,690 | 36,027 | 39,486 |
|
||||
| TLS+h2 | 29,770 | 38,205 | 37,410 | 38,136 |
|
||||
|
||||
45k 以上服务 CPU 约 0.96–1.00 核，客户端约 1.43–1.97 核；未发送请求主要因 256 个请求槽满，调度窗口过期数量相对很少。因此观测到当前拓扑约 36k–40k req/s 的吞吐平台，不能把它归结为纯服务端极限或把输入 90k 当成实际吞吐。28k 的首轮 60 秒成功率为 h2c 98.68%、TLS+h2 97.91%，未达 99% 验收门槛；24k 的 10 秒校准 TLS+h2 为 99.18%，余量仍小。20k 的 10 秒校准分别为 99.96% 和 99.92%，所以正式稳态统一采用 20k，低于较低观测平台的 80%。这些探索报告保留，不算正式稳态通过。
|
||||
|
||||
正式每个 case 预热 1 秒、测量 60 秒、计划 1,200,000 次请求。成功率是成功请求除以计划请求，容量丢弃仍在原始报告中，不称为零丢弃：
|
||||
|
||||
| 服务预算 | 传输 | 成功/计划 | 成功 QPS | 请求 p99 | 服务 RSS 高水位 | OOM/错误/FD 差 |
|
||||
| --- | --- | ---: | ---: | ---: | ---: | --- |
|
||||
| 128 MiB | h2c | 1,197,220/1,200,000 (99.77%) | 19,953 | 5.376 ms | 6.78 MiB | 0/0/0 |
|
||||
| 128 MiB | TLS+h2 | 1,199,036/1,200,000 (99.92%) | 19,983 | 4.800 ms | 7.83 MiB | 0/0/0 |
|
||||
| 64 MiB | h2c | 1,199,221/1,200,000 (99.94%) | 19,986 | 4.672 ms | 6.43 MiB | 0/0/0 |
|
||||
| 64 MiB | TLS+h2 | 1,197,545/1,200,000 (99.80%) | 19,958 | 5.184 ms | 7.90 MiB | 0/0/0 |
|
||||
|
||||
128 MiB 下在各自的同一服务进程上完成 h2c/TLS+h2 各 100 轮 16x16 恢复；每轮 0.1 秒、1000 req/s，随后观察断开 0.3 秒。两组各计划 10,000 次、成功 9,944 次，其余各 56 次为调度窗口过期，请求错误和 OOM 为 0。每轮至少成功 95（h2c）/98（TLS+h2）次，握手始终 16，200 次断开后的 FD 差全部为 0。断开后 RSS 从首轮到末轮为 h2c 4,845,568→5,357,568 B，TLS+h2 5,959,680→6,647,808 B；保留完整逐轮序列，不将分钟级增长或回落解释成小时级泄漏结论。邻接的 HTTP/HTTPS 各 30/30 成功，TCP/WS/SSE 各 16/16 连接完成关闭，零错误。
|
||||
|
||||
复现命令（输出目录必须换新名称；运行前确认 CPU 0–4 和 cgroup 权限，不接触现有服务）：
|
||||
|
||||
```bash
|
||||
python3 test/performance/run.py run --artifacts target/performance/typed-orm-build-v2 --output target/performance/http2-scan-next --suite http2 --duration 5 --warmup 1 --rates 30000,45000,60000,90000 --paths /plain --http2-connections 16 --repeats 2 --implementations dever --server-cpu 0 --client-cpus 1,2,3,4 --client-workers 4 --cgroup-parent /sys/fs/cgroup/system.slice --memory-mib 128 --cpu-quota 1
|
||||
python3 test/performance/run.py run --artifacts target/performance/typed-orm-build-v2 --output target/performance/http2-steady-128-next --suite http2 --duration 60 --warmup 1 --rates 20000 --paths /plain --http2-connections 16 --repeats 1 --implementations dever --server-cpu 0 --client-cpus 1,2,3,4 --client-workers 4 --cgroup-parent /sys/fs/cgroup/system.slice --memory-mib 128 --cpu-quota 1
|
||||
python3 test/performance/run.py run --artifacts target/performance/typed-orm-build-v2 --output target/performance/http2-steady-64-next --suite http2 --duration 60 --warmup 1 --rates 20000 --paths /plain --http2-connections 16 --repeats 1 --implementations dever --server-cpu 0 --client-cpus 1,2,3,4 --client-workers 4 --cgroup-parent /sys/fs/cgroup/system.slice --memory-mib 64 --cpu-quota 1
|
||||
python3 test/performance/run.py run --artifacts target/performance/typed-orm-build-v2 --output target/performance/http2-recovery-next --suite http2-recovery --duration 0.1 --cycles 100 --recovery-rate 1000 --recovery-connections 16 --repeats 1 --implementations dever --server-cpu 0 --client-cpus 1,2,3,4 --client-workers 4 --cgroup-parent /sys/fs/cgroup/system.slice --memory-mib 128 --cpu-quota 1
|
||||
```
|
||||
|
||||
本轮原始结果位于 `target/performance/http2-ceiling-{scan,steady-128m-20k,steady-64m-20k,recovery}-09-18/`；28k 首轮及 24k/20k 校准在相邻命名目录。吞吐平台、20k 持续负载和 100 轮恢复是不同问题，不能互相替代；服务与负载仍共享虚拟机和热缓存，结果不外推到真实网络、小物理机或小时级运行。
|
||||
@@ -0,0 +1,466 @@
|
||||
"""构建可搬运的基准产物；构建耗时与运行测量分开。"""
|
||||
|
||||
from copy import deepcopy
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from settings import PEER_CONFIGURATION, stage_peer, write_private_json
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
FIXTURES = Path(__file__).parent / "fixtures"
|
||||
TLS = ROOT / "test/dever-tests/fixtures/tls"
|
||||
SETTING_PATH = ROOT / "config/setting.json"
|
||||
POSTGRES_TEST_CONNECTION = "postgres_test"
|
||||
APPLICATION_CONFIGURATION = "llvm-cmd-v1"
|
||||
ENTRIES = {
|
||||
"runtime": ("sync_idle", "async_idle", "tasks", "tasks_on_worker", "channels", "cancel"),
|
||||
"http": ("http", "https", "http2", "https2"),
|
||||
"live": ("tcp", "ws", "sse"),
|
||||
}
|
||||
ORM_ENTRIES = ("orm_idle", "orm_crud", "orm_list", "orm_cursor", "orm_stream",
|
||||
"orm_pool", "orm_http")
|
||||
RUNTIME_PROFILES = {
|
||||
"base": ("runtime-api", "runtime-external"),
|
||||
"sqlite": ("runtime-api", "runtime-external", "runtime-sqlite"),
|
||||
"postgres": ("runtime-api", "runtime-external", "runtime-postgres"),
|
||||
"both": ("runtime-api", "runtime-external", "runtime-sqlite", "runtime-postgres"),
|
||||
}
|
||||
BENCHMARK_COMMAND = "benchmark.command.run"
|
||||
CONFIG_FUNCTIONS = {
|
||||
"runtime": ("lifetime", "idle_sample_ms", "pending", "channel_capacity", "input", "pending_input"),
|
||||
"http": ("lifetime", "limits", "http2_limits", "large_body", "certificate", "key"),
|
||||
"live": (),
|
||||
"orm": ("lifetime", "orm_input", "orm_rows", "orm_concurrency", "orm_cancel_work"),
|
||||
}
|
||||
|
||||
|
||||
def fingerprint(path):
|
||||
with path.open("rb") as binary:
|
||||
digest = hashlib.file_digest(binary, "sha256").hexdigest()
|
||||
return {"sha256": digest, "bytes": path.stat().st_size}
|
||||
|
||||
|
||||
def postgres_test_connection(document):
|
||||
try:
|
||||
connection = document["database"][POSTGRES_TEST_CONNECTION]
|
||||
except (KeyError, TypeError) as error:
|
||||
raise ValueError(
|
||||
f"config/setting.json must define database.{POSTGRES_TEST_CONNECTION}"
|
||||
) from error
|
||||
if not isinstance(connection, dict):
|
||||
raise ValueError(f"database.{POSTGRES_TEST_CONNECTION} must be an object")
|
||||
if connection.get("type") != "postgres":
|
||||
raise ValueError(f"database.{POSTGRES_TEST_CONNECTION} must use type 'postgres'")
|
||||
max_connections = connection.get("max_connections")
|
||||
if type(max_connections) is not int or not 1 <= max_connections <= 256:
|
||||
raise ValueError(
|
||||
f"database.{POSTGRES_TEST_CONNECTION}.max_connections must be an explicit 1..256 integer"
|
||||
)
|
||||
return connection
|
||||
|
||||
|
||||
def postgres_case_url(document, case_name):
|
||||
template = postgres_test_connection(document).get("url")
|
||||
if not isinstance(template, str):
|
||||
raise ValueError(f"database.{POSTGRES_TEST_CONNECTION}.url must be Text")
|
||||
parsed = urlsplit(template)
|
||||
database = parsed.path.removeprefix("/")
|
||||
if (template.count("{case}") != 1 or "{case}" not in database
|
||||
or not database or "/" in database):
|
||||
raise ValueError(
|
||||
f"database.{POSTGRES_TEST_CONNECTION}.url must contain one {{case}} placeholder "
|
||||
"in its database name"
|
||||
)
|
||||
if parsed.scheme not in ("postgres", "postgresql"):
|
||||
raise ValueError(
|
||||
f"database.{POSTGRES_TEST_CONNECTION}.url must use postgres:// or postgresql://"
|
||||
)
|
||||
return template.replace("{case}", case_name)
|
||||
|
||||
|
||||
def postgres_database_case(output, case_name):
|
||||
run = hashlib.sha256(str(output.resolve()).encode()).hexdigest()[:12]
|
||||
return f"{run}_{case_name.replace('-', '_')}"
|
||||
|
||||
|
||||
def load_postgres_setting(path=SETTING_PATH):
|
||||
if not path.is_file():
|
||||
return None
|
||||
try:
|
||||
document = json.loads(path.read_text())
|
||||
except (OSError, json.JSONDecodeError) as error:
|
||||
raise ValueError(f"cannot load '{path}': {error}") from error
|
||||
if not isinstance(document, dict):
|
||||
raise ValueError(f"'{path}' must contain a JSON object")
|
||||
connections = document.get("database")
|
||||
if connections is None:
|
||||
return None
|
||||
if not isinstance(connections, dict):
|
||||
raise ValueError(f"'{path}' database must be an object")
|
||||
if POSTGRES_TEST_CONNECTION not in connections:
|
||||
return None
|
||||
postgres_case_url(document, "validation")
|
||||
return document
|
||||
|
||||
|
||||
def configuration(args, postgres_setting=None):
|
||||
settings = {
|
||||
"workers": args.workers,
|
||||
"connections": args.connections,
|
||||
"pending": args.pending,
|
||||
"iterations": args.iterations,
|
||||
"channel_capacity": 64,
|
||||
"timeout_ms": 2000,
|
||||
"lifetime_ms": 3_600_000,
|
||||
"idle_sample_ms": 3000,
|
||||
"task_capacity": max(args.pending + 16, 2 * args.connections + 16),
|
||||
"max_blocking_threads": 2,
|
||||
"http2_streams": args.http2_streams,
|
||||
"http2_stream_window_bytes": args.http2_stream_window_bytes,
|
||||
"http2_connection_window_bytes": args.http2_connection_window_bytes,
|
||||
"orm_iterations": args.orm_iterations,
|
||||
"orm_rows": args.orm_rows,
|
||||
"orm_concurrency": args.orm_concurrency,
|
||||
"orm_cancel_work": args.orm_cancel_work,
|
||||
"orm_sqlite_connections": args.orm_sqlite_connections,
|
||||
"orm_postgres_connections": (
|
||||
postgres_test_connection(postgres_setting)["max_connections"]
|
||||
if postgres_setting is not None else None
|
||||
),
|
||||
}
|
||||
if settings["task_capacity"] > 65536:
|
||||
raise ValueError("task capacity exceeds the language limit")
|
||||
return settings
|
||||
|
||||
|
||||
def config_source(settings, fixture):
|
||||
names = CONFIG_FUNCTIONS[fixture]
|
||||
functions = {
|
||||
"lifetime": ("Int", str(settings["lifetime_ms"])),
|
||||
"idle_sample_ms": ("Int", str(settings["idle_sample_ms"])),
|
||||
"pending": ("Int", str(settings["pending"])),
|
||||
"channel_capacity": ("Int", str(settings["channel_capacity"])),
|
||||
"input": ("Bytes", f'dever.bytes.from_text({json.dumps("x" * settings["iterations"])})'),
|
||||
"pending_input": ("Bytes", f'dever.bytes.from_text({json.dumps("x" * settings["pending"])})'),
|
||||
"orm_input": ("Bytes", f'dever.bytes.from_text({json.dumps("x" * settings["orm_iterations"])})'),
|
||||
"orm_rows": ("Bytes", f'dever.bytes.from_text({json.dumps("r" * settings["orm_rows"])})'),
|
||||
"orm_concurrency": ("Int", str(settings["orm_concurrency"])),
|
||||
"orm_cancel_work": ("Int", str(settings["orm_cancel_work"])),
|
||||
"limits": ("dever.http.Limits", "dever.http.Limits {\nheader_bytes = 8192\n"
|
||||
f'body_bytes = 65536\ntimeout_ms = {settings["timeout_ms"]}\n'
|
||||
f'connections = {settings["connections"]}\nhttp2 = null\n}}'),
|
||||
"http2_limits": ("dever.http.Limits", "dever.http.Limits {\nheader_bytes = 8192\n"
|
||||
f'body_bytes = 65536\ntimeout_ms = {settings["timeout_ms"]}\n'
|
||||
f'connections = {settings["connections"]}\nhttp2 = dever.http.Http2Limits {{\n'
|
||||
f'streams = {settings["http2_streams"]}\n'
|
||||
f'stream_window_bytes = {settings["http2_stream_window_bytes"]}\n'
|
||||
f'connection_window_bytes = {settings["http2_connection_window_bytes"]}\n}}\n}}'),
|
||||
}
|
||||
if "certificate" in names:
|
||||
functions["certificate"] = ("Bytes", f'dever.bytes.from_text({json.dumps((TLS / "server.pem").read_text())})')
|
||||
if "key" in names:
|
||||
functions["key"] = ("Bytes", f'dever.bytes.from_text({json.dumps((TLS / "server-key.pem").read_text())})')
|
||||
source = ""
|
||||
for name in names:
|
||||
if name == "large_body":
|
||||
continue
|
||||
kind, expression = functions[name]
|
||||
source += f"\n{name}() (result: {kind}) {{\nresult = {expression}\n}}\n"
|
||||
if "large_body" in names:
|
||||
source += '\ndouble(value: Int, text: Text) (result: Text) { result = text + text }\n'
|
||||
source += '\nlarge_body() (result: Bytes) {\nresult = dever.bytes.from_text(reduce(double, '
|
||||
source += '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], "x"))\n}\n'
|
||||
return source
|
||||
|
||||
|
||||
def stage_application_command(module, capability):
|
||||
command = module / "benchmark/command"
|
||||
command.mkdir(parents=True, exist_ok=True)
|
||||
(command / "app.dever").write_text(
|
||||
f"run() (result: Bool) {{\n result = {capability}()\n}}\n"
|
||||
)
|
||||
(command / "api.dever").write_text("cmd run = app.run\n")
|
||||
|
||||
|
||||
def stage_benchmark_support(module, settings, fixture):
|
||||
if fixture in ("runtime", "orm"):
|
||||
bench = module / "benchmark/bench/app.dever"
|
||||
bench.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copyfile(FIXTURES / "bench.dever", bench)
|
||||
if CONFIG_FUNCTIONS[fixture]:
|
||||
config = module / "benchmark/config/app.dever"
|
||||
config.parent.mkdir(parents=True, exist_ok=True)
|
||||
config.write_text(config_source(settings, fixture))
|
||||
|
||||
|
||||
def database_setting(driver, settings, postgres_setting=None, case_name=None):
|
||||
if driver == "sqlite":
|
||||
database = {
|
||||
"type": "sqlite",
|
||||
"path": "data/db/orm.db",
|
||||
"max_connections": settings["orm_sqlite_connections"],
|
||||
"max_page_size": max(32, min(settings["orm_rows"], 10_000)),
|
||||
}
|
||||
elif driver == "postgres":
|
||||
if postgres_setting is None or case_name is None:
|
||||
raise ValueError("PostgreSQL requires config/setting.json")
|
||||
database = deepcopy(postgres_test_connection(postgres_setting))
|
||||
database["url"] = postgres_case_url(postgres_setting, case_name)
|
||||
else:
|
||||
raise ValueError(f"unknown database driver: {driver}")
|
||||
document = ({name: deepcopy(postgres_setting[name])
|
||||
for name in ("http", "log") if name in postgres_setting}
|
||||
if driver == "postgres" else {})
|
||||
document.setdefault("http", {})
|
||||
document.setdefault("log", {})
|
||||
document["database"] = {"default": database}
|
||||
document["auth"] = {"providers": {"benchmark": {
|
||||
"verify": "benchmark.item.verify",
|
||||
"jwtSecret": secrets.token_urlsafe(48),
|
||||
"ttlSeconds": 3600,
|
||||
}}}
|
||||
document["sites"] = {"benchmark": {"path": "", "auth": "benchmark"}}
|
||||
return document
|
||||
|
||||
|
||||
def profile_database_setting(profile):
|
||||
if profile == "base":
|
||||
return {}
|
||||
sqlite = {
|
||||
"type": "sqlite",
|
||||
"path": "data/db/profile.db",
|
||||
"max_connections": 1,
|
||||
"max_page_size": 100,
|
||||
}
|
||||
if profile == "sqlite":
|
||||
return {"database": {"default": sqlite}}
|
||||
if profile not in ("postgres", "both"):
|
||||
raise ValueError(f"unknown runtime profile: {profile}")
|
||||
postgres = {
|
||||
"type": "postgres",
|
||||
"url": "postgres://build-only.invalid/dever_profile",
|
||||
"tls": "disabled",
|
||||
"max_connections": 1,
|
||||
}
|
||||
if profile == "postgres":
|
||||
return {"database": {"default": postgres}}
|
||||
return {"database": {"default": sqlite, "postgres_profile": postgres}}
|
||||
|
||||
|
||||
def dependency_packages(text):
|
||||
return sorted({line.removesuffix(" (*)") for line in text.splitlines() if line})
|
||||
|
||||
|
||||
def runtime_dependencies(cargo, features):
|
||||
command = [cargo, "tree", "--offline", "--locked", "-p", "dever-backend-bridge",
|
||||
"--no-default-features", "--prefix", "none"]
|
||||
if features:
|
||||
command.extend(("--features", ",".join(features)))
|
||||
return dependency_packages(subprocess.check_output(command, cwd=ROOT, text=True))
|
||||
|
||||
|
||||
def prepare_native_project(destination, fixture, settings):
|
||||
module = destination / "module"
|
||||
domain = module / "benchmark" / fixture
|
||||
domain.mkdir(parents=True)
|
||||
shutil.copyfile(FIXTURES / fixture / "app.dever", domain / "app.dever")
|
||||
for role in ("port", "adapter", "api"):
|
||||
shutil.copyfile(FIXTURES / fixture / f"{role}.dever",
|
||||
domain / f"{role}.dever")
|
||||
stage_benchmark_support(module, settings, fixture)
|
||||
config = destination / "config"
|
||||
config.mkdir()
|
||||
write_private_json(config / "setting.json",
|
||||
native_runtime_setting(settings) if fixture == "live" else {})
|
||||
|
||||
|
||||
def prepare_project(destination, fixture, settings, driver, postgres_setting=None):
|
||||
module = destination / "module"
|
||||
domain = module / "benchmark/item"
|
||||
domain.mkdir(parents=True)
|
||||
shutil.copyfile(FIXTURES / fixture / "app.dever", domain / "app.dever")
|
||||
shutil.copyfile(FIXTURES / fixture / "app/model/item.dever", domain / "model.dever")
|
||||
if fixture == "orm":
|
||||
shutil.copyfile(FIXTURES / fixture / "app/api.dever", domain / "api.dever")
|
||||
stage_benchmark_support(module, settings, fixture)
|
||||
stage_application_command(module, "benchmark.item.orm_idle")
|
||||
config = destination / "config"
|
||||
config.mkdir()
|
||||
write_private_json(
|
||||
config / "setting.json",
|
||||
database_setting(driver, settings, postgres_setting, "build_only"),
|
||||
)
|
||||
(destination / "data").mkdir()
|
||||
|
||||
|
||||
def prepare_profile_project(destination, setting):
|
||||
shutil.copytree(FIXTURES / "profile", destination / "module")
|
||||
config = destination / "config"
|
||||
config.mkdir()
|
||||
write_private_json(config / "setting.json", setting)
|
||||
(destination / "data").mkdir()
|
||||
|
||||
|
||||
def prepare_cms_project(destination, source):
|
||||
shutil.copytree(ROOT / "examples/cms" / source, destination)
|
||||
|
||||
|
||||
def native_runtime_setting(settings):
|
||||
return {"adapter": {"benchmark.live": {"setting": {
|
||||
"limits": {
|
||||
"header_bytes": 8192,
|
||||
"body_bytes": 65536,
|
||||
"timeout_ms": settings["timeout_ms"],
|
||||
"connections": settings["connections"],
|
||||
"http2": None,
|
||||
},
|
||||
"lifetime_ms": settings["lifetime_ms"],
|
||||
}}}}
|
||||
|
||||
|
||||
def record_binary(manifest, name, binary, started, **metadata):
|
||||
manifest["binaries"][name] = {
|
||||
**fingerprint(binary),
|
||||
"build_seconds": time.monotonic() - started,
|
||||
**metadata,
|
||||
}
|
||||
|
||||
|
||||
def build_profile_report(args, manifest, log):
|
||||
report = {"source": "profile/benchmark/command/app.dever", "profiles": {}}
|
||||
for profile, features in RUNTIME_PROFILES.items():
|
||||
item = {"runtime_features": list(features),
|
||||
"dependencies": runtime_dependencies(args.cargo, features)}
|
||||
print(f"build profile-{profile}", flush=True)
|
||||
binary_name = f"profile-{profile}"
|
||||
binary = args.output.resolve() / binary_name
|
||||
started = time.monotonic()
|
||||
project = args.output.resolve() / "source" / f"profile-{profile}"
|
||||
setting = profile_database_setting(profile)
|
||||
prepare_profile_project(project, setting)
|
||||
subprocess.run(
|
||||
[args.compiler.resolve(), "build", project, "--output", binary],
|
||||
stdout=log,
|
||||
stderr=log,
|
||||
check=True,
|
||||
)
|
||||
record_binary(manifest, binary_name, binary, started,
|
||||
profile=profile, fixture="profile")
|
||||
item.update({"status": "built", "binary": fingerprint(binary)})
|
||||
report["profiles"][profile] = item
|
||||
path = args.output.resolve() / "profile-report.json"
|
||||
path.write_text(json.dumps(report, indent=2) + "\n")
|
||||
manifest["files"]["profile-report.json"] = fingerprint(path)
|
||||
|
||||
|
||||
def build(args):
|
||||
postgres_setting = load_postgres_setting() if args.selection == "all" else None
|
||||
settings = configuration(args, postgres_setting)
|
||||
destination = args.output.resolve()
|
||||
destination.mkdir(parents=True, exist_ok=False)
|
||||
compiler = args.compiler.resolve()
|
||||
cargo = shutil.which(str(args.cargo))
|
||||
if cargo is None:
|
||||
raise ValueError(f"cannot find Cargo command: {args.cargo}")
|
||||
args.cargo = Path(cargo)
|
||||
has_protocols = args.selection in ("all", "protocols")
|
||||
peers = ({name: getattr(args, name).resolve()
|
||||
for name in ("network_bench", "live_bench", "async_bench")}
|
||||
if has_protocols else {})
|
||||
required_tools = (compiler, *peers.values())
|
||||
for executable in required_tools:
|
||||
if not executable.is_file():
|
||||
raise ValueError(f"build the executable first: {executable}")
|
||||
# 在昂贵构建之前拒绝旧环境变量协议产物；这里只读配置，不启动网络。
|
||||
for name, peer in peers.items():
|
||||
if name == "async_bench":
|
||||
continue # The finite Rust reference takes explicit positional arguments.
|
||||
with tempfile.TemporaryDirectory(prefix="dever-peer-config-") as temporary:
|
||||
executable = stage_peer(peer, Path(temporary), {"workers": 2})
|
||||
result = subprocess.run([executable, "check-config"], capture_output=True,
|
||||
text=True, check=True, timeout=5)
|
||||
if json.loads(result.stdout) != {"workers": 2}:
|
||||
raise ValueError(f"benchmark peer does not implement {PEER_CONFIGURATION}: {peer}")
|
||||
manifest = {
|
||||
"peer_configuration": PEER_CONFIGURATION,
|
||||
"application_configuration": APPLICATION_CONFIGURATION,
|
||||
"selection": args.selection,
|
||||
"settings": settings,
|
||||
"cargo": subprocess.check_output([cargo, "-V"], text=True),
|
||||
"compiler": fingerprint(compiler),
|
||||
"optimization": "LLVM O2 application objects, LLD static link, selected prebuilt runtime archive",
|
||||
"runtime_scheduling": "Dever uses production runtime defaults; workers/task_capacity configure Rust references only",
|
||||
"binaries": {},
|
||||
"files": {},
|
||||
}
|
||||
with (destination / "build.log").open("w") as log:
|
||||
for fixture, entries in (ENTRIES.items() if has_protocols else ()):
|
||||
source = destination / "source" / fixture
|
||||
prepare_native_project(source, fixture, settings)
|
||||
print(f"build {fixture}", flush=True)
|
||||
started = time.monotonic()
|
||||
executable = source / "app"
|
||||
subprocess.run([compiler, "build", source, "--output", executable],
|
||||
stdout=log, stderr=log, check=True)
|
||||
for entry in entries:
|
||||
binary = destination / entry
|
||||
shutil.copy2(executable, binary)
|
||||
record_binary(manifest, entry, binary, started, profile="base", fixture=fixture,
|
||||
command=f"benchmark.{fixture}.{entry}", shared_application=fixture)
|
||||
drivers = ()
|
||||
if args.selection == "all":
|
||||
drivers = ("sqlite", "postgres") if postgres_setting is not None else ("sqlite",)
|
||||
for driver in drivers:
|
||||
project = destination / "source" / f"orm-{driver}"
|
||||
try:
|
||||
prepare_project(project, "orm", settings, driver, postgres_setting)
|
||||
subprocess.run([compiler, "fmt", project], stdout=log, stderr=log, check=True)
|
||||
for entry in ORM_ENTRIES:
|
||||
name = f"{driver}-{entry.removeprefix('orm_')}"
|
||||
print(f"build {name}", flush=True)
|
||||
binary = destination / name
|
||||
started = time.monotonic()
|
||||
command_entry = "benchmark.item.orm_seed" if entry == "orm_http" else f"benchmark.item.{entry}"
|
||||
stage_application_command(project / "module", command_entry)
|
||||
subprocess.run([compiler, "build", project, "--output", binary],
|
||||
stdout=log, stderr=log, check=True)
|
||||
record_binary(manifest, name, binary, started, profile=driver, fixture="orm",
|
||||
entry=entry)
|
||||
finally:
|
||||
if driver == "postgres":
|
||||
(project / "config/setting.json").unlink(missing_ok=True)
|
||||
cms_sources = {"all": ("dever",), "cms-profiles": ("dever", "md"),
|
||||
"profiles": (), "protocols": ()}[args.selection]
|
||||
for source in cms_sources:
|
||||
cms_project = destination / "source" / f"cms-{source}"
|
||||
prepare_cms_project(cms_project, source)
|
||||
config_name = f"source/cms-{source}/config/setting.json"
|
||||
manifest["files"][config_name] = fingerprint(cms_project / "config/setting.json")
|
||||
binary_name = "cms" if source == "dever" else "cms-md"
|
||||
print(f"build cms-{source}", flush=True)
|
||||
cms_binary = destination / binary_name
|
||||
started = time.monotonic()
|
||||
subprocess.run([compiler, "build", cms_project, "--output", cms_binary],
|
||||
stdout=log, stderr=log, check=True)
|
||||
record_binary(manifest, binary_name, cms_binary, started,
|
||||
profile="sqlite", fixture="cms", source=source)
|
||||
if args.selection != "protocols":
|
||||
build_profile_report(args, manifest, log)
|
||||
for name, peer in peers.items():
|
||||
shutil.copy2(peer, destination / name)
|
||||
manifest["binaries"][name] = fingerprint(destination / name)
|
||||
if has_protocols:
|
||||
shutil.copytree(TLS, destination / "tls")
|
||||
for path in sorted((destination / "tls").rglob("*")):
|
||||
if path.is_file():
|
||||
name = path.relative_to(destination).as_posix()
|
||||
manifest["files"][name] = fingerprint(path)
|
||||
(destination / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n")
|
||||
print(f"artifacts: {destination}", flush=True)
|
||||
@@ -0,0 +1,675 @@
|
||||
"""显式验收双源码 CMS 的真实身份、租户、发布链及专属内存预算。"""
|
||||
|
||||
import argparse
|
||||
from collections import Counter
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from copy import deepcopy
|
||||
import http.client
|
||||
from http.cookies import SimpleCookie
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import subprocess
|
||||
import tempfile
|
||||
from threading import Event
|
||||
import time
|
||||
from urllib.parse import urlencode
|
||||
|
||||
from build import ROOT, fingerprint
|
||||
from process import Budget, Process, observe, summarize, resource_trend
|
||||
from settings import write_private_json
|
||||
|
||||
FORMATS = ("dever", "md")
|
||||
ARTICLE_PATH = "/news/article/admin/manage"
|
||||
ARTICLE_LIST_PATH = f"{ARTICLE_PATH}/list?size=100"
|
||||
REVISION_PATH = "/content/revision/admin/manage/create"
|
||||
SESSION_PATH = "/user/account/{site}/session/login"
|
||||
PUBLICATIONS_PATH = "/news/article/front/browse/list?size=100"
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise RuntimeError(message)
|
||||
|
||||
|
||||
def execute(command, directory, log_name, *, envelope=False, timeout=180):
|
||||
child = subprocess.Popen([str(value) for value in command], cwd=directory,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE, start_new_session=True)
|
||||
failure = None
|
||||
try:
|
||||
stdout, stderr = child.communicate(timeout=timeout)
|
||||
except BaseException as error:
|
||||
failure = error
|
||||
try:
|
||||
signal_group(child.pid, signal.SIGTERM)
|
||||
stdout, stderr = child.communicate(timeout=3)
|
||||
except subprocess.TimeoutExpired:
|
||||
signal_group(child.pid, signal.SIGKILL)
|
||||
stdout, stderr = child.communicate(timeout=3)
|
||||
(directory / f"{log_name}.stdout.log").write_bytes(stdout)
|
||||
(directory / f"{log_name}.stderr.log").write_bytes(stderr)
|
||||
if failure is not None:
|
||||
raise failure
|
||||
require(child.returncode == 0, f"{log_name} exited {child.returncode}: {directory}")
|
||||
if envelope:
|
||||
return decode_response(stdout, 200)
|
||||
|
||||
|
||||
def signal_group(pid, signal_number):
|
||||
try:
|
||||
os.killpg(pid, signal_number)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
|
||||
|
||||
def decode_response(body, status):
|
||||
document = json.loads(body)
|
||||
require(isinstance(document, dict) and set(document) == {"code", "message", "data"},
|
||||
f"unexpected response envelope (HTTP {status})")
|
||||
require(status == 200 and document["code"] == 0,
|
||||
f"request failed: HTTP {status}, code={document['code']}, {document['message']}")
|
||||
return document["data"]
|
||||
|
||||
|
||||
def copy_cms_project(source, destination):
|
||||
destination.mkdir()
|
||||
for name in ("config", "module", "test"):
|
||||
shutil.copytree(source / name, destination / name)
|
||||
|
||||
|
||||
def build_cases(compiler, output):
|
||||
output.mkdir(parents=True, exist_ok=False)
|
||||
cases = {}
|
||||
for source_format in FORMATS:
|
||||
source = ROOT / "examples/cms" / source_format
|
||||
project = output / source_format
|
||||
copy_cms_project(source, project)
|
||||
started = time.monotonic()
|
||||
execute([compiler, "check", project], project, "check")
|
||||
execute([compiler, "build", project, "--output", project / "cms-app"], project, "build", timeout=900)
|
||||
executable = project / "cms-app"
|
||||
cases[source_format] = {
|
||||
"executable": str(executable), "config": str(project / "config/setting.json"),
|
||||
"fingerprint": fingerprint(executable),
|
||||
"build_seconds": time.monotonic() - started,
|
||||
}
|
||||
manifest = {"format": "dever-cms-publish-v1", "cases": cases}
|
||||
write_private_json(output / "manifest.json", manifest)
|
||||
return manifest
|
||||
|
||||
|
||||
def available_port():
|
||||
with socket.socket() as reservation:
|
||||
reservation.bind(("127.0.0.1", 0))
|
||||
return reservation.getsockname()[1]
|
||||
|
||||
|
||||
def isolated_settings(settings, port):
|
||||
settings = deepcopy(settings)
|
||||
settings["runtime"]["mode"] = "api"
|
||||
settings["http"] = {"host": "127.0.0.1", "port": port}
|
||||
settings["auth"]["providers"]["session"]["jwtSecret"] = secrets.token_hex(32)
|
||||
for site in settings["sites"].values():
|
||||
site["origin"] = f"http://127.0.0.1:{port}"
|
||||
database = settings["database"]["default"]
|
||||
if database["type"] == "sqlite":
|
||||
database["path"] = "data/db/platform.db"
|
||||
database["tenant_directory"] = "data/tenants"
|
||||
elif database["type"] == "postgres":
|
||||
require(bool(database.get("url")) and bool(database.get("tenant_database_prefix")),
|
||||
"PostgreSQL CMS acceptance requires an explicit isolated control database and tenant prefix")
|
||||
else:
|
||||
raise RuntimeError("CMS acceptance supports only SQLite and PostgreSQL")
|
||||
settings["log"]["level"] = "error"
|
||||
return settings
|
||||
|
||||
|
||||
class CmsClient:
|
||||
def __init__(self, port):
|
||||
self.connection = http.client.HTTPConnection("127.0.0.1", port, timeout=10)
|
||||
self.origin = f"http://127.0.0.1:{port}"
|
||||
self.cookies = {}
|
||||
self.timings = {}
|
||||
self.record_timings = True
|
||||
|
||||
def request(self, method, path, payload=None, *, site=None, expected=200):
|
||||
headers = {"Content-Type": "application/json", "Origin": self.origin}
|
||||
if site in self.cookies:
|
||||
headers["Cookie"] = self.cookies[site]
|
||||
body = None
|
||||
if payload is not None:
|
||||
if method in ("GET", "DELETE"):
|
||||
query = {name: value if isinstance(value, str) else json.dumps(value)
|
||||
for name, value in payload.items()}
|
||||
path += ("&" if "?" in path else "?") + urlencode(query)
|
||||
else:
|
||||
body = json.dumps(payload)
|
||||
started = time.monotonic()
|
||||
self.connection.request(method, path, body, headers)
|
||||
response = self.connection.getresponse()
|
||||
body = response.read()
|
||||
route = path.split("?")[0]
|
||||
label = f"{method} {route}" + (f" expected {expected}" if expected != 200 else "")
|
||||
if self.record_timings:
|
||||
self.timings.setdefault(label, []).append(
|
||||
(time.monotonic() - started) * 1000)
|
||||
require(response.status == expected,
|
||||
f"{method} {path}: HTTP {response.status}, expected {expected}: {body!r}")
|
||||
if expected != 200:
|
||||
document = json.loads(body)
|
||||
require(document.get("code") != 0, "rejected request returned success")
|
||||
return None
|
||||
data = decode_response(body, response.status)
|
||||
if path.endswith("/login"):
|
||||
cookie = SimpleCookie()
|
||||
for name, value in response.getheaders():
|
||||
if name.lower() == "set-cookie":
|
||||
cookie.load(value)
|
||||
require(bool(cookie), "login did not return a session cookie")
|
||||
self.cookies[site] = "; ".join(f"{name}={value.value}" for name, value in cookie.items())
|
||||
return data
|
||||
|
||||
def close(self):
|
||||
self.connection.close()
|
||||
|
||||
|
||||
def wait_ready(process, client):
|
||||
deadline = time.monotonic() + 20
|
||||
while time.monotonic() < deadline:
|
||||
process.sample()
|
||||
require(process.child.poll() is None, f"CMS exited before readiness: {process.directory}")
|
||||
try:
|
||||
client.request("GET", ARTICLE_LIST_PATH, expected=401)
|
||||
return time.monotonic() - process.started
|
||||
except (ConnectionError, http.client.HTTPException, OSError):
|
||||
client.close()
|
||||
time.sleep(0.02)
|
||||
raise TimeoutError(f"CMS readiness timeout: {process.directory}")
|
||||
|
||||
|
||||
def login_client(client, identity, password, tenant_key):
|
||||
for site in ("admin", "front"):
|
||||
data = client.request("POST", SESSION_PATH.format(site=site), {
|
||||
"credential_id": identity["credential_id"], "password": password, "tenant": tenant_key,
|
||||
}, site=site)
|
||||
require(data["account"]["id"] == identity["user_id"]
|
||||
and data["tenant_id"] == identity["tenant_id"] and data["site"] == site,
|
||||
"login changed the trusted identity or tenant/site")
|
||||
|
||||
|
||||
def publish_workload(client, process, identity, password, articles, *,
|
||||
tenant_key="bench", title_prefix="Article"):
|
||||
login_client(client, identity, password, tenant_key)
|
||||
titles = []
|
||||
ids = []
|
||||
for index in range(articles):
|
||||
slug, title = f"bench-{index}", f"{title_prefix} {index}"
|
||||
created = client.request("POST", f"{ARTICLE_PATH}/create", {
|
||||
"slug": f" {slug.upper()} ", "title": f" {title} ", "body": "Draft body",
|
||||
}, site="admin")
|
||||
require(created["slug"] == slug and created["title"] == title,
|
||||
"article creation did not normalize input")
|
||||
require("author_id" not in created and "status" not in created,
|
||||
"article creation leaked private owner or publication state")
|
||||
changed = client.request("POST", REVISION_PATH, {
|
||||
"selected_article_id": created["id"], "expected_version": created["version"],
|
||||
"title": f" {title} published ", "body": "Published body",
|
||||
}, site="admin")
|
||||
expected_title = f"{title} published"
|
||||
require(changed["title"] == expected_title
|
||||
and changed["article_id"] == created["id"]
|
||||
and changed["version"] == created["version"] + 1,
|
||||
"revision did not preserve normalized content and advance the article version")
|
||||
published = client.request("POST", f"{ARTICLE_PATH}/publish", {
|
||||
"article_id": created["id"],
|
||||
}, site="admin")
|
||||
require(published["slug"] == slug and published["title"] == expected_title
|
||||
and published["body"] == "Published body",
|
||||
"publication did not reflect the stored article")
|
||||
titles.append(expected_title)
|
||||
ids.append(created["id"])
|
||||
process.sample()
|
||||
listing = client.request("GET", PUBLICATIONS_PATH, site="front")
|
||||
require(listing["total"] == articles
|
||||
and sorted(value["title"] for value in listing["items"]) == sorted(titles),
|
||||
"front query did not return every published article exactly once")
|
||||
client.request("POST", f"{ARTICLE_PATH}/publish", {"article_id": ids[0]},
|
||||
site="admin", expected=409)
|
||||
return {"articles": articles, "titles": sorted(titles), "duplicate_publish_status": 409}
|
||||
|
||||
|
||||
def provision_identity(executable, directory, tenant_key):
|
||||
password = secrets.token_urlsafe(24)
|
||||
identity = execute([executable, "user.account.bootstrap", json.dumps({
|
||||
"tenant_key": tenant_key, "tenant_name": f"Benchmark {tenant_key}",
|
||||
"email": f"{tenant_key}@example.invalid", "display_name": "Benchmark",
|
||||
"password": password,
|
||||
})], directory, f"bootstrap-{tenant_key}", envelope=True)
|
||||
execute([executable, "--dever-tenant-migrate", identity["tenant_id"]],
|
||||
directory, f"migrate-{tenant_key}")
|
||||
for site in ("admin", "front"):
|
||||
execute([executable, "--dever-tenant-owner", identity["tenant_id"], site,
|
||||
identity["user_id"]], directory, f"owner-{tenant_key}-{site}")
|
||||
return identity, password
|
||||
|
||||
|
||||
def latency_summary(timings):
|
||||
result = {}
|
||||
for action, samples in timings.items():
|
||||
values = sorted(samples)
|
||||
result[action] = {"samples": len(values), "p50_ms": values[len(values) // 2],
|
||||
"max_ms": values[-1]}
|
||||
return result
|
||||
|
||||
|
||||
def catalog_key(catalog, component, domain, site, action, method):
|
||||
matches = [entry["key"] for entry in catalog
|
||||
if (entry["component"], entry["domain"], entry["site"],
|
||||
entry["action"], entry["method"]) == (component, domain, site, action, method)]
|
||||
require(len(matches) == 1 and bool(matches[0]),
|
||||
f"permission catalog must uniquely identify {component}.{domain}.{site}.{action}")
|
||||
return matches[0]
|
||||
|
||||
|
||||
def run_postgres_rbac(executable, directory, port, owner, identity):
|
||||
catalog = owner.request("GET", "/user/authorization/admin/manage/permissions", site="admin")
|
||||
front_catalog = owner.request("GET", "/user/authorization/front/manage/permissions", site="front")
|
||||
permissions = {
|
||||
"admin_read": catalog_key(catalog, "news", "article", "admin", "list", "GET"),
|
||||
"admin_permissions": catalog_key(catalog, "user", "authorization", "admin", "permissions", "GET"),
|
||||
"front_list": catalog_key(front_catalog, "news", "article", "front", "list", "GET"),
|
||||
}
|
||||
require(len(set(permissions.values())) == 3, "RBAC acceptance reused a permission key")
|
||||
password = secrets.token_urlsafe(24)
|
||||
member = execute([executable, "user.account.fixture_member", json.dumps({
|
||||
"selected_tenant_id": identity["tenant_id"], "email": f"member-{secrets.token_hex(8)}@example.invalid",
|
||||
"display_name": "Acceptance member", "password": password,
|
||||
})], directory, "rbac-member", envelope=True)
|
||||
require(isinstance(member["user_id"], int) and not isinstance(member["user_id"], bool)
|
||||
and member["user_id"] > 0 and member["user_id"] != identity["user_id"]
|
||||
and member["tenant_id"] == identity["tenant_id"],
|
||||
"RBAC acceptance did not create a distinct member in the owner's tenant")
|
||||
probes = (("admin", ARTICLE_LIST_PATH),
|
||||
("admin", "/user/authorization/admin/manage/permissions"),
|
||||
("front", PUBLICATIONS_PATH))
|
||||
client = CmsClient(port)
|
||||
try:
|
||||
login_client(client, member, password, "tenant_a")
|
||||
for site, path in probes:
|
||||
client.request("GET", path, site=site, expected=403)
|
||||
roles = (("admin", "fixture-reader", "admin_read"),
|
||||
("admin", "fixture-catalog", "admin_permissions"),
|
||||
("front", "fixture-reader", "front_list"))
|
||||
for site, role_id, permission in roles:
|
||||
base = f"/user/authorization/{site}/manage"
|
||||
owner.request("POST", f"{base}/save", {
|
||||
"id": role_id, "name": role_id, "all_permissions": False,
|
||||
"permission_keys": [permissions[permission]],
|
||||
}, site=site)
|
||||
owner.request("POST", f"{base}/grant", {
|
||||
"user_id": member["user_id"], "role_id": role_id,
|
||||
}, site=site)
|
||||
for site, path in probes:
|
||||
client.request("GET", path, site=site)
|
||||
owner.request("DELETE", "/user/authorization/admin/manage/revoke", {
|
||||
"user_id": member["user_id"], "role_id": "fixture-reader",
|
||||
}, site="admin")
|
||||
client.request("GET", ARTICLE_LIST_PATH, site="admin", expected=403)
|
||||
client.request("GET", probes[1][1], site="admin")
|
||||
client.request("GET", PUBLICATIONS_PATH, site="front")
|
||||
for site, role_id, path in (("admin", "fixture-catalog", probes[1][1]),
|
||||
("front", "fixture-reader", PUBLICATIONS_PATH)):
|
||||
owner.request("DELETE", f"/user/authorization/{site}/manage/revoke", {
|
||||
"user_id": member["user_id"], "role_id": role_id,
|
||||
}, site=site)
|
||||
client.request("GET", path, site=site, expected=403)
|
||||
finally:
|
||||
client.close()
|
||||
return {"member_user_id": member["user_id"], "permissions": permissions,
|
||||
"before_grant_denied": True, "both_admin_roles_allowed": True,
|
||||
"revoked_role_denied": True, "second_role_still_allowed": True,
|
||||
"front_same_named_role_still_allowed": True}
|
||||
|
||||
|
||||
class ReadLatency:
|
||||
"""固定内存的对数直方图；分位数报告桶上界，最大相邻桶比例约1.011。"""
|
||||
|
||||
def __init__(self):
|
||||
self.buckets = [0] * 2048
|
||||
self.count = 0
|
||||
|
||||
def add(self, milliseconds):
|
||||
index = math.ceil(math.log2(1 + milliseconds) * 64)
|
||||
require(0 <= index < len(self.buckets), "CMS latency exceeds histogram bounds")
|
||||
self.buckets[index] += 1
|
||||
self.count += 1
|
||||
|
||||
def merge(self, other):
|
||||
self.buckets = [left + right for left, right in zip(self.buckets, other.buckets)]
|
||||
self.count += other.count
|
||||
|
||||
def percentile(self, fraction):
|
||||
threshold = math.ceil(self.count * fraction)
|
||||
count = 0
|
||||
for index, samples in enumerate(self.buckets):
|
||||
count += samples
|
||||
if count >= threshold:
|
||||
return 2 ** (index / 64) - 1
|
||||
raise ValueError("empty latency histogram")
|
||||
|
||||
|
||||
def concurrent_reads(port, cookies, expected, process, *, concurrency, seconds, interval):
|
||||
"""Each closed-loop client keeps its own connection and verifies every reply."""
|
||||
stop = Event()
|
||||
started = time.monotonic()
|
||||
deadline = started + seconds
|
||||
first = len(process.samples)
|
||||
|
||||
def read_publications():
|
||||
client = CmsClient(port)
|
||||
client.cookies = cookies.copy()
|
||||
client.record_timings = False
|
||||
latency = ReadLatency()
|
||||
try:
|
||||
while not stop.is_set() and time.monotonic() < deadline:
|
||||
request_started = time.monotonic()
|
||||
listing = client.request("GET", PUBLICATIONS_PATH, site="front")
|
||||
require(listing == expected, "concurrent CMS read changed publication results")
|
||||
latency.add((time.monotonic() - request_started) * 1000)
|
||||
return latency
|
||||
except BaseException:
|
||||
stop.set()
|
||||
raise
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
with ThreadPoolExecutor(max_workers=concurrency) as clients:
|
||||
futures = [clients.submit(read_publications) for _ in range(concurrency)]
|
||||
try:
|
||||
while any(not future.done() for future in futures):
|
||||
process.check()
|
||||
observe([process], interval, interval)
|
||||
elapsed = time.monotonic() - started
|
||||
latency = ReadLatency()
|
||||
for future in futures:
|
||||
latency.merge(future.result())
|
||||
finally:
|
||||
stop.set()
|
||||
require(latency.count > 0, "concurrent CMS read completed without a successful request")
|
||||
require(elapsed >= seconds, "concurrent CMS read ended before the measurement window")
|
||||
process.sample()
|
||||
return {"mode": "closed_loop_authenticated_read", "concurrency": concurrency,
|
||||
"requested_seconds": seconds, "elapsed_seconds": elapsed,
|
||||
"succeeded": latency.count, "errors": 0, "result_checked_on_every_request": True,
|
||||
"actual_success_qps": latency.count / elapsed,
|
||||
"latency_measurement": "request and reply validation; logarithmic bucket upper bound, 64 buckets per octave",
|
||||
"request_latency_ms": {name: latency.percentile(fraction)
|
||||
for name, fraction in (("p50", .5), ("p95", .95), ("p99", .99))},
|
||||
"resources": summarize(process.samples[first:]),
|
||||
"trend": resource_trend(process.samples[first:]) if len(process.samples[first:]) >= 4 else None}
|
||||
|
||||
|
||||
def sqlite_pool_descriptor_limits(settings, directory):
|
||||
database = settings["database"]["default"]
|
||||
require(database["type"] == "sqlite", "CMS read benchmark requires isolated SQLite")
|
||||
databases = [directory / database["path"]]
|
||||
databases.extend((directory / database["tenant_directory"]).glob("*.db"))
|
||||
# 每个连接可持有主文件与WAL；同进程各连接共用一个SHM句柄。
|
||||
return {str(path.resolve()) + suffix: limit for path in databases
|
||||
for suffix, limit in (("", database["max_connections"]),
|
||||
("-wal", database["max_connections"]), ("-shm", 1))}
|
||||
|
||||
|
||||
def validate_read_descriptor_recovery(load):
|
||||
before = Counter(load["descriptors_before"].values())
|
||||
after = Counter(load["descriptors_after"].values())
|
||||
for path, limit in load["pooled_descriptor_limits"].items():
|
||||
require(after[path] <= limit, f"CMS SQLite descriptor count exceeds pool limit: {path}")
|
||||
before.pop(path, None)
|
||||
after.pop(path, None)
|
||||
require(before == after,
|
||||
f"CMS read descriptors did not recover: added={dict(after - before)}, missing={dict(before - after)}")
|
||||
|
||||
|
||||
def run_case(case, directory, *, articles, cgroup_parent, memory_mib,
|
||||
idle_seconds=0.2, interval=0.05, server_cpu=None, cpu_quota=None,
|
||||
concurrency=0, read_seconds=5):
|
||||
executable = Path(case["executable"])
|
||||
require(fingerprint(executable) == case["fingerprint"], "CMS artifact fingerprint changed")
|
||||
directory.mkdir(parents=True, exist_ok=False)
|
||||
target = directory / "cms-app"
|
||||
shutil.copy2(executable, target)
|
||||
(directory / "config").mkdir()
|
||||
(directory / "data").mkdir()
|
||||
port = available_port()
|
||||
settings = isolated_settings(json.loads(Path(case["config"]).read_text()), port)
|
||||
write_private_json(directory / "config/setting.json", settings)
|
||||
identity, password = provision_identity(target, directory, "bench")
|
||||
budget = Budget(cgroup_parent, memory_mib, cpu_quota)
|
||||
client = CmsClient(port)
|
||||
failure = None
|
||||
result = {}
|
||||
try:
|
||||
with Process([target], directory / "server", budget=budget,
|
||||
cpus=(server_cpu,) if server_cpu is not None else None) as process:
|
||||
result["raw"] = process.samples
|
||||
ready_seconds = wait_ready(process, client)
|
||||
started = time.monotonic()
|
||||
contract = publish_workload(client, process, identity, password, articles)
|
||||
elapsed = time.monotonic() - started
|
||||
result.update(contract=contract, ready_seconds=ready_seconds,
|
||||
workload_seconds=elapsed, published_articles_per_second=articles / elapsed,
|
||||
latency=latency_summary(client.timings))
|
||||
if concurrency:
|
||||
expected = client.request("GET", PUBLICATIONS_PATH, site="front")
|
||||
# 控制连接不参与长读负载，先关闭，避免其空闲超时改变基线。
|
||||
client.close()
|
||||
observe([process], 0.3, interval)
|
||||
descriptors_before = process.descriptors()
|
||||
pooled_limits = sqlite_pool_descriptor_limits(settings, directory)
|
||||
read_load = {"descriptors_before": descriptors_before,
|
||||
"pooled_descriptor_limits": pooled_limits}
|
||||
result["concurrent_reads"] = read_load
|
||||
read_load.update(concurrent_reads(port, client.cookies, expected, process,
|
||||
concurrency=concurrency, seconds=read_seconds, interval=interval))
|
||||
observe([process], 0.3, interval)
|
||||
descriptors_after = process.descriptors()
|
||||
read_load.update(descriptors_after=descriptors_after,
|
||||
fd_delta_after_disconnect=len(descriptors_after) - len(descriptors_before))
|
||||
validate_read_descriptor_recovery(read_load)
|
||||
observe([process], idle_seconds, interval)
|
||||
process.sample()
|
||||
process.check()
|
||||
require(process.child.poll() is None, "CMS exited during the publish workload")
|
||||
resources = summarize(process.samples)
|
||||
result.update(resources=resources, budget_before_stop=budget.sample())
|
||||
except BaseException as error:
|
||||
failure = error
|
||||
finally:
|
||||
client.close()
|
||||
try:
|
||||
budget.close()
|
||||
except BaseException as error:
|
||||
if failure is None:
|
||||
failure = error
|
||||
else:
|
||||
failure.add_note(f"cannot close owned CMS budget: {error}")
|
||||
result["cgroup"] = budget.final
|
||||
if failure is None and budget.final is not None and (
|
||||
budget.final["events"]["oom"] or budget.final["events"]["oom_kill"]):
|
||||
failure = RuntimeError("CMS exceeded its memory budget")
|
||||
result["status"] = "failed" if failure else "passed"
|
||||
if failure is not None:
|
||||
result["error"] = str(failure)
|
||||
write_private_json(directory / "report.json", result)
|
||||
if failure is not None:
|
||||
raise failure
|
||||
return result
|
||||
|
||||
|
||||
def run_postgres_case(project, output, compiler, *, articles, rbac_fixture=False):
|
||||
# 构建、临时密钥和端口只写测试副本，调用者的项目配置始终保持不变。
|
||||
with tempfile.TemporaryDirectory(prefix="dever-cms-postgres-") as temporary:
|
||||
staged = Path(temporary) / "project"
|
||||
copy_cms_project(project, staged)
|
||||
return run_staged_postgres_case(staged, output, compiler, articles=articles,
|
||||
rbac_fixture=rbac_fixture)
|
||||
|
||||
|
||||
def run_staged_postgres_case(project, output, compiler, *, articles, rbac_fixture):
|
||||
require(1 <= articles <= 100, "PostgreSQL CMS acceptance requires 1..100 articles")
|
||||
output.mkdir(parents=True, exist_ok=False)
|
||||
report = {"status": "running", "source": "dever", "tenants": {},
|
||||
"scope": "PostgreSQL HTTP owner flow and physical database tenant isolation; not non-owner RBAC revocation"}
|
||||
if rbac_fixture:
|
||||
report["scope"] = "PostgreSQL HTTP owner flow, physical database tenant isolation and non-owner multi-role/site grant-revoke"
|
||||
clients = {}
|
||||
try:
|
||||
setting_path = project / "config/setting.json"
|
||||
initial = json.loads(setting_path.read_text())
|
||||
require(initial["database"]["default"]["type"] == "postgres",
|
||||
"postgres-case requires a PostgreSQL project configuration")
|
||||
port = available_port()
|
||||
write_private_json(setting_path, isolated_settings(initial, port), replace=True)
|
||||
execute([compiler, "check", project], output, "check")
|
||||
executable = project / "cms-app"
|
||||
execute([compiler, "build", project, "--output", executable], output, "build", timeout=900)
|
||||
report["binary"] = fingerprint(executable)
|
||||
identities = {key: provision_identity(executable, output, key)
|
||||
for key in ("tenant_a", "tenant_b")}
|
||||
require(identities["tenant_a"][0]["tenant_id"] != identities["tenant_b"][0]["tenant_id"],
|
||||
"PostgreSQL acceptance reused a tenant identity")
|
||||
clients = {key: CmsClient(port) for key in identities}
|
||||
with Process([executable], output / "server") as process:
|
||||
report["ready_seconds"] = wait_ready(process, clients["tenant_a"])
|
||||
for key, (identity, password) in identities.items():
|
||||
contract = publish_workload(clients[key], process, identity, password, articles,
|
||||
tenant_key=key, title_prefix=key)
|
||||
permissions = clients[key].request(
|
||||
"GET", "/user/authorization/admin/manage/permissions", site="admin")
|
||||
require(bool(permissions), "CMS permission catalog is empty")
|
||||
report["tenants"][key] = {"tenant_id": identity["tenant_id"],
|
||||
"user_id": identity["user_id"], "contract": contract}
|
||||
for key, client in clients.items():
|
||||
listing = client.request("GET", PUBLICATIONS_PATH, site="front")
|
||||
contract = report["tenants"][key]["contract"]
|
||||
require(listing["total"] == articles
|
||||
and sorted(value["title"] for value in listing["items"]) == contract["titles"],
|
||||
"PostgreSQL tenant query leaked another tenant's articles")
|
||||
require(sorted(value["slug"] for value in listing["items"])
|
||||
== sorted(f"bench-{index}" for index in range(articles)),
|
||||
"PostgreSQL tenants did not preserve the same slug namespace")
|
||||
report["tenants"][key]["slugs"] = sorted(value["slug"] for value in listing["items"])
|
||||
identity, password = identities["tenant_a"]
|
||||
clients["tenant_a"].request("POST", SESSION_PATH.format(site="admin"), {
|
||||
"credential_id": identity["credential_id"], "password": password, "tenant": "tenant_b",
|
||||
}, site="admin", expected=401)
|
||||
front_cookie = clients["tenant_a"].cookies["front"]
|
||||
clients["tenant_a"].cookies["front"] = clients["tenant_a"].cookies["admin"]
|
||||
clients["tenant_a"].request("GET", PUBLICATIONS_PATH, site="front", expected=401)
|
||||
clients["tenant_a"].cookies["front"] = front_cookie
|
||||
if rbac_fixture:
|
||||
report["rbac"] = run_postgres_rbac(executable, output, port,
|
||||
clients["tenant_a"], identity)
|
||||
process.sample()
|
||||
process.check()
|
||||
require(process.child.poll() is None, "PostgreSQL CMS exited during tenant acceptance")
|
||||
report["resources"] = summarize(process.samples)
|
||||
report["cross_site_cookie_rejected"] = True
|
||||
report["cross_tenant_login_rejected"] = True
|
||||
report["status"] = "passed"
|
||||
except BaseException as error:
|
||||
report["status"] = "failed"
|
||||
report["error"] = str(error)
|
||||
raise
|
||||
finally:
|
||||
for client in clients.values():
|
||||
client.close()
|
||||
write_private_json(output / "report.json", report)
|
||||
return report
|
||||
|
||||
|
||||
def run_cases(manifest, output, *, articles, cgroup_parent, memory_mib,
|
||||
concurrency=0, read_seconds=5):
|
||||
require(manifest["format"] == "dever-cms-publish-v1", "unexpected CMS manifest format")
|
||||
require(1 <= articles <= 100, "articles must be 1..100")
|
||||
require(0 <= concurrency <= 64, "read concurrency must be 0..64")
|
||||
require(math.isfinite(read_seconds) and 0.1 <= read_seconds <= 3600,
|
||||
"read seconds must be finite and within 0.1..3600")
|
||||
require((memory_mib is None) == (cgroup_parent is None),
|
||||
"memory budget and cgroup parent must be provided together")
|
||||
output.mkdir(parents=True, exist_ok=False)
|
||||
report = {"format": manifest["format"], "memory_mib": memory_mib,
|
||||
"scope": "owned loopback HTTP/1, sequential publish, isolated SQLite; not a saturation result",
|
||||
"cases": {}}
|
||||
if concurrency:
|
||||
report["scope"] = "owned loopback HTTP/1, sequential publish and closed-loop authenticated reads, isolated SQLite"
|
||||
try:
|
||||
for source_format in FORMATS:
|
||||
report["cases"][source_format] = run_case(manifest["cases"][source_format],
|
||||
output / source_format, articles=articles,
|
||||
cgroup_parent=cgroup_parent, memory_mib=memory_mib,
|
||||
concurrency=concurrency, read_seconds=read_seconds)
|
||||
require(report["cases"]["dever"]["contract"] == report["cases"]["md"]["contract"],
|
||||
"Dever and Markdown CMS changed the publish contract")
|
||||
report["status"] = "passed"
|
||||
except BaseException as error:
|
||||
report["status"] = "failed"
|
||||
report["error"] = str(error)
|
||||
raise
|
||||
finally:
|
||||
for source_format in FORMATS:
|
||||
case_report = output / source_format / "report.json"
|
||||
if source_format not in report["cases"] and case_report.is_file():
|
||||
report["cases"][source_format] = json.loads(case_report.read_text())
|
||||
write_private_json(output / "report.json", report)
|
||||
return report
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
actions = parser.add_subparsers(dest="action", required=True)
|
||||
build = actions.add_parser("build")
|
||||
build.add_argument("--compiler", type=Path, default=ROOT / "target/debug/dever")
|
||||
build.add_argument("--output", type=Path, required=True)
|
||||
run = actions.add_parser("run")
|
||||
run.add_argument("--manifest", type=Path, required=True)
|
||||
run.add_argument("--output", type=Path, required=True)
|
||||
run.add_argument("--articles", type=int, default=16)
|
||||
run.add_argument("--cgroup-parent", type=Path)
|
||||
run.add_argument("--memory-mib", type=int, choices=(64, 128))
|
||||
run.add_argument("--concurrency", type=int, default=0,
|
||||
help="0 disables load; otherwise 1..64 independent authenticated read clients")
|
||||
run.add_argument("--read-seconds", type=float, default=5)
|
||||
postgres = actions.add_parser("postgres-case")
|
||||
postgres.add_argument("--project", type=Path, required=True)
|
||||
postgres.add_argument("--output", type=Path, required=True)
|
||||
postgres.add_argument("--compiler", type=Path, default=ROOT / "target/debug/dever")
|
||||
postgres.add_argument("--articles", type=int, default=2)
|
||||
postgres.add_argument("--rbac-fixture", action="store_true",
|
||||
help="run test-only member CMD provided by the Rust PostgreSQL fixture")
|
||||
args = parser.parse_args()
|
||||
if args.action == "build":
|
||||
result = build_cases(args.compiler.resolve(strict=True), args.output.resolve())
|
||||
elif args.action == "postgres-case":
|
||||
report = run_postgres_case(args.project.resolve(strict=True), args.output.resolve(),
|
||||
args.compiler.resolve(strict=True), articles=args.articles,
|
||||
rbac_fixture=args.rbac_fixture)
|
||||
result = {"status": report["status"], "report": str(args.output.resolve() / "report.json")}
|
||||
else:
|
||||
result = run_cases(json.loads(args.manifest.read_text()), args.output.resolve(),
|
||||
articles=args.articles, cgroup_parent=args.cgroup_parent,
|
||||
memory_mib=args.memory_mib, concurrency=args.concurrency,
|
||||
read_seconds=args.read_seconds)
|
||||
result = {"status": result["status"], "memory_mib": result["memory_mib"],
|
||||
"report": str(args.output / "report.json")}
|
||||
print(json.dumps(result, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,21 @@
|
||||
ready(port: Int) () {
|
||||
dever.io.println("READY|" + int.to_text(port))
|
||||
}
|
||||
|
||||
pause(millis: Int) () {
|
||||
dever.task.sleep(millis)
|
||||
}
|
||||
|
||||
report(
|
||||
begin: Int,
|
||||
finish: Int,
|
||||
name: Text,
|
||||
count: Int,
|
||||
checksum: Int
|
||||
) () recover("基准失败输出由运行器检查") {
|
||||
dever.io.println(
|
||||
"SAMPLE|" + name + "|" + int.to_text(count) + "|" + int.to_text(finish - begin) + "|" + int.to_text(
|
||||
checksum
|
||||
)
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
type Content {
|
||||
body: Bytes
|
||||
kind: Text
|
||||
}
|
||||
|
||||
listening(result: dever.net.ListenResult.Listening(listener)) (
|
||||
value: dever.system.Listener
|
||||
) {
|
||||
value = listener
|
||||
}
|
||||
|
||||
listening(result: dever.net.ListenResult.Failed(message)) (value: dever.system.Listener) {
|
||||
fail(app.WorkerFailure.Failed(message))
|
||||
}
|
||||
|
||||
bound(result: dever.net.PortResult.Bound(port)) (value: Int) {
|
||||
value = port
|
||||
}
|
||||
|
||||
bound(result: dever.net.PortResult.Failed(message)) (value: Int) {
|
||||
fail(app.WorkerFailure.Failed(message))
|
||||
}
|
||||
|
||||
paused(result: dever.time.SleepResult.Done) () {}
|
||||
|
||||
paused(result: dever.time.SleepResult.Failed(message)) () {
|
||||
fail(app.WorkerFailure.Failed(message))
|
||||
}
|
||||
|
||||
closed(result: dever.io.CloseResult.Closed) () {}
|
||||
|
||||
closed(result: dever.io.CloseResult.Failed(message)) () {
|
||||
fail(app.WorkerFailure.Failed(message))
|
||||
}
|
||||
|
||||
server_tls(result: dever.tls.ServerResult.Ready(tls)) (value: dever.system.ServerTls) {
|
||||
value = tls
|
||||
}
|
||||
|
||||
server_tls(result: dever.tls.ServerResult.Failed(message)) (
|
||||
value: dever.system.ServerTls
|
||||
) {
|
||||
fail(app.WorkerFailure.Failed(message))
|
||||
}
|
||||
|
||||
response_value(content: Content) (result: dever.http.Response) {
|
||||
result = dever.http.Response {
|
||||
status = 200
|
||||
headers = [dever.http.header("Content-Type", content.kind)]
|
||||
body = content.body
|
||||
}
|
||||
}
|
||||
|
||||
route(request: dever.http.Request, large: Bytes) (response: dever.http.Response) {
|
||||
response = response_value(select(request.target, large))
|
||||
}
|
||||
|
||||
select(path: "/plain", large: Bytes) (result: Content) {
|
||||
result = Content {
|
||||
body = dever.bytes.from_text("hello")
|
||||
kind = "text/plain"
|
||||
}
|
||||
}
|
||||
|
||||
select(path: "/json", large: Bytes) (result: Content) {
|
||||
result = Content {
|
||||
body = dever.bytes.from_text("{\"ok\":true}")
|
||||
kind = "application/json"
|
||||
}
|
||||
}
|
||||
|
||||
select(path: "/bytes", large: Bytes) (result: Content) {
|
||||
result = Content {
|
||||
body = large
|
||||
kind = "application/octet-stream"
|
||||
}
|
||||
}
|
||||
|
||||
select(path: other, large: Bytes) (result: Content) {
|
||||
result = Content {
|
||||
body = dever.bytes.from_text("unknown benchmark path")
|
||||
kind = "text/plain"
|
||||
}
|
||||
}
|
||||
|
||||
serve(
|
||||
listener: dever.system.Listener,
|
||||
tls: null,
|
||||
limits: dever.http.Limits,
|
||||
large: Bytes
|
||||
) () {
|
||||
dever.system.http_serve(route, listener, limits, large)
|
||||
}
|
||||
|
||||
serve(
|
||||
listener: dever.system.Listener,
|
||||
tls: dever.system.ServerTls,
|
||||
limits: dever.http.Limits,
|
||||
large: Bytes
|
||||
) () {
|
||||
dever.system.http_serve_tls(route, listener, limits, tls, large)
|
||||
}
|
||||
|
||||
worker(
|
||||
tls: dever.system.ServerTls?,
|
||||
limits: dever.http.Limits,
|
||||
large: Bytes,
|
||||
lifetime: Int
|
||||
) () {
|
||||
listener = listening(result(dever.net.listen("127.0.0.1", 0)))
|
||||
port = bound(result(dever.net.port(listener)))
|
||||
server = run(serve(listener, tls, limits, large))
|
||||
blocking(dever.io.println("READY|" + int.to_text(port)))
|
||||
paused(result(dever.task.sleep(lifetime)))
|
||||
closed(result(dever.net.close_listener(listener)))
|
||||
stop(server)
|
||||
}
|
||||
|
||||
port.plain(limits: dever.http.Limits, large: Bytes, lifetime: Int) () {
|
||||
worker(null, limits, large, lifetime)
|
||||
}
|
||||
|
||||
port.secure(
|
||||
limits: dever.http.Limits,
|
||||
large: Bytes,
|
||||
lifetime: Int,
|
||||
certificate: Bytes,
|
||||
key: Bytes
|
||||
) () {
|
||||
worker(
|
||||
server_tls(result(dever.tls.server(certificate, key))),
|
||||
limits,
|
||||
large,
|
||||
lifetime
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
cmd http = app.http
|
||||
|
||||
cmd https = app.https
|
||||
|
||||
cmd http2 = app.http2
|
||||
|
||||
cmd https2 = app.https2
|
||||
@@ -0,0 +1,35 @@
|
||||
type WorkerFailure {
|
||||
error Failed(message: Text)
|
||||
}
|
||||
|
||||
http() (result: Bool) {
|
||||
port.plain(config.limits(), config.large_body(), config.lifetime())
|
||||
result = true
|
||||
}
|
||||
|
||||
https() (result: Bool) {
|
||||
port.secure(
|
||||
config.limits(),
|
||||
config.large_body(),
|
||||
config.lifetime(),
|
||||
config.certificate(),
|
||||
config.key()
|
||||
)
|
||||
result = true
|
||||
}
|
||||
|
||||
http2() (result: Bool) {
|
||||
port.plain(config.http2_limits(), config.large_body(), config.lifetime())
|
||||
result = true
|
||||
}
|
||||
|
||||
https2() (result: Bool) {
|
||||
port.secure(
|
||||
config.http2_limits(),
|
||||
config.large_body(),
|
||||
config.lifetime(),
|
||||
config.certificate(),
|
||||
config.key()
|
||||
)
|
||||
result = true
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
plain(limits: dever.http.Limits, large: Bytes, lifetime: Int) () fails app.WorkerFailure
|
||||
|
||||
secure(
|
||||
limits: dever.http.Limits,
|
||||
large: Bytes,
|
||||
lifetime: Int,
|
||||
certificate: Bytes,
|
||||
key: Bytes
|
||||
) () fails app.WorkerFailure
|
||||
@@ -0,0 +1,213 @@
|
||||
setting {
|
||||
limits: dever.http.Limits
|
||||
lifetime_ms: Int
|
||||
}
|
||||
|
||||
type Service {
|
||||
Tcp
|
||||
Live
|
||||
}
|
||||
|
||||
listening(result: dever.net.ListenResult.Listening(listener)) (
|
||||
value: dever.system.Listener
|
||||
) {
|
||||
value = listener
|
||||
}
|
||||
|
||||
listening(result: dever.net.ListenResult.Failed(message)) (value: dever.system.Listener) {
|
||||
fail(app.WorkerFailure.Failed(message))
|
||||
}
|
||||
|
||||
bound(result: dever.net.PortResult.Bound(port)) (value: Int) {
|
||||
value = port
|
||||
}
|
||||
|
||||
bound(result: dever.net.PortResult.Failed(message)) (value: Int) {
|
||||
fail(app.WorkerFailure.Failed(message))
|
||||
}
|
||||
|
||||
written(result: dever.io.WriteResult.Done) () {}
|
||||
|
||||
written(result: dever.io.WriteResult.Failed(message)) () {
|
||||
fail(app.WorkerFailure.Failed(message))
|
||||
}
|
||||
|
||||
closed(result: dever.io.CloseResult.Closed) () {}
|
||||
|
||||
closed(result: dever.io.CloseResult.Failed(message)) () {
|
||||
fail(app.WorkerFailure.Failed(message))
|
||||
}
|
||||
|
||||
paused(result: dever.time.SleepResult.Done) () {}
|
||||
|
||||
paused(result: dever.time.SleepResult.Failed(message)) () {
|
||||
fail(app.WorkerFailure.Failed(message))
|
||||
}
|
||||
|
||||
tcp_chunk(event: dever.io.ReadEvent.Chunk(bytes), socket: dever.system.Socket) () recover("TCP 连接错误由运行器检查") {
|
||||
written(result(dever.net.write(socket, bytes)))
|
||||
}
|
||||
|
||||
tcp_chunk(event: dever.io.ReadEvent.Failed(message), socket: dever.system.Socket) () recover("TCP 连接错误由运行器检查") {
|
||||
blocking(dever.io.println("ERROR|" + message))
|
||||
}
|
||||
|
||||
tcp_stream(
|
||||
result: dever.net.ReadStreamResult.Streaming(stream),
|
||||
socket: dever.system.Socket
|
||||
) () {
|
||||
each(tcp_chunk, stream, socket)
|
||||
}
|
||||
|
||||
tcp_stream(
|
||||
result: dever.net.ReadStreamResult.Failed(message),
|
||||
socket: dever.system.Socket
|
||||
) () {
|
||||
fail(app.WorkerFailure.Failed(message))
|
||||
}
|
||||
|
||||
tcp_connection(
|
||||
event: dever.net.ConnectResult.Connected(socket),
|
||||
limits: dever.http.Limits
|
||||
) () recover("TCP 连接错误由运行器检查") {
|
||||
written(result(dever.net.timeout(socket, limits.timeout_ms)))
|
||||
tcp_stream(result(dever.net.chunks(socket, 4096)), socket)
|
||||
closed(result(dever.net.close(socket)))
|
||||
}
|
||||
|
||||
tcp_connection(
|
||||
event: dever.net.ConnectResult.Failed(message),
|
||||
limits: dever.http.Limits
|
||||
) () recover("TCP 连接错误由运行器检查") {
|
||||
blocking(dever.io.println("ERROR|" + message))
|
||||
}
|
||||
|
||||
tcp_server(listener: dever.system.Listener, limits: dever.http.Limits) () {
|
||||
stream = dever.net.connections(listener)
|
||||
parallel_each(tcp_connection, stream, limits.connections, limits)
|
||||
}
|
||||
|
||||
websocket_message(
|
||||
event: dever.websocket.MessageEvent.Read(message),
|
||||
socket: dever.system.WebSocket
|
||||
) () recover("WebSocket 连接错误由运行器检查") {
|
||||
websocket_echo(message, socket)
|
||||
}
|
||||
|
||||
websocket_message(
|
||||
event: dever.websocket.MessageEvent.Failed(message),
|
||||
socket: dever.system.WebSocket
|
||||
) () recover("WebSocket 连接错误由运行器检查") {
|
||||
blocking(dever.io.println("ERROR|" + message))
|
||||
}
|
||||
|
||||
websocket_echo(
|
||||
message: dever.websocket.Message.Text(value),
|
||||
socket: dever.system.WebSocket
|
||||
) () recover("WebSocket 连接错误由运行器检查") {
|
||||
written(result(dever.websocket.send(socket, dever.websocket.Message.Text(value))))
|
||||
}
|
||||
|
||||
websocket_echo(
|
||||
message: dever.websocket.Message.Binary(value),
|
||||
socket: dever.system.WebSocket
|
||||
) () recover("WebSocket 连接错误由运行器检查") {
|
||||
written(result(dever.websocket.send(socket, dever.websocket.Message.Binary(value))))
|
||||
}
|
||||
|
||||
websocket_echo(message: other, socket: dever.system.WebSocket) () recover("WebSocket 连接错误由运行器检查") {}
|
||||
|
||||
websocket_limits() (result: dever.websocket.Limits) {
|
||||
result = dever.websocket.Limits {
|
||||
message_bytes = 4096
|
||||
idle_ms = setting.lifetime_ms
|
||||
write_ms = setting.limits.timeout_ms
|
||||
}
|
||||
}
|
||||
|
||||
websocket_connection(result: dever.websocket.ConnectResult.Connected(socket)) () {
|
||||
each(websocket_message, dever.websocket.messages(socket), socket)
|
||||
}
|
||||
|
||||
websocket_connection(result: dever.websocket.ConnectResult.Failed(message)) () {
|
||||
fail(app.WorkerFailure.Failed(message))
|
||||
}
|
||||
|
||||
route(request: dever.http.Request, reply: dever.system.HttpReply) () {
|
||||
live_path(request.target, reply)
|
||||
}
|
||||
|
||||
live_path(target: "/ws", reply: dever.system.HttpReply) () recover("实时连接错误由运行器检查") {
|
||||
websocket_connection(result(dever.websocket.accept(reply, websocket_limits())))
|
||||
}
|
||||
|
||||
live_path(target: "/events", reply: dever.system.HttpReply) () recover("实时连接错误由运行器检查") {
|
||||
written(result(dever.sse.start(reply, [])))
|
||||
written(
|
||||
result(
|
||||
dever.sse.send(
|
||||
reply,
|
||||
dever.sse.Event {
|
||||
event = "live"
|
||||
data = "ready"
|
||||
id = "1"
|
||||
retry_ms = null
|
||||
}
|
||||
)
|
||||
)
|
||||
)
|
||||
paused(result(dever.task.sleep(setting.lifetime_ms)))
|
||||
}
|
||||
|
||||
live_path(target: other, reply: dever.system.HttpReply) () recover("实时连接错误由运行器检查") {
|
||||
written(
|
||||
result(
|
||||
dever.http.respond(
|
||||
reply,
|
||||
dever.http.Response {
|
||||
status = 404
|
||||
headers = []
|
||||
body = dever.bytes.from_text("unknown live benchmark path")
|
||||
}
|
||||
)
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
live_limits() (result: dever.http.LiveLimits) {
|
||||
result = dever.http.LiveLimits {
|
||||
chunk_bytes = 4096
|
||||
idle_ms = setting.lifetime_ms
|
||||
heartbeat_ms = 100
|
||||
}
|
||||
}
|
||||
|
||||
live_server(listener: dever.system.Listener) () {
|
||||
dever.system.http_serve_live(route, listener, setting.limits, live_limits())
|
||||
}
|
||||
|
||||
serve(service: Service.Tcp, listener: dever.system.Listener) () {
|
||||
tcp_server(listener, setting.limits)
|
||||
}
|
||||
|
||||
serve(service: Service.Live, listener: dever.system.Listener) () {
|
||||
live_server(listener)
|
||||
}
|
||||
|
||||
worker(service: Service) () {
|
||||
listener = listening(result(dever.net.listen("127.0.0.1", 0)))
|
||||
port = bound(result(dever.net.port(listener)))
|
||||
server = run(serve(service, listener))
|
||||
blocking(dever.io.println("READY|" + int.to_text(port)))
|
||||
paused(result(dever.task.sleep(setting.lifetime_ms)))
|
||||
stop(server)
|
||||
closed(result(dever.net.close_listener(listener)))
|
||||
}
|
||||
|
||||
port.tcp() () {
|
||||
worker(Service.Tcp)
|
||||
}
|
||||
|
||||
port.live() () {
|
||||
worker(Service.Live)
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
cmd tcp = app.tcp
|
||||
|
||||
cmd ws = app.ws
|
||||
|
||||
cmd sse = app.sse
|
||||
@@ -0,0 +1,18 @@
|
||||
type WorkerFailure {
|
||||
error Failed(message: Text)
|
||||
}
|
||||
|
||||
tcp() (result: Bool) {
|
||||
port.tcp()
|
||||
result = true
|
||||
}
|
||||
|
||||
ws() (result: Bool) {
|
||||
port.live()
|
||||
result = true
|
||||
}
|
||||
|
||||
sse() (result: Bool) {
|
||||
port.live()
|
||||
result = true
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
tcp() () fails app.WorkerFailure
|
||||
|
||||
live() () fails app.WorkerFailure
|
||||
@@ -0,0 +1,144 @@
|
||||
type Identity {
|
||||
id: Text
|
||||
user_id: model.id?
|
||||
tenant_id: model.id?
|
||||
}
|
||||
|
||||
verify(claims: dever.auth.Claims) (identity: Identity) {
|
||||
identity = Identity {
|
||||
id = claims.subject
|
||||
user_id = null
|
||||
tenant_id = null
|
||||
}
|
||||
}
|
||||
|
||||
ready() () {
|
||||
blocking(bench.ready(0))
|
||||
}
|
||||
|
||||
populate_one(value: Int) () {
|
||||
created = model.create({
|
||||
value = value
|
||||
label = "fixture"
|
||||
})
|
||||
}
|
||||
|
||||
populate() () {
|
||||
each(populate_one, config.orm_rows())
|
||||
}
|
||||
|
||||
crud_once(value: Int) () {
|
||||
begin = dever.time.monotonic_nanos()
|
||||
created = model.create({
|
||||
value = value
|
||||
label = "created"
|
||||
})
|
||||
changed = model.update(created.id, { label = "updated" })
|
||||
removed = model.delete(created.id)
|
||||
end = dever.time.monotonic_nanos()
|
||||
blocking(bench.report(begin, end, "crud", 1, changed + removed))
|
||||
}
|
||||
|
||||
list_once(value: Int) () {
|
||||
begin = dever.time.monotonic_nanos()
|
||||
page = model.list({
|
||||
order = id.asc
|
||||
page = 1
|
||||
size = 32
|
||||
})
|
||||
end = dever.time.monotonic_nanos()
|
||||
blocking(bench.report(begin, end, "list", 1, page.total + length(page.items)))
|
||||
}
|
||||
|
||||
cursor_once(value: Int) () {
|
||||
begin = dever.time.monotonic_nanos()
|
||||
page = model.cursor({
|
||||
order = id.asc
|
||||
size = 32
|
||||
})
|
||||
end = dever.time.monotonic_nanos()
|
||||
blocking(bench.report(begin, end, "cursor", 1, length(page.items)))
|
||||
}
|
||||
|
||||
count_item(value: model.Item, total: Int) (next: Int) {
|
||||
next = total + 1
|
||||
}
|
||||
|
||||
stream_once(value: Int) () {
|
||||
begin = dever.time.monotonic_nanos()
|
||||
rows = model.stream({
|
||||
order = id.asc
|
||||
size = 16
|
||||
})
|
||||
total = reduce(count_item, rows, 0)
|
||||
end = dever.time.monotonic_nanos()
|
||||
blocking(bench.report(begin, end, "stream", 1, total))
|
||||
}
|
||||
|
||||
pool_once(value: Int) () {
|
||||
begin = dever.time.monotonic_nanos()
|
||||
total = model.count()
|
||||
end = dever.time.monotonic_nanos()
|
||||
blocking(bench.report(begin, end, "pool", 1, total))
|
||||
}
|
||||
|
||||
transaction fill_cancelled_transaction() () {
|
||||
created = model.create({
|
||||
value = 1
|
||||
label = "cancelled"
|
||||
})
|
||||
delayed = model.delay(config.orm_cancel_work())
|
||||
}
|
||||
|
||||
orm_idle() (result: Bool) {
|
||||
total = model.count()
|
||||
blocking(bench.ready(total))
|
||||
bench.pause(config.lifetime())
|
||||
result = true
|
||||
}
|
||||
|
||||
orm_crud() (result: Bool) {
|
||||
ready()
|
||||
each(crud_once, config.orm_input())
|
||||
result = true
|
||||
}
|
||||
|
||||
orm_list() (result: Bool) {
|
||||
populate()
|
||||
ready()
|
||||
each(list_once, config.orm_input())
|
||||
result = true
|
||||
}
|
||||
|
||||
orm_cursor() (result: Bool) {
|
||||
populate()
|
||||
ready()
|
||||
each(cursor_once, config.orm_input())
|
||||
result = true
|
||||
}
|
||||
|
||||
orm_stream() (result: Bool) {
|
||||
ready()
|
||||
each(stream_once, config.orm_input())
|
||||
result = true
|
||||
}
|
||||
|
||||
orm_pool() (result: Bool) {
|
||||
ready()
|
||||
parallel_each(pool_once, config.orm_input(), config.orm_concurrency())
|
||||
result = true
|
||||
}
|
||||
|
||||
orm_seed() (result: Bool) {
|
||||
populate()
|
||||
result = true
|
||||
}
|
||||
|
||||
count() (total: Int) {
|
||||
total = model.count()
|
||||
}
|
||||
|
||||
cancel() (result: Bool) {
|
||||
fill_cancelled_transaction()
|
||||
result = true
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
public get count = app.count
|
||||
public post cancel = app.cancel
|
||||
cmd idle = app.orm_idle
|
||||
cmd crud = app.orm_crud
|
||||
cmd list = app.orm_list
|
||||
cmd cursor = app.orm_cursor
|
||||
cmd stream = app.orm_stream
|
||||
cmd pool = app.orm_pool
|
||||
cmd seed = app.orm_seed
|
||||
@@ -0,0 +1,13 @@
|
||||
type DelayResult {
|
||||
total: Int
|
||||
}
|
||||
|
||||
global type Item {
|
||||
value: Int index
|
||||
label: Text(1, 32)
|
||||
}
|
||||
|
||||
sql delay(work: Int) (result: DelayResult) {
|
||||
sqlite = "WITH RECURSIVE counter(value) AS (SELECT 1 UNION ALL SELECT value + 1 FROM counter WHERE value < ?1) SELECT SUM(value) AS total FROM counter"
|
||||
postgres = "SELECT SUM(value)::bigint AS total FROM generate_series(1::bigint, $1::bigint) AS value"
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
cmd run = app.run
|
||||
@@ -0,0 +1,4 @@
|
||||
run() (result: Bool) {
|
||||
dever.io.println("profile")
|
||||
result = true
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
port.blocking_pause(millis: Int) () {
|
||||
dever.time.sleep(millis)
|
||||
}
|
||||
|
||||
port.report_error(message: Text) () {
|
||||
dever.io.println("ERROR|" + message)
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
cmd sync_idle = app.sync_idle
|
||||
|
||||
cmd async_idle = app.async_idle
|
||||
|
||||
cmd tasks = app.tasks
|
||||
|
||||
cmd tasks_on_worker = app.tasks_on_worker
|
||||
|
||||
cmd channels = app.channels
|
||||
|
||||
cmd cancel = app.cancel
|
||||
@@ -0,0 +1,132 @@
|
||||
type Parking {
|
||||
ready: Channel<Int>
|
||||
gate: Channel<Int>
|
||||
}
|
||||
|
||||
sync_idle() (result: Bool) {
|
||||
bench.ready(0)
|
||||
port.blocking_pause(config.lifetime())
|
||||
result = true
|
||||
}
|
||||
|
||||
idle() () {
|
||||
blocking(bench.ready(0))
|
||||
bench.pause(config.lifetime())
|
||||
}
|
||||
|
||||
async_idle() (result: Bool) {
|
||||
idle()
|
||||
result = true
|
||||
}
|
||||
|
||||
identity(value: Int) (result: Int) {
|
||||
result = value
|
||||
}
|
||||
|
||||
task_sum(value: Int, state: Int) (result: Int) {
|
||||
job = run(identity(value))
|
||||
result = state + wait(job)
|
||||
}
|
||||
|
||||
sum(value: Int, state: Int) (result: Int) {
|
||||
result = state + value
|
||||
}
|
||||
|
||||
task_sample(index: Int, input: Bytes) () {
|
||||
begin = dever.time.monotonic_nanos()
|
||||
checksum = reduce(task_sum, input, 0)
|
||||
end = dever.time.monotonic_nanos()
|
||||
blocking(bench.report(begin, end, "tasks", length(input), checksum))
|
||||
}
|
||||
|
||||
send_byte(value: Int, output: Channel<Int>) () {
|
||||
send(output, value)
|
||||
}
|
||||
|
||||
produce(input: Bytes, output: Channel<Int>) () {
|
||||
each(send_byte, input, output)
|
||||
close(output)
|
||||
}
|
||||
|
||||
channel_sample(index: Int, input: Bytes) () {
|
||||
output = channel(Int, config.channel_capacity())
|
||||
begin = dever.time.monotonic_nanos()
|
||||
producer = run(produce(input, output))
|
||||
checksum = reduce(sum, stream(output), 0)
|
||||
wait(producer)
|
||||
end = dever.time.monotonic_nanos()
|
||||
blocking(bench.report(begin, end, "channels", length(input), checksum))
|
||||
}
|
||||
|
||||
wait_one(value: Int, parking: Parking) () {
|
||||
send(parking.ready, value)
|
||||
signaled(receive(parking.gate))
|
||||
}
|
||||
|
||||
signaled(value: Int) () {}
|
||||
|
||||
signaled(value: null) () {
|
||||
blocking(port.report_error("channel closed before expected signal"))
|
||||
}
|
||||
|
||||
hold(parking: Parking) () {
|
||||
parallel_each(wait_one, config.pending_input(), config.pending(), parking)
|
||||
}
|
||||
|
||||
started(value: Int, ready: Channel<Int>) () {
|
||||
signaled(receive(ready))
|
||||
}
|
||||
|
||||
cancel_worker() () {
|
||||
parking = Parking {
|
||||
ready = channel(Int, config.pending())
|
||||
gate = channel(Int, 1)
|
||||
}
|
||||
job = run(hold(parking))
|
||||
each(started, config.pending_input(), parking.ready)
|
||||
blocking(bench.ready(config.pending()))
|
||||
bench.pause(config.idle_sample_ms())
|
||||
begin = dever.time.monotonic_nanos()
|
||||
stop(job)
|
||||
end = dever.time.monotonic_nanos()
|
||||
close(parking.ready)
|
||||
close(parking.gate)
|
||||
blocking(bench.report(begin, end, "cancel", config.pending(), config.pending()))
|
||||
}
|
||||
|
||||
task_worker() () {
|
||||
input = config.input()
|
||||
blocking(bench.ready(0))
|
||||
each(task_sample, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], input)
|
||||
}
|
||||
|
||||
channel_worker() () {
|
||||
input = config.input()
|
||||
blocking(bench.ready(0))
|
||||
each(channel_sample, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], input)
|
||||
}
|
||||
|
||||
tasks() (result: Bool) {
|
||||
task_worker()
|
||||
result = true
|
||||
}
|
||||
|
||||
nested_task_worker() () {
|
||||
job = run(task_worker())
|
||||
wait(job)
|
||||
}
|
||||
|
||||
tasks_on_worker() (result: Bool) {
|
||||
nested_task_worker()
|
||||
result = true
|
||||
}
|
||||
|
||||
channels() (result: Bool) {
|
||||
channel_worker()
|
||||
result = true
|
||||
}
|
||||
|
||||
cancel() (result: Bool) {
|
||||
cancel_worker()
|
||||
result = true
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
blocking_pause(millis: Int) () fails dever.time.SleepResult
|
||||
|
||||
report_error(message: Text) () fails dever.io.WriteResult
|
||||
@@ -0,0 +1,97 @@
|
||||
"""长连接对端的阶段校验与采样；进程预算由统一运行器管理。"""
|
||||
|
||||
import json
|
||||
import time
|
||||
|
||||
from process import Process, summarize
|
||||
|
||||
PHASES = ("connecting", "connected", "closing", "disconnected")
|
||||
SETTLE_SECONDS = 0.3
|
||||
PROCESS_MARGIN_SECONDS = 10
|
||||
|
||||
|
||||
def phase_sequence(cycles):
|
||||
return [(cycle, phase) for cycle in range(cycles) for phase in PHASES]
|
||||
|
||||
|
||||
def lifecycle_seconds(cycles, hold_seconds, timeout_ms):
|
||||
timeout_seconds = timeout_ms / 1000
|
||||
# Each cycle has independent connect, hold-overrun and close deadlines.
|
||||
return cycles * (hold_seconds + 3 * timeout_seconds + SETTLE_SECONDS) + PROCESS_MARGIN_SECONDS
|
||||
|
||||
|
||||
def stable_phase(before, after):
|
||||
return before[-1] if before and before == after else None
|
||||
|
||||
|
||||
def parse_progress(text, cycles):
|
||||
phases = []
|
||||
result = None
|
||||
# stdout is sampled while its writer is alive; only consume complete lines.
|
||||
for line in text.splitlines(keepends=True):
|
||||
if not line.endswith("\n"):
|
||||
continue
|
||||
if result is not None:
|
||||
raise ValueError("live benchmark output continued after its final result")
|
||||
if line.startswith("PHASE|"):
|
||||
_, cycle, phase = line.strip().split("|")
|
||||
phases.append((int(cycle), phase))
|
||||
elif line.startswith("{"):
|
||||
result = json.loads(line)
|
||||
else:
|
||||
raise ValueError(f"unexpected live benchmark output: {line.strip()}")
|
||||
if phases != phase_sequence(cycles)[:len(phases)]:
|
||||
raise ValueError("live benchmark phases are missing, duplicated or out of order")
|
||||
return phases, result
|
||||
|
||||
|
||||
def validate_result(result, protocol, connections, cycles):
|
||||
counts = ("attempted_connections", "connected", "disconnected", "messages", "errors")
|
||||
if result is None or any(type(result[key]) is not int or result[key] < 0 for key in counts):
|
||||
raise ValueError("missing or invalid live counters")
|
||||
expected = connections * cycles
|
||||
if (result["protocol"] != protocol or result["connections"] != connections or result["cycles"] != cycles
|
||||
or any(result[key] != expected for key in counts[:3])
|
||||
or result["messages"] < expected or result["errors"] != 0 or result["elapsed_ms"] <= 0):
|
||||
raise ValueError("live benchmark did not validate and close all requested connections")
|
||||
|
||||
|
||||
def load_live(args, process, port, protocol, connections, timeout_ms):
|
||||
scheme, path = {"tcp": ("tcp", "/"), "ws": ("ws", "/ws"), "sse": ("http", "/events")}[protocol]
|
||||
hold_ms = round(args.duration * 1000)
|
||||
command = [args.artifacts / "live_bench", protocol, f"{scheme}://127.0.0.1:{port}{path}",
|
||||
connections, args.cycles, hold_ms, timeout_ms]
|
||||
phases = {phase: [] for phase in phase_sequence(args.cycles)}
|
||||
with Process(command, process.directory / "load", cpus=args.client_cpus,
|
||||
peer_settings={"workers": args.client_workers}) as client:
|
||||
deadline = time.monotonic() + lifecycle_seconds(args.cycles, args.duration, timeout_ms)
|
||||
while client.child.poll() is None:
|
||||
if time.monotonic() >= deadline:
|
||||
raise TimeoutError("long connection client exceeded its bounded lifecycle")
|
||||
progress_before, _ = parse_progress(client.text(), args.cycles)
|
||||
first = len(process.samples)
|
||||
process.sample()
|
||||
process.check()
|
||||
if process.child.poll() is not None:
|
||||
raise RuntimeError("server exited during long connection load")
|
||||
progress_after, _ = parse_progress(client.text(), args.cycles)
|
||||
if phase := stable_phase(progress_before, progress_after):
|
||||
phases[phase].extend(process.samples[first:])
|
||||
client.sample()
|
||||
time.sleep(args.interval)
|
||||
client.check()
|
||||
progress, result = parse_progress(client.text(), args.cycles)
|
||||
if progress != phase_sequence(args.cycles):
|
||||
raise ValueError("long connection client ended before all phases")
|
||||
validate_result(result, protocol, connections, args.cycles)
|
||||
# Connection setup/close can finish between polls. Only the two explicit
|
||||
# hold windows require observations; retain setup samples when present.
|
||||
observations = []
|
||||
for (cycle, phase), samples in phases.items():
|
||||
if phase in ("connected", "disconnected") and len(samples) < 2:
|
||||
raise ValueError(f"sampling interval missed {cycle}/{phase} window")
|
||||
observations.append({"cycle": cycle, "phase": phase,
|
||||
"resources": summarize(samples) if samples else None,
|
||||
"last": samples[-1] if samples else None, "raw": samples})
|
||||
return {"result": result, "phases": observations,
|
||||
"client_resources": summarize(client.samples), "client_raw": client.samples}
|
||||
@@ -0,0 +1,231 @@
|
||||
"""Linux 进程与可选专属 cgroup 的生命周期和采样。"""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import statistics
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from settings import stage_peer
|
||||
|
||||
|
||||
def key_values(path):
|
||||
return {fields[0].rstrip(":"): fields[1] for line in path.read_text().splitlines()
|
||||
if len(fields := line.split()) >= 2}
|
||||
|
||||
|
||||
def process_sample(pid):
|
||||
root = Path("/proc") / str(pid)
|
||||
status = key_values(root / "status")
|
||||
if "VmRSS" not in status:
|
||||
raise ProcessLookupError(f"process {pid} exited during sampling")
|
||||
memory = key_values(root / "smaps_rollup")
|
||||
# comm 字段可含空格和括号；从最后一个右括号之后解析固定 stat 字段。
|
||||
stat = (root / "stat").read_text().rsplit(")", 1)[1].split()
|
||||
switches = {"voluntary_ctxt_switches": 0, "nonvoluntary_ctxt_switches": 0}
|
||||
for thread in (root / "task").iterdir():
|
||||
try:
|
||||
thread_status = key_values(thread / "status")
|
||||
except FileNotFoundError:
|
||||
continue # A completed blocking thread can disappear during a sample.
|
||||
for name in switches:
|
||||
switches[name] += int(thread_status[name])
|
||||
return {
|
||||
"time": time.monotonic(),
|
||||
"rss_bytes": int(status["VmRSS"]) * 1024,
|
||||
"rss_high_water_bytes": int(status["VmHWM"]) * 1024,
|
||||
"pss_bytes": int(memory["Pss"]) * 1024,
|
||||
"threads": int(status["Threads"]),
|
||||
"fds": len(list((root / "fd").iterdir())),
|
||||
"cpu_seconds": (int(stat[11]) + int(stat[12])) / os.sysconf("SC_CLK_TCK"),
|
||||
**switches,
|
||||
}
|
||||
|
||||
|
||||
class Budget:
|
||||
def __init__(self, parent, memory_mib, cpu_quota):
|
||||
self.path = None
|
||||
self.final = None
|
||||
if parent is None:
|
||||
if memory_mib is not None or cpu_quota is not None:
|
||||
raise ValueError("memory/cpu quotas require --cgroup-parent")
|
||||
return
|
||||
self.path = parent.resolve() / ("dever-perf-" + uuid.uuid4().hex)
|
||||
self.path.mkdir()
|
||||
try:
|
||||
if memory_mib is not None:
|
||||
(self.path / "memory.max").write_text(str(memory_mib * 1024 * 1024))
|
||||
(self.path / "memory.swap.max").write_text("0")
|
||||
if cpu_quota is not None:
|
||||
(self.path / "cpu.max").write_text(f"{round(cpu_quota * 100000)} 100000")
|
||||
except BaseException:
|
||||
self.path.rmdir()
|
||||
raise
|
||||
|
||||
def sample(self):
|
||||
if self.path is None:
|
||||
return None
|
||||
return {
|
||||
"current_bytes": int((self.path / "memory.current").read_text()),
|
||||
"peak_bytes": int((self.path / "memory.peak").read_text()),
|
||||
"events": {k: int(v) for k, v in key_values(self.path / "memory.events").items()},
|
||||
"cpu": {k: int(v) for k, v in key_values(self.path / "cpu.stat").items()},
|
||||
"max": (self.path / "memory.max").read_text().strip(),
|
||||
"cpu_max": (self.path / "cpu.max").read_text().strip(),
|
||||
}
|
||||
|
||||
def close(self):
|
||||
if self.path is not None:
|
||||
try:
|
||||
self.final = self.sample()
|
||||
except OSError as error:
|
||||
try:
|
||||
self.path.rmdir()
|
||||
except OSError as cleanup_error:
|
||||
error.add_note(f"cannot remove owned cgroup {self.path}: {cleanup_error}")
|
||||
raise
|
||||
else:
|
||||
self.path.rmdir()
|
||||
|
||||
|
||||
class Process:
|
||||
def __init__(self, command, directory, *, peer_settings=None, cpus=None, budget=None):
|
||||
self.command = [str(arg) for arg in command]
|
||||
self.directory = directory
|
||||
self.budget = budget
|
||||
self.samples = []
|
||||
self.directory.mkdir(parents=True, exist_ok=False)
|
||||
if peer_settings is not None:
|
||||
self.command[0] = str(stage_peer(command[0], directory, peer_settings))
|
||||
self.stdout = (directory / "stdout.log").open("w")
|
||||
self.stderr = (directory / "stderr.log").open("w")
|
||||
self.started = time.monotonic()
|
||||
|
||||
def prepare_child():
|
||||
# 运行器单线程；只对即将 exec 的自有子进程设置 CPU/cgroup。
|
||||
if cpus is not None:
|
||||
os.sched_setaffinity(0, cpus)
|
||||
if budget is not None and budget.path is not None:
|
||||
(budget.path / "cgroup.procs").write_text(str(os.getpid()))
|
||||
|
||||
try:
|
||||
self.child = subprocess.Popen(self.command, stdout=self.stdout, stderr=self.stderr,
|
||||
preexec_fn=prepare_child, start_new_session=True)
|
||||
except BaseException:
|
||||
self.stdout.close()
|
||||
self.stderr.close()
|
||||
raise
|
||||
|
||||
def text(self):
|
||||
return (self.directory / "stdout.log").read_text()
|
||||
|
||||
def descriptors(self):
|
||||
descriptors = {}
|
||||
for path in (Path("/proc") / str(self.child.pid) / "fd").iterdir():
|
||||
try:
|
||||
descriptors[path.name] = os.readlink(path)
|
||||
except FileNotFoundError:
|
||||
continue # A descriptor may close while /proc is being read.
|
||||
return descriptors
|
||||
|
||||
def sample(self):
|
||||
if self.child.poll() is not None:
|
||||
return
|
||||
try:
|
||||
sample = process_sample(self.child.pid)
|
||||
except (FileNotFoundError, ProcessLookupError):
|
||||
# Linux can release /proc memory fields before waitpid reports exit.
|
||||
# This observation is unavailable; check() still owns exit status.
|
||||
return
|
||||
sample["elapsed_seconds"] = sample.pop("time") - self.started
|
||||
if self.budget is not None:
|
||||
sample["cgroup"] = self.budget.sample()
|
||||
self.samples.append(sample)
|
||||
|
||||
def ready(self, timeout=10):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
self.sample()
|
||||
for line in self.text().splitlines():
|
||||
if line.startswith("ERROR|"):
|
||||
raise RuntimeError(line)
|
||||
if line.startswith("READY|"):
|
||||
self.ready_seconds = time.monotonic() - self.started
|
||||
return int(line.split("|")[1])
|
||||
if self.child.poll() is not None:
|
||||
raise RuntimeError(f"process exited before readiness: {self.directory}")
|
||||
time.sleep(0.005)
|
||||
raise TimeoutError(f"readiness timeout: {self.directory}")
|
||||
|
||||
def check(self):
|
||||
code = self.child.poll()
|
||||
if code is not None and code != 0:
|
||||
raise RuntimeError(f"process exited {code}: {self.directory}")
|
||||
if "ERROR|" in self.text() or (self.directory / "stderr.log").stat().st_size:
|
||||
raise RuntimeError(f"benchmark reported an error: {self.directory}")
|
||||
|
||||
def close(self):
|
||||
try:
|
||||
if self.child.poll() is None:
|
||||
try:
|
||||
os.killpg(self.child.pid, signal.SIGTERM)
|
||||
except ProcessLookupError:
|
||||
# 子进程可能恰好在 poll 与 signal 之间自然退出。
|
||||
pass
|
||||
try:
|
||||
self.child.wait(timeout=3)
|
||||
except subprocess.TimeoutExpired:
|
||||
os.killpg(self.child.pid, signal.SIGKILL)
|
||||
self.child.wait(timeout=3)
|
||||
finally:
|
||||
self.stdout.close()
|
||||
self.stderr.close()
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *_):
|
||||
self.close()
|
||||
|
||||
|
||||
def observe(processes, seconds, interval):
|
||||
deadline = time.monotonic() + seconds
|
||||
while time.monotonic() < deadline:
|
||||
for process in processes:
|
||||
process.sample()
|
||||
process.check()
|
||||
time.sleep(min(interval, max(0, deadline - time.monotonic())))
|
||||
|
||||
|
||||
def summarize(samples):
|
||||
if not samples:
|
||||
raise ValueError("process ended before any resource sample")
|
||||
result = {"samples": len(samples)}
|
||||
for metric in ("rss_bytes", "pss_bytes", "threads", "fds"):
|
||||
values = sorted(sample[metric] for sample in samples)
|
||||
result[metric + "_median"] = values[len(values) // 2]
|
||||
result[metric + "_max"] = max(values)
|
||||
result["rss_high_water_bytes"] = max(sample["rss_high_water_bytes"] for sample in samples)
|
||||
elapsed = samples[-1]["elapsed_seconds"] - samples[0]["elapsed_seconds"]
|
||||
cpu = samples[-1]["cpu_seconds"] - samples[0]["cpu_seconds"]
|
||||
result["cpu_seconds_sampled"] = cpu
|
||||
result["cpu_cores_sampled"] = cpu / elapsed if elapsed > 0 else None
|
||||
for name in ("voluntary_ctxt_switches", "nonvoluntary_ctxt_switches"):
|
||||
result[name + "_sampled"] = samples[-1][name] - samples[0][name]
|
||||
return result
|
||||
|
||||
|
||||
def resource_trend(samples):
|
||||
"""报告四段中位数与后半程增长，避免将启动分配误记为长期泄漏。"""
|
||||
if len(samples) < 4:
|
||||
raise ValueError("resource trend requires at least four observations")
|
||||
windows = [samples[len(samples) * index // 4:len(samples) * (index + 1) // 4]
|
||||
for index in range(4)]
|
||||
medians = [statistics.median(sample["rss_bytes"] for sample in window) for window in windows]
|
||||
elapsed = samples[-1]["elapsed_seconds"] - samples[0]["elapsed_seconds"]
|
||||
return {"observed_seconds": elapsed, "rss_quarter_medians_bytes": medians,
|
||||
"rss_late_growth_bytes": medians[-1] - medians[2],
|
||||
"rss_late_growth_bytes_per_second": (medians[-1] - medians[2]) / (elapsed / 4) if elapsed > 0 else None,
|
||||
"fds_first": samples[0]["fds"], "fds_last": samples[-1]["fds"]}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,59 @@
|
||||
"""基准配置与隔离 peer 布置；配置只来自 config/setting.json。"""
|
||||
|
||||
import errno
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import tempfile
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
PEER_CONFIGURATION = "setting-json-v1"
|
||||
|
||||
|
||||
def write_private_json(path, value, *, replace=False):
|
||||
path = Path(path)
|
||||
temporary = None
|
||||
if replace:
|
||||
descriptor, temporary = tempfile.mkstemp(dir=path.parent, prefix=f".{path.name}-")
|
||||
else:
|
||||
descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
try:
|
||||
with os.fdopen(descriptor, "w") as output:
|
||||
os.fchmod(output.fileno(), 0o600)
|
||||
json.dump(value, output, indent=2)
|
||||
output.write("\n")
|
||||
if temporary is not None:
|
||||
os.replace(temporary, path)
|
||||
finally:
|
||||
if temporary is not None:
|
||||
Path(temporary).unlink(missing_ok=True)
|
||||
|
||||
|
||||
def stage_peer(executable, directory, settings):
|
||||
"""独立配置共用不可变程序；测量期间不得重写输入二进制。"""
|
||||
executable = Path(executable).resolve(strict=True)
|
||||
destination = directory / executable.name
|
||||
(directory / "config").mkdir()
|
||||
write_private_json(directory / "config/setting.json", {"benchmark": settings})
|
||||
try:
|
||||
os.link(executable, destination)
|
||||
except OSError as error:
|
||||
if error.errno != errno.EXDEV:
|
||||
raise
|
||||
# 配置目录可以位于另一文件系统；这种情况下硬链接不可用。
|
||||
with destination.open("xb") as output, executable.open("rb") as source:
|
||||
shutil.copyfileobj(source, output)
|
||||
shutil.copymode(executable, destination)
|
||||
return destination.resolve()
|
||||
|
||||
|
||||
def configured_peer(name, setting_path=ROOT / "config/setting.json"):
|
||||
document = json.loads(setting_path.read_text())
|
||||
section = document.get("performance") if isinstance(document, dict) else None
|
||||
value = section.get(name) if isinstance(section, dict) else None
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise ValueError(f"config/setting.json must define performance.{name} as a binary path")
|
||||
# 仓库设置中的相对路径以项目根目录为基准。
|
||||
path = Path(value)
|
||||
return (setting_path.parent.parent / path).resolve(strict=True)
|
||||
@@ -0,0 +1,255 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
from cms import (CmsClient, ReadLatency, concurrent_reads, decode_response, execute, isolated_settings, latency_summary,
|
||||
publish_workload, run_postgres_case, run_postgres_rbac)
|
||||
from settings import write_private_json
|
||||
|
||||
|
||||
class CmsMeasurementTests(unittest.TestCase):
|
||||
def test_long_read_histogram_is_bounded_and_reports_upper_bounds(self):
|
||||
latency = ReadLatency()
|
||||
for _ in range(10000):
|
||||
latency.add(3)
|
||||
other = ReadLatency()
|
||||
other.add(100)
|
||||
latency.merge(other)
|
||||
self.assertEqual(len(latency.buckets), 2048)
|
||||
self.assertEqual(latency.count, 10001)
|
||||
self.assertGreaterEqual(latency.percentile(.99), 3)
|
||||
self.assertLess(latency.percentile(.99), 3.05)
|
||||
client = CmsClient(12345)
|
||||
client.record_timings = False
|
||||
client.connection = SimpleNamespace(request=Mock(), getresponse=Mock(return_value=SimpleNamespace(
|
||||
status=200, read=Mock(return_value=b'{"code":0,"message":"ok","data":1}'))), close=Mock())
|
||||
try:
|
||||
self.assertEqual(client.request("GET", "/test"), 1)
|
||||
self.assertEqual(client.timings, {})
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
def test_concurrent_read_failure_closes_clients_and_cannot_be_counted_as_success(self):
|
||||
expected = {"total": 1, "items": [{"title": "published"}]}
|
||||
for reply in ({"total": 0, "items": []}, RuntimeError("HTTP 403")):
|
||||
with self.subTest(reply=reply):
|
||||
opened = []
|
||||
|
||||
def client(_port):
|
||||
request = Mock(side_effect=reply) if isinstance(reply, Exception) else Mock(return_value=reply)
|
||||
instance = SimpleNamespace(cookies={}, timings={}, request=request, close=Mock())
|
||||
opened.append(instance)
|
||||
return instance
|
||||
|
||||
process = SimpleNamespace(samples=[], check=Mock())
|
||||
with patch("cms.CmsClient", side_effect=client), patch("cms.observe"):
|
||||
with self.assertRaises(RuntimeError):
|
||||
concurrent_reads(12345, {"front": "test-only-cookie"}, expected, process,
|
||||
concurrency=2, seconds=.1, interval=.01)
|
||||
self.assertEqual(len(opened), 2)
|
||||
for instance in opened:
|
||||
instance.close.assert_called_once()
|
||||
|
||||
def test_delete_inputs_use_query_instead_of_json_body(self):
|
||||
client = CmsClient(12345)
|
||||
response = SimpleNamespace(status=200, read=Mock(return_value=b'{"code":0,"message":"ok","data":true}'))
|
||||
connection = SimpleNamespace(request=Mock(), getresponse=Mock(return_value=response), close=Mock())
|
||||
client.connection = connection
|
||||
try:
|
||||
self.assertIs(client.request("DELETE", "/revoke", {"user_id": 3, "role_id": "fixture reader"}), True)
|
||||
sent = connection.request.call_args.args
|
||||
self.assertEqual(sent[:3], ("DELETE", "/revoke?user_id=3&role_id=fixture+reader", None))
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
def test_postgres_rbac_checks_catalog_roles_and_same_session_revocation(self):
|
||||
catalog = [
|
||||
{"component": "news", "domain": "article", "site": "admin", "action": "list",
|
||||
"method": "GET", "key": "news.article.admin.list"},
|
||||
{"component": "user", "domain": "authorization", "site": "admin", "action": "permissions",
|
||||
"method": "GET", "key": "user.authorization.admin.permissions"},
|
||||
{"component": "news", "domain": "article", "site": "front", "action": "list",
|
||||
"method": "GET", "key": "news.article.front.list"},
|
||||
]
|
||||
identity = {"user_id": 1, "tenant_id": 2}
|
||||
member = {"user_id": 3, "tenant_id": 2, "credential_id": 4}
|
||||
login = [{"account": {"id": 3}, "tenant_id": 2, "site": site}
|
||||
for site in ("admin", "front")]
|
||||
for failure in (False, True):
|
||||
with self.subTest(failure=failure):
|
||||
owner = SimpleNamespace(request=Mock(side_effect=[
|
||||
[entry for entry in catalog if entry["site"] == "admin"],
|
||||
[entry for entry in catalog if entry["site"] == "front"],
|
||||
] + [True] * 9))
|
||||
responses = login + ([RuntimeError("missing denial")] if failure else [None] * 11)
|
||||
client = SimpleNamespace(request=Mock(side_effect=responses), close=Mock())
|
||||
with patch("cms.execute", return_value=member), patch("cms.CmsClient", return_value=client):
|
||||
if failure:
|
||||
with self.assertRaisesRegex(RuntimeError, "missing denial"):
|
||||
run_postgres_rbac(Path("owned-app"), Path("owned-output"), 12345, owner, identity)
|
||||
else:
|
||||
result = run_postgres_rbac(Path("owned-app"), Path("owned-output"), 12345, owner, identity)
|
||||
self.assertEqual(result["member_user_id"], 3)
|
||||
self.assertEqual(set(result["permissions"]), {"admin_read", "admin_permissions", "front_list"})
|
||||
self.assertNotIn("password", result)
|
||||
calls = client.request.call_args_list[2:]
|
||||
self.assertEqual([call.kwargs.get("expected", 200) for call in calls],
|
||||
[403, 403, 403, 200, 200, 200, 403, 200, 200, 403, 403])
|
||||
saves = [call for call in owner.request.call_args_list if call.args[1].endswith("/save")]
|
||||
self.assertEqual(owner.request.call_args_list[1].kwargs["site"], "front")
|
||||
self.assertEqual(saves[0].args[2]["id"], saves[2].args[2]["id"])
|
||||
self.assertNotEqual(saves[0].kwargs["site"], saves[2].kwargs["site"])
|
||||
client.close.assert_called_once()
|
||||
|
||||
def test_postgres_case_keeps_source_configuration_on_build_failure(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-cms-source-") as temporary:
|
||||
directory = Path(temporary)
|
||||
source = directory / "project"
|
||||
source.mkdir()
|
||||
for name in ("config", "module", "test"):
|
||||
(source / name).mkdir()
|
||||
original = {
|
||||
"runtime": {"mode": "all"},
|
||||
"database": {"default": {
|
||||
"type": "postgres", "url": "postgres://fixture@127.0.0.1:12345/control",
|
||||
"tenant_database_prefix": "dever_test_owned",
|
||||
}},
|
||||
"auth": {"providers": {"session": {"jwtSecret": "original"}}},
|
||||
"sites": {}, "log": {"level": "info"},
|
||||
}
|
||||
setting = source / "config/setting.json"
|
||||
write_private_json(setting, original)
|
||||
before = setting.read_bytes()
|
||||
staged_paths = []
|
||||
|
||||
def fail_check(command, *_args, **_kwargs):
|
||||
staged = command[2]
|
||||
staged_paths.append(staged)
|
||||
self.assertNotEqual(staged, source)
|
||||
changed = json.loads((staged / "config/setting.json").read_text())
|
||||
self.assertEqual(changed["runtime"]["mode"], "api")
|
||||
self.assertNotEqual(changed["auth"]["providers"]["session"]["jwtSecret"], "original")
|
||||
raise RuntimeError("owned check failed")
|
||||
|
||||
output = directory / "result"
|
||||
with patch("cms.execute", side_effect=fail_check):
|
||||
with self.assertRaisesRegex(RuntimeError, "owned check failed"):
|
||||
run_postgres_case(source, output, directory / "compiler", articles=2)
|
||||
self.assertEqual(setting.read_bytes(), before)
|
||||
self.assertFalse((source / "cms-app").exists())
|
||||
self.assertTrue(staged_paths)
|
||||
self.assertFalse(staged_paths[0].exists())
|
||||
self.assertEqual(json.loads((output / "report.json").read_text())["status"], "failed")
|
||||
|
||||
def test_private_config_replacement_is_atomic_and_keeps_exclusive_creation(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-cms-setting-") as temporary:
|
||||
directory = Path(temporary)
|
||||
setting = directory / "setting.json"
|
||||
write_private_json(setting, {"original": True})
|
||||
with self.assertRaises(FileExistsError):
|
||||
write_private_json(setting, {"overwrite": True})
|
||||
with self.assertRaises(TypeError):
|
||||
write_private_json(setting, {"invalid": object()}, replace=True)
|
||||
self.assertEqual(json.loads(setting.read_text()), {"original": True})
|
||||
self.assertEqual(list(directory.iterdir()), [setting])
|
||||
write_private_json(setting, {"postgres": True}, replace=True)
|
||||
self.assertEqual(json.loads(setting.read_text()), {"postgres": True})
|
||||
self.assertEqual(setting.stat().st_mode & 0o777, 0o600)
|
||||
self.assertEqual(list(directory.iterdir()), [setting])
|
||||
|
||||
def test_envelope_failure_cannot_count_as_publication(self):
|
||||
self.assertEqual(decode_response(b'{"code":0,"message":"ok","data":{"id":1}}', 200),
|
||||
{"id": 1})
|
||||
for document, status in (({"code": 0, "message": "ok", "data": {}}, 500),
|
||||
({"code": 409, "message": "conflict", "data": None}, 200),
|
||||
({"id": 1}, 200)):
|
||||
with self.assertRaises(RuntimeError):
|
||||
decode_response(json.dumps(document).encode(), status)
|
||||
|
||||
def test_isolation_preserves_site_contract_without_mutating_input(self):
|
||||
source = {
|
||||
"runtime": {"mode": "all"}, "http": {"port": 8080},
|
||||
"database": {"default": {"type": "sqlite", "path": "old.db"}},
|
||||
"auth": {"providers": {"session": {"jwtSecret": "example", "verify": "user.account.verify"}}},
|
||||
"sites": {"admin": {"path": "admin", "auth": "session", "origin": "https://example.invalid"}},
|
||||
"log": {"level": "info"},
|
||||
}
|
||||
staged = isolated_settings(source, 12345)
|
||||
self.assertEqual(source["runtime"]["mode"], "all")
|
||||
self.assertEqual(source["database"]["default"]["path"], "old.db")
|
||||
self.assertEqual(staged["runtime"]["mode"], "api")
|
||||
self.assertEqual(staged["sites"]["admin"]["path"], "admin")
|
||||
self.assertEqual(staged["sites"]["admin"]["origin"], "http://127.0.0.1:12345")
|
||||
self.assertNotEqual(staged["auth"]["providers"]["session"]["jwtSecret"], "example")
|
||||
self.assertEqual(staged["auth"]["providers"]["session"]["verify"], "user.account.verify")
|
||||
self.assertEqual(source["log"]["level"], "info")
|
||||
self.assertEqual(staged["log"]["level"], "error")
|
||||
source["database"]["default"]["type"] = "postgres"
|
||||
with self.assertRaises(RuntimeError):
|
||||
isolated_settings(source, 12345)
|
||||
|
||||
def test_postgres_isolation_preserves_explicit_control_database_and_prefix(self):
|
||||
source = {
|
||||
"runtime": {"mode": "all"},
|
||||
"database": {"default": {
|
||||
"type": "postgres", "url": "postgres://fixture@127.0.0.1:12345/control",
|
||||
"tenant_database_prefix": "dever_test_owned", "tls": "disabled",
|
||||
}},
|
||||
"auth": {"providers": {"session": {"jwtSecret": "example"}}},
|
||||
"sites": {"admin": {"origin": "https://example.invalid"}},
|
||||
"log": {"level": "info"},
|
||||
}
|
||||
staged = isolated_settings(source, 23456)
|
||||
self.assertEqual(staged["database"], source["database"])
|
||||
self.assertEqual(staged["http"], {"host": "127.0.0.1", "port": 23456})
|
||||
self.assertNotIn("path", staged["database"]["default"])
|
||||
source["database"]["default"].pop("tenant_database_prefix")
|
||||
with self.assertRaises(RuntimeError):
|
||||
isolated_settings(source, 23456)
|
||||
|
||||
def test_publish_workload_uses_tenant_and_title_parameters(self):
|
||||
identity = {"credential_id": 11, "user_id": 3, "tenant_id": 2}
|
||||
publication = {"slug": "bench-0", "title": "Tenant A 0 published",
|
||||
"body": "Published body"}
|
||||
client = SimpleNamespace(request=Mock(side_effect=[
|
||||
{"account": {"id": 3}, "tenant_id": 2, "site": "admin"},
|
||||
{"account": {"id": 3}, "tenant_id": 2, "site": "front"},
|
||||
{"id": 1, "slug": "bench-0", "title": "Tenant A 0", "version": 1},
|
||||
{"id": 7, "article_id": 1, "title": "Tenant A 0 published", "version": 2},
|
||||
publication, {"total": 1, "items": [publication]}, None,
|
||||
]))
|
||||
process = SimpleNamespace(sample=Mock())
|
||||
contract = publish_workload(client, process, identity, "fixture-password", 1,
|
||||
tenant_key="tenant_a", title_prefix="Tenant A")
|
||||
self.assertEqual(contract, {"articles": 1, "titles": ["Tenant A 0 published"],
|
||||
"duplicate_publish_status": 409})
|
||||
for call in client.request.call_args_list[:2]:
|
||||
self.assertEqual(call.args[2]["tenant"], "tenant_a")
|
||||
revision = client.request.call_args_list[3]
|
||||
self.assertEqual(revision.args[:2], ("POST", "/content/revision/admin/manage/create"))
|
||||
self.assertEqual(revision.args[2]["expected_version"], 1)
|
||||
self.assertEqual(revision.args[2]["selected_article_id"], 1)
|
||||
self.assertEqual(process.sample.call_count, 1)
|
||||
|
||||
def test_small_workload_reports_samples_without_implying_p99(self):
|
||||
self.assertEqual(latency_summary({"publish": [4, 1, 3]}),
|
||||
{"publish": {"samples": 3, "p50_ms": 3, "max_ms": 4}})
|
||||
|
||||
def test_command_timeout_retains_output_and_reaps_its_owned_process(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-cms-timeout-") as temporary:
|
||||
directory = Path(temporary)
|
||||
command = [sys.executable, "-u", "-c",
|
||||
"import time,sys; print('starting'); print('diagnostic',file=sys.stderr); time.sleep(10)"]
|
||||
with self.assertRaises(subprocess.TimeoutExpired):
|
||||
execute(command, directory, "timeout", timeout=0.2)
|
||||
self.assertEqual((directory / "timeout.stdout.log").read_text(), "starting\n")
|
||||
self.assertEqual((directory / "timeout.stderr.log").read_text(), "diagnostic\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,122 @@
|
||||
"""用独立 Python 对端检查 Rust 长连接客户端的正文与生命周期判定。"""
|
||||
|
||||
from contextlib import contextmanager
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
import json
|
||||
from pathlib import Path
|
||||
import socketserver
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
|
||||
from live import parse_progress, phase_sequence, validate_result
|
||||
from settings import configured_peer, stage_peer
|
||||
|
||||
|
||||
@contextmanager
|
||||
def serving(server):
|
||||
with server:
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
try:
|
||||
yield server.server_address[1]
|
||||
finally:
|
||||
server.shutdown()
|
||||
thread.join(timeout=2)
|
||||
if thread.is_alive():
|
||||
raise RuntimeError("owned test listener did not stop")
|
||||
|
||||
|
||||
class Echo(socketserver.BaseRequestHandler):
|
||||
def handle(self):
|
||||
self.request.settimeout(2)
|
||||
while payload := self.request.recv(4096):
|
||||
response = b"!" * len(payload) if self.server.wrong_body else payload
|
||||
self.request.sendall(response)
|
||||
if self.server.wrong_body:
|
||||
return
|
||||
|
||||
|
||||
class InvalidEvent(BaseHTTPRequestHandler):
|
||||
def log_message(self, *_):
|
||||
pass
|
||||
|
||||
def do_GET(self):
|
||||
body = b"event: live\nid: 1\ndata: wrong\n\n"
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/event-stream")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
try:
|
||||
self.wfile.write(body)
|
||||
except (BrokenPipeError, ConnectionResetError):
|
||||
# The client aborts sibling sessions after the first deliberate mismatch.
|
||||
return
|
||||
|
||||
|
||||
class SilentEvent(BaseHTTPRequestHandler):
|
||||
def log_message(self, *_):
|
||||
pass
|
||||
|
||||
def do_GET(self):
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/event-stream")
|
||||
self.end_headers()
|
||||
self.wfile.write(b"event: live\nid: 1\ndata: ready\n\n")
|
||||
self.wfile.flush()
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
class LiveClientTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
directory = tempfile.TemporaryDirectory(prefix="dever-live-client-")
|
||||
self.addCleanup(directory.cleanup)
|
||||
self.executable = stage_peer(configured_peer("live_peer"), Path(directory.name), {"workers": 1})
|
||||
|
||||
def execute(self, protocol, port, path):
|
||||
scheme = "http" if protocol == "sse" else protocol
|
||||
return subprocess.run([self.executable, protocol, f"{scheme}://127.0.0.1:{port}{path}", "2", "2", "300", "1000"],
|
||||
capture_output=True, text=True, timeout=8)
|
||||
|
||||
def test_tcp_reconnect_and_exact_echo(self):
|
||||
listener = socketserver.ThreadingTCPServer(("127.0.0.1", 0), Echo)
|
||||
listener.wrong_body = False
|
||||
with serving(listener) as port:
|
||||
result = self.execute("tcp", port, "/")
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
phases, counts = parse_progress(result.stdout, 2)
|
||||
self.assertEqual(phases, phase_sequence(2))
|
||||
validate_result(counts, "tcp", 2, 2)
|
||||
|
||||
def test_tcp_wrong_body_cannot_be_counted_as_held(self):
|
||||
listener = socketserver.ThreadingTCPServer(("127.0.0.1", 0), Echo)
|
||||
listener.wrong_body = True
|
||||
with serving(listener) as port:
|
||||
result = self.execute("tcp", port, "/")
|
||||
self.assert_failure(result)
|
||||
|
||||
def test_sse_wrong_event_cannot_pass_connection_readiness(self):
|
||||
with serving(ThreadingHTTPServer(("127.0.0.1", 0), InvalidEvent)) as port:
|
||||
result = self.execute("sse", port, "/events")
|
||||
self.assert_failure(result)
|
||||
|
||||
def test_sse_silent_hold_cannot_pass_heartbeat_validation(self):
|
||||
with serving(ThreadingHTTPServer(("127.0.0.1", 0), SilentEvent)) as port:
|
||||
result = self.execute("sse", port, "/events")
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
counts = json.loads(result.stdout.splitlines()[-1])
|
||||
self.assertGreater(counts["errors"], 0)
|
||||
self.assertIn("PHASE|0|connected\n", result.stdout)
|
||||
self.assertNotIn("PHASE|0|closing\n", result.stdout)
|
||||
|
||||
def assert_failure(self, result):
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
counts = json.loads(result.stdout.splitlines()[-1])
|
||||
self.assertGreater(counts["errors"], 0)
|
||||
self.assertNotIn("PHASE|0|connected\n", result.stdout)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,120 @@
|
||||
"""用独立的 Python HTTP 对端验证负载计数；只占用自有随机回环端口。"""
|
||||
|
||||
from contextlib import contextmanager
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
|
||||
from process import Process
|
||||
from run import validate_load
|
||||
from settings import configured_peer, stage_peer
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
TLS = ROOT / "test/dever-tests/fixtures/tls"
|
||||
|
||||
|
||||
@contextmanager
|
||||
def peer(body):
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
protocol_version = "HTTP/1.1"
|
||||
|
||||
def do_GET(self):
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/plain")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, *_):
|
||||
pass
|
||||
|
||||
server = HTTPServer(("127.0.0.1", 0), Handler)
|
||||
worker = threading.Thread(target=server.serve_forever, kwargs={"poll_interval": 0.01})
|
||||
worker.start()
|
||||
try:
|
||||
yield server.server_port
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
worker.join(timeout=2)
|
||||
if worker.is_alive():
|
||||
raise RuntimeError("owned HTTP peer did not stop")
|
||||
|
||||
|
||||
@contextmanager
|
||||
def benchmark_service(executable, mode):
|
||||
settings = {"workers": 1, "connections": 4, "task_capacity": 128,
|
||||
"timeout_ms": 1000, "lifetime_ms": 5000, "http2_streams": 16,
|
||||
"http2_stream_window_bytes": 65535, "http2_connection_window_bytes": 262144,
|
||||
"cert_path": str(TLS / "server.pem"), "key_path": str(TLS / "server-key.pem")}
|
||||
with tempfile.TemporaryDirectory(prefix="dever-h2-bench-") as directory:
|
||||
with Process([executable, mode], Path(directory) / "server", peer_settings=settings) as process:
|
||||
yield process.ready()
|
||||
process.check()
|
||||
|
||||
|
||||
class NetworkMeasurementTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
directory = tempfile.TemporaryDirectory(prefix="dever-network-client-")
|
||||
self.addCleanup(directory.cleanup)
|
||||
self.executable = stage_peer(configured_peer("network_peer"), Path(directory.name), {"workers": 1})
|
||||
|
||||
def load(self, port):
|
||||
executable = self.executable
|
||||
output = subprocess.run([str(executable), "load", "h1", f"http://localhost:{port}",
|
||||
"/plain", "20", "0.2", "1", "1", "500", "true", "-"],
|
||||
capture_output=True, text=True, timeout=5, check=True)
|
||||
self.assertEqual(output.stderr, "")
|
||||
result = json.loads(output.stdout)
|
||||
validate_load(result)
|
||||
self.assertEqual(result["scheduled"], 4)
|
||||
self.assertGreaterEqual(result["elapsed_seconds"], 0.2)
|
||||
return result
|
||||
|
||||
def test_success_requires_exact_body_and_full_window(self):
|
||||
with peer(b"hello") as port:
|
||||
result = self.load(port)
|
||||
self.assertEqual(result["succeeded"], 4)
|
||||
self.assertEqual(result["errors"], 0)
|
||||
self.assertEqual(result["dropped"], 0)
|
||||
self.assertEqual(result["latency_samples"], 4)
|
||||
self.assertEqual(result["http_version"], "h1")
|
||||
self.assertEqual(result["physical_connections"], 1)
|
||||
self.assertEqual(result["streams_per_connection"], 1)
|
||||
self.assertEqual(result["request_slots"], 1)
|
||||
|
||||
def test_wrong_body_is_an_error_with_no_success_latency(self):
|
||||
with peer(b"wrong") as port:
|
||||
result = self.load(port)
|
||||
self.assertEqual(result["succeeded"], 0)
|
||||
self.assertEqual(result["errors"], 4)
|
||||
self.assertEqual(result["latency_samples"], 0)
|
||||
self.assertIsNone(result["latency_ms"])
|
||||
|
||||
def test_http2_uses_one_physical_connection_for_sixteen_request_slots(self):
|
||||
executable = self.executable
|
||||
for mode, transport in (("runtime-http2", "http"), ("hyper-http2", "http"),
|
||||
("runtime-https2", "https"), ("hyper-https2", "https")):
|
||||
with self.subTest(mode=mode), benchmark_service(executable, mode) as port:
|
||||
output = subprocess.run(
|
||||
[str(executable), "load", "h2", f"{transport}://localhost:{port}",
|
||||
"/bytes", "200", "0.1", "1", "16", "1000", "true", str(TLS / "root.pem")],
|
||||
capture_output=True, text=True, timeout=5, check=True)
|
||||
self.assertEqual(output.stderr, "")
|
||||
result = json.loads(output.stdout)
|
||||
validate_load(result)
|
||||
self.assertEqual(result["errors"], 0)
|
||||
self.assertGreater(result["succeeded"], 0)
|
||||
self.assertEqual(result["physical_connections"], 1)
|
||||
self.assertEqual(result["streams_per_connection"], 16)
|
||||
self.assertEqual(result["request_slots"], 16)
|
||||
self.assertEqual(result["successful_handshakes"], 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,47 @@
|
||||
"""只执行 peer 的配置检查入口，不创建网络或连接数据库。"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from settings import configured_peer, stage_peer
|
||||
|
||||
|
||||
class PeerSettingsTests(unittest.TestCase):
|
||||
def check(self, peer, document):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-peer-config-test-") as temporary:
|
||||
root = Path(temporary)
|
||||
executable = stage_peer(configured_peer(peer), root, {})
|
||||
setting = root / "config/setting.json"
|
||||
if document is None:
|
||||
setting.unlink()
|
||||
else:
|
||||
setting.write_text(json.dumps(document))
|
||||
# 不在 executable 目录启动，验证读取位置不受工作目录影响。
|
||||
return subprocess.run([executable, "check-config"], cwd=root.parent,
|
||||
capture_output=True, text=True, timeout=5)
|
||||
|
||||
def test_both_peers_use_the_external_integer_setting(self):
|
||||
for peer in ("network_peer", "live_peer"):
|
||||
for workers in (1, 4, 64):
|
||||
with self.subTest(peer=peer, workers=workers):
|
||||
output = self.check(peer, {"benchmark": {"workers": workers}})
|
||||
self.assertEqual(output.returncode, 0, output.stderr)
|
||||
self.assertEqual(json.loads(output.stdout), {"workers": workers})
|
||||
|
||||
def test_both_peers_reject_missing_malformed_and_out_of_range_configuration(self):
|
||||
invalid = [None, {}, [], {"benchmark": []}, {"benchmark": {"unknown": 1}}]
|
||||
invalid.extend({"benchmark": {"workers": value}} for value in (0, 65, -1, 1.5, True, "4", None))
|
||||
for peer in ("network_peer", "live_peer"):
|
||||
for document in invalid:
|
||||
with self.subTest(peer=peer, document=document):
|
||||
output = self.check(peer, document)
|
||||
self.assertNotEqual(output.returncode, 0)
|
||||
self.assertEqual(output.stdout, "")
|
||||
self.assertTrue(output.stderr.strip())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,803 @@
|
||||
"""基准运行器的最小正确性检查，不启动网络压测或编译器。"""
|
||||
|
||||
import argparse
|
||||
import errno
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, Mock, patch
|
||||
|
||||
import cms
|
||||
from process import Budget, Process, observe, summarize
|
||||
from build import (APPLICATION_CONFIGURATION, BENCHMARK_COMMAND, ENTRIES, configuration, dependency_packages, fingerprint,
|
||||
load_postgres_setting, postgres_case_url, postgres_database_case,
|
||||
native_runtime_setting, prepare_cms_project, prepare_native_project,
|
||||
prepare_profile_project, prepare_project,
|
||||
profile_database_setting, stage_application_command)
|
||||
from run import (HTTP_BODY_BYTES, application_command, fixture_command, async_measurements, csv_positive_ints, http2_summary,
|
||||
cms_cases, cms_summary, expected_orm_checksum, orm_measurements, orm_summary,
|
||||
orm_count, parse_client_cpus, parse_implementations, parser, percentile,
|
||||
positive, report_arguments, required_binaries, require_no_oom, require_no_request_errors,
|
||||
require_successful_requests, server, stage_orm_bundle, transaction_cancel_measurement,
|
||||
validate_cpu_placement, validate_load)
|
||||
from run import stage_live_bundle, verify_artifacts
|
||||
from live import lifecycle_seconds, parse_progress, phase_sequence, stable_phase, validate_result
|
||||
from settings import PEER_CONFIGURATION, configured_peer, stage_peer
|
||||
|
||||
|
||||
class MeasurementTests(unittest.TestCase):
|
||||
def test_application_command_matches_generated_declaration(self):
|
||||
self.assertEqual(application_command(Path("app")),
|
||||
[Path("app"), BENCHMARK_COMMAND, "{}"])
|
||||
with tempfile.TemporaryDirectory(prefix="dever-perf-command-") as temporary:
|
||||
module = Path(temporary) / "module"
|
||||
stage_application_command(module, "benchmark.item.orm_list")
|
||||
self.assertEqual((module / "benchmark/command/api.dever").read_text(),
|
||||
"cmd run = app.run\n")
|
||||
self.assertIn("benchmark.item.orm_list()",
|
||||
(module / "benchmark/command/app.dever").read_text())
|
||||
self.assertFalse((module / "main.dever").exists())
|
||||
|
||||
def test_generated_projects_use_stock_cms_and_cmd_for_orm_profiles(self):
|
||||
settings = configuration(parser().parse_args(["build", "--output", "unused"]))
|
||||
with tempfile.TemporaryDirectory(prefix="dever-perf-project-") as temporary:
|
||||
root = Path(temporary)
|
||||
for source in ("dever", "md"):
|
||||
project = root / f"cms-{source}"
|
||||
prepare_cms_project(project, source)
|
||||
self.assertFalse((project / "module/main.dever").exists())
|
||||
self.assertFalse((project / "module/benchmark").exists())
|
||||
self.assertEqual((project / "config/setting.json").read_bytes(),
|
||||
(Path(__file__).parents[2] / "examples/cms" / source
|
||||
/ "config/setting.json").read_bytes())
|
||||
self.assertTrue((project / "test/news/article").is_dir())
|
||||
orm = root / "orm"
|
||||
prepare_project(orm, "orm", settings, "sqlite")
|
||||
self.assertFalse((orm / "module/main.dever").exists())
|
||||
self.assertTrue((orm / "module/benchmark/command/api.dever").is_file())
|
||||
self.assertTrue((orm / "module/benchmark/item/api.dever").is_file())
|
||||
for profile in ("base", "sqlite", "postgres", "both"):
|
||||
project = root / f"profile-{profile}"
|
||||
prepare_profile_project(project, profile_database_setting(profile))
|
||||
self.assertFalse((project / "module/main.dever").exists())
|
||||
self.assertTrue((project / "module/benchmark/command/api.dever").is_file())
|
||||
|
||||
def test_profile_free_generation_copies_roles_and_only_used_helpers(self):
|
||||
settings = configuration(parser().parse_args(["build", "--output", "unused"]))
|
||||
expected_config = {
|
||||
"runtime": ("lifetime", "idle_sample_ms", "pending", "channel_capacity",
|
||||
"input", "pending_input"),
|
||||
"http": ("lifetime", "limits", "http2_limits", "large_body",
|
||||
"certificate", "key"),
|
||||
"live": (),
|
||||
}
|
||||
with tempfile.TemporaryDirectory(prefix="dever-perf-sources-") as temporary:
|
||||
for fixture, functions in expected_config.items():
|
||||
destination = Path(temporary) / fixture
|
||||
prepare_native_project(destination, fixture, settings)
|
||||
domain = destination / "module/benchmark" / fixture
|
||||
self.assertTrue(all((domain / f"{role}.dever").is_file()
|
||||
for role in ("app", "port", "adapter", "api")))
|
||||
config = destination / "module/benchmark/config/app.dever"
|
||||
self.assertEqual(config.is_file(), bool(functions))
|
||||
if functions:
|
||||
source = config.read_text()
|
||||
self.assertTrue(all(f"\n{name}()" in source for name in functions))
|
||||
self.assertNotIn("orm_input()", source)
|
||||
self.assertEqual((destination / "module/benchmark/bench/app.dever").is_file(),
|
||||
fixture == "runtime")
|
||||
setting = json.loads((destination / "config/setting.json").read_text())
|
||||
self.assertEqual(setting, native_runtime_setting(settings) if fixture == "live" else {})
|
||||
declarations = (domain / "api.dever").read_text()
|
||||
for entry in ENTRIES[fixture]:
|
||||
self.assertIn(f"cmd {entry} = app.{entry}", declarations)
|
||||
self.assertEqual(fixture_command(Path("app"), entry),
|
||||
[Path("app"), f"benchmark.{fixture}.{entry}", "{}"])
|
||||
live = native_runtime_setting(settings)["adapter"]["benchmark.live"]["setting"]
|
||||
self.assertEqual(live["lifetime_ms"], settings["lifetime_ms"])
|
||||
self.assertEqual(live["limits"]["timeout_ms"], settings["timeout_ms"])
|
||||
self.assertNotIn("timeout_ms", live)
|
||||
|
||||
def test_live_adapter_setting_is_private_to_each_executable(self):
|
||||
settings = {"lifetime_ms": 4000, "timeout_ms": 2000, "connections": 8}
|
||||
with tempfile.TemporaryDirectory(prefix="dever-live-bundle-") as temporary:
|
||||
root = Path(temporary)
|
||||
artifacts = root / "artifacts"
|
||||
artifacts.mkdir()
|
||||
for protocol in ("tcp", "ws"):
|
||||
(artifacts / protocol).write_bytes(protocol.encode())
|
||||
output = root / "output"
|
||||
output.mkdir()
|
||||
args = SimpleNamespace(artifacts=artifacts, output=output)
|
||||
for protocol in ("tcp", "ws"):
|
||||
executable = stage_live_bundle(args, settings, f"live-{protocol}-0", protocol)
|
||||
self.assertEqual(executable.read_bytes(), protocol.encode())
|
||||
setting = executable.parent / "config/setting.json"
|
||||
self.assertEqual(json.loads(setting.read_text()), native_runtime_setting(settings))
|
||||
self.assertEqual(setting.stat().st_mode & 0o777, 0o600)
|
||||
self.assertFalse((artifacts / "config").exists())
|
||||
|
||||
def test_cms_cases_use_stock_config_and_publish_result(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-perf-cms-case-") as temporary:
|
||||
root = Path(temporary)
|
||||
artifacts = root / "artifacts"
|
||||
output = root / "output"
|
||||
artifacts.mkdir()
|
||||
output.mkdir()
|
||||
for source, name in (("dever", "cms"), ("md", "cms-md")):
|
||||
(artifacts / name).write_bytes(source.encode())
|
||||
config = artifacts / "source" / f"cms-{source}" / "config"
|
||||
config.mkdir(parents=True)
|
||||
(config / "setting.json").write_text("{}")
|
||||
args = SimpleNamespace(artifacts=artifacts, output=output, repeats=1,
|
||||
_cms_sources=("dever", "md"), cgroup_parent=None,
|
||||
memory_mib=None, idle_seconds=0.1, interval=0.01,
|
||||
server_cpu=None, cpu_quota=None)
|
||||
result = {"contract": {"articles": 16, "titles": [f"Article {i}" for i in range(16)],
|
||||
"duplicate_publish_status": 409},
|
||||
"ready_seconds": 0.2, "workload_seconds": 0.5,
|
||||
"published_articles_per_second": 32,
|
||||
"latency": {}, "resources": {"rss_bytes_median": 1,
|
||||
"pss_bytes_median": 1,
|
||||
"rss_high_water_bytes": 2},
|
||||
"raw": [], "budget_before_stop": None, "cgroup": None}
|
||||
with patch("cms.run_case", return_value=result) as run_case:
|
||||
cases = cms_cases(args, {}, {})
|
||||
self.assertEqual(len(cases), 2)
|
||||
self.assertEqual(run_case.call_count, 2)
|
||||
for call, source in zip(run_case.call_args_list, ("dever", "md")):
|
||||
identity = call.args[0]
|
||||
self.assertEqual(identity["config"], str(
|
||||
artifacts / "source" / f"cms-{source}" / "config/setting.json"))
|
||||
self.assertEqual(identity["fingerprint"], fingerprint(Path(identity["executable"])))
|
||||
self.assertEqual(call.kwargs["articles"], 16)
|
||||
self.assertEqual(cms_summary(cases)["dever"]["articles"], 16)
|
||||
self.assertTrue(all(case["workload_mode"] == "sequential_publish" for case in cases))
|
||||
self.assertTrue(all("ready_value" not in case for case in cases))
|
||||
|
||||
def test_focused_cms_build_selection_keeps_the_full_build_as_default(self):
|
||||
self.assertEqual(parser().parse_args(["build", "--output", "unused"]).selection, "all")
|
||||
self.assertEqual(parser().parse_args(
|
||||
["build", "--output", "unused", "--selection", "cms-profiles"]
|
||||
).selection, "cms-profiles")
|
||||
self.assertEqual(parser().parse_args(
|
||||
["build", "--output", "unused", "--selection", "profiles"]
|
||||
).selection, "profiles")
|
||||
|
||||
def test_cms_descriptor_recovery_bounds_pool_and_preserves_other_resources(self):
|
||||
before = {"0": "/dev/null", "3": "socket:[listener]", "4": "/fixture/platform.db"}
|
||||
after = {**before, "5": "/fixture/platform.db", "6": "/fixture/platform.db-wal"}
|
||||
load = {"descriptors_before": before, "descriptors_after": after,
|
||||
"pooled_descriptor_limits": {"/fixture/platform.db": 4,
|
||||
"/fixture/platform.db-wal": 4,
|
||||
"/fixture/platform.db-shm": 1}}
|
||||
cms.validate_read_descriptor_recovery(load)
|
||||
invalid = [
|
||||
{**after, "9": "socket:[request]"},
|
||||
{**after, "9": "socket:[listener]"},
|
||||
{**after, "3": "socket:[replacement]"},
|
||||
{**after, "0": "/unexpected/file"},
|
||||
{**after, **{str(index): "/fixture/platform.db" for index in range(10, 15)}},
|
||||
{**after, **{str(index): "/fixture/platform.db-wal" for index in range(10, 15)}},
|
||||
{**after, "10": "/fixture/platform.db-shm", "11": "/fixture/platform.db-shm"},
|
||||
]
|
||||
for descriptors in invalid:
|
||||
with self.subTest(descriptors=descriptors), self.assertRaises(RuntimeError):
|
||||
cms.validate_read_descriptor_recovery({**load, "descriptors_after": descriptors})
|
||||
|
||||
def test_cms_failed_recovery_keeps_case_and_aggregate_evidence(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-cms-failed-report-") as temporary:
|
||||
root = Path(temporary)
|
||||
executable = root / "app"
|
||||
executable.write_bytes(b"fixture")
|
||||
config = root / "setting.json"
|
||||
config.write_text("{}")
|
||||
identity = {"executable": str(executable), "fingerprint": fingerprint(executable),
|
||||
"config": str(config)}
|
||||
manifest = {"format": "dever-cms-publish-v1", "cases": {"dever": identity, "md": identity}}
|
||||
client = Mock(cookies={}, timings={})
|
||||
client.request.return_value = []
|
||||
snapshots = iter(({"3": "socket:[listener]"},
|
||||
{"3": "socket:[listener]", "4": "socket:[leaked]"}))
|
||||
|
||||
def descriptors():
|
||||
self.assertTrue(client.close.called, "control client must close before the baseline")
|
||||
return next(snapshots)
|
||||
|
||||
process = Mock(samples=[{"fds": 1}], descriptors=descriptors)
|
||||
context = MagicMock()
|
||||
context.__enter__.return_value = process
|
||||
context.__exit__.return_value = False
|
||||
budget = Mock(final={"events": {"oom": 0, "oom_kill": 0}})
|
||||
settings = {"database": {"default": {"type": "sqlite", "path": "data/platform.db",
|
||||
"tenant_directory": "data/tenants", "max_connections": 4}}}
|
||||
load = {"elapsed_seconds": 75, "succeeded": 100, "errors": 0}
|
||||
with (patch("cms.isolated_settings", return_value=settings),
|
||||
patch("cms.provision_identity", return_value=("actor", "password")),
|
||||
patch("cms.CmsClient", return_value=client), patch("cms.Budget", return_value=budget),
|
||||
patch("cms.Process", return_value=context), patch("cms.wait_ready", return_value=0.1),
|
||||
patch("cms.publish_workload", return_value={"articles": 1}), patch("cms.observe"),
|
||||
patch("cms.concurrent_reads", return_value=load) as reads):
|
||||
with self.assertRaisesRegex(RuntimeError, "read descriptors did not recover"):
|
||||
cms.run_cases(manifest, root / "output", articles=1, cgroup_parent=None,
|
||||
memory_mib=None, concurrency=8, read_seconds=75)
|
||||
snapshots = iter(({"3": "socket:[listener]"},))
|
||||
reads.side_effect = RuntimeError("read fixture failed")
|
||||
with self.assertRaisesRegex(RuntimeError, "read fixture failed"):
|
||||
cms.run_cases(manifest, root / "read-failure", articles=1, cgroup_parent=None,
|
||||
memory_mib=None, concurrency=8, read_seconds=75)
|
||||
case = json.loads((root / "output/dever/report.json").read_text())
|
||||
report = json.loads((root / "output/report.json").read_text())
|
||||
self.assertEqual(case["status"], "failed")
|
||||
self.assertEqual(case["raw"], [{"fds": 1}])
|
||||
self.assertEqual(case["concurrent_reads"]["succeeded"], 100)
|
||||
self.assertEqual(case["concurrent_reads"]["descriptors_after"]["4"], "socket:[leaked]")
|
||||
self.assertEqual(case["cgroup"], budget.final)
|
||||
self.assertEqual(report["cases"], {"dever": case})
|
||||
self.assertEqual(report["status"], "failed")
|
||||
early = json.loads((root / "read-failure/report.json").read_text())["cases"]["dever"]
|
||||
self.assertEqual(early["contract"], {"articles": 1})
|
||||
self.assertEqual(early["concurrent_reads"]["descriptors_before"], {"3": "socket:[listener]"})
|
||||
self.assertNotIn("succeeded", early["concurrent_reads"])
|
||||
self.assertEqual(context.__exit__.call_count, 2)
|
||||
self.assertEqual(budget.close.call_count, 2)
|
||||
|
||||
def test_cms_requires_every_declared_source_binary(self):
|
||||
args = SimpleNamespace(suite="cms", _postgres_setting=None,
|
||||
_cms_sources=("dever", "md"))
|
||||
self.assertEqual(required_binaries(args), {"cms", "cms-md"})
|
||||
args._cms_sources = ("dever",)
|
||||
self.assertEqual(required_binaries(args), {"cms"})
|
||||
|
||||
def test_peer_staging_preserves_binary_and_isolates_settings(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-peer-setting-") as temporary:
|
||||
root = Path(temporary)
|
||||
source = root / "peer"
|
||||
source.write_bytes(b"owned executable")
|
||||
source.chmod(0o700)
|
||||
for workers in (1, 4):
|
||||
directory = root / str(workers)
|
||||
directory.mkdir()
|
||||
executable = stage_peer(source, directory, {"workers": workers})
|
||||
self.assertEqual(executable.read_bytes(), source.read_bytes())
|
||||
self.assertTrue(executable.samefile(source))
|
||||
self.assertEqual(executable.stat().st_mode & 0o777, 0o700)
|
||||
setting = directory / "config/setting.json"
|
||||
self.assertEqual(json.loads(setting.read_text()), {"benchmark": {"workers": workers}})
|
||||
self.assertEqual(setting.stat().st_mode & 0o777, 0o600)
|
||||
with self.assertRaises(FileExistsError):
|
||||
stage_peer(source, directory, {"workers": 8})
|
||||
self.assertFalse((root / "config").exists())
|
||||
|
||||
def test_peer_staging_copies_only_across_filesystems(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-peer-filesystem-") as temporary:
|
||||
root = Path(temporary)
|
||||
source = root / "peer"
|
||||
source.write_bytes(b"owned executable")
|
||||
source.chmod(0o700)
|
||||
directory = root / "cross-filesystem"
|
||||
directory.mkdir()
|
||||
with patch("settings.os.link", side_effect=OSError(errno.EXDEV, "different device")):
|
||||
executable = stage_peer(source, directory, {})
|
||||
self.assertEqual(executable.read_bytes(), source.read_bytes())
|
||||
self.assertEqual(executable.stat().st_mode & 0o777, 0o700)
|
||||
self.assertFalse(executable.samefile(source))
|
||||
denied = root / "denied"
|
||||
denied.mkdir()
|
||||
with patch("settings.os.link", side_effect=PermissionError(errno.EACCES, "denied")):
|
||||
with self.assertRaises(PermissionError):
|
||||
stage_peer(source, denied, {})
|
||||
self.assertFalse((denied / "peer").exists())
|
||||
|
||||
def test_peer_paths_come_from_project_setting(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-peer-path-") as temporary:
|
||||
root = Path(temporary)
|
||||
(root / "config").mkdir()
|
||||
(root / "peer").write_bytes(b"binary")
|
||||
setting = root / "config/setting.json"
|
||||
setting.write_text(json.dumps({"performance": {"network_peer": "peer"}}))
|
||||
self.assertEqual(configured_peer("network_peer", setting), root / "peer")
|
||||
for value in ({}, {"performance": []}, {"performance": {"network_peer": 123}}):
|
||||
setting.write_text(json.dumps(value))
|
||||
with self.assertRaisesRegex(ValueError, "performance.network_peer"):
|
||||
configured_peer("network_peer", setting)
|
||||
|
||||
def test_process_stages_peer_before_starting_child(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-peer-process-") as temporary:
|
||||
root = Path(temporary)
|
||||
command = [sys.executable, "-c", (
|
||||
"import json, pathlib, sys; "
|
||||
"print((pathlib.Path(sys.executable).parent / 'config/setting.json').read_text())"
|
||||
)]
|
||||
with Process(command, root / "child", peer_settings={"workers": 3}) as process:
|
||||
self.assertEqual(process.child.wait(timeout=5), 0)
|
||||
process.check()
|
||||
self.assertEqual(json.loads(process.text()), {"benchmark": {"workers": 3}})
|
||||
|
||||
@staticmethod
|
||||
def postgres_setting():
|
||||
return {
|
||||
"database": {
|
||||
"postgres_test": {
|
||||
"type": "postgres",
|
||||
"url": "postgres://user:secret@localhost/dever_{case}",
|
||||
"tls": "disabled",
|
||||
"max_connections": 4,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def test_orm_samples_are_bounded_and_report_percentiles(self):
|
||||
result = orm_measurements(
|
||||
"SAMPLE|crud|1|1000000|2\nSAMPLE|crud|1|3000000|2\n",
|
||||
"crud", 2, 2, 0.5,
|
||||
)
|
||||
self.assertEqual(result["operations"], 2)
|
||||
self.assertEqual(result["throughput_operations_per_second"], 4)
|
||||
self.assertEqual(result["latency_ms"], {"p50": 1, "p95": 3, "p99": 3})
|
||||
with self.assertRaises(ValueError):
|
||||
orm_measurements("SAMPLE|crud|1|1|2\nSAMPLE|crud|1|1|2\n", "crud", 1, 2, 1)
|
||||
with self.assertRaises(ValueError):
|
||||
orm_measurements("SAMPLE|list|1|1|2\n", "crud", 1, 2, 1)
|
||||
with self.assertRaises(ValueError):
|
||||
orm_measurements("SAMPLE|crud|1|1|1\n", "crud", 1, 2, 1)
|
||||
|
||||
def test_orm_checksums_cover_empty_cancel_and_bounded_pages(self):
|
||||
settings = {"orm_rows": 20}
|
||||
self.assertEqual(expected_orm_checksum("crud", settings), 2)
|
||||
self.assertEqual(expected_orm_checksum("list", settings), 40)
|
||||
self.assertEqual(expected_orm_checksum("cursor", settings), 20)
|
||||
self.assertEqual(expected_orm_checksum("stream", settings), 20)
|
||||
self.assertEqual(expected_orm_checksum("pool", settings), 20)
|
||||
result = transaction_cancel_measurement(0, 0.01)
|
||||
self.assertEqual(result["raw"][0]["checksum"], 0)
|
||||
with self.assertRaises(RuntimeError):
|
||||
transaction_cancel_measurement(1, 0.01)
|
||||
|
||||
def test_postgres_cases_require_explicit_isolation_template(self):
|
||||
setting = self.postgres_setting()
|
||||
self.assertEqual(
|
||||
postgres_case_url(setting, "orm_postgres_crud_0"),
|
||||
"postgres://user:secret@localhost/dever_orm_postgres_crud_0",
|
||||
)
|
||||
for value in ("postgres://localhost/shared", "sqlite:///{case}",
|
||||
"postgres://localhost/shared?application_name={case}"):
|
||||
setting["database"]["postgres_test"]["url"] = value
|
||||
with self.assertRaises(ValueError):
|
||||
postgres_case_url(setting, "case")
|
||||
|
||||
def test_postgres_setting_is_loaded_from_the_project_config(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-setting-") as directory:
|
||||
path = Path(directory) / "setting.json"
|
||||
path.write_text(json.dumps(self.postgres_setting()))
|
||||
self.assertEqual(load_postgres_setting(path), self.postgres_setting())
|
||||
path.write_text(json.dumps({
|
||||
"database": {
|
||||
"default": {
|
||||
"type": "sqlite",
|
||||
"path": "data/db/dever.db",
|
||||
"max_connections": 1,
|
||||
"max_page_size": 100,
|
||||
}
|
||||
}
|
||||
}))
|
||||
self.assertIsNone(load_postgres_setting(path))
|
||||
path.write_text(json.dumps({
|
||||
"database": {"postgres_test": {"type": "sqlite"}}
|
||||
}))
|
||||
with self.assertRaises(ValueError):
|
||||
load_postgres_setting(path)
|
||||
self.assertIsNone(load_postgres_setting(path.with_name("missing.json")))
|
||||
|
||||
def test_profile_settings_cover_each_database_capability(self):
|
||||
base = profile_database_setting("base")
|
||||
sqlite = profile_database_setting("sqlite")
|
||||
postgres = profile_database_setting("postgres")
|
||||
both = profile_database_setting("both")
|
||||
self.assertEqual(base, {})
|
||||
self.assertEqual(tuple(sqlite["database"]), ("default",))
|
||||
self.assertEqual(sqlite["database"]["default"]["type"], "sqlite")
|
||||
self.assertEqual(postgres["database"]["default"]["type"], "postgres")
|
||||
self.assertEqual(postgres["database"]["default"]["url"],
|
||||
"postgres://build-only.invalid/dever_profile")
|
||||
self.assertEqual(tuple(both["database"]), ("default", "postgres_profile"))
|
||||
self.assertEqual(both["database"]["postgres_profile"]["type"], "postgres")
|
||||
self.assertNotIn("postgres_test", both["database"])
|
||||
|
||||
def test_postgres_cancel_fixture_binds_int_as_bigint(self):
|
||||
fixture = (Path(__file__).parent / "fixtures/orm/app/model/item.dever").read_text()
|
||||
|
||||
self.assertIn("generate_series(1::bigint, $1::bigint)", fixture)
|
||||
|
||||
def test_dependency_report_is_unique_and_stable(self):
|
||||
self.assertEqual(
|
||||
dependency_packages("z v1\na v1\nz v1 (*)\n"),
|
||||
["a v1", "z v1"],
|
||||
)
|
||||
|
||||
def test_report_omits_private_postgres_configuration(self):
|
||||
arguments = report_arguments(SimpleNamespace(
|
||||
_postgres_setting=self.postgres_setting(),
|
||||
artifacts=Path("artifacts"),
|
||||
client_workers=4,
|
||||
client_cpus=(1, 2, 3, 4),
|
||||
))
|
||||
self.assertNotIn("_postgres_setting", arguments)
|
||||
self.assertEqual(arguments["artifacts"], "artifacts")
|
||||
self.assertEqual(json.loads(json.dumps(arguments))["client_cpus"], [1, 2, 3, 4])
|
||||
self.assertEqual(arguments["client_workers"], 4)
|
||||
|
||||
def test_orm_bundle_uses_a_fresh_external_setting_per_case(self):
|
||||
settings = {"orm_sqlite_connections": 1, "orm_postgres_connections": 4,
|
||||
"orm_rows": 128}
|
||||
with tempfile.TemporaryDirectory(prefix="dever-orm-bundle-") as directory:
|
||||
root = Path(directory)
|
||||
artifacts = root / "artifacts"
|
||||
artifacts.mkdir()
|
||||
(artifacts / "sqlite-crud").write_bytes(b"binary")
|
||||
output = root / "output"
|
||||
output.mkdir()
|
||||
args = SimpleNamespace(output=output, artifacts=artifacts, _postgres_setting=None)
|
||||
first = stage_orm_bundle(args, settings, "sqlite", "crud", 0)
|
||||
second = stage_orm_bundle(args, settings, "sqlite", "crud", 1)
|
||||
self.assertNotEqual(first[1].parent, second[1].parent)
|
||||
first_setting = json.loads((first[1].parent / "config/setting.json").read_text())
|
||||
second_setting = json.loads((second[1].parent / "config/setting.json").read_text())
|
||||
self.assertEqual(first_setting["database"], second_setting["database"])
|
||||
self.assertNotEqual(first_setting["auth"], second_setting["auth"])
|
||||
self.assertEqual(first_setting["database"]["default"]["path"], "data/db/orm.db")
|
||||
self.assertEqual((first[1].parent / "config/setting.json").stat().st_mode & 0o777,
|
||||
0o600)
|
||||
|
||||
def test_orm_http_bundles_use_isolated_ports_and_private_auth(self):
|
||||
settings = {"orm_sqlite_connections": 1, "orm_rows": 17}
|
||||
with tempfile.TemporaryDirectory(prefix="dever-orm-http-bundle-") as directory:
|
||||
root = Path(directory)
|
||||
artifacts = root / "artifacts"
|
||||
artifacts.mkdir()
|
||||
(artifacts / "sqlite-http").write_bytes(b"binary")
|
||||
output = root / "output"
|
||||
output.mkdir()
|
||||
args = SimpleNamespace(output=output, artifacts=artifacts, _postgres_setting=None)
|
||||
bundles = [stage_orm_bundle(args, settings, "sqlite", entry, 0,
|
||||
binary_entry="http" if entry == "cancel" else None)
|
||||
for entry in ("http", "cancel")]
|
||||
documents = [json.loads((bundle[1].parent / "config/setting.json").read_text())
|
||||
for bundle in bundles]
|
||||
self.assertTrue(all(0 < document["http"]["port"] < 65536
|
||||
for document in documents))
|
||||
self.assertTrue(all(document["http"]["host"] == "127.0.0.1"
|
||||
for document in documents))
|
||||
self.assertNotEqual(documents[0]["auth"], documents[1]["auth"])
|
||||
self.assertTrue(all(document["sites"]["benchmark"]["path"] == ""
|
||||
for document in documents))
|
||||
self.assertTrue(all((bundle[1].parent / "config/setting.json").stat().st_mode & 0o777
|
||||
== 0o600 for bundle in bundles))
|
||||
|
||||
def test_orm_count_requires_the_current_api_envelope(self):
|
||||
with patch("run.http.client.HTTPConnection") as connection:
|
||||
response = connection.return_value.getresponse.return_value
|
||||
response.status = 200
|
||||
response.getheader.return_value = "application/json; charset=utf-8"
|
||||
response.read.return_value = b'{"code":0,"message":"ok","data":17}'
|
||||
self.assertEqual(orm_count(12345, 0.2), 17)
|
||||
connection.return_value.request.assert_called_with(
|
||||
"GET", "/benchmark/item/count", headers={"Connection": "close"}
|
||||
)
|
||||
for invalid in (b'{"ok":true}', b'{"code":0,"message":"ok","data":true}',
|
||||
b'{"code":0,"message":"ok","data":-1}'):
|
||||
response.read.return_value = invalid
|
||||
with self.assertRaises(ValueError):
|
||||
orm_count(12345, 0.2)
|
||||
|
||||
def test_postgres_bundle_uses_the_named_project_setting(self):
|
||||
settings = {"orm_sqlite_connections": 1, "orm_postgres_connections": 4,
|
||||
"orm_rows": 128}
|
||||
with tempfile.TemporaryDirectory(prefix="dever-postgres-bundle-") as directory:
|
||||
root = Path(directory)
|
||||
artifacts = root / "artifacts"
|
||||
artifacts.mkdir()
|
||||
(artifacts / "postgres-crud").write_bytes(b"binary")
|
||||
output = root / "output"
|
||||
output.mkdir()
|
||||
args = SimpleNamespace(
|
||||
output=output,
|
||||
artifacts=artifacts,
|
||||
_postgres_setting=self.postgres_setting(),
|
||||
)
|
||||
_, executable, _, _ = stage_orm_bundle(args, settings, "postgres", "crud", 2)
|
||||
setting_path = executable.parent / "config/setting.json"
|
||||
setting = json.loads(setting_path.read_text())
|
||||
self.assertEqual(tuple(setting["database"]), ("default",))
|
||||
case = postgres_database_case(output, "orm-postgres-crud-2")
|
||||
self.assertEqual(
|
||||
setting["database"]["default"]["url"],
|
||||
f"postgres://user:secret@localhost/dever_{case}",
|
||||
)
|
||||
self.assertEqual(setting_path.stat().st_mode & 0o777, 0o600)
|
||||
|
||||
def test_postgres_database_names_isolate_separate_reports(self):
|
||||
first = postgres_database_case(Path("first"), "orm-postgres-crud-0")
|
||||
second = postgres_database_case(Path("second"), "orm-postgres-crud-0")
|
||||
self.assertNotEqual(first, second)
|
||||
self.assertTrue(first.endswith("_orm_postgres_crud_0"))
|
||||
|
||||
def test_artifact_verification_rejects_missing_and_changed_files(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-artifacts-") as directory:
|
||||
root = Path(directory)
|
||||
binary = root / "app"
|
||||
certificate = root / "tls/root.pem"
|
||||
certificate.parent.mkdir()
|
||||
binary.write_bytes(b"binary")
|
||||
certificate.write_bytes(b"certificate")
|
||||
manifest = {
|
||||
"peer_configuration": PEER_CONFIGURATION,
|
||||
"application_configuration": APPLICATION_CONFIGURATION,
|
||||
"binaries": {"app": fingerprint(binary)},
|
||||
"files": {"tls/root.pem": fingerprint(certificate)},
|
||||
}
|
||||
verify_artifacts(root, manifest, {"app"})
|
||||
with self.assertRaisesRegex(ValueError, "obsolete configuration contract"):
|
||||
verify_artifacts(root, {**manifest, "peer_configuration": "old"}, {"app"})
|
||||
with self.assertRaisesRegex(ValueError, "obsolete application contract"):
|
||||
verify_artifacts(root, {**manifest, "application_configuration": "rust-native"}, {"app"})
|
||||
with self.assertRaises(ValueError):
|
||||
verify_artifacts(root, manifest, {"app", "missing"})
|
||||
certificate.write_bytes(b"changed")
|
||||
with self.assertRaises(ValueError):
|
||||
verify_artifacts(root, manifest, {"app"})
|
||||
|
||||
def test_server_cleanup_does_not_mask_primary_failure(self):
|
||||
args = SimpleNamespace(cgroup_parent=None, memory_mib=None, cpu_quota=None,
|
||||
server_cpu=None)
|
||||
with tempfile.TemporaryDirectory(prefix="dever-server-cleanup-") as directory:
|
||||
with patch("run.Budget") as budget_type, patch("run.Process"):
|
||||
budget_type.return_value.close.side_effect = OSError("cleanup")
|
||||
with self.assertRaisesRegex(ValueError, "primary") as caught:
|
||||
with server(args, ["unused"], Path(directory) / "case", {}):
|
||||
raise ValueError("primary")
|
||||
self.assertTrue(any("cannot close benchmark budget" in note
|
||||
for note in caught.exception.__notes__))
|
||||
|
||||
def test_orm_summary_keeps_driver_and_operation_separate(self):
|
||||
def case(driver, entry, latency):
|
||||
return {"suite": "orm", "driver": driver, "entry": entry,
|
||||
"binary": {"bytes": 10},
|
||||
"resources": {"rss_bytes_median": 20, "pss_bytes_median": 15,
|
||||
"rss_high_water_bytes": 25},
|
||||
"measurement": {"throughput_operations_per_second": 100,
|
||||
"latency_ms": {"p50": latency, "p95": latency + 1,
|
||||
"p99": latency + 2}}}
|
||||
|
||||
summary = orm_summary([case("sqlite", "crud", 1), case("postgres", "crud", 2)])
|
||||
self.assertEqual([(value["driver"], value["entry"]) for value in summary],
|
||||
[("postgres", "crud"), ("sqlite", "crud")])
|
||||
self.assertEqual(summary[1]["latency_p99_ms_median"], 3)
|
||||
|
||||
def test_cms_summary_reports_business_baseline(self):
|
||||
cases = [{
|
||||
"suite": "cms",
|
||||
"source": "dever",
|
||||
"binary": {"bytes": 30},
|
||||
"ready_seconds": 0.5,
|
||||
"contract": {"articles": 16},
|
||||
"workload_seconds": 0.25,
|
||||
"published_articles_per_second": 64,
|
||||
"resources": {"rss_bytes_median": 20, "pss_bytes_median": 15,
|
||||
"rss_high_water_bytes": 25},
|
||||
"cgroup": {"peak_bytes": 40},
|
||||
}]
|
||||
self.assertEqual(
|
||||
cms_summary(cases),
|
||||
{"dever": {"repeats": 1, "workload_mode": "sequential_publish",
|
||||
"binary_bytes": 30, "articles": 16,
|
||||
"ready_seconds_median": 0.5, "workload_seconds_median": 0.25,
|
||||
"published_articles_per_second_median": 64,
|
||||
"rss_bytes_median": 20,
|
||||
"pss_bytes_median": 15, "rss_high_water_bytes_median": 25,
|
||||
"cgroup_peak_bytes_median": 40}},
|
||||
)
|
||||
|
||||
def test_cms_summary_separates_source_variants(self):
|
||||
cases = [
|
||||
{"suite": "cms", "source": source, "binary": {"bytes": size},
|
||||
"ready_seconds": 0.5, "contract": {"articles": 16},
|
||||
"workload_seconds": 0.25, "published_articles_per_second": 64,
|
||||
"resources": {"rss_bytes_median": 20, "pss_bytes_median": 15,
|
||||
"rss_high_water_bytes": 25}, "cgroup": None}
|
||||
for source, size in (("dever", 30), ("md", 31))
|
||||
]
|
||||
summary = cms_summary(cases)
|
||||
self.assertEqual(summary["dever"]["binary_bytes"], 30)
|
||||
self.assertEqual(summary["md"]["binary_bytes"], 31)
|
||||
|
||||
def test_live_lifecycle_includes_all_three_phase_deadlines(self):
|
||||
self.assertAlmostEqual(lifecycle_seconds(2, 0.3, 2000), 23.2)
|
||||
|
||||
def test_live_phase_order_and_partial_writes(self):
|
||||
text = "PHASE|0|connecting\nPHASE|0|connected\nPHASE|0|clos"
|
||||
self.assertEqual(parse_progress(text, 1), (phase_sequence(1)[:2], None))
|
||||
with self.assertRaises(ValueError):
|
||||
parse_progress("PHASE|0|connected\n", 1)
|
||||
with self.assertRaises(ValueError):
|
||||
parse_progress("PHASE|0|connecting\nPHASE|0|connecting\n", 1)
|
||||
|
||||
def test_live_sample_is_only_attributed_to_a_stable_phase(self):
|
||||
connecting = phase_sequence(1)[:1]
|
||||
connected = phase_sequence(1)[:2]
|
||||
self.assertEqual(stable_phase(connected, connected), (0, "connected"))
|
||||
self.assertIsNone(stable_phase(connecting, connected))
|
||||
self.assertIsNone(stable_phase([], connecting))
|
||||
|
||||
def test_live_incomplete_cleanup_cannot_pass(self):
|
||||
result = {"protocol": "tcp", "connections": 4, "cycles": 3,
|
||||
"attempted_connections": 12, "connected": 12, "disconnected": 12,
|
||||
"messages": 24, "errors": 0, "elapsed_ms": 3000}
|
||||
validate_result(result, "tcp", 4, 3)
|
||||
for key, value in (("disconnected", 11), ("errors", 1), ("messages", 0)):
|
||||
with self.assertRaises(ValueError):
|
||||
validate_result({**result, key: value}, "tcp", 4, 3)
|
||||
|
||||
def test_missing_cgroup_controller_still_removes_owned_empty_group(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-perf-test-") as directory:
|
||||
budget = Budget(Path(directory), None, None)
|
||||
owned = budget.path
|
||||
with self.assertRaises(FileNotFoundError):
|
||||
budget.close()
|
||||
self.assertFalse(owned.exists())
|
||||
|
||||
def test_dropped_and_failed_requests_cannot_disappear_from_report(self):
|
||||
result = {"scheduled": 10, "sent": 4, "completed": 4, "succeeded": 0,
|
||||
"errors": 4, "dropped": 6, "dropped_capacity": 5, "dispatch_expired": 1,
|
||||
"latency_samples": 0, "request_latency_samples": 0, "dispatch_lag_samples": 0,
|
||||
"latency_ms": None, "request_latency_ms": None, "dispatch_lag_ms": None,
|
||||
"elapsed_seconds": 1, "actual_success_qps": 0, "http_version": "h2",
|
||||
"physical_connections": 1, "streams_per_connection": 16,
|
||||
"request_slots": 16, "successful_handshakes": 1}
|
||||
validate_load(result)
|
||||
with self.assertRaises(ValueError):
|
||||
validate_load({**result, "errors": 3})
|
||||
with self.assertRaises(ValueError):
|
||||
validate_load({**result, "latency_ms": {"p50": 0, "p95": 0, "p99": 0}})
|
||||
with self.assertRaises(ValueError):
|
||||
validate_load({**result, "dispatch_lag_samples": 1,
|
||||
"dispatch_lag_ms": {"sample_count": 2, "p50": 0,
|
||||
"p95": 0, "p99": 0}})
|
||||
with self.assertRaises(ValueError):
|
||||
validate_load({**result, "request_slots": 15})
|
||||
with self.assertRaises(ValueError):
|
||||
validate_load({**result, "successful_handshakes": 2})
|
||||
|
||||
def test_http2_run_fails_on_request_errors_or_oom(self):
|
||||
require_no_request_errors({"errors": 0}, "load")
|
||||
with self.assertRaises(RuntimeError):
|
||||
require_no_request_errors({"errors": 1}, "load")
|
||||
require_no_oom(None)
|
||||
require_no_oom({"events": {"oom": 0, "oom_kill": 0, "oom_group_kill": 0}})
|
||||
with self.assertRaises(RuntimeError):
|
||||
require_no_oom({"events": {"oom": 1, "oom_kill": 0, "oom_group_kill": 0}})
|
||||
|
||||
def test_orm_http_requires_at_least_one_success_without_a_qps_threshold(self):
|
||||
require_successful_requests({"succeeded": 1}, "ORM HTTP load")
|
||||
with self.assertRaises(RuntimeError):
|
||||
require_successful_requests({"succeeded": 0}, "ORM HTTP load")
|
||||
|
||||
def test_http2_summary_keeps_topologies_separate(self):
|
||||
def case(connections, repeat, qps, p99, rss):
|
||||
return {"suite": "http2", "implementation": "dever", "transport": "http",
|
||||
"path": "/plain", "rate": 5000.0, "physical_connections": connections,
|
||||
"streams_per_connection": 16, "repeat": repeat,
|
||||
"load": {"actual_success_qps": qps, "dropped": 0,
|
||||
"request_latency_ms": {"p99": p99}},
|
||||
"resources": {"rss_bytes_median": rss, "pss_bytes_median": rss - 10,
|
||||
"rss_high_water_bytes": rss + 10}}
|
||||
|
||||
summary = http2_summary([case(1, 0, 90, 2, 100), case(1, 1, 110, 4, 120),
|
||||
case(4, 0, 200, 1, 140)])
|
||||
self.assertEqual(len(summary), 2)
|
||||
self.assertEqual(summary[0]["physical_connections"], 1)
|
||||
self.assertEqual(summary[0]["actual_success_qps_median"], 100)
|
||||
self.assertEqual(summary[0]["request_latency_p99_ms_median"], 3)
|
||||
self.assertEqual(summary[0]["server_rss_bytes_median"], 110)
|
||||
self.assertEqual(summary[1]["physical_connections"], 4)
|
||||
|
||||
def test_warmup_and_checksum_are_checked_before_aggregation(self):
|
||||
lines = ["SAMPLE|tasks|4|400000|480"]
|
||||
lines += [f"SAMPLE|tasks|4|{nanos}|480" for nanos in range(40, 76, 4)]
|
||||
result = async_measurements("\n".join(lines), {"iterations": 4, "pending": 2})
|
||||
self.assertEqual(result["median_ns_per_operation"], 14)
|
||||
self.assertEqual(len(result["raw"]), 10)
|
||||
with self.assertRaises(ValueError):
|
||||
async_measurements("\n".join(lines).replace("|480", "|479"), {"iterations": 4})
|
||||
with self.assertRaises(ValueError):
|
||||
async_measurements("\n".join(lines[:-1]), {"iterations": 4})
|
||||
|
||||
def test_cancel_reports_full_group_drain_and_has_no_fake_warmup(self):
|
||||
result = async_measurements("SAMPLE|cancel|32|32000|32", {"pending": 32})
|
||||
self.assertFalse(result["warmup_samples"])
|
||||
self.assertEqual(result["median_ns_per_operation"], 1000)
|
||||
|
||||
def test_percentile_uses_nearest_rank(self):
|
||||
self.assertEqual(percentile([9, 2, 5, 1], 0.5), 2)
|
||||
self.assertEqual(percentile([9, 2, 5, 1], 0.99), 9)
|
||||
with self.assertRaises(argparse.ArgumentTypeError):
|
||||
positive("nan")
|
||||
|
||||
def test_http2_matrix_lists_reject_empty_duplicate_and_unknown_values(self):
|
||||
self.assertEqual(csv_positive_ints("1,4", "connections"), (1, 4))
|
||||
self.assertEqual(parse_implementations("dever,hyper"), ("dever", "hyper"))
|
||||
for value in ("", "1,", "1,1"):
|
||||
with self.assertRaises(ValueError):
|
||||
csv_positive_ints(value, "connections")
|
||||
for value in ("", "dever,", "dever,dever", "dever,other"):
|
||||
with self.assertRaises(ValueError):
|
||||
parse_implementations(value)
|
||||
|
||||
def test_client_cpu_configuration_rejects_invalid_sets_and_placement(self):
|
||||
self.assertEqual(parse_client_cpus("0,1,2"), (0, 1, 2))
|
||||
for value in ("", "1,", "1,,2", "1,1", "-1", "other"):
|
||||
with self.subTest(value=value), self.assertRaises(ValueError):
|
||||
parse_client_cpus(value)
|
||||
args = parser().parse_args(["run", "--artifacts", "artifacts", "--output", "output",
|
||||
"--client-workers", "4", "--client-cpus", "1,2,3,4"])
|
||||
self.assertEqual((args.client_workers, args.client_cpus), (4, (1, 2, 3, 4)))
|
||||
with self.assertRaises(ValueError):
|
||||
validate_cpu_placement(0, (1, 5), {0, 1, 2, 3, 4})
|
||||
with self.assertRaises(ValueError):
|
||||
validate_cpu_placement(0, (0, 1), {0, 1, 2, 3, 4})
|
||||
with self.assertRaises(ValueError):
|
||||
validate_cpu_placement(-1, (1,), {0, 1})
|
||||
validate_cpu_placement(0, (1, 2), {0, 1, 2})
|
||||
|
||||
def test_process_affinity_is_applied_only_to_owned_child(self):
|
||||
allowed = os.sched_getaffinity(0)
|
||||
cpus = tuple(sorted(allowed)[:2])
|
||||
with tempfile.TemporaryDirectory(prefix="dever-perf-test-") as directory:
|
||||
command = [sys.executable, "-c",
|
||||
"import os; print(sorted(os.sched_getaffinity(0)), flush=True)"]
|
||||
with Process(command, Path(directory) / "affinity", cpus=cpus) as process:
|
||||
self.assertEqual(process.child.wait(timeout=5), 0)
|
||||
process.check()
|
||||
self.assertEqual(json.loads(process.text()), list(cpus))
|
||||
self.assertEqual(os.sched_getaffinity(0), allowed)
|
||||
|
||||
def test_http_body_size_metadata_matches_fixtures(self):
|
||||
self.assertEqual(HTTP_BODY_BYTES, {"/plain": 5, "/json": 11, "/bytes": 65_536})
|
||||
|
||||
def test_live_process_sampling_and_owned_cleanup(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-perf-test-") as directory:
|
||||
command = [sys.executable, "-c", "import time; print('READY|0', flush=True); time.sleep(10)"]
|
||||
with Process(command, Path(directory) / "process") as process:
|
||||
self.assertEqual(process.ready(), 0)
|
||||
observe([process], 0.04, 0.01)
|
||||
result = summarize(process.samples)
|
||||
self.assertGreater(result["rss_bytes_median"], 0)
|
||||
self.assertGreater(result["pss_bytes_median"], 0)
|
||||
self.assertGreaterEqual(result["threads_median"], 1)
|
||||
self.assertGreaterEqual(result["fds_median"], 3)
|
||||
self.assertIsNotNone(process.child.poll())
|
||||
|
||||
def test_early_exit_does_not_become_successful_readiness(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-perf-test-") as directory:
|
||||
with Process([sys.executable, "-c", "raise SystemExit(7)"], Path(directory) / "failed") as process:
|
||||
with self.assertRaises(RuntimeError):
|
||||
process.ready()
|
||||
|
||||
def test_memory_teardown_can_precede_waitpid_exit(self):
|
||||
with tempfile.TemporaryDirectory(prefix="dever-perf-test-") as directory:
|
||||
command = [sys.executable, "-c", "import time; print('READY|0', flush=True); time.sleep(10)"]
|
||||
with Process(command, Path(directory) / "teardown") as process:
|
||||
process.ready()
|
||||
count = len(process.samples)
|
||||
with patch("process.process_sample", side_effect=ProcessLookupError):
|
||||
process.sample()
|
||||
self.assertEqual(len(process.samples), count)
|
||||
self.assertIsNone(process.child.poll())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user