feat: ship slim releases with shared on-demand extensions and locked restore

This commit is contained in:
Antigravity committed 2026-10-05 00:19:31 +00:00
1 parent 83b78a0db6
commit 4fda7c8c1a
58 files changed
+3774 -425

No files matched your search

+12 -6
View File
@@ -33,7 +33,7 @@ Process main is separate from callable roots. Initialize Unix SIGPIPE policy bef
- Project creation: `dever new <new-directory> [--markdown]`; `project::create(&Path, bool)` consumes the single template source under the pinned `skills/assets/template` submodule. The machine launcher selects the active core without reading nonexistent target configuration. - Project creation: `dever new <new-directory> [--markdown]`; `project::create(&Path, bool)` consumes the single template source under the pinned `skills/assets/template` submodule. The machine launcher selects the active core without reading nonexistent target configuration.
- AI guidance: `dever skill path [project-root]` resolves and verifies the active or project-pinned release's `skills/dever-language`; `dever skill install <new-directory>` atomically publishes a stable loader that invokes the absolute machine launcher. Skill resolution is a public read-only operation and must not open the root-only install lock. - AI guidance: `dever skill path [project-root]` resolves and verifies the active or project-pinned release's `skills/dever-language`; `dever skill install <new-directory>` atomically publishes a stable loader that invokes the absolute machine launcher. Skill resolution is a public read-only operation and must not open the root-only install lock.
- Application target: `dever build <root> --output <new-file> [--target linux-x86_64|linux-aarch64]`; explicit dependency preparation accepts the same target on `dever lib add/update/doctor`. Omission selects `BuildTarget::host()`. `CompileRequest.target: BuildTarget` is required and serialized canonically; test requests must use the host. - Application target: `dever build <root> --output <new-file> [--target linux-x86_64|linux-aarch64]`; explicit dependency preparation accepts the same target on `dever lib add/install/update/doctor`. Omission selects `BuildTarget::host()`. `CompileRequest.target: BuildTarget` is required and serialized canonically; test requests must use the host.
- Private author entry: `native-release <author-root> --output <new-dir>` calls `toolchain::packaging::create(&Path, &Path) -> Result<ReleaseManifest, String>`. It reads only `<author-root>/config/setting.json`; this is not an application configuration section or another public `dever` command. - Private author entry: `native-release <author-root> --output <new-dir>` calls `toolchain::packaging::create(&Path, &Path) -> Result<ReleaseManifest, String>`. It reads only `<author-root>/config/setting.json`; this is not an application configuration section or another public `dever` command.
@@ -57,7 +57,7 @@ Process main is separate from callable roots. Initialize Unix SIGPIPE policy bef
- `tls.system/client/server` own public roots, explicit custom CA and server PEM configuration. `http.serve_tls/serve_live_tls` and `websocket.open(ws/wss URL, ClientTls, Limits)` reuse the same verified TCP/TLS transport. All four system http_serve intrinsics accept an optional final transferable Context, appended to the static handler signature. - `tls.system/client/server` own public roots, explicit custom CA and server PEM configuration. `http.serve_tls/serve_live_tls` and `websocket.open(ws/wss URL, ClientTls, Limits)` reuse the same verified TCP/TLS transport. All four system http_serve intrinsics accept an optional final transferable Context, appended to the static handler signature.
- `native::compile_with_cache(..., cache_directory) -> NativeProgram`; `cache_hit()` reports reuse. This explicit development/test path uses the workspace `target/native-artifacts`; a released `dever` must request the protected machine `deverd` service and must not expose that directory as a fallback. - `native::compile_with_cache(..., cache_directory) -> NativeProgram`; `cache_hit()` reports reuse. This explicit development/test path uses the workspace `target/native-artifacts`; a released `dever` must request the protected machine `deverd` service and must not expose that directory as a fallback.
- `native::compile_test_suite(&Program, &SourceMap, rustc) -> NativeProgram`; every suite entry must come from successful checking and remains private. The suite selects one stable checked case index per process. - `native::compile_test_suite(&Program, &SourceMap, rustc) -> NativeProgram`; every suite entry must come from successful checking and remains private. The suite selects one stable checked case index per process.
- `toolchain::artifact_put(&Layout, &[u8]) -> ArtifactReceipt` and `artifact_get(&Layout, sha256, bytes) -> Vec<u8>` use authenticated daemon IPC. Receipts carry lowercase 64-hex SHA-256 and exact length, not a client-selected path. `CacheStore::summary()` returns metadata only with `verified:false`; administrator `status()` verifies bytes and returns `verified:true`. - `toolchain::artifact_put(&Layout, &[u8]) -> ArtifactReceipt` and `artifact_get(&Layout, sha256, bytes) -> Vec<u8>` use authenticated daemon IPC. `artifact_get_optional` reports genuine absence as None and retains corruption/permission/IPC errors; never string-match errors or convert every failure into a download. Receipts carry lowercase 64-hex SHA-256 and exact length, not a client-selected path. `CacheStore::summary()` returns metadata only with `verified:false`; administrator `status()` verifies bytes and returns `verified:true`.
- `toolchain::compile(&Layout, &CompileRequest) -> Result<Vec<u8>, String>` submits canonical JSON `{version, kind, target, sources, bindings, resources}`. Sources are logical path/test origin/text; bindings contain only database names/drivers, provider verify names and site paths/auth keys. Resources are normalized paths, base64 bytes, digests and executable flags. No project root, secret Settings, IR, output path or client cache key is accepted. `SourceMap::from_compilation_sources` forces strict layout and compiler-owned standard sources; `check_with_bindings` and `validate_database_bindings` share the ordinary configuration rules. - `toolchain::compile(&Layout, &CompileRequest) -> Result<Vec<u8>, String>` submits canonical JSON `{version, kind, target, sources, bindings, resources}`. Sources are logical path/test origin/text; bindings contain only database names/drivers, provider verify names and site paths/auth keys. Resources are normalized paths, base64 bytes, digests and executable flags. No project root, secret Settings, IR, output path or client cache key is accepted. `SourceMap::from_compilation_sources` forces strict layout and compiler-owned standard sources; `check_with_bindings` and `validate_database_bindings` share the ordinary configuration rules.
- `format::format(&SourceFile) -> Result<String, Vec<Diagnostic>>`; `dever fmt <project-root> [--check]` prepares both `module/` and optional `test/` before committing either source set. - `format::format(&SourceFile) -> Result<String, Vec<Diagnostic>>`; `dever fmt <project-root> [--check]` prepares both `module/` and optional `test/` before committing either source set.
@@ -83,9 +83,13 @@ Process main is separate from callable roots. Initialize Unix SIGPIPE policy bef
Developer onboarding has two official repositories: `shemic/dever-main` and `shemic/dever-main-skills`. The skill directory is an independent Git repository, pinned by the compiler's submodule; it distinguishes this language from Go Dever. Author `skill` is a required list of `{source,path,sha256}` entries published as signed `skills/dever-language/<path>` artifacts, including SKILL.md; at most 256 files, 1 MiB each and 8 MiB total. New-project source bytes come from the same skill templates; Markdown documentation wraps them without a second semantic copy. Keep the generated configuration CMD-only and free of database/auth prerequisites. Developer onboarding has two official repositories: `shemic/dever-main` and `shemic/dever-main-skills`. The skill directory is an independent Git repository, pinned by the compiler's submodule; it distinguishes this language from Go Dever. Author `skill` is a required list of `{source,path,sha256}` entries published as signed `skills/dever-language/<path>` artifacts, including SKILL.md; at most 256 files, 1 MiB each and 8 MiB total. New-project source bytes come from the same skill templates; Markdown documentation wraps them without a second semantic copy. Keep the generated configuration CMD-only and free of database/auth prerequisites.
Official `install` and `update` explicitly download from the compiled GitHub Releases origin; neither project setting.json nor environment variables choose that origin. Each platform publishes `dever-<platform>.manifest.json`, `.manifest.sig`, `.tar.gz`, with immutable version tags `v<major.minor.patch>`. Verify the pinned signature before unpacking, require a matching skill, reject unsigned/duplicate/nonregular/escaping archive entries, and bound both metadata and payload. USTAR contains only signed artifact files; the two metadata files are separate assets. Redirects remain on the enumerated official HTTPS asset hosts. Privately staged publication never overwrites an existing release; malformed downloads leave the active version unchanged. The existing active-version journal is the sole commit point for both core and skill. The stable skill entry resolves it on each task; do not maintain a second independently switched skill pointer or rewrite every user's home directory. Official `install` and `update` explicitly download from the compiled GitHub Releases origin; neither project setting.json nor environment variables choose that origin. `dever-release-v2` binds base `artifacts` and optional `extensions` in one signature. Each platform publishes `dever-<platform>.manifest.json`, `.manifest.sig`, `.tar.zst`, plus `.ext-<kind>-<target>.tar.zst`, under immutable version tags `v<major.minor.patch>`. Kind is runtime-pip/npm/go, build-pip/npm, or target; target holds nonhost native runtime/CRT and sandbox. Validate closed IDs, path namespaces, duplicate/overlapping paths, digests and size/count limits in the shared catalog owner. Base retains all four host database profiles, compiler closure, bootstrap, host sandbox and skill. Extensions cannot override base files. Do not fabricate an unsigned merged manifest.
The skill's first-install helper uses its independently distributed public key, verifies the full package before executing bootstrap, and generates only an owned temporary installer config. It never reads business configuration. Real-root installation explicitly starts the machine service unless --no-start is selected; image-root verification must not touch host services. Python/OpenSSL are installer prerequisites, not application runtime dependencies. `sdk/release-assets.py` publishes the three deterministic assets with Linux NOREPLACE directory rename and cannot clean another writer's public output. Authenticate the catalog before extraction. Zstandard is a single checksummed frame with window_log_max=27; verify complete compressed EOF/trailer and bounded expansion even after TAR EOF. USTAR contains only its signed ordinary artifact files, with no duplicate, escaping or extra entries; require complete terminator/padding and reject trailing content. Redirects remain on fixed official HTTPS hosts. Publication never replaces another writer's directory. First-install raw `.blob-<sha256>` assets include the launcher and every bootstrap/lib dependency, independently digest-checked before execution. The launcher owns extraction; Python must not implement a second TAR/Zstd parser or require system zstd.
Shared extensions live in service-writable `cache/extensions/<version>/<catalog-sha>/<id>`, outside native cache eviction and immutable core directories. IPC accepts only version and typed ID for an already authenticated installed catalog, never caller URLs/paths/keys. Prepare under the persistent install lock in private staging, verify complete bytes then publish readonly with NOREPLACE; corrupted reuse is an error. `SignedResources` owns verified read/list/root selection for Lib, daemon runtime verification and the actual compile worker. `dever target add linux-aarch64` and explicit dependency preparation may ensure resources; ordinary run/build/read cannot. Update prepares precisely the old active version's installed extension selection before switching the one core/skill active journal; any failure preserves the old selection. Stable launcher/daemon protocol changes require matching trusted bootstrap provisioning. Do not rewrite users' skill directories or add an independently switched skill pointer.
The skill's first-install helper uses its independently distributed public key, verifies the full package before executing bootstrap, and generates only an owned temporary installer config. It never reads business configuration. Real-root installation explicitly starts the machine service unless --no-start is selected; image-root verification must not touch host services. Python/OpenSSL are installer prerequisites, not application runtime dependencies. `sdk/release-assets.py` uses an explicitly supplied author zstd executable and publishes deterministic base/extension/raw assets with Linux NOREPLACE directory rename; it cannot clean another writer's public output.
Native author settings use `dever-native-release-input-v1`: version/target/signing_key, core and `core_libraries` inputs, required base/sqlite/postgres/both archive inputs, and ordered start/libraries/end. Input `source` is a normalized author-relative path and `sha256` is the exact lowercase digest; link inputs also specify the destination `path`. Core libraries use plain SONAME filenames and publish below `lib/`; runtime link inputs remain runtime-relative. Reuse the runtime v1 manifest types and validator. Copy into exclusively owned private staging, verify both source and copied digest, sort manifest artifacts, and sign those exact bytes with the supplied owner-private Ed25519 PKCS#8 key. Reject key content as payload, symlinks, path escape and reserved/duplicate destinations. Keep source paths, key material and timestamps out of release metadata. Linux publication uses atomic NOREPLACE; failures clean only owned staging. No tool invocation, environment configuration, downloads, input hard links or implicit trust-key installation belongs to this owner. Native author settings use `dever-native-release-input-v1`: version/target/signing_key, core and `core_libraries` inputs, required base/sqlite/postgres/both archive inputs, and ordered start/libraries/end. Input `source` is a normalized author-relative path and `sha256` is the exact lowercase digest; link inputs also specify the destination `path`. Core libraries use plain SONAME filenames and publish below `lib/`; runtime link inputs remain runtime-relative. Reuse the runtime v1 manifest types and validator. Copy into exclusively owned private staging, verify both source and copied digest, sort manifest artifacts, and sign those exact bytes with the supplied owner-private Ed25519 PKCS#8 key. Reject key content as payload, symlinks, path escape and reserved/duplicate destinations. Keep source paths, key material and timestamps out of release metadata. Linux publication uses atomic NOREPLACE; failures clean only owned staging. No tool invocation, environment configuration, downloads, input hard links or implicit trust-key installation belongs to this owner.
@@ -244,11 +248,13 @@ Create a response-body cancellation owner only after the pending-to-committed st
Cross build: unknown/duplicate/missing target arguments fail before execution; absent target packs, wrong ELF/archive machine, unsigned target leaves, incompatible lock/runtime/build receipt and changed target inputs fail before cache reuse or linking. Sandbox loader mode must survive both release packing and embedding; a hash-valid non-executable loader is not a usable Worker resource. Cross build: unknown/duplicate/missing target arguments fail before execution; absent target packs, wrong ELF/archive machine, unsigned target leaves, incompatible lock/runtime/build receipt and changed target inputs fail before cache reuse or linking. Sandbox loader mode must survive both release packing and embedding; a hash-valid non-executable loader is not a usable Worker resource.
Native maker: wrong format/target/version, unknown or duplicate fields, missing profile, non-regular or escaping source, SHA mismatch, exposed/invalid signing key or payload alias to key → explicit failure, no output, owned staging removed. Existing or concurrently created destination → no overwrite. Correct bytes → complete signed v1 directory, without private key/configuration. Unsupported host publication → explicit failure, never a weaker rename fallback. Native maker: wrong format/target/version, unknown or duplicate fields, missing profile, non-regular or escaping source, SHA mismatch, exposed/invalid signing key or payload alias to key → explicit failure, no output, owned staging removed. Existing or concurrently created destination → no overwrite. Correct bytes → complete signed v2 catalog with base and extension artifacts, without private key/configuration. Unsupported host publication → explicit failure, never a weaker rename fallback.
Managed input with duplicate fields/noncanonical encoding, unsafe paths or forged resource digest -> reject before publication; invalid source -> original checker diagnostic from worker; unavailable service -> explicit failure; signature/runtime omission or tampering even on a cache hit -> reject; full admission/quota -> bounded error; cancellation/expiry -> reap child and release staging/locks without publishing a partial executable. Managed input with duplicate fields/noncanonical encoding, unsafe paths or forged resource digest -> reject before publication; invalid source -> original checker diagnostic from worker; unavailable service -> explicit failure; signature/runtime omission or tampering even on a cache hit -> reject; full admission/quota -> bounded error; cancellation/expiry -> reap child and release staging/locks without publishing a partial executable.
External Lib declarations live only in the root `lib` array of `config/setting.json`; reject old `config/lib.json` rather than silently combining it. Canonical v2 locks bind checked Port/Adapter/schema/operations/capabilities and independent Worker roots. Doctor rejects a stale binding, missing/cyclic closure or within-environment version conflict; independent Workers may select different versions. Embedding compares complete normalized provider metadata, including dependencies, runtime, schema and artifacts, not just an artifact subset. External Lib declarations live only in the root `lib` array of `config/setting.json`; reject old `config/lib.json` rather than silently combining it. Canonical `dever-lock-v6` locks bind checked Port/Adapter/schema/operations/capabilities and independent Worker roots. Registry artifacts retain exact `{ecosystem,locator,filename}` sources. Build Receipt.inputs is the complete fixed LockFile: Python build requirements or npm's pre-pruning execution graph (no self-referential build receipt). Receipt.graph remains the post-build result. Doctor rejects stale bindings, missing/cyclic closure or within-environment version conflicts; independent Workers may select different versions. Embedding compares complete normalized provider metadata, including dependencies, runtime, schema and artifacts, not just an artifact subset.
`dever lib install <root>` requires a current canonical lock, rejects specs, and holds the existing project mutation lock. Restore exact Package ZIPs first, then check declaration/Worker closure and selected target before restoring Lib archives. It never invokes a version resolver or writes setting/lock. A true missing archive uses its recorded source with existing registry host/path/size rules and locked Go evidence. Missing build outputs replay fixed receipt inputs through the same bounded sandbox executor used by initial resolution; validate runtime/tools/frontend, result graph/outcomes, bytes and digest before publishing. A non-reproducible build is an explicit failure, not authority to rewrite the lock. Ordinary run/build and doctor/remove remain readonly; add/update/install and Package preparation ensure runtime resources before Worker binding, including no-Lib adapters. Build resources are ensured only when the source-build path actually requests them. Cached built outputs do not need build tools.
Production registry access derives from a verified installed version core, not host tools or development fixtures. Signed assets `runtime/<ecosystem>/<target>/manifest.json` (`dever-registry-runtime-v1`) bind a RuntimePack identity and the adjacent `runtime.pack`; Python metadata additionally supplies target markers/wheel tags. Resolver owners implement PyPI PEP 508/440, npm semver/SRI and Go proxy/MVS with fixed HTTPS origins, disabled proxies and bounded work. Redirects are rejected except for Go `/@v/*.zip`: manually follow at most two hops into the exact HTTPS `storage.googleapis.com/proxy-golang-org-prod/` bucket. Validate normalized host/port/path and reject userinfo, fragments, missing/duplicate Location and loops before issuing a subsequent request; all hops share one 30-second deadline and diagnostics must not echo signed URLs. Loopback mirrors are explicit test transport, never project configuration. Unsupported extras/sdist, install hooks and Go sumdb are not silently approximated. Only explicit Lib commands resolve/download; run/build consume locked cached bytes. Production registry access derives from a verified installed version core, not host tools or development fixtures. Signed assets `runtime/<ecosystem>/<target>/manifest.json` (`dever-registry-runtime-v1`) bind a RuntimePack identity and the adjacent `runtime.pack`; Python metadata additionally supplies target markers/wheel tags. Resolver owners implement PyPI PEP 508/440, npm semver/SRI and Go proxy/MVS with fixed HTTPS origins, disabled proxies and bounded work. Redirects are rejected except for Go `/@v/*.zip`: manually follow at most two hops into the exact HTTPS `storage.googleapis.com/proxy-golang-org-prod/` bucket. Validate normalized host/port/path and reject userinfo, fragments, missing/duplicate Location and loops before issuing a subsequent request; all hops share one 30-second deadline and diagnostics must not echo signed URLs. Loopback mirrors are explicit test transport, never project configuration. Unsupported extras/sdist, install hooks and Go sumdb are not silently approximated. Only explicit Lib commands resolve/download; run/build consume locked cached bytes.
@@ -0,0 +1,5 @@
{"file":".trellis/spec/backend/quality-guidelines.md","reason":"Focused checks and actual CLI verification"}
{"file":".trellis/spec/backend/toolchain-and-library.md","reason":"Distribution security and offline invariants"}
{"file":".trellis/tasks/10-04-slim-distribution/prd.md","reason":"Required deliverables and acceptance criteria"}
{"file":".trellis/tasks/10-04-slim-distribution/design.md","reason":"Implementation boundaries and exact restore contract"}
{"file":".trellis/tasks/10-04-slim-distribution/implement.md","reason":"Verification evidence and ownership"}
@@ -0,0 +1,37 @@
# 精简发行设计
## 修改边界与复用
现有 `packaging` 统一签名全部资源,`release_source` 下载单一 gzip,`MachineManager` 原子安装版本,daemon 保护共享缓存,`InstalledRegistry` 从安装版本取工具。改动必须落在这些 owner:不向 App/业务配置加入安装概念,不重复实现每种生态的下载器/进程监督器,不把 Rust 源码编译要求转给使用者。
保留源码签名、路径/类型/摘要校验、四种数据库 profile、系统服务权限范围、离线编译、失败前置和版本 journal 的既有语义。CLI/安装/缓存/SDK/skill 的改变属于已确认五项方案;语言语法、模型、业务运行时与其他宿主平台不变。
## 发行目录与认证
- 升级当前发行 schema:签名根 manifest 包含基础 `artifacts` 和封闭类型的扩展目录 `extensions`。每个扩展绑定 kind、target 和完整 artifact 列表,身份由根 manifest 精确字节摘要与扩展 ID 共同确定。单一根签名认证全部描述,不能信任单独下载的无绑定元数据。
- 基础包包含 host native runtime 四 profile、core/lib、bootstrap、host sandbox、skill;Runtime(pip/npm/go)、Build(pip/npm)、Target(linux-aarch64 native+target sandbox) 分开发布。纯 Python/npm 包不触发 Build。
- author maker 继续只复制/校验/签名显式输入。预备发行目录可同时包含待发布扩展,SDK publisher 输出 base `.tar.zst` 与每种扩展 `.tar.zst`,文件列表来自同一签名目录。既有 v0.1.0 tag 不覆盖,正式新产物使用新版本。
- Zstandard 固定长窗口上限 27、流式解压、压缩/解压/文件数/单文件/尾部边界仍受限;不以签名成功替代归档检查。Rust 使用维护中的 codec 库安全接口,不手写 codec 或新增 workspace unsafe。
- Python 首装沿用独立发行公钥+OpenSSL 认证根目录,再下载并逐字节校验 raw bootstrap launcher 及其已声明私有动态库到独立 scratch。使用已认证 launcher 的隐藏 extraction 入口解压 `.tar.zst`,然后运行现有 bootstrap;使用者无需系统 zstd。完整解压/路径规则复用 Rust owner,避免保留两套 TAR 安全实现。SDK 发布上述 raw blob 供引导,所有可执行字节必须在执行前经独立密钥和摘要验证。
## 扩展安装、读取与更新
- 资源存储位于现有 service 可写的 `cache/extensions` 下,按已安装签名 manifest 身份与扩展 ID 建立不可变目录;不扩大 systemd 写范围,不写用户家目录,不修改版本 core 树或签名数据。
- 新 IPC 只接收 `version + ExtensionId(kind,target)` 封闭枚举,不接受 URL、路径、脚本或解压参数。官方 HTTPS origin、redirect allowlist、字节/时限/并发限额由工具链固定。外部用户复用现有 peer authentication/admission,只能准备已安装可信版本目录里声明的扩展。
- 复用持久 install 锁,下载在私有 staging,验证完整列表和字节后原子 no-replace 发布;竞态仅在完整同身份校验通过后复用。缺失是准备提示,损坏必须拒绝,不能悄悄掩盖成 cache miss。
- 单一只读资源视图统一 base 与已安装扩展路径,所有 worker runtime/build/sandbox 和目标 runtime 读取共用它;编译和缓存命中不得触发下载。源 manifest 不被拼接改写成伪造签名数据。
- `target add linux-aarch64` 对当前 active 版本显式准备 target 扩展;Lib preparation 额外准备实际使用生态对应 target。目标是应用 ARM,不要求 ARM 宿主编译器。
- `update` 准备新 base,并把当前 active 版本已经安装的扩展选择全部准备到新版本身份下,再写同一个 active journal。任何前置失败保留旧 active/core/skill/扩展,已安装资源不重复下载;未使用扩展不被顺带安装。
## Lib 恢复
- 显式 add/update/install 及 Package 中真正准备 Lib 的动作进入同一 preparation scope。先取得未绑定 Worker 声明,按需准备 Runtime,然后通过现有 owner 绑定和解析;普通 check/doctor/remove/run/build 保持只读或既有离线语义。
- Build 仅在 resolver 已确认 sdist 或 npm hook 必需后由统一 build-input owner 准备,不能根据生态名称一律下载。
- `lib install` 消费当前格式准确锁:验证项目、Package 传递依赖与 checked Worker binding;缓存命中验证完整 bytes/hash;缺失时按锁定精确版本/源定位下载,不选择新版本。完成后 setting/lock 字节保持一致。
- 为原生构建收据补全准确源 locator/文件名、固定 build requirement closure、原始 npm 执行图和编译输入身份;当前格式一并迁移仓库 fixtures,不引入旧格式 fallback。重放只能使用锁定输入;最终产物必须符合原收据 bytes/hash,不能用不同产物改写原锁。不可重现明确失败。
## 质量与操作边界
主任务串行协调 Cargo,worker 不并发创建新 target。测试只选择发行/资源/Lib/安装路径,真实验收使用 owned 镜像和进程,不修改宿主服务、安全策略或真实数据库。正式优化构建与四种 runtime 档案来源分开记录。压缩收益不与重复内容字节直接相减;实际发布前报告 base download/install、raw bootstrap 开销及扩展增量。
公开 Release 和线上更新独立于本地签名包验收。若缺 GitHub API 登录,先完成可审查的源码与产物,再报告唯一外部阻塞,不能以 tag/源码 push 代替 Release。
@@ -0,0 +1,8 @@
{"file":".trellis/spec/backend/index.md","reason":"Rust compiler scope and check routing"}
{"file":".trellis/spec/backend/directory-structure.md","reason":"Module ownership and root test placement"}
{"file":".trellis/spec/backend/quality-guidelines.md","reason":"Focused validation and safe Rust"}
{"file":".trellis/spec/backend/error-handling.md","reason":"Explicit errors and resource cleanup"}
{"file":".trellis/spec/backend/toolchain-and-library.md","reason":"Signed toolchain, offline resources and Lib contracts"}
{"file":".trellis/tasks/10-04-slim-distribution/design.md","reason":"Approved scope and cross-owner contracts"}
{"file":".trellis/tasks/10-04-slim-distribution/research/release.md","reason":"Release call chain and security invariants"}
{"file":".trellis/tasks/10-04-slim-distribution/research/libs.md","reason":"Lock restore and preparation call chain"}
@@ -0,0 +1,56 @@
# 实施记录
## 已确认
- 用户已批准完整五项方案及任务创建/继续实施,无需重复确认相同范围。
- [x] 工作区起点核查和备份,`target/distribution-before.10dsis`,gzip 校验通过。
- [x] PRD 记录用户要求、范围、回滚和验收,非语言迁移任务。
- [x] research/release.md 和 research/libs.md 完成,按实际 owner 完成 API 对接。
- [x] 实施上下文配置、任务进入 in_progress。
## 实施顺序与责任
1. 发行/扩展 worker:根 catalog schema、Zstandard Rust owner、资源视图、共享扩展安装 IPC、target 命令、installed-extension update、Rust 隐藏 extraction 入口;拥有 toolchain 下相关模块、launcher binary、CLI Cargo dependency 和对应定向测试。不要修改 author packaging、SDK Python、skills 或 Lib resolver owner。
2. Lib worker:锁/收据当前格式与精确恢复、add/update/install/Package preparation scope、InstalledRegistry 资源读取接线、CLI lib help 与相关测试;拥有 libs/**、必要 workers 资源消费/Package 路径及 main.rs。与发行 worker对接 typed extension API,不重复实现下载/安装。
3. 主会话:author packaging 接 catalog、SDK asset publisher、Python 首装、skill/docs、最终构建及真实发行验收。三个 owner 先固定接口,避免同文件改动;若遇所有权交叉先沟通。
4. trellis-check 独立检查整个改动范围并修复;主会话核验关键结果,更新 canonical spec 和记录,然后按已有授权处理 Git 与可用的发行发布。
## 验证清单
- [x] codec/manifest:10 定向通过;bootstrap执行位另有断言。
- [x] install/cache/update:4 定向通过(并发/损坏/更新失败/已装集合/closedIPC/typedmiss);实际 UID65533/65534 共享扩展通过。
- [x] lib install:4 cold restore、2 npm cache/projection、1 Package exact closure、PEP517 nested replay、npm optional replay 定向通过;实际安装版三生态无 Lib Worker 的锁恢复/运行/构建通过。
- [x] 运行/构建缺资源只提示,不联网;target add 支持 ARM,实际 ARM64 ELF 交叉构建通过。
- [x] SDK/Python bootstrap 10/10、skill quick_validate、全部修改/新增 Rust 文件 rustfmt check、CLI lib/bins/author examples clippy -D warnings/type check 通过。
- [x] 正式 release 构建、core/launcher/daemon ELF closure、保留 runtime 档案链接符号。
- [x] 实际签名包基础首装:普通/Markdown new/fmt/check/test/run/build;不含可选生态和 ARM 文件。
- [x] 实际扩展准备/锁恢复/跨项目复用、ARM 构建、已准备资源离线和独立应用执行。
- [x] 安装/更新失败恢复和成功更新:定向回归通过;真实负载本地后继 catalog 的 4 阶段更新事务验收通过。公开网络单独阻塞,不混同。
- [x] 清理本次 owned 临时产物,保留 durable assets、hash、验收 JSON 和源码备份。
- [ ] 最终 diff/self review、任务进度、Git 提交/push 和 Release 状态准确记录。
## 构建资源
起始磁盘约 0.78 GiB 空闲,/dev/shm 约 1.4 GiB 空闲。必要时可回收上一轮由本会话创建的隔离安装镜像(正式包与验收记录已持久保存且先复核),不可删除其他项目、源码、keys、未知 cache 或实验。使用显式 Rust toolchain 路径,保留现有共享 target,Cargo 串行 -j1;不因磁盘不足跳过必要验收或宣称完成。
## 当前证据
- 首轮 SDK/Python installer 定向 10/10,通过基础/扩展资产分离、真实 Ed25519 验签、raw bootstrap 摘要/长度/私有库、发布竞态、流量边界。Rust 接管的归档拒绝测试由发行 worker 补齐,未以 Python mock 代替实际安装验收。
- Cargo 新增静态 zstd 0.13.3;工作区版本 0.1.1,release strip=symbols,runtime-pack 仍显式 strip=debuginfo。正式串行构建目录 `/dev/shm/dever-slim-cargo.PvjPZZ`,未运行全量 CI。
- 旧 v0.1.0 完整 archive SHA-256 重新核对为 `7b56e825fb2b0468a271276bffbd84775e9dd9ae219f7e16097d1ff3df22b1cc` 后回收 owned 1.5 GiB 临时镜像。正式旧包/验收记录/源码备份均保留。
- Cargo 自有 `clean -p dever-cli -p dever-core --profile dev --target-dir target` 回收 898.3 MiB 旧编译产物;仅 native-acceptance-init 验收助手另存 `target/release-0.1.1/tools/`。源码/依赖缓存/其它项目/全局服务未清理。
- 真实本地 sandbox PEP517 nested replay 142.15s、npm optional replay 61.70s,均通过。原生maker 25/25默认定向通过,10个独立重验收未在这条命令启动。旧cross sandbox fixture已补齐同组native input;不是删除失败检查。
- trellis-check 独立审查未发现其他Rust合同问题;主会话修复其SDK bounded metadata/path长度/namespace/hostTarget差异,补文件/目录重叠拒绝。主会话另修复解压bootstrap执行位。clippy唯一is_multiple_of诊断已修复并复验通过。
- 原正式build先完成了全部依赖和CLI类型检查;在最终ELF链接前因上述代码修正主动停止(exit130)。最终源码 release 构建已成功(6m39s),产物保存到 `target/release-0.1.1/tools/`。core 16,951,872 字节,launcher 3,330,288 字节,daemon 3,310,296 字节;ELF 已 strip,launcher/daemon 仅需私有 libgcc_s 和 OS ABI,core 的 LLVM/private closure 随包提供。
## 真实 0.1.1 产物与验收
- 已制作实际签名 v2 目录与全部 split assets。基础 `.tar.zst` 为 117,481,208 字节(112.04 MiB),SHA-256 `fbf04562012b40038c8d845dcf528070606b4fd8374edec34c6516ea4afdf167`;基础文件解压合计约 573 MiB。首装另有 raw helper 3,513,312 字节(3.35 MiB)和约 22 KiB 元数据。全部可选资产合计约 596.59 MiB,不代表默认下载这些资源。
- `target/release-0.1.1/sizes.json` 记录每个资产的字节数/摘要;`provenance.json` 记录显式输入及未修改 runtime 档案复用。host glibc 最低 2.39,ARM64 为应用交叉目标。
- 首次实包验收到普通/Markdown 项目和缺 ARM 资源提示均通过;跨 UID 步骤因 outer bwrap 单 UID 映射不能 setgid65534 而失败。修复的是验收夹具:使用既有测试模式,在私有 machine 启动独立 daemon,以真实 UID65533/65534 检查共享资源。产品代码/签名资产未改;重新从 durable assets 验收。
- 更新验收另用真实负载签署仅供本地测试的后继版本,传输指向本地真实扩展。Ed25519 PKCS#8 v2 由与 maker 相同的 ring 签署,不再使用不支持该作者密钥格式的 OpenSSL 私钥读取;正式安装的 OpenSSL 公钥验证未变。最终 `update-acceptance.json` 为 passed,4 阶段通过:安装/扩展基线、失败保留旧活动版本且项目可运行、成功激活核心/skill/恰好已装扩展、切回真实版本后项目继续运行。仅供验收的 0.1.2 catalog 不能冒充 0.1.2 编译器,托管编译器严格拒绝目录版本与编译版本不同;没有关闭这项产品校验,也不声称虚拟后继版本执行通过。
- 实包从 durable assets 复验最终 `acceptance.json` 为 passed,13 阶段全部通过:签名首装、可选资源排除、公共入口/版本、普通与 Markdown 各自完整新项目流程、skill、离线缺资源、ARM64、三生态、双 UID、五个程序脱离源码/机器目录且清空 Worker 缓存后执行。全程未启动全局服务、未替换全局命令;本地传输不表示线上更新通过。
- 独立 trellis-check 对新增验收夹具/作者助手再次限定复核,无阻塞。产品验证合计 Rust 48 项、Python 10 项;正式实包 13 阶段、更新事务 4 阶段通过。fmt/clippy/skill 检查通过;未运行无关全量 CI/压力测试。
- 仅为回收本轮已完成的测试产物,Cargo clean CLI dev profile 再释放 376.8 MiB;正式包、作者输入、源码备份保留。
- 所有本轮隔离安装镜像均已卸载,自有 daemon/namespace 已退出;空临时目录已移除。正式构建临时目录核对仅含本轮 Rust 缓存/产物,保留工具与正式资产后清理约 556 MiB tmpfs。该目录没有新版 Cargo clean 要求的 CACHEDIR.TAG,未补造标记,核对归属后按精确目录移除;可从源码重建。
- GitHub SSH 源码访问可用;GitHub Release API 尚未登录。不能把源码/tag push 或本地模拟传输当作公开发布及线上更新。
@@ -0,0 +1,33 @@
# Dever 精简发行与按需工具安装
## 目标与已确认范围
用户已确认五项实施方案,并于本轮确认创建任务和继续实施。默认 Linux x86_64 基础包目标约 120 MiB,功能保持完整:编译器、LLVM、四种本机运行库(base/sqlite/postgres/both)、CLI、sandbox、skill 与模板。Python/Node/Go、原生扩展构建工具和 ARM 应用交叉编译资源改为显式命令触发、自动判断的按需安装。整台机器多用户、多项目共享同版本资源。
## 事实与约束
- 当前完整发行压缩包 714.7 MiB;同一基础文件集 gzip 为 181.6 MiB,Zstandard 长窗口(128 MiB)为 118.0 MiB,解压约 624 MiB。这是压缩试算,实际正式构建与完整安装验证仍需执行。
- 源码备份:`target/distribution-before.10dsis/source.tar.gz`;独立 skill 备份 `skills.tar.gz`;发行记录备份 `release-records.tar.gz`,均已 gzip 完整性校验。工作起点主仓库 `83b78a0`、skill `9fd78fc`,工作区干净。
- 编译器、launcher、daemon 应改为正式优化构建,清理非必要符号;保持 runtime 链接需要的符号。
- 应用配置仅 `config/setting.json`;官方源属于工具链内置元数据,不能放进项目配置或新增环境变量配置。不得依赖系统 PATH 的 Python/Node/Go/Rust 来构建用户应用。
- `run/build` 保持离线,缺资源提供准确准备命令;Dever Package 保持源码发布,底层 Rust 运行库保持预编译,FFI、自举和其他宿主平台不在本任务内。
- 当前 v0.1.0 源码 tag 已推送但二进制 Release 未发布;不得覆盖已有 tag。用户已授权相关仓库 commit/push。公开发布需完整验收和可用 GitHub API 授权,不能把本地包或 tag 当作已发布。
- 不改宿主 AppArmor/sysctl/全局服务,不替换机器全局命令;验收使用隔离安装镜像和自有进程。磁盘有限,复用现有工具输入并避免重复 Cargo target。
## 需求
1. 正式基础发行包采用 Zstandard,安装器和更新器具备所需解压能力,用户无需手动安装 zstd。完整性校验、边界限制、拒绝路径逃逸与原子发布继续成立。
2. 扩展按 Dever 版本、目标平台和内容身份绑定;按需签名下载、原子安装、并发安全、缓存复用,扩展失败不破坏已有可用版本。
3. `dever lib add/update` 自动准备声明所需生态环境,只有确有必要才准备原生构建工具;`dever lib install <project-root>` 按现有锁文件恢复,不更新版本、不改 setting/lock。
4. `dever target add linux-aarch64` 为当前选定版本准备 ARM 交叉编译资源;使用第三方 Lib 的 ARM 项目仍显式准备目标依赖。既有 ARM 能力与离线构建保持。
5. `dever update` 同步核心、skill 和已安装扩展;新增版本准备失败时旧 active 版本可继续使用,不能隐式安装所有可选扩展。
6. 同步 SDK 发行工具、安装脚本、CLI 帮助、开发 skill 和文档,命令行为只有一套实现,不新增兼容别名或重复路径。
## 验收
- 实际正式构建,报告压缩包大小与安装占用;约 120 MiB 为基于已有内容的目标,超出时必须报告准确实测和原因。
- 基础安装不包含 Python/Node/Go、原生扩展构建工具或 ARM 专用资源;普通与 Markdown 项目 new/check/test/run/build 通过隔离的定向验收。
- 需要什么下载什么;已有资源复用;锁文件恢复前后字节不变;无锁、锁与声明不符、资源损坏与下载失败均明确失败。
- 全新机器式隔离环境验证首次安装、扩展安装、多项目共享、已装扩展随更新和失败恢复;资源准备后断网构建,最终程序脱离工具链可执行。
- 定向恶意/损坏元数据和压缩边界回归通过;ARM 至少验证交叉构建及既有适用的隔离证据,不能把构建成功当作物理 ARM 性能测试。
- 公开上传和线上更新单独记录为完成或具体阻塞,不混同本地验收;不运行无关全量测试/长测。
@@ -0,0 +1,159 @@
# Research: Lib 按需准备与精确锁恢复
- Query: 追踪 Lib、Worker、Package 的准备/离线消费路径,设计 `dever lib install <root>` 与扩展管理器接点。
- Scope: internal;仅源码阅读,无测试、构建、下载、Git 操作。
- Date: 2026-10-04
## Findings
### 归属与已读规范
实际研究仓库为 `/data/project/dever`,独立 Rust 语言编译器,不是 `/data/project/shemic` 的 Go 应用。已读仓库 `AGENTS.md`、`.trellis/workflow.md`、当前 `prd.md`、backend index 与 toolchain-and-library 中目标、生态、锁、缓存和 Worker 合同,以及仓库 `skills/SKILL.md`。全局 shemic-dever/framework reference 描述 Go 项目,不能覆盖本仓库事实。
`task.py current --source` 返回 none;输出目录由主代理明确指定,未建立/修改会话指针。
### Files found
| 文件 | 作用 |
| --- | --- |
| `crates/dever-cli/src/libs.rs` | 锁结构、显式命令、已安装 runtime 读取、校验与离线嵌入 |
| `crates/dever-cli/src/libs/mutation.rs` | 稳定 inode 的项目依赖互斥锁 |
| `crates/dever-cli/src/libs/registry.rs` | 官方源安全传输、PyPI/npm/Go 解析及产物缓存发布 |
| `crates/dever-cli/src/libs/registry/npm.rs` | 独立 npm 环境、精确实例图、归档展开与需构建判断 |
| `crates/dever-cli/src/libs/registry/sumdb.rs` | Go 已锁透明日志证据与 ZIP 校验 |
| `crates/dever-cli/src/libs/build.rs` | 构建 Inputs、Session、完整 Receipt、离线收据校验 |
| `crates/dever-cli/src/libs/build/python.rs` | sdist 解包、PEP517 build 依赖解析与隔离执行 |
| `crates/dever-cli/src/libs/build/npm.rs` | npm 生命周期执行、optional 剪枝、最终安装归档 |
| `crates/dever-cli/src/libs/build/npm/environment.rs` | Node/Python/native 工具与 sandbox 环境 |
| `crates/dever-cli/src/workers/runtime.rs` | maker/consumer 共享 runtime descriptor 校验 |
| `crates/dever-cli/src/workers/go_build.rs` | 已准备 host Go 工具编译 target Worker |
| `crates/dever-cli/src/packages.rs` | Package 解析、精确 ZIP、源码/Worker/Lib 汇入公共锁 |
| `crates/dever-cli/src/toolchain/service.rs` | daemon ArtifactGet/Put IPC |
| `crates/dever-cli/src/toolchain/launcher.rs`、`src/main.rs` | 选版本与公共参数路由 |
### 现有调用链与准备入口
1. `libs::execute_for_target` (`libs.rs:1870`) 在 add/update 中先读声明、`source_workers`、Package libs,再 `resolve_project`,最后写 setting/lock。不能将 install 实现为 add/update 别名。
2. `source_workers_with_packages` (`libs.rs:2173`) 加载源码、check,再 `bound_program_workers` (`1236`)。后者立即调用 `InstalledRegistry::runtime` 绑定每个非 exec Worker 的 runtime。因此只在 `resolve_project` (`612`) 加 ensure 太晚:全新精简安装会先失败。无 Lib 的 managed Worker 也需要 runtime。
3. `resolve_project` 为实际生态调用 `InstalledRegistry::runtime` (`648`),创建 `HttpRegistry::official`、sumdb、`build::Session`,交给 resolver。fixture 全 exec 分支完全离线,不应强制要求 managed 安装。
4. `packages::execute_locked` (`1109`) 的 add/update 解析 Package closure,然后 `source_workers_with_packages` (`1173 附近`) 和 `resolve_project` (`1193 附近`)。Package 的传递 Lib 与 Worker 必须享有同一显式准备能力;remove 则 `retain_locked_libs`,不得顺便安装扩展。
5. `prepare` (`libs.rs:1139`) 名称易误导:它只校验声明是否在锁中并统计大小,不下载、不校验缓存完整性。
6. `embedded_resources_for_target` (`1268`) 在离线路径核对 runtime identity、目标、每个归档摘要/长度和 Go sumdb,再把字节作为资源。`prepare_resources` (`1418`) 追加目标 sandbox,Go 另外需要 host sandbox,随后 `workers::prepare_for_target`。
7. `workers/go_build.rs:61` 的 `compile_worker` 是离线构建;Go 编译器/链接器/selector/target stdlib 目前在 Go **runtime pack** 中,非 `build/go` 原生扩展包。保持 host 工具 + target 输出区分。
### 扩展管理器最小 API 接点
主代理提出使用 daemon 可写的不可变 `cache/extensions`,catalog 绑定已签名核心。Lib 层可保留当前逻辑路径,无需另一份生态业务实现:
- `ensure_extension(version, kind, target)`:仅显式准备阶段调用;kind 至少 runtime(pip/npm/go)、build(pip/npm)、target。目标 extension 包含 native runtime/target sandbox;生态 target runtime 单独选取。
- `read_verified_extension_artifact(version, kind, target, relative)` 与有界 artifact listing:只读验证已安装 extension,返回缺资源的准确准备命令。`InstalledRegistry::runtime` (`791`)、`build::Inputs::tools` (`726`)、`sandbox_resources` (`839`) 共用这个验证 owner,避免保留各自手写签名读取。当前 `signed_artifact` (`817`) 仅从 core manifest.artifacts 查叶子,需切换到合并后的已验证资源读取。
- 显式 preparation wrapper 实现 `build::Inputs`,runtime/tools 可先 ensure 再读。普通 InstalledRegistry 永远只读;或者使用显式、不可隐式默认的 preparation policy,并沿 worker 绑定路径传递。
- `source_workers_with_packages` 保持只读供 doctor/remove/run/build;新增同一底层源码扫描的显式准备入口,先收集未绑定 Worker,再 ensure、绑定。Package add/update 必须走该入口。不要复制 SourceMap/check 逻辑。
- npm build environment 自身请求辅助 Python (`build/npm/environment.rs:57`);按需准备需覆盖此调用,不能只看顶层 npm 声明。
- cross target 的本机 runtime、sandbox 与目标资源由类型化 target 选择,不能全局修改 host identity。
### 何时真的需要 build extension
- Python:优先兼容 wheel;只有进入 `RegistryResolver::python_source_candidate` (`registry.rs:933`) 的 sdist 分支才调用 Session.python。`Environment::new` (`build/python.rs:239`) 读取 build/pip;提前为每个 pip 根都安装 build/pip 不合理。
- npm:`Session::npm` (`build/npm.rs:15`) 先展开户籍归档,`npm::requires_build` 为 false 在第 43 行前后返回。只有确需生命周期/native 检查才创建 Environment,读取 build/npm。
- 当前 npm 的任何需执行 hook 路径都加载 Python + native build pack;当前 Python 的任何 sdist 也加载完整 build/pip。若 PRD 的“确有必要”要求进一步区分纯 JS hook / 纯 Python sdist 与 native 编译,需要收紧现有 environment 合同,而不仅加 lazy ensure;不能声称已做到逐个原生编译器按需。
### 当前锁记录什么、未记录什么
| 记录 | 源码依据 | 能力/缺口 |
| --- | --- | --- |
| LockedArtifact | `libs.rs:238` | target/path/bytes/sha256;path 是生成的逻辑路径,不是下载 URL |
| RuntimePack | `libs.rs:247` | name/version/sha256;运行时元数据从已签名版本读取 |
| LockedLib | `libs.rs:255` | 精确 spec、依赖、runtime、artifacts、schema、build receipt identity |
| LockFile | `libs.rs:267` | libs/workers/go_sumdb/npm/builds/packages;decode 要求规范字节与当前 TOOLCHAIN (`353`) |
| npm Environment | `registry/npm.rs:73` | 每个 Worker 独立 roots、实例路径、owner archive/prefix、edges、omission、build id |
| build Receipt | `build.rs:52` | roots/sources/graph/runtime/aux runtime/tools/frontend/Python backend/dynamic deps/递归依赖锁/output |
| LockedPackage | `packages.rs:75` | name/version/zip sha256/bytes/精确依赖/manifest sha256;来源在 setting 中已有 registry |
普通 PyPI wheel filename/URL 在解析后丢失(`registry.rs:905`);npm dist URL/SRI 未持久化(`988`);Go ZIP 地址可由已锁 spec 按现有 go_path 规则导出(`1121` 附近)。不改锁格式的恢复可查询**已锁精确版本**的 metadata 并仅选匹配锁 SHA/长度的文件,绝不重新选择版本或依赖;若文件撤下或变字节,明确失败。下载仍通过现有官方源/redirect/长度策略。
构建产物不是远端 registry 文件,必须优先从共享 cache 恢复;缺失时用固定 receipt 重放并比对锁定输出 SHA/长度,不能把新输出写回锁。
### 锁恢复的完整最小流程
1. install 参数禁止 specs,保留明确 target 选择。读且保存原始 setting/lock 字节;无锁立即报错;`LockFile::decode` + `doctor` 校验。将 install 纳入 `ProjectMutation::for_command` 的稳定互斥锁 (`mutation.rs:13`),不调用 setting 或 lock writer。
2. Package 必须先恢复:从已有 registry 的 `/v1/packages/<name>/<version>.zip`(`packages.rs:467` 的现有格式)取**锁定** ZIP,以 lock 摘要/长度验证并发布共享 artifact cache;不读版本索引、不选版本。然后复用 `owned_files_with` (`726`)、`validate_locked_roots` (`811`) 核对 manifests、依赖、源码归属和本地冲突。
3. 收集 config Lib、Package manifest Lib、源码 Worker 合同,验证锁 roots/closure;现有 `prepare` 只查声明包含性,不拒绝无主锁 libs。可借 `retain_locked_libs` (`2072`) 的闭包投影与原锁比较(不发布),同时核对 workers;注意 npm receipt 的原始图与 remove 后视图允许不同,不误判合法离线 remove 结果。
4. 仅为锁中实际 runtime 及 Worker ensure 对应版本/target 扩展,然后比对 RuntimePack;目标错配先报错,不替用户切换锁。对 fixture 条目复用内置字节与完整 metadata 校验。
5. 实现 `restore_locked` 的精确归档恢复,接受 `RegistryTransport`、`ArtifactStore`、build Inputs,便于 fixture 注入。按 SHA/bytes 查缓存,真实 miss 才取精确版本 metadata/归档。Go 使用锁中 `go_sumdb` 验 ZIP,不调用新的 MVS/checkpoint 解析。
6. 对缺失的构建产物递归恢复 receipt.sources 和 receipt.dependencies;验证 frontend、tools、runtime identity;通过同一个 sandbox/environment 实现固定重放。Python 重放使用锁定 backend/static+dynamic 依赖,不调用 `build_requirements`;npm 使用锁定实例图与已记录 optional 结果,不再进行 semver 解析。
7. 恢复后复用 `embedded_resources_for_target` 验最终产物,确认 setting/lock 原字节未变。任何失败保留已存在可用数据;新增正确 cache 项可留作重试复用。
### 必须正视的重放缺口
- `Session::python` (`build/python.rs:37`) 在 73/94 行调用 resolver.build_requirements,直接重用会重新解析,违反 install 合同。应提取固定依赖的执行阶段,供首次解析和锁重放共用。
- Python receipt source 被写成 `build/source/<hash>/source` (`python.rs:144`),没有 filename/格式;`source_files` (`196`) 需要 `.tar.gz` 或 `.zip`。可用 receipt 根的精确 PyPI metadata 按 source hash 找回文件名,但若 metadata 不再包含该源,必须报缺失。
- npm `sources` 在 optional 构建失败剪枝**前**采集 (`build/npm.rs:46`),receipt.graph 在 `prune_failed` **后**保存 (`72/87`)。失败节点 spec/owner 可能已丢,sources 只有散列路径,故当前 receipt 不保证可以完整重放原环境;不能凭摘要路径猜原包名。
- 若需要对新生成锁完整恢复,应在首次解析时保存 source locator/filename 与 npm 原始执行图(不同于最终消费图),并在 doctor/identity 中校验。不要添加旧格式兼容分支;按仓库“新语言直接迁移合同”规则迁移 fixtures/callers。已有锁缺最终产物且缺恢复信息时,明确说明不可恢复,不能静默 resolve。
- 即使输入齐全,任意第三方 hook/wheel 打包不保证字节确定性。重放输出必须匹配原 SHA,否则明确失败;没有远端保存原产物的机制就不能保证所有第三方构建永远能在全新机器复现。PRD 的精确恢复应以这一合同验证,不能以“版本相同”代替“字节相同”。
- `ManagedArtifactStore::get_exact` (`libs.rs:893`) 调 `artifact_get(...).map(Some)`;当前 miss、损坏、IPC 失败都 Err。`service.rs:327/515` 未提供 typed miss。必须增加明确 absence 结果或专用有界 existence/read 协议,不能 `Err => 下载`、不能匹配错误文本。缓存损坏和 service 不可达应继续失败。
### target add 与 CLI
`main.rs:435` 的 Lib operation 白名单和两个 USAGE 都需 install;launcher `project_root` (`launcher.rs:174`) 已识别任意 lib operation,可原样按项目版本转发。
`dever target add linux-aarch64` 没有 root 参数,应在 launcher dispatch 使用当前选定的 active 版本并调用 extension owner;若期望 cwd pin,则需要与主设计明确一致,不能悄悄扩大既有 project_root 行为。它仅准备 target runtime/sandbox;第三方 ARM Lib 仍 `lib ... --target linux-aarch64`。
Python sdist/npm hook 在非 host target 上由 `build::require_execution_target` (`build.rs:13`) 明确拒绝,需要匹配隔离执行环境或完整目标产物。target add 不能解除这个约束。Go runtime 则支持 host tools + target stdlib。
### 最近的测试夹具(本次未执行)
- `test/dever-cli-tests/external_libs.rs:331/400/418`:LocalRegistry、FixtureArtifactStore、三生态精确归档;`:451` 目标选择;`:1333` 离线确定性;`:1397` setting 不变;`:1469` 错配配置;`:1508` Worker 合同;`:1649` 无 Lib Worker。
- `test/dever-cli-tests/external_libs/build.rs:241`:不兼容 sdist 在装工具前拒绝;`:260/359` backend fixture / PEP517 hooks;`:422` 官方 sdist 为显式重测试,不默认运行。
- `test/dever-cli-tests/external_libs/npm_build.rs:68`:完整固定 receipt;`:264` 不重跑 hooks 的锁图投影;`:421` optional 失败/bin/offline remove;可增加记录请求次数的精确 install 用例。
- `test/dever-cli-tests/packages.rs:121/174`:注入 transport/store、传递 Package→Lib;`:621` 移除不请求 registry;`:725` 原 setting 字节回滚;`:909` signed managed 全链路(重验收,不默认执行)。
- 新定向验收应覆盖冷 cache install 后锁字节不变、metadata 新增更高版本仍不选、Package→managed Worker、空 specs/无锁/声明漂移、typed miss 与损坏差异、固定 Python build deps、npm optional 原执行图、输出摘要不一致失败、ARM mismatch、已缓存构建输出不加载 build extension,以及普通 run/build 绝不发 ensure/download。
### 建议定稿的数据合同与实施所有权
当前源码实际是 `LOCK_FORMAT = "dever-lock-v5"` (`libs.rs:25`),部分 spec 文案仍写 v2;本次直接升级 **dever-lock-v6**,无旧格式 fallback。建议单一、最小模式:
```text
LockedArtifact += source: Option<RegistrySource>
RegistrySource = { ecosystem: Ecosystem, locator: String, filename: String }
Receipt.dependencies -> Receipt.inputs: Box<LockFile>
```
`source` 对 registry 原始归档必填,对最终本地构建 output/fixture 可为 None;locator 是受现有 RegistryTransport 官方 origin/path 策略约束的精确地址,filename 只允许单文件名并承担解包格式选择,不从 URL query 猜格式。SHA/长度/target 沿用同一 LockedArtifact,不引入第二份内容身份。既然只接受 v6,source 必须成为已知字段,不需要兼容性 serde default;可序列化为 null。receipt 没有独立外部文件,**锁 v6 就是其 schema 版本**,不要再加无独立生命周期的 receipt 格式版本。
`inputs` 复用现有 LockFile:Python 为固定 build requirements 的精确递归锁;npm 为剪枝前 `libraries + npm graph`,图中 build=None,inputs.builds 为空,彻底避免自身 receipt 环。当前 receipt.graph 继续保存完整构建结束后的结果图(包括 prune 结果),供离线消费者与 `remove` 后投影使用。`sources` 保留为执行输入摘要列表并由 doctor 确认等于 inputs 的原始归档集合(Python 为单 sdist,build requirements 另在 inputs)。npm receipt.input 图不可被 remove 剪枝;最终 lock.npm 可以投影,原 receipt 保持不变。
实施所有权建议:
| Owner | 文件/API | 验证责任 |
| --- | --- | --- |
| extension/cache worker | toolchain extension owner、service/cache;ensure/read/list 与 `artifact_get_optional` typed miss | 签名、只读缺失/损坏区分、跨 UID、并发、失败原子性 |
| libs integration worker | libs.rs、main.rs、packages.rs、mutation.rs;明确 preparation policy、`install_locked`、Package exact restoration | 公共命令、Package 先恢复、Worker 提前准备、锁字节不变、离线边界 |
| registry/receipt worker | libs/registry.rs、build.rs、build/python.rs、build/npm.rs;v6 source、`restore_locked_artifacts`、`Session::replay` | 固定来源/输入、拒绝重新解析、嵌套依赖、输出 hash 不同失败 |
这些是逻辑所有权,可由同一 implementer 串行完成;`libs.rs` 类型/API 应由主实现者统一修改,避免多个 worker 同写。
具体公共/内部接点建议:
```text
toolchain::artifact_get_optional(layout, sha256, bytes) -> Result<Option<Vec<u8>>, String>
packages::restore_locked(project_root, lock, transport, store) -> Result<(), String>
libs::restore::install(project_root, target, preparation_inputs, transport, store)
-> Result<PreparationReport, String>
registry::restore_artifact(source, artifact, transport, store) -> Result<(), String>
build::Session::replay(receipt, store, signed_runtimes) -> Result<(), String>
```
`Session::replay` 绝不接收可解析版本的 RegistryResolver;将 Environment.run 中只用到的 store/runtime 提取成参数,首次 resolve 与 replay 共用原隔离执行器。Python inspect/requires 若执行,只用来比对已锁 backend/requirements,不能更新 inputs;metadata/wheel 阶段沿用现有检查。npm 执行 input 图,验证产出的 optional outcomes/native entries/rejections/结果图及 output SHA 与 receipt 相等后 publish。先完成全部验证再发布最终 output,不能先 publish 后发现不一致。
最小可测直接路径:现有 `build.rs:260` 的内联 backend fixture 无网络包管理器,验证固定动态 build requirements;`npm_build.rs:68` 固定安装收据与 `:421` optional 生命周期 fixture 验证原始图;在内存/目录 ArtifactStore 清空最终 output 后重放,两次字节一致。另加 hook 有意生成变化输出的 fixture,要求明确失败且锁字节保持。真实 C extension/node-gyp 仍复用现有显式 native fixture,不能把纯脚本 replay 通过报告成原生编译验收。
### External references
未查外部网络;本研究以仓库实现为准。协议事实来自现有代码:PyPI PEP440/508 + wheel、npm semver/SRI/实例图、Go proxy/MVS/sumdb、PEP517。版本绑定以 lock.TOOLCHAIN 和 signed RuntimePack/pack::Descriptor 为准,不假定系统 Python/Node/Go 版本。
## Caveats / Not Found
- 未实现任何功能、未运行测试/build/download、未修改产品或 specs。
- Extension catalog/IPC 的最终结构由发行路径研究和主代理决定;本文提供实际 caller 需求,不重复设计签名/安装 owner。
- 现有锁可精确获取大多数普通 registry 归档,但并非所有构建收据都包含完整重放信息;这是当前实现完整性的实际缺口,不是通过更多 retry 可解决的问题。
@@ -0,0 +1,101 @@
# Research: 精简发行、签名扩展与机器更新
- Query: 核对现有签名发行、bootstrap、共享机器资源与更新契约,给出最小完整拆分方案。
- Scope: mixed;源码只读,外部仅查解码 API 文档。
- Date: 2026-10-04
## Findings
### 现状与文件归属
| 文件 | 归属与证据 |
| --- | --- |
| `crates/dever-cli/src/toolchain/release.rs` | `ReleaseManifest`(74)只有 format/version/platform/artifacts;`MachineManager` 安装(293)、更新(310)、只读 core(437)、编译租约(470)、签名验证(579)、文件验证(616)、安装权限(968);稳定 `install.lock` inode(713)。 |
| `crates/dever-cli/src/toolchain/release_source.rs` | 官方 URL/重定向白名单(16、119)、元数据先验签(184)、同版本清单字节不变(213)、有界 gzip 下载(223)、catalog(250)、严格 USTAR 文件白名单(280)、NOREPLACE 发布(343)。 |
| `crates/dever-cli/src/toolchain/packaging.rs` | 固定 author 输入与签名;assemble(141)将 native、生态、sandbox、build 全部加入同一清单(202–234);必须在此按语义分类,不在压缩脚本根据随机路径猜测归属。 |
| `toolchain/packaging/{ecosystems,builds,sandbox,bootstrap}.rs` | 各类资源本身的语义校验与打包复用点;bootstrap(18)已有独立 ELF loader closure。 |
| `sdk/release-assets.py` | 所有清单 artifact 输出一个 deterministic gzip USTAR(create);目录发布使用 renameat2 NOREPLACE(publish)。 |
| `skills/scripts/install.py` | Python/OpenSSL 独立锚定公钥;authenticate(79)先验签,copy_artifact(129)精确字节/SHA 校验,payload(148)严格 tar 流,最后执行 bootstrap/dever。 |
| `toolchain/bootstrap.rs` 与 `bootstrap/transaction.rs` | 固定 bootstrap 系统落点、可信 key pin、崩溃恢复;SERVICE(19)仅允许写 state/cache/run,不允许服务写 versions 或新顶层目录。 |
| `toolchain/service.rs` | IPC Operation(100)只有 Status/Clean/ArtifactPut/ArtifactGet/Compile;内核 UID 认证(440),外 UID 可编译/资源缓存,不能 Clean;socket 0666、8 workers(34–36、154)。 |
| `toolchain/service/compile.rs` | compilation 租约、cache identity(74–86)、verify_pack(120)逐项验证所选 runtime 文件属于签名 manifest,cache hit 前也检查。 |
| `toolchain/runtime_pack.rs` | load_manifest(99)接受资源根目录,内部追加 runtime/<platform>;公开 validate(142)、RuntimePack::load(173)。 |
| `crates/dever-cli/src/compile.rs` | daemon 子进程依然在 worker(149)用 compiler_directory 加载 runtime;仅改 daemon 的 verify_pack 不够。 |
| `crates/dever-cli/src/libs.rs` | InstalledRegistry(709);tools(726)、runtime(797)、signed_artifact(823)、sandbox_resources(841)直接读版本根与根清单。 |
| `toolchain/cache.rs` | CacheStore 根实际为 cache/native(118),其中 ArtifactPut 单文件上限 64 MiB(81);不能用此 opaque artifact 上传接口安装 256 MiB 运行包。 |
| `toolchain/launcher.rs` | install/update 先下载后调用 MachineManager(27);当前 update 仅 core+skill,target 命令尚不存在。 |
现有清单并非全目录严格白名单:verify_artifacts 验证声明文件,core_libraries 对 lib 目录另有精确 closure 检查。拆分后不能把“目录存在”或只读 root metadata 当作签名证明。所有消费者仍需验证所属声明和文件字节。
### 建议的数据契约:一个签名根清单
将现有根清单明确升级新格式,保留 `version/platform/artifacts`,增加有界 `extensions`。`artifacts` 仅基础必需项;`extensions` 每项为 `{kind, target, artifacts}`。enum `ExtensionKind = Runtime(Ecosystem) | Build(Ecosystem) | Target`;身份为 `(version, host_platform, kind, deployment_target, signed_catalog_sha256)`,内容身份可对该项规范化完整描述计算 SHA-256。根签名覆盖所有扩展文件完整 path/bytes/sha256,不需要第二套扩展密钥或可变 unsigned receipt 作为信任来源。
基础内容:core、LLVM 完整 private closure、本机四 profile runtime、host sandbox、bootstrap、skill/templates。Target(ARM) 包含 ARM native 四 profile/CRT 与 ARM sandbox。Runtime(pip/npm/go,target) 与 Build(pip/npm,target) 独立;Go runtime 已包含 host 工具和 target stdlib,不强造 build/go。无基础文件与扩展间路径重叠,也无扩展互相路径重叠。
根清单 validator 统一校验版本/host、枚举目标组合、重复 ID、规范路径、lowercase SHA、数量/字节溢出、每包上限与全部描述上限。按固定类型限制 logical path:runtime/<ecosystem>/<target>、build/<ecosystem>/<target>、Target 对应 runtime/<target> 与 sandbox/<target>。防止扩展声明覆盖 core、lib、bootstrap、skill 或另一个 target。元数据顶层需 deny_unknown_fields;Python 同步语义,不维持兼容分叉。
外部 asset 名从固定 enum 计算;客户端与 manifest 不传任意 URL。每包一个 tar.zst;基础清单和签名单独发布;bootstrap 解码器作为原始文件单独发布并由根清单绑定。先签名固定原始内容,再压缩;确定性 USTAR 排序、mtime/uid/gid 和完整文件集继续成立。
### 扩展存储、接口与权限
推荐 `/opt/dever/cache/extensions/<version>/<catalog-sha>/<kind-target>/`;位于当前 service 可写范围内,和 `cache/native` 的可清理缓存分开。扩展先在专用 owner-private staging 中完全解压/验签/校验语义,fsync 文件和子目录,再 NOREPLACE rename 到最终路径。终态目录 0755、数据 0644,root-owned、不可被组/其他用户写;无需把 /opt/dever/cache 或 state 改成 0700。终态存在时重验内容而不是静默覆盖;失败仅清理自己 staging。manifest 摘要不同但版本相同必须失败,不创建并行“修正版”身份。
```rust
enum Operation {
// 现有操作……
EnsureExtension { version: Version, extension: ExtensionId },
}
struct ExtensionId { kind: ExtensionKind, target: BuildTarget }
// 网络/落盘仅 daemon 或拥有机器写权限的更新 owner 调用。
fn ensure_extension(layout: &Layout, version: &Version, id: &ExtensionId) -> Result<(), String>;
// 只读;不得打开 root-only install.lock,也不得下载。
fn resolve_extension(layout: &Layout, version: &Version, id: &ExtensionId) -> Result<VerifiedExtension, String>;
```
IPC 只接受已安装版本与其已验签目录声明的有限扩展,返回成功/内容身份而非用户指定路径。复用内核 UID/admission 和严格官方 HTTPS 下载,按请求限时、压缩/解压字节和数量限额;拒绝 caller 提供 URL、任意 filesystem path、payload 或 trust key。没有资源时 readonly resolve 返回准确 `dever lib install ...` 或 `dever target add linux-aarch64` 准备命令;run/build/check 不隐式进入 ensure。
采用一个 `SignedResources` 只读 owner,负责 root catalog、路径归属和已安装 extension 定位,供 InstalledRegistry 与 native pack 共用。`read_artifact(logical_path)` 与 `artifacts(prefix)` 只看到选定基础/扩展文件;避免构造 unsigned merged ReleaseManifest。RuntimePack::load 接收其实际资源根(base root 或 Target extension root),保留 logical path 到签名 artifacts 的逐项绑定。
SignedResources 需同时接入 libs.rs runtime/tools/sandbox_resources、service/compile.rs verify_pack、compile.rs worker RuntimePack::load。LLVM 动态库仍从 core 根加载。daemon BuildIdentity 必须包含所选 target pack 的签名/内容身份;不要让未使用扩展改变本机编译 cache identity。cache hit 仍重验所选 target pack。
### 更新与原子性
在同一个安装事务 owner 下读取 active 版本和其已安装扩展 ID 集合,验证新根清单有对应完整定义,下载并准备新 core+skill+这些扩展,最终重验后才 append active-version。失败保持旧 journal、旧 core 和扩展;已完整准备的新 immutable bytes 可留待重试复用,不声称更新成功。不自动安装新清单里所有扩展。
扩展选择直接由严格校验后的已发布目录集合导出,无需增加一个独立可漂移 selection pointer。若更新期间允许并行 ensure,必须在提交前锁内重新读取旧 active 扩展集合并补齐;更简单是将 ensure 与 update 的集合采样/发布串行化,避免丢失更新期间新增的扩展。沿用现有持久 inode lock,但不能在持有 exclusive lock 时递归调用再次 acquire 的公用 API;拆 locked 内部路径。当前锁为 try_lock,冲突会明确失败;如需“并发两项目都成功”应有限等待并在获锁后重验,不引入无限等待。
私有 staging 与 release.cleanup_staging 的全目录清扫不能冲突:若沿用 cache/staging,所有操作必须持同一安装锁;独立下载阶段则必须独占子树并禁止被旧 cleanup 删除。cache clean 目前只清 cache/native,保持其不删除 installed extensions。卸载版本时同安装锁删除对应扩展命名空间,active 保护不变。
当前 `dever update` 不更新机器 `/opt/dever/bin/dever[d]`,只更新版本目录/active。新资源 IPC 必須在首装 bootstrap launcher/daemon 中就存在;后续若改变 IPC,旧 daemon 兼容性是单独的 bootstrap 生命周期问题,不能假设 active core 更新自动重启了新服务。
### Zstandard:统一 Rust 解码,Python 验证后执行已有 launcher
推荐 zstd crate 的安全 streaming wrapper,静态编入发行时所需 decoder library;不要手写 C FFI 或 ctypes 依赖宿主 libzstd。Rust release downloader 与 launcher 隐藏 extraction 入口复用一个 codec/archive 模块:固定 `window_log_max(27)` 对应 128 MiB、压缩输入上限、解压输出上限、精确 frame 结束与 EOF 检查。不得仅 tar 遍历到结束标记就成功,必须排空/校验 zstd frame 尾部,拒绝截断与尾随额外数据;默认 Decoder 会连续解多个 frame,单 frame 策略需显式实施。
首装 Python 保留 OpenSSL 签名锚:下载/验证根 manifest → 从基础 artifacts 的固定 bootstrap/dever 与 bootstrap/lib/* 集合,按 digest 命名的 raw assets 下载全部 launcher closure → 精确大小与 SHA 验证、权限落盘 → 以空环境、绝对路径执行隐藏 `--dever-extract <owned-config-root>` → launcher 再验证根清单签名并调用共享 Rust USTAR 解包 → 完整已校验后执行 bootstrap 安装。配置与 trust key 仅放 root-owned scratch,不从应用读取。raw helper 文件与 payload 目标必须是两个目录,避免 create_new 冲突。需要 subprocess timeout 与失败后 kill/wait,不能把解包失败当成可继续安装。
现有 staged manifest 声明 `bootstrap/lib/libgcc_s.so.1`(`target/release-0.1.0/assets/dever-linux-x86_64.manifest.json:27`)。因此直接下载裸 launcher 不成立;但按主 agent 收敛方案,额外发布已签名 raw launcher + 完整小型私有 lib closure 后,复用 launcher 优于新 decoder bin:没有额外执行文件/第二套 tar validator。raw asset 文件名由既有 SHA 推导,不能由远端任意 URL 或目录控制。作者阶段仍需验证 ELF DT_NEEDED、RPATH、完整私有 closure,确保 zstd 为静态且 LLVM 不被引入 launcher/daemon;Linux libc/loader 为已声明 OS ABI。Python 在执行前只能信赖独立签名和已固定路径规则,签名发行 maker 对 closure 的检查仍是必要发行门槛。
Python SDK maker 可以使用作者提供的 zstd 工具或编码库完成 deterministic streaming tar.zst;这属于 author 工具链,不把系统 zstd 加成最终用户前提。公共资产总大小应单独计入 raw bootstrap closure,而不是只报告主 tar.zst。
### External references
- [zstd Decoder API](https://docs.rs/zstd/latest/zstd/stream/read/struct.Decoder.html),本次页面为 0.14.0:Read streaming、single_frame、window_log_max;finish_frame 错误需要显式处理。版本选型应核对仓库 MSRV 并固定 Cargo.lock,本次没有安装或锁定依赖。
- 本地 SDK `target/backend-sdk/usr/include/zstd.h:107` 为 1.5.5;`:606` 说明 streaming window 上限控制、`:648` 提供 DCtx 参数 API。这是本地已有头文件证据,不代表最终打包采用此版本。
- GitHub zstd-sys Cargo.toml 页面读取失败;静态链接需在实施中核对选定 crate build 配置与产物 ELF,不能将本研究当作已验证链接证据。
### Related specs
- `.trellis/spec/backend/toolchain-and-library.md:23` compiler-relative runtime 与 daemon 签名闭包;`:27` bootstrap;`:84–96` release/skill/author/ELF closure;`:337–343` onboarding、native release、shared toolchain 定向验收。
- `.trellis/spec/backend/index.md`:本仓库是 Rust 语言编译器,不采用 Go Dever Model/Page/Service 规则。
- `.trellis/tasks/10-04-slim-distribution/prd.md`:离线 run/build、多用户共享、按需自动生态、target 命令、core+已装扩展更新、不改宿主服务。
## Caveats / Not Found
- 子 agent `task.py current --source` 返回 none;主 agent 已明确指定此任务与写入位置,未修改任务状态。
- 本次未下载发行/依赖、未运行构建/测试、未改产品文件/服务,也未读取私钥或授权凭据。仅查看公开解码 API 文档。
- 120 MiB 目标是已有压缩估算;codec 与新正式优化 core 会改变最终数字,尚无本次测量。
- 精简需同时调整 package/Lib 显式准备策略:另一研究分支已发现 `source_workers_with_packages` 在 resolve_project 前读 runtime;不能只在 resolver 内 ensure。remove/doctor/check/run/build 必须 readonly。
- 更新携带范围建议 active 版本已安装选择;若产品要求机器所有历史 pinned 版本的 union,应在设计明确,不应默默给新版本安装历史上任何扩展。
- 大包首次安装、损坏重试、多 UID 同时 ensure、更新失败仍运行旧版、ARM cache hit 校验、decoder 超大 window/截断/尾随数据均需定向验收;本研究没有执行验证。
@@ -0,0 +1,26 @@
{
"id": "slim-distribution",
"name": "slim-distribution",
"title": "Dever 精简发行与按需工具安装",
"description": "将基础包缩至约 120 MiB,拆分签名扩展并完成 lib install、target add、更新及实际发行验收",
"status": "in_progress",
"dev_type": null,
"scope": null,
"package": null,
"priority": "P2",
"creator": "Antigravity",
"assignee": "Antigravity",
"createdAt": "2026-10-04",
"completedAt": null,
"branch": "codex/slim-distribution",
"base_branch": "main",
"worktree_path": null,
"commit": null,
"pr_url": null,
"subtasks": [],
"children": [],
"parent": null,
"relatedFiles": [],
"notes": "",
"meta": {}
}
Generated
+47 -6
View File
@@ -150,6 +150,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "005ec2760ca554fae18df7a11195552ec576cd665632a881bc011d5bb2fd4d80" checksum = "005ec2760ca554fae18df7a11195552ec576cd665632a881bc011d5bb2fd4d80"
dependencies = [ dependencies = [
"find-msvc-tools", "find-msvc-tools",
"jobserver",
"libc",
"shlex", "shlex",
] ]
@@ -428,7 +430,7 @@ dependencies = [
[[package]] [[package]]
name = "dever-backend-bridge" name = "dever-backend-bridge"
version = "0.1.0" version = "0.1.1"
dependencies = [ dependencies = [
"cc", "cc",
"dever-core", "dever-core",
@@ -441,7 +443,7 @@ dependencies = [
[[package]] [[package]]
name = "dever-cli" name = "dever-cli"
version = "0.1.0" version = "0.1.1"
dependencies = [ dependencies = [
"base64 0.22.1", "base64 0.22.1",
"csv", "csv",
@@ -464,11 +466,12 @@ dependencies = [
"ureq", "ureq",
"url", "url",
"zip", "zip",
"zstd",
] ]
[[package]] [[package]]
name = "dever-core" name = "dever-core"
version = "0.1.0" version = "0.1.1"
dependencies = [ dependencies = [
"dever-runtime", "dever-runtime",
"pulldown-cmark", "pulldown-cmark",
@@ -478,7 +481,7 @@ dependencies = [
[[package]] [[package]]
name = "dever-runtime" name = "dever-runtime"
version = "0.1.0" version = "0.1.1"
dependencies = [ dependencies = [
"argon2", "argon2",
"base64 0.22.1", "base64 0.22.1",
@@ -511,7 +514,7 @@ dependencies = [
[[package]] [[package]]
name = "dever-sandbox" name = "dever-sandbox"
version = "0.1.0" version = "0.1.1"
dependencies = [ dependencies = [
"command-fds", "command-fds",
"goblin", "goblin",
@@ -524,7 +527,7 @@ dependencies = [
[[package]] [[package]]
name = "dever-tests" name = "dever-tests"
version = "0.1.0" version = "0.1.1"
dependencies = [ dependencies = [
"bytes", "bytes",
"dever-core", "dever-core",
@@ -1047,6 +1050,16 @@ version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "jobserver"
version = "0.1.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3"
dependencies = [
"getrandom 0.4.3",
"libc",
]
[[package]] [[package]]
name = "js-sys" name = "js-sys"
version = "0.3.105" version = "0.3.105"
@@ -2662,3 +2675,31 @@ dependencies = [
"log", "log",
"simd-adler32", "simd-adler32",
] ]
[[package]]
name = "zstd"
version = "0.13.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a"
dependencies = [
"zstd-safe",
]
[[package]]
name = "zstd-safe"
version = "7.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "64d80649ab6db9d9f6f9c80a40becd948eda4714a0a5ac8c4d157a32231c7882"
dependencies = [
"zstd-sys",
]
[[package]]
name = "zstd-sys"
version = "2.1.0+zstd.1.5.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ef0a8027ec3ee71300ab3bcbcd0393f434aa72b91ca6d635a39941deae8eea0"
dependencies = [
"cc",
"pkg-config",
]
+2 -1
View File
@@ -3,7 +3,7 @@ members = ["crates/dever-core", "crates/dever-runtime", "crates/dever-cli", "cra
resolver = "3" resolver = "3"
[workspace.package] [workspace.package]
version = "0.1.0" version = "0.1.1"
edition = "2024" edition = "2024"
rust-version = "1.95" rust-version = "1.95"
publish = false publish = false
@@ -46,6 +46,7 @@ chrono = { version = "=0.4.45", default-features = false, features = ["std"] }
[profile.release] [profile.release]
codegen-units = 1 codegen-units = 1
lto = "fat" lto = "fat"
strip = "symbols"
# Static archives retain linkable symbols; final program stripping belongs to LLD. # Static archives retain linkable symbols; final program stripping belongs to LLD.
[profile.runtime-pack] [profile.runtime-pack]
+3 -1
View File
@@ -182,7 +182,9 @@ external pip "worker/mail.py" {
Go Worker 使用锁定的目标专用 build pack 内的 compiler、linker 和标准库离线编译。入口源码声明 `package main`,按操作名导出 `func(context.Context, any, any) (any, error)`:`send` 对应 `Send`,`text.render` 对应 `TextRender`;生成入口静态绑定函数并由 Go 编译器检查签名,不要求业务手写 `main` 或注册表。构建按目标 build tags 选文件,支持 `go:embed`,导入只能来自标准库、当前 Adapter 源码、SDK 或锁定的 Go module 闭包。构建产物只携带 Worker 可执行文件与 checked 合同,启动不寻找系统 Go。参与解析的 `go.mod` 和最终 module ZIP 均验证官方 sumdb 签名、包含证明与 Go `h1`;项目现有锁保存 checkpoint,一致性和反回滚以该锁为锚,主动删除锁会重新建立锚。 Go Worker 使用锁定的目标专用 build pack 内的 compiler、linker 和标准库离线编译。入口源码声明 `package main`,按操作名导出 `func(context.Context, any, any) (any, error)`:`send` 对应 `Send`,`text.render` 对应 `TextRender`;生成入口静态绑定函数并由 Go 编译器检查签名,不要求业务手写 `main` 或注册表。构建按目标 build tags 选文件,支持 `go:embed`,导入只能来自标准库、当前 Adapter 源码、SDK 或锁定的 Go module 闭包。构建产物只携带 Worker 可执行文件与 checked 合同,启动不寻找系统 Go。参与解析的 `go.mod` 和最终 module ZIP 均验证官方 sumdb 签名、包含证明与 Go `h1`;项目现有锁保存 checkpoint,一致性和反回滚以该锁为锚,主动删除锁会重新建立锚。
`dever-lock-v5` 保存每个 Worker 的精确生态依赖。Python extras 使用规范化、排序后的 `pip:name[a,b]@version`,不同 Worker 不共享隐式 extras;实际选中 wheel 的 METADATA 决定依赖,原生 wheel 还必须符合签名解释器的 tags、扩展 ABI 和封闭动态库搜索规则。npm 保留每个安装实例的嵌套路径、peer 上下文、optional 省略原因和 bundled 归属,不能把不同依赖上下文压成一份全局版本表。 `dever-lock-v6` 保存每个 Worker 的精确生态依赖,以及原始归档的准确来源和文件名。Python extras 使用规范化、排序后的 `pip:name[a,b]@version`,不同 Worker 不共享隐式 extras;实际选中 wheel 的 METADATA 决定依赖,原生 wheel 还必须符合签名解释器的 tags、扩展 ABI 和封闭动态库搜索规则。npm 保留每个安装实例的嵌套路径、peer 上下文、optional 省略原因和 bundled 归属,不能把不同依赖上下文压成一份全局版本表。
`dever lib install <root>` 按已有锁恢复 Package、Lib 和必要运行环境,不重新选择版本,不改项目配置和锁文件。缺少构建产物时使用收据中的完整固定输入重放,并核对输出摘要;第三方构建不能重现相同字节时明确失败。运行环境和构建工具通过机器共享签名扩展按需准备,`run/build` 仍不联网。ARM 应用资源先用 `dever target add linux-aarch64` 准备。
显式 `dever lib add/update` 可用签名 build pack 构建 PEP 517 sdist、执行 npm registry 包的 `preinstall/install/postinstall` 和原生 addon 编译。build pack 必须提供目标运行时、头文件、编译器、sysroot、固定 shell 和工具闭包;缺少的工具或库明确报错,不寻找宿主环境。Python 的静态与动态 build requirements 使用独立精确锁,console scripts 安装到隔离构建 prefix;metadata 与 wheel hook 分进程执行,最终 wheel 必须匹配父进程保存的 metadata。npm publishing hooks 不作为 registry 安装钩子执行,可选依赖构建失败会裁剪其不可用闭包,必需依赖失败则整个准备失败。 显式 `dever lib add/update` 可用签名 build pack 构建 PEP 517 sdist、执行 npm registry 包的 `preinstall/install/postinstall` 和原生 addon 编译。build pack 必须提供目标运行时、头文件、编译器、sysroot、固定 shell 和工具闭包;缺少的工具或库明确报错,不寻找宿主环境。Python 的静态与动态 build requirements 使用独立精确锁,console scripts 安装到隔离构建 prefix;metadata 与 wheel hook 分进程执行,最终 wheel 必须匹配父进程保存的 metadata。npm publishing hooks 不作为 registry 安装钩子执行,可选依赖构建失败会裁剪其不可用闭包,必需依赖失败则整个准备失败。
+4 -1
View File
@@ -48,12 +48,15 @@ target/debug/examples/native-release <author-root> --output <new-release-directo
```bash ```bash
dever lib add <project-root> pip:fixture@1.0.0 dever lib add <project-root> pip:fixture@1.0.0
dever lib list <project-root> dever lib list <project-root>
dever lib install <project-root>
dever lib doctor <project-root> dever lib doctor <project-root>
dever lib update <project-root> dever lib update <project-root>
dever lib remove <project-root> pip:fixture@1.0.0 dever lib remove <project-root> pip:fixture@1.0.0
``` ```
当前仓库已接入 PyPI、npm registry 和 Go proxy 解析、校验下载及机器共享缓存。真实 provider 只从已验证的机器发行目录读取签名 runtime 描述和 pack;开发态缺少发行资产时明确报错,不使用宿主语言环境。v5 锁绑定 Worker 合同、Python extras、npm 嵌套实例和 Go sumdb 证据;不同 Worker 独立解析依赖。显式 Lib 准备可使用签名工具包构建 PEP 517 sdist、执行 npm 安装钩子及原生 addon 编译,源码、工具、构建依赖和产物统一绑定到收据;`run/build` 只复用已锁定输出。SDK 自动绑定 Port 同名操作并校验输入、结果与错误,见 `sdk/README.md` 和 `LANGUAGE.md`。Go Worker 使用锁定工具离线编译/链接,支持目标文件选择与 `go:embed`,启动不需要系统 Go。 当前仓库已接入 PyPI、npm registry 和 Go proxy 解析、校验下载及机器共享缓存。真实 provider 只从已验证的机器资源读取签名 runtime 描述和 pack;缺少资源明确报错,不使用宿主语言环境。v6 锁绑定 Worker 合同、Python extras、npm 嵌套实例、Go sumdb 证据、准确源文件与完整构建输入。`lib install` 按已有锁恢复,配置和锁文件字节不变;第三方构建无法重现锁定摘要时明确失败。不同 Worker 独立解析依赖。显式 Lib 准备可使用签名工具包构建 PEP 517 sdist、执行 npm 安装钩子及原生 addon 编译;`run/build` 只复用已锁定输出。SDK 自动绑定 Port 同名操作并校验输入、结果与错误,见 `sdk/README.md` 和 `LANGUAGE.md`。Go Worker 使用锁定工具离线编译/链接,支持目标文件选择与 `go:embed`,启动不需要系统 Go。
发行包分为基础包和可选扩展。基础包保留 Dever 编译器、本机四种数据库运行库和配套 skill;Python/Node/Go 由 `lib add/update/install` 按实际需要准备,源码构建路径再准备构建工具。`dever target add linux-aarch64` 准备 ARM 应用目标。同版本资源在机器上共享,普通 `run/build` 不下载;`dever update` 更新核心、skill 和当前已安装的扩展,全部准备成功后才切换活动版本。基础与扩展均采用 Zstandard,首装无需系统 zstd。
原生制作器已支持 Python/Node/Go runtime/build pack 的确定性打包与统一签名;Linux x86_64 三生态的签名安装、受管 `check/run/build` 和移除源码/机器目录后的无系统语言独立执行已通过,包含 Python/Node 原生依赖。Python sdist/extras/原生扩展、npm 高级依赖/安装钩子/native addon 和 Go sumdb 均已有实现及真实第三方依赖定向证据。Linux 签名首装、升级保护和两个项目/真实用户共享核心也已通过自有完整验收。Linux Worker 使用 OS 沙箱;当前 root 验收不代表受限 AppArmor 下非 root 部署通过。输入仍是私有作者资产和临时签名,尚未交付公开首次安装包或其他平台发行。具体流程见 [原生发行包制作流程](sdk/native-release.md)。 原生制作器已支持 Python/Node/Go runtime/build pack 的确定性打包与统一签名;Linux x86_64 三生态的签名安装、受管 `check/run/build` 和移除源码/机器目录后的无系统语言独立执行已通过,包含 Python/Node 原生依赖。Python sdist/extras/原生扩展、npm 高级依赖/安装钩子/native addon 和 Go sumdb 均已有实现及真实第三方依赖定向证据。Linux 签名首装、升级保护和两个项目/真实用户共享核心也已通过自有完整验收。Linux Worker 使用 OS 沙箱;当前 root 验收不代表受限 AppArmor 下非 root 部署通过。输入仍是私有作者资产和临时签名,尚未交付公开首次安装包或其他平台发行。具体流程见 [原生发行包制作流程](sdk/native-release.md)。
+6
View File
@@ -28,6 +28,7 @@ base64 = "=0.22.1"
zip = { version = "=2.4.2", default-features = false, features = ["deflate"] } zip = { version = "=2.4.2", default-features = false, features = ["deflate"] }
tar = { version = "=0.4.44", default-features = false } tar = { version = "=0.4.44", default-features = false }
flate2 = "=1.1.5" flate2 = "=1.1.5"
zstd = { version = "=0.13.3", default-features = false }
csv = "=1.4.0" csv = "=1.4.0"
goblin = { version = "=0.10.7", default-features = false, features = ["elf32", "elf64", "endian_fd", "std"] } goblin = { version = "=0.10.7", default-features = false, features = ["elf32", "elf64", "endian_fd", "std"] }
@@ -70,6 +71,11 @@ name = "native-acceptance-init"
path = "../../test/dever-cli-tests/support/native_acceptance_init.rs" path = "../../test/dever-cli-tests/support/native_acceptance_init.rs"
test = false test = false
[[example]]
name = "prepare-extensions"
path = "../../test/onboarding/prepare_extensions.rs"
test = false
[[test]] [[test]]
name = "native_release" name = "native_release"
path = "../../test/dever-cli-tests/native_release.rs" path = "../../test/dever-cli-tests/native_release.rs"
@@ -10,6 +10,14 @@ fn run() -> i32 {
if arguments.as_slice() == ["--dever-bootstrap-health"] { if arguments.as_slice() == ["--dever-bootstrap-health"] {
return 0; return 0;
} }
if let [flag, config] = arguments.as_slice()
&& flag == "--dever-extract"
{
return match dever_cli::toolchain::extract_config(std::path::Path::new(config)) {
Ok(()) => 0,
Err(error) => fail(error),
};
}
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
if let [flag, root] = arguments.as_slice() if let [flag, root] = arguments.as_slice()
&& flag == "--dever-install" && flag == "--dever-install"
+9 -1
View File
@@ -142,7 +142,15 @@ pub(super) fn worker_compile(
}, },
), ),
}; };
let pack = RuntimePack::load(&compiler_directory()?, profile, request.target)?; let (layout, version) =
managed_compiler()?.ok_or("compile worker requires a managed compiler")?;
let resources =
dever_cli::toolchain::SignedResources::load(&layout, &Version::parse(&version)?)?;
let root = resources.root(&format!(
"runtime/{}/manifest.json",
request.target.platform()
))?;
let pack = RuntimePack::load(&root, profile, request.target)?;
link_ir(&ir, &pack, build, output) link_ir(&ir, &pack, build, output)
} }
+121 -36
View File
@@ -15,6 +15,7 @@ use sha2::{Digest, Sha256};
pub mod build; pub mod build;
mod mutation; mod mutation;
mod registry; mod registry;
pub mod restore;
pub(crate) use mutation::ProjectMutation; pub(crate) use mutation::ProjectMutation;
pub use registry::npm; pub use registry::npm;
pub(crate) use registry::python_dependencies; pub(crate) use registry::python_dependencies;
@@ -22,7 +23,7 @@ pub use registry::sumdb;
pub use registry::{HttpRegistry, RegistryResolver, RegistryRuntime, RegistryTransport}; pub use registry::{HttpRegistry, RegistryResolver, RegistryRuntime, RegistryTransport};
pub(crate) use registry::{InstalledRegistryPack, archive_files}; pub(crate) use registry::{InstalledRegistryPack, archive_files};
const LOCK_FORMAT: &str = "dever-lock-v5"; const LOCK_FORMAT: &str = "dever-lock-v6";
const TOOLCHAIN: &str = env!("CARGO_PKG_VERSION"); const TOOLCHAIN: &str = env!("CARGO_PKG_VERSION");
#[derive(Clone, Debug, Eq, Ord, PartialEq, PartialOrd, Serialize, Deserialize)] #[derive(Clone, Debug, Eq, Ord, PartialEq, PartialOrd, Serialize, Deserialize)]
@@ -240,6 +241,15 @@ pub struct LockedArtifact {
pub path: String, pub path: String,
pub bytes: u64, pub bytes: u64,
pub sha256: String, pub sha256: String,
pub source: Option<RegistrySource>,
}
#[derive(Clone, Debug, Eq, Ord, PartialEq, PartialOrd, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct RegistrySource {
pub ecosystem: Ecosystem,
pub locator: String,
pub filename: String,
} }
#[derive(Clone, Debug, Eq, Ord, PartialEq, PartialOrd, Serialize, Deserialize)] #[derive(Clone, Debug, Eq, Ord, PartialEq, PartialOrd, Serialize, Deserialize)]
@@ -421,6 +431,7 @@ impl FixturePackage {
path: artifact.path.clone(), path: artifact.path.clone(),
bytes: artifact.bytes.len() as u64, bytes: artifact.bytes.len() as u64,
sha256: sha256(&artifact.bytes), sha256: sha256(&artifact.bytes),
source: None,
}) })
.collect::<Vec<_>>(); .collect::<Vec<_>>();
artifacts.sort(); artifacts.sort();
@@ -635,7 +646,7 @@ pub(crate) fn resolve_project(
lock.encode()?; lock.encode()?;
return Ok(lock); return Ok(lock);
} }
let managed = InstalledRegistry::current(target)?; let managed = PreparingRegistry(InstalledRegistry::current(target)?);
let mut runtimes = BTreeMap::new(); let mut runtimes = BTreeMap::new();
let mut ecosystems = all let mut ecosystems = all
.iter() .iter()
@@ -646,7 +657,10 @@ pub(crate) fn resolve_project(
ecosystems.insert(worker.ecosystem.parse()?); ecosystems.insert(worker.ecosystem.parse()?);
} }
for ecosystem in ecosystems { for ecosystem in ecosystems {
runtimes.insert(ecosystem.clone(), managed.runtime(ecosystem)?.0); runtimes.insert(
ecosystem.clone(),
build::Inputs::runtime(&managed, ecosystem)?.0,
);
} }
for worker in workers.iter().filter(|worker| worker.ecosystem != "exec") { for worker in workers.iter().filter(|worker| worker.ecosystem != "exec") {
let ecosystem: Ecosystem = worker.ecosystem.parse()?; let ecosystem: Ecosystem = worker.ecosystem.parse()?;
@@ -658,7 +672,7 @@ pub(crate) fn resolve_project(
} }
} }
let store = ManagedArtifactStore { let store = ManagedArtifactStore {
layout: managed.layout.clone(), layout: managed.0.layout.clone(),
}; };
let transport = HttpRegistry::official(); let transport = HttpRegistry::official();
let verified = sumdb::verify_chain(&prior_evidence, &sumdb::Verifier::official())?; let verified = sumdb::verify_chain(&prior_evidence, &sumdb::Verifier::official())?;
@@ -705,13 +719,40 @@ impl LibResolver for ProjectResolver<'_> {
} }
} }
#[derive(Clone)]
struct InstalledRegistry { struct InstalledRegistry {
layout: crate::toolchain::Layout, layout: crate::toolchain::Layout,
version_root: PathBuf, version: crate::toolchain::Version,
resources: crate::toolchain::SignedResources,
target: crate::toolchain::BuildTarget, target: crate::toolchain::BuildTarget,
} }
/// Only explicit dependency preparation constructs this capability. Offline
/// readers retain InstalledRegistry and cannot request an installation.
struct PreparingRegistry(InstalledRegistry);
impl PreparingRegistry {
fn ensure(&self, kind: crate::toolchain::ExtensionKind) -> Result<(), String> {
crate::toolchain::ensure_extension(&self.0.layout, &self.0.version, kind, self.0.target)
}
}
impl build::Inputs for PreparingRegistry {
fn runtime(&self, ecosystem: Ecosystem) -> Result<(RegistryRuntime, Vec<u8>), String> {
self.ensure(crate::toolchain::ExtensionKind::Runtime(ecosystem.clone()))?;
self.0.runtime(ecosystem)
}
fn tools(&self, ecosystem: Ecosystem) -> Result<(build::pack::Descriptor, Vec<u8>), String> {
self.ensure(crate::toolchain::ExtensionKind::Build(ecosystem.clone()))?;
build::Inputs::tools(&self.0, ecosystem)
}
fn assets(&self) -> Result<Vec<(String, Vec<u8>)>, String> {
build::Inputs::assets(&self.0)
}
fn diagnostic(&self, output: &[u8]) {
build::Inputs::diagnostic(&self.0, output);
}
}
impl build::Inputs for InstalledRegistry { impl build::Inputs for InstalledRegistry {
fn diagnostic(&self, output: &[u8]) { fn diagnostic(&self, output: &[u8]) {
if !output.is_empty() { if !output.is_empty() {
@@ -782,8 +823,9 @@ impl InstalledRegistry {
return Err("running Dever core is not the verified managed release".into()); return Err("running Dever core is not the verified managed release".into());
} }
Ok(Self { Ok(Self {
resources: crate::toolchain::SignedResources::load(&layout, &version)?,
layout, layout,
version_root: version_root.to_path_buf(), version,
target, target,
}) })
} }
@@ -815,40 +857,18 @@ impl InstalledRegistry {
} }
fn signed_artifact(&self, relative: &Path) -> Result<Vec<u8>, String> { fn signed_artifact(&self, relative: &Path) -> Result<Vec<u8>, String> {
let manifest: crate::toolchain::ReleaseManifest = serde_json::from_slice(
&fs::read(self.version_root.join("manifest.json"))
.map_err(|error| error.to_string())?,
)
.map_err(|error| format!("invalid signed release manifest: {error}"))?;
let name = relative.to_str().ok_or("runtime path is not UTF-8")?; let name = relative.to_str().ok_or("runtime path is not UTF-8")?;
let identity = manifest self.resources.read(name)
.artifacts
.iter()
.find(|artifact| artifact.path == name)
.ok_or_else(|| format!("signed release does not include runtime artifact '{name}'"))?;
let bytes = fs::read(self.version_root.join(relative))
.map_err(|error| format!("cannot read signed runtime artifact '{name}': {error}"))?;
if bytes.len() as u64 != identity.bytes || sha256(&bytes) != identity.sha256 {
return Err(format!(
"signed runtime artifact '{name}' failed identity verification"
));
}
Ok(bytes)
} }
fn sandbox_resources( fn sandbox_resources(
&self, &self,
target: crate::toolchain::BuildTarget, target: crate::toolchain::BuildTarget,
) -> Result<Vec<dever_core::native::EmbeddedResource>, String> { ) -> Result<Vec<dever_core::native::EmbeddedResource>, String> {
let manifest: crate::toolchain::ReleaseManifest = serde_json::from_slice(
&fs::read(self.version_root.join("manifest.json"))
.map_err(|error| error.to_string())?,
)
.map_err(|error| format!("invalid signed release manifest: {error}"))?;
let prefix = format!("sandbox/{}/", target.platform()); let prefix = format!("sandbox/{}/", target.platform());
let mut resources = Vec::new(); let mut resources = Vec::new();
let mut files = Vec::new(); let mut files = Vec::new();
for artifact in &manifest.artifacts { for artifact in self.resources.artifacts(&prefix)? {
let Some(relative) = artifact.path.strip_prefix(&prefix) else { let Some(relative) = artifact.path.strip_prefix(&prefix) else {
continue; continue;
}; };
@@ -896,7 +916,7 @@ impl ArtifactStore for ManagedArtifactStore {
bytes: u64, bytes: u64,
_target: &str, _target: &str,
) -> Result<Option<Vec<u8>>, String> { ) -> Result<Option<Vec<u8>>, String> {
crate::toolchain::artifact_get(&self.layout, sha256, bytes).map(Some) crate::toolchain::artifact_get_optional(&self.layout, sha256, bytes)
} }
} }
@@ -1237,13 +1257,28 @@ pub(crate) fn bound_program_workers(
program: &dever_core::hir::Program, program: &dever_core::hir::Program,
target: crate::toolchain::BuildTarget, target: crate::toolchain::BuildTarget,
) -> Result<Vec<LockedWorker>, String> { ) -> Result<Vec<LockedWorker>, String> {
let mut workers = program_workers(program)?; bind_workers(program_workers(program)?, target, false)
}
fn bind_workers(
mut workers: Vec<LockedWorker>,
target: crate::toolchain::BuildTarget,
prepare: bool,
) -> Result<Vec<LockedWorker>, String> {
if workers.iter().all(|worker| worker.ecosystem == "exec") { if workers.iter().all(|worker| worker.ecosystem == "exec") {
return Ok(workers); return Ok(workers);
} }
let managed = InstalledRegistry::current(target)?; let managed = InstalledRegistry::current(target)?;
for worker in &mut workers { for worker in &mut workers {
if worker.ecosystem != "exec" { if worker.ecosystem != "exec" {
if prepare {
crate::toolchain::ensure_extension(
&managed.layout,
&managed.version,
crate::toolchain::ExtensionKind::Runtime(worker.ecosystem.parse()?),
target,
)?;
}
worker.runtime = Some(managed.runtime(worker.ecosystem.parse()?)?.0.pack); worker.runtime = Some(managed.runtime(worker.ecosystem.parse()?)?.0.pack);
} }
} }
@@ -1876,9 +1911,22 @@ pub fn execute_for_target(
let _mutation = ProjectMutation::for_command(command, project_root)?; let _mutation = ProjectMutation::for_command(command, project_root)?;
let path = project_root.join("dever.lock"); let path = project_root.join("dever.lock");
match command { match command {
"install" => {
if !specs.is_empty() {
return Err(
"dever lib install accepts no Lib specs; it restores the existing dever.lock"
.into(),
);
}
restore::install(project_root, target).map(|report| report.summary())
}
"add" => { "add" => {
let mut requested = read_declarations(project_root)?; let mut requested = read_declarations(project_root)?;
let workers = source_workers(project_root, target)?; let workers = prepare_source_workers_with_packages(
project_root,
&crate::packages::sources(project_root)?,
target,
)?;
let package_libs = crate::packages::declared_libs(project_root)?; let package_libs = crate::packages::declared_libs(project_root)?;
if specs.is_empty() if specs.is_empty()
&& requested.is_empty() && requested.is_empty()
@@ -1920,7 +1968,11 @@ pub fn execute_for_target(
.map(|value| value.parse()) .map(|value| value.parse())
.collect::<Result<Vec<LibSpec>, _>>()? .collect::<Result<Vec<LibSpec>, _>>()?
}; };
let workers = source_workers(project_root, target)?; let workers = prepare_source_workers_with_packages(
project_root,
&crate::packages::sources(project_root)?,
target,
)?;
let package_libs = crate::packages::declared_libs(project_root)?; let package_libs = crate::packages::declared_libs(project_root)?;
if requested.is_empty() && workers.is_empty() && package_libs.is_empty() { if requested.is_empty() && workers.is_empty() && package_libs.is_empty() {
return Err("dever lib update requires declared libs or exact lib specs".into()); return Err("dever lib update requires declared libs or exact lib specs".into());
@@ -2174,6 +2226,29 @@ pub(crate) fn source_workers_with_packages(
project_root: &Path, project_root: &Path,
packages: &[dever_core::source::PackageSource], packages: &[dever_core::source::PackageSource],
target: crate::toolchain::BuildTarget, target: crate::toolchain::BuildTarget,
) -> Result<Vec<LockedWorker>, String> {
bind_workers(
unbound_source_workers(project_root, packages)?,
target,
false,
)
}
pub(crate) fn prepare_source_workers_with_packages(
project_root: &Path,
packages: &[dever_core::source::PackageSource],
target: crate::toolchain::BuildTarget,
) -> Result<Vec<LockedWorker>, String> {
bind_workers(
unbound_source_workers(project_root, packages)?,
target,
true,
)
}
fn unbound_source_workers(
project_root: &Path,
packages: &[dever_core::source::PackageSource],
) -> Result<Vec<LockedWorker>, String> { ) -> Result<Vec<LockedWorker>, String> {
let source_root = project_root.join("module"); let source_root = project_root.join("module");
if !source_root.exists() && packages.is_empty() { if !source_root.exists() && packages.is_empty() {
@@ -2193,7 +2268,7 @@ pub(crate) fn source_workers_with_packages(
.map(|error| error.render(&sources)) .map(|error| error.render(&sources))
.collect::<String>() .collect::<String>()
})?; })?;
bound_program_workers(&program, target) program_workers(&program)
} }
fn declaration_path(project_root: &Path) -> PathBuf { fn declaration_path(project_root: &Path) -> PathBuf {
@@ -2481,6 +2556,16 @@ fn validate_runtime(runtime: &RuntimePack) -> Result<(), String> {
} }
fn validate_artifact(artifact: &LockedArtifact) -> Result<(), String> { fn validate_artifact(artifact: &LockedArtifact) -> Result<(), String> {
if let Some(source) = &artifact.source {
crate::workers::valid_path(&source.filename)?;
if source.filename.contains('/')
|| source.locator.is_empty()
|| source.locator.len() > 8192
|| source.locator.chars().any(char::is_control)
{
return Err("invalid locked registry source locator or filename".into());
}
}
if artifact.target.is_empty() || artifact.bytes == 0 { if artifact.target.is_empty() || artifact.bytes == 0 {
return Err("artifact target/size is invalid".into()); return Err("artifact target/size is invalid".into());
} }
+65 -10
View File
@@ -63,7 +63,7 @@ pub struct Receipt {
pub frontend: String, pub frontend: String,
pub python: Option<PythonSystem>, pub python: Option<PythonSystem>,
pub dynamic_requires: Vec<String>, pub dynamic_requires: Vec<String>,
pub dependencies: Box<LockFile>, pub inputs: Box<LockFile>,
pub config_settings: BTreeMap<String, serde_json::Value>, pub config_settings: BTreeMap<String, serde_json::Value>,
pub output: LockedArtifact, pub output: LockedArtifact,
} }
@@ -90,6 +90,29 @@ pub struct Session<'a> {
} }
impl<'a> Session<'a> { impl<'a> Session<'a> {
pub(crate) fn replay(
&self,
receipt: &Receipt,
store: &dyn super::ArtifactStore,
) -> Result<(), String> {
if receipt.frontend != frontend(&receipt.ecosystem)? {
return Err("locked build frontend differs from this toolchain".into());
}
let bytes = match receipt.ecosystem {
Ecosystem::Pip => self.replay_python(receipt, store)?,
Ecosystem::Npm => self.replay_npm(receipt, store)?,
Ecosystem::Go => return Err("Go source build receipts are unsupported".into()),
};
if bytes.len() as u64 != receipt.output.bytes
|| super::sha256(&bytes) != receipt.output.sha256
{
return Err("fixed build replay is not byte reproducible: output differs from dever.lock; lock was not changed".into());
}
if store.publish(&bytes, &receipt.output.target)? != receipt.output.sha256 {
return Err("restored build cache identity mismatch".into());
}
Ok(())
}
pub fn new(inputs: &'a dyn Inputs) -> Self { pub fn new(inputs: &'a dyn Inputs) -> Self {
Self { Self {
inputs, inputs,
@@ -229,6 +252,9 @@ pub fn doctor(lock: &LockFile) -> Result<(), String> {
} }
fn doctor_at(lock: &LockFile, depth: usize) -> Result<(), String> { fn doctor_at(lock: &LockFile, depth: usize) -> Result<(), String> {
if lock.format != super::LOCK_FORMAT || lock.toolchain != super::TOOLCHAIN {
return Err("build input lock format or toolchain differs from this release".into());
}
if depth > 16 { if depth > 16 {
return Err("build dependency depth exceeds limit".into()); return Err("build dependency depth exceeds limit".into());
} }
@@ -361,7 +387,6 @@ fn doctor_at(lock: &LockFile, depth: usize) -> Result<(), String> {
&& (receipt.python.is_some() && (receipt.python.is_some()
|| receipt.python_markers.is_some() || receipt.python_markers.is_some()
|| !receipt.dynamic_requires.is_empty() || !receipt.dynamic_requires.is_empty()
|| !receipt.dependencies.libs.is_empty()
|| receipt.graph.is_none()) || receipt.graph.is_none())
{ {
return Err("npm build receipt contains Python backend state".into()); return Err("npm build receipt contains Python backend state".into());
@@ -372,20 +397,50 @@ fn doctor_at(lock: &LockFile, depth: usize) -> Result<(), String> {
if let Some(system) = &receipt.python { if let Some(system) = &receipt.python {
validate_requirements(receipt, system)?; validate_requirements(receipt, system)?;
} }
if !receipt.dependencies.workers.is_empty() if !receipt.inputs.workers.is_empty()
|| !receipt.dependencies.packages.is_empty() || !receipt.inputs.packages.is_empty()
|| receipt || receipt
.dependencies .inputs
.libs .libs
.iter() .iter()
.any(|lib| lib.spec.ecosystem != Ecosystem::Pip) .any(|lib| lib.spec.ecosystem != receipt.ecosystem)
{ {
return Err("build dependency closure is not a separate Python environment".into()); return Err("build input closure contains unrelated environments".into());
} }
doctor_at(&receipt.dependencies, depth + 1)?; if receipt.output.source.is_some()
|| receipt.sources.iter().any(|source| {
source
.source
.as_ref()
.is_none_or(|origin| origin.ecosystem != receipt.ecosystem)
})
{
return Err("build receipt lacks exact registry source identity".into());
}
if receipt.ecosystem == Ecosystem::Npm {
let mut sources = receipt
.inputs
.libs
.iter()
.flat_map(|lib| lib.artifacts.clone())
.collect::<Vec<_>>();
sources.sort();
sources.dedup();
if sources != receipt.sources
|| receipt.inputs.npm.len() != 1
|| receipt.inputs.npm[0].roots != receipt.roots
|| receipt.inputs.npm[0].build.is_some()
|| !receipt.inputs.builds.is_empty()
{
return Err("npm build input graph differs from its original sources".into());
}
} else if receipt.sources.len() != 1 || !receipt.inputs.npm.is_empty() {
return Err("Python build receipt has invalid source inputs".into());
}
doctor_at(&receipt.inputs, depth + 1)?;
// Full graph checks belong to the ordinary lock owner; recursion is // Full graph checks belong to the ordinary lock owner; recursion is
// bounded above before invoking that owner on a dependency lock. // bounded above before invoking that owner on a dependency lock.
let mut dependencies = (*receipt.dependencies).clone(); let mut dependencies = (*receipt.inputs).clone();
dependencies.builds.clear(); dependencies.builds.clear();
for lib in &mut dependencies.libs { for lib in &mut dependencies.libs {
lib.build = None; lib.build = None;
@@ -447,7 +502,7 @@ fn validate_requirements(receipt: &Receipt, system: &PythonSystem) -> Result<(),
let document = serde_json::json!({"info":{"requires_dist":requirements}}); let document = serde_json::json!({"info":{"requires_dist":requirements}});
let expected = super::python_dependencies(&document, markers, &[])?; let expected = super::python_dependencies(&document, markers, &[])?;
let libraries = receipt let libraries = receipt
.dependencies .inputs
.libs .libs
.iter() .iter()
.map(|lib| (lib.spec.name.as_str(), lib)) .map(|lib| (lib.spec.name.as_str(), lib))
+67 -22
View File
@@ -12,6 +12,43 @@ mod environment;
use environment::{Environment, Output}; use environment::{Environment, Output};
impl Session<'_> { impl Session<'_> {
pub(super) fn replay_npm(
&self,
receipt: &Receipt,
store: &dyn crate::libs::ArtifactStore,
) -> Result<Vec<u8>, String> {
let (runtime, _) = self.inputs.runtime(Ecosystem::Npm)?;
if runtime.pack != receipt.runtime {
return Err("locked npm build runtime changed".into());
}
let environment = Environment::new(self.inputs, &runtime)?;
if environment.descriptor != receipt.tools
|| Some(&environment.python_runtime) != receipt.auxiliary_runtime.as_ref()
{
return Err("locked npm build tools or auxiliary runtime changed".into());
}
let mut graph = receipt
.inputs
.npm
.first()
.ok_or("npm build input graph missing")?
.clone();
let (files, executables) = installation_inputs(store, &receipt.inputs, &graph)?;
let output = environment.run(self.inputs, &graph, &files, &executables)?;
prune_failed(&mut graph, &output.failed)?;
npm::validate_built_files(&graph, &output.files)?;
if Some(&graph) != receipt.graph.as_ref()
|| output.native != receipt.native_entries
|| output.rejections != receipt.native_rejections
{
return Err(
"npm fixed build replay changed graph, optional outcomes or native admission"
.into(),
);
}
archive(&output)
}
pub(crate) fn npm( pub(crate) fn npm(
&self, &self,
resolver: &RegistryResolver<'_>, resolver: &RegistryResolver<'_>,
@@ -21,26 +58,7 @@ impl Session<'_> {
) -> Result<(), String> { ) -> Result<(), String> {
let mut lock = LockFile::new(libraries.clone())?; let mut lock = LockFile::new(libraries.clone())?;
lock.npm.push(graph.clone()); lock.npm.push(graph.clone());
let archives = libraries let (files, executables) = installation_inputs(resolver.store, &lock, graph)?;
.iter()
.map(|lib| {
let artifact = lib.artifacts.first().ok_or("npm source archive missing")?;
Ok((
lib.spec.clone(),
resolver.store.verify_exact(
&artifact.sha256,
artifact.bytes,
&artifact.target,
)?,
))
})
.collect::<Result<BTreeMap<_, _>, String>>()?;
let files = npm::build_files(graph, &lock, |lib| {
archives
.get(&lib.spec)
.map(Vec::as_slice)
.ok_or("npm source archive missing".into())
})?;
if !npm::requires_build(&files)? { if !npm::requires_build(&files)? {
return Ok(()); return Ok(());
} }
@@ -66,7 +84,6 @@ impl Session<'_> {
retain_archives(libraries, graph); retain_archives(libraries, graph);
return Ok(()); return Ok(());
} }
let executables = npm::build_executables(graph, &archives)?;
let environment = Environment::new(self.inputs, runtime)?; let environment = Environment::new(self.inputs, runtime)?;
let output = environment.run(self.inputs, graph, &files, &executables)?; let output = environment.run(self.inputs, graph, &files, &executables)?;
prune_failed(graph, &output.failed)?; prune_failed(graph, &output.failed)?;
@@ -82,6 +99,7 @@ impl Session<'_> {
path: format!("build/npm/{digest}/installation.tgz"), path: format!("build/npm/{digest}/installation.tgz"),
bytes: bytes.len() as u64, bytes: bytes.len() as u64,
sha256: digest, sha256: digest,
source: None,
}; };
let receipt = Receipt { let receipt = Receipt {
ecosystem: Ecosystem::Npm, ecosystem: Ecosystem::Npm,
@@ -97,7 +115,7 @@ impl Session<'_> {
frontend: super::frontend(&Ecosystem::Npm)?, frontend: super::frontend(&Ecosystem::Npm)?,
python: None, python: None,
dynamic_requires: vec![], dynamic_requires: vec![],
dependencies: Box::new(LockFile::new(vec![])?), inputs: Box::new(lock),
config_settings: BTreeMap::new(), config_settings: BTreeMap::new(),
output: output_artifact, output: output_artifact,
}; };
@@ -114,6 +132,33 @@ impl Session<'_> {
} }
} }
type InstallationInputs = (Vec<(String, Vec<u8>)>, BTreeSet<String>);
fn installation_inputs(
store: &dyn crate::libs::ArtifactStore,
lock: &LockFile,
graph: &npm::Environment,
) -> Result<InstallationInputs, String> {
let archives = lock
.libs
.iter()
.map(|lib| {
let artifact = lib.artifacts.first().ok_or("npm source archive missing")?;
Ok((
lib.spec.clone(),
store.verify_exact(&artifact.sha256, artifact.bytes, &artifact.target)?,
))
})
.collect::<Result<BTreeMap<_, _>, String>>()?;
let files = npm::build_files(graph, lock, |lib| {
archives
.get(&lib.spec)
.map(Vec::as_slice)
.ok_or("npm source archive missing".into())
})?;
Ok((files, npm::build_executables(graph, &archives)?))
}
fn archive(output: &Output) -> Result<Vec<u8>, String> { fn archive(output: &Output) -> Result<Vec<u8>, String> {
let encoder = flate2::GzBuilder::new() let encoder = flate2::GzBuilder::new()
.mtime(0) .mtime(0)
+143 -54
View File
@@ -8,8 +8,8 @@ use std::path::{Path, PathBuf};
use super::{Product, PythonSystem, Receipt, Session, pack, process::Stage}; use super::{Product, PythonSystem, Receipt, Session, pack, process::Stage};
use crate::libs::{ use crate::libs::{
Ecosystem, LibSpec, LockFile, LockedArtifact, RegistryResolver, RegistryRuntime, archive_files, ArtifactStore, Ecosystem, LibSpec, LockFile, LockedArtifact, RegistryResolver, RegistryRuntime,
sha256, RegistrySource, archive_files, sha256,
}; };
use crate::workers::{python_wheel::Wheel, runtime}; use crate::workers::{python_wheel::Wheel, runtime};
use serde_json::{Value, json}; use serde_json::{Value, json};
@@ -23,6 +23,12 @@ struct Environment {
descriptor: pack::Descriptor, descriptor: pack::Descriptor,
} }
struct Execution<'a> {
store: &'a dyn ArtifactStore,
runtime: &'a RegistryRuntime,
inputs: &'a dyn super::Inputs,
}
struct Active<'a> { struct Active<'a> {
session: &'a Session<'a>, session: &'a Session<'a>,
key: String, key: String,
@@ -34,12 +40,69 @@ impl Drop for Active<'_> {
} }
impl Session<'_> { impl Session<'_> {
pub(super) fn replay_python(
&self,
receipt: &Receipt,
store: &dyn ArtifactStore,
) -> Result<Vec<u8>, String> {
let (runtime, _) = self.inputs.runtime(Ecosystem::Pip)?;
if runtime.pack != receipt.runtime || runtime.python_markers != receipt.python_markers {
return Err("locked Python build runtime or marker policy changed".into());
}
let environment = Environment::new(self, &runtime)?;
let execution = Execution {
store,
runtime: &runtime,
inputs: self.inputs,
};
if environment.descriptor != receipt.tools {
return Err("locked Python build tools changed".into());
}
let artifact = receipt
.sources
.first()
.ok_or("Python build source missing")?;
let origin = artifact
.source
.as_ref()
.ok_or("Python build source locator missing")?;
let bytes = store.verify_exact(&artifact.sha256, artifact.bytes, &artifact.target)?;
let source = source_files(&origin.filename, &bytes)?;
let system = receipt
.python
.as_ref()
.ok_or("Python build backend missing")?;
let empty = LockFile::new(vec![])?;
let inspected = environment.run(
&execution,
&empty,
&source,
json!({"phase":"inspect"}),
None,
)?;
let actual: PythonSystem =
serde_json::from_value(inspected.response).map_err(|error| error.to_string())?;
if &actual != system {
return Err("locked Python build backend configuration changed".into());
}
// The locked closure is final. Replaying never asks the resolver to
// discover or select static/dynamic build requirements again.
let (_, bytes) = environment.wheel(
&execution,
&receipt.inputs,
&source,
system,
receipt.roots.first().ok_or("Python build root missing")?,
)?;
Ok(bytes)
}
pub(crate) fn python( pub(crate) fn python(
&self, &self,
resolver: &RegistryResolver<'_>, resolver: &RegistryResolver<'_>,
spec: &LibSpec, spec: &LibSpec,
runtime: &RegistryRuntime, runtime: &RegistryRuntime,
filename: &str, source_origin: RegistrySource,
source: &[u8], source: &[u8],
) -> Result<Product, String> { ) -> Result<Product, String> {
let key = format!( let key = format!(
@@ -62,11 +125,21 @@ impl Session<'_> {
session: self, session: self,
key: key.clone(), key: key.clone(),
}; };
let sources = source_files(filename, source)?; let sources = source_files(&source_origin.filename, source)?;
let environment = Environment::new(self, runtime)?; let environment = Environment::new(self, runtime)?;
let execution = Execution {
store: resolver.store,
runtime,
inputs: self.inputs,
};
let empty = LockFile::new(vec![])?; let empty = LockFile::new(vec![])?;
let inspected = let inspected = environment.run(
environment.run(resolver, &empty, &sources, json!({"phase":"inspect"}), None)?; &execution,
&empty,
&sources,
json!({"phase":"inspect"}),
None,
)?;
let system: PythonSystem = serde_json::from_value(inspected.response) let system: PythonSystem = serde_json::from_value(inspected.response)
.map_err(|error| format!("invalid Python build configuration: {error}"))?; .map_err(|error| format!("invalid Python build configuration: {error}"))?;
let mut requirements = system.requires.clone(); let mut requirements = system.requires.clone();
@@ -75,7 +148,7 @@ impl Session<'_> {
let mut discovered = BTreeSet::new(); let mut discovered = BTreeSet::new();
for _ in 0..8 { for _ in 0..8 {
let result = environment.run( let result = environment.run(
resolver, &execution,
&dependencies, &dependencies,
&sources, &sources,
json!({"phase":"requires","system":system}), json!({"phase":"requires","system":system}),
@@ -99,38 +172,8 @@ impl Session<'_> {
dependencies = next; dependencies = next;
} }
let dynamic = stable.ok_or("Python dynamic build requirements did not converge")?; let dynamic = stable.ok_or("Python dynamic build requirements did not converge")?;
let metadata = environment.run( let (filename, bytes) =
resolver, environment.wheel(&execution, &dependencies, &sources, &system, spec)?;
&dependencies,
&sources,
json!({"phase":"metadata","system":system}),
None,
)?;
let result = environment.run(
resolver,
&dependencies,
&sources,
json!({"phase":"wheel","system":system,"metadata":metadata.response.get("metadata")}),
Some(&metadata.outputs),
)?;
let filename = result
.response
.get("filename")
.and_then(Value::as_str)
.ok_or("build backend omitted wheel filename")?;
crate::workers::valid_path(filename)?;
if filename.contains('/') || !filename.ends_with(".whl") {
return Err("build backend returned invalid wheel filename".into());
}
let bytes = result
.outputs
.get(&format!("wheels/{filename}"))
.ok_or("build backend did not produce its declared wheel")?
.clone();
Wheel::parse(spec, Some(filename), &runtime.python_wheel_tags, &bytes)?;
if let Some(prefix) = metadata.response.get("metadata").and_then(Value::as_str) {
compare_metadata(prefix, &metadata.outputs, &bytes)?;
}
let output = artifact( let output = artifact(
runtime, runtime,
format!( format!(
@@ -140,11 +183,12 @@ impl Session<'_> {
), ),
&bytes, &bytes,
); );
let source_artifact = artifact( let mut source_artifact = artifact(
runtime, runtime,
format!("build/source/{}/source", sha256(source)), format!("build/source/{}/source", sha256(source)),
source, source,
); );
source_artifact.source = Some(source_origin);
for (artifact, bytes) in [(&output, bytes.as_slice()), (&source_artifact, source)] { for (artifact, bytes) in [(&output, bytes.as_slice()), (&source_artifact, source)] {
if resolver.store.publish(bytes, &runtime.target)? != artifact.sha256 { if resolver.store.publish(bytes, &runtime.target)? != artifact.sha256 {
return Err("build artifact cache identity mismatch".into()); return Err("build artifact cache identity mismatch".into());
@@ -165,13 +209,13 @@ impl Session<'_> {
frontend: super::frontend(&Ecosystem::Pip)?, frontend: super::frontend(&Ecosystem::Pip)?,
python: Some(system), python: Some(system),
dynamic_requires: dynamic, dynamic_requires: dynamic,
dependencies: Box::new(dependencies), inputs: Box::new(dependencies),
config_settings: BTreeMap::new(), config_settings: BTreeMap::new(),
output, output,
}; };
let product = Product { let product = Product {
bytes, bytes,
filename: filename.into(), filename,
receipt, receipt,
}; };
self.remember(key, product.clone())?; self.remember(key, product.clone())?;
@@ -185,6 +229,7 @@ fn artifact(runtime: &RegistryRuntime, path: String, bytes: &[u8]) -> LockedArti
path, path,
bytes: bytes.len() as u64, bytes: bytes.len() as u64,
sha256: sha256(bytes), sha256: sha256(bytes),
source: None,
} }
} }
@@ -236,6 +281,54 @@ struct ResultFiles {
} }
impl Environment { impl Environment {
fn wheel(
&self,
execution: &Execution<'_>,
dependencies: &LockFile,
source: &Source,
system: &PythonSystem,
spec: &LibSpec,
) -> Result<(String, Vec<u8>), String> {
let metadata = self.run(
execution,
dependencies,
source,
json!({"phase":"metadata","system":system}),
None,
)?;
let result = self.run(
execution,
dependencies,
source,
json!({"phase":"wheel","system":system,"metadata":metadata.response.get("metadata")}),
Some(&metadata.outputs),
)?;
let filename = result
.response
.get("filename")
.and_then(Value::as_str)
.ok_or("build backend omitted wheel filename")?;
crate::workers::valid_path(filename)?;
if filename.contains('/') || !filename.ends_with(".whl") {
return Err("build backend returned invalid wheel filename".into());
}
let bytes = result
.outputs
.get(&format!("wheels/{filename}"))
.ok_or("build backend did not produce its declared wheel")?
.clone();
Wheel::parse(
spec,
Some(filename),
&execution.runtime.python_wheel_tags,
&bytes,
)?;
if let Some(prefix) = metadata.response.get("metadata").and_then(Value::as_str) {
compare_metadata(prefix, &metadata.outputs, &bytes)?;
}
Ok((filename.into(), bytes))
}
fn new(session: &Session<'_>, requested: &RegistryRuntime) -> Result<Self, String> { fn new(session: &Session<'_>, requested: &RegistryRuntime) -> Result<Self, String> {
super::require_execution_target(&requested.target, "Python source build")?; super::require_execution_target(&requested.target, "Python source build")?;
let (runtime, bytes) = session.inputs.runtime(Ecosystem::Pip)?; let (runtime, bytes) = session.inputs.runtime(Ecosystem::Pip)?;
@@ -291,12 +384,17 @@ impl Environment {
fn run( fn run(
&self, &self,
resolver: &RegistryResolver<'_>, execution: &Execution<'_>,
dependencies: &LockFile, dependencies: &LockFile,
source: &Source, source: &Source,
mut request: Value, mut request: Value,
prepared: Option<&BTreeMap<String, Vec<u8>>>, prepared: Option<&BTreeMap<String, Vec<u8>>>,
) -> Result<ResultFiles, String> { ) -> Result<ResultFiles, String> {
let Execution {
store,
runtime,
inputs,
} = execution;
let stage = Stage::new()?; let stage = Stage::new()?;
let input = stage.0.join("input"); let input = stage.0.join("input");
for (path, (bytes, executable)) in &self.files { for (path, (bytes, executable)) in &self.files {
@@ -322,10 +420,6 @@ impl Environment {
} }
} }
let mut installed = BTreeMap::<String, Vec<u8>>::new(); let mut installed = BTreeMap::<String, Vec<u8>>::new();
let runtime = resolver
.runtimes
.get(&Ecosystem::Pip)
.ok_or("build dependency runtime is missing")?;
let mut distributions = BTreeSet::new(); let mut distributions = BTreeSet::new();
for lib in &dependencies.libs { for lib in &dependencies.libs {
super::validate_python_policy( super::validate_python_policy(
@@ -342,10 +436,7 @@ impl Environment {
); );
} }
let artifact = &lib.artifacts[0]; let artifact = &lib.artifacts[0];
let bytes = let bytes = store.verify_exact(&artifact.sha256, artifact.bytes, &artifact.target)?;
resolver
.store
.verify_exact(&artifact.sha256, artifact.bytes, &artifact.target)?;
let wheel = Wheel::parse(&lib.spec, None, &runtime.python_wheel_tags, &bytes)?; let wheel = Wheel::parse(&lib.spec, None, &runtime.python_wheel_tags, &bytes)?;
wheel.validate_dependencies( wheel.validate_dependencies(
lib, lib,
@@ -432,9 +523,7 @@ impl Environment {
"isolated Python build hook", "isolated Python build hook",
&work, &work,
)?; )?;
if let Some(session) = resolver.build { inputs.diagnostic(&stage.diagnostics()?);
session.inputs.diagnostic(&stage.diagnostics()?);
}
let outputs = output_files(&work.join("output"))?; let outputs = output_files(&work.join("output"))?;
let response = serde_json::from_slice( let response = serde_json::from_slice(
outputs outputs
+1 -1
View File
@@ -10,7 +10,7 @@ pub(crate) struct ProjectMutation {
impl ProjectMutation { impl ProjectMutation {
pub(crate) fn for_command(command: &str, root: &Path) -> Result<Option<Self>, String> { pub(crate) fn for_command(command: &str, root: &Path) -> Result<Option<Self>, String> {
if !matches!(command, "add" | "update" | "remove") { if !matches!(command, "add" | "update" | "remove" | "install") {
return Ok(None); return Ok(None);
} }
let directory = fs::symlink_metadata(root) let directory = fs::symlink_metadata(root)
+44 -6
View File
@@ -19,7 +19,7 @@ pub mod sumdb;
use super::{ use super::{
ArtifactStore, Ecosystem, LibResolver, LibSpec, LockFile, LockedArtifact, LockedDependency, ArtifactStore, Ecosystem, LibResolver, LibSpec, LockFile, LockedArtifact, LockedDependency,
LockedLib, ProviderManifest, RuntimePack, sha256, validate_artifact, LockedLib, ProviderManifest, RegistrySource, RuntimePack, sha256, validate_artifact,
}; };
const MAX_METADATA_BYTES: usize = 16 * 1024 * 1024; const MAX_METADATA_BYTES: usize = 16 * 1024 * 1024;
@@ -553,7 +553,7 @@ impl RegistryResolver<'_> {
runtime, runtime,
"whl", "whl",
&candidate.bytes, &candidate.bytes,
"pypi-wheel-v1", candidate.source.clone(),
)?; )?;
lib.build = candidate.build.clone(); lib.build = candidate.build.clone();
distributions.insert(base.to_owned(), lib.clone()); distributions.insert(base.to_owned(), lib.clone());
@@ -925,6 +925,11 @@ impl RegistryResolver<'_> {
} }
Ok(Some(PythonCandidate { Ok(Some(PythonCandidate {
build: None, build: None,
source: Some(RegistrySource {
ecosystem: Ecosystem::Pip,
locator: url.into(),
filename: field(file, "filename")?.into(),
}),
bytes, bytes,
info: serde_json::json!({"requires_dist":wheel.requirements,"provides_extra":wheel.extras}), info: serde_json::json!({"requires_dist":wheel.requirements,"provides_extra":wheel.extras}),
})) }))
@@ -965,7 +970,17 @@ impl RegistryResolver<'_> {
if sha256(&bytes) != expected { if sha256(&bytes) != expected {
return Err("PyPI source SHA-256 mismatch".into()); return Err("PyPI source SHA-256 mismatch".into());
} }
let product = session.python(self, spec, runtime, field(file, "filename")?, &bytes)?; let product = session.python(
self,
spec,
runtime,
RegistrySource {
ecosystem: Ecosystem::Pip,
locator: field(file, "url")?.into(),
filename: field(file, "filename")?.into(),
},
&bytes,
)?;
let wheel = crate::workers::python_wheel::Wheel::parse( let wheel = crate::workers::python_wheel::Wheel::parse(
spec, spec,
Some(&product.filename), Some(&product.filename),
@@ -980,6 +995,7 @@ impl RegistryResolver<'_> {
} }
Ok(Some(PythonCandidate { Ok(Some(PythonCandidate {
bytes: product.bytes, bytes: product.bytes,
source: None,
info: serde_json::json!({"requires_dist":wheel.requirements,"provides_extra":wheel.extras}), info: serde_json::json!({"requires_dist":wheel.requirements,"provides_extra":wheel.extras}),
build: Some(product.receipt.identity()?), build: Some(product.receipt.identity()?),
})) }))
@@ -1012,7 +1028,7 @@ impl RegistryResolver<'_> {
runtime: &RegistryRuntime, runtime: &RegistryRuntime,
extension: &str, extension: &str,
bytes: &[u8], bytes: &[u8],
schema: &str, source: Option<RegistrySource>,
) -> Result<LockedLib, String> { ) -> Result<LockedLib, String> {
let files = archive_files(extension, bytes)?; let files = archive_files(extension, bytes)?;
if spec.ecosystem == Ecosystem::Go { if spec.ecosystem == Ecosystem::Go {
@@ -1035,6 +1051,7 @@ impl RegistryResolver<'_> {
path, path,
bytes: bytes.len() as u64, bytes: bytes.len() as u64,
sha256: sha256(bytes), sha256: sha256(bytes),
source,
}; };
validate_artifact(&artifact)?; validate_artifact(&artifact)?;
let published = self.store.publish(bytes, &runtime.target)?; let published = self.store.publish(bytes, &runtime.target)?;
@@ -1052,7 +1069,12 @@ impl RegistryResolver<'_> {
dependencies, dependencies,
runtime: runtime.pack.clone(), runtime: runtime.pack.clone(),
artifacts: vec![artifact], artifacts: vec![artifact],
schema: schema.into(), schema: match spec.ecosystem {
Ecosystem::Pip => "pypi-wheel-v1",
Ecosystem::Npm => "npm-instances-v1",
Ecosystem::Go => "go-sumdb-v1",
}
.into(),
build: None, build: None,
}) })
} }
@@ -1126,7 +1148,22 @@ impl RegistryResolver<'_> {
MAX_ARTIFACT_BYTES, MAX_ARTIFACT_BYTES,
)?; )?;
sumdb.verify_zip(name, version, &zip)?; sumdb.verify_zip(name, version, &zip)?;
self.lock_archive(&spec, dependencies, runtime, "zip", &zip, "go-sumdb-v1") self.lock_archive(
&spec,
dependencies,
runtime,
"zip",
&zip,
Some(RegistrySource {
ecosystem: Ecosystem::Go,
locator: format!(
"/{}/@v/v{}.zip",
go_path(name),
version.trim_start_matches('v')
),
filename: format!("v{}.zip", version.trim_start_matches('v')),
}),
)
}) })
.collect::<Result<Vec<_>, String>>()?; .collect::<Result<Vec<_>, String>>()?;
return Ok((libs, sumdb.finish())); return Ok((libs, sumdb.finish()));
@@ -1179,6 +1216,7 @@ struct PythonCandidate {
bytes: Vec<u8>, bytes: Vec<u8>,
info: Value, info: Value,
build: Option<String>, build: Option<String>,
source: Option<RegistrySource>,
} }
struct SearchState<'a> { struct SearchState<'a> {
@@ -128,6 +128,18 @@ impl Catalog<'_, '_> {
"npm archive cache exceeds its 256 MiB/package budget".into(), "npm archive cache exceeds its 256 MiB/package budget".into(),
)); ));
} }
let locator = document
.get("dist")
.and_then(|dist| dist.get("tarball"))
.and_then(serde_json::Value::as_str)
.ok_or_else(|| SearchError::Fatal("npm tarball locator missing".into()))?;
let filename = locator
.split(['?', '#'])
.next()
.unwrap_or(locator)
.rsplit('/')
.next()
.ok_or_else(|| SearchError::Fatal("npm tarball filename missing".into()))?;
let lib = self let lib = self
.resolver .resolver
.lock_archive( .lock_archive(
@@ -136,7 +148,11 @@ impl Catalog<'_, '_> {
self.runtime, self.runtime,
"tgz", "tgz",
&bytes, &bytes,
"npm-instances-v1", Some(crate::libs::RegistrySource {
ecosystem: Ecosystem::Npm,
locator: locator.into(),
filename: filename.into(),
}),
) )
.map_err(SearchError::Fatal)?; .map_err(SearchError::Fatal)?;
self.archives.insert(spec.clone(), (lib, archive)); self.archives.insert(spec.clone(), (lib, archive));
+278
View File
@@ -0,0 +1,278 @@
//! Exact restoration is a preparation operation, never a dependency resolver.
use std::collections::BTreeSet;
use std::fs;
use std::path::Path;
use super::{
ArtifactStore, FixtureRegistry, LockFile, LockedArtifact, RegistryTransport, build, sumdb,
};
pub(super) fn install(
project_root: &Path,
target: crate::toolchain::BuildTarget,
) -> Result<super::PreparationReport, String> {
super::reject_declaration_symlinks(project_root)?;
let lock_bytes = fs::read(project_root.join("dever.lock"))
.map_err(|error| format!("dever lib install requires an existing dever.lock: {error}"))?;
let setting_bytes = setting_snapshot(project_root)?;
let lock = LockFile::decode(&lock_bytes)?;
super::doctor(&lock)?;
validate_target(&lock, target.platform())?;
let fixtures = FixtureRegistry::builtin();
if lock.packages.is_empty()
&& lock
.libs
.iter()
.all(|lib| fixtures.package(&lib.spec).is_some())
&& lock.workers.iter().all(|worker| worker.ecosystem == "exec")
{
let store = super::FixtureArtifactStore::default();
validate_project(project_root, &lock, &store, target, false)?;
restore_fixtures(&lock, &store)?;
} else {
let inputs = super::PreparingRegistry(super::InstalledRegistry::current(target)?);
let store = super::ManagedArtifactStore {
layout: inputs.0.layout.clone(),
};
crate::packages::restore_locked(project_root, &lock, &store)?;
validate_project(project_root, &lock, &store, target, true)?;
for worker in lock
.workers
.iter()
.filter(|worker| worker.ecosystem != "exec")
{
let (runtime, _) = build::Inputs::runtime(&inputs, worker.ecosystem.parse()?)?;
if worker.runtime.as_ref() != Some(&runtime.pack) {
return Err("locked Worker runtime differs from signed release".into());
}
}
if target != crate::toolchain::BuildTarget::host()? && !lock.workers.is_empty() {
inputs.ensure(crate::toolchain::ExtensionKind::Target)?;
}
restore_artifacts(
&lock,
&super::HttpRegistry::official(),
&store,
&inputs,
target.platform(),
&sumdb::Verifier::official(),
)?;
super::embedded_resources_for_target(project_root, target)?;
}
if fs::read(project_root.join("dever.lock")).map_err(|error| error.to_string())? != lock_bytes
|| setting_snapshot(project_root)? != setting_bytes
{
return Err("project setting or lock changed during exact restoration".into());
}
let artifacts = lock
.libs
.iter()
.flat_map(|lib| &lib.artifacts)
.chain(build::npm_outputs(&lock).map(|receipt| &receipt.output))
.collect::<BTreeSet<_>>();
Ok(super::PreparationReport {
libs: lock.libs.len(),
artifacts: artifacts.len(),
runtimes: lock
.libs
.iter()
.map(|lib| &lib.runtime)
.chain(
lock.workers
.iter()
.filter_map(|worker| worker.runtime.as_ref()),
)
.collect::<BTreeSet<_>>()
.len(),
artifact_bytes: artifacts.iter().map(|artifact| artifact.bytes).sum(),
})
}
fn setting_snapshot(project_root: &Path) -> Result<Option<Vec<u8>>, String> {
match fs::read(project_root.join("config/setting.json")) {
Ok(bytes) => Ok(Some(bytes)),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(None),
Err(error) => Err(format!("cannot read project setting snapshot: {error}")),
}
}
fn validate_project(
project_root: &Path,
lock: &LockFile,
store: &dyn ArtifactStore,
target: crate::toolchain::BuildTarget,
prepare: bool,
) -> Result<(), String> {
let sources = crate::packages::sources_with(project_root, store)?;
let workers = super::unbound_source_workers(project_root, &sources)?;
let mut unbound = lock.clone();
for worker in &mut unbound.workers {
worker.runtime = None;
}
super::validate_worker_bindings(&unbound, &workers)?;
let mut roots = super::read_declarations(project_root)?;
roots.extend(crate::packages::declared_libs_with(project_root, store)?);
if super::retain_locked_libs(lock, &roots, lock.workers.clone())? != *lock {
return Err("dever.lock contains Lib entries or environments outside the declared closure; run 'dever lib update <project-root>'".into());
}
let workers = super::bind_workers(workers, target, prepare)?;
super::validate_worker_bindings(lock, &workers)?;
Ok(())
}
fn validate_target(lock: &LockFile, target: &str) -> Result<(), String> {
for artifact in lock.libs.iter().flat_map(|lib| &lib.artifacts).chain(
lock.builds
.iter()
.flat_map(|receipt| receipt.sources.iter().chain([&receipt.output])),
) {
if artifact.target != target && artifact.target != "host" {
return Err(format!(
"locked artifact targets {}, not {target}; install never switches lock targets",
artifact.target
));
}
}
for receipt in &lock.builds {
validate_target(&receipt.inputs, target)?;
}
Ok(())
}
fn restore_fixtures(lock: &LockFile, store: &dyn ArtifactStore) -> Result<(), String> {
let fixtures = FixtureRegistry::builtin();
for lib in &lock.libs {
if let Some(package) = fixtures.package(&lib.spec) {
if package.locked() != *lib {
return Err("locked fixture metadata differs from builtin package".into());
}
for artifact in &package.artifacts {
let expected = super::sha256(&artifact.bytes);
if store.publish(&artifact.bytes, &artifact.target)? != expected {
return Err("fixture artifact cache identity mismatch".into());
}
}
}
}
Ok(())
}
/// Shared by production preparation and isolated registry fixtures. Every
/// network request names the recorded archive; no metadata/version query runs.
pub fn restore_artifacts(
lock: &LockFile,
transport: &dyn RegistryTransport,
store: &dyn ArtifactStore,
inputs: &dyn build::Inputs,
target: &str,
verifier: &sumdb::Verifier,
) -> Result<(), String> {
super::doctor_with_go_verifier(lock, verifier)?;
validate_target(lock, target)?;
let session = build::Session::new(inputs);
restore_at(lock, transport, store, inputs, &session)?;
super::verify_locked_artifacts_with_go_verifier(lock, store, target, verifier)?;
Ok(())
}
fn restore_at(
lock: &LockFile,
transport: &dyn RegistryTransport,
store: &dyn ArtifactStore,
inputs: &dyn build::Inputs,
session: &build::Session<'_>,
) -> Result<(), String> {
restore_fixtures(lock, store)?;
let fixtures = FixtureRegistry::builtin();
for lib in &lock.libs {
if fixtures.package(&lib.spec).is_some() {
continue;
}
let (runtime, _) = inputs.runtime(lib.spec.ecosystem.clone())?;
if runtime.pack != lib.runtime
|| lib
.artifacts
.iter()
.any(|artifact| artifact.target != runtime.target)
{
return Err(format!(
"{} locked runtime differs from signed release",
lib.spec.key()
));
}
if let Some(markers) = &runtime.python_markers {
build::validate_python_policy(lock, lib, markers)?;
}
for artifact in &lib.artifacts {
if artifact
.source
.as_ref()
.is_some_and(|source| source.ecosystem != lib.spec.ecosystem)
{
return Err("locked registry source ecosystem differs from its Lib".into());
}
if artifact.source.is_some() {
restore_archive(artifact, transport, store)?;
} else if lib.build.is_none() {
return Err(format!("{} lacks an exact registry source", lib.spec.key()));
}
}
}
for receipt in &lock.builds {
if cached(&receipt.output, store)? {
continue;
}
restore_at(&receipt.inputs, transport, store, inputs, session)?;
for source in &receipt.sources {
restore_archive(source, transport, store)?;
}
session.replay(receipt, store)?;
}
Ok(())
}
fn cached(artifact: &LockedArtifact, store: &dyn ArtifactStore) -> Result<bool, String> {
match store.get_exact(&artifact.sha256, artifact.bytes, &artifact.target)? {
Some(bytes) => {
validate_bytes(artifact, &bytes)?;
Ok(true)
}
None => Ok(false),
}
}
fn validate_bytes(artifact: &LockedArtifact, bytes: &[u8]) -> Result<(), String> {
if bytes.len() as u64 != artifact.bytes || super::sha256(bytes) != artifact.sha256 {
return Err(format!(
"locked archive '{}' failed exact byte/hash verification",
artifact.path
));
}
Ok(())
}
fn restore_archive(
artifact: &LockedArtifact,
transport: &dyn RegistryTransport,
store: &dyn ArtifactStore,
) -> Result<(), String> {
if cached(artifact, store)? {
return Ok(());
}
let source = artifact
.source
.as_ref()
.ok_or("locked archive source is absent")?;
let limit =
usize::try_from(artifact.bytes).map_err(|_| "locked archive exceeds address space")?;
if limit > 64 * 1024 * 1024 {
return Err("locked registry archive exceeds byte budget".into());
}
let bytes = transport.get(source.ecosystem.clone(), &source.locator, limit)?;
validate_bytes(artifact, &bytes)?;
if store.publish(&bytes, &artifact.target)? != artifact.sha256 {
return Err("restored archive cache identity mismatch".into());
}
Ok(())
}
+2 -2
View File
@@ -18,7 +18,7 @@ mod worker;
#[path = "../../../test/dever-cli-tests/build_arguments.rs"] #[path = "../../../test/dever-cli-tests/build_arguments.rs"]
mod build_arguments; mod build_arguments;
const USAGE: &str = "Usage:\n dever check <project-root>\n dever api <project-root> [--output <new-file>]\n dever fmt <project-root> [--check]\n dever test <project-root>\n dever run <project-root> [-- <component>.<domain>.<cmd> '<json-object>']\n dever run <project-root> --tenant <positive-tenant-id> -- <component>.<domain>.<cmd> '<json-object>'\n dever build <project-root> --output <new-file> [--target linux-x86_64|linux-aarch64]\n dever clean <project-root>\n dever tenant migrate <project-root> <positive-tenant-id>\n dever tenant owner <project-root> <positive-tenant-id> <site> <positive-user-id>\n dever tenant component enable <project-root> <positive-tenant-id> <component>\n dever tenant component disable <project-root> <positive-tenant-id> <component>\n dever lib add|list|update|remove|doctor <project-root> [--target linux-x86_64|linux-aarch64] [spec ...]\n dever package add|list|update|remove|doctor <project-root> [spec ...]\n\nApplications use API, Job, and CMD declarations under module/ and config/setting.json. Native builds and tests are offline."; const USAGE: &str = "Usage:\n dever check <project-root>\n dever api <project-root> [--output <new-file>]\n dever fmt <project-root> [--check]\n dever test <project-root>\n dever run <project-root> [-- <component>.<domain>.<cmd> '<json-object>']\n dever run <project-root> --tenant <positive-tenant-id> -- <component>.<domain>.<cmd> '<json-object>'\n dever build <project-root> --output <new-file> [--target linux-x86_64|linux-aarch64]\n dever clean <project-root>\n dever tenant migrate <project-root> <positive-tenant-id>\n dever tenant owner <project-root> <positive-tenant-id> <site> <positive-user-id>\n dever tenant component enable <project-root> <positive-tenant-id> <component>\n dever tenant component disable <project-root> <positive-tenant-id> <component>\n dever lib add|install|list|update|remove|doctor <project-root> [--target linux-x86_64|linux-aarch64] [spec ...]\n dever package add|list|update|remove|doctor <project-root> [spec ...]\n\nApplications use API, Job, and CMD declarations under module/ and config/setting.json. Native builds and tests are offline.";
enum Action { enum Action {
Check, Check,
@@ -435,7 +435,7 @@ fn arguments_for(arguments: &[OsString]) -> Result<(PathBuf, Action), String> {
if command == "lib" if command == "lib"
&& matches!( && matches!(
operation.to_str(), operation.to_str(),
Some("add" | "list" | "update" | "remove" | "doctor") Some("add" | "install" | "list" | "update" | "remove" | "doctor")
) => ) =>
{ {
let command = operation.to_str().unwrap().to_owned(); let command = operation.to_str().unwrap().to_owned();
+65 -2
View File
@@ -12,7 +12,7 @@ use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256}; use sha2::{Digest, Sha256};
use url::Url; use url::Url;
use crate::libs::{self, ArtifactStore, LibSpec}; use crate::libs::{self, ArtifactStore, LibSpec, LockFile};
const INDEX_FORMAT: &str = "dever-package-index-v1"; const INDEX_FORMAT: &str = "dever-package-index-v1";
const MANIFEST_FORMAT: &str = "dever-package-v1"; const MANIFEST_FORMAT: &str = "dever-package-v1";
@@ -406,6 +406,64 @@ fn settings(project_root: &Path) -> Result<(String, Vec<PackageRequest>), String
} }
} }
pub(crate) fn restore_locked(
project_root: &Path,
lock: &LockFile,
store: &dyn ArtifactStore,
) -> Result<(), String> {
let (origin, roots) = settings(project_root)?;
validate_locked_roots(&roots, &lock.packages)?;
if lock.packages.is_empty() {
return Ok(());
}
restore_locked_with(
project_root,
lock,
&HttpPackageTransport::new(&origin)?,
store,
)
}
/// Exact Package archives are restored before source/Worker checking. No index
/// is consulted: both dependency versions and archive identities are locked.
pub fn restore_locked_with(
project_root: &Path,
lock: &LockFile,
transport: &dyn PackageTransport,
store: &dyn ArtifactStore,
) -> Result<(), String> {
validate_locked_roots(&settings(project_root)?.1, &lock.packages)?;
for package in &lock.packages {
let cached = store.get_exact(&package.sha256, package.bytes, "package")?;
let missing = cached.is_none();
let bytes = match cached {
Some(bytes) => bytes,
None => transport.get(
&format!("/v1/packages/{}/{}.zip", package.name, package.version),
package.bytes as usize,
)?,
};
if bytes.len() as u64 != package.bytes || digest(&bytes) != package.sha256 {
return Err(format!(
"Package {}@{} differs from dever.lock",
package.name, package.version
));
}
let archive = manifest(&bytes, &package.name, &package.version)?;
if archive.manifest_sha256 != package.manifest_sha256 {
return Err(format!(
"Package {} manifest differs from dever.lock",
package.name
));
}
if missing && store.publish(&bytes, "package")? != package.sha256 {
return Err("restored Package cache identity mismatch".into());
}
}
owned_files_with(project_root, store)?;
Ok(())
}
#[derive(Clone)] #[derive(Clone)]
struct SelectedVersion { struct SelectedVersion {
index: IndexVersion, index: IndexVersion,
@@ -1169,7 +1227,12 @@ fn execute_locked(
resolve_with(&roots, transport, store)? resolve_with(&roots, transport, store)?
}; };
check_project_collisions(project_root, &closure.packages, store)?; check_project_collisions(project_root, &closure.packages, store)?;
let workers = libs::source_workers_with_packages( let worker_loader = if command == "remove" {
libs::source_workers_with_packages
} else {
libs::prepare_source_workers_with_packages
};
let workers = worker_loader(
project_root, project_root,
&closure.sources, &closure.sources,
crate::toolchain::BuildTarget::host()?, crate::toolchain::BuildTarget::host()?,
+21 -2
View File
@@ -235,14 +235,33 @@ impl CacheStore {
} }
pub fn artifact(&self, expected: &ArtifactReceipt) -> Result<StoredArtifact, String> { pub fn artifact(&self, expected: &ArtifactReceipt) -> Result<StoredArtifact, String> {
self.artifact_optional(expected)?
.ok_or_else(|| "shared artifact unavailable: artifact is not cached".into())
}
pub fn artifact_optional(
&self,
expected: &ArtifactReceipt,
) -> Result<Option<StoredArtifact>, String> {
expected.validate()?; expected.validate()?;
let operation = CacheOperation::acquire(&self.root)?; let operation = CacheOperation::acquire(&self.root)?;
let directory = self.root.join("artifacts");
for path in [&directory, &directory.join(&expected.sha256)] {
match fs::symlink_metadata(path) {
Ok(_) => {}
Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None),
Err(error) => return Err(format!("cannot inspect shared artifact: {error}")),
}
if path == &directory {
ensure_real_directory(&directory)?;
}
}
let file = self.read_artifact_file(expected)?; let file = self.read_artifact_file(expected)?;
Ok(StoredArtifact { Ok(Some(StoredArtifact {
receipt: expected.clone(), receipt: expected.clone(),
file, file,
_operation: operation, _operation: operation,
}) }))
} }
fn read_artifact_file(&self, expected: &ArtifactReceipt) -> Result<File, String> { fn read_artifact_file(&self, expected: &ArtifactReceipt) -> Result<File, String> {
@@ -0,0 +1,506 @@
//! One signed catalog owns both the base release and immutable optional resources.
use std::collections::BTreeSet;
use std::fs;
use std::io;
use std::path::{Path, PathBuf};
use serde::{Deserialize, Serialize};
use super::BuildTarget;
use super::release::{
self, Artifact, InstallLock, Layout, MachineManager, ReleaseManifest, Version,
};
use crate::libs::Ecosystem;
/// Explicit author/test transport. Product IPC always uses the compiled official source.
pub trait ExtensionSource {
fn open(&self, version: &Version, extension: &ExtensionId)
-> Result<Box<dyn io::Read>, String>;
}
pub(super) const PAYLOAD_LIMIT: u64 = 2 * 1024 * 1024 * 1024;
pub(super) const ARTIFACT_LIMIT: usize = 4096;
#[derive(Clone, Debug, Eq, Ord, PartialEq, PartialOrd, Serialize, Deserialize)]
#[serde(
tag = "type",
content = "ecosystem",
rename_all = "snake_case",
deny_unknown_fields
)]
pub enum ExtensionKind {
Runtime(Ecosystem),
Build(Ecosystem),
Target,
}
#[derive(Clone, Debug, Eq, Ord, PartialEq, PartialOrd, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ExtensionId {
pub kind: ExtensionKind,
pub target: BuildTarget,
}
impl ExtensionId {
pub fn name(&self) -> String {
let kind = match &self.kind {
ExtensionKind::Runtime(ecosystem) => format!("runtime-{}", ecosystem.as_str()),
ExtensionKind::Build(ecosystem) => format!("build-{}", ecosystem.as_str()),
ExtensionKind::Target => "target".into(),
};
format!("{kind}-{}", self.target.platform())
}
pub fn asset_suffix(&self) -> String {
format!("ext-{}.tar.zst", self.name())
}
fn permits(&self, path: &str) -> bool {
let target = self.target.platform();
match &self.kind {
ExtensionKind::Runtime(ecosystem) => {
path.starts_with(&format!("runtime/{}/{target}/", ecosystem.as_str()))
}
ExtensionKind::Build(ecosystem) => {
path.starts_with(&format!("build/{}/{target}/", ecosystem.as_str()))
}
ExtensionKind::Target => {
path.starts_with(&format!("runtime/{target}/"))
|| path.starts_with(&format!("sandbox/{target}/"))
}
}
}
fn missing(&self) -> String {
match self.kind {
ExtensionKind::Target => format!(
"target resources are not installed; run 'dever target add {}'",
self.target.platform()
),
_ => format!(
"{} resources are not installed; run 'dever lib install <project-root> --target {}' (or 'dever lib add' for new dependencies)",
self.name(),
self.target.platform()
),
}
}
}
#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Extension {
pub kind: ExtensionKind,
pub target: BuildTarget,
pub artifacts: Vec<Artifact>,
}
impl Extension {
pub fn id(&self) -> ExtensionId {
ExtensionId {
kind: self.kind.clone(),
target: self.target,
}
}
}
pub fn validate_catalog(manifest: &ReleaseManifest) -> Result<(), String> {
Version::parse(manifest.version.as_str())?;
if manifest.format != "dever-release-v2" || manifest.platform != release::platform_identity() {
return Err("unsupported release format or host platform".into());
}
if manifest.extensions.len() > 12 {
return Err("release extension count exceeds its limit".into());
}
let mut paths = BTreeSet::new();
validate_artifacts(&manifest.artifacts, &mut paths)?;
let mut ids = BTreeSet::new();
for extension in &manifest.extensions {
let id = extension.id();
if !ids.insert(id.clone())
|| matches!(id.kind, ExtensionKind::Build(Ecosystem::Go))
|| matches!(id.kind, ExtensionKind::Target) && id.target == BuildTarget::host()?
{
return Err("release has a duplicate or unsupported extension identity".into());
}
validate_artifacts(&extension.artifacts, &mut paths)?;
if extension
.artifacts
.iter()
.any(|artifact| !id.permits(&artifact.path))
{
return Err(format!(
"extension '{}' contains a file outside its resource namespace",
id.name()
));
}
}
Ok(())
}
pub(super) fn validate_artifacts<'a>(
artifacts: &'a [Artifact],
paths: &mut BTreeSet<&'a str>,
) -> Result<(), String> {
if artifacts.is_empty() || artifacts.len() > ARTIFACT_LIMIT {
return Err("release artifact count is outside 1..=4096".into());
}
let mut bytes = 0_u64;
for artifact in artifacts {
if artifact.path.is_empty()
|| artifact.path.len() > 1024
|| artifact
.path
.split('/')
.any(|part| part.is_empty() || part == "." || part == "..")
|| artifact.path.contains(['\\', ':', '\0'])
|| matches!(artifact.path.as_str(), "manifest.json" | "manifest.sig")
|| !paths.insert(&artifact.path)
{
return Err("release has an unsafe or duplicate artifact path".into());
}
if artifact.sha256.len() != 64
|| !artifact
.sha256
.bytes()
.all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte))
{
return Err("release artifact has an invalid SHA-256".into());
}
bytes = bytes
.checked_add(artifact.bytes)
.ok_or("release size overflow")?;
if bytes > PAYLOAD_LIMIT {
return Err("release exceeds its two-GiB unpacked size limit".into());
}
}
// A signed file cannot also be the parent directory of another file.
for path in paths.iter() {
for (index, _) in path.match_indices('/') {
if paths.contains(&path[..index]) {
return Err("release artifact paths overlap as file and directory".into());
}
}
}
Ok(())
}
#[derive(Clone)]
pub struct SignedResources {
layout: Layout,
base: PathBuf,
manifest: ReleaseManifest,
catalog: String,
}
impl SignedResources {
pub fn load(layout: &Layout, version: &Version) -> Result<Self, String> {
Version::parse(version.as_str())?;
layout.validate_machine_permissions()?;
let base = layout.versions().join(version.as_str());
let manifest = MachineManager::new(layout.clone()).verify_release(&base, Some(version))?;
release::validate_installed_release(layout, &base, &manifest)?;
let catalog = release::sha256_file(&base.join("manifest.json"))?;
Ok(Self {
layout: layout.clone(),
base,
manifest,
catalog,
})
}
pub fn manifest(&self) -> &ReleaseManifest {
&self.manifest
}
pub fn root(&self, logical_path: &str) -> Result<PathBuf, String> {
if let Some(artifact) = self
.manifest
.artifacts
.iter()
.find(|artifact| artifact.path == logical_path)
{
verify_file(&self.base, artifact)?;
return Ok(self.base.clone());
}
let extension = self
.manifest
.extensions
.iter()
.find(|extension| {
extension
.artifacts
.iter()
.any(|artifact| artifact.path == logical_path)
})
.ok_or_else(|| format!("signed release does not declare resource '{logical_path}'"))?;
self.extension_root(extension)
}
pub fn path(&self, logical_path: &str) -> Result<PathBuf, String> {
Ok(self.root(logical_path)?.join(logical_path))
}
pub fn read(&self, logical_path: &str) -> Result<Vec<u8>, String> {
let path = self.path(logical_path)?;
let artifact = self
.manifest
.artifacts
.iter()
.chain(
self.manifest
.extensions
.iter()
.flat_map(|extension| &extension.artifacts),
)
.find(|artifact| artifact.path == logical_path)
.expect("root required signed artifact");
let bytes =
fs::read(path).map_err(|error| format!("cannot read signed resource: {error}"))?;
use sha2::{Digest, Sha256};
if bytes.len() as u64 != artifact.bytes
|| release::hex(&Sha256::digest(&bytes)) != artifact.sha256
{
return Err("signed resource changed while reading".into());
}
Ok(bytes)
}
pub fn artifacts(&self, prefix: &str) -> Result<Vec<Artifact>, String> {
let mut artifacts = Vec::new();
for artifact in self
.manifest
.artifacts
.iter()
.filter(|artifact| artifact.path.starts_with(prefix))
{
verify_file(&self.base, artifact)?;
artifacts.push(artifact.clone());
}
for extension in &self.manifest.extensions {
if extension
.artifacts
.iter()
.any(|artifact| artifact.path.starts_with(prefix))
{
self.extension_root(extension)?;
artifacts.extend(
extension
.artifacts
.iter()
.filter(|artifact| artifact.path.starts_with(prefix))
.cloned(),
);
}
}
Ok(artifacts)
}
fn store(&self) -> PathBuf {
self.layout
.extensions()
.join(self.manifest.version.as_str())
.join(&self.catalog)
}
fn destination(&self, id: &ExtensionId) -> PathBuf {
self.store().join(id.name())
}
fn extension(&self, id: &ExtensionId) -> Result<&Extension, String> {
self.manifest
.extensions
.iter()
.find(|extension| extension.id() == *id)
.ok_or_else(|| {
format!(
"Dever {} does not declare extension '{}'",
self.manifest.version,
id.name()
)
})
}
fn extension_root(&self, extension: &Extension) -> Result<PathBuf, String> {
let id = extension.id();
let root = self.destination(&id);
if !self.store_present()? || !exists(&root)? {
return Err(id.missing());
}
verify_extension(&root, &extension.artifacts)?;
Ok(root)
}
fn store_present(&self) -> Result<bool, String> {
let mut parent = self.layout.cache();
for directory in [
self.layout.extensions(),
self.layout
.extensions()
.join(self.manifest.version.as_str()),
self.store(),
] {
if !exists(&directory)? {
return Ok(false);
}
release::validate_shared_directory(&directory, &parent, "extension store")?;
parent = directory;
}
Ok(true)
}
pub(super) fn installed_extensions(&self) -> Result<Vec<ExtensionId>, String> {
if !self.store_present()? {
return Ok(Vec::new());
}
let mut installed = Vec::new();
for entry in fs::read_dir(self.store()).map_err(|error| error.to_string())? {
let entry = entry.map_err(|error| error.to_string())?;
let extension = self
.manifest
.extensions
.iter()
.find(|extension| entry.file_name() == extension.id().name().as_str())
.ok_or("extension store contains an undeclared entry")?;
self.extension_root(extension)?;
installed.push(extension.id());
}
Ok(installed)
}
}
pub(super) fn exists(path: &Path) -> Result<bool, String> {
match fs::symlink_metadata(path) {
Ok(_) => Ok(true),
Err(error) if error.kind() == io::ErrorKind::NotFound => Ok(false),
Err(error) => Err(format!(
"cannot inspect extension path '{}': {error}",
path.display()
)),
}
}
fn verify_file(root: &Path, artifact: &Artifact) -> Result<(), String> {
let mut parent = root.to_path_buf();
let relative = Path::new(&artifact.path);
for component in relative.parent().into_iter().flat_map(Path::components) {
let directory = parent.join(component);
release::validate_shared_directory(&directory, &parent, "signed resource directory")?;
parent = directory;
}
let path = root.join(relative);
release::validate_shared_file(&path, &parent, "signed resource", false)?;
if fs::metadata(&path)
.map_err(|error| error.to_string())?
.len()
!= artifact.bytes
|| release::sha256_file(&path)? != artifact.sha256
{
return Err(format!(
"signed resource '{}' failed integrity verification",
artifact.path
));
}
Ok(())
}
pub(super) fn verify_extension(root: &Path, artifacts: &[Artifact]) -> Result<(), String> {
release::validate_shared_directory(
root,
root.parent().ok_or("extension root has no parent")?,
"installed extension",
)?;
for artifact in artifacts {
verify_file(root, artifact)?;
}
fn entries(root: &Path, directory: &Path, expected: &BTreeSet<&str>) -> Result<(), String> {
for entry in fs::read_dir(directory).map_err(|error| error.to_string())? {
let entry = entry.map_err(|error| error.to_string())?;
let path = entry.path();
let kind = entry.file_type().map_err(|error| error.to_string())?;
let relative = path
.strip_prefix(root)
.map_err(|error| error.to_string())?
.to_str()
.ok_or("extension path is not UTF-8")?;
if kind.is_dir() {
release::validate_shared_directory(&path, directory, "extension directory")?;
if !expected
.iter()
.any(|artifact| artifact.starts_with(&format!("{relative}/")))
{
return Err("installed extension contains an unsigned directory".into());
}
entries(root, &path, expected)?;
} else if !kind.is_file() || !expected.contains(relative) {
return Err("installed extension contains an unsigned or non-regular file".into());
}
}
Ok(())
}
entries(
root,
root,
&artifacts
.iter()
.map(|artifact| artifact.path.as_str())
.collect(),
)
}
pub(super) fn ensure(layout: &Layout, version: &Version, id: &ExtensionId) -> Result<(), String> {
prepare_extension_with(
layout,
version,
id,
&super::release_source::ReleaseSource::official(),
)
}
pub fn prepare_extension_with(
layout: &Layout,
version: &Version,
id: &ExtensionId,
source: &dyn ExtensionSource,
) -> Result<(), String> {
Version::parse(version.as_str())?;
let _lock = InstallLock::acquire_wait(layout)?;
ensure_locked(layout, version, id, source)
}
pub(super) fn ensure_locked(
layout: &Layout,
version: &Version,
id: &ExtensionId,
source: &dyn ExtensionSource,
) -> Result<(), String> {
#[cfg(target_os = "linux")]
super::bootstrap::require_no_pending(layout)?;
let resources = SignedResources::load(layout, version)?;
let extension = resources.extension(id)?;
let destination = resources.destination(id);
if exists(&destination)? {
resources.extension_root(extension)?;
return Ok(());
}
let mut parent = layout.cache();
for directory in [
layout.extensions(),
layout.extensions().join(version.as_str()),
resources.store(),
] {
if !exists(&directory)? {
fs::create_dir(&directory)
.map_err(|error| format!("cannot create extension store: {error}"))?;
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::set_permissions(&directory, fs::Permissions::from_mode(0o755))
.map_err(|error| error.to_string())?;
}
release::sync_directory(&parent).map_err(|error| error.to_string())?;
}
release::validate_shared_directory(&directory, &parent, "extension store")?;
parent = directory;
}
super::release_source::prepare_extension(layout, version, extension, &destination, source)
}
+15 -1
View File
@@ -5,7 +5,7 @@ use std::process::Command;
use super::release::{Layout, MachineManager, Version, platform_contract}; use super::release::{Layout, MachineManager, Version, platform_contract};
const USAGE: &str = "Usage:\n dever install <version|latest>\n dever update\n dever use <version>\n dever uninstall <version>\n dever version [project-root]\n dever cache status\n dever cache clean\n dever lib add|list|update|remove|doctor <project-root> [--target linux-x86_64|linux-aarch64] [spec ...]\n dever package add|list|update|remove|doctor <project-root> [spec ...]\n dever check <project-root>\n dever api <project-root> [--output <new-file>]\n dever fmt <project-root> [--check]\n dever test <project-root>\n dever run <project-root> [-- <component>.<domain>.<cmd> '<json-object>']\n dever build <project-root> --output <new-file> [--target linux-x86_64|linux-aarch64]\n dever clean <project-root>\n dever tenant ..."; const USAGE: &str = "Usage:\n dever install <version|latest>\n dever update\n dever target add linux-aarch64\n dever use <version>\n dever uninstall <version>\n dever version [project-root]\n dever cache status\n dever cache clean\n dever lib add|list|update|install|remove|doctor <project-root> [--target linux-x86_64|linux-aarch64] [spec ...]\n dever package add|list|update|remove|doctor <project-root> [spec ...]\n dever check <project-root>\n dever api <project-root> [--output <new-file>]\n dever fmt <project-root> [--check]\n dever test <project-root>\n dever run <project-root> [-- <component>.<domain>.<cmd> '<json-object>']\n dever build <project-root> --output <new-file> [--target linux-x86_64|linux-aarch64]\n dever clean <project-root>\n dever tenant ...";
#[derive(Debug)] #[derive(Debug)]
pub enum CommandResult { pub enum CommandResult {
@@ -30,6 +30,20 @@ pub fn execute_in(layout: &Layout, arguments: &[OsString]) -> Result<CommandResu
super::bootstrap::require_no_pending(layout)?; super::bootstrap::require_no_pending(layout)?;
let manager = MachineManager::new(layout.clone()); let manager = MachineManager::new(layout.clone());
match arguments { match arguments {
[command, operation, target] if command == "target" && operation == "add" => {
let target = utf8(target, "target")?.parse::<super::BuildTarget>()?;
if target != super::BuildTarget::LinuxAarch64 {
return Err("target add supports linux-aarch64".into());
}
let version = manager
.active_version()?
.ok_or("no active Dever version; run 'dever install latest'")?;
super::ensure_extension(layout, &version, super::ExtensionKind::Target, target)?;
Ok(CommandResult::Message(format!(
"prepared {} for Dever {version}",
target.platform()
)))
}
[command, requested] if command == "install" => { [command, requested] if command == "install" => {
let requested = utf8(requested, "version")?; let requested = utf8(requested, "version")?;
super::release_source::prepare(layout, requested)?; super::release_source::prepare(layout, requested)?;
+11 -1
View File
@@ -3,15 +3,22 @@ pub mod bootstrap;
mod cache; mod cache;
pub mod compilation; pub mod compilation;
mod core_libraries; mod core_libraries;
mod extensions;
mod launcher; mod launcher;
pub mod packaging; pub mod packaging;
mod release; mod release;
mod release_archive;
mod release_source; mod release_source;
pub mod runtime_pack; pub mod runtime_pack;
mod service; mod service;
mod skill; mod skill;
mod target; mod target;
pub use extensions::{
Extension, ExtensionId, ExtensionKind, ExtensionSource, SignedResources,
prepare_extension_with, validate_catalog,
};
pub use release_source::extract_config;
pub use target::BuildTarget; pub use target::BuildTarget;
pub use cache::{ pub use cache::{
@@ -22,4 +29,7 @@ pub use release::{
Artifact, DownloadedReleases, Layout, MachineManager, PlatformContract, ReleaseManifest, Artifact, DownloadedReleases, Layout, MachineManager, PlatformContract, ReleaseManifest,
Version, platform_contract, platform_identity, sha256_file, Version, platform_contract, platform_identity, sha256_file,
}; };
pub use service::{artifact_get, artifact_put, cache_clean, cache_status, compile, serve}; pub use service::{
artifact_get, artifact_get_optional, artifact_put, cache_clean, cache_status, compile,
ensure_extension, serve,
};
+49 -8
View File
@@ -12,7 +12,10 @@ use serde::Deserialize;
use sha2::{Digest, Sha256}; use sha2::{Digest, Sha256};
use super::runtime_pack::{self, Manifest, Profile, Profiles, RuntimeArtifact}; use super::runtime_pack::{self, Manifest, Profile, Profiles, RuntimeArtifact};
use super::{Artifact, BuildTarget, ReleaseManifest, Version, sha256_file}; use super::{
Artifact, BuildTarget, Extension, ExtensionKind, ReleaseManifest, Version, sha256_file,
validate_catalog,
};
static NEXT_STAGING: AtomicU64 = AtomicU64::new(0); static NEXT_STAGING: AtomicU64 = AtomicU64::new(0);
@@ -205,33 +208,71 @@ fn assemble(
end: settings.end, end: settings.end,
}, },
)?); )?);
let mut extensions = Vec::new();
let mut target_artifacts = BTreeMap::new();
for (target, inputs) in settings.native_targets { for (target, inputs) in settings.native_targets {
artifacts.extend(assemble_native(&mut writer, &version, target, inputs)?); target_artifacts.insert(
target,
assemble_native(&mut writer, &version, target, inputs)?,
);
} }
for (ecosystem, input) in settings.runtimes.entries() { for (ecosystem, input) in settings.runtimes.entries() {
artifacts.extend(ecosystems::assemble(&mut writer, ecosystem, input, host)?); extensions.push(Extension {
kind: ExtensionKind::Runtime(ecosystem.clone()),
target: host,
artifacts: ecosystems::assemble(&mut writer, ecosystem, input, host)?,
});
} }
for (target, inputs) in settings.ecosystem_targets { for (target, inputs) in settings.ecosystem_targets {
for (ecosystem, input) in inputs.entries() { for (ecosystem, input) in inputs.entries() {
artifacts.extend(ecosystems::assemble(&mut writer, ecosystem, input, target)?); extensions.push(Extension {
kind: ExtensionKind::Runtime(ecosystem.clone()),
target,
artifacts: ecosystems::assemble(&mut writer, ecosystem, input, target)?,
});
} }
} }
if let Some(inputs) = settings.sandbox { if let Some(inputs) = settings.sandbox {
artifacts.extend(sandbox::assemble(&mut writer, inputs, host)?); artifacts.extend(sandbox::assemble(&mut writer, inputs, host)?);
} }
for (target, inputs) in settings.sandbox_targets { for (target, inputs) in settings.sandbox_targets {
artifacts.extend(sandbox::assemble(&mut writer, inputs, target)?); let selected = target_artifacts
.get_mut(&target)
.ok_or("target sandbox requires a matching native target")?;
selected.extend(sandbox::assemble(&mut writer, inputs, target)?);
}
for (target, artifacts) in target_artifacts {
extensions.push(Extension {
kind: ExtensionKind::Target,
target,
artifacts,
});
} }
for (ecosystem, input) in settings.builds.entries() { for (ecosystem, input) in settings.builds.entries() {
artifacts.extend(builds::assemble(&mut writer, ecosystem, input, host)?); extensions.push(Extension {
kind: ExtensionKind::Build(ecosystem.clone()),
target: host,
artifacts: builds::assemble(&mut writer, ecosystem, input, host)?,
});
} }
for (target, inputs) in settings.build_targets { for (target, inputs) in settings.build_targets {
for (ecosystem, input) in inputs.entries() { for (ecosystem, input) in inputs.entries() {
artifacts.extend(builds::assemble(&mut writer, ecosystem, input, target)?); extensions.push(Extension {
kind: ExtensionKind::Build(ecosystem.clone()),
target,
artifacts: builds::assemble(&mut writer, ecosystem, input, target)?,
});
} }
} }
artifacts.sort_by(|left, right| left.path.cmp(&right.path)); artifacts.sort_by(|left, right| left.path.cmp(&right.path));
let manifest = ReleaseManifest::new(version, artifacts); for extension in &mut extensions {
extension
.artifacts
.sort_by(|left, right| left.path.cmp(&right.path));
}
let mut manifest = ReleaseManifest::new(version, artifacts);
manifest.extensions = extensions;
validate_catalog(&manifest)?;
let bytes = write_metadata(&staging.join("manifest.json"), &manifest)?; let bytes = write_metadata(&staging.join("manifest.json"), &manifest)?;
let signature = key.pair.sign(&bytes); let signature = key.pair.sign(&bytes);
let hex: String = signature let hex: String = signature
+77 -17
View File
@@ -74,15 +74,17 @@ pub struct ReleaseManifest {
pub version: Version, pub version: Version,
pub platform: String, pub platform: String,
pub artifacts: Vec<Artifact>, pub artifacts: Vec<Artifact>,
pub extensions: Vec<super::extensions::Extension>,
} }
impl ReleaseManifest { impl ReleaseManifest {
pub fn new(version: Version, artifacts: Vec<Artifact>) -> Self { pub fn new(version: Version, artifacts: Vec<Artifact>) -> Self {
Self { Self {
format: "dever-release-v1".into(), format: "dever-release-v2".into(),
version, version,
platform: platform_identity(), platform: platform_identity(),
artifacts, artifacts,
extensions: Vec::new(),
} }
} }
} }
@@ -129,6 +131,9 @@ impl Layout {
pub fn cache(&self) -> PathBuf { pub fn cache(&self) -> PathBuf {
self.root.join("cache") self.root.join("cache")
} }
pub fn extensions(&self) -> PathBuf {
self.cache().join("extensions")
}
pub fn downloads(&self) -> PathBuf { pub fn downloads(&self) -> PathBuf {
self.cache().join("downloads") self.cache().join("downloads")
} }
@@ -307,6 +312,11 @@ impl MachineManager {
} }
pub fn update(&self) -> Result<Version, String> { pub fn update(&self) -> Result<Version, String> {
self.update_with(&super::release_source::ReleaseSource::official())
}
/// Uses the ordinary install transaction with an explicitly supplied author transport.
pub fn update_with(&self, source: &dyn super::ExtensionSource) -> Result<Version, String> {
let version = self.releases.latest()?; let version = self.releases.latest()?;
let _lock = InstallLock::acquire(&self.layout)?; let _lock = InstallLock::acquire(&self.layout)?;
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
@@ -321,6 +331,12 @@ impl MachineManager {
} }
self.cleanup_staging()?; self.cleanup_staging()?;
self.install_locked(&version)?; self.install_locked(&version)?;
if let Some(active) = self.active_version()? {
let resources = super::extensions::SignedResources::load(&self.layout, &active)?;
for extension in resources.installed_extensions()? {
super::extensions::ensure_locked(&self.layout, &version, &extension, source)?;
}
}
self.activate_locked(&version)?; self.activate_locked(&version)?;
Ok(version) Ok(version)
} }
@@ -468,6 +484,7 @@ impl MachineManager {
Ok(InstalledCompiler { Ok(InstalledCompiler {
core, core,
manifest, manifest,
resources: super::SignedResources::load(&self.layout, version)?,
identity: sha256_file(&root.join(MANIFEST_NAME))?, identity: sha256_file(&root.join(MANIFEST_NAME))?,
_lease: lease, _lease: lease,
}) })
@@ -486,6 +503,14 @@ impl MachineManager {
if destination.exists() { if destination.exists() {
let manifest = self.verify_release(&destination, Some(version))?; let manifest = self.verify_release(&destination, Some(version))?;
validate_installed_release(&self.layout, &destination, &manifest)?; validate_installed_release(&self.layout, &destination, &manifest)?;
if package.exists() {
self.verify_release(package, Some(version))?;
if read_regular_file(&destination.join(MANIFEST_NAME))?
!= read_regular_file(&package.join(MANIFEST_NAME))?
{
return Err("published Dever version changed its signed manifest".into());
}
}
return Ok(()); return Ok(());
} }
let manifest = self.verify_release(package, Some(version))?; let manifest = self.verify_release(package, Some(version))?;
@@ -548,7 +573,7 @@ impl MachineManager {
.map_err(|error| format!("cannot activate Dever {version}: {error}")) .map_err(|error| format!("cannot activate Dever {version}: {error}"))
} }
fn verify_release( pub(super) fn verify_release(
&self, &self,
root: &Path, root: &Path,
expected: Option<&Version>, expected: Option<&Version>,
@@ -601,7 +626,7 @@ impl MachineManager {
manifest_path.display() manifest_path.display()
) )
})?; })?;
if manifest.format != "dever-release-v1" { if manifest.format != "dever-release-v2" {
return Err("unsupported Dever release manifest format".into()); return Err("unsupported Dever release manifest format".into());
} }
if manifest.platform != platform_identity() { if manifest.platform != platform_identity() {
@@ -611,6 +636,7 @@ impl MachineManager {
platform_identity() platform_identity()
)); ));
} }
super::extensions::validate_catalog(&manifest)?;
Ok(manifest) Ok(manifest)
} }
@@ -706,6 +732,7 @@ pub(crate) struct InstalledCompiler {
pub(crate) core: PathBuf, pub(crate) core: PathBuf,
pub(crate) manifest: ReleaseManifest, pub(crate) manifest: ReleaseManifest,
pub(crate) identity: String, pub(crate) identity: String,
pub(crate) resources: super::SignedResources,
_lease: InstallLock, _lease: InstallLock,
} }
@@ -713,14 +740,18 @@ pub(super) struct InstallLock(File);
impl InstallLock { impl InstallLock {
pub(super) fn acquire(layout: &Layout) -> Result<Self, String> { pub(super) fn acquire(layout: &Layout) -> Result<Self, String> {
Self::open(layout, false) Self::open(layout, false, false)
}
pub(super) fn acquire_wait(layout: &Layout) -> Result<Self, String> {
Self::open(layout, false, true)
} }
fn shared(layout: &Layout) -> Result<Self, String> { fn shared(layout: &Layout) -> Result<Self, String> {
Self::open(layout, true) Self::open(layout, true, false)
} }
fn open(layout: &Layout, shared: bool) -> Result<Self, String> { fn open(layout: &Layout, shared: bool, wait: bool) -> Result<Self, String> {
let path = layout.state().join("install.lock"); let path = layout.state().join("install.lock");
let mut options = OpenOptions::new(); let mut options = OpenOptions::new();
options.read(true).write(true).create(true).truncate(false); options.read(true).write(true).create(true).truncate(false);
@@ -750,14 +781,29 @@ impl InstallLock {
} }
#[cfg(not(unix))] #[cfg(not(unix))]
let _ = current; let _ = current;
let locked = if shared { let started = std::time::Instant::now();
fs2::FileExt::try_lock_shared(&file) loop {
} else { let locked = if shared {
fs2::FileExt::try_lock_exclusive(&file) fs2::FileExt::try_lock_shared(&file)
}; } else {
locked.map_err(|error| { fs2::FileExt::try_lock_exclusive(&file)
format!("another Dever installation operation or compilation is active: {error}") };
})?; match locked {
Ok(()) => break,
Err(error)
if wait
&& error.kind() == io::ErrorKind::WouldBlock
&& started.elapsed() < std::time::Duration::from_secs(300) =>
{
std::thread::sleep(std::time::Duration::from_millis(50))
}
Err(error) => {
return Err(format!(
"another Dever installation operation or compilation is active: {error}"
));
}
}
}
Ok(Self(file)) Ok(Self(file))
} }
} }
@@ -912,7 +958,17 @@ fn read_regular_file(path: &Path) -> Result<Vec<u8>, String> {
if !metadata.file_type().is_file() { if !metadata.file_type().is_file() {
return Err(format!("'{}' must be a regular file", path.display())); return Err(format!("'{}' must be a regular file", path.display()));
} }
fs::read(path).map_err(|error| format!("cannot read '{}': {error}", path.display())) const LIMIT: u64 = 2 * 1024 * 1024;
let mut bytes = Vec::new();
File::open(path)
.map_err(|error| format!("cannot read '{}': {error}", path.display()))?
.take(LIMIT + 1)
.read_to_end(&mut bytes)
.map_err(|error| error.to_string())?;
if bytes.len() as u64 > LIMIT {
return Err("release metadata exceeds its size limit".into());
}
Ok(bytes)
} }
pub(super) fn copy_file(source: &Path, destination: &Path, executable: bool) -> Result<(), String> { pub(super) fn copy_file(source: &Path, destination: &Path, executable: bool) -> Result<(), String> {
@@ -964,7 +1020,7 @@ pub(super) fn sync_tree(root: &Path) -> Result<(), String> {
sync_directory(root).map_err(|error| error.to_string()) sync_directory(root).map_err(|error| error.to_string())
} }
fn validate_installed_release( pub(super) fn validate_installed_release(
layout: &Layout, layout: &Layout,
root: &Path, root: &Path,
manifest: &ReleaseManifest, manifest: &ReleaseManifest,
@@ -995,7 +1051,11 @@ fn validate_installed_release(
Ok(()) Ok(())
} }
fn validate_shared_directory(path: &Path, parent: &Path, label: &str) -> Result<(), String> { pub(super) fn validate_shared_directory(
path: &Path,
parent: &Path,
label: &str,
) -> Result<(), String> {
let metadata = fs::symlink_metadata(path) let metadata = fs::symlink_metadata(path)
.map_err(|error| format!("cannot inspect {label} '{}': {error}", path.display()))?; .map_err(|error| format!("cannot inspect {label} '{}': {error}", path.display()))?;
if metadata.file_type().is_symlink() || !metadata.is_dir() { if metadata.file_type().is_symlink() || !metadata.is_dir() {
@@ -0,0 +1,206 @@
//! Bounded single-frame Zstandard and exact signed USTAR extraction.
use std::collections::{BTreeMap, BTreeSet};
use std::fs::{self, OpenOptions};
use std::io::{self, BufRead, BufReader, Read};
use std::path::Path;
use super::extensions::{ARTIFACT_LIMIT, PAYLOAD_LIMIT, validate_artifacts};
use super::release::{Artifact, set_shared_file_permissions, sha256_file};
pub(super) fn extract(
reader: impl Read,
root: &Path,
artifacts: &[Artifact],
) -> Result<(), String> {
validate_artifacts(artifacts, &mut BTreeSet::new())?;
let mut input = Bounded {
input: reader,
remaining: PAYLOAD_LIMIT,
};
let mut header = [0_u8; 5];
input
.read_exact(&mut header)
.map_err(|error| format!("cannot read Zstandard header: {error}"))?;
if header[..4] != [0x28, 0xb5, 0x2f, 0xfd] || header[4] & 4 == 0 {
return Err("release requires a Zstandard frame with a content checksum".into());
}
let mut decoder = zstd::stream::read::Decoder::with_buffer(BufReader::new(
io::Cursor::new(header).chain(input),
))
.map_err(|error| format!("cannot initialize Zstandard decoder: {error}"))?
.single_frame();
decoder
.window_log_max(27)
.map_err(|error| format!("cannot bound Zstandard window: {error}"))?;
unpack(&mut decoder, root, artifacts)?;
// unpack drains the decoder through its checksum, including errors after TAR EOF.
let mut input = decoder.finish();
if !input
.fill_buf()
.map_err(|error| format!("cannot finish release frame: {error}"))?
.is_empty()
{
return Err("release contains data after its single Zstandard frame".into());
}
Ok(())
}
pub(super) fn unpack(reader: impl Read, root: &Path, artifacts: &[Artifact]) -> Result<(), String> {
validate_artifacts(artifacts, &mut BTreeSet::new())?;
let expected: BTreeMap<_, _> = artifacts
.iter()
.map(|artifact| (artifact.path.as_str(), artifact))
.collect();
let mut seen = BTreeSet::new();
let mut archive = Bounded {
input: reader,
remaining: PAYLOAD_LIMIT + ARTIFACT_LIMIT as u64 * 1024 + 10240,
};
while let Some(header) = read_header(&mut archive)? {
let name = std::str::from_utf8(&header.path_bytes())
.map_err(|_| "release archive path is not UTF-8")?
.to_owned();
let artifact = expected
.get(name.as_str())
.ok_or("release archive contains an unsigned file")?;
if !seen.insert(name) || header.size().map_err(|error| error.to_string())? != artifact.bytes
{
return Err("release archive contains a duplicate or wrong-sized file".into());
}
extract_file(&mut archive, root, artifact)?;
}
if seen.len() != expected.len() {
return Err("release archive is missing signed artifacts".into());
}
verify_trailer(&mut archive)
}
fn read_header(reader: &mut impl Read) -> Result<Option<tar::Header>, String> {
let mut header = tar::Header::new_ustar();
reader
.read_exact(header.as_mut_bytes())
.map_err(|error| format!("incomplete release TAR header: {error}"))?;
if header.as_bytes().iter().all(|byte| *byte == 0) {
return Ok(None);
}
let checksum = header.as_bytes()[..148]
.iter()
.chain(&header.as_bytes()[156..])
.map(|byte| u32::from(*byte))
.sum::<u32>()
+ 8 * u32::from(b' ');
if header.cksum().map_err(|error| error.to_string())? != checksum {
return Err("release TAR header checksum is invalid".into());
}
if !header.entry_type().is_file() || header.as_ustar().is_none() {
return Err("release archives may contain only USTAR regular files".into());
}
Ok(Some(header))
}
fn extract_file(reader: &mut impl Read, root: &Path, artifact: &Artifact) -> Result<(), String> {
let path = root.join(&artifact.path);
let mut parent = root.to_path_buf();
for part in Path::new(&artifact.path)
.parent()
.into_iter()
.flat_map(Path::components)
{
parent.push(part);
match fs::create_dir(&parent) {
Ok(()) => {}
Err(error) if error.kind() == io::ErrorKind::AlreadyExists => {
let metadata = fs::symlink_metadata(&parent).map_err(|error| error.to_string())?;
if !metadata.is_dir() || metadata.file_type().is_symlink() {
return Err("release artifact parent is not a real directory".into());
}
}
Err(error) => return Err(format!("cannot create release artifact directory: {error}")),
}
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::set_permissions(&parent, fs::Permissions::from_mode(0o755))
.map_err(|error| error.to_string())?;
}
}
let mut file = OpenOptions::new()
.write(true)
.create_new(true)
.open(&path)
.map_err(|error| format!("cannot create release artifact: {error}"))?;
if io::copy(&mut reader.by_ref().take(artifact.bytes), &mut file)
.map_err(|error| format!("cannot extract release artifact: {error}"))?
!= artifact.bytes
{
return Err("release archive contains a truncated file".into());
}
let padding_length = ((512 - artifact.bytes % 512) % 512) as usize;
let mut padding = [0; 512];
reader
.read_exact(&mut padding[..padding_length])
.map_err(|error| format!("incomplete TAR entry padding: {error}"))?;
if padding[..padding_length].iter().any(|byte| *byte != 0) {
return Err("release TAR entry padding must be zero".into());
}
if sha256_file(&path)? != artifact.sha256 {
return Err("release artifact failed integrity verification".into());
}
// First-install invokes the extracted signed bootstrap before machine install.
// Archive mode bits are untrusted; executable roles come from fixed paths.
let executable = matches!(
artifact.path.as_str(),
"dever-core" | "bootstrap/dever" | "bootstrap/deverd"
);
set_shared_file_permissions(&path, executable)?;
file.sync_all().map_err(|error| error.to_string())
}
fn verify_trailer(reader: &mut impl Read) -> Result<(), String> {
let mut tail_bytes = 0_usize;
let mut tail = [0_u8; 4096];
loop {
let length = reader
.read(&mut tail)
.map_err(|error| format!("invalid release archive trailer: {error}"))?;
if length == 0 {
break;
}
tail_bytes += length;
if tail_bytes > 10240 || tail[..length].iter().any(|byte| *byte != 0) {
return Err("release archive contains an invalid TAR trailer".into());
}
}
// The first zero header was consumed by read_header; require the second.
if tail_bytes < 512 || !tail_bytes.is_multiple_of(512) {
return Err("release archive is missing its complete TAR terminator".into());
}
Ok(())
}
struct Bounded<R> {
input: R,
remaining: u64,
}
impl<R: Read> Read for Bounded<R> {
fn read(&mut self, buffer: &mut [u8]) -> io::Result<usize> {
if buffer.is_empty() {
return Ok(0);
}
if self.remaining == 0 {
let mut probe = [0];
return match self.input.read(&mut probe)? {
0 => Ok(0),
_ => Err(io::Error::new(
io::ErrorKind::InvalidData,
"release byte limit exceeded",
)),
};
}
let limit = (self.remaining.min(buffer.len() as u64)) as usize;
let length = self.input.read(&mut buffer[..limit])?;
self.remaining -= length as u64;
Ok(length)
}
}
+134 -83
View File
@@ -1,31 +1,35 @@
//! Explicit installation/update downloads. Project commands never enter here. //! Explicit installation/update downloads. Project commands never enter here.
use std::collections::{BTreeMap, BTreeSet}; use std::fs::{self, File, OpenOptions};
use std::fs::{self, OpenOptions};
use std::io::{self, Read, Write}; use std::io::{self, Read, Write};
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, Instant}; use std::time::{Duration, Instant};
use super::extensions::Extension;
#[cfg(test)]
use super::extensions::PAYLOAD_LIMIT as RELEASE_LIMIT;
#[cfg(test)]
use super::release::ReleaseManifest;
use super::release::{ use super::release::{
InstallLock, Layout, MachineManager, ReleaseManifest, Version, decode_hex_file, InstallLock, Layout, MachineManager, Version, decode_hex_file, ensure_real_directory,
ensure_real_directory, platform_identity, reject_symlink_ancestors, platform_identity, reject_symlink_ancestors, set_shared_file_permissions, sync_directory,
set_shared_file_permissions, sync_directory, sync_tree, validate_shared_file, sync_tree, validate_shared_file,
}; };
use super::validate_catalog;
const OFFICIAL_RELEASES: &str = "https://github.com/shemic/dever-main/releases"; const OFFICIAL_RELEASES: &str = "https://github.com/shemic/dever-main/releases";
const DOWNLOAD_TIMEOUT: Duration = Duration::from_secs(300); const DOWNLOAD_TIMEOUT: Duration = Duration::from_secs(300);
const METADATA_LIMIT: u64 = 2 * 1024 * 1024; const METADATA_LIMIT: u64 = 2 * 1024 * 1024;
const RELEASE_LIMIT: u64 = 2 * 1024 * 1024 * 1024;
static NEXT_DOWNLOAD: AtomicU64 = AtomicU64::new(0); static NEXT_DOWNLOAD: AtomicU64 = AtomicU64::new(0);
struct ReleaseSource { pub(super) struct ReleaseSource {
base: String, base: String,
agent: ureq::Agent, agent: ureq::Agent,
} }
impl ReleaseSource { impl ReleaseSource {
fn official() -> Self { pub(super) fn official() -> Self {
Self::new(OFFICIAL_RELEASES.to_owned()) Self::new(OFFICIAL_RELEASES.to_owned())
} }
@@ -122,6 +126,23 @@ impl ReleaseSource {
} }
} }
impl super::ExtensionSource for ReleaseSource {
fn open(
&self,
version: &Version,
extension: &super::ExtensionId,
) -> Result<Box<dyn Read>, String> {
Ok(Box::new(
self.response(
self.asset(version.as_str(), &extension.asset_suffix()),
Instant::now(),
)?
.into_body()
.into_reader(),
))
}
}
fn redirect(current: &str, location: &str) -> Result<String, String> { fn redirect(current: &str, location: &str) -> Result<String, String> {
let next = url::Url::parse(current) let next = url::Url::parse(current)
.and_then(|base| base.join(location)) .and_then(|base| base.join(location))
@@ -218,12 +239,11 @@ fn prepare_from(
} }
Err(error) if error.kind() == io::ErrorKind::NotFound => { Err(error) if error.kind() == io::ErrorKind::NotFound => {
let mut response = let mut response =
source.response(source.asset(version.as_str(), "tar.gz"), started)?; source.response(source.asset(version.as_str(), "tar.zst"), started)?;
let compressed = response.body_mut().as_reader().take(RELEASE_LIMIT + 1); super::release_archive::extract(
unpack( response.body_mut().as_reader(),
flate2::read::GzDecoder::new(compressed),
&staging.0, &staging.0,
&manifest, &manifest.artifacts,
)?; )?;
MachineManager::verify_package(&staging.0, Some(&version), &key)?; MachineManager::verify_package(&staging.0, Some(&version), &key)?;
sync_tree(&staging.0)?; sync_tree(&staging.0)?;
@@ -250,80 +270,111 @@ fn prepare_from(
Ok(version) Ok(version)
} }
fn validate_catalog(manifest: &ReleaseManifest) -> Result<(), String> { #[cfg(test)]
if manifest.artifacts.is_empty() || manifest.artifacts.len() > 4096 {
return Err("release artifact count is outside 1..=4096".into());
}
let mut paths = BTreeSet::new();
let mut size = 0_u64;
for artifact in &manifest.artifacts {
if artifact
.path
.split('/')
.any(|part| part.is_empty() || part == "." || part == "..")
|| artifact.path.contains(['\\', ':', '\0'])
|| matches!(artifact.path.as_str(), "manifest.json" | "manifest.sig")
|| !paths.insert(&artifact.path)
{
return Err("release has an unsafe or duplicate artifact path".into());
}
size = size
.checked_add(artifact.bytes)
.ok_or("release size overflow")?;
if size > RELEASE_LIMIT {
return Err("release exceeds its two-GiB unpacked size limit".into());
}
}
Ok(())
}
fn unpack(reader: impl Read, root: &Path, manifest: &ReleaseManifest) -> Result<(), String> { fn unpack(reader: impl Read, root: &Path, manifest: &ReleaseManifest) -> Result<(), String> {
validate_catalog(manifest)?; validate_catalog(manifest)?;
let expected: BTreeMap<_, _> = manifest super::release_archive::unpack(reader, root, &manifest.artifacts)
.artifacts }
.iter()
.map(|artifact| (artifact.path.as_str(), artifact)) pub(super) fn prepare_extension(
.collect(); layout: &Layout,
let mut seen = BTreeSet::new(); version: &Version,
// Raw entries reject PAX/GNU extensions before tar can buffer arbitrary metadata. extension: &Extension,
let mut archive = tar::Archive::new(reader.take(RELEASE_LIMIT + 4096 * 1024)); destination: &Path,
for entry in archive source: &dyn super::ExtensionSource,
.entries() ) -> Result<(), String> {
.map_err(|error| format!("invalid release archive: {error}"))? let staging = Download::new(&layout.staging())?;
.raw(true) super::release_archive::extract(
source.open(version, &extension.id())?,
&staging.0,
&extension.artifacts,
)?;
#[cfg(unix)]
{ {
let mut entry = entry.map_err(|error| format!("invalid release archive entry: {error}"))?; use std::os::unix::fs::PermissionsExt;
if !entry.header().entry_type().is_file() { fs::set_permissions(&staging.0, fs::Permissions::from_mode(0o755))
return Err("release archives may contain only regular files".into()); .map_err(|error| error.to_string())?;
}
let bytes = entry.path_bytes();
let name = std::str::from_utf8(&bytes)
.map_err(|_| "release archive path is not UTF-8")?
.to_owned();
let artifact = expected
.get(name.as_str())
.ok_or("release archive contains an unsigned file")?;
if !seen.insert(name.clone()) || entry.size() != artifact.bytes {
return Err("release archive contains a duplicate or wrong-sized file".into());
}
let path = root.join(&name);
fs::create_dir_all(path.parent().expect("artifact has a parent"))
.map_err(|error| format!("cannot create release artifact directory: {error}"))?;
let mut file = OpenOptions::new()
.write(true)
.create_new(true)
.open(&path)
.map_err(|error| format!("cannot create release artifact: {error}"))?;
let bytes = io::copy(&mut entry, &mut file)
.map_err(|error| format!("cannot extract release artifact: {error}"))?;
if bytes != artifact.bytes {
return Err("release archive contains a truncated file".into());
}
set_shared_file_permissions(&path, false)?;
file.sync_all().map_err(|error| error.to_string())?;
} }
if seen.len() != expected.len() { super::extensions::verify_extension(&staging.0, &extension.artifacts)?;
return Err("release archive is missing signed artifacts".into()); if matches!(extension.kind, super::ExtensionKind::Target) {
super::runtime_pack::validate(&staging.0, version.as_str(), extension.target)?;
}
sync_tree(&staging.0)?;
publish(&staging.0, destination)?;
sync_directory(destination.parent().ok_or("extension has no parent")?)
.map_err(|error| error.to_string())
}
/// Private, independently authenticated installer entry; never accepted over IPC.
pub fn extract_config(config: &Path) -> Result<(), String> {
#[derive(serde::Deserialize)]
#[serde(deny_unknown_fields)]
struct Extraction {
release: PathBuf,
archive: PathBuf,
trusted_key: PathBuf,
}
trusted_input(config, false)?;
let mut bytes = Vec::new();
File::open(config)
.map_err(|error| error.to_string())?
.take(METADATA_LIMIT + 1)
.read_to_end(&mut bytes)
.map_err(|error| error.to_string())?;
if bytes.len() as u64 > METADATA_LIMIT {
return Err("extraction config exceeds its size limit".into());
}
let settings: Extraction = serde_json::from_slice(&bytes)
.map_err(|error| format!("invalid extraction config: {error}"))?;
trusted_input(&settings.release, true)?;
trusted_input(&settings.archive, false)?;
trusted_input(&settings.trusted_key, false)?;
if settings.trusted_key.starts_with(&settings.release) {
return Err("trusted key must be supplied independently outside the release".into());
}
let key = decode_hex_file(&settings.trusted_key, 32)?;
let manifest = MachineManager::signed_manifest(&settings.release, &key)?;
let archive = File::open(&settings.archive).map_err(|error| error.to_string())?;
super::release_archive::extract(archive, &settings.release, &manifest.artifacts)?;
MachineManager::verify_package(&settings.release, Some(&manifest.version), &key)?;
sync_tree(&settings.release)
}
fn trusted_input(path: &Path, directory: bool) -> Result<(), String> {
if !path.is_absolute()
|| path.components().any(|part| {
matches!(
part,
std::path::Component::ParentDir | std::path::Component::CurDir
)
})
{
return Err("extraction paths must be absolute and normalized".into());
}
reject_symlink_ancestors(path)?;
let metadata = fs::symlink_metadata(path)
.map_err(|error| format!("cannot inspect extraction input: {error}"))?;
if directory && !metadata.is_dir() || !directory && !metadata.is_file() {
return Err("extraction input has the wrong file type".into());
}
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
if metadata.uid() != 0
|| metadata.mode() & 0o022 != 0
|| !directory && metadata.nlink() != 1
{
return Err(
"extraction inputs must be root-owned and not writable by other users".into(),
);
}
for ancestor in path.ancestors().skip(1) {
let metadata = fs::symlink_metadata(ancestor).map_err(|error| error.to_string())?;
if metadata.uid() != 0 || metadata.mode() & 0o022 != 0 && metadata.mode() & 0o1000 == 0
{
return Err("extraction input ancestor is not protected by root".into());
}
}
} }
Ok(()) Ok(())
} }
+72 -5
View File
@@ -101,7 +101,13 @@ enum Operation {
Clean, Clean,
ArtifactPut(ArtifactReceipt), ArtifactPut(ArtifactReceipt),
ArtifactGet(ArtifactReceipt), ArtifactGet(ArtifactReceipt),
Compile { bytes: u64 }, EnsureExtension {
version: Version,
extension: super::ExtensionId,
},
Compile {
bytes: u64,
},
} }
#[derive(Debug, Serialize, Deserialize)] #[derive(Debug, Serialize, Deserialize)]
@@ -325,6 +331,15 @@ pub fn artifact_put(layout: &Layout, bytes: &[u8]) -> Result<ArtifactReceipt, St
} }
pub fn artifact_get(layout: &Layout, sha256: &str, bytes: u64) -> Result<Vec<u8>, String> { pub fn artifact_get(layout: &Layout, sha256: &str, bytes: u64) -> Result<Vec<u8>, String> {
artifact_get_optional(layout, sha256, bytes)?
.ok_or_else(|| "shared artifact unavailable: artifact is not cached".into())
}
pub fn artifact_get_optional(
layout: &Layout,
sha256: &str,
bytes: u64,
) -> Result<Option<Vec<u8>>, String> {
let expected = ArtifactReceipt { let expected = ArtifactReceipt {
sha256: sha256.to_owned(), sha256: sha256.to_owned(),
bytes, bytes,
@@ -341,7 +356,11 @@ pub fn artifact_get(layout: &Layout, sha256: &str, bytes: u64) -> Result<Vec<u8>
installed: Vec::new(), installed: Vec::new(),
}, },
)?; )?;
require_artifact(read_frame(&mut stream)?, &expected)?; let response: Response = read_frame(&mut stream)?;
if response.ok && response.artifact.is_none() {
return Ok(None);
}
require_artifact(response, &expected)?;
let mut bytes = vec![0; expected.bytes as usize]; let mut bytes = vec![0; expected.bytes as usize];
stream stream
.read_exact(&mut bytes) .read_exact(&mut bytes)
@@ -349,7 +368,7 @@ pub fn artifact_get(layout: &Layout, sha256: &str, bytes: u64) -> Result<Vec<u8>
if hex(&Sha256::digest(&bytes)) != expected.sha256 { if hex(&Sha256::digest(&bytes)) != expected.sha256 {
return Err("deverd artifact download failed SHA-256 verification".into()); return Err("deverd artifact download failed SHA-256 verification".into());
} }
Ok(bytes) Ok(Some(bytes))
} }
#[cfg(not(unix))] #[cfg(not(unix))]
{ {
@@ -358,6 +377,30 @@ pub fn artifact_get(layout: &Layout, sha256: &str, bytes: u64) -> Result<Vec<u8>
} }
} }
pub fn ensure_extension(
layout: &Layout,
version: &Version,
kind: super::ExtensionKind,
target: super::BuildTarget,
) -> Result<(), String> {
Version::parse(version.as_str())?;
let response = request(
layout,
Operation::EnsureExtension {
version: version.clone(),
extension: super::ExtensionId { kind, target },
},
&[],
)?;
if response.ok {
Ok(())
} else {
Err(response
.error
.unwrap_or_else(|| "extension preparation failed".into()))
}
}
fn require_artifact(response: Response, expected: &ArtifactReceipt) -> Result<(), String> { fn require_artifact(response: Response, expected: &ArtifactReceipt) -> Result<(), String> {
if !response.ok { if !response.ok {
return Err(response return Err(response
@@ -418,6 +461,11 @@ fn request(
#[cfg(unix)] #[cfg(unix)]
{ {
let (mut stream, token) = connect(layout)?; let (mut stream, token) = connect(layout)?;
if matches!(operation, Operation::EnsureExtension { .. }) {
stream
.set_read_timeout(Some(std::time::Duration::from_secs(620)))
.map_err(|error| error.to_string())?;
}
let request = Request { let request = Request {
token, token,
operation, operation,
@@ -513,8 +561,16 @@ fn handle(
}; };
} }
Operation::ArtifactGet(expected) => { Operation::ArtifactGet(expected) => {
let mut artifact = match store.artifact(&expected) { let mut artifact = match store.artifact_optional(&expected) {
Ok(artifact) => artifact, Ok(Some(artifact)) => artifact,
Ok(None) => {
return Response {
ok: true,
status: None,
artifact: None,
error: None,
};
}
Err(error) => return failure(error), Err(error) => return failure(error),
}; };
if let Err(error) = write_frame(stream, &artifact_response(artifact.receipt)) { if let Err(error) = write_frame(stream, &artifact_response(artifact.receipt)) {
@@ -530,6 +586,17 @@ fn handle(
error: None, error: None,
}; };
} }
Operation::EnsureExtension { version, extension } => {
return match super::extensions::ensure(layout, &version, &extension) {
Ok(()) => Response {
ok: true,
status: None,
artifact: None,
error: None,
},
Err(error) => failure(error),
};
}
Operation::Compile { bytes } => { Operation::Compile { bytes } => {
return match compile::serve(layout, stream, bytes) { return match compile::serve(layout, stream, bytes) {
Ok(()) => Response { Ok(()) => Response {
@@ -112,13 +112,15 @@ fn verify_pack(
target: super::super::BuildTarget, target: super::super::BuildTarget,
) -> Result<(), String> { ) -> Result<(), String> {
let root = compiler let root = compiler
.core .resources
.parent() .root(&format!("runtime/{}/manifest.json", target.platform()))?;
.ok_or("installed compiler has no directory")?;
let artifacts = let artifacts =
super::super::runtime_pack::validate(root, compiler.manifest.version.as_str(), target)?; super::super::runtime_pack::validate(&root, compiler.manifest.version.as_str(), target)?;
let signed_artifacts = compiler
.resources
.artifacts(&format!("runtime/{}/", target.platform()))?;
for input in artifacts { for input in artifacts {
if !compiler.manifest.artifacts.iter().any(|signed| { if !signed_artifacts.iter().any(|signed| {
signed.path == input.path signed.path == input.path
&& signed.bytes == input.bytes && signed.bytes == input.bytes
&& signed.sha256 == input.sha256 && signed.sha256 == input.sha256
+1 -1
View File
@@ -16,7 +16,7 @@ Run the opt-in host SDK check with explicit absolute runtime paths, for example:
Omitted runtime paths report `unavailable`; the script never discovers language runtimes through `PATH`. It builds the Go fixtures in a temporary offline cache. This check exercises both the lower-level and compiler-manifest-shaped typed paths on host interpreters. Managed runtime packaging and execution on a machine without host language runtimes require separate acceptance. Omitted runtime paths report `unavailable`; the script never discovers language runtimes through `PATH`. It builds the Go fixtures in a temporary offline cache. This check exercises both the lower-level and compiler-manifest-shaped typed paths on host interpreters. Managed runtime packaging and execution on a machine without host language runtimes require separate acceptance.
Managed dependencies use `dever-lock-v5`. Python extras and selected wheel METADATA, npm nested instances/peers/optional/bundled dependencies, and authenticated Go module evidence are part of the exact lock. Native Python wheels install into the packaged interpreter prefix and must satisfy its signed tags, extension ABI and loader closure. Managed dependencies use `dever-lock-v6`. Python extras and selected wheel METADATA, npm nested instances/peers/optional/bundled dependencies, and authenticated Go module evidence are part of the exact lock. Original archives retain their fixed source locator and filename; build receipts retain the complete execution input lock, including npm's graph before optional build pruning. `dever lib install <project>` restores these exact inputs and artifacts without resolving new versions or modifying configuration/lock bytes. Missing built outputs are replayed through the same sandbox executor and must match the recorded output digest; non-reproducible outputs fail explicitly. Native Python wheels install into the packaged interpreter prefix and must satisfy its signed tags, extension ABI and loader closure.
Cross builds select the application platform explicitly: `dever lib update <project> --target linux-aarch64`, followed by `dever build <project> --target linux-aarch64 --output <new-file>`. Worker source and Port contracts stay unchanged. Locks, interpreters, native dependencies and sandbox assets must match that target. A Go runtime pack additionally declares `build.host` for its compiler/linker/analyzer; `build.goos/goarch` and standard-library object headers describe the output architecture. The tools run with a cleared environment and must have the correct target baked in. Missing target assets are errors, with no host runtime fallback. `run` and ordinary tests remain host operations. Cross builds select the application platform explicitly: `dever lib update <project> --target linux-aarch64`, followed by `dever build <project> --target linux-aarch64 --output <new-file>`. Worker source and Port contracts stay unchanged. Locks, interpreters, native dependencies and sandbox assets must match that target. A Go runtime pack additionally declares `build.host` for its compiler/linker/analyzer; `build.goos/goarch` and standard-library object headers describe the output architecture. The tools run with a cleared environment and must have the correct target baked in. Missing target assets are errors, with no host runtime fallback. `run` and ordinary tests remain host operations.
+13 -4
View File
@@ -6,7 +6,7 @@ Build the four runtime archives separately, then assemble and sign a new directo
## Build inputs ## Build inputs
Build the compiler with `cargo build --offline --locked -p dever-cli --bin dever`; its author input becomes the installed `dever-core`. Build the stable machine launcher with `--bin dever-launcher` and the daemon with `--bin deverd`. Use `target/debug/dever-launcher` for the bootstrap launcher input, which the maker publishes as `bootstrap/dever`. The installed public command remains `dever`; `dever-launcher` is only an author build name. Preserve this distinction when preparing release inputs. Build the compiler with `cargo build --release --offline --locked -p dever-cli --bin dever`; its author input becomes the installed `dever-core`. Build the stable machine launcher with `--bin dever-launcher` and the daemon with `--bin deverd` in the same release profile. Use `target/release/dever-launcher` for the bootstrap launcher input, which the maker publishes as `bootstrap/dever`. The release profile uses fat LTO, one codegen unit and symbol stripping; `runtime-pack` overrides stripping to retain linkable archive symbols. The installed public command remains `dever`; `dever-launcher` is only an author build name.
Prepare the pinned Rust dependencies and compiler LLVM 18/LLD SDK explicitly. Use an explicitly selected Cargo/Rust toolchain for author builds. The root `runtime-pack` Cargo profile inherits the release optimization settings, uses one codegen unit and removes debug information while retaining archive symbols needed by LLD. It must not enable the bridge's `embedded` feature. Prepare the pinned Rust dependencies and compiler LLVM 18/LLD SDK explicitly. Use an explicitly selected Cargo/Rust toolchain for author builds. The root `runtime-pack` Cargo profile inherits the release optimization settings, uses one codegen unit and removes debug information while retaining archive symbols needed by LLD. It must not enable the bridge's `embedded` feature.
@@ -34,7 +34,7 @@ Prepare a matching compiler core, `libLLVM.so.18.1`, target CRT objects and genu
## Author settings ## Author settings
Application authors select `dever build <project-root> --target linux-aarch64 --output <new-file>`; omitted targets, `run`, and `test` use the host. This selects application output, not an ARM compiler installation. Missing target packs are errors. Product commands do not discover cross tools or download packs while building. Application authors first prepare `dever target add linux-aarch64`, then select `dever build <project-root> --target linux-aarch64 --output <new-file>`; omitted targets, `run`, and `test` use the host. This selects application output, not an ARM compiler installation. Missing target packs are errors with a preparation command. Product commands do not discover cross tools or download packs while building.
For an additional application target, add `native_targets.linux-aarch64` to the author settings. Its value has the same `profiles`, `start`, `libraries`, and `end` fields as the host inputs below. Build all four archives with Rust target `aarch64-unknown-linux-gnu` and an explicitly prepared cross linker/sysroot. Use ARM CRT and static libraries, never host files or linker scripts. Both standalone object and archive member ELF headers are checked against the declared target before publishing and on cache access. For an additional application target, add `native_targets.linux-aarch64` to the author settings. Its value has the same `profiles`, `start`, `libraries`, and `end` fields as the host inputs below. Build all four archives with Rust target `aarch64-unknown-linux-gnu` and an explicitly prepared cross linker/sysroot. Use ARM CRT and static libraries, never host files or linker scripts. Both standalone object and archive member ELF headers are checked against the declared target before publishing and on cache access.
@@ -96,7 +96,11 @@ Use a separate author root, not a Dever application's configuration. Place its e
`dever skill path [project-root]` validates and returns the skill belonging to the selected compiler. `dever skill install <new-directory>` installs a small AI entry that resolves the active signed skill on each task. It refuses existing directories. `dever update` and `dever use` change the one active-version journal used by both core and skill; they do not rewrite every user's AI directory. Ordinary users can read the signed skill without the install lock or administrator permissions. `dever skill path [project-root]` validates and returns the skill belonging to the selected compiler. `dever skill install <new-directory>` installs a small AI entry that resolves the active signed skill on each task. It refuses existing directories. `dever update` and `dever use` change the one active-version journal used by both core and skill; they do not rewrite every user's AI directory. Ordinary users can read the signed skill without the install lock or administrator permissions.
Publish official versions to `shemic/dever-main` GitHub Releases with tag `v<version>`. `sdk/release-assets.py` converts the signed directory into three assets named `dever-<platform>.manifest.json`, `dever-<platform>.manifest.sig`, and `dever-<platform>.tar.gz`. The archive contains exactly the manifest's ordinary artifact files, with no directory headers or embedded manifest. The installer and explicit `dever install/update` download these assets, verify the independent pinned public key before payload extraction, and never use application configuration or environment variables for the official address. Project `run/build` remain offline. Publish official versions to `shemic/dever-main` GitHub Releases with tag `v<version>`. The v2 signed root catalog separates base `artifacts` from `extensions`. Base includes core/private libraries, bootstrap, host native profiles/sandbox and skill. Each extension declares `{kind,target,artifacts}`; `kind` is `{"type":"runtime","ecosystem":"pip|npm|go"}`, `{"type":"build","ecosystem":"pip|npm"}` or `{"type":"target"}`. Target combines that target's native profiles/CRT and sandbox. Paths and identities cannot overlap. No extension has an independent unsigned manifest or can override base files.
`python3 sdk/release-assets.py <release> --output <new-assets> --zstd /absolute/author/zstd` writes `dever-<platform>.manifest.json`, `.manifest.sig`, `.tar.zst`, plus `.ext-<runtime-pip|runtime-npm|runtime-go|build-pip|build-npm|target>-<target>.tar.zst`. Each USTAR contains exactly its signed ordinary artifact files, no directory headers or embedded root metadata. Zstandard uses a checksummed single frame with a maximum 128 MiB window. The publisher also emits `.blob-<sha256>` for bootstrap/dever and every bootstrap/lib dependency. First installation authenticates these raw helpers before executing the launcher's bounded extractor; end users do not need a system zstd executable. Report helper bytes separately from base archive size.
Only explicit install/update, target add and dependency preparation download official resources. The source address is compiled into the toolchain, not application configuration or environment variables. Extension preparation uses closed typed IPC and the existing machine service, storing immutable version/catalog-bound resources under `cache/extensions`. Multiple projects/users share one verified tree. `run/build` and resource reads never download. `update` prepares the new core/skill and the active version's already installed extension selection before publishing the active-version journal; failure keeps the old selection usable. The stable machine launcher/daemon are provisioned by bootstrap, so changes to that protocol require a matching trusted bootstrap installation.
## Linux first installation ## Linux first installation
@@ -234,14 +238,19 @@ The third command requires root and namespace permission for a test-owned mount
The harness rejects a dynamically linked init. Inside the already pivoted namespace, it replaces bwrap's proc submounts with a complete private proc mount so the nested Worker can mount its own proc. It keeps the outer root fixture's existing capabilities; the product's inner namespace, capability removal and guard remain unchanged. It never binds the host proc into the final root. This private compiler test complements the managed daemon test; it does not install a machine service. The harness rejects a dynamically linked init. Inside the already pivoted namespace, it replaces bwrap's proc submounts with a complete private proc mount so the nested Worker can mount its own proc. It keeps the outer root fixture's existing capabilities; the product's inner namespace, capability removal and guard remain unchanged. It never binds the host proc into the final root. This private compiler test complements the managed daemon test; it does not install a machine service.
To accept an already prepared official release (including its real signing identity), use the owned-image verifier. `--assets` feeds the exact three download files through the real installer using local transport; signature checks, extraction and bootstrap execution are unchanged. It requires the explicit static bwrap/init fixtures described above, creates a new image, starts only its own daemon and records each passed stage. Online `dever update` and real-host service activation remain separate checks. To accept an already prepared official release (including its real signing identity), use the owned-image verifier. `--assets` feeds the exact split download files through the real installer using local transport; signature checks, extraction and bootstrap execution are unchanged. The `prepare-extensions` author example replaces only extension transport with actual local assets; it uses the product owner for validation and installation. Supply it with `--extension-helper` to exercise ARM preparation, CLI reuse and cross compilation. The verifier requires explicit static bwrap/init fixtures, creates a new image, starts only its own daemon and records each passed stage. Online update and real-host service activation remain separate checks.
```sh ```sh
python3 -B test/onboarding/verify_prepared_release.py \ python3 -B test/onboarding/verify_prepared_release.py \
--release /prepared/release --assets /prepared/download-assets \ --release /prepared/release --assets /prepared/download-assets \
--init /prepared/native-acceptance-init --extension-helper /prepared/prepare-extensions \
--image /new/owned-image --report /owned-records/acceptance.json --image /new/owned-image --report /owned-records/acceptance.json
``` ```
The image and its owned ancestor directories must be traversable by the two test UIDs (65533 and 65534). Their resource-reuse check starts a separate private-machine daemon with real kernel identities, because the minimal OS fixture's single-UID user namespace cannot represent another user. It does not touch the global service.
`test/onboarding/verify_release_update.py` accepts the same asset/init/helper paths plus `--key <explicit-author-pkcs8>`. It re-signs the real payload catalog as an acceptance-only next version, verifies that a missing extension preserves the old working selection, then verifies that a successful update activates core/skill and exactly the installed extension set. That fixture version is never published; this is local update-transaction evidence, not an online Release acceptance.
The existing fixture-based bootstrap regression additionally requires the built `dever` and `deverd` binaries: The existing fixture-based bootstrap regression additionally requires the built `dever` and `deverd` binaries:
```sh ```sh
+135 -40
View File
@@ -1,18 +1,132 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Turn a prepared signed release into the three official GitHub release assets.""" """Publish base, optional extension and authenticated bootstrap assets from a signed release."""
import argparse import argparse
import ctypes import ctypes
import gzip
import hashlib import hashlib
import json import json
import os import os
from pathlib import Path, PurePosixPath from pathlib import Path, PurePosixPath
import re
import shutil
import subprocess
import tarfile import tarfile
import tempfile import tempfile
def create(release, output): MAX_PACKAGE = 2 * 1024**3
def extension_suffix(extension):
kind = extension["kind"]
target = extension["target"]
if target not in ("linux-x86_64", "linux-aarch64"):
raise ValueError("unsupported extension target")
if kind == {"type": "target"}:
name = "target"
elif (set(kind) == {"type", "ecosystem"}
and kind["type"] in ("runtime", "build")
and kind["ecosystem"] in ("pip", "npm", "go")
and not (kind["type"] == "build" and kind["ecosystem"] == "go")):
name = f"{kind['type']}-{kind['ecosystem']}"
else:
raise ValueError("unsupported extension kind")
return f"ext-{name}-{target}"
def packages(manifest):
if (set(manifest) != {"format", "version", "platform", "artifacts", "extensions"}
or manifest["format"] != "dever-release-v2"
or manifest["platform"] not in ("linux-x86_64", "linux-aarch64")
or not re.fullmatch(r"(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)", manifest["version"])):
raise ValueError("unsupported release format, version or platform")
if not isinstance(manifest["extensions"], list) or len(manifest["extensions"]) > 12:
raise ValueError("release extension count exceeds limit")
selected = {"": manifest["artifacts"]}
for extension in manifest["extensions"]:
if set(extension) != {"kind", "target", "artifacts"}:
raise ValueError("invalid extension fields")
suffix = extension_suffix(extension)
if suffix in selected:
raise ValueError("duplicate extension identity")
target = extension["target"]
kind = extension["kind"]
if kind["type"] == "target":
if target == manifest["platform"]:
raise ValueError("host native resources belong to the base package")
prefixes = (f"runtime/{target}/", f"sandbox/{target}/")
else:
prefixes = (f"{kind['type']}/{kind['ecosystem']}/{target}/",)
if any(not artifact["path"].startswith(prefixes) for artifact in extension["artifacts"]):
raise ValueError("extension artifact is outside its resource namespace")
selected[suffix] = extension["artifacts"]
return selected
def verify_artifacts(release, groups):
names = set()
for artifacts in groups.values():
if (not 1 <= len(artifacts) <= 4096
or any(type(entry["bytes"]) is not int or entry["bytes"] < 0 for entry in artifacts)
or sum(entry["bytes"] for entry in artifacts) > MAX_PACKAGE):
raise ValueError("release exceeds the official asset limits")
for artifact in artifacts:
verify_artifact(release, artifact, names)
for name in names:
if any(str(parent) in names for parent in PurePosixPath(name).parents):
raise ValueError("release artifacts overlap as file and directory")
def verify_artifact(release, artifact, names):
name = artifact["path"]
if (set(artifact) != {"path", "bytes", "sha256"}
or not name or len(name.encode("utf-8")) > 1024
or str(PurePosixPath(name)) != name or name.startswith("/")
or any(part in ("", ".", "..") for part in name.split("/"))
or any(character in name for character in ("\\", ":", "\0"))
or name in names or name in ("manifest.json", "manifest.sig")
or not re.fullmatch(r"[0-9a-f]{64}", artifact["sha256"])):
raise ValueError("invalid release artifact path or digest")
names.add(name)
path = release / name
for parent in (path, *path.parents):
if parent.is_symlink():
raise ValueError("release artifacts must not traverse symlinks")
if parent == release:
break
if not path.is_file() or path.stat().st_size != artifact["bytes"]:
raise ValueError("release artifact type/size mismatch")
with path.open("rb") as stream:
if hashlib.file_digest(stream, "sha256").hexdigest() != artifact["sha256"]:
raise ValueError("release artifact digest mismatch")
def compress(release, artifacts, destination, zstd):
# Compression is an explicit author tool; installation uses the signed launcher codec.
with destination.open("xb") as output:
process = subprocess.Popen([str(zstd), "-q", "-10", "--long=27", "-c"],
stdin=subprocess.PIPE, stdout=output,
stderr=subprocess.PIPE, env={})
try:
with process.stdin:
with tarfile.open(fileobj=process.stdin, mode="w|", format=tarfile.USTAR_FORMAT) as archive:
for artifact in sorted(artifacts, key=lambda entry: entry["path"]):
header = tarfile.TarInfo(artifact["path"])
header.size = artifact["bytes"]
header.mode = 0o644
with (release / artifact["path"]).open("rb") as contents:
archive.addfile(header, contents)
errors = process.stderr.read(4096)
if process.wait() != 0:
raise ValueError(f"release compression failed: {errors.decode(errors='replace')}")
finally:
if process.poll() is None:
process.kill()
process.wait()
process.stderr.close()
def create(release, output, zstd):
release = release.absolute() release = release.absolute()
output = output.absolute() output = output.absolute()
if release.is_symlink() or not release.is_dir(): if release.is_symlink() or not release.is_dir():
@@ -21,49 +135,29 @@ def create(release, output):
raise ValueError("output already exists") raise ValueError("output already exists")
if not output.parent.is_dir(): if not output.parent.is_dir():
raise ValueError("output parent must exist") raise ValueError("output parent must exist")
manifest_bytes = (release / "manifest.json").read_bytes() if not zstd.is_absolute() or not zstd.is_file():
raise ValueError("zstd must be an explicit absolute author tool path")
with (release / "manifest.json").open("rb") as stream:
manifest_bytes = stream.read(2 * 1024**2 + 1)
if len(manifest_bytes) > 2 * 1024**2:
raise ValueError("release metadata exceeds limit")
manifest = json.loads(manifest_bytes) manifest = json.loads(manifest_bytes)
if manifest["format"] != "dever-release-v1" or manifest["platform"] not in ("linux-x86_64", "linux-aarch64"): groups = packages(manifest)
raise ValueError("unsupported release format or platform") verify_artifacts(release, groups)
artifacts = manifest["artifacts"]
if not 1 <= len(artifacts) <= 4096 or sum(entry["bytes"] for entry in artifacts) > 2 * 1024**3:
raise ValueError("release exceeds the official asset limits")
names = set()
for artifact in artifacts:
name = artifact["path"]
if (not name or str(PurePosixPath(name)) != name or name.startswith("/")
or any(part in ("", ".", "..") for part in name.split("/"))
or any(character in name for character in ("\\", ":", "\0"))
or name in names or name in ("manifest.json", "manifest.sig")):
raise ValueError("invalid release artifact path")
names.add(name)
path = release / name
for parent in (path, *path.parents):
if parent.is_symlink():
raise ValueError("release artifacts must not traverse symlinks")
if parent == release:
break
if not path.is_file() or path.stat().st_size != artifact["bytes"]:
raise ValueError("release artifact type/size mismatch")
with path.open("rb") as stream:
if hashlib.file_digest(stream, "sha256").hexdigest() != artifact["sha256"]:
raise ValueError("release artifact digest mismatch")
with tempfile.TemporaryDirectory(prefix=".dever-release-assets-", dir=output.parent) as scratch: with tempfile.TemporaryDirectory(prefix=".dever-release-assets-", dir=output.parent) as scratch:
prepared = Path(scratch) / "assets" prepared = Path(scratch) / "assets"
prepared.mkdir() prepared.mkdir()
stem = f"dever-{manifest['platform']}" stem = f"dever-{manifest['platform']}"
(prepared / f"{stem}.manifest.json").write_bytes(manifest_bytes) (prepared / f"{stem}.manifest.json").write_bytes(manifest_bytes)
(prepared / f"{stem}.manifest.sig").write_bytes((release / "manifest.sig").read_bytes()) (prepared / f"{stem}.manifest.sig").write_bytes((release / "manifest.sig").read_bytes())
with (prepared / f"{stem}.tar.gz").open("xb") as compressed: for suffix, artifacts in groups.items():
with gzip.GzipFile(filename="", fileobj=compressed, mode="wb", mtime=0) as stream: filename = f"{stem}{'.' + suffix if suffix else ''}.tar.zst"
with tarfile.open(fileobj=stream, mode="w|", format=tarfile.USTAR_FORMAT) as archive: compress(release, artifacts, prepared / filename, zstd)
for artifact in sorted(artifacts, key=lambda entry: entry["path"]): for artifact in manifest["artifacts"]:
path = release / artifact["path"] if artifact["path"] == "bootstrap/dever" or artifact["path"].startswith("bootstrap/lib/"):
header = tarfile.TarInfo(artifact["path"]) blob = prepared / f"{stem}.blob-{artifact['sha256']}"
header.size = artifact["bytes"] if not blob.exists():
header.mode = 0o644 shutil.copyfile(release / artifact["path"], blob)
with path.open("rb") as contents:
archive.addfile(header, contents)
publish(prepared, output) publish(prepared, output)
return sorted(path.name for path in output.iterdir()) return sorted(path.name for path in output.iterdir())
@@ -84,9 +178,10 @@ def main():
parser = argparse.ArgumentParser(description=__doc__) parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("release", type=Path) parser.add_argument("release", type=Path)
parser.add_argument("--output", required=True, type=Path) parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--zstd", required=True, type=Path, help="explicit author-side zstd executable")
args = parser.parse_args() args = parser.parse_args()
try: try:
for name in create(args.release, args.output): for name in create(args.release, args.output, args.zstd):
print(name) print(name)
except (OSError, ValueError, KeyError, TypeError, tarfile.TarError) as error: except (OSError, ValueError, KeyError, TypeError, tarfile.TarError) as error:
parser.exit(1, f"cannot prepare release assets: {error}\n") parser.exit(1, f"cannot prepare release assets: {error}\n")
+1 -1
Submodule skills updated: 9fd78fc820...27ac8f21c2.
+3
View File
@@ -30,6 +30,8 @@ mod native_elf;
mod npm_build_tests; mod npm_build_tests;
#[path = "external_libs/npm.rs"] #[path = "external_libs/npm.rs"]
mod npm_tests; mod npm_tests;
#[path = "external_libs/restore.rs"]
mod restore_tests;
#[path = "external_libs/sumdb.rs"] #[path = "external_libs/sumdb.rs"]
mod sumdb_tests; mod sumdb_tests;
#[path = "support/wheel.rs"] #[path = "support/wheel.rs"]
@@ -1694,6 +1696,7 @@ fn expanded_worker_resources_keep_archives_needed_by_exec_or_declarations() {
.collect(), .collect(),
runtime: registry_runtime(spec.ecosystem.clone()).pack, runtime: registry_runtime(spec.ecosystem.clone()).pack,
artifacts: vec![LockedArtifact { artifacts: vec![LockedArtifact {
source: None,
target: target.clone(), target: target.clone(),
path: path(spec), path: path(spec),
bytes: 1, bytes: 1,
+64 -15
View File
@@ -280,14 +280,16 @@ fn backend_fixture(mutate: bool) -> LocalRegistry {
), ),
], ],
); );
let key = (Ecosystem::Pip, "/pypi/buildhelper/1.0.0/json".into()); let helper_source = source_archive(vec![
let mut release: serde_json::Value = serde_json::from_slice(&registry.0[&key]).unwrap(); ("buildhelper-1.0.0/pyproject.toml", b"[build-system]\nrequires = []\nbuild-backend = 'backend'\nbackend-path = ['.']\n".to_vec()),
let url = release["urls"][0]["url"].as_str().unwrap().to_owned(); ("buildhelper-1.0.0/backend.py", b"import shutil\nfrom pathlib import Path\ndef build_wheel(directory, config_settings=None, metadata_directory=None):\n name = 'buildhelper-1.0.0-py3-none-any.whl'\n shutil.copy('result.whl', Path(directory) / name)\n return name\n".to_vec()),
release["urls"][0]["digests"]["sha256"] = serde_json::json!(digest(&helper)); ("buildhelper-1.0.0/result.whl", helper),
]);
registry.0.insert((Ecosystem::Pip, "/pypi/buildhelper/1.0.0/json".into()),
serde_json::to_vec(&serde_json::json!({"info":{},"urls":[{"packagetype":"sdist","filename":"buildhelper-1.0.0.tar.gz","url":"helper-source","digests":{"sha256":digest(&helper_source)}}]})).unwrap());
registry registry
.0 .0
.insert(key, serde_json::to_vec(&release).unwrap()); .insert((Ecosystem::Pip, "helper-source".into()), helper_source);
registry.0.insert((Ecosystem::Pip, url), helper);
let original = wheel_fixture::metadata("sample", "1.0.0", &[], &[]); let original = wheel_fixture::metadata("sample", "1.0.0", &[], &[]);
let changed = format!("{original}Summary: changed by wheel hook\n"); let changed = format!("{original}Summary: changed by wheel hook\n");
let wheel = wheel_fixture::pack( let wheel = wheel_fixture::pack(
@@ -334,11 +336,20 @@ def build_wheel(directory, config_settings=None, metadata_directory=None):
("sample-1.0.0/changed-metadata", changed.into_bytes()), ("sample-1.0.0/changed-metadata", changed.into_bytes()),
("sample-1.0.0/result.whl", wheel), ("sample-1.0.0/result.whl", wheel),
]; ];
let source = source_archive(sourcefiles);
registry.0.insert((Ecosystem::Pip,"/pypi/sample/1.0.0/json".into()),serde_json::to_vec(&serde_json::json!({"info":{},"urls":[{"packagetype":"sdist","filename":"sample-1.0.0.tar.gz","url":"sample-source","digests":{"sha256":digest(&source)}}]})).unwrap());
registry
.0
.insert((Ecosystem::Pip, "sample-source".into()), source);
registry
}
fn source_archive(files: Vec<(&str, Vec<u8>)>) -> Vec<u8> {
let mut tar = tar::Builder::new(flate2::write::GzEncoder::new( let mut tar = tar::Builder::new(flate2::write::GzEncoder::new(
Vec::new(), Vec::new(),
flate2::Compression::default(), flate2::Compression::default(),
)); ));
for (name, bytes) in sourcefiles { for (name, bytes) in files {
let mut header = tar::Header::new_gnu(); let mut header = tar::Header::new_gnu();
header.set_size(bytes.len() as u64); header.set_size(bytes.len() as u64);
header.set_mode(0o644); header.set_mode(0o644);
@@ -346,12 +357,7 @@ def build_wheel(directory, config_settings=None, metadata_directory=None):
tar.append_data(&mut header, name, bytes.as_slice()) tar.append_data(&mut header, name, bytes.as_slice())
.unwrap(); .unwrap();
} }
let source = tar.into_inner().unwrap().finish().unwrap(); tar.into_inner().unwrap().finish().unwrap()
registry.0.insert((Ecosystem::Pip,"/pypi/sample/1.0.0/json".into()),serde_json::to_vec(&serde_json::json!({"info":{},"urls":[{"packagetype":"sdist","filename":"sample-1.0.0.tar.gz","url":"sample-source","digests":{"sha256":digest(&source)}}]})).unwrap());
registry
.0
.insert((Ecosystem::Pip, "sample-source".into()), source);
registry
} }
#[test] #[test]
@@ -383,12 +389,55 @@ fn pep517_hooks_keep_requirements_cli_and_metadata_boundaries() {
let lock = result.unwrap(); let lock = result.unwrap();
dever_cli::libs::doctor(&lock).unwrap(); dever_cli::libs::doctor(&lock).unwrap();
assert_eq!(lock.builds[0].dynamic_requires, vec!["buildhelper==1.0.0"]); assert_eq!(lock.builds[0].dynamic_requires, vec!["buildhelper==1.0.0"]);
assert_eq!(
lock.builds[0].inputs.builds.len(),
1,
"nested build requirement receipt must be retained"
);
let before = lock.encode().unwrap();
let cold = FixtureArtifactStore::default();
let archives = super::restore_tests::archives_only(&lock, &transport);
dever_cli::libs::restore::restore_artifacts(
&lock,
&archives,
&cold,
&inputs,
&inputs.settings.runtime.target,
&dever_cli::libs::sumdb::Verifier::official(),
)
.unwrap();
assert_eq!(lock.encode().unwrap(), before);
let mut mismatch = lock.clone();
mismatch.builds[0].output.sha256 = "0".repeat(64);
mismatch.libs[0].artifacts = vec![mismatch.builds[0].output.clone()];
mismatch.libs[0].build = Some(mismatch.builds[0].identity().unwrap());
let rejected = FixtureArtifactStore::default();
assert!(
dever_cli::libs::restore::restore_artifacts(
&mismatch,
&archives,
&rejected,
&inputs,
&inputs.settings.runtime.target,
&dever_cli::libs::sumdb::Verifier::official()
)
.unwrap_err()
.contains("not byte reproducible")
);
assert!(
!rejected
.has(
&mismatch.builds[0].output.sha256,
&inputs.settings.runtime.target
)
.unwrap()
);
for mutation in 0..3 { for mutation in 0..3 {
let mut altered = lock.clone(); let mut altered = lock.clone();
match mutation { match mutation {
0 => altered.builds[0].tools.sha256 = "not-a-hash".into(), 0 => altered.builds[0].tools.sha256 = "not-a-hash".into(),
1 => altered.builds[0].dynamic_requires = vec!["buildhelper>=2".into()], 1 => altered.builds[0].dynamic_requires = vec!["buildhelper>=2".into()],
_ => *altered.builds[0].dependencies = LockFile::new(vec![]).unwrap(), _ => *altered.builds[0].inputs = LockFile::new(vec![]).unwrap(),
} }
altered.libs[0].build = Some(altered.builds[0].identity().unwrap()); altered.libs[0].build = Some(altered.builds[0].identity().unwrap());
assert!( assert!(
@@ -441,7 +490,7 @@ fn builds_official_python_source_with_isolated_pep517() {
let lock = dever_cli::libs::LibResolver::resolve(&resolver, &requested).unwrap(); let lock = dever_cli::libs::LibResolver::resolve(&resolver, &requested).unwrap();
dever_cli::libs::doctor(&lock).unwrap(); dever_cli::libs::doctor(&lock).unwrap();
assert_eq!(lock.builds.len(), 1); assert_eq!(lock.builds.len(), 1);
assert!(!lock.builds[0].dependencies.libs.is_empty()); assert!(!lock.builds[0].inputs.libs.is_empty());
let artifact = &lock let artifact = &lock
.libs .libs
.iter() .iter()
@@ -65,7 +65,7 @@ fn npm_registry_install_hooks_do_not_run_publishing_lifecycle() {
); );
} }
fn locked_installation_fixture() -> (LockFile, Vec<u8>) { fn locked_installation_fixture() -> (LockFile, Vec<u8>, FixtureArtifactStore) {
let mut registry = LocalRegistry(BTreeMap::new()); let mut registry = LocalRegistry(BTreeMap::new());
npm_tests::release( npm_tests::release(
&mut registry, &mut registry,
@@ -85,11 +85,13 @@ fn locked_installation_fixture() -> (LockFile, Vec<u8>) {
"npm:other@1.0.0".parse().unwrap(), "npm:other@1.0.0".parse().unwrap(),
]) ])
.unwrap(); .unwrap();
let sources = lock let mut sources: Vec<_> = lock
.libs .libs
.iter() .iter()
.flat_map(|lib| lib.artifacts.clone()) .flat_map(|lib| lib.artifacts.clone())
.collect(); .collect();
sources.sort();
sources.dedup();
let archives = lock let archives = lock
.libs .libs
.iter() .iter()
@@ -159,9 +161,10 @@ fn locked_installation_fixture() -> (LockFile, Vec<u8>) {
), ),
python: None, python: None,
dynamic_requires: vec![], dynamic_requires: vec![],
dependencies: Box::new(LockFile::new(vec![]).unwrap()), inputs: Box::new(lock.clone()),
config_settings: BTreeMap::new(), config_settings: BTreeMap::new(),
output: dever_cli::libs::LockedArtifact { output: dever_cli::libs::LockedArtifact {
source: None,
target: runtime.target, target: runtime.target,
path: format!("build/npm/{hash}/installation.tgz"), path: format!("build/npm/{hash}/installation.tgz"),
bytes: bytes.len() as u64, bytes: bytes.len() as u64,
@@ -171,7 +174,24 @@ fn locked_installation_fixture() -> (LockFile, Vec<u8>) {
lock.npm[0].build = Some(receipt.identity().unwrap()); lock.npm[0].build = Some(receipt.identity().unwrap());
lock.builds.push(receipt); lock.builds.push(receipt);
doctor(&lock).unwrap(); doctor(&lock).unwrap();
(lock, bytes) (lock, bytes, store)
}
#[test]
fn cached_npm_build_restore_does_not_load_build_tools_or_auxiliary_runtime() {
let (lock, bytes, store) = locked_installation_fixture();
store
.publish(&bytes, &lock.builds[0].output.target)
.unwrap();
dever_cli::libs::restore::restore_artifacts(
&lock,
&LocalRegistry(BTreeMap::new()),
&store,
&super::restore_tests::RuntimeOnly,
"linux-x86_64",
&sumdb_tests::verifier(),
)
.unwrap();
} }
#[test] #[test]
@@ -179,7 +199,7 @@ fn npm_expanded_outputs_are_pruned_only_when_no_raw_consumer_remains() {
use dever_cli::libs::{LockedWorker, prune_expanded_resources}; use dever_cli::libs::{LockedWorker, prune_expanded_resources};
use dever_core::native::EmbeddedResource; use dever_core::native::EmbeddedResource;
let (mut lock, bytes) = locked_installation_fixture(); let (mut lock, bytes, _) = locked_installation_fixture();
let roots = lock.npm[0].roots.clone(); let roots = lock.npm[0].roots.clone();
let other = vec!["npm:other@1.0.0".parse().unwrap()]; let other = vec!["npm:other@1.0.0".parse().unwrap()];
lock.npm = npm::retain(&lock, &[roots.clone(), other.clone()]).unwrap(); lock.npm = npm::retain(&lock, &[roots.clone(), other.clone()]).unwrap();
@@ -231,7 +251,7 @@ fn npm_expanded_outputs_are_pruned_only_when_no_raw_consumer_remains() {
#[test] #[test]
fn npm_native_admission_accounts_every_addon_and_prunes_unsupported_objects() { fn npm_native_admission_accounts_every_addon_and_prunes_unsupported_objects() {
let (lock, bytes) = locked_installation_fixture(); let (lock, bytes, _) = locked_installation_fixture();
let files = dever_cli::libs::build::npm_installation(&lock, &lock.npm[0], &bytes).unwrap(); let files = dever_cli::libs::build::npm_installation(&lock, &lock.npm[0], &bytes).unwrap();
assert!(files.iter().all(|(path, _, _)| !path.ends_with(".node"))); assert!(files.iter().all(|(path, _, _)| !path.ends_with(".node")));
let mut missing = lock.clone(); let mut missing = lock.clone();
@@ -262,7 +282,7 @@ fn npm_native_admission_accounts_every_addon_and_prunes_unsupported_objects() {
#[test] #[test]
fn npm_locked_output_projects_retained_roots_without_rerunning_hooks() { fn npm_locked_output_projects_retained_roots_without_rerunning_hooks() {
let (lock, bytes) = locked_installation_fixture(); let (lock, bytes, _) = locked_installation_fixture();
let project = TemporaryDirectory::new(); let project = TemporaryDirectory::new();
fs::create_dir(project.path().join("config")).unwrap(); fs::create_dir(project.path().join("config")).unwrap();
fs::write( fs::write(
@@ -468,6 +488,33 @@ fn npm_lifecycle_preserves_optional_failures_bins_and_offline_remove() {
.any(|edge| edge.omission == Some(npm::Omission::BuildFailure)) .any(|edge| edge.omission == Some(npm::Omission::BuildFailure))
); );
let receipt = &lock.builds[0]; let receipt = &lock.builds[0];
assert!(
receipt
.inputs
.libs
.iter()
.any(|lib| lib.spec.name == "broken")
);
assert!(
receipt
.inputs
.libs
.iter()
.any(|lib| lib.spec.name == "cycle")
);
assert!(receipt.inputs.npm[0].build.is_none());
let cold = FixtureArtifactStore::default();
let before = lock.encode().unwrap();
dever_cli::libs::restore::restore_artifacts(
&lock,
&super::restore_tests::archives_only(&lock, &transport),
&cold,
&inputs,
&inputs.settings.runtime.target,
&dever_cli::libs::sumdb::Verifier::official(),
)
.unwrap();
assert_eq!(lock.encode().unwrap(), before);
let bytes = store let bytes = store
.verify_exact( .verify_exact(
&receipt.output.sha256, &receipt.output.sha256,
@@ -0,0 +1,190 @@
use super::*;
use dever_cli::libs::build::{Inputs, pack};
use dever_cli::libs::restore::restore_artifacts;
pub(super) struct RuntimeOnly;
pub(super) fn archives_only(lock: &LockFile, registry: &LocalRegistry) -> LocalRegistry {
fn collect(
lock: &LockFile,
registry: &LocalRegistry,
files: &mut BTreeMap<(Ecosystem, String), Vec<u8>>,
) {
for artifact in lock
.libs
.iter()
.flat_map(|lib| &lib.artifacts)
.chain(lock.builds.iter().flat_map(|receipt| &receipt.sources))
{
if let Some(source) = &artifact.source {
let key = (source.ecosystem.clone(), source.locator.clone());
files.insert(key.clone(), registry.0[&key].clone());
}
}
for receipt in &lock.builds {
collect(&receipt.inputs, registry, files);
}
}
let mut files = BTreeMap::new();
collect(lock, registry, &mut files);
LocalRegistry(files)
}
impl Inputs for RuntimeOnly {
fn runtime(&self, ecosystem: Ecosystem) -> Result<(RegistryRuntime, Vec<u8>), String> {
Ok((registry_runtime(ecosystem), vec![]))
}
fn tools(&self, _: Ecosystem) -> Result<(pack::Descriptor, Vec<u8>), String> {
panic!("archive restoration must not load build extensions")
}
fn assets(&self) -> Result<Vec<(String, Vec<u8>)>, String> {
panic!("archive restoration must not execute a sandbox")
}
}
#[test]
fn cold_cache_restore_uses_only_locked_sources_and_never_registry_metadata() {
let mut registry = local_registry();
let original = FixtureArtifactStore::default();
let roots = ["pip:a@1.0.0", "npm:root@1.0.0", "go:example.com/a@1.0.0"]
.map(|spec| spec.parse().unwrap());
let lock = dever_cli::libs::LibResolver::resolve(&local_resolver(&registry, &original), &roots)
.unwrap();
let before = lock.encode().unwrap();
let locators = lock
.libs
.iter()
.flat_map(|lib| &lib.artifacts)
.map(|artifact| {
let source = artifact.source.as_ref().unwrap();
assert!(!source.filename.is_empty());
(source.ecosystem.clone(), source.locator.clone())
})
.collect::<std::collections::BTreeSet<_>>();
registry.0.retain(|key, _| locators.contains(key));
let cold = FixtureArtifactStore::default();
restore_artifacts(
&lock,
&registry,
&cold,
&RuntimeOnly,
"linux-x86_64",
&sumdb_tests::verifier(),
)
.unwrap();
assert_eq!(lock.encode().unwrap(), before);
assert_eq!(
verify_locked_artifacts(&lock, &cold, "linux-x86_64").unwrap(),
verify_locked_artifacts(&lock, &original, "linux-x86_64").unwrap()
);
// A warm restore must also succeed with no available registry at all.
restore_artifacts(
&lock,
&LocalRegistry(BTreeMap::new()),
&cold,
&RuntimeOnly,
"linux-x86_64",
&sumdb_tests::verifier(),
)
.unwrap();
}
#[test]
fn restored_archive_mismatch_fails_before_cache_publication() {
let mut registry = local_registry();
let lock = dever_cli::libs::LibResolver::resolve(
&local_resolver(&registry, &FixtureArtifactStore::default()),
&["pip:a@1.0.0".parse().unwrap()],
)
.unwrap();
let source = lock.libs[0].artifacts[0].source.as_ref().unwrap();
registry.0.insert(
(source.ecosystem.clone(), source.locator.clone()),
b"different bytes".to_vec(),
);
let cold = FixtureArtifactStore::default();
assert!(
restore_artifacts(
&lock,
&registry,
&cold,
&RuntimeOnly,
"linux-x86_64",
&sumdb_tests::verifier()
)
.unwrap_err()
.contains("byte/hash")
);
let artifact = &lock.libs[0].artifacts[0];
assert!(!cold.has(&artifact.sha256, &artifact.target).unwrap());
}
#[test]
fn corrupt_cache_is_an_error_and_never_falls_back_to_download() {
struct CorruptCache;
impl ArtifactStore for CorruptCache {
fn has(&self, _: &str, _: &str) -> Result<bool, String> {
unreachable!()
}
fn publish(&self, _: &[u8], _: &str) -> Result<String, String> {
panic!("corrupt cache cannot be replaced")
}
fn get(&self, _: &str, _: &str) -> Result<Option<Vec<u8>>, String> {
Ok(Some(b"corrupt".to_vec()))
}
}
let registry = local_registry();
let lock = dever_cli::libs::LibResolver::resolve(
&local_resolver(&registry, &FixtureArtifactStore::default()),
&["pip:a@1.0.0".parse().unwrap()],
)
.unwrap();
// No registry entries exist: a fallback would produce a distinct missing-path error.
assert!(
restore_artifacts(
&lock,
&LocalRegistry(BTreeMap::new()),
&CorruptCache,
&RuntimeOnly,
"linux-x86_64",
&sumdb_tests::verifier()
)
.unwrap_err()
.contains("byte/hash")
);
}
#[test]
fn lib_install_preserves_setting_and_lock_bytes_and_rejects_declaration_drift() {
let root = TemporaryDirectory::new();
fs::create_dir(root.path().join("config")).unwrap();
let setting = b"{\n \"lib\": [\"pip:fixture@1.0.0\"]\n}\n";
fs::write(root.path().join("config/setting.json"), setting).unwrap();
assert!(
dever_cli::libs::execute("install", root.path(), &[])
.unwrap_err()
.contains("existing dever.lock")
);
let lock = FixtureRegistry::builtin()
.resolve(&["pip:fixture@1.0.0".parse().unwrap()])
.unwrap();
lock.write_atomic(root.path()).unwrap();
let before = fs::read(root.path().join("dever.lock")).unwrap();
dever_cli::libs::execute("install", root.path(), &[]).unwrap();
assert_eq!(
fs::read(root.path().join("config/setting.json")).unwrap(),
setting
);
assert_eq!(fs::read(root.path().join("dever.lock")).unwrap(), before);
assert!(
dever_cli::libs::execute("install", root.path(), &["pip:fixture@1.0.0".into()])
.unwrap_err()
.contains("no Lib specs")
);
fs::write(root.path().join("config/setting.json"), b"{\"lib\":[]}").unwrap();
assert!(
dever_cli::libs::execute("install", root.path(), &[])
.unwrap_err()
.contains("declared closure")
);
assert_eq!(fs::read(root.path().join("dever.lock")).unwrap(), before);
}
+2
View File
@@ -262,6 +262,7 @@ fn fixture_for_target(
dependencies: vec![], dependencies: vec![],
runtime: runtime.clone(), runtime: runtime.clone(),
artifacts: vec![LockedArtifact { artifacts: vec![LockedArtifact {
source: None,
target: target.into(), target: target.into(),
path: path.clone(), path: path.clone(),
bytes: bytes.len() as u64, bytes: bytes.len() as u64,
@@ -794,6 +795,7 @@ fn run_native_python_fixture(fixture: &str) {
dependencies: vec![], dependencies: vec![],
runtime: runtime.clone(), runtime: runtime.clone(),
artifacts: vec![LockedArtifact { artifacts: vec![LockedArtifact {
source: None,
target: dever_cli::toolchain::platform_identity(), target: dever_cli::toolchain::platform_identity(),
path: probe_path.clone(), path: probe_path.clone(),
bytes: probe.len() as u64, bytes: probe.len() as u64,
@@ -358,6 +358,56 @@ fn install_release(layout: &Layout, version: &Version, key: &Ed25519KeyPair) {
MachineManager::new(layout.clone()) MachineManager::new(layout.clone())
.install_requested(version.as_str()) .install_requested(version.as_str())
.unwrap(); .unwrap();
// These opt-in whole-release fixtures explicitly prepare every declared resource.
// Product installation only installs the base; each preparation still exercises
// the ordinary signed extraction and immutable extension publication owner.
struct PreparedInputs(PathBuf);
impl dever_cli::toolchain::ExtensionSource for PreparedInputs {
fn open(
&self,
_version: &Version,
id: &dever_cli::toolchain::ExtensionId,
) -> Result<Box<dyn std::io::Read>, String> {
let catalog: dever_cli::toolchain::ReleaseManifest = serde_json::from_slice(
&fs::read(self.0.join("manifest.json")).map_err(|error| error.to_string())?,
)
.map_err(|error| error.to_string())?;
let extension = catalog
.extensions
.iter()
.find(|extension| extension.id() == *id)
.ok_or("fixture extension is undeclared")?;
let mut encoder = zstd::stream::write::Encoder::new(Vec::new(), 1)
.map_err(|error| error.to_string())?;
encoder
.include_checksum(true)
.map_err(|error| error.to_string())?;
let mut archive = tar::Builder::new(encoder);
for artifact in &extension.artifacts {
let mut header = tar::Header::new_ustar();
header.set_size(artifact.bytes);
header.set_mode(0o644);
header.set_cksum();
let file = fs::File::open(self.0.join(&artifact.path))
.map_err(|error| error.to_string())?;
archive
.append_data(&mut header, &artifact.path, file)
.map_err(|error| error.to_string())?;
}
let bytes = archive
.into_inner()
.map_err(|error| error.to_string())?
.finish()
.map_err(|error| error.to_string())?;
Ok(Box::new(std::io::Cursor::new(bytes)))
}
}
let resources = dever_cli::toolchain::SignedResources::load(layout, version).unwrap();
let source = PreparedInputs(layout.downloads().join(version.as_str()));
for extension in &resources.manifest().extensions {
dever_cli::toolchain::prepare_extension_with(layout, version, &extension.id(), &source)
.unwrap();
}
} }
fn archive_report(workspace: &Path) -> Value { fn archive_report(workspace: &Path) -> Value {
@@ -3,7 +3,7 @@ use super::*;
use dever_cli::libs::{ use dever_cli::libs::{
self, LibSpec, LockFile, LockedArtifact, LockedLib, LockedWorker, RegistryRuntime, self, LibSpec, LockFile, LockedArtifact, LockedLib, LockedWorker, RegistryRuntime,
}; };
use dever_cli::toolchain::{Artifact, ReleaseManifest}; use dever_cli::toolchain::{Artifact, Extension, ExtensionKind, ReleaseManifest};
use dever_core::hir::ExternalWorkerContract; use dever_core::hir::ExternalWorkerContract;
use sha2::{Digest, Sha256}; use sha2::{Digest, Sha256};
use std::collections::BTreeMap; use std::collections::BTreeMap;
@@ -68,6 +68,7 @@ fn sign_target_runtimes(
serde_json::to_vec(&json!({"format":"dever-registry-runtime-v1","runtime":runtime})) serde_json::to_vec(&json!({"format":"dever-registry-runtime-v1","runtime":runtime}))
.unwrap(), .unwrap(),
); );
let mut artifacts = Vec::new();
for leaf in ["manifest.json", "runtime.pack"] { for leaf in ["manifest.json", "runtime.pack"] {
let path = format!("{prefix}/{leaf}"); let path = format!("{prefix}/{leaf}");
assert!( assert!(
@@ -76,17 +77,21 @@ fn sign_target_runtimes(
.iter() .iter()
.any(|artifact| artifact.path == path) .any(|artifact| artifact.path == path)
); );
manifest.artifacts.push(Artifact { artifacts.push(Artifact {
bytes: fs::metadata(release.join(&path)).unwrap().len(), bytes: fs::metadata(release.join(&path)).unwrap().len(),
sha256: sha256_file(&release.join(&path)).unwrap(), sha256: sha256_file(&release.join(&path)).unwrap(),
path, path,
}); });
} }
manifest.extensions.push(Extension {
kind: ExtensionKind::Runtime(ecosystem.parse().unwrap()),
target: TARGET,
artifacts,
});
runtimes.insert(ecosystem.to_owned(), runtime); runtimes.insert(ecosystem.to_owned(), runtime);
} }
manifest manifest.extensions.sort_by_key(Extension::id);
.artifacts dever_cli::toolchain::validate_catalog(&manifest).unwrap();
.sort_by(|left, right| left.path.cmp(&right.path));
let encoded = serde_json::to_vec(&manifest).unwrap(); let encoded = serde_json::to_vec(&manifest).unwrap();
fs::write(release.join("manifest.json"), &encoded).unwrap(); fs::write(release.join("manifest.json"), &encoded).unwrap();
fs::write( fs::write(
@@ -167,6 +172,7 @@ fn pure_package_lock(
.unwrap(); .unwrap();
let sha256 = digest(&bytes); let sha256 = digest(&bytes);
let artifact = LockedArtifact { let artifact = LockedArtifact {
source: None,
target: TARGET.platform().into(), target: TARGET.platform().into(),
path: format!( path: format!(
"lib/{}/{}/{}/archive.{extension}", "lib/{}/{}/{}/archive.{extension}",
@@ -121,8 +121,9 @@ fn sign_build_runtimes(
for suffix in ["runtime.pack", "manifest.json"] { for suffix in ["runtime.pack", "manifest.json"] {
let name = format!("{prefix}/{suffix}"); let name = format!("{prefix}/{suffix}");
let artifact = manifest let artifact = manifest
.artifacts .extensions
.iter_mut() .iter_mut()
.flat_map(|extension| &mut extension.artifacts)
.find(|artifact| artifact.path == name) .find(|artifact| artifact.path == name)
.unwrap(); .unwrap();
artifact.bytes = fs::metadata(release.join(&name)).unwrap().len(); artifact.bytes = fs::metadata(release.join(&name)).unwrap().len();
@@ -311,8 +312,9 @@ fn signed_ecosystems_build_and_run_without_system_language_installations() {
for ecosystem in ECOSYSTEMS { for ecosystem in ECOSYSTEMS {
let path = format!("runtime/{ecosystem}/linux-x86_64/runtime.pack"); let path = format!("runtime/{ecosystem}/linux-x86_64/runtime.pack");
let artifact = signed let artifact = signed
.artifacts .extensions
.iter() .iter()
.flat_map(|extension| &extension.artifacts)
.find(|artifact| artifact.path == path) .find(|artifact| artifact.path == path)
.unwrap(); .unwrap();
runtimes.insert( runtimes.insert(
@@ -148,8 +148,9 @@ fn ecosystem_archives_are_deterministic_and_signed_with_computed_identities() {
); );
for path in [&metadata_path, &pack_path] { for path in [&metadata_path, &pack_path] {
let signed = manifest let signed = manifest
.artifacts .extensions
.iter() .iter()
.flat_map(|extension| &extension.artifacts)
.find(|artifact| &artifact.path == path) .find(|artifact| &artifact.path == path)
.unwrap(); .unwrap();
assert_eq!(signed.sha256, sha256_file(&first.join(path)).unwrap()); assert_eq!(signed.sha256, sha256_file(&first.join(path)).unwrap());
@@ -286,8 +287,9 @@ fn go_cross_pack_binds_host_tools_and_arm_standard_library_separately() {
let manifest = result.unwrap(); let manifest = result.unwrap();
assert!( assert!(
manifest manifest
.artifacts .extensions
.iter() .iter()
.flat_map(|extension| &extension.artifacts)
.any(|file| file.path == "runtime/go/linux-aarch64/runtime.pack") .any(|file| file.path == "runtime/go/linux-aarch64/runtime.pack")
); );
} }
@@ -295,6 +297,7 @@ fn go_cross_pack_binds_host_tools_and_arm_standard_library_separately() {
} }
#[test] #[test]
#[cfg(target_arch = "x86_64")]
fn sandbox_cross_assets_preserve_loader_mode_and_validate_deployment_target() { fn sandbox_cross_assets_preserve_loader_mode_and_validate_deployment_target() {
let mut files = vec![ let mut files = vec![
("bin/bwrap".into(), native_elf::static_executable()), ("bin/bwrap".into(), native_elf::static_executable()),
@@ -312,7 +315,7 @@ fn sandbox_cross_assets_preserve_loader_mode_and_validate_deployment_target() {
bytes[18..20].copy_from_slice(&183u16.to_le_bytes()); bytes[18..20].copy_from_slice(&183u16.to_le_bytes());
} }
dever_sandbox::validate_assets_for_target(&files, "linux-aarch64").unwrap(); dever_sandbox::validate_assets_for_target(&files, "linux-aarch64").unwrap();
let mut author = Author::new(); let mut author = super::targets::cross_author();
let inputs = files let inputs = files
.iter() .iter()
.map(|(path, bytes)| { .map(|(path, bytes)| {
@@ -3,7 +3,7 @@ use dever_cli::toolchain::runtime_pack::RuntimePack;
use dever_runtime::config::RuntimeProfile; use dever_runtime::config::RuntimeProfile;
#[cfg(target_arch = "x86_64")] #[cfg(target_arch = "x86_64")]
fn cross_author() -> Author { pub(super) fn cross_author() -> Author {
let mut author = Author::new(); let mut author = Author::new();
let target = BuildTarget::LinuxAarch64; let target = BuildTarget::LinuxAarch64;
let input = |name: &str, bytes: Vec<u8>| { let input = |name: &str, bytes: Vec<u8>| {
@@ -80,6 +80,12 @@ fn host_core_packages_both_native_targets_with_separate_profiles_and_signed_clos
manifest manifest
.artifacts .artifacts
.iter() .iter()
.chain(
manifest
.extensions
.iter()
.flat_map(|extension| &extension.artifacts)
)
.any(|artifact| artifact.path == input.path .any(|artifact| artifact.path == input.path
&& artifact.sha256 == input.sha256 && artifact.sha256 == input.sha256
&& artifact.bytes == input.bytes) && artifact.bytes == input.bytes)
+23
View File
@@ -245,6 +245,29 @@ fn package_add_resolves_transitive_versions_and_feeds_the_common_lib_lock() {
assert!(String::from_utf8_lossy(&files["module/catalog/value/app.dever"]).contains("two")); assert!(String::from_utf8_lossy(&files["module/catalog/value/app.dever"]).contains("two"));
assert!(execute_with("doctor", root.path(), &[], &registry, &store).is_ok()); assert!(execute_with("doctor", root.path(), &[], &registry, &store).is_ok());
let lock_before = fs::read(root.path().join("dever.lock")).unwrap();
let setting_before = fs::read(root.path().join("config/setting.json")).unwrap();
let archives_only = Registry(
registry
.0
.iter()
.filter(|(path, _)| path.ends_with(".zip"))
.map(|(path, bytes)| (path.clone(), bytes.clone()))
.collect(),
);
let restored = FixtureArtifactStore::default();
dever_cli::packages::restore_locked_with(root.path(), &lock, &archives_only, &restored)
.unwrap();
assert_eq!(owned_files_with(root.path(), &restored).unwrap(), files);
assert_eq!(
fs::read(root.path().join("dever.lock")).unwrap(),
lock_before
);
assert_eq!(
fs::read(root.path().join("config/setting.json")).unwrap(),
setting_before
);
let mut damaged = lock.clone(); let mut damaged = lock.clone();
damaged.packages[0].manifest_sha256 = "0".repeat(64); damaged.packages[0].manifest_sha256 = "0".repeat(64);
damaged.write_atomic(root.path()).unwrap(); damaged.write_atomic(root.path()).unwrap();
+129 -5
View File
@@ -140,9 +140,9 @@ fn official_asset_names_are_versioned_and_platform_specific() {
) )
); );
assert_eq!( assert_eq!(
source.asset("1.2.3", "tar.gz"), source.asset("1.2.3", "tar.zst"),
format!( format!(
"{OFFICIAL_RELEASES}/download/v1.2.3/dever-{}.tar.gz", "{OFFICIAL_RELEASES}/download/v1.2.3/dever-{}.tar.zst",
platform_identity() platform_identity()
) )
); );
@@ -219,9 +219,133 @@ fn compressed_release(core: &[u8]) -> Vec<u8> {
("dever-core", core), ("dever-core", core),
("skills/dever-language/SKILL.md", b"skill"), ("skills/dever-language/SKILL.md", b"skill"),
]); ]);
let mut gzip = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::fast()); compressed(&bytes)
gzip.write_all(&bytes).unwrap(); }
gzip.finish().unwrap()
fn compressed(bytes: &[u8]) -> Vec<u8> {
let mut encoder = zstd::stream::write::Encoder::new(Vec::new(), 1).unwrap();
encoder.include_checksum(true).unwrap();
encoder.write_all(bytes).unwrap();
encoder.finish().unwrap()
}
#[test]
fn zstandard_frames_require_checksum_bounded_window_and_exact_eof() {
let tar = archive(&[("dever-core", b"core")]);
let good = compressed(&tar);
let root = Download::new(&std::env::temp_dir()).unwrap();
super::super::release_archive::extract(good.as_slice(), &root.0, &manifest().artifacts)
.unwrap();
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
assert_eq!(
fs::metadata(root.0.join("dever-core"))
.unwrap()
.permissions()
.mode()
& 0o777,
0o755,
"signed executable roles retain execution after extraction"
);
}
let mut bad_checksum = good.clone();
*bad_checksum.last_mut().unwrap() ^= 1;
let mut concatenated = good.clone();
concatenated.extend(&good);
let mut garbage = good.clone();
garbage.push(0);
let oversized_window = vec![
0x28,
0xb5,
0x2f,
0xfd,
4,
(28 - 10) << 3,
1,
0,
0,
0,
0,
0,
0,
];
for bytes in [
bad_checksum,
concatenated,
garbage,
oversized_window,
good[..good.len() - 1].to_vec(),
zstd::stream::encode_all(tar.as_slice(), 1).unwrap(),
] {
let root = Download::new(&std::env::temp_dir()).unwrap();
assert!(
super::super::release_archive::extract(
bytes.as_slice(),
&root.0,
&manifest().artifacts
)
.is_err()
);
}
}
#[test]
fn tar_trailers_and_same_sized_corruption_are_rejected() {
let tar = archive(&[("dever-core", b"core")]);
let mut nonzero_tail = tar.clone();
nonzero_tail.push(1);
let mut nonzero_padding = tar.clone();
nonzero_padding[516] = 1;
for bytes in [
nonzero_tail,
nonzero_padding,
tar[..tar.len() - 512].to_vec(),
archive(&[("dever-core", b"evil")]),
] {
let root = Download::new(&std::env::temp_dir()).unwrap();
assert!(
super::super::release_archive::extract(
compressed(&bytes).as_slice(),
&root.0,
&manifest().artifacts
)
.is_err()
);
}
}
#[test]
fn signed_extensions_cannot_duplicate_or_escape_their_typed_namespace() {
use super::super::{BuildTarget, Extension, ExtensionKind};
use crate::libs::Ecosystem;
let mut catalog = manifest();
catalog.extensions.push(Extension {
kind: ExtensionKind::Runtime(Ecosystem::Pip),
target: BuildTarget::LinuxX86_64,
artifacts: vec![Artifact {
path: "runtime/pip/linux-x86_64/runtime.pack".into(),
bytes: 4,
sha256: catalog.artifacts[0].sha256.clone(),
}],
});
validate_catalog(&catalog).unwrap();
assert_eq!(
catalog.extensions[0].id().asset_suffix(),
"ext-runtime-pip-linux-x86_64.tar.zst"
);
let mut duplicate = catalog.clone();
duplicate.extensions.push(duplicate.extensions[0].clone());
assert!(validate_catalog(&duplicate).is_err());
for path in [
"dever-core",
"runtime/npm/linux-x86_64/runtime.pack",
"runtime/pip/linux-aarch64/runtime.pack",
"runtime/pip/linux-x86_64/../escape",
] {
catalog.extensions[0].artifacts[0].path = path.into();
assert!(validate_catalog(&catalog).is_err(), "{path}");
}
} }
#[test] #[test]
+4
View File
@@ -18,6 +18,10 @@ use sha2::{Digest, Sha256};
static NEXT_TEMPORARY: AtomicU64 = AtomicU64::new(0); static NEXT_TEMPORARY: AtomicU64 = AtomicU64::new(0);
#[cfg(unix)]
#[path = "shared_toolchain_extensions.rs"]
mod extensions;
struct TemporaryDirectory(PathBuf); struct TemporaryDirectory(PathBuf);
impl TemporaryDirectory { impl TemporaryDirectory {
@@ -0,0 +1,236 @@
use super::*;
use dever_cli::libs::Ecosystem;
use dever_cli::toolchain::{
BuildTarget, Extension, ExtensionId, ExtensionKind, ExtensionSource, SignedResources,
artifact_get_optional, prepare_extension_with,
};
use std::sync::atomic::AtomicUsize;
fn id(ecosystem: Ecosystem) -> ExtensionId {
ExtensionId {
kind: ExtensionKind::Runtime(ecosystem),
target: BuildTarget::host().unwrap(),
}
}
fn logical_path(id: &ExtensionId) -> String {
let ExtensionKind::Runtime(ecosystem) = &id.kind else {
panic!("runtime fixture")
};
format!(
"runtime/{}/{}/runtime.pack",
ecosystem.as_str(),
id.target.platform()
)
}
struct Source {
requests: AtomicUsize,
fail: bool,
}
impl Source {
fn new(fail: bool) -> Self {
Self {
requests: AtomicUsize::new(0),
fail,
}
}
}
impl ExtensionSource for Source {
fn open(&self, version: &Version, extension: &ExtensionId) -> Result<Box<dyn Read>, String> {
self.requests.fetch_add(1, Ordering::Relaxed);
if self.fail {
return Err("fixture extension download failed".into());
}
let mut tar = tar::Builder::new(Vec::new());
let mut header = tar::Header::new_ustar();
header.set_size(version.as_str().len() as u64);
header.set_mode(0o644);
header.set_cksum();
tar.append_data(
&mut header,
logical_path(extension),
version.as_str().as_bytes(),
)
.unwrap();
let mut encoder = zstd::stream::write::Encoder::new(Vec::new(), 1).unwrap();
encoder.include_checksum(true).unwrap();
encoder.write_all(&tar.into_inner().unwrap()).unwrap();
Ok(Box::new(std::io::Cursor::new(encoder.finish().unwrap())))
}
}
fn release(layout: &Layout, key: &Ed25519KeyPair, version: &str) {
write_release(
layout,
key,
version,
&layout.root().join("dispatch"),
true,
true,
);
let directory = layout.downloads().join(version);
let path = directory.join("manifest.json");
let mut manifest: ReleaseManifest = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap();
for ecosystem in [Ecosystem::Pip, Ecosystem::Npm] {
let id = id(ecosystem);
manifest.extensions.push(Extension {
kind: id.kind.clone(),
target: id.target,
artifacts: vec![Artifact {
path: logical_path(&id),
bytes: version.len() as u64,
sha256: hex(&Sha256::digest(version.as_bytes())),
}],
});
}
let bytes = serde_json::to_vec(&manifest).unwrap();
fs::write(path, &bytes).unwrap();
fs::write(
directory.join("manifest.sig"),
hex(key.sign(&bytes).as_ref()),
)
.unwrap();
}
fn machine() -> (TemporaryDirectory, Layout, MachineManager, Ed25519KeyPair) {
let root = TemporaryDirectory::new();
let layout = Layout::new(root.path().join("machine"));
layout.initialize().unwrap();
let key = signing_key();
fs::write(
layout.state().join("trusted-release-key"),
hex(key.public_key().as_ref()),
)
.unwrap();
let manager = MachineManager::new(layout.clone());
(root, layout, manager, key)
}
#[test]
fn signed_extension_preparation_is_shared_immutable_and_concurrent() {
let (_root, layout, manager, key) = machine();
release(&layout, &key, "1.0.0");
let version = manager.install_requested("1.0.0").unwrap();
let extension = id(Ecosystem::Pip);
let logical = logical_path(&extension);
let resources = SignedResources::load(&layout, &version).unwrap();
assert!(
resources
.read(&logical)
.unwrap_err()
.contains("dever lib install")
);
let source = Source::new(false);
std::thread::scope(|scope| {
for _ in 0..2 {
scope.spawn(|| prepare_extension_with(&layout, &version, &extension, &source).unwrap());
}
});
assert_eq!(source.requests.load(Ordering::Relaxed), 1);
assert_eq!(resources.read(&logical).unwrap(), b"1.0.0");
let path = resources.path(&logical).unwrap();
assert!(path.starts_with(layout.extensions()));
assert!(!layout.versions().join("1.0.0").join(&logical).exists());
fs::write(&path, b"other").unwrap();
assert!(resources.read(&logical).unwrap_err().contains("integrity"));
assert!(
prepare_extension_with(&layout, &version, &extension, &source)
.unwrap_err()
.contains("integrity")
);
assert_eq!(source.requests.load(Ordering::Relaxed), 1);
}
#[test]
fn update_prepares_only_installed_extensions_and_preserves_active_on_failure() {
let (_root, layout, manager, key) = machine();
for version in ["1.0.0", "2.0.0"] {
release(&layout, &key, version);
}
let old = manager.install_requested("1.0.0").unwrap();
let extension = id(Ecosystem::Pip);
prepare_extension_with(&layout, &old, &extension, &Source::new(false)).unwrap();
fs::write(layout.downloads().join("latest"), "2.0.0").unwrap();
let journal = fs::read(layout.state().join("active-version")).unwrap();
assert!(
manager
.update_with(&Source::new(true))
.unwrap_err()
.contains("download failed")
);
assert_eq!(
fs::read(layout.state().join("active-version")).unwrap(),
journal
);
assert_eq!(
SignedResources::load(&layout, &old)
.unwrap()
.read(&logical_path(&extension))
.unwrap(),
b"1.0.0"
);
let source = Source::new(false);
let new = manager.update_with(&source).unwrap();
assert_eq!(new.as_str(), "2.0.0");
assert_eq!(source.requests.load(Ordering::Relaxed), 1);
let resources = SignedResources::load(&layout, &new).unwrap();
assert_eq!(resources.read(&logical_path(&extension)).unwrap(), b"2.0.0");
assert!(
resources
.read(&logical_path(&id(Ecosystem::Npm)))
.unwrap_err()
.contains("not installed")
);
assert_eq!(manager.active_version().unwrap(), Some(new));
}
#[test]
fn optional_artifact_ipc_distinguishes_absence_corruption_and_service_failure() {
let (_root, layout, _manager, _key) = machine();
let mut daemon = TestDaemon::start(&layout);
let digest = hex(&Sha256::digest(b"payload"));
assert!(
artifact_get_optional(&layout, &digest, 7)
.unwrap()
.is_none()
);
artifact_put(&layout, b"payload").unwrap();
assert_eq!(
artifact_get_optional(&layout, &digest, 7).unwrap().unwrap(),
b"payload"
);
fs::write(
layout.native_cache().join("artifacts").join(&digest),
b"changed",
)
.unwrap();
assert!(
artifact_get_optional(&layout, &digest, 7)
.unwrap_err()
.contains("SHA-256")
);
daemon.stop();
assert!(artifact_get_optional(&layout, &digest, 7).is_err());
}
#[test]
fn extension_ipc_rejects_caller_paths_and_invalid_typed_identity() {
let (_root, layout, _manager, _key) = machine();
let _daemon = TestDaemon::start(&layout);
let token = fs::read_to_string(layout.service_token()).unwrap();
for selection in [
serde_json::json!({"version":"1.0.0","extension":{"kind":{"type":"runtime","ecosystem":"pip"},"target":"linux-x86_64"},"url":"http://localhost/"}),
serde_json::json!({"version":"../../outside","extension":{"kind":{"type":"target"},"target":"linux-aarch64"}}),
serde_json::json!({"version":"1.0.0","extension":{"kind":{"type":"shell"},"target":"linux-x86_64"}}),
] {
let result = raw_operation(
&layout,
token.trim(),
serde_json::json!({"ensure_extension":selection}),
);
assert_eq!(result["ok"], false, "{result}");
}
}
+91
View File
@@ -0,0 +1,91 @@
//! Opt-in actual-asset acceptance: only the download transport is local.
use std::fs::{self, File};
use std::io::Read;
use std::path::{Path, PathBuf};
use dever_cli::toolchain::{
BuildTarget, ExtensionId, ExtensionSource, Layout, MachineManager, ReleaseManifest,
SignedResources, Version, prepare_extension_with, validate_catalog,
};
use ring::signature::Ed25519KeyPair;
struct LocalAssets(PathBuf);
impl ExtensionSource for LocalAssets {
fn open(&self, _: &Version, extension: &ExtensionId) -> Result<Box<dyn Read>, String> {
let name = format!(
"dever-{}.{}",
BuildTarget::host()?.platform(),
extension.asset_suffix()
);
File::open(self.0.join(name))
.map(|file| Box::new(file) as Box<dyn Read>)
.map_err(|error| format!("cannot open actual extension asset: {error}"))
}
}
fn run() -> Result<(), String> {
let arguments: Vec<_> = std::env::args().skip(1).collect();
if let [operation, manifest, key] = arguments.as_slice()
&& operation == "--sign-catalog"
{
let bytes = fs::read(manifest).map_err(|error| error.to_string())?;
let catalog: ReleaseManifest =
serde_json::from_slice(&bytes).map_err(|error| error.to_string())?;
validate_catalog(&catalog)?;
let key = fs::read(key).map_err(|error| error.to_string())?;
let pair = Ed25519KeyPair::from_pkcs8(&key).map_err(|error| error.to_string())?;
let signature: String = pair
.sign(&bytes)
.as_ref()
.iter()
.map(|byte| format!("{byte:02x}"))
.collect();
return fs::write(
Path::new(manifest).with_file_name("manifest.sig"),
signature,
)
.map_err(|error| error.to_string());
}
let [machine, assets, version, names @ ..] = arguments.as_slice() else {
return Err(
"usage: prepare-extensions <machine> <assets> <version> <extension-name>...".into(),
);
};
if names.is_empty() {
return Err("at least one signed extension must be selected".into());
}
let layout = Layout::new(machine);
let version = Version::parse(version)?;
let source = LocalAssets(PathBuf::from(assets));
if names == ["--update"] {
let installed = MachineManager::new(layout).update_with(&source)?;
if installed != version {
return Err(format!(
"expected update to {version}, received {installed}"
));
}
println!("updated {installed}");
return Ok(());
}
let resources = SignedResources::load(&layout, &version)?;
for name in names {
let extension = resources
.manifest()
.extensions
.iter()
.find(|extension| extension.id().name() == *name)
.ok_or_else(|| format!("signed catalog does not declare extension '{name}'"))?;
prepare_extension_with(&layout, &version, &extension.id(), &source)?;
println!("prepared {}", extension.id().name());
}
Ok(())
}
fn main() {
if let Err(error) = run() {
eprintln!("prepare-extensions: {error}");
std::process::exit(1);
}
}
+91 -47
View File
@@ -2,7 +2,6 @@
import argparse import argparse
import contextlib import contextlib
import gzip
import hashlib import hashlib
import importlib.util import importlib.util
import io import io
@@ -40,6 +39,7 @@ class InstallerTests(unittest.TestCase):
self.payloads = { self.payloads = {
"dever-core": b"fixture core, never executed\n", "dever-core": b"fixture core, never executed\n",
"bootstrap/dever": b"fixture launcher, never executed\n", "bootstrap/dever": b"fixture launcher, never executed\n",
"bootstrap/lib/libfixture.so": b"fixture private closure\n",
"skills/dever-language/SKILL.md": b"fixture skill\n", "skills/dever-language/SKILL.md": b"fixture skill\n",
} }
self.catalog = {} self.catalog = {}
@@ -52,8 +52,9 @@ class InstallerTests(unittest.TestCase):
"sha256": hashlib.sha256(content).hexdigest(), "sha256": hashlib.sha256(content).hexdigest(),
} }
manifest = { manifest = {
"format": "dever-release-v1", "platform": self.platform, "format": "dever-release-v2", "platform": self.platform,
"version": "1.2.3", "artifacts": list(self.catalog.values()), "version": "1.2.3", "artifacts": list(self.catalog.values()),
"extensions": [],
} }
(self.release / "manifest.json").write_text(json.dumps(manifest)) (self.release / "manifest.json").write_text(json.dumps(manifest))
self.private = self.root / "test-key.pem" self.private = self.root / "test-key.pem"
@@ -82,24 +83,14 @@ class InstallerTests(unittest.TestCase):
return installer.authenticate(self.release, key or self.public, self.scratch, return installer.authenticate(self.release, key or self.public, self.scratch,
self.platform, "1.2.3") self.platform, "1.2.3")
def archive(self, entries, trailer=None): def create_assets(self, output):
stream = io.BytesIO() return assets.create(self.release, output, Path("/usr/bin/zstd"))
with tarfile.open(fileobj=stream, mode="w", format=tarfile.USTAR_FORMAT) as archive:
for name, content, kind in entries:
member = tarfile.TarInfo(name)
member.type = kind
member.size = len(content) if kind == tarfile.REGTYPE else 0
if kind == tarfile.SYMTYPE:
member.linkname = "../../outside"
archive.addfile(member, io.BytesIO(content) if member.size else None)
contents = stream.getvalue()
if trailer is not None:
contents += trailer
return gzip.compress(contents)
def extract(self, compressed, destination): def archive_payloads(self, path):
with mock.patch.object(installer, "download", return_value=io.BytesIO(compressed)): contents = subprocess.run(["/usr/bin/zstd", "-q", "-d", "-c", str(path)],
installer.payload(destination, None, "1.2.3", self.platform, self.catalog) env={}, capture_output=True, check=True, timeout=15).stdout
with tarfile.open(fileobj=io.BytesIO(contents)) as archive:
return {member.name: archive.extractfile(member).read() for member in archive}
def test_signature_accepts_independent_key_and_rejects_wrong_key_or_changed_manifest(self): def test_signature_accepts_independent_key_and_rejects_wrong_key_or_changed_manifest(self):
version, catalog, key = self.authenticate() version, catalog, key = self.authenticate()
@@ -119,19 +110,47 @@ class InstallerTests(unittest.TestCase):
first = self.root / "first" first = self.root / "first"
second = self.root / "second" second = self.root / "second"
(self.release / "unlisted").write_text("not published") (self.release / "unlisted").write_text("not published")
names = assets.create(self.release, first) names = self.create_assets(first)
self.assertEqual(names, assets.create(self.release, second)) self.assertEqual(names, self.create_assets(second))
self.assertEqual(len(names), 3) self.assertEqual(len(names), 5)
for name in names: for name in names:
self.assertEqual((first / name).read_bytes(), (second / name).read_bytes()) self.assertEqual((first / name).read_bytes(), (second / name).read_bytes())
extracted = self.directory("extracted") actual = self.archive_payloads(first / f"dever-{self.platform}.tar.zst")
self.extract((first / f"dever-{self.platform}.tar.gz").read_bytes(), extracted)
actual = {str(path.relative_to(extracted)): path.read_bytes()
for path in extracted.rglob("*") if path.is_file()}
self.assertEqual(actual, self.payloads) self.assertEqual(actual, self.payloads)
self.assertEqual((extracted / "bootstrap/dever").stat().st_mode & 0o777, 0o755) for name in ("bootstrap/dever", "bootstrap/lib/libfixture.so"):
blob = first / f"dever-{self.platform}.blob-{self.catalog[name]['sha256']}"
self.assertEqual(blob.read_bytes(), self.payloads[name])
with self.assertRaisesRegex(ValueError, "already exists"): with self.assertRaisesRegex(ValueError, "already exists"):
assets.create(self.release, first) self.create_assets(first)
def test_extension_assets_are_separate_and_cannot_overlap_base(self):
manifest_path = self.release / "manifest.json"
manifest = json.loads(manifest_path.read_text())
name = f"runtime/pip/{self.platform}/runtime.pack"
path = self.release / name
path.parent.mkdir(parents=True)
contents = b"optional Python bytes"
path.write_bytes(contents)
entry = {"path": name, "bytes": len(contents), "sha256": hashlib.sha256(contents).hexdigest()}
manifest["extensions"] = [{"kind": {"type": "runtime", "ecosystem": "pip"},
"target": self.platform, "artifacts": [entry]}]
manifest_path.write_text(json.dumps(manifest))
self.sign_manifest()
self.authenticate()
output = self.root / "split"
self.create_assets(output)
self.assertEqual(self.archive_payloads(output / f"dever-{self.platform}.tar.zst"), self.payloads)
extension = output / f"dever-{self.platform}.ext-runtime-pip-{self.platform}.tar.zst"
self.assertEqual(self.archive_payloads(extension), {name: contents})
manifest["extensions"][0]["artifacts"].append(self.catalog["dever-core"])
manifest_path.write_text(json.dumps(manifest))
with self.assertRaisesRegex(ValueError, "resource namespace"):
self.create_assets(self.root / "overlap")
manifest["extensions"] = [{"kind": {"type": "target"}, "target": self.platform,
"artifacts": [self.catalog["dever-core"]]}]
manifest_path.write_text(json.dumps(manifest))
with self.assertRaisesRegex(ValueError, "base package"):
self.create_assets(self.root / "host-target")
def test_signed_release_without_matching_skill_is_rejected(self): def test_signed_release_without_matching_skill_is_rejected(self):
path = self.release / "manifest.json" path = self.release / "manifest.json"
@@ -179,7 +198,7 @@ class InstallerTests(unittest.TestCase):
with mock.patch.object(assets, "publish", side_effect=publish), \ with mock.patch.object(assets, "publish", side_effect=publish), \
self.assertRaises(OSError): self.assertRaises(OSError):
assets.create(self.release, output) self.create_assets(output)
self.assertTrue(output.is_dir()) self.assertTrue(output.is_dir())
if populated: if populated:
self.assertEqual((output / "user-file").read_bytes(), b"concurrent user bytes") self.assertEqual((output / "user-file").read_bytes(), b"concurrent user bytes")
@@ -187,23 +206,48 @@ class InstallerTests(unittest.TestCase):
self.assertEqual(list(output.iterdir()), []) self.assertEqual(list(output.iterdir()), [])
self.assertEqual(list(self.root.glob(".dever-release-assets-*")), []) self.assertEqual(list(self.root.glob(".dever-release-assets-*")), [])
def test_archive_rejects_unsigned_duplicate_symlink_wrong_size_missing_and_bad_tail(self): def test_network_payload_executes_only_verified_helper_with_private_libraries(self):
normal = [(name, content, tarfile.REGTYPE) for name, content in self.payloads.items()] assets_path = self.root / "assets"
cases = { self.create_assets(assets_path)
"unsigned": normal + [("outside", b"extra", tarfile.REGTYPE)], calls = []
"duplicate": normal + [normal[0]],
"symlink": [(normal[0][0], b"", tarfile.SYMTYPE)] + normal[1:], def download(address):
"size": [(normal[0][0], b"short", tarfile.REGTYPE)] + normal[1:], return (assets_path / address.rsplit("/", 1)[-1]).open("rb")
"missing": normal[:-1],
} def extract(command, **kwargs):
for name, entries in cases.items(): self.assertEqual(command[1], "--dever-extract")
with self.subTest(name=name), self.assertRaises(ValueError): launcher = Path(command[0])
self.extract(self.archive(entries), self.directory(name)) self.assertEqual(launcher.read_bytes(), self.payloads["bootstrap/dever"])
with self.assertRaisesRegex(ValueError, "尾部"): self.assertEqual((launcher.parent / "lib/libfixture.so").read_bytes(),
self.extract(self.archive(normal, b"hidden"), self.directory("tail")) self.payloads["bootstrap/lib/libfixture.so"])
corrupted = [(normal[0][0], b"x" * len(normal[0][1]), tarfile.REGTYPE)] + normal[1:] self.assertEqual(launcher.stat().st_mode & 0o777, 0o755)
with self.assertRaisesRegex(ValueError, "摘要"): configuration = json.loads(Path(command[2]).read_text())
self.extract(self.archive(corrupted), self.directory("digest")) self.assertEqual(configuration["trusted_key"], str(self.public))
self.assertEqual(kwargs["env"], {})
self.assertTrue(Path(configuration["archive"]).is_file())
calls.append(command)
with mock.patch.object(installer, "download", side_effect=download), \
mock.patch.object(installer.subprocess, "run", side_effect=extract):
installer.payload(self.directory("network"), None, "1.2.3", self.platform,
self.catalog, self.public)
self.assertEqual(len(calls), 1)
self.assertEqual(list(self.root.glob(".extract-*")), [])
blob = assets_path / f"dever-{self.platform}.blob-{self.catalog['bootstrap/dever']['sha256']}"
for corrupt in (b"x" * len(self.payloads["bootstrap/dever"]), self.payloads["bootstrap/dever"] + b"extra"):
blob.write_bytes(corrupt)
with mock.patch.object(installer, "download", side_effect=download), \
mock.patch.object(installer.subprocess, "run") as execute, \
self.assertRaises(ValueError):
installer.payload(self.release, None, "1.2.3", self.platform, self.catalog, self.public)
execute.assert_not_called()
def test_download_archive_is_bounded(self):
with mock.patch.object(installer, "MAX_RELEASE", 4), \
mock.patch.object(installer, "download", return_value=io.BytesIO(b"12345")), \
self.assertRaisesRegex(ValueError, "超过"):
installer.download_archive("unused", self.root / "oversized.zst")
def test_local_payload_rejects_symlink_and_source_directory(self): def test_local_payload_rejects_symlink_and_source_directory(self):
copied = self.directory("copied") copied = self.directory("copied")
@@ -215,7 +259,7 @@ class InstallerTests(unittest.TestCase):
with self.assertRaisesRegex(ValueError, "符号链接"): with self.assertRaisesRegex(ValueError, "符号链接"):
installer.payload(self.directory("symlink"), self.release, "1.2.3", self.platform, self.catalog) installer.payload(self.directory("symlink"), self.release, "1.2.3", self.platform, self.catalog)
with self.assertRaisesRegex(ValueError, "symlinks"): with self.assertRaisesRegex(ValueError, "symlinks"):
assets.create(self.release, self.root / "invalid-assets") self.create_assets(self.root / "invalid-assets")
core.unlink() core.unlink()
core.mkdir() core.mkdir()
with self.assertRaisesRegex(ValueError, "类型"): with self.assertRaisesRegex(ValueError, "类型"):
+128 -22
View File
@@ -33,8 +33,8 @@ def execute(arguments, timeout=180):
return completed.stdout.strip() return completed.stdout.strip()
def response(output): def response(output, data="Hello, Dever"):
expected = {"code": 0, "message": "ok", "data": "Hello, Dever"} expected = {"code": 0, "message": "ok", "data": data}
if json.loads(output) != expected: if json.loads(output) != expected:
raise RuntimeError(f"unexpected application response: {output}") raise RuntimeError(f"unexpected application response: {output}")
@@ -50,30 +50,56 @@ def minimal_os(image):
(image / name).mkdir() (image / name).mkdir()
def isolated(image): def worker_project(image, ecosystem):
"""Small no-Lib Adapter: preparation must still acquire its managed runtime."""
entries = {
"pip": ("py", "import gzip\n\nasync def probe(payload, setting):\n return {\"okay\": gzip.decompress(gzip.compress(b'dever')) == b'dever'}\n"),
"npm": ("mjs", "import {createHash} from 'node:crypto';\nexport async function probe() { return {okay:createHash('sha256').update('dever').digest().length === 32}; }\n"),
"go": ("go", (WORKSPACE / "test/ecosystem-release/probe.go").read_text()),
}
extension, source = entries[ecosystem]
root = image / f"worker-{ecosystem}"
files = {
"config/setting.json": "{}\n",
"module/sample/worker/port.dever": "probe() (okay: Bool) fails app.ProbeResult\n",
"module/sample/worker/app.dever": "type ProbeResult { error Unavailable(message: Text) }\nprobe() (okay: Bool) { okay = port.probe() }\n",
"module/sample/worker/api.dever": "cmd probe = app.probe\n",
"module/sample/worker/adapter.dever": f'external {ecosystem} "worker/probe.{extension}" {{}}\n',
f"module/sample/worker/worker/probe.{extension}": source,
}
for relative, contents in files.items():
path = root / relative
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(contents)
return root
def isolated(image, init):
return [ return [
WORKSPACE / "target/sandbox-inputs/assets/bin/bwrap", WORKSPACE / "target/sandbox-inputs/assets/bin/bwrap",
"--unshare-all", "--die-with-parent", "--new-session", "--bind", image, "--unshare-all", "--die-with-parent", "--new-session", "--bind", image,
"/", "--proc", "/proc", "--dev", "/dev", "--chdir", "/", "--ro-bind", "/", "--proc", "/proc", "--dev", "/dev", "--chdir", "/", "--ro-bind",
WORKSPACE / "target/debug/examples/native-acceptance-init", "/fixture-proc-init", init, "/fixture-proc-init",
] ]
class InstalledDaemon: class InstalledDaemon:
"""All clients enter the same PID namespace so SO_PEERCRED stays valid.""" """All clients enter the same PID namespace so SO_PEERCRED stays valid."""
def __init__(self, image, records): def __init__(self, image, records, init):
self.image = image self.image = image
self.records = records self.records = records
self.init = init
self.child = None self.child = None
self.pidfd = None self.pidfd = None
self.pid = None self.pid = None
def __enter__(self): def __enter__(self):
self.records.mkdir(parents=True, exist_ok=True)
self.information = (self.records / "namespace.json").open("w+") self.information = (self.records / "namespace.json").open("w+")
self.errors = (self.records / "daemon.log").open("w+") self.errors = (self.records / "daemon.log").open("w+")
self.child = subprocess.Popen( self.child = subprocess.Popen(
[str(value) for value in isolated(self.image) + [ [str(value) for value in isolated(self.image, self.init) + [
"--info-fd", "1", "--", "/fixture-proc-init", "--info-fd", "1", "--", "/fixture-proc-init",
"/opt/dever/bin/deverd", "--root", "/opt/dever", "/opt/dever/bin/deverd", "--root", "/opt/dever",
]], env={}, stdout=self.information, stderr=self.errors, ]], env={}, stdout=self.information, stderr=self.errors,
@@ -134,6 +160,38 @@ class InstalledDaemon:
self.errors.close() self.errors.close()
def ordinary_uid_target_reuse(image):
# bwrap's single-UID user namespace cannot represent a second identity.
# Use the installed private machine with real kernel UIDs for this check;
# no host language tools or global service participate in target reuse.
machine = image / "opt/dever"
child = subprocess.Popen([str(machine / "bin/deverd"), "--root", str(machine)],
env={}, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
try:
deadline = time.monotonic() + 10
while not (machine / "state/deverd.sock").exists():
if child.poll() is not None or time.monotonic() >= deadline:
raise RuntimeError("private UID acceptance daemon did not become ready")
time.sleep(0.02)
for uid in (65533, 65534):
completed = subprocess.run(
[str(machine / "bin/dever"), "target", "add", "linux-aarch64"],
env={}, user=uid, group=uid, extra_groups=[], capture_output=True,
text=True, timeout=180, check=False,
)
if completed.returncode:
raise RuntimeError(f"UID {uid} target reuse failed: {completed.stderr}")
finally:
if child.poll() is None:
child.terminate()
try:
child.wait(timeout=10)
except subprocess.TimeoutExpired:
child.kill()
child.wait(timeout=10)
child.stderr.close()
def install_assets(assets, image, manifest): def install_assets(assets, image, manifest):
# Only transport is local: signature verification, full tar extraction and # Only transport is local: signature verification, full tar extraction and
# the final bootstrap executable are the real first-install implementation. # the final bootstrap executable are the real first-install implementation.
@@ -146,8 +204,12 @@ def install_assets(assets, image, manifest):
routes = { routes = {
f"{base}/latest/download/{stem}.manifest.json": assets / f"{stem}.manifest.json", f"{base}/latest/download/{stem}.manifest.json": assets / f"{stem}.manifest.json",
f"{base}/latest/download/{stem}.manifest.sig": assets / f"{stem}.manifest.sig", f"{base}/latest/download/{stem}.manifest.sig": assets / f"{stem}.manifest.sig",
f"{base}/download/v{manifest['version']}/{stem}.tar.gz": assets / f"{stem}.tar.gz", f"{base}/download/v{manifest['version']}/{stem}.tar.zst": assets / f"{stem}.tar.zst",
} }
for artifact in manifest["artifacts"]:
if artifact["path"] == "bootstrap/dever" or artifact["path"].startswith("bootstrap/lib/"):
name = f"{stem}.blob-{artifact['sha256']}"
routes[f"{base}/download/v{manifest['version']}/{name}"] = assets / name
installer.download = lambda address: routes[address].open("rb") installer.download = lambda address: routes[address].open("rb")
installer.install(argparse.Namespace( installer.install(argparse.Namespace(
version="latest", release=None, system_root=image, no_start=True, version="latest", release=None, system_root=image, no_start=True,
@@ -155,7 +217,7 @@ def install_assets(assets, image, manifest):
)) ))
def accept(release, assets, image, report): def accept(release, assets, image, report, init, extension_helper=None):
image.mkdir(mode=0o755) image.mkdir(mode=0o755)
report.parent.mkdir(parents=True, exist_ok=True) report.parent.mkdir(parents=True, exist_ok=True)
manifest = json.loads((release / "manifest.json").read_bytes()) manifest = json.loads((release / "manifest.json").read_bytes())
@@ -185,13 +247,19 @@ def accept(release, assets, image, report):
if (installed / "manifest.json").read_bytes() != (release / "manifest.json").read_bytes(): if (installed / "manifest.json").read_bytes() != (release / "manifest.json").read_bytes():
raise RuntimeError("installed package differs from the release under acceptance") raise RuntimeError("installed package differs from the release under acceptance")
passed("actual_signed_release_first_install") passed("actual_signed_release_first_install")
for extension in manifest["extensions"]:
if any((installed / artifact["path"]).exists() for artifact in extension["artifacts"]):
raise RuntimeError("base install unexpectedly includes optional extension bytes")
passed("base_install_excludes_optional_extensions")
launcher = image / "opt/dever/bin/dever" launcher = image / "opt/dever/bin/dever"
if execute([launcher, "version"]) != manifest["version"]: if execute([launcher, "version"]) != manifest["version"]:
raise RuntimeError("installed version mismatch") raise RuntimeError("installed version mismatch")
if (image / "usr/local/bin/dever").readlink() != Path("/opt/dever/bin/dever"): if (image / "usr/local/bin/dever").readlink() != Path("/opt/dever/bin/dever"):
raise RuntimeError("public launcher link mismatch") raise RuntimeError("public launcher link mismatch")
passed("installed_version_and_public_entry") passed("installed_version_and_public_entry")
with InstalledDaemon(image, report.parent) as daemon: standalone = [("first", "hello.greeting.greet", '{"name":"Dever"}', "Hello, Dever"),
("second", "hello.greeting.greet", '{"name":"Dever"}', "Hello, Dever")]
with InstalledDaemon(image, report.parent / f"{report.stem}-process", init) as daemon:
for project, markdown in (("first", False), ("second", True)): for project, markdown in (("first", False), ("second", True)):
root = "/" + project root = "/" + project
args = ["new", root] + (["--markdown"] if markdown else []) args = ["new", root] + (["--markdown"] if markdown else [])
@@ -211,27 +279,62 @@ def accept(release, assets, image, report):
if not (image / "ai-skill/SKILL.md").is_file(): if not (image / "ai-skill/SKILL.md").is_file():
raise RuntimeError("missing installed skill entry") raise RuntimeError("missing installed skill entry")
passed("versioned_skill_resolution_and_loader_install") passed("versioned_skill_resolution_and_loader_install")
daemon.command("build", "/first", "--target", "linux-aarch64", "--output", "/first/program.arm64") try:
with (image / "first/program.arm64").open("rb") as executable: daemon.command("build", "/first", "--target", "linux-aarch64", "--output", "/first/program.arm64")
header = executable.read(20) except RuntimeError as error:
if header[:4] != b"\x7fELF" or header[18:20] != b"\xb7\x00": if "dever target add linux-aarch64" not in str(error):
raise RuntimeError("cross build did not produce an ARM64 ELF") raise
passed("arm64_application_cross_build") else:
raise RuntimeError("unprepared cross build should fail without fetching resources")
passed("missing_target_fails_offline_with_preparation_command")
if assets and extension_helper:
execute([extension_helper, image / "opt/dever", assets, manifest["version"], "target-linux-aarch64"])
daemon.command("target", "add", "linux-aarch64")
daemon.command("build", "/first", "--target", "linux-aarch64", "--output", "/first/program.arm64")
with (image / "first/program.arm64").open("rb") as executable:
header = executable.read(20)
if header[:4] != b"\x7fELF" or header[18:20] != b"\xb7\x00":
raise RuntimeError("cross build did not produce an ARM64 ELF")
passed("actual_target_extension_install_reuse_and_arm64_application_cross_build")
for ecosystem in ("pip", "npm", "go"):
execute([extension_helper, image / "opt/dever", assets, manifest["version"],
f"runtime-{ecosystem}-{manifest['platform']}"])
project = worker_project(image, ecosystem)
root = "/" + project.name
daemon.command("lib", "update", root)
before = [(project / name).read_bytes() for name in ("config/setting.json", "dever.lock")]
daemon.command("lib", "install", root)
after = [(project / name).read_bytes() for name in ("config/setting.json", "dever.lock")]
if before != after:
raise RuntimeError("exact Lib installation modified project configuration or lock")
daemon.command("check", root)
response(daemon.command("run", root, "--", "sample.worker.probe", "{}"), True)
daemon.command("build", root, "--output", root + "/program")
standalone.append((project.name, "sample.worker.probe", "{}", True))
passed(f"actual_{ecosystem}_extension_no_lib_worker_exact_install_run_build")
result["cache"] = json.loads(daemon.command("cache", "status")) result["cache"] = json.loads(daemon.command("cache", "status"))
if assets and extension_helper:
ordinary_uid_target_reuse(image)
passed("two_ordinary_uids_reuse_shared_signed_target_extension")
# Remove sources and the entire machine from the visible OS root. Keep # Remove sources and the entire machine from the visible OS root. Keep
# them outside that root for a subsequent online-update acceptance. # them outside that root for a subsequent online-update acceptance.
moved = [] moved = []
try: try:
for relative in ("opt/dever", "first/module", "second/module"): for relative in ["opt/dever"] + [f"{project}/module" for project, *_ in standalone]:
source = image / relative source = image / relative
hidden = image.parent / ("hidden-" + relative.replace("/", "-")) hidden = image.parent / ("hidden-" + relative.replace("/", "-"))
source.rename(hidden) source.rename(hidden)
moved.append((source, hidden)) moved.append((source, hidden))
for project in ("first", "second"): for project, entry, parameters, data in standalone:
response(execute(isolated(image) + [ # These projects were created above; remove only their generated
# Worker cache so standalone execution must extract its own bundle.
cache = image / project / "data/cache/lib"
if cache.exists():
shutil.rmtree(cache)
response(execute(isolated(image, init) + [
"--", "/fixture-proc-init", f"/{project}/program", "--", "/fixture-proc-init", f"/{project}/program",
"hello.greeting.greet", '{"name":"Dever"}', entry, parameters,
])) ]), data)
passed("standalone_programs_without_sources_or_machine") passed("standalone_programs_without_sources_or_machine")
finally: finally:
for source, hidden in reversed(moved): for source, hidden in reversed(moved):
@@ -248,14 +351,17 @@ def accept(release, assets, image, report):
def main(): def main():
parser = argparse.ArgumentParser(description=__doc__) parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--release", required=True, type=Path) parser.add_argument("--release", required=True, type=Path)
parser.add_argument("--assets", type=Path, help="also exercise the three actual download assets") parser.add_argument("--assets", type=Path, help="exercise actual split download assets")
parser.add_argument("--extension-helper", type=Path, help="explicit prepare-extensions author fixture")
parser.add_argument("--init", type=Path, required=True, help="explicit static native-acceptance-init fixture")
parser.add_argument("--image", required=True, type=Path) parser.add_argument("--image", required=True, type=Path)
parser.add_argument("--report", required=True, type=Path) parser.add_argument("--report", required=True, type=Path)
args = parser.parse_args() args = parser.parse_args()
if os.geteuid() != 0: if os.geteuid() != 0:
parser.error("requires root for an owned image/namespace; never installs host services") parser.error("requires root for an owned image/namespace; never installs host services")
accept(args.release.resolve(), args.assets.resolve() if args.assets else None, accept(args.release.resolve(), args.assets.resolve() if args.assets else None,
args.image.absolute(), args.report.absolute()) args.image.absolute(), args.report.absolute(), args.init.absolute(),
args.extension_helper.absolute() if args.extension_helper else None)
if __name__ == "__main__": if __name__ == "__main__":
+114
View File
@@ -0,0 +1,114 @@
#!/usr/bin/env python3
"""Actual-asset update transaction in an owned image; never publishes the fixture version."""
import argparse
import hashlib
import json
import os
from pathlib import Path
from verify_prepared_release import InstalledDaemon, execute, install_assets, minimal_os, response
def prepare_update(image, manifest, key, helper):
# Reuse real payloads under a newly signed, acceptance-only catalog identity.
# This tests the update owner without claiming a second public release exists.
next_manifest = dict(manifest)
major, minor, patch = map(int, manifest["version"].split("."))
version = f"{major}.{minor}.{patch + 1}"
next_manifest["version"] = version
machine = image / "opt/dever"
package = machine / "cache/downloads" / version
package.mkdir()
installed = machine / "versions" / manifest["version"]
for artifact in manifest["artifacts"]:
destination = package / artifact["path"]
destination.parent.mkdir(parents=True, exist_ok=True)
os.link(installed / artifact["path"], destination)
catalog = package / "manifest.json"
catalog.write_text(json.dumps(next_manifest, separators=(",", ":")))
execute([helper, "--sign-catalog", catalog, key])
(machine / "cache/downloads/latest").write_text(version)
return version, hashlib.sha256(catalog.read_bytes()).hexdigest()
def accept(args):
manifest = json.loads((args.release / "manifest.json").read_bytes())
result = {"version": manifest["version"], "checks": [],
"transport": "local actual assets; acceptance-only next-version catalog",
"online_update": "not_run"}
def passed(name):
result["checks"].append(name)
args.report.write_text(json.dumps(result, indent=2) + "\n")
print(f"PASS {name}", flush=True)
try:
args.image.mkdir()
minimal_os(args.image)
install_assets(args.assets, args.image, manifest)
machine = args.image / "opt/dever"
extension = f"runtime-pip-{manifest['platform']}"
execute([args.extension_helper, machine, args.assets, manifest["version"], extension])
records = args.report.parent / f"{args.report.stem}-process"
with InstalledDaemon(args.image, records, args.init) as daemon:
daemon.command("new", "/project")
response(daemon.command("run", "/project", "--", "hello.greeting.greet", '{"name":"Dever"}'))
passed("actual_release_and_selected_extension_before_update")
version, catalog = prepare_update(args.image, manifest, args.key, args.extension_helper)
result["fixture_version"] = version
previous_active = (machine / "state/active-version").read_bytes()
empty_assets = args.image / "missing-extension-assets"
empty_assets.mkdir()
try:
execute([args.extension_helper, machine, empty_assets, version, "--update"])
except RuntimeError as error:
if "cannot open actual extension asset" not in str(error):
raise
else:
raise RuntimeError("update with a missing extension should fail")
if (machine / "state/active-version").read_bytes() != previous_active:
raise RuntimeError("failed update changed active version")
if daemon.command("version") != manifest["version"]:
raise RuntimeError("failed update changed the public launcher selection")
response(daemon.command("run", "/project", "--", "hello.greeting.greet", '{"name":"Dever"}'))
passed("failed_extension_update_keeps_old_active_and_application_working")
execute([args.extension_helper, machine, args.assets, version, "--update"])
if daemon.command("version") != version:
raise RuntimeError("successful update did not activate the new version")
if daemon.command("skill", "path") != f"/opt/dever/versions/{version}/skills/dever-language":
raise RuntimeError("skill did not follow the updated core")
extensions = machine / "cache/extensions" / version / catalog
if {path.name for path in extensions.iterdir()} != {extension}:
raise RuntimeError("update did not preserve exactly the installed extension selection")
passed("successful_update_activates_core_skill_and_only_installed_extensions")
# Only the catalog version changes in this transaction fixture.
# The managed compiler correctly requires its real build version;
# execute it again only after selecting the actual release.
daemon.command("use", manifest["version"])
response(daemon.command("run", "/project", "--", "hello.greeting.greet", '{"name":"Dever"}'))
passed("actual_release_remains_usable_after_update_and_explicit_switch_back")
result["status"] = "passed"
except BaseException as error:
result["status"] = "failed"
result["error"] = str(error)
raise
finally:
args.report.write_text(json.dumps(result, indent=2) + "\n")
def main():
parser = argparse.ArgumentParser(description=__doc__)
for name in ("release", "assets", "image", "report", "init", "extension-helper", "key"):
parser.add_argument(f"--{name}", required=True, type=Path)
args = parser.parse_args()
if os.geteuid() != 0:
parser.error("requires root for an owned image; never installs host services")
for name, value in vars(args).items():
setattr(args, name, value.absolute())
args.report.parent.mkdir(parents=True, exist_ok=True)
accept(args)
if __name__ == "__main__":
main()