feat: ship slim releases with shared on-demand extensions and locked restore

sdk/README.md

@@ -16,7 +16,7 @@ Run the opt-in host SDK check with explicit absolute runtime paths, for example:
 
 Omitted runtime paths report `unavailable`; the script never discovers language runtimes through `PATH`. It builds the Go fixtures in a temporary offline cache. This check exercises both the lower-level and compiler-manifest-shaped typed paths on host interpreters. Managed runtime packaging and execution on a machine without host language runtimes require separate acceptance.
 
-Managed dependencies use `dever-lock-v5`. Python extras and selected wheel METADATA, npm nested instances/peers/optional/bundled dependencies, and authenticated Go module evidence are part of the exact lock. Native Python wheels install into the packaged interpreter prefix and must satisfy its signed tags, extension ABI and loader closure.
+Managed dependencies use `dever-lock-v6`. Python extras and selected wheel METADATA, npm nested instances/peers/optional/bundled dependencies, and authenticated Go module evidence are part of the exact lock. Original archives retain their fixed source locator and filename; build receipts retain the complete execution input lock, including npm's graph before optional build pruning. `dever lib install <project>` restores these exact inputs and artifacts without resolving new versions or modifying configuration/lock bytes. Missing built outputs are replayed through the same sandbox executor and must match the recorded output digest; non-reproducible outputs fail explicitly. Native Python wheels install into the packaged interpreter prefix and must satisfy its signed tags, extension ABI and loader closure.
 
 Cross builds select the application platform explicitly: `dever lib update <project> --target linux-aarch64`, followed by `dever build <project> --target linux-aarch64 --output <new-file>`. Worker source and Port contracts stay unchanged. Locks, interpreters, native dependencies and sandbox assets must match that target. A Go runtime pack additionally declares `build.host` for its compiler/linker/analyzer; `build.goos/goarch` and standard-library object headers describe the output architecture. The tools run with a cleared environment and must have the correct target baked in. Missing target assets are errors, with no host runtime fallback. `run` and ordinary tests remain host operations.
 

sdk/native-release.md

@@ -6,7 +6,7 @@ Build the four runtime archives separately, then assemble and sign a new directo
 
 ## Build inputs
 
-Build the compiler with `cargo build --offline --locked -p dever-cli --bin dever`; its author input becomes the installed `dever-core`. Build the stable machine launcher with `--bin dever-launcher` and the daemon with `--bin deverd`. Use `target/debug/dever-launcher` for the bootstrap launcher input, which the maker publishes as `bootstrap/dever`. The installed public command remains `dever`; `dever-launcher` is only an author build name. Preserve this distinction when preparing release inputs.
+Build the compiler with `cargo build --release --offline --locked -p dever-cli --bin dever`; its author input becomes the installed `dever-core`. Build the stable machine launcher with `--bin dever-launcher` and the daemon with `--bin deverd` in the same release profile. Use `target/release/dever-launcher` for the bootstrap launcher input, which the maker publishes as `bootstrap/dever`. The release profile uses fat LTO, one codegen unit and symbol stripping; `runtime-pack` overrides stripping to retain linkable archive symbols. The installed public command remains `dever`; `dever-launcher` is only an author build name.
 
 Prepare the pinned Rust dependencies and compiler LLVM 18/LLD SDK explicitly. Use an explicitly selected Cargo/Rust toolchain for author builds. The root `runtime-pack` Cargo profile inherits the release optimization settings, uses one codegen unit and removes debug information while retaining archive symbols needed by LLD. It must not enable the bridge's `embedded` feature.
 
@@ -34,7 +34,7 @@ Prepare a matching compiler core, `libLLVM.so.18.1`, target CRT objects and genu
 
 ## Author settings
 
-Application authors select `dever build <project-root> --target linux-aarch64 --output <new-file>`; omitted targets, `run`, and `test` use the host. This selects application output, not an ARM compiler installation. Missing target packs are errors. Product commands do not discover cross tools or download packs while building.
+Application authors first prepare `dever target add linux-aarch64`, then select `dever build <project-root> --target linux-aarch64 --output <new-file>`; omitted targets, `run`, and `test` use the host. This selects application output, not an ARM compiler installation. Missing target packs are errors with a preparation command. Product commands do not discover cross tools or download packs while building.
 
 For an additional application target, add `native_targets.linux-aarch64` to the author settings. Its value has the same `profiles`, `start`, `libraries`, and `end` fields as the host inputs below. Build all four archives with Rust target `aarch64-unknown-linux-gnu` and an explicitly prepared cross linker/sysroot. Use ARM CRT and static libraries, never host files or linker scripts. Both standalone object and archive member ELF headers are checked against the declared target before publishing and on cache access.
 
@@ -96,7 +96,11 @@ Use a separate author root, not a Dever application's configuration. Place its e
 
 `dever skill path [project-root]` validates and returns the skill belonging to the selected compiler. `dever skill install <new-directory>` installs a small AI entry that resolves the active signed skill on each task. It refuses existing directories. `dever update` and `dever use` change the one active-version journal used by both core and skill; they do not rewrite every user's AI directory. Ordinary users can read the signed skill without the install lock or administrator permissions.
 
-Publish official versions to `shemic/dever-main` GitHub Releases with tag `v<version>`. `sdk/release-assets.py` converts the signed directory into three assets named `dever-<platform>.manifest.json`, `dever-<platform>.manifest.sig`, and `dever-<platform>.tar.gz`. The archive contains exactly the manifest's ordinary artifact files, with no directory headers or embedded manifest. The installer and explicit `dever install/update` download these assets, verify the independent pinned public key before payload extraction, and never use application configuration or environment variables for the official address. Project `run/build` remain offline.
+Publish official versions to `shemic/dever-main` GitHub Releases with tag `v<version>`. The v2 signed root catalog separates base `artifacts` from `extensions`. Base includes core/private libraries, bootstrap, host native profiles/sandbox and skill. Each extension declares `{kind,target,artifacts}`; `kind` is `{"type":"runtime","ecosystem":"pip|npm|go"}`, `{"type":"build","ecosystem":"pip|npm"}` or `{"type":"target"}`. Target combines that target's native profiles/CRT and sandbox. Paths and identities cannot overlap. No extension has an independent unsigned manifest or can override base files.
+
+`python3 sdk/release-assets.py <release> --output <new-assets> --zstd /absolute/author/zstd` writes `dever-<platform>.manifest.json`, `.manifest.sig`, `.tar.zst`, plus `.ext-<runtime-pip|runtime-npm|runtime-go|build-pip|build-npm|target>-<target>.tar.zst`. Each USTAR contains exactly its signed ordinary artifact files, no directory headers or embedded root metadata. Zstandard uses a checksummed single frame with a maximum 128 MiB window. The publisher also emits `.blob-<sha256>` for bootstrap/dever and every bootstrap/lib dependency. First installation authenticates these raw helpers before executing the launcher's bounded extractor; end users do not need a system zstd executable. Report helper bytes separately from base archive size.
+
+Only explicit install/update, target add and dependency preparation download official resources. The source address is compiled into the toolchain, not application configuration or environment variables. Extension preparation uses closed typed IPC and the existing machine service, storing immutable version/catalog-bound resources under `cache/extensions`. Multiple projects/users share one verified tree. `run/build` and resource reads never download. `update` prepares the new core/skill and the active version's already installed extension selection before publishing the active-version journal; failure keeps the old selection usable. The stable machine launcher/daemon are provisioned by bootstrap, so changes to that protocol require a matching trusted bootstrap installation.
 
 ## Linux first installation
 
@@ -234,14 +238,19 @@ The third command requires root and namespace permission for a test-owned mount
 
 The harness rejects a dynamically linked init. Inside the already pivoted namespace, it replaces bwrap's proc submounts with a complete private proc mount so the nested Worker can mount its own proc. It keeps the outer root fixture's existing capabilities; the product's inner namespace, capability removal and guard remain unchanged. It never binds the host proc into the final root. This private compiler test complements the managed daemon test; it does not install a machine service.
 
-To accept an already prepared official release (including its real signing identity), use the owned-image verifier. `--assets` feeds the exact three download files through the real installer using local transport; signature checks, extraction and bootstrap execution are unchanged. It requires the explicit static bwrap/init fixtures described above, creates a new image, starts only its own daemon and records each passed stage. Online `dever update` and real-host service activation remain separate checks.
+To accept an already prepared official release (including its real signing identity), use the owned-image verifier. `--assets` feeds the exact split download files through the real installer using local transport; signature checks, extraction and bootstrap execution are unchanged. The `prepare-extensions` author example replaces only extension transport with actual local assets; it uses the product owner for validation and installation. Supply it with `--extension-helper` to exercise ARM preparation, CLI reuse and cross compilation. The verifier requires explicit static bwrap/init fixtures, creates a new image, starts only its own daemon and records each passed stage. Online update and real-host service activation remain separate checks.
 
 ```sh
 python3 -B test/onboarding/verify_prepared_release.py \
   --release /prepared/release --assets /prepared/download-assets \
+  --init /prepared/native-acceptance-init --extension-helper /prepared/prepare-extensions \
   --image /new/owned-image --report /owned-records/acceptance.json
 ```
 
+The image and its owned ancestor directories must be traversable by the two test UIDs (65533 and 65534). Their resource-reuse check starts a separate private-machine daemon with real kernel identities, because the minimal OS fixture's single-UID user namespace cannot represent another user. It does not touch the global service.
+
+`test/onboarding/verify_release_update.py` accepts the same asset/init/helper paths plus `--key <explicit-author-pkcs8>`. It re-signs the real payload catalog as an acceptance-only next version, verifies that a missing extension preserves the old working selection, then verifies that a successful update activates core/skill and exactly the installed extension set. That fixture version is never published; this is local update-transaction evidence, not an online Release acceptance.
+
 The existing fixture-based bootstrap regression additionally requires the built `dever` and `deverd` binaries:
 
 ```sh

sdk/release-assets.py

@@ -1,18 +1,132 @@
 #!/usr/bin/env python3
-"""Turn a prepared signed release into the three official GitHub release assets."""
+"""Publish base, optional extension and authenticated bootstrap assets from a signed release."""
 
 import argparse
 import ctypes
-import gzip
 import hashlib
 import json
 import os
 from pathlib import Path, PurePosixPath
+import re
+import shutil
+import subprocess
 import tarfile
 import tempfile
 
 
-def create(release, output):
+MAX_PACKAGE = 2 * 1024**3
+
+
+def extension_suffix(extension):
+    kind = extension["kind"]
+    target = extension["target"]
+    if target not in ("linux-x86_64", "linux-aarch64"):
+        raise ValueError("unsupported extension target")
+    if kind == {"type": "target"}:
+        name = "target"
+    elif (set(kind) == {"type", "ecosystem"}
+          and kind["type"] in ("runtime", "build")
+          and kind["ecosystem"] in ("pip", "npm", "go")
+          and not (kind["type"] == "build" and kind["ecosystem"] == "go")):
+        name = f"{kind['type']}-{kind['ecosystem']}"
+    else:
+        raise ValueError("unsupported extension kind")
+    return f"ext-{name}-{target}"
+
+
+def packages(manifest):
+    if (set(manifest) != {"format", "version", "platform", "artifacts", "extensions"}
+            or manifest["format"] != "dever-release-v2"
+            or manifest["platform"] not in ("linux-x86_64", "linux-aarch64")
+            or not re.fullmatch(r"(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)", manifest["version"])):
+        raise ValueError("unsupported release format, version or platform")
+    if not isinstance(manifest["extensions"], list) or len(manifest["extensions"]) > 12:
+        raise ValueError("release extension count exceeds limit")
+    selected = {"": manifest["artifacts"]}
+    for extension in manifest["extensions"]:
+        if set(extension) != {"kind", "target", "artifacts"}:
+            raise ValueError("invalid extension fields")
+        suffix = extension_suffix(extension)
+        if suffix in selected:
+            raise ValueError("duplicate extension identity")
+        target = extension["target"]
+        kind = extension["kind"]
+        if kind["type"] == "target":
+            if target == manifest["platform"]:
+                raise ValueError("host native resources belong to the base package")
+            prefixes = (f"runtime/{target}/", f"sandbox/{target}/")
+        else:
+            prefixes = (f"{kind['type']}/{kind['ecosystem']}/{target}/",)
+        if any(not artifact["path"].startswith(prefixes) for artifact in extension["artifacts"]):
+            raise ValueError("extension artifact is outside its resource namespace")
+        selected[suffix] = extension["artifacts"]
+    return selected
+
+
+def verify_artifacts(release, groups):
+    names = set()
+    for artifacts in groups.values():
+        if (not 1 <= len(artifacts) <= 4096
+                or any(type(entry["bytes"]) is not int or entry["bytes"] < 0 for entry in artifacts)
+                or sum(entry["bytes"] for entry in artifacts) > MAX_PACKAGE):
+            raise ValueError("release exceeds the official asset limits")
+        for artifact in artifacts:
+            verify_artifact(release, artifact, names)
+    for name in names:
+        if any(str(parent) in names for parent in PurePosixPath(name).parents):
+            raise ValueError("release artifacts overlap as file and directory")
+
+
+def verify_artifact(release, artifact, names):
+    name = artifact["path"]
+    if (set(artifact) != {"path", "bytes", "sha256"}
+            or not name or len(name.encode("utf-8")) > 1024
+            or str(PurePosixPath(name)) != name or name.startswith("/")
+            or any(part in ("", ".", "..") for part in name.split("/"))
+            or any(character in name for character in ("\\", ":", "\0"))
+            or name in names or name in ("manifest.json", "manifest.sig")
+            or not re.fullmatch(r"[0-9a-f]{64}", artifact["sha256"])):
+        raise ValueError("invalid release artifact path or digest")
+    names.add(name)
+    path = release / name
+    for parent in (path, *path.parents):
+        if parent.is_symlink():
+            raise ValueError("release artifacts must not traverse symlinks")
+        if parent == release:
+            break
+    if not path.is_file() or path.stat().st_size != artifact["bytes"]:
+        raise ValueError("release artifact type/size mismatch")
+    with path.open("rb") as stream:
+        if hashlib.file_digest(stream, "sha256").hexdigest() != artifact["sha256"]:
+            raise ValueError("release artifact digest mismatch")
+
+
+def compress(release, artifacts, destination, zstd):
+    # Compression is an explicit author tool; installation uses the signed launcher codec.
+    with destination.open("xb") as output:
+        process = subprocess.Popen([str(zstd), "-q", "-10", "--long=27", "-c"],
+                                   stdin=subprocess.PIPE, stdout=output,
+                                   stderr=subprocess.PIPE, env={})
+        try:
+            with process.stdin:
+                with tarfile.open(fileobj=process.stdin, mode="w|", format=tarfile.USTAR_FORMAT) as archive:
+                    for artifact in sorted(artifacts, key=lambda entry: entry["path"]):
+                        header = tarfile.TarInfo(artifact["path"])
+                        header.size = artifact["bytes"]
+                        header.mode = 0o644
+                        with (release / artifact["path"]).open("rb") as contents:
+                            archive.addfile(header, contents)
+            errors = process.stderr.read(4096)
+            if process.wait() != 0:
+                raise ValueError(f"release compression failed: {errors.decode(errors='replace')}")
+        finally:
+            if process.poll() is None:
+                process.kill()
+            process.wait()
+            process.stderr.close()
+
+
+def create(release, output, zstd):
     release = release.absolute()
     output = output.absolute()
     if release.is_symlink() or not release.is_dir():
@@ -21,49 +135,29 @@ def create(release, output):
         raise ValueError("output already exists")
     if not output.parent.is_dir():
         raise ValueError("output parent must exist")
-    manifest_bytes = (release / "manifest.json").read_bytes()
+    if not zstd.is_absolute() or not zstd.is_file():
+        raise ValueError("zstd must be an explicit absolute author tool path")
+    with (release / "manifest.json").open("rb") as stream:
+        manifest_bytes = stream.read(2 * 1024**2 + 1)
+    if len(manifest_bytes) > 2 * 1024**2:
+        raise ValueError("release metadata exceeds limit")
     manifest = json.loads(manifest_bytes)
-    if manifest["format"] != "dever-release-v1" or manifest["platform"] not in ("linux-x86_64", "linux-aarch64"):
-        raise ValueError("unsupported release format or platform")
-    artifacts = manifest["artifacts"]
-    if not 1 <= len(artifacts) <= 4096 or sum(entry["bytes"] for entry in artifacts) > 2 * 1024**3:
-        raise ValueError("release exceeds the official asset limits")
-    names = set()
-    for artifact in artifacts:
-        name = artifact["path"]
-        if (not name or str(PurePosixPath(name)) != name or name.startswith("/")
-                or any(part in ("", ".", "..") for part in name.split("/"))
-                or any(character in name for character in ("\\", ":", "\0"))
-                or name in names or name in ("manifest.json", "manifest.sig")):
-            raise ValueError("invalid release artifact path")
-        names.add(name)
-        path = release / name
-        for parent in (path, *path.parents):
-            if parent.is_symlink():
-                raise ValueError("release artifacts must not traverse symlinks")
-            if parent == release:
-                break
-        if not path.is_file() or path.stat().st_size != artifact["bytes"]:
-            raise ValueError("release artifact type/size mismatch")
-        with path.open("rb") as stream:
-            if hashlib.file_digest(stream, "sha256").hexdigest() != artifact["sha256"]:
-                raise ValueError("release artifact digest mismatch")
+    groups = packages(manifest)
+    verify_artifacts(release, groups)
     with tempfile.TemporaryDirectory(prefix=".dever-release-assets-", dir=output.parent) as scratch:
         prepared = Path(scratch) / "assets"
         prepared.mkdir()
         stem = f"dever-{manifest['platform']}"
         (prepared / f"{stem}.manifest.json").write_bytes(manifest_bytes)
         (prepared / f"{stem}.manifest.sig").write_bytes((release / "manifest.sig").read_bytes())
-        with (prepared / f"{stem}.tar.gz").open("xb") as compressed:
-            with gzip.GzipFile(filename="", fileobj=compressed, mode="wb", mtime=0) as stream:
-                with tarfile.open(fileobj=stream, mode="w|", format=tarfile.USTAR_FORMAT) as archive:
-                    for artifact in sorted(artifacts, key=lambda entry: entry["path"]):
-                        path = release / artifact["path"]
-                        header = tarfile.TarInfo(artifact["path"])
-                        header.size = artifact["bytes"]
-                        header.mode = 0o644
-                        with path.open("rb") as contents:
-                            archive.addfile(header, contents)
+        for suffix, artifacts in groups.items():
+            filename = f"{stem}{'.' + suffix if suffix else ''}.tar.zst"
+            compress(release, artifacts, prepared / filename, zstd)
+        for artifact in manifest["artifacts"]:
+            if artifact["path"] == "bootstrap/dever" or artifact["path"].startswith("bootstrap/lib/"):
+                blob = prepared / f"{stem}.blob-{artifact['sha256']}"
+                if not blob.exists():
+                    shutil.copyfile(release / artifact["path"], blob)
         publish(prepared, output)
     return sorted(path.name for path in output.iterdir())
 
@@ -84,9 +178,10 @@ def main():
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("release", type=Path)
     parser.add_argument("--output", required=True, type=Path)
+    parser.add_argument("--zstd", required=True, type=Path, help="explicit author-side zstd executable")
     args = parser.parse_args()
     try:
-        for name in create(args.release, args.output):
+        for name in create(args.release, args.output, args.zstd):
             print(name)
     except (OSError, ValueError, KeyError, TypeError, tarfile.TarError) as error:
         parser.exit(1, f"cannot prepare release assets: {error}\n")