fix: restart Dever releases at 0.0.1 with complete signed assets

sdk/README.md

@@ -63,6 +63,26 @@ Worker 和依赖构建都直接在当前容器中运行，由内核启动真实
 
 GPU 使用父容器已经分配的设备和驱动，不再由 Dever 重新挂载。除已打包的私有库路径外，仅透传 `CUDA_VISIBLE_DEVICES`、`NVIDIA_VISIBLE_DEVICES`、`NVIDIA_DRIVER_CAPABILITIES`、`CUDA_DEVICE_ORDER` 与 `LD_LIBRARY_PATH`；其他宿主环境不自动传入。CUDA/PyTorch 等依赖仍需匹配目标平台。是否有 GPU、是否可加载驱动以及显存是否足够由真实部署决定，不会绕过 Kubernetes 调度或容器权限。设备验收须单独取得 GPU 准入；本次源码改造不等于已部署或模型推理通过。
 
+## Python 发行资源
+
+应用容器的系统库基线与编译器开发机不同。不要直接复制开发容器的 Python：解释器和每个动态扩展都必须满足目标 glibc 基线；Python 版本相同不代表系统 ABI 相同。保留真实解释器入口，不改用私有 GNU loader，也不回退到应用容器的系统 Python。
+
+`sdk/python-runtime.py` 从显式提供并锁定 SHA-256 的 Python Build Standalone `install_only_stripped` 归档生成输入，不联网下载、不安装服务。参数为 `--archive`、`--sha256`、`--version`、`--target`、`--readelf`、`--output`，默认 `--glibc-max 2.17`。输出目录必须不存在；工具检查路径、目标 ELF、所有 GLIBC 版本依赖、基础模块及迁移后的解释器身份，保留运行文件和 Python 许可证，不带 pip、第三方 site-packages 或测试缓存。
+
+输出的 `python-runtime.json` 对应既有作者配置 `runtimes.pip`，其 `source` 相对输出根；合入同一作者根后仍由 `native-release` 校验、打包和签名，Python 继续是独立扩展，不塞进基础包。wheel 标签包含同一 CPython 的原生 ABI、兼容的 abi3 和纯 Python 标签；manylinux 不超过目标 glibc 基线，通用 `linux_<架构>` 用于本机源码构建产物，不承诺跨系统 ABI 兼容。源码归档的来源、SHA-256 和实际目标环境验收结果归本次发行记录；本地候选目录不能充当正式源或修改现行安装。
+
+## 完整公开发行
+
+正式附件使用 `sdk/release-assets.py --editor-vsix <VSIX>` 导出，默认执行完整性检查。必须包含独立基础包、x86_64/ARM64 的 Python/Node/Go 运行扩展、宿主 Python/npm 构建扩展、ARM64 原生目标扩展、同版本编辑器扩展，以及签名清单和引导文件。缺任一项就失败；部分资源只允许通过显式的 `--partial-test-fixture` 用于定向测试，不能当作完整公开版本。Go 扩展由 `test/go-managed/make_pack.go` 导出完整 `std`，不能只导出样例依赖。
+
+`.tar.zst` 是 tar 文件集合经过 Zstandard 压缩，减少附件体积；它不是另一个运行环境。安装器使用已验签的引导程序解压，不要求使用者额外安装 zstd。Python、Node、Go 及构建工具继续按需单独下载，不并入基础包。
+
+当前发行序列经项目所有者确认从 `0.0.1` 重新开始，旧 Release 与附件在替代包验收后清理，不删除源代码提交历史。资源格式固定为现行 `process-runtime`，不再根据版本号推断旧沙箱布局。旧版 `0.1.x` 安装不能通过普通升级降到 `0.0.1`；须在新的安装根验收后明确切换，保留项目和凭据。不要关闭签名、同版本身份检查或防降级检查来绕过此规则。后续版本变更由项目所有者确认，不因每次修改自动递增版本号。
+
+定向测试：`python3 -B -m unittest discover -s test/onboarding -p test_python_runtime.py -v`。这些测试验证打包约束，不代替在目标容器中执行签名应用、真实模型和取消清理验收；x86_64 验收也不证明 ARM64 已验收。
+
+跨架构作者验证可显式传入 `--runner /absolute/qemu-aarch64 --sysroot /absolute/arm64-sysroot`，仍执行相同解释器身份和基础模块探测，不跳过 ELF 检查；须记录为 QEMU 验收，而不是原生 ARM64 硬件验收。这两个参数不进入应用运行配置。
+
 ## 依赖与跨平台构建
 
 `dever lib install <project>` 按现有锁恢复依赖，不重新选版本、不修改配置或锁。`dever-lock-v6` 记录 Python extras/wheel METADATA、npm 嵌套依赖及 peers/optional/bundled 依赖、Go 模块校验证据。原始包保留来源与文件名，构建记录保留完整输入，包括 npm 剪除可选构建依赖前的依赖图。
@@ -77,6 +97,8 @@ Go pack 的 `build.host` 描述编译器、链接器与分析器的运行平台
 
 只有显式准备 Lib 时才构建 Python sdist 或运行 npm 安装钩子。构建使用签名工具包，在独立临时工作目录内直接运行受管子进程；不再提供内层文件/网络隔离，因此只运行可信依赖及构建钩子。缺少工具时不回退宿主 PATH。
 
+直接执行 GCC/binutils 时，工具依赖的非 glibc 动态库从已验签的 SDK 在本次临时目录内建立硬链接，供既有受管库路径加载；不依赖宿主另装 libbfd 等作者工具库，不复制额外库数据，也不替换宿主 glibc/loader。Python 原生扩展和 npm 原生 addon 必须分别通过真实源码构建验收，纯 Python/JS 回退不算通过。
+
 Python PEP 517 的构建依赖单独锁定，每个 hook 使用新进程，最终 wheel 按保留的元数据核对。npm 的配套运行器处理 `preinstall/install/postinstall`、依赖 bin 和 node-gyp。
 
 构建记录绑定源码、目标运行环境、工具、构建前端、依赖图和输出摘要。Worker 准备阶段离线恢复结果，不重跑安装钩子。npm `.node` 产物须通过目标检查、动态加载检查和独立 Node 加载探测，才能进入 Worker 资源目录。不匹配平台或 ABI 的其他变体保留在锁定原始包中，不加入运行目录。运行时非系统依赖来自 Worker 目录，系统 ABI 由目标容器提供，不能借用编译器 sysroot。

sdk/python-runtime.py

@@ -0,0 +1,182 @@
+import argparse
+import hashlib
+import json
+from pathlib import Path, PurePosixPath
+import re
+import shutil
+import subprocess
+import tarfile
+
+
+MAX_BYTES = 256 * 1024 * 1024
+TARGETS = {
+    "linux-x86_64": ("x86_64", "Advanced Micro Devices X86-64"),
+    "linux-aarch64": ("aarch64", "AArch64"),
+}
+OMIT = {"__pycache__", "site-packages", "test", "tests", "tkinter", "turtledemo", "idlelib", "ensurepip"}
+PROBE = """import ctypes, hashlib, importlib.machinery, json, platform, sqlite3, ssl, sys, zlib
+print(json.dumps({"version": platform.python_version(), "machine": platform.machine(),
+                  "executable": sys.executable, "prefix": sys.prefix,
+                  "suffixes": importlib.machinery.EXTENSION_SUFFIXES}))
+"""
+
+
+def checksum(path):
+    with path.open("rb") as stream:
+        return hashlib.file_digest(stream, "sha256").hexdigest()
+
+
+def selected(name, minor):
+    path = PurePosixPath(name)
+    if path == PurePosixPath(f"python/bin/python{minor}"):
+        return True
+    if len(path.parts) == 3 and path.parts[:2] == ("python", "lib"):
+        return bool(re.fullmatch(r"libpython[\d.]+\.so(?:\.\d+)*", path.name))
+    prefix = ("python", "lib", f"python{minor}")
+    return (path.parts[:3] == prefix and not OMIT.intersection(path.parts[3:])
+            and not any(part.startswith("config-") for part in path.parts[3:])
+            and not path.name.startswith("_test") and path.suffix not in (".pyc", ".pyo", ".a"))
+
+
+def extract(archive, output, minor):
+    total = 0
+    seen = set()
+    with tarfile.open(archive, "r:gz") as source:
+        for member in source:
+            name = member.name.rstrip("/")
+            parts = name.split("/")
+            if (parts[0] != "python" or any(part in ("", ".", "..") for part in parts)
+                    or "\\" in name or name in seen or len(seen) >= 65536):
+                raise ValueError("invalid, repeated or excessive archive paths")
+            seen.add(name)
+            total += member.size
+            if total > MAX_BYTES:
+                raise ValueError("Python source exceeds the runtime byte budget")
+            if not member.isfile() or not selected(name, minor):
+                continue
+            destination = output / name
+            destination.parent.mkdir(parents=True, exist_ok=True)
+            with source.extractfile(member) as incoming, destination.open("xb") as outgoing:
+                shutil.copyfileobj(incoming, outgoing)
+            destination.chmod(0o755 if name == f"python/bin/python{minor}" else 0o644)
+
+
+def check_elf(path, readelf, target, baseline):
+    with path.open("rb") as stream:
+        if stream.read(4) != b"\x7fELF":
+            return
+    result = subprocess.run([str(readelf), "--wide", "--file-header", "--version-info", str(path)],
+                            check=True, capture_output=True, text=True, timeout=15)
+    if len(result.stdout) > 256 * 1024 or TARGETS[target][1] not in result.stdout:
+        raise ValueError("runtime ELF has the wrong target or excessive metadata")
+    if "Name: GLIBC_PRIVATE" in result.stdout:
+        raise ValueError("runtime ELF depends on private glibc interfaces")
+    versions = [tuple(map(int, value.split(".")))
+                for value in re.findall(r"Name: GLIBC_(\d+(?:\.\d+)+)", result.stdout)]
+    if versions and max(versions) > baseline:
+        raise ValueError(f"{path.name} requires GLIBC_{'.'.join(map(str, max(versions)))}")
+
+
+def wheel_tags(minor, target, baseline):
+    architecture = TARGETS[target][0]
+    oldest = 5 if architecture == "x86_64" else 17
+    if baseline[0] != 2 or baseline[1] < oldest:
+        raise ValueError("unsupported target glibc baseline")
+    aliases = {17: "manylinux2014", 12: "manylinux2010", 5: "manylinux1"}
+    platforms = []
+    for glibc_minor in range(baseline[1], oldest - 1, -1):
+        platforms.append(f"manylinux_2_{glibc_minor}_{architecture}")
+        if glibc_minor in aliases:
+            platforms.append(f"{aliases[glibc_minor]}_{architecture}")
+    platforms.append(f"linux_{architecture}")
+    python_minor = int(minor.split(".")[1])
+    abi = "cp" + minor.replace(".", "")
+    tags = [f"{abi}-{abi}-{platform}" for platform in platforms]
+    tags += [f"{abi}-abi3-{platform}" for platform in platforms]
+    tags += [f"{abi}-none-{platform}" for platform in platforms]
+    tags += [f"cp3{version}-abi3-{platform}"
+             for version in range(python_minor - 1, 1, -1) for platform in platforms]
+    interpreters = [f"py3{python_minor}", "py3"]
+    interpreters += [f"py3{version}" for version in range(python_minor - 1, -1, -1)]
+    tags += [f"{interpreter}-none-{platform}" for interpreter in interpreters for platform in platforms]
+    tags += [f"{abi}-none-any"]
+    tags += [f"{interpreter}-none-any" for interpreter in interpreters]
+    return tags
+
+
+def prepare(archive, expected, version, target, baseline, readelf, output, runner=None, sysroot=None):
+    if (not re.fullmatch(r"[0-9a-f]{64}", expected) or archive.stat().st_size > MAX_BYTES
+            or checksum(archive) != expected):
+        raise ValueError("Python archive SHA-256 or size mismatch")
+    if not re.fullmatch(r"3\.\d+\.\d+", version) or target not in TARGETS:
+        raise ValueError("invalid Python version or target")
+    if not readelf.is_absolute() or not readelf.is_file():
+        raise ValueError("readelf must be an explicit absolute executable")
+    if ((runner is None) != (sysroot is None)
+            or runner is not None and (not runner.is_absolute() or not runner.is_file()
+                                       or not sysroot.is_absolute() or not sysroot.is_dir())):
+        raise ValueError("cross probing requires an explicit QEMU runner and target sysroot")
+    minor = ".".join(version.split(".")[:2])
+    output.mkdir(mode=0o700)
+    try:
+        extract(archive, output, minor)
+        root = output / "python"
+        binary = root / "bin" / f"python{minor}"
+        if not binary.is_file() or not (root / "lib" / f"python{minor}" / "LICENSE.txt").is_file():
+            raise ValueError("Python archive lacks its interpreter or license")
+        for path in root.rglob("*"):
+            if path.is_file():
+                check_elf(path, readelf, target, baseline)
+        environment = {"HOME": str(root), "TMPDIR": str(output), "PATH": "/usr/bin:/bin"}
+        prefix = [str(runner), "-L", str(sysroot)] if runner else []
+        result = subprocess.run(prefix + [str(binary), "-I", "-S", "-B", "-c", PROBE],
+                                check=True, capture_output=True, text=True, env=environment, timeout=30)
+        probe = json.loads(result.stdout)
+        if (probe["version"] != version or probe["machine"] != TARGETS[target][0]
+                or Path(probe["prefix"]) != root or Path(probe["executable"]) != binary):
+            raise ValueError("Python runtime is not relocatable or has the wrong identity")
+        tags = wheel_tags(minor, target, baseline)
+        markers = {"implementation_name": "cpython", "implementation_version": version,
+                   "os_name": "posix", "platform_machine": TARGETS[target][0],
+                   "platform_python_implementation": "CPython", "platform_release": "",
+                   "platform_system": "Linux", "platform_version": "", "python_full_version": version,
+                   "python_version": minor, "sys_platform": "linux"}
+        descriptor = {"format": "dever-worker-runtime-v1", "ecosystem": "pip", "target": target,
+                      "executable": f"bin/python{minor}", "arguments": ["-I", "-S", "-B"],
+                      "python_markers": markers, "python_wheel_tags": tags,
+                      "python_extension_suffixes": probe["suffixes"]}
+        (root / "dever-runtime.json").write_text(json.dumps(descriptor, indent=2) + "\n")
+        files = [{"source": str(path.relative_to(output)), "path": str(path.relative_to(root)),
+                  "sha256": checksum(path)} for path in sorted(root.rglob("*")) if path.is_file()]
+        author = {"name": "cpython", "version": version, "python_markers": markers,
+                  "python_wheel_tags": tags, "files": files}
+        (output / "python-runtime.json").write_text(json.dumps(author, indent=2) + "\n")
+        return {"version": version, "target": target, "glibc_max": ".".join(map(str, baseline)),
+                "source_sha256": expected, "files": len(files)}
+    except BaseException:
+        shutil.rmtree(output)
+        raise
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Prepare verified portable Python inputs for native-release")
+    parser.add_argument("--archive", type=Path, required=True)
+    parser.add_argument("--sha256", required=True)
+    parser.add_argument("--version", required=True)
+    parser.add_argument("--target", choices=TARGETS, required=True)
+    parser.add_argument("--glibc-max", default="2.17")
+    parser.add_argument("--readelf", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--runner", type=Path, help="explicit QEMU user-mode executable for cross probing")
+    parser.add_argument("--sysroot", type=Path, help="target system libraries for the explicit QEMU runner")
+    arguments = parser.parse_args()
+    if not re.fullmatch(r"2\.\d+", arguments.glibc_max):
+        parser.error("glibc-max must be a 2.x baseline")
+    print(json.dumps(prepare(arguments.archive.resolve(), arguments.sha256, arguments.version,
+                             arguments.target, tuple(map(int, arguments.glibc_max.split("."))),
+                             arguments.readelf, arguments.output.absolute(), arguments.runner,
+                             arguments.sysroot), indent=2))
+
+
+if __name__ == "__main__":
+    main()

sdk/release-assets.py

@@ -12,11 +12,35 @@ import shutil
 import subprocess
 import tarfile
 import tempfile
+import zipfile
 
 
 MAX_PACKAGE = 2 * 1024**3
 
 
+def verify_complete(manifest, editor):
+    groups = packages(manifest)
+    expected = {f"ext-runtime-{ecosystem}-{target}"
+                for ecosystem in ("pip", "npm", "go")
+                for target in ("linux-x86_64", "linux-aarch64")}
+    expected.update(f"ext-build-{ecosystem}-{manifest['platform']}" for ecosystem in ("pip", "npm"))
+    expected.update(f"ext-target-{target}" for target in ("linux-x86_64", "linux-aarch64")
+                    if target != manifest["platform"])
+    missing = expected - groups.keys()
+    if missing:
+        raise ValueError("complete release is missing: " + ", ".join(sorted(missing)))
+    if editor is None or editor.is_symlink() or editor.stat().st_size > 16 * 1024**2:
+        raise ValueError("complete release requires a bounded editor VSIX")
+    with zipfile.ZipFile(editor) as archive:
+        info = archive.getinfo("extension/package.json")
+        if info.file_size > 64 * 1024:
+            raise ValueError("editor metadata exceeds limit")
+        package = json.loads(archive.read(info))
+        if (package.get("name") != "dever-language" or package.get("publisher") != "shemic"
+                or package.get("version") != manifest["version"]):
+            raise ValueError("editor identity/version does not match the release")
+
+
 def extension_suffix(extension):
     kind = extension["kind"]
     target = extension["target"]
@@ -126,7 +150,7 @@ def compress(release, artifacts, destination, zstd):
             process.stderr.close()
 
 
-def create(release, output, zstd):
+def create(release, output, zstd, complete=True, editor=None):
     release = release.absolute()
     output = output.absolute()
     if release.is_symlink() or not release.is_dir():
@@ -142,6 +166,8 @@ def create(release, output, zstd):
     if len(manifest_bytes) > 2 * 1024**2:
         raise ValueError("release metadata exceeds limit")
     manifest = json.loads(manifest_bytes)
+    if complete:
+        verify_complete(manifest, editor)
     groups = packages(manifest)
     verify_artifacts(release, groups)
     with tempfile.TemporaryDirectory(prefix=".dever-release-assets-", dir=output.parent) as scratch:
@@ -158,6 +184,8 @@ def create(release, output, zstd):
                 blob = prepared / f"{stem}.blob-{artifact['sha256']}"
                 if not blob.exists():
                     shutil.copyfile(release / artifact["path"], blob)
+        if complete:
+            shutil.copyfile(editor, prepared / f"dever-language-{manifest['version']}.vsix")
         publish(prepared, output)
     return sorted(path.name for path in output.iterdir())
 
@@ -179,11 +207,15 @@ def main():
     parser.add_argument("release", type=Path)
     parser.add_argument("--output", required=True, type=Path)
     parser.add_argument("--zstd", required=True, type=Path, help="explicit author-side zstd executable")
+    parser.add_argument("--partial-test-fixture", action="store_true", help="allow incomplete private test fixtures, not public releases")
+    parser.add_argument("--editor-vsix", type=Path, help="matching editor package required for public releases")
     args = parser.parse_args()
+    if args.partial_test_fixture and args.editor_vsix:
+        parser.error("partial test fixtures must not include a public editor package")
     try:
-        for name in create(args.release, args.output, args.zstd):
+        for name in create(args.release, args.output, args.zstd, not args.partial_test_fixture, args.editor_vsix):
             print(name)
-    except (OSError, ValueError, KeyError, TypeError, tarfile.TarError) as error:
+    except (OSError, ValueError, KeyError, TypeError, tarfile.TarError, zipfile.BadZipFile) as error:
         parser.exit(1, f"cannot prepare release assets: {error}\n")