feat: publish Dever core with project scaffolding and versioned AI skills
This commit is contained in:
Antigravity committed
1 parent
70fa73cdc2
commit
ac458ffa7f
960 files changed
+221319
-1163
No files matched your search
@@ -0,0 +1,27 @@
|
||||
# External Worker SDKs
|
||||
|
||||
Compiler authors: the private [native release workflow](native-release.md) builds and signs Dever runtime packs. It is separate from these external-language Worker SDKs and from application configuration.
|
||||
|
||||
These are stdio Worker libraries for Python, JavaScript (Node), and Go. The compiler remains the sole owner of Port schemas. Each checked external Adapter exposes `ExternalWorkerContract::sdk_manifest()`, including the exact identity and recursive setting, input, output and declared-error types. The managed Python and JavaScript runners package this manifest and bind each Port operation to a same-named function exported by the Adapter entry; JavaScript accepts ESM named exports or CommonJS `module.exports` properties. Application code does not maintain a second registration table. Python uses `Worker.from_manifest(manifest, handlers)`, JavaScript uses `Worker.fromManifest(manifest, handlers)`, and Go uses `NewFromManifest(manifest)` followed by `RegisterHandlers(handlers)`. These paths decode inputs and validate results and declared business-error payloads from the same compiler-generated contract. The SDK checks the identity during `hello` and never adopts a schema supplied by the supervisor.
|
||||
|
||||
The lower-level `Contract`/`Worker` registration API remains available for protocol fixtures. The setting decoder runs before `ready`; it receives only the `hello.setting` value. Handlers return JSON-compatible values or a declared `BusinessError`. Python handlers are async; JavaScript handlers receive an `AbortSignal`; Go handlers receive a `context.Context`. A cooperative cancellation receives one `dever.cancelled` protocol error and discards its handler result. If a handler ignores cancellation for one second, the Worker fails and admits no further calls. The supervisor owns the Worker process and its stdin lifetime; an in-process Go `Serve` caller must close or unblock its own `io.Reader` when returning.
|
||||
|
||||
The three `example` entries use the same fixture identity and contract to demonstrate the lower-level API. `fixture-schema-v1` and `test/external-sdk/typed_contract.json` are test data, not production Port hashes. Worker stdout is reserved for 4-byte big-endian length-prefixed UTF-8 JSON frames. Diagnostics go to stderr. The libraries use no external packages and do not read environment variables or `PATH`. A managed Worker entry reads only its explicitly packaged manifest path. Managed Python packs require fixed `-I -S -B` flags; the runner imports its packaged SDK before exposing Adapter source and dependencies on `sys.path`.
|
||||
|
||||
Run the opt-in host SDK check with explicit absolute runtime paths, for example:
|
||||
|
||||
```text
|
||||
/usr/bin/python3 -B test/external-sdk/check.py --python /usr/bin/python3 --node /absolute/path/to/node --go /absolute/path/to/go
|
||||
```
|
||||
|
||||
Omitted runtime paths report `unavailable`; the script never discovers language runtimes through `PATH`. It builds the Go fixtures in a temporary offline cache. This check exercises both the lower-level and compiler-manifest-shaped typed paths on host interpreters. Managed runtime packaging and execution on a machine without host language runtimes require separate acceptance.
|
||||
|
||||
Managed dependencies use `dever-lock-v5`. Python extras and selected wheel METADATA, npm nested instances/peers/optional/bundled dependencies, and authenticated Go module evidence are part of the exact lock. Native Python wheels install into the packaged interpreter prefix and must satisfy its signed tags, extension ABI and loader closure.
|
||||
|
||||
Cross builds select the application platform explicitly: `dever lib update <project> --target linux-aarch64`, followed by `dever build <project> --target linux-aarch64 --output <new-file>`. Worker source and Port contracts stay unchanged. Locks, interpreters, native dependencies and sandbox assets must match that target. A Go runtime pack additionally declares `build.host` for its compiler/linker/analyzer; `build.goos/goarch` and standard-library object headers describe the output architecture. The tools run with a cleared environment and must have the correct target baked in. Missing target assets are errors, with no host runtime fallback. `run` and ordinary tests remain host operations.
|
||||
|
||||
Only explicit Lib preparation may build Python sdists or run npm registry installation hooks. The shared build owner uses signed compiler/header/sysroot/frontend packs in a Linux sandbox with no network, immutable inputs and owned writable scratch space. Python PEP 517 build requirements have their own exact lock; each hook has a fresh process, and the parent retains the metadata snapshot used to validate the final wheel. npm uses its packaged lifecycle runner for `preinstall`, `install` and `postinstall`, including dependency bins and node-gyp. Missing tools do not fall back to host `PATH`.
|
||||
|
||||
Build receipts bind sources, target runtimes, tools, frontend, dependency graph and final output hashes. Worker preparation replays those outputs offline without running installation hooks. Every npm `.node` output is classified by target/loader checks and an isolated Node load probe; only admitted addons enter the immutable Worker tree, including trees used by threads or child processes. Rejected platform/ABI variants remain in the locked archive. Runtime native dependencies must come from that tree or the signed OS library set, never the compiler sysroot. These rules do not change the Port protocol or require an application-side registration layer.
|
||||
|
||||
A generated addon that fails admission fails the build; a pure JavaScript fallback is not evidence of successful native compilation. Build subprocesses have a five-minute deadline and separate 16 MiB stdout/stderr limits. Scratch usage is checked every 250 ms against 512 MiB and 65,536 entries, with owned-process cleanup on failure. This periodic check is not a kernel disk quota and may observe overshoot between checks. The final Worker installation remains limited to 256 MiB and 65,536 entries.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Based on AppArmor 4.0 extras/bwrap-userns-restrict (GPL-2.0-only).
|
||||
# Only signed static helpers published by the root-owned Dever installer attach.
|
||||
# The exec transition is stacked under no_new_privs; child capabilities are denied.
|
||||
# On restricted hosts, kernel.apparmor_restrict_unprivileged_unconfined=1 is
|
||||
# mandatory BEFORE installation/loading: attachment paths do not constrain an
|
||||
# explicit change_profile by an unconfined caller. Never enable this policy on
|
||||
# such a host with that flag unset. Rollback removes this policy before restoring
|
||||
# the previous strict flag. Dever never changes host sysctls automatically.
|
||||
abi <abi/4.0>,
|
||||
|
||||
@{DEVER_SANDBOX_HASH} = [0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]
|
||||
|
||||
profile dever-bwrap /opt/dever/sandbox/@{DEVER_SANDBOX_HASH}/bwrap flags=(attach_disconnected,mediate_deleted) {
|
||||
allow capability,
|
||||
allow file rwlkm /{**,},
|
||||
allow network,
|
||||
allow unix,
|
||||
allow ptrace,
|
||||
allow signal,
|
||||
allow mqueue,
|
||||
allow io_uring,
|
||||
allow userns,
|
||||
allow mount,
|
||||
allow umount,
|
||||
allow pivot_root,
|
||||
allow dbus,
|
||||
allow px /** -> dever-bwrap//&dever-bwrap-child,
|
||||
}
|
||||
|
||||
profile dever-bwrap-child flags=(attach_disconnected,mediate_deleted) {
|
||||
allow file rwlkm /{**,},
|
||||
allow network,
|
||||
allow unix,
|
||||
allow ptrace,
|
||||
allow signal,
|
||||
allow mqueue,
|
||||
allow io_uring,
|
||||
allow userns,
|
||||
allow mount,
|
||||
allow umount,
|
||||
allow pivot_root,
|
||||
allow dbus,
|
||||
allow pix /** -> &dever-bwrap-child,
|
||||
audit deny capability,
|
||||
}
|
||||
@@ -0,0 +1,699 @@
|
||||
// Package component implements the Dever external Worker process protocol.
|
||||
// The compiler owns schema and typed Port codecs; callers bind its identity here.
|
||||
package component
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"math"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
const (
|
||||
Version = "dever-component-1"
|
||||
maxBytes = 16 * 1024 * 1024
|
||||
maxDepth = 64
|
||||
maxElements = 65536
|
||||
cancelGrace = time.Second
|
||||
)
|
||||
|
||||
var (
|
||||
decimalPattern = regexp.MustCompile(`^[+-]?(?:[0-9]+(?:\.[0-9]*)?|\.[0-9]+)(?:[eE][+-]?[0-9]+)?$`)
|
||||
uuidPattern = regexp.MustCompile(`(?i)^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}$`)
|
||||
timePattern = regexp.MustCompile(`^[0-9]{2}:[0-9]{2}:[0-9]{2}(?:\.[0-9]{3})?$`)
|
||||
datetimePattern = regexp.MustCompile(`^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(?:\.[0-9]{1,3})?(?:Z|[+-][0-9]{2}:[0-9]{2})$`)
|
||||
)
|
||||
|
||||
type Contract struct {
|
||||
Port, Schema, Adapter string
|
||||
Operations, Capabilities, Errors []string
|
||||
}
|
||||
|
||||
type TypeSchema struct {
|
||||
Type string `json:"type"`
|
||||
Name string `json:"name,omitempty"`
|
||||
Value *TypeSchema `json:"value,omitempty"`
|
||||
Fields map[string]TypeSchema `json:"fields,omitempty"`
|
||||
}
|
||||
|
||||
type Manifest struct {
|
||||
Port string `json:"port"`
|
||||
Schema string `json:"schema"`
|
||||
Adapter string `json:"adapter"`
|
||||
Operations []string `json:"operations"`
|
||||
Capabilities []string `json:"capabilities"`
|
||||
Setting *TypeSchema `json:"setting"`
|
||||
Inputs map[string]TypeSchema `json:"inputs"`
|
||||
Outputs map[string]TypeSchema `json:"outputs"`
|
||||
Errors map[string]TypeSchema `json:"errors"`
|
||||
}
|
||||
|
||||
type BusinessError struct {
|
||||
Identity string
|
||||
Payload any
|
||||
}
|
||||
|
||||
func (e BusinessError) Error() string { return e.Identity }
|
||||
|
||||
type handler[S any] func(context.Context, any, S) (any, error)
|
||||
|
||||
type Worker[S any] struct {
|
||||
contract Contract
|
||||
decodeSetting func(any) (S, error)
|
||||
handlers map[string]handler[S]
|
||||
inputTypes map[string]TypeSchema
|
||||
outputTypes map[string]TypeSchema
|
||||
errorTypes map[string]TypeSchema
|
||||
}
|
||||
|
||||
func NewFromManifest(manifest Manifest) (*Worker[any], error) {
|
||||
if len(manifest.Inputs) != len(manifest.Operations) || len(manifest.Outputs) != len(manifest.Operations) {
|
||||
return nil, errors.New("incomplete typed Worker contract")
|
||||
}
|
||||
for _, operation := range manifest.Operations {
|
||||
if _, ok := manifest.Inputs[operation]; !ok {
|
||||
return nil, errors.New("missing operation input contract")
|
||||
}
|
||||
if _, ok := manifest.Outputs[operation]; !ok {
|
||||
return nil, errors.New("missing operation output contract")
|
||||
}
|
||||
}
|
||||
identities := make([]string, 0, len(manifest.Errors))
|
||||
for identity := range manifest.Errors {
|
||||
identities = append(identities, identity)
|
||||
}
|
||||
sort.Strings(identities)
|
||||
worker, err := New(Contract{
|
||||
Port: manifest.Port, Schema: manifest.Schema, Adapter: manifest.Adapter,
|
||||
Operations: manifest.Operations, Capabilities: manifest.Capabilities, Errors: identities,
|
||||
}, func(value any) (any, error) {
|
||||
if manifest.Setting == nil {
|
||||
if value != nil {
|
||||
return nil, errors.New("unexpected setting")
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
return decodeTyped(value, *manifest.Setting)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
worker.inputTypes, worker.outputTypes, worker.errorTypes = manifest.Inputs, manifest.Outputs, manifest.Errors
|
||||
return worker, nil
|
||||
}
|
||||
|
||||
// ServeManifest binds the checked contract to compile-time selected handlers.
|
||||
func ServeManifest(manifestBytes []byte, handlers map[string]func(context.Context, any, any) (any, error), input io.Reader, output io.Writer) error {
|
||||
var manifest Manifest
|
||||
if err := json.Unmarshal(manifestBytes, &manifest); err != nil {
|
||||
return err
|
||||
}
|
||||
worker, err := NewFromManifest(manifest)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := worker.RegisterHandlers(handlers); err != nil {
|
||||
return err
|
||||
}
|
||||
return worker.Serve(input, output)
|
||||
}
|
||||
|
||||
func (w *Worker[S]) RegisterHandler(operation string, run func(context.Context, any, S) (any, error)) error {
|
||||
schema, ok := w.inputTypes[operation]
|
||||
if !ok || run == nil {
|
||||
return errors.New("operation is not in a typed Worker contract")
|
||||
}
|
||||
if w.handlers[operation] != nil {
|
||||
return errors.New("duplicate component operation")
|
||||
}
|
||||
w.handlers[operation] = func(ctx context.Context, raw any, setting S) (any, error) {
|
||||
payload, err := decodeTyped(raw, schema)
|
||||
if err != nil {
|
||||
return nil, errors.New("invalid component payload")
|
||||
}
|
||||
result, err := run(ctx, payload, setting)
|
||||
if err != nil {
|
||||
var business BusinessError
|
||||
if errors.As(err, &business) {
|
||||
typeSchema, declared := w.errorTypes[business.Identity]
|
||||
if !declared {
|
||||
return nil, errors.New("undeclared business error identity")
|
||||
}
|
||||
business.Payload, err = decodeTyped(business.Payload, typeSchema)
|
||||
if err != nil {
|
||||
return nil, errors.New("invalid business error payload")
|
||||
}
|
||||
return nil, business
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
return decodeTyped(result, w.outputTypes[operation])
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w *Worker[S]) RegisterHandlers(handlers map[string]func(context.Context, any, S) (any, error)) error {
|
||||
if len(handlers) != len(w.contract.Operations) {
|
||||
return errors.New("component operations are not fully registered")
|
||||
}
|
||||
for _, operation := range w.contract.Operations {
|
||||
if err := w.RegisterHandler(operation, handlers[operation]); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func decodeTyped(value any, schema TypeSchema) (any, error) {
|
||||
switch schema.Type {
|
||||
case "nullable":
|
||||
if value == nil {
|
||||
return nil, nil
|
||||
}
|
||||
if schema.Value == nil {
|
||||
return nil, errors.New("invalid nullable contract")
|
||||
}
|
||||
return decodeTyped(value, *schema.Value)
|
||||
case "list":
|
||||
values, ok := value.([]any)
|
||||
if !ok || schema.Value == nil {
|
||||
return nil, errors.New("expected list")
|
||||
}
|
||||
result := make([]any, len(values))
|
||||
for index, item := range values {
|
||||
decoded, err := decodeTyped(item, *schema.Value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result[index] = decoded
|
||||
}
|
||||
return result, nil
|
||||
case "record":
|
||||
object, ok := value.(map[string]any)
|
||||
if !ok {
|
||||
return nil, errors.New("expected record")
|
||||
}
|
||||
for name := range object {
|
||||
if _, ok := schema.Fields[name]; !ok {
|
||||
return nil, errors.New("unknown record field")
|
||||
}
|
||||
}
|
||||
result := make(map[string]any, len(schema.Fields))
|
||||
for name, field := range schema.Fields {
|
||||
entry, exists := object[name]
|
||||
if !exists && field.Type != "nullable" {
|
||||
return nil, errors.New("missing record field")
|
||||
}
|
||||
decoded, err := decodeTyped(entry, field)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result[name] = decoded
|
||||
}
|
||||
return result, nil
|
||||
case "int64", "duration", "model_id":
|
||||
switch number := value.(type) {
|
||||
case json.Number:
|
||||
if _, err := number.Int64(); err == nil {
|
||||
return number, nil
|
||||
}
|
||||
case int64:
|
||||
return number, nil
|
||||
case int:
|
||||
return int64(number), nil
|
||||
}
|
||||
case "float64":
|
||||
switch number := value.(type) {
|
||||
case json.Number:
|
||||
parsed, err := number.Float64()
|
||||
if err == nil && !math.IsInf(parsed, 0) && !math.IsNaN(parsed) {
|
||||
return number, nil
|
||||
}
|
||||
case float64:
|
||||
if !math.IsInf(number, 0) && !math.IsNaN(number) {
|
||||
return number, nil
|
||||
}
|
||||
}
|
||||
case "bool":
|
||||
if boolean, ok := value.(bool); ok {
|
||||
return boolean, nil
|
||||
}
|
||||
case "json":
|
||||
return value, nil
|
||||
case "text", "id", "secret", "decimal", "uuid", "datetime", "date", "time":
|
||||
if text, ok := value.(string); ok {
|
||||
switch schema.Type {
|
||||
case "decimal":
|
||||
if !decimalPattern.MatchString(text) {
|
||||
return nil, errors.New("invalid decimal")
|
||||
}
|
||||
case "uuid":
|
||||
if !uuidPattern.MatchString(text) {
|
||||
return nil, errors.New("invalid UUID")
|
||||
}
|
||||
case "date":
|
||||
if _, err := time.Parse("2006-01-02", text); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
case "time":
|
||||
if !timePattern.MatchString(text) {
|
||||
return nil, errors.New("invalid Time")
|
||||
}
|
||||
layout := "15:04:05"
|
||||
if len(text) == 12 {
|
||||
layout = "15:04:05.000"
|
||||
}
|
||||
if _, err := time.Parse(layout, text); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
case "datetime":
|
||||
if !datetimePattern.MatchString(text) {
|
||||
return nil, errors.New("invalid DateTime")
|
||||
}
|
||||
parsed, err := time.Parse(time.RFC3339Nano, text)
|
||||
if err != nil || parsed.Nanosecond()%1_000_000 != 0 {
|
||||
return nil, errors.New("invalid DateTime")
|
||||
}
|
||||
}
|
||||
return text, nil
|
||||
}
|
||||
}
|
||||
return nil, errors.New("invalid typed Worker value")
|
||||
}
|
||||
|
||||
func New[S any](contract Contract, decodeSetting func(any) (S, error)) (*Worker[S], error) {
|
||||
if contract.Port == "" || contract.Schema == "" || contract.Adapter == "" || len(contract.Operations) == 0 || decodeSetting == nil || !unique(contract.Operations) || !unique(contract.Capabilities) || !unique(contract.Errors) {
|
||||
return nil, errors.New("incomplete component contract")
|
||||
}
|
||||
if contract.Capabilities == nil {
|
||||
contract.Capabilities = []string{}
|
||||
}
|
||||
if contract.Errors == nil {
|
||||
contract.Errors = []string{}
|
||||
}
|
||||
contract.Operations = append([]string(nil), contract.Operations...)
|
||||
contract.Capabilities = append([]string{}, contract.Capabilities...)
|
||||
contract.Errors = append([]string{}, contract.Errors...)
|
||||
return &Worker[S]{contract: contract, decodeSetting: decodeSetting, handlers: make(map[string]handler[S])}, nil
|
||||
}
|
||||
|
||||
func unique(values []string) bool {
|
||||
seen := make(map[string]bool, len(values))
|
||||
for _, value := range values {
|
||||
if value == "" || seen[value] {
|
||||
return false
|
||||
}
|
||||
seen[value] = true
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// RegisterTyped binds a compiler-declared operation to its application codec.
|
||||
func RegisterTyped[S, P any](worker *Worker[S], operation string, decode func(any) (P, error), run func(context.Context, P, S) (any, error)) error {
|
||||
if !contains(worker.contract.Operations, operation) || worker.handlers[operation] != nil || decode == nil || run == nil {
|
||||
return errors.New("unknown or duplicate component operation")
|
||||
}
|
||||
worker.handlers[operation] = func(ctx context.Context, raw any, setting S) (any, error) {
|
||||
payload, err := decode(raw)
|
||||
if err != nil {
|
||||
return nil, errors.New("invalid component payload")
|
||||
}
|
||||
return run(ctx, payload, setting)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func contains(values []string, target string) bool {
|
||||
for _, value := range values {
|
||||
if value == target {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
type event struct {
|
||||
message map[string]any
|
||||
err error
|
||||
}
|
||||
|
||||
type result struct {
|
||||
id int64
|
||||
payload any
|
||||
err error
|
||||
}
|
||||
|
||||
func (w *Worker[S]) Serve(input io.Reader, output io.Writer) error {
|
||||
if len(w.handlers) != len(w.contract.Operations) {
|
||||
return errors.New("component operations are not fully registered")
|
||||
}
|
||||
messages := make(chan event, 1)
|
||||
done := make(chan struct{})
|
||||
defer close(done)
|
||||
go func() {
|
||||
for {
|
||||
message, err := readFrame(input)
|
||||
select {
|
||||
case messages <- event{message, err}:
|
||||
case <-done:
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
first := <-messages
|
||||
if first.err != nil {
|
||||
return first.err
|
||||
}
|
||||
setting, err := w.handshake(first.message, output)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
results := make(chan result, 1)
|
||||
var cancel context.CancelFunc
|
||||
var active int64
|
||||
var activeDone chan struct{}
|
||||
defer func() {
|
||||
if active != 0 {
|
||||
_ = stopHandler(cancel, activeDone)
|
||||
}
|
||||
}()
|
||||
nextID := int64(1)
|
||||
lastCancelled := false
|
||||
for {
|
||||
select {
|
||||
case received := <-messages:
|
||||
if received.err != nil {
|
||||
return received.err
|
||||
}
|
||||
message := received.message
|
||||
kind, ok := message["kind"].(string)
|
||||
if !ok {
|
||||
return errors.New("invalid component kind")
|
||||
}
|
||||
switch kind {
|
||||
case "health":
|
||||
if !exact(message, "kind", "id") || requestID(message["id"]) != 0 {
|
||||
return errors.New("invalid component health")
|
||||
}
|
||||
if err := writeFrame(output, message); err != nil {
|
||||
return err
|
||||
}
|
||||
case "call":
|
||||
if !exact(message, "kind", "id", "operation", "payload") {
|
||||
return errors.New("invalid component call")
|
||||
}
|
||||
id := requestID(message["id"])
|
||||
operation, ok := message["operation"].(string)
|
||||
run := w.handlers[operation]
|
||||
if id != nextID || active != 0 || !ok || run == nil {
|
||||
return errors.New("invalid component call")
|
||||
}
|
||||
nextID++
|
||||
ctx, stop := context.WithCancel(context.Background())
|
||||
cancel, active = stop, id
|
||||
activeDone = make(chan struct{})
|
||||
payload := message["payload"]
|
||||
go func(finished chan struct{}) {
|
||||
defer close(finished)
|
||||
value, err := run(ctx, payload, setting)
|
||||
select {
|
||||
case results <- result{id, value, err}:
|
||||
case <-ctx.Done():
|
||||
case <-done:
|
||||
}
|
||||
}(activeDone)
|
||||
case "cancel":
|
||||
if !exact(message, "kind", "id") {
|
||||
return errors.New("invalid component cancel")
|
||||
}
|
||||
id := requestID(message["id"])
|
||||
if id <= 0 {
|
||||
return errors.New("invalid component cancel")
|
||||
}
|
||||
if id == active {
|
||||
if err := stopHandler(cancel, activeDone); err != nil {
|
||||
active = 0
|
||||
activeDone = nil
|
||||
return err
|
||||
}
|
||||
active = 0
|
||||
activeDone = nil
|
||||
lastCancelled = true
|
||||
if err := writeFrame(output, map[string]any{"kind": "error", "id": id, "error": "dever.cancelled", "payload": nil}); err != nil {
|
||||
return err
|
||||
}
|
||||
} else if id != nextID-1 || lastCancelled {
|
||||
return errors.New("unknown component cancel id")
|
||||
}
|
||||
case "shutdown":
|
||||
if !exact(message, "kind") {
|
||||
return errors.New("invalid component shutdown")
|
||||
}
|
||||
if active != 0 {
|
||||
if err := stopHandler(cancel, activeDone); err != nil {
|
||||
active = 0
|
||||
activeDone = nil
|
||||
return err
|
||||
}
|
||||
active = 0
|
||||
activeDone = nil
|
||||
}
|
||||
return writeFrame(output, map[string]any{"kind": "shutdown"})
|
||||
default:
|
||||
return errors.New("unknown component message")
|
||||
}
|
||||
case completed := <-results:
|
||||
if completed.id != active {
|
||||
continue
|
||||
}
|
||||
cancel()
|
||||
active = 0
|
||||
activeDone = nil
|
||||
lastCancelled = false
|
||||
if completed.err == nil {
|
||||
if err := writeFrame(output, map[string]any{"kind": "result", "id": completed.id, "payload": completed.payload}); err != nil {
|
||||
return err
|
||||
}
|
||||
continue
|
||||
}
|
||||
var business BusinessError
|
||||
if !errors.As(completed.err, &business) || !contains(w.contract.Errors, business.Identity) {
|
||||
return errors.New("component handler failed")
|
||||
}
|
||||
if err := writeFrame(output, map[string]any{"kind": "error", "id": completed.id, "error": business.Identity, "payload": business.Payload}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func stopHandler(cancel context.CancelFunc, finished <-chan struct{}) error {
|
||||
cancel()
|
||||
timer := time.NewTimer(cancelGrace)
|
||||
defer timer.Stop()
|
||||
select {
|
||||
case <-finished:
|
||||
return nil
|
||||
case <-timer.C:
|
||||
return errors.New("component handler did not stop after cancellation")
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Worker[S]) handshake(hello map[string]any, output io.Writer) (S, error) {
|
||||
var zero S
|
||||
if !exact(hello, "kind", "version", "port", "schema", "adapter", "capabilities", "operations", "setting") || hello["kind"] != "hello" || hello["version"] != Version || hello["port"] != w.contract.Port || hello["schema"] != w.contract.Schema || hello["adapter"] != w.contract.Adapter || !sameNames(hello["capabilities"], w.contract.Capabilities) || !sameNames(hello["operations"], w.contract.Operations) {
|
||||
return zero, errors.New("component handshake does not match registered contract")
|
||||
}
|
||||
setting, err := w.decodeSetting(hello["setting"])
|
||||
if err != nil {
|
||||
return zero, errors.New("invalid component setting")
|
||||
}
|
||||
if err := writeFrame(output, map[string]any{"kind": "ready", "version": Version, "port": w.contract.Port, "schema": w.contract.Schema, "adapter": w.contract.Adapter, "capabilities": w.contract.Capabilities, "operations": w.contract.Operations}); err != nil {
|
||||
return zero, err
|
||||
}
|
||||
return setting, nil
|
||||
}
|
||||
|
||||
func sameNames(raw any, expected []string) bool {
|
||||
values, ok := raw.([]any)
|
||||
if !ok || len(values) != len(expected) {
|
||||
return false
|
||||
}
|
||||
for i, value := range values {
|
||||
if value != expected[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func exact(message map[string]any, names ...string) bool {
|
||||
if len(message) != len(names) {
|
||||
return false
|
||||
}
|
||||
for _, name := range names {
|
||||
if _, ok := message[name]; !ok {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func requestID(value any) int64 {
|
||||
number, ok := value.(json.Number)
|
||||
if !ok {
|
||||
return -1
|
||||
}
|
||||
id, err := number.Int64()
|
||||
if err != nil || id < 0 {
|
||||
return -1
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func readFrame(input io.Reader) (map[string]any, error) {
|
||||
var header [4]byte
|
||||
if _, err := io.ReadFull(input, header[:]); err != nil {
|
||||
return nil, errors.New("incomplete component frame")
|
||||
}
|
||||
size := binary.BigEndian.Uint32(header[:])
|
||||
if size == 0 || size > maxBytes {
|
||||
return nil, errors.New("component frame exceeds byte limit")
|
||||
}
|
||||
body := make([]byte, size)
|
||||
if _, err := io.ReadFull(input, body); err != nil {
|
||||
return nil, errors.New("incomplete component frame")
|
||||
}
|
||||
return parseJSON(body)
|
||||
}
|
||||
|
||||
func writeFrame(output io.Writer, message map[string]any) error {
|
||||
body, err := json.Marshal(message)
|
||||
if err != nil {
|
||||
return errors.New("invalid handler response")
|
||||
}
|
||||
if len(body) == 0 || len(body) > maxBytes {
|
||||
return errors.New("component frame exceeds byte limit")
|
||||
}
|
||||
if _, err := parseJSON(body); err != nil {
|
||||
return err
|
||||
}
|
||||
var header [4]byte
|
||||
binary.BigEndian.PutUint32(header[:], uint32(len(body)))
|
||||
if err := writeAll(output, header[:]); err != nil {
|
||||
return err
|
||||
}
|
||||
return writeAll(output, body)
|
||||
}
|
||||
|
||||
func writeAll(output io.Writer, data []byte) error {
|
||||
for len(data) > 0 {
|
||||
count, err := output.Write(data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if count == 0 {
|
||||
return io.ErrShortWrite
|
||||
}
|
||||
data = data[count:]
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func parseJSON(body []byte) (map[string]any, error) {
|
||||
if !utf8.Valid(body) {
|
||||
return nil, errors.New("component frame is not UTF-8")
|
||||
}
|
||||
decoder := json.NewDecoder(bytes.NewReader(body))
|
||||
decoder.UseNumber()
|
||||
remaining := maxElements
|
||||
value, err := parseValue(decoder, 0, &remaining)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, err := decoder.Token(); err != io.EOF {
|
||||
return nil, errors.New("invalid component JSON")
|
||||
}
|
||||
object, ok := value.(map[string]any)
|
||||
if !ok {
|
||||
return nil, errors.New("component message must be an object")
|
||||
}
|
||||
return object, nil
|
||||
}
|
||||
|
||||
func parseValue(decoder *json.Decoder, depth int, remaining *int) (any, error) {
|
||||
if depth > maxDepth || *remaining <= 0 {
|
||||
return nil, errors.New("wire JSON exceeds limit")
|
||||
}
|
||||
*remaining--
|
||||
token, err := decoder.Token()
|
||||
if err != nil {
|
||||
return nil, errors.New("invalid component JSON")
|
||||
}
|
||||
delim, ok := token.(json.Delim)
|
||||
if !ok {
|
||||
if number, isNumber := token.(json.Number); isNumber && strings.ContainsAny(string(number), ".eE") {
|
||||
value, err := strconv.ParseFloat(string(number), 64)
|
||||
if err != nil || math.IsInf(value, 0) || math.IsNaN(value) {
|
||||
return nil, errors.New("non-finite wire Float")
|
||||
}
|
||||
}
|
||||
return token, nil
|
||||
}
|
||||
if depth == maxDepth {
|
||||
return nil, errors.New("wire JSON exceeds depth limit")
|
||||
}
|
||||
switch delim {
|
||||
case '{':
|
||||
object := make(map[string]any)
|
||||
for decoder.More() {
|
||||
key, err := decoder.Token()
|
||||
if err != nil {
|
||||
return nil, errors.New("invalid component JSON")
|
||||
}
|
||||
name, ok := key.(string)
|
||||
if !ok {
|
||||
return nil, errors.New("invalid component JSON")
|
||||
}
|
||||
if _, exists := object[name]; exists {
|
||||
return nil, errors.New("duplicate JSON field")
|
||||
}
|
||||
value, err := parseValue(decoder, depth+1, remaining)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
object[name] = value
|
||||
}
|
||||
if end, err := decoder.Token(); err != nil || end != json.Delim('}') {
|
||||
return nil, errors.New("invalid component JSON")
|
||||
}
|
||||
return object, nil
|
||||
case '[':
|
||||
array := make([]any, 0)
|
||||
for decoder.More() {
|
||||
value, err := parseValue(decoder, depth+1, remaining)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
array = append(array, value)
|
||||
}
|
||||
if end, err := decoder.Token(); err != nil || end != json.Delim(']') {
|
||||
return nil, errors.New("invalid component JSON")
|
||||
}
|
||||
return array, nil
|
||||
default:
|
||||
return nil, errors.New("invalid component JSON")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
component "dever-component"
|
||||
)
|
||||
|
||||
type payload struct {
|
||||
Text string
|
||||
DelayMS int64
|
||||
}
|
||||
|
||||
func setting(raw any) (string, error) {
|
||||
object, ok := raw.(map[string]any)
|
||||
if !ok || len(object) != 1 {
|
||||
return "", errors.New("invalid setting")
|
||||
}
|
||||
value, ok := object["prefix"].(string)
|
||||
if !ok {
|
||||
return "", errors.New("invalid setting")
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
|
||||
func decode(raw any) (payload, error) {
|
||||
object, ok := raw.(map[string]any)
|
||||
if !ok || len(object) != 2 {
|
||||
return payload{}, errors.New("invalid payload")
|
||||
}
|
||||
text, ok := object["text"].(string)
|
||||
if !ok {
|
||||
return payload{}, errors.New("invalid payload")
|
||||
}
|
||||
delay, ok := object["delay_ms"].(interface{ Int64() (int64, error) })
|
||||
if !ok {
|
||||
return payload{}, errors.New("invalid payload")
|
||||
}
|
||||
milliseconds, err := delay.Int64()
|
||||
if err != nil || milliseconds < 0 || milliseconds > 10000 {
|
||||
return payload{}, errors.New("invalid payload")
|
||||
}
|
||||
return payload{Text: text, DelayMS: milliseconds}, nil
|
||||
}
|
||||
|
||||
func main() {
|
||||
worker, err := component.New(component.Contract{
|
||||
Port: "example.Text", Schema: "fixture-schema-v1", Adapter: "example.TextAdapter",
|
||||
Operations: []string{"text.render", "text.fail", "number.echo", "text.stubborn"}, Errors: []string{"example.rejected"},
|
||||
}, setting)
|
||||
if err == nil {
|
||||
err = component.RegisterTyped(worker, "text.render", decode, func(ctx context.Context, value payload, prefix string) (any, error) {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
case <-time.After(time.Duration(value.DelayMS) * time.Millisecond):
|
||||
return map[string]any{"value": prefix + value.Text}, nil
|
||||
}
|
||||
})
|
||||
}
|
||||
if err == nil {
|
||||
err = component.RegisterTyped(worker, "text.fail", decode, func(context.Context, payload, string) (any, error) {
|
||||
return nil, component.BusinessError{Identity: "example.rejected", Payload: map[string]any{"reason": "rejected"}}
|
||||
})
|
||||
}
|
||||
if err == nil {
|
||||
err = component.RegisterTyped(worker, "number.echo", func(raw any) (json.Number, error) {
|
||||
object, ok := raw.(map[string]any)
|
||||
if !ok || len(object) != 1 {
|
||||
return "", errors.New("invalid number")
|
||||
}
|
||||
number, ok := object["number"].(json.Number)
|
||||
if !ok {
|
||||
return "", errors.New("invalid number")
|
||||
}
|
||||
if _, err := number.Int64(); err != nil {
|
||||
return "", errors.New("invalid number")
|
||||
}
|
||||
return number, nil
|
||||
}, func(_ context.Context, value json.Number, _ string) (any, error) {
|
||||
return map[string]any{"number": value}, nil
|
||||
})
|
||||
}
|
||||
if err == nil {
|
||||
err = component.RegisterTyped(worker, "text.stubborn", decode, func(_ context.Context, value payload, prefix string) (any, error) {
|
||||
time.Sleep(5 * time.Second)
|
||||
return map[string]any{"value": prefix + value.Text}, nil
|
||||
})
|
||||
}
|
||||
if err == nil {
|
||||
err = worker.Serve(os.Stdin, os.Stdout)
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "component worker:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
module dever-component
|
||||
|
||||
go 1.21
|
||||
@@ -0,0 +1,368 @@
|
||||
const VERSION = 'dever-component-1';
|
||||
const MAX_BYTES = 16 * 1024 * 1024;
|
||||
const MAX_DEPTH = 64;
|
||||
const MAX_ELEMENTS = 65536;
|
||||
const CANCEL_GRACE_MS = 1000;
|
||||
const decoder = new TextDecoder('utf-8', { fatal: true });
|
||||
|
||||
export class ProtocolError extends Error {}
|
||||
|
||||
export class BusinessError extends Error {
|
||||
constructor(identity, payload = null) {
|
||||
super(identity);
|
||||
this.identity = identity;
|
||||
this.payload = payload;
|
||||
}
|
||||
}
|
||||
|
||||
// JSON.parse accepts duplicate object keys; this parser enforces the runtime's
|
||||
// duplicate, depth and value budgets before returning ordinary JS values.
|
||||
function parseJSON(text) {
|
||||
let offset = 0;
|
||||
let remaining = MAX_ELEMENTS;
|
||||
const fail = () => { throw new ProtocolError('invalid component JSON'); };
|
||||
const space = () => { while (offset < text.length && /[ \t\r\n]/.test(text[offset])) offset++; };
|
||||
function string() {
|
||||
const start = offset++;
|
||||
while (offset < text.length) {
|
||||
const char = text[offset++];
|
||||
if (char === '"') return JSON.parse(text.slice(start, offset));
|
||||
if (char === '\\') offset++;
|
||||
else if (char.charCodeAt(0) < 32) fail();
|
||||
}
|
||||
fail();
|
||||
}
|
||||
function value(depth) {
|
||||
if (depth > MAX_DEPTH || --remaining < 0) throw new ProtocolError('wire JSON exceeds limit');
|
||||
space();
|
||||
const char = text[offset];
|
||||
if (char === '"') return string();
|
||||
if (char === '[' || char === '{') {
|
||||
if (depth === MAX_DEPTH) throw new ProtocolError('wire JSON exceeds depth limit');
|
||||
offset++;
|
||||
const array = char === '[';
|
||||
const result = array ? [] : {};
|
||||
const seen = new Set();
|
||||
space();
|
||||
if (text[offset] === (array ? ']' : '}')) { offset++; return result; }
|
||||
for (;;) {
|
||||
if (array) result.push(value(depth + 1));
|
||||
else {
|
||||
if (text[offset] !== '"') fail();
|
||||
const key = string();
|
||||
if (seen.has(key)) throw new ProtocolError('duplicate JSON field');
|
||||
seen.add(key);
|
||||
space();
|
||||
if (text[offset++] !== ':') fail();
|
||||
Object.defineProperty(result, key, { value: value(depth + 1), enumerable: true, writable: true, configurable: true });
|
||||
}
|
||||
space();
|
||||
const end = array ? ']' : '}';
|
||||
if (text[offset] === end) { offset++; return result; }
|
||||
if (text[offset++] !== ',') fail();
|
||||
space();
|
||||
}
|
||||
}
|
||||
const match = /^(?:true|false|null|-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?)/.exec(text.slice(offset));
|
||||
if (!match) fail();
|
||||
offset += match[0].length;
|
||||
if (match[0] === 'true' || match[0] === 'false' || match[0] === 'null') return JSON.parse(match[0]);
|
||||
if (match[0] === '-0') return -0;
|
||||
if (/^-?(?:0|[1-9]\d*)$/.test(match[0])) {
|
||||
const integer = BigInt(match[0]);
|
||||
return integer <= BigInt(Number.MAX_SAFE_INTEGER) && integer >= BigInt(Number.MIN_SAFE_INTEGER) ? Number(integer) : integer;
|
||||
}
|
||||
const number = JSON.parse(match[0]);
|
||||
if (!Number.isFinite(number)) throw new ProtocolError('invalid JSON number');
|
||||
return number;
|
||||
}
|
||||
const result = value(0);
|
||||
space();
|
||||
if (offset !== text.length || result === null || Array.isArray(result) || typeof result !== 'object') fail();
|
||||
return result;
|
||||
}
|
||||
|
||||
function encodeJSON(value) {
|
||||
let remaining = MAX_ELEMENTS;
|
||||
function encode(current, depth) {
|
||||
if (depth > MAX_DEPTH || --remaining < 0) throw new ProtocolError('wire JSON exceeds limit');
|
||||
if (current === null || typeof current === 'boolean') return String(current);
|
||||
if (typeof current === 'string') return JSON.stringify(current);
|
||||
if (typeof current === 'bigint') return current.toString();
|
||||
if (typeof current === 'number') {
|
||||
if (!Number.isFinite(current)) throw new ProtocolError('invalid JSON number');
|
||||
return Object.is(current, -0) ? '-0' : String(current);
|
||||
}
|
||||
if (depth === MAX_DEPTH) throw new ProtocolError('wire JSON exceeds depth limit');
|
||||
if (Array.isArray(current)) return `[${current.map(item => encode(item, depth + 1)).join(',')}]`;
|
||||
if (current && typeof current === 'object') {
|
||||
return `{${Object.keys(current).map(key => `${JSON.stringify(key)}:${encode(current[key], depth + 1)}`).join(',')}}`;
|
||||
}
|
||||
throw new ProtocolError('invalid handler response');
|
||||
}
|
||||
return encode(value, 0);
|
||||
}
|
||||
|
||||
function exact(message, names) {
|
||||
const actual = Object.keys(message);
|
||||
if (actual.length !== names.length || names.some(name => !Object.hasOwn(message, name))) {
|
||||
throw new ProtocolError('component message has missing or extra fields');
|
||||
}
|
||||
}
|
||||
|
||||
function requestID(value) {
|
||||
if (!Number.isSafeInteger(value) || value < 0) throw new ProtocolError('invalid request id');
|
||||
return value;
|
||||
}
|
||||
|
||||
function names(values) {
|
||||
return Array.isArray(values) && values.every(value => typeof value === 'string' && value.length) && new Set(values).size === values.length;
|
||||
}
|
||||
|
||||
function typed(value, schema) {
|
||||
switch (schema.type) {
|
||||
case 'nullable': return value === null ? null : typed(value, schema.value);
|
||||
case 'list':
|
||||
if (!Array.isArray(value)) throw new ProtocolError('expected list');
|
||||
return value.map(item => typed(item, schema.value));
|
||||
case 'record': {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).some(name => !Object.hasOwn(schema.fields, name))) throw new ProtocolError('invalid record fields');
|
||||
const result = {};
|
||||
for (const [name, field] of Object.entries(schema.fields)) {
|
||||
if (!Object.hasOwn(value, name) && field.type !== 'nullable') throw new ProtocolError('missing required record field');
|
||||
Object.defineProperty(result, name, { value: Object.hasOwn(value, name) ? typed(value[name], field) : null, enumerable: true });
|
||||
}
|
||||
return result;
|
||||
}
|
||||
case 'int64': case 'duration': case 'model_id': {
|
||||
if ((typeof value !== 'bigint' && !Number.isSafeInteger(value)) || BigInt(value) < -(1n << 63n) || BigInt(value) >= (1n << 63n)) throw new ProtocolError('expected signed 64-bit integer');
|
||||
return value;
|
||||
}
|
||||
case 'float64':
|
||||
if (typeof value !== 'number' || !Number.isFinite(value)) throw new ProtocolError('expected finite float');
|
||||
return value;
|
||||
case 'bool':
|
||||
if (typeof value !== 'boolean') throw new ProtocolError('expected boolean');
|
||||
return value;
|
||||
case 'json': return value;
|
||||
case 'text': case 'id': case 'secret': case 'decimal': case 'uuid': case 'datetime': case 'date': case 'time':
|
||||
if (typeof value !== 'string') throw new ProtocolError('expected text');
|
||||
if (schema.type === 'decimal' && !/^[+-]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?$/.test(value)) throw new ProtocolError('invalid decimal');
|
||||
if (schema.type === 'uuid' && !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(value)) throw new ProtocolError('invalid UUID');
|
||||
if (schema.type === 'date' && (!/^\d{4}-\d{2}-\d{2}$/.test(value) || Number.isNaN(Date.parse(`${value}T00:00:00Z`)))) throw new ProtocolError('invalid Date');
|
||||
if (schema.type === 'time' && !/^\d{2}:\d{2}:\d{2}(?:\.\d{3})?$/.test(value)) throw new ProtocolError('invalid Time');
|
||||
if (schema.type === 'datetime' && (!/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,3})?(?:Z|[+-]\d{2}:\d{2})$/.test(value) || Number.isNaN(Date.parse(value)))) throw new ProtocolError('invalid DateTime');
|
||||
return value;
|
||||
default: throw new ProtocolError('unknown worker type');
|
||||
}
|
||||
}
|
||||
|
||||
async function stopHandler(current) {
|
||||
current.controller.abort();
|
||||
let timeout;
|
||||
try {
|
||||
await Promise.race([
|
||||
current.settled,
|
||||
new Promise((_, reject) => {
|
||||
timeout = setTimeout(() => reject(new ProtocolError('component handler did not stop after cancellation')), CANCEL_GRACE_MS);
|
||||
}),
|
||||
]);
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
}
|
||||
|
||||
class Frames {
|
||||
constructor() {
|
||||
this.buffer = Buffer.alloc(0);
|
||||
this.queue = [];
|
||||
this.queuedBytes = 0;
|
||||
this.waiter = null;
|
||||
this.failed = null;
|
||||
process.stdin.on('data', chunk => this.receive(chunk));
|
||||
process.stdin.on('end', () => this.fail(new ProtocolError('incomplete component frame')));
|
||||
process.stdin.on('error', error => this.fail(error));
|
||||
}
|
||||
|
||||
fail(error) {
|
||||
this.failed = error;
|
||||
if (this.waiter) { this.waiter.reject(error); this.waiter = null; }
|
||||
}
|
||||
|
||||
receive(chunk) {
|
||||
if (this.failed) return;
|
||||
this.buffer = Buffer.concat([this.buffer, chunk]);
|
||||
try {
|
||||
while (this.buffer.length >= 4) {
|
||||
const size = this.buffer.readUInt32BE(0);
|
||||
if (!size || size > MAX_BYTES) throw new ProtocolError('component frame exceeds byte limit');
|
||||
if (this.buffer.length < size + 4) break;
|
||||
const message = parseJSON(decoder.decode(this.buffer.subarray(4, size + 4)));
|
||||
this.buffer = this.buffer.subarray(size + 4);
|
||||
if (this.waiter) { this.waiter.resolve(message); this.waiter = null; }
|
||||
else {
|
||||
if (this.queue.length === 4 || this.queuedBytes + size > MAX_BYTES) throw new ProtocolError('component input queue exceeds limit');
|
||||
this.queue.push({ message, size });
|
||||
this.queuedBytes += size;
|
||||
}
|
||||
}
|
||||
} catch (error) { this.fail(error); }
|
||||
}
|
||||
|
||||
next() {
|
||||
if (this.failed) return Promise.reject(this.failed);
|
||||
if (this.queue.length) {
|
||||
const next = this.queue.shift();
|
||||
this.queuedBytes -= next.size;
|
||||
return Promise.resolve(next.message);
|
||||
}
|
||||
return new Promise((resolve, reject) => { this.waiter = { resolve, reject }; });
|
||||
}
|
||||
|
||||
async send(message) {
|
||||
const body = Buffer.from(encodeJSON(message));
|
||||
if (!body.length || body.length > MAX_BYTES) throw new ProtocolError('component frame exceeds byte limit');
|
||||
parseJSON(decoder.decode(body));
|
||||
const frame = Buffer.allocUnsafe(body.length + 4);
|
||||
frame.writeUInt32BE(body.length);
|
||||
body.copy(frame, 4);
|
||||
await new Promise((resolve, reject) => process.stdout.write(frame, error => error ? reject(error) : resolve()));
|
||||
}
|
||||
}
|
||||
|
||||
export class Worker {
|
||||
constructor(contract, decodeSetting) {
|
||||
if (!contract || ![contract.port, contract.schema, contract.adapter].every(value => typeof value === 'string' && value.length) || !names(contract.operations) || !contract.operations.length || !names(contract.capabilities ?? []) || !names(contract.errors ?? []) || typeof decodeSetting !== 'function') {
|
||||
throw new Error('incomplete component contract');
|
||||
}
|
||||
this.contract = Object.freeze({
|
||||
port: contract.port, schema: contract.schema, adapter: contract.adapter,
|
||||
operations: Object.freeze([...contract.operations]),
|
||||
capabilities: Object.freeze([...(contract.capabilities ?? [])]),
|
||||
errors: Object.freeze([...(contract.errors ?? [])]),
|
||||
});
|
||||
this.decodeSetting = decodeSetting;
|
||||
this.handlers = new Map();
|
||||
this.outputTypes = null;
|
||||
this.errorTypes = null;
|
||||
}
|
||||
|
||||
static fromManifest(manifest, handlers = null) {
|
||||
const operations = manifest.operations;
|
||||
if (!names(operations) || Object.keys(manifest.inputs).sort().join() !== [...operations].sort().join() || Object.keys(manifest.outputs).sort().join() !== [...operations].sort().join()) throw new Error('incomplete typed Worker contract');
|
||||
const worker = new Worker({
|
||||
port: manifest.port, schema: manifest.schema, adapter: manifest.adapter,
|
||||
operations, capabilities: manifest.capabilities, errors: Object.keys(manifest.errors),
|
||||
}, value => {
|
||||
if (manifest.setting !== null) return typed(value, manifest.setting);
|
||||
if (value !== null) throw new ProtocolError('unexpected setting');
|
||||
return null;
|
||||
});
|
||||
worker.inputTypes = manifest.inputs;
|
||||
worker.outputTypes = manifest.outputs;
|
||||
worker.errorTypes = manifest.errors;
|
||||
if (handlers !== null) {
|
||||
if (Object.keys(handlers).sort().join() !== [...operations].sort().join()) throw new Error('component operations are not fully registered');
|
||||
for (const [operation, handler] of Object.entries(handlers)) worker.registerHandler(operation, handler);
|
||||
}
|
||||
return worker;
|
||||
}
|
||||
|
||||
registerHandler(operation, handler) {
|
||||
if (!this.inputTypes || !Object.hasOwn(this.inputTypes, operation)) throw new Error('operation is not in a typed Worker contract');
|
||||
this.register(operation, value => typed(value, this.inputTypes[operation]), handler);
|
||||
}
|
||||
|
||||
register(operation, decodePayload, handler) {
|
||||
if (!this.contract.operations.includes(operation) || this.handlers.has(operation) || typeof decodePayload !== 'function' || typeof handler !== 'function') throw new Error('unknown or duplicate operation');
|
||||
this.handlers.set(operation, { decodePayload, handler });
|
||||
}
|
||||
|
||||
async serve() {
|
||||
const contract = this.contract;
|
||||
if (this.handlers.size !== contract.operations.length) throw new Error('component operations are not fully registered');
|
||||
const frames = new Frames();
|
||||
const hello = await frames.next();
|
||||
exact(hello, ['kind', 'version', 'port', 'schema', 'adapter', 'capabilities', 'operations', 'setting']);
|
||||
if (hello.kind !== 'hello' || hello.version !== VERSION || hello.port !== contract.port || hello.schema !== contract.schema || hello.adapter !== contract.adapter || JSON.stringify(hello.capabilities) !== JSON.stringify(contract.capabilities) || JSON.stringify(hello.operations) !== JSON.stringify(contract.operations)) {
|
||||
throw new ProtocolError('component handshake does not match registered contract');
|
||||
}
|
||||
let setting;
|
||||
try { setting = this.decodeSetting(hello.setting); }
|
||||
catch { throw new ProtocolError('invalid component setting'); }
|
||||
await frames.send({ kind: 'ready', version: VERSION, port: contract.port, schema: contract.schema, adapter: contract.adapter, capabilities: contract.capabilities, operations: contract.operations });
|
||||
|
||||
let active = null;
|
||||
let nextID = 1;
|
||||
let lastCancelled = false;
|
||||
for (;;) {
|
||||
const message = await frames.next();
|
||||
switch (message.kind) {
|
||||
case 'health':
|
||||
exact(message, ['kind', 'id']);
|
||||
if (requestID(message.id) !== 0) throw new ProtocolError('invalid health id');
|
||||
await frames.send(message);
|
||||
break;
|
||||
case 'call': {
|
||||
exact(message, ['kind', 'id', 'operation', 'payload']);
|
||||
const id = requestID(message.id);
|
||||
const operation = this.handlers.get(message.operation);
|
||||
if (id !== nextID++ || active || !operation) throw new ProtocolError('invalid component call');
|
||||
let payload;
|
||||
try { payload = operation.decodePayload(message.payload); }
|
||||
catch { throw new ProtocolError('invalid component payload'); }
|
||||
const current = { id, controller: new AbortController() };
|
||||
active = current;
|
||||
current.settled = Promise.resolve().then(() => operation.handler(payload, setting, current.controller.signal)).then(
|
||||
async result => {
|
||||
if (active !== current) return;
|
||||
active = null;
|
||||
lastCancelled = false;
|
||||
if (this.outputTypes) result = typed(result, this.outputTypes[message.operation]);
|
||||
await frames.send({ kind: 'result', id, payload: result });
|
||||
},
|
||||
async error => {
|
||||
if (active !== current) return;
|
||||
active = null;
|
||||
lastCancelled = false;
|
||||
if (!(error instanceof BusinessError) || !contract.errors.includes(error.identity)) throw new ProtocolError('component handler failed');
|
||||
if (this.errorTypes) error.payload = typed(error.payload, this.errorTypes[error.identity]);
|
||||
await frames.send({ kind: 'error', id, error: error.identity, payload: error.payload });
|
||||
},
|
||||
).catch(error => { frames.fail(error); });
|
||||
break;
|
||||
}
|
||||
case 'cancel':
|
||||
exact(message, ['kind', 'id']);
|
||||
requestID(message.id);
|
||||
if (active?.id === message.id) {
|
||||
const current = active;
|
||||
active = null;
|
||||
await stopHandler(current);
|
||||
lastCancelled = true;
|
||||
await frames.send({ kind: 'error', id: message.id, error: 'dever.cancelled', payload: null });
|
||||
} else if (message.id !== nextID - 1 || lastCancelled) throw new ProtocolError('unknown component cancel id');
|
||||
break;
|
||||
case 'shutdown':
|
||||
exact(message, ['kind']);
|
||||
if (active) {
|
||||
const current = active;
|
||||
active = null;
|
||||
await stopHandler(current);
|
||||
}
|
||||
await frames.send({ kind: 'shutdown' });
|
||||
return;
|
||||
default:
|
||||
throw new ProtocolError('unknown component message');
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export async function main(worker) {
|
||||
try { await worker.serve(); }
|
||||
catch (error) {
|
||||
console.error(`component worker: ${error instanceof ProtocolError ? error.message : 'internal failure'}`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
import { BusinessError, Worker, main } from './dever_component.mjs';
|
||||
|
||||
function setting(value) {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).join() !== 'prefix' || typeof value.prefix !== 'string') throw new Error('invalid setting');
|
||||
return value.prefix;
|
||||
}
|
||||
|
||||
function payload(value) {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).sort().join() !== 'delay_ms,text' || typeof value.text !== 'string' || !Number.isInteger(value.delay_ms) || value.delay_ms < 0 || value.delay_ms > 10000) throw new Error('invalid payload');
|
||||
return value;
|
||||
}
|
||||
|
||||
function number(value) {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).join() !== 'number' || (typeof value.number !== 'bigint' && !Number.isSafeInteger(value.number))) throw new Error('invalid number');
|
||||
return value.number;
|
||||
}
|
||||
|
||||
const worker = new Worker({ port: 'example.Text', schema: 'fixture-schema-v1', adapter: 'example.TextAdapter', operations: ['text.render', 'text.fail', 'number.echo', 'text.stubborn'], errors: ['example.rejected'] }, setting);
|
||||
worker.register('text.render', payload, async (value, prefix, signal) => {
|
||||
await new Promise((resolve, reject) => {
|
||||
const timer = setTimeout(resolve, value.delay_ms);
|
||||
signal.addEventListener('abort', () => { clearTimeout(timer); reject(new Error('cancelled')); }, { once: true });
|
||||
});
|
||||
return { value: prefix + value.text };
|
||||
});
|
||||
worker.register('text.fail', payload, async () => { throw new BusinessError('example.rejected', { reason: 'rejected' }); });
|
||||
worker.register('number.echo', number, async value => ({ number: value }));
|
||||
worker.register('text.stubborn', payload, async () => new Promise(() => {}));
|
||||
await main(worker);
|
||||
@@ -0,0 +1,269 @@
|
||||
# Native release authoring
|
||||
|
||||
This is the private compiler-author workflow. Application projects continue to use `dever run/build`; they do not need Cargo, a signing key or author settings. A Linux x86_64 compiler release can contain both native x86_64 and cross-compiled ARM64 application packs. The compiler and its LLVM libraries remain host binaries.
|
||||
|
||||
Build the four runtime archives separately, then assemble and sign a new directory. The maker never builds source, downloads dependencies, searches `PATH`, reads configuration from environment variables, or modifies its inputs. It reuses the runtime manifest and signed release formats consumed by `deverd` and `MachineManager`.
|
||||
|
||||
## Build inputs
|
||||
|
||||
Build the compiler with `cargo build --offline --locked -p dever-cli --bin dever`; its author input becomes the installed `dever-core`. Build the stable machine launcher with `--bin dever-launcher` and the daemon with `--bin deverd`. Use `target/debug/dever-launcher` for the bootstrap launcher input, which the maker publishes as `bootstrap/dever`. The installed public command remains `dever`; `dever-launcher` is only an author build name. Preserve this distinction when preparing release inputs.
|
||||
|
||||
Prepare the pinned Rust dependencies and compiler LLVM 18/LLD SDK explicitly. Use an explicitly selected Cargo/Rust toolchain for author builds. The root `runtime-pack` Cargo profile inherits the release optimization settings, uses one codegen unit and removes debug information while retaining archive symbols needed by LLD. It must not enable the bridge's `embedded` feature.
|
||||
|
||||
All profiles include `runtime-api,runtime-external`; their dependencies enable `runtime-abi`. Database drivers vary:
|
||||
|
||||
| Profile | Additional features | Saved archive |
|
||||
| --- | --- | --- |
|
||||
| base | none | `base.a` |
|
||||
| sqlite | `runtime-sqlite` | `sqlite.a` |
|
||||
| postgres | `runtime-postgres` | `postgres.a` |
|
||||
| both | `runtime-sqlite,runtime-postgres` | `both.a` |
|
||||
|
||||
For example, build base for Linux x86_64:
|
||||
|
||||
```sh
|
||||
cargo build --offline --locked -p dever-backend-bridge --lib \
|
||||
--profile runtime-pack --no-default-features \
|
||||
--features runtime-api,runtime-external \
|
||||
--target x86_64-unknown-linux-gnu --target-dir target/native-runtime-abi
|
||||
```
|
||||
|
||||
Copy `target/native-runtime-abi/x86_64-unknown-linux-gnu/runtime-pack/libdever_backend_bridge.a` to a new `base.a` input, then repeat with each row's features and save its archive before the next build. Share this one Cargo target directory; do not create four dependency caches. Do not copy the both archive into all four inputs or strip a hard-linked fixture in place. Do not use `strip --strip-all` on link inputs.
|
||||
|
||||
Prepare a matching compiler core, `libLLVM.so.18.1`, target CRT objects and genuine static libraries. Linux GNU link order is explicit: start objects, runtime, static libraries, end objects. Linker scripts and thin archives are rejected. Preserve the compiler version and the exact compiler/CRT/SDK versions and input hashes in build records; the maker does not infer feature content or compiler versions from filenames.
|
||||
|
||||
## Author settings
|
||||
|
||||
Application authors select `dever build <project-root> --target linux-aarch64 --output <new-file>`; omitted targets, `run`, and `test` use the host. This selects application output, not an ARM compiler installation. Missing target packs are errors. Product commands do not discover cross tools or download packs while building.
|
||||
|
||||
For an additional application target, add `native_targets.linux-aarch64` to the author settings. Its value has the same `profiles`, `start`, `libraries`, and `end` fields as the host inputs below. Build all four archives with Rust target `aarch64-unknown-linux-gnu` and an explicitly prepared cross linker/sysroot. Use ARM CRT and static libraries, never host files or linker scripts. Both standalone object and archive member ELF headers are checked against the declared target before publishing and on cache access.
|
||||
|
||||
The parallel maps `ecosystem_targets.linux-aarch64`, `sandbox_targets.linux-aarch64`, and `build_targets.linux-aarch64` use the corresponding host `runtimes`, `sandbox`, and `builds` shapes. Python/Node interpreters, native wheel/addon files, the sandbox guard, static bwrap, and its OS libraries must match the application target. Go is different: `build.host` identifies the host executable tools; `build.goos` and `build.goarch` identify the output and standard-library archives. The compiler/linker must produce that target with a cleared environment. The private fixture maker supports `go run test/go-managed/make_pack.go <go-binary> <new-pack> linux-aarch64`; it uses an author-only Go overlay to bake ARM defaults into x86 tools without changing the installed Go SDK.
|
||||
|
||||
Prepare project Libs explicitly with `dever lib update <project-root> --target linux-aarch64` (or `lib add` with the same option). The resulting lock and cached resources bind the selected target. Offline `build` verifies that binding. Pure Python wheels/npm packages and Go cross compilation use this path; Python source distributions or npm lifecycle builds require already prepared matching target outputs and complete build receipts. An x86 tool must never silently produce a native extension labeled ARM.
|
||||
|
||||
The opt-in ARM acceptance uses explicit inputs under `target/arm64-cross/`: four archives/CRT libraries, target ecosystem and sandbox packs, locked upstream dependencies, and private QEMU/Alpine kernel/initramfs files. `native_release::acceptance::cross` exports independent applications under `guest/apps/` after deleting its source and machine store. Run `python3 -B test/ecosystem-release/arm64/simulate.py` to rebuild only the owned guest images and execute Python, Node, Go and HTTP under the complete ARM kernel. It requires all success/failure/cleanup markers and writes serial logs plus a result JSON under `logs/`. The VM has no external network or system language installations. Its initramfs switches into a normal root mount so bubblewrap can use `pivot_root`. This is functional emulation, not physical ARM performance or a universal kernel compatibility result.
|
||||
|
||||
Use a separate author root, not a Dever application's configuration. Place its explicit input files and private Ed25519 PKCS#8 key below that root. `config/setting.json` has this shape; replace every illustrative digest with the file's actual lowercase SHA-256:
|
||||
|
||||
```json
|
||||
{
|
||||
"format": "dever-native-release-input-v1",
|
||||
"version": "0.1.0",
|
||||
"target": "x86_64-unknown-linux-gnu",
|
||||
"signing_key": "private/release.pk8",
|
||||
"core": {"source": "inputs/dever-core", "sha256": "<sha256>"},
|
||||
"skill": [
|
||||
{"source": "inputs/skills/SKILL.md", "path": "SKILL.md", "sha256": "<sha256>"},
|
||||
{"source": "inputs/skills/references/development.md", "path": "references/development.md", "sha256": "<sha256>"}
|
||||
],
|
||||
"core_libraries": [
|
||||
{"source": "inputs/libLLVM.so.18.1", "path": "libLLVM.so.18.1", "sha256": "<sha256>"},
|
||||
{"source": "inputs/libstdc++.so.6", "path": "libstdc++.so.6", "sha256": "<sha256>"}
|
||||
],
|
||||
"profiles": {
|
||||
"base": {"source": "inputs/base.a", "sha256": "<sha256>"},
|
||||
"sqlite": {"source": "inputs/sqlite.a", "sha256": "<sha256>"},
|
||||
"postgres": {"source": "inputs/postgres.a", "sha256": "<sha256>"},
|
||||
"both": {"source": "inputs/both.a", "sha256": "<sha256>"}
|
||||
},
|
||||
"start": [
|
||||
{"source": "inputs/crt1.o", "path": "crt1.o", "sha256": "<sha256>"},
|
||||
{"source": "inputs/crti.o", "path": "crti.o", "sha256": "<sha256>"},
|
||||
{"source": "inputs/crtbeginT.o", "path": "crtbeginT.o", "sha256": "<sha256>"}
|
||||
],
|
||||
"libraries": [
|
||||
{"source": "inputs/libc.a", "path": "libc.a", "sha256": "<sha256>"},
|
||||
{"source": "inputs/libm.a", "path": "libm.a", "sha256": "<sha256>"},
|
||||
{"source": "inputs/libmvec.a", "path": "libmvec.a", "sha256": "<sha256>"},
|
||||
{"source": "inputs/libgcc.a", "path": "libgcc.a", "sha256": "<sha256>"},
|
||||
{"source": "inputs/libgcc_eh.a", "path": "libgcc_eh.a", "sha256": "<sha256>"}
|
||||
],
|
||||
"end": [
|
||||
{"source": "inputs/crtend.o", "path": "crtend.o", "sha256": "<sha256>"},
|
||||
{"source": "inputs/crtn.o", "path": "crtn.o", "sha256": "<sha256>"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`source` and `signing_key` are normalized relative paths below the author root, without symlinks. Link-input `path` is relative to the output's `runtime/<platform>` directory; reserved and duplicate destinations fail. The key must be owner-private on Unix. Unknown fields, missing profiles, invalid targets or digests fail explicitly. The key and author configuration are never included in the output.
|
||||
|
||||
`core_libraries` must list the full dynamic dependency closure, not just the two illustrative entries above. These `path` values are plain SONAMEs published under `lib/`. Current Linux x86_64 inputs additionally require gcc_s, z, ffi, edit, zstd, tinfo, xml2, bsd, md, icuuc, icudata and lzma. The compiler declares inherited `RPATH=$ORIGIN/lib`; library search entries may only name that same private directory (`$ORIGIN` or upstream LLVM's `$ORIGIN/../lib`). Maker and signed native installation reject missing dependencies, mismatched SONAME/architecture, unused libraries and escaping search paths. The declared system baseline is GNU libc and its companion libraries; the current prepared compiler requires glibc 2.39. The pack must not replace that system ABI. This replaces the single `llvm` author field directly.
|
||||
|
||||
## Versioned AI skill and public release assets
|
||||
|
||||
`skill` is required. Copy the complete standalone `dever-main-skills` checkout into the author inputs, excluding `.git` and temporary files, and list every skill file as `{source,path,sha256}`. Paths are relative to its skill root; the maker publishes them under `skills/dever-language/` in the signed manifest. The two entries above illustrate the shape, not the complete file list. `SKILL.md` is mandatory; at most 256 files, 1 MiB per file and 8 MiB total are allowed.
|
||||
|
||||
`dever skill path [project-root]` validates and returns the skill belonging to the selected compiler. `dever skill install <new-directory>` installs a small AI entry that resolves the active signed skill on each task. It refuses existing directories. `dever update` and `dever use` change the one active-version journal used by both core and skill; they do not rewrite every user's AI directory. Ordinary users can read the signed skill without the install lock or administrator permissions.
|
||||
|
||||
Publish official versions to `shemic/dever-main` GitHub Releases with tag `v<version>`. `sdk/release-assets.py` converts the signed directory into three assets named `dever-<platform>.manifest.json`, `dever-<platform>.manifest.sig`, and `dever-<platform>.tar.gz`. The archive contains exactly the manifest's ordinary artifact files, with no directory headers or embedded manifest. The installer and explicit `dever install/update` download these assets, verify the independent pinned public key before payload extraction, and never use application configuration or environment variables for the official address. Project `run/build` remain offline.
|
||||
|
||||
## Linux first installation
|
||||
|
||||
Include the stable launcher and daemon in the same signed release by adding this author field:
|
||||
|
||||
```json
|
||||
"bootstrap": {
|
||||
"launcher": {"source":"inputs/dever", "sha256":"<sha256>"},
|
||||
"daemon": {"source":"inputs/deverd", "sha256":"<sha256>"},
|
||||
"libraries": [
|
||||
{"source":"inputs/libgcc_s.so.1", "path":"libgcc_s.so.1", "sha256":"<sha256>"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Both Rust entry points must be built with their private `$ORIGIN/lib` RPATH. The maker verifies their separate ELF dependency closure and adds `bootstrap/dever`, `bootstrap/deverd`, `bootstrap/lib/*` and the fixed `bootstrap/deverd.service` to the existing signed manifest. Core-only releases may omit this field; first installation requires it.
|
||||
|
||||
An administrator must independently authenticate the first installer executable, its private libraries, and the release public key. A key shipped inside a downloaded release is not a trust source. Place the following configuration under an absolute, root-owned installer directory, in `config/setting.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"release": "/prepared/release",
|
||||
"system_root": "/owned/system-image",
|
||||
"trusted_key": "/independently-provisioned/release.pub",
|
||||
"activate_service": false
|
||||
}
|
||||
```
|
||||
|
||||
Run the independently trusted launcher with `--dever-install /absolute/installer-root`. It installs the signed compiler version, `/opt/dever/bin` bootstrap closure, `/usr/local/bin/dever` symlink, root-owned systemd unit and its enable link below `system_root`. The root daemon owns the shared machine store; ordinary projects and users share it through the existing authenticated service protocol. `install`, `update` and `use` retain their version-management contracts.
|
||||
|
||||
Offline image installation never invokes systemctl. `activate_service: true` is accepted only for the real `/` system root and explicitly performs daemon reload, restart and readiness checking. This session's acceptance uses owned image roots and manually started owned daemons; it does not activate any global service.
|
||||
|
||||
The initial key is written to a checked temporary file, synced, then published without replacing an existing key. A bounded journal records seven fixed destinations: bootstrap directory, sandbox helper store, AppArmor policy file, service unit, public entry, enable link and active version. File modes and child directories are persisted before publication. Interrupted uncommitted installs restore their recorded predecessors; committed installs finish cleanup without undoing publication. Rerun the explicit installer to recover. Normal launcher/version operations refuse an unfinished journal. A cleanup failure remains an explicit error with the journal retained. Existing service/policy ownership is checked against its previously signed template, allowing legitimate template upgrades while rejecting unrelated files. These ordering guarantees and interruption tests are not a claim of hardware power-loss testing.
|
||||
|
||||
## Ecosystem inputs
|
||||
|
||||
The same author configuration accepts an optional `runtimes` object with `pip`, `npm` and `go` entries. Each entry declares `name`, `version` and `files`: normalized archive `path`, author-root `source`, and exact `sha256`. Include `dever-runtime.json` in that list. Python additionally requires `python_markers` and nonempty `python_wheel_tags` matching the prepared interpreter; these are registry resolution metadata, not settings supplied by an application.
|
||||
|
||||
Python packs must contain a relocatable interpreter, its standard library, native extensions and their non-OS dependencies, with fixed `-I -S -B` arguments. Node packs include the interpreter with its embedded standard library. Go build packs contain `compile`, `link`, the Dever source analyzer and every standard-library archive referenced by their `importcfg`; applications embed only the resulting Go Worker. Interpreter portability must be tested against an explicit OS baseline. A copied host executable alone does not establish a complete pack.
|
||||
|
||||
The maker and Worker builder share descriptor, path and file-closure validation. Packs are deterministic gzip/tar files with sorted entries, fixed archive metadata and executable modes derived from the descriptor. Every runtime input is copied and rehashed; the private-key exclusion applies to these files too. The maker calculates the pack digest and emits `runtime/<ecosystem>/<platform>/{runtime.pack,manifest.json}`. Both files enter the existing top-level release signature. No new lock, installation or signature protocol is introduced.
|
||||
|
||||
Node registry metadata may declare `npm_libc` as `glibc` or `musl`, matching the explicitly prepared interpreter and target. This is signed author metadata for package eligibility, never host probing. The pinned GNU Node fixture declares `glibc`.
|
||||
|
||||
## Sandbox inputs
|
||||
|
||||
External Workers, including `exec`, require the installed release's sandbox assets. Add a `sandbox` array to author settings using the same `{source, path, sha256}` entries as other leaf inputs. Its paths are `bin/bwrap`, `bin/guard`, and flat `lib/<name>` libraries, including the platform loader. Build `dever-sandbox-guard` from the current source with the explicit author toolchain. The bwrap input must support bind-fd and be a static target ELF with neither PT_INTERP nor DT_NEEDED; runtime executes it directly. The guard and language runtimes retain their complete private loader/library closure. The maker validates every asset before publishing `sandbox/<platform>/<path>` under the existing release signature. The asset budget is 128 files / 32 MiB. Inputs must not use RPATH/RUNPATH to escape the supplied bootstrap library set.
|
||||
|
||||
The Linux backend requires user, mount, PID and network namespace permission, seccomp and `openat2`. An active host loader preload configuration is rejected. When `apparmor_restrict_unprivileged_userns=1`, a non-root launch selects only `/opt/dever/sandbox/<embedded-bwrap-sha256>/bwrap`. Every ancestor must be a real root-owned directory without group/other write permission; the regular executable must have no setuid/setgid or file capabilities, and its digest must equal the embedded verified asset. Missing or invalid helpers produce an installation error before spawning. Other launches execute the same static embedded bwrap directly; there is no PATH, environment, dynamic-loader or system-bwrap fallback.
|
||||
|
||||
The signed bootstrap includes `bootstrap/dever-sandbox`, copied to `/etc/apparmor.d/dever-sandbox` by the same installation transaction. The source is [sdk/apparmor/dever-sandbox](apparmor/dever-sandbox), derived from the [AppArmor 4.0 bwrap restriction profile](https://gitlab.com/apparmor/apparmor/-/raw/apparmor-4.0/profiles/apparmor/profiles/extras/bwrap-userns-restrict). Its attachment permits only 64-lowercase-hex directories below the trusted store. bwrap creates namespaces under the parent policy and exec stacks the capability-denying child policy, including under no-new-privileges. Attachment paths do not restrict an explicit profile transition from an unconfined process: on hosts with user namespace restriction enabled, `kernel.apparmor_restrict_unprivileged_unconfined=1` is therefore mandatory before installation or loading. See [Ubuntu's explanation of strict profile changes](https://discourse.ubuntu.com/t/understanding-apparmor-user-namespace-restriction/58007). Non-root launches check this flag before selecting a helper; a missing, disabled or unreadable flag fails closed. The real-host installer checks the same prerequisite before any installation changes and again before publishing the policy file, because an independent system policy reload could load even an initially inactive file. Offline image preparation does not change or require the author's host settings; the deployed image must satisfy this prerequisite before its policy becomes loadable.
|
||||
|
||||
The runtime also requires both named profiles in enforce mode using the read-only AppArmor policy filesystem; absent or inaccessible policy metadata is an explicit deployment error. No launch changes sysctl or loads policy. A container or restricted mount namespace must expose this policy metadata if it runs such non-root Workers. Hosts without the user namespace restriction do not require the strict-profile sysctl to exist.
|
||||
|
||||
Administrator order is explicit: first review and approve the host-wide effect of strict profile transitions, record the original sysctl value, and establish `kernel.apparmor_restrict_unprivileged_unconfined=1`; then install the matching signed helper/policy; only then load with `apparmor_parser -r -K /etc/apparmor.d/dever-sandbox`. Dever never changes this global setting automatically. Review and offline validation use `apparmor_parser -Q -K -j 1 sdk/apparmor/dever-sandbox`, which does not load policy or write its cache. First-install rollback must unload this exact policy with `apparmor_parser -R -K /etc/apparmor.d/dever-sandbox` and remove the newly installed policy file from reload discovery before restoring the original strict-profile sysctl value. For an upgrade, restore the previously approved policy instead, retaining strict mode for any policy that requires it. Keep all helpers needed by deployed applications. Each host operation requires authorization; never disable the user namespace restriction. Real non-root, NNP/aa-exec (including immediate mode) and capability-denial acceptance must follow loading; offline parsing and root-only acceptance do not establish those results.
|
||||
|
||||
The installer preserves at most 24 helper identities, each with its original signed manifest/signature, independently of compiler version directories. Unknown entries, incorrect signatures and digest mismatches fail without overwriting that store. The existing transaction's 128-entry/80-MiB tree budget also applies. Retained hashes support independently deployed applications after source or compiler-version removal. A new compiler version selecting different bwrap bytes requires the explicit trusted bootstrap installation before affected non-root Workers can start. Other OS backends and GPU device isolation are not currently supported.
|
||||
|
||||
For approved profile-transition acceptance, `test/dever-sandbox-tests/apparmor_transition.c` is a standalone negative probe: compile it statically with the explicit author C toolchain, then invoke it as a nonzero UID through `aa-exec -p dever-bwrap`, immediate `aa-exec -i -p dever-bwrap`, and no-new-privileges variants. It never executes another program; it attempts only its own user/network namespaces, UID/GID maps and loopback capability operation. Record aa-exec rejection separately from a probe result. Exit 0 reports a denied operation, exit 1 reports unsafe access and exit 2 means unavailable setup. Creating the network namespace already proves a capability check succeeded, so the probe returns 1 even if a later loopback operation fails. Preserve the initial label and exact denial stage for each case.
|
||||
|
||||
### Static Linux x86_64 author input
|
||||
|
||||
The current private fixture uses upstream [bubblewrap v0.11.0](https://github.com/containers/bubblewrap/tree/v0.11.0), SHA-256 `cfeeb15fcc47d177d195f06fdf0847e93ee3aa6bf46f6ac0a141fa142759e2c3` for `https://codeload.github.com/containers/bubblewrap/tar.gz/refs/tags/v0.11.0`, and Ubuntu Noble's `libcap-dev_2.66-5ubuntu2.4_amd64.deb`, SHA-256 `07f2462867569a2119a2ad0f1593232663f2d1612b791c230d22a8d73a15abee` from the authenticated apt package index. Its download URL is `https://archive.ubuntu.com/ubuntu/pool/main/libc/libcap2/libcap-dev_2.66-5ubuntu2.4_amd64.deb`. Extract with `dpkg-deb -x` into the author directory; do not install packages on the target host. Retain upstream `COPYING`, the libcap package's `usr/share/doc/libcap-dev/copyright`, sources and the exact build recipe with author inputs.
|
||||
|
||||
Below, `target/sandbox-author-inputs/` contains the extracted `bubblewrap-0.11.0/` and `libcap/`. Its source-local `config.h` contains only `#define PACKAGE_STRING "bubblewrap 0.11.0"` and `#define ENABLE_REQUIRE_USERNS 1`. Build with the explicitly selected author GNU C toolchain and static libc; no customer build tools are needed:
|
||||
|
||||
```sh
|
||||
/usr/bin/cc -static -O2 -D_GNU_SOURCE \
|
||||
-I target/sandbox-author-inputs/libcap/usr/include \
|
||||
-I target/sandbox-author-inputs/bubblewrap-0.11.0 \
|
||||
target/sandbox-author-inputs/bubblewrap-0.11.0/bubblewrap.c \
|
||||
target/sandbox-author-inputs/bubblewrap-0.11.0/bind-mount.c \
|
||||
target/sandbox-author-inputs/bubblewrap-0.11.0/network.c \
|
||||
target/sandbox-author-inputs/bubblewrap-0.11.0/utils.c \
|
||||
target/sandbox-author-inputs/libcap/usr/lib/x86_64-linux-gnu/libcap.a \
|
||||
-o target/sandbox-author-inputs/bwrap-static
|
||||
readelf -l target/sandbox-author-inputs/bwrap-static
|
||||
readelf -d target/sandbox-author-inputs/bwrap-static
|
||||
sha256sum target/sandbox-author-inputs/bwrap-static
|
||||
```
|
||||
|
||||
The 2026-10-02 private author build is 1,099,016 bytes with SHA-256 `c9e8bd758c2d01ca6668a842bad7b020bf700436e87b666cfaae3803be0be1b3`; readelf confirms no interpreter and no dynamic section. A different author compiler/libc may produce a different digest: sign the actual validated bytes and deploy their matching content-addressed helper. The current policy source SHA-256 is `a6d2ef490d12914e90a96272a197b8b78667403702b1324bae661a10b78125ec`; AppArmor parser 4.0.1 accepted it offline with the command above. These are input/parse results, not evidence that host policy has been loaded.
|
||||
|
||||
Application file access is explicit in `config/setting.json`, for example `adapter.notification.delivery.files.uploads = {"path":"data/upload", "write":true}` with checked `allow file`. The Worker sees `/data/uploads`. Grant paths cannot write the executable cache or other execution inputs; no environment configuration is introduced.
|
||||
|
||||
## Make and consume
|
||||
|
||||
Build the private author executable once, then run it directly. The command optimizes existing compression and digest dependencies even in a development build, so complete language distributions do not spend minutes in unoptimized byte loops. These author/test options do not change application configuration or compile-service limits.
|
||||
|
||||
```sh
|
||||
cargo build --offline --locked -p dever-cli --example native-release \
|
||||
--config 'profile.dev.package.miniz_oxide.opt-level=3' \
|
||||
--config 'profile.dev.package.flate2.opt-level=3' \
|
||||
--config 'profile.dev.package.sha2.opt-level=3'
|
||||
target/debug/examples/native-release <author-root> --output <new-release-directory>
|
||||
```
|
||||
|
||||
The output parent must exist. A private sibling staging directory holds copied inputs until the existing runtime validator succeeds and the manifest is signed. Publication atomically requires an absent destination, including under concurrent makers. Failures remove only owned staging. Output artifacts are independent copies; subsequent source changes do not change a completed pack.
|
||||
|
||||
The result contains `dever-core`, `lib/libLLVM.so.18.1`, `runtime/<platform>/manifest.json`, four `archives/<profile>.a` files, the configured link inputs, and top-level `manifest.json`/`manifest.sig`. The signed manifest covers the complete native closure. Place this version directory in the existing machine download store and use the established installer; trusted public-key provisioning remains explicit and separate. The maker does not trust a key merely because it accompanies a release, install a service, publish a catalog or update a global command.
|
||||
|
||||
For private `dever` development, the same verified `runtime/` and `lib/` trees may be copied beside a matching private compiler in a new directory. An executable named `dever-core` still requires the managed installation layout and authenticated daemon. This is not a service-unavailable fallback.
|
||||
|
||||
## Acceptance
|
||||
|
||||
Default `native_release` tests use synthetic input and temporary keys. The explicit Linux acceptance additionally consumes the four optimized archives in `target/native-release-inputs/`, the prepared native SDK and the built author executable:
|
||||
|
||||
Real ecosystem acceptance temporarily holds the signed release, installed version, compiler cache and Worker bundles simultaneously. Allow several GiB of free storage in addition to author inputs and Cargo artifacts. When the system temporary filesystem is too small, an optional author-only `target/native-release-inputs/config/setting.json` can select an existing executable filesystem:
|
||||
|
||||
```json
|
||||
{"machine_temporary_root":"/dev/shm"}
|
||||
```
|
||||
|
||||
The path must be absolute; unknown configuration fields are rejected. Only the ecosystem machine directory and bootstrap image use this setting. Author inputs remain on the normal temporary filesystem so their immutable hard links stay on the source filesystem. With no configuration, the existing temporary-directory behavior is unchanged. No environment variable is read for this setting. A memory-backed filesystem consumes real memory: provide both sufficient free capacity and an explicit acceptance-process memory budget, and run these large acceptances serially.
|
||||
|
||||
Before the no-host checks, build their test-only static init explicitly:
|
||||
|
||||
```sh
|
||||
cargo rustc --offline --locked -j1 -p dever-cli --example native-acceptance-init -- -C target-feature=+crt-static
|
||||
```
|
||||
|
||||
```sh
|
||||
cargo test --offline --locked -p dever-cli --test native_release
|
||||
cargo test --offline --locked -p dever-cli --test native_release \
|
||||
acceptance::optimized_profiles_make_reproducible_signed_release_and_run_through_daemon \
|
||||
-- --exact --ignored --nocapture
|
||||
cargo test --offline --locked -p dever-cli --test native_release \
|
||||
acceptance::signed_core_builds_without_host_compiler_libraries_or_tools \
|
||||
-- --exact --ignored --nocapture
|
||||
```
|
||||
|
||||
The second command starts only a test-owned daemon and SQLite databases. It checks actual driver symbols, identical manifests/signatures for repeated packaging, signed installation, four profile executions through the daemon, and independent programs after source removal and daemon shutdown. PostgreSQL server behavior is a separate acceptance, not implied by linking the PostgreSQL profile.
|
||||
|
||||
The third command requires root and namespace permission for a test-owned mount namespace root. It checks source and builds a SQLite program inside a root containing only the signed compiler closure, selected native archives and named GNU OS ABI libraries, then executes the program after removing its source. No host LLVM, C++ library, language tool, compiler SDK or package manager is visible there. The harness uses the explicitly prepared bwrap/loader assets to pivot into that root.
|
||||
|
||||
The harness rejects a dynamically linked init. Inside the already pivoted namespace, it replaces bwrap's proc submounts with a complete private proc mount so the nested Worker can mount its own proc. It keeps the outer root fixture's existing capabilities; the product's inner namespace, capability removal and guard remain unchanged. It never binds the host proc into the final root. This private compiler test complements the managed daemon test; it does not install a machine service.
|
||||
|
||||
The complete first-install acceptance additionally requires the built `dever` and `deverd` binaries:
|
||||
|
||||
```sh
|
||||
cargo test --offline --locked -p dever-cli --test native_release \
|
||||
acceptance::bootstrap::signed_bootstrap_installs_and_serves_two_projects_without_host_libraries \
|
||||
-- --exact --ignored --nocapture
|
||||
```
|
||||
|
||||
It installs only into a new owned image, verifies stable entry and systemd assets, exercises a legitimate older signed unit upgrade and rejects tampering, then runs two projects through the installed daemon in the minimal OS root. Separate clients retain two real kernel UIDs to check shared-cache use and denied machine mutation. Finally the independently built programs run after source and machine-store removal. Neither the global service nor a public signing identity is involved.
|
||||
|
||||
Identical prepared inputs and signing key produce identical payloads, manifests and signatures. This establishes deterministic packaging, not reproducible source builds across different machines, CRTs or Rust toolchains. Temporary test signatures and Linux execution do not establish public release, six-platform delivery, an OS sandbox or performance capacity.
|
||||
|
||||
The explicit three-ecosystem acceptance uses fixed upstream archives and the retained Go build fixture listed with exact digests in `test/ecosystem-release/prepare.py`. Supply those inputs explicitly, then prepare a new directory:
|
||||
|
||||
| Input | Explicit source |
|
||||
| --- | --- |
|
||||
| `downloads/cpython-3.12.14.tar.gz` | [python-build-standalone 20260901, Linux x86_64 install-only stripped](https://github.com/astral-sh/python-build-standalone/releases/download/20260901/cpython-3.12.14%2B20260901-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz) |
|
||||
| `downloads/node-v24.15.0-linux-x64.tar.xz` | [Node v24.15.0 Linux x64](https://nodejs.org/dist/v24.15.0/node-v24.15.0-linux-x64.tar.xz), verified against the release's `SHASUMS256.txt` |
|
||||
| `target/go-managed/runtime.pack` | Go 1.26.3 private host fixture with explicit `build.host=linux-x86_64`; the pinned archive preserves the original tools/stdlib and repacks the updated descriptor. New fixtures use `test/go-managed/make_pack.go`; this is not an upstream published pack. |
|
||||
|
||||
The two download paths are below `target/ecosystem-release-inputs/`. No command here performs implicit downloads; the preparation script rejects different digests.
|
||||
|
||||
```sh
|
||||
/usr/bin/python3 test/ecosystem-release/prepare.py target/ecosystem-release-inputs/prepared
|
||||
cargo test --offline --locked -p dever-cli --test native_release \
|
||||
--config 'profile.dev.package.miniz_oxide.opt-level=3' \
|
||||
--config 'profile.dev.package.flate2.opt-level=3' \
|
||||
--config 'profile.dev.package.sha2.opt-level=3' \
|
||||
acceptance::ecosystems::signed_ecosystems_build_and_run_without_system_language_installations \
|
||||
-- --exact --ignored --nocapture
|
||||
```
|
||||
|
||||
This opt-in test requires Linux root and mount/PID/user namespace capability. It signs and installs an owned release, uses the managed daemon for each ecosystem's lock/check/run/build, then removes project sources and the entire machine store. The resulting programs run inside an owned mount namespace root containing only named Linux OS ABI libraries, a private proc mount and synthetic devices. Python checks bundle-local imports, standard-library native modules and the source-built simplejson C extension; Node loads the source-built bufferutil addon; Go retains its managed probe. The already prepared source-build locks and exact output artifacts come from `target/source-build/dependency` and `target/npm-build/dependency`; this acceptance never rebuilds hooks or downloads dependencies. It signs the exact runtime-pack bytes bound by those build receipts, so this test's author signature differs from the separate deterministic-maker evidence. No system Python, Node, Go, compiler, SDK or package manager is visible there. Other operating systems and a public release identity remain separate acceptance.
|
||||
@@ -0,0 +1,27 @@
|
||||
//! Private release-author entry point; never distributed as a user command.
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
fn main() {
|
||||
if let Err(error) = run() {
|
||||
eprintln!("native-release: {error}");
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
fn run() -> Result<(), String> {
|
||||
let arguments: Vec<_> = std::env::args_os().skip(1).collect();
|
||||
let [author_root, flag, output] = arguments.as_slice() else {
|
||||
return Err("usage: native-release <author-root> --output <new-dir>".into());
|
||||
};
|
||||
if flag != "--output" {
|
||||
return Err("usage: native-release <author-root> --output <new-dir>".into());
|
||||
}
|
||||
let manifest =
|
||||
dever_cli::toolchain::packaging::create(Path::new(author_root), Path::new(output))?;
|
||||
println!(
|
||||
"created Dever {} for {}",
|
||||
manifest.version, manifest.platform
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,404 @@
|
||||
"""Python Worker for the Dever Component Protocol.
|
||||
|
||||
The compiler owns the Port schema. Pass its exact identity to Contract; this
|
||||
module only transports and decodes values supplied by the application.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import struct
|
||||
import sys
|
||||
import uuid
|
||||
from contextlib import suppress
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Callable
|
||||
|
||||
|
||||
VERSION = "dever-component-1"
|
||||
MAX_BYTES = 16 * 1024 * 1024
|
||||
MAX_DEPTH = 64
|
||||
MAX_ELEMENTS = 65_536
|
||||
CANCEL_GRACE = 1
|
||||
|
||||
|
||||
class ProtocolError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class BusinessError(Exception):
|
||||
def __init__(self, identity: str, payload: Any = None):
|
||||
super().__init__(identity)
|
||||
self.identity = identity
|
||||
self.payload = payload
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Contract:
|
||||
port: str
|
||||
schema: str
|
||||
adapter: str
|
||||
operations: tuple[str, ...]
|
||||
capabilities: tuple[str, ...] = ()
|
||||
errors: tuple[str, ...] = ()
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
for name in ("operations", "capabilities", "errors"):
|
||||
values = getattr(self, name)
|
||||
if not isinstance(values, (list, tuple)):
|
||||
raise ValueError("component contract entries must be lists")
|
||||
object.__setattr__(self, name, tuple(values))
|
||||
if not all(isinstance(value, str) and value for value in (self.port, self.schema, self.adapter)) or not self.operations:
|
||||
raise ValueError("incomplete component contract")
|
||||
for names in (self.operations, self.capabilities, self.errors):
|
||||
if any(not isinstance(name, str) or not name for name in names) or len(set(names)) != len(names):
|
||||
raise ValueError("invalid or duplicate component contract entry")
|
||||
|
||||
|
||||
def _typed(value: Any, schema: dict[str, Any]) -> Any:
|
||||
kind = schema["type"]
|
||||
if kind == "nullable":
|
||||
return None if value is None else _typed(value, schema["value"])
|
||||
if kind == "list":
|
||||
if not isinstance(value, list):
|
||||
raise ValueError("expected list")
|
||||
return [_typed(item, schema["value"]) for item in value]
|
||||
if kind == "record":
|
||||
fields = schema["fields"]
|
||||
if not isinstance(value, dict) or not value.keys() <= fields.keys():
|
||||
raise ValueError("invalid record fields")
|
||||
return {name: _typed(value[name], field) if name in value else
|
||||
(None if field["type"] == "nullable" else _missing_field())
|
||||
for name, field in fields.items()}
|
||||
if kind in ("int64", "duration", "model_id"):
|
||||
if type(value) is not int or not -(2**63) <= value < 2**63:
|
||||
raise ValueError("expected signed 64-bit integer")
|
||||
elif kind == "float64":
|
||||
if type(value) not in (int, float) or not math.isfinite(value):
|
||||
raise ValueError("expected finite float")
|
||||
elif kind == "bool":
|
||||
if type(value) is not bool:
|
||||
raise ValueError("expected boolean")
|
||||
elif kind == "json":
|
||||
_bounded(value, 0, [MAX_ELEMENTS])
|
||||
elif kind in ("text", "id", "secret", "decimal", "uuid", "datetime", "date", "time"):
|
||||
if not isinstance(value, str):
|
||||
raise ValueError("expected text")
|
||||
if kind == "decimal" and not re.fullmatch(r"[+-]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?", value):
|
||||
raise ValueError("invalid decimal")
|
||||
if kind == "uuid":
|
||||
if not re.fullmatch(r"[0-9a-fA-F]{8}(?:-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}", value):
|
||||
raise ValueError("invalid UUID")
|
||||
uuid.UUID(value)
|
||||
if kind == "date":
|
||||
import datetime
|
||||
if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", value):
|
||||
raise ValueError("invalid Date")
|
||||
datetime.date.fromisoformat(value)
|
||||
if kind == "time":
|
||||
import datetime
|
||||
if not re.fullmatch(r"\d{2}:\d{2}:\d{2}(?:\.\d{3})?", value):
|
||||
raise ValueError("invalid Time")
|
||||
datetime.time.fromisoformat(value)
|
||||
if kind == "datetime":
|
||||
import datetime
|
||||
if not re.fullmatch(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,3})?(?:Z|[+-]\d{2}:\d{2})", value):
|
||||
raise ValueError("invalid DateTime")
|
||||
datetime.datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
else:
|
||||
raise ValueError("unknown worker type")
|
||||
return value
|
||||
|
||||
|
||||
def _missing_field() -> Any:
|
||||
raise ValueError("missing required record field")
|
||||
|
||||
|
||||
def _pairs(values: list[tuple[str, Any]]) -> dict[str, Any]:
|
||||
result = {}
|
||||
for key, value in values:
|
||||
if key in result:
|
||||
raise ProtocolError("duplicate JSON field")
|
||||
result[key] = value
|
||||
return result
|
||||
|
||||
|
||||
def _reject_constant(_: str) -> None:
|
||||
raise ProtocolError("invalid JSON number")
|
||||
|
||||
|
||||
def _bounded(value: Any, depth: int, budget: list[int]) -> None:
|
||||
if depth > MAX_DEPTH or (depth == MAX_DEPTH and isinstance(value, (dict, list))):
|
||||
raise ProtocolError("wire JSON exceeds depth limit")
|
||||
budget[0] -= 1
|
||||
if budget[0] < 0:
|
||||
raise ProtocolError("wire JSON exceeds element limit")
|
||||
if isinstance(value, dict):
|
||||
for child in value.values():
|
||||
_bounded(child, depth + 1, budget)
|
||||
elif isinstance(value, list):
|
||||
for child in value:
|
||||
_bounded(child, depth + 1, budget)
|
||||
elif isinstance(value, float) and not math.isfinite(value):
|
||||
raise ProtocolError("non-finite wire Float")
|
||||
|
||||
|
||||
def _decode(body: bytes) -> dict[str, Any]:
|
||||
try:
|
||||
value = json.loads(body.decode("utf-8"), object_pairs_hook=_pairs,
|
||||
parse_constant=_reject_constant)
|
||||
except (UnicodeError, ValueError, RecursionError) as error:
|
||||
raise ProtocolError("invalid component JSON") from error
|
||||
_bounded(value, 0, [MAX_ELEMENTS])
|
||||
if not isinstance(value, dict):
|
||||
raise ProtocolError("component message must be an object")
|
||||
return value
|
||||
|
||||
|
||||
def _fields(message: dict[str, Any], names: set[str]) -> None:
|
||||
if message.keys() != names:
|
||||
raise ProtocolError("component message has missing or extra fields")
|
||||
|
||||
|
||||
def _id(value: Any) -> int:
|
||||
if type(value) is not int or not 0 <= value <= 2**63 - 1:
|
||||
raise ProtocolError("invalid request id")
|
||||
return value
|
||||
|
||||
|
||||
async def _read_frame(reader: asyncio.StreamReader) -> dict[str, Any]:
|
||||
try:
|
||||
size = struct.unpack(">I", await reader.readexactly(4))[0]
|
||||
if not 0 < size <= MAX_BYTES:
|
||||
raise ProtocolError("component frame exceeds byte limit")
|
||||
return _decode(await reader.readexactly(size))
|
||||
except asyncio.IncompleteReadError as error:
|
||||
raise ProtocolError("incomplete component frame") from error
|
||||
|
||||
|
||||
def _write_frame(message: dict[str, Any]) -> None:
|
||||
try:
|
||||
body = json.dumps(message, ensure_ascii=False, allow_nan=False, separators=(",", ":")).encode("utf-8")
|
||||
except (TypeError, ValueError) as error:
|
||||
raise ProtocolError("invalid handler response") from error
|
||||
_bounded(_decode(body), 0, [MAX_ELEMENTS])
|
||||
if not 0 < len(body) <= MAX_BYTES:
|
||||
raise ProtocolError("component frame exceeds byte limit")
|
||||
sys.__stdout__.buffer.write(struct.pack(">I", len(body)) + body)
|
||||
sys.__stdout__.buffer.flush()
|
||||
|
||||
|
||||
async def _stop_handler(task: asyncio.Task) -> None:
|
||||
task.cancel()
|
||||
completed, _ = await asyncio.wait({task}, timeout=CANCEL_GRACE)
|
||||
if not completed:
|
||||
raise ProtocolError("component handler did not stop after cancellation")
|
||||
|
||||
|
||||
class Worker:
|
||||
def __init__(self, contract: Contract, decode_setting: Callable[[Any], Any]):
|
||||
if not callable(decode_setting):
|
||||
raise ValueError("component setting decoder is required")
|
||||
self.contract = contract
|
||||
self.decode_setting = decode_setting
|
||||
self.handlers: dict[str, tuple[Callable[[Any], Any], Callable[..., Any]]] = {}
|
||||
self.output_types: dict[str, Any] = {}
|
||||
self.error_types: dict[str, Any] = {}
|
||||
|
||||
@classmethod
|
||||
def from_manifest(cls, manifest: dict[str, Any], handlers: dict[str, Callable[..., Any]] | None = None) -> "Worker":
|
||||
operations = tuple(manifest["operations"])
|
||||
inputs = manifest["inputs"]
|
||||
outputs = manifest["outputs"]
|
||||
errors = manifest["errors"]
|
||||
if set(inputs) != set(operations) or set(outputs) != set(operations):
|
||||
raise ValueError("incomplete typed Worker contract")
|
||||
contract = Contract(manifest["port"], manifest["schema"], manifest["adapter"],
|
||||
operations, tuple(manifest["capabilities"]), tuple(errors))
|
||||
setting = manifest["setting"]
|
||||
worker = cls(contract, lambda value: _typed(value, setting) if setting is not None else
|
||||
(None if value is None else _missing_field()))
|
||||
worker.input_types = inputs
|
||||
worker.output_types = outputs
|
||||
worker.error_types = errors
|
||||
if handlers is not None:
|
||||
if set(handlers) != set(operations):
|
||||
raise ValueError("component operations are not fully registered")
|
||||
for operation, handler in handlers.items():
|
||||
worker.register_handler(operation, handler)
|
||||
return worker
|
||||
|
||||
def register_handler(self, operation: str, handler: Callable[..., Any]) -> None:
|
||||
if not hasattr(self, "input_types") or operation not in self.input_types:
|
||||
raise ValueError("operation is not in a typed Worker contract")
|
||||
self.register(operation, lambda value: _typed(value, self.input_types[operation]), handler)
|
||||
|
||||
def register(self, operation: str, decode_payload: Callable[[Any], Any], handler: Callable[..., Any]) -> None:
|
||||
if operation not in self.contract.operations or operation in self.handlers:
|
||||
raise ValueError("unknown or duplicate operation")
|
||||
if not callable(decode_payload):
|
||||
raise ValueError("component payload decoder is required")
|
||||
if not asyncio.iscoroutinefunction(handler):
|
||||
raise ValueError("component handler must be async")
|
||||
self.handlers[operation] = (decode_payload, handler)
|
||||
|
||||
async def serve(self) -> None:
|
||||
if set(self.handlers) != set(self.contract.operations):
|
||||
raise ValueError("component operations are not fully registered")
|
||||
loop = asyncio.get_running_loop()
|
||||
reader = asyncio.StreamReader()
|
||||
transport, _ = await loop.connect_read_pipe(
|
||||
lambda: asyncio.StreamReaderProtocol(reader), sys.stdin.buffer
|
||||
)
|
||||
try:
|
||||
await self._serve(reader)
|
||||
finally:
|
||||
transport.close()
|
||||
|
||||
async def _serve(self, reader: asyncio.StreamReader) -> None:
|
||||
hello = await _read_frame(reader)
|
||||
_fields(hello, {"kind", "version", "port", "schema", "adapter", "capabilities", "operations", "setting"})
|
||||
expected = self.contract
|
||||
if (hello["kind"], hello["version"], hello["port"], hello["schema"], hello["adapter"], hello["capabilities"], hello["operations"]) != ("hello", VERSION, expected.port, expected.schema, expected.adapter, list(expected.capabilities), list(expected.operations)):
|
||||
raise ProtocolError("component handshake does not match registered contract")
|
||||
try:
|
||||
setting = self.decode_setting(hello["setting"])
|
||||
except Exception as error:
|
||||
raise ProtocolError("invalid component setting") from error
|
||||
_write_frame({"kind": "ready", "version": VERSION, "port": expected.port, "schema": expected.schema, "adapter": expected.adapter, "capabilities": list(expected.capabilities), "operations": list(expected.operations)})
|
||||
|
||||
messages: asyncio.Queue = asyncio.Queue(maxsize=4)
|
||||
|
||||
async def read_messages() -> None:
|
||||
while True:
|
||||
try:
|
||||
message = await _read_frame(reader)
|
||||
except (OSError, ProtocolError) as error:
|
||||
await messages.put(("fault", error))
|
||||
return
|
||||
await messages.put(("frame", message))
|
||||
|
||||
receiver = asyncio.create_task(read_messages())
|
||||
try:
|
||||
await self._dispatch(messages, setting)
|
||||
finally:
|
||||
# Cancel and join the pipe reader before Python finalizes stdio.
|
||||
receiver.cancel()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await receiver
|
||||
|
||||
async def _dispatch(self, messages: asyncio.Queue, setting: Any) -> None:
|
||||
loop = asyncio.get_running_loop()
|
||||
active_id: int | None = None
|
||||
active_task: asyncio.Task | None = None
|
||||
active_operation: str | None = None
|
||||
next_id = 1
|
||||
last_cancelled = False
|
||||
completions: set[asyncio.Task] = set()
|
||||
|
||||
def on_done(task: asyncio.Task) -> None:
|
||||
try:
|
||||
task.exception()
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
notification = loop.create_task(messages.put(("done", task)))
|
||||
completions.add(notification)
|
||||
notification.add_done_callback(completions.discard)
|
||||
|
||||
try:
|
||||
while True:
|
||||
event, value = await messages.get()
|
||||
if event == "fault":
|
||||
raise value
|
||||
if event == "done":
|
||||
task = value
|
||||
if task is active_task:
|
||||
try:
|
||||
payload = task.result()
|
||||
if self.output_types:
|
||||
payload = _typed(payload, self.output_types[active_operation])
|
||||
_write_frame({"kind": "result", "id": active_id, "payload": payload})
|
||||
except BusinessError as error:
|
||||
if error.identity not in self.contract.errors:
|
||||
raise ProtocolError("undeclared business error identity") from error
|
||||
if self.error_types:
|
||||
error.payload = _typed(error.payload, self.error_types[error.identity])
|
||||
_write_frame({"kind": "error", "id": active_id, "error": error.identity, "payload": error.payload})
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
except Exception as error:
|
||||
raise ProtocolError("component handler failed") from error
|
||||
active_id, active_task, active_operation = None, None, None
|
||||
last_cancelled = False
|
||||
continue
|
||||
message = value
|
||||
kind = message.get("kind")
|
||||
if kind == "health":
|
||||
_fields(message, {"kind", "id"})
|
||||
if _id(message["id"]) != 0:
|
||||
raise ProtocolError("invalid health id")
|
||||
_write_frame(message)
|
||||
elif kind == "call":
|
||||
_fields(message, {"kind", "id", "operation", "payload"})
|
||||
request_id = _id(message["id"])
|
||||
operation = message["operation"]
|
||||
if request_id != next_id or active_task is not None or not isinstance(operation, str) or operation not in self.handlers:
|
||||
raise ProtocolError("invalid component call")
|
||||
next_id += 1
|
||||
decoder, handler = self.handlers[operation]
|
||||
try:
|
||||
payload = decoder(message["payload"])
|
||||
except Exception as error:
|
||||
raise ProtocolError("invalid component payload") from error
|
||||
async def invoke() -> Any:
|
||||
return await handler(payload, setting)
|
||||
active_id = request_id
|
||||
active_operation = operation
|
||||
active_task = loop.create_task(invoke())
|
||||
active_task.add_done_callback(on_done)
|
||||
elif kind == "cancel":
|
||||
_fields(message, {"kind", "id"})
|
||||
request_id = _id(message["id"])
|
||||
if request_id == 0:
|
||||
raise ProtocolError("invalid cancel id")
|
||||
if active_id == request_id and active_task is not None:
|
||||
stopped_task = active_task
|
||||
active_id, active_task, active_operation = None, None, None
|
||||
await _stop_handler(stopped_task)
|
||||
last_cancelled = True
|
||||
_write_frame({"kind": "error", "id": request_id, "error": "dever.cancelled", "payload": None})
|
||||
elif request_id != next_id - 1 or last_cancelled:
|
||||
raise ProtocolError("unknown component cancel id")
|
||||
elif kind == "shutdown":
|
||||
_fields(message, {"kind"})
|
||||
if active_task is not None:
|
||||
stopped_task = active_task
|
||||
active_id, active_task = None, None
|
||||
await _stop_handler(stopped_task)
|
||||
_write_frame({"kind": "shutdown"})
|
||||
return
|
||||
else:
|
||||
raise ProtocolError("unknown component message")
|
||||
|
||||
finally:
|
||||
if active_task is not None:
|
||||
active_task.remove_done_callback(on_done)
|
||||
for notification in completions:
|
||||
notification.cancel()
|
||||
await asyncio.gather(*completions, return_exceptions=True)
|
||||
|
||||
|
||||
def main(worker: Worker) -> None:
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
asyncio.set_event_loop(loop)
|
||||
loop.run_until_complete(worker.serve())
|
||||
except (ProtocolError, OSError, ValueError) as error:
|
||||
print(f"component worker: {error}", file=sys.stderr)
|
||||
raise SystemExit(1) from None
|
||||
finally:
|
||||
for task in asyncio.all_tasks(loop):
|
||||
task.cancel()
|
||||
loop.close()
|
||||
asyncio.set_event_loop(None)
|
||||
@@ -0,0 +1,55 @@
|
||||
"""Runnable protocol example with a fixed compiler-owned fixture schema."""
|
||||
|
||||
import asyncio
|
||||
from dever_component import BusinessError, Contract, Worker, main
|
||||
|
||||
|
||||
def setting(value):
|
||||
if not isinstance(value, dict) or set(value) != {"prefix"} or not isinstance(value["prefix"], str):
|
||||
raise ValueError("invalid setting")
|
||||
return value["prefix"]
|
||||
|
||||
|
||||
def payload(value):
|
||||
if not isinstance(value, dict) or set(value) != {"text", "delay_ms"}:
|
||||
raise ValueError("invalid payload")
|
||||
if not isinstance(value["text"], str) or type(value["delay_ms"]) is not int or not 0 <= value["delay_ms"] <= 10_000:
|
||||
raise ValueError("invalid payload")
|
||||
return value
|
||||
|
||||
|
||||
async def render(value, prefix):
|
||||
await asyncio.sleep(value["delay_ms"] / 1000)
|
||||
return {"value": prefix + value["text"]}
|
||||
|
||||
|
||||
async def fail(value, prefix):
|
||||
raise BusinessError("example.rejected", {"reason": "rejected"})
|
||||
|
||||
|
||||
def number(value):
|
||||
if not isinstance(value, dict) or set(value) != {"number"} or type(value["number"]) is not int:
|
||||
raise ValueError("invalid number")
|
||||
return value["number"]
|
||||
|
||||
|
||||
async def echo_number(value, prefix):
|
||||
return {"number": value}
|
||||
|
||||
|
||||
async def stubborn(value, prefix):
|
||||
try:
|
||||
await asyncio.sleep(5)
|
||||
except asyncio.CancelledError:
|
||||
await asyncio.sleep(5)
|
||||
return {"value": prefix + value["text"]}
|
||||
|
||||
|
||||
worker = Worker(Contract("example.Text", "fixture-schema-v1", "example.TextAdapter", ("text.render", "text.fail", "number.echo", "text.stubborn"), errors=("example.rejected",)), setting)
|
||||
worker.register("text.render", payload, render)
|
||||
worker.register("text.fail", payload, fail)
|
||||
worker.register("number.echo", number, echo_number)
|
||||
worker.register("text.stubborn", payload, stubborn)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main(worker)
|
||||
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Turn a prepared signed release into the three official GitHub release assets."""
|
||||
|
||||
import argparse
|
||||
import ctypes
|
||||
import gzip
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path, PurePosixPath
|
||||
import tarfile
|
||||
import tempfile
|
||||
|
||||
|
||||
def create(release, output):
|
||||
release = release.absolute()
|
||||
output = output.absolute()
|
||||
if release.is_symlink() or not release.is_dir():
|
||||
raise ValueError("release must be a real prepared directory")
|
||||
if output.exists() or output.is_symlink():
|
||||
raise ValueError("output already exists")
|
||||
if not output.parent.is_dir():
|
||||
raise ValueError("output parent must exist")
|
||||
manifest_bytes = (release / "manifest.json").read_bytes()
|
||||
manifest = json.loads(manifest_bytes)
|
||||
if manifest["format"] != "dever-release-v1" or manifest["platform"] not in ("linux-x86_64", "linux-aarch64"):
|
||||
raise ValueError("unsupported release format or platform")
|
||||
artifacts = manifest["artifacts"]
|
||||
if not 1 <= len(artifacts) <= 4096 or sum(entry["bytes"] for entry in artifacts) > 2 * 1024**3:
|
||||
raise ValueError("release exceeds the official asset limits")
|
||||
names = set()
|
||||
for artifact in artifacts:
|
||||
name = artifact["path"]
|
||||
if (not name or str(PurePosixPath(name)) != name or name.startswith("/")
|
||||
or any(part in ("", ".", "..") for part in name.split("/"))
|
||||
or any(character in name for character in ("\\", ":", "\0"))
|
||||
or name in names or name in ("manifest.json", "manifest.sig")):
|
||||
raise ValueError("invalid release artifact path")
|
||||
names.add(name)
|
||||
path = release / name
|
||||
for parent in (path, *path.parents):
|
||||
if parent.is_symlink():
|
||||
raise ValueError("release artifacts must not traverse symlinks")
|
||||
if parent == release:
|
||||
break
|
||||
if not path.is_file() or path.stat().st_size != artifact["bytes"]:
|
||||
raise ValueError("release artifact type/size mismatch")
|
||||
with path.open("rb") as stream:
|
||||
if hashlib.file_digest(stream, "sha256").hexdigest() != artifact["sha256"]:
|
||||
raise ValueError("release artifact digest mismatch")
|
||||
with tempfile.TemporaryDirectory(prefix=".dever-release-assets-", dir=output.parent) as scratch:
|
||||
prepared = Path(scratch) / "assets"
|
||||
prepared.mkdir()
|
||||
stem = f"dever-{manifest['platform']}"
|
||||
(prepared / f"{stem}.manifest.json").write_bytes(manifest_bytes)
|
||||
(prepared / f"{stem}.manifest.sig").write_bytes((release / "manifest.sig").read_bytes())
|
||||
with (prepared / f"{stem}.tar.gz").open("xb") as compressed:
|
||||
with gzip.GzipFile(filename="", fileobj=compressed, mode="wb", mtime=0) as stream:
|
||||
with tarfile.open(fileobj=stream, mode="w|", format=tarfile.USTAR_FORMAT) as archive:
|
||||
for artifact in sorted(artifacts, key=lambda entry: entry["path"]):
|
||||
path = release / artifact["path"]
|
||||
header = tarfile.TarInfo(artifact["path"])
|
||||
header.size = artifact["bytes"]
|
||||
header.mode = 0o644
|
||||
with path.open("rb") as contents:
|
||||
archive.addfile(header, contents)
|
||||
publish(prepared, output)
|
||||
return sorted(path.name for path in output.iterdir())
|
||||
|
||||
|
||||
def publish(prepared, output):
|
||||
# Linux renameat2 publishes the entire owned tree without replacing even an
|
||||
# empty directory created by another publisher. Failure cleans only staging.
|
||||
libc = ctypes.CDLL("libc.so.6", use_errno=True)
|
||||
rename = libc.renameat2
|
||||
rename.argtypes = [ctypes.c_int, ctypes.c_char_p, ctypes.c_int, ctypes.c_char_p, ctypes.c_uint]
|
||||
rename.restype = ctypes.c_int
|
||||
if rename(-100, os.fsencode(prepared), -100, os.fsencode(output), 1) != 0:
|
||||
error = ctypes.get_errno()
|
||||
raise OSError(error, os.strerror(error), str(output))
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("release", type=Path)
|
||||
parser.add_argument("--output", required=True, type=Path)
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
for name in create(args.release, args.output):
|
||||
print(name)
|
||||
except (OSError, ValueError, KeyError, TypeError, tarfile.TarError) as error:
|
||||
parser.exit(1, f"cannot prepare release assets: {error}\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in new issue
Block a user