Skip to content
All work
SYSTEMS RESEARCH2024

Adytum

A self-hosted, event-driven orchestration runtime, built around the distributed-systems problems behind autonomous task execution.

A self-hosted orchestration runtime built to explore four distributed-systems problems: tiered task delegation without recursion failure, a dual-layer persistence model over SQLite and FTS5, a hot-reloading plugin architecture, and process lifecycle management on a fixed local compute budget.

3
Execution tiers with enforced separation
2
Persistence layers with distinct query shapes

The problem

Strip away the framing and this is a scheduling and state problem. A single worker that can both plan and execute will loop on a task it cannot finish. A process that accumulates context without a consolidation strategy runs out of memory budget. A plugin system that requires a restart to load a capability is not a plugin system. And a runtime on one machine cannot keep every worker resident. Adytum exists to work through those four problems in code rather than in a design document.

The approach

  • Tiered delegation with a hard separation of concerns

    A three-tier command structure where the planning tier is programmatically prohibited from executing tools and must emit a delegation plan. Planners that cannot act cannot loop; executors that cannot plan cannot wander.

  • Dual-layer persistence

    An episodic log with FTS5 full-text plus vector retrieval for relevance, and a separate entity-relationship graph for durable facts. Two access patterns with genuinely different query shapes, so they get two stores rather than one compromise.

  • Hot-reloading plugin architecture

    A loader discovers manifests at runtime, a filesystem watcher triggers a live refresh of the tool registry, and dependency guards keep a capability inactive unless its OS binaries and secrets are actually present.

  • Lifecycle management on a fixed budget

    Idle workers are serialised to disk and evicted from memory; completed ones move to an append-only historical log. Local compute is finite, so process lifecycle is a design problem rather than an operational one.

System architecture

How it is put together

  1. Orchestration tier

    Deterministic planning layer that emits delegation plans and is barred from tool execution

  2. Worker tiers

    Managers fan out parallel batches; ephemeral executors run one task and report back

  3. Persistence

    SQLite episodic log with FTS5 and vector retrieval, plus a separate entity-relationship fact graph

  4. Plugin loader

    Manifest discovery, filesystem-watched hot reload, and dependency guards on OS binaries and secrets

  5. Lifecycle manager

    Serialise-and-evict for idle processes, append-only history for completed ones

  6. Event backbone

    Scheduled heartbeats, idle-time consolidation passes and filesystem sensors driving work without a request

  7. Observability

    Next.js control surface over Socket.io, streaming live runtime metrics

Engineering decisions

The calls worth defending

A planner that cannot execute

The failure mode of a single worker that both plans and acts is an infinite retry on a task it cannot complete. Enforcing the separation in the runtime, rather than in instructions, means the loop is structurally impossible rather than discouraged.

Two stores, because there are two query shapes

'What happened recently that resembles this?' and 'what do we know to be true?' are a relevance search and a graph traversal. Forcing both through one schema gives you a store that is mediocre at each. Two stores with a consolidation pass between them is more code and a great deal less compromise.

Serialise and evict, rather than cap concurrency

Capping workers makes the system fail at a threshold. Serialising idle ones to disk and restoring on demand makes it degrade instead: slower, not broken, which is the correct behaviour when the compute budget is one machine.

Interface

What shipped

  • Adytum: Runtime topology

    Runtime topology

    Live view of the tiered worker graph and delegation edges

  • Adytum: Execution trace

    Execution trace

    Streaming task output with step-level inspection

  • Adytum: Plugin registry

    Plugin registry

    Manifest discovery, hot reload and dependency guards

In depth

What this project is actually about

Adytum is a research prototype, and the honest framing is that the interesting parts are not the autonomous behaviour. They are the four systems problems underneath it. Each one has a well-understood analogue in ordinary backend engineering, which is why the project was worth building.

Tiered delegation, because planners that act will loop

A single process that both decides what to do and does it has a predictable failure mode: it retries a task it cannot complete, forever, with no external signal that anything is wrong.

The runtime splits that into three tiers with enforced boundaries. The top tier holds the goal and is programmatically prevented from calling any tool: its only legal output is a delegation plan. The middle tier orchestrates workflows and fans work out in parallel batches. The bottom tier is ephemeral executors that do one thing and report back.

The important word is programmatically. An instruction not to loop is a suggestion. A runtime in which the planning tier has no tool-calling capability at all makes the loop structurally impossible, which is the difference between a guardrail and a wish.

Two persistence layers, because there are two questions

The system needs to answer "what happened recently that resembles this?" and "what do we know to be true?". Those are a relevance search and a graph traversal. They want different indexes, different write paths and different retention rules.

So there are two stores. An episodic log in SQLite, indexed with FTS5 and vector retrieval, holds every message, call and intermediate step; retrieval is by relevance to the current context. A fact graph holds discrete entity-relationship claims extracted from that log: durable, small, and queried directly rather than searched.

A consolidation pass runs during idle time: it compacts verbose history into summaries and promotes stable observations from the log into the graph. It is the same pattern as an analytics rollup job, and it exists for the same reason: the raw log grows without bound and the useful shape of it does not.

Hot reload, because a restart is a failed abstraction

Capabilities are plugins. A loader scans the workspace for manifests at runtime; a filesystem watcher triggers a live refresh of the tool registry and the runtime's instruction set when a plugin changes on disk.

Dependency guards are the part that matters in practice. A plugin declares what it needs (an operating system, a binary on the path, an API secret) and stays inactive if any of it is missing. The alternative is a capability that registers successfully and then fails at the moment it is first used, which is a much more expensive way to find out.

Lifecycle management on one machine

The runtime is self-hosted, so the compute budget is fixed and small. Capping concurrency would make the system fail sharply at a threshold. Instead, idle workers are serialised to disk and evicted from memory, restored on demand; completed ones are moved to an append-only historical log for later inspection.

The result degrades under load rather than breaking, which is the property you want when the ceiling is one box and there is no autoscaler to call.

The event backbone

Work is not only request-driven. Scheduled heartbeats evaluate progress against long-running goals, idle periods trigger the consolidation pass, and filesystem sensors turn external changes, such as a new commit or a modified file, into scheduled work. It is a job queue with a cron and a watcher attached, and describing it that way is more useful than describing it as autonomy.

Stack: Node.js, Fastify and TypeScript with constructor injection; SQLite with FTS5 for the episodic layer and JSON-backed storage for lifecycle state; a Next.js control surface over Socket.io streaming live runtime metrics.