openwork/docs/alchemy-effect-testkit-migration.md

Alchemy + Effect migration proposal for OpenWork testkit

Status: Proposed Decision requested: approve an Effect-first refactor and one gated Alchemy provider pilot; do not approve a wholesale testkit rewrite. Research snapshot: 2026-08-21, Alchemy 2.0.0-beta.72.

Executive summary

Alchemy, Effect, and @openwork/testkit solve different problems:

• Effect is a runtime and composition model for typed failures, dependencies, concurrency, retries, interruption, and scoped acquisition and release.
• Alchemy is a desired-state engine built on Effect. It adds resource identity, dependency graphs, plan/apply/destroy, provider lifecycle methods, and persisted state.
• OpenWork testkit is an application/E2E harness. It decides placement, starts local or Daytona resources, drives OpenWork, records witness evidence, and maps outcomes to Passed/Incomplete/Failed.

The current testkit is mostly ephemeral orchestration, not infrastructure reconciliation. Effect is therefore a strong fit for its internals. Alchemy is a fit only where a resource is external, durable enough to outlive the Node process, discoverable after interruption, and safely reconcilable. The first plausible Alchemy candidate is the desktop sandbox created by the Daytona suite preparer; local child processes, ports, temporary profiles, mocks, users, and organizations should remain scoped Effect resources or application fixtures.

The recommendation is:

1. Keep the public @openwork/testkit API, Vitest runner, witnesses, claims, ambient evidence, and verdict rules unchanged.
2. Introduce Effect behind that API and migrate one lifecycle at a time.
3. Build one custom Alchemy provider for a suite-owned Daytona desktop sandbox only after the Effect boundary is stable.
4. Compare reliability, cleanup, runtime, and maintenance cost against the existing implementation before expanding.
5. Remove the Alchemy pilot if it does not materially improve interrupted-run recovery or leak prevention. Do not keep it merely for architectural consistency.

This is intentionally not a big-bang migration.

Why consider a change

The authoring contract is already good. Specs acquire resources in dependency order with needs() -> server() -> app(), use normal observable assertions, and get ambient evidence without passing recorder handles (evals/README.md). The migration should preserve that.

The maintenance pressure is below the facade:

• server() contains attached, Daytona, and local lifecycles in one function, with hand-written acquisition rollback and disposal paths (evals/packages/testkit/src/server.ts).
• Local Den provisioning coordinates MySQL, Redis, an ephemeral database, ports, two child processes, readiness probes, organization fixtures, logs, and teardown.
• Daytona provisioning coordinates sandbox creation or reuse, checkout, install, display/browser readiness, public URLs, seeded identities, mocks, and conditional deletion (evals/packages/hosts/src/provision.ts).
• Most resources expose AsyncDisposable, but every composition site still decides how partial acquisition, retry, cancellation, and cleanup errors are represented.
• Placement and command execution are concrete dependencies, which makes failure-path testing harder than it needs to be.

Effect can centralize those mechanics without changing what a spec sees. Alchemy may additionally help recover a remotely created sandbox after the test process is interrupted, where in-process disposal cannot run, provided a retry uses stable ownership identity or a janitor can rediscover the resource.

What Alchemy would and would not replace

Alchemy's core loop is:

Stack program
  -> desired resource graph
  -> diff against persisted state
  -> dependency-ordered reconcile/delete
  -> updated state

A custom provider implements required reconcile and delete behavior plus optional hooks such as read, diff, and list as an Effect Layer. A stage isolates physical resources and state. Alchemy's Vitest adapter can deploy and destroy stacks, but it does not provide OpenWork's application-driving, witness, evidence, or verdict semantics.

Alchemy would not replace:

• evals/specs/**/*.test.ts or Vitest;
• @openwork/test-evidence, briefTest, prove.<claim>(), screenshots, or PR publication;
• @openwork/labs deterministic provider witnesses;
• @openwork/behaviors, CDP surfaces, or matchers;
• requirement checks and named skips from needs();
• product code or production deployment;
• Daytona, Docker, MySQL, Redis, Electron, or their underlying APIs.

It would replace only selected imperative external-resource lifecycle code after an OpenWork-specific provider exists. Alchemy has no built-in Daytona provider, so adopting it does not remove the need to understand or test the Daytona lifecycle.

There is also a known runner incompatibility in the reviewed versions. OpenWork evals use Vitest ^3.2.4. alchemy@2.0.0-beta.72 requires Effect >=4.0.0-beta.105 and depends on the matching Effect 4 @effect/vitest, whose peer range requires Vitest >=4.1.0 <5. The Alchemy Vitest adapter therefore cannot be installed as the OpenWork test fixture without upgrading the eval runner. This proposal does not upgrade Vitest: the pilot may use only Alchemy's core programmatic lifecycle behind OpenWork's existing fixture. If pnpm cannot isolate Alchemy's transitive Vitest 4 from the Vitest 3 runner, the Alchemy pilot stops. The repository's existing Effect 4.0.0-beta.83 also does not satisfy Alchemy's peer floor, so the isolated eval workspace would initially carry a newer pinned Effect version.

Decision axes

Effect adoption and Alchemy adoption are separate decisions.

Concern	Effect	Alchemy	Current testkit
Primary abstraction	Typed program with services and scopes	Desired resource with provider lifecycle	Promise-returning test fixture/handle
Normal lifetime	Request, test, process, or explicit Scope	Across deploys through persisted state	One test or prepared suite
Cleanup	Scope finalizers and interruption	destroy/provider delete	AsyncDisposable plus manual rollback
Dependency graph	Service/Layer composition	Resource inputs/outputs	Call order and nested helpers
Recovery after process death	No; the process is gone	Possible through state or deterministic identity plus read	No automatic recovery
Drift/reconciliation	Application-defined	Provider-defined plan/apply	Recreate, reuse, or health-check
Best OpenWork fit	Processes, DB handles, mocks, desktop, fixtures	Owned remote sandboxes/snapshots if discoverable	Spec API, evidence, user journeys

Alchemy is not an always-on controller. A killed CI job still needs a later rerun or janitor to invoke reconciliation/deletion. Its normal plan is also based on persisted state; provider code must observe live state when it reconciles. It must not be presented as universal drift detection.

Goals

1. Preserve the existing spec-authoring and evidence contracts.
2. Make acquisition, partial failure, interruption, and reverse-order release explicit and testable.
3. Replace unstructured lifecycle errors with a small typed error taxonomy while preserving actionable messages at the Vitest boundary.
4. Make local, attached-Den, and Daytona implementations swappable through Layers rather than branches spread through resource constructors.
5. Determine with evidence whether Alchemy improves remote-resource recovery enough to justify its state and provider cost.
6. Keep pnpm and Node.js 22+ as the supported eval toolchain. Bun must not become required.

Non-goals

• Rewriting all specs to Effect.gen.
• Replacing OpenWork's test fixture with alchemy/Test/Vitest.
• Treating users, org membership, OAuth grants, or seeded product data as IaC.
• Persisting test credentials in Alchemy props, outputs, or state.
• Managing an attached Den, an externally supplied reuse, or a borrowed suite-owned sandbox from per-test code.
• Moving eval state to Cloudflare or adding production cloud dependencies.
• Adopting Alchemy for local ports, temporary directories, child processes, or desktop profiles solely because they can be modeled as resources.
• Changing skip, evidence, or verdict semantics as part of the migration.

Proposed architecture

evals/specs + Vitest
        |
        v
@openwork/testkit public Promise/AsyncDisposable facade       unchanged
        |
        v
internal Effect programs and Layers                           new
  - EvalConfig / Place / TestCancellation
  - CommandExecutor / Clock / PortAllocator
  - LocalDatabase / LocalProcess / MockRuntime
  - DenRuntime / DesktopRuntime / DaytonaClient
        |
        +--> existing @openwork/hosts, @openwork/labs,
        |    @openwork/behaviors, @openwork/cdp
        |
        `--> optional Alchemy stack adapter                    gated pilot
               `--> OpenWork DaytonaDesktopSandbox provider

Stable public boundary

Existing specs continue to look like this:

const requirements = needs({ placement: "local" });
await using den = await server({ place, ...requirements });
await using desktop = await app({ den, as: "admin", place });

Internally, the facade opens an Effect Scope, acquires the resource, and returns the existing handle shape. Calling Symbol.asyncDispose closes that Scope. This lets migration happen resource by resource without forcing Effect types into every spec or coupling evidence recording to a new test adapter.

That disposal bridge covers normal lexical exit, not a timeout by itself. The test fixture must retain Vitest's abort signal, interrupt the owning Effect fiber when the signal fires, and close all registered scopes after the test body. The current wrapper narrows the callback context to place, evidence, and skip; carrying cancellation internally is part of Phase 1. Process death still requires startup cleanup or an external janitor.

Internal Effect services

The first internal services should be narrow capabilities, not a mirror of every existing module:

Service	Responsibility	Implementations
EvalConfig	Validated run ID, placement, paths, timeouts	Environment-backed test Layer
TestCancellation	Interrupt fibers and close scopes on runner abort	Vitest signal, deterministic fake
CommandExecutor	Run local or sandbox commands with bounded output	Local, Daytona, fake
PortAllocator	Reserve/release ports	Local, deterministic fake
DatabaseRuntime	Acquire/drop an isolated database	Local MySQL, unavailable on Daytona
MockRuntime	Start/expose/stop a witness	Local, Daytona
DenRuntime	Acquire an attached, local, or Daytona Den	Layer selected once from Place
DesktopRuntime	Acquire and stop an isolated app surface	Local, Daytona

Use scoped acquisition/release for resources and bounded Schedule-based retry for readiness. Expected operational failures should be tagged errors such as RequirementUnavailable, ProvisionFailed, ReadinessTimedOut, and CleanupFailed. Unknown defects remain defects. At the public boundary:

• RequirementUnavailable maps to the existing named SkipError only when it represents a declared requirement;
• provisioning/readiness failures fail the test;
• cleanup behavior remains compatible with each existing handle: several disposers currently log and swallow cleanup errors, and this RFC does not change their verdict semantics;
• error messages retain the command context and bounded log tail used today.

Dedicated lifecycle specs and pilot post-run audits must still fail when they observe a leaked owned resource. That verifies the migration without silently changing the generic Passed/Incomplete/Failed contract. A separate decision is required if all cleanup errors should become test failures.

What belongs in Alchemy

A resource is eligible only if all answers are yes:

1. Can it outlive the test Node process?
2. Can the provider find it again by stable identity without trusting stale local memory?
3. Can reconcile safely run more than once after partial success?
4. Can delete safely run more than once and refuse foreign resources?
5. Is there value in planning/recovery beyond scoped acquire/release?

Initial classification:

Resource	Owner	Rationale
Local port or temporary directory	Effect Scope	Process-local and cheap; persisted state adds no recovery value
Local MySQL database	Effect Scope + startup cleanup	Random per-run name; scoped teardown handles live runners, while a TTL cleanup path is needed after process death
Den API/Web child processes	Effect Scope + startup cleanup	Process supervision, readiness, and interruption are runtime concerns, but detached children can survive a killed runner
MCP mock/fault proxy	Effect Scope	Deterministic witness with a short test lifetime
Electron profile/process	Effect Scope	Caller drives a live process; not desired cloud state
Den org/users/grants	Behavior fixture	Application data with authorization semantics, not infrastructure
Attached Den or externally supplied reuse	External reference	Caller-owned; testkit must never reconcile or delete it
Suite-prepared Daytona sandbox	Runner-owned resource borrowed by tests	Created and deleted in evals/runner/prepare-stack.ts; the normal Daytona path and first Alchemy pilot candidate
On-demand Daytona sandbox	Test-owned resource	Also eligible later, but ownership must remain distinct from external and suite-prepared reuse
Shared Daytona base snapshot	Later candidate	Durable desired artifact, but cache/version/ownership policy must be designed first

Environment variables currently make a suite-prepared sandbox look similar to reuse at the server()/app() layer. The migration must carry provenance explicitly: external, suite-owned, or test-owned, plus the component that is allowed to delete it. A borrowed per-test handle never deletes a suite-owned sandbox; global setup owns its cleanup.

The first provider should expose one OpenWork.DaytonaDesktopSandbox resource inside the suite preparer. Its desired props contain only non-secret identity and configuration: requested git ref, snapshot name, retry-stable ownership ID, and a resolved Daytona organization/account/target fingerprint. Returned attributes contain the sandbox ID, ownership marker, and that fingerprint only. Preview URLs are resolved by a scoped Effect service after deployment and are not persisted in Alchemy state.

Den is deliberately not the first provider. Its provisioner obtains 24-hour signed preview hostnames and bakes those exact hosts into OAuth/MCP identity. Every fresh daytona preview-url call can mint a different hostname. Persisting that URL risks storing a signed capability and later returning an expired endpoint on an Alchemy no-op; regenerating it can break issuer identity. A Den provider requires a separate endpoint-lifetime design.

Provider rules:

• read verifies the sandbox exists and carries the expected run ownership when state is absent or interrupted;
• every operation validates the active Daytona context against the persisted non-secret organization/account/target fingerprint and fails before create or delete on context drift;
• a foreign same-name sandbox fails with a provider-specific ownership error; it is not returned as Unowned, because Alchemy's --adopt mode can take over an Unowned resource;
• reconcile revalidates context, fence, and ownership before its first remote mutation, then observes/ensures and converges checkout and readiness;
• delete treats not-found as success and deletes only owned resources;
• diff updates a git ref in place only if the existing checkout/install path proves convergence; snapshot or ownership changes replace the sandbox;
• replacement uses a generation-specific physical name so create-before-delete cannot collide, or explicitly uses delete-first if Daytona cannot support overlapping generations;
• the first provider omits list; account-wide alchemy unsafe nuke remains unsupported, and TTL cleanup uses a separately ownership-filtered janitor;
• caller-provided reuse is represented as an external reference outside the Stack, never as a managed Resource;
• alchemy unsafe nuke is not part of any OpenWork workflow.

If Daytona CLI/API behavior cannot support those rules, the Alchemy pilot is a no-go. Wrapping the same create/delete script in a provider without reliable read and ownership would add state without adding reconciliation.

Restored terminal Alchemy state is not live proof: unchanged state normally plans a no-op, so neither read nor reconcile is guaranteed to run. Before a restored suite slot is published, the adapter performs a read-only preflight of Daytona context, fence, and ownership; only then may it invoke a supported forced reconcile. The provider repeats those checks immediately before its first mutation to avoid a check/use race. A final live preflight verifies existence, running status, requested checkout, and readiness. A stopped owned sandbox is restarted, a missing owned sandbox is recreated, and a foreign-owned sandbox fails with zero mutations. If Alchemy's core API cannot force this path without the incompatible Vitest adapter, same-state recovery is a no-go.

State, identity, and cleanup

The pilot uses a retry-stable recovery ID supplied by the eval orchestrator, not Date.now(). In CI it should derive from the workflow run, job, explicit matrix/shard identity, and worker slot while intentionally excluding the retry attempt. A rerun of one shard retains its key; concurrent shards must differ. Local runs may generate an invocation ID when cross-run recovery is not requested. A unique stage and resource owner derive from that value, for example eval-<recovery>-w<worker>.

Alchemy beta.72's built-in localState() does not accept a path and captures process.cwd() at module load, so it cannot directly satisfy run-scoped state under the eval result area. Phase 0 must choose one of two isolated mechanisms:

• a conformant custom State Layer rooted under that run's result directory; or
• an Alchemy helper subprocess whose cwd is the run result directory before its first Alchemy import.

Do not mutate cwd inside the shared Vitest process. Preserve the isolated state as a diagnostic artifact on failure and remove it after verified destroy on success. Local state alone does not recover across a new CI machine.

A custom State Layer must prove the same safety properties the pilot depends on: atomic replacement, recovery after interrupted write/delete, defensive handling of missing directories/records, and concurrent disjoint stages. The helper subprocess is preferred if it can reuse Alchemy's built-in local store; path convenience alone does not justify maintaining a state backend.

Cross-machine recovery is a separate gate and does not inherently require remote state. There are three mechanisms to test independently:

• preserve the run-scoped state artifact and resume with the same recovery ID;
• start with empty state but the same deterministic identity and let provider read rediscover the owned sandbox;
• run a Daytona ownership/TTL janitor for jobs that are never retried.

A remote state store is optional. It is useful only if retaining Alchemy's full intermediate lifecycle is worth the additional locking, access-control, and retention surface, and only after parallel workers are proven safe.

Retries that intentionally share a recovery ID need a single-writer lease and fencing token scoped by Daytona context, recovery ID, shard, and worker. Every reconcile, delete, janitor action, and borrowed-host mutation validates the current token. The token is carried into CommandExecutor/the Daytona Host; lease loss interrupts the old test worker and rejects its next operation. A takeover must use a new sandbox generation unless prior-attempt termination is positively confirmed, so an old in-flight command cannot mutate the new attempt's sandbox. The lease is heartbeated by the active attempt; a janitor may delete only after both lease expiry and ownership checks. If this cannot be enforced with the Daytona API and runner, shared-identity retry is a no-go.

The janitor is required before claiming that the migration prevents leaks. Alchemy only cleans up when a process runs destroy; it cannot finalise a job after SIGKILL by itself.

No token, cookie, password, API key, signed preview URL, or raw environment map may enter Alchemy state. Secrets and transient endpoints stay in Effect services and are resolved inside or immediately after provider operations. Remote state is out of scope until its encryption, access control, retention, and locking are reviewed.

Effect Scope also cannot clean detached local children or databases after SIGKILL. Phase 2 may only claim in-process cleanup improvements unless it adds an ownership-aware startup/TTL cleanup path and proves that path against a forcibly terminated runner.

Migration plan

Phase 0: compatibility and boundary spike (2-3 engineer-days)

No production lifecycle moves in this phase.

1. Pin exact alchemy, effect, and @effect/platform-node versions in the isolated evals/ workspace; do not depend on @next and do not add @effect/vitest to OpenWork's runner.
2. Use an Effect version satisfying Alchemy's reviewed >=4.0.0-beta.105 peer range. Verify pnpm + Node.js 22 execution, TypeScript 5.9, Vitest 3.2, ESM exports, and the existing eval tsconfig. Bun remains optional.
3. Prove an internal Effect Scope can return an existing AsyncDisposable facade, that ambient evidence survives Effect fibers, and that a Vitest abort signal interrupts the fiber and closes the Scope.
4. Exercise a fake lifecycle provider from a normal @openwork/testkit app-less spec, using Alchemy core rather than alchemy/Test/Vitest. Cover create, update, replace, repeated reconcile, partial failure, delete, and not-found delete.
5. Prove either the custom State Layer or pre-import helper-process approach can isolate state under one eval result directory; the built-in localState() is not configurable enough in the reviewed release. If custom, test atomic writes, interrupted write/delete recovery, missing records/directories, and concurrent disjoint stages.
6. Prove the Alchemy core API can force reconciliation of unchanged terminal state without adopting alchemy/Test/Vitest.
7. Measure install-size and cold-import impact of Alchemy's transitive graph.

Gate: stop Alchemy work if it requires replacing the OpenWork test fixture, requires upgrading Vitest, leaks a second Vitest into runner resolution, requires Bun, or cannot run through the existing pnpm/Vitest lane. Effect-only work may continue.

Phase 1: Effect foundation and one scoped resource (3-5 engineer-days)

1. Add the internal Scope-to-AsyncDisposable bridge and typed lifecycle errors.
2. Carry Vitest cancellation into the fixture and interrupt all registered Effect fibers/scopes while the process remains alive.
3. Add fake CommandExecutor, Clock, and PortAllocator Layers.
4. Migrate mcpMock() acquisition/readiness/stop behind the existing API.
5. Fault-test failure before acquisition, during readiness, in the test body, on timeout/cancellation, and during finalization.

Gate: existing local specs are source-compatible; evidence output and skip classification are unchanged; timeout interrupts the fiber; no process or port remains after any in-process injected failure.

Phase 2: local Den orchestration (5-8 engineer-days)

Migrate the local server() branch to a composed Effect program:

requirements
  -> mocks + database + ports
  -> db push
  -> den-api + optional den-web
  -> health/auth readiness
  -> org/member fixtures
  -> Den handle

Keep organization provisioning as an application fixture in testkit; moving the current private provisioning/deletion functions to @openwork/behaviors is not part of this phase. Keep attached Den as a non-owning Layer. Preserve existing logs and endpoint shapes.

Gate: targeted specs pass in both the normal path and injected failures; database, services, mocks, and organizations are released in reverse order; cleanup logging and verdict behavior match the existing handles. Dedicated lifecycle specs assert that every finalizer is attempted. Any claim about cleanup after process death additionally requires a local stale-resource janitor and forced-termination proof.

Phase 3: one Alchemy Daytona pilot (5-8 engineer-days)

1. Add the OpenWork.DaytonaDesktopSandbox resource/provider in a pilot-only internal package or module.
2. Use an in-memory state store for provider lifecycle tests and run-scoped local state for one real E2E journey.
3. Integrate it at evals/runner/prepare-stack.ts, where suite ownership and global cleanup actually live. Per-test consumers continue borrowing the prepared desktop sandbox and cannot delete it.
4. Add an explicit force-suite-preparation mode for the comparison. A single explicitly selected test currently bypasses suite preparation, so each arm must assert that it received a suite-owned slot and record whether the old or Alchemy preparer created it.
5. Put the provider behind an explicit pilot selection; default Daytona tests continue using the current suite provisioner during comparison.
6. Run the same selected journey through old and pilot paths against the same git ref and resolved Daytona context fingerprint.
7. Coordinate the current Den branch and Alchemy desktop branch so an outer failure waits for both acquisitions to settle, then destroys every owned resource before returning. Inject a Den failure while desktop creation is in flight and require the ownership audit to find nothing.
8. Simulate interruption after remote create, then prove both same-state resume through forced live reconcile and empty-state read recovery with the same recovery ID. Seed terminal state against stopped, missing, wrong-checkout, unready, and foreign-owned sandboxes; the foreign-owned case must record zero remote mutations. Prove the TTL janitor handles a run that is never resumed and does not delete an active newer retry sharing that recovery ID.
9. Keep Den provisioning on the current path; signed endpoint identity is a separate design gate.
10. Run overlapping retry attempts: lease loss must interrupt the old worker, reject its next host mutation, and prevent it from changing the new generation.

Gate: do not make Alchemy the default until the go/no-go criteria below are met. Do not migrate desktop and Den sandboxes simultaneously in the first pilot.

Phase 4: decide, then expand or remove (2 engineer-days for decision)

Possible outcomes:

• Effect only: remove Alchemy dependencies and retain the scoped runtime refactor.
• Selective Alchemy: make the proven desktop sandbox provider the Daytona suite-preparation default; consider a separately reviewed snapshot provider.
• Stop/revert: keep the original lifecycle behind the stable facade and remove the pilot modules/state.

A full conversion of all testkit resources is not an expected outcome and would require a new RFC.

Go/no-go criteria

The pilot is a go only when all are true:

Correctness

• Existing selected specs require no source changes.
• Every existing claim still has observable test evidence.
• Comparison evidence proves suite preparation ran and identifies the selected old or Alchemy preparer; an on-demand fallback cannot satisfy the pilot.
• External reuses and per-test borrows of suite resources are never deleted; suite global cleanup deletes the suite-owned sandbox exactly once.
• Owned resources are deleted after pass, assertion failure, readiness failure, and runner cancellation while the process remains alive.
• A post-create interruption has a tested recovery or TTL janitor path.
• Restored terminal state is forced through live reconciliation/preflight; stopped, missing, stale-checkout, unready, and foreign-owned resources cannot pass as a no-op.
• Read-only ownership/context/fence validation runs before forced reconcile and again before its first mutation; the foreign-owned case makes zero mutations.

Reliability

• Repeated reconcile and delete are idempotent in provider lifecycle specs.
• A 20-run comparison leaves zero local processes/databases and zero owned Daytona sandboxes in both paths.
• Parallel workers use disjoint names and state; no state corruption or cross-run deletion is observed.
• Concurrent matrix shards receive distinct recovery identities, while a rerun of the same shard retains its identity.
• Retrying under a different Daytona organization/account/target fails before reconcile or delete and cannot leak or duplicate a resource silently.
• Overlapping attempts with the same recovery ID are fenced to one writer, and a TTL janitor cannot delete the active newer attempt's sandbox.
• Lease loss interrupts the old worker and blocks borrowed-host mutations; a takeover does not hand the same sandbox generation to both attempts.
• Injected Den failure during desktop creation waits for the desktop branch and leaves no suite-owned sandbox.
• A dedicated post-run ownership audit fails the pilot comparison if any sandbox remains. Generic disposer errors retain current log-and-continue behavior until a separate verdict-policy decision.
• Same-state resume and fresh-state read recovery both use a documented, retry-stable identity rather than a new timestamp.

Performance

• Local testkit cold start regresses by no more than 10% at p50.
• The selected Daytona journey regresses by no more than 10% at p50 and 20% at p95, unless the added time demonstrably replaces flaky retry work.
• Dependency install and cache impact is measured and accepted explicitly.

Maintainability

• The custom provider plus adapter is smaller and easier to fault-test than the imperative suite lifecycle it replaces.
• Error output remains at least as actionable as current timed-step and log-tail messages.
• At least two maintainers can modify the provider without relying on Alchemy internals or undocumented APIs.

Failing any ownership, endpoint-security, cleanup, state-concurrency, or fixture-compatibility criterion is a no-go, not a follow-up to absorb during rollout.

Tradeoffs

Benefits of Effect-first

• One model for acquisition, interruption, timeout, retry, and finalization.
• Dependencies become replaceable Layers, enabling deterministic failure-path tests without shelling out to real infrastructure.
• Typed operational errors make skip/fail classification deliberate.
• Local and Daytona implementations can share orchestration while differing at capability boundaries.
• Incremental adoption is possible behind Promise and AsyncDisposable APIs.
• OpenWork already has limited Effect 4 usage in packages/codemode and ee/apps/den-api, so the concepts are not entirely new to the repository.

Costs of Effect-first

• Effect, Layer, Scope, Cause, and interruption add a learning curve.
• Poorly chosen services can turn straightforward code into indirection.
• Effect failure channels plus Vitest errors plus evidence verdicts create three taxonomies that must have one explicit boundary.
• Stack traces and debugging can worsen if every helper is wrapped rather than only lifecycle boundaries.
• Effect 4 is currently beta in this repository, so version pinning and upgrade work remain real.
• Alchemy's peer floor is newer than the Effect beta currently pinned elsewhere in OpenWork, creating temporary version skew even though evals are isolated.
• Scoped finalizers improve timeout and in-process failure handling but do not clean detached processes or databases after a killed runner; a janitor is a separate mechanism.

Additional benefits of selective Alchemy

• Stable resource identity and a dependency graph replace implicit remote creation order.
• Plan/apply/destroy makes ownership and replacement decisions inspectable.
• Persisted intermediate lifecycle state can support recovery after partial remote provisioning.
• Provider lifecycle tests exercise create/update/replace/delete consistently.
• Stages give parallel runs an explicit isolation key.

Additional costs and risks of Alchemy

• Alchemy's README labels the project alpha and its current v2 package is beta; breaking changes should be expected.
• The reviewed Alchemy/Effect Vitest adapter requires Vitest 4.1+, while OpenWork evals use Vitest 3.2; using the adapter would force an unrelated runner migration.
• It brings a broad IaC dependency graph into an otherwise narrow eval workspace.
• There is no built-in Daytona provider. OpenWork must own read, diff, reconcile, delete, auth, ownership, and upgrade compatibility.
• Most testkit resources are short-lived runtime resources; modeling them in persisted desired state would be conceptual and operational overhead.
• State introduces retention, locking, concurrency, corruption, and secret handling concerns that do not exist for in-memory handles.
• The reviewed local state backend has no configurable path; isolation requires a helper process or a correctly crash-safe custom State Layer.
• Retry reuse needs leasing and fencing across both IaC lifecycle calls and the test processes borrowing a sandbox, which may outweigh the provider benefit.
• Daytona Den preview hosts are expiring signed values and OAuth/MCP identity; persisting them as attributes or refreshing them on a no-op is unsafe without a dedicated endpoint-lifetime design.
• A custom provider can become the same homegrown provisioner behind a more complex interface. Alchemy is valuable only if the provider truly converges.
• Alchemy's documented test posture favors real cloud resources. OpenWork must retain deterministic witnesses because identity, failure, and policy claims require controlled observations.
• Alchemy cannot clean up after a dead runner without a subsequent command or external janitor.
• Adopting Alchemy's Vitest adapter directly could conflict with OpenWork's evidence fixture and skip mapping; this proposal avoids that coupling.

Alternatives considered

Option	Advantages	Disadvantages	Decision
Keep current implementation	No migration or dependency cost; familiar debugging	Manual rollback and concrete dependencies remain; no interrupted remote recovery	Valid fallback
Effect-only internals	Best fit for ephemeral resources; incremental; deterministic failure testing	No persisted desired state or post-process recovery	Recommended baseline
Effect + selective Alchemy	Adds recovery/plan semantics where remote resources justify them	Provider/state/maturity cost	Recommended gated pilot
Full Alchemy rewrite	One resource vocabulary everywhere	Poor fit for app fixtures and local processes; highest churn; risks evidence contract	Reject
Terraform/Pulumi/CDK for evals	Mature IaC/state ecosystems	Separate language/runtime or heavier bridge; still does not replace test orchestration	Reject for this scope
Kubernetes/Docker-only test environments	Strong declarative runtime where supported	Does not cover Electron/Daytona/attached Den and changes current lanes	Out of scope

Rollback strategy

• The public testkit facade remains the rollback seam throughout.
• Each migrated resource keeps one implementation selected at composition time; specs do not branch on implementation.
• Pilot resource names and state are isolated from current Daytona names.
• No production or shared long-lived state is converted.
• Reverting means selecting the existing provisioner, destroying pilot-owned resources, deleting pilot state, and removing Alchemy dependencies.
• Effect phases are independently reversible; no spec should require Effect types to compile.

Open questions before Phase 3

1. Does the Daytona API expose stable metadata/tags, or only CLI names? This determines whether ownership-safe read is possible.
2. What exact workflow/job/matrix-shard/worker fields form a retry-stable and shard-unique recovery ID across local, GitHub, and Daytona lanes?
3. Which stable Daytona organization/account/target identifiers can be resolved without persisting credentials?
4. Where does the single-writer lease/fencing record live, and how is its heartbeat distinguished from a cancelled attempt? Can takeover isolate the new sandbox generation from old in-flight commands?
5. What is the concurrency/locking contract of the selected state store under parallel Vitest workers and retried CI jobs?
6. Can Alchemy's programmatic deploy/destroy path be used without installing a second Vitest fixture around the test, despite its transitive Effect 4 @effect/vitest dependency?
7. Which exact Alchemy/Effect versions are mutually compatible with the eval workspace at implementation time?
8. Where should the ownership/TTL janitor run so it still executes when a CI job is cancelled and never retried?
9. Can a future Den provider avoid persisting signed preview hosts while preserving the exact issuer identity baked into the sandbox?

Whether generic cleanup errors should become Failed or Incomplete is a separate evidence-policy question. The current implementation often logs and swallows them; this migration preserves that behavior rather than silently choosing a new verdict.

Requested decision

Approve Phases 0-2 as an Effect-first internal migration and approve Phase 3 only as a time-boxed Alchemy experiment. Do not approve Alchemy as the new testkit foundation until the provider meets every go/no-go criterion.

The likely end state is a hybrid:

Vitest + OpenWork evidence contract
  + Effect for test resource orchestration
  + Alchemy for a small set of owned, recoverable remote resources

That uses each tool at the layer it is designed for and avoids replacing a working E2E harness with an IaC engine that does not solve its core evidence problem.

References

OpenWork:

• evals/README.md
• evals/packages/testkit/src/fixture.ts
• evals/packages/testkit/src/server.ts
• evals/packages/testkit/src/app.ts
• evals/packages/testkit/src/place.ts
• evals/packages/hosts/src/provision.ts
• evals/packages/test-evidence/src/ambient.ts
• evals/runner/prepare-stack.ts
• evals/runner/stack-env.ts
• evals/runner/stack-suite.ts
• .devcontainer/test-server-on-daytona.sh

Alchemy (reviewed 2026-08-21):

• Repository and alpha notice
• What is Alchemy?
• Custom providers
• Resource lifecycle
• State store
• Test harness
• Local development
• Getting started and Node.js support