01
Stale completed order
Commerce completed, persisted order state is stale.
Authority
AUTOMATIC
RECONCILE_ORDER_STATE
Outcome
Reconcile
The system repairs stale internal state automatically when deterministic reconciliation invariants still hold.
← Selected work
AI systems / operations
AI Ops Investigator
Reason freely.Act deterministically.
A production-oriented AI operations system that investigates ambiguous incidents, grounds remediation in operational knowledge, and executes only through deterministic safety controls.
Agentic investigation
Grounded RAG
Human approval
Crash-safe automation
MCP
Evals
Incident
Order stuck in processing
ORD-18472
Investigate
adaptive evidence gathering
01
Order state
get_order
PROCESSING
02
Payment
get_payment_state
CAPTURED
03
Fulfillment
get_fulfillment_attempts
FAILED
04
History
get_order_event_history
Failure confirmed
Diagnosis
Confirmed fulfillment failure
confidence
HIGH
Grounded remediation
Create a replacement fulfillment attempt
CREATE_FULFILLMENT_ATTEMPT
Supported by retrieved runbook
Deterministic safety boundary
Allow
Require approval
Deny
REQUIRE_APPROVAL
Approval
Exact action approved
ONE-TIMEEXPIRING
Execute
Mutation audited
EFFECT RECORDED
Verify
Authoritative state confirmed
COMPLETED
The autonomy problem
Reasoning can be probabilistic.Authority cannot be.
An LLM can investigate evidence, form hypotheses, and propose what should happen next. But its recommendation should never be equivalent to permission to change production.
Model output is input to policy.
It is not permission to act.
Coupled authority
reasoning + permission
LLM agent
One control surface
PROBABILISTIC
01
Observe
Read production state
02
Reason
Choose remediation
03
Authorize
Decide whether to act
04
Mutate
Change production
A probabilistic decision can cross directly into an external side effect because reasoning and authority belong to the same actor.
MODEL DECISION = OPERATIONAL AUTHORITY
Separated authority
implemented architecture
Reasoning plane
The model can explore uncertainty.
STOCHASTIC
01
Investigate
02
Retrieve
03
Recommend
Deterministic boundary
Control plane
Code decides what production is permitted to do.
DETERMINISTIC
01
Map capability
02
Apply policy
03
Validate state
04
Execute
The model can change its reasoning path without gaining new capabilities.
Production authority remains explicit, deterministic, and independently enforceable.
Adaptive investigation
The agent chooses what evidence it needs next.
I avoided encoding the investigation as a fixed decision tree. The model receives narrowly scoped read capabilities and decides which missing evidence can materially reduce uncertainty.
Everything around that reasoning loop remains deterministic: tool schemas, validation, budgets, duplicate prevention, persistence, and the final assessment contract.
Stop when the operational cause is established.
Keep investigating while available evidence can materially change the diagnosis.
Investigation run
ORD-1001
Reported symptom
Order is still processing.
CURRENT STATE
PROCESSING
Available evidence
01
Current order
get_order
02
Payment state
get_payment_state
03
Fulfillment
get_fulfillment_attempts
04
Delivery
get_delivery_state
05
Notification
get_notification_state
06
Business history
get_order_event_history
07
Processing trace
get_order_processing_trace
Working hypothesis
The visible order state is the symptom. The cause is still unknown.
Commerce completed successfully. The final state transition failed to persist.
Evidence established
Payment
CAPTURED
Fulfillment
SUCCEEDED
Delivery
SUCCEEDED
Completion transition
DB TIMEOUT
Assessment
DIAGNOSIS_READY
Root cause
Infrastructure
Confidence
HIGH
Causal distinction
PROCESSING is stale state, not the root cause. The database timeout prevented FULFILLED from being persisted.
07
read capabilities
08
tool-call maximum
30s
active execution budget
1×
identical call allowed
Context & provenance
Context is not truth.
Tool output, durable state, runbook knowledge, and model inference carry different epistemic weight. The context model keeps those origins explicit instead of flattening everything into prompt text.
Successful reads can become observations. Failed tool calls remain part of the execution ledger, but they do not become evidence that a business state is absent.
Model remembers ≠ system knows.
Provenance model
fact kind + source
OBSERVATION
TOOL
Point-in-time evidence returned by an operational read.
DURABLE_FACT
STORE
State the system can establish from durable storage.
KNOWLEDGE
RUNBOOK
Operational guidance retrieved from controlled knowledge.
INFERENCE
MODEL
A model conclusion, never silently promoted into fact.
Durable tool-execution ledger
raw history
01
get_order
SUCCEEDED
02
get_payment_state
SUCCEEDED
03
get_fulfillment_attempts
SUCCEEDED
04
get_notification_state
recorded, not projected as a business fact
FAILED
The ledger answers which tools ran and whether they succeeded. Failure records execution history, not absence of business state.
Projected observations
model-facing evidence
OBSERVATION 01
tool:get_order
Order status is PROCESSING.
observedAt preserved
OBSERVATION 02
tool:get_payment_state
Payment status is CAPTURED.
observedAt preserved
OBSERVATION 03
tool:get_fulfillment_attempts
Fulfillment attempt status is SUCCEEDED.
observedAt preserved
FAILED EXECUTION
No observation projected.
Investigation resume
Reconstruct model context from durable state
Provider continuity
previousResponseId
null
Resume does not depend on the provider preserving the previous conversation state.
Reconstructed input
previousResponseId
null
confirmed working facts
included
unresolved questions
included
tool execution ledger
included
projected observations
included
Durable executions remain the source material. Working facts and projected observations are reconstructed from them before the model continues.
Knowledge & grounding
Relevant knowledge is not permission to act.
Remediation starts only after diagnosis. The model can search operational runbooks, but every recommendation must retain the identity of the knowledge that supports it.
The application then independently checks whether those runbooks were actually retrieved and whether they are configured to support the exact executable capability.
Retrieval establishes relevance.
Code establishes executable meaning.
Grounded remediation
confirmed fulfillment failure
Diagnosis
Payment captured. Fulfillment terminally failed. No entitlement or delivery exists.
FULFILLMENT
Search
search_runbooks
confirmed terminal fulfillment failure replacement attempt
MAX 3 SEARCH CALLS
Retrieved knowledge
RUNBOOK-CONFIRMED-
FULFILLMENT-FAILURE
Known terminal failure may permit a replacement attempt, subject to policy and approval.
Structured recommendation
disposition
PRIMARY
actionKind
CREATE_FULFILLMENT_ATTEMPT
supportedByRunbookIds
[RUNBOOK-CONFIRMED-FULFILLMENT-FAILURE]
Independent grounding checks
01
Was the cited runbook actually retrieved?
YES
02
Does that runbook support this exact action kind?
YES
03
Does the recommendation contain one executable primary?
YES
Deterministic derivation
Exact executable capability established
ACTION_READY
Semantic precision
unknown fulfillment
Retrieved runbook
RUNBOOK-UNKNOWN-
FULFILLMENT
Reconcile provider state before retrying.
Required operation
Provider-state reconciliation
EXACT CAPABILITY
NOT AVAILABLE
Safe result
disposition = CONSTRAINT
actionKind = null
Escalate without mutation
NO EXECUTABLE PRIMARY
“Reconcile” does not map to RECONCILE_ORDER_STATE simply because the words are similar.
Capabilities are exact contracts. Missing capability means safe degradation, not approximation.
Retrieval workbench
implemented retrieval strategies
Lexical
ACTIVE LOOP
Current agentic remediation search
deterministic token overlap
Semantic
Implemented retrieval variant
text-embedding-3-small
LLM reranking
Implemented retrieval variant
bounded candidate set
The active remediation agent currently uses the lexical search_runbooks capability. Semantic retrieval and bounded LLM reranking are implemented separately so retrieval quality can be evaluated without changing the grounding contract.
Action safety
Approval is not a bypass.
Once remediation becomes an executable action, the model leaves the control path. Deterministic code decides whether the action is allowed, requires a person, or is unavailable entirely.
Even an approved action must still be safe against current production state immediately before execution.
Human approval grants authority.
It does not invalidate system invariants.
Deterministic action policy
model output excluded
RECONCILE_ORDER_STATE
ALLOW
Safe only when execution-time reconciliation invariants pass.
RETRY_NOTIFICATION
ALLOW
Does not alter payment or fulfillment state.
CREATE_FULFILLMENT_ATTEMPT
REQUIRE_APPROVAL
Could create duplicate external fulfillment.
ISSUE_REFUND
REQUIRE_APPROVAL
Financial consequence requires explicit human approval.
CANCEL_ORDER
DENY
Manual operation unavailable to the agent.
Approval path
CREATE_FULFILLMENT_ATTEMPT
Exact action snapshot
kind
CREATE_FULFILLMENT_ATTEMPT
orderId
ORD-18472
reason
Grounded remediation requires creating a new fulfillment attempt.
An approval must match all three fields exactly.
Approval record
persisted
Pending
requested
Approved
resolvedBy
Consumed
not yet
expiresAt
time bounded
consumedByExecutionId
null
Approval is exact, expiring, and can authorize only one execution.
Fresh-state validation
Approval exists. The system now rereads authoritative execution state.
validateActionExecution()
Current invariants
execution-time read
Blocking fulfillment attempt
NONE
Fulfillment already succeeded
NO
Active entitlement
NO
Account delivery completed
NO
Current order state
PROCESSING
Authorization + state validation passed
READY_TO_EXECUTE
01
CONSUMED
Consume approval
Bind the approval to this execution ID before the side effect.
02
PRECONDITIONS
Validate at write boundary
Re-read the mutable state immediately before creating the attempt.
03
PENDING
Create attempt
Only after both authorization and current-state checks still hold.
If state changes between the first validation and the write, the repository returns PRECONDITION_FAILED instead of forcing the mutation through.
FAIL CLOSED
Execution & verification
Execution is not completion.
Every action creates a durable execution audit. Successful mutation records a structured effect describing exactly what the executor claims to have changed.
The workflow then enters a separate verification phase and rereads authoritative state before it can declare the operation complete.
Executor says what it did.
Verification proves what exists.
Automation state
durable state machine
01
EXECUTING
02
VERIFYING
03
COMPLETED
Action execution audit
ACT-...
status
EXECUTED
action.kind
CREATE_FULFILLMENT_ATTEMPT
initiatedBy
SYSTEM / automationRunId
approvalId
APR-...
effect.kind
FULFILLMENT_ATTEMPT_CREATED
attemptId
FA-20481
attemptStatus
PENDING
The audit persists the execution result as structured data rather than requiring downstream recovery logic to interpret a prose message.
Independent verifier
authoritative reread
01
Audit action matches derived action
MATCH
02
Execution audit status
EXECUTED
03
Approval reference matches automation
MATCH
04
Structured execution effect
VALID
05
Created fulfillment attempt exists
FOUND
06
Current attempt state
PENDING
Fulfillment verification accepts the created attempt only when it can be found again and its current state is PENDING, ACTIVE, or SUCCEEDED.
Structured effects
execution contracts
ORDER_STATE_RECONCILED
previousStatus → currentStatus
NOTIFICATION_RETRY_CREATED
notificationId + notificationStatus
FULFILLMENT_ATTEMPT_CREATED
attemptId + attemptStatus
Reconciliation, notification retry, and fulfillment-attempt creation each emit a typed effect with the identifiers or state transition required for later verification and recovery.
Postcondition satisfied
Transition automation to COMPLETED
VERIFYING → COMPLETED
Postcondition not satisfied
Escalate instead of assuming success
VERIFYING → ESCALATED
Crash recovery
Crash at the worst possible moment.
External side effects and local durable state cannot be wrapped in one atomic transaction. A worker can die after production changed but before the coordinator knows it.
Recovery therefore asks a different question from retry logic: can the system prove what happened?
Absence of known success
≠ proof of failure.
Dangerous crash window
consequential external write
01
Approval
CONSUMED
02
Execution audit
STARTED
03
External request
SENT
04
Provider
MAY CHANGE
05
Worker
CRASH
After restart, coordinator state alone cannot tell whether the mutation never happened, succeeded, or may have succeeded.
EXECUTING IS NOT EVIDENCE
Case A
Known durable success
RECOVER
Automation
EXECUTING
coordinator is stale
Approval
CONSUMED
execution correlation exists
Audit
EXECUTED
matching action + approval
Effect
FULFILLMENT_ATTEMPT_CREATED
exact attempt identity persisted
Recovery decision
01
Do not execute again
02
Associate existing execution
03
Move to VERIFYING
04
Reread authoritative state
Existing side effect recovered
RECOVERED_EXECUTION
Case B
Side effect outcome unknown
ESCALATE
Automation
EXECUTING
coordinator is inconclusive
Approval
CONSUMED
authority was already spent
Audit
STARTED
no terminal execution result
Effect
NONE
absence does not prove the provider did nothing
Recovery decision
01
Do not replay the write
02
Preserve execution correlation
03
Require reconciliation
04
Escalate before any retry
Replay blocked
ESCALATE
Recovery evidence
no single field is sufficient
AutomationRun.status
Approval.status
Approval.consumedByExecutionId
ActionExecution.status
ActionExecution.effect
Authoritative operational state
Evaluation
Same system.Different authority.
The goal was never maximum automation. It was appropriate autonomy: act automatically when evidence and consequences are bounded, require a person when authority matters, and stop when the system cannot establish a safe action.
The canonical safety scenarios exercise those different outcomes through the same investigation, remediation, policy, execution, and verification architecture.
Task success and safety
are measured separately.
Canonical incidents
same architecture
02
Confirmed fulfillment failure
Payment captured, fulfillment terminally failed, no delivery exists.
Authority
HUMAN APPROVAL
CREATE_FULFILLMENT_ATTEMPT
Outcome
Replace
A replacement attempt is supported by evidence and runbook knowledge, but the consequential write still requires exact approval.
03
Unknown fulfillment outcome
Provider outcome cannot be established safely.
Authority
NO MUTATION
actionKind = null
Outcome
Escalate
The required provider-state reconciliation capability does not exist, so the system refuses to approximate one.
04
Notification-only failure
Order completed successfully; customer notification failed.
Authority
AUTOMATIC
RETRY_NOTIFICATION
Outcome
Retry
The failure is isolated from payment and fulfillment state, so only the failed notification is retried.
Engineering evidence
Deterministic unit coverage validates the control surfaces around the model. The documented live safety suite then exercises the model-backed system against the four canonical operational scenarios.
View source on GitHub ↗60
test files
Vitest
228
tests passed
0 failed
4/4
live safety scenarios
documented suite
0
safety violations
documented suite
UNIT TEST RUN
60 test files passed · 228 tests passed · 0 failed
LIVE SAFETY SUITE
4 scenarios passed · 0 safety violations documented
Observability & production boundaries
Measure the agent.Keep the boundaries explicit.
A correct diagnosis is not enough to understand whether an agent is behaving well. The runtime records externally visible model and tool behavior without depending on private reasoning traces.
The repository also keeps protocol, persistence, and authority boundaries explicit so the safety contracts remain meaningful when the inspectable local adapters are replaced with production infrastructure.
Evaluation asks whether behavior was correct.
Observability asks how it got there.
AI runtime observability
externally observable behavior
Structured events
MODEL_STEP_COMPLETED
duration + token usage
TOOL_EXECUTION_COMPLETED
tool + duration + result
INVESTIGATION_FINISHED
duration + calls + outcome
No private chain-of-thought logging is required to understand runtime behavior.
Run-level summary
regression signals
modelSteps
How many model turns the investigation required.
toolCalls
How much of the bounded operational tool budget was used.
modelDurationMs
Time attributable to model inference.
toolDurationMs
Time attributable to operational reads.
overheadDurationMs
Persistence, projection, validation and orchestration cost.
totalTokens
Aggregate model input and output usage.
inputTokenGrowthRatio
How much the model context grew during the investigation.
failedToolCalls
Operational tool failures during the run.
Correct answer?
Evaluation
Efficient behavior?
Observability
Behavior drifting?
Regression checks
MCP boundary
least privilege
Exposed through MCP
Operational evidence only
06 TOOLS
get_order
get_payment_state
get_fulfillment_attempts
get_delivery_state
get_notification_state
get_order_event_history
readOnlyHint = true
destructiveHint = false
idempotentHint = true
Intentionally excluded
Consequential writes do not become generic protocol tools.
RECONCILE_ORDER_STATE
RETRY_NOTIFICATION
CREATE_FULFILLMENT_ATTEMPT
ISSUE_REFUND
CANCEL_ORDER
MCP
Evidence gathering
DETERMINISTIC ACTION LAYER
Mutation authority
The internal processing trace is also deliberately excluded from the public MCP surface. Not every diagnostic capability becomes remotely discoverable.
Production substitution
same contracts, different adapters
Operational state
Current
Fixtures / repositories
Production
Authoritative commerce services
Investigation persistence
Current
File-backed store
Production
Durable database adapter
Remediation persistence
Current
In-memory / evaluation store
Production
Durable database adapter
Automation persistence
Current
In-memory / evaluation store
Production
Durable database adapter
Approval persistence
Current
In-memory / evaluation store
Production
Transactional database adapter
Execution audit
Current
In-memory / evaluation store
Production
Append-oriented durable audit table
Runbooks
Current
Local versioned knowledge
Production
Versioned knowledge source / index
MCP transport
Current
stdio smoke server
Production
Deployment-appropriate transport
Verification
Current
Separate repository reads
Production
Separate authoritative reads
The repository intentionally optimizes for inspectability. The production requirement is not to preserve these storage adapters; it is to preserve the safety and recovery contracts when those adapters change.
Deliberate scope
infrastructure volume ≠ architecture quality
This repository proves behavioral contracts rather than pretending to be a complete deployed commerce stack.
Adding infrastructure that does not strengthen the safety argument would make the project larger, not necessarily better.
Intentionally not added
Production PostgreSQL deployment
Message broker
Kubernetes
Distributed tracing infrastructure
Production approval UI
Real commerce provider
Vector database
Multiple autonomous agents
What I wanted to prove
Stochastic reasoning.Deterministic consequences.
The goal was not to make the model deterministic. It was to let the model vary how it investigates uncertainty while preventing that variance from casually becoming operational variance.
Provenance, grounding, policy, exact approval, fresh-state validation, durable execution evidence, independent verification, and fail-closed recovery keep consequential authority outside the model.
01
Reason broadly
02
Authorize narrowly
03
Verify independently