Model gateway
The in-process Anthropic gateway and worker accounting path exist. Automatic fallback and additional adapters are not implemented.
The model gateway translates a reasoning request into a provider call and returns structured output with normalized usage. It runs inside product-worker, in internal/modelgateway. The worker calls it directly through Go interfaces; the outbound HTTP boundary is between the provider adapter and the model provider.
Components and ownership
Section titled “Components and ownership”flowchart TB
subgraph Worker["product-worker process"]
Cap["Capabilities: research, synthesis, assessment"]
Client["ModelClient: prompts, schemas, result decoding"]
Gateway["Gateway: route selection and request validation"]
Policy["Embedded policy.json"]
Adapter["Anthropic adapter: HTTP and response parsing"]
Cap --> Client --> Gateway --> Adapter
Policy --> Gateway
end
Cap <-->|"Operations, admission, receipts"| DB[(Product PostgreSQL)]
Adapter <-->|"POST /v1/messages"| Provider["Anthropic API"]
| Component | Responsibility |
|---|---|
| Capability orchestration | Task authorization, credit reservation, provider concurrency, recovery and publication |
ModelClient | Build purpose-specific prompts and output schemas; decode results and check their meaning |
Gateway | Resolve a configured route, validate request fields and call its registered adapter |
| Provider adapter | Authentication headers, provider payloads, response bounds, structured-result extraction and usage normalization |
| PostgreSQL | Durable operation state, shared admission across worker replicas and retained stage results |
Embedding and reranking clients have their own paths. This gateway currently handles the three reasoning purposes below. Its routing profiles are separate from embedding profiles and search configurations.
Route selection
Section titled “Route selection”At startup, NewAnthropic loads the embedded policy.json, verifies that routed models have prices, registers the Anthropic adapter and validates that each profile has all three purposes. The checked-in policy.json is the source for the current policy version, profile, model and lane. Its purposes are research (plan), synthesis (answer/proposal) and assessment (evidence gaps). Only the Anthropic adapter is registered today.
Resolve(purpose) selects the first route for that purpose in the default profile. Route arrays do not implement fallback, load balancing or health-based selection. Changing the embedded policy requires rebuilding and deploying the worker.
A gateway request contains a purpose, system instructions, input, a named tool with its JSON schema, and an output-token limit. Provider, model, lane and policy version come from the checked-in policy. Credentials and base URL are injected at worker startup; they are not request fields.
One paid request
Section titled “One paid request”The worker resolves the route before reserving credits so accounting knows which provider and model will be used. Gateway.Execute resolves it again from the same immutable in-memory policy.
sequenceDiagram
participant C as Capability
participant D as Product PostgreSQL
participant G as ModelClient / Gateway
participant P as Provider adapter / API
C->>D: Look for retained stage result
alt Completed stage exists
D-->>C: Result and usage receipt
else New provider call
C->>G: Resolve purpose
G-->>C: Provider, model, lane
C->>D: Reserve credits and operation
C->>D: Acquire provider/lane capacity
C->>D: Recheck authority and mark dispatched
C->>G: Execute structured request
G->>P: Send provider request
P-->>G: Structured output and usage
G-->>C: Decoded result or error
C->>D: Release provider capacity
alt Valid result
C->>D: Commit stage result and settlement
else Outcome uncertain
C->>D: Record uncertainty for recovery
end
end
Admission is shared through PostgreSQL and keyed by provider and lane. The worker waits for capacity, then rechecks permissions, task generation and recovery ownership immediately before dispatch. If that check fails, it releases admission and the reservation without sending the request.
The adapter requests a named Anthropic tool result. It caps response bodies using maximumResponseBytes in anthropic.go and rejects non-success responses, invalid JSON, missing tool output, a mismatched tool name or multiple tool results. It returns raw tool JSON plus token/cache usage, request ID and latency. ModelClient decodes that JSON into the purpose’s result type. Evidence validation and final publication checks remain in the capability layer.
Accounting and recovery
Section titled “Accounting and recovery”pricing.go provides versioned model costs and converts normalized usage into credits and monetary cost. Startup rejects a routed model with no price entry. Reservation and settlement are worker/database operations; calling the gateway alone does not create a financial record.
Successful stage results and their settlement normally commit in one transaction. On recovery, a retained stage is loaded and its digest checked, so completed reasoning is reused. If publication is fenced after a known provider result, the worker still attempts to settle its usage.
This section owns uncertain paid-call outcomes. A crash or timeout after dispatch can leave the provider outcome unknown. Recovery quarantines that uncertainty instead of automatically issuing another paid call. There is currently no gateway retry or cross-provider fallback. Error settlement also depends on the calling stage: research can settle an invalid plan when a provider request ID is available; synthesis and assessment currently mark execution errors uncertain.
TestAIXUnknownOutcomeCannotRedispatch and TestAIXUncertainCreditsCrossMonth in services/product-worker/internal/capabilities/service_integration_test.go exercise uncertainty and retained liability. TestAIXRetainedStages verifies receipt reuse. These checks exist because retrying after a lost response can charge for the same logical work twice.
See configuration for environment wiring and worker recovery for execution ownership.