Skip to content

Model gateway

Implemented

The in-process Anthropic gateway and worker accounting path exist. Automatic fallback and additional adapters are not implemented.

Code and tests

The model gateway translates a reasoning request into a provider call and returns structured output with normalized usage. It runs inside product-worker, in internal/modelgateway. The worker calls it directly through Go interfaces; the outbound HTTP boundary is between the provider adapter and the model provider.

flowchart TB
  subgraph Worker["product-worker process"]
    Cap["Capabilities: research, synthesis, assessment"]
    Client["ModelClient: prompts, schemas, result decoding"]
    Gateway["Gateway: route selection and request validation"]
    Policy["Embedded policy.json"]
    Adapter["Anthropic adapter: HTTP and response parsing"]
    Cap --> Client --> Gateway --> Adapter
    Policy --> Gateway
  end
  Cap <-->|"Operations, admission, receipts"| DB[(Product PostgreSQL)]
  Adapter <-->|"POST /v1/messages"| Provider["Anthropic API"]
ComponentResponsibility
Capability orchestrationTask authorization, credit reservation, provider concurrency, recovery and publication
ModelClientBuild purpose-specific prompts and output schemas; decode results and check their meaning
GatewayResolve a configured route, validate request fields and call its registered adapter
Provider adapterAuthentication headers, provider payloads, response bounds, structured-result extraction and usage normalization
PostgreSQLDurable operation state, shared admission across worker replicas and retained stage results

Embedding and reranking clients have their own paths. This gateway currently handles the three reasoning purposes below. Its routing profiles are separate from embedding profiles and search configurations.

At startup, NewAnthropic loads the embedded policy.json, verifies that routed models have prices, registers the Anthropic adapter and validates that each profile has all three purposes. The checked-in policy.json is the source for the current policy version, profile, model and lane. Its purposes are research (plan), synthesis (answer/proposal) and assessment (evidence gaps). Only the Anthropic adapter is registered today.

Resolve(purpose) selects the first route for that purpose in the default profile. Route arrays do not implement fallback, load balancing or health-based selection. Changing the embedded policy requires rebuilding and deploying the worker.

A gateway request contains a purpose, system instructions, input, a named tool with its JSON schema, and an output-token limit. Provider, model, lane and policy version come from the checked-in policy. Credentials and base URL are injected at worker startup; they are not request fields.

The worker resolves the route before reserving credits so accounting knows which provider and model will be used. Gateway.Execute resolves it again from the same immutable in-memory policy.

sequenceDiagram
  participant C as Capability
  participant D as Product PostgreSQL
  participant G as ModelClient / Gateway
  participant P as Provider adapter / API
  C->>D: Look for retained stage result
  alt Completed stage exists
    D-->>C: Result and usage receipt
  else New provider call
    C->>G: Resolve purpose
    G-->>C: Provider, model, lane
    C->>D: Reserve credits and operation
    C->>D: Acquire provider/lane capacity
    C->>D: Recheck authority and mark dispatched
    C->>G: Execute structured request
    G->>P: Send provider request
    P-->>G: Structured output and usage
    G-->>C: Decoded result or error
    C->>D: Release provider capacity
    alt Valid result
      C->>D: Commit stage result and settlement
    else Outcome uncertain
      C->>D: Record uncertainty for recovery
    end
  end

Admission is shared through PostgreSQL and keyed by provider and lane. The worker waits for capacity, then rechecks permissions, task generation and recovery ownership immediately before dispatch. If that check fails, it releases admission and the reservation without sending the request.

The adapter requests a named Anthropic tool result. It caps response bodies using maximumResponseBytes in anthropic.go and rejects non-success responses, invalid JSON, missing tool output, a mismatched tool name or multiple tool results. It returns raw tool JSON plus token/cache usage, request ID and latency. ModelClient decodes that JSON into the purpose’s result type. Evidence validation and final publication checks remain in the capability layer.

pricing.go provides versioned model costs and converts normalized usage into credits and monetary cost. Startup rejects a routed model with no price entry. Reservation and settlement are worker/database operations; calling the gateway alone does not create a financial record.

Successful stage results and their settlement normally commit in one transaction. On recovery, a retained stage is loaded and its digest checked, so completed reasoning is reused. If publication is fenced after a known provider result, the worker still attempts to settle its usage.

This section owns uncertain paid-call outcomes. A crash or timeout after dispatch can leave the provider outcome unknown. Recovery quarantines that uncertainty instead of automatically issuing another paid call. There is currently no gateway retry or cross-provider fallback. Error settlement also depends on the calling stage: research can settle an invalid plan when a provider request ID is available; synthesis and assessment currently mark execution errors uncertain.

TestAIXUnknownOutcomeCannotRedispatch and TestAIXUncertainCreditsCrossMonth in services/product-worker/internal/capabilities/service_integration_test.go exercise uncertainty and retained liability. TestAIXRetainedStages verifies receipt reuse. These checks exist because retrying after a lost response can charge for the same logical work twice.

See configuration for environment wiring and worker recovery for execution ownership.