For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Route agentgateway LLM traffic to foundation models on Amazon Bedrock.

Configure Amazon Bedrock as an LLM provider in agentgateway.

Authentication

Before you can use Bedrock as an LLM provider, you must authenticate by using the standard AWS authentication sources. Agentgateway will automatically detect the local ambient credentials, but these can be explicitly configured with auth.aws.

Configuration

# yaml-language-server: $schema=https://agentgateway.dev/schema/config

llm:
  models:
  - name: "*"
    provider: bedrock
    params:
      awsRegion: us-west-2

Passthrough

If your applications directly use the AWS Converse or Invoke APIs, Agentgateway cannot translate these APIs to other providers. However, it can pass the request through to Bedrock itself following the passthrough approach.

This can provide telemetry data for these requests.

First, setup passthrough mode:

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
llm:
  models:
  - name: us.anthropic*
    provider: bedrock
    params:
      awsRegion: us-west-2
    passthrough: detect

Then, you can send native Converse and Invoke requests:

import json

import boto3

client = boto3.client(
    'bedrock-runtime',
    region_name='us-west-2',
    endpoint_url='http://localhost:4000',
)
response = client.converse(
    modelId='us.anthropic.claude-sonnet-4-6',
    messages=[
        {
            'role': 'user',
            'content': [{'text': 'give 1 word answer'}]
        }
    ]
)
print('converse response:')
print(response)

import json

import boto3

client = boto3.client(
    'bedrock-runtime',
    region_name='us-west-2',
    endpoint_url='http://localhost:4000',
)
response = client.invoke_model(
    modelId='us.anthropic.claude-sonnet-4-6',
    body=json.dumps({
        'anthropic_version': 'bedrock-2023-05-31',
        'max_tokens': 10,
        'messages': [
            {
                'role': 'user',
                'content': [{'type': 'text', 'text': 'give 1 word answer'}],
            }
        ],
    }),
)
body = json.loads(response['body'].read())

print('invoke response:')
print(body)

Claude Platform on AWS

See here for connect to Claude Platform on AWS.

Bedrock Mantle

Bedrock serves models on two API surfaces: the Runtime endpoint, which carries the Converse and Invoke APIs, and the Mantle endpoint, which carries the native OpenAI and Anthropic APIs. Some models are served on only one of the two.

For chat requests, the endpoint is chosen per model from the runtime and mantle tags in your model cost catalog. Run agctl catalog import to populate those tags, because the default sources include aws-bedrock-mantle, which reads them from the AWS model cards. Without a catalog, no model carries either tag, so every chat request falls back to the preference alone.

Set params.bedrockEndpointPreference to choose how the tags are applied.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config

llm:
  models:
  - name: "*"
    provider: bedrock
    params:
      awsRegion: us-west-2
      bedrockEndpointPreference: runtimePreferred

The Kubernetes API takes the same four values capitalized, such as RuntimePreferred, under spec.ai.provider.bedrock.endpointPreference. A value that you copy from one mode to the other fails to load.

The preference applies to four route types: chat completions, messages, responses, and Anthropic token counting. Every other route type ignores the preference and uses a fixed endpoint.

• Model listing always uses Mantle.
• Embeddings, realtime, Gemini token counting, detection, passthrough, and content generation always use Runtime.
• Reranking uses a separate Bedrock agent-runtime host rather than the Runtime or Mantle endpoint.

Whether the preference changes the request format that a model accepts depends on the endpoint that it selects. A model that resolves to Runtime accepts the Bedrock Converse format only, and its chat format tags do not apply. A model that resolves to Mantle accepts the formats in its tags, except for anthropic.claude* models, which always take the Anthropic Messages format. For more information, see Chat format tags.

Token counting

Bedrock supports token counting for Anthropic models via the count_tokens endpoint. Agentgateway automatically handles the required formatting for Bedrock’s count-tokens endpoint.

curl -X POST http://localhost:4000/v1/messages/count_tokens \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
    "messages": [{"role": "user", "content": "Hello!"}],
    "system": "You are a helpful assistant."
  }'

Example response:

{
  "input_tokens": 15
}

Extended thinking and reasoning

Extended thinking and reasoning lets models reason through complex problems before generating a response. To opt in, add the OpenAI reasoning_effort field to your request. The value is added to the additionalModelRequestFields of the Bedrock request, in a form that depends on the model family. The family is chosen by matching the model ID.

For Claude models that support adaptive thinking, the request is sent with thinking.type set to adaptive and the effort level in output_config.effort, instead of a token budget. The value minimal is sent as low. Which models take this form depends on the adaptive_thinking tag in the model cost catalog. The built-in catalog sets this tag for these models.

Other Claude models, such as us.anthropic.claude-opus-4-20250514-v1:0, receive a thinking budget.

Note that max_tokens must be greater than the thinking budget, and the minimum thinking budget is 1,024 tokens.

curl "localhost:4000/v1/chat/completions" -H content-type:application/json -d '{
  "model": "",
  "max_tokens": 6000,
  "reasoning_effort": "high",
  "messages": [
    {
      "role": "user",
      "content": "Explain the trade-offs between consistency and availability in distributed systems."
    }
  ]
}' | jq

Structured outputs

Structured outputs constrain the model to respond with a specific JSON schema. Provide the schema definition in the OpenAI response_format field of your request. Agentgateway translates this to Bedrock’s native format automatically.

curl "localhost:4000/v1/chat/completions" -H content-type:application/json -d '{
  "model": "",
  "max_tokens": 256,
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "answer_schema",
      "schema": {
        "type": "object",
        "properties": {
          "answer": { "type": "string" },
          "confidence": { "type": "number" }
        },
        "required": ["answer", "confidence"],
        "additionalProperties": false
      }
    }
  },
  "messages": [
    {
      "role": "user",
      "content": "Is the sky blue? Respond with your answer and a confidence score between 0 and 1."
    }
  ]
}' | jq