For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Debug your agentgateway environment.

Inspect and troubleshoot agentgateway proxies through the admin endpoints and the agctl command-line tool.

Admin endpoints

Each agentgateway pod runs an admin server on port 15000. The admin server provides the following endpoints for inspection and debugging.

To inspect the configuration that a gateway proxy has loaded and to capture per-request traces, use the agctl command-line tool. agctl resolves the proxy pod for you, opens a port-forward, and renders the admin output in formats that are easier to scan than raw JSON.

Before you begin

Install agctl.

Check the gateway, route, and policy status

Most routing and policy issues surface in the status of the corresponding Kubernetes resource. Check these first.

1. Verify that the agentgateway control plane and proxy pods are running.

   Replace with your ownQuickstart example

   kubectl get pods -n <namespace>

   kubectl get pods -n agentgateway-system

2. Verify the Gateway is Accepted and Programmed.

   Replace with your ownQuickstart example

   kubectl get gateway -A
   kubectl get gateway <name> -n <namespace> -o yaml

   kubectl get gateway agentgateway-proxy -n agentgateway-system -o yaml

3. Check the HTTPRoute for Accepted and ResolvedRefs conditions.

   kubectl get httproute -A
   Replace with your ownQuickstart example

   kubectl get httproute <name> -n <namespace> -o yaml

   kubectl get httproute openai -n agentgateway-system -o yaml

   Common issues to check for:

   • The wrong backend is selected.
   • The wrong parent Gateway is referenced.
   • Multiple HTTPRoutes conflict by having identical matchers or by having no matchers (and so default to /).

4. Check AgentgatewayBackend and AgentgatewayPolicy resources for partial acceptance. A resource can report Accepted=True with reason: PartiallyValid when the controller keeps the usable parts of a backend or policy and reports the invalid part in the condition message.

   1. Find the backends and policies that report PartiallyValid. The ACCEPTED column of kubectl get shows True for these resources, so filter on the reason instead.

      kubectl get AgentgatewayBackend -A -o json | jq -r '.items[] | .metadata as $m | .status.conditions[]? | select(.type == "Accepted" and .reason == "PartiallyValid") | "\($m.namespace)/\($m.name): \(.message)"'
      kubectl get AgentgatewayPolicy -A -o json | jq -r '.items[] | .metadata as $m | .status.ancestors[]?.conditions[]? | select(.type == "Accepted" and .reason == "PartiallyValid") | "\($m.namespace)/\($m.name): \(.message)"'

   2. Inspect the resource that you want to debug, such as the openai backend in the following example.

      kubectl get AgentgatewayBackend openai -n agentgateway-system -o yaml

   3. Find the Accepted condition in the output. For a backend, look in status.conditions. For a policy, look in status.ancestors[].conditions, which has one entry per Gateway that the policy attaches to.

      Example output:

      status:
        conditions:
        - type: Accepted
          status: "True"
          reason: PartiallyValid
          message: 'failed to translate backend: secret agentgateway-system/openai-secret not found'
          observedGeneration: 1
          lastTransitionTime: "2026-09-21T14:02:28Z"

   4. Read the reason, not only the status. A check that tests the Accepted status alone passes in this state because the status stays True. When the reason is PartiallyValid, the message names the configuration that could not be translated.

   5. Fix the cause that the message names. The rest of the resource is still translated and served, so the symptom usually shows up at request time instead of at apply time.

      Invalid part	What the proxy loads instead	How to fix it
      The Secret that holds LLM provider credentials does not exist	The backend keeps the credential field, but empty. Provider calls go out unauthenticated, so the failure comes back from the provider rather than from agentgateway.	Create the Secret in the namespace that secretRef names or correct the reference.
      A prompt guard webhook.backendRef names a Service that does not exist	The webhook stays in the policy with no target to call. Guarded requests then take the webhook failureMode, which is FailClosed unless you set it. With FailOpen, prompt guarding is skipped instead.	Create the Service or point backendRef at one that exists.
      A remote JWKS is not available yet	The authentication policy keeps an empty key set, {"keys":[]}, so the policy trusts no keys. JWT authentication defaults to Strict mode, which rejects a request that carries no token and rejects any token that does arrive because no key can verify a signature.	Check that the JWKS endpoint is reachable from the control plane. The controller retries the fetch with a backoff and installs the keys when it succeeds.
      A CEL expression does not compile	The rest of the policy is kept and the expression is replaced with one that always fails. An authorization rule that depends on that replacement therefore allows nothing.	Fix the expression. The condition message names the field, then the expression that did not compile.

   6. Check the condition again. The controller recomputes it on its own when the missing Secret, Service, or key set appears, so you do not need to reapply the resource.

Inspect the loaded configuration

Sometimes a route is Accepted but the proxy still does not behave as expected. To see what the proxy actually loaded, dump its runtime configuration.

1. Render a summary of the routes, backends, and policies that the gateway has loaded.

   Replace with your ownQuickstart example

   agctl proxy config all gateway/<gateway-name> -n <namespace> -o yaml

   agctl proxy config all gateway/agentgateway-proxy -n agentgateway-system -o yaml

2. Inspect the backends that the gateway is sending traffic to and their endpoint health.

   Replace with your ownQuickstart example

   agctl proxy config backends gateway/<gateway-name> -n <namespace>

   agctl proxy config backends gateway/agentgateway-proxy -n agentgateway-system

   Example output:

   TYPE     NAME       NAMESPACE            ENDPOINT                    HEALTH  REQUESTS  LATENCY
   Backend  openai     agentgateway-system  backend                     1.00    1         4682.37ms
   Service  ext-authz  backend-extauth      ext-authz-7c7596b5f6-tvs28  0.70    4         0.00ms
   Service  httpbin    backend-extauth      httpbin-7dc88b5fbc-zqrfn    1.00    2         3.06ms

   A backend or policy that reports PartiallyValid still appears here because the part of it that translated successfully is still served. Compare what the proxy loaded against the condition message from the previous section. For example, an authentication policy whose remote JSON Web Key Set (JWKS) was not available loads with an empty key set, {"keys":[]}.

For complete steps, see Inspect agentgateway configuration.

Trace requests

To see how a specific request flows through agentgateway, use agctl proxy trace. The trace shows you the route that was selected, the policies that were applied, the backend that was chosen, and the response status. Tracing helps you understand why a request did or did not match a route, why a policy was or was not applied, or why a request returned an unexpected status.

agctl proxy trace gateway/<gateway-name> -n <namespace> --port <listener-port> -- http://<host>/<path>

agctl proxy trace gateway/agentgateway-proxy -n agentgateway-system --port 8080 -- http://httpbin.example.com/

agctl opens a port-forward to the proxy pod, captures the trace, sends the request, and renders the result in a text-based terminal user interface (TUI). Use --raw to print JSON Lines instead.

For complete steps, see Trace requests with agctl.

Enable debug logs

Agentgateway uses the same level syntax as RUST_LOG: error, warn, info, debug, and trace. Use agctl to read and change log levels at runtime for both the proxy and the controller. agctl resolves the pod and opens a port-forward for you, so you do not need to manage kubectl port-forward yourself.

Proxy logs

1. Show the proxy’s current log filter directive.

   Replace with your ownQuickstart example

   agctl proxy log gateway/<gateway-name> -n <namespace>

   agctl proxy log gateway/agentgateway-proxy -n agentgateway-system

2. Set the global log level for the proxy.

   agctl proxy log gateway/agentgateway-proxy -n agentgateway-system --level debug

   Example output:

   current log level is typespec_client_core::http::policies::logging=warn,hickory_server::server::server_future=off,rmcp=warn,debug

3. Tail the proxy logs to see the added detail.

   Replace with your ownQuickstart example

   kubectl logs -n <namespace> deploy/<gateway-name> -f

   kubectl logs -n agentgateway-system deploy/agentgateway-proxy -f

Controller logs

The agentgateway controller tracks a log level per component, such as the translator, syncer, and gateway controller.

1. Show the controller’s current log level for each component. The -n flag defaults to agentgateway-system.

   agctl controller log -n agentgateway-system

   Example output (truncated):

   current log levels:
   ---
   agentgateway/syncer: info
   agentgateway/translator: info
   gateway-controller: info
   deployer: info
   default: info

2. Set the log level. Change all components at once with --level, or target a single component with --set component=level.

   # Set all components to debug
   agctl controller log -n agentgateway-system --level debug
   
   # Set a single component to debug
   agctl controller log -n agentgateway-system --set agentgateway/syncer=debug

3. Tail the controller logs.

   kubectl logs -n agentgateway-system deploy/agentgateway -f

Capture profiles

Agentgateway includes pprof endpoints to help you investigate CPU and memory issues. Use the agctl proxy profile commands to capture a profile. agctl resolves the proxy pod, opens the port-forward, reads the admin endpoint, and writes the profile to a file, so you do not need to manage kubectl port-forward yourself.

1. Optional: If you have not already, download Graphviz to visualize the profiles.

2. Capture a CPU profile. The default duration is 30 seconds. Send traffic through the gateway while the profile runs. Otherwise, the profile contains no samples.

   Replace with your ownQuickstart example

   agctl proxy profile cpu gateway/<gateway-name> -n <namespace> --seconds 30 -o ./cpu.pprof

   agctl proxy profile cpu gateway/agentgateway-proxy -n agentgateway-system --seconds 30 -o ./cpu.pprof

   Example output:

   Wrote cpu profile to ./cpu.pprof

3. Capture a heap profile.

   Replace with your ownQuickstart example

   agctl proxy profile heap gateway/<gateway-name> -n <namespace> -o ./heap.pprof

   agctl proxy profile heap gateway/agentgateway-proxy -n agentgateway-system -o ./heap.pprof

   Example output:

   Wrote heap profile to ./heap.pprof

   If you omit -o, agctl writes the profile to agentgateway-cpu-<timestamp>.pb.gz or agentgateway-heap-<timestamp>.pb.gz in the current directory.

4. Inspect the profiles with go tool pprof.

   CPU profile

   go tool pprof -http=: cpu.pprof

   Heap profile

   go tool pprof -http=: heap.pprof

   Graphviz opens on your web browser to a UI on localhost. Example: