Endpoint traces (OpenTelemetry)

Every endpoint call is traced. The gateway opens a span for the endpoint, the component that actually ran opens a child span, and the OTel instrumentation adds a child span for each downstream call the component makes (HTTP via httpx/requests, Redis, Milvus, PostgreSQL, gRPC). Those spans are exported to the Peach collector, which feeds the request-rate/latency/error metrics behind your dashboard and stores the traces themselves.

The dashboard shows traces per endpoint (downstream breakdown, flame graph, slowest recent traces). What it cannot do is give you the trace for one specific call you just made. For that, ask for the spans in the response.

Getting the spans back in the response

Send your codops secret in the X-Peach-Debug-Secret header. The response then carries a debug_spans block with every span the request produced:

curl -s -H "X-Peach-Debug-Secret: $CODOPS_SECRET" \
  "https://peach.ebu.io/api/v1/zzebu/my_recommendations?u=00000000-0000-0000-0000-000000000000&size=5" \
  | jq .debug_spans

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "spans": [
    {
      "name": "my_recommendations/main",
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
      "span_id": "00f067aa0ba902b7",
      "parent_id": "a3ce929d0e0e4736",
      "kind": "SpanKind.INTERNAL",
      "start_ms": 1757240000123.4,
      "duration_ms": 41.2,
      "attributes": {
        "peach.codops": "zzebu",
        "peach.endpoint": "my_recommendations",
        "peach.component": "main"
      }
    }
  ]
}

Fields per span:

Field	Meaning
name	Span name. <endpoint> for the gateway span, <endpoint>/<component> for the component, library-specific for downstream calls
span_id / parent_id	Rebuild the tree with these; the root span has parent_id: null
duration_ms	Wall time of that span, so you can see which downstream call cost the latency
attributes	Span attributes, including peach.component.chosen and peach.component.selection (A/B or override), and peach.fallback.used / peach.fallback.reason when a fallback ran

Spans are ordered by when they finished, so the deepest downstream call usually comes first and the gateway span last.

The secret is your existing codops auth secret. Contact the Peach Core team if you do not have it.

Note

Use the header, not a query parameter. A secret in the query string ends up in access logs and browser history. The header is stripped by the gateway and never reaches your component code or anything downstream.

Using the trace_id

trace_id is the same ID the trace was stored under, so you can hand it to the Peach Core team to look the call up on the storage side, and it is what ties the gateway, component and downstream spans together.

Traces are also joined to logs through a peach.request_id attribute on the root span, which is the Ray Serve request ID that appears in your endpoint logs.

Note

Storage is sampled per endpoint, sized from the endpoint's measured request rate: quiet endpoints keep every trace, a busy one keeps a percentage. So a trace_id from a hot endpoint may not be in trace storage. The inline debug_spans block is unaffected by sampling, it is captured in the process before the sampling decision is applied, so it always returns the full span list.

Limitations

• Streaming endpoints return nothing. A call with stream=true returns a streaming response, which leaves no place to attach debug_spans. Call the same endpoint non-streaming to get its trace.
• The header also switches on the regular debug output. You will get the normal debug block in the response alongside debug_spans.
• Personal data is stripped from the spans. Query strings, database statements and Milvus filter expressions are removed before the spans are returned, the same scrubbing that is applied before traces are stored. Do not expect to see the values you passed in.
• An empty spans list means tracing is not active on that deployment (for example a codops that has opted out). Nothing else about the endpoint is affected.

Warning

debug_spans is a debugging aid for a hand-made call, not something to enable in a client. It only works with the codops secret, so it must never be wired into a browser or mobile app.