Trace a multi-step agent run end to end

An agent runs twelve steps across three tool calls and something goes wrong at step eight. Logs alone will not tell you which lines belong to the same run. A trace ID will.

The trace context

Both SDKs carry an ambient trace context. Anything published while it is active inherits its trace ID, so correlation is the default rather than something you remember to do.

from axonpush import AxonPush, EventType, get_or_create_trace

with AxonPush() as client:
  trace = get_or_create_trace()

  client.events.publish(
      "web_search",
      {"query": "AI frameworks"},
      channel_id="ch_...",
      agent_id="researcher",
      event_type=EventType.AGENT_TOOL_CALL_START,
  )

  client.events.publish(
      "summarize",
      {"input_tokens": 1200},
      channel_id="ch_...",
      agent_id="researcher",
      event_type=EventType.AGENT_TOOL_CALL_START,
  )

  detail = client.traces_v2.detail(trace.trace_id)
  summary = detail.summary
  print(summary.event_count, summary.duration_ms, summary.error_count)

Neither publish call names a trace. Both land in the same one, because get_or_create_trace() installed it and the SDK reads it on the way out. It also generates a fresh span ID per event, so ordering within the trace is preserved without your tracking a counter.

What a trace ID actually is

A UUID4 string. Not a prefixed identifier, not a sequence, just a UUID. Span IDs are 16-character lowercase hex, which is the W3C span format.

That matters because a trace ID has to survive the trip into OpenTelemetry, where a trace ID is 32 hex characters. TraceContext handles the conversion:

trace = get_or_create_trace()

trace.trace_id        # '9f2c4e1b-7a3d-4f5e-8c0b-1a2d3e4f5a6b'
trace.w3c_trace_id()  # '9f2c4e1b7a3d4f5e8c0b1a2d3e4f5a6b'
trace.next_span_id()  # 'a3f19c02b7d84e51'
trace.traceparent()   # '00-9f2c4e1b7a3d4f5e8c0b1a2d3e4f5a6b-a3f19c02b7d84e51-01'

A UUID that is already 32 valid hex characters is used as-is; anything else is hashed to 32 hex characters, deterministically. So the same axonpush trace always maps to the same W3C trace, and back.

Propagating across services

The trace context lives in a ContextVar in Python and in AsyncLocalStorage in Node, so it follows function calls and spawned tasks inside one process for free. Crossing a process boundary is where you have to do something.

Pass the ID explicitly:

# Service A - start the run and put the ID on the outbound request.
trace = get_or_create_trace()
requests.post(downstream, headers={"traceparent": trace.traceparent()}, json=body)

# Service B - adopt it.
incoming = request.headers["traceparent"].split("-")[1]
trace = get_or_create_trace(incoming)

get_or_create_trace(trace_id) installs the given ID as the current context, overwriting whatever was there. Called with no argument it returns the existing context or makes one. That asymmetry is the whole propagation mechanism.

TypeScript exposes the same pair as getOrCreateTrace and currentTrace from @axonpush/sdk, or as client.getOrCreateTrace() on the client:

import { getOrCreateTrace } from "@axonpush/sdk";

app.post("/run", async (req, res) => {
  getOrCreateTrace(req.header("traceparent")?.split("-")[1]);
  await handleRun(req.body);
});

The canonical headers for this are traceparent and baggage (W3C), with X-Axonpush-Trace-Id and X-Axonpush-Span-Id as the axonpush-native pair.

The read model

/traces is the read surface the dashboard uses. It models spans as spans rather than as events, which is what you want for a waterfall.

On the client this surface is client.traces_v2 in Python and client.tracesV2 in TypeScript. Every method needs the traces:read scope.

detail = client.traces_v2.detail("9f2c4e1b-…")
summary = detail.summary

print(summary.event_count, summary.duration_ms, summary.error_count)

for e in client.traces_v2.events("9f2c4e1b-…").data:
    print(f"  [{e.span_id}] {e.identifier} ({e.event_type})")

The summary returns trace_id, event_count, agents, start_time, end_time, duration_ms, error_count, tool_call_count, handoff_count, total_tokens, status, and cost_usd when the pricing for every model involved is known. Cost is omitted rather than guessed.

list() takes the filter query as a mapping (environment, limit, cursor, plus the facet filters); stats() takes app_id and environment and returns the dashboard rollups.

facets is the one worth knowing about: rather than guessing which model, service or release to filter by, ask what actually appears in the window you care about.

Read a trace back in the Observe pillar, or over the Traces API.

Async

from axonpush import AsyncAxonPush, EventType, get_or_create_trace

async with AsyncAxonPush() as client:
    trace = get_or_create_trace()
    await client.events.publish(
        "web_search",
        {"query": "AI agents"},
        channel_id="ch_...",
        agent_id="researcher",
        event_type=EventType.AGENT_TOOL_CALL_START,
    )

The context propagates to tasks spawned from the current task. It does not propagate to a task created before the context was installed, or to one running on a different thread, install it again there with get_or_create_trace(known_id).

Next

• Add observability to your agent framework, every integration is trace-aware already
• OpenTelemetry over OTLP, joining axonpush traces to spans from your HTTP and database layers
• Observe, reading the trace back as a waterfall