Get notified when your agent fails

You cannot watch a stream all day. When an agent breaks in production you want it pushed to you. axonpush has two ways to do that, and they answer different questions:

Most teams end up with both: webhooks for a raw feed into a channel nobody is required to read, alert rules for the things that should interrupt someone.

Webhook endpoints

Create one

from axonpush import AxonPush

with AxonPush() as client:
  endpoint = client.webhooks.create_endpoint(
      url="https://ops.example.com/hooks/axonpush",
      channel_id="ch_...",
      event_types=["agent.error"],
      secret="a-long-random-string-you-generated",
      description="Slack alert on agent errors",
  )

  print(endpoint.endpoint_id, endpoint.active)

Endpoints are scoped to one channel. eventTypes is an allow-list; omit it or send an empty array and every event on that channel is delivered, which is almost never what you want.

The full surface, all requiring the webhooks:manage scope:

DELETE deactivates rather than erasing, so delivery history survives.

Supply your own secret

If you omit secret, axonpush generates one and returns it once as rawSecret (whsec_ followed by 48 hex characters). It is not retrievable afterwards, later reads only tell you hasSecret: true.

There is a wrinkle worth being explicit about. When axonpush generates the secret, the value it stores and signs with is the SHA-256 hex digest of rawSecret, not rawSecret itself. Your verifier would have to hash it first. When you supply secret yourself, the stored value is exactly what you sent and there is nothing to derive.

Supply your own. It is one fewer thing to get wrong, and it means you can rotate on your own schedule.

Verify the signature

Every delivery carries four headers:

The signed payload is the raw request body, verbatim. There is no timestamp and no version prefix. Compare in constant time.

import hashlib
import hmac
import os

SECRET = os.environ["AXONPUSH_WEBHOOK_SECRET"].encode()

def verify(raw_body: bytes, signature: str) -> bool:
  expected = hmac.new(SECRET, raw_body, hashlib.sha256).hexdigest()
  return hmac.compare_digest(expected, signature)

Read the body as bytes before any JSON middleware touches it. Re-serialising parsed JSON will not reproduce the same bytes and the signature will not match.

A known-good vector, so you can test your verifier without sending traffic:

secret    whsec_test_1
body      {"hello":"world"}
signature a8760ee84868116729e475b6c56cd67b0c4f82b86755678f19a62a4d49795238

What the body actually is

This is the part most people get wrong on the first try. The POST body is the event’s payload, on its own, not a wrapper object, not the whole event record. The event’s identity travels in the headers.

So a handler looks like this:

@app.post("/hooks/axonpush")
async def hook(request):
    raw = await request.body()
    if not verify(raw, request.headers["x-axonpush-signature"]):
        return Response(status_code=401)

    event_type = request.headers["x-axonpush-event"]   # "agent.error"
    delivery = request.headers["x-axonpush-delivery"]
    payload = json.loads(raw)                          # your event's payload

    if seen(delivery):        # deliveries can repeat; make this idempotent
        return Response(status_code=200)
    ...

Delivery and retries

GET /webhooks/deliveries/{endpointId} returns the 50 most recent attempts:

for d in client.webhooks.deliveries(endpoint.endpoint_id):
    print(d.delivery_id, d.status, d.status_code, d.attempts, d.error)

status is pending, success, failed or retrying. A non-2xx response is a failure, and the first 4 KB of your response body is stored in responseBody so you can see what your own handler complained about.

Retry behaviour depends on deployment shape, and it is worth knowing which one you are on:

• Cloud. Deliveries are queued. A failure is retried by the queue and ends in a dead-letter queue if it keeps failing. X-AxonPush-Attempt reflects the redelivery count.
• Self-host and local. There is no worker process, so delivery happens inline on the ingest path and a failure is logged, not retried. Attempt is always 1.

In neither case does a broken webhook slow down or fail ingest, dispatch is isolated and never propagates an error back to the write.

Endpoint lists are cached for five seconds, so a newly created endpoint can take that long to start receiving.

Inbound webhooks

The same endpoint record also works in reverse. POST /webhooks/ingest/{endpointId} is a public route that turns an inbound HTTP call into an axonpush event, useful for getting CI results, deploy notifications or a third-party service’s callbacks onto the same timeline as your agent runs.

curl -X POST https://api.axonpush.xyz/webhooks/ingest/$ENDPOINT_ID \
  -H "Content-Type: application/json" \
  -H "x-webhook-signature: $(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -r | cut -d' ' -f1)" \
  -d "$BODY"

• Signed with the same HMAC-SHA256-hex scheme, but read from x-webhook-signature rather than X-AxonPush-Signature. Required whenever the endpoint has a secret.
• Body fields identifier, payload, agentId, traceId and eventType are lifted onto the event. If there is no payload key, the whole body becomes the payload. Defaults are webhook.inbound and custom.
• The event lands in the endpoint’s channel, in the organisation’s default environment, inbound ingest does not resolve an environment override.
• Rate limited to 30 requests per minute per endpoint.
• Returns { "status": "accepted", "eventId": "…" }.

Alert rules

An alert rule watches finished traces rather than individual events. When a trace is compacted, every enabled rule whose filters match is evaluated against that trace’s summary.

Scope: alerts:manage.

curl -X POST https://api.axonpush.xyz/v2/alerts \
  -H "Authorization: Bearer $AXONPUSH_JWT" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "prod error rate",
        "metric": "error_rate",
        "operator": "gt",
        "threshold": 5,
        "environmentId": "env_...",
        "destinationType": "email",
        "destination": "oncall@example.com"
      }'

error_rate is a percentage, errorCount / eventCount * 100, so a threshold of 5 means five percent. latency_ms is the trace’s duration. cost_usd and score are skipped when the trace has no known value for them, so a rule on either simply does not fire rather than firing on zero.

The service, model and release filters test membership in the trace’s lists, so a rule on model: claude-sonnet-4-6 matches any trace that used that model anywhere.

A webhook destination must be a webhook endpoint ID belonging to the same organisation; creation is rejected otherwise.

What an alert delivers

Alerts sent to a webhook endpoint go through the same signed delivery path, but carry a structured body rather than an event payload:

{
  "type": "alert.triggered",
  "alertRuleId": "…",
  "ruleName": "prod error rate",
  "traceId": "…",
  "metric": "error_rate",
  "value": 8.3,
  "threshold": 5
}

X-AxonPush-Event is alert.triggered. Verify the signature exactly as above.

Deduplication

An occurrence is keyed on (alertRuleId, traceId, traceRevision), and creation is conditional. A trace that is re-compacted at the same revision cannot fire the same rule twice. Occurrences are retained for 90 days and record whether delivery succeeded.

This is a per-trace guarantee, not a rate limit. A rule with a low threshold on high-volume traffic will fire per trace. Tighten the filters, not the threshold, when that happens.

Next

• Handle errors and rate limits, what the SDK does when the API is unreachable
• Alerts, the dashboard pillar for threshold alerting
• Moderation, rules over the content flowing through your agents