Operate and troubleshoot the sidecar

Start the AAC journey · All references

This reference is for a deployment or integration you have chosen to configure. For the first authenticated reservation, use the packaged Compose journey above.

Use the actual paths printed by your setup and run. The executable offline audit recipes below refer to the optional protocol fixture and its AAC_DEMO_DIR; the reservation demo instead prints its retained agent state paths and saved mint response. Do not guess a tenant's local file path.

Diagnostics and support

Work in this order: exact installed version → configuration/file permissions → /healthz and /readyz → trust publication and certificate validity → workload projection → pairing → workflow outcome → local and central evidence. A 401 at an agent callback means pairing failed before business execution. A trust or recipient rejection should leave the agent untouched. A healthy process with missing trust can remain unable to accept an authenticated workflow.

Contact support@cascadeauth.com with the version, image/bundle digest, platform, timestamp/time zone, request/root identifier if appropriate, stable error code and sanitized reproduction. Beta support is best-effort without a 24/7 SLA. For a suspected compromise, stop affected admission, preserve local evidence and contact the same support address; your operator owns credential revocation and recovery. License questions go to legal@cascadeauth.com.

Optional central trace forwarding

The sidecar always follows its configured local telemetry sink. To also send eligible chain metadata to AAC, add this block using the supplied data-plane URL and a private tenant API-key file:

sidecar:
  telemetry:
    sink: "/var/lib/aac/telemetry.jsonl"
    central_forward:
      control_plane_url: "https://api.stage.cascadeauth.dev"
      api_key_file: "/etc/aac/keys/tenant-api-key"

The parent directory must exist. The sidecar exchanges that API key for a short-lived telemetry-ingest session and sends only the allowed central metadata. Business payloads, task references, human claims, raw AAC/DPoP credentials and terminal attestations remain local. A2A protocol diagnostics without chain identifiers also remain local. Local sink: "none" may be used with central forwarding when local retention is intentionally disabled.

Forwarding uses a bounded background queue and never waits on the authorization path. Outages, queue saturation or shutdown can lose central events; retain local audit data according to your needs. Coarse local warnings identify exchange/forward failures without printing credentials or response bodies. The API-key file is reread on each token exchange, so an atomic replacement takes effect without a restart; already-issued sessions retain their own expiry.

After a real workflow, use aac chain --help and the returned root token identifier to inspect its central trace. Allow for asynchronous delivery; successful local events alone do not prove central delivery.

Audit your workflows

Configure a tenant-local JSONL sink and retain it alongside your application's business audit. Each line is one JSON object; fields not applicable to an event are omitted. Start with root_token_id from the client, then follow token IDs, hop indices, workload identities, decisions and failures across your sidecars. Wall-clock arrival order across hosts is not a causal guarantee.

Native events use result: success or failure; A2A diagnostics may use accepted, rejected or failed. Event availability depends on how far the request progressed. A request rejected before authority can be decoded may have no root/token ID. A bad pairing signature at the application is an application HTTP 401, not proof of a chain-verification event at the sidecar.

Local event fields

Not every supported field is populated by every current path. Treat optional absence as unknown, not a successful verification or a zero measurement. Ordinary access/application logs are separate from this JSONL schema.

These abbreviated, synthetic examples show the shape, not complete captured events. Ellipses indicate omitted values and must not be fed into a verifier:

{"timestamp_unix_seconds":1700000000,"event_type":"mint","result":"success","root_token_id":"...","agent_decision_action":"forward"}
{"timestamp_unix_seconds":1700000001,"event_type":"receive","result":"failure","failure_code":"ERR_DPOP_REPLAY"}
{"timestamp_unix_seconds":1700000002,"event_type":"receive","result":"failure","failure_code":"ERR_RECIPIENT_NOT_AUTHORIZED"}
{"timestamp_unix_seconds":1700000003,"event_type":"respond","result":"success","root_token_id":"...","agent_decision_action":"settle","terminal_attestation":"..."}

Investigate three common failures

Keep access logs or request correlation when a pre-verification failure has no root ID. Do not invent a root association from unverified caller headers. After a successful two-agent flow, correlate sender dispatch and receiver receive/respond, then verify the terminal JWS below. An HTTP 200 alone is insufficient evidence of the intended recipient or settlement.

Retain and inspect safely

The sidecar writes file sinks with mode 0600 and fails startup for an unusable path. Restrict access, encrypt backups, define a retention period, and forward to an append-only/tamper-evident store if your audit requirements need it. JSONL itself is not a signed or tamper-proof ledger. Preserve clock/source metadata and test your collector's loss detection. For file rotation, coordinate stop/reopen/restart; renaming an open file does not make the process reopen a new path. Do not rely on log retention as a substitute for retained A2A state.

Central forwarding sends a smaller metadata envelope asynchronously. Payloads, task references, human claims, raw authority/DPoP and terminal attestations stay local. Central outages can lose events without stopping authorization. Use:

aac chain show --help
aac chain show --profile "$AAC_PROFILE" --token-id '<root_token_id>' --output json

Only participating tenants can inspect a trace. A missing central trace can mean delayed/lost forwarding, an unknown ID, or no access; investigate local evidence and coarse forwarding warnings before changing credentials.

Verify terminal evidence offline

Save the local terminal_attestation string from a respond event into terminal.jws. Verify it using your already trusted CA certificates for that workload's SPIFFE domain and the expected tenant, workload, root and task from your own registration/request records. Never derive those expectations or trust anchors from the unverified JWS itself.

Install python -m pip install --upgrade cryptography in an isolated verifier environment. Save the following as verify_terminal.py. It implements the supported direct-CA, single-leaf Ed25519/P-256 profile, a 64-KiB input bound and a fixed 30-second certificate clock tolerance. It checks the signature and identity/correlation bindings; it does not reconstruct the entire AAC chain or prove the business action happened. Pair it with the workflow and application audit records.

import argparse
import base64
import hashlib
import json
import re
import time
from pathlib import Path
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import ec, ed25519
from cryptography.hazmat.primitives.asymmetric.utils import encode_dss_signature
from cryptography.x509.oid import SignatureAlgorithmOID

ORDER = int("FFFFFFFF00000000FFFFFFFFFFFFFFFFBCE6FAADA7179E84F3B9CAC2FC632551", 16)


def require(ok, message):
    if not ok:
        raise ValueError(message)


def unb64(value):
    require(bool(re.fullmatch(r"[A-Za-z0-9_-]+", value)), "Invalid base64url")
    return base64.urlsafe_b64decode(value + "=" * (-len(value) % 4))


def verify(compact, ca_pem, tenant, spiffe, root, task, at):
    require(len(compact) <= 65536, "Attestation too large")
    parts = compact.split(".")
    require(len(parts) == 3, "Expected compact JWS")
    header, claims = [json.loads(unb64(p)) for p in parts[:2]]
    require(isinstance(header, dict) and isinstance(claims, dict), "Expected JSON objects")
    require(
        header.get("typ") == "aac-terminal-attestation+jwt" and "crit" not in header,
        "Unsupported protected header",
    )
    require(header.get("alg") in ("EdDSA", "ES256"), "Unsupported algorithm")
    chain = header.get("x5c")
    require(isinstance(chain, list) and len(chain) == 1, "Expected one x5c leaf")
    leaf = x509.load_der_x509_certificate(base64.b64decode(chain[0], validate=True))
    public = leaf.public_key()
    spki = public.public_bytes(
        serialization.Encoding.DER, serialization.PublicFormat.SubjectPublicKeyInfo
    )
    kid = base64.urlsafe_b64encode(hashlib.sha256(spki).digest()).rstrip(b"=").decode()
    require(header.get("kid") == kid, "Leaf-key fingerprint mismatch")
    uris = leaf.extensions.get_extension_for_class(
        x509.SubjectAlternativeName
    ).value.get_values_for_type(x509.UniformResourceIdentifier)
    require(
        [u for u in uris if u.startswith("spiffe://")] == [spiffe], "Unexpected workload identity"
    )

    def valid(cert):
        return (
            cert.not_valid_before_utc.timestamp() - 30
            <= at
            <= cert.not_valid_after_utc.timestamp() + 30
        )

    require(valid(leaf), "Leaf outside certificate validity")
    trusted = False
    for ca in x509.load_pem_x509_certificates(ca_pem):
        if not valid(ca):
            continue
        key = ca.public_key()
        try:
            if (
                isinstance(key, ed25519.Ed25519PublicKey)
                and leaf.signature_algorithm_oid == SignatureAlgorithmOID.ED25519
            ):
                key.verify(leaf.signature, leaf.tbs_certificate_bytes)
            elif (
                isinstance(key, ec.EllipticCurvePublicKey)
                and isinstance(key.curve, ec.SECP256R1)
                and leaf.signature_algorithm_oid == SignatureAlgorithmOID.ECDSA_WITH_SHA256
            ):
                key.verify(leaf.signature, leaf.tbs_certificate_bytes, ec.ECDSA(hashes.SHA256()))
            else:
                continue
            trusted = True
            break
        except Exception:
            continue
    require(trusted, "Leaf not signed by a currently valid trusted CA")
    signature = unb64(parts[2])
    data = ".".join(parts[:2]).encode("ascii")
    require(len(signature) == 64, "Expected 64-byte signature")
    if header["alg"] == "EdDSA":
        require(isinstance(public, ed25519.Ed25519PublicKey), "Algorithm/key mismatch")
        public.verify(signature, data)
    else:
        require(
            isinstance(public, ec.EllipticCurvePublicKey)
            and isinstance(public.curve, ec.SECP256R1),
            "Algorithm/key mismatch",
        )
        r, s = int.from_bytes(signature[:32], "big"), int.from_bytes(signature[32:], "big")
        require(0 < r < ORDER and 0 < s <= ORDER // 2, "Invalid or noncanonical ES256 signature")
        public.verify(encode_dss_signature(r, s), data, ec.ECDSA(hashes.SHA256()))
    expected = {
        "iss": tenant,
        "terminal_agent_svid": spiffe,
        "root_token_id": root,
        "task_ref": task,
    }
    require(isinstance(root, str) and bool(root), "Expected root must be nonempty")
    for name, value in expected.items():
        require(claims.get(name) == value, "Claim mismatch: " + name)
    require(
        bool(re.fullmatch(r"tnt-[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}", tenant)),
        "Invalid expected tenant",
    )
    require(
        type(claims.get("iat")) is int and -(2**63) <= claims["iat"] < 2**63,
        "Invalid issued-at time",
    )
    require(
        isinstance(claims.get("settlement_id"), str) and bool(claims["settlement_id"]),
        "Missing settlement ID",
    )
    require(
        "action_summary" not in claims or isinstance(claims["action_summary"], str),
        "Invalid action summary",
    )
    return {"verified": True, **expected, "settlement_id": claims["settlement_id"]}


def read_compact(path):
    with Path(path).open("rb") as stream:
        raw = stream.read(65537)
    require(len(raw) <= 65536, "Attestation file exceeds 64 KiB")
    return raw.decode("ascii").strip()


if __name__ == "__main__":
    p = argparse.ArgumentParser()
    for flag in ("jws", "ca", "tenant", "spiffe", "root", "task"):
        p.add_argument("--" + flag, required=True)
    p.add_argument("--at", type=int, default=None)
    a = p.parse_args()
    result = verify(
        read_compact(a.jws),
        Path(a.ca).read_bytes(),
        a.tenant,
        a.spiffe,
        a.root,
        a.task,
        int(time.time()) if a.at is None else a.at,
    )
    print(json.dumps(result))

python verify_terminal.py --jws terminal.jws --ca "$AAC_DEMO_DIR/pki/dev-ca.crt" \
  --tenant "$AAC_TENANT_ID" --spiffe "spiffe://${AAC_TRUST_DOMAIN}/demo/agent" \
  --root '<root_token_id returned by your client>' --task '<task_ref from your request>'

For the normal current-time check, omit --at. For historical analysis, use an independently trusted observation time and archived trusted CA material; never use an unverified iat to choose the certificate time. Historical verification does not establish that a key is trusted or a credential active now. The attestation has no expiry claim; iat is signed metadata, not a replay defense. Check it against your independently recorded workflow timeline. For a result from the two-agent example's peer, use its registered spiffe://<trust-domain>/demo/peer identity instead of /demo/agent.

Sidecar error reference

Application errors use this envelope; detail may be null and diagnostic wording may change. Use code for automation and request_id for support. The HTTP status and code together identify the response; the same code can appear at different operation boundaries.

{"error":{"code":"ERR_RECIPIENT_NOT_AUTHORIZED","message":"Recipient verification failed","detail":null,"request_id":"req-aac-sdk-example"}}

Authenticated A2A protocol errors can instead use JSON-RPC errors under HTTP 200: -32700 parse error, -32600 invalid request, -32601 unknown method, -32602 invalid parameters, -32004 unsupported operation and -32009 unsupported version. Inspect the JSON body even when HTTP transport succeeded. Admission/authentication/body-size errors may use non-200 HTTP responses.

CLI/control-plane errors are a separate surface. For example, ERR_API_KEY_LAST_ACTIVE belongs to API-key retirement, not sidecar receiving; its documented recovery is to issue/migrate a second active key first.

Upgrade, rollback, and uninstall

• Verify a new immutable version before placing it in a new version directory.
• Stop admission, point the service symlink at the new version, restart, and require health/readiness before restoring traffic.
• Roll back by restoring the previous verified symlink; never replace an existing version directory in place.
• Preserve bbolt, Shared durable replay, trust, and configuration state across binary/image replacement according to the tenant retention policy. Basic replay history is lost on every restart; choose Shared durable if that is unacceptable.
• Remove the image or versioned binary only after stopping the service and completing the chosen credential-retention or retirement plan below. Public artifacts already downloaded by others cannot be recalled.

Developer-binary uninstall:

rm "${HOME}/.local/bin/aac-sidecar"
rm -rf "${HOME}/.local/lib/aac-sidecar/releases/v0.4.4"

Do not use those commands for an operator-owned production directory or state volume. Follow the tenant's change, retention, and secure-deletion procedures.

Rotate or revoke credentials deliberately

A binary upgrade does not rotate tenant credentials. Before revocation, identify all clients of the exact key and distinguish planned rotation from compromise:

aac tenant api-key list --profile "$AAC_PROFILE" --output table
aac tenant api-key issue --help
aac tenant api-key retire --help
aac tenant reissue-api-key --help
aac trust-anchor list --profile "$AAC_PROFILE" --output table
aac trust-anchor revoke --help

For planned API-key rotation, issue the second active key, migrate all clients, verify them, then retire the exact old key. The control plane refuses to retire the last ACTIVE API key (ERR_API_KEY_LAST_ACTIVE); issue and migrate a second key first. For a lost/compromised key, follow the recovery command's separate semantics. The telemetry API-key file is reread on token exchange; existing sessions keep their own expiry.

Root-key rotation uses a new key ID, publishes its public key, waits for trust visibility, moves the workload to it, verifies a real workflow, then revokes the old key. Revoking the last root-signing key is permitted for compromise or teardown; the API-key last-ACTIVE protection does not apply to root keys. Revoked roots leave the public trust set, so verifiers lose trust in chains rooted at that key when their trust view refreshes. Healthy control-plane-backed caches poll every five minutes by default; outage stale-serve behavior remains, and there is no instantaneous global revocation guarantee. Stop affected workflows and coordinate trust refresh/replacement with peers. Do not assume removing a public root erases a workload's locally held signing key.

Replace workload/terminal keys with matching valid certificates and update trust as required. Replace a compromised pairing secret in both paired processes and restart them together. For remote signers, disable/revoke the exact key version and verify failure without provider/file fallback.

An unsafe beta is removed from recommendations and replaced by a newly reviewed immutable version; its old digest is never overwritten. Pause affected traffic and preserve evidence while deciding whether rollback is safe.

Local uninstall removes software, not a tenant account. For reusable testing, stop the processes and retain tenant registration, credentials in your secret manager, and necessary private state. Remove local credential copies only after verifying custody. Permanent workload/credential retirement is a separate operator action; this guide does not promise a tenant-account deletion API or a complete tenant-offboarding procedure.