LLM observability for AI apps on Lambda

What you see

• Each model call: provider, model, settings, the messages sent, the response, input and output tokens, and time to first token for streamed calls.
• Agent steps and tools: the steps of a multi-step generation, each tool call with its arguments and result, and the AWS SDK or HTTP calls the tool made, nested under it.
• The provider's HTTP request (POST api.openai.com, …) under the model call that made it.
• Totals: tokens in and out for the whole trace in its header, and an "LLM usage" table per model.
• In the service graph: one node per provider, so you see where an invocation spent its time.

The same data is available to agents: get_trace_summary prints lines like chat gpt-4o-mini 812→24 tok tool:getOrder.

Setup

Attach the Ivy layer as usual (auto-instrumentation). Then turn on the AI SDK's telemetry:

AI SDK 3 to 6: pass experimental_telemetry on each call.

import { generateText } from "ai";

const result = await generateText({
  model,
  prompt,
  experimental_telemetry: { isEnabled: true },
});

AI SDK 7: register the OpenTelemetry integration once, at the top of the handler module.

import { registerTelemetry } from "ai";
import { OpenTelemetry } from "@ai-sdk/otel";

registerTelemetry(new OpenTelemetry());

You don't need an OpenTelemetry SDK. The layer registers a minimal tracer provider that libraries using @opentelemetry/api report to. If your app registers its own provider (for example with the AWS Distro for OpenTelemetry), Ivy steps aside. IVY_CAPTURE_OTEL=false turns this off.

Supported conventions

Attributes are stored as the library sent them and interpreted when read, so new instrumentations only need a change in how Ivy displays them.

Privacy and limits

• Prompts and responses follow the same rules as other payloads: capped per attribute (IVY_MAX_PAYLOAD_BYTES), and string values under keys that match your redaction list are replaced. Token counts are kept.
• Long conversations can exceed the cap; raise IVY_MAX_PAYLOAD_BYTES if you need the full text.
• Ivy shows tokens, not cost estimates: model prices change too often to hard-code.
• Response streaming Lambda handlers (awslambda.streamifyResponse) are not traced yet. Streaming from the model inside a normal handler is.