# Ivy: full documentation The text of every page on https://ivytrace.dev, in markdown. The short index is https://ivytrace.dev/llms.txt. # Simple Lambda observability, for the AI age Ivy is open-source (MIT) tracing and error monitoring for AWS Lambda. A Lambda layer records every Node.js invocation with its payloads, AWS SDK and HTTP calls, and the failures that happen outside your handler. You read the traces in the dashboard; your coding agent reads them through an MCP server and an HTTP API. It is free for 100,000 traces a month per workspace, then €5 per million, and free to self-host. ## Get started Paste this prompt into your coding agent (Claude Code, Cursor, Codex or any agent that can fetch a URL and use a terminal). It signs you up, connects your AWS account and adds tracing; you approve twice in your browser. ``` Set up Ivy for my AWS Lambda functions using https://ivytrace.dev/llms.txt ``` ## Key facts - Setup takes one prompt to your coding agent, with no code changes. - No measurable delay on warm invocations; about 20–45 ms on cold starts, because spans are sent after your function has responded. - Free tier: 100,000 traces a month per workspace. After that, €5 per million traces. One trace is one Lambda invocation. - Self-hosting is free and MIT licensed: `docker compose up` runs everything. - Runtime: Node.js on AWS Lambda (with AWS SDK v3, `fetch` and `node:http`). Other runtimes can send OpenTelemetry (OTLP) directly. ## How it works 1. **Paste the prompt.** Your agent reads [llms.txt](https://ivytrace.dev/llms.txt), which tells it how to sign you up and which API calls to make. 2. **Approve in your browser.** You approve the sign-in, then create a CloudFormation stack that gives Ivy a role in your AWS account. No access keys change hands. 3. **See traces.** You pick which functions to trace, by CloudFormation stack, name pattern or tag. New matching functions are traced automatically. The agent invokes one and checks that its trace arrived. ## What you get - [Lambda tracing](https://ivytrace.dev/features/lambda-tracing): every invocation with its event, response, errors, cold start and logs, and every AWS SDK and HTTP call with payloads. - [Auto-instrumentation](https://ivytrace.dev/features/auto-instrumentation): one Lambda layer, no code changes, attached by Ivy to the functions you choose. - [Lambda errors](https://ivytrace.dev/features/lambda-errors): init errors, out-of-memory kills, crashes, timeouts and throttles, grouped by fingerprint. - [Alerts](https://ivytrace.dev/features/alerts): a Slack message when an error group is first seen, and a weekly email digest. - [For AI agents](https://ivytrace.dev/features/ai-agents): a remote MCP server across all your workspaces, an HTTP API with OpenAPI, `llms.txt` onboarding and an agent skill. - [LLM observability](https://ivytrace.dev/features/llm-observability): Vercel AI SDK and OpenTelemetry GenAI spans with prompts, tool calls, tokens and time to first token. - [OpenTelemetry](https://ivytrace.dev/features/opentelemetry): OTLP ingest, so any OpenTelemetry SDK can send traces. - [Self-hosting](https://ivytrace.dev/features/self-hosting): the same images as ivytrace.dev, under the MIT license. ## Frequently asked questions ### Does Ivy slow my functions down? No measurable delay on warm invocations, and about 20–45 ms on cold starts, because spans are sent by a small Lambda extension after your function has responded. If Ivy can't be reached, the layer retries briefly, drops the spans and records how many it dropped; it never blocks or crashes your function. See the [measurements](https://ivytrace.dev/features/auto-instrumentation#performance). ### Which runtimes are supported? Node.js on Lambda (tested on `nodejs22.x`, `nodejs20.x` should work), with the AWS SDK v3, `fetch` and `node:http`, including bundled code. Other runtimes can already send traces with any OpenTelemetry SDK, since ingest is plain OTLP. A Python layer is next on the roadmap. ### What does Ivy get access to in my AWS account? The CloudFormation stack creates a role that Ivy assumes with an external ID, so it can list your functions and update the layers and environment of the ones you choose to trace. It also adds an EventBridge rule so new functions that match your rules get traced. Nothing is traced until you pick functions. Delete the stack to revoke access. ### What about secrets and personal data in payloads? Payloads are capped in size, and keys that match your redaction list are replaced inside the function, before anything is sent. Keys like `authorization`, `cookie`, `password` and `token` are redacted by default. If the data must stay in your network, self-host. ### How much does it cost? The hosted version is free for 100,000 traces a month per workspace, then €5 per million traces. One trace is one Lambda invocation. Self-hosting is free. See [pricing](https://ivytrace.dev/pricing). ### Is it really open source? Yes, MIT licensed. The layer, the API, the dashboard and the MCP server are all in [github.com/ivytrace/ivy](https://github.com/ivytrace/ivy). The hosted version runs the same code. --- Source: https://ivytrace.dev/ · Updated 2026-11-14 # Ivy docs Ivy traces AWS Lambda functions through a Lambda layer and shows the traces in a dashboard, an MCP server and an HTTP API. Most people start by pasting one prompt into their coding agent; everything it does is also documented here. ## Quick start Paste this into your coding agent, in the repository that deploys your Lambda functions: ``` Set up Ivy for my AWS Lambda functions using https://ivytrace.dev/llms.txt ``` The agent follows [llms.txt](https://ivytrace.dev/llms.txt): it signs you up, creates a workspace and project, connects your AWS account through a CloudFormation stack, lets you choose which functions to trace, and checks that the first trace arrived. You approve the sign-in and the stack in your browser. Read [what it does, step by step](https://ivytrace.dev/docs/agent-setup). ## Guides - [Agent setup](https://ivytrace.dev/docs/agent-setup): Paste one prompt and your coding agent sets up Ivy: OAuth device-flow sign-in, workspace and project, a CloudFormation role in your AWS account, auto-trace rules with a dry run, and a verified first trace. - [Manual setup](https://ivytrace.dev/docs/manual-setup): Attach Ivy's Node.js Lambda layer yourself or in infrastructure code. The layer, the four environment variables, all optional settings, and how to get an ingest key. - [Set up alerts](https://ivytrace.dev/docs/alerts): Connect Slack to get a message when a Lambda error group is first seen, tune the thresholds per project, and manage the weekly email digest. Self-hosted setup included. - [MCP server](https://ivytrace.dev/docs/mcp): Add Ivy's remote MCP server to Claude Code, Claude, Cursor or VS Code. OAuth sign-in, read-only tools across all your workspaces, and interactive trace views with MCP Apps. - [HTTP API](https://ivytrace.dev/docs/api): Ivy's HTTP API covers everything the dashboard does, described by an OpenAPI 3.1 spec. Authentication, workspace and project routes, and traces as compact text for LLMs. - [Self-hosting](https://ivytrace.dev/docs/self-hosting): Run Ivy on your own server with docker compose. Services and ports, the environment variables that matter, OIDC sign-in, ingest keys, auto-trace and alerts. - [Span conventions](https://ivytrace.dev/docs/span-conventions): The OpenTelemetry attributes Ivy's Lambda layer emits for invocations, AWS SDK and HTTP calls, logs, failures outside the handler and LLM spans, plus payload capping and redaction defaults. ## Concepts - **Trace:** one Lambda invocation with everything it did. Billing counts traces. - **Span:** one operation inside a trace: the invocation itself, an AWS SDK call, an HTTP request, an LLM call. - **Workspace:** your team. It owns members, billing and connected AWS accounts. - **Project:** one app or environment inside a workspace (`checkout-prod`, `checkout-staging`). Every function belongs to one project, and each project has its own ingest keys. - **Auto-trace rule:** a saved choice of functions (by CloudFormation stack, name pattern or tag) and the project they trace into. New matching functions are traced automatically. - **Error group:** failures with the same function, error type and normalized message. ## Reference - [API reference](https://ivytrace.dev/api/docs) and [OpenAPI spec](https://ivytrace.dev/api/openapi.json) - [Span conventions](https://ivytrace.dev/docs/span-conventions) - [Layer environment variables](https://ivytrace.dev/docs/manual-setup#environment-variables) - [Ivy agent skill](https://ivytrace.dev/skills/ivy/SKILL.md) - [Source code on GitHub](https://github.com/ivytrace/ivy) --- Source: https://ivytrace.dev/docs · Updated 2026-11-14 # Set up Ivy with your coding agent Ivy's setup is written for coding agents. One prompt points your agent to `https://ivytrace.dev/llms.txt`, and the agent does the rest through Ivy's HTTP API, asking you to approve two things in your browser. ## The prompt ``` Set up Ivy for my AWS Lambda functions using https://ivytrace.dev/llms.txt ``` Any agent that can fetch a URL and make HTTP requests works: Claude Code, Cursor, Codex, Windsurf and others. Run it in the repository that deploys your functions, so the agent can check your infrastructure code. ## What the agent does 1. **Signs you in.** It registers itself with Ivy's sign-in service and starts an OAuth device flow with your email address. You open the link, check the code, sign in (or create an account) and click **Allow access**. The agent never sees a password. 2. **Picks a workspace.** New users get a personal workspace. If you have several, it asks which one. 3. **Picks or creates a project.** Projects are named after the app or stack, one per environment (`checkout-prod`, `checkout-staging`). 4. **Connects your AWS account.** It asks for the account ID and regions, then gives you a **CloudFormation quick-create link**. You open it in the AWS console, tick the IAM acknowledgement and click **Create stack**. The stack creates a role that Ivy assumes with an external ID, and an EventBridge rule that tells Ivy about function changes. No access keys change hands, and nothing is traced yet. 5. **Chooses functions with you.** It lists your functions grouped by CloudFormation stack, asks what to trace, shows a **dry run** of the rule, and creates it when you confirm. Ivy attaches the layer within seconds. 6. **Checks your infrastructure code.** If CDK, SST, the Serverless Framework, SAM, Terraform or Pulumi sets your functions' layers or environment, the next deploy could remove Ivy's layer. The agent offers to add the layer and variables to your code instead ([manual setup](https://ivytrace.dev/docs/manual-setup)). 7. **Verifies the first trace.** With your permission it invokes a function (or asks you to), waits for the trace, and gives you a link and a one-line summary. 8. **Offers the extras.** The [Ivy skill](https://ivytrace.dev/skills/ivy/SKILL.md) for your repo, a two-line pointer in `AGENTS.md`, and the [MCP server](https://ivytrace.dev/docs/mcp) in `.mcp.json`. It commits nothing without asking. ## Rules the agent follows `llms.txt` tells agents to never ask for passwords or AWS keys, to send Ivy tokens only to `https://ivytrace.dev/api`, to ask before invoking functions or changing infrastructure code, and to stop and show you the error if a step fails. ## Doing it in the dashboard instead Everything the agent does is also in the dashboard: sign in at [ivytrace.dev](https://ivytrace.dev/login), and the quick-start wizard walks you through the workspace, project, AWS connection and first trace. ## Self-hosted If you run Ivy yourself, tell the agent your URL; `llms.txt` explains which steps change. See [self-hosting](https://ivytrace.dev/docs/self-hosting). --- Source: https://ivytrace.dev/docs/agent-setup · Updated 2026-11-14 # Pricing Ivy costs **€5 per million traces**, and the first **100,000 traces each month are free** for every workspace. One trace is one Lambda invocation, however many spans it contains. Every feature is in every plan, and self-hosting is free under the MIT license. | Plan | Price | What you get | |---|---|---| | Free | €0 | 100,000 traces a month per workspace, every feature, unlimited projects and members. At the limit, new traces are rejected until you upgrade or the month resets; nothing stored is lost. | | Pay as you go | €5 per million traces | The first 100,000 traces each month are still free. Never blocked; billed monthly through Stripe. | | Self-hosted | Free (MIT) | Unlimited. The same images as ivytrace.dev, run with `docker compose up`. | ## Cost calculator Enter how many Lambda invocations you trace in a month. The cost is (invocations − 100,000) ÷ 1,000,000 × €5. | Traced invocations per month | Cost per month | |---|---| | 100,000 | €0 | | 1 million | €4.50 | | 10 million | €49.50 | | 50 million | €249.50 | | 100 million | €499.50 | ## What counts as a trace - **One trace is one Lambda invocation.** The AWS SDK calls, HTTP calls, logs and LLM spans inside it are free. - Usage is counted per **workspace**, across all its projects, per calendar month. It resets on the 1st at 00:00 UTC. - Only functions you choose to trace count. Connecting an AWS account traces nothing by itself. - Failures outside the handler (an init error, a crash, a timeout) are one trace each, like any other invocation. A healthy invocation never gets extra traces. ## What's included in every plan - [Lambda tracing](https://ivytrace.dev/features/lambda-tracing) with payloads, AWS-aware spans and logs - [Auto-instrumentation](https://ivytrace.dev/features/auto-instrumentation) and opt-in auto-trace by CloudFormation stack, name or tag - [Failures outside the handler](https://ivytrace.dev/features/lambda-errors) and throttles, grouped into error groups - [Alerts](https://ivytrace.dev/features/alerts): Slack messages per error group and a weekly email digest - [MCP server and HTTP API](https://ivytrace.dev/features/ai-agents), across all your workspaces - [LLM spans](https://ivytrace.dev/features/llm-observability) from the Vercel AI SDK and OpenTelemetry GenAI - [OTLP ingest](https://ivytrace.dev/features/opentelemetry) for anything else that speaks OpenTelemetry - Workspaces, projects and team members, with sign-in through OIDC ## Frequently asked questions ### What happens when a free workspace reaches 100,000 traces? New traces for that workspace are rejected until you upgrade or the month resets on the 1st (UTC). Nothing already stored is lost. The dashboard warns you at 80% and 100% of the quota, ingest answers with HTTP 429 and an upgrade link instead of dropping traces silently, and your functions keep running normally. ### How do I upgrade? In the dashboard, open Settings → Usage and click **Upgrade**. Checkout and invoices go through Stripe, and you're billed monthly for the traces beyond the free 100,000. Manage or cancel billing from the same page. ### Is a paying workspace ever blocked? No. A workspace on pay as you go has no cap. You pay €5 per million for the traces beyond the free 100,000 each month. ### How is a trace different from a span? A span is one operation, such as a DynamoDB query or an HTTP request. A trace is everything one Lambda invocation did. Ivy charges per trace, so instrumenting more calls does not cost more. ### Can I set a budget? Stay on the free plan to cap usage at 100,000 traces a month per workspace. On pay as you go, choose which functions to trace with auto-trace rules; nothing else is counted. ### Does self-hosting cost anything? No. Self-hosted Ivy is MIT licensed and has no limits and no payment code. You pay only for the server you run it on. See [self-hosting](https://ivytrace.dev/docs/self-hosting). ### Do I need a credit card to start? No. Sign up (your coding agent can do it for you) and use the free tier. You add billing details only when you upgrade a workspace to pay as you go. --- Source: https://ivytrace.dev/pricing · Updated 2026-11-14 # Set up alerts Ivy sends a Slack message when an error group is first seen, and a weekly email digest per workspace. Both cover failures outside the handler as well as handler errors. ## Slack 1. In the dashboard, open your project's alert settings and click **Add to Slack**. 2. Approve the Slack app and pick the channel to post to. 3. That's it. When a new error group appears, Ivy posts it with a **View in Ivy** link. **When Ivy posts.** The first occurrence of a new error group posts right away. After that, the same group posts again at most every **60 minutes** or every **100 occurrences**, whichever comes first. You can change both thresholds per project. **What Ivy posts.** The error group and a link to it. No payloads (events, responses, bodies) are sent to Slack. **What alerts cover.** Handler errors, init errors, out-of-memory kills, crashes and timeouts. Throttle alerts are optional. ## Weekly email digest Each week, workspace members get an email with new error groups, the top error groups with their trend, and totals of failures and throttles. Members can opt out of the digest. ## Self-hosted - **Slack:** create a Slack app from the manifest included in the repository, then set the Slack environment variables on the API. - **Email:** set `IVY_SMTP_URL` and `IVY_EMAIL_FROM` so Ivy can send the digest. - **Anything else:** the `on-error-group-created` hook runs an executable from your hooks directory (`IVY_HOOKS_DIR`) whenever a new error group appears, for example to open an issue or call your paging tool. The repository's deploy README lists the exact variables. ## Not supported Metric, latency or cost alerts, custom alert rules beyond the thresholds, and paging integrations such as PagerDuty are not part of Ivy. For those, use CloudWatch alarms, or the `on-error-group-created` hook on a self-hosted install. --- Source: https://ivytrace.dev/docs/alerts · Updated 2026-11-14 # HTTP API Everything in Ivy's dashboard goes through its HTTP API, which you and your agents can use directly. It is described by an OpenAPI 3.1 document at [/api/openapi.json](https://ivytrace.dev/api/openapi.json) and browsable at [/api/docs](https://ivytrace.dev/api/docs). ## Authentication Send an OIDC access token as `Authorization: Bearer `. On ivytrace.dev: - **Coding agents** get a token through the OAuth device flow described in [llms.txt](https://ivytrace.dev/llms.txt) (section 1). Tokens last an hour and refresh for up to 30 days. - **Scripts** can use the same flow, or the authorization code flow with PKCE on a local port. Ingest keys (`IVY_API_KEY`) are separate: they only send spans and can't read data. ## Routes Data routes are scoped to a workspace and project: ``` P=https://ivytrace.dev/api/w//p/ ``` | Route | What | |---|---| | `GET /api/workspaces` | Your workspaces | | `GET /api/w//projects` | Projects in a workspace | | `GET $P/functions?since=24h` | Per-function counts, error rates and latency percentiles | | `GET $P/invocations?function=&status=error&since=1h` | Invocations, filtered | | `GET $P/errors?since=24h` | Error groups (`/api/w//errors` covers every project) | | `GET $P/traces/` | One trace as JSON | | `GET $P/traces/?format=text` | One trace as a compact text tree | | `POST $P/keys` | Create an ingest key for the project | | `GET /api/w//aws/accounts` | Connected AWS accounts, with `POST`, `verify` and function listing under it | | `POST /api/w//aws/rules/dry-run` | Preview an auto-trace rule | The [API reference](https://ivytrace.dev/api/docs) has every route, parameter and response shape. ## Traces as text `?format=text` returns a trace in a form that is cheap for language models to read: ``` checkout-createOrder · API Gateway POST /orders → 500 · cold start · 212ms · ERROR trace 6af84d409a1b… · request 8f1c2d3e-… · eu-west-1 · 96 MB used · 4 spans +0.0ms 212ms checkout-createOrder [invocation] API Gateway POST /orders → 500 ERROR ! TypeError: Cannot read properties of undefined (reading 'id') at createPaymentIntent (/var/task/src/payments.js:42:31) req: {"routeKey":"POST /orders","headers":{"authorization":"[redacted]",…} +2.0ms log info: Creating order for cart cart_u_8812 +4.0ms 11.6ms DynamoDB.GetItem checkout-users (1 item returned) ok +36.0ms 135ms POST api.stripe.com/v1/payment_intents → 402 ok res: {"error":{"code":"card_declined",…}} ``` Add `detail=summary` for less, or `detail=full` for every payload and message. ## Self-hosted The same API runs at `http://localhost:3000/api` on a self-hosted install. Without OIDC configured it needs no token. --- Source: https://ivytrace.dev/docs/api · Updated 2026-11-14 # Manual layer setup Auto-trace attaches Ivy's layer for you. If your infrastructure code owns your functions' layers and environment, add Ivy there instead, so deploys keep it. You need one layer and four environment variables. ## What to add | Setting | Value | |---|---| | Layer | `ivy-trace-nodejs` (see "Getting the layer" below) | | `AWS_LAMBDA_EXEC_WRAPPER` | `/opt/ivy-wrapper` | | `IVY_ENDPOINT` | Your ingest endpoint. Spans go to `/v1/traces` | | `IVY_API_KEY` | A project ingest key. It decides which workspace and project the traces belong to | No code changes. Without `IVY_ENDPOINT` the wrapper leaves your handler alone. **Getting the layer.** When you connect an AWS account, Ivy publishes `ivy-trace-nodejs` in your account and region. Copy the exact layer version ARN from a function Ivy traces (its configuration lists the layer), or from the dashboard. Self-hosting without a connected account? Build the layer from source and publish it yourself: see the [layer README](https://github.com/ivytrace/ivy/blob/main/packages/layer/README.md). **Getting the endpoint and key.** The ingest endpoint is shown in Workspace settings → AWS accounts (and in `GET /api/w//aws/setup` as `ingest_endpoint`). Create an ingest key per project in Settings → Projects, or with `POST /api/w//p//keys`. The key is shown once; store it like your other secrets. ## Examples **AWS CLI** (`--environment` replaces all variables, so include the function's existing ones): ``` aws lambda update-function-configuration \ --function-name checkout-createOrder \ --layers arn:aws:lambda:eu-west-1:123456789012:layer:ivy-trace-nodejs:1 \ --environment 'Variables={AWS_LAMBDA_EXEC_WRAPPER=/opt/ivy-wrapper,IVY_ENDPOINT=https://,IVY_API_KEY=}' ``` **AWS CDK:** ```ts const ivyLayer = lambda.LayerVersion.fromLayerVersionArn(this, "Ivy", ivyLayerArn); new NodejsFunction(this, "CreateOrder", { entry: "src/create-order.ts", layers: [ivyLayer], environment: { AWS_LAMBDA_EXEC_WRAPPER: "/opt/ivy-wrapper", IVY_ENDPOINT: ivyEndpoint, IVY_API_KEY: ivyApiKey, // from Secrets Manager or SSM, not plain code }, }); ``` **Serverless Framework:** ```yaml provider: layers: - arn:aws:lambda:eu-west-1:123456789012:layer:ivy-trace-nodejs:1 environment: AWS_LAMBDA_EXEC_WRAPPER: /opt/ivy-wrapper IVY_ENDPOINT: ${param:ivyEndpoint} IVY_API_KEY: ${ssm:/ivy/api-key} ``` If auto-trace also covers these functions, set the tag `ivy:trace=false` on them, or leave them out of your rules, so Ivy doesn't change them behind your deploys. ## Environment variables | Variable | Default | | |---|---|---| | `AWS_LAMBDA_EXEC_WRAPPER` | | `/opt/ivy-wrapper` turns tracing on | | `IVY_ENDPOINT` | | Ingest base URL | | `IVY_API_KEY` | | Sent as `Authorization: Bearer ` | | `IVY_DISABLE` | `false` | Turn tracing off without removing the layer | | `IVY_MAX_PAYLOAD_BYTES` | `16384` | Cap for each payload (event, response, bodies) | | `IVY_REDACT_KEYS` | see [span conventions](https://ivytrace.dev/docs/span-conventions#redaction) | Comma-separated; a key is redacted if it contains one of these (case-insensitive, `-` and `_` ignored) | | `IVY_CAPTURE_LOGS` | `true` | Record `console.*` calls on the trace | | `IVY_CAPTURE_OTEL` | `true` | Capture spans that libraries create through `@opentelemetry/api` (Vercel AI SDK, …) | | `IVY_PLATFORM_TELEMETRY` | `true` | Use the Lambda Telemetry API to report init errors, crashes, out of memory and timeouts | | `IVY_REPORT_METRICS` | `true` | Send billed duration and max memory used | | `IVY_TIMEOUT_MARGIN_MS` | `100` | How long before the deadline to snapshot spans for a possible timeout | | `IVY_EXPORT_TIMEOUT_MS` | `1000` | Per-request export timeout | | `IVY_FLUSH_BUDGET_MS` | `2000` | Most time the extension spends exporting after each invocation | | `IVY_MAX_QUEUE_SPANS` | `2000` | Spans buffered while the endpoint is unreachable | | `IVY_DEBUG` | `false` | Log layer diagnostics | ## Requirements - Node.js runtime: `nodejs22.x` is tested, `nodejs20.x` should work (Node 20.16 or later). - x86_64 or arm64; one layer zip serves both. - Your function must be able to reach the ingest endpoint (outbound HTTPS, or a route to your self-hosted Ivy). - Not traced yet: response streaming handlers and Lambda multi-concurrency. --- Source: https://ivytrace.dev/docs/manual-setup · Updated 2026-11-14 # MCP server Ivy's MCP server lets your coding agent read production Lambda traces, errors and payloads. It is a remote server at `https://ivytrace.dev/mcp` with OAuth sign-in, and it covers all of your workspaces at once. ## Add it to your client **Claude Code:** ``` claude mcp add --transport http ivy https://ivytrace.dev/mcp ``` Or commit it for your team in `.mcp.json`: ```json { "mcpServers": { "ivy": { "type": "http", "url": "https://ivytrace.dev/mcp" } } } ``` **Claude (web and desktop):** Settings → Connectors → **Add custom connector**, with the URL `https://ivytrace.dev/mcp`. **Cursor:** in `.cursor/mcp.json` (project) or `~/.cursor/mcp.json` (global): ```json { "mcpServers": { "ivy": { "url": "https://ivytrace.dev/mcp" } } } ``` **VS Code:** in `.vscode/mcp.json`: ```json { "servers": { "ivy": { "type": "http", "url": "https://ivytrace.dev/mcp" } } } ``` **Other clients:** any client that supports remote (streamable HTTP) MCP servers with OAuth. The first time a tool runs, the client opens a browser window to sign in with your Ivy account. ## Tools All tools are read-only. Every tool except `list_workspaces` takes optional `workspace` and `project` (slug or ID); without them it searches every workspace you belong to, and each result says which workspace it came from. | Tool | Inputs | Returns | |---|---|---| | `list_workspaces` | | Your workspaces and their projects | | `find_failing_invocations` | `function`, time range | Recent failures, grouped by error | | `search_invocations` | function, status, request ID, path, error message | Matching invocations | | `list_error_groups` | time range | Error groups with counts and first and last seen, including throttles | | `get_trace_summary` | `trace_id`, detail level | The trace as a compact text tree | | `compare_invocations` | two trace IDs | What differs between a good and a bad run | | `render_waterfall` | `trace_id` | Interactive waterfall view (MCP Apps), text elsewhere | | `render_graph` | `trace_id` | Interactive service graph (MCP Apps), text elsewhere | `render_waterfall` and `render_graph` are **MCP Apps**: in hosts that support them, they show the dashboard's own trace components in the conversation. The view gets its data from the tool result and makes no calls of its own. Hosts without MCP Apps show the text version. ## Example prompts - *"Checkout returns 500s since this morning's deploy. Check Ivy."* - *"Why is `orders-api-prod-listOrders` slow today? Compare a slow and a fast invocation."* - *"Show me the waterfall of the last failed `createOrder` invocation."* - *"Did the fix work? Look for new errors in the last 30 minutes."* Add the [Ivy skill](https://ivytrace.dev/skills/ivy/SKILL.md) to your repo so the agent knows these workflows and which workspace and project belong to the repo. ## Self-hosted Use your own URL, for example `claude mcp add --transport http ivy http://localhost:3000/mcp`. With OIDC configured, the MCP endpoint is an OAuth protected resource and clients sign in through your provider (which must support dynamic client registration or client ID metadata documents). Without OIDC it is open, like the rest of a self-hosted install. --- Source: https://ivytrace.dev/docs/mcp · Updated 2026-11-14 # Self-hosting quickstart Ivy runs anywhere Docker runs. One `docker compose up` starts the ingest router, the OpenTelemetry Collector, ClickHouse, and the API with the dashboard and MCP server. It is MIT licensed and has no limits. ## Start it ``` git clone https://github.com/ivytrace/ivy && cd ivy docker compose -f deploy/docker-compose.yml up -d --build ``` | Service | Address | |---|---| | Dashboard, API and MCP server | `http://localhost:3000` (API at `/api`, MCP at `/mcp`) | | OTLP/HTTP ingest | `http://localhost:4318/v1/traces`, with `Authorization: Bearer ` | | ClickHouse HTTP | `http://localhost:8123` | On startup Ivy creates a `default` workspace, its `default` project and an ingest key (`IVY_INGEST_API_KEY`, `dev-ingest-key` unless you set it). Stop with `docker compose -f deploy/docker-compose.yml down` (add `-v` to delete the data). ## Settings that matter Set these in your shell or in `deploy/.env`. **Change the defaults before exposing Ivy to a network.** | Variable | Default | | |---|---|---| | `IVY_INGEST_API_KEY` | `dev-ingest-key` | Key of the default project | | `IVY_COLLECTOR_TOKEN` | `dev-collector-token` | Shared secret between the ingest router and the collector | | `IVY_CLICKHOUSE_USER`, `IVY_CLICKHOUSE_PASSWORD` | `ivy`, `ivy` | ClickHouse credentials | | `IVY_RETENTION_DAYS` | `30` | How long traces are kept (set before the first start) | | `IVY_PUBLIC_URL` | | The URL people use to reach Ivy | | `IVY_INGEST_PUBLIC_URL` | | The ingest URL your Lambda functions can reach (needed for auto-trace) | | `IVY_SECRET_KEY` | | Encrypts per-project auto-trace keys | The [deploy README](https://github.com/ivytrace/ivy/blob/main/deploy/README.md) lists every variable. ## Sign-in with OIDC Without OIDC, the dashboard, API and MCP server run open (for setups behind your own authenticating proxy) and the dashboard shows an "authentication disabled" banner. To require sign-in: 1. Register a public SPA client with your provider (Keycloak, Auth0, Okta, Entra ID, Authentik, Zitadel, …) with the redirect URI `/auth/callback`, and an API audience whose access tokens are JWTs. 2. Set `IVY_OIDC_ISSUER` and `IVY_OIDC_CLIENT_ID` (and `IVY_OIDC_AUDIENCE` if it differs from the client ID). 3. For several workspaces with members and invitations, set `IVY_SINGLE_TENANT=0`. For invitation emails, set `IVY_SMTP_URL` and `IVY_EMAIL_FROM`. Ivy never uses passwords of its own. ## Send traces - **Lambda, by hand:** build and publish the layer ([layer README](https://github.com/ivytrace/ivy/blob/main/packages/layer/README.md)), then set `IVY_ENDPOINT` to your ingest URL and `IVY_API_KEY` to a project key ([manual setup](https://ivytrace.dev/docs/manual-setup)). - **Lambda, with auto-trace:** give Ivy an AWS identity (`IVY_AWS_PRINCIPAL_ARN` and AWS credentials) that can assume the role the CloudFormation template creates, and set `IVY_INGEST_PUBLIC_URL`. Then connect accounts in the dashboard. Without a hook that publishes templates, the dashboard offers the template as a download. - **Anything else:** any OpenTelemetry SDK, over OTLP/HTTP to port 4318 ([OpenTelemetry](https://ivytrace.dev/features/opentelemetry)). ## Alerts Turn on Slack and the weekly digest with environment variables, and use the `on-error-group-created` hook for anything else. See [setting up alerts](https://ivytrace.dev/docs/alerts). ## Agents Point your agent at your instance: `claude mcp add --transport http ivy https://ivy.example.com/mcp`, and tell it to use your URL instead of `https://ivytrace.dev` when following `llms.txt`. ## Updating Pull the new version, apply the schema (it is idempotent) and restart: ``` git pull docker compose -f deploy/docker-compose.yml up -d --build docker compose -f deploy/docker-compose.yml exec -T clickhouse /docker-entrypoint-initdb.d/init.sh ``` --- Source: https://ivytrace.dev/docs/self-hosting · Updated 2026-11-14 # Span conventions Ivy stores standard OpenTelemetry spans. This is a summary of the attributes its Lambda layer emits and the ones Ivy reads, so you can send compatible spans from any OpenTelemetry SDK. The full reference is in the repository. ## Resource | Attribute | Example | |---|---| | `service.name`, `faas.name` | `checkout-api-prod-createOrder` | | `cloud.provider`, `cloud.platform` | `aws`, `aws_lambda` | | `cloud.region`, `cloud.account.id` | `eu-west-1`, `123456789012` | | `faas.version`, `faas.max_memory` | `$LATEST`, memory in bytes | ## Invocation (root span) One per invocation; Ivy's invocation list, error groups and billing are built from it. | Attribute | | |---|---| | `ivy.invocation = true` | Required marker | | `faas.invocation_id` | AWS request ID | | `faas.coldstart`, `ivy.init_duration_ms` | Cold start and init duration | | `faas.trigger`, `ivy.trigger.source` | `http`, `pubsub`, …; `aws:apigateway`, `aws:sqs`, `aws:eventbridge`, … | | `cloud.resource_id` | Function ARN | | `http.request.method`, `url.path`, `http.route`, `http.response.status_code` | HTTP triggers | | `ivy.payload.request`, `ivy.payload.response` | Event and result, JSON, capped | | `ivy.billed_duration_ms`, `ivy.memory_max_used_mb` | From the Lambda platform report | Errors set span status `ERROR`, `error.type`, and an `exception` event with type, message and stack trace. ## Failures outside the handler Set by the extension from the Lambda Telemetry API: | Attribute | | |---|---| | `error.type` | `Runtime.ImportModuleError`, `Runtime.OutOfMemory`, `Runtime.ExitError`, `Sandbox.Timedout`, … | | `ivy.failure.phase` | `init` or `invoke` | | `ivy.platform.status` | `failure`, `error` or `timeout` | | `ivy.synthetic = true` | The extension created the span because the tracer couldn't | The last log lines (up to 50 lines or 8 KB, redacted) are attached as `log` events. ## Outgoing calls **AWS SDK:** span name `DynamoDB.GetItem`; `rpc.system = aws-api`, `rpc.service`, `rpc.method`, `aws.request_id`, `cloud.region`, plus `aws.dynamodb.table_names`, `aws.s3.bucket` and `aws.s3.key`, `messaging.system` and `messaging.destination.name` (SQS, SNS), `faas.invoked_name` (Lambda), `aws.eventbridge.bus`. **HTTP:** span name `POST api.stripe.com`; `http.request.method`, `url.full` (query string redacted), `server.address`, `http.response.status_code`, and `ivy.http.request.headers` and `ivy.http.response.headers` with sensitive headers redacted. Both carry `ivy.payload.request` and `ivy.payload.response`. ## Logs `console.*` calls become `log` events on the invocation span, with `log.severity` and `log.message` (capped at 4 KB). ## LLM spans Spans that libraries create through `@opentelemetry/api` keep their own names and attributes. Ivy reads three families: the Vercel AI SDK's `ai.*` attributes, OpenTelemetry GenAI `gen_ai.*`, and their variants, to show messages, tool calls, tokens and time to first token. See [LLM observability](https://ivytrace.dev/features/llm-observability). ## Redaction - Each `ivy.payload.*` value is capped at `IVY_MAX_PAYLOAD_BYTES` (16 KB). Truncated values end with `…[truncated N bytes]` and the span gets `ivy.payload.truncated = true`. - Keys matching `IVY_REDACT_KEYS` are replaced with `"[redacted]"` at any depth before anything leaves the function. The default list: `password, secret, token, authorization, cookie, apikey, api_key, accesskey, secretaccesskey, sessiontoken, creditcard, cardnumber, cvv`. Full reference: [span-conventions.md](https://github.com/ivytrace/ivy/blob/main/docs/implementation/span-conventions.md). --- Source: https://ivytrace.dev/docs/span-conventions · Updated 2026-11-14 # Features Ivy is open-source observability for AWS Lambda, built around one idea: a trace should show exactly what an invocation did, with its payloads, in a form that both you and your coding agent can read. These are the parts. - [Lambda tracing](https://ivytrace.dev/features/lambda-tracing): Trace every AWS Lambda invocation with its event, response, logs, errors and cold start, plus every DynamoDB, S3, SQS, SNS, EventBridge and HTTP call with request and response bodies. - [Auto-instrumentation](https://ivytrace.dev/features/auto-instrumentation): Ivy's Lambda layer traces Node.js functions without code changes. Connect your AWS account once and pick functions by CloudFormation stack, name or tag; new matches are traced automatically. - [Lambda errors](https://ivytrace.dev/features/lambda-errors): Ivy catches AWS Lambda failures outside the handler (init errors, out-of-memory kills, crashes and timeouts) with their last log lines, counts throttles, and groups everything into error groups. - [Alerts](https://ivytrace.dev/features/alerts): Ivy sends a Slack message when a Lambda error group is first seen, then at most every 60 minutes or 100 occurrences, and a weekly email digest per workspace. Covers init errors, OOM, crashes, timeouts and throttles. - [For AI agents](https://ivytrace.dev/features/ai-agents): Ivy gives coding agents production context for AWS Lambda: a remote MCP server across all your workspaces, interactive trace views (MCP Apps), an HTTP API with OpenAPI, llms.txt onboarding and an agent skill. - [LLM observability](https://ivytrace.dev/features/llm-observability): Trace Vercel AI SDK and OpenTelemetry GenAI calls in AWS Lambda with Ivy. See prompts, responses, tool calls, token counts and time to first token in the same trace as your DynamoDB and HTTP calls. - [OpenTelemetry](https://ivytrace.dev/features/opentelemetry): Ivy ingests standard OpenTelemetry (OTLP over HTTP) and uses OpenTelemetry semantic conventions, so any OTel SDK can send traces, and the Ivy layer is an optional, lighter producer for Node.js Lambda. - [Self-hosting](https://ivytrace.dev/features/self-hosting): Run Ivy yourself with docker compose up. The OpenTelemetry Collector, ClickHouse, the API, dashboard and MCP server, MIT licensed, with OIDC sign-in and no limits. ## What Ivy covers today | Area | Supported | |---|---| | Runtimes | Node.js on AWS Lambda (`nodejs22.x` tested, `nodejs20.x` should work), ESM and CommonJS, bundled or not | | Instrumented calls | AWS SDK v3 (including bundled copies), `fetch`, `node:http` and `node:https`, `console.*` logs | | Triggers recognized | API Gateway (REST and HTTP), ALB, function URLs, SQS, SNS, S3, DynamoDB Streams, EventBridge, schedules, Kinesis, direct invokes | | Failures | Handler errors, init errors, out of memory, crashes, timeouts, throttles | | Alerts | Slack per error group, weekly email digest | | Agents | Remote MCP server with OAuth, HTTP API with OpenAPI 3.1, `llms.txt`, agent skill | | LLM spans | Vercel AI SDK 3–7, OpenTelemetry GenAI semantic conventions | | Ingest | OTLP over HTTP (protobuf or JSON) | | Sign-in | OIDC (hosted: email sign-in; self-hosted: your own provider) | ## Not yet Ivy is young, and these are not part of it yet: - **Runtimes other than Node.js.** A Python layer is next. Until then, other runtimes can send traces through any OpenTelemetry SDK. - **Advanced alerting.** Alerts cover error groups (Slack and a weekly digest). There are no metric or latency alerts, custom alert rules beyond the thresholds, or paging integrations such as PagerDuty. - **Response streaming handlers** (`awslambda.streamifyResponse`) and Lambda multi-concurrency are not traced. - **Trace context propagation** across functions (incoming and outgoing `traceparent` or X-Ray headers). - **Logs and metrics as products.** Ivy keeps the `console.*` lines of each invocation and the last log lines of a failure, but it does not ship or index all of your CloudWatch logs. --- Source: https://ivytrace.dev/features · Updated 2026-11-14 # Lambda observability for AI coding agents Ivy is built so that coding agents such as Claude Code, Cursor and Codex can debug production Lambda functions on their own. An agent sets Ivy up from one prompt through `llms.txt`, then reads traces, errors and payloads through a **remote MCP server** or the **HTTP API**, and shows traces as interactive views right in the chat. ## Set up from one prompt ``` Set up Ivy for my AWS Lambda functions using https://ivytrace.dev/llms.txt ``` The prompt points the agent at [llms.txt](https://ivytrace.dev/llms.txt), a setup script written for agents. The agent signs you up with an OAuth device flow (you approve in your browser, and never hand it a password or AWS key), creates a workspace and project, gives you a CloudFormation link to connect your AWS account, previews which functions a rule would trace, and invokes one function to check that its trace arrived. At the end it offers to add the Ivy skill, an `AGENTS.md` pointer and the MCP server to your repo. It commits nothing without asking. See [agent setup](https://ivytrace.dev/docs/agent-setup). ## The MCP server A remote MCP server at `https://ivytrace.dev/mcp`, with OAuth sign-in. It covers **all of your workspaces** at once, and every result says which workspace and project it came from. | Tool | What it does | |---|---| | `list_workspaces` | The workspaces and projects you can read | | `find_failing_invocations` | Recent failures for a function, grouped by error | | `search_invocations` | Invocations by function, status, request ID, path or error message | | `list_error_groups` | Errors by fingerprint, with counts and first and last seen | | `get_trace_summary` | A whole trace as a compact text tree with timings, errors and trimmed payloads | | `compare_invocations` | What differs between a good and a bad run | | `render_waterfall` | The trace's waterfall as an interactive view in the chat | | `render_graph` | The trace's service graph as an interactive view in the chat | The tools are read-only and answer in compact text, so they spend few tokens. `render_waterfall` and `render_graph` are **MCP Apps**: hosts that support them show the real dashboard components inline, and other hosts get the text version. Add it to Claude Code with: ``` claude mcp add --transport http ivy https://ivytrace.dev/mcp ``` Setup for Claude, Cursor and other clients: [MCP server docs](https://ivytrace.dev/docs/mcp). ## The HTTP API Everything the dashboard does goes through an open HTTP API, described by an OpenAPI 3.1 document at [/api/openapi.json](https://ivytrace.dev/api/openapi.json) and browsable at [/api/docs](https://ivytrace.dev/api/docs). Any trace is available as text with `?format=text`: ``` GET https://ivytrace.dev/api/w//p//traces/?format=text ``` There is no Ivy CLI on purpose: agents already speak MCP and HTTP. ## The agent skill The [Ivy skill](https://ivytrace.dev/skills/ivy/SKILL.md) is a `SKILL.md` file for your repo. It tells your agent when to reach for Ivy (a Lambda error, a slow endpoint, checking a deploy), which tools to call in which order, and which workspace and project belong to this repo. ## A typical session > Checkout is returning 500s since this morning's deploy. Can you check Ivy? The agent calls `find_failing_invocations` for the checkout function, opens one trace with `get_trace_summary`, sees that `SQS.SendMessage` to `order-events.fifo` fails with `MissingParameter: MessageGroupId`, finds the line in your code that dropped it, and fixes it. With MCP Apps, it shows you the waterfall in the chat while it explains. ## Why this matters Your agent already has the code. What it usually lacks is production: the failing request, the payload that caused it, the stack trace, and what the function called. Ivy gives it that, scoped by the same sign-in and permissions you have. --- Source: https://ivytrace.dev/features/ai-agents · Updated 2026-11-14 # Lambda error alerts in Slack, and a weekly digest Ivy alerts you about Lambda errors in two ways. A **Slack message per error group** when an error is first seen, and again at most every 60 minutes or every 100 occurrences by default. And a **weekly email digest** per workspace with new errors, the top error groups and their trend, and totals of failures and throttles. Alerts are included in every plan. ## Slack alerts per error group - **Add to Slack** and pick the channel. Each alert describes the error group (the function and its error) with a **View in Ivy** link. - **First seen, then throttled.** The first occurrence of a new error group sends a message right away. After that, the same group posts again at most every **60 minutes** or every **100 occurrences**, whichever comes first. Both thresholds can be changed per project. - **No payloads in Slack.** Messages carry the error, not your request or response bodies. The data stays in Ivy. - **Covers failures outside the handler:** init errors, out-of-memory kills, crashes and timeouts alert like any handler error. Throttle alerts are optional. ## Weekly email digest Once a week, each workspace member gets a summary: - new error groups that week; - the top error groups, with their trend against the week before; - totals of failures and throttles. Members can opt out of the digest. ## Self-hosted On a self-hosted install, Slack and email are turned on with environment variables. A Slack app manifest is included, so creating your own Slack app takes a minute, and email goes through your SMTP server. For anything else, the `on-error-group-created` [hook](https://github.com/ivytrace/ivy) runs your own script when a new error group appears, for example to open an issue in your tracker. See [setting up alerts](https://ivytrace.dev/docs/alerts). ## What alerts don't do (yet) Alerts are deliberately simple. Ivy does not have: - metric, latency or cost alerts; - custom alert rules beyond the per-project thresholds above; - paging integrations such as PagerDuty or Opsgenie. If you need those today, pair Ivy with CloudWatch alarms, or use the `on-error-group-created` hook on a self-hosted install. --- Source: https://ivytrace.dev/features/alerts · Updated 2026-11-14 # Lambda auto-instrumentation Ivy instruments Node.js Lambda functions with one Lambda layer and no code changes. Connect an AWS account through a CloudFormation stack, choose which functions to trace by stack, name pattern or tag, and Ivy attaches the layer for you, including to new functions that match later. ## How the layer works The layer has three small parts: 1. **An exec wrapper** (`AWS_LAMBDA_EXEC_WRAPPER=/opt/ivy-wrapper`) that loads Ivy's tracer before your handler. Without `IVY_ENDPOINT` it changes nothing. 2. **An in-process tracer** (32 KB) that records the invocation and patches the AWS SDK v3, `fetch` and `node:http`. It works with ESM and CommonJS handlers and with bundled code. 3. **A Lambda extension written in Rust** that exports the spans after your function has responded, and reads the Lambda Telemetry API to catch [failures outside the handler](https://ivytrace.dev/features/lambda-errors). One layer zip serves both x86_64 and arm64. ## Opt-in auto-trace Connecting an AWS account traces nothing by itself. You decide what to trace: 1. **Connect.** Ivy gives you a CloudFormation quick-create link. The stack creates an IAM role that Ivy assumes with an external ID, and an EventBridge rule that reports function changes. No access keys change hands. 2. **Pick.** Ivy lists your functions grouped by **CloudFormation stack**. Trace a whole stack, a name pattern (`checkout-*`) or a tag (`team=core`), optionally limited to an account and region. Each choice is saved as a rule that also names the [project](https://ivytrace.dev/docs/agent-setup) the traces go to. A dry run shows what a rule would select first. 3. **Stay in sync.** New functions that match a rule are traced automatically. Remove a rule, or exclude a function, and the next sync removes the layer and Ivy's environment variables again. Function tags override rules: `ivy:trace=true` or `ivy:trace=false`, and `ivy:project=`. ## If you deploy with infrastructure as code If CDK, SST, the Serverless Framework, SAM, Terraform or Pulumi manages your functions' layers and environment, the next deploy can remove what Ivy attached. In that case, add the layer and four environment variables to your infrastructure code instead. The [manual setup](https://ivytrace.dev/docs/manual-setup) page lists them, and the [agent setup](https://ivytrace.dev/docs/agent-setup) offers to do it for you. ## Performance Your callers don't wait for Ivy: spans are sent by the extension after your function has responded. Measured on AWS Lambda (eu-west-1, Node.js 22, arm64 and x86_64, 128 MB and 1024 MB) in November 2026: | Measure | Result | |---|---| | Warm response latency | No measurable difference at 1024 MB; about 1 ms at the median at 128 MB | | Cold start (init duration) | About 20–45 ms at the median | | Billed duration | About 15–20 ms per invocation at 1024 MB, for the export after the response | The tracer is 32 KB, and the extension is written in Rust because a Node.js extension added about 30 ms to every cold start. ## Limits - Node.js only, for now. A Python layer is next. - Response streaming handlers and Lambda multi-concurrency are not traced. - `undici` used directly is not traced (Node's global `fetch` is). --- Source: https://ivytrace.dev/features/auto-instrumentation · Updated 2026-11-14 # Lambda errors, including the ones your handler never sees Ivy reports every failed Lambda invocation, including the failures an in-process tracer can't see. Init errors like `Runtime.ImportModuleError`, out-of-memory kills, crashes and timeouts show up as traces with the function's last log lines, throttles are counted per function, and all of it is grouped into error groups that can [alert you in Slack](https://ivytrace.dev/features/alerts). ## Why some Lambda failures are invisible Most tracers live inside your function's process. That works for an exception thrown by your handler, but not when: | Failure | Why an in-process tracer misses it | |---|---| | Init errors: `Runtime.ImportModuleError`, `Runtime.HandlerNotFound`, `Runtime.UserCodeSyntaxError`, a throw at module load | Your code never finished loading, so there is no wrapped handler | | Out of memory (`Runtime.OutOfMemory`) and crashes (`Runtime.ExitError`) | The process dies, and spans in memory die with it | | Timeouts (`Sandbox.Timedout`) | The function is stopped before it can report | | Throttles | The request never reaches the function | ## How Ivy catches them Ivy's Lambda extension runs outside your function's process and subscribes to the **Lambda Telemetry API**. When the platform reports a failure, the extension: - completes the invocation's trace (or creates one, if the tracer sent nothing) with the platform's error type: `Runtime.OutOfMemory`, `Sandbox.Timedout`, `Runtime.ImportModuleError` and so on; - attaches the **last log lines** of the function (at most 50 lines or 8 KB, with your redaction rules applied), so you see `Error: Cannot find module 'left-pad'` or V8's heap limit message, not just "failed"; - keeps the spans recorded before a crash or timeout, so you can see how far the invocation got. The dashboard labels these failures plainly: **Init failed**, **Out of memory**, **Timed out** or **Crashed**. **Throttles** never reach a function, so Ivy reads them from CloudWatch through the same cross-account role: every 5 minutes, an account-level check per region, and per-function counts only when something was throttled. Throttles appear on the function overview, in the function's chart and as a "Throttled" error group. The polling runs in your AWS account (a few cents a month per region in CloudWatch API calls) and can be switched off per account. ## No log shipping, no extra cost for healthy invocations Ivy does not ship your function logs. Log lines are kept in a small buffer inside the extension and only sent when an invocation fails. A healthy invocation adds no extra trace, so it costs nothing extra. ## Error groups Every failure, inside or outside the handler, lands in an **error group**, fingerprinted by function, error type and normalized message. Each group shows its count, first and last seen, and the traces behind it. - In the dashboard, under **Errors**. - For coding agents, through the MCP tools `list_error_groups` and `find_failing_invocations` (see [for AI agents](https://ivytrace.dev/features/ai-agents)). - In Slack and in a weekly email digest, through [alerts](https://ivytrace.dev/features/alerts). ## Example: an init error after a deploy A deploy ships a bundle that is missing a dependency. Every cold start fails before your handler runs, so callers get errors and an in-process tracer reports nothing. In Ivy you see an error group like: ``` checkout-createOrder · Init failed · Runtime.ImportModuleError Error: Cannot find module 'left-pad' 12 occurrences · first seen 10:04 · last seen 10:09 ``` with a trace per failed attempt and the log lines the runtime printed. --- Source: https://ivytrace.dev/features/lambda-errors · Updated 2026-11-14 # AWS Lambda tracing with payloads Ivy traces every invocation of the AWS Lambda functions you choose, with the event that triggered it, the response it returned, its logs and errors, and every AWS SDK and HTTP call it made, including the request and response bodies. A DynamoDB query looks like a DynamoDB query, not like an anonymous HTTPS POST. ## What one trace contains Each Lambda invocation becomes one trace: an invocation span with a child span for each outgoing call. - **The invocation:** function name, AWS request ID, trigger (API Gateway, SQS, S3, EventBridge, a schedule, …), HTTP method, path and status code for HTTP triggers, cold start with the init duration, memory used, billed duration, and status. - **Payloads:** the event and the handler's response, as JSON, capped at 16 KB each by default. - **Logs:** every `console.*` call during the invocation, with its level, on the timeline. - **Errors:** the exception type, message and stack trace (with the layer's own frames removed). - **Outgoing calls:** every AWS SDK v3 call, `fetch` and `node:http` request, with its duration, status and bodies. ## AWS-aware spans Ivy's layer names and labels AWS calls the way you think about them: | Service | What the span shows | |---|---| | DynamoDB | `DynamoDB.GetItem` on `checkout-carts`, the key, the item returned | | S3 | bucket and key | | SQS and SNS | queue or topic name, message ID, the message body | | EventBridge | bus and detail types | | Lambda | the invoked function and region | | Other AWS services | `.` with request and response | | HTTP | `POST api.stripe.com`, the URL (query string redacted), status, headers and bodies | The dashboard draws each trace as a **service graph** (your function and every service it called) next to a **waterfall** timeline. Click a node or a row to see its payloads. The [home page](https://ivytrace.dev/) has an interactive copy of the real trace page. This also works when your bundler inlines the AWS SDK: Ivy recognizes signed AWS requests and still shows them as AWS calls. ## Find the invocation you need - **Invocations:** filter by function, status and time range; search by request ID, HTTP path or error message. - **Functions:** invocations, errors, cold starts, throttles and duration percentiles per function. - **Errors:** error groups by fingerprint (function, error type and normalized message), with counts and first and last seen. - **Workspaces and projects:** keep `checkout-prod` and `checkout-staging` apart, invite your team, and switch with ⌘K. ## Safe by default - Keys like `authorization`, `cookie`, `password`, `secret`, `token` and `apikey` are replaced with `[redacted]` at any depth, **inside your function**, before anything is sent. Add your own keys with `IVY_REDACT_KEYS`. - Payloads are capped (`IVY_MAX_PAYLOAD_BYTES`, 16 KB by default) and marked when truncated. - Spans are exported after your function has responded, by a small Lambda extension written in Rust. If Ivy can't be reached, the extension retries briefly and then drops the spans; it never blocks or crashes the function. - Need the data to stay in your network? [Self-host](https://ivytrace.dev/features/self-hosting) the same software. ## Fast by design Spans are sent after your function has responded, so warm invocations show no measurable delay, and cold starts take about 20–45 ms longer. The export after the response adds its round trip to billed duration. See the [measurements](https://ivytrace.dev/features/auto-instrumentation#performance). ## For your coding agent too Every trace is also available as a compact text tree through the [MCP server and the HTTP API](https://ivytrace.dev/features/ai-agents), so an agent can read the failing request and its payload and fix the bug in the same session: ``` checkout-createOrder · API Gateway POST /orders → 500 · cold start · 212ms · ERROR +0.0ms 212ms checkout-createOrder [invocation] API Gateway POST /orders → 500 ERROR ! TypeError: Cannot read properties of undefined (reading 'id') +4.0ms 11.6ms DynamoDB.GetItem checkout-users (1 item returned) ok +36.0ms 135ms POST api.stripe.com/v1/payment_intents → 402 ok ``` --- Source: https://ivytrace.dev/features/lambda-tracing · Updated 2026-11-14 # LLM observability for AI apps on Lambda Ivy records the LLM calls your Lambda functions make, from the **Vercel AI SDK** or any library that emits **OpenTelemetry GenAI** spans, inside the same trace as everything else the invocation did. You see the conversation, the response, tool calls and their results, token counts and time to first token, next to the DynamoDB query a tool ran. ## What you see - **Each model call:** provider, model, settings, the messages sent, the response, input and output tokens, and time to first token for streamed calls. - **Agent steps and tools:** the steps of a multi-step generation, each tool call with its arguments and result, and the AWS SDK or HTTP calls the tool made, nested under it. - **The provider's HTTP request** (`POST api.openai.com`, …) under the model call that made it. - **Totals:** tokens in and out for the whole trace in its header, and an "LLM usage" table per model. - **In the service graph:** one node per provider, so you see where an invocation spent its time. The same data is available to agents: [`get_trace_summary`](https://ivytrace.dev/features/ai-agents) prints lines like `chat gpt-4o-mini 812→24 tok tool:getOrder`. ## Setup Attach the Ivy layer as usual ([auto-instrumentation](https://ivytrace.dev/features/auto-instrumentation)). Then turn on the AI SDK's telemetry: **AI SDK 3 to 6:** pass `experimental_telemetry` on each call. ```ts import { generateText } from "ai"; const result = await generateText({ model, prompt, experimental_telemetry: { isEnabled: true }, }); ``` **AI SDK 7:** register the OpenTelemetry integration once, at the top of the handler module. ```ts import { registerTelemetry } from "ai"; import { OpenTelemetry } from "@ai-sdk/otel"; registerTelemetry(new OpenTelemetry()); ``` You don't need an OpenTelemetry SDK. The layer registers a minimal tracer provider that libraries using `@opentelemetry/api` report to. If your app registers its own provider (for example with the AWS Distro for OpenTelemetry), Ivy steps aside. `IVY_CAPTURE_OTEL=false` turns this off. ## Supported conventions | Source | Spans | |---|---| | Vercel AI SDK 3–6 | `ai.generateText`, `ai.streamText`, `ai.generateObject`, `ai.toolCall`, `ai.embed`, … | | Vercel AI SDK 7 with `@ai-sdk/otel` | OpenTelemetry GenAI conventions: `invoke_agent`, `chat`, `execute_tool`, … | | Other OpenTelemetry GenAI instrumentations | OpenAI, Anthropic and LangChain instrumentations that use `@opentelemetry/api` | Attributes are stored as the library sent them and interpreted when read, so new instrumentations only need a change in how Ivy displays them. ## Privacy and limits - Prompts and responses follow the same rules as other payloads: capped per attribute (`IVY_MAX_PAYLOAD_BYTES`), and string values under keys that match your redaction list are replaced. Token counts are kept. - Long conversations can exceed the cap; raise `IVY_MAX_PAYLOAD_BYTES` if you need the full text. - Ivy shows tokens, not cost estimates: model prices change too often to hard-code. - Response streaming Lambda handlers (`awslambda.streamifyResponse`) are not traced yet. Streaming from the model inside a normal handler is. --- Source: https://ivytrace.dev/features/llm-observability · Updated 2026-11-14 # OpenTelemetry for AWS Lambda Ivy is built on OpenTelemetry. Ingest accepts standard **OTLP over HTTP**, spans use OpenTelemetry semantic conventions (`rpc.system=aws-api`, `faas.*`, `http.*`, `gen_ai.*`), and storage uses the OpenTelemetry Collector's ClickHouse exporter. The Ivy Lambda layer is one producer of those spans, tuned for Lambda; any OpenTelemetry SDK can be another. ## Send OTLP from anything Point any OpenTelemetry SDK or Collector at Ivy's ingest endpoint with a project's ingest key: ``` OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https:///v1/traces OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer ``` The key decides which workspace and project the spans land in. Create one in the dashboard (Settings → Projects) or through the API. Ingest takes protobuf or JSON over HTTP; there is no gRPC endpoint. To appear as a Lambda invocation in Ivy's invocation list and error groups, a trace's root span needs `ivy.invocation=true` plus the usual `faas.*` attributes; other spans are stored and shown as they are. The [span conventions](https://ivytrace.dev/docs/span-conventions) list the attributes. ## Why a layer of its own, then? The OpenTelemetry Lambda layers are general purpose and configurable. Ivy's layer makes a few Lambda-specific choices out of the box: - **Small cold-start cost.** About 20–45 ms added to init, with a 32 KB tracer and an extension written in Rust. - **Export after the response.** Spans leave through a Lambda extension once your function has responded, so warm responses are not measurably slower. - **Payloads by default.** Events, responses and AWS SDK and HTTP bodies, capped and redacted inside the function. - **Failures outside the handler.** Init errors, out of memory, crashes and timeouts through the Lambda Telemetry API. - **OTel libraries without an OTel SDK.** Libraries that use `@opentelemetry/api`, like the Vercel AI SDK, report into the layer directly. If you already run the AWS Distro for OpenTelemetry or your own OpenTelemetry SDK, keep it and send its spans to Ivy over OTLP. If your app registers its own tracer provider, the Ivy layer leaves it alone. ## Standard attributes | Span | Attributes | |---|---| | Invocation | `faas.invocation_id`, `faas.coldstart`, `faas.trigger`, `cloud.resource_id`, `http.request.method`, `url.path`, `http.response.status_code` | | AWS SDK call | `rpc.system=aws-api`, `rpc.service`, `rpc.method`, `aws.request_id`, `aws.dynamodb.table_names`, `messaging.destination.name` | | HTTP call | `http.request.method`, `url.full`, `server.address`, `http.response.status_code` | | LLM call | `gen_ai.*` (OpenTelemetry GenAI) or the Vercel AI SDK's `ai.*` | Ivy-specific data, such as payloads and failure details, lives under `ivy.*`. ## No lock-in The data is OpenTelemetry in ClickHouse, the code is MIT licensed, and the layer only needs two environment variables (`IVY_ENDPOINT`, `IVY_API_KEY`) to point somewhere else. --- Source: https://ivytrace.dev/features/opentelemetry · Updated 2026-11-14 # Self-hosted Lambda tracing Ivy is open source under the MIT license, and the hosted service at ivytrace.dev runs the same images you can run yourself. `docker compose up` starts the OpenTelemetry Collector, ClickHouse, the API, the dashboard and the MCP server. Self-hosted Ivy has no limits and no payment code, and your traces never leave your servers. ## One command ``` git clone https://github.com/ivytrace/ivy && cd ivy docker compose -f deploy/docker-compose.yml up -d --build ``` - Dashboard, API and MCP server on `http://localhost:3000` - OTLP/HTTP ingest on `http://localhost:4318` The full guide, with environment variables, sign-in and AWS setup, is in the [self-hosting docs](https://ivytrace.dev/docs/self-hosting). ## What runs | Service | Role | |---|---| | Ingest router | Checks the ingest key and decides which workspace and project the spans belong to | | OpenTelemetry Collector (contrib) | Batches spans with a persistent on-disk queue and writes them to ClickHouse | | ClickHouse | Stores spans, one row per invocation for fast search, and hourly rollups | | API | The HTTP API, the dashboard and the MCP server | The backend needs no AWS services. It only needs network access to the AWS API if you want it to attach the layer to your functions (auto-trace), and your functions need to reach its ingest endpoint. ## Same features as hosted - The dashboard, MCP server (including MCP Apps views) and HTTP API. - Auto-trace through a cross-account role, or attach the layer yourself. - Workspaces, projects, members and invitations. - [Alerts](https://ivytrace.dev/features/alerts): Slack and the weekly email digest, turned on with environment variables, plus an `on-error-group-created` hook for your own integrations. - Retention is yours to choose (`IVY_RETENTION_DAYS`, 30 days by default). ## Sign-in Ivy uses **OIDC or no auth**, never passwords: - **With OIDC:** register a client with Keycloak, Auth0, Okta, Entra ID, Authentik, Zitadel or another provider that issues JWT access tokens, and set `IVY_OIDC_ISSUER` and `IVY_OIDC_CLIENT_ID`. Coding agents sign in through the same provider. - **Without OIDC:** the dashboard and API run open, for setups behind your own authenticating proxy, and the UI shows a permanent "authentication disabled" banner. Ingest always needs an API key. ## Hooks Hosted-only behaviour, like billing, lives in **hooks**: executables in a directory, git-hooks style. Self-hosted installs can use the same mechanism, for example `on-error-group-created` to open a ticket, or `publish-cfn-template` for one-click CloudFormation links. ## Hosted or self-hosted? | Feature | ivytrace.dev | Self-hosted | |---|---|---| | Price | Free for 100,000 traces a month, then €5 per million | Free (MIT), plus your server | | Operations | None | You run and upgrade it | | Data location | Ivy's hosted service | Your servers | | Sign-in | Email sign-in, ready to use | Your OIDC provider, or none | | Features | Same | Same | You can move between them at any time: point the layer's `IVY_ENDPOINT` and `IVY_API_KEY` somewhere else. --- Source: https://ivytrace.dev/features/self-hosting · Updated 2026-11-14 # How Ivy compares Ivy is a narrow tool on purpose. It traces AWS Lambda functions (Node.js today) with full payloads, catches failures outside the handler, and lets coding agents read production through MCP, for €5 per million traces or free when self-hosted. Broader platforms do more. These pages say where each tool is the better choice. - [Ivy vs Lumigo](https://ivytrace.dev/compare/lumigo): Looking for a Lumigo alternative before the Lumigo platform shuts down on December 31, 2026? Compare Ivy and Lumigo for AWS Lambda tracing: features, pricing as of November 2026, and how to migrate. - [Ivy vs Datadog](https://ivytrace.dev/compare/datadog): Compare Ivy and Datadog Serverless Monitoring for AWS Lambda tracing. Features, list prices as of November 2026 (per active function plus per traced invocation vs €5 per million traces), and when each fits. - [Ivy vs AWS X-Ray](https://ivytrace.dev/compare/aws-x-ray): Compare Ivy with AWS X-Ray and CloudWatch Application Signals for Lambda tracing. Payloads, sampling, failures outside the handler, pricing as of November 2026, and the X-Ray SDK maintenance timeline. - [Ivy vs Sentry](https://ivytrace.dev/compare/sentry): Compare Ivy and Sentry for AWS Lambda. Sentry is error and performance monitoring across your whole stack; Ivy traces Lambda invocations with payloads and AWS-aware spans. Features and pricing as of November 2026. - [Ivy vs Baselime](https://ivytrace.dev/compare/baselime): Baselime joined Cloudflare in 2024. If you used Baselime for AWS Lambda observability, compare it with Ivy, an open-source Lambda tracer with payloads, failures outside the handler and an MCP server. - [Migrate from Lumigo](https://ivytrace.dev/migrate/lumigo): A practical guide to moving AWS Lambda tracing from Lumigo to Ivy before Lumigo shuts down on December 31, 2026. Remove the Lumigo layer and variables, attach Ivy, and map transactions, issues and alerts. ## At a glance As of November 2026, from each vendor's public pages (linked on each comparison). | Feature | Ivy | Lumigo | Datadog Serverless | AWS X-Ray | Sentry | |---|---|---|---|---|---| | Focus | Lambda tracing and errors, for people and agents | Serverless and Kubernetes observability | Full observability platform | AWS-native distributed tracing | Error and performance monitoring | | Pricing unit | Traces (invocations) | Traces, by plan | Active functions plus traced invocations | Traces recorded and scanned | Errors and spans, by plan | | Free tier | 100,000 traces a month | 150,000 traces a month (Basic plan) | Free trial | 100,000 traces recorded a month | 5,000 errors and 5M spans (Developer) | | Open source, self-hostable | Yes (MIT) | No | No | No | Self-hosted version available | | Lambda runtimes | Node.js (others through OTLP) | Node.js, Python, Java, Go | Most Lambda runtimes | Most Lambda runtimes (OpenTelemetry) | Node.js and Python integrations | | Status | Active | Shuts down December 31, 2026 | Active | SDKs in maintenance mode, OpenTelemetry recommended | Active | ## Choosing - **You want Lambda traces with payloads, cheaply, and your coding agent in the loop:** Ivy. - **You need logs, metrics, infrastructure, many runtimes and enterprise alerting in one place:** Datadog, or another full platform. - **You want to stay entirely inside AWS:** X-Ray and CloudWatch Application Signals. - **Your main need is error tracking across frontend and backend:** Sentry. - **You're on Lumigo:** its platform shuts down at the end of 2026. Read [Ivy vs Lumigo](https://ivytrace.dev/compare/lumigo) and the [migration guide](https://ivytrace.dev/migrate/lumigo). --- Source: https://ivytrace.dev/compare · Updated 2026-11-14 # Ivy vs AWS X-Ray AWS X-Ray is the tracing service built into AWS, now instrumented through OpenTelemetry and shown in CloudWatch. Ivy records more per Lambda invocation (the event, the response and every AWS SDK and HTTP body), traces every invocation instead of a sample, and makes it all readable by coding agents. X-Ray stays inside AWS and traces across many AWS services natively. ## Feature comparison As of November 2026, from AWS's public documentation and pricing. | Feature | Ivy | AWS X-Ray (with CloudWatch) | |---|---|---| | Where it runs | ivytrace.dev or self-hosted | Inside your AWS account | | Lambda setup | Opt-in auto-trace attaches a layer; no code changes | Active tracing toggle; OpenTelemetry (ADOT) for detailed spans | | What each invocation records | Event, response, logs, errors, every AWS SDK and HTTP call with bodies | Segments and subsegments with timings; payloads only if you add them as metadata | | Sampling | Every invocation | Sampling rules (by default, the first request each second and 5% of the rest) | | Failures outside the handler | Init errors, out of memory, crashes, timeouts with log tails; throttles | Lambda metrics and CloudWatch Logs | | Cross-service traces | Within one invocation | Across API Gateway, SQS, SNS, Step Functions and other AWS services | | Agents | MCP server, MCP Apps views, HTTP API | AWS APIs and AWS's own tooling | | Open source | Yes (MIT) | No (OpenTelemetry instrumentation is) | **Where X-Ray is stronger:** no third party, no data leaving AWS, IAM-native access control, and traces that follow a request across AWS services. CloudWatch Application Signals adds service maps and SLOs on top. **Where Ivy is stronger:** much more detail per Lambda invocation without writing instrumentation, every invocation kept rather than a sample, failures outside the handler as traces with their last log lines, a dashboard built only for Lambda, and an MCP server for your coding agent. ## X-Ray SDK maintenance AWS put the X-Ray SDKs and daemon into maintenance mode on February 25, 2026, with end of support on February 25, 2027, and recommends OpenTelemetry instead ([AWS announcement](https://aws.amazon.com/blogs/mt/announcing-aws-x-ray-sdks-daemon-end-of-support-and-opentelemetry-migration), [support timeline](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-daemon-timeline.html)). If you are rewriting X-Ray SDK instrumentation anyway, that is a good moment to choose where the traces go. Ivy accepts OpenTelemetry over OTLP, so an OpenTelemetry SDK can send to Ivy instead of, or as well as, X-Ray. ## Pricing | Pricing | Ivy | AWS X-Ray | |---|---|---| | Free each month | 100,000 traces | 100,000 traces recorded; 1,000,000 traces retrieved or scanned | | Beyond that | €5 per million traces | $5.00 per million traces recorded; $0.50 per million retrieved or scanned | | What a trace is | One Lambda invocation, all spans included | One request's trace, as sampled | The list prices per million traces are similar ($5.00 and €5). X-Ray's sampling means you record fewer traces, with less detail in each; Ivy keeps every invocation with its payloads. ## Using both Ivy's layer does not use X-Ray, so you can keep X-Ray active tracing on for cross-service maps and use Ivy for invocation detail. Ivy does not read or forward X-Ray trace headers yet, so the two show separate traces. ## Sources Checked in November 2026: - [Amazon CloudWatch pricing](https://aws.amazon.com/cloudwatch/pricing/) (X-Ray tracing) - [X-Ray SDKs and daemon end of support](https://aws.amazon.com/blogs/mt/announcing-aws-x-ray-sdks-daemon-end-of-support-and-opentelemetry-migration) - [X-Ray SDK and daemon support timeline](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-daemon-timeline.html) - [Migrating from X-Ray instrumentation to OpenTelemetry](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-migration.html) AWS, AWS Lambda and AWS X-Ray are trademarks of Amazon.com, Inc. or its affiliates. Ivy is not affiliated with AWS. --- Source: https://ivytrace.dev/compare/aws-x-ray · Updated 2026-11-14 # Ivy vs Baselime Baselime was a serverless observability tool with strong AWS Lambda support that joined Cloudflare in April 2024. Ivy covers similar ground for AWS Lambda (traces with payloads, errors and alerts) as an open-source project you can use hosted or run yourself. ## Baselime's status Cloudflare [announced the acquisition of Baselime](https://blog.cloudflare.com/cloudflare-acquires-baselime-expands-observability-capabilities) in April 2024. It said existing users could keep using the product with ongoing support, and made Baselime's paid features free. Since then, Cloudflare's observability work has centred on Workers. As of November 2026, [baselime.io](https://baselime.io/) still shows "Baselime has joined Cloudflare" and lists AWS integrations; check there for the current state of its AWS support. ## Feature comparison | Feature | Ivy | Baselime | |---|---|---| | Owner | Open-source project, MIT licensed | Cloudflare | | Focus | AWS Lambda | Serverless, with Cloudflare Workers as Cloudflare's focus | | Lambda instrumentation | Layer attached by opt-in auto-trace rules | OpenTelemetry-based Lambda instrumentation | | Ingest | OTLP over HTTP | OpenTelemetry and other sources | | Payloads | Events, responses, AWS SDK and HTTP bodies | See Baselime's docs | | Failures outside the handler | Init errors, out of memory, crashes, timeouts, throttles | See Baselime's docs | | Alerts | Slack per error group, weekly digest | See Baselime's docs | | Agents | MCP server, MCP Apps views, HTTP API | See Baselime's docs | | Self-hosting | Yes | No | | Price | Free for 100,000 traces a month, then €5 per million | Free (paid features made free in 2024) | **Where Baselime was stronger:** a query language and dashboards over events and logs from many sources, and, today, a price of zero. **Where Ivy is stronger:** an actively developed AWS Lambda product, payloads and failures outside the handler by default, an MCP server for coding agents, and the option to self-host. ## Moving from Baselime to Ivy Remove Baselime's layer or OpenTelemetry configuration from your functions, then connect your AWS account to Ivy and pick functions to trace ([agent setup](https://ivytrace.dev/docs/agent-setup)). If your functions already send OpenTelemetry, you can point the OTLP exporter at Ivy instead ([OpenTelemetry](https://ivytrace.dev/features/opentelemetry)). ## Sources Checked in November 2026: - [Cloudflare blog: Cloudflare acquires Baselime](https://blog.cloudflare.com/cloudflare-acquires-baselime-expands-observability-capabilities) - [baselime.io](https://baselime.io/) Baselime and Cloudflare are trademarks of their owners. Ivy is not affiliated with either. --- Source: https://ivytrace.dev/compare/baselime · Updated 2026-11-14 # Ivy vs Datadog Serverless Ivy is a focused, open-source alternative to Datadog Serverless Monitoring for teams that mainly need AWS Lambda traces with payloads and error groups. Datadog is a full observability platform with logs, metrics, infrastructure, many runtimes and mature alerting. Ivy does less, costs much less per invocation, and can be self-hosted. ## Feature comparison As of November 2026, from Datadog's public pricing and product pages. | Feature | Ivy | Datadog Serverless | |---|---|---| | Scope | Lambda traces, errors and alerts | Full platform: APM, logs, metrics, infrastructure, RUM, security and more | | Lambda runtimes | Node.js (others through OTLP) | Most Lambda runtimes | | Instrumentation | One layer, attached by opt-in auto-trace rules | Datadog Lambda extension and libraries, with several install methods | | Payloads | Events, responses, AWS SDK and HTTP bodies by default, redacted in the function | Available, configurable | | Failures outside the handler | Init errors, out of memory, crashes, timeouts, throttles | Enhanced Lambda metrics and logs | | Alerting | Slack per error group, weekly digest | Monitors on any metric, log or trace, many integrations | | Agents | MCP server across workspaces, MCP Apps views, HTTP API with OpenAPI | MCP server (generally available since March 2026) | | Open source and self-hosting | Yes (MIT), `docker compose up` | No | **Where Datadog is stronger:** breadth. If you need logs, metrics, dashboards, monitors on everything, many runtimes and one vendor for all of it, Datadog does far more than Ivy. **Where Ivy is stronger:** a small, predictable bill for Lambda tracing, payloads on by default, failures outside the handler as first-class traces, an open-source codebase you can run yourself, and setup by your coding agent from one prompt. ## Pricing Datadog bills Lambda on two meters (list prices, November 2026): | Datadog item | Billed annually | Month-to-month | On demand | |---|---|---|---| | Serverless Workload Monitoring, per active function per month | $5 | $6 | $7.20 | | Serverless Functions APM, per 1M traced invocations per month | $10 | $12 | $15 | Ivy bills one meter: €5 per million traces (one trace is one invocation), with the first 100,000 a month free. **Example:** 40 functions and 10 million traced invocations a month. - Datadog, billed annually: 40 × $5 + 10 × $10 = **$300 a month**, before any other Datadog products. - Ivy: (10,000,000 − 100,000) ÷ 1,000,000 × €5 = **€49.50 a month**. Datadog's price includes far more than tracing. Compare against what you would actually use. ## Migrating from Datadog to Ivy 1. Remove the Datadog layers and extension, or the Datadog CloudFormation macro or plugin, from the functions you want to move, and restore their original handlers if the Datadog wrapper replaced them. 2. Connect your AWS account to Ivy and pick functions by stack, name or tag (see [agent setup](https://ivytrace.dev/docs/agent-setup)), or add the Ivy layer in your infrastructure code ([manual setup](https://ivytrace.dev/docs/manual-setup)). 3. Keep Datadog for logs and metrics if you need them; Ivy's layer does not conflict with CloudWatch. ## Sources Checked in November 2026: - [Datadog pricing list](https://www.datadoghq.com/pricing/list/) - [Datadog serverless billing docs](https://docs.datadoghq.com/account_management/billing/serverless/) - [Datadog launches MCP Server](https://www.datadoghq.com/about/latest-news/press-releases/datadog-launches-mcp-server/), March 2026 Datadog is a trademark of its owner. Ivy is not affiliated with Datadog. --- Source: https://ivytrace.dev/compare/datadog · Updated 2026-11-14 # Ivy vs Lumigo Ivy is an open-source (MIT) alternative to Lumigo for tracing AWS Lambda functions. Like Lumigo, it attaches a Lambda layer to the functions you choose, records every invocation with its payloads and AWS calls, and catches failures outside the handler. Unlike Lumigo, it is self-hostable, priced at €5 per million traces, and built for coding agents through an MCP server. Lumigo has announced that its platform shuts down on December 31, 2026. ## Lumigo's status Lumigo was acquired by Dash0, announced on [February 4, 2026](https://dash0.com/blog/dash0-acquires-lumigo-to-expand-agentic-observability-across-aws-and-serverless). As of November 2026, [lumigo.io](https://lumigo.io/) says: *"Lumigo is now part of Dash0. The Lumigo platform shuts down on December 31, 2026."* Dash0's [acquisition FAQ](https://www.dash0.com/lumigo-acquisition-faq) says that all Lumigo customers have to transition to Dash0 by the end of 2026, and that the Lumigo platform will be sunset after they have migrated. So every Lumigo user needs a new home for their Lambda traces in the next few weeks. Lumigo's own recommended path is **Dash0**, which also has a Lambda extension and an MCP server, with usage-based pricing per span. If you want the same team and a broad platform, that is a reasonable choice. Ivy is for teams that want something smaller: Lambda-focused, open source, cheap per invocation, and ready for coding agents. ## Feature comparison As of November 2026, from Lumigo's public website and documentation. | Feature | Ivy | Lumigo | |---|---|---| | Availability | Active, MIT licensed | Platform shuts down December 31, 2026 | | Lambda instrumentation | Lambda layer with an exec wrapper (`/opt/ivy-wrapper`) | Lambda layer with an exec wrapper (`/opt/lumigo_wrapper`) | | Auto-trace | Opt-in rules by CloudFormation stack, name pattern or tag; `ivy:trace` tag | Select functions in the UI, or the `lumigo:auto-trace` tag | | Lambda runtimes | Node.js (Python next; others via OTLP) | Node.js, Python, Java, Go | | Payloads | Events, responses, AWS SDK and HTTP bodies, redacted in the function | Yes | | Failures outside the handler | Init errors, out of memory, crashes, timeouts (Telemetry API), throttles | See Lumigo's docs | | Alerts | Slack per error group, weekly email digest | Broader alerting | | Logs and metrics | Invocation logs and failure log tails only | Log management and metrics products | | Kubernetes and containers | No | Yes | | AI assistant | Your own agent, through an MCP server, MCP Apps views and an HTTP API | Lumigo Copilot | | LLM spans | Vercel AI SDK, OpenTelemetry GenAI | AI agent observability | | Self-hosting | Yes, `docker compose up` | No | | Open source | Yes, MIT | No | Where Lumigo is stronger: more runtimes, Kubernetes, log management and metrics, more mature alerting, and years of production use. Where Ivy is stronger: open source and self-hostable, a lower price per invocation, an agent-first workflow, and a future past December 2026. ## Pricing | Pricing | Ivy | Lumigo (until December 31, 2026) | |---|---|---| | Free | 100,000 traces a month per workspace | Basic: 150,000 traces, 14-day trace retention | | Paid | €5 per million traces beyond the free tier | Standard: $119 a month ($99 billed annually) for 1M traces; Plus: $359 a month ($299 annually) for 5M traces; Custom above that | | 5 million traces a month | €24.50 | $299–$359 a month (Plus) | | Self-hosted | Free | Not available | Dash0, Lumigo's successor, prices per span and log record rather than per trace (see [Dash0 pricing](https://www.dash0.com/pricing)). ## Migrating Moving from Lumigo to Ivy takes a few minutes per AWS account: untrace the functions in Lumigo (or remove its layer and variables), connect the account to Ivy, and choose what to trace. Your coding agent can do most of it. The [migration guide](https://ivytrace.dev/migrate/lumigo) has the steps and a concept map: | Lumigo | Ivy | |---|---| | Transactions | Traces (one per Lambda invocation) | | Issues | Error groups | | Alerts | Slack alerts per error group and the weekly digest | | Projects | Workspaces and projects | | Auto-trace | Auto-trace rules | | Lumigo Copilot | Your coding agent with Ivy's MCP server | ## Sources Checked in November 2026: - [lumigo.io](https://lumigo.io/): site banner on the shutdown date - [Lumigo pricing](https://lumigo.io/pricing/) - [Lumigo docs: Lambda layers](https://docs.lumigo.io/docs/lambda-layers) and [auto-tracing](https://docs.lumigo.io/docs/auto-instrumentation) - [Dash0: Lumigo acquisition FAQ](https://www.dash0.com/lumigo-acquisition-faq) - [Dash0 blog: Dash0 acquires Lumigo](https://dash0.com/blog/dash0-acquires-lumigo-to-expand-agentic-observability-across-aws-and-serverless), February 4, 2026 - [Dash0 pricing](https://www.dash0.com/pricing) Lumigo and Dash0 are trademarks of their owners. Ivy is not affiliated with either. --- Source: https://ivytrace.dev/compare/lumigo · Updated 2026-11-14 # Ivy vs Sentry for AWS Lambda Sentry is error and performance monitoring for your whole stack, frontend and backend, with a Lambda integration. Ivy is narrower: it traces every AWS Lambda invocation with its payloads and AWS calls, and catches failures outside the handler. Many teams would use both, Sentry for application errors and Ivy for what happened inside each Lambda invocation. ## Feature comparison As of November 2026, from Sentry's public pricing and documentation. | Feature | Ivy | Sentry | |---|---|---| | Scope | AWS Lambda traces, errors and alerts | Errors, tracing, session replay, profiling, logs and more, across many platforms | | Lambda setup | Opt-in auto-trace attaches a layer; no code changes | Sentry's AWS Lambda integration (SDK or Lambda layer), configured per function | | Unit of data | One trace per invocation, every invocation | Error events and sampled spans | | Payloads | Events, responses and AWS SDK and HTTP bodies by default | Request context on errors; configurable | | AWS-aware spans | DynamoDB, S3, SQS, SNS, EventBridge, Lambda calls with service details | Depends on SDK instrumentation | | Failures outside the handler | Init errors, out of memory, crashes, timeouts with log tails; throttles | Timeout warnings in the SDK; see Sentry's docs | | Error workflow | Error groups, Slack alerts, weekly digest | Issues with assignment, releases, ownership rules, many alert integrations | | Agents | MCP server, MCP Apps views, HTTP API | See Sentry's docs | | Self-hosting | Yes, MIT licensed | Self-hosted Sentry is available | **Where Sentry is stronger:** errors across your whole product, not just Lambda; a mature issue workflow with releases, assignment and many integrations; frontend monitoring and session replay. **Where Ivy is stronger:** everything an invocation did, with payloads, without sampling; failures that never reach your code; AWS calls shown as AWS calls; setup without touching each function; and a price per invocation. ## Pricing | Pricing | Ivy | Sentry | |---|---|---| | Free | 100,000 traces a month | Developer: 5,000 errors and 5M spans a month | | Paid | €5 per million traces | Team from $26 a month (annual) or $29 (monthly); Business from $80 (annual) or $89 (monthly); usage beyond the plan billed extra | ## Using both Ivy's layer and Sentry's SDK can run in the same function. Sentry reports exceptions to its issue workflow; Ivy records the full invocation. If you only want one tool for Lambda errors, Ivy's error groups and Slack alerts cover the basics. ## Sources Checked in November 2026: - [Sentry pricing](https://sentry.io/pricing/) - [Sentry docs: AWS Lambda](https://docs.sentry.io/platforms/javascript/guides/aws-lambda/) Sentry is a trademark of Functional Software, Inc. Ivy is not affiliated with Sentry. --- Source: https://ivytrace.dev/compare/sentry · Updated 2026-11-14 # Migrating from Lumigo to Ivy Moving Lambda tracing from Lumigo to Ivy takes a few minutes per AWS account. You stop Lumigo's tracing (its layer, its wrapper and its token variable), connect the account to Ivy, and pick which functions to trace. Your coding agent can do most of it. Lumigo has announced that its platform shuts down on December 31, 2026, so plan the switch before then. ## Before you start - **Check what Ivy supports.** Ivy's layer instruments **Node.js** functions today. If you also trace Python, Java or Go functions with Lumigo, those need another tool for now, or an OpenTelemetry SDK that sends OTLP to Ivy. - **Find out how Lumigo was attached.** Either Lumigo's **auto-trace** added its layer (from the Lumigo UI or the `lumigo:auto-trace` tag), or your **infrastructure code** adds it (CDK, Serverless Framework, SAM, Terraform and so on). The steps differ slightly. - **Export what you want to keep.** Ivy does not import Lumigo's historical data. Note your alert rules and any dashboards you rely on. ## Step 1: Stop Lumigo's tracing Ivy and Lumigo both use `AWS_LAMBDA_EXEC_WRAPPER`, and a function can only have one, so remove Lumigo's before adding Ivy's. **If Lumigo's auto-trace attached it:** untrace the functions in the Lumigo UI, or set the tag `lumigo:auto-trace` to `false` on them (Lumigo's docs say changes take 15 to 20 minutes). Then check one function's configuration to confirm the Lumigo layer and variables are gone. **If your infrastructure code adds it,** remove from each function: | Remove | What it is | |---|---| | The `lumigo-node-tracer` layer | Lumigo's Node.js tracer layer (its ARN names the region and version) | | `AWS_LAMBDA_EXEC_WRAPPER=/opt/lumigo_wrapper` | Lumigo's exec wrapper | | `LUMIGO_TRACER_TOKEN` | Your Lumigo token | | `LUMIGO_ORIGINAL_HANDLER` | Only if you used handler redirection: set the function's handler back to this value | | Any other `LUMIGO_*` variables | Optional tracer settings | | `@lumigo/tracer` imports and `lumigo.trace(...)` wrappers | Only if you wrapped handlers in code | These names come from [Lumigo's layer documentation](https://docs.lumigo.io/docs/lambda-layers); check them against your own configuration. Lumigo's CDK constructs and Serverless plugin add the same pieces, so removing the construct or plugin is usually enough. ## Step 2: Connect Ivy **The quick way:** paste this into your coding agent in the repository that deploys your functions. ``` Set up Ivy for my AWS Lambda functions using https://ivytrace.dev/llms.txt ``` The agent signs you up (you approve in your browser), creates a workspace and project, and gives you a CloudFormation quick-create link that creates Ivy's role in your AWS account. Then it lists your functions grouped by CloudFormation stack and asks which to trace. If it finds infrastructure code that sets layers or environment variables, it offers to add Ivy there instead, so the next deploy doesn't remove it. You can also tell it: *"We're moving off Lumigo: remove the Lumigo layer and variables first."* **By hand:** connect the account in the dashboard (Workspace settings → AWS accounts), or add the layer yourself: | Add | Value | |---|---| | Layer | `ivy-trace-nodejs`, published in your own account by Ivy | | `AWS_LAMBDA_EXEC_WRAPPER` | `/opt/ivy-wrapper` | | `IVY_ENDPOINT` | Your ingest endpoint | | `IVY_API_KEY` | A project ingest key | Details in [manual setup](https://ivytrace.dev/docs/manual-setup). ## Step 3: Choose what to trace Ivy's auto-trace is **opt-in**, like a Lumigo auto-trace selection, but saved as rules: - a whole **CloudFormation stack** (`{"stack": "checkout-prod"}`), - a **name pattern** (`{"name": "checkout-*"}`), - or a **tag** (`{"tag": "team=core"}`). New functions that match a rule are traced automatically. The tag `ivy:trace=true` or `false` on a function overrides rules, much like `lumigo:auto-trace`. ## Step 4: Verify Invoke one function, or wait for traffic, and open **Invocations** in Ivy. The trace should show the event, the response and the AWS calls. Ask your agent to *"find failing invocations in the last hour"* to check the MCP connection. ## Step 5: Set up alerts Connect Slack (**Add to Slack**, then pick a channel) to get a message when an error group is first seen. The weekly email digest is on for workspace members by default. See [setting up alerts](https://ivytrace.dev/docs/alerts). ## Concept map | Lumigo | Ivy | |---|---| | Transaction | Trace: one Lambda invocation with all its spans | | Invocation | Invocation (the trace's root span) | | Issue | Error group, fingerprinted by function, error type and message | | Alert | Slack alert per error group, plus the weekly email digest | | Project | Workspace (team, billing, AWS accounts) and project (one app or environment) | | Auto-trace and `lumigo:auto-trace` tag | Auto-trace rules and the `ivy:trace` tag | | `LUMIGO_TRACER_TOKEN` | `IVY_API_KEY` (a project ingest key) | | Lumigo Copilot | Your coding agent with Ivy's MCP server | | Timeouts and out of memory | Failures outside the handler: init errors, OOM, crashes, timeouts, throttles | ## What's different - **Alerting is simpler.** Ivy alerts on error groups in Slack and sends a weekly digest. There are no metric alerts, custom rules beyond the thresholds, or PagerDuty integration. - **No log management.** Ivy keeps each invocation's `console.*` lines and the last log lines of failures, not all of your logs. - **Node.js only** for the layer, today. - **Payload redaction runs in the function.** Ivy redacts keys like `authorization`, `password` and `token` by default. Add your own with `IVY_REDACT_KEYS`. - **Pricing.** €5 per million traces after 100,000 free a month per workspace, or self-host for free. See [pricing](https://ivytrace.dev/pricing). ## Frequently asked questions ### Can I run Lumigo and Ivy side by side? Not on the same function: both rely on `AWS_LAMBDA_EXEC_WRAPPER`, and only one wrapper can be set. You can move functions over stack by stack, keeping Lumigo on the rest until you're done. ### Will removing Lumigo's layer change my function's behaviour? It shouldn't. If you used Lumigo's handler redirection, set the handler back to the value of `LUMIGO_ORIGINAL_HANDLER` before removing that variable, or the function will fail to start (Ivy would show it as an init error). ### Can Ivy import my Lumigo history? No. Ivy starts from the first trace after you switch. Keep exports of anything you need from Lumigo before December 31, 2026. ### Is Dash0 the better choice for Lumigo users? It depends on what you need. Dash0 is Lumigo's official successor and a broad observability platform. Ivy is smaller, focused on Lambda, open source, and priced per invocation. See [Ivy vs Lumigo](https://ivytrace.dev/compare/lumigo). --- Source: https://ivytrace.dev/migrate/lumigo · Updated 2026-11-14 # About Ivy Ivy is an open-source (MIT) observability tool for AWS Lambda. It exists because debugging a Lambda function should mean reading what the invocation actually did, and because the one reading it is increasingly a coding agent. ## Why Ivy Serverless teams often end up with one of two things: a broad platform that is expensive and complex for a handful of Lambda functions, or CloudWatch logs and guesswork. Ivy sits in between. It records every invocation with its payloads and AWS calls, shows failures that never reach your code, and keeps the price per invocation low. Coding agents changed what an observability tool is for. Your agent already has the code; what it lacks is production. Ivy gives it the failing request, the payload that caused it and the stack trace, through an MCP server and an HTTP API, and lets it set Ivy up from a single prompt. ## Principles - **Open source.** The layer, the API, the dashboard and the MCP server are MIT licensed at [github.com/ivytrace/ivy](https://github.com/ivytrace/ivy). The hosted service runs the same images. - **Small and focused.** Lambda first, Node.js first, done well, rather than everything at once. - **Cheap to run.** A healthy invocation is one trace and nothing more. Failures add data only when they happen. - **Standards.** OpenTelemetry in, OpenTelemetry semantic conventions, OIDC for sign-in. - **Agent-native.** No CLI, no click-only flows: everything is available through MCP or the HTTP API. - **Honest about gaps.** The [features page](https://ivytrace.dev/features) lists what Ivy doesn't do yet. ## Hosted and self-hosted [ivytrace.dev](https://ivytrace.dev/) is the hosted version: free for 100,000 traces a month per workspace, then €5 per million. Or run it yourself with `docker compose up` ([self-hosting](https://ivytrace.dev/features/self-hosting)). ## Get in touch Questions, bugs and ideas go to [GitHub issues](https://github.com/ivytrace/ivy/issues). --- Source: https://ivytrace.dev/about · Updated 2026-11-14