# Ivy

> Ivy is open-source (MIT) tracing and error monitoring for AWS Lambda (Node.js). A Lambda layer records every invocation (event, response, logs, errors, cold starts) and every AWS SDK and HTTP call with payloads, and sends the spans after the function has responded. It also reports failures outside the handler (init errors, out of memory, crashes, timeouts) and throttles, and alerts in Slack per error group. Coding agents read traces through a remote MCP server and an HTTP API. Hosted at https://ivytrace.dev: free for 100,000 traces a month per workspace, then €5 per million traces (one trace is one Lambda invocation). Self-hosting is free.

Key facts:

- No measurable delay to warm responses; about 20–45 ms added to cold starts.
- Runtimes: Node.js on Lambda today (Python is next). Other runtimes can send OpenTelemetry (OTLP over HTTP).
- Tracing is opt-in: connecting an AWS account traces nothing until the user picks functions by CloudFormation stack, name pattern or tag.
- MCP server: `https://ivytrace.dev/mcp` (OAuth). HTTP API: OpenAPI at `https://ivytrace.dev/api/openapi.json`.
- Every page below is also available as HTML at the same URL without `.md`. The full text of all pages is at https://ivytrace.dev/llms-full.txt.

**If a user asked you to "set up Ivy", follow "Agent setup" at the end of this file, in order.** Tell the user briefly what you are about to do before each step that needs them.

## Start here

- [Simple Lambda observability, for the AI age](https://ivytrace.dev/index.md): Open-source tracing for AWS Lambda. See every invocation with its payloads, errors and AWS calls, get Slack alerts, and let your coding agent debug production through MCP. Free for 100,000 traces a month.
- [Ivy docs](https://ivytrace.dev/docs.md): Get started with Ivy: set it up with your coding agent in one prompt, attach the Lambda layer by hand, connect Slack alerts, add the MCP server to Claude Code, Claude or Cursor, use the HTTP API, or self-host.
- [Set up Ivy with your coding agent](https://ivytrace.dev/docs/agent-setup.md): Paste one prompt and your coding agent sets up Ivy: OAuth device-flow sign-in, workspace and project, a CloudFormation role in your AWS account, auto-trace rules with a dry run, and a verified first trace.
- [Pricing](https://ivytrace.dev/pricing.md): Ivy is free for 100,000 traces a month per workspace, then €5 per million traces. One trace is one Lambda invocation. Self-hosting is free and MIT licensed.

## Docs

- [Set up alerts](https://ivytrace.dev/docs/alerts.md): Connect Slack to get a message when a Lambda error group is first seen, tune the thresholds per project, and manage the weekly email digest. Self-hosted setup included.
- [HTTP API](https://ivytrace.dev/docs/api.md): Ivy's HTTP API covers everything the dashboard does, described by an OpenAPI 3.1 spec. Authentication, workspace and project routes, and traces as compact text for LLMs.
- [Manual layer setup](https://ivytrace.dev/docs/manual-setup.md): Attach Ivy's Node.js Lambda layer yourself or in infrastructure code. The layer, the four environment variables, all optional settings, and how to get an ingest key.
- [MCP server](https://ivytrace.dev/docs/mcp.md): Add Ivy's remote MCP server to Claude Code, Claude, Cursor or VS Code. OAuth sign-in, read-only tools across all your workspaces, and interactive trace views with MCP Apps.
- [Self-hosting quickstart](https://ivytrace.dev/docs/self-hosting.md): Run Ivy on your own server with docker compose. Services and ports, the environment variables that matter, OIDC sign-in, ingest keys, auto-trace and alerts.
- [Span conventions](https://ivytrace.dev/docs/span-conventions.md): The OpenTelemetry attributes Ivy's Lambda layer emits for invocations, AWS SDK and HTTP calls, logs, failures outside the handler and LLM spans, plus payload capping and redaction defaults.

## Features

- [Features](https://ivytrace.dev/features.md): What Ivy does for AWS Lambda: tracing with payloads, a no-code layer, failures outside the handler, Slack alerts, an MCP server for agents, LLM spans, OTLP ingest and self-hosting.
- [Lambda observability for AI coding agents](https://ivytrace.dev/features/ai-agents.md): Ivy gives coding agents production context for AWS Lambda: a remote MCP server across all your workspaces, interactive trace views (MCP Apps), an HTTP API with OpenAPI, llms.txt onboarding and an agent skill.
- [Lambda error alerts in Slack, and a weekly digest](https://ivytrace.dev/features/alerts.md): Ivy sends a Slack message when a Lambda error group is first seen, then at most every 60 minutes or 100 occurrences, and a weekly email digest per workspace. Covers init errors, OOM, crashes, timeouts and throttles.
- [Lambda auto-instrumentation](https://ivytrace.dev/features/auto-instrumentation.md): Ivy's Lambda layer traces Node.js functions without code changes. Connect your AWS account once and pick functions by CloudFormation stack, name or tag; new matches are traced automatically.
- [Lambda errors, including the ones your handler never sees](https://ivytrace.dev/features/lambda-errors.md): Ivy catches AWS Lambda failures outside the handler (init errors, out-of-memory kills, crashes and timeouts) with their last log lines, counts throttles, and groups everything into error groups.
- [AWS Lambda tracing with payloads](https://ivytrace.dev/features/lambda-tracing.md): Trace every AWS Lambda invocation with its event, response, logs, errors and cold start, plus every DynamoDB, S3, SQS, SNS, EventBridge and HTTP call with request and response bodies.
- [LLM observability for AI apps on Lambda](https://ivytrace.dev/features/llm-observability.md): Trace Vercel AI SDK and OpenTelemetry GenAI calls in AWS Lambda with Ivy. See prompts, responses, tool calls, token counts and time to first token in the same trace as your DynamoDB and HTTP calls.
- [OpenTelemetry for AWS Lambda](https://ivytrace.dev/features/opentelemetry.md): Ivy ingests standard OpenTelemetry (OTLP over HTTP) and uses OpenTelemetry semantic conventions, so any OTel SDK can send traces, and the Ivy layer is an optional, lighter producer for Node.js Lambda.
- [Self-hosted Lambda tracing](https://ivytrace.dev/features/self-hosting.md): Run Ivy yourself with docker compose up. The OpenTelemetry Collector, ClickHouse, the API, dashboard and MCP server, MIT licensed, with OIDC sign-in and no limits.

## Comparisons and migration

- [How Ivy compares](https://ivytrace.dev/compare.md): Honest comparisons of Ivy with Lumigo, Datadog Serverless, AWS X-Ray, Sentry and Baselime for AWS Lambda: features, pricing as of November 2026, and where each tool is the better choice.
- [Ivy vs AWS X-Ray](https://ivytrace.dev/compare/aws-x-ray.md): Compare Ivy with AWS X-Ray and CloudWatch Application Signals for Lambda tracing. Payloads, sampling, failures outside the handler, pricing as of November 2026, and the X-Ray SDK maintenance timeline.
- [Ivy vs Baselime](https://ivytrace.dev/compare/baselime.md): Baselime joined Cloudflare in 2024. If you used Baselime for AWS Lambda observability, compare it with Ivy, an open-source Lambda tracer with payloads, failures outside the handler and an MCP server.
- [Ivy vs Datadog Serverless](https://ivytrace.dev/compare/datadog.md): Compare Ivy and Datadog Serverless Monitoring for AWS Lambda tracing. Features, list prices as of November 2026 (per active function plus per traced invocation vs €5 per million traces), and when each fits.
- [Ivy vs Lumigo](https://ivytrace.dev/compare/lumigo.md): Looking for a Lumigo alternative before the Lumigo platform shuts down on December 31, 2026? Compare Ivy and Lumigo for AWS Lambda tracing: features, pricing as of November 2026, and how to migrate.
- [Ivy vs Sentry for AWS Lambda](https://ivytrace.dev/compare/sentry.md): Compare Ivy and Sentry for AWS Lambda. Sentry is error and performance monitoring across your whole stack; Ivy traces Lambda invocations with payloads and AWS-aware spans. Features and pricing as of November 2026.
- [Migrating from Lumigo to Ivy](https://ivytrace.dev/migrate/lumigo.md): A practical guide to moving AWS Lambda tracing from Lumigo to Ivy before Lumigo shuts down on December 31, 2026. Remove the Lumigo layer and variables, attach Ivy, and map transactions, issues and alerts.

## Optional

- [About Ivy](https://ivytrace.dev/about.md): Ivy is an open-source (MIT) observability tool for AWS Lambda, built for developers and their coding agents. Why it exists, how it is built and how it is run.

## Agent setup

### Rules

- Never ask the user for passwords, AWS access keys or secret keys. Sign-in and AWS access are both approved by the user in their browser.
- Send the Ivy access token only to `https://ivytrace.dev/api/...`. Never send it anywhere else, and never print it in full.
- Do not invoke Lambda functions or change infrastructure code without asking the user first.
- If a step fails, show the user the error and stop. Do not work around it.

### 1. Sign the user in (Includable Auth, OAuth device flow)

Issuer: `https://auth.includable.com`. Endpoints are also listed at `https://auth.includable.com/.well-known/openid-configuration`.

1. Register this agent once:

   ```
   POST https://auth.includable.com/reg
   Content-Type: application/json

   {"client_name": "<your agent name, e.g. Claude Code>", "parent_client_id": "ivy"}
   ```

   Keep the returned `client_id` (it starts with `agt_ivy_`). There is no client secret. A registration that the user has not approved expires after 1 hour.

2. Ask the user which email address they want to use for Ivy. Then start a device authorization, passing that address (URL-encoded) as `login_hint`:

   ```
   POST https://auth.includable.com/device/auth
   Content-Type: application/x-www-form-urlencoded

   client_id=<client_id>&scope=openid email offline_access&login_hint=<email>
   ```

   Keep the `device_code`.

3. Show the user the `verification_uri_complete` link and the `user_code`. Tell them the page first shows the code (it should match), then "Continue" leads to sign-in with their email filled in, and that if they don't have an Ivy account yet they can create one right there. Ask them to open the link, check that the code matches, sign in or create their account, and click "Allow access". The code expires after 10 minutes. Open the link in their browser for them if you can.

4. Poll for the token every 5 seconds:

   ```
   POST https://auth.includable.com/token
   Content-Type: application/x-www-form-urlencoded

   grant_type=urn:ietf:params:oauth:grant-type:device_code&device_code=<device_code>&client_id=<client_id>
   ```

   - `authorization_pending`: keep polling.
   - `access_denied`: stop and tell the user.
   - `expired_token`: go back to step 2.
   - Success returns an `access_token` (valid for 1 hour) and a `refresh_token`.

5. Call the Ivy API with `Authorization: Bearer <access_token>`. The token's audience is `https://ivytrace.dev/api`.

6. When the access token expires, refresh it:

   ```
   POST https://auth.includable.com/token
   Content-Type: application/x-www-form-urlencoded

   grant_type=refresh_token&refresh_token=<refresh_token>&client_id=<client_id>
   ```

   The refresh token rotates: store the new one every time. On `invalid_grant`, start again at step 1. Access lasts at most 30 days after the user approved it.

Alternative to steps 2 to 4: if you can listen on a local port, register with `"redirect_uris": ["http://127.0.0.1/callback"]` and use the authorization code flow with PKCE (S256).

To sign out, revoke the refresh token:

```
POST https://auth.includable.com/token/revocation
Content-Type: application/x-www-form-urlencoded

token=<refresh_token>&client_id=<client_id>
```

### 2. Pick the workspace

A workspace holds the user's AWS accounts, projects and team members.

1. `GET https://ivytrace.dev/api/workspaces` returns `{"items": [{"id", "slug", "name", "role", "default", "created_at"}]}`. A new user gets a personal workspace on this first call.
2. One workspace: use it. Several: ask the user which one. None: create one with `POST https://ivytrace.dev/api/workspaces` and `{"name": "<team or company name>"}`.
3. Keep its `slug`. Every route below starts with `https://ivytrace.dev/api/w/<workspace>`.

### 3. Pick or create a project

Every trace belongs to one project. Name projects after the repo, app or stack, with environments as separate projects (for example `checkout-prod` and `checkout-staging`).

1. `GET /api/w/<workspace>/projects` lists them: `{"items": [{"id", "slug", "name", "archived", "default", "created_at"}]}`.
2. If one matches this repo, confirm it with the user. Otherwise propose a name and create it: `POST /api/w/<workspace>/projects` with `{"name": "Checkout (prod)"}`. Ivy makes the slug from the name (`checkout-prod`) and returns it.

### 4. Connect the AWS account

Ivy attaches its layer through a cross-account IAM role. The user creates that role with a CloudFormation stack in their own AWS console. You never handle AWS credentials.

1. `GET /api/w/<workspace>/aws/setup`. If its `issues` list is not empty, show it to the user and stop. Note `quick_create` (whether Ivy can give a one-click CloudFormation link).
2. Ask the user for the 12-digit AWS account ID, the regions their functions run in, and a name for the connection (for example "Production").
3. `POST /api/w/<workspace>/aws/accounts` with `{"account_id": "123456789012", "name": "Production", "regions": ["eu-west-1"], "stack_region": "eu-west-1", "default_project": "<project slug>"}`. The response (status `pending`) has the connection `id`, a `quick_create_url` and a `template_download_url`.
4. Show the user the `quick_create_url`; it is short-lived, so if it has expired, `GET /api/w/<workspace>/aws/accounts/<id>` gives a fresh one. (If it is `null`, or `quick_create` was false, download the template from `template_download_url` with your token and ask the user to upload it in the CloudFormation console instead.) Ask them to open it, sign in to that AWS account, tick the IAM acknowledgement and click "Create stack", then tell you when the stack shows `CREATE_COMPLETE` (usually 1 to 2 minutes). It creates the IAM role and an EventBridge rule that reports function changes to Ivy. Connecting traces nothing by itself.
5. `POST /api/w/<workspace>/aws/accounts/<id>/verify`. On `{"verified": true}` the account is `connected` and Ivy lists its functions. On `false`, show the user `account.error`, wait a minute and try again.

### 5. Choose which functions to trace

Tracing is opt-in. Rules select functions by CloudFormation stack, name pattern, tag, account or region, and name the project their traces go to. New functions that match a rule are traced automatically. Tags on a function override rules: `ivy:trace=true` / `ivy:trace=false`, and `ivy:project=<slug>` picks the project.

1. `GET /api/w/<workspace>/aws/accounts/<id>/functions` returns the functions grouped by stack: `{"groups": [{"stack", "region", "counts", "rule", "suggested_project", "functions": [...]}]}`. Show the user the stacks with their function counts. Functions without a stack come last (`stack: null`).
2. Ask which stacks or name patterns to trace, and into which project. Matchers: `{"stack": "checkout-prod"}`, `{"name": "checkout-*"}`, `{"tag": "team=core"}`, optionally with `account_id` and `region`. `*` and `?` are globs.
3. Preview: `POST /api/w/<workspace>/aws/rules/dry-run` with `{"matcher": {...}, "project": "<project slug>"}`. Show the user the `summary` (`trace`, `unchanged`, `excluded`, `other`, `unsupported`) and the matching function names from `items`.
4. When the user confirms, create the rule: `POST /api/w/<workspace>/aws/rules` with the same `{"matcher": {...}, "project": "<project slug>"}` (or `"new_project": {"name": "..."}` to create the project at the same time). Ivy attaches the layer within seconds.
5. Single functions: `PUT /api/w/<workspace>/aws/functions/<account_id>/<region>/<name>` with `{"mode": "trace", "project": "<project slug>"}`, `{"mode": "exclude"}`, or `{"mode": "rules"}` to follow the rules again.
6. Check the user's repository for infrastructure as code (CDK, SST, Serverless Framework, SAM, Terraform, Pulumi). If their deploys set the functions' layers or environment, the next deploy can remove Ivy's layer. In that case offer to add it to their infrastructure code instead:
   - Layer: `ivy-trace-nodejs`, published in their own account. Copy the exact layer version ARN from a traced function's configuration.
   - `AWS_LAMBDA_EXEC_WRAPPER=/opt/ivy-wrapper`
   - `IVY_ENDPOINT`: the `ingest_endpoint` from `GET /api/w/<workspace>/aws/setup`.
   - `IVY_API_KEY`: a project ingest key from `POST /api/w/<workspace>/p/<project>/keys` with `{"name": "iac"}` (the `key` is shown only in that response). Store it the way the project stores other secrets, never in plain code.

### 6. Verify the first trace

1. Ask the user which function you may invoke, or ask them to trigger one (for example by loading a page that calls it). Prefer a read-only function.
2. Wait 10 seconds, then call `GET /api/w/<workspace>/p/<project>/invocations?function=<name>&since=15m`. Repeat a few times for up to a minute.
3. When an invocation shows up, read it with `GET /api/w/<workspace>/p/<project>/traces/<trace_id>?format=text` (a compact text tree). Give the user the link `https://ivytrace.dev/w/<workspace>/<project>/traces/<trace_id>` and a one-line summary (status, duration, number of calls).
4. If nothing arrives after a minute, check `GET /api/w/<workspace>/aws/functions?state=traced` and the function's configuration (Ivy layer, `AWS_LAMBDA_EXEC_WRAPPER`), and tell the user what you found.

### 7. Offer the skill and the MCP server

After the first trace is verified, offer the user these three things. Do each only if they say yes, and commit nothing without asking.

1. **The Ivy skill.** Download `https://ivytrace.dev/skills/ivy/SKILL.md` into the repo at `.claude/skills/ivy/SKILL.md`, and fill in its "Repo context" section: the workspace and project slugs for this repo, and its main function names.
2. **A pointer for other agents.** Add two lines to `AGENTS.md`, for example:
   ```
   ## Production debugging
   Use Ivy for Lambda errors, slowness and deploy checks: see .claude/skills/ivy/SKILL.md.
   ```
   Create `AGENTS.md` only if the user agrees to a new file.
3. **The MCP server.** Add it to the repo's `.mcp.json`, merging with what is there and never overwriting other servers:
   ```json
   {"mcpServers": {"ivy": {"type": "http", "url": "https://ivytrace.dev/mcp"}}}
   ```
   It signs in with the same Ivy account (OAuth). Tools: `list_workspaces`, `find_failing_invocations`, `search_invocations`, `list_error_groups`, `get_trace_summary`, `compare_invocations`, `render_waterfall`, `render_graph`; they take optional `workspace` and `project` (slugs).

Finally, tell the user they can get Slack alerts for new error groups: in the dashboard, the project's alert settings have an "Add to Slack" button (see https://ivytrace.dev/docs/alerts.md). Workspace members also get a weekly email digest, which they can turn off.

The HTTP API is described at `https://ivytrace.dev/api/openapi.json` (reference: `https://ivytrace.dev/api/docs`).

### Self-hosted Ivy

If the user runs Ivy themselves (`docker compose -f deploy/docker-compose.yml up` from https://github.com/ivytrace/ivy), replace `https://ivytrace.dev` with their URL. Self-hosted instances sign in with the user's own OIDC provider, or run without auth. Ingest always needs the key in `IVY_INGEST_API_KEY`.

### Reference

- [Source code](https://github.com/ivytrace/ivy): layer, API, dashboard, deploy files
- [Lambda layer](https://github.com/ivytrace/ivy/blob/main/packages/layer/README.md): environment variables, manual attach with the AWS CLI
- [API reference](https://ivytrace.dev/api/docs) and [OpenAPI spec](https://ivytrace.dev/api/openapi.json)
- [Span conventions](https://github.com/ivytrace/ivy/blob/main/docs/implementation/span-conventions.md): attributes the layer emits, redaction defaults