NeMo Compass

The goal of NeMo Compass is to efficiently find signals in agent traces that lead to meaningful improvements in quality, cost, safety, privacy, and/or user satisfaction.

A critical input to improving any agent is understanding what it’s doing, especially once you turn it on in production. Evals often capture only a small subset of the tasks an agent actually needs to be good at. Agent developers have discovered that production trace analysis is an essential part of improving an agent, but today there's no standard for how this type of analysis gets validated and measured.

NeMo Compass is an applied research project with a goal of improving the tools and techniques available to developers for trace analysis at scale. It uses a range of techniques to identify different types of issues in a large set of traces, then summarizes findings as actionable insights. We call each of those discovery techniques "Compass Signals."

Learn more about each current Compass Signal: Architecture

We are sharing the implementation at this stage to encourage feedback and testing by others as we continue to improve on it.

Start here

Install uv and Git, then choose a guide:

The distribution is named nemo-compass; the command is compass. If compass is not found after installation, run uv tool update-shell and restart your terminal.

After your first run

• Read your results — understand insights, skipped Compass Signals, and saved output.
• Compass Signals — supply business rules, configure sentiment, or check findings against code.
• Data and model access — configure your inference endpoint and understand where data goes.
• Configuration reference — overrides, limits, and repeatable runs.

Development · Releasing · Architecture

Research Topics

Questions we are exploring

• How can developers efficiently find recurring, fixable agent problems in trace datasets containing hundreds of thousands or millions of executions?
• How can developers estimate how often a problem occurs and how much it matters without asking an LLM to inspect every trace?
• How can trace evidence, evaluation results, and read-only checks against agent code help distinguish real problems from misleading patterns?
• How should we measure whether an insight is correct, useful, and worth a developer's time to investigate?

This repository explores these questions. The architecture guide explains what each part does and where judgment or coverage limits remain.

Help shape the direction

We are especially interested in feedback from teams building agent observability or evaluation tools, and from developers investigating their own agents:

• Which types of recurring issues are hardest to identify today?
• How do you decide whether an identified issue matters to your agent?
• What would make a finding more "actionable": examples, frequency, estimated impact, something else?
• How would you ideally use these tools in your own workflows? What would you need to integrate it?
• What types of additional insights or findings would be most valuable to you?

If an NVIDIA contact shared this preview with you, please send your feedback through that contact and ask them to route it to the NeMo Compass research team. Otherwise, use the NVIDIA Developer contact form with NeMo Compass research preview feedback in the subject. Please avoid sending raw traces, prompts, credentials, or user data.

Contributing

This project is currently not accepting code contributions, pull requests, or GitHub issues from external contributors. Please do not submit external pull requests or open issues, including bug reports and feature requests.

For research feedback, use the channels described below. To report a security vulnerability, follow SECURITY.md.

License

Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This project is licensed under the Apache License, Version 2.0. See NOTICE for project attributions, THIRD_PARTY_LICENSES.md for human-readable dependency license disclosures, and third_party/licenses.jsonl for the machine-readable inventory.