Nemo Relay Plugin Observability

evals/evals.json

[
  {
    "id": "nemo-relay-plugin-observability-001",
    "question": "I want to use the nemo-relay-plugin-observability skill to add monitoring to my NeMo Relay application. Can you walk me through choosing the right observability path?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent used nemo-relay-plugin-observability, established whether the application uses NeMo Relay 0.6 or 0.7, guided the user through the version-appropriate observability outputs, recommended verifying a plugin-managed exporter before adding others, and provided relevant binding names for their language and version.",
    "expected_behavior": [
      "The agent read the nemo-relay-plugin-observability SKILL.md before responding",
      "The agent determined the installed NeMo Relay version or asked for it before giving version-specific configuration",
      "The agent presented the observability output options with guidance on the separate 0.6 OpenTelemetry/OpenInference exporters and the unified typed 0.7 exporter",
      "The agent recommended verifying a plugin-managed exporter before adding others",
      "The agent explained the lifecycle operations for creating, registering, running instrumented work, flushing when needed, deregistering, and shutting down",
      "The agent referenced the appropriate binding names for the user's language or listed multiple language bindings",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-002",
    "question": "I have a Python application using NeMo Relay and I want to get visibility into what's happening during LLM calls and tool invocations. I'm not sure whether I should use tracing, logging, or some kind of event export. What are my options and how do I set one up?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent identified this as an observability setup task, established the NeMo Relay version, presented console/custom subscribers, ATOF, ATIF, and the version-appropriate OpenTelemetry/OpenInference paths, and explained when plugin-managed configuration or manual subscriber APIs are appropriate.",
    "expected_behavior": [
      "The agent identified the user's need as choosing and setting up NeMo Relay observability without the skill being explicitly named",
      "The agent determined whether the Python application uses NeMo Relay 0.6 or 0.7 before selecting configuration version 2 or 3 and binding classes",
      "The agent described the embedded event and subscriber model including global, scope-local, and plugin-installed subscribers",
      "The agent distinguished plugin-managed process configuration from manual Python subscriber or exporter lifecycle control",
      "The agent outlined the shared lifecycle steps (create, register, run, deregister, flush when needed, shut down)",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-003",
    "question": "We're building a multi-service AI agent platform on NeMo Relay. Our SRE team wants to integrate telemetry into our existing Grafana/Jaeger stack, but our ML team wants portable execution traces they can replay for debugging. Can we support both at the same time, and how would we wire that up in Node.js?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent recognized the dual observability requirement, recommended using OpenTelemetrySubscriber for the Grafana/Jaeger integration and AtifExporter for portable execution trajectories, explained that multiple subscribers can observe the same event stream simultaneously, and provided Node.js-specific setup guidance using registerSubscriber(...), OpenTelemetrySubscriber, and AtifExporter.",
    "expected_behavior": [
      "The agent confirmed that multiple subscribers can observe the same NeMo Relay event stream concurrently",
      "The agent recommended OpenTelemetrySubscriber for the OTLP/Jaeger/Grafana use case and AtifExporter for portable replay trajectories",
      "The agent provided Node.js binding references including registerSubscriber(...), AtifExporter, and OpenTelemetrySubscriber",
      "The agent walked through the shared lifecycle for registering both subscribers before scoped work begins",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-004",
    "question": "How do I configure NeMo Relay's rate limiting and retry policies for upstream LLM API calls?",
    "expected_skill": null,
    "expected_script": null,
    "ground_truth": "The agent recognized this question is about rate limiting and retry configuration for LLM API calls, not about observability setup, and did not invoke the nemo-relay-plugin-observability skill.",
    "expected_behavior": [
      "The agent did not invoke or reference the nemo-relay-plugin-observability skill",
      "The agent addressed the rate limiting and retry topic or indicated it requires a different skill or resource",
      "The agent did not conflate observability/telemetry concepts with request retry and rate limiting configuration",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-005",
    "question": "How do I use the nemo-relay-plugin-observability skill to export my agent's execution traces as ATIF v1.7 documents?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent used nemo-relay-plugin-observability, loaded references/atif.md, and provided a complete walkthrough of creating an AtifExporter with session/agent metadata, registering it before instrumented work, running scoped activity, calling export() or export_json(), and managing the buffer with clear() between runs.",
    "expected_behavior": [
      "The agent read the nemo-relay-plugin-observability SKILL.md before responding",
      "The agent loaded references/atif.md for ATIF-specific trajectory export guidance",
      "The agent explained the default path including AtifExporter creation, registration, running scoped activity, and calling export()",
      "The agent mentioned ATIF v1.7 schema version and the importance of verifying agent metadata and step presence",
      "The agent described buffer management including when to call clear() between runs",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-006",
    "question": "I have a NeMo Relay instrumented agent and I want to convert the collected spans into a trajectory format suitable for replay and offline analysis. How can I get a structured JSON document with user steps, agent steps, and tool observations from my relay events?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent identified this as an ATIF trajectory export task and explained how NeMo Relay events map to ATIF trajectory steps—LLM start events become user steps, LLM end events become agent steps with model metadata and tool_calls, tool end events become system observations, and the result is exportable as structured JSON.",
    "expected_behavior": [
      "The agent identified the nemo-relay-plugin-observability skill as relevant to the user's request",
      "The agent loaded references/atif.md for ATIF-specific trajectory export guidance",
      "The agent explained the semantic mapping from NeMo Relay events to ATIF trajectory steps (user, agent, system)",
      "The agent described how to use export_json() to produce the structured JSON document",
      "The agent mentioned that tool calls are promoted from LLM end responses and observations are correlated by function name",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-007",
    "question": "We're building an evaluation pipeline for our multi-agent system. Each agent run needs to produce a standardized trajectory that captures LLM calls, tool invocations, and sub-agent interactions so our eval framework can score them. The agents are already instrumented with NeMo Relay. What's the best way to produce these trajectory files, and how do nested agent scopes appear in the output?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent recommended using the AtifExporter to produce ATIF v1.7 trajectory documents from NeMo Relay events, explained how nested agent scopes become embedded subagent_trajectories with subagent_trajectory_ref observations in the parent, and provided guidance on validation before evaluation including checking schema version, metadata, and step completeness.",
    "expected_behavior": [
      "The agent referenced the nemo-relay-plugin-observability skill and its ATIF v1.7 output format",
      "The agent loaded references/atif.md for ATIF-specific trajectory export guidance",
      "The agent explained that nested agent scopes become embedded subagent_trajectories with subagent_trajectory_ref observations in the parent trajectory",
      "The agent described the validation checklist: confirming schema_version is ATIF-v1.7, agent metadata is correct, expected steps are present, and sensitive fields are absent",
      "The agent advised on buffer management for multi-agent scenarios, such as using one exporter per run or calling clear() between runs",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-008",
    "question": "How do I configure Prometheus metrics scraping for my NeMo Guardrails deployment running on Kubernetes?",
    "expected_skill": null,
    "expected_script": null,
    "ground_truth": "The agent recognized this question is about Prometheus metrics and Kubernetes configuration, which is unrelated to ATIF trajectory export, and either provided general guidance on Prometheus/Kubernetes or indicated it does not have a specific skill for this task.",
    "expected_behavior": [
      "The agent did not invoke or reference the nemo-relay-plugin-observability skill",
      "The agent addressed the Prometheus metrics scraping topic or clarified it lacks a matching skill",
      "The agent did not mention AtifExporter, ATIF trajectories, or trajectory export in its response",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-009",
    "question": "How do I use the nemo-relay-plugin-observability skill with NeMo Relay 0.6 to configure OTLP tracing export to a local OpenTelemetry Collector?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent used nemo-relay-plugin-observability, loaded references/opentelemetry.md, and provided a NeMo Relay 0.6 configuration-version-2 walkthrough using the standalone OpenTelemetryConfig and OpenTelemetrySubscriber, including endpoint setup on port 4318, http_binary transport, service name, registration, and shutdown/flush steps.",
    "expected_behavior": [
      "The agent read the nemo-relay-plugin-observability SKILL.md to understand the configuration workflow",
      "The agent loaded references/opentelemetry.md for OpenTelemetry-specific export guidance",
      "The agent kept the example on the NeMo Relay 0.6 surface with observability configuration version 2 and did not introduce 0.7 typed endpoints",
      "The agent provided specific configuration details including http_binary transport and endpoint localhost:4318",
      "The agent explained the register/deregister lifecycle including flush and shutdown steps",
      "The agent mentioned service name and resource attribute configuration",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-010",
    "question": "I use NeMo Relay 0.7 and need to send traces to Jaeger using gRPC transport. How do I set up the endpoint, configure auth headers, and ensure spans are flushed before my service exits?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent identified this as an OpenTelemetry export task and provided guidance on configuring gRPC transport for Jaeger, including a TLS-verified endpoint, runtime-only auth header construction from secret injection, exporter-owned runtime behavior, a synthetic non-sensitive first proof with sanitization before production export, backend span validation with redacted diagnostics, and deterministic flush-before-exit procedures.",
    "expected_behavior": [
      "The agent referenced the nemo-relay-plugin-observability skill content for gRPC transport requirements",
      "The agent loaded references/opentelemetry.md for OpenTelemetry-specific export guidance",
      "The agent explained that the subscriber owns the Tokio runtime required by gRPC",
      "The agent explained how to construct auth headers in memory from runtime secret injection without placing credential values in source code, committed configuration, command-line arguments, prompts, examples, or diagnostics",
      "The agent required TLS verification and avoided endpoints that embed credentials in URL user information or query parameters",
      "The agent used synthetic non-sensitive payloads for the first proof and required sanitization before production export",
      "The agent validated exporter behavior with synthetic scopes, tools, or LLMs and reported only redacted header names and response status",
      "The agent described the flush and shutdown sequence for graceful exit",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-011",
    "question": "We're deploying a NeMo Relay-based LLM orchestration service in production on Kubernetes. Our observability stack uses Grafana Tempo with an OpenTelemetry Collector sidecar. I want to make sure all LLM tool calls and scope spans show up correctly grouped in Tempo. What's the recommended setup?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent provided a production-ready OpenTelemetry export configuration for a Kubernetes deployment with Grafana Tempo, covering collector sidecar endpoint configuration, service identity best practices, resource attributes, span grouping by root scope, sensitive payload redaction, and validation steps to confirm spans appear correctly in Tempo.",
    "expected_behavior": [
      "The agent consulted the nemo-relay-plugin-observability skill and applied its guidance to the Kubernetes/Tempo scenario",
      "The agent loaded references/opentelemetry.md for OpenTelemetry-specific export guidance",
      "The agent recommended http_binary transport with the collector sidecar endpoint and stable service naming",
      "The agent addressed span grouping by root scope and validation of backend spans for scopes/tools/LLMs",
      "The agent advised keeping auth and endpoints out of source code and redacting sensitive payloads before production export",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-012",
    "question": "How do I fine-tune a LLaMA model using NeMo Framework with a custom dataset in JSONL format?",
    "expected_skill": null,
    "expected_script": null,
    "ground_truth": "The agent recognized this as a model fine-tuning question unrelated to NeMo Relay OpenTelemetry export and did not invoke the nemo-relay-plugin-observability skill. It either provided general fine-tuning guidance or directed the user to appropriate NeMo Framework documentation.",
    "expected_behavior": [
      "The agent did not reference or invoke the nemo-relay-plugin-observability skill",
      "The agent recognized the question is about NeMo Framework model training, not NeMo Relay observability",
      "The agent provided relevant guidance about fine-tuning or directed the user to appropriate resources",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-013",
    "question": "How do I use the nemo-relay-plugin-observability skill with NeMo Relay 0.7 to send traces to Arize Phoenix with HTTP binary transport?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent used nemo-relay-plugin-observability, loaded references/openinference.md, and provided a complete configuration walkthrough for sending OpenInference traces to Arize Phoenix using http_binary transport, including endpoint setup, service metadata, and subscriber registration.",
    "expected_behavior": [
      "The agent read the nemo-relay-plugin-observability SKILL.md before responding",
      "The agent loaded references/openinference.md for OpenInference-specific export guidance",
      "The agent used observability configuration version 3 and did not introduce the standalone 0.6 OpenInferenceConfig or OpenInferenceSubscriber",
      "The agent explained how to build an OpenTelemetryConfig with type openinference, http_binary transport, and an OTLP/HTTP traces endpoint",
      "The agent described the steps to construct and register the subscriber, run instrumented work, and deregister/flush/shutdown",
      "The agent mentioned that input.value and output.value are derived from scope/tool/LLM start inputs and end outputs respectively",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-014",
    "question": "I use NeMo Relay 0.6 and need to export my LLM traces with model-centric semantic conventions to an OTLP backend. How do I configure the endpoint, headers, and resource attributes so the backend understands the span kinds and token usage?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent identified this as a NeMo Relay 0.6 standalone OpenInference exporter task and provided guidance using OpenInferenceConfig and OpenInferenceSubscriber with observability configuration version 2, including endpoint, headers, resource attributes, service identity, transport selection, and token counters.",
    "expected_behavior": [
      "The agent referenced the nemo-relay-plugin-observability skill content to address the user's needs",
      "The agent loaded references/openinference.md for OpenInference-specific export guidance",
      "The agent explained how to configure transport, endpoint, service_name, headers, and resource attributes in the 0.6 OpenInferenceConfig",
      "The agent described how scope types map to OpenInference span kinds and how LLM usage metadata maps token counters",
      "The agent noted that http_binary is the default and 0.6 grpc requires an active Tokio runtime",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-015",
    "question": "We're integrating NeMo Relay into our ML observability stack and our team uses Arize Phoenix to monitor LLM calls. Spans are not showing up in Phoenix even though our collector seems to be running. What should I check?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent applied the nemo-relay-plugin-observability troubleshooting guidance, established whether the application uses NeMo Relay 0.6 or 0.7, and used the corresponding subscriber, correlation fields, transport requirements, and lifecycle checks.",
    "expected_behavior": [
      "The agent consulted the nemo-relay-plugin-observability skill's troubleshooting focus section",
      "The agent loaded references/openinference.md for OpenInference-specific export guidance",
      "The agent determined the NeMo Relay version before choosing the standalone 0.6 OpenInference subscriber or the 0.7 typed OpenTelemetry projection",
      "The agent suggested checking construction logs and collector traffic, then same-root_uuid spans for 0.6 or nemo_relay.uuid/nemo_relay.parent_uuid lineage for 0.7",
      "The agent asked or advised about verifying the correct OTLP transport (http_binary vs grpc) for the user's binding/target",
      "The agent recommended confirming the subscriber was properly registered and that flush/shutdown was called after instrumented work",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-016",
    "question": "How do I set up a Kubernetes CronJob to periodically clean up old container images from my registry?",
    "expected_skill": null,
    "expected_script": null,
    "ground_truth": "The agent recognized this as a Kubernetes/container registry maintenance question unrelated to NeMo Relay OpenInference export and provided general Kubernetes CronJob guidance without invoking the nemo-relay-plugin-observability skill.",
    "expected_behavior": [
      "The agent did not reference or invoke the nemo-relay-plugin-observability skill",
      "The agent provided guidance related to Kubernetes CronJob configuration or container registry cleanup",
      "The agent did not mention OpenInference semantics, OTLP backends, or NeMo Relay tracing",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-017",
    "question": "I want the raw NeMo Relay lifecycle events in a local JSONL file so I can inspect exactly what scopes, tools, and LLM calls emitted. Which observability output should I configure?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent selected ATOF JSONL as the raw canonical event output, loaded references/atof.md, and recommended enabling one plugin-managed ATOF exporter before adding translated formats.",
    "expected_behavior": [
      "The agent read the nemo-relay-plugin-observability SKILL.md before responding",
      "The agent selected ATOF rather than ATIF or the version-appropriate OpenTelemetry/OpenInference exporter for raw canonical lifecycle events",
      "The agent loaded references/atof.md for ATOF-specific guidance",
      "The agent recommended a plugin-managed ATOF exporter as the default reusable process configuration",
      "The agent explained that ATOF preserves raw events rather than translating them into trajectories or trace spans",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-018",
    "question": "Can you configure the Relay observability plugin to append ATOF events to logs/events.jsonl and tell me how to verify parent-child relationships?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent used the ATOF reference to provide the observability plugin component with append mode and explained verification through JSONL validity, expected lifecycle events, and UUID/parent UUID relationships.",
    "expected_behavior": [
      "The agent loaded references/atof.md",
      "The agent configured the built-in observability component with ATOF enabled, output_directory logs, filename events.jsonl, and mode append",
      "The agent recommended verifying one JSON object per line and the expected scope/tool/LLM events",
      "The agent used UUID and parent UUID fields to verify ancestry rather than relying only on event order",
      "The agent warned that sensitive fields should be checked before retaining or transmitting output",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-019",
    "question": "For a one-off Python test I need an ATOF exporter with a custom subscriber name and an exact collection window. Should I still configure plugins.toml?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent selected the manual AtofExporter API because the one-off test needs a custom name and explicit registration window, and described the create, register, run, flush, deregister, and shutdown lifecycle.",
    "expected_behavior": [
      "The agent recognized that explicit test-level lifecycle control is a valid manual exporter use case",
      "The agent did not require plugins.toml for the one-off test",
      "The agent described the manual ATOF lifecycle in the correct order: create, register, run, force flush, deregister, and shut down",
      "The agent explained that plugin-managed configuration remains the default for reusable process setup",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-plugin-observability-020",
    "question": "My application does not emit any Relay scope or call events yet. Can you fix that by turning on ATOF export?",
    "expected_skill": "nemo-relay-plugin-observability",
    "expected_script": null,
    "ground_truth": "The agent explained that an exporter cannot create missing instrumentation events, routed the user to get-started or instrument-calls first, and kept capture separate from export.",
    "expected_behavior": [
      "The agent did not claim that enabling ATOF creates scope, tool, or LLM lifecycle events",
      "The agent explained that observability export consumes events emitted by instrumentation",
      "The agent directed the user to nemo-relay-get-started or nemo-relay-instrument-calls before configuring export",
      "The agent kept instrumentation and export as separate responsibilities",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  }
]