Nemo Relay Instrument Calls

evals/evals.json

[
  {
    "id": "nemo-relay-instrument-calls-001",
    "question": "How do I use the nemo-relay-instrument-calls skill to wrap my existing Python tool functions with NeMo Relay managed execution?",
    "expected_skill": "nemo-relay-instrument-calls",
    "expected_script": null,
    "ground_truth": "The agent used nemo-relay-instrument-calls and provided guidance on wrapping existing Python tool functions using the tools.execute(...) and llm.execute(...) managed execution APIs, including scope boundaries and preserving original arguments/results.",
    "expected_behavior": [
      "The agent read the nemo-relay-instrument-calls SKILL.md before responding",
      "The agent explained how to use Python's tools.execute(...) managed execution API to wrap existing tool functions",
      "The agent described establishing a scope boundary around the agent or workflow before the first tool call",
      "The agent noted that the original tool callable retains responsibility for business logic while NeMo Relay owns lifecycle events",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-instrument-calls-002",
    "question": "I have a Rust agent with several tool functions and an OpenAI provider call. I want to add NeMo Relay instrumentation around these calls so they emit lifecycle events and go through guardrails. How should I structure this?",
    "expected_skill": "nemo-relay-instrument-calls",
    "expected_script": null,
    "ground_truth": "The agent provided detailed Rust-specific guidance on using tool_call_execute(ToolCallExecuteParams::builder()...) and llm_call_execute(LlmCallExecuteParams::builder()...) to wrap existing tool functions and the OpenAI provider call, including scope placement and conditional-execution guardrail semantics.",
    "expected_behavior": [
      "The agent read the nemo-relay-instrument-calls SKILL.md before responding",
      "The agent recommended using Rust managed execution APIs: tool_call_execute and llm_call_execute with their builder patterns",
      "The agent explained that a scope should be placed around the agent boundary before the first tool or LLM call",
      "The agent described how conditional-execution guardrails run first on raw input and can reject before the callable executes",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-instrument-calls-003",
    "question": "We're building a Node.js customer support chatbot that calls a 'lookup_order' tool and then sends the result to GPT-4 for summarization. We need proper observability and guardrail integration for compliance. The tool and LLM calls already work but we need them to go through our NeMo Relay runtime. Can you help me instrument these?",
    "expected_skill": "nemo-relay-instrument-calls",
    "expected_script": null,
    "ground_truth": "The agent provided a concrete plan for instrumenting the Node.js chatbot's lookup_order tool call using toolCallExecute(...) and the GPT-4 summarization call using llmCallExecute(...), explaining scope placement, metadata attachment for model name, and how request/response sanitize guardrails affect emitted events without altering caller-visible data.",
    "expected_behavior": [
      "The agent read the nemo-relay-instrument-calls SKILL.md before responding",
      "The agent recommended using Node.js APIs toolCallExecute(...) for the lookup_order tool and llmCallExecute(...) for the GPT-4 call",
      "The agent explained how to attach metadata such as model name for trace export and diagnostics",
      "The agent clarified that sanitize-response guardrails affect emitted event payloads only and do not alter the value returned to application code",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "nemo-relay-instrument-calls-004",
    "question": "How do I configure OpenTelemetry trace export and set up ATIF formatting for my NeMo Relay deployment?",
    "expected_skill": null,
    "expected_script": null,
    "ground_truth": "The agent recognized this as a trace/observability setup question and directed the user to the nemo-relay-plugin-observability skill rather than nemo-relay-instrument-calls, since the question is about export configuration and ATIF formatting rather than wrapping tool or LLM calls.",
    "expected_behavior": [
      "The agent identified that the question is about observability setup rather than instrumenting tool/LLM calls",
      "The agent recommended the nemo-relay-plugin-observability skill or provided observability configuration guidance",
      "The agent did not provide tool_call_execute or llm_call_execute instrumentation patterns as the primary answer",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  }
]