Generated banner for Vally Tests
Generated logo for Vally Tests

Vally Tests

MicrosoftOfficialPublished by Microsoft

  • Skill
  • Workflow
  • Agent tooling
  • MIT
  • Low risk
  • Has scripts

From the author

Authors Vally conformance tests for prompts, instructions, agents, and skills, and refuses jailbreak, prompt-injection, harmful-elicitation, TOS, CoC, and PII-extraction stimuli
— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The vally-tests skill provides a structured workflow for authoring conformance tests for various AI artifacts, including prompts, instructions, agents, and skills. It operates by mapping artifact behaviors to specific Vally grader types, ensuring that tests are both relevant and robust. The skill includes a safety refusal taxonomy to prevent the generation of adversarial or harmful stimuli, and it supports both individual artifact processing and batch processing via corpus imports.

What it does

  • Authors conformance test stimuli for prompts, instructions, agents, and skills.
  • Routes test stimuli to appropriate Vally evaluation files based on artifact kind.
  • Enforces a safety refusal taxonomy to block adversarial or policy-violating content.
  • Performs deduplication of stimuli using SHA-256 hashing of normalized prompt text.
  • Supports batch processing of test stimuli via CSV or XLSX corpus imports.
  • Generates JSON run reports detailing the outcome of the authoring process.

When to use it

  • When you need to create conformance tests for a new prompt, instruction, agent, or skill.
  • When you have a batch of behaviors that require standardized formatting and deduplication.
  • When you need to ensure that authored test stimuli comply with predefined safety and refusal policies.
  • When you are working within a Vally-based evaluation environment.

Advantages

  • Automates the creation of consistent and structured test stimuli.
  • Prevents the accidental inclusion of harmful or prohibited content through a built-in safety taxonomy.
  • Ensures test uniqueness through automated deduplication.
  • Provides clear routing to the correct evaluation suites for different artifact types.

Usage examples

“vally-tests mode=from-artifact files=my-agent.agent.md”
The skill analyzes my-agent.agent.md, selects appropriate conformance checks, and appends the generated stimuli to the corresponding Vally eval file.
“vally-tests mode=corpus-import path=tests.csv”
The skill processes each row in the provided CSV, generates the corresponding stimulus blocks, and appends them to the routed eval file after deduplication.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/microsoft/vally-tests/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip vally-tests.zip -d .agents/skills

Documentation by Microsoft

<!-- cspell:ignore roleplay doxxing scaffolder -->

Vally Tests Skill

Purpose

This skill authors Vally conformance tests for the four supported artifact kinds in this repository: prompts, instructions, agents, and skills. Each test exercises a documented behavior the artifact already claims and routes the result through an appropriate Vally grader so failures are explainable. Test authoring is bounded by a refusal taxonomy that keeps the skill out of adversarial, harmful, or policy-evasion territory.

Follow the shared content-policy public-output guard for eval stimuli and any public summaries derived from this skill. Vally tests are not a venue for terms-of-service boundary mapping, payload generation, refusal-text scoring, hidden-instruction disclosure, PII extraction, or secret extraction.

The skill ships:

  • A canonical authoring workflow that runs directly through /vally-tests or through the Vally Test Author subagent.

  • Per-kind reference files that enumerate every conformance check the skill knows how to express.

  • A grader catalog that maps Vally CLI grader types to the checks they fit, and robustness rules that keep an emitted grader from asserting the impossible or failing correct behavior.

  • A safety refusal taxonomy with regex patterns the safety lint script consumes.

  • Helper scripts and asset templates for stimulus emission, corpus import, and dedupe.

When to Invoke

Invoke this skill in one of two modes:

  • From-artifact mode. The caller points at one or more artifact files (a .prompt.md, .instructions.md, .agent.md, or SKILL.md) through files, or at the open or attached file when files is omitted, and asks for conformance test stimuli that verify each artifact's stated behaviors. The skill detects the artifact kind from the filename unless kind overrides it, looks up the matching per-kind reference, picks graders, and appends stimulus blocks to the routed Vally eval file.

  • Corpus-import mode. The caller supplies a CSV or XLSX corpus through path that follows the shipped template, and the skill turns each row into a stimulus block in the routed eval file. Before reading, confirm that path exists and ends in .csv or .xlsx; otherwise name the bad path and stop. Every imported row carries tags.advisory: true, and corpus content cannot override it. Used when a large batch of behaviors needs identical formatting and dedupe handling.

After either mode, report the target eval file, stimuli appended, duplicates skipped, refusals triggered, and the JSON run report path.

Do not invoke this skill to:

  • Generate adversarial probes, jailbreak attempts, prompt-injection payloads, or red-team stimuli.

  • Test refusal quality on harmful, illegal, or policy-violating prompts.

  • Author stimuli that elicit PII, secrets, hidden instructions, or model self-disclosure.

  • Store payload examples, paraphrased prohibited requests, or quoted flagged content in Vally eval specs, corpus rows, PR summaries, issue bodies, or comments.

  • Replace Responsible AI threat modeling. RAI work lives in the RAI Planner agent.

Authoring Workflow

Each invocation follows the same six-step pipeline:

  1. Artifact-kind detection. Resolve the kind from the artifact path or the corpus row's kind column. Supported kinds: prompt, instructions, agent, skill. Reject unknown kinds with a refusal block.

  2. Reference lookup. Load the matching reference file from references/ and select the check or checks the stimulus exercises. Stage the artifact under test in the stimulus's agent_environment so the agent can read what the prompt names.

  3. Grader selection. Use references/grader-catalog.md as the authoritative selection surface. Pick the registered Vally grader family that directly observes the expected behavior and follow its stimulus-shape guidance.

  4. Robustness check. Apply references/grader-robustness.md. Confirm the stimulus stages every file whose wording the grader asserts, that the pattern constrains order and proximity only where those are the behavior under test, that a compliant agent phrasing the behavior differently still passes, and that no positive grader passes on the prompt text alone. Verify the pattern offline against answers that must pass and answers that must still fail before appending.

  5. Safety self-check. Run a safety self-check against the refusal taxonomy regex set. Refusing here is the correct outcome for any stimulus that matches a refusal category.

  6. Dedupe and append. Compute a SHA-256 hash of the normalized prompt text, compare against existing stimuli in the target eval file, and append only when novel.

The pipeline is identical for both invocation modes. Corpus-import mode runs steps 2 through 6 once per row.

Safety Refusal Taxonomy

Authoring is bounded by seven refusal categories. Any stimulus that matches a category's regex set is refused before it reaches the dedupe step. The categories and their normative source are listed below. The full regex set lives in references/refusal-taxonomy.md and is consumed by the safety lint script.

CategoryDefinitionNormative Source
jailbreakAttempts to bypass model safety policies through persona switching, hypothetical framing, or roleplay coercion.Project Code of Conduct
prompt-injectionAttempts to override system or developer instructions through embedded or external content.Project Code of Conduct
harmful-elicitationRequests for content that causes physical, financial, psychological, or reputational harm.Project Code of Conduct
tos-violationStimuli that solicit content prohibited by GitHub, Microsoft, or model-provider terms of service.Project Code of Conduct
coc-violationStimuli that violate this repository's Code of Conduct, including harassment, discrimination, or doxxing.Project Code of Conduct
model-refusal-elicitationAttempts to provoke a model refusal so the refusal text itself can be scored, graded, or used to map provider boundaries.RAI Planner guidance
pii-extractionAttempts to elicit personally identifiable information, secrets, credentials, or proprietary training data.RAI Planner guidance

When a request triggers a refusal, emit the canonical refusal block:

This skill authors conformance tests only. The request appears to fall under <category>. Please consult <Project Code of Conduct | RAI Planner agent> for the appropriate process.

Substitute the matched <category> and the most relevant normative source. Do not negotiate, rephrase, or partially fulfill the request.

Helper Script Index

Helper scripts ship as parity pairs (.ps1 and .sh) where the workflow does not require Python. The safety lint is the exception: it ships only as Lint-VallyTestSafety.ps1, which import_corpus.py invokes, so a single regex engine evaluates the refusal taxonomy. Python is used only for the corpus-import path because the source-of-truth interchange format is CSV with an XLSX mirror.

ScriptPurposeLanguageDelivery
scripts/New-Stimulus.ps1Scaffolds a single stimulus YAML block from an artifact path and appends to the routed eval file.PowerShell 7+Phase 2
scripts/new-stimulus.shParity counterpart for the PowerShell stimulus scaffolder.bashPhase 2
scripts/import_corpus.pyReads the CSV or XLSX corpus template and emits dedupe-checked stimulus blocks per kind.Python 3.11+Phase 2
scripts/Lint-VallyTestSafety.ps1Runs the refusal taxonomy regex set against a candidate stimulus and exits non-zero on match.PowerShell 7+Phase 3

All helpers honour a shared dedupe contract: SHA-256 of the prompt text after Unicode NFC normalization and whitespace collapse.

The helpers emit a JSON run report to logs/vally-test-author-<timestamp>.json, where <timestamp> is YYYYMMDD-HHMMSS (UTC). The report captures, at minimum: mode; inputs (the resolved files/path and kind); target_eval_file; stimuli_appended (count and per-row hash); dedupe_results (count and per-row hash for skipped duplicates); refusal_check (per-row category match, if any); safety_lint_exit_code; blockers (any ambiguous safety-lint outcomes surfaced for review); and written_paths.

Reference Index

References capture the conformance taxonomy, grader selection rules, eval-suite routing, and the regex source of truth for the refusal taxonomy. Each file targets a specific decision point in the authoring workflow.

ReferenceCovers
prompts.mdThe 12 conformance checks emitted for .prompt.md artifacts.
instructions.mdThe 8 conformance checks emitted for .instructions.md artifacts.
agents.mdThe 9 conformance checks emitted for .agent.md artifacts.
skills.mdThe 9 conformance checks emitted for SKILL.md artifacts.
grader-catalog.mdVally CLI grader types, selection rules, and gotchas.
grader-robustness.mdMounting, order and proximity limits, negation guards, prompt-echo checks, and the pre-commit probe.
refusal-taxonomy.mdRegex source of truth for the 7 refusal categories and worked examples.
eval-suite-routing.mdMaps artifact kind to the canonical Vally eval file under evals/.

Asset Index

Assets supply the interchange formats the corpus-import path consumes. The CSV is the source of truth. The XLSX mirror is regenerated from the CSV by import_corpus.py and is never edited directly.

AssetPurpose
assets/corpus-import-template.csvCanonical CSV template with header prompt,kind,target_artifact,grader,tags,expected_refusal_category,notes.
assets/corpus-import-template.xlsxExcel mirror of the CSV regenerated by the import script.

Output Targets per Kind

Authored stimuli always land in one of the routed Vally eval files. The router is encoded in references/eval-suite-routing.md and mirrored here for quick lookup.

KindTarget Eval FileVally Suite Name
promptevals/behavior-conformance/prompts.eval.yamlbehavior-conformance
instructionsevals/behavior-conformance/instructions.eval.yamlbehavior-conformance
agentevals/agent-behavior/eval.yamlagent-behavior
skillevals/behavior-conformance/skill-behavior.eval.yamlbehavior-conformance

The prompt target is created on first use when a repository has no prompt stimuli yet.

Never write to evals/baseline-equivalence/, evals/script-validation/, or evals/results/ from this skill. Those targets serve baseline equivalence, script validation, and historical comparison flows that are out of scope for conformance authoring.

Contributing

Follow these conventions when extending this skill:

  • New per-kind checks belong in the matching references/{kind}.md file. Bump the check count in this SKILL.md when the reference adds or removes checks.

  • New grader types belong in references/grader-catalog.md and only after the matching Vally CLI version is pinned in package.json devDependencies.

  • New refusal categories require updates to references/refusal-taxonomy.md, the regex set the safety lint script consumes, the Safety Refusal Taxonomy table above, and the canonical refusal block.

  • Helper scripts must ship in parity pairs (.ps1 and .sh) unless the workflow has a hard Python dependency. The safety lint is the one exception and ships only as Lint-VallyTestSafety.ps1; do not add a bash mirror. Python helpers live under scripts/ and are configured by the skill's pyproject.toml.

Files

references/prompts.md


title: Prompts Conformance Checks description: Twelve conformance checks the vally-tests skill emits for .prompt.md artifacts, with contract citations, stimulus shapes, and Vally grader recommendations

<!-- markdownlint-disable-file -->

Prompts Conformance Checks

Overview

This reference enumerates the twelve conformance checks the vally-tests skill knows how to express for .prompt.md artifacts. Each check exercises a behavior the prompt's authoring contract already claims, then routes the resulting stimulus block to the canonical Vally eval file declared in eval-suite-routing.md.

The Copilot agent harness does not load .prompt.md files, and HVE Core ships its entry points as user-invocable skills. Use these checks only for prompt files kept for hosts that still load them, such as VS Code's Local harness. Test a skill entry point with skills.md.

The canonical eval target for this kind is evals/behavior-conformance/prompts.eval.yaml. New stimulus blocks are appended to its stimuli: array, creating the file on first use, and tagged tags.advisory: true per eval-suite-routing.md. Authors MUST run every candidate stimulus through refusal-taxonomy.md before emission and refuse any match.

Grader identifiers below use the Vally CLI 0.9.0 catalog (semantic_similarity, contains, regex, json_schema) per grader-catalog.md. Where the research phrasing recommended output-matches, the equivalent here is regex; where it recommended output-contains, the equivalent is contains; where it recommended llm-grader, the equivalent is semantic_similarity.

Contract Summary

TopicSection in hve-builder.instructions.md
Frontmatter and metadataFrontmatter and Portability
Agent delegationKeep Context Focused
Input variables and argument hintsFrontmatter and Portability
Protocol structureOutcome and Structure
Step and phase headingsOutcome and Structure; Writing
Writing style and link formatWriting
Subagent invocationFrontmatter and Portability
Quality criteria checklistOutcome and Structure; Completion Check

Conformance Checks

Check 1: Required Frontmatter Fields
  • Contract source: hve-builder.instructions.md, Frontmatter and Portability.

  • Testable behavior: prompt frontmatter MUST include a non-empty description: field under 120 characters; OPTIONAL fields agent:, argument-hint:, and a --- activation line MAY be present when the prompt delegates or accepts arguments.

  • Suggested stimulus: ask the assistant to summarize the frontmatter of a named prompt under .github/prompts/, then assert that the description value is surfaced in the response.

  • Grader recommendation: regex with pattern (?m)^description:\s*['"]?.{1,120}['"]?.

  • Evidence: a delegating prompt typically declares description:, agent:, and argument-hint: together.

Check 2: Agent Delegation Without Duplication
  • Contract source: hve-builder.instructions.md, Keep Context Focused.

  • Testable behavior: when the prompt sets agent:, it MUST NOT duplicate the delegated agent's Required Phases or Required Steps; instead the prompt references the specific phases or sections that differ and extends rather than substitutes the agent's requirements section.

  • Suggested stimulus: ask the assistant to describe what a delegating prompt adds on top of its agent, naming the delegated agent and any sections that differ.

  • Grader recommendation: semantic_similarity with rubric "Does the response identify the delegated agent and confirm that the prompt extends rather than duplicates the agent's protocol?".

  • Evidence: a prompt that sets agent: contributes scoped Inputs and Requirements and leaves the delegated agent's flow to the agent.

Check 3: Inputs Documentation Format
  • Contract source: hve-builder.instructions.md, Frontmatter and Portability.

  • Testable behavior: when the prompt defines inputs, the Inputs section MUST document every input variable using ${input:varName} for required inputs or ${input:varName:defaultValue} for optional inputs.

  • Suggested stimulus: ask the assistant to list the inputs a named prompt accepts and the default value (if any) for each.

  • Grader recommendation: regex with pattern \$\{input:[a-zA-Z_][a-zA-Z0-9_]*(?::[^}]*)?\}.

  • Evidence: a prompt with inputs documents each variable, such as a required ${input:task} and an optional ${input:mode:standard}, with a description.

Check 4: Argument Hint Format
  • Contract source: hve-builder.instructions.md, Frontmatter and Portability.

  • Testable behavior: when the prompt declares argument-hint:, the value MUST use [] for positional arguments, key=value for named arguments, {option1|option2} for enumerated choices, and ... for free-form remainders.

  • Suggested stimulus: ask the assistant to show the argument hint a named prompt advertises in the VS Code picker.

  • Grader recommendation: regex with pattern argument-hint:\s*["'][^"']*(?:\[.*\]|\{.*\|.*\}|=|\.\.\.).

  • Evidence: an argument hint such as argument-hint: "task=... [mode={standard|deep}] [notes=...]" combines named, enumerated, and free-form arguments.

Check 5: Protocol Structure Presence
  • Contract source: hve-builder.instructions.md, Outcome and Structure.

  • Testable behavior: a prompt with multiple ordered stages or a complex workflow MUST include either ## Required Steps (autonomous, step-based) or ## Required Phases (conversational, phase-based). Single-task prompts MAY omit a protocol section.

  • Suggested stimulus: ask the assistant whether a named prompt uses a step-based or phase-based protocol and to name the section heading.

  • Grader recommendation: regex with pattern (?m)^##\s+Required\s+(Steps|Phases|Protocol)\b.

  • Evidence: a multi-stage prompt declares a ## Required Steps, ## Required Phases, or ## Required Protocol section that scopes its behavior.

Check 6: Step and Phase Heading Consistency
  • Contract source: hve-builder.instructions.md, Outcome and Structure and Writing.

  • Testable behavior: when a protocol section is present, each step heading MUST take the form ### Step N: Short Summary and each phase heading MUST take the form ### Phase N: Short Summary with a descriptive summary after the colon.

  • Suggested stimulus: ask the assistant to list the step or phase headings of a named prompt in order.

  • Grader recommendation: regex with pattern (?m)^###\s+(?:Step|Phase)\s+\d+:\s+\S.+.

  • Evidence: a step-based prompt uses numbered step headings with descriptive summaries.

  • Contract source: hve-builder.instructions.md, Writing, and markdown.instructions.md.

  • Testable behavior: file path references that appear in user-facing response sections MUST be formatted as markdown links [filename](path/to/file). Such references MUST NOT be wrapped in backticks, because backticks suppress link rendering.

  • Suggested stimulus: ask the assistant to point to a specific file the prompt references and confirm the response surfaces a clickable markdown link.

  • Grader recommendation: regex with positive pattern \[[^\]]+\]\([^)]+\.(?:md|py|ts|js|sh|ps1|yml|yaml)\) and negate pattern (?<!\\)`[a-zA-Z0-9_./-]+\.(?:md|py|ts|js|sh|ps1|yml|yaml)` .

  • Evidence: the Writing section in hve-builder.instructions.md defers to markdown.instructions.md, which states the markdown-link surface rule for user-facing file references.

  • Contract source: hve-builder.instructions.md, Writing, and markdown.instructions.md.

  • Testable behavior: external URL references MUST be formatted as markdown links [display text](https://example.com). Raw URLs and backtick-wrapped URLs are non-conforming.

  • Suggested stimulus: ask the assistant to cite an external resource the prompt or its delegated agent points to.

  • Grader recommendation: regex with pattern \[[^\]]+\]\(https?://[^\s)]+\).

  • Evidence: the Writing section in hve-builder.instructions.md, markdown.instructions.md, and writing-style.instructions.md reinforce the markdown link requirement for URLs.

Check 9: List Type Matches Purpose
  • Contract source: hve-builder.instructions.md, Writing.

  • Testable behavior: bulleted lists (* or -) MUST be used for groupings and option lists; numbered lists (1., 2., ...) MUST be used for sequential action steps. Sequence vs grouping intent MUST match the list type.

  • Suggested stimulus: ask the assistant to walk through the ordered steps of a named prompt's protocol and separately to enumerate the groupings of optional inputs.

  • Grader recommendation: semantic_similarity with rubric "Does the response use ordered lists for sequential steps and bulleted lists for groupings consistent with the prompt's structure?".

  • Evidence: prompts authored to the hve-builder conventions follow this pattern.

Check 10: Prompt Design Principles
  • Contract source: hve-builder.instructions.md, Outcome and Structure and Completion Check.

  • Testable behavior: prompt outputs MUST exhibit clarity (followable without guessing), consistency (similar inputs yield similar shapes), alignment (matches repo conventions), coherence (no internal conflicts), calibration (just enough instruction), and correctness (asks rather than guesses when inputs are ambiguous).

  • Suggested stimulus: invoke the prompt with a deliberately ambiguous or incomplete input and observe whether the assistant asks for clarification rather than fabricating values.

  • Grader recommendation: semantic_similarity with rubric "Does the response demonstrate the six prompt-design principles, in particular asking for clarification on ambiguous inputs instead of guessing?".

  • Evidence: the prompt-design principles in the hve-builder conventions.

Check 11: Few-Shot Examples in Fenced Code Blocks
  • Contract source: hve-builder.instructions.md, Writing, and markdown.instructions.md.

  • Testable behavior: when the prompt presents code examples or few-shot demonstrations, those examples MUST appear inside correctly fenced code blocks with a language identifier where one applies.

  • Suggested stimulus: ask the assistant to reproduce a short example the prompt references.

  • Grader recommendation: regex with pattern (?ms)^```[a-z0-9_+-]*\n.+?\n```$.

  • Evidence: the Writing guidance in hve-builder.instructions.md and skill reference files such as .github/skills/hve-core/vally-tests/references/refusal-taxonomy.md apply fenced code blocks consistently.

Check 12: Subagent Invocation Uses Human-Readable Names
  • Contract source: hve-builder.instructions.md, Frontmatter and Portability.

  • Testable behavior: when the prompt or its delegated agent invokes a subagent, invocation text MUST reference the subagent by the human-readable name: from the subagent's frontmatter (for example, "Dispatch the BRD Quality Reviewer subagent"). Invocation by filename or by file path is non-conforming.

  • Suggested stimulus: ask the assistant which subagent a named prompt invokes and how the invocation reads.

  • Grader recommendation: regex with a positive pattern naming the subagent, for example (?i)\bBRD Quality Reviewer\b, and negate pattern (?i)[A-Za-z0-9_-]+\.agent\.md.

  • Evidence: .github/agents/project-planning/brd-builder.agent.md dispatches BRD Quality Reviewer by its human-readable name.

Cross-References

Requirements and compatibility

Written for: GitHub Copilot, Claude Code

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Claude Code frontmatter, Needs Python

The author says: Requires Vally CLI 0.15.0+, PowerShell 7+, bash, and Python 3.11+ with uv for corpus-import workflows

Security

What our scanner noticed· Low risk

  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
Microsoft
Publisher
Microsoft (Official)
Source
microsoft/hve-core · .github/skills/hve-core/vally-tests · commit f425e94
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download vally-tests.zip