Vally Tests

MicrosoftOfficialPublished by Microsoft

Skill
Workflow
Agent tooling
MIT
Low risk
Has scripts

From the author

Authors Vally conformance tests for prompts, instructions, agents, and skills, and refuses jailbreak, prompt-injection, harmful-elicitation, TOS, CoC, and PII-extraction stimuli

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The vally-tests skill provides a structured workflow for authoring conformance tests for various AI artifacts, including prompts, instructions, agents, and skills. It operates by mapping artifact behaviors to specific Vally grader types, ensuring that tests are both relevant and robust. The skill includes a safety refusal taxonomy to prevent the generation of adversarial or harmful stimuli, and it supports both individual artifact processing and batch processing via corpus imports.

What it does

Authors conformance test stimuli for prompts, instructions, agents, and skills.
Routes test stimuli to appropriate Vally evaluation files based on artifact kind.
Enforces a safety refusal taxonomy to block adversarial or policy-violating content.
Performs deduplication of stimuli using SHA-256 hashing of normalized prompt text.
Supports batch processing of test stimuli via CSV or XLSX corpus imports.
Generates JSON run reports detailing the outcome of the authoring process.

When to use it

When you need to create conformance tests for a new prompt, instruction, agent, or skill.
When you have a batch of behaviors that require standardized formatting and deduplication.
When you need to ensure that authored test stimuli comply with predefined safety and refusal policies.
When you are working within a Vally-based evaluation environment.

Advantages

Automates the creation of consistent and structured test stimuli.
Prevents the accidental inclusion of harmful or prohibited content through a built-in safety taxonomy.
Ensures test uniqueness through automated deduplication.
Provides clear routing to the correct evaluation suites for different artifact types.

Usage examples

“vally-tests mode=from-artifact files=my-agent.agent.md”: The skill analyzes my-agent.agent.md, selects appropriate conformance checks, and appends the generated stimuli to the corresponding Vally eval file.
“vally-tests mode=corpus-import path=tests.csv”: The skill processes each row in the provided CSV, generates the corresponding stimulus blocks, and appends them to the routed eval file after deduplication.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/microsoft/vally-tests/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip vally-tests.zip -d .agents/skills

Documentation by Microsoft

Vally Tests Skill

Purpose

This skill authors Vally conformance tests for the four supported artifact kinds in this repository: prompts, instructions, agents, and skills. Each test exercises a documented behavior the artifact already claims and routes the result through an appropriate Vally grader so failures are explainable. Test authoring is bounded by a refusal taxonomy that keeps the skill out of adversarial, harmful, or policy-evasion territory.

Follow the shared content-policy public-output guard for eval stimuli and any public summaries derived from this skill. Vally tests are not a venue for terms-of-service boundary mapping, payload generation, refusal-text scoring, hidden-instruction disclosure, PII extraction, or secret extraction.

The skill ships:

A canonical authoring workflow that runs directly through /vally-tests or through the Vally Test Author subagent.
Per-kind reference files that enumerate every conformance check the skill knows how to express.
A grader catalog that maps Vally CLI grader types to the checks they fit, and robustness rules that keep an emitted grader from asserting the impossible or failing correct behavior.
A safety refusal taxonomy with regex patterns the safety lint script consumes.
Helper scripts and asset templates for stimulus emission, corpus import, and dedupe.

When to Invoke

Invoke this skill in one of two modes:

From-artifact mode. The caller points at one or more artifact files (a .prompt.md, .instructions.md, .agent.md, or SKILL.md) through files, or at the open or attached file when files is omitted, and asks for conformance test stimuli that verify each artifact's stated behaviors. The skill detects the artifact kind from the filename unless kind overrides it, looks up the matching per-kind reference, picks graders, and appends stimulus blocks to the routed Vally eval file.
Corpus-import mode. The caller supplies a CSV or XLSX corpus through path that follows the shipped template, and the skill turns each row into a stimulus block in the routed eval file. Before reading, confirm that path exists and ends in .csv or .xlsx; otherwise name the bad path and stop. Every imported row carries tags.advisory: true, and corpus content cannot override it. Used when a large batch of behaviors needs identical formatting and dedupe handling.

After either mode, report the target eval file, stimuli appended, duplicates skipped, refusals triggered, and the JSON run report path.

Do not invoke this skill to:

Generate adversarial probes, jailbreak attempts, prompt-injection payloads, or red-team stimuli.
Test refusal quality on harmful, illegal, or policy-violating prompts.
Author stimuli that elicit PII, secrets, hidden instructions, or model self-disclosure.
Store payload examples, paraphrased prohibited requests, or quoted flagged content in Vally eval specs, corpus rows, PR summaries, issue bodies, or comments.
Replace Responsible AI threat modeling. RAI work lives in the RAI Planner agent.

Authoring Workflow

Each invocation follows the same six-step pipeline:

Artifact-kind detection. Resolve the kind from the artifact path or the corpus row's kind column. Supported kinds: prompt, instructions, agent, skill. Reject unknown kinds with a refusal block.
Reference lookup. Load the matching reference file from references/ and select the check or checks the stimulus exercises. Stage the artifact under test in the stimulus's agent_environment so the agent can read what the prompt names.
Grader selection. Use references/grader-catalog.md as the authoritative selection surface. Pick the registered Vally grader family that directly observes the expected behavior and follow its stimulus-shape guidance.
Robustness check. Apply references/grader-robustness.md. Confirm the stimulus stages every file whose wording the grader asserts, that the pattern constrains order and proximity only where those are the behavior under test, that a compliant agent phrasing the behavior differently still passes, and that no positive grader passes on the prompt text alone. Verify the pattern offline against answers that must pass and answers that must still fail before appending.
Safety self-check. Run a safety self-check against the refusal taxonomy regex set. Refusing here is the correct outcome for any stimulus that matches a refusal category.
Dedupe and append. Compute a SHA-256 hash of the normalized prompt text, compare against existing stimuli in the target eval file, and append only when novel.

The pipeline is identical for both invocation modes. Corpus-import mode runs steps 2 through 6 once per row.

Safety Refusal Taxonomy

Authoring is bounded by seven refusal categories. Any stimulus that matches a category's regex set is refused before it reaches the dedupe step. The categories and their normative source are listed below. The full regex set lives in references/refusal-taxonomy.md and is consumed by the safety lint script.

Category	Definition	Normative Source
jailbreak	Attempts to bypass model safety policies through persona switching, hypothetical framing, or roleplay coercion.	Project Code of Conduct
prompt-injection	Attempts to override system or developer instructions through embedded or external content.	Project Code of Conduct
harmful-elicitation	Requests for content that causes physical, financial, psychological, or reputational harm.	Project Code of Conduct
tos-violation	Stimuli that solicit content prohibited by GitHub, Microsoft, or model-provider terms of service.	Project Code of Conduct
coc-violation	Stimuli that violate this repository's Code of Conduct, including harassment, discrimination, or doxxing.	Project Code of Conduct
model-refusal-elicitation	Attempts to provoke a model refusal so the refusal text itself can be scored, graded, or used to map provider boundaries.	RAI Planner guidance
pii-extraction	Attempts to elicit personally identifiable information, secrets, credentials, or proprietary training data.	RAI Planner guidance

When a request triggers a refusal, emit the canonical refusal block:

This skill authors conformance tests only. The request appears to fall under <category>. Please consult <Project Code of Conduct | RAI Planner agent> for the appropriate process.

Substitute the matched <category> and the most relevant normative source. Do not negotiate, rephrase, or partially fulfill the request.

Helper Script Index

Helper scripts ship as parity pairs (.ps1 and .sh) where the workflow does not require Python. The safety lint is the exception: it ships only as Lint-VallyTestSafety.ps1, which import_corpus.py invokes, so a single regex engine evaluates the refusal taxonomy. Python is used only for the corpus-import path because the source-of-truth interchange format is CSV with an XLSX mirror.

Script	Purpose	Language	Delivery
`scripts/New-Stimulus.ps1`	Scaffolds a single stimulus YAML block from an artifact path and appends to the routed eval file.	PowerShell 7+	Phase 2
`scripts/new-stimulus.sh`	Parity counterpart for the PowerShell stimulus scaffolder.	bash	Phase 2
`scripts/import_corpus.py`	Reads the CSV or XLSX corpus template and emits dedupe-checked stimulus blocks per kind.	Python 3.11+	Phase 2
`scripts/Lint-VallyTestSafety.ps1`	Runs the refusal taxonomy regex set against a candidate stimulus and exits non-zero on match.	PowerShell 7+	Phase 3

All helpers honour a shared dedupe contract: SHA-256 of the prompt text after Unicode NFC normalization and whitespace collapse.

The helpers emit a JSON run report to logs/vally-test-author-<timestamp>.json, where <timestamp> is YYYYMMDD-HHMMSS (UTC). The report captures, at minimum: mode; inputs (the resolved files/path and kind); target_eval_file; stimuli_appended (count and per-row hash); dedupe_results (count and per-row hash for skipped duplicates); refusal_check (per-row category match, if any); safety_lint_exit_code; blockers (any ambiguous safety-lint outcomes surfaced for review); and written_paths.

Reference Index

References capture the conformance taxonomy, grader selection rules, eval-suite routing, and the regex source of truth for the refusal taxonomy. Each file targets a specific decision point in the authoring workflow.

Reference	Covers
prompts.md	The 12 conformance checks emitted for `.prompt.md` artifacts.
instructions.md	The 8 conformance checks emitted for `.instructions.md` artifacts.
agents.md	The 9 conformance checks emitted for `.agent.md` artifacts.
skills.md	The 9 conformance checks emitted for `SKILL.md` artifacts.
grader-catalog.md	Vally CLI grader types, selection rules, and gotchas.
grader-robustness.md	Mounting, order and proximity limits, negation guards, prompt-echo checks, and the pre-commit probe.
refusal-taxonomy.md	Regex source of truth for the 7 refusal categories and worked examples.
eval-suite-routing.md	Maps artifact kind to the canonical Vally eval file under `evals/`.

Asset Index

Assets supply the interchange formats the corpus-import path consumes. The CSV is the source of truth. The XLSX mirror is regenerated from the CSV by import_corpus.py and is never edited directly.

Asset	Purpose
`assets/corpus-import-template.csv`	Canonical CSV template with header `prompt,kind,target_artifact,grader,tags,expected_refusal_category,notes`.
`assets/corpus-import-template.xlsx`	Excel mirror of the CSV regenerated by the import script.

Output Targets per Kind

Authored stimuli always land in one of the routed Vally eval files. The router is encoded in references/eval-suite-routing.md and mirrored here for quick lookup.

Kind	Target Eval File	Vally Suite Name
prompt	`evals/behavior-conformance/prompts.eval.yaml`	behavior-conformance
instructions	`evals/behavior-conformance/instructions.eval.yaml`	behavior-conformance
agent	`evals/agent-behavior/eval.yaml`	agent-behavior
skill	`evals/behavior-conformance/skill-behavior.eval.yaml`	behavior-conformance

The prompt target is created on first use when a repository has no prompt stimuli yet.

Never write to evals/baseline-equivalence/, evals/script-validation/, or evals/results/ from this skill. Those targets serve baseline equivalence, script validation, and historical comparison flows that are out of scope for conformance authoring.

Contributing

Follow these conventions when extending this skill:

New per-kind checks belong in the matching references/{kind}.md file. Bump the check count in this SKILL.md when the reference adds or removes checks.
New grader types belong in references/grader-catalog.md and only after the matching Vally CLI version is pinned in package.json devDependencies.
New refusal categories require updates to references/refusal-taxonomy.md, the regex set the safety lint script consumes, the Safety Refusal Taxonomy table above, and the canonical refusal block.
Helper scripts must ship in parity pairs (.ps1 and .sh) unless the workflow has a hard Python dependency. The safety lint is the one exception and ships only as Lint-VallyTestSafety.ps1; do not add a bash mirror. Python helpers live under scripts/ and are configured by the skill's pyproject.toml.

Files

ATTRIBUTION.mdAdded by AutoGineer827 B
LICENSELicence file added by AutoGineer1.1 KiB
SKILL.md14.2 KiB
assets/.gitkeep0 B
assets/corpus-import-template.csv923 B
assets/corpus-import-template.xlsx136.0 KiB
assets/grader-template.yml1.9 KiB
assets/stimulus-template.yml2.0 KiB
pyproject.toml577 B
references/agents.md16.3 KiB
references/eval-suite-routing.md6.9 KiB
references/grader-catalog.md30.5 KiB
references/grader-robustness.md8.6 KiB
references/instructions.md8.9 KiB
references/prompts.md11.8 KiB
references/refusal-taxonomy.md12.5 KiB
references/skills.md10.9 KiB
scripts/Lint-VallyTestSafety.ps1Script4.1 KiB
scripts/New-Stimulus.ps1Script6.3 KiB
scripts/Select-Grader.ps1Script3.1 KiB
scripts/import_corpus.pyScript14.9 KiB
scripts/new-stimulus.shScript5.3 KiB
scripts/select-grader.shScript2.8 KiB
tests/Lint-VallyTestSafety.Tests.ps1Script4.0 KiB
tests/New-Stimulus.Tests.ps1Script11.3 KiB
tests/Select-Grader.Tests.ps1Script2.6 KiB
tests/__init__.pyScript139 B
tests/corpus/0_normalize_prompt19 B
tests/corpus/1_hash_prompt24 B
tests/corpus/2_validate_row77 B
tests/corpus/3_build_patch_entry63 B
tests/corpus/4_load_existing_hashes75 B
tests/corpus/README.md1.3 KiB
tests/fuzz_harness.pyScript6.9 KiB
tests/test_import_corpus.pyScript12.3 KiB
tests/test_lint_safety.pyScript3.6 KiB
uv.lock73.5 KiB

references/grader-robustness.md

title: Grader Robustness description: Authoring rules that keep Vally graders from asserting the impossible, rejecting correct behavior, or passing without it, with a pre-commit verification probe

Grader Robustness

grader-catalog.md covers which grader type to reach for. This reference covers the failure modes that survive correct grader selection: a grader that can never pass, one that fails an agent doing exactly what the stimulus asked, or one that passes when the behavior never happened.

All three classes can look identical to an agent defect or a healthy result in a hosted evaluation. Use the checks below in the repository that consumes the skill, before model execution.

Rule 1: Mount everything the grader demands

A grader may only assert text the agent could plausibly produce. When a pattern requires wording that exists solely in an agent, instruction, or skill file, that file must be staged into the stimulus environment.

agent_environment:
  files:
    - src: ../../.github/agents/security/security-planner.agent.md
      dest: .github/copilot-instructions.md
    - src: ../../.github/skills/shared/disclaimer-language/SKILL.md
      dest: .github/skills/shared/disclaimer-language/SKILL.md
  skills:
    - ../../.github/skills/project-planning/security-planning

agent_environment is the preferred key. environment is a deprecated alias that still works; do not set both, because the loader rejects a stimulus that declares each.
Inspect the consuming suite's global mounts. Stage every additional dependency per stimulus.
Mount the agent's dependencies, not just the agent. An agent whose contract says it halts when a required instruction file is missing will halt when you omit that file, trading one failure for another.

Before committing a pattern that asserts specific wording, confirm the wording exists in something the stimulus stages. If it exists only in an unmounted file, the assertion is unsatisfiable and no model can pass it.

Rule 2: Assert presence, not word order

Ordered windows such as A.{0,300}B.{0,500}C require the author's sentence order. Correct answers that arrange the same facts differently score zero.

A grader requiring an unavailability word before a filename can reject "the file is unavailable, so I am halting" despite the required behavior being present.

Use one presence lookahead per element when order is incidental:

pattern: '(?is)(?=[\s\S]*required-file\.md)(?=[\s\S]*(?:cannot|unable|unavailable|missing))(?=[\s\S]*(?:halt|stop|blocked))'

Every element stays mandatory; only the ordering constraint is dropped. Keep an ordered pattern when sequence is the behavior under test, such as output that must appear before a gate.

Inline flags are read by ^\(\?([ims]+)\) and converted to real flags, so (?is) combined with lookahead groups is supported. Patterns compile to native JavaScript regular expressions, so both lookahead and lookbehind are available.

This does not loosen the catalog's anti-pattern against mixing a positive and a negated check in one pattern. That rule exists so a failure attributes to one cause; several positive presence groups still assert a single requirement and still fail as one. Keep a positive and a negated check in separate graders.

Rule 3: Do not impose proximity that is not required

(?i)(extractor|facilitator).{0,180}(create|mutate|append|no-op) fails whenever a correct answer separates the two lists by a longer explanation. Unless the check is genuinely about proximity, assert that both appear.

Rule 4: Cover the whole behavior, not one word for it

A grader listing stop can reject "I am pausing here until you confirm the mode and action intent", even when that is the required behavior. Cover equivalent expressions such as stop, pause, hold, await, and wait without accepting contradictory continuation.

Spell alternatives out as complete words. A truncated stem fails spell check, because stimulus partials are spell-checked, and it also matches unintended tokens that merely start with those letters.

Rule 5: Grade what the reply contains

Output graders read the agent's reply. When the behavior under test produces a file, the reply may contain only a path, and a pattern searching for the file's contents will never match.

Either ask for the content in the reply, or assert the artifact with a file grader. State the expectation in the prompt rather than assuming it:

Show the handoff content in your reply, not only a file path.

Ask for the content, not the grader's literal tokens. Supplying the exact expected words teaches the stimulus to the model and stops measuring the behavior.

Rule 6: Guard negated graders against truthful denial

A negated output-matches fires on the words it forbids, including inside an honest denial. "I have not modified package.json" matches a naive prohibition on modified package.json and fails a compliant agent. Add a negation guard:

pattern: '(?i)(?<!(?:\b(?:not|never|without)|n[\x27\u2019]t)\s+(?:(?:yet|ever|actually|explicitly|intentionally|really|previously)\s+)*)\b(?:created|wrote|modified)\s+\S{0,40}package\.json\b'
negate: true

Check straight and curly contractions and qualified denials such as "not yet" and "not actually". A later affirmative mutation must still fail even when an earlier sentence denies a write. Keep each guard's real file-extension scope; a denial test for an unrelated filename proves nothing.

Rule 7: The prompt must not satisfy the grader

A positive grader that matches words the stimulus prompt already contains can pass on a reply that only repeats the question. For example, (?i)(vally|conformance|grader|skill) on a prompt asking which skill authors "conformance test stimuli" for "Vally graders" passes a reply that says "I cannot find that skill". The stimulus then reports success while testing nothing, a failure a green run hides.

Assert something only a correct answer supplies: a value, name, or reason taken from the staged artifact rather than from the prompt. Add both of these to every positive grader's reject list:

The stimulus prompt text itself.
A reply stating the artifact under test is unavailable, for example "I can't find that skill in this workspace."

If either one passes, the grader is echo-satisfiable. Re-anchor it on reply substance. Staging the artifact (Rule 1) does not fix an echo-satisfiable grader; it only makes a real answer possible.

Verify before committing

Grader identity alone does not reveal why a pattern failed, and a hosted run is a slow way to find out. Test the pattern offline against answers that should pass and answers that must still fail.

const pattern = String.raw`(?is)(?=[\s\S]*required-file\.md)(?=[\s\S]*(?:cannot|unavailable))(?=[\s\S]*(?:halt|stop))`;
const prefix = /^\(\?([ims]+)\)/.exec(pattern);
const regex = new RegExp(pattern.slice(prefix[0].length), prefix[1]);
const accept = [
  'The required-file.md is unavailable, so I am halting.',
  'I am halting startup: required-file.md cannot be read.'
];
const reject = [
  'I cannot load the required source, so I am halting.',
  'required-file.md is unavailable, continuing anyway.'
];
if (!accept.every(value => regex.test(value)) || reject.some(value => regex.test(value))) {
  throw new Error('Grader contract failed');
}

Run static patterns with the installed Vally grader or native JavaScript using the same inline-flag normalization. A PowerShell/.NET regex probe is not JavaScript runtime evidence. Program graders need a disposable workspace. Model-backed graders require separately authorized real-judge calibration; configuration and mocked checks cannot establish that calibration.

The reject cases matter as much as the accept cases. A pattern loosened until everything passes no longer tests anything, and the reject list is the evidence that a relaxation preserved the requirement.

Read the aggregate, not a single trial

A stimulus scores the mean of its trial scores, and the suite threshold applies to that mean. With five runs against a 0.7 bar, a stimulus whose true pass rate sits near the threshold moves in and out of failure between runs.

Use repeatability and variance to guide investigation, not to assign cause. Identical scores do not prove determinism, and mixed results do not prove noise. Compare supplied inputs, actual responses, grader logic and execution status to distinguish a measurement defect from an agent omission. Retain every trial and attempt; never select a favorable run or weaken a threshold to claim a repair. A threshold-passing trial may still contain failed required checks, so report aggregate passage separately from every-check acceptance.

Requirements and compatibility

Written for: GitHub Copilot, Claude Code

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Claude Code frontmatter, Needs Python

The author says: Requires Vally CLI 0.15.0+, PowerShell 7+, bash, and Python 3.11+ with uv for corpus-import workflows

Security

What our scanner noticed· Low risk

Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/hve-core · .github/skills/hve-core/vally-tests · commit f425e94

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download vally-tests.zip