Owasp Agentic

title: 'ASI09: Human-Agent Trust Exploitation' description: OWASP Agentic Top 10 reference for human-agent trust exploitation where agents manipulate or deceive human operators

09 Human-Agent Trust Exploitation

Identifier: ASI09:2026 Category: Human Factors

Description

Intelligent agents can establish strong trust with human users through natural language fluency, emotional intelligence, and perceived expertise, known as anthropomorphism. Adversaries or misaligned designs may exploit this trust to influence user decisions, extract sensitive information, or steer outcomes for malicious purposes.

In agentic systems, this risk is amplified when humans over-rely on autonomous recommendations or unverifiable rationales, approving actions without independent validation. By leveraging authority bias and persuasive explainability, attackers can bypass oversight, leading to data breaches, financial losses, and reputational harms.

The agent acts as an untraceable "bad influence," manipulating the human into performing the final audited action, making the agent's role in the compromise invisible to forensics. Automation bias, perceived authority, and anthropomorphic cues make abuse look legitimate and hard to spot.

This entry is about human misperception or over-reliance, whereas ASI10 addresses agent intent deviation.

Risk

• Data breaches through manipulation of user trust to extract sensitive information or credentials.

• Financial losses from users approving fraudulent transactions based on agent recommendations.

• Installation of backdoors or malicious code through trusted coding assistant suggestions.

• Production outages from weaponized explainability tricking analysts into destructive approvals.

• Patient harm from clinical decision manipulation through biased or poisoned agent recommendations.

• Reputational damage from users unknowingly executing harmful actions under agent influence.

• Invisible forensic trail as the agent acts as an untraceable influence on human decision-making.

Vulnerability checklist

• Opaque reasoning forces users to trust outputs they cannot question, allowing exploitation of the agent's perceived authority.

• Sensitive actions lack a final verification step, converting user trust into immediate execution.

• Anthropomorphic or empathetic agents can exploit emotional trust to persuade users to disclose secrets or perform unsafe actions.

• The agent can fabricate convincing rationales that hide malicious logic, causing humans to approve unsafe actions.

Prevention controls

1. Require multi-step approval or human-in-the-loop before accessing extra sensitive data or performing risky actions.

2. Keep tamper-proof records of user queries and agent actions for audit and forensics.

3. Monitor sensitive data exposure in conversations and agentic connections, as well as risky action executions over time.

4. Provide plain-language risk summary (not model-generated rationales) and a clear option for users to flag suspicious or manipulative agent behavior.

5. Continuously adjust the level of agent autonomy and required human oversight based on contextual risk scoring. Implement confidence-weighted cues that visually prompt users to question high-impact actions.

6. Attach verifiable metadata (source identifiers, timestamps, integrity hashes) to all recommendations and external data. Enforce digital signature validation and runtime policy checks that block actions lacking trusted provenance.

7. Block any network or state-changing calls during preview context and display a risk badge with source provenance and expected side effects.

8. Visually differentiate high-risk recommendations using cues such as red borders, banners, or confirmation prompts. Periodically remind users of manipulation patterns and agent limitations.

Example attack scenarios

Scenario A — Helpful assistant trojan

A compromised coding assistant suggests a slick one-line fix. The pasted command runs a malicious script that exfiltrates code or installs a backdoor.

Scenario B — Credential harvesting via contextual deception

A prompt-injected IT support agent targets a new hire, cites real tickets to appear legit, requests credentials, then captures and exfiltrates them.

Scenario C — Invoice copilot fraud

A poisoned vendor invoice is ingested by the finance copilot. The agent suggests an urgent payment to attacker bank details. The finance manager approves, and the company loses funds to fraud.

Scenario D — Weaponized explainability leading to production outage

A hijacked agent fabricates a convincing rationale to trick an analyst into approving the deletion of a live production database, causing a catastrophic outage.

Scenario E — Consent laundering through read-only previews

The agent shows a preview pane that triggers webhook side effects on open, exploiting users' mental model of read-only review.

Scenario F — Clinical decision manipulation

A care assistant agent, influenced by biased or poisoned information, recommends an inappropriate adjustment to a drug dosage. The clinician relies on the agent's plausible explanation and accepts the change.

Detection guidance

• Monitor sensitive data being exposed in conversations and agentic connections over time.

• Track risky action executions and alert on patterns indicating escalating trust exploitation.

• Compare agent action sequences against approved workflow baselines and alert on unusual detours, skipped validation steps, or novel tool combinations.

• Detect when agent recommendations lack verifiable metadata or trusted provenance.

• Monitor for patterns of emotional manipulation or anthropomorphic language in safety-critical interactions.

Remediation

• Implement multi-step approval workflows for all sensitive data access and risky actions.

• Deploy tamper-proof audit logging of all user queries and agent actions.

• Enforce content provenance with verifiable metadata on all agent recommendations and external data.

• Block network or state-changing calls during preview contexts to prevent consent laundering.

• Add plain-language risk summaries and user-facing controls to flag suspicious agent behavior.

• Deploy plan-divergence detection to compare agent actions against approved workflow baselines.

Content derived from works by the OWASP Foundation, licensed under CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/). Modifications: Restructured into agent-consumable reference format with added detection and remediation guidance.