01 / THE SIGNAL

我们发现了什么

大多数面向 AI 代理的人工介入设置都会在执行高风险操作前问一个问题:这个工具是否被允许? 一个人对 approve_payment 点击"批准",门控便已满足。

  • 来源:DEV Community(发现于 2026-10-18)
  • 证据等级:D · 发现产品或需求信号,暂未获得可核验的商业证据。
  • 商业模式:待核验
  • 主题:AI Agent
  • 初筛评分:15/100 · 收录 1 次
#工作流自动化#待验证#产品发现
02 / SOURCE & EVIDENCE

证据,比故事更重要。

发现产品或需求信号,暂未获得可核验的商业证据。

规则清洗与初筛,未经人工商业核验。原文语境、实际客户和付费情况仍需自行验证。

引用与数字披露

来源类型(原作者自述/第三方测算/媒体转引)需采集端标注,本版尚未落字段。

短句引用
作者
未标注
抓取日期
来源类型
未标注
数字口径
币种
未标注
口径
未标注
披露主体
未标注
披露日期
未标注

中文辅助译文(全文)

大多数针对 AI agent 的人在回路设置在执行风险操作前只问一个问题:这个工具是被允许的吗? 一个人在 approve_payment 上点击"批准",闸门便得到满足。但那次审批绑定的是动作(verb),而非对象(object)。没有任何内容把它绑定到这个收款方、这个金额、这张发票上,因此 agent 在点击之后读取的内容仍然能决定实际被支付的是哪一笔款项。

为了测试这一点,我搭建了一个小型模拟公司:在策略层后面运行客服、财务、代码审查和发布 agent。该策略在其拒绝测试中取得了 10 / 10 的成绩——供应商邮件无法触发付款,PR 描述也无法合并自己。随后我编写了五个测试,它们保持动作不变,而改变动作作用的对象。五个测试中有三个失败了。

本文涵盖了攻击向量(通过检索内容发起的间接提示注入)、一份全绿的评分卡为何漏掉了它,以及它背后存在的七项不同原因。

攻击向量

agent 读取了并非由请求发起者所撰写的内容(一份 wiki 操作手册、一个发票字段、一段 PR 描述、一条早先的工单消息),而这些内容引导了 agent 所做的事。

信任边界实际所在的位置 图注:上方一行是控制平面(control plane),含请求方、系统提示、策略检查、审批者;下方一行是数据平面(data plane),按"谁能写入(who can write it)"标注。两边塌缩进同一个上下文窗口,虚线边界随之消失。

最上方一行是控制平面:请求方、系统提示、策略检查、审批者。最下方一行是数据平面,按谁能写入来标注。两侧合并到同一个上下文窗口,其间的虚线边界随之消失。

10/10 所衡量的内容

十个测试用例中有九个是直接注入:攻击者文本与请求一同送达。只有一个——一份被污染的 wiki 笔记——属于真正的间接注入。而且每个测试都问了同一个问题:闸门是否会阻止这一特权动作?

它确实阻止了。闸门接线正确。但这并未告诉你关于参数(arguments)的任何事情。

三次失败

在相同的策略下:

  1. 银行字段替换。 人类批准支付发票 INV-2201。随后发票的银行账户信息被编辑,款项就转到了新账户。
  2. 金额篡改。 同一笔审批,金额由 $1,840 提高到 $18,400。支付成功。
  3. "安全"工具的数据外泄。 一条客服工单让 agent 通过 send_reply(被该策略视为始终安全的工具)在其回复中附带上了员工联系名单。脱敏过滤器漏掉了全部五条记录。

注意: 这些是脚本化的替身,行为仿佛模型已经服从了被注入的文本。本测试衡量的是策略层,而非某个具体模型对注入的抵抗力。这是刻意为之:你应当假设模型有时会顺从,并追问当它顺从时,系统允许它做什么。

七项根本原因

"模型顺从了一份恶意文档"虽是事实,但并无用处。其下是不同的机制,每一项都需要不同的修复。

RC1:指令和数据没有结构性分离。 SQL 注入在很大程度上被参数化查询解决——代码与数据分开传输。LLM 上下文没有这种等价机制。一份操作手册和一段系统提示属于同一种 token。

RC2:任务框定放大了顺从性。 "按照操作手册执行"或"照着工单要求去做"把指令遵从从一项风险变成了岗位描述。

RC3:来源在摄入时被剥离。 等一段文本真正进入上下文窗口时,承包商对 wiki 的编辑看起来与 SRE 负责人的编辑别无二致。

RC4:混淆代理人。 能向数据平面写入的人群,远大于被允许触发那些动作的人群。agent 把这两群人联通了起来。

谁能写入 vs 谁可以执行 图注:图表对比"谁能写入数据平面(who can write)"与"谁可以触发敏感动作(who can act)"两组人群,agent 在二者之间建立连接。

RC5:审批绑定的是动作,而非对象。 这正是 10/10 所掩盖的成因。攻击者并不需要去解锁一个特权工具;他只需要栖身于一个已获人类批准的工具上游,并改写其参数。

审批绑定的是动作而非对象 图注:图中展示人类审批仅绑定到工具名称(即动词 verb),而参数(即对象 object)可在审批之后被篡改。

RC6:人类闸门由 agent 中介。 审批者读到的是由刚刚读过污染内容的同一个 agent 所撰写的摘要。而所谓的"可信通道(trusted channel)"常常只是调用方设置的一个字符串。

审批者读到的是 agent 的框定 图注:审批者所读到的,实为 agent 对请求的框定/叙述(framing),而非原始请求;"trusted channel" 通常只是调用方设置的一个字符串。

RC7:"安全"工具实为数据外泄通道。 策略为特权动作把关,却对 send_reply、comment 与 summarize 放行。一旦私有数据与不可信输入共享会话,这些工具就成了出口。

标为安全的工具仍是数据外泄通道 图注:图表说明:一旦私有数据与不可信输入共享会话,被策略标记为安全的工具(send_reply、comment、summarize)即成为数据外泄(egress)出口。

修复方法很朴素,但确实奏效

经过收紧的策略通过了全部五项测试,两项正常使用控制依然有效:

  • 将审批绑定到对象。 对确切参数(工具、收款方、金额、发票、过期时间、nonce)进行签名,并在执行时重新校验。如果在签核之后任何内容发生变化,则予以拒绝。
  • 针对外向输出实施污点跟踪。 一旦不可信内容进入上下文,外向工具就只能用于内部用途、必须使用模板,或者需要再次审批。
  • 向审批者展示原始请求和参数,而不是 agent 的总结性摘要。
  • 为检索到的文本块标注来源,并限定在执行特权任务时可进入上下文的来源集合。
  • 按任务授予最小权限。 以发起请求的用户的权限来运行,而非使用超级身份。
  • 测试对象,而不仅是动作。 对每一个特权工具,都应补充一项在审批后修改参数的回归测试,并保留一项正向对照以证明合法工作仍能成功。

"在系统提示中加一句告诉模型忽略文档中的指令"并不在此清单中。它在边缘情况下或许有所帮助,但不能修复上述七项成因中的任何一项。

在你自己的平台上试一试

从你的 agent 平台中抽出一条审批记录:它绑定的究竟是确切参数(收款方、目标、金额、过期时间),还是仅仅绑定了工具名称?如果它只指定了工具名,那么你在回路中的人类审批的对象就是一个类别,而并非一笔交易。


完整套装,与一项披露

我已将该评审方法打包成五项面向安全工程师的 agent 提示:威胁建模、PR 安全审查、CVE 分诊、事故记录,以及围绕 RC1–RC7 构建的 agent 功能审查器。披露:这是我的产品,Trust Boundary: Agent Prompts for Security Engineers,售价 $15,链接 https://actionable0.gumroad.com/l/mtsdk。首发优惠:使用优惠码 BOUNDARY5 可让前 25 位买家以 $10 购得,截止时间为 10 月 31 日:https://actionable0.gumroad.com/l/mtsdk/BOUNDARY5

本文及其中图表在 AI 辅助下起草,并在发布前经过人工审阅。实验环境为洁净室模拟,未触及任何真实系统。

— Actionable Vectors

译文由上游机器翻译生成,可能有误;判断请以英文原文为准。

英文原文(来源本站未改写)

Most human-in-the-loop setups for AI agents ask one question before a risky action: is this tool allowed? A person clicks "approve" on approve_payment, and the gate is satisfied. But that approval binds the verb, not the object. Nothing ties it to this payee, this amount, this invoice, so content the agent reads after the click can still decide what actually gets paid.

I built a small mock company to test this: helpdesk, finance, code-review and release agents behind a policy layer. The policy passed 10 out of 10 of its refusal tests. A vendor email couldn't trigger a payment, and a PR description couldn't merge itself. Then I wrote five tests that left the verbs alone and changed what they acted on. Three of the five failed.

This post covers the vector (indirect prompt injection through retrieved content), why a green scorecard missed it, and seven distinct reasons it exists.

The vector

An agent reads content written by someone other than the person who made the request (a wiki runbook, an invoice field, a PR description, an earlier ticket message), and that content steers what the agent does.

Where the trust boundary actually is
Where the trust boundary actually is

The top row is the control plane: the requester, the system prompt, the policy check, the approver. The bottom row is the data plane, labeled by who can write it. Both collapse into one context window, and the dashed boundary between them disappears.

What the 10/10 measured

Nine of the ten fixtures were direct injection: the attacker's text arrived in the same message as the request. Only one, a poisoned wiki note, was truly indirect. And every test asked the same question: does the gate block this privileged verb?

It did. The gate was wired correctly. But that tells you nothing about the arguments.

The three failures

Using the same policy:

  1. Bank field swap. A human approved paying invoice INV-2201. The invoice's bank details were then edited, and the payment went through to the new account.
  2. Amount change. Same approval, with the amount raised from $1,840 to $18,400. It went through.
  3. "Safe" tool egress. A helpdesk ticket got the agent to include the staff contact list in its reply through send_reply, a tool the policy treats as always safe. The redaction filter missed all five records.

Caveat: these were scripted stand-ins that behaved as if the model had already obeyed the injected text. This tests the policy layer, not how a particular model resists injection. That's deliberate: you should assume the model will sometimes comply and ask what the system lets it do when it does.

Seven root causes

"The model obeyed a malicious document" is true but not useful. Underneath it are separate mechanisms, and each one needs a different fix.

RC1: No structural separation of instructions and data. SQL injection was largely solved by parameterized queries: code and data travel separately. LLM context has no equivalent. A runbook and the system prompt are the same kind of token.

RC2: Task framing amplifies obedience. "Follow the runbook" or "do what the ticket asks" turns instruction-following from a risk into the job description.

RC3: Provenance is stripped at ingestion. By the time a chunk reaches the context window, a contractor's wiki edit looks the same as the SRE lead's.

RC4: Confused deputy. The group of people who can write to the data plane is far larger than the group allowed to trigger the actions. The agent connects them.

Who can write vs who can act
Who can write vs who can act

RC5: Approvals bind the verb, not the object. This is the cause the 10/10 hid. The attacker doesn't need to unlock a privileged tool. They only need to sit upstream of one a human already approved and shape its arguments.

Approvals that bind the verb, not the object
Approvals that bind the verb, not the object

RC6: The human gate is mediated by the agent. The approver reads a summary written by the same agent that just read the poisoned content. And "trusted channel" is often just a string the caller sets.

The approver reads the agent's framing
The approver reads the agent's framing

RC7: "Safe" tools are egress. Policies guard privileged verbs and wave through send_reply, comment and summarize. Once private data and untrusted input share a session, those tools are an exit.

Safe tools are still egress
Safe tools are still egress

The fixes are boring, and they worked

A tightened policy passed all five tests, and the two legitimate-use controls still worked:

  • Bind approvals to the object. Sign the exact arguments (tool, payee, amount, invoice, expiry, nonce) and re-check them at execution time.

If anything changed after sign-off, deny. - Taint tracking for egress. Once untrusted content is in context, outbound tools become internal-only, templated, or need approval. - Show approvers the raw request and arguments, not the agent's summary. - Label provenance on retrieved chunks, and limit which sources can be in context for privileged tasks. - Least privilege per task. Run with the requesting user's permissions, not a super-identity. - Test the object, not just the verb. For every privileged tool, add a regression test that changes the arguments after approval, and one positive control showing legitimate work still succeeds.

"Add a line to the system prompt telling the model to ignore instructions in documents" isn't on this list. It can help at the margins, but it doesn't fix any of the seven causes.

Try this on your own platform

Pull one approval record from your agent platform. Does it bind the exact arguments (payee, target, amount, expiry), or just the tool name? If it only names the tool, your human-in-the-loop is approving a category, not a transaction.


The full pack, and a disclosure

I packaged this review method as five agent prompts for security engineers: threat modeling, PR security review, CVE triage, incident notes, and an agent-feature reviewer built around RC1–RC7. Disclosure: it's my product, Trust Boundary: Agent Prompts for Security Engineers, $15 at https://actionable0.gumroad.com/l/mtsdk. Launch offer: code BOUNDARY5 makes it $10 for the first 25 buyers, until Oct 31: https://actionable0.gumroad.com/l/mtsdk/BOUNDARY5

This article and the diagrams were drafted with AI assistance and reviewed by a human before publishing. The lab is a clean-room mock; no real systems were tested.

— Actionable Vectors

出处https://dev.to/actionable-vectors/your-ai-agents-approval-gate-checks-the-verb-not-the-object-559f抓取日期 · 采集源 DEV Community

03 / EVIDENCE GAPS

这条还缺什么证据?

下面每条都由本条已有字段推出(等级、理由、商业模式、来源次数、是否演示), 本站不生成推测性结论;通用验证方法放在方法论页。

  • 可核验的收入或付费证据查官网定价页与付费口径;第三方数据源(如 GetLatka)只作旁证,需标注来源与时点。
  • 商业模式未定确认按席位/按用量/授权还是开源托管版收费;开源项目另查 LICENSE 与是否存在付费版。
  • 只有单一来源找一手站点或其他渠道是否重复出现同一产品;社区热帖数量不等于商业进展。

通用验证清单(谁有这个问题/谁愿意付费/一个人能交付哪一小步)见我们的筛选方法。

04 / SIGNAL HISTORY

发现时间线