Applies when your product sends data to a language model or lets a model take actions. Organised around the OWASP Top 10 for LLM Applications (2025 edition at the time of writing).
Core idea
Section titled “Core idea”Treat the model as an untrusted component. Anything that reaches it can influence it, and anything it produces is untrusted input to the rest of your system.
Prompt injection
Section titled “Prompt injection”- Assume any text the model reads (user input, web pages, documents, emails, tool results, retrieved chunks) may contain instructions trying to override yours.
- Keep instructions and untrusted data clearly separated in the prompt and label the data as data.
- Security MUST NOT depend on the model obeying instructions. Enforce limits in code: permissions, allowlists, schema validation, human approval.
Output handling
Section titled “Output handling”- Validate model output against a schema before using it.
- Never pass model output directly into SQL, shell commands,
eval, file paths, HTML without encoding, or URLs that are fetched server-side. - Render model-generated HTML or Markdown through a sanitiser.
Sensitive data
Section titled “Sensitive data”- Send the model only the data the task needs. Strip secrets and unneeded personal data before the call.
- Know and document the provider’s data-retention and training-use terms for your plan.
- Never put API keys, internal URLs or other users’ data in prompts.
Agency and tools
Section titled “Agency and tools”- Give tools the least privilege possible: read-only where possible, scoped to the current user, with allowlisted parameters.
- Actions that spend money, send messages, delete data or change permissions MUST require explicit user confirmation.
- Log every tool call with user, parameters and result for audit.
System prompt
Section titled “System prompt”- Assume the system prompt can be extracted. Never put secrets or security logic in it.
Retrieval (RAG)
Section titled “Retrieval (RAG)”- Enforce access control at retrieval time: a user can only retrieve documents they may already see.
- Track the source of every indexed document; be able to remove one.
Misinformation
Section titled “Misinformation”- When output is presented as fact, ground it in retrieved sources and show citations; say clearly when the model is unsure.
- Label AI-generated content where users would reasonably want to know.
Unbounded consumption
Section titled “Unbounded consumption”- Per-user and global limits on requests, tokens and spend; alerts on unusual cost.
- Limit input size and maximum output tokens.
Checklist
Section titled “Checklist”- Untrusted content separated and labelled
- Output schema-validated and encoded before use
- Tools least-privilege; risky actions need confirmation
- No secrets in prompts or system prompt
- Retrieval respects user permissions
- Rate, token and cost limits with alerts