Skip to main content

What is it?

Insecure output handling (also called Improper Output Handling) is a vulnerability that happens when an application trusts LLM output too much and passes it directly into other systems without proper validation, sanitisation, or encoding.  

Because an LLM’s output can be influenced by user prompts (including prompt injection), treating the model’s response as safe can be similar to giving a user indirect access to functions they should not control.  

If exploited, insecure output handling can lead to classic security issues such as: 

  • Cross-site scripting (XSS) and CSRF in browsers 
  • Server-side request forgery (SSRF) 
  • Privilege escalation 
  • Remote code execution (RCE) on backend systems 

This risk is not the same as overreliance. Overreliance is about people trusting accuracy, insecure output handling is about technical handling of outputs before they reach downstream systems. 

 

Who is at risk?

Individuals 

  • People using AI tools that generate HTML, messages, or helpful instructions can be harmed if they copy/paste output into sensitive places (admin consoles, scripts, terminals) or if a tool renders unsafe output automatically.  
  • Anyone using an AI tool with preview features (e.g. showing generated HTML) may be exposed if the preview renders unsafe content.  

Businesses and organisations 

  • Any organisation that embeds LLMs into websites, customer chatbots, internal tools, document search/summarisation, or agent workflows is at risk, especially if LLM output is:  
    • Rendered as HTML 
    • Used to build database queries 
    • Used in scripts/commands 
    • Used to generate emails/messages 
    • Passed into other APIs/tools automatically 

 

How attacks work

Example 1: XSS via unsafe rendering 

A chatbot or summariser generates content that includes markup or scripts. If your application renders the output directly in a web page (or a preview), malicious content may execute in a user’s browser. 

Example 2: SQL injection via generated queries 

An internal tool asks the LLM to write a database query. If that generated query is executed without parameterisation and validation, it can create SQL injection risks because the attacker can influence what the model produces through prompts. 

Example 3: Command or code execution via automation 

An LLM is used to generate command lines, scripts, or code snippets (e.g. “run this to fix the issue”). If the organisation’s workflow automatically runs that output, it can result in remote code execution or privilege escalation.  

Example 4: SSRF or unsafe API calls 

An LLM produces a URL, file path, or request that is passed into a backend system without checks. The backend may then make requests to internal services or restricted locations (SSRF-style behaviour), or access files it shouldn’t.  

 

Controls

People  

  • Train users and staff: LLM output should not be treated as safe to run or safe to paste into admin tools, terminals, or scripts without review.  
  • Encourage a “stop and check” habit: if an output contains code, links, scripts, or instructions to disable security controls, escalate for review.  

Process  

  • Define safe vs high-risk uses of LLM output:  
    • Safer: drafting text for human review 
    • Higher risk: anything that executes, changes systems, queries databases, or sends external communications automatically  
  • Require human approval for high-risk actions (emails sent externally, scripts executed, database updates, permission changes).  
  • Test your LLM features (including previews, HTML rendering, database helpers, automation) as part of security assurance because LLM output becomes an attack path when integrated into downstream systems.  

Tech  

  • Output validation & sanitisation: validate outputs against expected formats before passing them downstream.  
  • Context-aware output encoding: encode/escape output appropriately depending on where it will be used (HTML, Markdown, JavaScript contexts, etc.).  
  • Never auto-execute LLM output: avoid piping model output into shells, exec, eval, or privileged functions. OWASP lists shell execution as a common example leading to RCE. 
  • Use safe database patterns: don’t execute LLM-generated SQL directly, use parameterised queries and strict query builders where possible.  
  • Limit privileges and tool access: apply least privilege so even if output handling fails, the blast radius is smaller.  
  • Logging and monitoring: log prompts, outputs, tool calls, and admin changes so you can detect anomalies and investigate incidents. 
  • Rate limiting / anomaly detection: apply basic controls to reduce automated probing and repeated attempts to force malicious output.