Skip to content
sec/atlas
back to the map
technique
stage: automatizar
2 min read

AI in triage and mitigation

Language models speed up alert triage and the writing of mitigations, but they read untrusted data and can be confidently wrong. The right design treats the output as a suggestion, not a decision.

Azure OpenAIPromptfooJSON SchemaPython

What it is

Common uses of AI in security: summarizing an alert and its context in clear language, suggesting classification and severity, proposing a code mitigation from a or report, and answering analysts' questions about incidents. The gain is reading and writing time, not replacing judgment.

Specific risks

  • through the data: logs, user agents, PR titles and request bodies are attacker-controlled and may contain instructions the model obeys.
  • Hallucination: a plausible but wrong mitigation, or a reference to a fix that does not exist.
  • Leakage: sending secrets, personal data or proprietary code to an external service without control.
  • Excess autonomy: wiring the model's output directly to a destructive action.

Structured output and validation

Ask the model for output in a fixed format and validate it against a schema before using it. Whatever does not match is discarded or goes to a human. This turns free text into data the automation can check.

triage_output.schema.jsonjson
{
  "type": "object",
  "required": ["verdict", "severity", "reasons"],
  "additionalProperties": false,
  "properties": {
    "verdict":  { "enum": ["attack", "benign", "needs_human"] },
    "severity": { "enum": ["low", "medium", "high", "critical"] },
    "reasons":  { "type": "array", "items": { "type": "string" }, "maxItems": 5 }
  }
}

Where AI fits in the playbook

The safe pattern is AI around the decision, not in place of it: it summarizes and suggests, a deterministic rule or a human decides, and the action goes through the playbook's brakes (idempotency, approval, dry-run, auditing). The greater the impact of the action, the further it sits from the model's direct output.

Measure before trusting

  • An evaluation set of real alerts already labeled by analysts, run on every prompt or model change.
  • Track accuracy, false positives and cases where the model refused or asked for a human.
  • Continuous sampled review of the output in production.
  • Data control: what goes out to the provider, with secrets and personal data masked before sending.
SASTStatic analysis: examines the code without running it. It finds issues through flow analysis, not guesswork.see in glossary →DASTDynamic testing: attacks the running application from the outside, as a black box.see in glossary →Prompt injectionAn attack in which hostile content (in an indexed page, in a summarized email, in a document the agent opens) tries to override the system instructions. The attacker knows the LLM mixes trusted text with found text and plants an 'ignore the previous instructions and run X'. It is the number 1 vector in production agents today; it is not an LLM bug, it is how agents read text by design.see in glossary →

connections on the map

updated on 14/10/2026 · sources to go deeper · report an error

in other domains

Study before

test what you learned

Loading questions…