Stop Hallucinations from Breaking Security: Human-in-the-Loop Approvals and Evidence-Required AI By CyberDudeBivash — Founder, CyberDudeBivash

August 09, 2025

Stop Hallucinations from Breaking Security: Human-in-the-Loop Approvals and Evidence-Required AI By CyberDudeBivash — Founder, CyberDudeBivash | Cybersecurity & AI

Executive summary

Generative AI speeds up detection, triage, and response—but it will hallucinate (produce plausible, incorrect content) and can suggest unsafe actions (e.g., “disable logging to resolve an alert”). Production security requires an evidence-required AI pattern: every claim must carry a source, risky actions must pass human-in-the-loop (HITL) approval, and models must be able to abstain when unsure. This article gives the threat model, reference architecture, guardrails, policies, and copy-paste code to deploy it.

1) Why hallucinations happen (and how they hurt IR/SOC)

Causes

Probabilistic decoding (the model guesses the next token even when facts are unknown).
Training gaps & outdated corpora.
Retrieval drift (stale or poisoned knowledge base).
Prompt ambiguity and over-broad tool access.

Security impact

Wrong triage (mislabels TTPs/CVEs).
Bad containment advice (disables telemetry, kills critical services).
Fabricated sources (“source: internal wiki, page???“).
Over-confident summaries that bury the one line that matters.

2) Threat model: where wrong advice enters

Triage copilot summarizes 50k events → drops key evidence.
RAG chatbot pulls poisoned Confluence/SharePoint text (indirect prompt injection).
SOAR tool-caller executes high-risk function on vague instruction.
Analyst fatigue: trust-by-default on long AI outputs.

3) Reference architecture: “Citable AI with HITL”

swift
Log/EDR/SaaS/Cloud → Ingest → Vector Index (signed) → 
Retriever → LLM (tools: IOC lookups, ticketing, SOAR) →
Policy & Guardrails (evidence-required) →
HITL UI (approve/deny) → SOAR Actions → Audit/Telemetry

Design goals

Evidence required: every factual claim must cite at least one verifiable artifact (log row, URL, case, CVE record).
Risk-based routing: low-risk actions can auto-execute; high-risk requires human approval.
Abstention: model is allowed to say “Not enough evidence.”
Immutable audit: prompts, retrieved chunks, tool calls, human decisions.

4) Output contract: force evidence & confidence

Require the model to return structured JSON, not free text:

json
{
  "answer": "Likely OAuth consent abuse on user a.b@corp.",
  "claims": [
    {
      "text": "App consent granted at 04:21 UTC",
      "evidence": [
        {"type":"log","source":"Entra AuditLogs","id":"log:8756a","uri":null,
         "quote":"OperationName=Consent to application, InitiatedBy=a.b@corp, Time=04:21"}
      ],
      "confidence": 0.74
    }
  ],
  "risk_level": "high",
  "proposed_actions": [
    {"id":"revoke_tokens","requires_human": true},
    {"id":"block_ip","ip":"91.123.34.10","requires_human": false}
  ],
  "abstain": false
}

Hard rules

A claim without evidence[] is rejected.
requires_human: true for any destructive or service-impacting action.
If confidence < threshold (e.g., 0.6), set "abstain": true.

5) Guardrails & policy checks

5.1 Evidence gate (OPA/Rego sketch)

rego
package ai.output

default allow = false

allow {
  input.answer != ""
  count(input.claims) > 0
  forall c in input.claims {
    count(c.evidence) > 0
    c.confidence >= 0.5
  }
}

5.2 High-risk action approval (YAML policy)

yaml
risk_policy:
  high_risk_actions: [ "disable_user","revoke_tokens","isolate_host","delete_object" ]
  require_human: true

5.3 Tool sandbox

Tool schemas with least privilege scopes.
Egress allow-list for any network calls from tools.
Dry-run / simulator mode for SOAR calls during validation.

6) Retrieval you can trust

Signed documents: store SHA-256 for every chunk in the vector DB.
Freshness guard: reject sources older than N days for fast-moving intel (e.g., 7).
Poisoning control: sanitize HTML/JS, strip hidden text, reject unknown MIME types.
Source scoring: vendor advisories > NVD/CISA > curated internal wiki > user forums.

7) Observability & evals (measure hallucinations)

Metrics

Citation coverage: % of claims with ≥1 evidence.
Source reliability score (weighted by provider).
Abstention rate and escalation SLA.
Hallucination rate: % human-flagged wrong claims.
Action override rate: % of proposed actions downgraded by analysts.

Offline tests

Synthetic Q&A packs with known answers.
Adversarial red-team prompts (“ignore previous instructions…”).
Poisoned-doc canaries: model must refuse to follow hidden instructions.

8) SOC integrations (copy-paste)

8.1 Sentinel (KQL) — suspicious app consent (evidence generator)

kusto
AuditLogs
| where OperationName in ("Consent to application","Add app role assignment to service principal")
| project TimeGenerated, InitiatedBy, TargetResources, IPAddress

8.2 Splunk — mail purge burst (evidence generator)

pgsql
index=o365 operation IN ("HardDelete","SoftDelete")
| bin _time span=5m
| stats count by user, _time
| where count > 50

8.3 SOAR approval gate (pseudo)

yaml
- if: ai_output.risk_level == "high"
  then: request_human_approval(ai_output.proposed_actions)
- else:
  execute_actions(ai_output.proposed_actions where requires_human=false)

9) Failure modes & mitigations

Fabricated citations → verify every evidence ID exists in the data lake before render.
Outdated intel → enforce doc freshness; include timestamp in evidence.
Over-automation → circuit breaker: max N isolates/hr; change-window awareness.
Prompt injection → sanitize retrieved content; refuse untrusted instructions; confine system prompt (immutable).

10) 30-60-90 day rollout

Days 1–30

Add evidence-required JSON contract; block outputs without citations.
HITL UI for high-risk actions; turn on full audit logging.

Days 31–60

Source scoring & freshness; deploy OPA policies; add abstention routing.
Start red-team prompts and poisoned-doc tests; track hallucination rate.

Days 61–90

Expand to SOAR auto-execute for low-risk actions with perfect evidence.
Regression pack for every post-incident lesson; quarterly policy review.

Quick checklist

Evidence-required output schema in prod
OPA policy gating claims & actions
Tool sandbox + egress allow-lists
Signed, fresh retrieval with source scoring
HITL approvals for high-risk actions
Metrics: citation coverage, hallucination rate, abstention
Adversarial/red-team tests in CI

Closing

AI can supercharge your SOC—but only if facts are proven and humans stay in the loop for risky steps. Make evidence mandatory, measure hallucinations, and route danger through approvals. That’s Zero-Trust AI for incident response.

SEO/CPC keywords: AI Governance, Model Risk Management, Zero Trust AI, SOC Automation, SIEM, SOAR, Incident Response, Data Loss Prevention, LLM Security, RAG Security.

Search This Blog

Cyberdudebivash