What We Learned Building an AI-Assisted Threat Modeling Platform

AI is quickly moving from experimentation into core risk and security workflows. One area where this is especially powerful is threat modeling. An AI-assisted threat modeling platform can accelerate architecture review, identify potential threats, map evidence to risks, suggest controls, and support risk decisions.

But this also creates a governance challenge: what happens when AI starts influencing how risks are identified, assessed, scored, and accepted?

This is where the three lines of defence model becomes critical.

For an AI-assisted threat modeling platform, the right operating model is not “let AI assess the risk.”

The safer and more defensible model is:

AI assists. The deterministic engine scores. Humans remain accountable.

That distinction matters because threat modeling is not just a technical exercise. It influences remediation priorities, risk acceptance, audit evidence, and potentially regulatory conversations.

The project context

The AI-assisted threat modeling solution I am referring to is not a simple chatbot. It is a security and risk decision-support platform that may include:

• architecture and data-flow ingestion;

• graph-based threat modeling;

• LLM-assisted threat identification;

• deterministic risk scoring;

• evidence grounding;

• multi-tenant client isolation;

• immutable audit records;

• Jira and scanner integrations;

• secrets management;

• governed egress to third-party AI providers.

  
  

Because of this, the risk profile is broader than traditional application security. The project introduces cyber risk, AI/model risk, data confidentiality risk, third-party risk, operational risk, auditability risk, and governance risk.

That means each line of defence has a specific role to play.

1LOD: Own, build, operate, and evidence the controls

The first line of defence owns the AI solution and the risks created by using it.

1LOD should start with an AI use-case assessment. The team should clearly document what the AI is used for, what data is sent to the model, whether the AI output is advisory or decision-making, who relies on the output, and what happens if the AI is wrong.

For this type of platform, the correct classification should be:

AI-assisted security/risk decision support — not autonomous final risk decisioning.

1LOD should then identify the inherent risks before controls. These include hallucinated threats, missed threats, prompt injection, output injection, automation bias, model drift, third-party provider exposure, tenant isolation failure, incorrect scoring, audit tampering, and over-reliance on AI-generated outputs.

Once the risks are identified, 1LOD must design and implement controls directly into the architecture. Key examples include:

• LLM output is advisory only;

• final risk scoring is performed by a deterministic Decision Engine;

• prompts are redacted and tokenized before leaving the boundary;

• AI outputs are validated against a strict schema;

• suggested threats are grounded against graph evidence;

• tenant context is enforced across all services and data queries;

• secrets are stored in a vault, not in the browser or source code;

• third-party calls go through controlled egress proxies;

• risk decisions are written to immutable, hash-chained audit records;

• rule changes, prompt changes, and model changes follow change governance.

The key point is that 1LOD cannot simply state that these controls exist. They must prove they operate through test evidence, logs, configuration exports, control traceability, and monitoring.

2LOD: Challenge whether the risk and controls are adequate

The second line of defence does not own or build the solution. Its role is effective challenge.

2LOD should challenge whether the AI use case has been classified correctly, whether the inherent risks are complete, whether the control design is sufficient, and whether residual risk is within appetite.

For example, 2LOD should ask:

• Is the LLM truly advisory, or is it becoming the de facto risk assessor?

• Can AI output directly change a score, close a finding, accept risk, or create a remediation ticket?

• How is hallucination detected?

• How is prompt injection handled?

• What fields are allowed to leave the organization’s boundary?

• How do we know secrets, IP addresses, hostnames, and client identifiers are redacted?

• Can one tenant’s architecture ever appear in another tenant’s prompt?

• Are scoring rules versioned, tested, approved, and reproducible?

• Is there a human-in-the-loop for High or Critical risks?

• What KRIs will be monitored after go-live?

2LOD should also review evidence, not just design statements. Evidence may include redaction test results, prompt-injection test results, cross-tenant negative tests, Decision Engine regression tests, egress firewall rules, vault access logs, immutable audit verification, and sample redacted prompts/responses.

The 2LOD challenge should be formal. A strong challenge opinion may say that the solution is satisfactory for go-live, satisfactory with conditions, or not satisfactory until specific control gaps are addressed.

3LOD: Independently assess whether the lifecycle worked

The third line of defence provides independent assurance.

3LOD should not be the project approver and should not become part of 1LOD control execution. Its value comes from independence.

Internal Audit should assess whether 1LOD identified and managed the risks, whether 2LOD performed effective challenge, and whether the control environment was designed and operating effectively.

This means 3LOD should review the full lifecycle:

• Was the AI use case assessed and approved?

• Was the inherent risk assessment complete?

• Were AI/model risks considered?

• Were data confidentiality and tenant isolation risks assessed?

• Were controls mapped to risks?

• Were pre-production tests completed?

• Were residual risks accepted by the right authority?

• Were monitoring metrics defined?

• Were model, prompt, and scoring rule changes governed?

• Did 2LOD provide meaningful challenge?

• Is there evidence that controls operated over time?

3LOD should independently test selected controls. For example, it may reperform cross-tenant access tests, review prompt redaction evidence, sample immutable audit records, inspect rule-change approvals, or validate whether risk acceptances followed policy.

The risk of using AI as the risk assessor

The biggest risk is not simply that AI may be wrong. The bigger risk is that organizations may begin treating AI-generated output as authoritative without proper grounding, accountability, and governance.

If AI becomes the actual risk assessor, several risks increase:

• accountability becomes unclear;

• explanations may sound convincing but be unsupported;

• outputs may change across model versions;

• humans may stop challenging AI recommendations;

• false negatives may leave material risks unidentified;

• false positives may overwhelm remediation teams;

• auditors may challenge unsupported risk decisions;

• sensitive architecture data may be exposed to providers;

• formal risk methodology may be bypassed.

That is why the operating model should be explicit:

AI can assist risk assessment. AI should not own risk accountability. AI should not be the final authority for material risk decisions.

The accountable risk owner must remain human. The final score should be produced by a governed, deterministic, explainable engine. Material decisions should require human review, clear evidence, and an immutable audit trail.

Managing controls through the lifecycle

Controls should not be treated as one-time design features. They need to be operationalized.

For each control, the project should define:

• the control objective;

• the control design;

• the control owner;

• implementation evidence;

• test procedure;

• operating frequency;

• monitoring metric;

• failure handling;

• residual risk;

• assurance approach.

For example, the control “LLM advisory only” should have a technical design, code enforcement, negative tests proving the LLM cannot write final scores, audit events showing the Decision Engine set the score, and monitoring for exceptions.

Similarly, “redaction before egress” should have redaction rules, test corpora, sampled redacted prompts, failure handling, and metrics for redaction failures.

Recommended governance gates

A strong lifecycle should include risk gates:

1. AI use case approved.

2. Inherent risk assessed.

3. Architecture and control design challenged by 2LOD.

4. Build controls implemented.

5. Pre-production testing passed.

6. Residual risk accepted by the correct authority.

7. Go-live approved.

8. 30/60/90-day post-go-live control review.

9. Periodic independent audit.

At each gate, the organization should require evidence. Design statements alone are not enough.

Final thought

AI-assisted threat modeling can materially improve the speed and consistency of security risk analysis. But if the governance model is weak, it can also create false confidence.

The right approach is not to avoid AI. The right approach is to govern it properly.

1LOD should own and operate the risk.

2LOD should challenge the risk and controls.

3LOD should independently assess whether the lifecycle worked.

Most importantly, the organization should preserve a clear accountability principle:

AI can assist the risk assessor, but it should not replace accountable human risk ownership.