Verifying AI in your global security stack
Decisions have real-world consequences in global security. Lives can be on the line, and teams are under considerable pressure to make the right call. As it becomes harder to know what’s true, your credibility matters more than ever.
AI is rapidly improving, but it still makes mistakes. Striking a balance is a moving target. How much should you use AI for security tasks? How do you know if it’s producing the right answer? How much effort should you invest to validate it?
These are hard questions for global security professionals who work in time-sensitive situations. At a recent Factal workshop, “A practical guide to verifying AI,” I walked through a framework that balances risk and judgment. The higher the risk, the more you should verify. The more judgment is required, the more a human needs to own the decision.
(If you missed the workshop, security professionals can request a link to the video.)
Verifying AI in a chat conversation is relatively straightforward. You see each response as it happens, and mistakes are easier to catch. Agents are different. They combine steps and complete tasks in a single workflow, and mistakes are often hidden behind the scenes. One mistake in the chain cascades all the way through.
The same applies to your security stack. An error in one tool can disrupt the rest. For example, an incorrectly geolocated incident can trigger unnecessary response plans and send the wrong mass notifications. The more automated the stack, the more the errors propagate, the harder it is to backtrack to what went wrong.
As security stacks and platforms become more automated, there’s also the risk that security teams start acting more like approvers than reviewers. It’s much easier to just accept the AI’s output without double-checking it first. Researchers at Wharton call this behavior “cognitive surrender.” Others call it a meat proxy.
As AI moves from answering questions to taking actions, verification has to expand beyond what a human does to something the system also provides. Here are three key areas:
1. Get the inputs right
AI is only as accurate as the data you feed it. Since errors at the beginning of a decision chain have disproportionate consequences down the line, pay special attention to your inputs. What data is supporting your most critical decisions?
Constrain your AI to high-quality, well-cited source material whenever you can. It significantly increases accuracy and makes the output easier to check.
For example, at Factal we built a new AI discovery product called Explorer. We constrained the inputs to Factal verified data only – no social media posts and no AI-written content – with every detail cited back to its original source. In our testing with risk events, it outperforms AI chatbots in both accuracy and speed.
2. Put friction around consequential actions
As vendors add more automation, security teams should establish protocols about when to use AI and what steps they should take to verify it. These protocols should align with the settings and checkpoints in your security stack.
For example, your stack should let you decide when mass notifications fire automatically and what approvals are required. Sometimes a little friction can help reduce consequential errors without significantly slowing a response.
3. Make every action traceable
Automation is increasing faster than visibility. Security teams should be able to see what the system knew, what it decided, what it did and who approved it. When something goes wrong, you should be able to inspect a log of errors, changes and corrections. Vendors are often hesitant to admit mistakes or provide corrections, but mission-critical tools need this level of transparency.
Much of the AI race has focused on speed, productivity and how much more we can automate. But in global security, the most important measure is making the right call from accurate information. As AI takes on more consequential work, accuracy can’t be treated as a feature. It has to be the foundation.