AI is catching up to intel analysts. But here’s what it still can’t do.
Former Acting CIA Director Michael Morell told a story that grabbed my attention. In a fireside chat at the Silicon Valley Security Group’s (SVSG) Create conference last week, he explained how an unnamed AI startup was beginning to produce strategic intelligence on par with expert CIA analysis. He had tested it himself with questions about Iran.
“I have no doubt that AI can do the kind of quality analysis that the best analysts at CIA did,” he said. “I think it’s right around the corner.”
But Morell was quick to add a caveat.
“I think it’s going to be a long time before it gets adopted because there’s a level of trust you have to have,” he said. “If you’re going to trust the answer, if you’re going to walk into the Oval Office, you have to know where the answer came from.”
That gap between capability and trust came up again and again across several days of conversations. At the Analyst Roundtable in San Francisco earlier in the week, some analysts agreed that junior roles will face the most pressure, while senior analysts will evolve into AI orchestrators and validators, ensuring the right questions are being asked and the quality of intelligence remains high.
Listening across both rooms, it became clear that several analysis tasks require more of a human touch than others. These skills will be increasingly assisted and enhanced by AI, but for the immediate future, are unlikely to be replaced by it.
1. Critical decision-making
As we’ve seen with enterprise AI, what the technology can do and putting it to work in practice are two different journeys. This is especially the case with the high-stakes decisions that are common in crisis. Trust is foundational, and it’s built through accountability, a reliable track record and the soft skills that AI can’t shortcut.
Or as one CSO put it bluntly: “We can’t be wrong.”
In another SVSG fireside chat, I spoke with Matt Abrahams, a popular strategic communicator, author and lecturer at Stanford. He explained how relationships and in-person communication skills are more important than ever as AI advances. Without it, even the smartest intelligence won’t land with the decision-makers who need it most. (I recommend his podcast, “Think Fast, Talk Smart,” for some great practical advice.)
2. Real-time verification
AI needs trusted data to produce quality results. For slower-moving strategic intelligence, it can draw on a deep corpus of information. For fast-moving events, data is scarce and it’s increasingly altered or generated by AI on social media. When you ask an AI if it’s real, the answer is a coin flip.
Verifying an event in real time is one of the hardest human-led jobs in intelligence, and for an industry that “can’t be wrong,” it’s only getting harder. Every layer of automation built on top of unverified data inherits and amplifies its mistakes. That’s the gap our team at Factal was built to close.
3. Violent threat assessment
Some threats of violence are straightforward, but others are puzzles embedded in sub-cultures. A grievance may escalate through violent slang, memes and coded references spanning fringe groups, emerging platforms and the dark corners of the internet. These references are deeply nuanced, fragmented and always changing with the times.
Several trust and safety leaders speaking at SVSG explained how internet culture knowledge and a “Spidey sense” are highly valuable skills. While AI assists in research, it struggles with piecing together changing human behaviors and differentiating legitimate threats from humor and fandom.
4. Benchmarking
As one Analyst Roundtable panelist explained, “Your AI won’t know how other companies are responding to the same crisis.” This remains the domain of human conversations: the Signal groups, Factal chats, OSAC channels and old-fashioned phone calls. Security, privacy and trust are paramount, and benchmarking is often driven by relationships built over the years. It’s safe to say it will be a long time before crisis benchmarking is outsourced to AI agents calling each other.
5. Real-world access and experience
If a model can’t see it, it doesn’t know it exists. There’s no substitute for an on-the-ground team – better yet, a local one – with first-person experience and the tribal knowledge that comes from actually being there. That’s especially true in parts of the world with a thin digital footprint.
The same goes for what happens inside an organization. Hallway conversations, side comments after a meeting, the context a colleague gives you over coffee — none of it shows up in meeting notes, email or internal documentation. It’s some of the most useful information an analyst has, and it’s invisible to any AI that wasn’t in the room.
As AI begins to absorb some parts of analysis roles, it’s also making others more valuable than ever: judgment under uncertainty, real-world experience, verification and trust built through relationships. They’re the foundation everything else gets built on. Someone still has to walk into the CEO’s office and stand behind the answer.