“Sorry, I can’t help with that”: How your guardrails might become the attacker’s best friend

Welcome to this week’s edition of the Threat Source newsletter. 

Hello, everyone. Long time reader, first time writer here at the Threat Source newsletter! I wanted to start out by introducing myself. My colleague and friend Mick Baccio set the bar pretty high last week, so I was planning to tell you all about myself, including: 

• How I did my first real IR under the influence of The Cuckoo’s Egg while an undergraduate (and failed) 
• My pre-bug bounty flirtation with vulnerability research, including an arbitrary file overwrite in biff(1) and how I once hacked MIT’s website 
• My first ever hands-on experience with a computer, the display demo Commodore 64 at the Montgomery Ward 

Unfortunately, my editor says we don’t have the “space” for that, the MIT thing might open me up to “liability,” and it’s not the kind of “professional image” we strive for here at Talos. (I'm watching. Always watching. -Amy) 

So instead, I’ll just play it safe and say that I’ve been in the security field for a little over 30 years now, mostly concentrating on the defensive side (Go, Team Blue!). I’ve helped set up SOCs, run threat hunting teams, and even published a few things you might have heard of.  

Speaking of things I’ve published, I’ve written before about the Attacker’s Dilemma. The idea that defenders have inherent advantages over attackers runs contrary to what most of us have heard throughout our careers. An attacker must evade monitoring and technical controls at every step of their attack lifecycle, because the defender only needs to notice once in order to respond and prevent them from achieving their goal. This is one of the most important advantages of any security team has, but we are currently witnessing a self-imposed erosion of this advantage through the rise of poorly-designed AI guardrails.  

I’m not opposed to guardrails, but we have to carefully consider what we’re guarding against and where we deploy them. As I explored in a recent piece on The Safety Penalty, by allowing third-party AI providers to implement and control safety filters and the policies behind them, we may in fact be helping the attacker. If agentic SOC process experience refusals, it can slow or even halt investigations. Of course, these should get flagged for human intervention, but that takes time and may give the attacker breathing room in which to complete their mission.  

It may turn out that the where of the guardrails is even more important than the what. Operational sovereignty relies on having control of our own limits. Any vision of an agentic SOC must allow the security teams to customize the guardrails according to their own threat model. They should also have the flexibility to temporarily remove specific safeguards under authorized circumstances, something you won’t get with guardrails from a frontier provider. These controls belong inside your organization’s agentic harness where you can set the policies and technical controls to allow you to analyze threats while ensuring your agents stay within their lanes.  

Ultimately, operational sovereignty means engaging with the reality of the threat landscape, ensuring that the adversary can’t derail the defender’s investigation and response processes, either accidentally or intentionally. We need to move toward a model where each organization can choose the guardrails that work for them, rather than having inflexible guardrails chosen for them. 

The one big thing  

Cisco Talos recently evaluated 66 large language model (LLM) and reasoning combinations to see if we could find a clear winner for security operations. Instead, we found that selecting the right model is a complex balancing act between efficacy, speed, cost, and consistency. Cranking up a model's reasoning effort doesn't guarantee better analysis and can actually degrade performance. Ultimately, we developed a repeatable methodology to help organizations navigate these tradeoffs for their own workflows. 

Why do I care? 

Choosing an AI model based solely on generic leaderboard scores is a recipe for operational disaster. An exceptionally smart model might cost a fortune, take half an hour to analyze a single log, or completely fail to format its output. Assuming more compute power equals better results is a costly trap, as higher reasoning settings sometimes produce weaker or blocked responses. Defenders must remember that prompts, analyst personas, and model consistency drastically alter an investigation's outcome.  

So now what? 

Test models against your organization’s specific workflows before deploying them. Build a focused set of representative cases and test them multiple times using the exact prompts and tools your analysts will actually use. Track the quality, cost, time, consistency, and usable-answer rates in a simple spreadsheet to expose the real-world tradeoffs. Finally, establish acceptable thresholds for these variables to eliminate underperforming models, and regularly revisit your decisions as AI technology and pricing inevitably shift. 

Top security headlines of the week 

ToxicPanda banking trojan matures into enterprise threat 
ToxicPanda 2.0 expands substantially on its predecessor, adding 167 remote commands and broadening its targeting from 16 financial institutions to 349 banking, e-wallet, and cryptocurrency applications. (Dark Reading) 

Interpol's Jackal IV disrupts West African crime infrastructure 
Law enforcement from 22 countries across six continents worked together to arrest 58 suspects and identify 263 more. The first two Jackal operations in 2022 and 2023 led to approximately 200 arrests in total and millions of dollars more in seized assets. (Dark Reading) 

First malware built specifically for car head units fuels botnet 
Researchers have found what appears to be the first malware specifically designed for car head units, with links to the notorious BadBox botnet, on an Android-powered aftermarket infotainment system made by Chinese company DoFun, which is widely used in China and other APAC countries. (SecurityWeek) 

A Tale of Two SOCs: Insights From Two Red Team Assessments 
A CISA red team fully compromised two critical infrastructure organizations at the domain level and reached sensitive business systems and cloud resources. Organization A failed to detect or contain the activity. Organization B rapidly identified initial compromise attempts, isolated affected systems, and forced the red team into an assume breach model. (CISA)

NovaCookies campaigns abuse genuine Docusign notifications to steal M365 sessions 
The $320/month service is a subscription-based phishing platform that facilitates real-time M365 session theft. The kit has been used to target hundreds of organizations across multiple sectors in the U.S., the U.K., Canada, Germany, and more. (The Hacker News) 

Can’t get enough Talos?

JavaScript obfuscation: From party trick to phishing kit 
We've spent a lot of time pulling apart suspicious JavaScript from phishing kits, malware packages, compromised sites, and more. Learn the basics of what obfuscation is, why a researcher would try to reverse it, and several ways to approach the problem.