Can employees safely use AI agents? AI pentesting agent liabilities, and the news - ESW #473
Ayman discusses the use of AI in pentesting, emphasizing the need for human oversight and the value of custom harnesses to ensure accurate and novel testing. He also highlights the challenges of using AI for security, including the need for open source models and the potential for AI to both create and solve security issues. Additionally, he comments on the implications of AI in content creation and the importance of transparency in its use.
Transcript
Only Ayman's parts of this conversation are included, with the question before each answer for context.
I got some Deloitte stuff over there. Let me grab that real quick and see if I can leverage that as a, you know, an alternate channel. And nobody would have seen that coming, right? Like, there's no way to know that it's going to do that. And so that's my biggest concern about offensive kind of AI driven. And our resident pro, Ayman, has a million things I'm sure that he would like to comment around that.
I've been on both sides of the fence of pentesting. There's a few things to unpack here. One, I spoke to an automated pentest founder, and I asked him, how are you different? There's so many of you. His take was that every finding has to be vetted by a human, like an OSCP. So that's not bad. And, you know, if you go on GitHub, you have a whole bunch of automated pentest repos, like one every month, new one just coming out, right? 39:44 I spoke to a pentester, and here's the thing, when I commission a pentest, I am expecting someone to be using AI. So I do have that expectation, but I want them to be using it in a novel way. And I talked to one person at Black Hat, which was fascinating. This guy creates a new harness for every pentest.
Oh, wow.
Which is really cool. And essentially, that's what it is. You need a heart.
And presumably, the unique limiting parameters.
Exactly. It's just that's smart. Like that, I think that is the way to go, right? So this person, he's a legit pentester. You know, he's done his thing. But for every customer he has, he creates a custom harness and throws it at it. And so then now you have that really good mix of AI and human blend. So I found that really fascinating.
Like, you get a pentest from these guys, you know, they're going to break your business. And, you know, I worry that, you know, has kind of a chilling effect on using AI at all. So you want to have some guardrails to make sure that you're not just, you know, beating up stuff at your customers every time you go in.
I do wonder if like, oh, sorry.
Go ahead, Ayman.
Yeah, I do wonder if the pentest industry, like how the pentest industry has been affected by AI, you know, like, have they seen less demand? Because I mean,
Lots of folks we could bring on to have that conversation.
Yeah, yeah.
What do you think, Ayman?
I mean, you know, if we're talking about offense, like from the front door, then we will need AI because the volume is so high. We do need security chain models for defense. I mean, like, you know, in my job, like the number of PRs that happen, like people are outputting more now. So I need a model to help keep up with that, right? Or, you know, a trained model specific. The same thing at the front door. 58:28 If there's a novel bot attack, I needed to inspect that bot attack and update my Cloudflare rules to prevent that attack until Cloudflare updates its rules. I could see that as far as closing the gap, kind of like fighting fire with fire.
I think we're going to need a plan B, though, because if that doesn't work... If AI models can't keep up with all the PRs, if they can't do that defense piece, I don't know that. Well, they're going to assist.
It's kind of like assisted. I mean, we have open source. There's a lot of really good open source right now models. There's even open source harnesses, right? So I think it can be done. I wouldn't rely on regulatory... I don't know what that would look like. That would be scary. I think the hard part is, and I was just writing some notes internally, it's hard to keep up. If AI is causing the problem, then we'll need AI to solve the problem. 59:37 I don't know. It's a bit circular, but I can't think of anything else.
And I did not catch this in the article, so somebody who reads it in more detail may correct me. But they have to give it away. They have to give it to that same foundation that Anthropic donated the MCP concept to, and I forget what the foundation specifically is called, but it's an open source foundation specific to support MCP, etc. This needs to be done in the same way. Even if it is somewhat self-serving to work better in their systems, if they want mass adoption, they have to go here. This is for the community. This is for the betterment of the world to operate in.
Yeah, I'm excited about it. I mean, you know, sometimes I decide not to use an MCP because it takes up too much context. So I'd rather have the MCP to a specific agent than like in my in my main harness or sometimes I just go with APIs. So if this helps like also like reduce context and make it Talk on information security covering CISO, vCISO, red team, threat modeling, Enterprise Security Weekly, Security Cafe, Getting Into Infosec, Ayman Elsawah, Adrian Sanabria talk on information security covering CISO, vCISO, red team, threat modeling, Enterprise Security Weekly, Security Cafe, Getting Into Infosec, Ayman Elsawah, Adrian Sanabria
But I can't believe when you're doing an exact measurement, when the LLM is doing an exact measurement of choosing A or B, that it doesn't come... Are you saying that if you take a 20-digit measurement of accuracy of word choice, 20 digits are going to hit it? You're going to nail it? Like, there's always going to be little fluctuations of better or worse. So there will be some degradation. But what they're saying is that when they present it to a human reviewer, there's no difference. They can't tell, right? Which is probably true.
Oh, you're on mute, Adrian.
Correct. I think that's a very interesting way to look at it, right? Yeah. Yeah, I think it's a very interesting way to look at it.
You guys think this is a good thing? I mean, you know, as creatives yourselves, and I'm sure we all don't like AI slop.
And this was regulation, Tyler. That's my read on why this exists is for the EU AI Act.
And then there's PanGram, too. There's a whole industry around this, like AI detection, deepfake, all that kind of stuff.
The European market, if they don't want it, now if it makes no difference and the market says it's fine, that's fine. But if the market says we don't want that, we don't want that regulation, it's just going to drive people away from that business. And I think the market finds ways, right? And they'll use Kimmy or Quinn or some other open weight model to create the exact same thing that's not watermarked. So I don't think it's a net positive and I don't think it achieves the results that the EU wants in this case.
Yeah, maybe. We'll find out.
Transnational cyber-enabled crime.
This is just a Pandora's box, man.
Explain.
What do you mean? I mean, you know, just transferring that capability. I mean, you know, what... You're not, first of all, I mean, no private company or entity is allowed to do it, like as far as a US, if you're on US web, because you will get sued for hacking. So it's, it's, it's, it's, How is that going to affect that law, the CFAA, right?
So Rob does point out that the CFAA says this section does not prohibit any lawfully authorized investigative, protective, or intelligence activity of a, it does say law enforcement agency of the US, a state, or political subdivision of a state, or of an intelligence agency.
So have we all been deputized now?
I was very negative on it after reading Rob's report. I actually, I think there's a lot of potential positive for this to work well.
So wasn't that happening anyway? It seems that this is just kind of like making it clear. Probably. This is just making it like official? Is that, I mean, it's been happening.
You get to keep half of everything you seize. That doesn't happen here. The companies don't get to steal Bitcoin found on hackers' computers for themselves. However... It happens with whistleblowers. However, the memo is silent on where those seized assets do go. So the memo, the EO doesn't say where the assets end up when they do get seized. So there is some potential there for some pirate booty sharing.
And the memo does say if you end up going to US citizen or US territory, you have to cease and stop. So that's good. At least that actually follows the law of what it should be versus what we used to do. Side note, in Rob's article, he has like an AI disclaimer, which talks about, which kind of was what we just spoke about. And he actually includes the prompt that he tells Claude, which I find very fascinating.
This is very common now. Anybody who does a lot of writing puts in some kind of AI statement to say they use it, they don't use it, here's how much they use it, here's how they use it. Yeah, there's been a lot of push for people to be transparent about how they're using AI.
I was wondering if I had to say that, like, you know, my last article was like, I didn't even throw it through AI at all. I guess I should include those disclaimers. That's interesting.
Yeah, I'm thinking about adding something as well because for me, it would be like going to a nice restaurant and having somebody pre-chew my food. I like that part. Or like Elon hiring somebody to play Diablo for me. That's missing the whole point. Why would I want to outsource the enjoyment? But I enjoy writing. I understand not everybody does.
Well, I thought it was a given that my stuff was 100% written, but I guess I have to say it now. So I don't know.
Yeah.
Yeah, I didn't read it either. Sorry.
All right. Number five is just a fun read. Snopes put together a how-to on how to spot AI images, and I actually learned a couple things from it. I think it's gotten very difficult these days with some of them, but there's some interesting stuff in there because this is, you know, the people at Snopes are doing this day in and day out, so it was kind of cool to see what they look for when they're trying to figure out if an image is fake or not. Yeah, we've come a long way.
Sorry, I was going to say, I love the first line. People have lied on the internet for as long as it has existed. Love it.
And I'm like, why have we not Right, because this predictive component could now take MoveIt, which was, or PTC Windchill, whatever it is, which was potentially a very low risk system that nobody really even thought about. It was an afterthought that happened to sit there.
I've never heard of it.
And one of the things a regular mentions is that the turnaround time is extremely short. It's like 48 to 72 hours. Like in some of these cases, like, you know, just transferring the model onto their systems might take a couple hours. Some of these models are huge. But yeah, I found that very interesting that's done at a third party and that, you know, not within Anthropic and OpenAI and Meta's own labs.
Yeah, yeah, I thought they'd built their own, their own, like...
I'm sure they do, but for some reason, like, I think what Irregular is selling here is the testing itself. Yeah. It's maybe the, like, the testing methodology, right? Like, you'd send something to UL Labs or something like that.
Right, I think they're advanced evaluation sets, which had a naming error, and they're supposed to have a fake domain, but actually was a real domain.
Pretty much anywhere there are Wi-Fi signals, even if you're not the one that has the Wi-Fi, like you can turn all your Wi-Fi off and there's enough Wi-Fi around us in businesses, other homes and stuff like that, that it can still detect your movement. So... That's a thing now that is never going to be undone, right? Unless you turn your home into a Faraday cage, I guess.
Yeah, bad enough.
And then our squirrel story, Liquid Death and a beer company also, I think they teamed up with Garage Beer, I think is the name of the company. And they just have this ridiculous marketing where they said, hey, we solved the problem of AI data centers using too much water. We're just going to collect all our urine and send it to the data centers.
I'm looking at the website. It's just so nasty.
Yeah, so Liquid Death has always had this like these edgy marketing things. They do a great job with it. And it's a lot of fun. And this one is, I was not expecting it to be this ridiculous, but it's pretty ridiculous.
Although, although it could be plausible. In some crazy dystopian future.
So I saw on LinkedIn a writer at TechCrunch is like, hey, data center experts, urologists, like, I need you all to hit me up. I want to write an article on this. I want to see if it's possible.
I mean, this in gray water, I don't know.
I mean, it is gray water, right? Like, technically.
Yeah. It's a little yellow water, actually, but. Yellow water. I wonder what the radiation, like, the absorption of urine is to absorb heat and retain heat.