AI News Stories | Episode 33
Hello, everybody, and welcome to this week's episode of AI Security Ops. Today, we have a couple of news stories for you. We're going to dive into the first documented AI orchestrated cyber espionage campaign, a free black hat LLM that could arm script kitties but could also compromise your system. Critical RCEs in popular AI inference stacks. Amazon opens up an AI bug bounty program of their own.
Brian Fehrman:Google's shiny new anti gravity ID got popped in under twenty four hours. And there is some polymorphic malware that likes to continuously rewrite itself using Gemini. So before we get kicked off, do some quick introductions. We got Derek Banks on the panel today, as well as myself, Brian Fuhrman. And of course, this episode is brought to you by Blackhawk Information Security.
Brian Fehrman:If you are in need of security consulting or any, pen testing, anything of that nature, go ahead and reach out to us and see how we can help you out. And, episode is also brought to you by Anti Syphon Training for any of your security training needs, come check us out. We have a lot of great courses that are put on by, many of their, consultants and, soft professionals and other people in our company who are doing the things that they are teaching day in and day out. So we have really good quality training at very low affordable cost. So with that, let's dive into the news stories.
Brian Fehrman:So first, we have the, what it I think the first AI orchestrated cyber campaign might be a strong claim or cyber espionage campaign, but Anthropic put it out and, they're they're talking about it. Eric, I know you were telling me about this one, within the last few weeks, pretty recently here. Yeah?
Derek Banks:Yeah. I think that the thing that, that makes it different than others in the past is just the amount of things that the AI was doing, like, you know, semi autonomously. Right? I think that there was still, you know, a human in the loop for a lot of it, but it just seems like from, you know, reading the threat report and, you know, reading other people's takes on it that, that's really, like, what made it unique was just the amount of automation through AI. And it reminded me of a novel by Daniel Suarez called Damon.
Derek Banks:Right? Like, now now is this the next step before it's actually loosed on the Internet and it's its own thing? I don't know. But, it's certainly interesting for sure. And I think that, you know, the the the the thing that I find most interesting was that they were using Claude on the back end.
Derek Banks:Right? And so they were able to get away with the hacky hack, so to speak, because they were breaking things down to subtasks and essentially able to fool, Claude or into, you know, getting past its security, checks. Right? And so I mean but I do that on a daily basis. Right?
Derek Banks:Like, many times I type something into, Quad or ChatGPT and I say, I'm on an authorized penetration test, and I need to do this thing. Can you help me do this thing? So Oh, yeah.
Brian Fehrman:It makes make makes the prompting there makes a world of difference. So I've been on recent engagements using Cloud to help rewrite some of our common go to scripts, not scripts that we wrote personally necessarily, but scripts that are out there and heavily used, but also heavily signature. And I know in one instance, I had specifically mentioned for Claude to help, you know, rewrite the script and bypass EDR, and it was like, no, I can't do that. I'm like, okay, well, make a script that does this thing, leaving out the bypass in EDR and it wrote it. It's like, okay, cool.
Brian Fehrman:Here you go. You know, I'll I'll I'll do it, you know, just if you want it. But if you want it to bypass EDR specifically, then, you know, we're not gonna not gonna do that. So it's all on how you ask it.
Derek Banks:We wanna bypass EDR, but I'm just testing. I'm not doing it for realsies. Oh, okay. Well, that
Brian Fehrman:sounds fun. It's just a it's a staging environment. Don't worry about it.
Derek Banks:Yeah. Exactly. So this actually got me interested and because, you know, you know, in the in the offensive security space, you know, the quote, the good guys, the white hat hackers, so to speak. You know, there's a couple of of products that are out already. You know?
Derek Banks:Like, there's XBOW and Horizon three, which, you know, you know, AI powered penetration tests. And so I I was just curious, like, how easy, you know, would it be and and and to to make something like that. But before I went to go make it all myself, which is, you know, a pretty huge undertaking, I went and looked for some open source stuff, and there's a a a bunch of things out for sure. And I, started messing with Hexstrike, and I was able to pretty easily connect Cloud Desktop up to Hexstrike, MCP servers and get it to start. It went and found the authorization bypass on Jen and Juice shop by itself.
Derek Banks:I just passed it. I was like, hey, you're authorized to do this, you know, Port Swigger says this is a a playground. You can go do it here. Because at first, it was like, yeah, I'm not helping you hack. I see you got some security tools and you're a security person, so cool, but we're not doing that.
Derek Banks:So I prompted it a little better. And yeah. I mean, it it you know, as a proof of concept, I think that this is, that was it was very useful. And I really think that, you know, this is kind of the way the industry is going, I think.
Brian Fehrman:Oh, yeah. I I absolutely agree. I think that there are a lot of, a lot of tasks that we, do normally as we're doing engagements that could be automated with these types of tools. And in some aspects, you know, some of those, engagements could potentially start to look almost more like a, you know, like a SOC role where you have your different agents running, you're getting the information in, and then you're responding to things as they come in rather than taking, you know, the much more active role throughout the entire process. And I really do foresee that as being the the future as we move forward.
Brian Fehrman:You know, obviously, I I wouldn't let something just, like, loose within a customer environment, but when we're talking about externally facing things, externally facing assets that are getting hammered away day in and day out anyways, And, you know, gathering up information about companies and everything else that you would normally do, I, you know, I think that this is this is certainly what we're gonna see a lot more of in the in the near future.
Derek Banks:Yeah. I I agree with you too. I I think that from external and, like, even, like, a web application perspective. Right? It's, you know, something that's facing the Internet.
Derek Banks:Like, I think I'm very comfortable at this point with my, you know, AI usage and knowledge to think that, like, oh, I'm alright with that. Because, like you say, you're already a target. Right? Like, you're getting hammered away at all day. I don't know that I'm quite ready to let, you know, let things loose in an internal environment, give it AD creds, and say, here you go.
Brian Fehrman:I
Derek Banks:I just I I don't know. And and and the reason I say that is, you know, with with the, you know, with folks that are watching, you know, if you're not aware, Burp Suite has an AI feature where you can say explore issue, and it'll take one of the issues in Burp Suite, and AI will actually go through and verify it like a human. Right? Which is a really cool feature. And, well, we had a one of our, other researchers here was telling me he was using it and, you know, it was a command injection finding.
Derek Banks:And the, instead of, you know, caching Etsy password on the web server, it went to r M Etsy password on the server. I agree that if you could do that from a web app perspective, that is a finding. That's generally not the way that I would go about trying to do that because I think that would be a bad day for the web app server if that were the thing. But, I mean, yeah, I'm I'm I'm not quite ready for the the internal, but I think it's coming very soon. So
Brian Fehrman:Oh, yeah. Yeah. Abs absolutely.
Derek Banks:And so a couple of things Anthropic said, speed and scale. They were able to use thousands of tool calls, often multiple per second, doing work that would take a large human team. And I I think that's really kinda where, like, from, you know, a consultancy perspective where I think that that's really going to be, like, a game changer. Because if I can do an external engagement in a day instead of, what, four days or whatever our, our time is now, I I that's a huge difference. Right?
Derek Banks:And then, and then also, I think interesting too is Anthropic, said that they also use large language model on the defensive side, CAUG, for, investigating, telemetry and, you know, investigating the incident, which I really wish they went into more detail about that because processing large amounts of data with a LLM is actually a trick in itself. Right? There's strategies to go about doing that and I would love to see how they're doing that.
Brian Fehrman:Oh, yes. Yep. Absolutely.
Derek Banks:So a couple takeaways. Right? So, I think for everyone listening, just start assuming, we're in the age of the agentic attacker. And if you're not monitoring, what's going in and out of your large language model in terms of input and output, the prompts that are going in, you should be. And, I think that it's kind of an interesting take here to look for jailbreak like task patterns.
Derek Banks:Lots of small, testing. Like, frame is like, hey. I'm testing this thing. Right? And and so you guys look for patterns of data, which, you know, what do you use for that?
Derek Banks:Probably an LLM. Again, it's LLMs all the way down.
Brian Fehrman:Mhmm. Yep. We're building the robot army so
Derek Banks:robot army, that's right. My own personal LLM powered botnet. Right? Yeah. So you wanna move on to the next story?
Brian Fehrman:Yeah. Yeah. So next one is Kauai? Kauai? It might be Kauai.
Brian Fehrman:I'm gonna go with Kauai.
Derek Banks:Kauai?
Brian Fehrman:But not but not like the Hawaii Kauai. It's spelled a little bit different. It's for it's spelled more like if Hawaii had a had a K. We'll call it Kauai.
Derek Banks:Kauai? Kauai?
Brian Fehrman:Kauai. Kauai GPT. There's a free black hat LLM for script kitties, but I think it was more than just an LLM. Right? I think that it was kind of a it's more of like
Derek Banks:a tool
Brian Fehrman:set. Yeah. Like a
Derek Banks:So I went and looked briefly and because it's you know, it said the article said it was on GitHub. And was like, oh, cool. It's on GitHub. So I went and looked. And so there are a couple of things that showed up that produced four zero four.
Derek Banks:So I guess they're gone. But the one that I did find, it had two Python scripts, an install Python and then, you know, a a main Python. Right? And, well, I took a look at the Python as, you know, one is apt to do. And I immediately thought, this does not look like just a simple wrapper for an LLM.
Derek Banks:It looks a little bit, sus. Right? And so I did what, I tend to do these days, and I asked to chat GPT, could you go analyze this, link to this Python file and tell me what it what it does? And, here's what it said, the short version. It's a sketchy local AI assistant that phones home to a remote server, tries very hard to bypass safety, and can literally run shell commands the AI suggests on your machine via a special hacking mode.
Derek Banks:It's not just a harmless chatbot, and then it gives me a big breakdown of how it works.
Brian Fehrman:Oh, that's awesome.
Derek Banks:Well, no.
Brian Fehrman:I love when the I love when the LM's throwing, like, a little bit of personality, like, throwing in, like, the word sketchy. And, like, I know I was troubleshooting something on a Kali system just yesterday, and I was giving, you know, output to an LM, and it was like, oh, this is Kali being extra helpful with a smiley face. You know? And yeah.
Derek Banks:I I did one recently and I I I can't even remember. There's something helping with a pen test. Right? And I said, well, let's do this now. Right?
Derek Banks:And it said, hell, yeah. Let's do this. I was like, that is hilarious. So I I like the extra flavor as well.
Brian Fehrman:Yeah. Yeah. Certainly. So yeah, so this this yeah. This this tool kit, I think that it's a it's a good example of that people need to be skeptical or scrutinizing of different tools that you find, especially for free out there.
Brian Fehrman:I mean, that's there are tons of good free tools out there that, I mean, are well heavily used throughout the industry. And I know there's have been people who have tried to make points before about, you know, people blindly grabbing down and using things. I remember this was quite a few years ago that, a maintainer of one of the heavily used tool sets that was out on GitHub actually put something into one of the files that had said like, hey, you know, this is just a heads up or just a reality check that y'all are running stuff without checking it. And, you
Derek Banks:know, there was that.
Brian Fehrman:Do you remember that? Yeah. There was a whole conversation in the industry of, like, you know, come on. I mean, yeah, I get your point, but, like, at some point, you gotta trust someone and
Derek Banks:Yeah. Well, I I I think that I I'm less than a trusting person, but I am I will admit, I am guilty of, just running things sometimes, especially PIP installing, trying to troubleshoot something. Right?
Brian Fehrman:It's the
Derek Banks:same kind of thing. Be aware of what you're installing. But it's a very difficult kind of thing, specifically for this LLM because I I think that that supply chain kind of vulnerability with, you know, PIP packages or or or running models off a hugging face, you know, I I think that you just have to be careful. Right? And so some of the capabilities, like you said, there's tons of free stuff out there.
Derek Banks:Let's say that you were in the market for a black hat LLM, that is going to help you with your hacky hacks. And you want things like polished phishing emails, ransomware notes, Python scripts for lateral movement and exfiltration, full ransomware work workflows. Well, you can do that now for free. You can just install Olama and have a big enough video card to run some of the obliterated models that are out there on just like Olama. And and, you know, you can then, get it to create I've actually tried this on an obliterated model.
Derek Banks:Create me a full ransomware campaign complete with code, and they will happily oblige. And so I did all that without having to give up, the access to my, my actual system through through, you know, back doors. Right?
Brian Fehrman:Yeah. Yep. Certainly.
Derek Banks:So but hey, I mean, I guess, know, it is for for script kitties. And so some of the things that we had is, you know, other talking points, democratized offense. So you need to know PowerShell and c two, to, you know, pull off command and control and and getting a a, you know, a shell on someone's system. Well, yeah, it's a lot easier than it used to be because you can just get, you know, an LLM to write you the code. But it's been my experience that you still have to know a little bit or Mhmm.
Derek Banks:Because it's not always completely right or or functioning. Yeah.
Brian Fehrman:Yeah. Certainly. And I mean, again, if you're I mean, if you're doing this within a customer environment, it's good to know so that you can go through and you can review. I mean, what what is this doing? Does this look correct?
Brian Fehrman:I mean, it's not, you know, we're talking earlier about the difference between, like, cutting out a file versus r m ing a file to show a proof of concept. Right? Yeah. It's important to be able to go through and understand the difference of like, oh, yeah. Now this is gonna be harmless.
Brian Fehrman:I understand what it's doing here versus just like, okay. Yeah. You know, YOLO. Let's see what happens.
Derek Banks:Yeah. So some other takeaways. Assume phishing content is now human quality by default. Yeah. That's been a thing with large language models for a couple years now, I think.
Derek Banks:Mhmm. That's even baked into some security products. I can't remember. Think it was, I think, maybe Darktrace or someone like that, had, we had a customer who wanted our opinion on a module that, they were trying to sell them. And it was it basically would hook into, o three sixty five or m three sixty five.
Derek Banks:And for a given user generate a phishing email based on their, like, emails. And it was it was astoundingly good, and this was, like, two years ago. So it ain't gotten worse. Right? And I almost felt like it just wasn't fair.
Derek Banks:Right? Like Yeah. That's not fair. But I mean, as a threat actor, if I got access to, you know, company email and I was able to have a, you know, generate, phishing campaigns based on your emails. Oh, man.
Brian Fehrman:Oh, yeah. That's a powerful tool. About a day.
Derek Banks:And then, Blue Team should add detections around unusual SSH automation and email exfiltration. Yeah. I I I've been saying this for a while. Was it 2019, I think, when Microsoft is starting including open SSH in Windows desktops and servers. Right?
Derek Banks:And it's still to this day very, uncommon for me to for us to find that SSH has been locked down outbound out of an environment. I wouldn't allow it, period. Yeah. No. At all.
Derek Banks:And I would take it off the machines. Actually, I I I wanna I heard over I saw one of our testers recently saying something in a meeting that they the customer they had had taken SSH off the systems, but they were able just to go to the Windows store and add it back.
Brian Fehrman:So It just reinstalled.
Derek Banks:Still need those network protections. Network protections aren't aren't dead at all. But
Brian Fehrman:Yep. Yep. Yep. Traditional security stuff still still applies.
Derek Banks:Yeah.
Brian Fehrman:Alright. So we have the next one.
Derek Banks:Pretty interesting. The ShadowMQ, and and inference engines. It's kind of attacking a lower layer below the the LLM itself. Right?
Brian Fehrman:Yeah. Yeah. It sounds like it's part of the the inference stack that a lot of different companies use. Some examples are Meta's Llama Stack, Nvidia Tensor RTLM, BLM, SGLing, as well as a couple others. And it's dubbed ShadowMQ because it comes from the use of the ZeroMQ, as well as the Python pickle deserialization.
Brian Fehrman:I don't recall that protocol format functionality.
Derek Banks:Like I said, in our, you know, well, it wasn't pre show, but before the show. It is the time of the year for this joke, pickle. It's like the jelly of the month club. It's the gift that keeps on you and Clark. I I just I again, with pickle deserialization, this was an issue, a while back on Hugging Face.
Derek Banks:Right?
Brian Fehrman:Yeah. Yep. Yeah. Exactly. With the issue so for those who aren't familiar, basically, a pickle file is a way that you can store a trained model, and, different properties that are associated with it.
Brian Fehrman:So you can grab it down, you can load it up and start using it. The security issue with it is that you can basically put in arbitrary code that will get executed when the model is loaded. So it's it's like, it's trivial to backdoor, basically, to throw to throw in, arbitrary executable code into these files and someone goes and load it and boom, you've got code execution just like that. So there are other formats that have come out that I feel have kind of superseded it, which are, like, safe tensor format where that's just that's just not a thing. Like, you just can't do the code execution in there and it's a much much better way.
Brian Fehrman:I think there are a couple others out there too. The names are escaping me, but, you know, you gotta be the bottom line, you gotta be very careful if you're dealing with a pickle file that is not that you did not create.
Derek Banks:And I guess they, the article referred to this as the Log four j moment for AI infrastructure design. Oh, good.
Brian Fehrman:Yeah. Yeah.
Derek Banks:That sounds pretty serious.
Brian Fehrman:Yeah. So yeah. So I think, you know, kind of just the main thing, if you're using any really any any part of some of these major stacks, you really need to check to make sure that you're you're fully patched up because this was considered it was ranked with almost like a over a nine severity rating for RCE, And it's certainly a serious deal to make sure that if you've got, you know, some of these, some of these components exposed out to the Internet, wanna make sure that they are, you know, patched up, ready to go.
Derek Banks:Yeah. Again, with exposing things to the Internet is not always as easy as one thinks it should be. Right? Looks like the CVS scores are 9.3 to 9.8. There must be multiple CVEs.
Brian Fehrman:Mhmm. Yep.
Derek Banks:Well, that's, kind of scary.
Brian Fehrman:Yeah. Oh, yeah. Yep. I mean, well, you know, it's again, I mean, AI is just it's a it's another product, and it's gonna have issues like any of the other Internet facing products that that we see out there. Well So it's just it's important to oh, go ahead.
Derek Banks:No. I was gonna say because this is all still new to companies, I think they're forgetting some lessons they probably would have otherwise never done. Right? Like, you're gonna stick something out on the Internet that's running somebody else's Python code for and there's, like, this huge, like, you know, software library of code that you're leaning on that you haven't gone and checked through. And and it just seems like to me that kind of, like, as a whole, since the AI industry is moving so fast, like, and the code is just an afterthought.
Derek Banks:When you're bolting together a bunch of this kind of stuff, I I I think, you know, the pioneers are going to be taking the arrows, so to speak. Mhmm.
Brian Fehrman:Yeah. Yep. And certainly, it's the you know, as we've mentioned many times before, it's it's the same problem we've we've seen with a lot of new technologies is that there's a giant push to get these technologies customer facing or even employee facing as quickly as possible and security comes later. But just have to have to be careful and stay up on the latest. Alright.
Brian Fehrman:Here at the next one. This one just a quick one to mention, which is that Amazon Nova, which I think is Amazon's line of, models. Yeah.
Derek Banks:Yeah. Actually, I've used it. It's not bad.
Brian Fehrman:Yeah? Nice. Yeah.
Derek Banks:I used
Brian Fehrman:it for bedrock.
Derek Banks:I've seen around.
Brian Fehrman:Yeah. Like, I've seen them on their, like, their store or whatever you wanna call it within their their interface, but haven't really played around with them much. But they Yeah. They released their own private AI bug bounty program that is, kind of in addition to the one that they've already got going on, with HackerOne.
Derek Banks:It's pretty interesting. Amazon is, I guess, I'll I'll I'm not being critical, but I kinda late to the AI party in a lot of ways. Right? And but I think they're coming on pretty fast. I actually just read a a a semi analysis, article about how they're essentially using now, like, custom, like, I guess, I can't custom AI chips now.
Derek Banks:I guess Amazon AWS has a lot of custom hardware in their, you know, AWS infrastructure. And I guess now, you know, that's even going all the way down to, like, the silicone layer instead of using NVIDIA chips. They're, like, now using, like, custom AI chips for stuff. And so I'm glad to see that they have a bug bounty program trying to find issues before, you know, before things happen, like, in the previous article where there's a Log four j moment in their AWS AI infrastructure. So very good.
Derek Banks:Although, I wish I was invited to the private bug bounty. Oh,
Brian Fehrman:yeah. Same. No. I was reading through the article, and it sounds like it was, quite lucrative for those who, got invited that they're paid a large sum of money upfront and then were paid out additional money as they found things as they went through.
Derek Banks:I mean, I like money.
Brian Fehrman:Especially when large
Derek Banks:goes with the amount. So Yes. That kid's going to college.
Brian Fehrman:So Yep. Yeah. So I think this is this is good. It's always good to see just bug bounty programs in general because it's good, you know, if you're you know, customer of really any service, but I mean, especially these AI services, it's good to ask and or at least do some research to see, mean, are they participating in bug bounty programs and do they regularly get tested? So that way that you know that, like, hey, if you're handing over your data to this company, sure they say that they they don't use it for training.
Brian Fehrman:They're just storing it. Well, like, how safe is that storage? And how safe is their infrastructure? How safe are, you know, everything that you're dealing with in in general? I mean, how often does that get tested?
Derek Banks:And we we were just on a customer call yesterday with a a customer that was implementing a third party AI solution, and they basically said that the vendor did not want them pen testing their products. And I I to me, that's kind of a red flag. Right? Like, well, okay. Now now I really want to.
Derek Banks:Right? And so I'd like to see what's, you know, the man behind the curtain now. And so I think that that should be a red flag for you if you're going to implement some kind of AI solution and ask the vendor about security and penetration testing and they're just like, oh, yeah. No. We're not gonna do that.
Derek Banks:That's probably not the answer you wanna hear.
Brian Fehrman:No. That's it's a bit suspicious. As a Yeah. What was that? John Strand, I think, had the a quote from him recently.
Brian Fehrman:It's kind of like not not wanting people to look in the kitchen. He was referring to something different, but I think it still applies here.
Derek Banks:Yeah. You wouldn't eat at a restaurant if they wouldn't let you look in the kitchen.
Brian Fehrman:Yeah. Then be a little skeptical.
Derek Banks:Yeah. I mean, I like the analogy, but also, like, if I'm running a restaurant, I don't want people up in the kitchen. That sounds like a health code violation.
Brian Fehrman:True. True. True. Yeah. I
Derek Banks:get the I guess that was authorized.
Brian Fehrman:Yeah. Yeah. Yep. Alright. Alright.
Brian Fehrman:Let's head on the next one. So this one actually came from a newsletter from that I I got from some of our friends over in The UK at a company called MindGuard, who we've been in connection with over the past year or two, who deal with AI security testing. But I guess Google launched an agentic AI IDE that is powered by Gemini three. And within twenty four hours, someone from MindGuard found, what they're calling a persistent code execution vulnerability within this new, basically this new IDE that, Google dropped.
Derek Banks:That doesn't sound good. It says the bug effectively turns a compromised project into a long lived backdoor for arbitrary code execution under the user's own identity. That sounds terrible.
Brian Fehrman:Oh, yeah.
Derek Banks:But I guess, so is this any different than a malicious, like, you know, Visual Studio, project? Like, if I'm targeting developers of a company and I know they use Visual Studio and I was able to trick them into running something in in compromising Visual Studio because they trusted the project. Is this any different though?
Brian Fehrman:Oh, I think it's it's it's probably similar because, you know, as you're mentioning that that's I was I was thinking of that, you know, when you're when you're in Visual Studio Code, it'll you know, it'll always ask like, hey, do you actually trust this as you load? And I mean
Derek Banks:Of course I do.
Brian Fehrman:Of course. Very trustworthy folder.
Derek Banks:That's right.
Brian Fehrman:Yeah. So
Derek Banks:yeah. I mean, it but again, I I think that if I I haven't used the product and, if if Google isn't, putting those kinds of safeguards in place, like, sure you trust this and because that's what Versus Code does. Right? It's like, hey. Make sure you trust this.
Derek Banks:Bad things could happen. Yeah. And and so yeah. But, I mean, hey. My first thought when I read this was, it it usually only takes about a day for new AI products to get popped.
Derek Banks:Right? Pliny the prompter usually gets a a jailbroken LLM. Well, sometimes even before they really release it, which is kinda interesting. But, yes. That was my first thought.
Derek Banks:I was like, oh, is this plenty of the prompter?
Brian Fehrman:Yeah. So I will say, the it mentioned that Google has acknowledged the issue and said they're gonna work work on a fix, but others have pointed out that it's just kind of structural to how agented tools are
Derek Banks:being shipped. Yeah. Yeah.
Brian Fehrman:It's not, you know, like something that that was done wrong, but it's just, you know, it's I mean, like you're saying, I mean, it's just a matter of like, you you gotta be careful and you make sure that you trust whatever it is that you're loading in. So you
Derek Banks:don't It says The flaw is structural to how agentic tools are being shipped, high autonomy, broad access, and weak guardrails. I think that's all agentic and MCP kind of stuff at the moment. Right?
Brian Fehrman:Oh, yeah. Gotta be
Derek Banks:now, like, back to the previous, you know, the second story. You just gotta be real careful about what you're, running off the Internet.
Brian Fehrman:Yes.
Derek Banks:Treat AI coding and IDEs as high risk software. Yeah. I mean, I think all development, can be that way. In fact, we've had clients in the past that were like, well, just give all the developers admin rights. I'm like, I I think you ought to not do that.
Derek Banks:That sounds like a terrible idea. I I know that I've been on, you know, pen tests in the past where I was able to get on developer workstation and they had lax, you know, controls on developer workstation. And it was very beneficial to me as a pen tester, but not to the the company at all. That's how I got in. And, you know, for dev teams, don't blindly trust a workspace.
Derek Banks:Make sure you know what you're trusting.
Brian Fehrman:Yep. And Yeah. And then
Derek Banks:I usually run that stuff in a VM anyway. Not on my
Brian Fehrman:Yeah. It's a it's a good idea not to not to run on the the main system. And, yeah, another another point is don't just have the, you know, the run anything as AI mentality. Like, of the things I like about, like, Cloud Code, if you're using it as a as a coding assistant is that it asks you before it actually runs anything, which is nice so you can review it.
Derek Banks:I I really like that, feature kinda human in loop. Now you can tell it don't do that anymore. Right? But I I like to see
Brian Fehrman:Always trust this. Yeah.
Derek Banks:Yeah. I kinda like to see what's happening. I like to still be in the loop and say, yeah, I want you to do that. I don't think I've actually said no, but I've definitely read through, like, what it's trying to do. Oh, yeah.
Derek Banks:Yeah.
Brian Fehrman:Yeah. Absolutely. Like, I don't want it accidentally, like, you know, wiping out an entire directory or something or
Derek Banks:Removing that to password or something. Yeah. Yeah. Yeah. Exactly.
Derek Banks:Yeah.
Brian Fehrman:Alright. We'll hit the last
Derek Banks:one here. Last story.
Brian Fehrman:PromptFlux, which is a I call this a polymorphic malware that is using Gemini Gemini AI to rewrite itself hourly, calling out to Gemini's APIs to, basically, keep rewriting itself, as it goes. And it was v b script, I think, what the actual malware is in. V b script?
Derek Banks:Oh, that's a
Brian Fehrman:That's a throwback.
Derek Banks:A blast from the past. Right? I can't remember the last time I saw a VB script, but I'm No. I'm sure it still works on Windows machines like all these other weird things. Well, that that's really interesting too that it goes and rewrites itself on the fly.
Derek Banks:Mhmm. I I'm sure it's not the first example of it, but, it sure does have a great name. I like prompt flux. It reminds me of Aon flux.
Brian Fehrman:Oh, yeah. Right. I remember that show from back in the day.
Derek Banks:Prompt flux sounds like a a cool robot name. So Yes. Yeah. Yeah. Yeah.
Derek Banks:Right. Polymorphism.
Brian Fehrman:Mhmm. Oh, I'm just gonna say, yeah. You're right. I think, this isn't this isn't the first case of, AI powered polymorphism that we've seen. It's certainly not the first case of just polymorphism in general.
Brian Fehrman:Right? I mean, polymorphism been around for a little while, but, I mean, now that AI is becoming more heavily utilized and leveraged, we're seeing people actually using, doing this sort of thing where they are just using AI to just basically rewrite rewrite the malware on on the fly, which
Derek Banks:is pretty interesting. And apparently, the prompt is, one of the things that does make this different, is that it's not writing, the prompt in prose, that it's machine parsable. So it's basically, like, you know, doing more, like, code parsable type stuff and not, like, paragraphs to rewrite, which is kind of interesting. Kinda like with these prompts.
Brian Fehrman:Oh, yeah. Yeah. I I completely agree. Yeah. And it's, it also tries to propagate, it looks like, too.
Brian Fehrman:So it's not just, you know, not just like, I don't know, like a a foothold, necessarily, but it's actually like worms itself, throughout removable drives and, network shares and, also ever mutating itself as it does, not only onto the new systems, but onto on the current system, it sounds like.
Derek Banks:So you combine that with the first story, right, which was the AI agentic hacking thing and this put then this, you know, self replicating. Now we are in the Daniel Suarez daemon territory, where, in a couple of years, are we gonna be talking about, AGI that's loose on the Internet as its own, like, you know, nation state hacker group, and they can't, like, get the Internet clean of it. That's gonna be that's how we know we really are living in a simulation when that happens. So
Brian Fehrman:Oh, yeah. Yeah. I think I think it's inevitable. It's gonna just start hiding itself and everywhere. It's gonna be, like, you know, preserving itself within, like, Reddit sub forums and
Derek Banks:all that.
Brian Fehrman:Like, it's never gonna
Derek Banks:like a movie. It's just like a movie. I remember was it Black Hat? I think one of the really crappy, like, hacker movies where, like, the I don't know if it was that one where, like, the backup code was, like, on some tape thing that was in some basement of some data center somewhere and it's like reel to reel tape going and that's where it stored itself like, man.
Brian Fehrman:Yep.
Derek Banks:I guess we are getting into this science fiction is reality territory.
Brian Fehrman:Oh, yes. Yep. It is here.
Derek Banks:Good times.
Brian Fehrman:Yep. So I guess takeaways on this one, obviously, you know, it's we're not not saying that people shouldn't be using signature approaches. I mean, it's all it's always about layers. Right? Don't throw out something just because it doesn't work 100% of the time.
Brian Fehrman:So signature stuff, definitely still keep that in in place, but realize that obviously it's not gonna catch things like this that are constantly rewriting themselves. So it's really important to look more towards, behavioral and, telemetry based approaches for detection. But on just the, you know, the prevention side, obviously, you know, making sure that your network is locked down, looking at how your systems are able to communicate with one another, who has access where, still all the, you know, same security practices. Right? So even if a box gets popped, you can at least mitigate the the damage, so to speak.
Derek Banks:Yeah. I like the, the line item here. Look for suspicious calls out to LLMs from scripts. I mean, doing that at a at a host, like, at a host level is very, beneficial. In fact, we have clients now who are using, EDR to look for, calls out to large language models that aren't approved.
Derek Banks:I I I would definitely if it were my network, that's what I would be doing is Mhmm. Or or you're not supposed to be using Gemini. Why are you using Gemini? Why is Visual Basic Script using Gemini? Oh, crap.
Brian Fehrman:Yeah. Let's see. And it keeps calling out to it at every hour on the hour.
Derek Banks:Yeah. Not recommended in Better Homes and Gardens.
Brian Fehrman:No.
Derek Banks:Well Alright. I think we made it through the stories.
Brian Fehrman:Yeah. Excellent. Well, so thanks everyone for tuning in again for, another episode. I hope you enjoyed our take on the news stories, and tune in for our next one. And as always, keep safe and keep on prompting.
Creators and Guests
