This transcript was generated automatically by AI. If you find any mistakes, please email us.
Disesdi Shoshana Cox (00:00)
These box bases are absolutely massive. We've determined that it's not feasible computationally to run a search through them. So what that means is we can't search and find all the attacks. So with a traditional pen test, you can pretty much find all the attacks that you know may be relevant to a system. You cannot do that with AI systems. It's not really possible to do. So what you can do, because there are effectively an infinite number of attacks, is
You know, spray a bunch of prompts at something and say, my God, see look, we all these attacks worked. And then look, I blocked them all now. Like you can do that, but did you actually really provide meaningful security? ⁓ I would argue no, you absolutely did not. I think that is the crux of what's wrong with AI red teaming right now.
Ganesh The Awesome (00:48)
Hello everyone, you're listening to Cloud Next, your go-to source for cloud innovation and leaders insight, brought to you by Global Dots.
Our guest today is Shoshana Cox, an AI security researcher focused on how AI systems actually fail in practice. Back in 2022, she approached this space from a defense perspective, assuming many of the core problems were already understood. But over time, she realized the industry's understanding of AI security was far more broken than it seemed, something that led her to challenge some of the dominant approaches we see today. I'm going to show you some solutions architect at GlobalDots, where we research innovations every day so you don't have to.
We're not here to talk about him. Me, we're here to talk with Shoshana. Shoshana, thanks for joining us. So great to have you.
Disesdi Shoshana Cox (01:36)
I'm so happy to be here. Thank you so much for having me.
Ganesh The Awesome (01:39)
Before we get into the juicy stuff, I really want you to outline your history in AI and how you came to this point where you are today, because I know you're a rock star and I followed you on LinkedIn and read a lot of your posts, but a lot of people would claim to be AI experts and they're definitely not. And I want ⁓ the listener to understand that you definitely are and for you to lay down the credentials so that people know they should be listening, basically.
Disesdi Shoshana Cox (02:07)
Yeah, so I've
Disesdi Shoshana Cox (02:09)
been
doing this for a really long time. I I started working with AI and security back before it was cool, ⁓ before the Gen AI revolution in 2022. ⁓ so I was back in about 2010 working on ⁓ looking at how these systems could break. And what I saw was very disturbing and ⁓ there was an a great amount of literature ⁓ out there for how to mathematically you know attack these systems back then and I just sort of
Disesdi Shoshana Cox (02:26)
What's
Disesdi Shoshana Cox (02:38)
dove into it. So we found that there were ⁓ serious ⁓ vectors for attack for all AI systems from predictive AI to now including generative AI and obviously ⁓ agentic inherits those vulnerabilities as well. But go back ⁓ to the beginning of AI itself. So there and anyone who's been working on this for a very long time certainly recognizes the new quote unquote new vulnerabilities in prompt injections.
Disesdi Shoshana Cox (02:41)
Found
These go.
Disesdi Shoshana Cox (03:08)
but yeah, that's how I got started. I was a red teamer initially and then started poking around in AI. ⁓ and since then ⁓ I wrote a paper in 2022 that was the first ⁓ AI security operations paper, machine learning security operations, whatever you want to call it, ⁓ with the first architecture for exactly how to implement this defensive ⁓ security. And then I ⁓ I got a patent ⁓ in 2024 for ⁓ defensive federated architecture.
And I've been working on the OWASP AI Exchange, serving on the core author team there. ⁓ I do work on federated AI. ⁓ I am also the AI policy lead, and I served on the team that helped prepare requirements for the EU AI Act's technical sections. So that's kind of what I've been working on up until now. I started my own startup a few months ago. So we're building in stealth and
Yeah, I I kind of enjoy working at the bleeding edge of AI research and security.
Ganesh The Awesome (04:05)
Very cool, very long list of accolades that nobody could possibly argue with. And specifically around this point, I'd love for you to flesh out AI red teaming. if you can start with so people understand what AI red teaming is before we jump into it, just so they know we're talking about the right thing. And then if you can break down for us the myth of that and how that has pervaded into software today, because I think it's very interesting.
Disesdi Shoshana Cox (04:34)
Yeah, so I mean
Disesdi Shoshana Cox (04:36)
That's kind of problem though, isn't it? Like what is AI red teaming? Well, I I don't know how much people know about security testing and so forth. There's offensive and defensive security. And ⁓ the defenders are the ones that are hardening systems and trying to make sure hackers don't get in. ⁓ and then you also have good guys who attack systems professionally. And those are the red teamers, those are the offensive security professionals.
Disesdi Shoshana Cox (04:39)
Up so for people I don't know
Disesdi Shoshana Cox (05:01)
so the idea ⁓ is you can do anything from, you know, a penetration test and test someone's application or network security all the way up to a full red team engagement, which typically in security meant that you would be ⁓ testing all types of systems potentially, like not just the computer systems, but also potentially physical security. ⁓ social engineering might be on the table in the rules of engagement.
Disesdi Shoshana Cox (05:25)
So this idea of red
teaming
Disesdi Shoshana Cox (05:27)
Is the idea that we are the red team, that's the offensive team, and we are going to just attack this from all possible angles. So ⁓ AI didn't used to have that back in the day. It was this AI security was its own thing, but it wasn't called red teaming. You could attack these models, you could test for them. When it started becoming ⁓ red teaming, to my understanding, is after the Gen AI ⁓ big launch and the adoption. And what happened was people found that these models had vulnerability.
vulnerabilities, excuse me. How did they find that? ⁓ they found it by typing in natural language prompts to see if they could trick the model. So I my understanding is that this was analogized to the social engineering aspect because they kind of said, okay, prompt engineering or prompt engineering, prompt injection, so forth, this is social engineering, right? So when we do it for security, we are red teaming the AI. There's your
Disesdi Shoshana Cox (06:04)
believe
So
Disesdi Shoshana Cox (06:23)
Long winded back history of why it came to be called that. If you haven't noticed by now, I disagree with that nomenclature. I do not think that is the right name. I think it should be called something else, but not that.
Ganesh The Awesome (06:34)
Let's pick a name just for fun. What would it be if we called it something sensible?
Disesdi Shoshana Cox (06:38)
Ha
ha
Disesdi Shoshana Cox (06:39)
So what is being done right now on the internet?
Disesdi Shoshana Cox (06:40)
Administrate
nothing, you should not do that at all. if you were going to actually test AI systems, I would call it AI security test. ⁓ if you really wanted to do a red team operation, to my thinking you need to like show up on site, like on prem, and see if you can, I don't know, ⁓ attack the model somehow that way. But ⁓ yeah, that's that's how it got its name. I prefer security testing. It's more straightforward. So and hopefully conveys what you're actually trying to do.
Ganesh The Awesome (07:09)
So like, definition, the real AI red teaming would be turning up with a parcel at the people's office and seeing if you can get in there and then attack the AI machine with a baseball bat or something like that.
Disesdi Shoshana Cox (07:22)
Right one to badge me in that's usually the easiest way to get in is follow somebody and then like I don't know, I'll find the the Gibson server room and
Ganesh The Awesome (07:31)
proper Hollywood style, just with like the all-gray boiler suit, you come in as the cleaner or something like that. So let's go back to why it doesn't work though, because let's just say that I work in an industry where I've had people demonstrate things to me that definitely products claiming to stop this. And if you talk about prompt injection, that's…
Disesdi Shoshana Cox (07:40)
Ooh, I like it.
Ganesh The Awesome (07:58)
I wouldn't say totally understood, but maybe understood to some level. And you know that you can stop some prompts coming in, ironically, by using AI to listen to the language that's coming in or stopping certain keywords. essentially that is unstoppable in like mathematically provable that it is not something you can defend against. That's what you're saying. That's correct.
Disesdi Shoshana Cox (08:21)
Yeah, and and you know I would argue that using AI to to monitor these things is ⁓ just a bad idea all around. ⁓ shouldn't be done. Not a good idea from an ar architectural or security standpoint, but that's another conversation, maybe. ⁓ yeah, so mathematically we understand that ⁓ every model, let me back up a little bit actually. Every model comes with ⁓ a set of attacks that will work against it. ⁓ and
Disesdi Shoshana Cox (08:49)
This is like it
Disesdi Shoshana Cox (08:50)
called a subspace, an adversarial subspace, is you can see a box that's attached to the model. And these are all the attacks that'll work against that model, right? And absolutely massive. We've determined that it's not feasible computationally to run a search through them. ⁓ They're they're pretty huge. They're sparsely clustered. It's it's a mess down there. So what that means we can't search and find all the attacks. So with a
Disesdi Shoshana Cox (08:53)
Think of it like
These these boxes, these spaces are actually determined.
Traditional pen test you
Disesdi Shoshana Cox (09:17)
can pretty much
Disesdi Shoshana Cox (09:18)
pretty much
Disesdi Shoshana Cox (09:18)
find all the attacks that you know may be relevant to it. You cannot do that with AI systems. It's not really possible to do. So ⁓ what you can do, because there are effectively an infinite number of attacks is, you know, spray a bunch of prompts at something and say, my God, see look, we all these attacks worked. And then look, I blocked them all now. Like you can do that, but did you actually really provide meaningful security? ⁓ I would argue no, you absolutely did not.
Disesdi Shoshana Cox (09:22)
System.
Disesdi Shoshana Cox (09:47)
I think that is the crux of what's wrong with AI red teaming right now.
Ganesh The Awesome (09:51)
So these are like basically the Donald Rumsfelds of AIs. These are like the unknown unknowns basically.
Disesdi Shoshana Cox (09:58)
Well the dumb the drums of AI. ⁓
Disesdi Shoshana Cox (10:00)
Donald.
Ganesh The Awesome (10:03)
No, you never heard that. There's a famous quote about Donald Rumsfeld when he was talking about the no-nones. It was talking about the war in Iraq or something. He was like, we've got no-nones, we've got unknown unknowns, and we've got unknown unknowns. It was like the most ridiculous statement ever, humans ever made in politics. Anyway.
Disesdi Shoshana Cox (10:19)
No, I I'm sorry. I was only aware that Al Gore invented the internet since we are talking about ridiculous statements. So I've learned something today. Thank you.
Disesdi Shoshana Cox (10:27)
Political state.
Ganesh The Awesome (10:30)
Tim Berners-Lee might have had something to say about that, but that's okay. He's a penniless guy somewhere in the UK living in a very conservative house. these unknown unknowns though, because you don't… I'm sorry, I forget what you called the space. What did you call this like?
Disesdi Shoshana Cox (10:50)
Adversarial. Like it's trying to fight.
Ganesh The Awesome (10:53)
I
had to say, but if you don't know they're there, or we don't know the size of them, like, just for my brain and probably everybody else's brain who's new to this as a topic, like, how do they get there? Or how do you ever measure this space? Or a better question, what the hell do you do about it if it's this like unknown mystery box that lives within inside these models?
Disesdi Shoshana Cox (11:18)
So that's three questions. they get there because whenever you item
Disesdi Shoshana Cox (11:20)
I will start with the first one. ⁓
Compress a high dimensional
space into a lower-dimensional one, ⁓ you break certain assumptions, right? You create angular representations among the concepts that the model so quote unquote learns, right? But those are not as rich in information as the actual relationships, right? So if we model something from our daily life, we are ⁓ inherently presenting less detail than the detail we ourselves understand.
Disesdi Shoshana Cox (11:38)
Those as rich.
Disesdi Shoshana Cox (11:52)
Because we have to, because you know, you're simplifying something for modeling purposes. Same in AI. so when you make these models, ⁓ boom, all of that compression, right? So you create this adversarial space. Like now you have these angles where attacks are possible. ⁓ and that's just an artifact of changing the dimensionality of a representation. ⁓ it's I can say that and people are like, what? So I I'm happy to go into more detail, but on to question two.
Disesdi Shoshana Cox (11:56)
Exactly.
Ganesh The Awesome (12:22)
I like it. It sounds very, I was going to say it sounds very Stephen Hawkins. I don't want to talk about that guy because he now appeared in the Epstein. It sounds very quantum mechanicsy, which, you know, there's lots of very cool words there that all came together in a beautiful sentence, it totally makes sense. Sorry, I interrupted. On to the next part.
Disesdi Shoshana Cox (12:47)
Yeah, so papers
Disesdi Shoshana Cox (12:48)
My paper that
came out a few months ago with my co-author, Nicholas Spenzel. If you're not following him on LinkedIn, you should be. ⁓ he's he's done brilliant foundational work in this space, like actual mathematics and engineering. ⁓ so that paper laid out a system by which you can start to map the boundaries ⁓ using some pretty cool new techniques. One's called centered kernel alignment to measure similarities of models and some kind of hacker techniques.
Disesdi Shoshana Cox (13:06)
of these subspaces. Cool.
So
didn't
Disesdi Shoshana Cox (13:16)
There,
⁓ kind of doing things the way hackers do it a little bit.
Disesdi Shoshana Cox (13:20)
So the marriage of these of these two
Disesdi Shoshana Cox (13:22)
methodologies,
the mathematics and the just a a little teeny bit of black hat if you will, ⁓ wink, ⁓ I think is a a good for anyone who wants to do serious research on that. So that's how do we measure them, but more to the point, what do we do about that? ⁓ this is where threat modeling comes in. Absolutely cannot defend against every possible attack to your system if you are running AI. ⁓ we can back up a little bit maybe you don't need AI for that, right? Like
Disesdi Shoshana Cox (13:30)
Good starting place?
This is
Back and same.
If
you don't you shouldn't put it because these are intractable
Disesdi Shoshana Cox (13:51)
You don't need AI, you just put it in because trackable
vulnerabilities that you will be responsible for. All right, if you're dead set and you and you need AI and that's the solution for you, you need to start with threat modeling because the threat model is going to tell you what to test for. And that way you're not trying to bowl the ocean, you're not trying to just spray random prompts that somebody told you were good. You know, you're not wasting money on the tokens, the time, the red team, the engagement.
Disesdi Shoshana Cox (14:16)
Money.
Disesdi Shoshana Cox (14:20)
You have a threat model that says, This is what I want to test for and this is how let's go.
Ganesh The Awesome (14:24)
The horses bolted on that one because it's too late. Everybody decided they wanted AI everywhere all the time. now everybody's obviously backpedaling trying to solve it and scrambling to do anything to help. And Lord knows, I'm 43 years old at this point. I'm like paying catch up. I see all these kids doing things and I'm like, my God.
Why is an open core running my whole life? Maybe I'm doomed. But the people who are in the high positions are even older than me and they definitely don't have a clue what's going on. And even something like, hey, you just need to provide a threat model for this. I'm pretty sure the thing they're going to do is go straight to like ChatGPT or Gemini and say, how do I build a threat model? Because nobody really knows. So people are just desperate. And this is why I totally…
Hook, Line and Sinker fell for all of these various prompt injection, AI gateway, AI red team, whatever. There's a whole suite of people making a whole lot of money out of venture capitalists building these things. And to some extent they appear to work and they've got a flashy UI and blah, blah. But like, it's it's better than doing nothing, but I feel like, you know, it's a bit like a…
Well, in the UK I'd say it's like a chocolate fire guard, basically. That's what it appears to be.
Disesdi Shoshana Cox (15:54)
What?
Disesdi Shoshana Cox (15:55)
I said the American
Ganesh The Awesome (15:57)
A chocolate fire guard. So a fire guard is like a piece of metal that you put in front of your fireplace so that the sparks don't come into your house. I could have said a chocolate teapot. That probably makes more sense. It's basically like a chocolate teapot. So what apart from like offering people that they do some threat modeling, know, for the people whose the horse is bolted, like what?
Disesdi Shoshana Cox (16:06)
Okay, okay, I get it now.
Ganesh The Awesome (16:24)
What's the, what's what's the, what's basic advice? What's like real stuff that they can do where they're not gonna waste money?
Disesdi Shoshana Cox (16:33)
⁓ you got a threat model, man. That's it. That's it. That's you you you guys you guys is out. If you wouldn't put it like that, I would put it a little less delicately personally, but you guys are gonna you're gonna have to do the threat model or else you're gonna be liable when this thing breaks. Now now now consider if it's an internal chat bot or something, maybe you have a a limited blast radius for what can go wrong when you start deploying agents, when you start deploying something publicly, ⁓ you're effective
giving it hands and mouth to speak to the world. So responsible for that. And I if if I were you I would take that responsibility seriously because governments are already showing that they are.
Disesdi Shoshana Cox (17:05)
You're gonna be responsible.
Ganesh The Awesome (17:15)
Well, I seem to see something in the news literally on a daily basis about something atrocious that a chatbot has said or some ridiculous, you know, 99 % discount on giving a hotel room or whatever. It's basically like a non-stop reel of stupid things that people have, like AI has basically done because people have tricked it.
Disesdi Shoshana Cox (17:41)
That's
all it ever does is it tells people to kill them
Disesdi Shoshana Cox (17:43)
themselves
and then like gives them incredible discounts. I don't know.
Disesdi Shoshana Cox (17:45)
Those are the two you changed.
Ganesh The Awesome (17:49)
Yeah, ⁓
well, I mean, they're like machines that are trained on 10 years of Reddit posts, you know, they're, I could only imagine they're highly sarcastic and all kinds of things.
Disesdi Shoshana Cox (18:03)
God, it's way worse than that. It's way worse. I won't I I this is my very depressing, but the things you're trained on way worse than Reddit. They're trained on the open internet. Like Reddit is nice compared to some of the things these things have seen. That's why they have the reinforcement learning with human feedback. And that's why the people that do that get PTSD. These are not nice machines and they must literally be trained to not be totally evil.
Disesdi Shoshana Cox (18:08)
My soapbox that's very
Ganesh The Awesome (18:30)
Yeah, I have seen like a mini documentary about that because there's basically paid by the penny kind of our workers out in wherever name a sort of third world country where it's cheap for the labor and they just have to sit there and go through these obscene responses and filter the results. yeah, this is like…
Well, yeah, people are getting sick, but humans are getting literally sick training these things because they're so vile what comes out of them. I would love to go down that rabbit hole, but maybe it's a bit too depressing.
Disesdi Shoshana Cox (19:08)
It's a little I told you.
Disesdi Shoshana Cox (19:09)
That I know nobody
wants to hear this because I I I'd say it and people are like, Wow, that's
Disesdi Shoshana Cox (19:14)
Anyways, what can we
Ganesh The Awesome (19:17)
Well,
you know, we live in a very, very strange industry because all of the, you know, I'm chatting on a Mac book and a whatever Phillips screen full of microchips that probably were artisanally mined by some poor bastard out in the Democratic Republic of Congo. And that's a whole massive shit show also. So we're just like stacking bad karma on top of bad karma, basically, to build a super AI bad karma machine. But that's like a…
That's a spiritual vent that is definitely probably not going to resonate too much with the tech audience, but who knows? Maybe they will.
Disesdi Shoshana Cox (19:53)
No, let's go.
Disesdi Shoshana Cox (19:54)
Down that rabbit hole, let's construct
the Tower of Babel. Why not?
Ganesh The Awesome (19:59)
Great, let's go there. Well, basically, guys, it looks like this. Karma is a real thing. You have a soul and what you do to the planet is going to come back to get you. So if you build everything out of bad things that's crushing loads of people and then you build other stuff on top of that, well, surprise, surprise, the end result is a bad thing. So that's where we're heading. Super AI evil end of game machine.
Disesdi Shoshana Cox (20:21)
We sound old and also you kids
Disesdi Shoshana Cox (20:22)
I need
to get off of my lawn too, another thing.
Ganesh The Awesome (20:29)
Perfect. My daffodils are supposed to be coming up there. Stop playing board games, et cetera. So I'll try and pull us slightly back in. So horses bolted. These poor people, they need to do threat modeling. What about the people who actively knew that this was a snake oil operation and…
We can and we can't leave this in. It's totally up to you because I know you basically some interesting stuff came out because you were talking about it and saying, Hey, this is this whole thing is snake oil. And people came for you and said, you better stop saying that snake oil. then it turned out that it was snake oil and you were right. And you did a very long post on LinkedIn. was basically like, I was right. And you were wrong. Give us a breakdown of that. And what sort of, well, if you're allowed to, what sort of nonsense came your way because of that and from what kinds of
Disesdi Shoshana Cox (21:25)
Yeah,
so I have to be very careful what I say here. ⁓ a lot of money was being made on this and I will go ahead and I was dumb and did not realize what they were doing because I thought, okay, everyone knows this, right? I'm gonna publish this first and let's get on it, guys. And ⁓ I just assume you guys are on it, right? I didn't think about it further because I was you know working on the EU AI Act and a bunch of other stuff. So
Disesdi Shoshana Cox (21:26)
So,
Didn't confess that
Paper in defense twenty twenty two
Okay, AI red teamers got
In
the course of
Disesdi Shoshana Cox (21:53)
doing
that, it came out and came to my attention because I'm on the r the sets red teaming requirements, ⁓ what the state of the art was and what these people were doing and my head absolutely caught fire. So the first thing I did obviously was start to going to people privately because like again I'm so dumb. I was like, you guys, did you know? I thought maybe they just didn't realize this is not real, right?
Disesdi Shoshana Cox (21:57)
Because team that
Like I'm you know this
Disesdi Shoshana Cox (22:22)
And ⁓ time after time I was told that we don't care as long as people pay for it. we're trying to build a business here, doesn't matter to us. A few people trying to like I guess neg me into join Yeah, I know. I'm like they did that verbatim. I was told that by a CEO, I won't name the name, but anyway, it I can't like you said, people I I I can't believe.
Disesdi Shoshana Cox (22:37)
Looking at your face. Verbato my
Anyway.
Ganesh The Awesome (22:47)
I can't believe that people would just be in an industry to make money without caring. That is such a shock to the capitalist mentality.
Disesdi Shoshana Cox (22:57)
I was was murdered in my feelings by that point. ⁓
Disesdi Shoshana Cox (22:57)
I know. I too.
Ganesh The Awesome (23:04)
It was the shattering moment.
Disesdi Shoshana Cox (23:07)
Yeah, yeah, I grew up a lot that day.
Ganesh The Awesome (23:09)
the idealist in Wonderland moment. So basically they told you to ignore it and shut up because we're making cash money, then what?
Disesdi Shoshana Cox (23:17)
Yeah, or works some of them tried to like neg me into like telling them how it worked and some of them tried I people were mean, like really mean. And I I eventually was like, Okay, well I have to do something about this. ⁓ so I ended up writing ⁓ a couple of posts about it and in my newsletter and ⁓ that's I started getting threats. Like I I actually said something to a couple of people publicly because at this point I was getting pretty frustrated with this because I'm watching
Disesdi Shoshana Cox (23:18)
Yeah.
Hello.
Disesdi Shoshana Cox (23:46)
n I don't know even how much money has been made off of people charging enterprise for red teaming services. I know for a fact that these guys selling courses out here are like three thousand dollars ahead, basically. And
Disesdi Shoshana Cox (23:56)
Like two thousand fifteen hundred basis. And you
pack a course with like twenty people, you do this multiple times a year, you are raking cash off of these poor people who think they're learning a skill and none of it's real. And you're sending them industry to do what? To do and then I had to say not just doing nothing. You're not just charging for nothing. You're publishing these repos of prompts which make it easier for real hackers to do real.
Disesdi Shoshana Cox (24:04)
Are you in
Disesdi Shoshana Cox (24:25)
tax. So anyone, any system that you red teams, that any logo you stuck on your stupid site.
Disesdi Shoshana Cox (24:31)
on
is now vulnerable. So I feel pretty passionately about this. And ⁓ yeah, that was the point at which I got threatened. That was the point at which ⁓ an organization ⁓ reached out to me and was like, you gotta shut up about this or else. And and I won't say what the or else was. ⁓ but it it was bad and it was in writing. And ⁓ I couldn't believe it. I was absolutely shocked.
And then that's when I published, I wrote fifteen thousand words about it and dropped six or five different ⁓ volumes on my newsletter because I was like, Don't threaten me. Don't threaten me. That's how you get fifteen thousand words written about you.
Ganesh The Awesome (25:12)
It sounds like one of those situations where you also need to stay openly that you're not feeling suicidal and you've got no plans to end your life and you're quite happy. I don't know, it sounds very dark.
Disesdi Shoshana Cox (25:24)
Yeah,
yeah, I've been a little bit on the edge here, and I will I will and to to your point, I will ⁓ affirm that statement. I am not suicidal. I have no plans to disappear. I'm in Washington DC. I plan to stay here loud and proud. So let's just make that known.
Ganesh The Awesome (25:39)
hats off to you, it's very difficult to be, it's very difficult to have like a North Compass, a moral North in when that, money's involved and especially where a job is involved, you know, I won't go into like ins and outs, but I know I've definitely done things that weren't totally aligned with my North Compass due to the fact that I was employed with a certain company or my role was to, you know.
You're in a certain role. You don't rock the boat if you don't want to get sacked basically. it takes, I was going to say it takes some balls, but you're a woman so it doesn't take balls. It takes a lot of courage to do that. So for that, chuchba indeed. Takes a lot of chuchba for that. And we definitely salute you for that. Or I definitely salute you for that. it's pretty awful that you're basically, you know.
Disesdi Shoshana Cox (26:26)
Aw thing.
Ganesh The Awesome (26:32)
It's classic whistleblower mentality. The whistleblower comes out and they go after the whistleblower basically. And I can't imagine how much money is… I mean, you talk about the training courses, which is, okay, that's like definitely snake oil, but these VC, the guys making startup applications and software, the amount of money they're taking from VCs and they regularly announce, yay, we did 25 million funding round B, 50 million funding round C.
and those tools are doing exactly what you said. very, very murky waters. ⁓
Disesdi Shoshana Cox (27:10)
be real with you this is asterisk but I
Disesdi Shoshana Cox (27:11)
not financial advice ask like
my number one w one of my largest demographics of readers is investors ⁓ on my newsletter ⁓ and because i've had so many people that have started coming to me more and more and being like hey is this real shoche and i'm like that is vaporware my friends if you want to invest in it hoping it gets acquired by someone stupider that's your business not mine but that is vaporware so
There's a lot of it going around and you hate to see it.
Ganesh The Awesome (27:41)
Definitely interested, should I buy silver?
Disesdi Shoshana Cox (27:48)
Gold. Everyone on gold on YouTube told me so. Gold and the green drink.
Ganesh The Awesome (27:52)
You can't go wrong with
Disesdi Shoshana Cox (27:55)
gotta
get some of the green drink.
Ganesh The Awesome (27:56)
What, AG1? I dunno. What's the green drink?
Disesdi Shoshana Cox (27:58)
Is that what it is?
I don't know.
Ganesh The Awesome (28:04)
Yeah, it's AG1, Athletic Greens, AG1. Pulling us back and we go way back to actually talking about building the boundaries of these things. So, I'm sorry, I forget the name of your colleague you mentioned who's building the ways to find the boundaries in a mathematical way.
Disesdi Shoshana Cox (28:08)
My bad.
Disesdi Shoshana Cox (28:23)
That
was me.
Disesdi Shoshana Cox (28:24)
with my colleague Nicholas Funzel and he provided the ⁓ technique called centered kernel alignment and it's a means of measuring the similarity of two neural networks. ⁓ we found that there's potentially other applications. ⁓ Nicholas's work demonstrated that you can use it to measure these things and it does actually in fact provide a good measure of similarity when you're atta evaluating attack transferability. So this
Disesdi Shoshana Cox (28:29)
The so basically it's this
The mean measure.
Disesdi Shoshana Cox (28:51)
kind of a a a major breakthrough. And then ⁓ in our paper together we were able to apply that to say, okay, here's a method you can use to find the actual boundaries of these subspaces, like the boundaries of this box. So ⁓ anyway, sorry, that's a little bit more of the the background on how that works.
Ganesh The Awesome (29:09)
No, that's very cool. So now that we've got the boundaries, like what does that mean in real world terms? Does it mean that you can see something, even if you don't know what it is, you can analyze it and say it falls within the boundary. If you know the boundary space, can you know if something's inside the boundary? Does that actually help to solve the problem in the future?
Disesdi Shoshana Cox (29:30)
Well
this is not something I've really talked about.
Disesdi Shoshana Cox (29:33)
much publicly before but one of the big methodologies for attacking AI in the wild is to construct a surrogate model first. ⁓ if if I were to attack AI, that would be my first stop. I would surrogate model of anything that I wanted to attack, I would craft attacks against that model. Those attacks most likely transfer. So what we're talking about here is attack transferability. And while don't need your model
Disesdi Shoshana Cox (29:48)
Make a search anything.
Well, ⁓
Attackers in the water.
Disesdi Shoshana Cox (30:04)
Need your data,
they don't need any of that. They need to understand the domain that your model is trained on in order to construct these attacks. So you can see why every model is very vulnerable. Begin to map out the boundaries of these subspaces. Right now they're very, very large. ⁓ but begin to understand how they overlap, we could begin.
Disesdi Shoshana Cox (30:11)
See
If we can
Again.
I think we could begin to un begin
to un
Disesdi Shoshana Cox (30:29)
understand
and have some idea of attack transferability, right? If I know that the more similar a model is to my model, the more likely attacks are to transfer, I can start testing different models and their attacks, right? So I can test one that is very dissimilar to my model and see if the attacks will transfer. If they do, I know, my God, okay, that's a boundary. A model that is
Disesdi Shoshana Cox (30:33)
So I know.
Then
Very
Very
Disesdi Shoshana Cox (30:55)
similar to
mine will still have attacks transfer, move the boundary out when we're start when we're crafting these attacks. ⁓ and in that way we can start to maybe map this a little bit. Now keep in mind it's high dimensional space. So it's a it's a challenging problem. But the intention here is to be in to understand what attacks will transfer and what are the implications of that for my system.
Ganesh The Awesome (31:17)
I can't even imagine how that will apply in the real world. I'm too thick for that information to go in, beautifully presented. We definitely hit the bar of where I was able to follow you in the conversation. What's the, if you had to throw a magical number at the problem or…
Well, first of all, is it solvable ever, basically? Because initially, speaking to you and getting the feedback, was like, is a mathematical flaw in the systems. It's just there. It's not solvable. And then you talk about surrogate models in high dimensional space and looking at attack models in different things. And then to the layman, which I definitely am, you start thinking, this sounds like she's onto something. And actually, maybe it is solvable after all. But the…
What is there a time frame on it if something magical like this can actually solve the problem?
Disesdi Shoshana Cox (32:18)
Yeah, it is solvable. That's why I'm not I mean, I'm not actually an AI whatever pessimist or
Disesdi Shoshana Cox (32:23)
or anything,
you just have to do it right. Like I just want this industry to
Disesdi Shoshana Cox (32:27)
To
adopt real engineering standards instead of treating everything like it's silly pretend sci-fi. Like we're making real systems, let's act like adults, let's make them real. I don't think it's wrong at all. I think the problem is solvable, but it's not using the old toolkit, right? And it's not solvable if we pretend like it doesn't exist. Combine ⁓ these threat modeling for our system designed for tests, right? So
Disesdi Shoshana Cox (32:30)
I'm treating
So yeah, I agree with you. I don't think you're wrong at all. I think the problem
It's not
So when we compare these ideas of threat
We designed c
Disesdi Shoshana Cox (32:53)
systems
for testability. We threat models so we know what to test. And then our testing is focused and efficient. We can also test to understand what attacks will train how easy this is going to be for hackers. And then into our model monitoring. So whether we're monitoring monitoring a model itself ⁓ or a an agentic system that's deployed
Disesdi Shoshana Cox (33:03)
Transfer to see how it we put all of these things into security. And this goes
Disesdi Shoshana Cox (33:21)
have in place a a certain set of thresholds, right? Like in which everything is normal. So I will make up a magic number. This is a meaningless number, but but for some purposes, bear with me. Say I have, you know, ⁓ a ratio of red blots, red dots to blue dots going through my system, right? And if my ratio of red dots to blue dots becomes twenty seven percent higher for the red ones, then I know
Disesdi Shoshana Cox (33:25)
The these are the ranges.
Demonstration.
And I know.
Disesdi Shoshana Cox (33:50)
because of my predetermined thresholds, uh-oh, we're under attack. Uh-oh, something's going wrong in my system. So because of that, I'm able to trigger, say, human in the loop, human review, some type of model review, some other type of data engineering system to remedy this. That's how we deploy AI securely.
Ganesh The Awesome (34:10)
Mmm. My favourite part of that was human in the loop because it's just, that solves all the problems of people losing their jobs to AI. They can just be humans in the loop.
Disesdi Shoshana Cox (34:22)
She had we would meet a lot of them.
Ganesh The Awesome (34:26)
Well, that's all right. There's going to be a lot of lawyers and consultants at work soon. mean, you think? What about McKinsey? McKinsey has just dropped off a cliff like that because all they did was stuff that AI does automatically for them now. So we can put all the McKinsey people and we can put them in AI in the loops and then they can just be AI monitors. I don't know, maybe that's just my personal feeling on people who work for McKinsey.
Disesdi Shoshana Cox (34:33)
I don't think
not an economist, so I'll yield to your expertise here.
Disesdi Shoshana Cox (34:58)
Well
Ganesh The Awesome (35:00)
Super cool, but it's a nice way to sort of come to the close of the podcast because at least it has a light at the end of the tunnel positive feel about it where it's not because I don't believe in being an AI doom and gloom either. I've never there's so many so many reasons that people think it's the apocalypse and this and that and the other. I just don't believe it whatsoever.
I think the statement, you we just need to be adults in the room. I think that's it basically. Instead of just like releasing these power toys like children, it's like having a moment of being adults and then using them correctly. So very cool. ⁓ Anything you want to get off your chest that we didn't get off your chest is part of this.
Disesdi Shoshana Cox (35:46)
Man.
Disesdi Shoshana Cox (35:47)
All types of stuff.
Disesdi Shoshana Cox (35:49)
No, but I really want ⁓ people to take message seriously. Serious engineering. It requires serious engineering. ⁓ it's time to take it seriously. It's going to become very expensive to not take it seriously very soon. And I would really rather people talk to me about that now than me have to say I told you so later because I don't actually enjoy that. I would rather see everything work.
Ganesh The Awesome (36:12)
Maybe you enjoy that I told you so a little bit. don't know but but
Disesdi Shoshana Cox (36:17)
By the time you make me
Disesdi Shoshana Cox (36:19)
We
have to say it. Yeah, I enjoy it. I don't know if you're mine.
Ganesh The Awesome (36:22)
So we always like to ask every person the DeLorean question. I'll say that again, the DeLorean question, which is if you could go back in time to when you first started in this space or just in your professional career and give yourself one piece of advice, what would it be?
Disesdi Shoshana Cox (36:41)
Mm, don't give up. Be mean or faster.
Ganesh The Awesome (36:45)
Be meaner faster. That's a new one. I like that. Thank you. And then we have five quickfire questions, which are pretty fun to ask.
Disesdi Shoshana Cox (36:59)
Okay.
Disesdi Shoshana Cox (36:59)
Is are there rules that I need to know to answer this or can I just go? Cool IV.
Ganesh The Awesome (37:04)
I mean, they're supposed to be quick fires, so it's the thing that comes to your brain, so try not to think too much about it. No, no, no, no,
Disesdi Shoshana Cox (37:10)
Thank you
Disesdi Shoshana Cox (37:10)
Got it. A candia.
I don't I don't have any of those. I don't care about tools or app.
Ganesh The Awesome (37:22)
One tool or app that you hate.
Disesdi Shoshana Cox (37:25)
all of them. I hate all of them. I think that they were all a mistake. Except Grep. Grep is good.
Ganesh The Awesome (37:30)
Grep is good. You know, I once met a guy who was heavily involved in helping build Linux kernels and he was such a super nerd and I was like, I was quite young at the time, I was like, what's your favorite Linux command? What's the best thing about computers? And his answer was Grep, actually.
Disesdi Shoshana Cox (37:52)
Yeah.
Disesdi Shoshana Cox (37:52)
Yes, that's a sensible person. Thank you.
Ganesh The Awesome (37:55)
Yeah,
well, someone who programs Linux kernels tends to be pretty sensible. They're not, you know, they're not too rock and roll. ⁓ What keeps you up at night professionally or unprofessionally?
Disesdi Shoshana Cox (38:09)
Definitely thinking about
Disesdi Shoshana Cox (38:10)
influence
operations and how AI can be used adversarily, not just adversarial attacks against AI. ⁓ from a national security perspective, that is an evergreen problem and I need everyone to be more serious. That's long, but
Ganesh The Awesome (38:25)
Got it.
Disesdi Shoshana Cox (38:26)
Sorry that
Ganesh The Awesome (38:27)
That's, you could be as long, you can be as long as you like. Can you share the worst ever experience of your career?
Disesdi Shoshana Cox (38:34)
⁓ probably getting threatened for talking about this. I don't know, man. It's been rough out there. That's why I would I s I say to girls that are coming into this space, be meaner, faster. I got your back, cause like it's rough out here. I could I there's I got stories I that range from everything from people stealing my work to sexual harassment to on and on and on and on and on and you just you take it.
Disesdi Shoshana Cox (39:00)
T.
Disesdi Shoshana Cox (39:01)
You update your priors, you move along, you keep it rolling. the yeah, it's hard point.
Disesdi Shoshana Cox (39:07)
Card to say at this point.
Ganesh The Awesome (39:08)
No, definitely this is a side pitch, but we've also done a couple of podcasts with some like influential CISOs and stuff in the UK. One of the topics we came back to quite a few times was women in tech, the lack of women in tech, the problems that arise with women trying to come into tech. I didn't want to touch on it too much to assume that there were problems. I don't know. you know, the problem is middle-aged white men and I'm a middle-aged white man. So it's some somehow.
Whatever. I'm it basically, but I'm not it also.
Disesdi Shoshana Cox (39:43)
I don't know. I don't want to blame men for this problem. I don't actually think that's the case. ⁓ I I do think I think is systemic, which means that it doesn't necessarily need conscious maintainers. Like I don't think there's like a cabal of dudes sitting around like planning, well, we've we're going to make sure that Shoshana has a hard time. I don't think that happens, but there are micro decisions and there are systems that are set in place. And yeah, you come to like I'm not I I I would not a
Disesdi Shoshana Cox (39:49)
Think of the problem
Big vlog.
Disesdi Shoshana Cox (40:12)
cry about things that happened to me kind of person, but any woman that's been in this space for more than five minutes is like, Yeah, the isn't was real. It turned out to be real.
Disesdi Shoshana Cox (40:18)
Thanks.
Ganesh The Awesome (40:21)
I mean, it's just the only feedback that every woman agrees with, basically. And loads of good women leave the space because of it. Like, smart. I've worked with very smart women. They said they couldn't work in an XYZ department anymore. And then they left and it's kind of, the churn is unbelievable. And particularly if you're high profile, I think I would consider you, I don't know, quite high profile. But, you know,
With that also comes more pushback. you, did you, was it, was it, it I have seen your comments, in particularly, so we're going right back to when you talking about raising these problems initially and you were like a bit naive about the fact that they knew and what difference do you think it would have made if you were a dude bringing up those problems? How different do think your response would have been or?
or the feedback or anything, does that play into it? I feel like it must have done.
Disesdi Shoshana Cox (41:24)
In my opinion, a first time I brought up, I would have been immediately hired and handed a fat salary, in my opinion. Yeah, that's how I feel. And I'm gonna tell you this. I've been ⁓ I've had every role in the data stack up to chief data officer as a red team lead. I've interviewed men, I've been on team men interviewing men, and I've seen the different ways that men get interviewed versus how I get interviewed, and I'm telling you what, man, it's different. And again, I'm not gonna sit here and complain because me better. I'm better at
Disesdi Shoshana Cox (41:26)
The first
Disesdi Shoshana Cox (41:52)
everything because it was all harder for me. But like so yeah, I have to think that if I had been a dude and I had come and, you know, talked to these people about it, it would have been received very differently. But that's just my opinion.
Disesdi Shoshana Cox (41:55)
It's different. So I
Ganesh The Awesome (42:06)
No,
I mean, it totally resonates. We had, a we we talked about it a good number of times on the podcast, to be honest, including one girl who's the head of DevOps who someone started like cracking onto her in an interview, like chatting her up in the interview. And I was just like, it's so fucking ridiculous. It's unbelievable. But
Disesdi Shoshana Cox (42:28)
Dude,
I had a f there's a very well known gentleman in the space. I won't say more because people know who he is, that ⁓ asked me out to ⁓ a resume review, ⁓ bought me dinner and this I was struggling at this time. Like I this was this was a while back.
me and real that I was Jewish. He went off on like a giant anti Semitic grant. So like what a night for me. I tell you what, no review happened and I got to experience all that great times. Great times.
Disesdi Shoshana Cox (43:02)
Okay, right.
Exactly.
Ganesh The Awesome (43:07)
Great, out of pure interest, was the guy so autistic that he couldn't pick up on body language that you were being repulsed? mean, having, being anti-semitic, like the whole thing, you know?
Disesdi Shoshana Cox (43:20)
I'm gonna be real with you, I don't think they can think they can.
Disesdi Shoshana Cox (43:22)
care. I don't think
care because I'm not buying I'm autistic or I can tell when someone doesn't want to talk about something. Like, okay, you I don't think they care. And again, I don't blame men. Like I'm not like, ooh, men or the the devil here or anything, but I do them that rewards certain kinds of behaviors and punishes others. And that's it was based on my observation.
Disesdi Shoshana Cox (43:25)
Second.
But again we're getting into this is my opinion.
Sorry.
Disesdi Shoshana Cox (43:49)
and I think that it's hurting the tech industry because it it cuts out a lot of really talented and ⁓ viewpoints, not diverse for diversity's sake, but diverse technologically.
Disesdi Shoshana Cox (43:56)
Diverse view.
Ganesh The Awesome (44:00)
We
do and it's totally true and weirdly, not weirdly, factually, there are places that don't have this problem. One of them is Israel, where I work with a lot of Israeli tech companies and it's about 50 % women in the, maybe not 50%, but a very much higher percentage than is in Europe generally. ⁓
I'm sure it's because of the military, because the military just trains people equally and then churns them out and then everybody arrives with a similar sort of mindset and that's how it goes. But once you get into the professional workplace, it's different there. It is different. It's just different, basically. So it's not a matter that women can't live in the tech world because Israel is a perfect example of where there are loads of women in very high tech positions who are super smart and it's not a problem or they don't need to leave and there isn't this high churn.
But elsewhere, you have to be a badass like you that takes the challenge and then, you know, sticks the finger up and has to like trawl through the mud to become a lepous flower and pop out the other side.
Disesdi Shoshana Cox (45:08)
But I have to I'm sorry.
Disesdi Shoshana Cox (45:09)
say that I sorry to interrupt,
but it's very important to me to say this. I would not be where I am if men had not stood up for me and helped me and and whatever else. So there have been a ton of good dudes that I'm I'm not said don't make the perfect the enemy of the good, you know what I mean? Who have or like boosted my thing when I needed it and so forth.
Disesdi Shoshana Cox (45:26)
Like, hey, let her talk.
So the reason I bring that up is just be one of those dudes. Be one of those dudes.
Disesdi Shoshana Cox (45:36)
For a lady and that's all you gotta do.
Ganesh The Awesome (45:39)
Yeah, very sound advice. I don't know how many deaf ears that falls on. We have got a couple of minutes left. I'll go back to the final speed question. If you weren't working in tech, what career would you choose?
Disesdi Shoshana Cox (45:50)
sorry.
Disesdi Shoshana Cox (45:54)
Whoa, policy, I'd be in policy, but if I wasn't gonna do any of that, I would be in music. ⁓ so I guess I am kinda policy. So yeah, I would love to be like performing and music stuff on stage. Super fun.
Disesdi Shoshana Cox (45:55)
Probably
Ganesh The Awesome (46:07)
very cool. That's it, we've come to the close. Totally awesome chatting to you. It was one of my favorite episodes. I really thank you for giving us the time. Totally loved it. Any parting words?
Disesdi Shoshana Cox (46:20)
had a wonderful time. Thank you so much for having me. This was
Disesdi Shoshana Cox (46:23)
a minute in the making and I'm glad you pinged me again and ⁓ had the patience and forbearance to keep trying.
Disesdi Shoshana Cox (46:29)
Thank you.
Ganesh The Awesome (46:31)
That's it for today. Thank you all for listening. If you're ready to rethink your cloud practices and cybersecurity strategy, then the team and I at GlobalDots are at your disposal. We've been doing it for over 20 years. It's what we do. And if I don't say so myself, we do it pretty well. So have a word with the experts. Don't be shy. And remember that conversations are always for free. I'm Ganesh Leeawasam, and this episode was produced and edited by Tomer Morvidzon. Sound editing and mixed by Bren Russell.
Stay tuned for more episodes.
