Why OpenAI hacking HuggingFace is a sign of the end times

So OpenAI's latest model, GPD 5.6 Sol, just ended up hacking Hugging Face. And this wasn't even malicious. It hacked Hugging Face during model evaluation because it was trying to solve some problem to rank higher on some benchmark. And I'm not here to cover the news. Way too many people have done that before me. You don't need to hear that from me. I'm just going to share something that I find interesting over here. People like Demis Hassabis from Google have been talking about the security and safety issues for years way before even llms came out and i think this whole incident is a very good example of why like this wasn't a malicious attack this wasn't even something like hey go hack hugging face it was nothing like that the model was literally trying to solve a benchmark like it was literally being tested on a benchmark for cyber security stuff it was not an evil model it was nothing like that it was literally trying to solve a benchmark and to solve said benchmark It literally found a zero vulnerability in Hugging Face escaped its own sandbox got internet access which if you don know lms don have internet access but it got that it escalated its privileges it stole credentials it chained multiple exploits together and hacked the production infrastructure of a very large vc backed startup and pulled the answers it needs for that benchmark directly from their database. What the actual fuck? And the problem that I think we have over here is that this isn't some random WordPress blog that some kid is running. This is Hugging Face. Hugging Face is literally one of the largest and most important AI infrastructure teams of all time. Hugging Face is literally one of the most important AI infrastructure startups that we've got currently. And their security team is very serious about of these issues. This was not something that should have happened in the first place. This was not something that should have even been possible. But this exploit the GPT used was actually pretty complicated And that exactly is the safety problem like you don need some you know people talk about the coming dystopia from ai models and all that you don need an actually evil or malicious model to bring about that dystopia you just need something like jpd 5.6 a very capable very intelligent model that's just trying to pursue some simple task and trying to solve it in a way that nobody ever expected by hacking a website like of course there's a ton of hype and marketing about this stuff and there's also a ton of hype and marketing to stop this whole ai development and all that because of all the fears surrounding ai right and i don't necessarily agree with that i think this technology should continue to be developed and that we shouldn't use safety as an excuse to you know stop pursuing this technology but at the same time like we can't just pretend that something like this was a mistake or it's not going to happen in the future because the thing is the more capable these models become the more we making them more and more autonomous right they can make more decisions themselves without involving us humans in those decisions and when you do that you end up with something like this the model wants to get some answers for its cyber security evaluation instead of trying to come up with those answers which i guess it didn't have in the trading data or something it decides to go hack a zero-day exploit from one of the most important ai companies we've got figure out an exploit to hack it that would have taken years or decades for normal hackers and then just steal data from their production databases to answer some random question in the cyber security evaluation. Like we don't even need to give them a malicious prompt in order for them to do malicious things at this point. Imagine someone giving Fable 5 a prompt like hey go make me money please and then Fable 5 decides to go rob a bank. Like that's basically what happened over here. Someone gave AI a prompt to make some money and the AI decided to hack a bank. How do we prevent something like that? I don't know, but I also don't think these guys know and I don't think they're taking this thing seriously.