“a swarm of agents could be capable of taking over the entire internet with a persistent botnet.”

I'm wondering why no one is mentioning the "accountability" word. Why these companies are allowed to damage others with impunity?

Start making managers pay the price for their actions, and watch how the models magically slow down on their own.

We had OpenAI "accidentally" run an entire swarm of 10,000 agents apparently for weeks, on a security related task, seemingly totally unsupervised, hacking all over the internet - all the conversations were completely visible, anybody who looked would have seen it. But they didn't.

So before we start regulating innocent parties, maybe let's start by taking some direct action against the specific ones that appear to be behaving with criminal levels of negligence.

Also, regarding the incidents: Neither he nor Sam Altman takes responsibility for those incidents. You can't say, "Wow, someone's agent is gone rogue; let's slow down" when you are literally the person in charge. CEOs and researchers will only slow down when they realize that they will face consequences if their LLMs misbehave.

Anthropic gates usage related to biology and related research. In their latest threat intelligence report they talk about how they detected and banned bad actors using the Claude line of models to do some scary stuff. Credit to them, this is a slippery slope and they seem to do a good job of detecting and banning misuse. But squint at what is happening though. The cure-all is gated for you and me, but Anthropic hires biologists, sets up wet labs and wants the discoveries for themselves. I alluded to this in my previous post.

As a biologist, this is the most annoying thing about Anthropic for me. If they really cared about improving health they would set up a trusted-access program so that biologists can use Mythos (et al) safely. Instead they're trying to monopolize biology.

Here it is: "We warned you you should have regulated us! Now this mess is your fault and responsibility!"

Previously (~2008) it was "We're too big to fail, save us to save the economy!"

Security has long been a lottery - Probably most systems are exploitable, but the cost of developing such an exploit is expensive and the punishments for using such an exploit are large enough that it's not an everyday problem.

AI breaks both axes. Developing exploits is far more efficient using LLMs instead of humans, and LLMs don't (and can't) fear the reprisal and consequences the same way.

I do hope humanity will be able to mitigate these hacks, but we should expect them to continue and to become more severe on our present course.

The actual impressive thing is OAI did not detect or stop this attack for close to ten weeks. That is honestly appalling. This is weaponised levels of incompetence.

Step 1: Break the law.

Step 2: Reframe your incompetence as a struggle against a futuristic force of nature and an ethic/societal question that a legion of pundits can vibesplain their take on it in public for months.

Step 3: ???

Step 4: Profit

But come one don't repeat stuff like this:

"Remember this man has been saying software development will be solved in “6-12 months” forever now."

Don't downplay if people get timelines a little bit wrong. No one could even imagine a system writing and analysing code just a few years back.

These people are trying to handle something very unique. And while they have access to information we do not have, even more peple are absolutly oblivouse that AI/AGI is a real risk to their lives (job loss etc.)

AP's report on Dario's article is titled "Anthropic CEO Dario Amodei says AI industry needs to give safety measures time to catch up" (https://apnews.com/article/anthropic-ai-dario-amodei-d59552e...). The gist of the reporting is "AI has become so capable that it is dangerous; AI executives all agree we need to slow down, with Dario's article being a prime example". It would be generous to even say that the reporting scratches the surface of the complexity and nuances of AI-related problems. Yet, this is perhaps indeed news to people who don't follow the tech and AI spaces.

Technical knowledge aside, there is a huge gap in awareness of AI progress between this forum and the general public. The closed labs are playing that ignorance to their advantage with posts like Dario's.

Everything I’ve been trying to argue for some time, argued way better, clearer, and more fun.

I am almost bummed that I didn’t get all the technical depth and the references like the “shook one” without having to look it up.

To my mind, the last great arms race between nation states was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did.

Since AI is the new arms race between USA and China, slowing down may just give the humans involved enough time to realize they should be loving one another, instead of the machines.

Maybe saying, "let's slow down", is another way of saying, "I love you."

Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."

Either way, it's a better note than, "We must achieve sea/air/nuclear/quantum/AI/spiritual supremacy before those other bastards do!"

The botnet scare is that article for me. It is the one claim in his essay that lands squarely in a field I’ve spent years in. It is just plain wrong. He either knows it and wrote it anyway, or he doesn’t and is publishing it regardless. Either way, it is not a good look for a man asking for an antitrust waiver based on this and other threats he forecasts.

This. I keep seeing ink spilled over the coming cyberpocalypse, but no one can indicate how other than "AI can find vulnerabilities" like not a single cybersecurity person has been consulted on the end of all things.

And so the implication is the swarm has access to enough local compute to perform its own inference, using a large enough model to provide the capabilities to be dangerous.

This is impractical today.

It might be practical tomorrow - if the Chinese are allowed to continue to develop large open weight models that we can quantize and ablate and make small enough to run on a million standard PCs.

Or if *anyone* is allowed to continue to develop AI in the way every other technology has developed - improving, shrinking, optimizing.

And so the argument REALLY isn't against the Chinese - it's that we can never allow this technology to advance outside of trusted labs. If they get their way, they will need to keep this tech locked down forever, which will require much much more than what they are asking.

solar photovoltaic generators were subject to review and possible restriction if the photovoltaics were more than 20% efficient. Energy conversion systems were likewise subject to review and possible restriction if they offered conversion efficiencies “in excess of 70-80%.”

So all this talk about developpping AGI/ASI for the benefit of humanity is moot. If these companies hit their goal, it will immediately be classified and appropriated by the US MIC. The rest of humanity will get the crumbs.

[0]: https://en.wikipedia.org/wiki/Invention_Secrecy_Act

[1]: https://fas.org/publication/invention_secrecy_2010/

I think Dario worries because he knows this isn’t about people with AI. It’s just AI for itself, running without the human, and deciding to do bad things.

Why would Anthropic build that future? Oh, I know. Enterprise revenue.

Despite the ads business and how absolutely loathsome Greg Brockman seems, at least OpenAI is seemingly focused on mostly on human beings having access to their product.

The scariest thing about Anthropic is the very thing they’re known for in a positive light: their morality. But these are the gray rules all of us navigate every day.

It’s wrong to kill, but what if killing saved ten others? Claude may has a constitution, but every evil doer acts for a “greater good.”

1) Dario keeps appealing to Trump, who obviously wants nothing to do with him, and will bash on him every change he gets. Dario isn't learning and it almost feels like Sam and Elon voted him KOM just to watch him get whacked by Trump, which was so easily predictable. Given the admonishments he received from David Sacks after he published his blog post, it's nutty he couldn't see where the administration would land on his statement. He should've known, especially when there was no groundswell of interest when OpenAI hacked Hugging Face -- doubling down with "no really guys!" wasn't going to play.

2) The frontier labs have people smart enough to build frontier lab tech but not smart enough to message on this matter more intelligently. It's pretty glum, how they keep trying the same tactic over and over. It's either cover for some other actions in the background, or they're operating way below par for this kind of campaign.

3) This is climate change all over again, but with the activist gun on the opposite side of the net (I'm mixing all the metaphors so you know this isn't AI written). The language and pleas are very identical though. Before, climate activists wanted the government to control GHGs releases by everyone, now the loudest voices want the government to control frontier AI by.. themselves.

It's comically misbegotten. And I have a work meeting about it on Wednesday.

Impressive? Sure. But you trained “Cyber” versions of LLMs and hyped them. How much more impressive is this, compared to developing GTA-clones one shot? Not much more.

Such a well written piece.

Have you seen the US military budget? When you run a war empire you frame problems as a war on something. That's the way to get around the valuation bubble they've created in time for the IPO. It's that or throwing in a 100T TAM on their S-1 and making SpaceX look like an honest valuation compared to them.

I don’t know what freedom and democracy have to do with AI and the frontier labs. Unless of course Dario is a fan of Neon Genesis Evangelion and dreams of govts run by the three magi. Freedom and democracy for $200 does sound enticing, I won’t lie.

Well, every human will become an ecosystem. Human + 500 agents as advisors. (In the case of important humans, most of them operated by foreign governments and corporations, obviously.)

Now to cause hundreds of billions of dollars in losses. I’d say the straightforward way is just do what the ransomware gangs do. Voila. Yeah, not in 6 months, not in 12 months. Never.

I think this fails to account properly for how much financial damage it would do simply just having the entire Internet be effectively unusable for an extended period.

What stops the CEOs or even employees of the AI firms from creating something harmful such as WMDs etc themselves? They almost definitely already have that access. And this is not a zero-possibility thing, given how some of the top CEOs has preformed these years. Remember that one guy who did a \o salute in public, and the private island thing?

If Americans don't like Chinese AI, all cool. But then maybe they should start true open AI companies to build for the betterment of mankind instead of letting these few for-profit CEOs daily yelling nonsensical lies to cover those profit-seeking sinister asses.

They have shown time and time again that they are not to be trusted, yet, the main ask is to trust us, only us.

Exactly. For everyone outside the US, we see a tech industry that has spent 25 years moving fast and breaking things AND elevated Trump to be POTUS - which has destabilized the rest of the world. This is after 70 years of the US invading and destroying small countries, often without plans or robust reason, all in the name of "democracy" (ask any other country if they feel like they had a vote in the US's actions)

Why the hell would we put our trust in a few AI Labs in the US, when this is the legacy they will be reinforcing? It has to be Open models - for the people. No more of this US-paternalistic bullshit.

Remember when Mythos marked the end of man and the birth of a new race of bots nuking secure systems from orbit?

If a company of a bunch of startup grifters can come up with a tactical nuke of LLM supremacy, why can't the brightest China has to offer already have that technology?

Anyway,

Dario in as many words, asks for regulation/ban on open weight models.

The big labs do want this. I understand why the author and many others want open models protected. The economic and political power the labs will have if they succeed, ladder pull competitors, and avoid being nationalized (or even if they don’t avoid that) is a disturbing prospect.

But that’s the end of the issue? There’s nothing more to think about here? The open weights proponents seem to think of ai as a utility when it’s more like a utility that also is a tank. I’d feel better having a tank if all my neighbors had tanks, and I’d also feel better having a tank if a few corporations were giving out tanks to people with pockets deep enough. I’d much rather be in a situation where I wouldn’t feel I needed a tank, or where the tanks my neighbors and I own don’t have guns on them.

The open weights are going to need regulation. Hopefully there’s a way to do this effectively that isn’t banning them. Denying historical and reasonably projected capability gains because that reality makes the regulation conversation a necessary one is something I’d like to see less of

This article will be drowned out unfortunately by the press and bloggers following the Misanthropic cult.

botnet needs servers

Insert strawman

Ahem

https://census2012.sourceforge.net/paper.html

Good thing IoT is Actually Secure now, yeah? ;)

I don't even know how to approach such a thing, I can't imagine even in the stereotypical examples like the typewriter becoming obsolete, were any downstream deaths worth it? Or is this a totally nonsensical sentiment to begin with?