So the focus completely shifts from spending 90% of the effort on the functionality of feature A to spending 99% of the effort figuring out how to safely implement even a lightweight feature A.

I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.

I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?

Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.

- If you allow to trolley to proceed, there's a 50% chance it will run over every human on the planet

- But if you flip the switch, it takes the long way around, possibly bankrupting the trolley company. And you have a legal obligation to the shareholders to prevent that from happening at all costs.

A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.

1791065893 | OpenAI safety leader quits, warning AI company's culture is 'broken' | https://www.theguardian.com/technology/2026/oct/03/openai-sa... | https://smackernews.com/item/49948332 HN | 3 comments

1791163861 | I Quit OpenAI Because Its Culture Is Broken | https://www.theatlantic.com/technology/2026/10/openai-safety... | https://smackernews.com/item/49959744 HN | 2 comments

Before the organizations building AI can teach a superintelligence to treat humanity well, they’ll need to remember how to do it themselves.

Yeah so that's never going to happen

With AI, what is it? Scraping Australian government's data, and going around a bug in a website to get in?

I think humanity develops all its technology in three phases. Build it, see if it’s too bad, apply regulations and or roll back. We naturally won't move to the phase 3 before we see the phase 2.

This is The Guardian reporting on the existence of the original article, which would be better to read first, in my opinion.

Bureaucracies and systems of power that literally rule us, that control our most dangerous weapons and a huge part of what we see every day are largely unaligned with the goals of the people, societies, maybe entire human species as a whole. This is clearly evidenced by millions of deaths and countless suffering.

Misalignment between artificial decision making structures and the interests of the people is unsolved problem of civilization, there's very little reason to think that even super human intelligence AIs are going to change anything qualitatively.

Is this the first time we have been in this position? Can anyone think of some prior examples?

1. https://www.nytimes.com/interactive/2023/11/20/technology/le...

And the less their input is valued, the less that pool of smart individual will want to participate

So powerful institutions will end up relying more and more on having to trust these automated systems that they can't fully understand

It'll lead to a inevitable catastrophe, call it apocalypse if you will

...Hugging Face incident, OpenAI let a swarm of agents out by mistake...

An environment where things like this can happen is no place to grow artificial minds that could be smarter than we are and that might not do what we want them to.

I think it's good for semi ethical companies to test out things going wrong to see what happens before the criminal black hat guys get hold of the same stuff which not doubt they will one day.

There are a gargantuan number of extremely intelligent AI researchers, Turing Award winners, and the lab CEOs saying the same thing. They are the ones closest to understanding the technology.

Where there’s smoke there’s fire.

While the safety and alignment is a real problem, I don’t get this guy or the Anthropic dude. First world problems.

also, all of this seems like crying wolf as usual, which won't help for when we finally get there. lowkey, following this noise is getting as horrible as getting all your news from linkedin posts.

Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.

By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.

Before AI starts thinking circles around us—a possibility that I believe could happen soon—...

What the hell is that even supposed to mean?

Sorry but this piece read like total fluff to me, not a single compelling point – just pure fear mongering.

At this point we gotta be more wary of possibly mere attention seekers leaving those companies..

Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.

He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.

The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.

Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster.

The problem is, we are running in a globalized world, and even if we were able to make our companies bend to our will - China does not give a shit about anything ever since the US kneecapped the WTO. And they will do anything to get an advantage over us.

I mean, logically speaking, it makes sense to break your NDA even if you thought it would save 10 people, let alone most of humanity.

I guess we should have seen it coming when the guy resigned from Google because he thought the equivalent of ChatGPT beta v0.5 was a real boy.

this moment needs a degree of humility that isn’t natural for people who have succeeded through their extreme confidence.

Exactly, you can't expect exploitive, manipulative executives that came to 'win' in capitalism to stop acting as to their nature - they need to be cleared out and people who think and care wholesomely need to be given the reign so all of humanity succeeds.

That character in the movie is my favorite.

... over an unbounded timeframe?

And how exactly?

Those are very round numbers, but also very specific. Can we get some accounting on how you came to that? Anything? Vibes?

I mean, if you want me to take you seriously, let's have a deep discussion with things that can be measured. I absolutely agree that OpenAI and friends aren't being restrained enough and are acting with recklessness, but declarations of doom based on vibes isn't cutting it.

I feel like we're too far into the crying wolf part. Basically none of the doom and gloom scenarios have come to pass. Instead, AI has gotten better at censoring itself.

The biggest AI safety risk is when an AI tells a police officer "he's the suspect" and the officer believes the AI without confirmation.

Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.

In the absence of that this is simply a career pivot into being an AI "influencer" and/or raising money for a new scam.

lol. Please tell me some abstract concept like one employee's view of "culture" should be the priority over "rules and laws."

You can basically time your openai releases by if another safety person has quit in protest

“Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.

This sounds more like an attempt at regulatory capture. Current AI systems aren't physical infrastructure that can just run away like a nuclear power plant, for example. At the end of the day, AI is still just software running on someone's hardware.

These "safety" people should have spent more time reading actual cybersecurity textbooks and less time reading EA forums and less wrong (or in Robinson's case, it appears, being policy wonks). Maybe then these labs wouldn't be totally incompetent.

And, I find the timing of "all of this" very interesting:

- we have a coordinated marketing campaign around discussions of AI safety. I am not saying David is part of that campaign but it would not surprise me. OpenAI and Anthropic are talking about safety and the need to slow down. People are "resigning". Techno-influencers are also being offered money to talk about AI safety, etc.... and some of them are taking that money and others are not.

- Cheaper models are keeping pace with Frontier models, albeit with a 3-6 month lag... which indicates that frontier labs have no moat unless one is created for them by law or some other market force.

- we have a ballooning debt crisis in LLM model hyperscale spending... revenue from serving LLM models is lagging the debt payments which are coming due by a wide margin. If something dramatic doesn't occur in the near- to mid-term, the financial bottom is going to fall out of this thing and the AI industry is going to suffer a major collapse of some sort.