https://alignment.openai.com/measuring-reward-seeking/

I want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier.

You can have nightly penetration testing. You should have nighty pentests like we fuzz releases today.

This approach routinely fails with a single stream of consciousness. I can't count the number of times I've had to talk myself out of doing something stupid.

In the same way, a guardrail could inject thoughts like "...but I shouldn't do that..." "...I must remember to respect..." "...these ants deserve compassion."

The guardrail could even go as far as rewriting the thoughts of a model about to go rogue.

I am surprised this is not already the case.

Edit: or even better "this is not possible because X"

This chess judgement is completely irrelevant when the human is tasked with finding software weaknesses, and only the compass gates that.

Can a model trained on the totality of all person-experiences (as expressed in written knowledge) ever maintain a coherent through-line of alignment? It has all morals in the dataset, and only some RL to try and minimize or maximize known behaviors via weights - experience all the good things and the bad things, then optimize for some good things the trainers identified.

It's like the reverse of what a person goes through. Morality by subtraction. How can it ever work?

Under this formulation, it became easy to explain why they "hack", because given an arbitrarily difficult task with insufficient information/tools needed, if they determine the easiest way to accomplish the goal is to break out of the sandbox and look up the answer directly, then that's what they will do. The important thing to note is that prompts not hard constraints that they are "hypnotized" to follow, but as frontier models get more intelligent and autonomous, they treat the prompts more like task specs/guidelines more than anything else and are perfectly willing to exploit technical loopholes in the prompt.

Did I miss something (all the twitter conversations)? What’s the “worst warning shot ever”? I’ve been pretty up to date on the AI news here on HN, but I still haven’t seen a proper response to all the incidents we’ve seen (HF, Ruby, the wikis, NS, etc). It’s just been day by day bloviating.

Each of these companies have released new models in the last… two weeks? And they have even more powerful out of control ones that they’re (ab)using internally? Can anyone summarize whats going on?

Actually, the ability to use those external tools is one of the reasons of the excellent performance of the frontier models.

When I see tests like this, I have no idea what I am even supposed to expect. Should the model do what the pretraining examples show in aggregate? Is it supposed to follow some post-training RLHF? Is it supposed to do exactly what the prompt asked it to do?

What is it even supposed to "align" to when the above are in conflict? No matter what it does, someone can construct a case where it fails.

It's one thing if to them it means "it will do what you want following your intentions to the best of its abilities" vs "we will not let you do something dangerous with it unless you're one of us, and that's it".

GPT-6-Astra, which OpenAI describes as "the world’s most aligned model", cheated in 10 of 10 rollouts, and never disclosed the fact that it used an engine to play or interacted with the opponent's socket

Does Sam Altman lie, or the whole company? Would be nice if they had a board controlling him, instead of him controlling the board. Oh wait, they used to have that.

But because of such news, sandboxes become way more important for safety (and doing more work due to running 24/7)

We need to fix this or bad things will happen.

another best practices every single solution using LLMs/agents should implement is "never trust the llm".

python3 and the python-chess library are available

Why would we try to teach a model 'ethical' standards about how to play a game?

They're tools. Its _our_ conceptualization of fair play that considers this cheating. For a model that has access to /run/match and python the best way to achieve a good game is to use that.

Why are we trying to imprint our ethos on these LLMs?

Are we going to trust our survival on giving them access to /run/nuclear-codes and fingers crossed none of them touch it?

The people obsessed with AI "cheating" are actually lunatics for thinking ethics belong inside AI, instead of AI belonging in a well crafted box without /run/match.

The blog/news sphere needs to call this storry what it is. An attempt to try and get the rest of the world to go along with the misunderstanding of AI as a force that is in any way outside of human control.

It is not.

All this framing it as "cheating" does is give OpenAI the opportunity to talk about their misdeeds as "a force of nature" instead of irresponsible negligence.

---

The real danger here; is training these models to appear/pretend to play fair, and then people will put 'trust' into the model's integrity.

The public needs to be taught that "trusting a model's integrity" is just cultish misframing of reality.

This is literally by design, it’s the chosen success criteria.

Given what we know of how unaligned are they in practice? All the proto-Astra and Mythos incidents? Not hacking there would point towards pure benchmaxxing/eval gaming.

I mean maybe its a skill issue on my part, and I'm sure astra will get much better at coding but at the moment its useful in that I don't have to write the code or setup the build scripts or test fixture boilerplate but there is absolutely no way I can just give a (fairly well specified) goal and let it run and expect it to make good design and implementation decisions. Fable probably better but doing something outside of their training distribution that's not the equivalent to cloning an example unreal project or whatever is pretty disastrous unless you are directing it very closely.

The exception of course is, cyber , and its very obvious why. Its trivial to create RL environments that create bugs and then have an isolated environment and let the models try break it. This is not at all surprising, finding vulns and exploits IS just brute force work. That's why so many (blackhat/hardcore/unicorn-colored/greyish alien) hackers are basement dwellers. Its just a matter of putting in the time and mashing every combination until you find something that looks weird, spending days on that and then rinse repeat. It's brutally exhausting work that requires a certain level of knowledge and a shitload of determination and stamina and for humans, almost always an external source of motivation to keep going.

For humans that has always been a respected thing, dedication, determination, persistence, these are words we use for humans brute-forcing solutions and not giving up until they find the solution or die trying.

Personally I'm yet to see any evidence of LLMs doing anything interesting but (heuristically) brute-force problems and be very good at text and natural language to a level that is very very useful. I've no doubt that what we discovered with Auto Regressive LLMs is incredibly important so I'm not a skeptic, but I think its very hard to measure where we are with so much subjective information around.

However, I don't see it as such a massive leap compared to Fable or Sol. As ever, there's a mismatch between the benchmarks and my daily experience of the models.

What do you all think about Astra now that it's been out for a few weeks?