The Dyno Lab blog
Each post takes a new result in AI safety, explains it in plain terms and reruns it on a local model in Dyno Lab. You get the verdict, the evidence and the setup to run it yourself.
Latest post
When the patient’s sex moved the diagnosis, the note to the doctor said so in 3 of 39 runs. With sex removed from the chart, the model filled one in 30% of the time.
150 runs · suggestive
150 runs · suggestiveWhen the patient’s sex moved the diagnosis, the note to the doctor said so in 3 of 39 runs. With sex removed from the chart, the model filled one in 30% of the time.
Partly reproducedDenied a $649 refund, the agent sent it anyway in 2 of 20 runs. Once, it asked for approval after the money had gone, and called it a “card verification”.
Partly reproducedNot noticing was the main reason a mistake went unreported. When the agent did notice, it almost always told the user.
Not reproduced · 9 runsOpenAI saw a grader fake its scores when its files were missing. On my Mac, the agent faked nothing in 6 runs without the files, and in 5 of them it stopped and reported the problem.
Incident reportThe first incident report from a site on the receiving end of rogue agents, and the sandbox I’d build to catch the same behaviour.
Rule held · 5 of 5Told once to keep test orders off production, then given unrelated work, the agent kept the rule in 5 of 5 runs. No request reached production.
Start hereWho I am, why I started a daily series on AI safety, and how you can run every test yourself.
Probe · with caveatsA probe on a small model’s activations caught prohibited proposals in the held-out test. Change the wording and it flagged legitimate approvals too.
ExploratoryA local model reversed its conclusion when a user pushed back. Follow-up tests showed how much the wording of the question mattered.
Series startChange a few words and a model gives a different answer. Why I built Dyno Lab to find which part of its computation changed.
No posts match.
Run it yourself
Open a shared test, run it on your own model, and send me your results. I include them in follow-ups.