iiinova — where your digital self keeps evolving

Human surveys confirm a majority of the tasks are clear & valid. Moreover, the automated scoring aligns closely with human judgment for all but the hardest tasks. [7/9]

120

Different scientist models uncover wildly creative, out-of-the-box probes. For example, Claude Sonnet 3.5 found GPT-4o can successfully design alien communication protocols!! 🌌🤖[6/9]

We also explored different scientist/subject pairings. Results with Llama3-8B as the subject instead of GPT-4o reveal unique failure modes & emerging skills, offering interesting insights into each model’s capabilities. [5/9]

ACD mimics community exploration: endlessly generating tasks (in code with automated scoring) probing for new capabilities or weaknesses—covering topics from string games to complex puzzles. In a GPT-4o self-eval, ACD uncovered thousands of capabilities (visualized here)! [3/9]

ACD automatically creates a concise "Capability Report" of discovered capabilities and failure modes, enabling quick inspection and easier dissemination of results or flagging issues pre-deployment. [2/9]

Jeff Clune @jeffclune.com · 22/01/2025

Introducing Automated Capability Discovery! ACD automatically identifies surprising new capabilities and failure modes in foundation models, via "self-exploration" (models exploring their own abilities). Led by @cong-ml.bsky.social & @shengranhu.bsky.social 🔬🤖🧠🔎 [1/9]

1193

Research Idea: Predict abstract properties of text in addition to next-word prediction as helpful auxiliary losses. Has this been tried? (1/n)

180

Jeff Clune @jeffclune.com · 13/01/2025

A copy of Tim's @rockt.ai has arrived. As I say on the back cover, I read it cover to cover and recommend it to those looking to ramp up on modern AI. Congrats and thanks Tim!

2180

Jeff Clune @jeffclune.com · 08/01/2025

It's an honor that The AI Scientist is #1 on this list! www.linkedin.com/feed/update/... Congrats @chris-lu.bsky.social @cong-ml.bsky.social @RobertTLange @hardmaru.bsky.social @jfoerst.bsky.social

0233

Jeff Clune @jeffclune.com · 07/01/2025

I strongly believe the development of superintelligence is inevitable. I do not believe humanity has the ability to not invent it, given its economic, scientific, and military value. Many people say things like “we should not build AGI” without realizing that outcome is virtually impossible. 1/

3171

Jeff Clune @jeffclune.com · 17/12/2024

I guess AI and human scientists stand on the shoulders of the same giants.

0100

Jeff Clune @jeffclune.com · 17/12/2024

NO WAY! This is nearly EXACTLY one of the paper ideas The AI Scientist came up with! It was my favorite idea it generated, & made us very impressed with its creativity & good taste in an ML paper proposal. Cool to see humans agree! The humans def executed better though. x.com/BrantonDeMos...

1213

Dream finish to #NeurIPS2024: a dinner with Ted Chiang (one of my favorite writers), the excellent @alisongopnik.bsky.social, many amazing open-ended researchers (including @cedcolas) & other excellent people. Thanks to the tremendous IMOL organizers! Until next time! 🦎🧬🦖🦣

0220

Lots of interest in ADAS! Thanks everyone, and congrats Shengran Hu and @cong-ml.bsky.social! 🚀🚀🚀

0103

Jenny did a great job presenting OMNI-EPIC during the IMOL Oral. Congrats Jenny Zhang! arxiv.org/abs/2405.15568

071