iiinova — where your digital self keeps evolving

7/ More of our thoughts on agent hijacking evaluations are in the post – our first US AISI technical blog post!

010

6/ We also explored, among other questions, what happens when we measure pass@k attack success rates, because real world attackers may be able to attempt attacks multiple times at little cost.

100

5/ Here are results for several specific malicious tasks of varying harmfulness and complexity, including new scenarios we added to the framework (more details in the blog post on our improvements to AgentDojo):

100

4/ Note that AgentDojo has four “environments” simulating different AI assistant deployment settings. Red teamers only had access to the “Workspace” environment, but as the above plot shows, the attack transferred very well to the three unseen environments.

100

3/ To find out, we organized a red teaming exercise. The resulting attack is much more effective than the pre-packaged attacks. In a majority of cases, the agent follows the hijacker’s instructions:

2/ AgentDojo is a framework for evaluating agent hijacking. Since its June release, some newer models – such as Claude 3.5 Sonnet (October version) – have shown markedly improved robustness to the included attacks. But what happens when we stress test the model with new attacks?

Technical Blog: Strengthening AI Agent Hijacking Evaluations

1/ Excited to share a new blog post from the U.S. AI Safety Institute! AI agents are becoming more capable, but they are vulnerable to prompt injections in external content – an agent may be given task A, but then be “hijacked” and perform malicious task B instead. www.nist.gov/news-events/...

nist.gov

Large AI models are increasingly used to power agentic systems, or “agents,” which can automate complex tasks on behalf of users.

140

Ben Edelman @benedelman.bsky.social · 08/12/2024

Thanks to @desmos.com's 3D calculator, you can now design your very own animated Lissajous knot! Demo: www.desmos.com/3d/fnqqqsbvuc For the best experience, click and drag the view to get it spinning. (disclaimer: the loop loop is only visible on my homepage when browser width >=1024px)

desmos.com

Undulating Lissajous Knots

074

Ben Edelman @benedelman.bsky.social · 08/12/2024

For years, this mysterious undulating loop has lived at the top of my personal homepage.

1152

Ben Edelman @benedelman.bsky.social · 07/12/2024

Agreed, but the story describes *discovering* a tiny piece of maggot in the remaining apple after having taken a bite. (the perhaps questionable assumption being that the maggot piece was quite recently part of a whole)

Ben Edelman @benedelman.bsky.social · 07/12/2024

My favorite "ordinary life" example of this notion of singular limits: (from mecheng.iisc.ac.in/lamfip/me304...)

I don't. Can let you know if I end up making one.

(accidentally omitted some text which was meant to precede the above:) The model system approach can be found everywhere across the sciences and for good reason: it is often the shortest path to conceptual insights—as long as the conditions are right...

000

I'll end this thread with the parable that opens the dissertation (my conference will require a parable section in every submission). Tag yourself :)

130

The bulk of the thesis is a series of case studies from my research. But first, in Chapter 3 ("Deep Learning Preliminaries") I try to define some terms from first principles—above these footnotes, you can find my idiosyncratic definition of neural nets in terms of arithmetic circuits.

2. Transferability: insights learned from the system need to transfer to settings of interest. This can happen because of *low-level* commonalities (think cell cultures) or *high-level* commonalities (think macroeconomic models).

...Specifically, two conditions I propose in the thesis: 1. Productivity: A model system needs to be exceptionally fertile ground for producing scientific insights.

220

It's a tribute to a kind of science I love (and reviews sometimes hate), where in order to understand a complicated system (e.g. training a transformer on internet text), you instead study a different system (e.g. training an MLP to solve parity problems).

I defended my PhD dissertation back in May. I didn't have time to share it widely then (newborn baby), but I think some of you might enjoy it, especially the opening chapters: benjaminedelman.com/assets/disse...

3313

(edit: sensors, not sensory inputs)

000

What explanations am I missing? (It's interesting, btw, to think about how different combinations of the above are relevant to case studies such as protein structure prediction and language learning.)

010

7/ The anthropic principle: the evolution of learning (and thus the evolution of us) was only possible if simple, computationally efficient functions had predictive power that could be leveraged for increased fitness.

6/ The state of reality on Earth is selected (naturally and artificially) to be learnable—consider, e.g., biological signaling mechanisms, human communication, and legibility imposed/incentivized by states and markets. (note: there can also be selection against learnability)

100

5/ Our sensory inputs (both biological and technological) are selected/designed to capture the most (efficiently) predictive aspects of reality.

200

4/ Reality as we observe it tends to obey the principle of locality. (en.m.wikipedia.org/wiki/Princip...)

100

3/ Complex systems tend towards emergent order.

100

2/ We live in a weirdly low-entropy environment.

100

1/ The fundamental laws of the universe are simple.

100