iiinova — where your digital self keeps evolving

Aly Lidayan

@aliday.bsky.social

38 followers 30 following 10 posts

CS PhD student at UC Berkeley studying RL and cognitive science alyd.github.io

Posts Replies Media

Reposted by Aly Lidayan

Luigi Acerbi @lacerbi.bsky.social · 30/09/2026

1/ A short film about how your brain solves the "one thing or two?" problem, based on our recent paper with Shuze Liu, Trevor Holland and @weijima.bsky.social in PLoS Comp Biol.

173

Aly Lidayan @aliday.bsky.social · 04/08/2026

We investigate this with a novel task where we manipulate how small a shift in frame is required to switch goals. See our new CogSci paper for more! escholarship.org/uc/item/4mv3... Work done with @gaiamolinaro.bsky.social and @annecollins.bsky.social ✨

escholarship.org

041

Aly Lidayan @aliday.bsky.social · 04/08/2026

New paper! Setting goals is so useful in complex settings because it can reduce the environment's complexity to only the goal-relevant features. But for humans, reframing comes at a cost. Could this explain why we're so bad at abandoning goals that are no longer worthwhile? 🧵

1133

Aly Lidayan @aliday.bsky.social · 25/04/2025

I'm presenting this 3-5:30pm on Saturday, Hall 3 #396 🌞 come chat about designing rewards and intrinsic motivation for RL + meta RL!

000

Reposted by Aly Lidayan

Christian Guckelsberger @creativeendvs.bsky.social · 08/04/2025

1/3 Out now: new paper on people's perception of AI (robot) creativity! Core finding: we attribute more creativity to a creative act if people not only see the final artwork, but also its creation process & the robot making it. Video: vimeo.com/1073134853 Open-access paper: doi.org/10.1145/3711...

1105

Aly Lidayan @aliday.bsky.social · 26/03/2025

Get all the details in our paper: arxiv.org/abs/2409.05358 🚀 This work was a joint effort with Michael Dennis and Stuart Russell at UC Berkeley!

arxiv.org

BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping

Intrinsic motivation and reward shaping guide reinforcement learning (RL) agents by adding pseudo-rewards, which can lead to useful emergent behaviors. However, they can also encourage counterproducti...

000

Aly Lidayan @aliday.bsky.social · 26/03/2025

5️⃣We demonstrate our framework in Mountain Car. We set the potential to the maximum displacement the agent learnt to reach so far, signaling the value of its training. Rewarding displacement directly (pink) led to reward-hacking but the BAMPF (green) preserved optimality✅

100

Aly Lidayan @aliday.bsky.social · 26/03/2025

4️⃣We get a new typology for intrinsic motivation & reward shaping terms based on which BAMDP value component they signal! They hinder exploration if they align poorly with actual value, e.g., prediction error is high for watching a noisy TV but no valuable information is gained.

100

Aly Lidayan @aliday.bsky.social · 26/03/2025

3️⃣To guide more efficient exploration, BAMPF potentials should encode BAMDP state value. To gain further insights, we decompose BAMDP value into the value of the information gathered🧠 and the value of the MDP state given prior knowledge only🌎.

100

Aly Lidayan @aliday.bsky.social · 26/03/2025

2️⃣Harmful reward-hacking policies maximize modified rewards to the detriment of true rewards. We prove that converting IM and reward shaping terms to BAMDP potential-based shaping functions (BAMPFs) prevents hacking, and empirically validate this in both RL and meta-RL.

100

Aly Lidayan @aliday.bsky.social · 26/03/2025

1️⃣We cast RL agents as policies in Bayes-Adaptive MDPs, which augment the MDP state with the history of all environment interactions. Optimal exploration maximizes BAMDP state value, and pseudo-rewards guide RL agents by rewarding them for going to more valuable BAMDP states.

100

Aly Lidayan @aliday.bsky.social · 26/03/2025

🚨Our new #ICLR2025 paper presents a unified framework for intrinsic motivation and reward shaping: they signal the value of the RL agent’s state🤖=external state🌎+past experience🧠. Rewards based on potentials over the learning agent’s state provably avoid reward hacking!🧵

1103