Sign in

Marlos C. Machado

@marloscmachado.bsky.social
927 followers 226 following 70 posts

Assistant Professor at the University of Alberta. Amii Fellow, Canada CIFAR AI chair. ML researcher. All things reinforcement learning. Photo by Ampersand Grey. 📍 Edmonton, Canada 🇨🇦 🔗 mcmachado.github.io 🗓️ Joined November, 2024

PostsRepliesMedia
Marlos C. Machado @marloscmachado.bsky.social · 05/10/2026
I am so excited about this paper. Wayfarer fulfills a vision I had 10 years ago when I wrote Learning Purposeful Behaviour in the Absence of Rewards. It fulfills the promise of options, maybe for the first time. End of thread. 9/9 arXiv link: arxiv.org/abs/2610.03604
arxiv.org
Mastering Atari 2600 Games with Discovered Options
Temporal abstractions, often instantiated as options, have long been regarded as a mechanism for accelerating credit assignment, facilitating exploration, and enabling generalisation in reinforcement ...
041 0
Marlos C. Machado @marloscmachado.bsky.social · 05/10/2026
Some results are quite compelling, such as learning a representation that considers only what's under the agent's control. In Freeway, instead of learning about cars, it naturally learns about the only degree of freedom in the game, and the agent simply learns to go up. 8/9
130 0
Marlos C. Machado @marloscmachado.bsky.social · 05/10/2026
These results were obtained on a challenging set of Atari 2600 games, highlighting games where progress has historically been much slower than average due to difficult exploration and credit assignment problems. We also tried to characterize each component's contribution. 7/9
110 0
Marlos C. Machado @marloscmachado.bsky.social · 05/10/2026
The options Wayfarer discovers also generalize across different settings. For example, in Montezuma's Revenge, the agent discovers an option that consists of walking across the tiled floor and going down the stairs. It works even in rooms the agent had never seen! 6/9
110 0
Marlos C. Machado @marloscmachado.bsky.social · 05/10/2026
The benefit of using options for credit assignment is also easy to see. In Montezuma's Revenge, for example, the agent discovers options (without domain knowledge) that it can sequence to reach the 3rd screen in 2 orders of magnitude fewer decisions. 5/9 www.youtube.com/watch?v=RpCn...
youtube.com
Wayfarer: Montezuma's Revenge — primitive actions vs. options
YouTube video by Erik Lintunen
120 0
Marlos C. Machado @marloscmachado.bsky.social · 05/10/2026
Wayfarer enables temporally extended exploration with options that generate observations that are hard to obtain from the agent's perspective. For example, a single option executed in the game Private Eye lets the agent scan the entire environment! 4/9 www.youtube.com/shorts/RByU_...
youtube.com
Wayfarer: Private Eye — discovered option
YouTube video by Erik Lintunen
110 0
Marlos C. Machado @marloscmachado.bsky.social · 05/10/2026
Beyond being quite general and actually improving performance over primitive actions (and outperforming algorithms such as DreamerV3, IQN, and Rainbow), Wayfarer discovers options that simultaneously improve exploration, credit assignment, and transfer. 3/9
110 0
Marlos C. Machado @marloscmachado.bsky.social · 05/10/2026
Our algorithm, Wayfarer, learns an agent-centric Laplacian representation and uses it to define an intrinsic reward that shapes options, which in turn shape the agent's experience, leading to updated representations and options in a virtuous cycle. 2/9
120 0
Marlos C. Machado @marloscmachado.bsky.social · 05/10/2026
I'm excited to share Erik Lintunen's work! Mastering Atari 2600 Games with Discovered Options We have a general-purpose option-discovery method that works w/ high-dimensional observations, tabula rasa, using a single stream of experience! It outperforms DreamerV3, Rainbow, and IQN! 1/9
2195 0
Marlos C. Machado @marloscmachado.bsky.social · 28/08/2026
"It was part of the essential deceit of the Ring to fill minds with imaginations of supreme power." - Tolkien, Letter 246
021 0
Marlos C. Machado @marloscmachado.bsky.social · 11/08/2026
Every year, we have the RLAI Tea Time Talks at the @ualberta.bsky.social. Last year, I talked about how some of our work can be seen from a physics perspective. I gave the same talk at the RLC'25 IBRL Workshop. I uploaded the recording to make it broadly accessible. www.youtube.com/watch?v=_a5q...
youtube.com
Heat Diffusion as an Inductive Bias for Discovering Temporal Abstractions in RL
YouTube video by Marlos C. Machado
093 0
Marlos C. Machado @marloscmachado.bsky.social · 06/07/2026
Here's another paper we'll be presenting at ICML'26 - @icmlconf.bsky.social. Laplacian Representations for Decision-Time Planning openreview.net/forum?id=6Jw... This work was led by Dikshant Shehmar, and he'll be presenting on 8th July at 2 pm, Korea time. This video summarizes the idea well ↓
0102 0
Marlos C. Machado @marloscmachado.bsky.social · 21/05/2026
I'm honoured to have been renewed as one of the Canada CIFAR AI Chairs. It continues to be an honour to be part of such an amazing group. The ecosystem created by CIFAR, @amiithinks.bsky.social, and @ualberta.bsky.social continues to be a major factor for our success. cifar.ca/cifarnews/20...
cifar.ca
Government of Canada and CIFAR announce $24M investment in top AI Talent - CIFAR
CIFAR
050 0
Marlos C. Machado @marloscmachado.bsky.social · 06/05/2026
Our paper, “The Cell Must Go On! AgarCL as an Evaluation Platform for Continual RL“, has been accepted at @rl-conference.bsky.social'26! This work was led by Mohamed Ayman as a master's student at Univ. of Alberta and Amii. Preprint: arxiv.org/abs/2505.18347 Blog post: medium.com/@marlos.chol...
medium.com
The Cell Must Go On! AgarCL as an Evaluation Platform for Continual RL
Continual learning is becoming more popular, with people from diverse research ideologies promoting it. Naturally, as with everything else…
092 0
Reposted by Marlos C. Machado
RL and Video Games Workshop @rlvgworkshop.bsky.social · 20/04/2026
Wake up Samurai, RLVG IS BACK! 😎 We are happy to announce the awesome speakers of the second edition of the RLVG workshop: @alexkearney.com, @marloscmachado.bsky.social, Martin Singh-Bloom, Sam Devlin, Pete Wurman and @neuroai.bsky.social! Join us August 15th at @rl-conference.bsky.social!
1196 0
Marlos C. Machado @marloscmachado.bsky.social · 25/03/2026
This work was led by Diego Gomez (tarod13.github.io), in collaboration with @michaelhbowling.bsky.social and @jquinnlee.bsky.social. What an environment we have here at @ualberta.bsky.social and @amiithinks.bsky.social! Preprint: biorxiv.org/content/10.1... Blog post: medium.com/@marlos.chol...
medium.com
From Pixels to Place Cells: Where Representation Learning Meets Neuroscience
I have been itching to talk about a paper we put out on bioRxiv last year, and I finally found the time to write about it. This is a…
010 0
Marlos C. Machado @marloscmachado.bsky.social · 25/03/2026
Are these results real? We then replicated experiments that measured cell activations in the hippocampus of mice during remapping experiments. We observed the NNs we trained performing quite similarly to the neural data from previous work! 🤯🤯🤯 Ref: www.cell.com/neuron/fullt...
110 0
Marlos C. Machado @marloscmachado.bsky.social · 25/03/2026
In other words, the successor representation shows up immediately before grid cells, similarly to how the hippocampus (where place cells are) comes after the entorhinal cortex (to understand the direction, think about what influences what when performing backprop in an ANN).
110 0
Marlos C. Machado @marloscmachado.bsky.social · 25/03/2026
Moreover, we see activations resembling place cells in the 3rd layer and border and corner cells in the 1st and 2nd layers (and many other cell types). These results showed the formation of a representational loop consistent with anatomical and functional circuits in the brain.
110 0
Marlos C. Machado @marloscmachado.bsky.social · 25/03/2026
First, we showed that when training a neural network to estimate, from high-dimensional observations, the eigenvectors of the successor representation (or graph Laplacian), we see activation patterns resembling those in the entorhinal cortex. In fact, we see many of those!
110 0
Marlos C. Machado @marloscmachado.bsky.social · 25/03/2026
A couple of months ago, we released a preprint of one of my favourite papers I’ve ever written. It lies at the intersection of representation learning and neuroscience. I have now written a blog post about it. Preprint: biorxiv.org/content/10.1... Blog post: medium.com/@marlos.chol...
medium.com
From Pixels to Place Cells: Where Representation Learning Meets Neuroscience
I have been itching to talk about a paper we put out on bioRxiv last year, and I finally found the time to write about it. This is a…
140 0
Marlos C. Machado @marloscmachado.bsky.social · 10/12/2025
"Canada Impact+ Research Chairs program—a new $1 billion investment that will provide Canadian institutions the opportunity to recruit top-tier international researchers with expertise in key areas ..." www.canada.ca/en/innovatio...
canada.ca
Government of Canada launches new initiative to recruit world-leading researchers - Canada.ca
Canada will invest $1.7 billion to attract top global talent
020 0
Reposted by Marlos C. Machado
Erin Grant @eringrant.me · 06/12/2025
Thrilled to start 2026 as faculty in Psych & CS @ualberta.bsky.social + Amii.ca Fellow! 🥳 Recruiting students to develop theories of cognition in natural & artificial systems 🤖💭🧠. Find me at #NeurIPS2025 workshops (speaking coginterp.github.io/neurips2025 & organising @dataonbrainmind.bsky.social)
410928 0
Marlos C. Machado @marloscmachado.bsky.social · 27/11/2025
The Computing Science Dept. at the University of Alberta has multiple faculty job openings. Please share this broadly. We have a great environment! - CS Theory: tinyurl.com/zrh9mk69 - Network/Cyber Security: tinyurl.com/renxazzy - Robotics/CV/Graphics: tinyurl.com/ypcsfbff
092 0
Marlos C. Machado @marloscmachado.bsky.social · 20/11/2025
The Department of Computing Science at the University of Alberta at the University of Alberta has an opening for another tenure-track faculty in robotics. Please, spread the word. I can attest to how awesome our department and @amiithinks.bsky.social are! (Official job posting coming soon.)
030 0
Marlos C. Machado @marloscmachado.bsky.social · 07/10/2025
Ratatouille (2007)
060 0
Marlos C. Machado @marloscmachado.bsky.social · 18/09/2025
This paper has now been accepted @neuripsconf.bsky.social ! Huge congratulations, Hon Tik (Rick) Tse and Siddarth Chandrasekar.
073 0
Marlos C. Machado @marloscmachado.bsky.social · 27/08/2025
2/2: “Conquerors live in dread of the day when they are shown to be, not superior, but simply lucky.” ― N.K. Jemisin, The Stone Sky
020 0
Marlos C. Machado @marloscmachado.bsky.social · 27/08/2025
1/2: But there are none so frightened, or so strange in their fear, as conquerors. They conjure phantoms endlessly, terrified that their victims will someday do back what was done to them—even if, in truth, their victims couldn’t care less about such pettiness and have moved on.”
121 0
Reposted by Marlos C. Machado
Reinforcement Learning Conference @rl-conference.bsky.social · 07/08/2025
Excited to announce the RLC best paper awards! Like last year, we wanted to highlight the many excellent ways you can do research. rl-conference.cc/RLC2025Award...
rl-conference.cc
RLC 2025 - Outstanding Paper Awards
0106 0
Marlos C. Machado @marloscmachado.bsky.social · 04/08/2025
* RLC Journal to Conference Track:* (Originally published at TMLR) - Deep RL track (Thu): AGaLiTe: Approximate Gated Linear Transformers for Online Reinforcement Learning by S. Pramanik
010 0
Marlos C. Machado @marloscmachado.bsky.social · 04/08/2025
* RLC Full Papers:* (These are great papers!) - Deep RL track (Thu): Deep Reinforcement Learning with Gradient Eligibility Traces by E. Elelimy - Foundations track (Fri): An Analysis of Action-Value Temporal-Difference Methods That Learn State Values by B. Daley and P. Nagarajan
110 0
Marlos C. Machado @marloscmachado.bsky.social · 04/08/2025
* RLC Workshop Papers (2/2):* Inductive Biases in RL sites.google.com/view/ibrl-wo... - A Study of Value-Aware Eigenoptions by H. Kotamreddy
100 0
Marlos C. Machado @marloscmachado.bsky.social · 04/08/2025
* RLC Workshop Papers (1/2):* RL Beyond Rewards rlbrew2-workshop.github.io - Tue 11:59 (spotlight talk): Towards An Option Basis To Optimize All Rewards by S. Chandrasekar - The World Is Bigger: A Computationally-Embedded Perspective on the Big World Hypothesis by A. Lewandowsi
rlbrew2-workshop.github.io
Workshop on Reinforcement Learning Beyond Rewards: Ingredients for Developing Generalist Agents
100 0
Marlos C. Machado @marloscmachado.bsky.social · 04/08/2025
Here's what our group will be presenting at RLC'25. * Invited Talks at Workshops:* Tue 10:00: The Causal RL Workshop sites.google.com/uci.edu/crlw... Tue 14:30: Inductive Biases in RL (IBRL) Workshop sites.google.com/view/ibrl-wo... Tue 15:00: Panel Discussion at IBRL Workshop
100 0
Marlos C. Machado @marloscmachado.bsky.social · 04/08/2025
RLC starts tomorrow here in Edmonton. I couldn't be more excited! It has a fantastic roll of speakers, great papers, and workshops. And this time, it is in Edmonton 😁 @rl-conference.bsky.social is my favourite conference, and no, it is not because I am one of its organizers this year.
0123 0
Marlos C. Machado @marloscmachado.bsky.social · 27/06/2025
This was a great long-term effort from @martinklissarov.bsky.social, Akhil Bagaria, and @ray-luo.bsky.social, and it led to a great overview of the ideas behind leveraging temporal abstractions in AI. If anything, I think this is very useful resource for anyone interested in this field!
071 0
Reposted by Marlos C. Machado
Reinforcement Learning Conference @rl-conference.bsky.social · 30/05/2025
To align better with workshop acceptance dates, 𝐑𝐋𝐂 𝐢𝐬 𝐞𝐱𝐭𝐞𝐧𝐝𝐢𝐧𝐠 𝐢𝐭𝐬 𝐞𝐚𝐫𝐥𝐲 𝐫𝐞𝐠𝐢𝐬𝐭𝐫𝐚𝐭𝐢𝐨𝐧 𝐝𝐞𝐚𝐝𝐥𝐢𝐧𝐞 𝐭𝐨 𝐉𝐮𝐧𝐞 𝟐𝟑𝐫𝐝!
083 0
Marlos C. Machado @marloscmachado.bsky.social · 27/05/2025
9/9: I genuinely think AgarCL might unlock new research avenues in CRL, including loss of plasticity, exploration, representation learning, and more. I do hope you consider using it. Repo: github.com/machado-rese... Website: agarcl.github.io Preprint: arxiv.org/abs/2505.18347
060 0
Marlos C. Machado @marloscmachado.bsky.social · 27/05/2025
8/9: Well, if you are still interested, you should probably consider reading the paper, but it is interesting to see that most of the agents we considered were able to reach human-level performance only in the most benign settings. And we did use a lot of computing here!
141 0
Marlos C. Machado @marloscmachado.bsky.social · 27/05/2025
7/9: Through mini-games, we tried to quantify and isolate some of the challenges AgarCL poses, including partial observability, non-stationarity, exploration, hyperparameter tuning, and the non-episodic nature of the environment (so easy to forget!). Where do our agents "break"?
110 0
Marlos C. Machado @marloscmachado.bsky.social · 27/05/2025
6/9: Importantly, this is a challenge problem that forces us to deal with many problems we often avoid, such as hyperparameter sweeps and exploration in CRL. It is perhaps no surprise that the classic algorithms we considered couldn't really make much progress in the full game.
120 0
Marlos C. Machado @marloscmachado.bsky.social · 27/05/2025
5/9: Over time, even the agent's observation will change, as the camera needs to zoom out to accommodate more agents; not to mention that there are other agents in the environment. I'm very excited about AgarCL because I think it allows us to ask questions we couldn't before.
120 0
Marlos C. Machado @marloscmachado.bsky.social · 27/05/2025
4/9: AgarCL is an adaptation of agar.io, a game with simple mechanics that lead to complex interactions. It's non-episodic, and a key aspect is that the agent dynamics change as it accumulates mass: It becomes slower, gains new affordances, sheds more mass, etc.
140 0
Marlos C. Machado @marloscmachado.bsky.social · 27/05/2025
3/9: AgarCL is our attempt at an environment with the complexity of a "big world" but in a smooth way, where the "laws of physics" don't change. It has complex dynamics, is partially observable, with non-stationarity, pixel-based observations, and a hybrid action space.
130 0
Marlos C. Machado @marloscmachado.bsky.social · 27/05/2025
2/9: CRL is often motivated by the idea that the world is bigger than the agent, requiring tracking. We usually simulate this with non-stationarity by cycling through classic episodic problems. I've written papers like this, but it feels too artificial. arxiv.org/abs/2303.07507
130 0
Marlos C. Machado @marloscmachado.bsky.social · 27/05/2025
📢 I'm very excited to release AgarCL, a new evaluation platform for research in continual reinforcement learning‼️ Repo: github.com/machado-rese... Website: agarcl.github.io Preprint: arxiv.org/abs/2505.18347 Details below 👇
1298 0
Marlos C. Machado @marloscmachado.bsky.social · 25/05/2025
This is great, thanks for sharing! We will read your paper carefully.
010 0
Marlos C. Machado @marloscmachado.bsky.social · 24/05/2025
7/7: We just scratched the surface here, but I think this could be the beginning of something interesting; that might be relevant to research questions ranging from safety in RL all the way to cognitive sciences. Again, here's the preprint by Tse et al.: arxiv.org/abs/2505.16217
arxiv.org
Reward-Aware Proto-Representations in Reinforcement Learning
In recent years, the successor representation (SR) has attracted increasing attention in reinforcement learning (RL), and it has been used to address some of its key challenges, such as exploration, c...
090 0
Marlos C. Machado @marloscmachado.bsky.social · 24/05/2025
6/7: We also show that, when compared to the SR, the DR gives rise to qualitatively different behavior in all sorts of tasks, such as reward shaping, exploration, & option discovery. Similar to what we did w/ STOMP, sometimes there's value in being aware of the reward function 😁
140 0