iiinova — where your digital self keeps evolving

Gaspard Lambrechts @gsprd.be · 20/02/2026

Congratulations to the hardworking folks at UPenn! Thank you Edward for including me and for all the nice discussions. 🌐 penn-pal-lab.github.io/aawr 📝 openreview.net/forum?id=Rkd... 💻 github.com/penn-pal-lab... More theory details in Appendix A-E and on slide 30 (orbi.uliege.be/handle/2268/...)

010

Gaspard Lambrechts @gsprd.be · 20/02/2026

As seen from the results and videos, AAWR improves significantly on (i) foundation policies, (ii) behavior cloning policies, and (iii) AWR policies, providing good policies even in partially observable environments with non Markovian inputs.

110

Gaspard Lambrechts @gsprd.be · 20/02/2026

It is the case here, where we use additional cameras, position estimates, or bounding boxes from pretrained models. These features i are used as additional input of the critic Q(i, z, a) to provide a better advantage estimate and policy improvement direction.

100

Gaspard Lambrechts @gsprd.be · 20/02/2026

In fact, it is a common assumption in asymmetric RL, which distinguishes the execution information from the training information. In practice, while we do not always know the exact state, it is common to have more information available about the state at training time.

120

Gaspard Lambrechts @gsprd.be · 20/02/2026

Unfortunately, we show that we cannot just learn the symmetric critic Q(z, a) = E[G | z, a]. Instead, we need an asymmetric critic Q(s, z, a) = E[G | s, z, a] for a valid policy iteration. This is because, unlike for policy gradients, the AWR objective is not linear in Q.

100

Gaspard Lambrechts @gsprd.be · 20/02/2026

To learn a good policy (for this specific input z), we may want to rely on existing RL algorithms such as policy gradient or policy iteration. Here, because we perform offline-to-online training, we rely on AWR, a policy iteration algorithm going offline to online seamlessly.

100

Gaspard Lambrechts @gsprd.be · 20/02/2026

When learning in real-world scenarios, it is common to have constraints on the input available to the policy at execution time (e.g., last observation only, wrist camera only, etc). In the general case (POMDP), the input z is a function of the observation history h: z = f(h).

100

Gaspard Lambrechts @gsprd.be · 16/07/2025

At #ICML2025, we will present a theoretical justification for the benefits of « asymmetric actor-critic » algorithms (#W1008 Wednesday at 11am). 📝 Paper: hdl.handle.net/2268/326874 💻 Blog: damien-ernst.be/2025/06/10/a...

ICML poster of the paper « A Theoretical Justification for Asymmetric Actor-Critic Algorithms » by Gaspard Lambrechts, Damien Ernst and Aditya Mahajan.

083

Gaspard Lambrechts @gsprd.be · 11/07/2025

Last week, I gave an invited talk on "asymmetric reinforcement learning" at the BeNeRL workshop. I was happy to draw attention to this niche topic, which I think can be useful to any reinforcement learning researcher. Slides: hdl.handle.net/2268/333931.

063

Gaspard Lambrechts @gsprd.be · 13/06/2025

Two months after my PhD defense on RL in POMDP, I finally uploaded the final version of my thesis :) You can find it here: hdl.handle.net/2268/328700 (manuscript and slides). Many thanks to my advisors and to the jury members.

Cover page of the PhD thesis "Reinforcement Learning in Partially Observable Markov Decision Processes: Learning to Remember the Past by Learning to Predict the Future" by Gaspard Lambrechts

082

Gaspard Lambrechts @gsprd.be · 09/06/2025

Now, as far as the actor suboptimality is concerned, we obtained the following finite-time bounds. In addition to the average critic error, which is also present in the actor bound, the symmetric actor-critic algorithm suffers from an additional "inference term".

Theorem showing the finite-time suboptimality bound for the asymmetric and symmetric actor-critic algorithms. The asymmetric algorithm has four terms: the natural actor-critic term, the gradient estimation term, the residual gradient term, and the average critic error. The symmetric algorithm has an additional term: the inference term.

100

Gaspard Lambrechts @gsprd.be · 09/06/2025

By adapting the finite-time bound from the symmetric setting to the asymmetric setting, we obtain the following error bounds for the critic estimates. The symmetric temporal difference learning algorithm has an additional "aliasing term".

Theorem showing the finite-time error bound for the asymmetric and symmetric temporal difference learning algorithms. The asymmetric algorithm has three terms: the temporal difference learning term, the function approximation term, and the bootstrapping shift term. The symmetric algorithm has an additional term: the aliasing term.

100

Gaspard Lambrechts @gsprd.be · 09/06/2025

While this algorithm is valid/unbiased (Baisero & Amato, 2022), a theoretical justification for its benefit is still missing. Does it really learn faster than symmetric learning? In this paper, we provide theoretical evidence for this, based on an adapted finite-time analysis (Cayci et al., 2024).

Title page of the paper "A Theoretical Justification for Asymmetric Actor-Critic Algorithms", written by Gaspard Lambrechts, Damien Ernst and Aditya Mahajan.

100

Gaspard Lambrechts @gsprd.be · 09/06/2025

However, with actor-critic algorithms, it can be noticed that the critic is not needed at execution! As a result, the state can be an input of the critic, which becomes Q(s, z, a) in the asymmetric setting instead of Q(z, a) in the symmetric setting.

Figure showing the policy being passed the feature z = f(h) of the history, and the critic being passed both the feature z = f(h) of the history and the state s as input. The asymmetric critic and the policy are used together to form the sample policy-gradient expression.

100

Gaspard Lambrechts @gsprd.be · 09/06/2025

In a POMDP, the goal is to find an optimal policy π(a|z) that maps a feature z = f(h) of the history h to an action a. In a privileged POMDP, the state can be used to learn a policy π(a|z) faster. But note that the state cannot be an input of the policy, since it is not available at execution.

Figure showing the history h being compressed into a feature z = f(h) for then being passed to a policy g(a | z).

100

Gaspard Lambrechts @gsprd.be · 09/06/2025

Typically, classical RL methods assume: - MDP: full state observability (too optimistic), - POMDP: partial state observability (too pessimistic). Instead, asymmetric RL methods assume: - Privileged POMDP: asymmetric state observability (full at training, partial at execution).

Table showing that the MDP assumes full state observability both during training and execution and that the POMDP assumes partial state observability both during training and execution, while the privileged POMDP assumes full state observability during training but partial state observability during execution.

100

Gaspard Lambrechts @gsprd.be · 09/06/2025

📝 Our paper "A Theoretical Justification for Asymmetric Actor-Critic Algorithms" was accepted at #ICML! Never heard of "asymmetric actor-critic" algorithms? Yet, many successful #RL applications use them (see image). But these algorithms are not fully understood. Below, we provide some insights.

Slide showing three recent successes of reinforcement learning that have used an asymmetric actor-critic algorithm:
- Magnetic Control of Tokamak Plasma through Deep RL (Degrave et al., 2022).
- Champion-Level Drone Racing using Deep RL (Kaufmann et al., 2023).
- A Super-Human Vision-Based RL Agent in Gran Turismo (Vasco et al., 2024).

1177

Gaspard Lambrechts @gsprd.be · 03/04/2025

Slydst, my Typst package for making simple slides, just got its 100th star on Github. While I would not advise using Typst for papers yet, its markdown-like syntax allows to create slides in a few minutes, while supporting everything we love from LaTeX: equations. github.com/glambrechts/...

Typst interface showing an example of Slydst code and the resulting slides.

030