iiinova — where your digital self keeps evolving

Pedro Santos @pedrosantospps.bsky.social · 28/08/2026

Last week, I attended the Reinforcement Learning Conference in Montreal to present our work, "Risk-Aware General-Utility Markov Decision Processes" (arxiv.org/abs/2607.09298). More details about our work are in the following blog post: ppsantos.github.io/posts/risk-a...

ppsantos.github.io

Bringing risk-awareness to general-utility MDPs

We motivate and explore risk-aware general-utility MDPs, where we aim to find an optimal policy with respect to a risk measure of the distribution of objective values induced by its interaction with t...

000

Pedro Santos @pedrosantospps.bsky.social · 10/02/2026

We provide experimental results showing that our MCTS-based approach to solve GUMDPs in the single-trial regime is successful in tasks such as exploration, imitation learning or adversarial MDPs. N/N

010

Pedro Santos @pedrosantospps.bsky.social · 10/02/2026

Then, we explore how online planning techniques can be used to solve GUMDPs in the single-trial regime. In particular, we show that we can use an MCTS algorithm to provably solve GUMDPs in the single-trial regime.

110

Pedro Santos @pedrosantospps.bsky.social · 10/02/2026

In our work, under the discounted infinite-horizon setting, we first provide fundamental results for policy optimization in the single-trial regime. We show that non-Markovianity matters, connect single-trial optimization with solving a particular MDP, and prove a hardness result.

110

Pedro Santos @pedrosantospps.bsky.social · 10/02/2026

In particular, the optimal policy for the single-trial regime can differ from the optimal policy for the multiple-trial regime. This is unfortunate since the single-trial regime is important in the real-world, where policy performance is usually assessed based on a single trajectory.

110

Pedro Santos @pedrosantospps.bsky.social · 10/02/2026

However, previous works (jmlr.org/papers/volum..., arxiv.org/pdf/2409.15128) pointed out that the performance of a policy depends on the number of trials/trajectories drawn to evaluate its performance.

110

Pedro Santos @pedrosantospps.bsky.social · 10/02/2026

GUMDPs generalize the MDP framework by allowing the performance of a given policy to depend on a (possibly non-linear) function of the frequency of visitation of state-action pairs induced by the policy.

110

Reposted by Pedro Santos

GAIPS Lab @gaipslab.bsky.social · 30/01/2026

Here’s Pedro at yet another international conference! 🙌✨ GAIPS member Pedro P. Santos presented “Centralized training with hybrid execution in multi-agent reinforcement learning via predictive observation imputation” at #AAAI2026, Singapore 🇸🇬 📄 Check out his paper: doi.org/10.1016/j.ar...

011

Reposted by Pedro Santos

GAIPS Lab @gaipslab.bsky.social · 03/10/2025

Here’s some photos of GAIPS member @pedrosantospps.bsky.social presenting his work on ICML 2025 in Vancouver and EWRL 2025 in Tübingen, Germany. His poster was selected as a "spotlight poster" (top 2.6% of the papers)! 🙌 Read his work here: icml.cc/virtual/2025...

011

Reposted by Pedro Santos

Mirco Mutti @mircomutti.bsky.social · 24/07/2025

Walking around posters at @icmlconf.bsky.social, I was happy to see some buzz around convex RL—a topic I’ve worked on and strongly believe in. Thought I’d share a few ICML papers on this direction. Let’s dive in👇 But first… what is convex RL? 🧵 1/n

151

Pedro Santos @pedrosantospps.bsky.social · 03/05/2025

The paper can be found here: arxiv.org/pdf/2409.15128

arxiv.org

000

Pedro Santos @pedrosantospps.bsky.social · 03/05/2025

We provide lower and upper bounds on the mismatch between the finite and infinite trials formulations for GUMDPs, as well as empirical results to support our claims, highlighting how the number of trajectories and the structure of the underlying GUMDP influence policy evaluation.

100

Pedro Santos @pedrosantospps.bsky.social · 03/05/2025

We show that the number of trials plays a key role in infinite-horizon GUMDPs, and the expected performance of a given policy depends, in general, on the number of trials.

100

Pedro Santos @pedrosantospps.bsky.social · 03/05/2025

We contribute the first analysis on the impact of the number of trials, i.e., the number of randomly sampled trajectories, in infinite-horizon GUMDPs (considering both discounted and average formulations).

100

Pedro Santos @pedrosantospps.bsky.social · 03/05/2025

The general-utility Markov decision processes (GUMDPs) framework generalizes the MDPs framework by considering objective functions that depend on the frequency of visitation of state-action pairs induced by a given policy.

110

Pedro Santos @pedrosantospps.bsky.social · 03/05/2025

Happy to share that our paper "The Number of Trials Matters in Infinite-Horizon General-Utility Markov Decision Processes" got accepted as a spotlight poster at the International Conference on Machine Learning (ICML).

251