Exploration under sparse rewards, intrinsic motivation and language-guided agents built on pretrained foundation models.
This line studies how reinforcement learning (RL) agents can learn efficiently when rewards are sparse, and how pretrained foundation models (vision-language models and LLMs) can guide them through intrinsic motivation, high-level language prompts and structured task decomposition.
It grew out of Alain Andrés Fernández’s doctoral thesis (2023), which studied intrinsic motivation mechanisms for sample-efficient deep RL in hard-exploration environments. That work covered self-imitation learning with prioritization and diversity, the use of offline data in procedurally generated environments, and collaborative training of heterogeneous agents. Unai Ruiz-Gonzalez’s ongoing thesis extends it to open-ended learning that uses the latent knowledge of foundation models. Other contributions in this line address multitask RL through evolutionary multifactorial optimization, curiosity-driven continual offline RL, and safe RL when agents meet out-of-distribution observations.
Main contributions
Exploration and intrinsic motivation: an evaluation of intrinsic motivation techniques in hard-exploration environments (Andres et al., 2022), intrinsic motivation for self-imitation learning (Andres et al., 2022), prioritization and diversity in self-imitation over procedural environments (Andrés et al., 2023) and offline data to speed up RL (Andres et al., 2025).
Collaborative, multitask and continual RL: what and when heterogeneous agents should share (Andres et al., 2023), adaptive multifactorial evolutionary optimization for multitask RL (Martinez et al., 2022) and curiosity for an even representation of tasks in continual offline RL (Pathmanathan et al., 2024).
Foundation models as guides: intrinsic motivation with pretrained foundation models (NeurIPS 2024 workshop) (Andres & Del Ser, 2024), guiding agents with high-level language prompts (NeurIPS 2024 workshop) (Ruiz-Gonzalez et al., 2024) and LLMs for structured task decomposition under sparse rewards (Ruiz-Gonzalez et al., 2025).
Safe RL: detection of and adaptation to out-of-distribution observations in sparse-reward environments (Martínez-Seras et al., 2024).
Publications in this area (11)
All my publications related to this research line, in reverse chronological order, including those mentioned above.
2025
Neurocomputing
Using Offline Data to Speed-up Reinforcement Learning in Procedurally Generated Environments
Alain Andres, Lukas Schäfer, Stefano V. Albrecht, and Javier Del Ser
One of the key challenges of Reinforcement Learning (RL) is the ability of an agent to generalize its learned policy to unseen settings. Moreover, training an RL agent requires large numbers of interactions with the environment. Motivated by the success of Imitation Learning (IL), we conduct a study to investigate whether an agent can leverage offline data in the form of trajectories to improve the sample-efficiency in procedurally generated environments. We consider two settings of using IL from offline data for RL: (1) pre-training a policy before online RL training and (2) concurrently training a policy with online RL and IL from offline data. We analyze the impact of the quality (optimality of trajectories), quantity and diversity of available offline trajectories on the effectiveness of both approaches. Across four well-known sparse reward tasks in the MiniGrid environment, we find that using IL for both pre-training and concurrently during online RL training, consistently improves sample-efficiency, and in some tasks achieves higher returns compared to using either IL or RL alone. Furthermore, we show that training a policy from as few as two trajectories can make the difference between learning an optimal policy at the end of online training and not learning at all. Evaluation in two tasks of the Procgen environment further highlights that the diversity of the training data is more important than its quality. Our findings motivate the widespread adoption of IL for pre-training and concurrent IL in procedurally generated environments whenever offline trajectories are available or can be generated.
@article{andres2023offline,title={{Using Offline Data to Speed-up Reinforcement Learning in Procedurally Generated Environments}},author={Andres, Alain and Schäfer, Lukas and Albrecht, Stefano V. and Del Ser, Javier},journal={Neurocomputing},volume={618},year={2025},pages={129079},doi={10.1016/j.neucom.2024.129079},}
Machine Learning and Knowledge Extraction
Large Language Models for Structured Task Decomposition in Reinforcement Learning Problems with Sparse Rewards
Unai Ruiz-Gonzalez, Alain Andrés, and Javier Del Ser
Reinforcement learning (RL) agents face significant challenges in sparse-reward environments, as insufficient exploration of the state space can result in inefficient training or incomplete policy learning. To address this challenge, this work proposes a teacher–student framework for RL that leverages the inherent knowledge of large language models (LLMs) to decompose complex tasks into manageable subgoals. The capabilities of LLMs to comprehend problem structure and objectives, based on textual descriptions, can be harnessed to generate subgoals, similar to the guidance a human supervisor would provide. For this purpose, we introduce the following three subgoal types: positional, representation-based, and language-based. Moreover, we propose an LLM surrogate model to reduce computational overhead and demonstrate that the supervisor can be decoupled once the policy has been learned, further lowering computational costs. Under this framework, we evaluate the performance of three open-source LLMs (namely, Llama, DeepSeek, and Qwen). Furthermore, we assess our teacher–student framework on the MiniGrid benchmark—a collection of procedurally generated environments that demand generalization to previously unseen tasks. Experimental results indicate that our teacher–student framework facilitates more efficient learning and encourages enhanced exploration in complex tasks, resulting in faster training convergence and outperforming recent teacher–student methods designed for sparse-reward environments.
@article{ruizgonzalez2025large,title={{Large Language Models for Structured Task Decomposition in Reinforcement Learning Problems with Sparse Rewards}},author={Ruiz-Gonzalez, Unai and Andrés, Alain and Del Ser, Javier},journal={Machine Learning and Knowledge Extraction},volume={7},number={4},pages={126},year={2025},doi={10.3390/make7040126},}
2024
Cognitive Computation
Using Curiosity for an Even Representation of Tasks in Continual Offline Reinforcement Learning
Pankayaraj Pathmanathan, Natalia Díaz-Rodríguez, and Javier Del Ser
In this work, we investigate the means of using curiosity on replay buffers to improve offline multi-task continual reinforcement learning when tasks, which are defined by the non-stationarity in the environment, are non labeled and not evenly exposed to the learner in time . In particular, we investigate the use of curiosity both as a tool for task boundary detection and as a priority metric when it comes to retaining old transition tuples , which we respectively use to propose two different buffers. Firstly, we propose a Hybrid Reservoir Buffer with Task Separation (HRBTS), where curiosity is used to detect task boundaries that are not known due to the task-agnostic nature of the problem. Secondly, by using curiosity as a priority metric when it comes to retaining old transition tuples, a Hybrid Curious Buffer (HCB) is proposed. We ultimately show that these buffers, in conjunction with regular reinforcement learning algorithms, can be used to alleviate the catastrophic forgetting issue suffered by the state of the art on replay buffers when the agent’s exposure to tasks is not equal along time. We evaluate catastrophic forgetting and the efficiency of our proposed buffers against the latest works such as the Hybrid Reservoir Buffer (HRB) and the Multi-Time Scale Replay Buffer (MTR) in three different continual reinforcement learning settings. These settings are defined based on how many times the agent encounters the same task, how long they last, and how different new tasks are when compared to the old ones (i.e., how large the task drift is). The three settings are namely, 1. prolonged task encounter with substantial task drift, and no task re-visitation , 2. frequent, short-lived task encounter with substantial task drift and task re-visitation , and 3. every timestep task encounter with small task drift and task re-visitation . Experiments were done on classical control tasks and Metaworld environment. Experiments show that our proposed replay buffers display better immunity to catastrophic forgetting compared to existing works in all but the every time step task encounter with small task drift and task re-visitation. In this scenario curiosity will always be higher, thus not being an useful measure in both proposed buffers, making them not universally better than other approaches across all types of CL settings, and thereby opening up an avenue for further research.
@article{pathmanathan2023using,title={{Using Curiosity for an Even Representation of Tasks in Continual Offline Reinforcement Learning}},author={Pathmanathan, Pankayaraj and Díaz-Rodríguez, Natalia and Del Ser, Javier},journal={Cognitive Computation},volume={16},number={1},pages={425--453},year={2024},doi={10.1007/s12559-023-10213-9},}
NeurIPS 2024 Workshop on Intrinsically Motivated Open-ended Learning (IMOL)
Fostering Intrinsic Motivation in Reinforcement Learning with Pretrained Foundation Models
A. Andres and Javier Del Ser
In NeurIPS 2024 Workshop on Intrinsically Motivated Open-ended Learning (IMOL), 2024
Exploration remains a significant challenge in reinforcement learning, especially in environments where extrinsic rewards are sparse or non-existent. The recent rise of foundation models, such as CLIP, offers an opportunity to leverage pretrained, semantically rich embeddings that encapsulate broad and reusable knowledge. In this work we explore the potential of these foundation models not just to drive exploration, but also to analyze the critical role of the episodic novelty term in enhancing exploration effectiveness of the agent. We also investigate whether providing the intrinsic module with complete state information – rather than just partial observations – can improve exploration, despite the difficulties in handling small variations within large state spaces. Our experiments in the MiniGrid domain reveal that intrinsic modules can effectively utilize full state information, significantly increasing sample efficiency while learning an optimal policy. Moreover, we show that the embeddings provided by foundation models are sometimes even better than those constructed by the agent during training, further accelerating the learning process, especially when coupled with the episodic novelty term to enhance exploration.
@inproceedings{andres2024fostering,title={{Fostering Intrinsic Motivation in Reinforcement Learning with Pretrained Foundation Models}},author={Andres, A. and Del Ser, Javier},booktitle={NeurIPS 2024 Workshop on Intrinsically Motivated Open-ended Learning (IMOL)},year={2024},doi={10.48550/arxiv.2410.07404},}
NeurIPS 2024 Workshop on Open-World Agents (OWA)
Words as Beacons: Guiding RL Agents with High-Level Language Prompts
U. Ruiz-Gonzalez, A. Andres, P. G. Bascoy, and Javier Del Ser
In NeurIPS 2024 Workshop on Open-World Agents (OWA), 2024
Sparse reward environments in reinforcement learning (RL) pose significant challenges for exploration, often leading to inefficient or incomplete learning processes. To tackle this issue, this work proposes a teacher-student RL framework that leverages Large Language Models (LLMs) as "teachers" to guide the agent’s learning process by decomposing complex tasks into subgoals. Due to their inherent capability to understand RL environments based on a textual description of structure and purpose, LLMs can provide subgoals to accomplish the task defined for the environment in a similar fashion to how a human would do. In doing so, three types of subgoals are proposed: positional targets relative to the agent, object representations, and language-based instructions generated directly by the LLM. More importantly, we show that it is possible to query the LLM only during the training phase, enabling agents to operate within the environment without any LLM intervention. We assess the performance of this proposed framework by evaluating three state-of-the-art open-source LLMs (Llama, DeepSeek, Qwen) eliciting subgoals across various procedurally generated environment of the MiniGrid benchmark. Experimental results demonstrate that this curriculum-based approach accelerates learning and enhances exploration in complex tasks, achieving up to 30 to 200 times faster convergence in training steps compared to recent baselines designed for sparse reward environments.
@inproceedings{ruizgonzalez2024words,title={{Words as Beacons: Guiding RL Agents with High-Level Language Prompts}},author={Ruiz-Gonzalez, U. and Andres, A. and Bascoy, P. G. and Del Ser, Javier},booktitle={NeurIPS 2024 Workshop on Open-World Agents (OWA)},year={2024},doi={10.48550/arxiv.2410.08632},}
International Joint Conference on Neural Networks (IJCNN)
Advancing Towards Safe Reinforcement Learning over Sparse Environments with Out-of-Distribution Observations: Detection and Adaptation Strategies
Aitor Martínez-Seras, Alain Andrés, and Javier Del Ser
In International Joint Conference on Neural Networks (IJCNN), 2024
Safety in AI-based systems is among the highest research priorities, particularly when such systems are deployed in real-world scenarios subject to uncertainties and unpredictable inputs. Among them, the presence of long-tailed stimuli (Out-of-Distribution data, OoD) has captured much interest in recent times, giving rise to many proposals over the years to detect unfamiliar inputs to the model and adapt its knowledge accordingly. This work analyzes several OoD detection and adaptation strategies for Reinforcement Learning agents over environments with sparse reward signals. The sparsity of rewards and the impact of OoD objects on the state transition distribution learned by the agent are shown to be crucial when it comes to the design of effective knowledge transfer methods once OoD objects are detected. Furthermore, different approaches to detect OoD elements within the observation of the agent are also outlined, stressing on their benefits and potential downsides. Experiments with procedurally generated environments are performed to assess the performance of the considered OoD detection techniques, and to gauge the impact of the adaptation strategies on the generalization capability of the RL agent. The results pave the way towards further research around the provision of safety guarantees in sparse open-world Reinforcement Learning environments.
@inproceedings{martinezseras2024advancing,title={{Advancing Towards Safe Reinforcement Learning over Sparse Environments with Out-of-Distribution Observations: Detection and Adaptation Strategies}},author={Martínez-Seras, Aitor and Andrés, Alain and Del Ser, Javier},booktitle={International Joint Conference on Neural Networks (IJCNN)},pages={1--10},year={2024},doi={10.1109/ijcnn60899.2024.10650670},}
2023
IEEE Symposium Series on Computational Intelligence (SSCI)
Enhanced Generalization Through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewards
Alain Andrés, Daochen Zha, and Javier Del Ser
In IEEE Symposium Series on Computational Intelligence (SSCI), 2023
Exploration poses a fundamental challenge in Reinforcement Learning (RL) with sparse rewards, limiting an agent’s ability to learn optimal decision-making due to a lack of informative feedback signals. Self-Imitation Learning (self-IL) has emerged as a promising approach for exploration, leveraging a replay buffer to store and reproduce successful behaviors. However, traditional self-IL methods, which rely on high-return transitions and assume singleton environments, face challenges in generalization, especially in procedurally-generated (PCG) environments. Therefore, new self-IL methods have been proposed to rank which experiences to persist, but they replay transitions uniformly regardless of their significance, and do not address the diversity of the stored demonstrations. In this work, we propose tailored self-IL sampling strategies by prioritizing transitions in different ways and extending prioritization techniques to PCG environments. We also address diversity loss through modifications to counteract the impact of generalization requirements and bias introduced by prioritization techniques. Our experimental analysis, conducted over three PCG sparse reward environments, including MiniGrid and ProcGen, highlights the benefits of our proposed modifications, achieving a new state-of-the-art performance in the MiniGrid-MultiRoom-N12-S10 environment.
@inproceedings{andres2023enhanced,title={{Enhanced Generalization Through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewards}},author={Andrés, Alain and Zha, Daochen and Del Ser, Javier},booktitle={IEEE Symposium Series on Computational Intelligence (SSCI)},pages={1414--1420},year={2023},doi={10.1109/ssci52147.2023.10371796},}
Neural Computing and Applications
Collaborative Training of Heterogeneous Reinforcement Learning Agents in Environments with Sparse Rewards: What and When to Share?
Alain Andres, Esther Villar-Rodriguez, and Javier Del Ser
In the early stages of human life, babies develop their skills by exploring different scenarios motivated by their inherent satisfaction rather than by extrinsic rewards from the environment. This behavior, referred to as intrinsic motivation, has emerged as one solution to address the exploration challenge derived from reinforcement learning environments with sparse rewards. Diverse exploration approaches have been proposed to accelerate the learning process over single- and multi-agent problems with homogeneous agents. However, scarce studies have elaborated on collaborative learning frameworks between heterogeneous agents deployed into the same environment, but interacting with different instances of the latter without any prior knowledge. Beyond the heterogeneity, each agent’s characteristics grant access only to a subset of the full state space, which may hide different exploration strategies and optimal solutions. In this work we combine ideas from intrinsic motivation and transfer learning. Specifically, we focus on sharing parameters in actor-critic model architectures and on combining information obtained through intrinsic motivation with the aim of having a more efficient exploration and faster learning. We test our strategies through experiments performed over a modified ViZDooM’s My Way Home scenario, which is more challenging than its original version and allows evaluating the heterogeneity between agents. Our results reveal different ways in which a collaborative framework with little additional computational cost can outperform an independent learning process without knowledge sharing. Additionally, we depict the need for modulating correctly the importance between the extrinsic and intrinsic rewards to avoid undesired agent behaviors.
@article{andres2022collaborative,title={{Collaborative Training of Heterogeneous Reinforcement Learning Agents in Environments with Sparse Rewards: What and When to Share?}},author={Andres, Alain and Villar-Rodriguez, Esther and Del Ser, Javier},journal={Neural Computing and Applications},year={2023},pages={16753--16780},number={23},volume={35},doi={10.1007/s00521-022-07774-5},}
2022
International Cross-Domain Conference, CD-MAKE 2022
An Evaluation Study of Intrinsic Motivation Techniques Applied to Reinforcement Learning over Hard Exploration Environments
Alain Andres, Esther Villar-Rodriguez, and Javier Del Ser
In International Cross-Domain Conference, CD-MAKE 2022, 2022
In the last few years, the research activity around reinforcement learning tasks formulated over environments with sparse rewards has been especially notable. Among the numerous approaches proposed to deal with these hard exploration problems, intrinsic motivation mechanisms are arguably among the most studied alternatives to date. Advances reported in this area over time have tackled the exploration issue by proposing new algorithmic ideas to generate alternative mechanisms to measure the novelty. However, most efforts in this direction have overlooked the influence of different design choices and parameter settings that have also been introduced to improve the effect of the generated intrinsic bonus, forgetting the application of those choices to other intrinsic motivation techniques that may also benefit of them. Furthermore, some of those intrinsic methods are applied with different base reinforcement algorithms (e.g. PPO, IMPALA) and neural network architectures, being hard to fairly compare the provided results and the actual progress provided by each solution. The goal of this work is to stress on this crucial matter in reinforcement learning over hard exploration environments, exposing the variability and susceptibility of avant-garde intrinsic motivation techniques to diverse design factors. Ultimately, our experiments herein reported underscore the importance of a careful selection of these design aspects coupled with the exploration requirements of the environment and the task in question under the same setup, so that fair comparisons can be guaranteed.
@inproceedings{andres2022evaluation,title={{An Evaluation Study of Intrinsic Motivation Techniques Applied to Reinforcement Learning over Hard Exploration Environments}},author={Andres, Alain and Villar-Rodriguez, Esther and Del Ser, Javier},booktitle={International Cross-Domain Conference, CD-MAKE 2022},series={Lecture Notes in Computer Science},year={2022},pages={201--220},doi={10.1007/978-3-031-14463-9_13},}
IEEE Symposium Series on Computational Intelligence (SSCI)
Towards Improving Exploration in Self-Imitation Learning Using Intrinsic Motivation
Alain Andres, Esther Villar-Rodriguez, and Javier Del Ser
In IEEE Symposium Series on Computational Intelligence (SSCI), 2022
Reinforcement Learning has emerged as a strong alternative to solve optimization tasks efficiently. The use of these algorithms highly depends on the feedback signals provided by the environment in charge of informing about how good (or bad) the decisions made by the learned agent are. Unfortunately, in a broad range of problems the design of a good reward function is not trivial, so in such cases sparse reward signals are instead adopted. The lack of a dense reward function poses new challenges, mostly related to exploration. Imitation Learning has addressed those problems by leveraging demonstrations from experts. In the absence of an expert (and its subsequent demonstrations), an option is to prioritize well-suited exploration experiences collected by the agent in order to bootstrap its learning process with good exploration behaviors. However, this solution highly depends on the ability of the agent to discover such trajectories in the early stages of its learning process. To tackle this issue, we propose to combine imitation learning with intrinsic motivation, two of the most widely adopted techniques to address problems with sparse reward. In this work intrinsic motivation is used to encourage the agent to explore the environment based on its curiosity, whereas imitation learning allows repeating the most promising experiences to accelerate the learning process. This combination is shown to yield an improved performance and better generalization in procedurally-generated environments, outperforming previously reported self-imitation learning methods and achieving equal or better sample efficiency with respect to intrinsic motivation in isolation.
@inproceedings{andres2022improving,title={{Towards Improving Exploration in Self-Imitation Learning Using Intrinsic Motivation}},author={Andres, Alain and Villar-Rodriguez, Esther and Del Ser, Javier},booktitle={IEEE Symposium Series on Computational Intelligence (SSCI)},year={2022},pages={890--899},doi={10.1109/ssci51031.2022.10022199},}
IEEE Transactions on Evolutionary Computation
Adaptive Multi-Factorial Evolutionary Optimization for Multi-Task Reinforcement Learning
Aritz D. Martinez, Javier Del Ser, Eneko Osaba, and Francisco Herrera
IEEE Transactions on Evolutionary Computation, 2022
Evolutionary computation has largely exhibited its potential to complement conventional learning algorithms in a variety of machine learning tasks, especially those related to unsupervised (clustering) and supervised learning. It has not been until lately when the computational efficiency of evolutionary solvers has been put in prospective for training reinforcement learning models. However, most studies framed so far within this context have considered environments and tasks conceived in isolation, without any exchange of knowledge among related tasks. In this manuscript we present A-MFEA-RL, an adaptive version of the well-known MFEA algorithm whose search and inheritance operators are tailored for multitask reinforcement learning environments. Specifically, our approach includes crossover and inheritance mechanisms for refining the exchange of genetic material, which rely on the multilayered structure of modern deep-learning-based reinforcement learning models. In order to assess the performance of the proposed approach, we design an extensive experimental setup comprising multiple reinforcement learning environments of varying levels of complexity, over which the performance of A-MFEA-RL is compared to that furnished by alternative nonevolutionary multitask reinforcement learning approaches. As concluded from the discussion of the obtained results, A-MFEA-RL not only achieves competitive success rates over the simultaneously addressed tasks, but also fosters the exchange of knowledge among tasks that could be intuitively expected to keep a degree of synergistic relationship.
@article{martinez2021adaptive,title={{Adaptive Multi-Factorial Evolutionary Optimization for Multi-Task Reinforcement Learning}},author={Martinez, Aritz D. and Del Ser, Javier and Osaba, Eneko and Herrera, Francisco},journal={IEEE Transactions on Evolutionary Computation},volume={26},number={2},pages={233--247},year={2022},doi={10.1109/tevc.2021.3083362},}