APRÈN. Portal de la producció docent del professorat de la UPC

The vast collections within library networks can make it difficult for users to find books that match their interests. This project addresses this challenge by developing a prototype recommendation system (RS) for the library network of the Diputació de Barcelona, powered by Deep Reinforcement Learning (DRL). The system personalises book recommendations based on users' borrowing history and profiles, overcoming the lack of explicit ratings through implicit feedback mechanisms. The study addresses two key challenges commonly faced by recommender systems: providing personalised suggestions that match individual preferences, and ensuring diversity in recommendations to avoid repetition or overly popular content. The recommendation task is framed as a Markov Decision Process (MDP), selected after an extensive review of state-of-the-art reinforcement learning-based recommender systems (RL-based RS). By applying DRL, the system aims to optimise long-term user engagement by focusing on cumulative rewards rather than immediate results. The project involves extensive data analysis, DRL model design, and iterative experimentation to refine the system's performance. This work helps to advance the application of DRL in the context of library systems and demonstrates its potential to enhance personalised user experiences in recommendation tasks. Although the project highlights the potential of DRL in library RS, challenges such as the high-dimensional action space and the scarcity of interaction data from real library environments prevented the model from achieving its initial performance goals. These limitations emphasise the inherent complexity of implementing DRL in large-scale, sparsely populated environments.

Driven by the motivation to save human lives in a world with the increasing frequency of natural disasters and technological threats, this thesis explores the use of Multi-Agent Reinforcement Learning (MARL) to generate urban evacuation strategies. By using a simulated environment based on the intricate layout of Venice, Italy, citizens represented by individual independent agents learn to navigate and collaborate to reach safety in an efficient way. This work proposes to explore the effectiveness of different MARL techniques evaluated based on factors such as average evacuation time and robustness against dynamic capacity restrictions in the street network. The theoretical foundation and preliminary results provides insights into how effective MARL is compared to central RL and fixed heuristics for evacuations. It also discusses the limitations of the solutions and addresses some of the challenges related to a potential implementation.

Recently, Deep Reinforcement Learning (DRL) has found numerous applications in robotics control enabling robots to learn complex tasks and adapt to new environments, however, it is notorious for being extremely sensitive to hyperparameters and also suffers from the problem of low sample efficiency. Meanwhile, striking the right balance between exploring the environment and exploiting known strategies (known as the exploration-exploitation dilemma) poses a significant challenge for the application of DRL. In the field of robotics, symmetry plays a significant role in designing and controlling robotic systems. By holding the equivariant and invariant properties, the morphological symmetries in robots show great potential to alleviate the dilemma of exploration vs. exploitation dramatically and improve the sample efficiency significantly in the DRL learning process. Meanwhile, it can also greatly enhance the model's generalization ability since the symmetric architectures can allow robots to better generalize their learned policies to new, unseen scenarios. In order to exploit the above benefits, we extend this symmetric inductive bias into the Reinforcement Learning (RL) scheme by defining a concept of symmetric Markov Decision Process (MDP) in robot systems. In our project, we incorporate symmetric MDP in DRL pipeline in 2 ways: (1) imposing the symmetric constraints in the design of neural layers the design of neural layers to construct equivariant/invariant networks or (2) performing data augmentation based on the symmetric transformations to enlarge the streaming replay buffer or the fixed offline dataset during training. In our work, we evaluated the effectiveness of the two aforementioned 2 methods under both online and RL settings. We found that both of them can significantly enhance the performance of DRL models in locomotion control tasks. Under certain conditions, such as with a small training set in offline RL learning, combining the two can help the agent learn a superior control strategy. By incorporating the equivariant neural network into the Behavior Cloning algorithm, we got extremely effective policies for Cartpole control task, the policy converged to the optimal solution after the first epoch and remained behaving steadily in the subsequent training epochs. In TriFinger task, We found that when training with 10% to 30% of the original dataset with symmetric data augmentation, the results of the Implicit Q-Learning (IQL) algorithm can be improved largely. By using 30% of the data for data augmentation, we achieved the highest return value of 1353. For online RL, in the training results based on the Soft Actor-Critic algorithm in IsaacGym Cartpole environment, we got 40 more scores continuously in return at the last 100 training epochs with the symmetric data augmentation approach.

This thesis is part of the SIA-REMU project, which aims to optimize the management of medical emergency resources using artificial intelligence tools. The specific focus of this work is on employing Reinforcement Learning (RL) to optimize the selection of ambulances dispatched during emergencies. To achieve this, we first create a virtual environment that simulates real-world scenarios, where ambulances are distributed among various ambulance stations. Subsequently, different RL algorithms are trained to select the appropriate ambulance dispatch station, based on the position and priority level of the emergency. The algorithms take into account that once an ambulance is dispatched, it becomes unavailable for a certain period, necessitating careful decisions to not compromise the response to subsequent high-priority emergencies. In this study, we compare various RL algorithms, ranging from simpler approaches to state-of-the-art methods. The attempted algorithms include Q-Learning with linear function approximation, Deep Q-Learning, Double Deep Q-Learning, Soft Actor-Critic, and Safe Model-Based Policy Optimization. By implementing these algorithms, we observe distinct behaviors with different properties. Generally, these behaviors entail selecting ambulances from the closest stations for high-priority emergencies and choosing ambulances from larger stations for less urgent situations.

This master's thesis is focused on applying offline reinforcement learning techniques to the ambulance dispatch problem, which involves selecting the most appropriate ambulance to dispatch when an incident occurs. The research is part of the SIA-REMU project, conducted both in France and Switzerland. An incident has an associated priority level. Incidents of priority 0 are vital emergencies, incidents of priority 1 are non-vital emergencies, and incidents of priority 2 are non-emergencies. The primary objective of the presented work is to train reinforcement learning agents capable of prioritizing incidents appropriately when dispatching ambulances. Initially, a dataset of experiences was constructed using data provided by the Centre de régulation du Centre Hospitalier Universitaire Vaudois (CHUV), which contains valuable information about incidents, interventions, and resources. This dataset of experiences served as a static dataset for training reinforcement learning agents in an offline setting, without interacting with an environment. State-of-the-art offline reinforcement learning algorithms were employed to train the agents, and their hyperparameters were tuned performing a Random Search. To evaluate and test the trained agents, a virtual environment was implemented. Finally, the policies learned by the agents were analyzed to draw meaningful conclusions from the obtained results.

This thesis explores the field of multi-agent reinforcement learning (MARL). It presents a solution to tackle the problems single-agent methods face when they cannot solve a task because the action dimensionality is too high. As they train one network to output an action, if that action has too many dimensions, it needs to explore exponential possibilities to find the optimal one to execute at each state of the environment. For this reason, we propose dividing those actions among multiple agents, each controlling one or more dimensions. Thus, we can train several networks to output the sub-action dimensions each agent controls, each having to explore fewer combinations. With that solution, we have studied different MARL methods to demonstrate how they can solve those tasks that single-agent methods cannot. We have also studied which type of MARL method works better. We divide a single-agent task into multiple agents but have a shared reward function that we cannot usually decompose into each agent's reward function. Hence, different methods tackle the problem of learning each agent's policy with a shared reward that depends on the actions of all agents. In this thesis, we have focused on those learning to decompose that shared reward into the contributions of each agent. Thus, we have studied several MARL methods belonging to that type and demonstrated how they work better than those not decomposing the joint reward and those that train by ignoring the existence of the other agents, even though their actions affect the environment. Finally, we have focused on methods working with continuous actions because there is less research. Thanks to finding very few, we have developed a new one. We have developed that method using another as the basis because there were some points about it we thought we could improve by doing some modifications. Thus, we have also compared our new method with the others, obtaining the best results in most cases.

This is a B-mode project carried out under an employment contract with the company Capgemini. In the recent years, it has been observed that clients are increasingly drawn to use cloud resources due to their scalability, cost-effectiveness and accessibility. Most of them have migrated their environments there, or are in the process of doing so. Simultaneously, the field of machine learning (ML) has expanded rapidly, becoming a standard for many customer processes and applications. Although having huge advantages, it is a method that requires a large amount of data to work, which is not always available in all areas or from the customers themselves, and also one that can arise some privacy concerns, as traditionally the data is centralized in a certain location. In order to circumvent this drawbacks, a privacy-preserved method to train ML models on decentralized data that can be used is federated learning (FL). After a thorough analysis of the situation of the company's current customers and the possible new services that could be offered, it was considered that it would be very beneficial to have the knowledge of how to develop a system where FL can be applied in a productive environment in the cloud. Therefore, the objective of this project is to design and implement a federated architecture in the cloud where models can be trained in a federated way, determining its feasibility, and additionally, also create a package that will be the basis from which coworkers can learn about this field and when facing a similar project, take into account all the options to adapt as much as possible to the customer's situation. For the first, a multi-cloud federated system based on VMs and the framework NVFlare has been deployed. Multiple experiments have been conducted in it, such as validating that federated models achieve similar results as central ones or how a federated model achieves better results than models trained with the isolate data. Results suggesting a similar or even superior performance than with the centralized design. For the later, in the thesis a solid FL background is provided, with two additionally studies on, first, all the possible cloud services from the three most popular cloud providers that could be used to deploy a federated architecture and how they may interconnect, and second, all the existing federated frameworks, evaluating them on its possible productive use.

Political science is interested in understanding where political parties are positioned on several socioeconomic scales and how these evolve through time. A huge number of resources, mostly human, are spent on codifying the manifestos that the parties publish each election and therefore, following this idea, several automatic codification systems have been proposed but few use state-of-the-art techniques. This thesis addresses this challenge by providing a tool for the automatic classification of political messages to subsequently study the variation of parties' discourse across years and contexts. Concretely, this thesis has developed deep learning models based on state-of-the-art Transformers, capable of characterizing political statements and classifying them into the categories defined in the political party manifestos. With the objective of better capturing the relationships between different political statements on the same topic, we also propose a new technique for applying Data Augmentation in the field of Natural Language Processing named, Context Data Augmentation (CDA). CDA is a type of Data Augmentation where samples are generated by concatenating two or more sentences of the same category stated by a set of parties belonging to the same ideological axis. %Using data from party manifestos in Spain from 2000 to 2019 coming from the Comparative Manifesto Project, the use of deep learning techniques together with CDA grants us 1.4 points improvement in terms of accuracy and 1.5 points improvement in terms of F1 Score with respect to the base approach. Finally, our technique has demonstrated effective to characterize the Regional Party Manifestos and evaluate the Right to Left (RILE) position difference. Using data from party manifestos in Spain from 2000 to 2019 coming from the Comparative Manifesto Project, the use of deep learning techniques together with CDA grants us some points of improvement in terms of accuracy and F1 Score with respect to the base approach. Political messages, posted either on social media or traditional media, are starting to differ considerably from their corresponding political party manifestos, making it very difficult to position a party on an ideological scale. As a result, citizens can be easily confused by political messages and vote for political parties that might be contrary to their actual social and economic needs. A tool, such as the one we have presented, opens the door to analyzing potential discursive differences between parties across different media and social networks, as well as assessing the potential bias in media when talking about political parties.

Nowadays, e-commerce is facing the problem of information overload, where users are exposed to a vast amount of content, making it more difficult for users to effectively take quality decisions. The need for delivering the right items at the right moment to each user has resulted in making recommendation systems one of the hot topics in research and technology trends, where a good recommendation system might give a key advantage to an e-commerce over its competitors. Industry-leading companies such as Youtube or Amazon introduced the concept of large-scale recommendation systems, where the number of candidate items to consider for recommendation is enormous, and efficient techniques must be applied. The most common way to deal with large-scale recommendation systems nowadays is to build a retrieval model that retrieves a subset of relevant items for the user, and a ranking model that scores and ranks the set of retrieved items. Research on recommendation systems is continuously evolving, where new approaches produce state-of-the-art results, which especially happens thanks to the rise of deep learning. In this thesis, we describe classical and current approaches to recommendation systems, from content-based methods without assuming latent factors and collaborative-filtering methods like matrix factorization, to hybrid approaches, deep learning-based methods, and state-of-the-art approaches. Precisely, we focus on the concept of context-aware methods and multitask methods, which aim to optimize more than one task at a time. In this master thesis, we focus on developing a recommendation system for Stayforlong as a proof of concept. Firstly, we analyze their data and see that we can opt for a hybrid context-aware model since we have at our disposal user features, item features and context features. Another thing that characterizes the data set that we work with is the abundance of implicit feedback and the scarcity of explicit feedback. This drives us to experiment with different model architectures and approaches, focusing on developing a hybrid model that performs a retrieval task and another hybrid model that performs a ranking task. In addition, we check in our experiments the benefits of adding context to our models, the benefits of jointly training a model that optimizes multiple tasks, and the benefits of training a model on an abundant data set, like implicit feedback, and applying transfer learning to fine-tune on explicit feedback the learned representations. Results show that, in rich data scenarios, a context-aware multitask hybrid model trained on implicit feedback and fine-tuned with explicit feedback outperforms other approaches such as training separate retrieval and ranking models, disregarding implicit feedback or not including context features in the models. Finally, we propose as future work a data pipeline for the recommendation system to be used in production, taking into account data freshness and model re-training.

This thesis presents a study on Hierarchical Reinforcement Learning, where several different approaches for learning are researched, developed and tested. Specifically, the algorithm Reverse Curriculum Vicinity Learning (RCVL) resulted with an excellent performance in the tested environments. It is built on two level hierarchies, where the high hierarchy learns to suggest adequate subgoals to the lower level recursively, the latter learns the sequence of needed primitive actions to achieve the subgoals, and finally the ultimate goal. Currently it is designed only for discrete Reinforcement Learning environments. RCVL algorithm has shown outperformance over the State of the Art algorithms: DDQN and DDQN combined with HER in more complex tested environment. It has reached to a success rate of above 97%, while avoiding unfeasible subgoals suggestion, and constructing optimal paths. Finally, it has shown to be robust to most of hyperparameters changes. Hierarchical Learning allows to break a task into several smaller sub-tasks, which results with faster learning, since smaller tasks are easier to master. Each of the levels in the hierarchy has its own resolution (i.e. different time scales) of the problem, while the low policy is the one to interact with the environment exclusively. In our problem the subgoals proposed by the high policy can be seen as milestones that break the big task into several shorter tasks. The proposed algorithm also integrates the concept of Reverse Curriculum Learning. Its learning begins from states around the goal, and gradually expands to more difficult tasks from further states, until mastering the whole state space. With this curricular approach, the agent is able to learn faster: first it masters the easy tasks, and then challenged with harder tasks. In the proposed algorithm the high hierarchy stores neighbours from the vicinity of each goal (collected by low hierarchy interactions) such that the goal is reachable from them with a limited number of actions. In the meantime, the low policy learns simple actions to solve the mini-trajectories from the neighbours to the goal. Then with the accumulation of knowledge of both hierarchies, the high policy learns to draw a path from the goal to the state backwards recursively, suggesting the subgoals along the way. By long-term return estimation learning, the agent is able to decide which is the best subgoal for each given pair of state and goal (or subgoal). More concepts are integrated in the algorithm to accelerate the learning and to allow sample efficiency. First, It is an off-policy algorithm. Secondly, the reward system is designed to exploit the maximum information when rolling-out the collected experience so that all possible ordered combinations are stored with a non-sparse reward. the algorithm uses Hindsight Experience Relabelling, allowing exploitation of the accumulated experience in a more efficient way.

In this study, the potential of using Reinforcement Learning for Portfolio Optimization is investigated, considering the constraints set by the stock market, such as liquidity, slippage, and transaction costs. Five Deep Reinforcement Learning (DRL) agents are trained in two different environments to test the agents' ability to learn the best trading strategies to allocate assets, expecting to generate higher cumulative returns. All agents used are model-free and already optimized for financial problems, using the FinRL library. Therefore, the state-space has a high dimension, as found in the financial market environments. The two proposed environments use market data from US stocks, and one of them also uses Finsent data, an alternative data source that contains the news sentiment for all the stocks that are part of Dow Jones Industrial Average (DJIA). A series of backtesting experiments were performed from the beginning of 2019 to the beginning of 2020 and compared the two environments and how the agents performed against the DJIA. All the results were assessed with the pyfolio Python library, which uses all standard metrics to evaluate portfolio performance. Some algorithms increased the cumulative returns compared to the first dataset. The best result obtained outperformed DJIA by a significant amount and a smaller drawdown.

In this project, we want to investigate how Artificial Intelligence can help solve real-world problems, such as minimising the number of injuries and improving the technique of a beginner, and how this solution can be used by everyone. The aim is to develop software with Machine Learning techniques (such as Computer Vision) to detect the correct execution of physical exercises such as squat. We will carry out the process of creating and training a model (an algorithm based on automatic learning that has learned to solve a specific problem) to detect whether a squat is well done or not, and if not, the different kinds of errors. In order to create this model we also create a dataset, which is a set of examples that serve to help the model learn and which contains recorded videos of people doing correct and incorrect squats. Finally, we will test this model with other videos to find a model that has a high accuracy and therefore is a good solution. We will also create a REST API (Application Program Interface) that contains the obtained solution and all the necessary data preprocessing in order to be able to communicate with the model and make it accessible to any application that connects to the API. In the end, we will design and implement a small web/mobile application that connects to the API and process the response obtained to show it to the user in an understandable way. By making it human readable, we get the user to use the services offered by our artificial intelligence system. The application is designed to be accessible from any device (computer or mobile). This report details the whole process we have carried out and how we have designed, implemented and tested all the components of the service

Reinforcement Learning's unique way of learning has led to remarkable successes like Alpha Zero or Alpha Go, mastering the games of Chess and Go, and being able to beat the respective World Champions. Notwithstanding, Reinforcement Learning algorithms are underused in real-world applications compared to other techniques such as Supervised or Unsupervised learning. One of the most significant problems that could limit the applicability of Reinforcement Learning in other areas is its sample inefficiency, i.e., its need for vast amounts of interactions to obtain good behaviors. Off-policy algorithms, those that can learn a different policy than the one they use for exploration, take advantage of Replay Buffers to store the gathered knowledge and reuse it, already making a step into sample efficiency. However, in complex tasks, they still need lots of interactions with the environment to explore and learn good policies. This master thesis presents Learning Recursive Goal Proposal (LRGP), a new hierarchical algorithm based on two levels in which the higher one serves as a goal proposal for the lower one, which interacts with the environment following the proposed goals. The main idea of this novel method is to break a task into two parts, speeding and easing the learning process. In addition to this, LRGP implements a new reward system that takes advantage of non-sparse rewards to increase its sample efficiency by generating more transitions per episode, which are stored and reused thanks to Experience Replay. LRGP, which has the flexibility to be used with a wide variety of Reinforcement Learning algorithms in environments of different nature, obtains State-of-the-Art results both in performance and efficiency when compared to methods such as Double DQN or Soft Actor Critic (SAC) in Simple MiniGrid and Pendulum environments.

In this work, we proposed a new approach for learning legged locomotion for any legged robot, in the sagittal plane, by using a combination of classical control techniques and reinforcement learning. Specifically, we use optimal control of the low-order model Spring-Loaded Inverted Pendulum (SLIP), for the planning and generation of expert reference trajectories that resemble the ideal dynamics of animals in nature, and a control policy that learns to imitate these ideal dynamics. The objective of this approach is to provide a generic methodology for learning legged locomotion, that is flexible enough to be applied to robots with different morphological properties, and reduces the impact of simulation inaccuracies in the emergence of unnatural controlled gaits.

Meta-reinforcement learning has the potential to enable artificial agents to master new skills with improved sample-efficiency by leveraging previous learning experience in tasks that are diverse but share common structure. Our focus is to study the application of such algorithms to task distributions where the function that controls the dynamics of the environment is the main factor of variation. We start by providing an introductory background for related fields, including deep reinforcement learning, variational inference, and meta-learning. Then, we conduct a non-systematic review of the state-of-the-art algorithms for meta-reinforcement learning and perform an empirical investigation of PEARL, a method that combines soft actor-critic with latent task variables. Based on our review, we propose and implement two algorithmic modifications for PEARL: one that aims to improve the meta-training sample complexity by automatically adjusting a critical hyperparameter, and a second one focused on improving the meta-testing asymptotic performance by fine-tuning the policy during adaptation. Using a new multi-task environment suite for simulated robotics continuous control tasks, we compare the original version of PEARL and our proposed modifications, obtaining favourable results. Finally, we ponder our findings and suggest future research directions.

Applications of deep reinforcement learning in multi-agent systems are a rapidly developing scientific field that focuses on constructing computational frameworks and models for how various agents can interact efficiently to achieve their goals. In this thesis, we tackle the challenging coordination problem in which the agents cooperate to achieve a common task. First, we evaluate the robustness of the state-of-the-art algorithms: VDN and MADDPG in more complex scenarios of higher agent cardinality. We will analyze if the agent populations can discover various physical and informational coordination strategies since the models' variance increases proportionally to the number of the agents. To overcome the limitations derived from their low cooperation and their slow convergence, we introduce the MAGAT model. This new actor-critic model incorporates graph neural layers to encourage a better representation of the agent relations. The MADDPG algorithm demonstrates a poor performance on the initial scenario, motivating us to modify the current rewarding scheme. Although our model yielded some promising preliminary results, it seemed more challenging to fine-tune the self-attention architecture and requires further analysis. Designing optimal policies for coordination problems is a highly active and open research area. Through our insights, we hope that we will contribute to the evolution of the cooperation analysis, and our work will be the building block for future studies.

The autonomous driving research area has gained popularity over the past decade, even more with the launch of the first autonomous vehicle from Tesla, Inc. Different research branches are currently being studied, and one of the most innovative is the one in the direction of the Reinforcement Learning. However, Reinforcement Learning models do not ensure doing the safest decisions due to the unknown decision making process, making it impossible to apply these research lines in the real world, since being sure of the safety of the system is what allows autonomous driving to trespass the theoretical knowledge to practice. The aim of this project is to define a Reinforcement Learning model which ensures safety and allows to have control and awareness of the decision making process given possible unsafe situations, being trained and evaluated over a driving simulator named CARLA. The model architecture is composed of a Variational Autoencoder, in charge of reducing the dimensionality of the input images given by the simulator, a Mixture Density Recurrent Neural Network, which forecasts the most probable future state, and a Soft Actor-Critic who predicts the next action of the car agent based on past experience. Moreover, a security mask is applied to modify the actor's policy given a dangerous situation. This safety mask ensures a supervised behavior in this kind of situations providing Reinforcement-Learning- based autonomous driving systems of the security they were lacking to be applied in the real world. In addition, it has been analyzed if the agent would be able to learn the safety constraints provided by the safety mask, therefore learning to safely drive. The main contributions of this project start with proving the efficiency of using the Rein- forcement Learning Soft Actor-Critic algorithm in an autonomous driving task, which has never been done before. Additionally, several reward functions were defined which outper- forms the current state of the art. Moreover, this thesis also provides an exhaustive analysis of the relevance of forecasting in a self-driving task. To conclude, this thesis proves that using security masks in Reinforcement-Learning-based autonomous driving systems is, to the best of our knowledge, the best option to avoid the uncertain actions of Reinforcement Learning agents in unsafe situations. This fact could be the first step for promoting the application of Reinforcement Learning in the real world since it ensures safe behavior.