APRÈN. Portal de la producció docent del professorat de la UPC

Neural Machine Translation models reproduce gender biases present in their training data, es- pecially when translating from gender-neutral languages like English to grammatical gendered languages like Spanish. This thesis investigates gender debiasing by adapting the state-of- the-art No Language Left Behind model into a gender-balanced and synthetically-generated domain. Our approach involves synthetic data augmentation using a Large Language Model (LLM) guided by a structured prompt, as it has strong benefits for scalable data augmenta- tion. We first identify the LLM and prompt combination that generates the most accurate and least biased counterfactual sentences. Experiments reveal that the Llama 2 - 13B model performs best when guided by a wider variety of examples, step-by-step reasoning, and uses the same language the model needs to complete the task. Using this setup, we augment a dataset with gender-modified sentences and then fine- tune the NLLB model using LoRA, a parameter-efficient approach that trains just 1.5% of the model's parameters. Our experiments show a statistically significant reduction in gender bias without compromising translation quality. Larger datasets combining both factual and synthetic counterfactual, and filtering out low-quality generated examples help generalize gendered language patterns more effectively. These results have broader implications: first, parameter-efficient fine-tuning can reduce gender biases at a smaller fraction of computa- tional cost; second, LLM-augmented datasets can match the effectiveness of other synthetic augmentation methods.

Cognitive distortions, also called cognitive errors or distorted thoughts, are biased interpretations and information processing errors that distort an individual's perception of their experiences. Cognitive restructuring therapy aims to address these distorted thoughts by generating alternative, more adaptive thoughts. However, there is a notable gap in the literature regarding the comparative analysis of distorted and alternative thoughts. This study aims to fill this gap by examining the structural and emotional patterns inherent in cognitive distortions, with a particular focus on the differences between distorted thoughts and their respective alternative thoughts. Utilizing Natural Language Processing (NLP) techniques such as n-grams, Part-Of-Speech tagging, dependency tagging, TF-IDF, Bag-of-Words (BoW), and cognitive network analysis, we investigate the emotional organization of concepts and individuals' cognitive frameworks. Our findings provide new insights into the nature of cognitive distortions and the effectiveness of cognitive restructuring techniques, enhancing our understanding of how distorted thoughts can be transformed into healthier, more balanced alternatives.

Machine Translation (MT), is one of the most important tasks related to natural language processing which facilitates seamless communication across different languages by automating the translation process, thereby significantly reducing communication costs. In this context, a multilingual neural machine translation (MNMT) system emerges as a cost-effective solution, capable of handling translations across multiple language pairs. An interesting property of MNMT systems is their ability to perform zero-shot translation, where the model translates between language pairs it has not explicitly seen before. This feature is particularly promising as it could potentially reduce the extensive resources and parameters typically required for MNMT models. Traditionally, MNMT models were thought to utilize a representation space independent of any specific language. This interlingua representation would allow the model to encode phrases conveying the same meaning similarly, regardless of the language, thereby aiding the translation process. However, in practice, it has been shown that this interlingua representation space is not as robust as previously assumed. In this project, we aim to study how MNMT representations still exhibit language dependent information and devise techniques to improve the model's representation. To achieve this goal, we propose some methods based on language projections.

The main objective of this thesis is to apply advanced process mining techniques to a real case study of an internal revenue recognition process for a type of service offered by HP Printing and Computing Solutions, S.L.U (which must remain confidential) to identify bottlenecks, teams that need to improve their performance to avoid delays either in revenue recognition, enabling service delivery in a timely manner, and improving forecasting regarding which quarter the delivery is planned for, thus facilitating financial recognition. All of this is achieved through the application of advanced process mining techniques using the Apromore analysis program and subsequently Python for automating the mentioned process. To achieve this, and based on the data shared by the company, a significant challenge exists in evaluating and acquiring data for process mining in a more efficient manner, observe patterns, trends, understand how different categories of services behave in these cases, and make better decisions when planning and executing them. Currently, this process is performed manually, which requires more time dedication as it involves merging different datasets from various systems into an Excel file, where exceptions are processed, and patterns are individually identified. This can also lead to manual errors in the process and require each verification to be manual. Therefore, through the development of this thesis, it will be possible not only to visualize the flow of the end-to-end process by analyzing the event logs but also to implement its automation, using Python as the main program, specifically the PM4PY library, while also ensuring the data's quality and appropriateness for process mining. This study's findings indicate that preprocessing techniques significantly affect process mining tasks'efficiency. Data cleaning needs vary based on event log characteristics, such as bulkiness, a broad range in trace set size, and activity duration changes. Hence, deploying automation could drastically cut down on work time. Moreover, as this involves a revenue recognition process, information can be accessible anytime within the month, not just at its conclusion. This facilitates the enactment of improvements or action plans by identifying opportunities and mapping out the flow.

Aquest estudi presenta SignScribe Net, un model innovador en el camp de la traducció de llenguatge de signes (SLT), integrant tècniques avançades de processament de llenguatge natural. La tesi aborda el desafiament de la comunicació inclusiva a través del desenvolupament d'una nova arquitectura SLT, combinant una Inflated 3D ConvNet amb capes convolucionals, adapters i un model transformer multilingüe. SignScribe Net demostra millores substancials sobre els models existents, amb la versió Gran aconseguint un augment del 50.06% en els resultats en BLEU i un increment del 8.57% en els resultats en COMET, destacant la seva eficàcia en traduir gestos complexos de llenguatge de signes mantenint la coherència semàntica. Malgrat el seu èxit, el model revela limitacions en la traducció zero-shot, indicant àrees per a recerques futures. La tesi contribueix significativament al camp de SLT, establint nous estàndards de precisió i inclusivitat en la comunicació per a les comunitats de sords i amb dificultats auditives.

This thesis evaluates gender bias in Machine Translation using Large Language Models. A comprehensive benchmarking process assessed various encoder-decoder NMT models and decoder-only base LLMs for English-to-Catalan and English-to-Spanish translations. Distinct test sets, including FLoRes, WinoMT, Gold BUG, and MuST-SHE, were employed to evaluate translation quality and gender bias, obtaining metrics such as BLEU, COMET, Gender Accuracy, F1-male, F1-female, ?G, and ?S. Results indicated the presence of gender bias across all models, with base LLMs exhibiting more bias than NMT models. To mitigate this bias, prompting engineering techniques were implemented in an instruction-tuned LLM. After multiple attempts, a prompt resulting in a significant reduction in gender bias across the test sets was identified, achieving remarkable gender scores. The prompt followed a simplified step-by-step approach with 5-shots on anti-stereotypical content and increased female representation. This prompt enabled the instructed LLM to perform competitively, achieving results comparable to NMT models, even surpassing some of them. However, a loss in translation quality using this prompt was observed.

Driven by the ambition to eliminate language barriers worldwide, Machine Translation has become a central area of interest in today's artificial intelligence research. Despite significant advancements, the concentration of research and resources has been predominantly on a highresource languages. This discrepancy in language coverage points out a critical gap in the field. Recent breakthroughs in Machine Translation have seen the emergence of Multilingual Large Pre-Trained models, which have set new benchmarks across the field by enabling low-resource languages benefit from zero-shot translation. However, these models obtain high performances at the cost of requiring huge amounts of data and hardware resources. The focus of this thesis is to explore and formulate a fine-tuning strategy for a multilingual machine translation model, such as M2M100. Specifically, the project aims to extend their linguistic capabilities by incorporating new low-resource languages by fine-tuning specific language adapters using Low-Rank Adaptation methods. To evaluate the performance of the strategy, state-of-the-art techniques and evaluation metrics are employed, considering factors like scalability, catastrophic forgetting and zero-shot translation. The implemented approach has successfully developed a M2M100 language translator in a low-resource context resulting in a SacreBLEU score of 5.6 when just training a 13 % of the parameters while the full fine-tuning methodology reach a score of 7.43. Furthermore, this framework demonstrates its capability to produce more advanced and efficient machine translation models, which can deliver high-quality translations with reduced computational demands.

In today's data-driven business landscape, the volume of information available and generated is unprecedented. The growing availability of data has sparked increased interest in process mining - a powerful approach for leveraging this data to gain valuable insights and make informed decisions. Process mining can reveal patterns, bottlenecks, and inefficiencies, enabling continuous improvement in business operations while also having the capability to train models for simulation and prediction. However, a significant challenge lies in assessing and obtaining suitable data for process mining, known as event logs, in a more efficient way. Currently, this task is performed manually, involving the laborious process of merging datasets and identifying relevant attributes through individual validation checks. This thesis addresses this challenge by focusing on the development of an automation tool, using Python and the PM4PY library, that accelerates the analysis of the raw data and the approach to transform it into event logs, while ensuring the quality and suitability of this data for process mining. To evaluate the performance of the automated approach, state-of-the-art techniques and evaluation metrics are employed, assessing accuracy, scalability, efficiency, and usefulness. The tool is evaluated using four datasets provided by participants who requested a process mining data assessment. The implemented approach has successfully developed an automated data preparation tool for process mining analysis. Its utilization has resulted in notable cost savings and a significant time reduction of up to 5.5 times faster than the manual approach. Moreover, the automated tool has proven to be highly effective in minimizing effort and errors, ensuring greater data accuracy and reliability.

Els treballadors mèdics han de fer front a grans quantitats de casos de pacients cada dia. L'ús de tècniques d'intel·ligència artificial com el resum automàtic pot ajudar a reduir la seva càrrega i millorar l'atenció al pacient. Aquest és el problema a resoldre proposat a la Shared Task darrere de la proposta de projecte. Aquesta tasca proporciona un conjunt de dades perquè els participants puguin entrenar les seves solucions i després provar-les en un conjunt de proves privat, que conté 768 notes de progrés diaris de l'hospital a partir de les quals es generen un resum de les diferents notes del pacient que enumera els diferents diagnòstics i problemes de salut. La proposta es basa en el fine-tuning del model pre-entrenat de llenguatge BERT, per al Named Entity Recognition per tal d'extreure els tokens més importants de les notes i combinar totes aquestes paraules clau per aconseguir una llista resum de problemes i diagnòstic. Abans del fine-tuning, les dades passen per una anàlisi exhaustiva i neteja per fer-les viable, i també, per adaptar el corpus a la classificació de tokens, cal anotar les dades. Els experiments se centren en l'impacte de diferents passos de preprocessament en els conjunts de dades, els canvis en l'entrenament del model utilitzat com a base per a l'ajustament, així com l'addició de tècniques de postprocessament i tècniques d'augment de dades.

In the context of advanced Natural Language Processing (NLP), this study delves into the interpretability domain, aiming to understand the logic behind models' decision making process. It focuses on three NLP tasks - Question Answering (QA), Text Summarization (TS), and Error Detection (ED). BERT and DistilBERT models were fine-tuned to address each task. The research introduces tailor-made datasets, placing special emphasis on the ED task for misspelling to enhance transparency. Additionally, novel faithfulness metrics are introduced to assess interpretability methods. A variety of gradient-based methods and an attention-based method, Aggregation of Layer-wise Token-to-token Interactions (ALTI), were evaluated. ALTI outperformed the gradient-based methods in all tasks albeit its higher computational cost. Integrated Gradients (IG) showed the highest variability, performing well in shorter ED sequences but less so in longer QA and TS sequences. The work stresses the need to broaden the task spectrum in interpretability evaluations and the significance of ensuring the robustness of emerging metrics.

This work provides a DNABERT-based solution inspired by NLP for the phage-bateria interaction (PBI) problem, as well as comprehensive literature review on the applied Deep Learning (DL) techniques. Intuitively, DNABERT is a novel approach to tackle DNA-based tasks built upon the popular BERT Transformer model. The report initially focuses on introducing and explaining core DL concepts that have been of paramount importance for developing the new ad-hoc models, providing also some background for understanding the current importance of Phage Therapy (PT) and how the PBI problem relates to PT. In brief, PT involves using phages to treat bacterial infections instead of antibiotics, and the focus of this work is to build a classifier based on DL capable of working with DNA information in text format from phages and bacteria, and of determining if a certain phage is likely to interact with (attack/neutralize) a certain bacterium. The first concept discussed is RNNs and, more concretely, LSTM networks. This type of neural network will be used as a starting point to reach the Transformer model by following the evolution of DL in the context of the NLP field. The attention mechanism and its implementation in the Transformer model are also discussed in detail. The document explains how this mechanism can be used to improve performance in various NLP tasks such as machine translation (MT), text summarization, and sentiment analysis, and, more importantly, how this mechanism can be also used to highlight the most relevant areas of a DNA sequence. This has been the main objective of the project, together with improving previous existing performance results of other neural models for the PBI problem. The motivation behind this research is also presented, along with important research questions that were explored and which guided the development, and there's an extensive section devoted to presenting the obtained results. Overall, this report aims to contribute to the field of Data Science by providing insights into how DL and NLP-oriented algorithms can be successfully applied to Genomics research for defeating great threats to human health such as superbugs. As a result of the developed work, 6 new DL models based on Transformers have been created, trained and evaluated, and the results have been satisfactory, achieving better results than previous neural models solving the same PBI problem. With the sake of encouraging future developments, at the end of the document there's a section devoted to pointing out possible promising future extensions of the work that has been carried out.

This project provides a spoken language translation system trained with UN Parallel Corpus and MuST-C, aiming at study the correlation between languages of different linguistic families and the performance of the translation tasks. This SLT system consists of a text-to-text Neural Machine Translation model, whose dataset includes six languages from five linguistic families, and a Automated Speech Recognition model, using dataset that contains four languages from four linguistic families. The combined SLT system is an end2end system, which is a relatively new task, and in this project, the idea is to analyze how would different linguistic families perform when training under the same conditions. Apart from measuring the performance using BLEU score system, this project also performs fine-tuning and zero-shot translation tasks. In general, the obtained BLEU scores are good and similar to original baseline models studies in UNPC and MuST-C papers. Finetuning and zero-shot translation experiments also obtained reasonable results, proving the hypothesized positive correlation between the closeness of languages and the performances of the translation tasks.

Multilingual Machine Translation has been approached from different perspectives including the shared and the language-specific encoders-decoders. The shared one uses a single encoder and decoder for all languages but the language-specific encoders-decoders allocate encoder and decoder for each language. Both perspectives have their benefits and drawbacks on translation quality and resource consumption aspect. To find a balance between these two factors, this project explores a new approach that is to share the encoders and decoders for language families. The new model was trained and tested on the TED2020 dataset with 21 chosen languages to form 4 language families. Comparison between the all-language shared baseline and our model shows a great improvement in BLEU score which can from 3 points to a maximum of 10 points according to the family pairs. The new model also has a good performance of zero-shot translation, which outperforms that of the baseline model and the improvement follows the rule of growth concluded from the model training.

En un món on l'Internet ens dóna accés a qualsevol tipus d'informació, encara hi ha limitacions quan la font d'aquesta informació es presenta en un altre idioma. Els traductors en línia són lluny de ser perfectes i és per això que la traducció automàtica entre idiomes és un tema de tendència en el camp de l'aprenentatge profund. L'objectiu d'aquest projecte és utilitzar l'arquitectura del Transformer, desenvolupada per Google al 2017, en el context de la traducció automàtica multilingüe. El model es centra en el concepte de l'atenció i està compost per un codificador i un decodificador basats en una representació intermèdia del llenguatge d'origen. Amb l'objectiu d'augmentar la BLEU, puntuació que defineix la qualitat de la traducció, i millorar la representació intermèdia comuna, hem introduït l'etiquetatge de categories gramaticals (Part-Of-Speech) al codificador del model. Hem realitzat experiments amb quatre idiomes (anglès, castellà, francès i alemany) tant en traducció automàtica com en Comprensió del Llenguatge Natural (CLN). Finalment, hem visualitzat la representació intermèdia i hem fet experiments per veure com les representacions de paraules en vectors codifiquen la informació de gènere. En comparar un model base sense etiquetatge Part-Of-Speech amb els nous codis etiquetats, la BLEU de traducció ha disminuït 0.50 punts de mitjana. En el cas de la CLN, les precisions també han disminuït un 8%, demostrant que els models etiquetats no milloren el rendiment d'aquestes tasques. No obstant això, la precisió de la classificació de gènere per a professions ha augmentat en un 1.1%.

Els sistemes de traducció automàtica neuronal presenten diferents tipologies de biaixos socials, com el racial i el de gènere, que s'associen principalment al baix present en les dades utilitzades per a l'entrenament d'aquests sistemes. Molts treballs en aquest àmbit tenen com a objectiu reduir aquests biaixos, especialment el de gènere, que perpetua estereotips i prejudicis. En aquest treball hem volgut mitigar el biaix de gènere mitjançant l'aprenentatge conjunt de la traducció, la categoria gramatical i el gènere de les paraules de l'idioma destí. Aquest plantejament ha donat bons resultats quant a la precisió del gènere, amb millores de fins a 6,8 punts respecte del model utilitzat com a referència i sense afectar significativament la qualitat de la traducció.

L'aprenentatge profund ha sorgit durant els darrers anys com una tècnica prometedora en IA. La implementació d'algoritmes que permeten els sistemes aprendre amb aplicacions en diversos camps ha marcat la diferència amb la codificació de regles convencionals que tenia múltiples limitacions. Atès que s'han dut a terme experiments amb èxit dins d'aquest àmbit, els límits del desenvolupament de màquines intel·ligents semblen haver esvaït. Tot i això, aquest no és encara un procediment impecable: hi ha un gran nombre de millores per assolir per obtenir millors resultats. Un d'aquests problemes és la presència d'estereotips en grans conjunts de dades, un problema generalitzat en moltes aplicacions de Deep Learning. En el cas de la traducció automàtica neuronal, aquest fenomen condueix a traduccions inexactes esbiaixades en quant a gènere. En aquesta tesi, el nostre compromís és treballar en el desenvolupament d'una modificació del mecanisme d'atenció (dins de l'arquitectura de NMT) que pugui mitigar aquest biaix. L'atenció controla la quantitat de context que s'utilitza a la traducció automàtica de neurones. La motivació d'aquest treball és que el mecanisme d'atenció de diferents tipus de paraules, com per exemple, les paraules amb diferents quantitats de dependències, haurien de ser diferents. La concentració o diversificació de la informació de context tant a nivells de codificació com de descodificació pot variar en funció dels biaixos i del llenguatge estudiats. Estudiem l'efecte de controlar aquesta quantitat d'atenció modificant el mecanisme d'atenció a nivell de codificació i descodificació, per a l'interès i la interatenció. Ho avaluem per al cas de biaixos mitigadors en un entorn multilingüe sobre una arquitectura NMT recent: el transformador. Els resultats mostren una qualitat de traducció similar a la del model de referència en termes de precisió, però supera el model de referència en gairebé tots els indicadors de biaix de gènere, donant resultats que són encoratjadors per continuar explorant aquesta línia d'investigació

En los últimos años, la industria de la inteligencia artificial ha aprovechado el poder de la computación conjuntamente con los modelos de aprendizaje profundo para construir aplicaciones de vanguardia. Algunas de estas aplicaciones, tales como asistentes personales o chat-bots, dependen en gran medida de Bases de Datos de Conocimiento, un depósito de datos sobre dominios específicos. Sin embargo, estas bases de datos no sólo deben ingerir hechos nuevos constantemente para actualizarse con la información más reciente sobre su dominio, sino que también tienen que recuperar el conocimiento ingerido de manera comprensible para la gente, en la mayoría de los casos. Centrado en este último punto, hacer que el conocimiento sea fácilmente accesible por los humanos, este trabajo se caracteriza por dar acceso automáticamente a este conocimiento mediante el lenguaje natural. Lo hacemos construyendo un modelo único capaz de extraer conocimientos dado unos enunciados en lenguaje natural, así como de generar texto dado un cierto conocimiento. La solución propuesta, una arquitectura basada en un Transformer eficiente, se entrena en un entorno semi-supervisado de múltiples tareas, siguiendo un régimen de entrenamiento cíclico. Nuestros resultados superan el estado del arte en la extracción de conocimiento para modelos sin supervisión, y también se alcanzan resultados satisfactorios para la tarea de generación de texto. El modelo resultante se puede entrenar fácilmente en cualquier nuevo dominio, con datos no paralelos, simplemente añadiendo texto y conocimientos al respecto, gracias a nuestro marco de entrenamiento cíclico. Además, este entorno semi-supervisado es útil para conseguir un aprendizaje permanente

L'objectiu d'aquest projecte és trobar les relacions entre factors d'estrès psicosocial i els atacs de cor que es van produir a Catalunya entre 2010 i 2016. Aquests factors es mesuren a través de les notícies publicades al compte de Twitter de La Vanguardia, processades amb tècnques de Machine Learning, com Word Embeddings i Clustering.