Committed to research
that brings advanced results to language technology
Committed to research
that brings advanced results to language technology
Showing 61-64 of 64 publications (page 7 of 7)
2011
Using Apertium linguistic data for tokenization to improve Moses SMT performance
Sergio Ortiz Rojas, Santiago Cortés Vaıllo, UMH Campus
+1 more
Edficio Quorum IIIThis paper describes a new method to tokenize texts, both to train a Moses SMT system and to be used during the translation process. The new method involves reusing the morphological analyser and part...
-of-speech tagger of the Apertium rule-based machine translation system to enrich the default tokenization used in Moses with part-of-speech-based truecasing, multi-word-unit chunking, number preprocessing and fixed translation patterns. Figures of the experimental results show an improvement of the final quality similar to the improvement attained by using minimumerror-rate training (MERT) as well as an increase of the overall consistency of the output.
Read more
2016
Prompsit’s submission to WMT 2018 Parallel Corpus Filtering shared task
Víctor M Sánchez-Cartagena, Marta Bañón, Sergio Ortiz Rojas
+1 more
Gema Ramírez-SánchezThis paper describes Prompsit Language Engineering’s submissions to the WMT 2018 parallel corpus filtering shared task. Our four submissions were based on an automatic classifier for identifying pairs...
of sentences that are mutual translations. A set of hand-crafted hard rules for discarding sentences with evident flaws were applied before the classifier. We explored different strategies for achieving a training corpus with diverse vocabulary and fluent sentences: language model scoring, an active-learning-inspired data selection algorithm and n-gram saturation. Our submissions were very competitive in comparison with other participants on the 100 million word training corpus.
Read more
2020
Bifixer and Bicleaner: two open-source tools to clean your parallel data
Marta Bañón and Sergio Ortiz Rojas Gema Ramírez-Sánchez, Jaume Zaragoza-Bernabeu
This paper shows the utility of two open-source tools designed for parallel data cleaning: Bifixer and Bicleaner. Already used to clean highly noisy parallel content from crawled multilingual websites...
, we evaluate their performance in a different scenario: cleaning publicly available corpora commonly used to train machine translation systems. We choose four English–Portuguese corpora which we plan to use internally to compute paraphrases at a later stage. We clean the four corpora using both tools, which are described in detail, and analyse the effect of some of the cleaning steps on them. We then compare machine translation training times and quality before and after cleaning these corpora, showing a positive impact particularly for the noisiest ones.
Read more
2022
Bicleaner AI: Bicleaner Goes Neural
Jaume Zaragoza-Bernabeu, Gema Ramírez‐Sánchez, Marta Bañón
+1 more
Sergio Ortiz-RojasThis paper describes the experiments carried out during the development of the latest version of Bicleaner, named Bicleaner AI, a tool that aims at detecting noisy sentences in parallel corpora. The t...
ool, which now implements a new neural classifier, uses state-of-the-art techniques based on pre-trained transformer-based language models fine-tuned on a binary classification task. After that, parallel corpus filtering is performed, discarding the sentences that have lower probability of being mutual translations. Our experiments, based on the training of neural machine translation (NMT) with corpora filtered using Bicleaner AI for two different scenarios, show significant improvements in translation quality compared to the previous version of the tool which implemented a classifier based on Extremely Randomized Trees.