Smart datasets for LLMs & NMT
Data collection, cleaning & normalisation
We gather text from public and client-provided sources, scrub duplicates and noise, and normalise encodings and formats. Then we tailor the corpus to your domain and preferred language or language flavour, including low-resource European languages where we hold unique expertise. The outcome is a gold-standard dataset that lets your LLM or NMT models train faster, cheaper, and with higher accuracy. Our data workflows are already being used in the OpenEuroLLM project, where Prompsit is in charge of collecting and preparing multilingual corpora.
Parallel document & segment alignment
Our algorithms precisely align sentence pairs across languages — taking into account context, markup, domain-specific terminology, and even structural mismatches. The result is a high-quality parallel corpus optimised for fine-tuning including terminology consistency, and reliable evaluation. This process is crucial for domain adaptation and especially valuable for low-resource language pairs — one of Prompsit's long-standing areas of expertise.
Dataset analysis & evaluation
Our specialised tools help you analyse big datasets gathering quantitative and qualitative insights. They evaluates key metrics such as the ratio of unique segments, the volume of potential personal identifiable data (PII), genre distribution, average and median sentence length, and over a dozen other parameters. Our reports highlight noisy areas, shows which parts should be cleaned or enriched, and helps you prioritise next steps — saving both time and budget on data preparation. This tool has already proven its reliability in the HPLT project, where Prompsit audited corpora containing billions of documents and segments — and it's ready to do the same for your data.
Enrichment & synthetic data generation
We are ready to enrich your data with valuable metadata and to act when data is scarce — especially in specialised domains or low-resource languages — by producing a synthetic data. Using carefully combined LLM + NMT ensembles, we annotate and generate data and then apply automated and rule-based filters to select only those that match the required style, terminology and domain coverage. This method allows you to enrich and scale your dataset rapidly without adding noise, making it suitable for training, evaluation, or internal benchmarking.
Alignment with EU regulations
We follow closely the requirements to adapt our datasets and models to current EU regulations — including data source documentation, PII identification and anonymisation, and auditability of dataset curation. We care so that your dataset and models stay both effective and legally deployable in the EU.
