SynthDetoxM: Modern LLMs are Few-Shot Parallel Detoxification Data Annotators

Existing approaches to multilingual text detoxification are hampered by the scarcity of parallel multilingual datasets. In this work we introduce a pipeline for cross-lingual parallel detoxification data generation. We also introduce SynthDetoxM, a manually collected and synthetically generated multilingual parallel text detoxification dataset comprising 16,000 high-quality detoxification sentence pairs across German, French, Spanish and Russian. The data was sourced from different toxicity evaluation datasets and then rewritten with nine modern open-source LLMs in few-shot setting. Our experiments demonstrate that models trained on our data achieve superior performance to those trained on the human-annotated MultiParaDetox dataset even in data limited setting. Models, trained on SynthDetoxM outperform all evaluated LLMs in few-shot setting. We release our dataset and code to help further research in multilingual text detoxification.

This work introduces a pipeline for cross-lingual parallel detoxification data generation. The approach leverages large language models (LLMs) to create synthetic data, addressing the lack of parallel multilingual detoxification datasets. The methodology involves the following steps:

1. Data Collection: Toxic texts were manually collected from publicly available datasets in German, French, Spanish, and Russian. Only samples marked as toxic by human annotators were selected. Filtering based on Perspective API based Style Transfer Accuracy (STA) and LaBSE-based Similarity (SIM) metrics, was used to enhance data quality.
2. Parallel Data Generation: Several open-source LLMs were used in a few-shot setting to generate detoxified versions of the collected toxic texts. The models used were: Qwen 2.5 32B, Command-R 32B, Gemma 2 27B, Aya Expanse (32B and 8B), Mistral Small 22B, Mistral Nemo 12B, and Llama 3.1 (70B and 8B).
3. Few-Shot Example Mining: The best toxic/non-toxic pairs for few-shot prompting were selected by calculating a score based on STA and SIM metrics from a multilingual toxicity detection dataset. For French, 10 sentences were manually detoxified due to a lack of representation in the existing dataset.
4. Filtering and Ranking: Generated detoxifications were filtered using a refusal classification model, and a threshold-based non-detoxifiability metric. The remaining detoxifications were ranked by the product of their STA and SIM metrics, and the top-scoring examples were selected.
5. Dataset Composition: The final dataset, SynthDetoxM, consists of 16,000 parallel toxic/non-toxic text pairs across Spanish, German, Russian, and French (4,000 per language).
6. Evaluation: The quality of SynthDetoxM was evaluated by training sequence-to-sequence models (mT0-XL) on different folds of the dataset and comparing their performance to models trained on the human-annotated MultiParaDetox dataset. The evaluation used the metrics defined in the MultiParaDetox shared task:
• Style Transfer Accuracy (STA): Uses a multilingual XLM-R text classification model to measure toxicity reduction.
• Content Similarity (SIM): Calculates the cosine distance between LaBSE embeddings of the source and generated texts.
• Fluency (FL): Uses ChrF1 score (though limitations of this metric are discussed in Appendix B of the paper).
• Joint Score (J): Combines STA, SIM, and ChrF1 into a single score: \[\textbf{J} = \frac{1}{n}\sum\limits_{i=1}^{n}\textbf{STA}(y_i) \cdot \textbf{SIM}(x_i,y_i) \cdot \textbf{ChrF1}(x_i, y_i)\]
7. SBS evaluation: To further evaluate the applicability of the proposed dataset for training detoxification language models, a Side-by-side evaluation was carried out using GPT-4o as a judge.

The table below presents the Joint (J) scores from the automatic evaluation of different multilingual text detoxification approaches. The models were evaluated on Spanish, German, and Russian using the test set from MultiParaDetox.

Key findings: