Code-Switching Curricula Improve Cross-Lingual Alignment in Small LMs

What happened

Researchers ran controlled experiments on small-parameter decoder-class Transformer models, comparing two sets of 100 million-word English-Dutch-Chinese trilingual pre-training corpora: standard mixed corpora, and LLM-generated augmented corpora with word- and sentence-level code-switching. They found that models trained with a progressive curriculum — moving sequentially from word-level code-switching, to sentence-level code-switching, to monolingual text — outperformed baselines trained without code-switching on multilingual tasks, and retained cross-lingual text alignment gains even after subsequent monolingual training.

Key facts

Pre-training corpus size

100 million words per corpus, covering English, Dutch, and Chinese

Evaluation benchmark

BabyLM evaluation suite

Open-source released artifacts

Associated code, training data, and pre-trained model weights are publicly available on GitHub

Paper identifier

arXiv preprint 2609.30535v1

Background

Prior work found that cross-lingual alignment gains from multilingual pre-training often failed to persist through later monolingual training, particularly for languages with distinct writing systems, where alignment was historically costly and inconsistent in outcomes.

Why it matters

For developers, this curricular code-switching approach offers a low-cost multilingual data augmentation strategy that enables cross-lingual representation alignment without reliance on expensive parallel corpora, cutting training costs for small-parameter multilingual models. For end users, this may eventually lead to AI interactions with smoother cross-lingual understanding and more natural code-switching behavior. For the broader industry, the approach also provides a replicable training paradigm for adapting models to low-resource languages.

What to watch

Future research will likely explore the approach’s adaptability to larger models, more language settings, and real-world performance in production multilingual products.