SIMBig 2026 Shared Task

Data and Baselines

Datasets, baseline systems and evaluation protocol for Task 1 (Puno Quechua ASR)

Code, baselines and evaluation scripts: github.com/QuechuaBase/asr-puno-quechua (Apache-2.0)
Trained models: huggingface.co/QuechuaBase

Training data

Task 1 uses the Mozilla Common Voice datasets for Puno Quechua (ISO 639-3 qxp), released to participants on 24 August 2026. Both are available on Mozilla Data Collective under CC0 1.0.

Scripted speech

Common Voice Scripted Speech: Puno Quechua

Read sentences recorded and validated by volunteer speakers on the Common Voice platform.

Current release 27.0 (11 Sep 2026): 34.9 h total · 31.2 h validated · 25,423 clips · 81 speakers
Mozilla Data Collective

Spontaneous speech

Common Voice Spontaneous Speech: Puno Quechua

Unscripted answers to open questions, transcribed by the community.

Current release 5.0 (11 Sep 2026): 35.4 h total · 5.2 h validated · 7,305 clips · 110 speakers
Mozilla Data Collective

Train and dev splits were chosen by each team from the Common Voice data and are described in their system description papers.

Test set

All systems were evaluated on the same held-out test set of Puno Quechua speech, released (audio only) on 30 September 2026: 659 clips · 59 minutes · 7 speakers.

Type	Clips	Duration	Description
Scripted	438	25.7 min	Read-aloud sentences
Spontaneous	221	33.5 min	Elder speakers answering open questions
Total	659	59 min	7 speakers

Audio is 16 kHz, mono, 16-bit PCM WAV. The metadata file qxp_test_dataset.tsv gives each clip's file_name, speaker gender (female, male), age band (twenties, thirties, sixties = 60 or older) and type (scripted, spontaneous); these fields drive the leaderboard's breakdowns by gender and speech type.

Recording conditions: speakers recorded the clips themselves in a web browser on their own devices; rooms, microphones and background noise were not controlled.

Baselines

Baselines played two roles in the shared task.

1. Initial baselines (released at the start)

At the start of the task the organisers released a complete, reproducible pipeline and trained models, so that teams could begin from a working system: continued pre-training (CPT) of wav2vec 2.0 XLS-R on Puno Quechua audio, fine-tuning, and evaluation (pipeline.sh in the repository; models on Hugging Face). These results were reported in the repository on its own evaluation data, before the shared-task test set existed:

Model	Description	Scripted WER / CER (%)	Spontaneous WER / CER (%)
ft_cpt_validated	CPT checkpoint fine-tuned on validated (gold) data only	1.19 / 0.19	13.61 / 1.73
ft_cpt_silver	CPT checkpoint, plus automatically transcribed (silver) spontaneous speech	2.11 / 0.30	3.15 / 0.41
ft_xlsr_validated	Public XLS-R 300M fine-tuned on validated data only	2.06 / 0.30	13.58 / 1.71
ft_xlsr_silver	Public XLS-R 300M, plus silver spontaneous speech	4.36 / 0.57	6.68 / 0.81
omniASR LLM 300M v2	Omnilingual ASR, zero-shot	7.07 / 0.89	11.98 / 1.54
omniASR CTC 300M v2	Omnilingual ASR with CTC head, zero-shot	24.47 / 3.05	20.22 / 2.43

Source: QuechuaBase/asr-puno-quechua README. These numbers are not comparable with the leaderboard, which uses the new test set.

2. Baselines on the leaderboard (new test set)

For the leaderboard, the organisers took their models and omniASR models and evaluated them on the new shared-task test set, under the same conditions as the participants. They appear on the leaderboard as unranked reference rows marked Baseline.

Leaderboard name	Model	WER (%)	CER (%)	Code / model
omniLLM7B_v2	Generic Omnilingual ASR model by Meta (`omniASR_LLM_7B_v2`, 7B parameters), evaluated as is, without any Puno Quechua training	18.81	3.50	Code & model
xlsr-cpt-silver	XLS-R with continued pre-training (CPT) on Puno Quechua audio, fine-tuned on validated transcriptions plus automatically transcribed (silver) spontaneous speech	27.42	5.42	Code · Model
xlsr-cpt-validated	The same CPT checkpoint, fine-tuned only on validated (gold) transcriptions	30.53	6.27	Code · Model

Evaluation

Word error rate (WER) and character error rate (CER) are the main metrics; teams are ranked by WER. The leaderboard also reports:

CER variants ignoring diacritics and ignoring spaces;
chrF: character n-gram F-score (higher is better);
Levenshtein distance per utterance;
word-level substitution, deletion and insertion rates;
WER broken down by speaker gender and by scripted vs spontaneous speech.

Text normalisation. Before scoring, both the reference and the system output are normalised:

Unicode NFC;
lowercase;
punctuation removed, except the apostrophe ', which marks ejective consonants (k'anka ≠ kanka); ñ is also kept.

Submission format. A tab-separated file named qxp.tsv, with no header and one line per test audio file. The test set includes a qxp.tsv template listing every file; teams replaced the placeholder text with their system's output.

qxp_008067c9b0cb.wav	your predicted transcription here

Scoring script: in the eval/ folder of the baseline repository.

Task 2 data

Task 2 teams built and released their own datasets. Task 2 has been postponed due to a lack of entries, so no Task 2 datasets are listed here (see Task 2 outcome).

Citation and licence

Licences: the Common Voice datasets are released by Mozilla under CC0 1.0. Baseline code: Apache-2.0.

How to cite

If you use this data, the baseline models or the results in your research, please cite the following papers, together with the shared task overview:

Cordova, J., Wang, I., Zaruma, J., Chelbi, O., Malderez, A., & Huaman, E. (2026). Creating small ASR models for low-resource languages from similar languages: The cases of Betawi and Puno Quechua [Preprint]. HAL. https://hal.science/hal-05624964

@unpublished{CordovaWZCMH26,
  author = {Johanna Cordova and Ilaine Wang and Jocelyn Zaruma and Oumaya Chelbi and Alexis Malderez and Elwin Huaman},
  title  = {{Creating Small ASR Models For Low-resource Languages From Similar Languages: The Cases Of Betawi and Puno Quechua}},
  note   = {Preprint, HAL Id: hal-05624964},
  year   = {2026},
  month  = may,
  url    = {https://hal.science/hal-05624964}
}

Huaman, E., Gamarra Lafuente, A., Cordova, J., & Korhonen, A. (2026). Building community-centred NLP resources for Puno Quechua. In Proceedings of the Sixth Workshop on NLP for Indigenous Languages of the Americas (AmericasNLP) (pp. 147–152). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.americasnlp-6.13

@inproceedings{HuamanGCK2026,
  title     = {Building Community-Centred {NLP} Resources for {P}uno {Q}uechua},
  author    = {Huaman, Elwin and Gamarra Lafuente, Adrian and Cordova, Johanna and Korhonen, Anna},
  booktitle = {Proceedings of the Sixth Workshop on {NLP} for Indigenous Languages of the {A}mericas ({A}mericas{NLP})},
  month     = {jul},
  year      = {2026},
  address   = {San Diego, California, USA},
  publisher = {Association for Computational Linguistics},
  url       = {https://aclanthology.org/2026.americasnlp-6.13/},
  doi       = {10.18653/v1/2026.americasnlp-6.13},
  pages     = {147--152},
  isbn      = {979-8-89176-415-6}
}

Huaman, E., Huaman, J. L., Huaman, W., & Quispe, N. (2026). Quechua speech datasets in Common Voice: The case of Puno Quechua. In Information Management and Big Data (SIMBig 2025) (pp. 184–193). Communications in Computer and Information Science. Springer Nature Switzerland.

@inproceedings{HuamanHHQ2026,
  author    = {Elwin Huaman and Jorge Luis Huaman and Wendi Huaman and Ninfa Quispe},
  title     = {Quechua Speech Datasets in Common Voice: The Case of Puno Quechua},
  booktitle = {Proceedings of the Information Management and Big Data (SIMBig 2025), Lima, Peru, October 29-31, 2025},
  year      = {2026},
  publisher = {Springer Nature Switzerland},
  pages     = {184--193},
  isbn      = {978-3-032-20322-9},
  series    = {Communications in Computer and Information Science}
}

Download all references (.bib)

We thank the Puno Quechua speakers who contributed their voices, and Mozilla Common Voice for hosting the data.