# 走样 Zouyang

> A Chinese ASR benchmark that scores errors by meaning, not by character
> distance. 31 speech-to-text models on 100 real Chinese recordings,
> judged against a 16-category rubric where a flipped negation costs
> 35 points and a filler word costs 1.

Chinese ASR leaderboards are usually English-first or recorded in studio
conditions, and WER (CER in Chinese) counts characters without asking whether
the meaning survived. "我喜欢" and "我不喜欢" are one character of edit distance
apart and mean opposite things. Zouyang scores that difference.

## Scoring

Every model transcribes the same 100 recordings (78.3 minutes of audio).
An LLM judge marks each transcript against a rubric of 16 error
categories, grouped into 4:

- meaning-flip (Meaning flips): polarity, semantic-inversion, hallucination, numeric
- entity (Entities): brand, tech, proper-noun
- cultural (Cultural): culture, dialect
- fidelity (Fidelity): mishear-severe, content-drop, paraphrase, mishear-minor, filler, format

The headline number is a fidelity score, not a raw deduction count:

    fidelity = 100 * (1 - deductions_per_100_chars / 20)

where 20 deductions per 100 ground-truth characters is the calibrated
unusability floor. Higher is better, 100 is a clean transcript.

Judge: deepseek_flash. Rubric: v6.8, v6.9. Last updated: 2026-09-23.

## Current board

Dedicated ASR systems, best first:

 1.  90.4  MAI-Transcribe-2  [mai_transcribe_2]
 2.  87.5  MiMo-V2.5-ASR  [mimo]
 3.  84.5  Qwen3-ASR-Flash  [flash]
 4.  84.4  Qwen-Audio-3.1-ASR  [qwen_audio_3_1_asr]
 5.  83.9  Scribe v2  [elevenlabs]
 6.  82.1  Doubao 2.0  [doubao]
 7.  81.9  Confucius4-R2T2  [r2t2]
 8.  81.0  Soniox stt-async-v5  [soniox]
 9.  80.9  GPT Transcribe  [gpt_transcribe]
10.  80.4  VibeVoice-ASR  [vibevoice]
11.  76.6  GPT-4o-mini-transcribe  [whisper]
12.  76.1  Qwen3-ASR-1.7B  [qwen3_oss]
13.  74.3  Gemini 3.5 Transcribe  [gemini_3_5_transcribe]
14.  73.1  GLM-ASR-Nano  [glm_asr_nano]
15.  71.3  Step-Audio 2.5 ASR  [stepfun]
16.  69.3  FireRedASR2-AED  [firered_aed]
17.  63.5  Fun-ASR-Nano  [funasr_nano]
18.  58.8  Whisper-Large-v3-turbo  [whisper_v3_turbo]
19.  48.6  Apple SpeechAnalyzer (macOS 26)  [apple_speechanalyzer]
20.  35.7  FireRedASR2-CTC  [firered]

General-purpose multimodal LLMs that take audio directly are scored and published
too, but kept off the board above, because latency and parameter count make a
direct comparison with purpose-built ASR unfair. They are listed separately on the
portal. Note that a vendor can appear in both places: MiMo-V2.5-ASR is a dedicated
ASR model and ranks on the board, while the general MiMo LLM is scored as
LLM-as-ASR. One product gets one row: extra experimental arms of the same model
are excluded rather than stacked.

## Data

- Portal: https://jizhang.io/articles/zouyang-bench/
- Full aggregate JSON: https://jizhang.io/articles/zouyang-bench/eval_aggregate.json

The JSON carries per-model scores, per-category and per-group deductions,
per-sample results, pricing and model metadata. Top-level keys worth knowing:
`models`, `per_sample`, `groups`, `pricing`, `model_meta`, `fidelity_floor`,
`_judges`, `_rubric_versions`, `generated_date`.

Not published: the audio, the reference transcripts, the per-sample key-point
lists, and the judge's written critiques. This benchmark is meant to be read,
not trained on.

## Attribution

Built by Ji Zhang. If you cite a number, please link the portal so readers can
see the rubric it came from.