There is a comfortable story we tell about fairness in AI. The systems are not yet accurate enough, the story goes, and once we measure the gaps and feed the models better data, the gaps will close. Accuracy and fairness are assumed to move together. Two failures in speech AI suggest that assumption is wrong, and they fail in ways that point past accuracy entirely. Speech recognition mishears dialect speakers at a rate that has been documented, published, and conceded by vendors for six years without closing. Captioning systems transcribe words but ignore almost everything else a soundtrack carries, and for deaf and hard-of-hearing (DHH) viewers that silence has barely been measured. One gap we can quantify precisely and have chosen not to fix. The other we cannot quantify, because there is no single right answer to quantify against. The real threshold for fair speech AI is not how accurate the system is. It is who gets to decide what the right answer was in the first place.
Begin with the dialect gap, because the evidence is unusually clean. In 2020, a Stanford team fed matched recordings of Black and white American speakers into the five leading commercial speech recognition services and found an average error rate of 35 percent for Black speakers against 19 percent for white speakers, almost exactly double. The decisive detail came from a subset of identical phrases spoken by both groups, where the disparity held. That rules out the convenient explanation of slang or unfamiliar vocabulary and points instead at the acoustic model1, the layer that turns sound into candidate words. As Koenecke and colleagues (2020) report, more than one in five samples from Black speakers had at least half their words garbled into unusable transcripts; for white speakers, fewer than one in fifty crossed that line.
The cause has never been a mystery. Most Black Americans, on the order of thirty-five to forty million people, speak African American Vernacular English2, yet training datasets these systems learn from severely underrepresent it. As Koenecke and colleagues (2020) document, in some widely used training datasets Black speakers comprised only a small fraction of the total. A model trained on a diet of standardized white American English will treat everything else as deviation, something to be corrected back toward a center it was never asked to question.
What makes this gap worth dwelling on is not that it exists but that it has survived being measured. Six years and several model generations later, it has migrated rather than disappeared. OpenAI’s own documentation for Whisper3 concedes that accuracy varies across accents and dialects, with higher error rates possible across race, gender, and age (OpenAI, 2022); Graham and Roll’s (2024) independent evaluation of Whisper confirmed this finding across controlled samples of diverse speakers. The instructive move is what the field has done in response. Rather than rebalance the training data, a common industry response has been to attach a text-correction tool to the output and have it smooth accent errors after they occur, treating the symptom rather than the cause. Measurement was supposed to create accountability. Here the industry named the disparity, quantified it, and absorbed it as a known limitation. Measuring the gap did not produce the will to close it. It produced a more efficient way to live with it.
That pattern of evidence absorbed without action assumes a problem that itself is legible enough to be measured. The second frontier inverts that assumption entirely. Words are only part of what a soundtrack carries. A door slamming offscreen, a phone going off, a score tilting toward dread before anything visibly happens: these are signals too, and for DHH viewers they are largely absent. Early work by Alonzo and colleagues (2022) on captioning and visualizing non-speech sounds4 in user-generated video established both the feasibility and the complexity of the challenge. May and colleagues (2024), analyzing captioning patterns across a large collection of popular YouTube videos, found that meaningful non-speech sound was captioned in roughly four percent of cases. For the billion-plus people worldwide living with some degree of hearing loss, that figure marks the difference between following a video and having to guess at it.
It is tempting to read this as the dialect gap at an earlier stage, simply underbuilt and awaiting its turn. A three-year National Science Foundation project now running at NJIT and NYU (Cartwright et al., 2025) is among the first serious efforts to build systems that identify and caption the non-speech sounds that matter. Investment like this matters, but the gap is stuck for a reason no amount of funding dissolves. The obstacle is not that the system has yet to be built. It is that no one can specify what the system should output. The dialect problem has a ground truth: a speaker says a word, and word error rate5 measures the distance from it. Non-speech captioning has no such anchor. As May and colleagues (2025) observe, current models try to describe every sound a person would hear, yet caption users do not want every sound described, and the right description depends on the scene and the viewer.
And the users do not agree among themselves, which rules out the obvious workaround of simply asking them. A survey of 168 DHH individuals (May et al., 2025) found preferences so varied, over which sounds belong in a caption and how to render them, that the researchers concluded a fixed best-practice guide would amount to a one-size-fits-none solution. One Deaf participant located the goal not in correctness but in control: the dream, she said, was to have choices. Kafle and Huenerfauth (2017) had already shown that DHH users’ satisfaction with captions bore little relation to error counts. The metric that disciplines the dialect problem here measures something close to the wrong thing.
So the two frontiers are not two points on a single trajectory. They are stuck in opposite ways, and the contrast is the argument. The dialect gap is a failure of will operating in conditions of near-total knowledge: the yardstick exists, the figures are damning, the remedy is understood, and still the gap holds. The non-speech gap is a failure of definition operating before will even becomes relevant: you cannot demand accuracy until you have decided whose answer counts as accurate, and that decision genuinely varies from person to person. The acts of exclusion differ in character, one rooted in data choices, the other embedded in the definition of the task itself, but both operate by the same logic: whoever builds the system decides in advance whose signal is worth processing. System designers cast dialect speakers as noise around a white-American-English mean. They defined non-speech sound out of the captioning task before it ever began.
The practical implications diverge accordingly, and they fall to three groups. For those building dialect systems, the task is unglamorous but well charted: assemble training data that represents the people who actually use the product, and publish error rates broken down by speaker group as a matter of routine rather than under pressure, since a gap no one is required to report is a gap no one is required to close. For those building non-speech captioning, the discipline runs the opposite way: resist the urge to manufacture a single right answer where none exists, and make adjustable output, rather than one canonical caption, the target. For the platforms that ship both at scale, the obligation is to stop mistaking the standard speaker and the spoken word for the natural boundary of the problem. Some have already begun: the NJIT-NYU project counts Adobe, Google/YouTube, and New York Public Radio among its partners. But that kind of commitment needs to become standard practice, not an exception noted in a press release.
Read together, the two frontiers retire a comfortable belief. The hard problem in speech AI was never raising accuracy, and accuracy was never going to deliver fairness on its own. The hard problem is deciding what counts as the right answer, and conceding that the authority to make that decision belongs, first and finally, to the people the system is built to serve.
Notes
1 Acoustic model: The component of a speech recognition system that maps raw audio into candidate speech sounds, the first stage in turning sound into text. Its accuracy depends heavily on how wide a range of voices appears in its training data.
2 African American Vernacular English (AAVE): A rule-governed dialect of English spoken by many Black Americans, with its own consistent grammar, pronunciation, and vocabulary. Linguists treat it as a fully systematic variety of English, not a deviation from a standard form.
3 Whisper: An automatic speech recognition model released by OpenAI in 2022 and widely used for transcription across many languages. Its public model card acknowledges uneven accuracy across accents, dialects, and speaker groups.
4 Non-speech sound: Meaningful audio other than spoken words, such as music, sound effects, ambient noise, or shifts in tone. In captioning, these cues carry information that a word-for-word transcript leaves out.
5 Word error rate (WER): The standard measure of speech recognition accuracy: the proportion of words a system transcribes incorrectly, counting substitutions, insertions, and deletions against a reference transcript. A lower rate means a more accurate transcription.
References
Alonzo, O., Shin, H.V. and Li, D. (2022) ‘Beyond Subtitles: Captioning and Visualizing Non-speech Sounds to Improve Accessibility of User-Generated Videos’, Proceedings of ASSETS ’22, ACM. Available at: https://research.adobe.com/publication/beyond-subtitles-captioning-and-visualizing-non-speech-sounds-to-improve-accessibility-of-user-generated-videos
Cartwright, M., Lee, S. and Fuentes, M. (2025) NJIT and NYU Researchers Advance AI Audio Captioning for Deaf and Hard-of-Hearing with NSF Grant. NJIT News, 29 July. Available at: https://news.njit.edu/njit-and-nyu-researchers-advance-ai-audio-captioning-deaf-and-hard-hearing-nsf-grant
Graham, C. and Roll, N. (2024) ‘Evaluating OpenAI’s Whisper ASR: Performance analysis across diverse accents and speaker traits’, JASA Express Letters, 4(2), 025206. Available at: https://doi.org/10.1121/10.0024876
Kafle, S. and Huenerfauth, M. (2017) ‘Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing’, Proceedings of ASSETS ’17, ACM, pp. 165–174. Available at: https://doi.org/10.1145/3132525.3132542
Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J.R., Jurafsky, D. and Goel, S. (2020) ‘Racial disparities in automated speech recognition’, Proceedings of the National Academy of Sciences, 117(14), pp. 7684-7689. Available at: https://www.pnas.org/doi/10.1073/pnas.1915768117
May, L., Ohshiro, K., Dang, K., Sridhar, S., Pai, J., Fuentes, M., Lee, S. and Cartwright, M. (2024) ‘Unspoken Sound: Identifying Trends in Non-Speech Audio Captioning on YouTube’, Proceedings of CHI ’24, ACM. Available at: https://doi.org/10.1145/3613904.3642162
May, L., Clemens, M., Dang, K., Ohshiro, K., Sridhar, S., Wee, P., Fuentes, M., Lee, S. and Cartwright, M. (2025) ‘“Choices? That’s the dream”: challenges and opportunities in non-speech information closed-captioning’, Frontiers in Computer Science, 7. Available at: https://doi.org/10.3389/fcomp.2025.1575176
OpenAI (2022) Whisper Model Card. OpenAI. Available at: https://github.com/openai/whisper/blob/main/model-card.md
