Can Software Hear an Accent?

I have been learning Russian. Vocabulary I can grind. Grammar I can study. The part that humbles me every week is the accent. I say a word slowly, feel like I nailed it, and my tutor smiles and says "close." Close where? Which sound? By how much? A human ear knows instantly and cannot always explain it. So I started wondering whether software could tell me instead. Could a machine hear my accent the way my tutor can?

That is the question I wanted to test, so I wrote it down as a claim I could try to break.

Hypothesis: an ordinary speech engine can catch a single mispronounced sound as reliably as a trained ear can.

What "hearing an accent" even means

Before testing it, it helps to be clear about what we are asking the machine to hear, because an accent is not one thing. It is at least two.

The first is the individual sounds, the phonemes. Russian has a vowel, ы, that English simply does not, and English speakers reach for the nearest thing they know, и, the "ee" in "see." Swap one for the other and you have said a different word.

The second, and the one people underrate, is prosody: the melody, stress and rhythm of speech. Prosody is the tune you hum when you have forgotten the words. It is how "really" can mean agreement or sarcasm with the same three syllables. It is why a sentence with every individual sound correct can still land as foreign, because the rises and falls sit in the wrong places.

For humans, prosody does a lot of quiet work. It marks a question without a single word changing. It puts the emphasis that tells you which part of a sentence matters. It carries emotion, and it manages the back and forth of a conversation. Research on second language speech keeps landing on the same finding: getting the melody and rhythm right does more for being understood than getting every individual sound perfect. We forgive a foreign vowel. We stumble when the stress lands on the wrong syllable.

Which is exactly what makes prosody the interesting problem for software. A single sound is a short, local event you can line up against a reference and grade. Prosody is a shape spread across a whole phrase: pitch rising and falling, syllables stretching and compressing. To score it, a machine has to follow the tune of your voice over time and judge the whole contour, not a snapshot. Getting a machine to hear that shape, and then say something useful about it, is genuinely hard.

So the test has two halves. Can the machine hear a single wrong sound, and can it hear the melody around it.

How you test whether a machine can hear a sound

The tool for the first half is pronunciation assessment. You hand the engine a recording and the words you meant to say, and it scores how closely the sounds match, right down to each phoneme. I used Azure's pronunciation assessment, because it is one of the few that claims to support Russian, not just English.

The trap with any scorer is that it might simply be lenient. Feed it anything and it says "great, 95." So the test is built around minimal pairs. A minimal pair is two words that differ by exactly one sound. Russian мыл (washed) and мил (nice) differ only in that one vowel, ы against и. If the engine can really hear an accent, then a recording of мил, submitted as an attempt at мыл, should score badly, and badly because of that one vowel. If it waves it through, the whole idea is dead on arrival.

The Russian test

Here is the reference word мыл, said cleanly, next to мил, the exact slip an English speaker makes, scored against мыл. Press play on both and watch the number.

The correct vowel scores 100. The single wrong vowel drops it to under five. Same story with быть (to be) and бить (to hit), which again part ways on that one vowel.

Is it just failing everything?

A scorer that flags errors is only useful if it also passes good speech and rejects nonsense. So, two controls. Clean native recordings of ordinary words all score around 100, so the ceiling is where it should be. And when the audio does not match the claimed words at all, the score falls to zero rather than coasting in the eighties. It is discriminating, not merely harsh.

A single wrong vowel took the score from 100 to under 5. In a language the engine barely supports, it heard the accent.

English, and the melody

English is the engine's home turf, and it behaves a little differently there. Take sheep and ship, a pair split by one vowel, and think and sink, split by their first sound, the English "th" that a lot of languages lack.

Here the errors are caught but scored more gently. The engine hands out partial credit and points at the offending sound rather than collapsing to zero. Arguably that is the more honest behaviour: someone saying "ship" for "sheep" is wrong, but not unintelligible.

English also unlocks the second half of the hypothesis, the prosody. On English the engine returns a separate score for intonation and rhythm, the melody from earlier. This next clip is a whole sentence, and the number is not about its individual sounds at all. It is the machine judging the tune.

That score is the engine tracking the shape of a voice across a whole phrase and rating it, which is exactly the thing that is hard to teach and hard to measure.

Where this breaks down

The demo looks cleaner than the reality, so three honest caveats.

The voices here are synthesized, not human. Real learners make graded, halfway errors that should land in the middle of the scale, not at zero. The engine clearly has the sensitivity. How it behaves on messy human speech is the next experiment.

The engine will tell you a word is wrong, but it is coarse about which sound and by how much, and for some languages it does not label the sounds at all. Turning "this word scored low" into "your ы specifically drifted toward и" has to be reconstructed on top of it.

And prosody scoring is only offered for some accents. The melody is the hardest part to hear, and it is also the least evenly supported, which is a little ironic given how much of an accent it carries.

So, can software hear an accent?

On this evidence, more than I expected. It hears a single wrong vowel in a language it is barely trained on. It tells the difference between a real sound and a nonsense noise. And in English it hears the prosody, the melody and rhythm that carry so much of what makes an accent sound foreign in the first place.

What it does not do yet is coach. Hearing the error is the easy half. Telling a learner what to do with their tongue, and being kind about it, is the harder and more human half. But the part I assumed would be science fiction, a machine hearing the exact sound I got wrong, already works today. That is a good place to start.