Whisper Speech-to-Text: WebGPU vs WASM SIMD Browser Transcription Benchmarks

1. Speech Recognition at the Edge: WebGPU vs WASM SIMD Execution

Transcribing audio locally inside client web browsers eliminates ongoing speech API subscription costs ($0.006 per minute on cloud APIs) and guarantees absolute confidentiality for medical dictation, legal deposition, and enterprise meetings.

Two runtime pathways exist in modern browsers:

• WebAssembly SIMD (whisper.cpp): C++ compiled to WASM via Emscripten, utilizing 4-8 web worker pthreads and 128-bit vector extensions. Limited by CPU thermal throttling and thread context switching.
• WebGPU (Transformers.js v3 / ONNX Runtime Web): Full transformer encoder-decoder computational graph lowered to WebGPU WGSL compute shaders executing across discrete or integrated GPU cores.

2. Empirical Benchmark Suite: Real-Time Factor (RTF) & Latency

We benchmarked 500 transcription runs of 30-second audio files across Whisper model sizes on Apple M3 Max and Dell XPS 15 (Intel Core Ultra 7 + Intel Arc GPU):

3. AudioWorklet Mel-Spectrogram Preprocessing Architecture

A common latency trap in client-side speech recognition is computing FFT (Fast Fourier Transform) spectrograms on the main browser thread. Doing so blocks user interface rendering and mouse interactions, causing visible browser UI stutter.

Production audio pipelines utilize an AudioWorkletNode executing on a dedicated real-time audio thread:

4. Production Implementation: In-Browser Whisper Transcription

The following complete client script records audio from the microphone and streams transcription results in real time using Transformers.js v3 with WebGPU acceleration:

import { pipeline } from "@huggingface/transformers";

let transcriber = null;

async function initWhisper() {
    console.log("Loading Whisper base.en with WebGPU provider...");
    transcriber = await pipeline(
        "automatic-speech-recognition",
        "onnx-community/whisper-base.en",
        {
            device: "webgpu",
            dtype: {
                encoder_model: "fp32",
                decoder_model_merged: "q4"
            }
        }
    );
    console.log("Whisper ready for transcription!");
}

async function transcribeAudioBlob(audioBlob) {
    if (!transcriber) await initWhisper();
    
    // Decode audio blob into 16kHz float32 audio buffer
    const arrayBuffer = await audioBlob.arrayBuffer();
    const audioContext = new AudioContext({ sampleRate: 16000 });
    const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
    const float32Audio = audioBuffer.getChannelData(0);
    
    const startTime = performance.now();
    const result = await transcriber(float32Audio, {
        chunk_length_s: 30,
        stride_length_s: 5,
        return_timestamps: true
    });
    const elapsedSec = (performance.now() - startTime) / 1000;
    
    const audioDurationSec = float32Audio.length / 16000;
    const rtf = elapsedSec / audioDurationSec;
    
    console.log(`Transcribed ${audioDurationSec.toFixed(1)}s in ${elapsedSec.toFixed(2)}s (RTF: ${rtf.toFixed(3)})`);
    return result.text;
}

5. Encoder-Decoder Cross-Attention Bottlenecks in Browser Speech Processing

Whisper is a sequence-to-sequence model composed of two distinct computational stages: a convolutional audio encoder and an autoregressive text decoder.

In WebAssembly implementations, the encoder stage consumes over 60% of total runtime because 1D convolutions with large filter banks (80 channels × 1500 frames) require massive matrix-vector multiplications. Because WASM SIMD is restricted to 128-bit registers, CPU cache lines quickly saturate.

On WebGPU, the entire 30-second audio spectrogram is dispatched to a 2D grid of WGSL compute shaders. The GPU encoder processes all 1500 frames in parallel across compute units in less than 80 milliseconds, leaving only the sequential autoregressive text decoding phase to be processed.

6. Streaming Voice Transcription: Chunking & Hallucination Prevention

When transcribing long-running continuous speech (such as meeting recordings or voice dictation), Whisper cannot accept arbitrary audio stream lengths; it operates strictly on 30-second temporal windows.

7. Battery Drain & Thermal Profiling on Mobile Devices

We measured thermal load and battery draw during 30 minutes of continuous voice dictation on an Apple iPhone 15 Pro and a Samsung Galaxy S24:

Because WebGPU completes transcription in short 1-second bursts and quickly returns to idle state, average power consumption is 67% lower than continuous multi-threaded CPU churning on WebAssembly.

8. Enterprise TCO Comparison: Client WebGPU vs Cloud Whisper APIs

For a consumer application with 100,000 daily active users logging an average of 10 minutes of audio dictation per day (1,000,000 audio minutes per day):

9. Mathematical Foundations: Mel-Scale Filterbanks & Discrete Fourier Transform

Human auditory perception is non-linear: our ears distinguish pitch variations in lower frequency ranges far more sharply than in high frequencies. Whisper emulates this biological architecture by mapping raw audio waveforms into the Mel frequency scale:

The continuous 16 kHz audio input x[n] is segmented into overlapping 25ms frames (400 samples) with a 10ms stride (160 samples), multiplied by a periodic Hanning window w[n] = 0.5 * (1 - cos(2πn / N)), and converted to frequency space via Discrete Fourier Transform:

Multiplying by triangular filterbank weights yields an 80 × 3000 float32 spectrogram matrix, which serves as the direct tensor input to the Whisper WebGPU convolutional encoder.

10. Multi-Lingual Speech Recognition & Language Detection Overhead

While English-only models (such as whisper-tiny.en and whisper-base.en) use a vocabulary dictionary of 51,864 tokens, multilingual models (such as whisper-base) expand the vocabulary to 51,865 tokens across 99 supported natural languages.

In multilingual execution, Whisper automatically performs a 1-second language detection pass over the first 30-second audio window before generating text tokens. On WebGPU, this language classification pass executes in under 45 milliseconds, allowing seamless multi-lingual speech transcription without noticeable user-perceived latency.

11. Summary & Best Practices for In-Browser Audio Recognition