Running Whisper Locally in Browser with WebGPU: Complete Step-by-Step Architecture

Empirical Whisper Benchmarks Across In-Browser WebGPU Models

Tested on Chrome 128 (WebGPU) on Apple M3 Max (Metal Backend). 30-second conversational audio sample.

1. Off-Thread Audio Preprocessing with AudioWorklet

A common mistake in browser-based speech synthesis or recognition is running Fast Fourier Transforms (FFT) on the main UI thread, which causes noticeable stutter and dropped frames. In EdgeRuntimeHQ architectures, microphone audio is routed through an AudioWorkletProcessor:

• Resampling from the native input sample rate (e.g., 44.1kHz or 48kHz) down to 16,000 Hz.
• Framing raw audio into 25ms windows with 10ms stride (400-sample window size with 160-sample hop).
• Applying Hann windowing and extracting 80 log-Mel filterbank channels.
• Transferring typed arrays to WebGPU storage buffers using zero-copy ArrayBuffer transfers.

2. Complete In-Browser Whisper Implementation

The code below shows how to configure Transformers.js with WebGPU acceleration and streaming callbacks inside an Astro or modern frontend web application:

import { pipeline, env } from '@huggingface/transformers';

// Configure runtime to prioritize WebGPU WGSL compute
env.allowLocalModels = false;
env.backends.onnx.wasm.numThreads = 4;

export class InBrowserWhisperPipeline {
  private transcriber: any = null;

  async init(onProgress?: (progress: any) => void) {
    // Initialize speech-to-text pipeline with WebGPU execution
    this.transcriber = await pipeline(
      'automatic-speech-recognition',
      'onnx-community/whisper-tiny',
      {
        device: 'webgpu',
        dtype: 'fp32', // or fp16 if browser supports shader-f16
        progress_callback: onProgress
      }
    );
    return true;
  }

  async transcribeChunk(audioFloat32Array: Float32Array): Promise<string> {
    if (!this.transcriber) {
      throw new Error('Whisper pipeline not initialized');
    }

    const output = await this.transcriber(audioFloat32Array, {
      chunk_length_s: 30,
      stride_length_s: 5,
      language: 'english',
      task: 'transcribe',
      return_timestamps: false
    });

    return output.text.trim();
  }
}

3. Biometric Audio Privacy & Zero Cloud Egress

Voice data carries distinct biometric markers, acoustic environments, and potentially sensitive personal identifiers. Traditional cloud ASR APIs (such as OpenAI Whisper Cloud or Google Speech-to-Text) require transmitting raw PCM audio streams over WAN connections.

By performing Whisper inference locally via WebGPU: