Denoising System for Audio

Design a system to denoise audio for real-time and batch applications, balancing latency, quality, and deployability.

Design a system to denoise audio for real-time and batch applications, balancing latency, quality, and deployability. Use this guide to structure the discussion, test the design under pressure, and practise explaining trade-offs clearly.

Problem and Scope

Design a system that removes noise from audio streams and audio files. The system should support both real-time low-latency use cases (e.g., video calls) and high-quality batch processing (e.g., post-processing of uploaded videos). Consider accuracy, latency, throughput, model deployment, monitoring, and upgrade paths.

Start by confirming the core user journey, exclusions, success criteria, and the constraints that materially affect the architecture.

Requirements to Clarify

A strong answer should establish scope before choosing components.

Functional requirements:

Real-time denoising API with low end-to-end latency.
Batch denoising API for higher-quality processing.
Support mono/stereo, sample rates up to 48 kHz.
Model update and A/B testing support.

Non-functional requirements:

Real-time latency: typical target < 50 ms added processing.
Throughput: scale to 100k+ concurrent real-time streams (example target).
Accuracy: measurable improvements in SNR/PESQ and human MOS.
Deployability: run on server GPU/CPU and optionally on-device/mobile.
Robustness: handle packet loss, jitter, and variable input levels.
Metrics for latency, error rates, quality metrics.
Logging and tools for subjective evaluation.

Scale and Capacity

Use the workload to justify storage, partitioning, caching, and reliability decisions. Clarify or challenge these assumptions rather than treating them as unquestionable facts:

Concurrent real-time streams: 100,000
Average throughput per stream: 16 kbps processed (voice), inference 10–50 ms per 20–40 ms frame
Batch jobs: 10,000 files/hour, average file 5 minutes
Model size: 5–200 MB depending on architecture
Storage: store models and metadata; audio storage depends on retention—if storing raw/before/after for 1% of traffic for quality eval, use tens of TB
Real-time: optimized CPU instances or small GPU/TPU pods with model quantization or DSP front-end to reduce CPU/GPU cost.
Batch: GPU-backed workers for high-quality models (Demucs/Conv-TasNet) for faster throughput.

Architecture Discussion

Walk through the important read and write paths, identify ownership boundaries, and explain how the design behaves when dependencies fail. Cover these areas explicitly:

Audio framing, buffering, jitter handling, and the real-time processing path
Separate low-latency streaming and high-quality batch architectures
Model selection, quantization, hardware placement, and capacity planning
Objective and subjective quality evaluation across representative noise conditions
Model registry, rollout, rollback, drift monitoring, and A/B testing
Privacy, retention, observability, and graceful degradation

Follow-up Questions

Expect the interviewer to test the consequences of your choices. Practise answering these questions with a concrete decision, its benefit, and its cost:

What are the target applications? (real-time conferencing, live streaming, post-processing)
- Focus: Assess the candidate's answer to "What are the target applications? (real-time conferencing, live streaming, post-processing)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
What noise types must be handled? (stationary: hum, hiss; non-stationary: traffic, music, other speakers)
- Focus: Assess the candidate's answer to "What noise types must be handled? (stationary: hum, hiss; non-stationary: traffic, music, other speakers)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Latency SLO for real-time path (e.g., <20 ms, <100 ms)?
- Focus: Assess the candidate's answer to "Latency SLO for real-time path (e.g., <20 ms, <100 ms)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Throughput targets: concurrent streams, batch jobs/hour?
- Focus: Assess the candidate's answer to "Throughput targets: concurrent streams, batch jobs/hour" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Allowed compute: CPU-only edge, mobile, GPUs in cloud?
- Focus: Assess the candidate's answer to "Allowed compute: CPU-only edge, mobile, GPUs in cloud" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Success metrics: objective (SNR, PESQ) and subjective (MOS) thresholds?
- Focus: Assess the candidate's answer to "Success metrics: objective (SNR, PESQ) and subjective (MOS) thresholds" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Privacy/regulatory constraints on sending raw audio to cloud?
- Focus: Assess the candidate's answer to "Privacy/regulatory constraints on sending raw audio to cloud" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Personalization: adapt models to speaker voice over time.
- Focus: Assess the candidate's answer to "Personalization: adapt models to speaker voice over time." for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Multi-microphone support: beamforming + denoising.
- Focus: Assess the candidate's answer to "Multi-microphone support: beamforming + denoising." for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Dereverberation and separation of multiple speakers.
- Focus: Assess the candidate's answer to "Dereverberation and separation of multiple speakers." for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
On-device training or federated learning for privacy-preserving improvements.
- Focus: Assess the candidate's answer to "On-device training or federated learning for privacy-preserving improvements." for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Support for music and complex acoustic scenes.
- Focus: Assess the candidate's answer to "Support for music and complex acoustic scenes." for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Cost model: estimate per-hour GPU costs vs. CPU inference cost per stream.
- Focus: Assess the candidate's answer to "Cost model: estimate per-hour GPU costs vs. CPU inference cost per stream." for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.

Evaluation Rubric

MockMe evaluates the answer across the following dimensions. A complete answer should connect claims to requirements and explain consequential trade-offs.

Requirements and scope (15%): Clarifies and prioritizes the required behavior for Real-time denoising API with low end-to-end latency; Batch denoising API for higher-quality processing; Support mono/stereo, sample rates up to 48 kHz. Establishes the constraints that materially affect Denoising System for Audio, including Real-time latency: typical target < 50 ms added processing; Throughput: scale to 100k+ concurrent real-time streams (example target). Strong evidence includes Separates the critical path from secondary features and resolves ambiguous requirements before choosing components.
Architecture and interfaces (20%): Presents coherent ownership boundaries and end-to-end flows covering Audio framing, buffering, jitter handling, and the real-time processing path; Separate low-latency streaming and high-quality batch architectures; Model selection, quantization, hardware placement, and capacity planning; Objective and subjective quality evaluation across representative noise. Strong evidence includes Defines interfaces and traces important success, retry, and failure paths across the proposed components.
Data and scaling (25%): Uses workload assumptions such as Concurrent real-time streams: 100,000; Average throughput per stream: 16 kbps processed (voice), inference 10–50 ms per 20–40 ms frame to justify capacity and partitioning decisions. Explains the data, state, or model strategy for Separate low-latency streaming and high-quality batch architectures; Model selection, quantization, hardware placement, and capacity planning; Model registry, rollout, rollback, drift monitoring, and A/B testing. Strong evidence includes Quantifies a dominant workload, identifies the first bottleneck, and explains how the design evolves as that workload grows.
Reliability, correctness, and safety (20%): Explains concrete failure behavior, recovery, and operational safeguards for Model registry, rollout, rollback, drift monitoring, and A/B testing; Privacy, retention, observability, and graceful degradation. Strong evidence includes States the required correctness or consistency boundary and covers retries, partial failure, observability, and safe degradation.
Communication and trade-offs (20%): Drives a structured discussion and compares consequential alternatives for Objective and subjective quality evaluation across representative noise conditions; Model registry, rollout, rollback, drift monitoring, and A/B testing; Privacy, retention, observability, and graceful degradation. Strong evidence includes Makes assumptions explicit, answers the question asked, and explains both the benefit and cost of major decisions.