Next-Word Prediction System Design

Problem and Scope

Design a next-word prediction system for a mobile keyboard. It should return useful suggestions within the typing latency budget, support multiple languages, learn safely from user behavior, and continue providing a reasonable experience when the network is unavailable.

Start by confirming the core user journey, exclusions, success criteria, and the constraints that materially affect the architecture.

Requirements to Clarify

A strong answer should establish scope before choosing components.

Functional requirements:

• Given recent typed tokens (context), return top-k next-word suggestions.
• Support real-time suggestions while typing on keyboard (low latency).
• Update model periodically with new data; allow personalization.
• Latency: 50–100ms round-trip for suggestion generation on server-assisted flow; <=5ms additional client processing.
• Accuracy: maximize top-1 accuracy; report top-3/top-5 as secondary.
• Availability: 99.9% service availability.
• Privacy: minimize sensitive data transmission; allow on-device inference and apply differential privacy for aggregated training if needed.
• Throughput: handle peak QPS consistent with user base (see scale estimates).

Non-functional requirements:

• Keep suggestions within the typing latency budget, with an on-device fallback when the network or service is unavailable
• Protect typed text through data minimization, explicit consent, retention controls, and privacy-preserving training
• Support language and device diversity without unacceptable model-size, battery, or memory cost
• Prevent offensive or sensitive suggestions and monitor quality regressions across important user slices
• Roll out model and vocabulary updates safely with measurable accuracy and product-impact guardrails

Scale and Capacity

Use the workload to justify storage, partitioning, caching, and reliability decisions. Clarify or challenge these assumptions rather than treating them as unquestionable facts:

• Active users: 10M daily active users (DAU).
• Average sessions per user: 5 per day.
• Average suggestion requests per session: 50.
• Requests/day = 10M * 5 * 50 = 2.5B requests/day.
• Requests/sec (avg) ≈ 2.5B / 86400 ≈ 28935 RPS.
• Peak factor 3 → target peak RPS ≈ 90k.
• Global training corpus: several TB (web + public corpora + anonymized keyboard logs).
• Model size examples: n-gram model compressed few hundred MB; Transformer baseline (e.g., 100M params) ~400MB FP32; quantized to 8-bit or 4-bit reduces size significantly.

Architecture Discussion

Walk through the important read and write paths, identify ownership boundaries, and explain how the design behaves when dependencies fail. Cover these areas explicitly:

• Product objective, top-k metrics, latency budget, privacy boundary, and supported languages
• Consent-aware data collection, cleaning, tokenization, labeling, and training pipelines
• N-gram, neural, and transformer model trade-offs under device constraints
• On-device, server-assisted, and hybrid serving with caching and degraded operation
• Personalization, cold start, federated learning, and user-level state
• Safety filtering, experiment design, model monitoring, drift, and rollback

Follow-up Questions

Expect the interviewer to test the consequences of your choices. Practise answering these questions with a concrete decision, its benefit, and its cost:

• Target platforms: mobile, desktop, or both?
  • Focus: Assess the candidate's answer to "Target platforms: mobile, desktop, or both" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• Supported languages and whether multilingual support is required.
  • Focus: Assess the candidate's answer to "Supported languages and whether multilingual support is required." for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• Max acceptable latency (ms) for suggestions to appear.
  • Focus: Assess the candidate's answer to "Max acceptable latency (ms) for suggestions to appear." for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• Memory/compute limits on client devices.
  • Focus: Assess the candidate's answer to "Memory/compute limits on client devices." for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• Privacy constraints (can typed text leave device? differential privacy required?).
  • Focus: Assess the candidate's answer to "Privacy constraints (can typed text leave device? differential privacy required?)." for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• Cold-start definition: no historical data per new user, but global corpora available?
  • Focus: Assess the candidate's answer to "Cold-start definition: no historical data per new user, but global corpora available" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• Success metric details: top-1 accuracy, top-k accuracy, or BLEU-like metrics?
  • Focus: Assess the candidate's answer to "Success metric details: top-1 accuracy, top-k accuracy, or BLEU-like metrics" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• How to personalize suggestions while preserving privacy? (federated learning, on-device fine-tuning, or per-user lightweight caches)
  • Focus: Assess the candidate's answer to "How to personalize suggestions while preserving privacy? (federated learning, on-device fine-tuning, or per-user lightweight caches)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• Support for multiple languages and code-switching?
  • Focus: Assess the candidate's answer to "Support for multiple languages and code-switching" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• How to handle offensive or sensitive suggestions? (filtering, safety classifiers, context-aware suppression)
  • Focus: Assess the candidate's answer to "How to handle offensive or sensitive suggestions? (filtering, safety classifiers, context-aware suppression)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• Can we provide multi-token completions or whole-phrase suggestions?
  • Focus: Assess the candidate's answer to "Can we provide multi-token completions or whole-phrase suggestions" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• How to integrate user feedback signals (accept/reject) into online learning?
  • Focus: Assess the candidate's answer to "How to integrate user feedback signals (accept/reject) into online learning" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• A/B test strategies to measure impact on typing speed and user satisfaction.
  • Focus: Assess the candidate's answer to "A/B test strategies to measure impact on typing speed and user satisfaction." for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.

Evaluation Rubric

MockMe evaluates the answer across the following dimensions. A complete answer should connect claims to requirements and explain consequential trade-offs.

• Requirements and scope (15%): Clarifies and prioritizes the required behavior for Given recent typed tokens (context), return top-k next-word suggestions; Support real-time suggestions while typing on keyboard (low latency); Update model periodically with new data; allow personalization. Establishes the constraints that materially affect Next-Word Prediction System Design, including Keep suggestions within the typing latency budget, with an on-device fallback when the network or service is unavailable; Protect typed text through data minimization, explicit consent, retention controls, and privacy-preserving training. Strong evidence includes Separates the critical path from secondary features and resolves ambiguous requirements before choosing components.
• Architecture and interfaces (20%): Presents coherent ownership boundaries and end-to-end flows covering Product objective, top-k metrics, latency budget, privacy boundary, and supported languages; Consent-aware data collection, cleaning, tokenization, labeling, and training pipelines; N-gram, neural, and transformer model trade-offs under device constraints; On-device, server-assisted, and hybrid. Strong evidence includes Defines interfaces and traces important success, retry, and failure paths across the proposed components.
• Data and scaling (25%): Uses workload assumptions such as Active users: 10M daily active users (DAU); Average sessions per user: 5 per day to justify capacity and partitioning decisions. Explains the data, state, or model strategy for Consent-aware data collection, cleaning, tokenization, labeling, and training pipelines; N-gram, neural, and transformer model trade-offs under device constraints; Safety filtering, experiment design, model monitoring, drift, and rollback. Strong evidence includes Quantifies a dominant workload, identifies the first bottleneck, and explains how the design evolves as that workload grows.
• Reliability, correctness, and safety (20%): Explains concrete failure behavior, recovery, and operational safeguards for Product objective, top-k metrics, latency budget, privacy boundary, and supported languages; Safety filtering, experiment design, model monitoring, drift, and rollback; Keep suggestions within the typing latency budget, with an on-device fallback when the network or service is unavailable. Strong evidence includes States the required correctness or consistency boundary and covers retries, partial failure, observability, and safe degradation.
• Communication and trade-offs (20%): Drives a structured discussion and compares consequential alternatives for On-device, server-assisted, and hybrid serving with caching and degraded operation; Personalization, cold start, federated learning, and user-level state; Safety filtering, experiment design, model monitoring, drift, and rollback. Strong evidence includes Makes assumptions explicit, answers the question asked, and explains both the benefit and cost of major decisions.