Log Ordered Messages System Design

Problem and Scope

Design a system that accepts messages from producers and persists them such that the messages are logged in the exact order they were produced. The system should be durable, highly available, and scalable to large message volumes. Consider trade-offs between strict ordering, throughput, latency, and fault tolerance.

Start by confirming the core user journey, exclusions, success criteria, and the constraints that materially affect the architecture.

Requirements to Clarify

A strong answer should establish scope before choosing components.

Functional requirements:

• Accept messages from many producers and persist them.
• Preserve ordering guarantees as specified (global or per-key).
• Support read/query of logged messages.

Non-functional requirements:

• Durability: no logged message loss within retention window.
• Availability: system tolerates node failures.
• Scalability: handle spikes and growth in throughput.
• Performance: bounded write latency; acceptable read latency for queries.
• Security: authentication, authorization, encryption in transit and at rest.

Scale and Capacity

Use the workload to justify storage, partitioning, caching, and reliability decisions. Clarify or challenge these assumptions rather than treating them as unquestionable facts:

• Example baseline: 100k messages/sec, average message 1 KB → ~100 MB/s write throughput.
• Peak might be 2–3x baseline.
• Storage: 100 MB/s ≈ 8.6 TB/day. Factor retention policy to determine disk.
• Number of partitions/servers sized to handle throughput and IOPS. Each partition target ~10k–20k msgs/sec depending on hardware.

Architecture Discussion

Walk through the important read and write paths, identify ownership boundaries, and explain how the design behaves when dependencies fail. Cover these areas explicitly:

• Definition of global, partition, producer, or key-level ordering guarantees
• Partition ownership, sequencing, append APIs, acknowledgements, and producer identity
• Durable log layout, replication, quorum, batching, and fsync trade-offs
• Idempotency, duplicate handling, retries, and exactly-once claims
• Leader failure, recovery, re-election, and prevention of split-brain ordering
• Consumer offsets, retention, compaction, querying, and cross-partition merge limits

Follow-up Questions

Expect the interviewer to test the consequences of your choices. Practise answering these questions with a concrete decision, its benefit, and its cost:

• Are messages single-stream (global ordering) or per-key ordering (e.g., per user/session)?
  • Focus: Assess the candidate's answer to "Are messages single-stream (global ordering) or per-key ordering (e.g., per user/session)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• What is the expected write throughput (messages/sec) and size distribution?
  • Focus: Assess the candidate's answer to "What is the expected write throughput (messages/sec) and size distribution" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• Is ordering required across geographically distributed producers, or only per-region?
  • Focus: Assess the candidate's answer to "Is ordering required across geographically distributed producers, or only per-region" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• What are SLAs for write latency and durability (e.g., ack semantics)?
  • Focus: Assess the candidate's answer to "What are SLAs for write latency and durability (e.g., ack semantics)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• Are duplicates acceptable, or do we need exactly-once semantics?
  • Focus: Assess the candidate's answer to "Are duplicates acceptable, or do we need exactly-once semantics" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• How long must logs be retained and what query patterns are expected?
  • Focus: Assess the candidate's answer to "How long must logs be retained and what query patterns are expected" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• How to support global ordering across regions? (Use single-region write or a global sequencer — trade-offs.)
  • Focus: Assess the candidate's answer to "How to support global ordering across regions? (Use single-region write or a global sequencer — trade-offs.)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• How to provide low-latency reads while ensuring ordering? (Keep recent data in local fast store.)
  • Focus: Assess the candidate's answer to "How to provide low-latency reads while ensuring ordering? (Keep recent data in local fast store.)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• Support for exactly-once delivery to sinks (use transactions and idempotent sinks).
  • Focus: Assess the candidate's answer to "Support for exactly-once delivery to sinks (use transactions and idempotent sinks)." for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• How to handle schema evolution? (Use schema registry and versioning.)
  • Focus: Assess the candidate's answer to "How to handle schema evolution? (Use schema registry and versioning.)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
• Can we provide ordered queries across multiple keys? (Requires merge-join and time-based ordering; more complex.)
  • Focus: Assess the candidate's answer to "Can we provide ordered queries across multiple keys? (Requires merge-join and time-based ordering; more complex.)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.

Evaluation Rubric

MockMe evaluates the answer across the following dimensions. A complete answer should connect claims to requirements and explain consequential trade-offs.

• Requirements and scope (15%): Clarifies and prioritizes the required behavior for Accept messages from many producers and persist them; Preserve ordering guarantees as specified (global or per-key); Support read/query of logged messages. Establishes the constraints that materially affect Log Ordered Messages System Design, including Durability: no logged message loss within retention window; Availability: system tolerates node failures. Strong evidence includes Separates the critical path from secondary features and resolves ambiguous requirements before choosing components.
• Architecture and interfaces (20%): Presents coherent ownership boundaries and end-to-end flows covering Definition of global, partition, producer, or key-level ordering guarantees; Partition ownership, sequencing, append APIs, acknowledgements, and producer identity; Durable log layout, replication, quorum, batching, and fsync trade-offs; Idempotency, duplicate handling, retries, and exactly-once. Strong evidence includes Defines interfaces and traces important success, retry, and failure paths across the proposed components.
• Data and scaling (25%): Uses workload assumptions such as Example baseline: 100k messages/sec, average message 1 KB → ~100 MB/s write throughput; Peak might be 2–3x baseline to justify capacity and partitioning decisions. Explains the data, state, or model strategy for Definition of global, partition, producer, or key-level ordering guarantees; Partition ownership, sequencing, append APIs, acknowledgements, and producer identity; Consumer offsets, retention, compaction, querying, and cross-partition merge limits. Strong evidence includes Quantifies a dominant workload, identifies the first bottleneck, and explains how the design evolves as that workload grows.
• Reliability, correctness, and safety (20%): Explains concrete failure behavior, recovery, and operational safeguards for Durable log layout, replication, quorum, batching, and fsync trade-offs; Idempotency, duplicate handling, retries, and exactly-once claims; Leader failure, recovery, re-election, and prevention of split-brain ordering. Strong evidence includes States the required correctness or consistency boundary and covers retries, partial failure, observability, and safe degradation.
• Communication and trade-offs (20%): Drives a structured discussion and compares consequential alternatives for Idempotency, duplicate handling, retries, and exactly-once claims; Leader failure, recovery, re-election, and prevention of split-brain ordering; Consumer offsets, retention, compaction, querying, and cross-partition merge limits. Strong evidence includes Makes assumptions explicit, answers the question asked, and explains both the benefit and cost of major decisions.