Evaluation Framework for Ads Ranking

Framework to evaluate and optimize ads ranking using KPIs, A/B testing, ML models, attribution, and ethics.

Framework to evaluate and optimize ads ranking using KPIs, A/B testing, ML models, attribution, and ethics. Use this guide to structure the discussion, test the design under pressure, and practise explaining trade-offs clearly.

Problem and Scope

Design an evaluation framework for an ads-ranking system. It should help teams decide whether a ranking change improves user and advertiser outcomes without violating product, marketplace, privacy, or safety guardrails.

Start by confirming the core user journey, exclusions, success criteria, and the constraints that materially affect the architecture.

Requirements to Clarify

A strong answer should establish scope before choosing components.

Functional requirements:

Provide per-ad and per-segment KPIs (CTR, CVR, ROAS, Quality Score).
Support offline evaluation (batch) and online experiments (A/B, multi-armed bandits).
Produce dashboards and automated alerts for KPI regressions.

Non-functional requirements:

Scalable to billions of impressions/day.
Low-cost reproducible offline pipelines.
Privacy-preserving (compliant with GDPR/CCPA).
Explainable metrics for stakeholders.

Scale and Capacity

Use the workload to justify storage, partitioning, caching, and reliability decisions. Clarify or challenge these assumptions rather than treating them as unquestionable facts:

Impressions: 1B+/day (example at Meta scale).
Active ads: 10M+ creatives running concurrently.
Unique users: 100M+ daily active users.
Events: 10–100B raw events/day (impressions, clicks, conversions).
Storage: petabyte-scale event stores; aggregated telemetry in data warehouses.

Architecture Discussion

Walk through the important read and write paths, identify ownership boundaries, and explain how the design behaves when dependencies fail. Cover these areas explicitly:

Business objective, user value, advertiser value, and guardrail metric definition
Exposure, click, conversion, auction, and counterfactual logging quality
Offline ranking metrics, calibration, replay, bias, and slice analysis
A/B test design, randomization unit, power, duration, and interference
Long-term holdouts, novelty effects, network effects, and practical significance
Experiment governance, metric integrity, monitoring, and launch decisions

Follow-up Questions

Expect the interviewer to test the consequences of your choices. Practise answering these questions with a concrete decision, its benefit, and its cost:

What business objective dominates (revenue, ROAS, engagement, long-term retention)?
- Focus: Assess the candidate's answer to "What business objective dominates (revenue, ROAS, engagement, long-term retention)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Are we evaluating ad-level ranking or campaign-level allocation?
- Focus: Assess the candidate's answer to "Are we evaluating ad-level ranking or campaign-level allocation" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
What surfaces (newsfeed, stories, search, sidebar) and placements must be covered?
- Focus: Assess the candidate's answer to "What surfaces (newsfeed, stories, search, sidebar) and placements must be covered" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Do we have ground-truth conversions or use proxy signals (clicks, viewability)?
- Focus: Assess the candidate's answer to "Do we have ground-truth conversions or use proxy signals (clicks, viewability)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
What latency and privacy constraints exist (PII restrictions, differential privacy)?
- Focus: Assess the candidate's answer to "What latency and privacy constraints exist (PII restrictions, differential privacy)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Is offline historical data available with treatment labels and impression logs?
- Focus: Assess the candidate's answer to "Is offline historical data available with treatment labels and impression logs" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
How to evaluate long-term impact (e.g., user lifetime value) vs. short-term conversions?
- Focus: Assess the candidate's answer to "How to evaluate long-term impact (e.g., user lifetime value) vs. short-term conversions" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Should we integrate counterfactual policy learning (offline RL) into the evaluation pipeline?
- Focus: Assess the candidate's answer to "Should we integrate counterfactual policy learning (offline RL) into the evaluation pipeline" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
How to surface explainability for advertisers (why an ad ranked lower)?
- Focus: Assess the candidate's answer to "How to surface explainability for advertisers (why an ad ranked lower)" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
How to incorporate creative quality (NLP/image scoring) into ranking metrics?
- Focus: Assess the candidate's answer to "How to incorporate creative quality (NLP/image scoring) into ranking metrics" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Can we use multi-armed bandits for exploration while minimizing regret?
- Focus: Assess the candidate's answer to "Can we use multi-armed bandits for exploration while minimizing regret" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
What privacy-preserving ML techniques (federated learning, differential privacy) should be adopted?
- Focus: Assess the candidate's answer to "What privacy-preserving ML techniques (federated learning, differential privacy) should be adopted" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.

Evaluation Rubric

MockMe evaluates the answer across the following dimensions. A complete answer should connect claims to requirements and explain consequential trade-offs.

Requirements and scope (15%): Clarifies and prioritizes the required behavior for Provide per-ad and per-segment KPIs (CTR, CVR, ROAS, Quality Score); Support offline evaluation (batch) and online experiments (A/B, multi-armed bandits); Produce dashboards and automated alerts for KPI regressions. Establishes the constraints that materially affect Evaluation Framework for Ads Ranking, including Scalable to billions of impressions/day; Low-cost reproducible offline pipelines. Strong evidence includes Separates the critical path from secondary features and resolves ambiguous requirements before choosing components.
Architecture and interfaces (20%): Presents coherent ownership boundaries and end-to-end flows covering Business objective, user value, advertiser value, and guardrail metric definition; Exposure, click, conversion, auction, and counterfactual logging quality; Offline ranking metrics, calibration, replay, bias, and slice analysis; A/B test design, randomization unit, power, duration, and interference. Strong evidence includes Defines interfaces and traces important success, retry, and failure paths across the proposed components.
Data and scaling (25%): Uses workload assumptions such as Impressions: 1B+/day (example at Meta scale); Active ads: 10M+ creatives running concurrently to justify capacity and partitioning decisions. Explains the data, state, or model strategy for Offline ranking metrics, calibration, replay, bias, and slice analysis. Strong evidence includes Quantifies a dominant workload, identifies the first bottleneck, and explains how the design evolves as that workload grows.
Reliability, correctness, and safety (20%): Explains concrete failure behavior, recovery, and operational safeguards for Experiment governance, metric integrity, monitoring, and launch decisions; Privacy-preserving (compliant with GDPR/CCPA). Strong evidence includes States the required correctness or consistency boundary and covers retries, partial failure, observability, and safe degradation.
Communication and trade-offs (20%): Drives a structured discussion and compares consequential alternatives for A/B test design, randomization unit, power, duration, and interference; Long-term holdouts, novelty effects, network effects, and practical significance; Experiment governance, metric integrity, monitoring, and launch decisions. Strong evidence includes Makes assumptions explicit, answers the question asked, and explains both the benefit and cost of major decisions.