How to evaluate prediction quality

This article gives a practical guide to deciding whether a Prediction is “good enough” to use.

If you are not familiar with Predictions, refer to our Predictions documentation to learn more.

It’s structured around three core steps:

• Data checks – Is there enough and the right kind of data for a model to learn from?

• Model metrics – Do the evaluation metrics in the Results tab show a strong, stable model?

• Reporting and business validation – Does the prediction actually improve campaigns and outcomes when you use it?

1. Data checks

Before looking at any metrics, confirm that your data setup makes sense. A model cannot learn patterns that are not present in the data.

Minimum data requirements

• At least 1,000 unique customers are involved in the behavior you’re predicting.

• At least 2 weeks of history and 10,000+ relevant events (purchases, opens, visits, and more).

• Among eligible customers, roughly 5–95% should hit the target in the training window; if almost nobody or everybody does, the model can’t learn useful patterns.

Time windows

For typical binomial classification predictions (all standard templates and “Custom – Binomial”):

• Target window:

  • When you expect the target behavior to happen (for example, “will purchase in the next 30 days”).

• Feature (training) window:

  • Which past behavior is used to predict the target (for example, “what they did in the 60 days before the 30‑day target window”).

Rules to follow:

• For future-behavior predictions, the feature window should end before the target window (for example, “last 60 days before next 30 days”) – this avoids data leakage.

• For “now” predictions (in‑session), feature and target windows can overlap on purpose (you are predicting something that happens during the current session or day).

If windows are misaligned, the model can appear accurate during training but perform poorly with real customers later.

2. Model metrics (Results tab)

Only binomial and multinomial classification predictions use these metrics properly. Pure regression predictions (for example, numeric CLTV) are evaluated using different error measures.

Note: For quality, focus on Test metrics and charts as they estimate performance on new customers.

AUC ROC and ROC Curve

What does it tell us:

If I randomly pick one customer who did the target action and one who did not, how often does the model assign a higher score to the customer who did it?

AUC ROC (single number)

• Number between 0 and 1.

• Interpretation:

  • 0.5 ≈ random guessing (no better than flipping a coin).

  • < 0.5: worse than random (it systematically ranks non‑buyers above buyers).

  • > 0.5: better than random; higher means better ranking.

Test vs Train AUC ROC

• They should be reasonably close.

• If Train AUC is very high (for example, 0.95) and Test AUC is much lower (for example, 0.65), the model is probably overfitting.

  • Overfitting means the model has essentially memorized the training customers instead of learning general patterns, so it looks great on the data it was trained on, but performs worse on “new” customers – that is, customers or time periods that were not used for training (the held‑out test set and future traffic).

ROC curve (visual)

• Green line = your model’s performance (Prediction preview).
• Red line = random selection baseline.
• A perfect model would go almost straight up the left side and then across the top. 
  • This is quite rare and could suggest a setup mistake rather than a genuinely perfect model.
• A random model tends to follow the diagonal (from bottom‑left to top‑right).

PR Curve and Area Under PR

What does it tell us:

Among the customers I select as “likely” (those with high prediction scores), how many actually do the target action (this is precision), and how many of all real target customers sit outside this selected group and are missed (this is recall).

PR curve

• Plots:

  • X‑axis: Recall – share of all buyers that you catch.

  • Y‑axis: Precision – of those you flag as likely, what share really buys?

• You want the PR curve clearly above the baseline across a useful range of recall - for example, for the top 10–30% of customers by score, precision should be significantly higher than the baseline event rate.

Area under PR

• Single‑number summary of how far the PR curve sits above the baseline precision.

• Example: Suppose you have a purchase prediction with 100,000 eligible customers and 5,000 target customers.
  • The event rate (baseline) is 5,000 / 100,000 = 5% = 0.05. This 0.05 is both:
    • the baseline precision (what random targeting gives you), and
    • roughly the baseline AUCPR (area under PR for a random model).
• If your Test AUCPR = 0.20, that’s 4× baseline (0.20 / 0.05), meaning that across thresholds the model concentrates buyers about four times better than random, which is usually a solid, usable model.

Red flags

• PR curve hugging the baseline → you’re hardly better than random at finding real positives among your “high probability” customers.

F1 Score and F1 Curve

What does it tell us:

If I pick a specific threshold, how good is the balance between catching positives and avoiding false alarms?

F1 score (single value at one threshold)

• Combines Precision and Recall into one number between 0 and 1:

  • High only if both precision and recall are high.

• Typically computed at a particular threshold – not across all thresholds.

F1 curve

• Shows how F1 changes as you move the decision threshold.

• Often peaks at the “best compromise” threshold.

What numbers to look for (for that threshold)

• ≈ 0.5+ – good balance; often a useful working threshold.
• ≈ 0.3–0.5 – borderline but can still be acceptable, especially for rare events.
• ≈ 0 or very low – at this threshold, the model is not capturing positives effectively.

Matthews Correlation Coefficient (MCC) Curve

What does it tell us:

If I look at all outcomes together – true positives (correctly predicted buyers), false positives (predicted as buyers but did not buy), true negatives (correctly predicted non‑buyers), and false negatives (missed buyers) - how well do the model’s predictions line up with reality overall?

MCC basics

• A single number between -1 and 1 that measures how correlated predictions are with reality.
• 1 = perfectly right.
• 0 = random.
•  –1 = perfectly wrong (everything flipped).

What to look for in the MCC curve

• Height of the peak (Y‑axis)
  • You want the maximum MCC value on the Test set to be clearly above 0.
  • As a rule of thumb, a peak around 0.3 or higher usually indicates a meaningful signal; values staying very close to 0 across thresholds mean the model is barely better than random.
• Where the peak is (X‑axis)
  • The X‑axis represents different decision thresholds, ranging from 0 to 100 on a 0–100 scale, from very strict on the left (almost nobody predicted positive) to very loose on the right (almost everyone predicted positive).
  • A useful model typically has a visible peak in the middle range, not only at the extremes. This tells you there is a reasonable threshold region where the model balances catching positives and avoiding false alarms well.

Lift curve

What does it tell us:

If I sort customers by prediction score and start targeting from the top, how much better than random am I at finding converters in each slice of the list?

How it works

1. Sort all customers by their prediction scores from highest to lowest.
2. Split them into equal‑sized buckets (for example, deciles: top 10%, next 10%, …).
3. For each bucket, compare:
   • Conversion rate in that bucket
   • Overall conversion rate in the whole Test population
4. Lift for a bucket =
   bucket conversion rate ÷ overall conversion rate
   • Lift = 3 → “This bucket converts 3× better than average”.

What can you see in the Test data lift chart:

• X‑axis (Customers %) – how big a top slice you take (from best 0–100% of customers).
• Y‑axis (Target %) – “of all buyers in the test set, what % have we already captured in this slice?”
• Green line = your model; red line = random selection.

Example - interpreting the screenshot above

In the highlighted point, the tooltip says:

• Customers picked: 1%
• Prediction preview: 40.69
• Random selection: 1

This means:

• If you take just the top 1% of customers by prediction score, you already capture about 40% of all buyers in the test set.
• If you picked 1% of customers at random, you’d capture only about 1% of buyers.

So the top 1% segment is converting at roughly 40× the random baseline, which is an extremely strong result and a very clear sign that the model is doing something useful.

Red flags

• Green line sits on top of (or very close to) the red line across the whole chart → top‑scored customers are not converting noticeably better than random.
• No sharp jump on the left side → even the first few % of customers don’t capture a clearly higher share of buyers.

Report

What does it tell us:

The Report view takes a chosen evaluation date and shows, for each score band, how many customers are in it and how many target events they generated in the historical target window associated with that prediction.

3. Reporting and business validation

Even a model with great metrics can fail in practice if used incorrectly. We recommend combining predictions with A/B tests and structured evaluation.

Four‑group framework (A, B, C, D)

For a prediction used in a campaign, divide customers into four groups based on:

1. Predicted probability (high vs low).

2. Campaign inclusion (in campaign vs in control group).

• Groups A and B have a high predicted probability.

• Groups C and D have low predicted probability.

• A and C receive the campaign; B and D are controls.

After the campaign, compute performance (for example, conversion rate) in each group and measure it as described in the Evaluation and Testing Predictions article.

Note: Perf is a placeholder for whatever performance metric you choose to evaluate the campaign, for example conversion rate, revenue per customer, average order value, etc.

Campaign‑level A/B tests

Beyond the four‑group framework, we recommend:

• Running A/B tests on:

  • The incentive or communication itself (does it work at all?).

  • Using vs not using the prediction to target segments (does the prediction add value?).

• Reading Prediction + A/B test results in dashboards/reports to ensure:

  • Uplift is higher in high‑probability segments than in low or random ones.

If offline metrics look good and online A/B tests show clear, prediction‑driven uplift, you can be very confident that the model is well‑trained and valuable.