---
skill: ab-test-planner
title: Ab Test Planner
input: "Test whether moving the signup CTA above the fold on the pricing page increases free-trial starts. Current trial-start rate 8%, ~20k weekly visitors. We want to detect a 10% relative lift."
source: generated by claude-sonnet-4-6
---

# A/B Test Plan — Pricing Page CTA Position Test — 2025

---

## Hypothesis

> "We believe that **moving the signup CTA above the fold on the pricing page** will cause **free-trial starts** to **increase by ~10% relative (from 8.0% to 8.8%)** for **all visitors to the pricing page**, because users who arrive with purchase intent shouldn't have to scroll to find the primary action — reducing friction at the moment of peak motivation."

---

## Variants

| | Experience |
|---|---|
| **Control (A)** | Current pricing page — signup CTA positioned below the fold (existing layout) |
| **Treatment (B)** | Pricing page with signup CTA repositioned above the fold, visible without scrolling on desktop and mobile — no other changes to copy, color, or surrounding elements |

> ⚠️ **Isolation note:** Change only the CTA position. Do not alter button copy, color, pricing display, or plan layout in the same variant. If those elements are changed simultaneously, you cannot attribute results to position alone.

---

## Metrics

**Primary Metric:**
Free-trial start rate — defined as unique visitors who complete trial account creation divided by unique pricing page visitors, measured server-side at account creation confirmation.

**Guardrail Metrics:**
1. **Trial-to-paid conversion rate** — a position change that inflates low-intent sign-ups could depress downstream conversion. Must not degrade by more than 5%.
2. **Checkout abandonment rate** — ensures the CTA change doesn't create confusion or false-start behavior in the signup flow.

> Do not use clicks on the CTA as the primary metric. A click is not a trial start. Optimize for the outcome, not the behavior.

---

## Targeting & Traffic

| Parameter | Value |
|---|---|
| **Target Segment** | All pricing page visitors (new + returning) |
| **Traffic Split** | 50% Control / 50% Treatment |
| **Exclusions** | Existing paid subscribers (already converted — would dilute signal); any users currently in another pricing-page test |
| **Device scope** | Desktop and mobile (verify "above the fold" is correctly implemented per viewport — this is a common implementation error) |

---

## Sample Size & Duration

**Inputs:**
- Baseline trial-start rate: **8.0%**
- MDE: **10% relative lift → detect a move from 8.0% → 8.8%**
- Statistical power: **80%**
- Significance level: **95% (p < 0.05)**

**Required sample per variant: ~38,000 visitors**
**Total sample required: ~76,000 visitors**

**Duration calculation:**
```
Weekly visitors:     20,000
50% exposed:         10,000 per variant per week
Weeks required:      38,000 ÷ 10,000 = 3.8 weeks → round up to 4 weeks
```

**Recommended test duration: 4 weeks**

This captures two full weekday/weekend cycles and accounts for weekly traffic seasonality. Four weeks is also the outer boundary for novelty effects — do not extend beyond this. If you haven't reached significance at 4 weeks, the effect is likely smaller than your MDE and a different approach is warranted.

> 🔖 **Tool recommendation:** Validate this sample size at [Evan Miller's A/B test calculator](https://www.evanmiller.org/ab-testing/sample-size.html) using: Baseline = 8%, Absolute MDE = 0.8pp, Power = 80%, Alpha = 0.05.

---

## Launch & Monitoring Plan

**Pre-launch checklist:**
- [ ] QA variant on Chrome, Safari, Firefox — desktop and mobile viewports
- [ ] Confirm "above the fold" renders correctly at 1280px, 1440px, 375px (iPhone), 390px breakpoints
- [ ] Verify tracking fires correctly on trial creation (not on CTA click)
- [ ] Confirm no other pricing page tests are running
- [ ] Baseline metric snapshot taken 2 weeks prior for sanity check

**During the test:**
- Do not look at results until Week 4. Set a calendar reminder and lock the dashboard if your tool allows.
- Peeking at results before the planned endpoint and stopping early when you see significance is the single most common cause of false positives in A/B testing — it can inflate your false positive rate from 5% to over 25%.
- The only valid reason to check results early is to monitor guardrail metrics for a rollback trigger (below).

**Rollback trigger:**
> If trial-to-paid conversion rate in the Treatment group degrades by **more than 5% relative** at any monitoring checkpoint (check at end of Week 2 only), pause the test, investigate cohort quality, and escalate before proceeding.

---

## Results Interpretation Guide

Once the test reaches the planned 4-week endpoint and full sample size:

| Outcome | Condition | Action |
|---|---|---|
| ✅ **Ship it** | Treatment shows ≥10% relative lift on trial-start rate at 95% confidence **AND** guardrail metrics are stable (within ±5%) | Deploy to 100% of traffic |
| 🔄 **Iterate** | Treatment shows positive direction but does not reach significance | Do not ship. The effect may be real but smaller than your MDE. Consider redesigning the CTA element itself (copy, contrast, size) rather than just repositioning |
| ❌ **Reject** | No lift or negative direction at significance | Position is not the problem. Investigate pricing clarity, plan structure, or trust signals instead |
| ⚠️ **Inconclusive** | Test ended before reaching sample size, or results are within noise | Do not ship. Do not call it a win. Do not call it a loss. Rerun with a revised plan |

> **On "almost significant" results:** A p-value of 0.07 is not a win. It is not "trending positive." It means you do not have sufficient evidence. Shipping on p=0.07 results in roughly a 1-in-14 chance you're deploying a change that does nothing or causes harm.

---

## If This Test Doesn't Move the Needle

A null result on CTA position is genuinely useful signal. It tells you intent isn't the bottleneck — users aren't failing to find the button. Investigate:

1. **Pricing clarity** — are users confused about plan differences or what's included in the trial?
2. **Trust signals** — is there sufficient social proof near the decision point (testimonials, logos, review counts)?
3. **Trial friction** — how many fields does signup require? What happens immediately after clicking the CTA?
4. **Scroll depth analytics** — if >70% of users already scroll past the current CTA location, position was never the problem.
