Hypothesis Testing: t-test & p-value

In analytics, we often ask: “Is this change real or just random?”
Hypothesis Testing is the formal method to answer this, and tools like the t-test and p-value help us decide.

What is Hypothesis Testing?

Hypothesis testing is a statistical method to make decisions using data. It starts with two statements:

• Null Hypothesis (H₀): No difference, no effect, or status quo
• Alternative Hypothesis (H₁ or Hₐ): There is a difference, or some effect exists

We collect sample data and use statistical tests to determine whether to reject the null hypothesis.

What is a t-test?

A t-test checks if the means of two groups are significantly different.

Types of t-tests:

• One-sample t-test: Compare sample mean to a known population mean
• Two-sample t-test (independent t-test): Compare means of two independent groups
• Paired t-test: Compare means from the same group at two different times

Example:

Suppose NextLeap launches a new course and wants to test if students spend more time on the new course than the previous one.

• H₀: There is no difference in average time spent
• H₁: There is a difference
  A two-sample t-test can help test this.

What is a p-value?

The p-value tells us how likely our sample data would occur if the null hypothesis were true.

• Low p-value (≤ 0.05): Strong evidence against the null → Reject H₀
• High p-value (> 0.05): Weak evidence against the null → Fail to reject H₀

Example:

You test whether users spend more time on the updated Adidas product page.
You run a t-test and get p = 0.03 → This means there’s only a 3% chance of observing this result if there was no difference. So you reject H₀.

Steps in Hypothesis Testing:

1. Define H₀ and H₁
2. Choose a significance level (α), often 0.05
3. Collect and summarize data
4. Run a t-test and get the test statistic + p-value
5. Compare p-value with α

• If p ≤ α → Reject H₀
• If p > α → Fail to reject H₀

Real-World Applications:

• A/B testing in marketing or UI design
• Customer behavior analysis (e.g., app feature usage)
• Product experiments (e.g., comparing delivery speeds of two logistics methods)