Learning Objectives

  • Distinguish between descriptive statistics and statistical inference, and explain when each applies
  • Formulate null and alternative hypotheses from a research question, specifying one- and two-tailed tests
  • Define and correctly interpret significance level (α), p-values, and confidence levels
  • Identify and contrast Type I and Type II errors, and explain their consequences in a research context
  • Explain the relationship between α, β, and statistical power, and describe how sample size affects this trade-off
  • Apply the hypothesis testing procedure to a computational or cybersecurity scenario
  • Recognise common misuses of statistical inference including p-hacking, multiple testing, and conflating statistical with practical significance
  • Identify the key assumptions underlying parametric tests and describe how violations affect inference
1

Core Input

Read each tab. These are the building blocks of statistical inference that every activity in this unit builds upon.

Data collected in a study is a sample — a subset of a larger population. The population is the full set of cases your research question is about. An observation is a single data point within the sample.

In computational research, the distinction matters more than it first appears:

  • Population: all possible executions of the algorithm; sample: the 50 benchmark runs you recorded
  • Population: all phishing emails sent globally this year; sample: the 10,000 you analysed
  • Population: all network traffic on enterprise firewalls; sample: the logs from three partner organisations

Descriptive statistics summarise the sample itself — mean, median, standard deviation, frequencies. No claim is made beyond the data you have.

Inferential statistics use the sample to draw conclusions about the population. This requires both a statistical test and an argument that your sample is representative.

Parametric tests assume the data meet specific distributional conditions (usually normality) and are measured on an interval or ratio scale. They are more powerful when the assumptions hold. Non-parametric tests impose fewer restrictions, working with ranked or categorical data, and are more appropriate when assumptions cannot be verified or when samples are small.

Hypothesis testing is a formal procedure for deciding whether evidence in a sample is strong enough to support a claim about a population. The standard eight-step procedure:

  1. Identify the parameter of interest — a population mean, proportion, correlation coefficient, or other estimable quantity
  2. Define H₀ and H₁ — the null hypothesis asserts no effect or no difference (the conservative default); the alternative asserts the effect you are looking for. One-tailed tests specify a direction; two-tailed tests do not
  3. Set the significance level α — the Type I error rate you are willing to accept, commonly 0.05 or 0.01
  4. Define the minimum interesting effect (δ*) — the smallest difference that would be practically meaningful
  5. Calculate the required sample size — from α, desired power (1–β), and δ*
  6. Collect data
  7. Compute the test statistic and p-value
  8. Make and interpret the decision — reject H₀ or fail to reject H₀. Never "accept" H₀, and never claim the result "proves" anything

The p-value is the probability of observing a test statistic at least as extreme as the one calculated, assuming H₀ is true. A small p-value means the observed data would be surprising under H₀ — not that H₀ is false, and not that the effect is large. It answers one question: how consistent are the data with H₀?

Parametric tests are only valid when the data meet their underlying assumptions. Violating assumptions does not automatically break the analysis — but it can bias results, inflate false positive rates, or reduce power. Always check before testing.

1. Normality

Many tests (z-test, t-test, ANOVA) assume the residuals — or the variable itself — follow a normal distribution. This matters most in small samples; the Central Limit Theorem means the sampling distribution of the mean approaches normality in large samples regardless of the underlying distribution.

  • Visual check: Q-Q plot (quantile-quantile plot comparing observed quantiles to normal quantiles)
  • Formal test: Shapiro-Wilk (most powerful for small samples), Anderson-Darling, Kolmogorov-Smirnov
  • Violation? Consider a non-parametric alternative (e.g., Mann-Whitney instead of independent t-test)

2. Independence

Observations must be independent — the value of one observation must not influence another. This assumption is frequently violated in CS research:

  • Multiple benchmark runs on the same hardware share environmental conditions
  • Network traffic logs from the same organisation share infrastructure
  • Repeated vulnerability assessments of the same system are not independent

There is no universal test for independence — it must be established through study design. The Durbin-Watson test can detect serial autocorrelation in time-ordered data.

3. Homogeneity of Variance (Homoscedasticity)

Two-sample tests often assume both groups have similar variances. Check with Levene's test or Bartlett's test. If violated, use Welch's t-test, which does not require equal variances and is now widely recommended as the default.

2

Hypothesis Formulation Workshop

For each scenario below, formulate the null and alternative hypotheses before clicking to reveal. Consider whether a one-tailed or two-tailed test is appropriate and why.

Scenario 1 — Phishing awareness training

A security awareness team runs a mandatory phishing simulation programme across their organisation. Before the programme, 18% of employees clicked on simulated phishing links. After six months of training, they want to test whether the click-through rate has decreased.

H₀: p ≥ 0.18 — Training has no effect; the click-through rate is at least as high as before
H₁: p < 0.18 — Training reduced the click-through rate below 18%
One-tailed (lower) — the claim is directional: we are specifically interested in a reduction, not just any change. Test: one-sample proportion z-test.
Scenario 2 — Algorithm runtime comparison

A research team argues that an IP-address-optimised radix sort (Algorithm A) has a lower mean runtime than standard quicksort (Algorithm B) when processing network packet logs. They run 60 benchmark trials with each algorithm on the same hardware.

H₀: μ_A ≥ μ_B — Algorithm A is no faster than Algorithm B on average
H₁: μ_A < μ_B — Algorithm A has a lower mean runtime
One-tailed (lower) — the team is making a specific directional claim about runtime. Test: independent samples t-test (Welch's unless equal variance is verified).
Scenario 3 — Malware file entropy

A malware analyst hypothesises that the byte entropy of benign executable files differs from the byte entropy of malicious executables (which tend to be packed or encrypted). They have collected 200 benign and 200 malicious samples from a curated dataset.

H₀: μ_benign = μ_malicious — Mean entropy does not differ between benign and malicious files
H₁: μ_benign ≠ μ_malicious — Mean entropy differs between the two groups
Two-tailed — no specific direction is stated; entropy could plausibly differ in either direction. Test: independent samples t-test (check normality; consider Mann-Whitney for non-normal data).
Scenario 4 — Deep packet inspection latency

A network administrator is concerned that adding a deep packet inspection (DPI) module will add more than 2 ms to mean packet processing time. The vendor claims the overhead is negligible. The administrator runs 80 trials to test whether the increase exceeds this operational threshold.

H₀: μ_increase ≤ 2 ms — The DPI module does not add more than 2 ms on average
H₁: μ_increase > 2 ms — The DPI module adds more than 2 ms on average
One-tailed (upper) — the concern is a threshold exceedance; a reduction below 2 ms is irrelevant to the decision. Test: one-sample t-test against μ₀ = 2.
3

Error Types Visualiser

A security research team is testing a new fast-path packet inspection algorithm. The claim: the new algorithm reduces mean inspection time compared to the baseline. They run a one-tailed lower test at α = 0.05.

  • H₀ distribution (orange) — the sampling distribution of the test statistic if the new algorithm is no better than baseline
  • H₁ distribution (blue) — the sampling distribution if the new algorithm genuinely outperforms baseline by a worthwhile margin
  • The dashed vertical line marks the critical threshold (z = −1.645). Results to the left of this line lead to rejecting H₀.
Key insight: Lowering α (moving the critical line to the left) reduces Type I error but increases Type II error — and reduces power. Raising α has the opposite effect. The only way to simultaneously reduce both error types is to increase the sample size, which narrows both distributions.
4

p-value Pitfalls

The p-value is one of the most cited and most misunderstood statistics in research. Each accordion below addresses a common misuse. These errors appear regularly in published papers, vendor reports, and security benchmarks.

p < 0.05 means that, if H₀ were true, you would see a test statistic at least this extreme about 5% of the time. It is not a guarantee that the effect is genuine. One in twenty correctly-designed experiments will produce a false positive simply by chance.

If a team tests 20 independent firewall configurations at α = 0.05, they should expect approximately one to appear significant even if none truly outperforms the baseline. Without pre-registration, it is impossible for a reader to know whether a significant result is one of twenty or the only test run.

Failing to reject H₀ is a weak conclusion — it means the evidence was insufficient, not that the effect does not exist. A study that is underpowered (sample too small to detect a real effect) will routinely produce p > 0.05 even when the effect is genuine.

Example: a small-scale penetration test of 20 scenarios showing no significant difference between two security tools does not prove they are equivalent — it may simply mean there were too few test cases. Always report power or confidence intervals alongside non-significant results.

This is the most dangerous misconception. The p-value is a property of the data given H₀ — P(data this extreme | H₀ true). It is not the probability that H₀ is true given the data — P(H₀ | data). The latter is a Bayesian posterior probability, which requires a prior and a different framework entirely.

Claiming "p = 0.03 means there is only a 3% chance the null hypothesis is true" is logically incorrect. The correct statement: "If H₀ were true, there would be only a 3% chance of observing a result this extreme or more extreme."

When you run many hypothesis tests, the probability that at least one produces a false positive grows rapidly. With 20 independent tests at α = 0.05, the probability of at least one false positive is 1 − (1 − 0.05)²⁰ ≈ 64%.

In security research, this arises when: comparing many ML models in a benchmark without correction; scanning network logs for significant anomalies across many features; or running post-hoc pairwise comparisons between all pairs of groups after an initial test.

Corrections include: Bonferroni (divide α by the number of tests — conservative but simple); Benjamini-Hochberg (controls the false discovery rate — better for large-scale testing). Pre-registration is the most powerful protection: commit to your hypotheses before data collection.

With a large enough sample, any effect, however tiny, will produce a significant p-value. This is not a limitation of the data — it is a mathematical certainty. p-values tell you nothing about the size of the effect.

Example: in a dataset of 500,000 network connections, you discover that encrypted traffic is statistically significantly more likely to arrive between 02:00 and 03:00 (p = 0.0001). But the effect size is 0.3% higher prevalence. This would not trigger any change in operational policy — the statistical significance is a consequence of the sample size, not evidence of a meaningful pattern.

Always report effect sizes (Cohen's d, odds ratio, relative risk, etc.) alongside p-values. Effect sizes are independent of sample size and directly answer the question: "How big is this?"

HARKing (Hypothesising After Results are Known): running an exploratory analysis, finding interesting patterns, then writing them up as though you had hypothesised them in advance. The confirmatory framing misrepresents the exploratory work and inflates confidence in the finding. Exploratory and confirmatory analyses are both legitimate — but they must be clearly distinguished.

p-hacking: repeatedly adjusting the analysis — adding or removing variables, trying different subsets, switching test variants, adding more data — until p < 0.05 appears. Because each decision is effectively another test, the actual Type I error rate becomes much larger than the nominal α.

Both practices are widespread and usually unintentional. The antidote is pre-registration: publicly commit your hypotheses, sample size, analysis plan, and stopping criteria before collecting data. Any post-hoc analysis should be clearly labelled as exploratory.

5

Check Your Understanding

A cybersecurity researcher runs 20 independent significance tests on different network anomaly detectors, all at α = 0.05. Even if no detector genuinely outperforms the baseline, approximately how many tests would you expect to yield a 'significant' result by chance alone?

Correct! At α = 0.05, the probability of a Type I error on each individual test is 5%. With 20 independent tests, you expect 20 × 0.05 = 1 false positive. This is the multiple testing problem: without correction, the family-wise error rate accumulates rapidly.
Not quite — review the material and try again. At α = 0.05, the probability of a Type I error on each individual test is 5%. With 20 independent tests, you expect 20 × 0.05 = 1 false positive. This is the multiple testing problem: without correction, the family-wise error rate accumulates rapidly.

An IDS evaluation paper reports: 'The new classifier achieves 0.3% higher detection rate than the baseline (p = 0.002).' Which statement is the most accurate assessment of this finding?

Correct! The result is statistically significant (p = 0.002 is well below α = 0.05), but the effect size is 0.3 percentage points. Whether this matters operationally depends on the deployment context — but a 0.3pp improvement is usually not practically meaningful and does not justify the resources required to replace a baseline system.
Not quite — review the material and try again. The result is statistically significant (p = 0.002 is well below α = 0.05), but the effect size is 0.3 percentage points. Whether this matters operationally depends on the deployment context — but a 0.3pp improvement is usually not practically meaningful and does not justify the resources required to replace a baseline system.

A malware classification study reports: 'We found no significant difference between classifiers A and B (p = 0.23).' What is the most defensible conclusion?

Correct! Failing to reject H₀ is a weak conclusion — it means the evidence was insufficient, not that the classifiers are equivalent. The study may have been underpowered. The correct framing is: 'There is insufficient evidence to conclude the classifiers differ.' Claiming equivalence requires equivalence testing (TOST procedure), not just a non-significant p-value.
Not quite — review the material and try again. Failing to reject H₀ is a weak conclusion — it means the evidence was insufficient, not that the classifiers are equivalent. The study may have been underpowered. The correct framing is: 'There is insufficient evidence to conclude the classifiers differ.' Claiming equivalence requires equivalence testing (TOST procedure), not just a non-significant p-value.
6

Case Study: The Patch That Wasn't

A cybersecurity vendor has published a whitepaper claiming their kernel-level security patch reduces privilege escalation exploit success rates by 40% compared to unpatched systems. Your organisation is evaluating whether to deploy this patch across 3,000 endpoints at significant operational cost.

Vendor marketing claim

"Independent benchmark testing shows our patch reduces successful privilege escalation exploits by 40% — the most significant protection improvement in this category for over a decade."

Underlying benchmark data (from the vendor's own technical appendix)

Unpatched (control)

Penetration test runs: 200

Successful exploits: 23

Exploit rate: 11.5%

Patched (treatment)

Penetration test runs: 200

Successful exploits: 17

Exploit rate: 8.5%

Work through the investigation steps below. Each reveals a key aspect of evaluating this claim.

The vendor claims a 40% reduction. Let us verify this against the data.

Absolute reduction: 11.5% − 8.5% = 3.0 percentage points
Relative reduction: (11.5 − 8.5) / 11.5 × 100 = 26.1%
The relative reduction is 26.1%, not 40%. The vendor's figure is a significant overstatement of what the data show. This may reflect a calculation error, a different baseline definition, or selective reporting of a more favourable subset of the data.

Before testing, we need to specify exactly what claim we are evaluating.

H₀: p_patched ≥ p_control — The patch does not reduce the exploit rate
H₁: p_patched < p_control — The patch reduces the exploit rate
One-tailed (lower) — the vendor's claim is directional. Test: two-sample proportion z-test. α = 0.05.

We use a two-sample proportion z-test to test the hypotheses.

Sample proportions:
p̂₁ = 23/200 = 0.115 (control)  |  p̂₂ = 17/200 = 0.085 (patched)

Pooled proportion:
p̂_pool = (23 + 17) / (200 + 200) = 40/400 = 0.10

Standard error:
SE = √(0.10 × 0.90 × (1/200 + 1/200)) = √(0.0009) = 0.030

z-statistic:
z = (0.115 − 0.085) / 0.030 = 0.030 / 0.030 = 1.000

Critical value (α = 0.05, one-tailed): z_crit = 1.645
p-value: P(Z > 1.000) ≈ 0.159
z = 1.000 < z_crit = 1.645. We fail to reject H₀ at α = 0.05 (p = 0.159). The data do not provide sufficient evidence that the patch reduces exploit rates. The vendor is reporting a non-significant result as though it were established evidence.

Even if the result had been statistically significant, we should ask: is 3 percentage points operationally meaningful?

Context matters enormously here. Consider two framings:

  • In a pen-test with 200 runs, 6 fewer successful exploits (from 23 to 17). Relative to test volume, this is a 26% reduction — potentially meaningful if exploits are catastrophic (e.g., critical infrastructure).
  • Translated to 3,000 endpoints: if the annual attack surface corresponds to ~200 real privilege escalation attempts per year, the patch would prevent approximately 6 successful exploits. Whether this justifies the deployment cost depends on your threat model and the cost per successful exploit.
The statistical question (is the evidence strong enough to believe the patch works?) and the practical question (is 3pp worth the deployment cost?) are distinct. Here, the evidence base fails the statistical test before we even reach the practical question.

Statistical significance aside, are there threats to validity in this study design?

  • Same testers? If the same pen-testing team conducted both phases, their familiarity with the system between phases creates a learning effect. Post-patch tests may reflect improved tester skill, not improved patch efficacy.
  • Same exploit types? If the patched and unpatched tests used different exploit categories (e.g., the patched system was tested with older CVEs), the comparison is not like-for-like.
  • Blinded? Testers who know which system they are attacking may consciously or unconsciously vary their effort.
  • Sequential vs simultaneous testing: If unpatched testing preceded patched testing, any temporal confounds (new exploits discovered, testers more skilled over time) would systematically bias the comparison.
  • Vendor-managed study: The study was conducted and reported by the vendor, with no independent replication or audit. Financial conflicts of interest apply.

Based on the full analysis, what should you advise the procurement committee?

"The vendor's '40% reduction' claim does not match their own data, which shows a 26.1% relative reduction (11.5% → 8.5%). More critically, the difference is not statistically significant at α = 0.05 (z = 1.00, p = 0.159), meaning this result is reasonably likely to be chance variation in a small sample. The study design has several validity threats including potential tester learning effects and vendor management of the evaluation. The evidence does not currently justify the procurement cost. Recommendation: request an independently conducted, pre-registered evaluation with ≥ 500 pen-test runs per group, blind testers, specified exploit categories, and simultaneous parallel testing. Revisit the procurement decision once credible evidence is available."

Notice the structure: finding → why the claim is wrong → statistical verdict → design limitations → decision → conditions for reconsideration. This is the format of a defensible evidence-based recommendation.

Review

Use the self-assessment below to gauge your confidence on each learning objective. Marking an objective as needing revisiting reveals a direct link back to the relevant activity.

0 of 8 objectives assessed

  • Descriptive statistics summarise a sample; inferential statistics use a sample to draw conclusions about a population — a justified claim requires a representative sample and a valid test
  • H₀ is the conservative default (no effect); H₁ is the effect being claimed — you reject or fail to reject H₀; you never "accept" or "prove" either
  • Type I error (α): reject H₀ when it is true — a false alarm; Type II error (β): fail to reject H₀ when it is false — a miss
  • Statistical power (1−β) is the probability of correctly detecting a real effect; it increases with sample size, effect magnitude, and significance level
  • Failing to reject H₀ is a weak conclusion — it does not prove equivalence; a study may simply be underpowered

  • The p-value is the probability of observing data this extreme if H₀ were true — not the probability that H₀ is true; these are logically distinct
  • Statistical significance and practical significance are independent: large samples make arbitrarily small effects statistically significant
  • Multiple testing at α = 0.05 produces, on average, one false positive per twenty tests — corrections (Bonferroni, Benjamini-Hochberg) or pre-registration are required
  • HARKing and p-hacking inflate the apparent false positive rate without obvious detection; pre-registration is the most robust protection
  • Parametric tests assume normality, independence, and homogeneity of variance — violations bias results or inflate error rates; check with Q-Q plots, Shapiro-Wilk, Levene's test