Unit 9: Statistical Inference
Learning Objectives
- Distinguish between descriptive statistics and statistical inference, and explain when each applies
- Formulate null and alternative hypotheses from a research question, specifying one- and two-tailed tests
- Define and correctly interpret significance level (α), p-values, and confidence levels
- Identify and contrast Type I and Type II errors, and explain their consequences in a research context
- Explain the relationship between α, β, and statistical power, and describe how sample size affects this trade-off
- Apply the hypothesis testing procedure to a computational or cybersecurity scenario
- Recognise common misuses of statistical inference including p-hacking, multiple testing, and conflating statistical with practical significance
- Identify the key assumptions underlying parametric tests and describe how violations affect inference
Core Input
Read each tab. These are the building blocks of statistical inference that every activity in this unit builds upon.
Data collected in a study is a sample — a subset of a larger population. The population is the full set of cases your research question is about. An observation is a single data point within the sample.
In computational research, the distinction matters more than it first appears:
- Population: all possible executions of the algorithm; sample: the 50 benchmark runs you recorded
- Population: all phishing emails sent globally this year; sample: the 10,000 you analysed
- Population: all network traffic on enterprise firewalls; sample: the logs from three partner organisations
Descriptive statistics summarise the sample itself — mean, median, standard deviation, frequencies. No claim is made beyond the data you have.
Inferential statistics use the sample to draw conclusions about the population. This requires both a statistical test and an argument that your sample is representative.
Parametric tests assume the data meet specific distributional conditions (usually normality) and are measured on an interval or ratio scale. They are more powerful when the assumptions hold. Non-parametric tests impose fewer restrictions, working with ranked or categorical data, and are more appropriate when assumptions cannot be verified or when samples are small.
Hypothesis testing is a formal procedure for deciding whether evidence in a sample is strong enough to support a claim about a population. The standard eight-step procedure:
- Identify the parameter of interest — a population mean, proportion, correlation coefficient, or other estimable quantity
- Define H₀ and H₁ — the null hypothesis asserts no effect or no difference (the conservative default); the alternative asserts the effect you are looking for. One-tailed tests specify a direction; two-tailed tests do not
- Set the significance level α — the Type I error rate you are willing to accept, commonly 0.05 or 0.01
- Define the minimum interesting effect (δ*) — the smallest difference that would be practically meaningful
- Calculate the required sample size — from α, desired power (1–β), and δ*
- Collect data
- Compute the test statistic and p-value
- Make and interpret the decision — reject H₀ or fail to reject H₀. Never "accept" H₀, and never claim the result "proves" anything
The p-value is the probability of observing a test statistic at least as extreme as the one calculated, assuming H₀ is true. A small p-value means the observed data would be surprising under H₀ — not that H₀ is false, and not that the effect is large. It answers one question: how consistent are the data with H₀?
Parametric tests are only valid when the data meet their underlying assumptions. Violating assumptions does not automatically break the analysis — but it can bias results, inflate false positive rates, or reduce power. Always check before testing.
1. Normality
Many tests (z-test, t-test, ANOVA) assume the residuals — or the variable itself — follow a normal distribution. This matters most in small samples; the Central Limit Theorem means the sampling distribution of the mean approaches normality in large samples regardless of the underlying distribution.
- Visual check: Q-Q plot (quantile-quantile plot comparing observed quantiles to normal quantiles)
- Formal test: Shapiro-Wilk (most powerful for small samples), Anderson-Darling, Kolmogorov-Smirnov
- Violation? Consider a non-parametric alternative (e.g., Mann-Whitney instead of independent t-test)
2. Independence
Observations must be independent — the value of one observation must not influence another. This assumption is frequently violated in CS research:
- Multiple benchmark runs on the same hardware share environmental conditions
- Network traffic logs from the same organisation share infrastructure
- Repeated vulnerability assessments of the same system are not independent
There is no universal test for independence — it must be established through study design. The Durbin-Watson test can detect serial autocorrelation in time-ordered data.
3. Homogeneity of Variance (Homoscedasticity)
Two-sample tests often assume both groups have similar variances. Check with Levene's test or Bartlett's test. If violated, use Welch's t-test, which does not require equal variances and is now widely recommended as the default.
Hypothesis Formulation Workshop
For each scenario below, formulate the null and alternative hypotheses before clicking to reveal. Consider whether a one-tailed or two-tailed test is appropriate and why.
A security awareness team runs a mandatory phishing simulation programme across their organisation. Before the programme, 18% of employees clicked on simulated phishing links. After six months of training, they want to test whether the click-through rate has decreased.
A research team argues that an IP-address-optimised radix sort (Algorithm A) has a lower mean runtime than standard quicksort (Algorithm B) when processing network packet logs. They run 60 benchmark trials with each algorithm on the same hardware.
A malware analyst hypothesises that the byte entropy of benign executable files differs from the byte entropy of malicious executables (which tend to be packed or encrypted). They have collected 200 benign and 200 malicious samples from a curated dataset.
A network administrator is concerned that adding a deep packet inspection (DPI) module will add more than 2 ms to mean packet processing time. The vendor claims the overhead is negligible. The administrator runs 80 trials to test whether the increase exceeds this operational threshold.
Error Types Visualiser
A security research team is testing a new fast-path packet inspection algorithm. The claim: the new algorithm reduces mean inspection time compared to the baseline. They run a one-tailed lower test at α = 0.05.
- H₀ distribution (orange) — the sampling distribution of the test statistic if the new algorithm is no better than baseline
- H₁ distribution (blue) — the sampling distribution if the new algorithm genuinely outperforms baseline by a worthwhile margin
- The dashed vertical line marks the critical threshold (z = −1.645). Results to the left of this line lead to rejecting H₀.
p-value Pitfalls
The p-value is one of the most cited and most misunderstood statistics in research. Each accordion below addresses a common misuse. These errors appear regularly in published papers, vendor reports, and security benchmarks.
p < 0.05 means that, if H₀ were true, you would see a test statistic at least this extreme about 5% of the time. It is not a guarantee that the effect is genuine. One in twenty correctly-designed experiments will produce a false positive simply by chance.
If a team tests 20 independent firewall configurations at α = 0.05, they should expect approximately one to appear significant even if none truly outperforms the baseline. Without pre-registration, it is impossible for a reader to know whether a significant result is one of twenty or the only test run.
Failing to reject H₀ is a weak conclusion — it means the evidence was insufficient, not that the effect does not exist. A study that is underpowered (sample too small to detect a real effect) will routinely produce p > 0.05 even when the effect is genuine.
Example: a small-scale penetration test of 20 scenarios showing no significant difference between two security tools does not prove they are equivalent — it may simply mean there were too few test cases. Always report power or confidence intervals alongside non-significant results.
This is the most dangerous misconception. The p-value is a property of the data given H₀ — P(data this extreme | H₀ true). It is not the probability that H₀ is true given the data — P(H₀ | data). The latter is a Bayesian posterior probability, which requires a prior and a different framework entirely.
Claiming "p = 0.03 means there is only a 3% chance the null hypothesis is true" is logically incorrect. The correct statement: "If H₀ were true, there would be only a 3% chance of observing a result this extreme or more extreme."
When you run many hypothesis tests, the probability that at least one produces a false positive grows rapidly. With 20 independent tests at α = 0.05, the probability of at least one false positive is 1 − (1 − 0.05)²⁰ ≈ 64%.
In security research, this arises when: comparing many ML models in a benchmark without correction; scanning network logs for significant anomalies across many features; or running post-hoc pairwise comparisons between all pairs of groups after an initial test.
Corrections include: Bonferroni (divide α by the number of tests — conservative but simple); Benjamini-Hochberg (controls the false discovery rate — better for large-scale testing). Pre-registration is the most powerful protection: commit to your hypotheses before data collection.
With a large enough sample, any effect, however tiny, will produce a significant p-value. This is not a limitation of the data — it is a mathematical certainty. p-values tell you nothing about the size of the effect.
Example: in a dataset of 500,000 network connections, you discover that encrypted traffic is statistically significantly more likely to arrive between 02:00 and 03:00 (p = 0.0001). But the effect size is 0.3% higher prevalence. This would not trigger any change in operational policy — the statistical significance is a consequence of the sample size, not evidence of a meaningful pattern.
Always report effect sizes (Cohen's d, odds ratio, relative risk, etc.) alongside p-values. Effect sizes are independent of sample size and directly answer the question: "How big is this?"
HARKing (Hypothesising After Results are Known): running an exploratory analysis, finding interesting patterns, then writing them up as though you had hypothesised them in advance. The confirmatory framing misrepresents the exploratory work and inflates confidence in the finding. Exploratory and confirmatory analyses are both legitimate — but they must be clearly distinguished.
p-hacking: repeatedly adjusting the analysis — adding or removing variables, trying different subsets, switching test variants, adding more data — until p < 0.05 appears. Because each decision is effectively another test, the actual Type I error rate becomes much larger than the nominal α.
Both practices are widespread and usually unintentional. The antidote is pre-registration: publicly commit your hypotheses, sample size, analysis plan, and stopping criteria before collecting data. Any post-hoc analysis should be clearly labelled as exploratory.
Check Your Understanding
A cybersecurity researcher runs 20 independent significance tests on different network anomaly detectors, all at α = 0.05. Even if no detector genuinely outperforms the baseline, approximately how many tests would you expect to yield a 'significant' result by chance alone?
An IDS evaluation paper reports: 'The new classifier achieves 0.3% higher detection rate than the baseline (p = 0.002).' Which statement is the most accurate assessment of this finding?
A malware classification study reports: 'We found no significant difference between classifiers A and B (p = 0.23).' What is the most defensible conclusion?
Case Study: The Patch That Wasn't
A cybersecurity vendor has published a whitepaper claiming their kernel-level security patch reduces privilege escalation exploit success rates by 40% compared to unpatched systems. Your organisation is evaluating whether to deploy this patch across 3,000 endpoints at significant operational cost.
"Independent benchmark testing shows our patch reduces successful privilege escalation exploits by 40% — the most significant protection improvement in this category for over a decade."
Unpatched (control)
Penetration test runs: 200
Successful exploits: 23
Exploit rate: 11.5%
Patched (treatment)
Penetration test runs: 200
Successful exploits: 17
Exploit rate: 8.5%
Work through the investigation steps below. Each reveals a key aspect of evaluating this claim.
The vendor claims a 40% reduction. Let us verify this against the data.
3.0 percentage pointsRelative reduction: (11.5 − 8.5) / 11.5 × 100 =
26.1%
Before testing, we need to specify exactly what claim we are evaluating.
We use a two-sample proportion z-test to test the hypotheses.
p̂₁ = 23/200 =
0.115 (control) | p̂₂ = 17/200 = 0.085 (patched)Pooled proportion:
p̂_pool = (23 + 17) / (200 + 200) = 40/400 =
0.10Standard error:
SE = √(0.10 × 0.90 × (1/200 + 1/200)) = √(0.0009) =
0.030z-statistic:
z = (0.115 − 0.085) / 0.030 = 0.030 / 0.030 =
1.000Critical value (α = 0.05, one-tailed): z_crit =
1.645p-value: P(Z > 1.000) ≈
0.159
Even if the result had been statistically significant, we should ask: is 3 percentage points operationally meaningful?
Context matters enormously here. Consider two framings:
- In a pen-test with 200 runs, 6 fewer successful exploits (from 23 to 17). Relative to test volume, this is a 26% reduction — potentially meaningful if exploits are catastrophic (e.g., critical infrastructure).
- Translated to 3,000 endpoints: if the annual attack surface corresponds to ~200 real privilege escalation attempts per year, the patch would prevent approximately 6 successful exploits. Whether this justifies the deployment cost depends on your threat model and the cost per successful exploit.
Statistical significance aside, are there threats to validity in this study design?
- Same testers? If the same pen-testing team conducted both phases, their familiarity with the system between phases creates a learning effect. Post-patch tests may reflect improved tester skill, not improved patch efficacy.
- Same exploit types? If the patched and unpatched tests used different exploit categories (e.g., the patched system was tested with older CVEs), the comparison is not like-for-like.
- Blinded? Testers who know which system they are attacking may consciously or unconsciously vary their effort.
- Sequential vs simultaneous testing: If unpatched testing preceded patched testing, any temporal confounds (new exploits discovered, testers more skilled over time) would systematically bias the comparison.
- Vendor-managed study: The study was conducted and reported by the vendor, with no independent replication or audit. Financial conflicts of interest apply.
Based on the full analysis, what should you advise the procurement committee?
Notice the structure: finding → why the claim is wrong → statistical verdict → design limitations → decision → conditions for reconsideration. This is the format of a defensible evidence-based recommendation.
Review
Use the self-assessment below to gauge your confidence on each learning objective. Marking an objective as needing revisiting reveals a direct link back to the relevant activity.
- Descriptive statistics summarise a sample; inferential statistics use a sample to draw conclusions about a population — a justified claim requires a representative sample and a valid test
- H₀ is the conservative default (no effect); H₁ is the effect being claimed — you reject or fail to reject H₀; you never "accept" or "prove" either
- Type I error (α): reject H₀ when it is true — a false alarm; Type II error (β): fail to reject H₀ when it is false — a miss
- Statistical power (1−β) is the probability of correctly detecting a real effect; it increases with sample size, effect magnitude, and significance level
- Failing to reject H₀ is a weak conclusion — it does not prove equivalence; a study may simply be underpowered
- The p-value is the probability of observing data this extreme if H₀ were true — not the probability that H₀ is true; these are logically distinct
- Statistical significance and practical significance are independent: large samples make arbitrarily small effects statistically significant
- Multiple testing at α = 0.05 produces, on average, one false positive per twenty tests — corrections (Bonferroni, Benjamini-Hochberg) or pre-registration are required
- HARKing and p-hacking inflate the apparent false positive rate without obvious detection; pre-registration is the most robust protection
- Parametric tests assume normality, independence, and homogeneity of variance — violations bias results or inflate error rates; check with Q-Q plots, Shapiro-Wilk, Levene's test