Learning Objectives

  • Evaluate data quality, suitability and limitations for a given research context
  • Select and justify appropriate analytical methods for a research question
  • Apply descriptive statistics and interpret distributions, central tendency and spread
  • Evaluate computational systems using performance metrics including confusion matrices, precision, recall and F1
  • Construct and critically assess data visualisations, selecting chart types appropriate to the data and purpose
  • Recognise misleading analyses, deceptive visualisations and threats to validity
  • Explain the relationship between association and causation, and identify confounding variables
  • Communicate evidence-based findings clearly and acknowledge limitations
1

Core Input

Read each tab. These ideas underpin every activity in this unit.

Data analysis serves very different purposes depending on the question being asked. Knowing which type of analysis you are doing — and being honest about it — is the first responsibility of a research analyst.

  • Exploratory — you are scanning for patterns without a specific hypothesis. Example: examining six months of network logs to understand what normal traffic looks like before building a detection model. Findings are provisional and must be confirmed in independent data.
  • Descriptive — summarising what happened in a specific dataset. Example: a monthly security operations report counting phishing attempts by category and time of day. Describes, does not generalise.
  • Inferential — using a sample to draw conclusions about a broader population. Example: testing whether a new intrusion detection algorithm performs significantly better than the baseline on a representative sample of network traffic. Requires careful attention to sampling.
  • Causal — establishing that one thing causes another. The hardest and most powerful type. Requires controlled experiments or strong quasi-experimental designs — not just correlation. Example: a randomised study showing that mandatory phishing simulation training causes a measurable reduction in click rates, controlling for prior exposure.

Before analysing data, you must understand what kind of data you have. The measurement scale of a variable determines which statistics and visualisations are meaningful — and which are actively misleading.

  • Nominal — categories with no meaningful order. Example: type of cyber attack (phishing, ransomware, DDoS, insider threat). You can count frequencies; computing a mean is meaningless.
  • Ordinal — ordered categories, but gaps between levels are unequal. Example: vulnerability severity (Low / Medium / High / Critical). You can rank; arithmetic is unreliable.
  • Interval — equal spacing, no true zero. The ratio of two values is not interpretable. Rare in CS research but arises in some composite scores.
  • Ratio — equal spacing with a true zero. Example: response time (ms), packet size (bytes), number of failed login attempts. All arithmetic operations are valid.

Data quality issues to check before analysis:

  • Missing data — missing values are rarely random. A system that logs failures only when an analyst is on duty will underrepresent night-time incidents.
  • Outliers — a single misconfigured server logging 10M requests/sec can destroy the mean and inflate the standard deviation for an entire dataset.
  • Duplicates — the same alert appearing in two log formats is not two alerts.
  • Dependency structures — vulnerability reports from the same organisation are not independent observations. Treating them as independent inflates your effective sample size.
  • Class imbalance — in security datasets, attack events are typically a tiny fraction of all traffic. A classifier that labels everything "normal" can achieve 99% accuracy while catching zero attacks.

Data visualisation is not decoration. It is a tool for reasoning — both for yourself (exploratory) and for your audience (explanatory). The two purposes demand different standards.

  • Exploratory visualisation is fast and provisional. You generate many plots, most of which you will discard. The goal is to find patterns, not to communicate them. Standards are relaxed — axes can be unlabelled, scales imprecise.
  • Explanatory visualisation is a finished argument. Every element must earn its place. It answers a specific question for a specific audience and must not mislead.

Edward Tufte's core principles (summarised):

  • Graphical excellence — give the viewer the greatest number of ideas in the shortest time, with the smallest space and the least ink.
  • Graphical integrity — the visual representation of numbers should be directly proportional to the numerical quantities represented. Distortion is a form of dishonesty.
  • Data-ink ratio — maximise the proportion of ink used to convey actual data. Remove gridlines, borders, backgrounds and decorations that add no information.
  • Avoid chartjunk — 3D effects, excessive colour, decorative patterns and irrelevant imagery all reduce the signal-to-noise ratio of a chart.
2

Chart Types Workshop

Expand each chart type. For each: when to use it, what it reveals, and what can go wrong.

Use when: you want to explore or demonstrate the relationship between two continuous variables. Can encode a third variable through dot size or colour.

Reveals: correlation direction and strength, clusters, outliers, non-linear relationships.

Watch for:

  • Overplotting — with large datasets, thousands of points stack on top of each other, making the chart look denser than it is. Use transparency, jitter, or a 2D density plot instead.
  • Apophenia — the human visual system is wired to find patterns. A cloud of truly random points will still look like it has structure.
  • Confounding — a strong visual correlation does not imply causation. A third variable may be driving both axes.

CS example: plotting memory usage against scan duration for antivirus processes across 1,000 endpoints to identify whether resource-hungry scans are completing faster or slower.

Use when: you want to understand how a continuous variable is distributed across its range — not just its mean, but its shape.

Reveals: skewness, bimodality, spread, typical values, outliers, gaps.

Watch for:

  • Bin width sensitivity — bin width is a subjective choice that dramatically changes the apparent structure. Too wide: hides multimodality. Too narrow: creates noisy spikes that look like structure but are sampling artefacts. Always try multiple bin widths.
  • Confusing histograms with bar charts — a histogram shows a continuous distribution; a bar chart compares discrete categories. The x-axis of a histogram is continuous and ordered; bars should touch, not have gaps.

CS example: visualising the distribution of time-to-detection (seconds) across 10,000 security alerts to understand whether most threats are caught quickly or whether a long tail of slow detections inflates the mean.

Use when: you want to compare the distribution of a continuous variable across several groups, particularly to show medians and spread.

Reveals: median, interquartile range (IQR), symmetry, outliers. Directly comparable across groups.

Watch for:

  • Hidden shape — a bimodal distribution and a uniform distribution can produce identical box plots. The box hides the actual shape of the data. Use a violin plot when the distribution shape matters.
  • Small samples — a box plot from 8 observations conveys false precision. The quartiles are unstable with small n.

CS example: comparing the distribution of model inference times (ms) across three hardware configurations to decide which is most suitable for a real-time threat detection pipeline.

Use when: you want to compare a numeric quantity (count, rate, mean) across a set of discrete categories.

Reveals: relative magnitudes, rankings, proportions of a whole (stacked variant).

Watch for:

  • Truncated y-axis — the single most common form of misleading chart. Starting the y-axis at a non-zero value makes small differences appear enormous. The y-axis of a bar chart should almost always start at zero, because bar height encodes magnitude proportionally.
  • Arbitrary ordering — sorting bars randomly or alphabetically hides patterns. Order by value unless there is a natural ordering (e.g., time, severity).
  • Too many categories — more than 8–10 bars becomes difficult to read. Consider grouping.

CS example: comparing the number of critical vulnerabilities patched per quarter across five software products to report progress to a risk committee.

Use when: you need to compare distributions across groups and the shape of the distribution matters, not just the median and spread.

Reveals: full density estimate, multimodality, spread, median — more than a box plot.

Watch for:

  • Small samples — the kernel density estimate that generates the violin shape is unreliable with small n. The smooth curves look authoritative but may be entirely artefactual.
  • Unfamiliar to general audiences — violin plots require explanation for non-technical readers. If your audience is unfamiliar with them, a box plot with individual data points overlaid may communicate more clearly.

CS example: comparing the distribution of prediction confidence scores across three malware classifiers to understand not just which classifier is more accurate, but whether its high-confidence predictions are reliably correct.

Use when: you want to show how a variable changes over a continuous ordered axis — typically time.

Reveals: trends, seasonality, sudden events, rate of change.

Watch for:

  • Cherry-picked time window — selecting a window that shows a convenient trend while hiding the broader context is one of the most common forms of selective reporting.
  • Non-zero baseline — unlike bar charts, line charts do not always need to start at zero (the focus is on change, not magnitude). But a compressed y-axis can exaggerate volatility and make noise look like a signal.
  • Dual y-axes — placing two variables on the same chart with independently scaled axes makes them appear correlated by construction. Treat dual-axis charts with significant suspicion.

CS example: plotting daily failed SSH login attempts on an internet-facing server over six months to identify whether a recent hardening change had a detectable effect.

3

Chart Crime Lab

Each chart below has been published in a real-world context — a security report, a vendor whitepaper, or an academic poster. Each contains a visual design decision that misleads the reader. Identify the problem before revealing the verdict.

Crime #1 The Truncated Axis

A security team reports dramatic progress after rolling out mandatory password manager adoption across the organisation.

The crime

The y-axis starts at 90%, not 0%. The visual height of the "After" bar is 3× the "Before" bar — implying a tripling. The actual improvement is 2 percentage points (91% → 93%). A bar chart's y-axis must start at zero, because bar height encodes the quantity. Starting elsewhere breaks that proportional relationship.

Crime #2 The Twin Deception

Two organisations publish daily phishing attempt data. A consultant claims their threat profiles are "almost identical."

The crime

The two charts use different y-axis scales. Organisation Alpha receives 8–14 phishing attempts per day. Organisation Beta receives 800–1,400. The visual patterns are identical because both datasets have been independently scaled to fill the same chart space. Side-by-side charts that do not share a y-axis scale cannot be visually compared.

Crime #3 The Cherry-Pick

A security operations team publishes its performance dashboard. "Response times have improved by 25% since we restructured the team."

The crime

The chart shows only days 10–15 — a local downward run within a noisy 30-day trend that is actually rising. Response times went from ~47 min (day 1) to ~82 min (day 30). Selecting the window that best supports the narrative is selective reporting — a form of data manipulation even when no individual number is fabricated.

Crime #4 The Dual-Axis Trick

An analyst presents "compelling evidence" that reduced IT training budgets correlate with rising security incidents.

The crime

The chart has two independent y-axes — one for incident counts (40–90), one for training hours (1.5–6.0) — each scaled to fill the same visual space. This forces any two variables that broadly increase or decrease together to appear correlated. Dual-axis charts manufacture visual correlation by construction. In this case, both variables happen to trend upward over time (a common confound), but they have no meaningful causal relationship.

Crime #5 The Percentage Illusion

Two research teams publish their attack type breakdowns as pie charts. A journalist concludes that both organisations face the same threat profile.

The crime

Team A's chart is based on n = 12 incidents; Team B's on n = 1,200. The percentage breakdown looks nearly identical, but Team A's percentages are statistically meaningless — a single additional phishing incident would shift their "phishing" slice from 42% to 46%. Presenting percentages without sample sizes invites comparison between distributions that cannot legitimately be compared. Always report n alongside proportions.

Crime #6 The Radius Illusion

A security vendor ranks their product tiers using circles. The bigger the circle, the better the tier. Something looks wrong about the size differences.

The crime

Score values have been encoded as circle radii, not areas. Because we perceive circle area rather than radius, the apparent size difference is the square of the actual value ratio. Score 6 looks 9× bigger than score 2 — but represents only 3× the value. The correct encoding maps √score to radius, so circle area remains proportional to what is being compared. This crime is widespread in product marketing and infographics.

4

Confusion Matrix Lab

An intrusion detection system (IDS) has been deployed to monitor network traffic at a financial services company. After one week, you audit its performance on 1,000 traffic samples. The results are shown below. Calculate each metric in turn, then answer the critical question.

Predicted: ATTACK Predicted: NORMAL
Actual: ATTACK 42True Positive 13False Negative ⚠
Actual: NORMAL 8False Positive 937True Negative

Total samples: 1,000  |  Real attacks: 55  |  Real normal traffic: 945

Select a metric to calculate:
5

Check Your Understanding

A cybersecurity researcher finds that organisations with larger security teams report more data breaches per year. Which interpretation is most defensible?

Correct! Larger organisations tend to have both larger security teams and more incidents — team size and breach count are both driven by organisation size (a confound). Inferring causation from this correlation would be an ecological fallacy.
Not quite — review the material and try again. Larger organisations tend to have both larger security teams and more incidents — team size and breach count are both driven by organisation size (a confound). Inferring causation from this correlation would be an ecological fallacy.

You are comparing the false positive rates of three malware classifiers. Which visualisation would best show both the typical false positive rate and the spread of that rate across test runs?

Correct! A box plot (or violin plot) shows both the median and the spread. A bar chart of means hides the distribution of values across test runs, potentially concealing high variability. Pie and line charts are not appropriate for comparing distributions across groups.
Not quite — review the material and try again. A box plot (or violin plot) shows both the median and the spread. A bar chart of means hides the distribution of values across test runs, potentially concealing high variability. Pie and line charts are not appropriate for comparing distributions across groups.

An IDS achieves 99.1% accuracy on a network dataset where 99% of all traffic samples are normal. What is the most important limitation of this result?

Correct! On heavily imbalanced data, a classifier can achieve near-perfect accuracy by simply predicting the majority class every time. This IDS could be labelling everything as 'normal' and still hit 99%. Recall — the proportion of actual attacks correctly detected — is the critical metric here.
Not quite — review the material and try again. On heavily imbalanced data, a classifier can achieve near-perfect accuracy by simply predicting the majority class every time. This IDS could be labelling everything as 'normal' and still hit 99%. Recall — the proportion of actual attacks correctly detected — is the critical metric here.
6

Case Study: The Security Efficiency Report

A cybersecurity vendor has published a whitepaper claiming their endpoint protection platform (EPP) reduces false positives by 40% compared to leading competitors, calling it "the most significant efficiency breakthrough in enterprise security this decade." Your organisation is considering a £200,000 procurement. You have been asked to scrutinise the evidence.

Marketing claim

"CyberShield EPP generated 40% fewer false positives than the market-leading competitor in independent benchmark testing — giving your analysts back the time they need to focus on real threats."

The chart
Raw benchmark data (5,000 samples each)

CyberShield EPP

Pred: ATTACKPred: NORMAL
Actual: ATTACK18045
Actual: NORMAL124763

Competitor

Pred: ATTACKPred: NORMAL
Actual: ATTACK21015
Actual: NORMAL204755

* Performance data from vendor-managed benchmark dataset, Q4 only. Threat mix not disclosed.

Work through each investigation step below. Use the confusion matrices above to calculate the metrics.

The marketing chart displays false positive counts — CyberShield (12) versus Competitor (20) — but with a y-axis starting at 10, not 0. The CyberShield bar occupies 2 units of visual height; the Competitor bar occupies 10 units. This makes CyberShield appear to generate five times fewer false positives when the actual ratio is 20:12 ≈ 5:3.

The 40% reduction figure is arithmetically correct: (20−12)/20 = 40%. But the visual representation amplifies this into something that looks like an order-of-magnitude difference. This is a textbook truncated-axis bar chart.

False positives are legitimate alert fatigue concerns — analysts who spend hours investigating benign traffic miss real threats. CyberShield generates 12 FPs per 5,000 samples vs 20 for the Competitor. In a real network processing 100,000 samples per day, this represents 240 vs 400 spurious alerts daily. The difference is real and potentially valuable.

However, false positives are only one type of error. The confusion matrix contains a second type — false negatives — that the marketing material does not mention.

Recall = TP / (TP + FN) — of all real attacks, what fraction were detected?

  • CyberShield EPP: 180 / (180 + 45) = 180/225 = 80.0%
  • Competitor: 210 / (210 + 15) = 210/225 = 93.3%

CyberShield misses 45 out of 225 real attacks (20%). The Competitor misses only 15 (6.7%). For every 1,000 genuine attacks, CyberShield leaves 200 undetected; the Competitor leaves 67 undetected. This is not a minor difference — it is a fundamental security trade-off that the marketing material conceals entirely by reporting only false positive counts.

CyberShield has higher precision, lower recall than the Competitor. It raises fewer false alarms (fewer wasted analyst hours) but catches fewer real attacks. The Competitor raises more false alarms but misses significantly fewer real threats.

This is a genuine engineering trade-off — not a flaw. Some security environments genuinely prioritise reducing alert fatigue (e.g., understaffed SOCs drowning in noise). Others cannot tolerate missed detections (e.g., critical infrastructure, financial services). Neither system is universally better. The claim that CyberShield represents "a breakthrough" elides this trade-off entirely.

The footnote reads: "Performance data from vendor-managed benchmark dataset, Q4 only. Threat mix not disclosed."

This is a critical red flag. When a vendor controls the evaluation dataset:

  • They may select a threat mix that favours their system's strengths
  • They may evaluate on data similar to their training data, inflating performance estimates
  • Q4 only may not represent the full range of threats across the year
  • The competitor may not have been tested under the same conditions

Independent, peer-reviewed evaluation on representative, uncontrolled data is the minimum standard for research integrity. Vendor-supplied benchmarks should be treated as marketing material — informative, but not authoritative.

A defensible recommendation acknowledges the evidence and its limitations:

"CyberShield EPP reduces false positives by 40% compared to the competitor, which may meaningfully reduce analyst workload. However, it also exhibits significantly lower recall (80% vs 93.3%), missing 20% of real attacks. This trade-off may be acceptable in environments where alert fatigue is the primary operational risk, but would be concerning in high-stakes contexts where missed detections carry severe consequences. The evaluation data was vendor-managed and the threat mix was not disclosed, which limits confidence in these figures. Independent evaluation on organisation-representative data is recommended before procurement."

Notice the structure: finding → caveat → context-dependency → limitation → recommendation. This is the format of a defensible evidence-based conclusion.

Review

Use the self-assessment below to gauge your confidence on each learning objective. Marking an objective as needing revisiting reveals a direct link back to the relevant activity.

0 of 8 objectives assessed

  • Measurement scales (Nominal, Ordinal, Interval, Ratio) determine which statistics and visualisations are meaningful — a mean on nominal data is an error
  • Exploratory analysis generates hypotheses; confirmatory analysis tests them — keeping these separate prevents inflated false positive rates
  • Correlation is not causation; a confounding variable can drive two correlated outcomes independently
  • Anscombe's Quartet: four datasets with identical summary statistics, completely different structures — always visualise before summarising
  • Class imbalance makes accuracy misleading: a classifier labelling everything "normal" on a 99% normal dataset scores 99% while catching zero attacks
  • Precision, recall, F1, and specificity each capture different aspects of classifier performance on imbalanced data

  • Bar chart y-axes must start at zero — bar height encodes magnitude proportionally, so truncation directly distorts the perceived difference
  • Two line charts with different y-axis scales can look visually identical while representing values differing by orders of magnitude
  • Cherry-picking a time window can reverse the apparent direction of a trend — always show the full context
  • Dual y-axis charts manufacture visual correlation by scaling each axis independently to fill the same visual space
  • Percentages without sample sizes prevent meaningful comparison — n = 12 and n = 1,200 may produce identical pie charts while one is statistically meaningless
  • Circle size should encode area, not radius — radius encoding squares the apparent difference, making 3× look like 9×