Evidence Collection and Investigation Methods

Introduction

Root cause analysis (RCA) is only as credible as the evidence on which it rests. A plausible explanation is not necessarily a defensible explanation. In quality, safety, healthcare, manufacturing, service, and public-sector environments, investigators must show what happened, under what conditions, how confidently it is known, and why alternative explanations were rejected.

Evidence collection is therefore more than gathering facts after an undesirable event. It is a disciplined inquiry into process reality. It combines direct observation, interviews, document and record review, sampling, measurement-system assessment, chronology construction, and data-integrity controls. The investigator must also manage human judgment: hindsight bias can make outcomes appear inevitable; confirmation bias can make investigators search selectively for supporting evidence; and attribution bias can cause attention to shift prematurely from system conditions to individual blame.

This chapter presents an evidence-centred approach to RCA. It treats an incident as an observable outcome generated by a process operating within a system. The purpose is not merely to identify a proximate error, but to establish a reliable chain from conditions and controls to the observed outcome.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish evidence, interpretation, assumption, allegation, and conclusion.
2. Design an investigation plan that preserves objectivity and traceability.
3. Use observation, interviews, documents, sampling, and timelines systematically.
4. Evaluate whether measurement systems and data sources are fit for RCA.
5. Apply basic sampling and data-integrity principles without overstating certainty.
6. Recognize and control hindsight, confirmation, attribution, availability, and anchoring biases.
7. Integrate multiple evidence streams into a defensible causal argument.
8. Conduct an investigation ethically, fairly, and proportionately.

1. Evidence as the Foundation of RCA

1.1 Evidence, information, and inference

Evidence is information that bears on a question and can be examined for reliability. It may be physical, digital, documentary, observational, testimonial, or quantitative. Evidence becomes useful when its source, time, context, and limitations are known.

A useful distinction is:

Investigators should label these categories explicitly. A statement such as “the operator ignored the alarm” combines an observation, an untested mental-state claim, and an attribution. A more defensible formulation is: “The alarm was active at 14:03, acknowledgement was not recorded, and the procedure did not specify a required response within a defined time.”

1.2 The evidence chain

An evidence chain links the event to the causal explanation:

Event → conditions → actions or non-actions → controls → organizational context → outcome

The chain should answer five questions:

• What exactly occurred?
• When and where did it occur?
• What was the process supposed to do?
• What actually happened?
• Which conditions made the outcome possible, likely, or difficult to detect?

A strong causal claim is proportional to the evidence. “The revised software caused the failure” is weak if based only on temporal proximity. A stronger claim identifies the changed function, reproduces the failure or demonstrates a mechanism, compares performance before and after the change, and rules out competing explanations.

2. Investigation Design Before Data Collection

2.1 Define the investigation question

An investigation should begin with a bounded question rather than a preferred cause. For example:

“How did a conforming product pass final inspection on 18 March, and why was the escape not detected before shipment?”

This is superior to “Who made the mistake?” because it directs attention to process performance and detection controls.

Define the scope by:

• product, service, process, or population;
• event period and relevant preceding period;
• locations, shifts, systems, and suppliers;
• known impact and possible impact;
• decisions that the investigation must support.

A scope statement should also identify exclusions. Scope control prevents an investigation from becoming either an unfocused audit or an artificially narrow search for a convenient explanation.

2.2 Preserve evidence early

Evidence can disappear through routine cleaning, system overwriting, maintenance, staff turnover, or modification of the process. Early actions may include:

• preserving relevant electronic logs and access records;
• quarantining affected material when justified;
• photographing conditions before restoration;
• recording equipment status, settings, and alarms;
• securing samples using documented identification and custody;
• preventing unauthorized alteration of original records.

Preservation must be proportionate. It should not create unnecessary operational risk or compromise safety. The investigator should record who preserved what, when, by what method, and under whose authority.

2.3 Build an evidence matrix

An evidence matrix connects investigation questions to sources and tests.

This matrix helps prevent overreliance on a single source and makes missing evidence visible.

3. Process Observation: Seeing Work as Performed

3.1 Documented work and actual work

Procedures describe intended work; observation reveals performed work. The difference is not automatically noncompliance. It may indicate that the procedure is impractical, ambiguous, obsolete, or inconsistent with equipment and workload.

Observation should examine:

• sequence and timing of tasks;
• interfaces between people, equipment, software, and materials;
• interruptions, handoffs, and workarounds;
• environmental conditions such as lighting, noise, temperature, and congestion;
• information available to the worker at the decision point;
• normal variation and abnormal recovery actions;
• barriers that prevent, detect, or mitigate error.

The observer should describe actions neutrally. “The technician carelessly skipped step 7” is an interpretation. “The technician moved from step 6 to step 8 while responding to a telephone call; no visual or electronic prompt required completion of step 7” is an observation with contextual detail.

3.2 Observation methods

A gemba walk examines the place where value-creating work occurs. A job safety or task analysis decomposes work into steps, hazards, controls, and recovery opportunities. Shadowing follows an individual or transaction across interfaces. Video or system replay, where lawful and ethically approved, can support precise sequence analysis.

Observation should include routine successful cases, not only failures. Comparing successful and unsuccessful execution helps identify discriminating conditions. Observation of a single event is valuable for reconstruction but insufficient for generalizing process frequency.

4. Interviews as Structured Evidence

Interviews provide access to knowledge that may not exist in records, especially informal practices and weak signals. They are also vulnerable to memory distortion, social pressure, status effects, and retrospective rationalization.

4.1 Interview sequence and questioning

Interview as close as practical to the event, before group discussion reshapes recollection. Begin with an open prompt:

“Please describe what happened from the beginning, in your own words.”

Then use non-leading probes:

• “What did you see or hear?”
• “What information was available at that moment?”
• “What did you expect the system to do?”
• “What happened next?”
• “Had this situation occurred before?”
• “What would normally make the problem visible?”

Avoid questions that imply the answer, such as “Why did you ignore the warning?” Replace them with “What did you understand the warning to mean, and what action did you take?”

Separate fact from opinion and record uncertainty. A witness who says “I think” is not necessarily unreliable; uncertainty honestly expressed is more useful than false precision.

4.2 Fairness and psychological safety

Interviews should be conducted in a manner consistent with just-culture principles. The investigator must distinguish human error, at-risk behaviour, and reckless behaviour rather than treating every adverse outcome as culpable misconduct. Confidentiality limits should be explained, and participants should be told how records will be used.

Interview notes should identify date, interviewer, participant role, relevant time references, and whether the account is firsthand or hearsay. Important claims should be corroborated where possible, without treating inconsistency alone as proof of dishonesty.

5. Document, Record, and Digital Evidence Review

Documents include procedures, specifications, training materials, maintenance instructions, risk assessments, change records, and contracts. Records include completed forms, transactions, inspection results, alarms, laboratory results, and complaint data.

Review should address both content and control:

• Was the document approved and current at the time?
• Was the relevant version available to the worker?
• Were changes authorized, assessed, and communicated?
• Does the record show who entered data and when?
• Can entries be altered without an audit trail?
• Are timestamps synchronized across systems?
• Are missing, duplicated, or impossible values present?

5.1 Data integrity principles

Reliable data should be attributable, legible or readable, contemporaneous, original or a verified copy, and accurate. For electronic systems, investigators should also examine completeness, consistency, persistence, and availability.

A timestamp is meaningful only if the clock is controlled and its time zone and synchronization status are known. A completed checkbox is evidence that a field was completed, not necessarily that the underlying activity occurred correctly. Audit trails, source records, exception logs, and system metadata may be needed to assess authenticity.

Data cleaning must be reproducible. Never silently delete an outlier because it conflicts with the proposed cause. Record the original value, reason for exclusion, rule applied, and impact on conclusions.

6. Sampling and Quantitative Evidence

When the population is too large for a census, sampling can estimate conditions efficiently. The design must match the question.

• Random sampling gives each population unit a known chance of selection and supports statistical inference.
• Systematic sampling selects every k-th unit after a random start; it can be efficient but may be distorted by periodic patterns.
• Stratified sampling samples separately from important groups such as shift, product, supplier, or location.
• Purposive sampling deliberately selects informative cases; it is useful for exploration but does not support unbiased prevalence estimates.
• Convenience sampling is easy but particularly vulnerable to selection bias.

For a simple estimate of a proportion, the observed proportion is:

where x is the number of sampled units with the characteristic and n is the total number sampled. The quantity p̂ is dimensionless and estimates the population proportion only under an appropriate sampling design.

For planning a large-sample estimate with approximate margin of error E at confidence level associated with critical value z, a commonly used formula is:

Here n is the required sample size, p is the anticipated population proportion, E is the desired absolute margin of error, and z is the standard-normal critical value. This approximation assumes independent observations and a sufficiently large population; finite-population correction, clustering, stratification, nonresponse, and expected data loss may require adjustment. If p is unknown, p = 0.5 gives the largest value of p(1 − p) and therefore a conservative simple-random sample size.

For RCA, statistical precision is not the only criterion. A small number of high-severity cases may justify investigation even when prevalence cannot be estimated precisely. Conversely, a large sample of poorly defined or biased records may create false confidence.

Worked example: interpreting a sample

An investigator reviews 80 randomly selected service cases and finds 12 with a missing verification. The estimated proportion is:

Thus, 15% of sampled cases contained the observed deficiency. This does not prove that exactly 15% of all cases were deficient. The investigator must check whether the sample frame excluded cases, whether records were independently reviewed, and whether the 80 cases represent relevant shifts, products, and time periods.

7. Measurement-System Assessment

A measurement result is not the same as the characteristic being measured. It is the result of the characteristic, the instrument, the method, the operator, the environment, and the data-processing rule.

Important questions include:

• Is the instrument suitable for the required range and resolution?
• Is it calibrated or verified against a traceable reference?
• Is there evidence of bias, drift, repeatability, or reproducibility problems?
• Were sampling location, preparation, and method standardized?
• Did environmental conditions influence the result?
• Were acceptance limits wider than measurement uncertainty?
• Could the measurement process itself have caused contamination or damage?

For repeated measurements, the sample mean is:

where xᵢ is the i-th measurement and n is the number of measurements. The mean summarizes central tendency but does not by itself establish accuracy. Repeated values may be consistently wrong if the instrument is biased.

A practical measurement-system study separates repeatability, variation when the same operator measures the same item, from reproducibility, variation between operators or measurement conditions. The exact study design should reflect the measurement method and applicable standards. Investigators should not infer that a calibration certificate proves adequate discrimination for the decision being made.

8. Timeline Reconstruction and Event Chronology

A timeline converts scattered facts into a sequence while preserving uncertainty. It should include:

• planned process milestones;
• actual actions and system events;
• alarms, changes, handoffs, and decisions;
• when information became available;
• discovery, containment, and escalation;
• gaps and conflicting timestamps.

Construct the first timeline using only information known or reasonably available at each time. Then create a separate analytical timeline incorporating later findings. This separation is a direct control against hindsight bias.

A timeline should distinguish event time from recording time. For example, a note entered at 16:00 may describe an action taken at 15:20. Do not force an exact order when the evidence supports only an interval. Use qualifiers such as “before,” “after,” “approximately,” or “order unresolved.”

9. Bias Control in Investigation

9.1 Hindsight bias

Hindsight makes an outcome appear predictable after it has occurred. Control it by asking what information was available at the decision point, what signals were observable, and what reasonable alternatives existed then.

9.2 Confirmation bias

Confirmation bias leads investigators to seek evidence supporting an initial theory. Use a hypothesis log that records the initial explanation, supporting evidence, contradicting evidence, confidence, and tests still required. Require at least one serious alternative explanation to be examined.

9.3 Attribution and fundamental attribution error

Attribution bias overemphasizes personal traits and underemphasizes system conditions. Ask whether another competent person, under the same conditions, might have acted similarly. Examine workload, interface design, training quality, incentives, supervision, equipment state, and procedure usability before assigning individual responsibility.

9.4 Other controls

Anchoring can be reduced through independent initial assessments. Availability bias can be reduced by reviewing base-rate data rather than memorable examples. Groupthink can be challenged through a devil’s-advocate review or independent peer assessment. Blinded or partially blinded data review may be appropriate when analyst expectations could influence classification.

10. Integrating Evidence into a Causal Argument

RCA should not end with a list of observations. The investigator must explain how evidence supports a causal model. A useful structure is:

1. Claim: what causal proposition is being made?
2. Evidence: which observations, records, tests, or comparisons support it?
3. Mechanism: how could the condition produce the outcome?
4. Boundary: under what circumstances would the claim not apply?
5. Alternatives: what other explanations were considered and why were they weaker?
6. Confidence: high, moderate, or low, with reasons.

Causal confidence increases when evidence is temporally appropriate, independently corroborated, mechanistically plausible, consistent across sources, and capable of distinguishing the proposed cause from alternatives. A correlation or sequence alone is insufficient to establish causation.

11. Illustrative Case: Escaped Incorrect Medication Label

Background

A hospital pharmacy released a medication package with an incorrect strength label. The error was detected on a ward before administration. No patient harm occurred, but the potential severity was high.

Problem

The initial report stated that “the technician selected the wrong label.” Management requested an RCA to determine whether retraining was required.

Analysis

The investigation preserved the electronic dispensing record, printer logs, label stock, workstation layout, shift roster, and remaining packages. Investigators interviewed the technician privately before conducting a group review.

Observation showed that two products with similar names were stored adjacent to one another. The dispensing software displayed the correct product, but the confirmation screen used abbreviated names and required a rapid sequence of keyboard commands. The printer queue retained an earlier label after a temporary network interruption. The technician reported seeing a familiar product name and did not realize that the queued label belonged to a previous transaction.

Document review found that the procedure required visual verification but did not specify verification against the original order, product package, and printed label as three distinct checks. Training records were current. Review of 200 prior transactions found two similar near misses, both during network interruptions, although the organization had not previously classified them as a common failure mode.

The measurement issue in this case was not physical measurement but information integrity: the system displayed and printed different transaction states. Timestamp comparison showed that the label was printed 41 seconds before the network recovery event, while the dispensing record was finalized afterward.

Approach and solution

The causal explanation was framed as a system interaction rather than a single-person failure:

• similar products increased selection risk;
• abbreviated display names reduced discrimination;
• the print queue did not clearly identify transaction status;
• the procedure did not define a robust reconciliation check;
• near-miss data were not aggregated by failure mode.

Corrective actions included redesigning display and label identifiers, clearing or quarantining stale print jobs after communication loss, adding an independent barcode comparison, revising the procedure, and monitoring similar near misses. Retraining was included as reinforcement, not as the principal control.

Outcome and learning points

The investigation changed the organization’s understanding from “operator selected the wrong label” to “the process permitted a stale label to appear credible.” The revised controls addressed prevention, detection, and learning. The case demonstrates why interviews, system logs, observation, and trend review must be integrated; any single source could have produced an incomplete conclusion.

12. Governance, Ethics, and Technology

Evidence collection must comply with privacy, employment, safety, records-retention, and sector-specific requirements. Access to personal data should be limited to a legitimate purpose. Investigators should avoid collecting irrelevant sensitive information and should define retention and disposal rules.

Artificial intelligence can assist with log clustering, transcription, anomaly detection, timeline extraction, and document comparison. However, AI output is an analytical aid, not automatically verified evidence. Models may reproduce classification bias, omit context, hallucinate links, or expose confidential data. Maintain the original records, document prompts and model versions where relevant, validate important outputs against source evidence, and require human accountability for conclusions.

Contemporary investigations increasingly combine operational technology, enterprise systems, sensor streams, and human accounts. This creates opportunities for richer reconstruction but also raises issues of clock synchronization, interoperability, cybersecurity, algorithmic opacity, and excessive surveillance. Good governance treats data minimization and explainability as part of evidence quality.

13. Best-Practice Protocol

A defensible investigation commonly follows this sequence:

1. Define the question, scope, impact, and decision required.
2. Preserve physical, documentary, digital, and testimonial evidence.
3. Establish an initial timeline using contemporaneous knowledge.
4. Map the intended process and observe actual work.
5. Collect interviews using open, neutral, sequenced questions.
6. Review documents, records, metadata, and change history.
7. Assess sampling adequacy and measurement-system fitness.
8. Generate competing hypotheses and record disconfirming evidence.
9. Triangulate sources and identify unresolved contradictions.
10. State causal conclusions with boundaries and confidence.
11. Separate corrective action from disciplinary judgement.
12. Archive evidence, assumptions, analyses, and decision rationale.

Challenges and Limitations

Evidence is often incomplete, contradictory, or generated by systems not designed for investigation. Memory decays, records may be copied without provenance, and rare events provide little statistical information. Observation can alter behaviour, while extensive data collection can overwhelm investigators and delay containment. A well-designed investigation does not eliminate uncertainty; it makes uncertainty visible and prevents unsupported certainty from being presented as fact.

Chapter Summary

Defensible RCA depends on disciplined evidence collection rather than attractive explanations. Investigators should define a neutral question, preserve evidence, distinguish observations from inferences, and integrate process observation, interviews, document review, digital records, sampling, measurement-system assessment, and timeline reconstruction. Sampling results require an appropriate design and should not be treated as exact population facts without qualification. Measurement systems must be assessed for suitability, bias, repeatability, reproducibility, and data integrity. Hindsight, confirmation, attribution, anchoring, and availability biases must be actively controlled through chronological reconstruction, competing hypotheses, independent review, and fair questioning. The final causal argument should identify evidence, mechanism, alternatives, boundaries, and confidence. Ethical governance, privacy, traceability, and human oversight are essential, especially when AI supports analysis.

Key Terms

Evidence: Information that bears on an investigative question and can be evaluated for reliability.

Observation: A directly perceived or recorded fact.

Inference: An interpretation drawn from evidence.

Evidence matrix: A table linking investigation questions, sources, reliability checks, and limitations.

Triangulation: Comparison of multiple independent evidence sources.

Gemba walk: Direct examination of work at the place where it occurs.

Interview bias: Distortion introduced by memory, questioning, status, or social pressure.

Data integrity: The completeness, consistency, authenticity, and accuracy of data throughout its lifecycle.

Audit trail: A record of system actions, changes, users, and timestamps.

Sampling frame: The defined population or list from which a sample is selected.

Random sampling: Selection in which population units have known selection probabilities.

Stratified sampling: Sampling separately within defined subgroups.

Measurement system: The instrument, method, operator, environment, and data process producing a measurement.

Repeatability: Variation when the same operator measures the same item using the same method.

Reproducibility: Variation associated with different operators or measurement conditions.

Hindsight bias: The tendency to view an outcome as more predictable after it occurs.

Confirmation bias: The tendency to seek or interpret evidence in support of an existing belief.

Attribution bias: The tendency to overemphasize personal causes and underemphasize situational causes.

Causal mechanism: The process by which a condition produces or contributes to an outcome.

Just culture: An approach that promotes learning and fair accountability while distinguishing error from reckless conduct.

Review Questions

Short Answer Questions

1. Distinguish an observation from an inference and provide one example of each.
2. Why should an investigator create an initial timeline using only information available at the time?
3. What is the difference between repeatability and reproducibility?
4. State two methods for reducing confirmation bias in RCA.
5. Why does a completed checklist field not necessarily prove that a task was performed correctly?

Long Answer Questions

1. Explain how an evidence matrix improves the quality and defensibility of an RCA investigation.
2. Discuss the strengths and limitations of interviews as an evidence source.
3. Explain how sampling design affects the interpretation of an observed defect proportion.
4. Describe the principal components of a measurement-system assessment.
5. Evaluate how just-culture principles should influence the treatment of individual performance during RCA.

Analytical / Application Questions

1. A production record shows a completed inspection, but a physical sample fails the specification. Design an investigation using at least four evidence sources and identify possible data-integrity risks.
2. A manager claims that a software change caused a rise in complaints because the complaints increased immediately afterward. Develop a competing-hypotheses plan that tests causation without relying on temporal proximity alone.
3. During an interview, an operator says a warning was “always ignored,” while system data show only three recorded alarms. Explain how you would reconcile the accounts, assess evidence quality, and avoid attribution bias.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes evidence from inference?

A. Evidence is always numerical, whereas inference is always verbal.

B. Evidence is an examinable source of information, whereas inference is meaning assigned to that information.

C. Evidence is obtained only from documents, whereas inference is obtained only from interviews.

D. Evidence is a conclusion, whereas inference is an observation.

Correct Answer: B. Evidence is an examinable source of information, whereas inference is meaning assigned to that information.

Question 2

Which interview question is least leading?

A. Why did you ignore the alarm?

B. You knew the procedure, so why did you skip it?

C. Please describe what happened from the beginning in your own words.

D. Did the workload cause you to make the error?

Correct Answer: C. Please describe what happened from the beginning in your own words.

Question 3

A sample contains 12 deficient cases out of 80 reviewed cases. What is the observed proportion?

A. 0.015

B. 0.067

C. 0.150

D. 0.800

Correct Answer: C. 0.150

Question 4

Which sampling method deliberately samples important subgroups separately?

A. Convenience sampling

B. Stratified sampling

C. Purposive sampling

D. Unrestricted census sampling

Correct Answer: B. Stratified sampling

Question 5

What does repeatability primarily describe?

A. Variation between different measurement operators

B. Variation caused by changes in the measured product

C. Variation when the same operator measures the same item under the same conditions

D. The difference between a measurement and a specification limit

Correct Answer: C. Variation when the same operator measures the same item under the same conditions

Question 6

Which practice most directly controls hindsight bias?

A. Interviewing only senior managers

B. Using the final outcome to identify what should have been obvious

C. Reconstructing what information was available at each decision point

D. Removing contradictory records from the analysis

Correct Answer: C. Reconstructing what information was available at each decision point

Question 7

Why is a timestamp potentially unreliable as chronological evidence?

A. All timestamps are estimates by definition.

B. System clocks may be unsynchronized or recording time may differ from event time.

C. Timestamps cannot be used in electronic systems.

D. A timestamp proves that the recorded action was performed correctly.

Correct Answer: B. System clocks may be unsynchronized or recording time may differ from event time.

Question 8

Which conclusion is most consistent with a just-culture approach?

A. Any adverse outcome proves individual negligence.

B. Training must always be the only corrective action.

C. Human error, at-risk behaviour, and reckless behaviour should be distinguished.

D. Individual accountability should never be considered.

Correct Answer: C. Human error, at-risk behaviour, and reckless behaviour should be distinguished.

Question 9

What is the principal purpose of testing competing hypotheses?

A. To guarantee that the first hypothesis is correct

B. To reduce confirmation bias and examine alternative explanations

C. To replace evidence with expert opinion

D. To avoid collecting contradictory evidence

Correct Answer: B. To reduce confirmation bias and examine alternative explanations

Question 10

Which statement about AI-generated investigation output is correct?

A. AI output automatically becomes verified evidence.

B. AI can replace human accountability for causal conclusions.

C. AI output should be validated against source evidence and governed for privacy and bias.

D. AI analysis eliminates the need for data-integrity controls.

Correct Answer: C. AI output should be validated against source evidence and governed for privacy and bias.