Evaluation, Learning Analytics and Decision-Making

Introduction

Training evaluation is the systematic collection and interpretation of evidence about whether learning has occurred, whether it has been transferred to work, and whether it has created organisational value. It is more than asking participants whether they enjoyed a course. Effective evaluation connects learning objectives with evidence, decisions and improvement.

For human resource practitioners, evaluation serves four purposes. First, it establishes accountability for expenditure and working time. Second, it identifies whether a programme should be continued, modified or discontinued. Third, it improves the design and delivery of future learning. Fourth, it helps managers understand the conditions required for learning to influence performance.

Learning analytics extends evaluation by using data generated throughout the learning process—for example, enrolment, attendance, assessment, participation, practice and workplace-performance data. Analytics can reveal patterns, but data do not speak for themselves. Their meaning depends on valid measures, appropriate comparison groups, ethical data use and careful interpretation.

Learning objectives

By the end of this chapter, you should be able to:

• distinguish evaluation, measurement, assessment and learning analytics;
• apply Kirkpatrick’s four levels and related evaluation approaches;
• interpret completion, engagement, skills and performance evidence;
• design a practical evaluation plan using appropriate indicators;
• estimate training cost-effectiveness and return on investment;
• recognise limitations, bias, privacy risks and ethical responsibilities in people analytics; and
• make evidence-informed recommendations about learning programmes.

1. Foundations of training evaluation

1.1 Key distinctions

Measurement is the assignment of values to characteristics according to defined rules. A test score, completion percentage or error rate is a measurement. Assessment is the process of gathering evidence about knowledge, skills or performance and judging it against criteria. Evaluation is the broader judgement of the value, quality or effectiveness of a programme. Learning analytics is the collection, analysis and use of data about learners and learning contexts to understand and improve learning and its outcomes.

These concepts should not be confused. A high assessment score may show that a learner can answer questions, but it does not prove that the learner applies the skill at work. Similarly, a high completion rate may indicate persistence, but it does not demonstrate competence.

1.2 Evaluation questions

A useful evaluation begins with decisions rather than with available data. Typical questions include:

1. Did the programme reach the intended employees?
2. Did participants value the learning experience?
3. What knowledge, skills or attitudes changed?
4. Did behaviour change in the workplace?
5. Did organisational results improve?
6. Was the value obtained reasonable in relation to the cost?
7. What should be changed, expanded or stopped?

The evaluation question determines the indicator, data source, timing and method. For example, a question about immediate reaction may require a survey after a workshop, whereas a question about behaviour transfer may require observation or manager ratings several weeks later.

2. Kirkpatrick’s model and related approaches

2.1 Kirkpatrick’s four levels

Kirkpatrick’s model organises evaluation into four levels:

Reaction evaluation is useful for identifying problems in relevance, accessibility, facilitation and learning conditions. It is not a sufficient measure of effectiveness. Learners may enjoy an ineffective course, or dislike a demanding but valuable one.

Learning evaluation should use objectives that can be demonstrated. A knowledge test may be appropriate for rules or concepts; a simulation or observed performance is more suitable for communication, technical or safety skills. Pre-training and post-training measures are stronger than post-training opinions alone.

Behaviour evaluation examines transfer. It should specify the expected workplace behaviour, the context in which it should occur and the evidence that would demonstrate it. Transfer may be limited by workload, supervisor support, equipment, incentives or organisational culture, even when learning has been successful.

Results evaluation links training with organisational indicators. Results should be selected carefully because many factors influence them. A change in sales, for example, may reflect pricing, market demand, staffing or seasonality rather than training alone.

The levels are logically related but not guaranteed to form a simple causal chain. Positive reaction does not guarantee learning; learning does not guarantee transfer; and transfer does not guarantee measurable organisational results.

2.2 Kaufman’s expanded perspective

Kaufman and colleagues expanded the traditional outcomes perspective by distinguishing organisational results from wider societal value. A programme may improve internal performance while producing undesirable effects for customers, communities or the environment. This encourages HR professionals to ask whether learning contributes to responsible and sustainable outcomes, not merely short-term organisational gain.

2.3 The CIRO approach

The CIRO framework evaluates Context, Input, Reaction and Outcome. Context analysis establishes the need and desired outcomes. Input evaluation examines the quality and suitability of the programme design, resources and delivery strategy. Reaction concerns participants’ responses, while outcome evaluation examines immediate, intermediate and longer-term effects. CIRO is particularly useful before a programme begins because it emphasises needs and design, not only post-course measurement.

2.4 The CIPP model

The CIPP model evaluates Context, Input, Process and Product. Context asks what need exists; input asks what plan and resources are appropriate; process monitors implementation; and product examines outcomes. CIPP treats evaluation as a continuing management process and is useful where programmes are complex, developmental or implemented over time.

2.5 Brinkerhoff’s Success Case Method

The Success Case Method identifies particularly successful and unsuccessful cases, investigates what happened, and examines the conditions that enabled or blocked success. It does not estimate average programme impact by itself. Its strength is diagnostic: it helps explain why a programme worked for some participants and not for others.

3. Designing an evaluation system

3.1 Begin with a logic model

A logic model connects resources and activities to outcomes:

Inputs → Activities → Outputs → Learning outcomes → Workplace behaviour → Organisational results

Inputs may include trainers, technology, time and budget. Activities include instruction, practice, coaching and assessment. Outputs include attendance, completed assignments and coaching sessions. Outcomes progress from immediate learning to behaviour and results.

The model makes assumptions visible. For example, it may assume that supervisors provide opportunities to practise, that employees have access to the required systems, and that the performance indicator is sensitive to the targeted behaviour.

3.2 Align objectives, evidence and decisions

Each learning objective should have a matching assessment and workplace indicator. A useful evaluation plan specifies:

Decision rules prevent evaluation from becoming data collection without action. They should be established before results are known where possible.

3.3 Timing of evaluation

• Baseline: before training; establishes existing performance or knowledge.
• Immediate: during or immediately after training; captures reaction and short-term learning.
• Follow-up: commonly several weeks or months later; examines transfer and retention.
• Longitudinal: repeated measurement over time; examines sustainability and organisational trends.

The appropriate interval depends on the behaviour. A software procedure may be assessed within weeks, whereas leadership behaviour may require several months.

4. Measuring reaction, learning, behaviour and results

4.1 Reaction data

A reaction instrument should include more than an overall satisfaction question. It may assess relevance, clarity, psychological safety, accessibility, facilitator effectiveness, opportunity to practise and confidence to apply learning. Open comments can explain ratings, but they should be coded systematically rather than selected only when they support a preferred conclusion.

Avoid leading questions and excessive survey length. Report response rate because non-response may bias findings. For a proportion such as completion rate, the basic calculation is:

The numerator and denominator must be defined consistently. If employees are allowed to withdraw legitimately, a separate withdrawal category may be more informative than treating every non-completer as a failure.

4.2 Learning data

Learning can be assessed through written tests, oral questioning, demonstrations, simulations, portfolios and work products. Assessment quality depends on validity, reliability, fairness and alignment with the objective.

A simple gain score is:

If scores are expressed as percentages, gain is measured in percentage points, not percent. For example, an increase from 60% to 75% is a gain of 15 percentage points. A relative percentage increase would be 25%, calculated as 15 ÷ 60 × 100%, but this is a different statistic.

A normalised gain is sometimes used when the maximum possible score matters:

This measure assumes that the scores have a meaningful equal-interval interpretation and that ceiling effects are relevant. It should not be treated as universally superior to a simple gain score.

4.3 Behaviour and transfer data

Behaviour evidence should be collected from multiple sources where feasible: direct observation, work samples, system records, customer feedback and structured manager ratings. Self-report is useful for perceptions but may overestimate transfer.

A behaviour checklist should define observable standards. “Uses effective communication” is vague; “confirms the customer’s request, explains the next step and records the interaction in the approved system” is more assessable.

Transfer is affected by the work environment. A transfer audit should therefore examine resources, supervisor coaching, peer norms, workload, incentives and opportunities to practise. If employees learned a procedure but cannot access the relevant software, a low behaviour score should not be attributed automatically to poor training.

4.4 Results data

Results indicators should be operationally defined. Examples include defects per 1,000 units, average handling time, absenteeism days per employee, safety incidents, sales conversion or employee retention. Definitions, measurement periods and data sources must remain stable when comparing groups or time periods.

A result measure should be tested for sensitivity to the intervention. If a programme targets accurate data entry, customer satisfaction may be too distant or confounded to serve as the only outcome.

5. Learning analytics and data interpretation

5.1 Common data categories

Engagement data require cautious interpretation. Time online may reflect difficulty, distraction or an open browser rather than productive learning. The number of clicks is not equivalent to cognitive engagement. Meaningful interpretation combines behavioural data with assessment and contextual information.

5.2 Descriptive and comparative analysis

Descriptive analysis summarises what happened: counts, percentages, averages, medians, distributions and trends. Comparative analysis examines differences between groups or time periods. When reporting an average, analysts should consider variation and sample size, not just the highest or lowest value.

Averages can hide important patterns. If most learners score highly but a small group scores very poorly, the mean may conceal an accessibility or prerequisite problem. Segmenting results by role, location, experience or accessibility needs can reveal inequities, but small groups should be protected from identification.

5.3 Attribution and evaluation design

A before-and-after comparison is informative but vulnerable to alternative explanations. Stronger designs include:

• Comparison group: compares trained employees with similar untrained employees.
• Randomised design: assigns eligible participants randomly where ethically and operationally feasible.
• Difference-in-differences: compares change over time in a trained group with change over time in a comparison group.
• Interrupted time series: examines whether a trend changes after implementation using repeated observations.

In a difference-in-differences design, the estimated effect is:

Here, Y₁,post and Y₁,pre are outcomes for the trained group after and before training, while Y₀,post and Y₀,pre are outcomes for the comparison group. The method assumes that, without training, both groups would have followed sufficiently similar trends—the parallel-trends assumption. It also requires consistent measurement and careful consideration of other interventions.

Evaluation designs rarely prove causation perfectly in routine organisational settings. The responsible conclusion is usually that training contributed to, or is consistent with, an observed change, rather than that training alone caused it.

5.4 Dashboards and predictive analytics

A dashboard should support a defined decision. It might display participation funnel data, assessment outcomes, transfer indicators and business results, with filters for relevant populations. It should show definitions, dates, denominators and data limitations.

Predictive analytics may identify learners at risk of non-completion or low performance. Such predictions should trigger supportive intervention, not automatic punishment. Models can reproduce historical discrimination if training access, assessment or performance data reflect unequal opportunities. Human review, fairness testing and an appeal process are essential.

6. Cost-effectiveness and return on investment

6.1 Identifying costs and benefits

Training costs may include design, facilitators, materials, technology, facilities, travel, administration, learner wages and the opportunity cost of time. Costs should be classified consistently and documented.

Benefits may include avoided errors, reduced waste, increased output, improved retention, reduced accidents or additional revenue. Only benefits credibly connected to the programme should be monetised.

6.2 Cost-effectiveness

Cost-effectiveness compares cost with a non-monetary outcome, such as cost per competent employee or cost per avoided defect:

The units must be stated. A result might be £400 per employee reaching the required standard or £25 per avoided error. Lower cost per equivalent outcome is generally preferable, provided quality and equity are not reduced.

6.3 ROI

Return on investment expresses net monetary benefit relative to investment cost:

The numerator and denominator must use the same currency and time basis. If benefits and costs occur over several years, discounting may be appropriate. A positive ROI does not prove that the programme was the best option; an alternative intervention may have produced greater value or lower risk.

6.4 Worked example

An organisation spends £48,000 on a customer-service programme, including delivery, materials, administration and paid learning time. After adjusting for seasonal demand and a comparison branch, the evaluation estimates £72,000 in attributable annual benefits.

Net benefit = £72,000 − £48,000 = £24,000.

The benefit–cost ratio is:

The interpretation is that each £1 of programme cost is associated with £1.50 of estimated monetary benefit, before considering uncertainty and any unmonetised effects. The estimate should be accompanied by assumptions, confidence in attribution and sensitivity analysis. If only £60,000 of benefits can be defended, ROI falls to 25%; if benefits are £40,000, ROI is negative 16.7% approximately.

7. An illustrative case: evaluating a compliance-learning programme

Background

A fictional logistics company introduced mandatory training on safe manual handling after an increase in minor injuries. The programme consisted of a digital module, a practical demonstration and supervisor coaching.

Problem

After three months, completion was 96%, and participant satisfaction averaged 4.5 out of 5. Senior managers concluded that the programme was successful. However, injury reports had not fallen consistently across depots.

Analysis

The HR analyst examined evidence at all four Kirkpatrick levels. A knowledge test showed that the average score increased from 68% to 86%. Practical demonstrations showed that 88% met the required technique standard. Follow-up observations found that only 61% consistently used the technique. Interviews revealed that some depots lacked adjustable equipment and that supervisors prioritised speed during peak periods. Injury rates were also affected by staffing levels and changes in delivery volume.

Approach and solution

The organisation avoided blaming the training alone. It supplied appropriate equipment, briefed supervisors, introduced weekly observation with coaching, and revised the performance dashboard to distinguish training completion from safe-work behaviour. A comparison of similar depots was planned, with injury rates normalised by hours worked rather than raw counts.

Outcome

After six months, observed correct behaviour rose to 84%. The injury rate declined, although the evaluation reported the result as a contribution rather than exclusive causation because workload and staffing had also changed. The programme was retained, but resources were shifted from repeated online reminders to equipment and supervisor support.

Learning points

The case demonstrates that completion and satisfaction are weak substitutes for transfer evidence. It also shows how learning analytics can identify a performance-support problem: employees had learned the technique but lacked enabling conditions. Finally, it illustrates why results must be normalised and interpreted with contextual information.

8. Governance, ethics and professional practice

Evaluation involves employee data and therefore requires responsible governance. Organisations should collect only data necessary for a clear purpose, explain that purpose, limit access, protect data securely and retain it only as long as justified. Applicable data-protection, employment, equality and records-management requirements must be followed.

Important principles include:

• Transparency: learners should know what is collected, why and how it will be used.
• Purpose limitation: data collected to improve learning should not silently become a disciplinary surveillance system.
• Data minimisation: collect the least information needed for the decision.
• Fairness: check whether measures disadvantage groups because of language, disability, technology access or work conditions.
• Confidentiality: report small groups cautiously and anonymise data where possible.
• Validity: do not use a convenient proxy as if it directly measured competence.
• Human oversight: automated recommendations require professional review and a route for challenge.

Ethical evaluation also includes the right to fair treatment of comparison groups. If withholding training would create unacceptable risk, a delayed-training or alternative-support design may be more appropriate than a pure control group.

9. Common limitations and best practices

Evaluation can be weakened by low response rates, poorly designed tests, inconsistent definitions, small samples, self-report bias, selection bias, contamination between groups, changing business conditions and pressure to show positive results. Monetary estimates may involve uncertain assumptions, particularly for intangible outcomes such as morale or trust.

Best practice is to:

1. define the decision and success criteria before collecting data;
2. align objectives, assessments, workplace indicators and business measures;
3. establish a baseline and use comparison evidence where feasible;
4. combine quantitative and qualitative evidence;
5. distinguish association from causation;
6. report uncertainty, missing data and limitations;
7. evaluate transfer conditions, not only learner capability;
8. segment results carefully while protecting privacy;
9. use dashboards to support decisions rather than to create surveillance; and
10. close the evaluation loop by communicating findings and implementing improvements.

Chapter Summary

Training evaluation examines the quality and value of learning interventions. Kirkpatrick’s four levels—reaction, learning, behaviour and results—provide a useful organising structure, but they are not a guaranteed causal chain. CIRO, CIPP, Kaufman’s expanded approach and the Success Case Method provide complementary perspectives on context, design, implementation, outcomes and enabling conditions.

Effective evaluation aligns objectives, evidence and decisions. Completion and engagement data describe participation, while tests, demonstrations and work samples provide stronger evidence of competence. Behaviour evaluation requires observation in context and attention to transfer conditions. Organisational results must be interpreted cautiously because many factors influence them.

Learning analytics can improve visibility and decision-making, but valid definitions, comparison designs, ethical governance and human judgement are essential. Cost-effectiveness expresses cost per outcome, whereas ROI compares estimated net monetary benefit with programme cost. Both require transparent assumptions and should be supplemented by consideration of quality, risk and equity.

Key Terms

Assessment: Gathering and judging evidence against defined criteria.

Attribution: The extent to which an outcome can reasonably be assigned to an intervention.

Behaviour transfer: Application of learning in the workplace.

Baseline: Measurement taken before an intervention.

CIPP: Evaluation framework based on Context, Input, Process and Product.

Completion rate: The percentage of a defined enrolled population that completes a programme.

Cost-effectiveness: Comparison of cost with a specified non-monetary outcome.

Difference-in-differences: A comparison of changes over time between an intervention group and a comparison group.

Engagement: Observable participation or involvement in learning activities.

Evaluation: Systematic judgement of a programme’s quality, value or effectiveness.

Kirkpatrick model: Four-level framework covering reaction, learning, behaviour and results.

Learning analytics: Analysis and use of data about learners and learning contexts.

Logic model: Representation linking inputs, activities, outputs and outcomes.

Normalised gain: Learning gain adjusted for the remaining possible score.

Reaction: Participants’ responses to a learning experience.

Return on investment: Net monetary benefit expressed as a percentage of programme cost.

Results: Organisational or wider outcomes associated with learning.

Transfer climate: Workplace conditions that support or obstruct application of learning.

Validity: The extent to which evidence supports the intended interpretation or use of a measure.

Review Questions

Short Answer Questions

1. Distinguish measurement, assessment and evaluation.
2. What are the four levels of Kirkpatrick’s model?
3. Why does completion not prove learning?
4. Define cost-effectiveness in a training context.
5. State two ethical principles relevant to learning analytics.

Long Answer Questions

1. Explain how a logic model can guide the evaluation of a learning programme.
2. Compare Kirkpatrick’s model with the CIRO or CIPP framework.
3. Discuss the strengths and limitations of using learning analytics dashboards.
4. Explain why workplace conditions influence behavioural transfer.
5. Describe the information needed to produce a credible training ROI estimate.

Analytical / Application Questions

1. A programme has a 95% completion rate and excellent satisfaction ratings, but workplace performance has not changed. Analyse at least four possible explanations and recommend an evaluation plan.
2. A company reports that trained employees improved productivity by 12%, while an untrained comparison group improved by 5%. Explain how a difference-in-differences estimate could be used and identify the assumptions that must be examined.
3. Design a four-level evaluation plan for a customer-service programme. Specify one indicator, data source and timing point for each level, and explain how ethical data governance would be applied.

Multiple Choice Questions with Answers

Question 1

Which Kirkpatrick level examines whether employees apply learning in the workplace?

A. Reaction

B. Learning

C. Behaviour

D. Results

Correct Answer: C. Behaviour

Question 2

A programme enrols 80 employees and 68 complete it. What is the completion rate?

A. 80%

B. 85%

C. 88%

D. 95%

Correct Answer: B. 85%

Question 3

Which measure provides the strongest direct evidence that a practical skill has been learned?

A. A satisfaction rating

B. A login count

C. An observed demonstration scored against criteria

D. A participant’s intention to practise

Correct Answer: C. An observed demonstration scored against criteria

Question 4

A pre-test score is 60% and a post-test score is 75%. What is the gain in percentage points?

A. 15

B. 20

C. 25

D. 35

Correct Answer: A. 15

Question 5

What is a key assumption of difference-in-differences analysis?

A. All participants must receive identical training

B. The groups would have followed similar trends without the intervention

C. The post-test must be completed immediately

D. The outcome must be expressed in monetary units

Correct Answer: B. The groups would have followed similar trends without the intervention

Question 6

A programme costs £20,000 and produces estimated monetary benefits of £30,000. What is its ROI?

A. 10%

B. 33.3%

C. 50%

D. 150%

Correct Answer: C. 50%

Question 7

Which data interpretation is most appropriate?

A. More time online always means better engagement

B. High satisfaction proves organisational impact

C. Completion data alone demonstrate competence

D. Engagement data should be interpreted with assessment and context

Correct Answer: D. Engagement data should be interpreted with assessment and context

Question 8

Which framework includes Context, Input, Process and Product?

A. CIPP

B. CIRO

C. Kirkpatrick

D. Success Case Method

Correct Answer: A. CIPP

Question 9

Which practice best supports ethical learning analytics?

A. Collecting every available data point

B. Using predictions automatically for disciplinary action

C. Explaining data purposes and limiting collection to what is necessary

D. Publishing individual learner scores publicly

Correct Answer: C. Explaining data purposes and limiting collection to what is necessary

Question 10

An organisation spends £48,000 and estimates £72,000 in attributable benefits. What is the benefit–cost ratio?

A. 0.67:1

B. 1.5:1

C. 2:1

D. 24:1

Correct Answer: B. 1.5:1