Enterprise Risk Management and Risk Appetite

Introduction and context

Enterprise risk management (ERM) is the coordinated governance, identification, measurement, control, monitoring and reporting of risks that may affect a bank’s objectives. It is not simply a consolidated risk register or a compliance exercise. ERM connects risk-taking with strategy, capital, liquidity, profitability, reputation and the bank’s obligations to depositors, customers, shareholders and regulators.

Banking makes ERM especially important. Banks transform maturities, create credit and liquidity exposures, operate highly leveraged balance sheets and depend on confidence. A decision that appears profitable at business-unit level may create excessive concentration, liquidity vulnerability, conduct exposure or capital consumption at enterprise level. ERM therefore asks not only whether a transaction is profitable, but also whether it is consistent with the institution’s capacity, appetite, strategy and control environment.

A mature ERM system operates through a cycle:

1. define objectives and strategy;
2. identify and classify risks;
3. establish risk appetite and limits;
4. measure exposures and risk-adjusted returns;
5. monitor indicators and emerging threats;
6. escalate breaches and decisions;
7. report to management and the board; and
8. learn through review, stress testing and corrective action.

The central managerial challenge is to permit well-understood risk-taking while preventing uncontrolled accumulation of risk.

Learning objectives

By the end of this chapter, the reader should be able to:

• construct an enterprise risk taxonomy for a banking group;
• distinguish risk capacity, risk appetite, risk limits and risk tolerances;
• draft a clear risk appetite statement;
• design limits, key risk indicators (KRIs) and escalation protocols;
• connect ERM with strategy, budgeting and performance management;
• calculate and interpret RAROC and economic profit;
• evaluate risk reporting, stress testing and governance arrangements; and
• identify implementation, ethical, regulatory and technological challenges.

1. Enterprise risk as a strategic discipline

1.1 Risk, uncertainty and exposure

Risk is commonly understood as the possibility that an event or condition will affect objectives. In finance, risk may be represented through probability distributions of loss, but important risks are also qualitative, interconnected and difficult to quantify. Uncertainty includes situations in which probabilities are poorly known; ambiguity concerns disagreement about the meaning or consequences of information.

A bank’s exposure is the amount or sensitivity subject to a risk. For example, a loan portfolio has exposure to borrower default, a fixed-rate bond portfolio has exposure to interest-rate changes, and a digital channel has exposure to operational disruption and cyberattack. A control reduces the likelihood or impact of an event but does not eliminate risk.

1.2 Integrated versus siloed risk management

Siloed risk management assigns credit, market, operational, liquidity and compliance risks to separate specialists. Expertise is valuable, but silos can conceal correlations and risk transfers. ERM adds an enterprise perspective. For example, a credit downgrade may generate higher collateral calls, market-value losses, liquidity outflows, reputational damage and capital pressure simultaneously.

Integration does not mean that every risk must be converted into a single number. Rather, it requires common definitions, clear ownership, consistent aggregation where possible, scenario analysis where aggregation is unreliable and decision processes that recognize interdependence.

1.3 Three lines of responsibility

A practical governance model distinguishes:

• First line: business and operational management own risks arising from their activities and operate controls.
• Second line: risk management and compliance set frameworks, challenge decisions, monitor exposures and advise management.
• Third line: internal audit provides independent assurance on governance, risk management and controls.

The board remains responsible for oversight, including approval of strategy, risk appetite and the effectiveness of senior management. The model is not a substitute for accountability: assigning a risk to a function does not transfer ownership away from the business that creates it.

2. Building an enterprise risk taxonomy

A risk taxonomy is a structured classification of risk types, subtypes, drivers, events, consequences, controls, owners and measures. It creates a shared vocabulary and supports reporting, scenario analysis, data architecture and accountability.

A useful banking taxonomy may contain the following primary classes:

These categories are not mutually exclusive. A cyber incident is operational in origin but may create legal, conduct, liquidity and reputational consequences. Taxonomies should therefore support both a primary risk owner and linked consequences.

2.1 Design principles

An effective taxonomy is:

• complete: it includes material financial and non-financial risks;
• non-overlapping enough for accountability: ownership and measurement are not confused by excessive duplication;
• hierarchical: board-level categories can be decomposed into actionable subcategories;
• stable: definitions do not change merely to improve reported results;
• decision-oriented: each material risk has an owner, measure, limit and response; and
• data-compatible: risk events and exposures can be traced to systems and legal entities.

Risk identification should combine workshops, loss-event data, audit findings, regulatory publications, horizon scanning, business-plan review, stress tests and interviews with customers and frontline staff. Quantitative history alone will miss emerging risks and low-frequency, high-impact events.

3. Risk capacity, appetite, tolerance and limits

These terms are related but distinct.

• Risk capacity is the maximum risk the bank can absorb while remaining within legal, regulatory, capital, liquidity and operational constraints.
• Risk appetite is the aggregate level and types of risk the bank is willing to accept in pursuit of its objectives.
• Risk tolerance is the acceptable variation around a target or appetite position.
• Risk limit is a quantified or qualitative boundary allocated to a portfolio, product, legal entity, desk or process.
• Risk target is the desired operating level, usually below the limit.
• Risk trigger is an early-warning threshold that prompts investigation before a formal limit breach.

Capacity is a constraint; appetite is a strategic choice. A well-capitalized bank may have high capacity but choose a conservative appetite because of its mission, ownership, reputation or uncertain environment. Conversely, appetite must never exceed capacity.

A useful hierarchy is:

Figure: Risk capacity constrains appetite, which is translated into limits, targets and early-warning triggers.

4. Constructing a risk appetite statement

A risk appetite statement (RAS) communicates the risks the bank will accept, avoid or manage within boundaries. It should be approved by the board and translated into business-unit limits and decision rules. A high-quality RAS normally includes:

1. qualitative principles, such as refusal to tolerate deliberate regulatory breaches or customer exploitation;
2. quantitative measures, such as capital ratios, liquidity measures, non-performing exposure, concentration, earnings volatility and operational-loss tolerance;
3. scope, including group, subsidiary, portfolio and material legal-entity dimensions;
4. time horizon, distinguishing normal conditions from stress conditions;
5. management actions, including escalation, remediation and risk reduction; and
6. accountability, specifying owners, reporting frequency and approval authority.

A statement should be specific enough to govern decisions but not so detailed that it becomes a static catalogue. For example:

The bank will pursue diversified relationship banking and maintain sufficient capital and liquidity to withstand severe but plausible stress. It will not knowingly tolerate material breaches of law, sanctions requirements, customer-protection standards or data-security obligations. Credit growth will remain consistent with underwriting capacity, portfolio concentration limits and approved stress-loss tolerance. Material new products require documented risk assessment, control readiness and approval before launch.

This qualitative statement should be supported by a schedule of metrics. A metric without a decision rule is reporting, not appetite management.

5. Limits, KRIs and escalation

5.1 Designing effective limits

Limits should reflect the risk mechanism and decision level. Examples include single-obligor exposure, sector concentration, value at risk, duration, liquidity outflow, intraday overdraft, non-performing-loan ratio, system recovery time and unresolved high-risk audit issues.

A limit should specify the numerator, denominator, currency, aggregation method, time horizon, data source, owner and breach authority. For example, “large exposure” is inadequate unless the bank defines whether exposure is gross or net of eligible mitigation, how connected counterparties are aggregated and when collateral is recognized.

Limits may be:

• hard limits, which cannot be exceeded without formal approval;
• soft limits, where temporary excess is permitted under documented authority;
• trigger limits, which prompt pre-emptive action; and
• concentration limits, which restrict dependence on a borrower, sector, geography, product or funding source.

5.2 Key risk indicators

A KRI is a metric that signals changing risk conditions. Effective KRIs are relevant, timely, reliable, sensitive to deterioration and linked to action. They may be leading indicators—such as application exceptions, staff turnover, failed security patches or deposit concentration—or lagging indicators—such as losses, complaints, defaults and regulatory findings.

A KRI specification should document:

No single KRI is sufficient. A falling delinquency ratio could reflect genuine improvement, aggressive write-offs, reduced lending or delayed recognition. KRIs must be interpreted with context and qualitative challenge.

5.3 Escalation protocols

Escalation should be predetermined rather than improvised during a crisis. A protocol normally establishes:

• immediate notification for serious or potentially irreversible events;
• thresholds for business, risk, executive committee and board notification;
• required information, including exposure, cause, uncertainty, scenario impact and proposed action;
• decision rights for limit waivers, trading suspension, product withdrawal or contingency funding; and
• closure criteria and post-incident review.

A breach is not necessarily evidence of misconduct: markets can move rapidly and data can be revised. However, failure to detect, report or remediate a breach is itself a governance failure. Temporary waivers should have an expiry date, compensating controls and independent approval.

6. Integrating ERM with strategy and budgeting

Risk appetite becomes meaningful when it influences strategic choices. During strategy formulation, management should assess the risk implications of growth targets, acquisitions, geographic expansion, product innovation, outsourcing and technology dependence. Each strategic initiative should identify required capital, liquidity, operational capability, risk data and stress outcomes.

Budgeting should include a risk capacity allocation. Business plans should be challenged under adverse scenarios rather than evaluated only on expected income. Capital and liquidity planning should be linked to the balance-sheet forecast, including risk-weighted assets, funding maturity, collateral needs and earnings sensitivity.

Performance management should avoid rewarding gross growth without recognizing risk. Balanced scorecards can include risk-adjusted profitability, limit discipline, customer outcomes, control effectiveness and conduct measures. Incentive deferral and malus arrangements may be appropriate where losses emerge after revenue recognition.

6.1 Risk-adjusted return on capital

RAROC compares risk-adjusted income with the capital required to support risk. A common formulation is:

Expected income may include interest margin and fees after funding costs. Expected loss is the prospective average loss over the selected horizon. Operating cost and taxes must be treated consistently. Economic capital is the capital allocated to unexpected loss at a chosen confidence level and time horizon.

RAROC is not a universal accounting ratio. Its interpretation depends on the loss horizon, confidence level, diversification assumptions, allocation method and treatment of capital costs. Comparisons are valid only when methodologies are consistent.

Worked example: RAROC

A corporate loan portfolio generates interest and fee income of £12 million. Funding cost is £4 million, operating cost is £2 million, expected loss is £3 million and tax is £0.6 million. Economic capital is £20 million.

Risk-adjusted income is:

£12m − £4m − £2m − £3m − £0.6m = £2.4m

Therefore:

RAROC = £2.4m ÷ £20m = 0.12 = 12%

The result means that the portfolio produces 12% risk-adjusted return on allocated economic capital under the stated assumptions. It does not, by itself, establish that the portfolio should be expanded. Management must compare 12% with the bank’s hurdle rate, strategic value, liquidity use, concentration effects and stress performance.

6.2 Economic profit

Economic profit measures value created after charging for the capital employed:

Using the previous example and a hurdle rate of 10%:

£2.4m − (£20m × 10%) = £0.4m

The positive £0.4 million indicates value creation relative to the required return on economic capital. If the hurdle rate were 15%, economic profit would be £2.4m − (£20m × 15%) = −£0.6m, despite the same accounting income.

RAROC and economic profit answer related but different questions. RAROC is a percentage useful for comparing opportunities of different scale; economic profit is an absolute value measure useful for assessing total contribution. Both can be distorted by inaccurate expected loss, model risk, capital allocation or failure to price tail risk.

6.3 Stress testing and reverse stress testing

Stress testing examines how risk measures and strategic outcomes change under adverse assumptions. A scenario may combine recession, property-price decline, interest-rate shock, currency depreciation, cyber disruption and deposit outflows. It should be severe but plausible, with explicit assumptions and management actions.

Reverse stress testing begins with an unacceptable outcome—such as insolvency, a liquidity failure or a major conduct event—and asks which combinations of conditions could produce it. This reveals vulnerabilities that ordinary forecasting may overlook and can test the credibility of recovery options.

Stress results should influence limits, capital buffers, contingency funding plans, pricing, portfolio composition and board challenge. They should not be treated as forecasts or precise probabilities unless the methodology supports such claims.

7. Risk reporting and information quality

A board risk report should be concise but decision-useful. It should show current position, trend, appetite, limit utilization, stress results, breaches, emerging risks, management actions and unresolved data limitations. Traffic-light displays are useful for scanning but can hide nonlinear exposure and uncertainty. Commentary should explain why an indicator moved and what decision is required.

A sound report distinguishes:

• actual results from forecasts;
• gross exposure from mitigated exposure;
• expected loss from unexpected loss;
• current risk from emerging risk; and
• model output from management judgment.

Risk data aggregation requires common identifiers, legal-entity mappings, reconciliations, lineage, access controls and documented overrides. Data quality is a risk control: incomplete collateral data, stale ratings or missing related-party links can make a formally sophisticated report misleading.

8. Governance, ethics and regulation

The board should approve appetite, understand material risks and challenge optimistic assumptions. Senior management should implement the framework, allocate resources and ensure that commercial pressure does not override risk controls. Independent risk and compliance functions require appropriate authority, expertise and access to decision forums.

Ethical ERM extends beyond avoiding fines. It considers whether products are understandable, whether vulnerable customers are treated fairly, whether algorithms discriminate, whether collections practices cause avoidable harm and whether losses are transferred unfairly to depositors or taxpayers. A culture that celebrates revenue while marginalizing challenge creates “normalization of deviance,” in which repeated small exceptions become accepted practice.

Regulatory expectations commonly emphasize board accountability, capital and liquidity adequacy, operational resilience, model governance, recovery planning, outsourcing oversight, consumer protection and accurate risk data. Exact requirements vary by jurisdiction; banks must therefore map the ERM framework to applicable law, supervisory rules and recognized prudential standards rather than relying on generic labels.

9. Technology, models and artificial intelligence

Technology can improve KRI timeliness, anomaly detection, scenario analysis, limit monitoring and reporting automation. Machine-learning models may identify unusual transactions or early signs of borrower stress. However, technology introduces model, data, cyber, third-party and explainability risks.

AI used in credit, fraud, pricing or customer management should have a defined purpose, validated data, performance monitoring, bias testing, human accountability, access control and a documented override process. A model’s predictive accuracy does not prove that its decision is fair, stable under stress or legally permissible. Generative AI used for risk reporting should not be allowed to invent figures or obscure source limitations; controlled retrieval, reconciliation and human approval are essential.

10. Illustrative case: Northbridge Bank’s growth decision

Background

Northbridge Bank is a fictional mid-sized bank with strong retail deposits and a growing commercial-lending division. Its three-year strategy proposes 18% annual growth in commercial real-estate lending. The division’s historical RAROC is 14%, above the 11% hurdle rate.

Problem

The reported RAROC excludes an emerging concentration in urban offices, assumes stable collateral values and uses a two-year expected-loss estimate. Risk management identifies rising vacancy, refinancing needs and dependence on a small group of property sponsors. Funding concentration is also increasing because several large depositors account for a high share of new balances.

Analysis

The enterprise taxonomy links credit, concentration, valuation, liquidity, interest-rate and strategic risks. A baseline analysis shows attractive expected profit, but a stress scenario assumes falling property values, higher refinancing rates, increased defaults and partial deposit runoff. Expected loss rises, collateral recovery falls and economic capital increases. Under stress, RAROC falls below the hurdle rate and the bank approaches its concentration and liquidity limits.

The issue is not solved by deleting the stressed scenario or by relying on the historical average. The scenario reveals interaction: credit deterioration reduces earnings and capital, while deposit outflows increase the cost and scarcity of funding.

Approach and solution

The board approves a revised appetite: commercial-real-estate growth may continue, but only within sector, sponsor, geography and loan-to-value limits. Management introduces amber triggers based on vacancy, refinancing concentration and deposit outflow; red triggers require a credit committee review and possible origination pause. Pricing is revised to reflect expected loss, economic capital and liquidity usage. New lending is subject to stress-tested debt-service capacity, and a portion of capital is allocated to emerging concentration risk.

The risk report now shows baseline and stressed RAROC, economic profit, limit utilization, data-quality exceptions and management actions. Incentive measures include portfolio quality and limit discipline rather than volume alone.

Outcome

Growth slows to 11%, but the portfolio becomes more diversified and risk-adjusted economic profit improves. The bank exits several marginal transactions, strengthens funding contingency options and detects deteriorating sponsors earlier. The outcome illustrates that ERM may reduce nominal growth while improving resilience and sustainable value creation.

Learning points

1. Business-unit profitability can conceal enterprise concentration.
2. Risk appetite must be translated into actionable limits and triggers.
3. Stress testing should influence pricing and resource allocation.
4. RAROC is only as credible as its loss, capital and data assumptions.
5. Good governance treats challenge as part of value creation, not as an obstacle to growth.

11. Challenges and limitations

ERM faces several persistent difficulties. Risk categories may overlap, making aggregation uncertain. Correlations increase during crises and historical data may be unrepresentative. Economic capital models can create false precision. Appetite statements may become boilerplate if boards do not debate trade-offs. Limit frameworks can encourage boundary-seeking behaviour, in which managers optimize just below thresholds. KRIs may be lagging, gamed or disconnected from decisions. Excessive centralization can suppress local knowledge, while excessive decentralization can permit hidden concentrations.

There is also a fundamental tension between simplicity and completeness. A dashboard with a small number of indicators is usable but may omit material tail risks; a comprehensive dashboard may overwhelm decision-makers. The solution is layered reporting: a focused board view supported by drill-down evidence, clear escalation and periodic deep reviews.

12. Best-practice implementation roadmap

A practical implementation sequence is:

1. obtain board sponsorship and define risk principles;
2. inventory legal entities, products, processes and material exposures;
3. agree the taxonomy and risk ownership model;
4. assess capacity, vulnerabilities and strategic objectives;
5. draft qualitative and quantitative appetite statements;
6. translate appetite into limits, triggers, KRIs and escalation rules;
7. embed risk assumptions in strategy, budgeting, pricing and incentives;
8. establish data lineage, reconciliations and reporting controls;
9. conduct baseline, stress and reverse-stress analysis;
10. test the framework through incidents, limit breaches and internal audit; and
11. review appetite at least annually and whenever strategy or conditions materially change.

The framework should be proportionate. A small bank does not need the same model complexity as a global trading institution, but it does need clear accountability, reliable information and credible responses to material risks.

Chapter Summary

Enterprise risk management integrates risk identification, governance, measurement, appetite, limits, monitoring, escalation and reporting with the bank’s strategy and performance system. A risk taxonomy supplies a common structure for financial and non-financial risks, while the hierarchy of capacity, appetite, tolerance, limits, targets and triggers converts strategic intent into operational control. KRIs are useful only when they are well defined, timely, owned and linked to action.

Risk appetite statements should combine qualitative prohibitions and quantitative boundaries, apply across relevant entities and portfolios, and distinguish normal from stressed conditions. ERM becomes economically meaningful when it affects budgeting, pricing, capital allocation, incentives and product approval. RAROC measures risk-adjusted return as a percentage, whereas economic profit measures value after a capital charge. Both depend on consistent assumptions and should be supplemented by stress testing and judgment.

Effective ERM requires board oversight, independent challenge, accountable business ownership, reliable risk data and ethical attention to customers and stakeholders. Technology and AI can strengthen monitoring but create additional model, data, cyber, fairness and governance risks. The ultimate test of ERM is not the sophistication of its documents; it is whether the bank recognizes changing risk early and makes disciplined decisions before resilience is compromised.

Key Terms

Risk appetite: The aggregate level and types of risk a bank is willing to accept in pursuit of objectives.

Risk capacity: The maximum risk the bank can absorb while meeting legal, regulatory and solvency constraints.

Risk tolerance: Acceptable variation around a target or appetite position.

Risk limit: A quantified or qualitative boundary allocated to an activity, portfolio or entity.

Risk trigger: An early-warning threshold prompting investigation or action.

Enterprise risk management: Integrated governance and management of risks affecting enterprise objectives.

Risk taxonomy: A structured classification of risks, drivers, consequences, controls and owners.

Key risk indicator: A metric signalling changing exposure or control conditions.

Expected loss: The prospective average loss over a specified horizon.

Unexpected loss: Adverse loss beyond expected loss, commonly supported by capital.

Economic capital: Capital allocated for unexpected loss at a specified confidence level and horizon.

RAROC: Risk-adjusted income divided by economic capital.

Economic profit: Risk-adjusted income less the capital charge based on a hurdle rate.

Concentration risk: Risk arising from excessive dependence on a borrower, sector, geography, product or funding source.

Stress testing: Analysis of outcomes under adverse but plausible assumptions.

Reverse stress testing: Analysis beginning with a failure outcome and identifying conditions that could cause it.

Three lines model: A governance model distinguishing business ownership, risk oversight and independent assurance.

Risk data lineage: The documented path from source data through transformation to reported information.

Model risk: Risk of adverse consequences from flawed model design, data, implementation or use.

Operational resilience: The ability to continue important services through disruption and recover within acceptable parameters.

Review Questions

Short Answer Questions

1. Distinguish risk capacity from risk appetite.
2. What purposes does an enterprise risk taxonomy serve?
3. Identify three characteristics of an effective KRI.
4. Why should a risk limit include a definition of its numerator and denominator?
5. State the difference between RAROC and economic profit.

Long Answer Questions

1. Explain how a bank can translate a board-approved risk appetite statement into business-unit limits and escalation protocols.
2. Discuss the advantages and limitations of integrated ERM compared with siloed risk management.
3. Explain how risk management should influence strategy, budgeting, pricing and incentive design.
4. Evaluate the role of stress testing and reverse stress testing in enterprise risk governance.
5. Discuss the ethical and governance risks associated with using artificial intelligence in banking risk decisions.

Analytical / Application Questions

1. A portfolio has risk-adjusted income of £3 million, economic capital of £25 million and a hurdle rate of 10%. Calculate RAROC and economic profit, and interpret both results.
2. Design a KRI and escalation protocol for a bank experiencing increasing concentration in a single funding source. Specify the metric, thresholds, owner, reporting frequency and management actions.
3. Using the Northbridge Bank case, explain why business-unit RAROC may remain attractive while enterprise risk increases. Recommend two additional limits and two stress assumptions.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes risk capacity from risk appetite?

A. Capacity is the desired risk level, while appetite is the legal maximum.

B. Capacity is the maximum absorbable risk, while appetite is the risk the bank chooses to accept.

C. Capacity applies only to operational risk, while appetite applies only to credit risk.

D. Capacity and appetite are identical terms.

Correct Answer: B. Capacity is the maximum absorbable risk, while appetite is the risk the bank chooses to accept.

Question 2

Which function is primarily responsible for independent assurance under the three lines model?

A. Business and operational management

B. The treasury desk

C. Internal audit

D. The product sales team

Correct Answer: C. Internal audit

Question 3

A portfolio has risk-adjusted income of £2.4 million and economic capital of £20 million. What is its RAROC?

A. 8%

B. 10%

C. 12%

D. 20%

Correct Answer: C. 12%

Question 4

Using risk-adjusted income of £2.4 million, economic capital of £20 million and a 10% hurdle rate, what is economic profit?

A. £0.2 million

B. £0.4 million

C. £2.0 million

D. £4.4 million

Correct Answer: B. £0.4 million

Question 5

Which is the best example of a leading KRI?

A. Historical realized loan losses

B. Last year’s regulatory penalty

C. The number of failed security patches

D. Cumulative write-offs already recorded

Correct Answer: C. The number of failed security patches

Question 6

What is the principal purpose of a risk appetite statement?

A. To replace all business planning documents

B. To communicate accepted risk and translate strategy into boundaries and principles

C. To guarantee that no loss will occur

D. To calculate accounting depreciation

Correct Answer: B. To communicate accepted risk and translate strategy into boundaries and principles

Question 7

What does reverse stress testing begin with?

A. The bank’s annual budget

B. A normal operating forecast

C. An unacceptable outcome and the conditions that could cause it

D. A historical average loss only

Correct Answer: C. An unacceptable outcome and the conditions that could cause it

Question 8

Why can a cyber incident belong to more than one risk category?

A. Taxonomies must classify every risk as market risk

B. One event can have operational origins and legal, conduct, liquidity or reputational consequences

C. Cyber risk is never measurable

D. Risk categories cannot have owners

Correct Answer: B. One event can have operational origins and legal, conduct, liquidity or reputational consequences

Question 9

Which feature most improves the usefulness of a limit?

A. Leaving its calculation method unspecified

B. Defining its scope, data source, owner, threshold and breach response

C. Changing its definition whenever exposure rises

D. Reporting it only after a crisis

Correct Answer: B. Defining its scope, data source, owner, threshold and breach response

Question 10

Why should RAROC comparisons use consistent methodology?

A. Because RAROC is an accounting balance-sheet total

B. Because different horizons, confidence levels and capital allocations can produce incomparable results

C. Because RAROC excludes all expected losses

D. Because RAROC is always equal to economic profit

Correct Answer: B. Because different horizons, confidence levels and capital allocations can produce incomparable results