Data Infrastructure, Analytics and Decision Systems

Introduction

Financial institutions increasingly compete through the quality, speed and responsible use of data. A digital bank may process millions of payment events, authentication requests, account updates and customer interactions each day. These events support decisions about credit, fraud, pricing, liquidity, customer service and regulatory reporting. Yet data do not create value automatically. Value depends on architecture, governance, quality, analytical methods and the ability to embed decisions safely into operational processes.

Data infrastructure is the technological and organizational foundation through which data are collected, stored, integrated, protected and made available. Analytics converts data into descriptions, predictions and recommended actions. Decision systems connect analytical outputs to human or automated decisions. In finance, this connection must be auditable, explainable, resilient and compliant.

This chapter examines the design of modern financial data environments, including warehouses, lakes, lakehouses and real-time pipelines. It then develops descriptive, predictive and prescriptive analytics for customer, risk, profitability and operational decisions.

Learning Objectives

By the end of this chapter, readers should be able to:

1. Explain the principal components of a financial data architecture.
2. Distinguish data warehouses, data lakes, lakehouses and operational data stores.
3. Evaluate data quality, governance, lineage, privacy and security controls.
4. Explain batch and real-time data pipelines and their design trade-offs.
5. Apply descriptive, predictive and prescriptive analytics to financial decisions.
6. Interpret common performance metrics and recognize model limitations.
7. Design an analytically enabled decision process that is controlled and auditable.

1. From Data Events to Financial Decisions

A data event is a recorded occurrence, such as a card authorization, loan repayment, login, customer complaint or market-price update. Events usually contain a timestamp, an entity identifier, attributes and sometimes a location or device identifier. A transaction record is not the same as information: it becomes informative only when its meaning, quality and context are understood.

A typical decision chain is:

event → data record → curated dataset → analytical feature or measure → decision → outcome → feedback

For example, a series of missed repayments may be transformed into a delinquency feature, used by a credit model, and then incorporated into a collections strategy. The subsequent repayment outcome should return to the data environment so that the institution can evaluate the decision and improve future models.

Four properties determine whether the chain is useful:

• Timeliness: data arrive early enough to support the decision.
• Accuracy: values represent the underlying event correctly.
• Completeness: required records and attributes are present.
• Traceability: users can identify the source, transformations and responsible controls.

A technically sophisticated model cannot compensate for a poorly defined target, duplicated accounts or data collected after the decision being predicted. This is the principle of decision-relevant data quality: quality is assessed in relation to the intended use, not merely by abstract cleanliness.

2. Reference Architecture for Digital Banking Data

A modern architecture commonly contains the following layers:

1. Source systems. Core banking, cards, payments, lending, customer relationship management, mobile applications, treasury, general ledger, call centres and external data providers.
2. Ingestion. Connectors, application programming interfaces, file transfers, change-data capture and event brokers.
3. Storage. Operational databases, data warehouses, data lakes, lakehouses and backup repositories.
4. Processing. Transformation, validation, enrichment, aggregation and feature computation.
5. Metadata and governance. Catalogues, business glossaries, ownership records, lineage, classifications and access policies.
6. Analytics and decision services. Business intelligence, statistical models, machine-learning services, rules engines and optimization systems.
7. Consumption channels. Dashboards, regulatory reports, employee applications, customer interfaces and automated actions.

The architecture should separate systems optimized for transaction processing from systems optimized for analysis. Operational databases prioritize rapid, consistent updates. Analytical platforms prioritize large scans, historical analysis and integration across domains. Directly running intensive analytical queries on a core transaction system can impair customer-facing services.

Warehouses, lakes and lakehouses

A data warehouse stores integrated, structured and governed data for reporting and analysis. It often applies an extract-transform-load or extract-load-transform process and uses defined schemas. It is appropriate for financial reporting, management information and standardized performance measures.

A data lake stores large volumes of structured, semi-structured and unstructured data, often in relatively raw form. It supports exploratory analysis, machine learning and retention of detailed event histories. Without governance, however, a lake can become a “data swamp,” containing duplicated, undocumented or unreliable assets.

A lakehouse combines low-cost, flexible storage with warehouse-like management features such as schema enforcement, transaction controls and data-quality expectations. The distinction is architectural rather than absolute; products and designs vary.

An operational data store provides a current, integrated view for operational use, often with lower latency than a warehouse but less historical depth. A data mart is a subject-oriented subset, such as a credit-risk or profitability repository.

Data modelling

A relational model represents entities in tables connected through keys. In analytical environments, a star schema commonly contains a central fact table, such as transactions, linked to dimension tables such as customer, account, product, branch and date. A transaction fact may include amount, fee, currency and quantity, while dimensions provide descriptive context.

Designers must define the grain of a fact table explicitly. “One row per card authorization” differs from “one row per settled card transaction.” Mixing grains causes double counting. Measures must also be classified as additive, semi-additive or non-additive. Transaction amounts may be additive across transactions; account balances may be additive across accounts but not across time; ratios generally should be recomputed from their components rather than summed.

3. Pipelines and Real-Time Data

A data pipeline moves data from source to destination while applying controls. Batch pipelines process accumulated records at scheduled intervals. Streaming pipelines process events continuously or in short windows.

Batch processing is economical and suitable for daily profitability, monthly regulatory returns and end-of-day reconciliation. Streaming is appropriate for card fraud detection, payment monitoring, cybersecurity and real-time customer communication. It introduces challenges involving event ordering, duplicate delivery, late-arriving data, temporary outages and state management.

Important pipeline design concepts include:

• Idempotency: processing the same event more than once does not change the final result beyond the first valid processing.
• Exactly-once business effect: although infrastructure may deliver messages more than once, controls ensure that the business outcome occurs once.
• Event time: when the event happened, as distinct from processing time.
• Watermark: an operational estimate of how complete event-time data are for a given period.
• Schema evolution: controlled changes to data structures without unexpectedly breaking downstream users.
• Dead-letter handling: isolation of records that fail validation so that they can be investigated rather than silently discarded.

For a queue with arrival rate λ and service rate μ, a basic stability condition is:

Here λ and μ are measured in events per unit of time. The relationship applies to a simplified single-server queue under stable operating assumptions; real systems require capacity planning for bursts, multiple workers and failure recovery. If λ approaches μ, delays can increase sharply even when average capacity appears adequate.

A pipeline service-level objective might specify that 99% of payment events become available for fraud scoring within a defined number of seconds. This is an operational commitment, not merely a technical statistic; it should be linked to the financial loss and customer impact of delay.

4. Governance, Lineage, Quality and Privacy

Data governance

Data governance allocates decision rights and accountability. A robust governance model usually identifies:

• Data owner: accountable for business meaning, acceptable use and risk.
• Data steward: manages definitions, quality rules and issue resolution.
• Data custodian: operates storage, access and technical controls.
• Data consumer: uses data within approved purposes.

A business glossary should define terms such as “active customer,” “default,” “available balance,” “gross revenue” and “chargeback.” Definitions should include scope, time period, currency, exclusions and authoritative source. A single metric may legitimately have different versions for finance, risk and marketing, but the differences must be explicit.

Lineage

Data lineage records where data originated, how it changed and where it was used. Technical lineage follows tables, fields, jobs and interfaces. Business lineage explains how a reported or modelled measure relates to a business concept. Lineage is essential for impact analysis, regulatory response, model validation and incident investigation.

For a credit decision, lineage should make it possible to identify the source of each material input, the transformation date, the model version, the policy threshold and the resulting customer action.

Data quality

Common dimensions are accuracy, completeness, consistency, timeliness, validity, uniqueness and integrity. Quality rules should be measurable. For example:

In ordinary text, the same metric is: completeness rate = non-missing required values divided by all required values, multiplied by 100%. The denominator must be defined, and the rule must distinguish legitimately inapplicable values from missing values.

Other useful measures include duplicate rate, reconciliation difference, invalid-code rate and freshness delay. A quality dashboard should show thresholds, trends, affected consumers and remediation owners. Controls should operate at ingestion, transformation and consumption stages.

Privacy and security

Financial data frequently contain personal, sensitive and legally protected information. Privacy-by-design principles include purpose limitation, data minimization, retention limits, transparency, accuracy, access rights and secure disposal. Pseudonymization replaces direct identifiers with controlled tokens, but it does not guarantee anonymity if other attributes permit re-identification.

Security controls include encryption in transit and at rest, strong authentication, least-privilege access, segregation of duties, privileged-access monitoring, key management, tokenization and tested recovery procedures. Access should be evaluated by role, purpose, data sensitivity and sometimes geographic location. Analysts should ordinarily receive only the fields and level of detail necessary for the approved use.

Privacy and analytics can conflict. A highly granular dataset may improve fraud detection while increasing surveillance and re-identification risk. Governance therefore requires proportionality, documented purpose and review of both benefits and harms.

5. Analytics for Financial Management

Descriptive analytics

Descriptive analytics answers “what happened?” It includes reports, dashboards, segmentation, cohort analysis and variance analysis. Examples include approval rates, customer activity, net interest income, complaints per thousand accounts, fraud losses and processing times.

A mean is calculated as:

where xᵢ is observation i and n is the number of observations. The mean is sensitive to extreme values; medians and percentiles may better describe skewed transaction values or waiting times.

A conversion rate, for example, is:

The numerator must be a subset of the denominator and the observation window must be consistent. A dashboard should expose definitions rather than present percentages without context.

Profitability analysis may use contribution margin:

For a customer, revenue may include fees and allocated interest income, while variable costs may include payment-network charges, expected credit loss allocation and service costs. Allocation methods are assumptions, not neutral facts.

Predictive analytics

Predictive analytics estimates an unknown or future outcome. A binary credit or fraud outcome may be modelled using logistic regression:

Here Y is the binary outcome, X is the vector of predictors, β₀ is the intercept, βⱼ are estimated coefficients, and e is the base of natural logarithms. The formula produces a value between 0 and 1. It assumes a log-odds relationship that is linear in the predictors and requires appropriate sampling, target definition and validation. A predicted probability is not automatically a decision; a threshold and policy are also required.

For a lending portfolio, expected loss is commonly represented as:

PD is probability of default, LGD is loss given default expressed as a proportion of exposure, and EAD is exposure at default in currency units. The simplified expression assumes compatible definitions, a common time horizon and adequate treatment of discounting, recoveries and dependence. If PD = 2%, LGD = 40% and EAD = $50,000, expected loss is 0.02 × 0.40 × $50,000 = $400.

Model evaluation must match the decision. Classification metrics include precision, recall, specificity, area under the receiver operating characteristic curve and calibration. Calibration asks whether cases assigned a probability of 20% experience the event approximately 20% of the time in a suitable population. A model can rank customers well while producing poorly calibrated probabilities.

Validation should use out-of-time data where possible, because random splits can conceal changing economic conditions. Analysts should test drift, stability, subgroup performance, missingness, leakage and adverse effects. Data leakage occurs when information unavailable at decision time enters model development, producing deceptively strong results.

Prescriptive analytics

Prescriptive analytics addresses “what should be done?” It combines predictions, constraints, objectives and possible actions. A general optimization problem is:

Here x represents decision variables, f(x) is the objective, gₖ(x) are constraint functions, bₖ are permitted bounds and m is the number of constraints. In banking, x might represent credit limits, collections capacity, liquidity allocation or fraud-review assignments. The objective must include relevant costs, revenues, risk limits and service commitments.

For instance, assigning investigators to alerts should not simply maximize the number of reviewed alerts. It may optimize expected prevented loss subject to staff hours, response-time requirements, legal constraints and minimum coverage of high-risk segments. Human review remains important where consequences are material or explanations are required.

6. Decision Systems and Model Risk

A decision system may consist of rules, a statistical model, a policy layer, a workflow engine and a monitoring service. Separating these components improves governance. A model estimates risk; policy determines what risk is acceptable; a workflow determines who acts and records the outcome.

Decision systems should provide:

• versioned inputs, models, rules and outputs;
• reason codes or explanations appropriate to the use;
• approval and change-control records;
• access and override logs;
• fallback procedures when data or services fail;
• monitoring for performance, fairness, drift and operational incidents.

Model risk arises from incorrect specifications, unsuitable data, implementation errors, misuse, changing populations and inadequate monitoring. Independent validation should examine conceptual soundness, data, implementation, performance, limitations and intended use. A model should have an owner, inventory entry, risk rating, validation schedule and retirement criteria.

Fairness analysis requires careful choice of population, outcome, metric and legal context. Equal error rates, equal selection rates and calibrated probabilities cannot generally all be achieved when groups differ in base rates. Therefore, fairness is a governance and policy question, not a single technical score. Human oversight must be meaningful rather than a nominal approval step.

7. Illustrative Case: Meridian Digital Bank

Background

Meridian Digital Bank, a fictional institution, offers instant consumer payments and unsecured loans. It stores payment events in a cloud data lake, maintains a warehouse for finance and uses a real-time fraud service. Management observes rising fraud losses, inconsistent customer profitability reports and long delays in regulatory data requests.

Problem

Investigation finds three issues. First, the mobile application and card processor use different customer identifiers, causing duplicate profiles. Second, the profitability dashboard combines settled transactions with authorization records, double counting some revenue. Third, the fraud model uses a device attribute that is sometimes updated after the transaction, creating data leakage during testing.

Analysis

A cross-functional data council maps critical data elements and defines a canonical customer and transaction identifier. Lineage analysis identifies the dashboard’s conflicting sources. A time-based model-validation design is introduced: every feature must be available at the instant of the fraud decision. The bank also evaluates precision, recall, calibration, review capacity and subgroup impacts rather than maximizing a single metric.

Approach and solution

Meridian establishes a curated transaction layer with documented grain: one record per settled transaction, linked to authorization events through a controlled relationship. A reconciliation control compares daily settled totals with the general ledger. A streaming pipeline scores events using event-time data, while late corrections are processed through a controlled adjustment process. The model produces a risk score, but a policy layer sets review thresholds based on expected loss and available investigator capacity. High-impact declines receive reason codes and an appeal pathway.

For profitability, the bank creates a governed metric catalogue. Customer contribution is reported with separate views for direct revenue, allocated funding cost, expected loss and service cost. Allocation assumptions are reviewed by finance and risk.

Outcome

The bank reduces duplicate customer records, removes double counting and improves the auditability of fraud decisions. It does not treat the model as a complete solution: operational controls, staff training, customer communication and monitoring remain necessary. The case illustrates that architecture, definitions and governance often create more value than selecting a more complex algorithm.

Learning points

1. Data grain and identifiers are foundational controls.
2. A model must use only information available at decision time.
3. A metric requires a definition, owner and reconciliation method.
4. Real-time systems require explicit treatment of late and duplicate events.
5. Responsible automation combines prediction with policy, explanation and appeal.

8. Contemporary Developments and Best Practice

Data products are increasingly organized around domains such as payments, lending and customers. A domain team publishes a reliable, documented product with an owner and service expectations. This can reduce dependence on a single centralized team, but federated ownership requires common standards and strong governance.

Data mesh principles emphasize domain ownership, data as a product, self-service infrastructure and federated computational governance. They are not a replacement for architecture; they redistribute responsibility. Similarly, generative AI can assist with data discovery, documentation and analyst productivity, but generated code and explanations require testing, access control and human review. Sensitive financial data should not be sent to an unapproved model service.

Practical best practices include:

• begin with critical decisions and data elements rather than collecting data indiscriminately;
• document definitions, grain, ownership, lineage and permitted uses;
• design for batch and real-time needs separately where their reliability requirements differ;
• reconcile analytical totals to authoritative financial records;
• test models out of time and monitor drift after deployment;
• separate model outputs from policy decisions;
• minimize personal data and enforce retention schedules;
• maintain human escalation for consequential or uncertain cases;
• rehearse outage, cyberattack, corruption and recovery scenarios;
• measure business outcomes, not only technical accuracy.

Challenges and Limitations

Data architecture involves trade-offs among latency, cost, flexibility, consistency and control. More data can increase analytical power while increasing privacy, storage and governance burdens. Real-time processing may improve fraud response but can be less complete than a reconciled batch view. Complex models may improve discrimination but reduce interpretability and operational resilience.

Analytics also faces selection bias, changing customer behaviour, confounding, feedback loops and measurement error. If a bank offers credit only to customers predicted as safe, observed repayment data may not reveal how rejected customers would have behaved. If a fraud model sends certain communities for greater scrutiny, the resulting investigation data may reflect the model’s own decisions rather than underlying fraud alone.

Consequently, data-driven management should combine statistical evidence with domain knowledge, control testing, qualitative investigation and professional judgement. Good infrastructure makes uncertainty visible; it does not eliminate uncertainty.

Chapter Summary

Financial data infrastructure connects source events to governed analytical and operational decisions. Warehouses provide curated reporting, lakes provide flexible retention, and lakehouses seek to combine flexibility with stronger controls. Effective pipelines address latency, duplicates, ordering, schema change, failure and reconciliation. Governance establishes ownership, definitions, lineage, quality rules, privacy protection and access accountability.

Descriptive analytics explains past performance; predictive analytics estimates future or unknown outcomes; prescriptive analytics selects actions under objectives and constraints. In finance, expected-loss calculations, profitability measures, risk models and optimization support customer, credit, fraud and operational decisions. However, models require appropriate targets, time-valid data, independent validation, calibration, monitoring and fairness review. Decision systems must combine models with policy, workflow, explanation, logging and human oversight.

The central managerial lesson is that reliable decisions depend on the entire data lifecycle. Architecture and governance are not administrative afterthoughts: they are core components of financial control, competitive capability and responsible digital banking.

Key Terms

Data architecture: The structure of systems, storage, pipelines, metadata and interfaces used to manage data.

Data warehouse: A governed analytical repository containing integrated, structured data.

Data lake: A scalable repository for raw and varied data formats.

Lakehouse: An architecture combining flexible lake storage with warehouse-like management controls.

Data lineage: The record of data origins, transformations and destinations.

Data governance: Allocation of authority, accountability, standards and controls for data.

Data quality: The fitness of data for a defined purpose, including accuracy and completeness.

Event time: The time at which a business event occurred.

Idempotency: The property that repeated processing does not create additional unintended effects.

Schema evolution: Controlled change to the structure or meaning of data.

Descriptive analytics: Analysis that summarizes what has happened.

Predictive analytics: Analysis that estimates future or unknown outcomes.

Prescriptive analytics: Analysis that recommends actions under objectives and constraints.

Data leakage: Use of information in model development that was unavailable at decision time.

Calibration: Agreement between predicted probabilities and observed frequencies.

Model risk: Risk of loss or harm arising from model error, misuse or changing conditions.

Expected loss: Estimated loss commonly expressed as PD × LGD × EAD.

Data minimization: Limiting collection and use to what is necessary for an approved purpose.

Review Questions

Short Answer Questions

1. What is the difference between a data warehouse and a data lake?
2. Why must the grain of an analytical fact table be documented?
3. Define data lineage and explain one financial use of it.
4. What is data leakage in predictive modelling?
5. Distinguish descriptive, predictive and prescriptive analytics.

Long Answer Questions

1. Explain the layers of a reference data architecture for a digital bank and discuss the purpose of each layer.
2. Compare batch and streaming pipelines, including their operational risks and suitable financial applications.
3. Discuss how governance, privacy and security controls should be integrated into an analytical platform.
4. Explain the expected-loss formula and evaluate the assumptions required for its use in lending decisions.
5. Critically assess the role of model validation, monitoring and human oversight in automated banking decisions.

Analytical / Application Questions

1. A fraud model has high ranking performance but is poorly calibrated and uses a device feature updated after the transaction. Diagnose the problems and propose a validation plan.
2. Design a governed profitability data product for a digital bank. Specify its grain, sources, quality controls, ownership, lineage and principal measures.
3. A bank must allocate a limited team of investigators across fraud alerts. Formulate an objective and constraints for a prescriptive decision system, and explain what ethical and operational safeguards are required.

Multiple Choice Questions with Answers

Question 1

Which platform is primarily optimized for governed, integrated analytical reporting?

A. Operational database

B. Data warehouse

C. Event broker

D. Dead-letter queue

Correct Answer: B. Data warehouse

Question 2

What does idempotency mean in a data pipeline?

A. Every event is processed in random order

B. A repeated event creates a new business effect each time

C. Repeating processing does not create unintended additional effects

D. All events are permanently deleted after processing

Correct Answer: C. Repeating processing does not create unintended additional effects

Question 3

A queue receives events at λ = 900 events per second and can process μ = 1,000 events per second under stable conditions. Which conclusion is correct?

A. The basic stability condition is satisfied

B. The queue is necessarily losing 100 events per second

C. The service rate is lower than the arrival rate

D. The system cannot process burst traffic

Correct Answer: A. The basic stability condition is satisfied

Question 4

If PD = 2%, LGD = 40% and EAD = $50,000, what is the simplified expected loss?

A. $40

B. $200

C. $400

D. $4,000

Correct Answer: C. $400

Question 5

Which analytical approach answers the question “What should be done?”

A. Descriptive analytics

B. Predictive analytics

C. Prescriptive analytics

D. Data lineage

Correct Answer: C. Prescriptive analytics

Question 6

Why is a time-based validation split often important for banking models?

A. It guarantees that the model is unbiased

B. It tests performance under temporal change and better reflects deployment

C. It removes the need for a target definition

D. It ensures every subgroup has identical outcomes

Correct Answer: B. It tests performance under temporal change and better reflects deployment

Question 7

Which example best illustrates data leakage?

A. Using a customer’s age recorded before loan approval

B. Using a transaction amount at the time of authorization

C. Using a device attribute updated after the fraud decision to predict that decision

D. Using historical repayment status from the training period

Correct Answer: C. Using a device attribute updated after the fraud decision to predict that decision

Question 8

Who is generally accountable for the business meaning and acceptable use of a data asset?

A. Data owner

B. Network router

C. External auditor only

D. End-user device

Correct Answer: A. Data owner

Question 9

Which measure is generally not additive across time?

A. Number of transactions

B. Total fee revenue

C. Account balance

D. Units sold

Correct Answer: C. Account balance

Question 10

What is the principal purpose of a business glossary?

A. To encrypt all customer records

B. To define and standardize important business terms and metrics

C. To replace disaster recovery procedures

D. To increase network bandwidth

Correct Answer: B. To define and standardize important business terms and metrics