11. Data Warehousing, OLAP and Analytical DatabasesIntroduction and ContextOperational databases are designed to record business events efficiently. A banking system processes deposits and withdrawals, an online store records orders, and a hospital information system captures registrations, diagnoses, and treatments. These systems are optimized for frequent inserts, updates, and short transactions. Decision-makers, however, ask a different class of questions: Which products are becoming less profitable? How do customer segments behave across regions and months? What factors explain delayed deliveries? What is the long-term trend in patient readmissions? Answering such questions directly from operational databases can be inefficient and risky. Analytical queries often scan large volumes of historical data, join many tables, calculate aggregates, and compare periods. They may consume resources required by transaction-processing applications. Data warehousing addresses this problem by collecting, integrating, cleaning, organizing, and preserving data for analysis and decision support. A modern analytical environment is broader than a traditional warehouse. It may include data marts, online analytical processing (OLAP) tools, semantic layers, columnar storage, in-memory engines, cloud warehouses, data lakes, and lakehouses. The central design challenge remains the same: transform diverse operational data into trustworthy information that users can interpret and use. This chapter examines dimensional modelling, fact and dimension tables, star and snowflake schemas, data marts, semantic layers, OLAP operations, and contemporary analytical database technologies. It also considers governance, performance, security, and managerial implications. Learning ObjectivesAfter studying this chapter, the learner should be able to: 1. Explain the purpose, architecture, and characteristics of a data warehouse. 2. Distinguish operational databases from analytical databases. 3. Design fact tables, dimension tables, and dimensional models. 4. Compare star schemas, snowflake schemas, and related modelling approaches. 5. Explain data marts, semantic layers, and the ETL/ELT process. 6. Perform and interpret roll-up, drill-down, slice, dice, and pivot operations. 7. Assess columnar, in-memory, cloud, and lakehouse technologies for analytical workloads. 8. Evaluate performance, governance, security, quality, and ethical issues in decision-support systems. 9. Apply dimensional modelling principles to a realistic organizational case. From Transaction Processing to Analytical Decision SupportA transaction-processing system, commonly called an OLTP system, supports the immediate operation of an organization. Its characteristics include high concurrency, short transactions, detailed current data, and strong controls for consistency. A normalized relational design is commonly used to minimize redundancy and protect update integrity. An analytical database, by contrast, supports complex queries over large volumes of historical or integrated data. It is generally read-intensive and must deliver fast aggregation, filtering, comparison, and trend analysis. Analytical systems may tolerate some denormalization because the priority is efficient querying rather than minimizing every repeated value. The distinction is not absolute. Operational systems increasingly include analytical features, while modern analytical platforms can support near-real-time data. Nevertheless, the workload priorities differ: | Dimension | OLTP | Analytical database | |---|---|---| | Primary purpose | Run business transactions | Support analysis and decisions | | Data focus | Current operational state | Historical, integrated, and derived data | | Query pattern | Short, predictable transactions | Complex scans, joins, and aggregations | | Design tendency | Normalized | Dimensional, denormalized, or columnar | | Users | Clerks, applications, operational staff | Analysts, managers, data scientists | | Optimization goal | Transaction throughput and integrity | Query speed, scalability, and insight | A data warehouse is a subject-oriented, integrated, time-variant, and relatively non-volatile collection of data used to support management decisions. Subject orientation means that data is organized around areas such as sales, customers, finance, or supply chains rather than individual applications. Integration requires common definitions, formats, codes, and identifiers. Time variance means that history is retained so that trends and changes can be studied. Non-volatility means that data is primarily loaded and queried rather than continuously updated through operational transactions. The Analytical Data PipelineA warehouse is not created simply by copying tables from an operational database. It requires a controlled data pipeline. Traditionally, this pipeline is described as ETL: extract, transform, and load. During extraction, data is collected from enterprise resource planning systems, customer relationship management systems, point-of-sale applications, files, APIs, sensors, and external providers. Extraction may be full or incremental. Incremental extraction identifies only new or changed records, often using timestamps, change-data-capture mechanisms, transaction logs, or source-system versioning. Transformation applies business rules. Examples include standardizing dates, converting currencies, resolving duplicate customers, validating product codes, handling missing values, and deriving measures. This stage is also where organizational definitions must be made explicit. For example, “net sales” might mean gross sales less discounts and returns, but different departments may use different interpretations. Loading places prepared data into staging areas, warehouse tables, aggregates, or analytical storage. Loads may occur in batches, micro-batches, or streams. In ELT, common in cloud platforms, raw or lightly processed data is loaded first and transformed using the computational power of the target platform. A robust pipeline includes data-quality tests, reconciliation totals, error handling, lineage records, audit logs, and restart capability. A failed load should not silently produce a dashboard that appears complete but excludes an important source system. Dimensional Modelling: Facts, Dimensions, and GrainDimensional modelling organizes data so that business users can understand it and analytical queries can execute efficiently. Its two central structures are the fact table and the dimension table. A fact table records measurable business events. Typical facts include sales quantity, revenue, cost, discount, duration, balance, or number of service incidents. A fact table usually contains foreign keys to dimensions and one or more numeric measures. A dimension table describes the context of a fact. Common dimensions include date, customer, product, store, employee, supplier, location, and promotion. Dimensions contain descriptive attributes used for filtering, grouping, and labeling results. The most important modelling decision is the grain: the exact meaning of one row in a fact table. For example, “one row per product per order line” is a precise grain. “One row per order” is a different grain. Measures, keys, and relationships must be consistent with the selected grain. Many warehouse errors arise because a model combines facts with incompatible grains and thereby double-counts measures. Consider a retail sales model. A sales fact table might contain DateKey, ProductKey, StoreKey, CustomerKey, Quantity, GrossAmount, DiscountAmount, and CostAmount. Dimensions provide the product name and category, the store region, customer segment, and calendar attributes such as month, quarter, and financial year. Measures may be additive, semi-additive, or non-additive. A fully additive measure, such as sales revenue, can be summed across all relevant dimensions. An account balance is generally additive across customers but not across time; balances at different dates should not be added together. Ratios, percentages, and unit prices are non-additive and must usually be recomputed from appropriate numerators and denominators. Fact tables may represent transactions, periodic snapshots, accumulating snapshots, or factless events. A transaction fact records individual events. A periodic snapshot records a condition at regular intervals, such as daily inventory. An accumulating snapshot follows a process with milestones, such as an order moving from placement to shipment and delivery. A factless fact table records that an event or relationship occurred even when there is no numeric measure, such as student attendance or product eligibility for a promotion. Surrogate keys are warehouse-generated identifiers used in dimension tables. They separate warehouse identity from unstable source-system keys and support historical versions of dimension records. A natural key, such as a customer number assigned by an operational application, may be retained as an attribute but should not always serve as the warehouse primary key. Star and Snowflake SchemasA star schema places a central fact table around which relatively denormalized dimension tables are arranged. It is called a star because the visual structure resembles a star. The design is easy for analysts to understand: sales can be grouped by product category, store region, or calendar month with straightforward joins. The principal strengths of a star schema are usability, predictable query paths, and efficient aggregation. Its principal cost is repeated descriptive data. For example, a product dimension may store category and department names repeatedly for every product. A snowflake schema normalizes one or more dimensions into related subdimensions. A product dimension might be separated into Product, Brand, Category, and Department tables. This reduces redundancy and may improve control over shared hierarchies, but it increases the number of joins and can make self-service analysis less intuitive. The choice should be based on workload, governance, storage, and user needs. A star schema is often preferable for a business-facing semantic model. Snowflaking can be appropriate where dimensions are very large, hierarchies are complex, or reference data is managed independently. Hybrid designs are common. A dimension may also change over time. Slowly changing dimension techniques manage these changes. In a Type 1 approach, the old value is overwritten, so history is not retained. In Type 2, a new dimension row is created with effective dates or a current-record indicator, preserving historical context. Type 3 stores limited previous values in additional columns. Type 2 is particularly important when reports must reflect the organizational state that existed when a transaction occurred. Data Marts and Semantic LayersA data mart is a subject-oriented analytical store serving a department, function, or business domain. A sales mart may contain sales, product, customer, and territory data; a finance mart may focus on budgets, actuals, and cost centers. A dependent data mart is derived from an enterprise warehouse, whereas an independent mart is built directly from source systems. Data marts can deliver focused performance and faster implementation, but isolated marts may create inconsistent definitions. If one department defines “active customer” differently from another, enterprise reporting becomes difficult. Conformed dimensions—shared dimensions with consistent keys, attributes, and definitions—help integrate marts. A conformed customer or date dimension enables comparison across sales, service, and finance facts. A semantic layer presents business-friendly concepts above physical tables. It may define measures such as Net Revenue, Customer Retention Rate, or Average Resolution Time; specify permitted relationships; apply security rules; and hide technical complexity. The semantic layer is important because a technically correct SQL query can still produce a misleading result if the metric definition is unclear. Semantic models also support governed self-service analytics. Users can explore data without independently rebuilding joins and calculations. However, semantic layers must be documented and managed carefully. Poorly designed models can hide important assumptions, create ambiguous relationships, or encourage excessive abstraction. OLAP and Multidimensional AnalysisOnline analytical processing enables users to examine data across multiple dimensions. A conceptual OLAP cube may have dimensions such as time, product, geography, and customer, with measures such as revenue and quantity. The cube is a logical model; it may be implemented through precomputed aggregates, relational queries, columnar storage, or hybrid techniques. The main OLAP operations are: - Roll-up summarizes data to a higher level, such as moving from city to region or month to quarter.
- Drill-down moves from a summary to greater detail, such as from annual revenue to monthly and daily revenue.
- Slice selects one value of a dimension, such as sales for a particular year.
- Dice selects a subcube using multiple filters, such as sales for two regions, three product categories, and one quarter.
- Pivot rotates the dimensional view, changing rows and columns to reveal a different perspective.
- Drill-through moves from an aggregate result to the underlying detailed records.
OLAP may be multidimensional (MOLAP), relational (ROLAP), or hybrid (HOLAP). MOLAP stores specialized multidimensional structures and can provide rapid responses for known analyses, but processing and storage may become difficult when dimensions change frequently. ROLAP uses relational tables and SQL, offering scalability and flexibility but sometimes requiring careful tuning. HOLAP combines approaches, often storing summaries in specialized structures and detail in relational or columnar tables. OLAP is not merely a collection of buttons. The quality of analysis depends on valid hierarchies, correct aggregation rules, consistent time periods, and appropriate treatment of missing data. A roll-up from daily data to month is not necessarily valid if the measure is a daily average that should be weighted by volume. Analytical Database TechnologiesTraditional warehouses often used row-oriented relational systems. Row storage is effective when queries retrieve many columns for a small number of records, as in transaction processing. Columnar systems store values from the same column together. Analytical queries frequently select a few columns from millions or billions of rows, making columnar storage attractive. It can reduce I/O, improve compression, and accelerate vectorized aggregation. In-memory processing keeps frequently used data or intermediate results in RAM. It can reduce disk access and support interactive analysis, but memory is more expensive and volatile than persistent storage. Effective systems use caching, compression, partitioning, and workload management rather than assuming that all data can remain in memory. Massively parallel processing distributes storage and computation across nodes. Large analytical queries are divided into tasks that execute in parallel. Performance depends on partitioning, data distribution, network movement, concurrency, and query planning. A poorly chosen distribution key can create data skew, leaving one node overloaded while others remain idle. Cloud data warehouses separate, to varying degrees, storage and compute. This supports elastic scaling and consumption-based provisioning, but it introduces cost-management responsibilities. Queries that scan unnecessary data, retain unused compute clusters, or create excessive copies can produce significant expense. Governance must therefore include both performance and financial controls. Data lakes store large quantities of structured, semi-structured, and unstructured data, often in relatively raw form. They are flexible and useful for data science, exploration, and machine learning, but unmanaged lakes can become difficult to discover, trust, and govern. A lakehouse seeks to combine the flexibility of a lake with warehouse capabilities such as transactional consistency, schema management, cataloguing, and reliable analytical performance. Modern platforms also support streaming and near-real-time analytics. These are valuable for fraud detection, logistics monitoring, industrial telemetry, and personalized services. However, low latency increases architectural complexity. Systems must address out-of-order events, duplicate messages, late-arriving data, changing schemas, and reconciliation with batch history. Performance and Quality EngineeringWarehouse performance begins with correct modelling. A clearly defined grain prevents unnecessary joins and double counting. Partitioning large fact tables by date or another selective attribute can reduce the data scanned. Clustering or sorting can improve the retrieval of commonly filtered columns. Aggregation tables, materialized views, caching, and workload prioritization may further improve response time. Performance tuning should be evidence-based. Query plans, execution time, scan volume, concurrency, storage use, and resource consumption should be monitored. Indexing strategies differ between row-oriented and columnar platforms, so techniques should not be transferred uncritically from OLTP systems. Data quality is equally important. Relevant dimensions of quality include accuracy, completeness, consistency, timeliness, uniqueness, validity, and integrity. Reconciliation checks can compare warehouse totals with source-system totals. Referential-integrity tests can identify facts whose dimension keys are missing. Freshness indicators should show users when data was last successfully loaded. A warehouse should communicate uncertainty rather than conceal it. If a source system was unavailable or a late file has not been processed, dashboards should display an appropriate status. Trust is damaged when visually polished reports imply a level of precision that the underlying data does not support. Governance, Security, and Ethical ConsiderationsAnalytical databases frequently contain personal, financial, health, employment, or commercially sensitive information. Access should follow least privilege, purpose limitation, and separation of duties. Row-level and column-level security may restrict users by region, department, or sensitivity. Sensitive attributes may require masking, tokenization, encryption, or controlled environments. Governance includes ownership, stewardship, metadata, lineage, retention, classification, and approved definitions. A data catalogue should help users discover datasets, understand their meaning, assess quality, and identify responsible owners. Retention and deletion procedures should reflect legal requirements and organizational policy. Ethical concerns extend beyond unauthorized access. Historical data may encode discrimination, underrepresentation, or unequal service patterns. A predictive model trained on warehouse data can reproduce these patterns. Analysts should examine representativeness, document assumptions, test for disparate effects where appropriate, and avoid using sensitive attributes without a legitimate and carefully governed purpose. Metric governance is also an ethical issue. Changing the definition of “profit,” “customer churn,” or “successful treatment” can materially influence decisions. Definitions, calculation logic, and version changes should be documented so that performance comparisons remain honest. Applied Case: Designing an Analytical Platform for a Retail ChainBackgroundIllustrative retailer Northstar operates 120 stores and an online channel. Its operational systems include point-of-sale software, an e-commerce platform, a loyalty application, and a supply-chain system. Managers receive weekly spreadsheets, but the figures often disagree. Situation and ProblemThe chief executive wants to understand declining margins. Store managers want local inventory visibility, while marketing wants to measure promotion effectiveness. Existing reports cannot distinguish online from store sales consistently, customer identifiers vary across systems, and returns are recorded differently by channel. Directly querying the point-of-sale database has also slowed checkout operations during peak periods. AnalysisThe first modelling task is to define business processes and grains. Northstar identifies sales transactions, returns, inventory snapshots, and promotions as separate processes. The sales fact is defined at one row per sold product per order line. Returns are modelled separately because their event timing and measures differ. Inventory is a daily periodic snapshot, not a transaction fact. Conformed dimensions include Date, Product, Customer, Store, Channel, Promotion, and Geography. A surrogate CustomerKey is introduced because loyalty identifiers and online account identifiers are not stable across all systems. Product hierarchy attributes are managed in a governed product dimension. Type 2 history is selected for customer segment and store-region changes so that historical sales retain the context applicable at the time. Approach and SolutionData is extracted incrementally from each source into a cloud staging area. Standardization rules reconcile currencies, time zones, product codes, and return status. A matching process creates a mastered customer identity, while unmatched records are assigned an explicitly governed unknown member rather than being silently discarded. A star-schema sales mart is built from the enterprise warehouse. A semantic layer defines Gross Sales, Discounts, Returns, Net Sales, Cost, Gross Margin, and Margin Percentage. The model specifies that margin percentage is calculated from aggregated net margin divided by aggregated net sales, rather than averaging line-level percentages. Row-level security restricts store managers to their own stores, while executives can view enterprise results. Columnar storage and date partitioning support large sales scans. Frequently used weekly summaries are materialized, while detailed records remain available for drill-through. Data-quality checks compare daily sales totals with source-system control totals and flag missing product mappings. A dashboard displays data freshness and unresolved records. OutcomeThe redesigned platform provides a consistent view of sales and margin across channels. Managers can roll up from product to category and drill down from region to store and order line. Marketing can compare promoted and non-promoted sales, while supply-chain staff combine inventory snapshots with sales facts. The case does not assume that technology alone solves the organization’s problems: agreement on definitions, stewardship, and ownership is treated as part of the solution. Learning PointsThe case demonstrates the importance of grain, conformed dimensions, separate fact tables for different processes, governed measures, and transparent quality controls. It also shows why a semantic layer and security design should be planned with the warehouse rather than added after deployment. Common Failure Modes and Practical Best PracticesA frequent mistake is beginning with dashboard screens instead of business processes and grain. This produces tables shaped around one report and makes later analysis difficult. Another mistake is copying operational schemas without redesigning them for historical analysis. A third is creating independent departmental marts with incompatible definitions. Other common failures include treating all measures as additive, ignoring slowly changing dimensions, loading duplicate records, omitting late-arriving data, using ambiguous date fields, and failing to preserve source lineage. Performance problems often arise from scanning all columns, excessive joins, skewed partitions, poorly governed ad hoc queries, or insufficient workload isolation. Effective practice includes: - define business questions and grain before physical implementation;
- use conformed dimensions and a shared vocabulary;
- separate facts with different grains or business processes;
- make additive behaviour explicit for every measure;
- automate quality, reconciliation, lineage, and freshness checks;
- design security and retention into the architecture;
- use representative workloads for performance testing;
- document assumptions in the semantic layer; and
- balance speed of delivery with maintainability and governance.
Current DevelopmentsAnalytical architecture is increasingly moving toward unified governance across warehouses, lakes, and streaming platforms. Open table formats, metadata-driven pipelines, and interoperable catalogues seek to reduce dependence on a single storage engine. Data products and data mesh approaches emphasize domain ownership, although they still require shared standards for security, quality, and interoperability. Artificial intelligence is being applied to query assistance, anomaly detection, schema matching, data-quality monitoring, and natural-language interfaces. These tools can improve productivity, but generated SQL and automated metric interpretations must be validated. An AI assistant may select an inappropriate join, overlook slowly changing history, or confuse a ratio with an additive measure. Human accountability remains essential for consequential decisions. The strategic direction is therefore not simply toward faster databases. It is toward trustworthy, explainable, governed, and economically sustainable analytical systems that connect data engineering with organizational decision-making. Chapter SummaryData warehouses and analytical databases separate decision-support workloads from transaction processing and organize integrated, historical data for analysis. Dimensional modelling uses fact tables to record measurable events and dimensions to provide descriptive context. Grain is the foundation of a valid model, while additive behaviour determines how measures can be aggregated. Star schemas emphasize simplicity and query usability; snowflake schemas normalize selected dimensions. Data marts provide focused analytical environments, and semantic layers define governed business measures and relationships. ETL and ELT pipelines integrate source data, while OLAP operations—roll-up, drill-down, slice, dice, pivot, and drill-through—support multidimensional exploration. Columnar storage, in-memory processing, massively parallel execution, cloud warehouses, data lakes, and lakehouses provide alternative technological foundations. Their suitability depends on workload, scalability, latency, cost, governance, and data variety. High-quality analytical systems require secure architecture, lineage, validation, ethical controls, and clear ownership. The essential professional principle is that trustworthy decisions depend on both sound technology and sound definitions. Key TermsData warehouse: An integrated, historical data store designed primarily for analysis and decision support. OLTP: Online transaction processing that supports operational, short-duration business transactions. OLAP: Online analytical processing for multidimensional exploration and aggregation. Fact table: A table recording measurable business events or states. Dimension table: A table containing descriptive context used to filter and group facts. Grain: The precise meaning represented by one row in a fact table. Measure: A quantitative value used in analysis, such as revenue or quantity. Additive measure: A measure that can be summed across all relevant dimensions. Surrogate key: A warehouse-generated identifier used to identify a dimension record. Star schema: A dimensional design with a central fact table and denormalized dimensions. Snowflake schema: A dimensional design in which one or more dimensions are normalized. Data mart: A focused analytical store serving a department, subject, or business domain. Semantic layer: A governed business-facing abstraction defining entities, relationships, and measures. ETL: Extract, transform, and load data into a target analytical store. ELT: Extract, load, and transform data using the target platform’s processing capabilities. Slowly changing dimension: A dimension whose descriptive attributes change over time. Columnar storage: Storage that organizes values by column to support analytical scans and compression. Data lakehouse: An architecture combining data-lake flexibility with warehouse-style reliability and management. Conformed dimension: A shared dimension with consistent definitions across analytical models. Data lineage: Documentation of a data element’s origins, transformations, and destinations. Review QuestionsShort Answer Questions1. Why are analytical workloads usually separated from OLTP workloads? 2. What is the grain of a fact table, and why is it important? 3. Distinguish between a fact table and a dimension table. 4. What is the difference between a star schema and a snowflake schema? 5. Define a semantic layer and state one benefit it provides. Long Answer Questions1. Explain the architecture and purpose of a data warehouse, including extraction, transformation, loading, integration, and historical preservation. 2. Discuss dimensional modelling and explain transaction, periodic snapshot, accumulating snapshot, and factless fact tables. 3. Compare MOLAP, ROLAP, and HOLAP, including their respective strengths and limitations. 4. Explain how columnar storage, in-memory processing, and massively parallel processing improve analytical workloads. 5. Discuss governance, security, data quality, and ethical issues in analytical databases. Analytical / Application Questions1. Design a dimensional model for a university course-registration system. State the grain, fact tables, dimensions, and at least three measures. 2. A company averages daily margin percentages to calculate quarterly margin. Explain why this may be misleading and propose a better calculation. 3. An organization has three departmental data marts reporting different values for “active customer.” Diagnose the architectural and governance problem and recommend a solution. 22. Multiple Choice Questions with AnswersQuestion 1Which modelling decision most directly prevents double-counting in a fact table? A. Choosing an appropriate grain B. Adding more descriptive attributes C. Using only natural keys D. Removing all dimension tables Correct Answer: A. Choosing an appropriate grain Question 2A daily inventory balance is generally classified as which type of measure? A. Fully additive across time B. Semi-additive across time C. Non-numeric D. Always non-additive across every dimension Correct Answer: B. Semi-additive across time Question 3Which OLAP operation moves from annual revenue to monthly revenue? A. Slice B. Pivot C. Drill-down D. Dice Correct Answer: C. Drill-down Question 4What is a principal advantage of a columnar analytical database? A. It is optimized for updating one complete transaction row at a time B. It can scan selected columns efficiently and compress similar values C. It eliminates the need for data-quality controls D. It guarantees that every query runs in constant time Correct Answer: B. It can scan selected columns efficiently and compress similar values Question 5A new customer-dimension row is created whenever a customer’s region changes, preserving the old region for historical reports. Which technique is being used? A. Slowly changing dimension Type 1 B. Slowly changing dimension Type 2 C. Factless modelling D. Database normalization Correct Answer: B. Slowly changing dimension Type 2 Question 6Which statement best describes a semantic layer? A. A temporary area for raw files only B. A business-facing model that governs definitions, relationships, and measures C. A physical backup of an OLTP database D. A network protocol for streaming events Correct Answer: B. A business-facing model that governs definitions, relationships, and measures Question 7A retailer wants to analyse sales by product, store, customer, and date while keeping the design easy for business users. Which schema is usually the most suitable starting point? A. Star schema B. Fully normalized OLTP schema C. Network schema without dimensions D. Unstructured file collection Correct Answer: A. Star schema Question 8Which measure should generally be recalculated from aggregated values rather than averaged across rows? A. Quantity sold B. Gross revenue C. Margin percentage D. Number of orders Correct Answer: C. Margin percentage Question 9What is a major governance risk of independent departmental data marts? A. They always require columnar storage B. They may create inconsistent definitions and duplicated data C. They cannot contain historical information D. They prevent all forms of self-service analysis Correct Answer: B. They may create inconsistent definitions and duplicated data Question 10Why might a lakehouse be selected for a modern analytical platform? A. It combines flexible data-lake storage with stronger warehouse-style management and reliability B. It supports only highly structured transactional data C. It removes the need for metadata and access control D. It guarantees that raw data requires no transformation Correct Answer: A. It combines flexible data-lake storage with stronger warehouse-style management and reliability
|