Foundations of Data Mining and Knowledge Discovery

Introduction

Modern organizations generate data continuously through transactions, web activity, sensors, mobile devices, scientific instruments, enterprise systems, and social platforms. The existence of data, however, does not automatically create knowledge. Data must be prepared, examined, modeled, interpreted, and connected to decisions. Data mining provides a systematic set of computational and analytical methods for discovering useful patterns in data.

Data mining is often discussed together with knowledge discovery in databases, machine learning, statistics, and analytics. These fields overlap, but they are not identical. A sound understanding of their relationships is essential because successful data-mining projects require more than selecting an algorithm. They require a clear problem definition, appropriate data, valid evaluation, organizational acceptance, ethical safeguards, and deployment into practice.

This chapter introduces the conceptual foundations of data mining and knowledge discovery. It explains descriptive and predictive tasks, distinguishes data mining from machine learning and analytics, presents the CRISP-DM and SEMMA process models, and develops a practical method for identifying objectives, data requirements, constraints, and success criteria.

Learning Objectives

After studying this chapter, you should be able to:

1. Define data mining, knowledge discovery, machine learning, and analytics.
2. Explain the relationship between data, information, patterns, knowledge, and decisions.
3. Distinguish descriptive, diagnostic, predictive, and prescriptive analytical tasks.
4. Classify common data-mining tasks, including classification, regression, clustering, association analysis, and anomaly detection.
5. Describe the stages of CRISP-DM and SEMMA and compare their strengths.
6. Translate a business or organizational problem into a data-mining problem.
7. Identify data requirements, constraints, risks, and measurable success criteria.
8. Explain important ethical, legal, operational, and technical limitations of data mining.

From Data to Knowledge

A useful conceptual hierarchy is:

• Data: recorded observations, such as transaction amounts, timestamps, product identifiers, or sensor readings.
• Information: organized data that answers a basic question, such as monthly revenue by region.
• Pattern: a regularity, relationship, or structure identified in data, such as customers who purchase a printer frequently purchasing ink.
• Knowledge: an interpreted and validated pattern that supports understanding or action.
• Decision: a choice made using knowledge, such as offering a retention incentive to a customer segment.

The hierarchy is not completely linear. Domain knowledge influences what data are collected, which patterns are considered meaningful, and how results are interpreted. A statistically strong pattern may be operationally irrelevant, while a modest pattern may be valuable if it supports a high-impact decision.

Data mining therefore involves both technical and human processes. Algorithms can detect correlations, clusters, rules, and predictive relationships, but subject-matter experts must judge whether those results are plausible, useful, ethical, and actionable.

What Is Data Mining?

Data mining is the computational process of discovering valid, novel, useful, and understandable patterns in large or complex datasets. The definition emphasizes four qualities:

• Validity: the pattern should be supported by appropriate evidence and should not be an artifact of errors or random variation.
• Novelty: the pattern should provide information that was not already obvious or known.
• Usefulness: the pattern should support a meaningful decision, explanation, prediction, or process improvement.
• Understandability: users should be able to interpret the result sufficiently for its intended purpose.

Data mining can be applied to structured tables, text, images, graphs, time series, spatial data, and streaming data. Typical tasks include predicting whether a loan applicant will default, grouping customers by behavior, identifying fraudulent transactions, recommending products, and discovering relationships among biological variables.

Data mining is not simply the same as querying a database. A query retrieves information specified in advance; data mining searches for regularities that may not have been explicitly specified. It is also not equivalent to producing a chart. Visualization is an important tool for exploration and communication, but data mining generally involves systematic pattern discovery, modeling, or both.

Knowledge Discovery in Databases

Knowledge discovery in databases, commonly abbreviated KDD, is the broader end-to-end process of obtaining useful knowledge from data. Data mining is usually treated as one central step within KDD.

A conventional KDD process includes:

1. Understanding the application domain: learning the organizational context, terminology, objectives, and decision processes.
2. Selecting data: identifying relevant records, variables, time periods, and sources.
3. Cleaning data: addressing errors, missing values, duplicates, inconsistent formats, and outliers.
4. Integrating data: combining information from multiple sources while preserving meaning and lineage.
5. Transforming data: constructing features, scaling values, encoding categories, or aggregating observations.
6. Mining data: applying algorithms to discover patterns or build models.
7. Evaluating patterns: testing technical validity, practical value, stability, and ethical acceptability.
8. Presenting and using knowledge: communicating findings and incorporating them into decisions or systems.

These stages are iterative rather than strictly sequential. Evaluation may reveal that the wrong population was selected; domain review may suggest a new feature; deployment may expose data-quality problems. Effective projects repeatedly move between technical analysis and domain understanding.

Figure: KDD as an iterative process from domain understanding to action and feedback.

Data Mining, Machine Learning, Statistics, and Analytics

These disciplines have different emphases:

The boundaries are porous. A data-mining project may use machine-learning algorithms, statistical tests, database technologies, and visualization. Machine learning often emphasizes predictive performance, whereas data mining also emphasizes pattern discovery and human interpretation. Statistics provides principles for uncertainty and sampling, but not every data-mining objective is inferential. Analytics connects methods to organizational action and value.

A model that predicts accurately is not necessarily causal. If two variables are associated, changing one may not cause the other to change. Confounding variables, selection bias, measurement error, and temporal effects can produce misleading relationships. Consequently, data-mining findings should be described as associations or predictions unless a suitable causal design supports causal interpretation.

Descriptive, Predictive, and Prescriptive Tasks

Descriptive analytics

Descriptive tasks summarize what has happened or what is currently present. Examples include sales dashboards, customer segmentation, frequency distributions, and association rules. Descriptive mining is often exploratory: it helps analysts understand structure before formal modeling.

Common descriptive tasks include:

• Clustering: dividing observations into groups whose members are relatively similar according to selected variables.
• Association analysis: identifying items or events that occur together.
• Summarization: producing compact descriptions of records or groups.
• Anomaly detection: identifying observations that differ substantially from expected behavior.

Predictive analytics

Predictive tasks estimate an unknown or future target using known explanatory variables. The target may be categorical or numerical.

• Classification predicts a category, such as approved or rejected.
• Regression predicts a numerical value, such as delivery time or demand.
• Time-series forecasting predicts future values using temporal structure.

Predictive performance must be assessed on data that were not used to fit the model. Otherwise, performance estimates may be excessively optimistic because the model has partially memorized the training data.

Prescriptive analytics

Prescriptive analytics recommends actions by combining predictions with objectives, constraints, and decision rules. For example, a model may predict demand, while an optimization procedure recommends inventory levels subject to storage and budget limits. Data mining can contribute to prescriptive systems, but prediction alone does not determine the best action.

Common Data-Mining Tasks

Classification

Classification learns a mapping from input variables to a finite set of classes. A training dataset contains examples with known class labels. The learned model is then applied to new cases. Applications include spam detection, medical screening, credit-risk categorization, and image recognition.

Important concerns include class imbalance, unequal error costs, threshold selection, interpretability, and fairness. Accuracy alone can be misleading when one class is much more common than another.

Regression

Regression estimates a continuous numerical target. Examples include predicting electricity consumption, house prices, or processing time. Evaluation may use mean absolute error, root mean squared error, or another measure appropriate to the application. The choice of metric should reflect the cost of errors and the scale of the target.

Clustering

Clustering is an unsupervised task because class labels are not supplied. The result depends on the representation of observations, the distance or similarity measure, the algorithm, and the chosen number of clusters. A cluster is a mathematical grouping, not automatically a naturally existing category. Domain experts must determine whether the grouping is stable and useful.

Association-rule mining

Association analysis discovers co-occurrence relationships, often in transaction data. A rule has the form “if X occurs, Y tends to occur.” Three commonly used measures are support, confidence, and lift.

For a rule X → Y, with N transactions:

Here, count(X) is the number of transactions containing X, count(X ∪ Y) is the number containing both X and Y, and N is the total number of transactions. These measures assume a clearly defined transaction population. Lift greater than 1 indicates positive association relative to the baseline occurrence of Y, but it does not establish causation.

Worked example. Suppose 1,000 transactions exist. Bread is present in 200, butter in 300, and both in 120. For the rule bread → butter:

• Support = 120 ÷ 1,000 = 0.12, or 12%.
• Confidence = 120 ÷ 200 = 0.60, or 60%.
• Lift = 0.60 ÷ 0.30 = 2.0.

Thus, 12% of all transactions contain both items; 60% of bread transactions contain butter; and butter is twice as common among bread transactions as in the full transaction population. The result remains an association, not proof that buying bread causes the purchase of butter.

Anomaly detection

Anomaly detection identifies observations that are rare, inconsistent, or unusually distant from a reference distribution. Applications include fraud detection, network intrusion detection, equipment monitoring, and quality control. A rare observation is not necessarily erroneous or malicious; investigation is required. Thresholds should account for changing conditions and the cost of false alarms.

Process Frameworks for Data-Mining Projects

CRISP-DM

CRISP-DM, or Cross-Industry Standard Process for Data Mining, is a widely used process model with six phases:

1. Business understanding: define objectives, constraints, stakeholders, and success criteria.
2. Data understanding: collect initial data, describe variables, explore distributions, and identify quality problems.
3. Data preparation: select, clean, integrate, construct, and format data for modeling.
4. Modeling: choose techniques, generate models, and tune parameters.
5. Evaluation: assess whether the model meets technical and business requirements and whether risks remain.
6. Deployment: deliver reports, decision rules, applications, or monitoring processes.

CRISP-DM is valuable because it begins with the problem rather than the algorithm. Its phases are iterative. For instance, evaluation may show that the target variable is poorly defined, requiring a return to business understanding or data preparation.

SEMMA

SEMMA is an analytical process associated with SAS and consists of:

• Sample: create one or more representative samples.
• Explore: investigate relationships, distributions, missingness, and outliers.
• Modify: transform variables and engineer useful features.
• Model: fit models using suitable techniques.
• Assess: compare performance and determine usefulness.

SEMMA provides a practical modeling workflow, particularly for data preparation and model comparison. Compared with CRISP-DM, it gives less explicit attention to business understanding and deployment. The frameworks can be used together: CRISP-DM can structure the overall project, while SEMMA-like activities can guide exploration and modeling.

Translating Objectives into a Data-Mining Problem

A technically sophisticated model can fail if it addresses the wrong question. Problem formulation should therefore precede algorithm selection. A useful project statement identifies:

1. Decision owner: Who will use the result?
2. Decision: What action or choice will be improved?
3. Population: Which entities and time period are included?
4. Target: What outcome is to be predicted, described, or explained?
5. Prediction horizon: How far into the future must the result apply?
6. Action window: How much time is available to act?
7. Constraints: What limits exist regarding cost, privacy, latency, interpretability, or regulation?
8. Success criteria: How will technical and organizational success be measured?

For example, “use data mining to improve customer retention” is too vague. A more precise formulation is: “For subscription customers active for at least 90 days, estimate the probability of cancellation within the next 30 days, using information available at the beginning of each week, so that the retention team can prioritize outreach within a fixed budget.”

This formulation defines the population, target, horizon, timing, and operational use. It also reduces a major modeling risk: data leakage, in which information unavailable at prediction time is accidentally used to train the model.

Data Requirements and Data Quality

Data requirements should be specified before modeling. They include:

• entities and identifiers;
• variables and their definitions;
• target-label construction;
• observation period and update frequency;
• required historical depth;
• acceptable missingness;
• data ownership and access permissions;
• integration keys and source systems;
• storage, computation, and latency requirements;
• documentation, lineage, and versioning.

Data quality is multidimensional. Important dimensions include accuracy, completeness, consistency, timeliness, uniqueness, validity, and representativeness. A dataset can be complete but inaccurate, or accurate for one subgroup but unrepresentative of the population in which the model will be used.

Missing values require careful treatment. Missingness may be random, related to other measured variables, or related to the unobserved value itself. Replacing all missing values with zero can create false meaning. Similarly, removing every record with a missing value may bias the sample. The treatment should reflect the data-generating process and the intended model.

Data preparation can include normalization, categorical encoding, aggregation, text processing, feature construction, and outlier treatment. Every transformation should be reproducible and documented. The same transformation logic must be applied consistently during deployment.

Success Criteria and Evaluation

Success criteria should include both technical and business measures. A classification project might use recall for a costly-to-miss condition, precision when investigation capacity is limited, and calibration when predicted probabilities guide resource allocation. A regression project might use mean absolute error when errors should be interpreted in the original units.

For binary classification, a confusion matrix contains true positives, false positives, true negatives, and false negatives. Common measures include:

Here TP, TN, FP, and FN denote the four confusion-matrix counts. These formulas assume a defined classification threshold and a clearly identified positive class. Accuracy can be inappropriate for severe class imbalance. Precision and recall should be interpreted together with the operational cost of each error.

Evaluation should use an appropriate validation design. Random splitting may be unsuitable for time-dependent data because it can allow future information to influence evaluation. Time-based evaluation, grouped splitting, or cross-validation may be more appropriate. The final test set should remain untouched until model selection is complete.

Business success may be measured through reduced losses, improved service levels, increased conversion, shorter processing time, or user adoption. A model can have excellent statistical performance but produce no value if users distrust it, cannot act on its outputs, or face incompatible workflows.

Illustrative Case: Predicting Late Deliveries

Background

A regional online retailer experiences customer dissatisfaction because some deliveries arrive later than promised. Management wants to prioritize potentially late orders for intervention.

Problem

The initial request is “build a late-delivery model.” During business understanding, the team clarifies that the relevant decision is whether to expedite, contact the customer, or adjust the promise date. The target is defined as delivery after the promised date, and the prediction must be made at order dispatch, before the package reaches the carrier network.

Analysis

Available data include destination region, product category, warehouse, carrier, dispatch time, promised date, historical carrier performance, weather indicators available at dispatch, and package characteristics. Information recorded after dispatch—such as later tracking scans—must not be used for the initial model because it would not be available when the decision is made.

The team discovers that late deliveries are only 8% of all orders. Therefore, accuracy alone could be misleading: a model that predicts “on time” for every order would be 92% accurate but operationally useless. The team selects recall and precision at the number of orders the logistics staff can realistically investigate each day.

Approach and solution

Following CRISP-DM, the team establishes a time-based training and test split. It cleans inconsistent carrier codes, constructs features from historical performance, and compares a transparent baseline model with more complex alternatives. The model is evaluated not only on aggregate performance but also across regions and carrier groups. A dashboard displays risk scores, principal contributing factors, and an escalation recommendation.

The project includes a human review process. Staff may override a recommendation, and override reasons are recorded. The model is monitored after deployment because carrier behavior, seasonal demand, and service policies may change.

Outcome

In an illustrative pilot, the model identifies a manageable subset of orders containing a large proportion of late deliveries. The organization uses the output to prioritize interventions rather than automatically penalize carriers or customers. This distinction is important: a prediction supports a decision, but the organization remains accountable for the decision and its consequences.

Learning points

1. The operational decision determines the target and prediction time.
2. Preventing data leakage is essential.
3. Imbalanced outcomes require suitable evaluation measures.
4. Business capacity determines the useful operating threshold.
5. Fairness, monitoring, explainability, and human oversight matter after deployment.

Challenges, Ethics, and Governance

Data mining can reproduce or amplify social inequalities. If historical decisions reflect discrimination, a model trained on those decisions may learn discriminatory patterns. Protected attributes should not be assumed harmless merely because they are excluded; proxy variables can encode similar information.

Important governance practices include:

• lawful and transparent data collection;
• purpose limitation and data minimization;
• access control and secure storage;
• documentation of data sources and transformations;
• consent or another valid legal basis where required;
• fairness testing across relevant groups;
• explainability appropriate to the decision context;
• human review for high-impact decisions;
• audit trails and appeal mechanisms;
• monitoring for drift, degradation, and unexpected effects.

Privacy risks include re-identification, excessive collection, unauthorized secondary use, and insecure sharing. Technical approaches such as aggregation, access restrictions, de-identification, differential privacy, and federated learning may reduce risk, but none is a universal solution. Governance must combine technology, policy, accountability, and legal compliance.

Other limitations include spurious correlations, sampling bias, concept drift, adversarial manipulation, insufficient labels, and overfitting. A pattern discovered in historical data may disappear when policies change. Models should therefore be monitored and periodically reassessed.

Contemporary Directions and Professional Practice

Current data-mining practice increasingly involves cloud platforms, streaming data, graph analytics, natural-language processing, automated machine learning, and generative-AI-assisted workflows. These technologies can improve speed and scale, but they do not remove the need for problem formulation, data governance, validation, or human judgment.

Professionals should adopt the following practices:

• begin with a decision and measurable objective;
• establish a simple baseline before using complex models;
• separate training, validation, and final testing data;
• document assumptions, transformations, and limitations;
• evaluate performance by subgroup and operating condition;
• distinguish correlation from causation;
• design deployment and monitoring before final model selection;
• communicate uncertainty rather than presenting predictions as facts;
• involve domain experts, affected stakeholders, security personnel, and governance specialists.

Data mining is best understood as a socio-technical discipline. Algorithms are important, but value arises only when reliable data, appropriate methods, responsible governance, and organizational action operate together.

Chapter Summary

Data mining discovers valid, useful, novel, and understandable patterns in data. It forms a central component of the broader knowledge-discovery process, which also includes domain understanding, data selection, preparation, evaluation, communication, and use. Machine learning supplies many algorithms for learning from data; statistics supports uncertainty and inference; analytics connects results to decisions.

Descriptive tasks summarize and explore existing data, predictive tasks estimate unknown or future outcomes, and prescriptive tasks recommend actions under objectives and constraints. Classification, regression, clustering, association analysis, and anomaly detection are major data-mining tasks. CRISP-DM provides an end-to-end, business-oriented project framework, while SEMMA emphasizes sampling, exploration, modification, modeling, and assessment.

Successful projects define the decision, population, target, prediction time, data requirements, constraints, and technical and business success criteria. Evaluation must account for class imbalance, temporal structure, leakage, subgroup performance, operational capacity, and uncertainty. Ethical and governance concerns—including privacy, fairness, transparency, security, and accountability—remain essential throughout the project lifecycle.

Key Terms

Data mining: Computational discovery of useful patterns in data.

Knowledge discovery in databases (KDD): The broader iterative process of obtaining knowledge from data.

Pattern: A regularity, relationship, or structure found in data.

Machine learning: Methods that learn relationships or decision rules from data or experience.

Analytics: The use of data and analytical methods to support understanding and decisions.

Descriptive analytics: Analysis concerned with what has happened or what is present.

Predictive analytics: Analysis that estimates unknown or future outcomes.

Prescriptive analytics: Analysis that recommends actions under objectives and constraints.

Classification: Prediction of a categorical target.

Regression: Prediction of a numerical target.

Clustering: Grouping observations according to similarity.

Association rule: A co-occurrence relationship expressed as X → Y.

Support: Proportion of transactions containing all items in an association rule.

Confidence: Proportion of X transactions that also contain Y.

Lift: Confidence divided by the baseline support of the consequent.

Data leakage: Use of information in model development that would not be available at prediction time.

Overfitting: Learning training-specific variation that harms performance on new data.

Concept drift: Change over time in the relationship between inputs and outcomes.

CRISP-DM: A six-phase process model for data-mining projects.

SEMMA: A process model consisting of Sample, Explore, Modify, Model, and Assess.

Review Questions

Short Answer Questions

1. Define data mining and state four qualities of a valuable discovered pattern.
2. How does KDD differ from data mining?
3. Distinguish classification from regression.
4. What is data leakage, and why is it dangerous?
5. Name the six phases of CRISP-DM.

Long Answer Questions

1. Explain the relationship among data mining, machine learning, statistics, and analytics.
2. Compare descriptive, predictive, diagnostic, and prescriptive analytics with examples.
3. Explain the major data-quality dimensions and describe how poor quality can affect a model.
4. Compare CRISP-DM and SEMMA in terms of scope, strengths, and limitations.
5. Discuss ethical and governance issues that should be addressed before and after deploying a data-mining model.

Analytical / Application Questions

1. A university wants to identify students at risk of failing a course. Formulate the decision, population, target, prediction horizon, data requirements, constraints, and success criteria. Identify at least two possible leakage variables.
2. In a transaction database of 2,000 purchases, item A appears in 500 transactions, item B in 400, and both in 150. Calculate support, confidence for A → B, and lift. Interpret the result without claiming causation.
3. A fraud model has high accuracy but misses most fraudulent transactions. Explain why accuracy may be misleading and recommend an evaluation and deployment strategy appropriate for limited investigation capacity.

Multiple Choice Questions with Answers

Question 1

Which statement best describes the relationship between KDD and data mining?

A. KDD is a database storage method, whereas data mining is data encryption.

B. Data mining is one central activity within the broader KDD process.

C. KDD refers only to predictive modeling, whereas data mining refers only to visualization.

D. They are unrelated disciplines with no overlapping activities.

Correct Answer: B. Data mining is one central activity within the broader KDD process.

Question 2

Which task is an example of classification?

A. Grouping customers by similarity without labels

B. Estimating tomorrow's electricity consumption

C. Predicting whether an email is spam or legitimate

D. Finding products frequently purchased together

Correct Answer: C. Predicting whether an email is spam or legitimate

Question 3

For an association rule X → Y, what does confidence measure?

A. The proportion of all transactions containing X and Y

B. The proportion of X transactions that also contain Y

C. The ratio of X transactions to the total number of transactions

D. The difference between support of X and support of Y

Correct Answer: B. The proportion of X transactions that also contain Y

Question 4

Which CRISP-DM phase focuses primarily on defining objectives, stakeholders, constraints, and business success criteria?

A. Business understanding

B. Data preparation

C. Modeling

D. Deployment

Correct Answer: A. Business understanding

Question 5

Which statement about clustering is correct?

A. It always requires predefined class labels.

B. It predicts only continuous numerical targets.

C. Its results depend partly on representation and similarity measures.

D. It proves that discovered groups are natural social categories.

Correct Answer: C. Its results depend partly on representation and similarity measures.

Question 6

A dataset contains 1,000 transactions; X occurs in 200 and both X and Y occur in 50. What is the confidence of X → Y?

A. 5%

B. 20%

C. 25%

D. 50%

Correct Answer: B. 20%

Question 7

Why can a model with high accuracy still be unsuitable for fraud detection?

A. Accuracy cannot be calculated for classification models.

B. Fraud datasets may be imbalanced, allowing a model to miss most fraud.

C. Fraud detection never uses labeled data.

D. Accuracy measures only model training time.

Correct Answer: B. Fraud datasets may be imbalanced, allowing a model to miss most fraud.

Question 8

Which situation is an example of data leakage?

A. Using a customer's age recorded before a prediction date

B. Splitting data into training and test sets

C. Using a delivery scan recorded after dispatch to predict late delivery at dispatch

D. Removing duplicate records before modeling

Correct Answer: C. Using a delivery scan recorded after dispatch to predict late delivery at dispatch

Question 9

What is the primary emphasis of SEMMA?

A. Legal compliance and policy writing only

B. Sampling, exploration, modification, modeling, and assessment

C. Database indexing and transaction recovery

D. Replacing all human decisions with automated systems

Correct Answer: B. Sampling, exploration, modification, modeling, and assessment

Question 10

Which statement is most appropriate when interpreting a discovered association?

A. Association automatically establishes causation.

B. Association is irrelevant unless it is causal.

C. Association may be useful, but causal claims require suitable evidence.

D. Association measures eliminate the need for domain expertise.

Correct Answer: C. Association may be useful, but causal claims require suitable evidence.