2. Data Types, Structures and Program Control

Introduction and Context

Data science begins with the disciplined representation and manipulation of information. Before a model can estimate a parameter, a visualisation can reveal a pattern, or an algorithm can make a prediction, data must be stored in suitable forms and processed through logically correct instructions. Data types determine what a value means and which operations are valid. Data structures determine how values are organised. Program-control mechanisms determine the order in which instructions are executed.

Python and R provide overlapping capabilities but express them through different design philosophies. Python emphasises general-purpose programming, object-oriented abstractions, and a rich ecosystem of libraries. R was designed primarily for statistical computing and data analysis, and consequently gives particular importance to vectors, factors, matrices, and data frames. Effective data scientists need more than syntactic familiarity: they must understand how representation affects computation, memory, missing-value behaviour, comparison, and analytical validity.

This chapter develops the foundations required to write reliable data-processing programs in both languages. It considers scalar values, collections, arrays, matrices, tabular objects, indexing, type conversion, missing values, conditionals, iteration, vectorization, comprehensions, and error control. These ideas form the bridge between basic programming and later work with data manipulation, statistical modelling, and machine learning.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Explain the role of variables, operators, scalar types, and type systems in Python and R.

2. Select appropriate structures for one-dimensional, multidimensional, and tabular data.

3. Index, slice, filter, reshape, and combine data safely.

4. Distinguish vectorized operations from explicit iteration and choose between them appropriately.

5. Use conditionals, loops, comprehensions, and function-level control flow.

6. Detect, represent, convert, and appropriately handle missing or invalid values.

7. Apply error-handling techniques and identify common programming mistakes.

8. Evaluate the implications of data representation for reproducibility, performance, and professional analytics.

Values, Variables, and Operators

A variable is a name associated with an object or value. Assignment does not merely label a piece of text; it creates or reuses a reference to an object in memory. In Python, assignment uses `=` and multiple assignment is possible, as in `a, b = 10, 20`. In R, assignment is commonly written as `<-`, although `=` is also accepted in many contexts: `a <- 10`.

The principal scalar types in Python include integers (`int`), floating-point numbers (`float`), complex numbers (`complex`), Boolean values (`bool`), strings (`str`), and the null-like value `None`. R includes numeric values, integer values, logical values, character strings, complex values, and special values such as `NA`, `NaN`, `Inf`, and `NULL`. The distinctions matter. For example, `NA` means that a value is unavailable, whereas `NaN` generally indicates an undefined numerical result, such as zero divided by zero.

Operators express computation and comparison. Arithmetic operators include addition, subtraction, multiplication, division, exponentiation, and remainder. Python uses `**` for exponentiation and `/` for ordinary division; R uses `^` and also uses `/` for division. Comparison operators produce Boolean or logical results. Python uses `and`, `or`, and `not` for scalar logical combinations, while R commonly uses `&`, `|`, and `!`. Python also distinguishes identity (`is`) from equality (`==`); R has related concepts through equality and object comparison functions. Confusing identity with equality is a frequent source of errors.

Operator precedence determines how expressions are evaluated. Parentheses should be used when they improve clarity or prevent ambiguity. In analytical code, explicit expressions are generally preferable to relying on detailed knowledge of precedence rules.

Collections and Data Structures

A data structure is a patterned organisation of values that supports particular operations. The appropriate structure depends on dimensionality, homogeneity, ordering, mutability, and the required analytical tasks.

Python collections

A list is an ordered, mutable collection that may contain values of different types: `scores = [72, 81, 68]`. Lists support indexing from zero, slicing, insertion, deletion, and iteration. A tuple is ordered but immutable: `point = (4, 7)`. Tuples are useful when a fixed grouping of values should not be modified.

A dictionary stores key–value associations: `student = {'id': 14, 'name': 'Mira'}`. Keys should be unique and hashable. Dictionaries are useful for metadata, lookup tables, configuration, and records with named attributes. A set stores unique elements and supports operations such as union and intersection. Sets are useful for membership tests and deduplication, but they should not be used when positional order is central.

Strings are ordered sequences of characters. They can be indexed and sliced, but they are immutable. A data scientist should distinguish textual content from categorical information: a string such as `"High"` has no inherent numerical ordering unless that ordering is explicitly defined.

For numerical analysis, NumPy arrays are more appropriate than ordinary lists in many situations. An array normally contains elements of a compatible data type and supports efficient vectorized operations. A two-dimensional array may represent a matrix, while arrays with more dimensions can represent images, simulations, or tensors. Array shape and data type should be checked before computation because incompatible shapes can produce errors or unintended broadcasting.

R vectors, lists, and factors

The fundamental R structure is the vector. An atomic vector contains values of one basic type, such as numeric, integer, logical, or character. Operations on vectors are naturally element-wise. For example, if `x <- c(2, 4, 6)`, then `x * 3` produces `6, 12, 18` without an explicit loop.

An R list can contain objects of different types and lengths. It is therefore suitable for complex records, model results, or nested structures. A factor represents categorical data through a set of levels. For example, `factor(c("low", "high", "low"), levels = c("low", "high"))` records both observed values and the permitted categories. Factors may be nominal or ordered. Treating a nominal factor as if its numeric codes were meaningful can create serious analytical errors.

Matrices, arrays, and tabular objects

An R matrix is a two-dimensional homogeneous structure. It is commonly created with `matrix()` and is especially useful for linear algebra. An R array generalises this idea to more than two dimensions. A data frame is a tabular object in which columns may have different types but must have compatible lengths. A tibble, provided by the tidyverse, is a modern data-frame variant with more consistent printing and subsetting behaviour.

In Python, tabular analysis is commonly performed with a pandas `DataFrame`. A DataFrame consists of labelled columns that may have different data types. A pandas `Series` is a one-dimensional labelled object. Unlike a NumPy matrix, a DataFrame gives prominence to column names and row labels, which is valuable in data preparation and reporting.

Choosing between a matrix and a table is conceptually important. Matrices are appropriate when all elements participate in a common mathematical structure. Tables are appropriate when columns represent distinct variables such as age, region, and income. Forcing a heterogeneous table into a homogeneous matrix may cause unwanted conversion to strings or a less useful common type.

Indexing, Slicing, and Selection

Indexing selects one or more elements. Python uses zero-based positional indexing: the first element of `x` is `x[0]`. A slice such as `x[1:4]` selects positions 1, 2, and 3 because the upper boundary is excluded. Negative indices count from the end. NumPy and pandas extend indexing with multidimensional selection, labels, Boolean masks, and methods such as `.loc` and `.iloc`.

R generally uses one-based indexing: the first element is `x[1]`. A slice such as `x[2:4]` includes both boundaries. R uses `[` to return a subset that may retain structure, while `[[` extracts a single component from a list or data frame, and `$` accesses a named component. These differences make direct translation between Python and R dangerous.

Boolean selection is central to data analysis. In Python, a condition such as `df[df['age'] >= 18]` filters rows. In R, `df[df$age >= 18, ]` has a similar purpose. Missing values must be considered because a comparison involving a missing value may not yield an ordinary `True` or `False` result. Robust filtering explicitly identifies missingness before applying a condition.

Names and labels improve readability but can introduce ambiguity if labels are duplicated or changed during transformation. Professional code should validate expected dimensions, column names, and index uniqueness rather than assuming that they remain stable.

Type Conversion and Missing Values

Type conversion, or coercion, changes an object from one data type to another. Python uses constructors such as `int()`, `float()`, `str()`, and `bool()`, although conversion may fail or produce surprising results. For example, `bool("False")` is `True` because every non-empty string is truthy; it does not interpret the semantic meaning of the word False. Safe conversion often requires explicit validation.

R provides functions such as `as.numeric()`, `as.character()`, `as.integer()`, and `as.factor()`. Conversion from factor to numeric requires care: `as.numeric(factor(c("10", "20")))` returns internal level codes rather than the displayed numbers. A safer approach is to convert first to character and then to numeric.

Missing values are not simply zeros or empty strings. In Python, `None`, `numpy.nan`, and pandas' nullable values have different behaviours. In R, `NA` propagates through many operations. Python commonly uses `is None`, `math.isnan()`, or pandas functions such as `isna()`. R uses `is.na()` and related functions. Testing equality with a missing value is generally inappropriate: the correct operation is a missingness test.

Handling missing data requires an analytical decision. Possible strategies include removing incomplete records, replacing values through imputation, adding a missingness indicator, or using algorithms that model missingness directly. The choice depends on the mechanism and extent of missingness, the purpose of the analysis, and the consequences of bias. Silent removal is particularly risky because it can alter the population represented by the data.

Program Control: Decisions and Repetition

A conditional executes different instructions according to a logical test. Python uses `if`, `elif`, and `else`; R uses `if`, `else if`, and `else`. The condition in a scalar `if` should resolve to one logical value. In R, attempting to use a vector with more than one logical value in `if` is an error or warning-prone practice; vectorized selection should instead use `ifelse()` or related tools.

Loops repeat a block of instructions. Python provides `for` and `while`; R also provides `for`, `while`, and `repeat`. `break` terminates a loop, while `continue` in Python and `next` in R skip to the next iteration. Loops are appropriate when operations are sequential, state-dependent, or difficult to express otherwise. They may be less efficient and less expressive than vectorized operations for large numerical data.

Vectorization applies an operation to an entire collection using an implementation optimized below the language level. In Python, NumPy expressions such as `x * 2 + 1` avoid explicit element-by-element Python loops. In R, vectorized arithmetic and functions such as `ifelse()` provide similar benefits. Vectorization can improve speed and clarity, but it is not automatically correct: recycling in R and broadcasting in NumPy may produce results that are syntactically valid but conceptually wrong. Shapes and lengths must therefore be checked.

Python comprehensions provide compact construction of collections. For example, `[x**2 for x in values if x > 0]` creates squares of positive values. Dictionary and set comprehensions are also available. Comprehensions should remain readable; deeply nested expressions are often less maintainable than ordinary loops or functions.

Functions, Scope, and Error Control

Functions package a computation into a reusable unit. Parameters should have meaningful names, and a function should have a clear contract concerning inputs, outputs, types, and possible errors. Python functions use `def`; R functions use `function()`. Local variables reduce unintended interactions with the surrounding program.

Python handles anticipated runtime problems with `try`, `except`, `else`, and `finally`. Exceptions should be caught narrowly—for example, handling `ValueError` during numeric conversion rather than catching every possible exception. R supports condition handling through mechanisms including `try()`, `tryCatch()`, warnings, and messages. Error control should not hide failures. Logging, informative messages, and validation are preferable to silently returning an incorrect result.

Assertions express assumptions, such as a requirement that a table contain a particular column or that an array have a specified shape. Assertions are useful during development, but critical production validation should not depend solely on assertions that may be disabled or bypassed. Input checks, unit tests, and reproducible examples provide additional protection.

An Applied Scenario: Preparing a Customer-Transaction Table

Background

An illustrative retail company receives transaction records from several branches. Each record contains a customer identifier, transaction amount, branch, customer segment, and sometimes a missing amount. Python and R analysts must prepare the data for a monthly report.

Situation and problem

The raw data contains numeric amounts stored as text, inconsistent branch labels, missing transaction amounts, and segment values such as `"standard"`, `"Standard"`, and `"premium"`. A direct calculation of the average amount produces either an error or an unreliable result.

Analysis

The first task is to inspect the structure, not immediately calculate statistics. The analysts identify column types, unique category values, missingness, and duplicate identifiers. They determine that an amount such as `"1,250.50"` requires cleaning before conversion. They also establish that a missing amount cannot automatically be treated as zero because the absence may indicate an incomplete transaction record rather than a free transaction.

Approach and solution

The analysts standardise text by trimming whitespace and applying a documented case convention. They remove thousands separators only after confirming that the field represents a monetary amount. In Python, conversion is performed with an explicitly controlled numeric function and invalid values are recorded for review. In R, character data are converted through a validated transformation rather than relying on factor codes.

They represent customer segment as a factor in R with defined levels, or as a categorical column in pandas. They use Boolean filtering to separate valid transactions from records requiring investigation. For the monthly total, they choose a policy that excludes unknown amounts from the numerical sum while reporting the count of missing amounts separately. They use vectorized calculations for standard transformations and reserve a loop for generating a detailed audit message for each rejected record.

Outcome

The resulting table contains consistent types, documented category levels, an explicit missingness indicator, and a reproducible cleaning process. The report presents both the computed revenue and the number of incomplete records, allowing managers to interpret the result rather than treating it as an unquestionable number.

Learning points

The case illustrates that data structures are analytical decisions. Type conversion is part of data quality management, not merely a technical precondition. Missing-value handling must be documented, and vectorization should be combined with validation rather than used as a substitute for reasoning.

Common Errors, Risks, and Professional Practice

Several mistakes recur in Python and R programs. Off-by-one errors arise when analysts transfer one-based R assumptions into zero-based Python code. A mutable default argument in Python can retain values across function calls. In R, unintended vector recycling can combine vectors of incompatible lengths. Ambiguous truth testing, accidental factor-code conversion, and treating missing values as ordinary values can corrupt results without producing obvious errors.

Another risk is modifying a view or copy unintentionally, especially when working with tabular libraries. Analysts should understand whether an operation creates a new object or changes an existing one. Chained transformations should be tested with small examples and checked for expected dimensions.

Data governance also begins at this level. Sensitive identifiers should not be printed unnecessarily, temporary files should be controlled, and transformations affecting regulated or consequential decisions should be traceable. Reproducible scripts should record assumptions, package versions where relevant, input checks, and decisions about exclusions or imputation. Automated data-quality tests can check ranges, allowed categories, uniqueness, missingness thresholds, and schema changes before a pipeline proceeds.

Current analytical practice increasingly combines vectorized data-frame operations with declarative pipelines, typed schemas, nullable data types, and automated validation. Array-oriented libraries and data-frame engines continue to improve execution through parallelism, compilation, and columnar storage. These developments do not remove the need to understand basic structures; rather, they make foundational knowledge more important because high-level tools still depend on correct types, dimensions, and missing-value semantics.

Best Practices for Reliable Data Programs

1. Choose a structure that matches the meaning and dimensionality of the data.

2. Inspect types, shapes, levels, and missingness before computation.

3. Use explicit conversion and validate the result.

4. Prefer vectorization for independent numerical operations, but check broadcasting or recycling.

5. Use loops when state, sequencing, or complex control flow genuinely requires them.

6. Write small functions with clear inputs, outputs, and error behaviour.

7. Distinguish missing, zero, empty, invalid, and not-applicable values.

8. Avoid silent exception handling and document data-cleaning decisions.

9. Test boundary cases, including empty collections, duplicate labels, wrong types, and all-missing columns.

10. Preserve an audit trail for transformations that affect business or public decisions.

Chapter Summary

Data types describe the nature of values, while data structures organise those values for computation. Python provides lists, tuples, dictionaries, sets, NumPy arrays, Series, and DataFrames; R provides vectors, lists, factors, matrices, arrays, data frames, and tibbles. Their differences in indexing, coercion, logical operations, and missing-value semantics require deliberate language-specific reasoning.

Indexing and Boolean selection support targeted analysis, while vectorization and comprehensions provide concise and efficient transformations. Conditionals and loops control program execution, and functions provide reusable abstractions. Type conversion, missing-value handling, validation, and exception control are essential to trustworthy results. In professional data science, these are not merely programming details: they determine whether an analytical result is interpretable, reproducible, and defensible.

Key Terms

Variable: A name associated with a value or computational object.

Scalar: A single value rather than a collection of values.

Operator: A symbol or keyword that performs computation, comparison, or logical combination.

List: An ordered, mutable Python collection that may contain heterogeneous objects.

Tuple: An ordered, immutable Python collection.

Dictionary: A collection of key–value associations in Python.

Set: A collection of unique elements supporting membership and set operations.

Vector: A one-dimensional R structure whose atomic elements generally share a type.

Factor: An R structure representing categorical values and their defined levels.

Matrix: A two-dimensional homogeneous numerical or logical structure.

Array: A potentially multidimensional collection with a regular shape.

Data frame: A tabular structure whose columns may have different types.

Indexing: Selecting elements, rows, columns, or components from a structure.

Vectorization: Applying an operation to a collection without an explicit user-level loop.

Comprehension: A compact Python expression for constructing a collection.

Coercion: Conversion of an object from one data type to another.

Missing value: A value that is unavailable, unknown, or not recorded.

Broadcasting: Applying array operations across compatible but differently shaped dimensions.

Exception handling: Detecting and managing runtime errors without uncontrolled program failure.

Schema: A specification of expected fields, types, dimensions, and constraints in data.

Review Questions

Short Answer Questions

1. How does a Python tuple differ from a Python list?

2. Why is a factor useful for representing categorical data in R?

3. Distinguish `NA` from zero in an analytical dataset.

4. What is vectorization, and why can it improve performance?

5. Why should equality testing not normally be used to detect missing values?

Long Answer Questions

1. Compare Python and R indexing conventions, including positional selection and slicing.

2. Explain the differences among vectors, matrices, arrays, and data frames in R.

3. Discuss type coercion in Python and R and identify situations in which automatic conversion may be dangerous.

4. Explain how conditionals, loops, vectorization, and comprehensions support different forms of program control.

5. Describe a robust workflow for cleaning a transaction table containing invalid types, inconsistent categories, and missing amounts.

Analytical / Application Questions

1. A pandas column contains the strings `"10"`, `"20"`, `"unknown"`, and an empty string. Design a conversion and validation strategy that preserves an audit trail.

2. An R vector has length 10 and is multiplied by a vector of length 3. Explain the potential problem and propose checks that should occur before the operation.

3. A business report excludes all rows with missing income values. Analyse when this decision might bias the report and suggest alternative strategies.

22. Multiple Choice Questions with Answers

Question 1

Which Python structure is most appropriate for storing unique customer identifiers and testing membership efficiently?

A. Set

B. Tuple

C. Float

D. Matrix

Correct Answer: A. Set

Question 2

What is the main analytical danger of converting an R factor directly with as.numeric()?

A. It always creates missing values

B. It may return internal level codes rather than displayed numeric values

C. It converts all values to logical values

D. It changes the factor into a matrix

Correct Answer: B. It may return internal level codes rather than displayed numeric values

Question 3

Which statement correctly distinguishes a matrix from a data frame?

A. A matrix must contain columns of different types

B. A data frame cannot contain character data

C. A matrix is generally homogeneous, while a data frame may have heterogeneous columns

D. A data frame can contain only two dimensions

Correct Answer: C. A matrix is generally homogeneous, while a data frame may have heterogeneous columns

Question 4

Which Python expression selects values greater than zero and squares them using a comprehension?

A. [x**2 for x in values if x > 0]

B. [x for x**2 in values if x > 0]

C. (x**2 if x > 0 for x in values)

D. values[x > 0] ** 2

Correct Answer: A. [x2 for x in values if x > 0]**

Question 5

Why is replacing every missing transaction amount with zero potentially inappropriate?

A. Zero is not a valid Python number

B. Missingness may represent an unknown or incomplete transaction rather than no transaction

C. Zeros cannot be stored in data frames

D. It always causes a syntax error

Correct Answer: B. Missingness may represent an unknown or incomplete transaction rather than no transaction

Question 6

Which technique is generally suitable for independent numerical operations on an entire NumPy array?

A. Vectorization

B. Manual string parsing

C. Recursive exception handling

D. Factor coding

Correct Answer: A. Vectorization

Question 7

What is the principal purpose of a narrowly written exception handler?

A. To hide all program failures

B. To convert every value to a string

C. To manage an anticipated error while avoiding concealment of unrelated failures

D. To eliminate the need for input validation

Correct Answer: C. To manage an anticipated error while avoiding concealment of unrelated failures

Question 8

In Python, what is the index of the first element in an ordinary list?

A. -1

B. 0

C. 1

D. The list length

Correct Answer: B. 0

Question 9

A data scientist wants to represent customer segment values as ordered categories such as low, medium, and high. Which representation is most appropriate in R?

A. An ordered factor with defined levels

B. A complex scalar

C. A matrix of character codes only

D. A set with no level information

Correct Answer: A. An ordered factor with defined levels

Question 10

What should an analyst do before applying a vectorized operation to arrays of different shapes?

A. Assume broadcasting always expresses the intended meaning

B. Convert all values to strings

C. Check shape compatibility and confirm that broadcasting produces the intended result

D. Delete the shorter array

Correct Answer: C. Check shape compatibility and confirm that broadcasting produces the intended result