Core Data Structures and Comprehensions

Introduction

Data science depends on the ability to represent, organize, transform, and inspect information. Python provides several built-in data structures for these purposes. Strings represent textual data; lists represent ordered collections that may change; tuples represent ordered collections intended to remain stable; sets represent unique, unordered elements; and dictionaries represent associations between keys and values.

Choosing an appropriate structure is not merely a matter of syntax. It affects readability, correctness, memory use, computational efficiency, and the reliability of data-processing pipelines. A data scientist who understands mutability, indexing, slicing, equality, identity, hashing, and comprehensions can write programs that are both expressive and robust.

This chapter develops these concepts from first principles and applies them to practical data-science tasks such as cleaning text, filtering observations, removing duplicates, constructing feature mappings, and preparing data for analysis.

Learning Objectives

By the end of this chapter, you should be able to:

• manipulate strings, lists, tuples, sets, and dictionaries;
• apply indexing and slicing correctly, including negative indices and steps;
• distinguish mutable from immutable objects;
• explain equality, identity, and aliasing;
• select a suitable data structure for a given problem;
• construct and evaluate list, set, and dictionary comprehensions;
• avoid common errors involving shallow copying, missing dictionary keys, and unintended mutation;
• design readable and maintainable data-structure operations for data-science workflows.

1. Objects, References, and Collection Types

In Python, a variable name refers to an object rather than containing a value in the same sense as a fixed memory box. For example:

scores = [72, 81, 95]

The name scores refers to a list object. The list contains references to integer objects. Assignment generally creates another reference to the same object:

other_scores = scores
other_scores.append(66)
print(scores)  # [72, 81, 95, 66]

Both names refer to the same list. This behavior is called aliasing.

Python collection types differ along several dimensions:

“Insertion order preserved” for dictionaries means that iteration follows the order in which keys were inserted in modern Python. It does not mean that a dictionary is indexed by integer position.

Equality and identity

The operator == tests whether two objects have equivalent contents. The operator is tests whether two names refer to the same object.

x = [1, 2]
y = [1, 2]
z = x

print(x == y)  # True
print(x is y)  # False
print(x is z)  # True

Use == for value comparison. Use is mainly for identity tests such as value is None.

2. Strings: Text as an Immutable Sequence

A string is an immutable sequence of Unicode characters. Immutability means that an existing string cannot be changed in place. Operations that appear to modify a string actually create a new string.

text = "Data Science"
text = text.lower()
text = text.replace("science", "analysis")
print(text)  # data analysis

Indexing and slicing strings

Sequence positions begin at zero. Negative indices count from the end.

word = "Python"
print(word[0])    # P
print(word[-1])   # n
print(word[1:4])  # yth
print(word[:2])   # Py
print(word[2:])   # thon
print(word[::2])  # Pto
print(word[::-1]) # nohtyP

For a sequence of length n, valid positive indices are from 0 through n − 1; valid negative indices are from −n through −1. A slice sequence[start:stop:step] includes start when reachable but excludes stop. The step cannot be zero.

Slicing normally returns a new sequence and does not raise an error when a boundary lies outside the sequence, although an integer index outside the valid range raises IndexError.

Useful string methods

raw = "  Nairobi, Kenya  "
clean = raw.strip()
city, country = clean.split(", ")
label = f"{city} ({country})"

Important operations include:

• strip(), lstrip(), and rstrip() for surrounding whitespace;
• lower() and upper() for case conversion;
• replace(old, new) for substitution;
• split(separator) for converting text into a list;
• join(iterable) for combining strings;
• startswith() and endswith() for pattern checks;
• find() for locating a substring;
• count() for counting non-overlapping occurrences.

join() is called on the separator, not on the list:

words = ["clean", "data", "carefully"]
sentence = " ".join(words)

A frequent data-cleaning error is confusing textual replacement with numerical conversion. The string "42" is not the integer 42 until it is explicitly converted using int("42").

3. Lists: Flexible, Mutable Sequences

Lists are ordered and mutable. They are appropriate when elements may be added, removed, sorted, or replaced.

temperatures = [21.4, 22.1, 20.8]
temperatures.append(23.0)
temperatures[1] = 22.5

The most common list operations are shown below.

append() and extend() behave differently:

items = [1, 2]
items.append([3, 4])   # [1, 2, [3, 4]]

items = [1, 2]
items.extend([3, 4])   # [1, 2, 3, 4]

List copying and aliasing

The expression copy = original does not copy a list. Use original.copy() or original[:] for a shallow copy.

original = [[1, 2], [3, 4]]
copy_1 = original.copy()
copy_1.append([5, 6])       # original is unaffected at top level
copy_1[0].append(99)        # nested inner list is shared

A shallow copy duplicates only the outer list. Nested mutable objects remain shared. When independent nested structures are required, copy.deepcopy() may be appropriate, although it should be used deliberately because it can be expensive and may have special behavior for complex objects.

Sorting and keys

sorted(iterable) returns a new list, whereas list.sort() modifies a list and returns None.

records = [("Amina", 82), ("Jon", 74), ("Li", 91)]
ranked = sorted(records, key=lambda record: record[1], reverse=True)

The key function determines the comparison value. In professional code, sorting by a named function or a clear lambda is preferable to relying on the internal arrangement of records.

4. Tuples: Stable Ordered Records

Tuples are immutable ordered sequences. They are useful for fixed collections such as coordinates, dates, database rows, or function return values.

location = (−1.2864, 36.8172)
name, score = ("Amina", 88)

A one-element tuple requires a trailing comma:

not_a_tuple = (5)
a_tuple = (5,)

Tuple unpacking improves readability:

for student, mark in [("Amina", 88), ("Jon", 76)]:
    print(student, mark)

Tuples can contain mutable objects. Therefore, tuple immutability means that the tuple’s references cannot be replaced, not that every object reachable through the tuple is immutable.

container = ([1, 2], "fixed")
container[0].append(3)  # permitted
# container[0] = [9]    # TypeError

A tuple can be used as a dictionary key only when all its elements are hashable. A tuple containing a list cannot be a key because lists are mutable and unhashable.

5. Sets: Uniqueness and Membership

A set is a mutable collection of unique, hashable elements. Sets do not support positional indexing because they represent membership rather than sequence position.

departments = {"sales", "finance", "sales"}
print(departments)  # {'sales', 'finance'}; display order may vary

An empty set is created with set(), not {}, because {} creates an empty dictionary.

Set operations express common data-processing tasks:

a = {1, 2, 3}
b = {3, 4, 5}

print(a | b)  # union: {1, 2, 3, 4, 5}
print(a & b)  # intersection: {3}
print(a - b)  # difference: {1, 2}
print(a ^ b)  # symmetric difference: {1, 2, 4, 5}

Membership testing with a set is generally efficient because sets are implemented using hash-based lookup. Exact performance depends on implementation and workload, but average-case membership is commonly described as O(1), while scanning a list is O(n), where n is the number of elements. These are asymptotic descriptions, not guarantees of identical running time for every input.

Sets are valuable for removing duplicates, checking whether categories are allowed, and comparing groups. They should not be used when order or repeated occurrences are meaningful.

6. Dictionaries: Key–Value Data Models

A dictionary maps unique, hashable keys to values. It is suitable when data must be retrieved by a meaningful identifier rather than by position.

student = {
    "id": "ST104",
    "name": "Amina",
    "score": 88
}
print(student["score"])

Dictionary keys must be hashable. Strings, numbers, and suitable tuples can be keys; lists and dictionaries cannot.

Access and safe retrieval

Indexing with square brackets raises KeyError when the key is absent:

score = student["score"]

Use get() when a missing key is expected and a fallback is appropriate:

email = student.get("email", "not supplied")

The methods keys(), values(), and items() provide views of dictionary contents:

for field, value in student.items():
    print(field, value)

To update one or more entries, use assignment or update():

student["score"] = 91
student.update({"status": "pass", "verified": True})

Counting with dictionaries

A standard data-science pattern is frequency counting:

words = ["model", "data", "model", "feature"]
counts = {}
for word in words:
    counts[word] = counts.get(word, 0) + 1

The result is {'model': 2, 'data': 1, 'feature': 1}. For larger applications, collections.Counter offers a specialized and expressive alternative.

7. Indexing, Slicing, and Nested Structures

Nested data structures occur frequently in datasets. Consider:

sales = [
    {"region": "East", "units": [10, 12, 9]},
    {"region": "West", "units": [8, 11, 13]}
]

The expression sales[1]["units"][2] returns 13. Reading such expressions from left to right is useful: select the second dictionary, retrieve its units value, then select the third unit.

For maintainability, unpack complex structures:

west_record = sales[1]
west_units = west_record["units"]
latest_west_units = west_units[-1]

This style makes assumptions visible and simplifies debugging.

8. Comprehensions: Declarative Collection Construction

A comprehension constructs a collection from an iterable, optionally transforming and filtering its elements. It is concise, but it should not sacrifice clarity.

List comprehensions

The general pattern is:

[expression for item in iterable if condition]

For example:

squares = [x * x for x in range(1, 6)]
even_squares = [x * x for x in range(1, 11) if x % 2 == 0]

The equivalent loop is sometimes clearer for complex logic:

squares = []
for x in range(1, 6):
    squares.append(x * x)

A comprehension is not automatically faster in every context, but it is often efficient and communicates a simple transformation well.

Set comprehensions

Set comprehensions create unique results:

cities = ["Lagos", "Lima", "Lagos", "Lisbon"]
initials = {city[0] for city in cities}

The result contains unique initials. Because sets are unordered, do not depend on the display or iteration order for meaning.

Dictionary comprehensions

Dictionary comprehensions produce key–value mappings:

marks = {"Amina": 82, "Jon": 74, "Li": 91}
passed = {name: mark for name, mark in marks.items() if mark >= 50}

If a comprehension produces the same key more than once, the later value replaces the earlier value. This behavior must be considered carefully when transforming data.

Nested comprehensions

Nested comprehensions can represent nested loops:

matrix = [[1, 2], [3, 4], [5, 6]]
flat = [value for row in matrix for value in row]

The order corresponds to an outer loop over rows followed by an inner loop over values. Deeply nested comprehensions are usually less readable than explicit loops.

Generator expressions

A generator expression uses parentheses and produces values lazily:

total = sum(value * value for value in range(1_000_000))

Unlike a list comprehension, it does not construct the entire collection in memory at once. This can be beneficial for large streams, although a generator can generally be consumed only once.

9. Selecting a Suitable Structure

Structure selection should follow the data’s conceptual requirements.

Consider the following questions:

1. Does order matter?
2. Are duplicates meaningful?
3. Must the collection change after creation?
4. Will elements be retrieved by position or by name?
5. Must membership testing be frequent?
6. Are the elements hashable?
7. Will other programmers understand the representation?

A list of dictionaries is often useful for small tabular datasets, but analytical workloads commonly benefit from specialized libraries such as pandas, whose Series and DataFrame structures provide columnar operations, missing-value handling, alignment, and vectorization. Python built-ins remain essential because they underpin and complement such tools.

10. Worked Example: Cleaning and Summarizing Survey Responses

Suppose a survey stores responses as strings:

responses = [
    " Python ", "python", "R", "SQL", "Python", " sql ", "Java"
]

The objective is to normalize the responses, identify unique technologies, count occurrences, and retain only technologies appearing at least twice.

First normalize whitespace and case:

normalised = [response.strip().lower() for response in responses]

The result is:

['python', 'python', 'r', 'sql', 'python', 'sql', 'java']

Create the unique set:

unique_technologies = set(normalised)

This produces the members python, r, sql, and java; the display order is not analytically meaningful.

Count frequencies:

frequencies = {}
for technology in normalised:
    frequencies[technology] = frequencies.get(technology, 0) + 1

The resulting dictionary is:

{'python': 3, 'r': 1, 'sql': 2, 'java': 1}

Filter frequent technologies:

frequent = {
    technology: count
    for technology, count in frequencies.items()
    if count >= 2
}

The result is {'python': 3, 'sql': 2}. This example illustrates a pipeline in which a list preserves observations, a set represents unique categories, and a dictionary stores aggregated counts.

11. Applied Case: Preparing Product Events for Analysis

Background

An online retailer records product events as dictionaries. Each event contains a customer identifier, a product code, and an event type.

events = [
    {"customer": "C1", "product": "P7", "event": "view"},
    {"customer": "C1", "product": "P7", "event": "cart"},
    {"customer": "C2", "product": "P3", "event": "view"},
    {"customer": "C1", "product": "P7", "event": "view"},
]

Problem

The analyst needs to determine which customers interacted with product P7, remove duplicate customer identifiers, and count event types.

Analysis and approach

A list is appropriate for the raw event sequence because each observation is retained. A set is appropriate for the unique customers associated with P7:

p7_events = [event for event in events if event["product"] == "P7"]
customers = {event["customer"] for event in p7_events}

The result is {'C1'}. The repeated view event does not create a duplicate customer.

A dictionary can count event types:

event_counts = {}
for event in p7_events:
    kind = event["event"]
    event_counts[kind] = event_counts.get(kind, 0) + 1

The result is {'view': 2, 'cart': 1}.

Outcome

The retailer can report that one unique customer interacted with P7, generating two views and one cart event. The representation also supports later extension, such as grouping by customer or product.

Learning points

• Raw records and derived summaries may require different structures.
• Sets remove duplicate identities but should not replace event records.
• Dictionaries make grouped counts explicit.
• A comprehension is appropriate for a short, transparent filter; a conventional loop may be preferable when validation, logging, or multiple operations are required.

12. Common Errors and Best Practices

Accidental mutation

Avoid modifying a collection while iterating over it. Create a filtered result instead:

values = [1, 2, 3, 4]
values = [value for value in values if value % 2 == 0]

Mutable default arguments

Do not use a mutable object as a default parameter when the intention is to create a fresh object for each call:

def add_item(item, items=None):
    if items is None:
        items = []
    items.append(item)
    return items

Dictionary access

Use get() when absence is normal; use square brackets when absence indicates an error that should be detected.

Readability and validation

Comprehensions should normally contain one clear transformation and, at most, simple filtering. Validate input before conversion, document assumptions about missing values, and use descriptive names. For sensitive data, avoid exposing personal identifiers in debugging output or logs.

Ethics and governance

Data structures do not eliminate ethical responsibilities. When processing customer, student, health, or employee data:

• collect and retain only necessary information;
• protect identifiers and access permissions;
• avoid using dictionary keys or set membership as a substitute for authorization controls;
• preserve provenance when transforming records;
• document cleaning rules because normalization can alter interpretation;
• ensure that deduplication does not incorrectly merge distinct individuals.

For example, lowercasing names may be suitable for matching in one context but may erase meaningful distinctions in another. Technical convenience must not override domain knowledge or privacy requirements.

13. Contemporary Practice and Extension

Modern data-science systems often move from Python’s built-in structures to typed models, NumPy arrays, pandas tables, databases, and distributed dataframes as scale increases. Nevertheless, built-ins remain the language-level foundation for configuration, parsing, API responses, feature engineering, and test fixtures.

Type hints can communicate intended structure:

from typing import TypedDict

class Event(TypedDict):
    customer: str
    product: str
    event: str

Type hints do not automatically validate runtime data, but they improve documentation, editor support, and static analysis. In production pipelines, combine clear types with explicit validation and tests.

Chapter Summary

Strings are immutable ordered sequences used for textual data. Lists are mutable ordered collections suitable for observations that may change. Tuples are immutable ordered records, while sets represent unique hashable elements and support union, intersection, and difference. Dictionaries map unique hashable keys to values and are especially useful for lookup, grouping, and counting.

Indexing selects one element and slicing selects a range using start:stop:step; slices exclude the stop boundary. Assignment creates references, so aliasing and shallow copies must be understood when mutable objects are involved. Equality compares contents, whereas identity compares object references.

List, set, and dictionary comprehensions express concise transformations and filters. Generator expressions provide lazy evaluation for potentially large sequences. The appropriate structure depends on order, duplication, mutability, access pattern, and membership requirements. In professional data science, structure choice must also consider readability, scalability, privacy, provenance, and the risk of incorrect deduplication or normalization.

Key Terms

Object: A value managed by Python that may be referenced by one or more names.

Reference: A relationship in which a name refers to an object.

Aliasing: The condition in which multiple names refer to the same object.

Mutable: Capable of being changed after creation.

Immutable: Not changeable in place after creation.

Indexing: Selecting an individual element by position or key.

Slicing: Selecting a subsequence using start, stop, and step boundaries.

String: An immutable Unicode sequence of characters.

List: A mutable ordered collection.

Tuple: An immutable ordered collection.

Set: A collection of unique hashable elements.

Dictionary: A mutable mapping of unique keys to values.

Hashable: Suitable for use in hash-based structures such as sets and dictionary keys.

Comprehension: A compact expression for constructing a collection.

List comprehension: A comprehension that creates a list.

Set comprehension: A comprehension that creates a set of unique results.

Dictionary comprehension: A comprehension that creates key–value pairs.

Generator expression: A lazy expression that produces values progressively.

Shallow copy: A copy of an outer collection whose nested objects may remain shared.

Normalization: Transforming data into a consistent representation, such as standard case and whitespace.

Review Questions

Short Answer Questions

1. What is the difference between equality and identity in Python?
2. Why does append() differ from extend() when used with a list?
3. Why can a set not be indexed by position?
4. What is the purpose of the default value in dictionary.get(key, default)?
5. Why is a trailing comma required to create a one-element tuple?

Long Answer Questions

1. Compare lists, tuples, sets, and dictionaries in terms of ordering, mutability, duplication, and typical data-science use.
2. Explain Python slicing, including start, stop, step, negative indices, and boundary behavior.
3. Discuss aliasing and shallow copying, using nested lists to illustrate the difference.
4. Explain list, set, and dictionary comprehensions and provide an appropriate use case for each.
5. Evaluate the factors that should guide data-structure selection in a production data-processing pipeline.

Analytical / Application Questions

1. Given a list of inconsistent product categories containing whitespace, mixed case, and duplicates, design a pipeline that normalizes the values, identifies unique categories, and counts their frequencies. Justify the structure used at each stage.
2. A team uses a list to test membership among several million permitted identifiers. Analyze the likely performance and design implications, and recommend a more suitable structure where appropriate.
3. A customer dataset contains repeated events and similar names. Explain how careless use of sets, lowercasing, or dictionary keys could produce incorrect analytical results, and propose safeguards.

Multiple Choice Questions with Answers

Question 1

Which Python structure is most suitable for storing unique department names when order is not important?

A. List

B. Tuple

C. Set

D. String

Correct Answer: C. Set

Question 2

What does the slice text[1:5] include?

A. Positions 1 through 5

B. Positions 1 through 4

C. Positions 0 through 5

D. Only position 5

Correct Answer: B. Positions 1 through 4

Question 3

What is the result of a = [1, 2]; b = a; b.append(3); print(a)?

A. [1, 2]

B. [3]

C. [1, 2, 3]

D. An empty list

Correct Answer: C. [1, 2, 3]

Question 4

Which operation safely supplies a fallback when a dictionary key may be absent?

A. d.key()

B. d.get(key, fallback)

C. d.index(key)

D. d.value(key)

Correct Answer: B. d.get(key, fallback)

Question 5

What collection is produced by {x * 2 for x in [1, 2, 2]}?

A. A list containing 2, 4, 4

B. A tuple containing 2, 4

C. A set containing 2, 4

D. A dictionary mapping 1 to 2

Correct Answer: C. A set containing 2, 4

Question 6

Which statement about sorted(values) is correct?

A. It always modifies values in place

B. It returns a new sorted list

C. It returns a set

D. It can only sort strings

Correct Answer: B. It returns a new sorted list

Question 7

Which object can serve as a dictionary key?

A. A list

B. A dictionary

C. A mutable set

D. A tuple containing only hashable elements

Correct Answer: D. A tuple containing only hashable elements

Question 8

What is the main advantage of a generator expression over a list comprehension for a large one-pass calculation?

A. It guarantees sorted output

B. It stores all results permanently

C. It can produce values lazily

D. It removes all duplicates

Correct Answer: C. It can produce values lazily

Question 9

What does " ".join(["data", "science"]) return?

A. data science

B. datascience

C. ['data', 'science']

D. data, science

Correct Answer: A. data science

Question 10

In the frequency-counting pattern counts[word] = counts.get(word, 0) + 1, what value is used when word is not already a key?

A. None

B. 1

C. 0

D. The length of counts

Correct Answer: C. 0