Collections and Data Organization

Introduction

Data science rarely operates on one value at a time. A program may need to store a sequence of measurements, the fields of one record, a collection of unique categories, or a mapping between identifiers and values. Python provides several built-in collection types for these purposes: lists, tuples, ranges, dictionaries and sets.

Choosing an appropriate collection is a design decision. It affects readability, correctness, memory use, speed and the ease with which data can be transformed. This chapter develops practical foundations for organizing data with Python collections. It covers indexing, slicing, mutation, copying, sorting, nested structures and the selection of suitable collections for data-processing tasks.

Learning Objectives

By the end of this chapter, you should be able to:

• explain the characteristics of lists, tuples, ranges, dictionaries and sets;
• use indexing and slicing safely;
• distinguish mutable from immutable objects;
• modify, copy, sort and transform collections;
• construct and process nested data structures;
• select a suitable collection for a given data-processing problem;
• recognize common aliasing, ordering and missing-data errors.

1. The Collection Model

A collection stores multiple objects. Python collections differ along several important dimensions.

Lists, tuples and ranges are sequence types. Dictionaries store key–value associations. Sets store unique elements. Python also supports nested combinations, such as a list of dictionaries or a dictionary containing lists.

A collection’s length is obtained with len(collection). Iteration is usually performed with a for loop:

scores = [72, 81, 68]
for score in scores:
    print(score)

The in operator tests membership. For a list, membership generally requires checking elements one by one. Dictionaries and sets are designed for efficient membership testing based on hashing, although exact performance depends on implementation and data conditions.

2. Lists: Flexible, Mutable Sequences

A list is an ordered, mutable collection. It can contain values of the same or different types, although homogeneous lists are often easier to process.

temperatures = [21.5, 22.0, 19.8, 23.1]
items = ["pen", 12, True]
empty = []

Indexing

Python uses zero-based indexing: the first element has index 0. Negative indices count from the end.

values = [10, 20, 30, 40]
print(values[0])    # 10
print(values[-1])   # 40

An index outside the valid range raises IndexError. Therefore, code that receives unknown-length input should validate the index or use iteration.

Slicing

A slice has the form sequence[start:stop:step]. The start index is included, while the stop index is excluded.

values = [10, 20, 30, 40, 50]
print(values[1:4])   # [20, 30, 40]
print(values[:3])    # [10, 20, 30]
print(values[::2])   # [10, 30, 50]
print(values[::-1])  # [50, 40, 30, 20, 10]

The stop-exclusive rule is useful because adjacent slices can join without overlap: values[:2] + values[2:] reconstructs the original list.

A slice creates a new list, but it is a shallow copy. The outer list is independent; nested objects remain shared.

Mutation

Lists can be changed in place.

numbers = [3, 5, 7]
numbers[1] = 6
numbers.append(9)
numbers.insert(0, 1)
numbers.extend([11, 13])

Important operations include:

remove raises ValueError if the value is absent, while pop raises IndexError if the list is empty or the index is invalid. Production code should handle these conditions when input is uncertain.

List Comprehensions

A list comprehension constructs a list from an iterable, optionally applying a condition.

raw = ["  12", "7", "  19 "]
cleaned = [int(text.strip()) for text in raw]
large = [x for x in cleaned if x >= 10]

The equivalent conventional loop may be clearer when several operations or error checks are required. Readability is more important than reducing the number of lines.

3. Tuples: Fixed-Structure Sequences

A tuple is an ordered, immutable sequence.

point = (4, 7)
record = ("DS101", "Python", 78)

Parentheses are optional in many contexts, but the comma creates a tuple:

one_item = (42,)
not_a_tuple = (42)

Tuples support indexing, slicing, iteration and membership testing, but their elements cannot be replaced, appended or removed. Immutability communicates that the structure should not change and allows a tuple containing only hashable elements to be used as a dictionary key or set element.

Tuple unpacking assigns elements to variables:

student = ("Amina", 84, "Pass")
name, mark, result = student

The number of variables must normally match the number of values. Extended unpacking is useful when the middle contains an unknown number of items:

first, *middle, last = [1, 2, 3, 4, 5]

Tuples are suitable for fixed records such as coordinates, RGB colour values and database rows. A tuple does not make nested mutable objects immutable:

data = ([1, 2], "A")
data[0].append(3)  # valid: the list inside changed

The tuple’s reference cannot be replaced, but the referenced list remains mutable.

4. Ranges: Representing Integer Progressions

range represents an arithmetic progression of integers without storing every integer as a list. Its general form is range(start, stop, step), where stop is excluded.

range(5)          # 0, 1, 2, 3, 4
range(2, 8)       # 2, 3, 4, 5, 6, 7
range(10, 2, -2)  # 10, 8, 6, 4

The step cannot be zero. A positive step with a start greater than or equal to the stop produces an empty range; a negative step requires the start to be greater than the stop.

for index in range(len(scores)):
    print(index, scores[index])

When both index and value are required, enumerate is usually clearer:

for index, score in enumerate(scores):
    print(index, score)

A range is memory-efficient because it stores its parameters rather than materializing all values. Convert it to a list only when a concrete list is needed: list(range(5)).

5. Dictionaries: Key–Value Organization

A dictionary maps hashable keys to values. Keys are unique; assigning an existing key replaces its value.

student = {
    "id": "S104",
    "name": "Amina",
    "score": 84
}
print(student["name"])
student["score"] = 87
student["status"] = "Pass"

Modern Python dictionaries preserve insertion order. This is useful for predictable iteration, but dictionaries should primarily be understood as mappings rather than positional sequences.

Accessing a missing key with student["email"] raises KeyError. Safer alternatives include:

email = student.get("email")
country = student.get("country", "Unknown")

Useful dictionary operations include keys(), values(), items(), update(), pop() and setdefault().

for key, value in student.items():
    print(key, value)

Dictionary comprehensions create mappings concisely:

squares = {n: n * n for n in range(1, 6)}

Keys must be hashable. Strings, numbers and suitable tuples are common keys; lists and dictionaries cannot be keys because they are mutable and unhashable.

6. Sets: Unique Membership and Set Algebra

A set is an unordered collection of unique hashable elements.

subjects = {"Python", "Statistics", "Databases"}
subjects.add("Visualisation")
subjects.discard("Networks")

An empty set must be created with set(), because {} creates an empty dictionary.

Sets support mathematical operations:

For example:

python_students = {"Amina", "Jon", "Wei"}
statistics_students = {"Jon", "Mira", "Wei"}
common = python_students & statistics_students

Sets are appropriate for removing duplicates and testing membership. They do not provide positional indexing, and their display order should not be relied upon for presentation or data meaning. Use a list when order or duplicates matter.

7. Copying, Aliasing and Nested Structures

Assignment does not copy an object; it creates another reference to the same object.

original = [1, 2, 3]
alias = original
alias.append(4)
# original is now [1, 2, 3, 4]

A shallow copy creates a new outer collection:

copy_one = original.copy()
copy_two = original[:]
copy_three = list(original)

For nested mutable structures, a shallow copy still shares inner objects:

records = [["Amina", 80], ["Jon", 75]]
shallow = records.copy()
shallow[0][1] = 95
# records[0][1] is also 95

A deep copy recursively copies nested objects:

from copy import deepcopy
independent = deepcopy(records)

Deep copying should be used deliberately. It can consume substantial memory and may be inappropriate for objects connected to files, databases or external resources. Often, constructing a new data structure explicitly is clearer and safer.

A common mistake is multiplying a nested list:

matrix = [[0] * 3] * 2
matrix[0][0] = 1
# Both rows appear changed because they reference the same inner list.

The correct construction is:

matrix = [[0] * 3 for _ in range(2)]

8. Sorting and Ordering Data

sorted(iterable) returns a new sorted list. The list method sort() changes a list in place and returns None.

values = [4, 1, 9, 2]
ascending = sorted(values)
values.sort(reverse=True)

Both use ascending order by default. A key function defines the property used for comparison.

students = [
    {"name": "Jon", "score": 75},
    {"name": "Amina", "score": 91},
    {"name": "Wei", "score": 84}
]
ranked = sorted(students, key=lambda item: item["score"], reverse=True)

Python sorting is stable: records with equal keys retain their original relative order. This permits multi-stage sorting, although a single key returning a tuple is often clearer:

ranked = sorted(students, key=lambda item: (-item["score"], item["name"]))

The negative score gives descending score order while names remain ascending. Sorting values of incompatible types can raise TypeError; data should be validated before ordering.

9. Nested and Structured Data

Real data is frequently hierarchical. A list of dictionaries is a practical representation of tabular records:

sales = [
    {"product": "A", "region": "North", "units": 12},
    {"product": "B", "region": "South", "units": 8},
    {"product": "A", "region": "South", "units": 15}
]

Filtering and aggregation can be expressed directly:

north = [row for row in sales if row["region"] == "North"]
total_units = sum(row["units"] for row in sales)

A dictionary of lists may represent grouped observations:

scores_by_course = {
    "Python": [78, 84, 91],
    "Statistics": [72, 80, 86]
}

A nested dictionary can represent hierarchical lookup data, but excessive nesting makes validation and access difficult. If a structure becomes complex, consider defining a class, using dataclasses, or adopting a tabular data library such as pandas. Built-in collections remain valuable for inputs, intermediate transformations and small datasets.

10. Selecting the Right Collection

Selection should follow the data’s semantic requirements.

Ask the following questions:

1. Does order carry meaning?
2. Are duplicates meaningful?
3. Must the structure change after creation?
4. Is lookup by a key required?
5. Is fast membership testing important?
6. Will the data be used as a dictionary key or set element?

For example, a student registry indexed by student ID is better represented as a dictionary than as a list requiring a search. A collection of distinct course codes is naturally a set. A fixed latitude–longitude pair is naturally a tuple, while a time series that receives new observations is naturally a list.

Applied Case: Processing a Course-Assessment Dataset

Background

This illustrative case concerns a diploma data-science course that stores assessment records for reporting.

Problem

The lecturer needs to identify passed students, calculate course averages, remove duplicate course codes and rank students by score. The data contains records with fields that should be accessed by name rather than by uncertain numeric position.

Approach

A list of dictionaries is selected because the data consists of multiple records with named fields.

records = [
    {"id": "S01", "name": "Amina", "course": "Python", "score": 86},
    {"id": "S02", "name": "Jon", "course": "Python", "score": 73},
    {"id": "S03", "name": "Wei", "course": "Statistics", "score": 91},
    {"id": "S04", "name": "Mira", "course": "Python", "score": 86}
]

passed = [r for r in records if r["score"] >= 50]
average = sum(r["score"] for r in records) / len(records)
courses = {r["course"] for r in records}
ranking = sorted(records, key=lambda r: (-r["score"], r["name"]))
by_id = {r["id"]: r for r in records}

The pass threshold is a business rule, not a property of Python. It should be documented and confirmed with the institution. The average is the arithmetic mean, applicable here because each record contributes one score with equal weight:

Here, xᵢ is the score of record i, and n is the number of records. For the four scores 86, 73, 91 and 86, the total is 336 and the mean is 84.0 score points.

Outcome

passed contains all four records, courses contains the two unique course names, ranking places Wei first, Amina and Mira next in alphabetical order, and Jon last. by_id supports direct lookup such as by_id["S03"].

Learning Points

The example demonstrates that one dataset can be viewed through several collections: the list preserves records, dictionaries provide named fields and direct identifier lookup, a set provides unique course names, and sorting creates a ranked view without changing the original order. In a larger system, validation should check missing fields, invalid scores, duplicate IDs and privacy permissions before processing.

Challenges, Quality and Governance

Collections do not automatically guarantee valid data. Programs should handle missing keys, unexpected types, duplicate identifiers and empty collections. Division by len(records) is invalid when the list is empty, so code should test for that condition.

Data governance is also relevant. Student identifiers and assessment results are personal information in many jurisdictions. Applications should minimize collected data, restrict access, avoid exposing records through debugging output and follow institutional retention policies. A dictionary keyed by a student ID improves lookup but does not itself provide security.

For larger datasets, built-in collections may become inefficient or difficult to query. Array libraries provide compact numerical storage, while database systems provide indexing, transactions and access control. Such tools complement rather than replace understanding of Python’s fundamental collections.

Best Practices

• Choose a collection based on meaning, not habit.
• Prefer descriptive keys over unexplained numeric positions for records.
• Use tuples to communicate fixed structure and lists for changing sequences.
• Use sets for uniqueness, not for ordered reporting.
• Remember that assignment creates an alias, not a copy.
• Use deepcopy only when nested independence is genuinely required.
• Prefer sorted when the original list must remain unchanged.
• Validate external data before indexing, sorting or aggregation.
• Avoid modifying a collection while directly iterating over it; build a new collection or iterate over a copy.
• Keep nested structures understandable and document their schema.

Chapter Summary

Python collections provide complementary ways to organize data. Lists are ordered and mutable; tuples are ordered and immutable; ranges represent integer progressions compactly; dictionaries associate unique keys with values; and sets represent unique membership. Indexing and slicing support sequence access, while comprehensions support concise transformation. Mutation, aliasing and shallow copying require careful attention, especially for nested structures. Sorting can create ordered views using key functions, and nested combinations such as lists of dictionaries model practical datasets. Effective data processing depends on selecting a collection whose ordering, uniqueness, mutability and lookup behavior match the problem. Professional use also requires validation, readable schemas and responsible handling of personal data.

Key Terms

Collection: An object that groups multiple values.

List: An ordered, mutable sequence.

Tuple: An ordered, immutable sequence.

Range: A compact representation of an integer progression.

Dictionary: A mapping from unique hashable keys to values.

Set: An unordered collection of unique hashable elements.

Index: An integer position used to access a sequence element.

Slice: A selected subsequence defined by start, stop and step.

Mutable: Capable of being changed after creation.

Immutable: Not changeable after creation.

Hashable: Suitable for use as a dictionary key or set element.

Aliasing: Multiple references pointing to the same object.

Shallow copy: A new outer collection whose nested objects may remain shared.

Deep copy: A recursively independent copy of nested objects.

Comprehension: A compact expression for constructing a collection.

Key function: A function defining the property used for sorting.

Nested structure: A collection containing other collections.

Membership: Whether an object belongs to a collection.

Schema: The intended structure and fields of organized data.

Review Questions

5 Short Answer Questions

1. What is the difference between a list and a tuple?
2. Why is the stop index excluded in a Python slice?
3. What happens when a dictionary is accessed with a missing key using square brackets?
4. Why is set() used to create an empty set?
5. What is the difference between sorted() and the list method sort()?

5 Long Answer Questions

1. Explain the characteristics and typical applications of lists, tuples, ranges, dictionaries and sets.
2. Discuss aliasing, shallow copying and deep copying with nested Python collections.
3. Explain how indexing, slicing and comprehensions support data transformation.
4. Describe how dictionaries and sets support efficient data-processing workflows.
5. Explain the factors that should guide collection selection in a professional data-science application.

3 Analytical / Application Questions

1. Design a collection structure for storing customer orders, where each order has an ID, customer name, multiple products and a total value. Justify every collection choice.
2. A program contains a list of monthly sales records and must produce unique regions, a region-wise total and a ranking by total sales. Describe an appropriate processing strategy using Python collections.
3. A developer copies a two-dimensional list using .copy() and finds that changing one row changes the original data. Explain the cause and propose two valid solutions.

Multiple Choice Questions with Answers

Question 1

Which collection is most appropriate for storing unique course codes where order is not important?

A. List

B. Tuple

C. Set

D. Range

Correct Answer: C. Set

Question 2

What is returned by values[1:4] for values = [10, 20, 30, 40, 50]?

A. [10, 20, 30]

B. [20, 30, 40]

C. [20, 30, 40, 50]

D. [10, 20, 30, 40]

Correct Answer: B. [20, 30, 40]

Question 3

Which statement about dictionary keys is correct?

A. Keys may be duplicated without replacing values

B. Keys must be hashable

C. Keys must always be strings

D. Keys are accessed only by position

Correct Answer: B. Keys must be hashable

Question 4

What does sorted(data) do to a list named data?

A. Sorts the list in place and returns None

B. Deletes duplicate elements

C. Returns a new sorted list

D. Converts the list into a set

Correct Answer: C. Returns a new sorted list

Question 5

Which expression creates two independent inner lists?

A. [[0] 3] 2

B. list([0, 0, 0]) * 2

C. [[0] * 3 for _ in range(2)]

D. ([0, 0, 0],) * 2

Correct Answer: C. [[0] * 3 for _ in range(2)]

Question 6

What is the result of set([2, 2, 3, 4])?

A. {2, 3, 4}

B. [2, 3, 4]

C. {2, 2, 3, 4}

D. (2, 3, 4)

Correct Answer: A. {2, 3, 4}

Question 7

For scores = [72, 81, 68], which expression retrieves the last score?

A. scores[3]

B. scores[0]

C. scores[-1]

D. scores[1]

Correct Answer: C. scores[-1]

Question 8

What is the arithmetic mean of the scores 86, 73, 91 and 86?

A. 82.0

B. 83.0

C. 84.0

D. 85.0

Correct Answer: C. 84.0

Question 9

Which structure is most suitable for direct lookup of a student record by student ID?

A. A dictionary keyed by student ID

B. A set of student names

C. A range of numeric positions

D. A tuple of all records

Correct Answer: A. A dictionary keyed by student ID

Question 10

What happens when alias = original is followed by alias.append(4) for a list?

A. Only alias changes

B. Only original changes

C. Both names refer to the changed list

D. Python creates a deep copy automatically

Correct Answer: C. Both names refer to the changed list