Collections, Text and Data Manipulation

Introduction

Python programs rarely operate on one value at a time. Artificial intelligence and machine-learning systems process collections of observations, features, labels, tokens, records, and configuration values. Python’s built-in collection types provide the basic structures for representing and transforming such data before specialized libraries such as NumPy, pandas, or PyTorch are introduced.

This chapter examines lists, tuples, dictionaries, sets, and strings. It explains their underlying purposes, indexing and slicing, comprehensions, unpacking, sorting, searching, and selection criteria. The central principle is that a data structure should be selected according to the operations a program must perform, not merely according to personal preference.

Python collections are objects. A variable stores a reference to an object, and several variables may refer to the same mutable object. This distinction is important when modifying lists or dictionaries. In contrast, strings and tuples are immutable: their contents cannot be changed after creation.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the characteristics and appropriate uses of lists, tuples, dictionaries, sets, and strings.

2. Apply indexing, negative indexing, slicing, and sequence traversal.

3. Construct collections using comprehensions and unpacking.

4. Sort and search collection data accurately and efficiently.

5. Distinguish mutation, reassignment, copying, and aliasing.

6. Select suitable collections for data-processing tasks.

7. Manipulate textual data while recognizing issues involving Unicode, normalization, and privacy.

8. Design a small collection-based data-processing solution.

1. A Model of Python Collections

A collection can be described by several properties:

• Ordering: whether elements have a meaningful iteration order.
• Multiplicity: whether duplicate values are retained.
• Mutability: whether the object can be changed in place.
• Access model: whether values are accessed by position, key, membership, or another relation.
• Hashability: whether an object can be used as a dictionary key or set element.

The principal built-in collections have these general characteristics:

| Type | Ordered iteration | Duplicates | Mutable | Typical access |

|---|---|---:|---:|---|

| `list` | Yes | Yes | Yes | Integer index |

| `tuple` | Yes | Yes | No | Integer index |

| `dict` | Insertion order retained | Keys unique | Yes | Key |

| `set` | No meaningful positional order | No | Yes | Membership |

| `str` | Yes | Yes | No | Character index |

Dictionary insertion order is guaranteed by modern Python language specifications, but dictionary order should not be confused with sorted order. A set is iterable, but it should not be treated as a sequence with stable positional access.

Mutability and Aliasing

Consider:

```python

scores = [70, 82, 91]

other_scores = scores

other_scores.append(88)

print(scores) # [70, 82, 91, 88]

```

Both names refer to the same list. Assignment does not copy the object. To create a shallow copy, use `scores.copy()`, `list(scores)`, or `scores[:]`:

```python

scores = [70, 82, 91]

other_scores = scores.copy()

other_scores.append(88)

print(scores) # [70, 82, 91]

```

A shallow copy duplicates the outer collection but not nested mutable objects. For deeply nested structures, `copy.deepcopy` may be required, although copying should be used deliberately because it can be expensive and may have semantic complications.

2. Lists: Flexible Ordered Collections

A list represents an ordered, mutable sequence. It is suitable when data must be traversed in order, duplicates matter, or elements may be added, removed, or updated.

```python

features = [2.4, 1.8, 3.1]

features.append(2.9)

features[0] = 2.5

```

Common list operations include:

```python

items = [10, 20, 30]

items.append(40) # Add one item at the end

items.extend([50, 60]) # Add all items from an iterable

items.insert(1, 15) # Insert before index 1

items.remove(30) # Remove first matching value

last = items.pop() # Remove and return final item

items.clear() # Remove all items

```

`remove(value)` raises `ValueError` if the value is absent, while `pop(index)` raises `IndexError` for an invalid index. These distinctions matter in robust programs.

Indexing and Slicing

Python uses zero-based indexing. For `data = ['a', 'b', 'c', 'd', 'e']`:

```python

data[0] # 'a'

data[-1] # 'e'

data[1:4] # ['b', 'c', 'd']

data[:3] # ['a', 'b', 'c']

data[2:] # ['c', 'd', 'e']

data[::2] # ['a', 'c', 'e']

data[::-1] # reversed copy

```

A slice has the form `sequence[start:stop:step]`. The start index is included and the stop index is excluded. A slice creates a new list, so assigning to a slice can replace a region:

```python

values = [1, 2, 3, 4]

values[1:3] = [20, 30, 40]

[1, 20, 30, 40]

```

An index accesses one element; a slice produces a sequence, possibly empty. Attempting to access an invalid index raises `IndexError`, whereas an out-of-range slice generally returns an empty or shortened sequence.

List Traversal

Use direct iteration when the index is unnecessary:

```python

for name in names:

print(name)

```

Use `enumerate` when both position and value are needed:

```python

for position, name in enumerate(names, start=1):

print(position, name)

```

`enumerate` is clearer and safer than manually maintaining a counter.

3. Tuples: Fixed-Structure Sequences

A tuple is an immutable ordered sequence:

```python

point = (4, 7)

record = ('A17', 82.5, True)

```

Parentheses are often optional; the comma creates the tuple:

```python

single = (42,)

not_a_tuple = (42)

```

Tuples support indexing, slicing, iteration, and unpacking, but not operations that modify their contents. They are useful for fixed records, coordinates, function results, and values that should not be changed accidentally.

```python

x, y = point

```

Tuple unpacking requires a compatible number of variables unless starred unpacking is used:

```python

first, *middle, last = [10, 20, 30, 40]

first = 10, middle = [20, 30], last = 40

```

Tuples can be dictionary keys or set elements if all their contents are hashable. A tuple containing a list cannot be used as a key because the list is mutable and unhashable.

Immutability does not make every referenced object immutable. For example:

```python

record = ('student-1', [80, 90])

record[1].append(95) # The list inside the tuple changes

```

The tuple’s references cannot be replaced, but the nested list remains mutable.

4. Dictionaries: Key–Value Representation

A dictionary maps unique, hashable keys to values:

```python

student = {

'id': 'S104',

'name': 'Amina',

'score': 86

}

```

Dictionaries are appropriate for records, lookup tables, frequencies, configuration, and relationships between identifiers and values.

```python

student['score'] = 90

student['grade'] = 'A'

```

Accessing a missing key with square brackets raises `KeyError`. The `get` method allows a default:

```python

attendance = student.get('attendance', 0)

```

Other useful methods include:

```python

student.keys()

student.values()

student.items()

student.update({'score': 92, 'active': True})

student.pop('active')

```

The views returned by `keys`, `values`, and `items` are dynamic views rather than independent lists. If an independent list is required, explicitly convert the view.

Dictionary Traversal

```python

for key, value in student.items():

print(key, value)

```

A dictionary comprehension constructs mappings concisely:

```python

squares = {n: n * n for n in range(1, 6)}

```

A frequency table is a common data-processing pattern:

```python

counts = {}

for word in words:

counts[word] = counts.get(word, 0) + 1

```

For more specialized counting, `collections.Counter` is often preferable, but understanding the basic dictionary pattern is fundamental.

Keys must be hashable. Strings, numbers, and suitable tuples are hashable; lists, dictionaries, and sets are not. Hashability means, in practical terms, that an object has a stable hash value and equality behavior suitable for a hash table.

5. Sets: Membership and Uniqueness

A set is an unordered collection of distinct hashable elements:

```python

skills = {'Python', 'SQL', 'Statistics'}

skills.add('Linux')

skills.discard('Java')

```

`discard` does nothing if an element is absent; `remove` raises `KeyError`. An empty set must be created with `set()`, because `{}` creates an empty dictionary.

Set operations express relationships directly:

```python

required = {'Python', 'SQL', 'Git'}

applicant = {'Python', 'Git', 'Linux'}

required - applicant # {'SQL'}

required & applicant # {'Python', 'Git'}

required | applicant # union of all skills

required <= applicant # subset test

```

Sets are particularly useful for membership testing and duplicate removal:

```python

unique_ids = set(ids)

if user_id in unique_ids:

...

```

A set does not preserve a meaningful order. If reproducible display order is required, use `sorted(my_set)`, which returns a list.

6. Strings and Text Processing

A string is an immutable sequence of Unicode characters. It supports indexing and slicing:

```python

text = 'Machine Learning'

text[0] # 'M'

text[-1] # 'g'

text[:7] # 'Machine'

```

Because strings are immutable, operations such as `lower` and `replace` return new strings:

```python

cleaned = text.lower().replace(' ', '_')

```

Important methods include:

```python

text.strip() # Remove leading/trailing whitespace

text.split() # Split on whitespace

text.split(',') # Split on commas

'-'.join(['A', 'B', 'C']) # 'A-B-C'

text.startswith('Machine')

text.endswith('Learning')

text.find('Learn') # Index, or -1 if absent

text.count('i')

```

`str.split()` without an argument treats runs of whitespace as one separator and removes surrounding whitespace. `str.split(',')` has different behavior and may produce empty fields when delimiters are adjacent.

Formatting Text

Formatted string literals are usually the clearest way to construct output:

```python

name = 'Ravi'

score = 87.456

message = f'{name} achieved {score:.1f}%'

```

Formatting should be separated from data processing where possible. This makes programs easier to test and supports alternative output formats.

Text Normalization

Text from users or external files may differ in case, whitespace, punctuation, or Unicode representation. A basic normalization pipeline might be:

```python

import unicodedata

raw = ' Café '

normalized = unicodedata.normalize('NFC', raw).strip().casefold()

```

`casefold()` is designed for caseless comparison more broadly than `lower()`. Normalization must be chosen according to the application: destroying capitalization or punctuation may be inappropriate if the original text is needed for presentation, legal evidence, or sentiment analysis.

Text processing also requires attention to encoding. Python strings represent Unicode text, while files and network messages use encoded bytes. Specify encodings explicitly when reading and writing files, commonly UTF-8:

```python

with open('records.txt', encoding='utf-8') as file:

lines = file.read().splitlines()

```

7. Comprehensions and Unpacking

Comprehensions provide a compact syntax for constructing collections from iterables. A list comprehension has the conceptual form:

```python

[expression for item in iterable if condition]

```

Examples:

```python

values = [3, 8, 11, 14]

even_squares = [x * x for x in values if x % 2 == 0]

[64, 196]

```

Set and dictionary comprehensions use `{}` with different internal forms:

```python

unique_lengths = {len(word) for word in words}

length_map = {word: len(word) for word in words}

```

A comprehension should remain readable. Deeply nested or condition-heavy comprehensions are often better expressed with ordinary loops or helper functions. A comprehension creates the resulting collection immediately; it is not inherently a memory-saving technique. For very large data, a generator expression may be more suitable:

```python

total = sum(x * x for x in range(1_000_000))

```

Unpacking assigns elements from an iterable to variables:

```python

name, score = ('Leila', 91)

first, *remaining = [1, 2, 3, 4]

```

It also works in function calls and dictionary construction:

```python

a = [1, 2]

b = [3, 4]

combined = [a, b]

base = {'timeout': 10}

options = {**base, 'retries': 3}

```

When keys overlap in dictionary unpacking, later entries take precedence. This is useful for configuration overrides but should be documented because silent overwriting can hide mistakes.

8. Sorting and Searching

The `sorted` function returns a new list, while the list method `sort` modifies a list in place and returns `None`:

```python

scores = [72, 95, 81]

ascending = sorted(scores)

scores.sort(reverse=True)

```

A key function determines the comparison value:

```python

students = [

{'name': 'Bo', 'score': 78},

{'name': 'Ana', 'score': 92},

{'name': 'Chen', 'score': 85}

]

ranked = sorted(students, key=lambda student: student['score'], reverse=True)

```

Python sorting is stable: if two elements have equal keys, their original relative order is preserved. Stable sorting supports multi-stage sorting, although a tuple key is often clearer:

```python

ranked = sorted(students, key=lambda s: (-s['score'], s['name']))

```

The key function is evaluated in a way that makes sorting efficient for ordinary use. Sorting a collection of `n` elements generally requires time proportional to `n log n` in the comparison-based model; exact performance depends on the implementation and input.

For membership tests, `in` is simple and expressive:

```python

if target in values:

print('Found')

```

Membership in a list or tuple generally requires scanning elements, whereas membership in a set or dictionary is typically constant-time on average, assuming a well-behaved hash function. This is an average-case property, not a guarantee for every possible implementation or adversarial input.

For a sorted list, binary search can reduce search time. Python’s `bisect` module supports insertion points and binary-search-style operations:

```python

from bisect import bisect_left

values = [10, 20, 30, 40]

position = bisect_left(values, 30)

found = position < len(values) and values[position] == 30

```

The list must already be sorted according to the same ordering used by the search. Binary search does not make insertion into a Python list inexpensive, because shifting elements may still require linear time.

9. Choosing the Appropriate Collection

Selection should follow the dominant operation:

• Use a list for ordered, changeable sequences and repeated values.
• Use a tuple for fixed-position records or values that should be immutable.
• Use a dictionary for key-based lookup and structured records.
• Use a set for uniqueness and fast membership tests.
• Use a string for immutable Unicode text.

For example, a student-data system might represent one student as a dictionary, a class as a list of student dictionaries, the set of enrolled course codes as a set, and a coordinate as a tuple. These structures can be nested:

```python

students = [

{'id': 'S1', 'name': 'Asha', 'courses': {'AI', 'Python'}},

{'id': 'S2', 'name': 'Jon', 'courses': {'Python', 'Databases'}}

]

```

The choice should also consider invariants. If order is meaningful, do not use a set. If keys may repeat, a dictionary alone cannot represent all records without grouping values. If frequent positional insertion at the beginning is required, a list may be inefficient and `collections.deque` may be more suitable.

10. Worked Application: Processing a Course-Feedback Dataset

The following is an illustrative case. A university wants to summarize anonymous course-feedback records. Each record contains a student identifier, a comment, and a rating:

```python

feedback = [

{'student': 'S01', 'rating': 5, 'comment': 'Clear explanations and useful examples.'},

{'student': 'S02', 'rating': 4, 'comment': 'Useful examples, but the pace was fast.'},

{'student': 'S03', 'rating': 5, 'comment': 'Clear explanations and supportive labs.'},

{'student': 'S04', 'rating': 3, 'comment': 'The pace was fast.'}

]

```

Problem

The coordinator needs to identify the average rating, frequent words, unique ratings, and comments containing the word “clear,” while preserving the original records.

Analysis

The outer data is an ordered list because each feedback record is retained. Each record is naturally represented by a dictionary. A set is appropriate for unique ratings, while a dictionary is appropriate for word frequencies. Text must be normalized before comparison.

Solution

```python

import re

ratings = [record['rating'] for record in feedback]

average_rating = sum(ratings) / len(ratings)

unique_ratings = set(ratings)

word_counts = {}

for record in feedback:

words = re.findall(r"[A-Za-z]+", record['comment'].casefold())

for word in words:

word_counts[word] = word_counts.get(word, 0) + 1

clear_comments = [

record for record in feedback

if 'clear' in record['comment'].casefold()

]

ranking = sorted(

feedback,

key=lambda record: (-record['rating'], record['student'])

)

```

There are four records, and their ratings sum to 17. Therefore, under the assumption that every record has a numeric rating and that the arithmetic mean is the desired summary, the average rating is `17 / 4 = 4.25`. The unique rating set is `{3, 4, 5}`. The word-frequency dictionary counts normalized words without treating capitalization as significant. The ranking sorts by descending rating and then ascending student identifier.

Outcome and Learning Points

The solution separates representation from processing. Lists preserve records, dictionaries provide named fields and counts, sets provide uniqueness, and strings support normalization and extraction. The original feedback is not modified. In a production system, additional validation would be required for missing fields, invalid ratings, multilingual text, personally identifiable information, and comments containing punctuation or non-Latin scripts.

11. Professional Practice, Ethics, and Limitations

Collections are simple to use but can produce serious data-quality problems when assumptions remain implicit. Important practices include:

• Validate external data before indexing required fields.
• Avoid assuming that text consists only of ASCII characters.
• Preserve original data when transformations are not reversible.
• Use explicit schemas for records shared between teams or services.
• Do not expose personal identifiers unnecessarily in logs or examples.
• Treat dictionary keys and text labels as case-sensitive or case-insensitive intentionally, not accidentally.
• Document whether duplicate records are meaningful.
• Use deterministic sorting when results are used in reports or tests.

In AI and ML workflows, preprocessing decisions can affect model outcomes. Converting text to lowercase, removing punctuation, or deduplicating records may alter minority-language content, names, identifiers, or legally significant statements. A set-based deduplication step can also discard repeated observations that represent genuine frequency. Data manipulation is therefore not merely mechanical; it embodies assumptions about meaning and quality.

For large datasets, built-in collections may not be sufficient. NumPy arrays provide homogeneous numerical storage and vectorized operations; pandas DataFrames provide labeled tabular operations; databases provide indexing, transactions, and concurrent access. Nevertheless, Python’s built-in collections remain essential for glue code, configuration, model inputs, parsing, and algorithmic logic.

12. Best Practices Checklist

1. Choose a collection based on access patterns and invariants.

2. Use direct iteration and `enumerate` rather than manual index management where possible.

3. Distinguish `sort()` from `sorted()` and remember that `sort()` returns `None`.

4. Use `get` when a missing dictionary key is expected and meaningful.

5. Use sets for membership and uniqueness, not for ordered output.

6. Use tuple keys only when all nested components are hashable.

7. Keep comprehensions short and readable.

8. Normalize text only when the application requires it.

9. Copy mutable collections deliberately to avoid accidental aliasing.

10. Test empty collections, missing keys, duplicate values, unusual Unicode, and invalid input.

Chapter Summary

Python collections provide complementary models for data manipulation. Lists are mutable ordered sequences; tuples are immutable ordered sequences; dictionaries map unique keys to values; sets represent distinct elements and support efficient membership operations; and strings are immutable Unicode sequences. Indexing and slicing support sequence access, while comprehensions express common transformations concisely. Unpacking improves assignment and composition, and sorting supports ordered analysis through key functions and stable ordering. Searching should reflect the data structure: lists require scanning in general, whereas sets and dictionaries provide efficient average-case membership testing. Text processing requires careful attention to immutability, encoding, normalization, and language diversity. In professional AI and ML programming, appropriate collection selection, validation, reproducibility, privacy awareness, and preservation of data meaning are as important as syntactic correctness.

Key Terms

Collection: An object that groups multiple values for storage or processing.

Sequence: An ordered collection supporting positional access.

Mutable: Capable of being changed in place after creation.

Immutable: Not capable of having its contents changed after creation.

List: A mutable ordered sequence that permits duplicates.

Tuple: An immutable ordered sequence.

Dictionary: A mutable mapping from unique hashable keys to values.

Set: A mutable collection of unique hashable elements.

String: An immutable sequence of Unicode characters.

Indexing: Accessing an element by position or key.

Slicing: Extracting a subsequence using start, stop, and step values.

Comprehension: Compact syntax for constructing a collection from an iterable.

Unpacking: Assigning elements of an iterable or mapping to separate targets.

Hashable: Having a stable hash value and equality behavior suitable for hash-based collections.

Aliasing: The situation in which multiple names refer to the same object.

Normalization: Transforming data into a consistent representation.

Stable sort: A sort that preserves the relative order of equal-key elements.

Membership: Testing whether an element occurs in a collection.

Shallow copy: A copy of an outer collection whose nested references remain shared.

Unicode: A standard system for representing characters from many writing systems.

Review Questions

Short Answer Questions

1. What is the difference between a list and a tuple in terms of mutability?

2. Why must an empty set be created with `set()` rather than `{}`?

3. What is the difference between `sort()` and `sorted()`?

4. Why are lists generally unsuitable as dictionary keys?

5. What does the stop index mean in a Python slice?

Long Answer Questions

1. Explain the main properties of lists, tuples, dictionaries, sets, and strings, and give one suitable application for each.

2. Describe dictionary construction, lookup, updating, deletion, and traversal, including the handling of missing keys.

3. Explain list, set, and dictionary comprehensions. Discuss when a normal loop may be more appropriate.

4. Discuss string immutability, Unicode, encoding, normalization, and the risks of simplistic text preprocessing.

5. Compare list, set, dictionary, and binary-search approaches to membership testing and explain how data structure choice affects performance.

Analytical / Application Questions

1. Design a data representation for an online bookstore containing books, authors, prices, categories, and customer orders. Justify every collection choice.

2. A data-cleaning program removes duplicate records by converting complete records to sets. Analyze why this may fail and propose a safer approach.

3. Given a collection of transaction dictionaries, design a procedure to calculate category totals, identify categories appearing in every transaction batch, and produce a deterministic ranking. State how you would handle missing or invalid data.

Multiple Choice Questions with Answers

Question 1

Which collection is most appropriate for testing membership while ensuring that duplicate values are removed?

A. List

B. Tuple

C. Set

D. String

Correct Answer: C. Set

Question 2

What is the result of the slice expression `[10, 20, 30, 40][1:3]`?

A. [10, 20]

B. [20, 30]

C. [20, 30, 40]

D. [30, 40]

Correct Answer: B. [20, 30]

Question 3

Which statement correctly describes `sorted(values)`?

A. It always changes `values` in place.

B. It returns a new sorted list.

C. It works only with tuples.

D. It returns `None` after sorting.

Correct Answer: B. It returns a new sorted list.

Question 4

Which expression safely obtains a missing dictionary key with a default value of zero?

A. `data['count', 0]`

B. `data.get('count', 0)`

C. `data.value('count', 0)`

D. `data.find('count', 0)`

Correct Answer: B. `data.get('count', 0)`

Question 5

Why does `a = b` not create an independent copy when `b` is a list?

A. Lists cannot be assigned to variables.

B. Assignment creates a reference to the same list object.

C. Python automatically converts lists to tuples.

D. Only strings can be copied.

Correct Answer: B. Assignment creates a reference to the same list object.

Question 6

What does `first, *middle, last = [1, 2, 3, 4]` assign to `middle`?

A. 1

B. [1, 2]

C. [2, 3]

D. 4

Correct Answer: C. [2, 3]

Question 7

Which statement about a Python set is correct?

A. It preserves duplicate elements.

B. It supports reliable positional indexing.

C. Its elements must be hashable.

D. It is created with `{}` when empty.

Correct Answer: C. Its elements must be hashable.

Question 8

What does `'-'.join(['A', 'B', 'C'])` return?

A. `ABC`

B. `A-B-C`

C. `['A-B-C']`

D. `A B C`

Correct Answer: B. `A-B-C`

Question 9

For the list `values = [10, 20, 30, 40]`, what does `bisect_left(values, 30)` return?

A. 0

B. 1

C. 2

D. 3

Correct Answer: C. 2

Question 10

Which collection is most suitable for a fixed two-dimensional coordinate such as `(4, 7)`?

A. Tuple

B. Set

C. Dictionary

D. String

Correct Answer: A. Tuple