9. Querying, Indexing and Data Processing

Introduction and Context

A NoSQL database is valuable not merely because it stores data without a relational schema, but because it supports access patterns that are difficult, expensive, or inflexible in traditional systems. Querying, indexing, and data processing therefore form a connected design discipline. A query expresses what an application needs; an index determines how efficiently the database can locate relevant data; and a processing framework transforms, combines, ranks, or traverses that data to produce useful results.

In a document database, a request may involve equality matching, range filtering, array conditions, projections, sorting, grouping, or full-text search. In a graph database, the equivalent task may be expressed as a traversal through vertices and edges. In a wide-column database, efficient access commonly depends on partition-key equality followed by restrictions on clustering columns. These differences mean that query design cannot be separated from data modeling. A model that supports fast writes but ignores retrieval patterns may produce unacceptable response times. Conversely, excessive indexing can make reads fast while increasing storage consumption and write latency.

This chapter develops a practical and analytical approach to querying NoSQL systems. MongoDB-style document examples are used for queries and aggregation, while graph and distributed wide-column concepts are introduced for comparison. The examples are illustrative rather than tied to one vendor. The central principle is that performance results from alignment among data shape, query pattern, index structure, workload, and operational constraints.

Learning Objectives

By the end of this chapter, students should be able to:

1. Construct equality, range, array, projection, sorting, aggregation, traversal, and search queries.

2. Explain the roles of primary, secondary, compound, geospatial, and full-text indexes.

3. Select indexes from documented access patterns rather than intuition alone.

4. Interpret query plans, selectivity, cardinality, examined records, and execution cost.

5. Design aggregation pipelines and graph traversals for practical applications.

6. Identify indexing trade-offs, operational risks, and governance requirements.

7. Evaluate an applied NoSQL querying and indexing strategy.

Querying as an Access-Pattern Problem

A query is an executable description of an information requirement. It may ask for documents satisfying conditions, fields to return, an order, a limit, or a transformation. In NoSQL development, the first question should not be “What tables would represent this data?” but “Which questions must the system answer, and at what scale?”

Consider a collection named `orders`:

```javascript

{

_id: 501,

customerId: "C42",

status: "PAID",

placedAt: ISODate("2025-03-10T11:30:00Z"),

total: 185.50,

items: [

{ sku: "KB7", quantity: 1, price: 120 },

{ sku: "MS2", quantity: 2, price: 32.75 }

],

shipping: { city: "Pune", country: "IN" }

}

```

An equality query might retrieve one customer's paid orders:

```javascript

db.orders.find({ customerId: "C42", status: "PAID" })

```

A range query can identify recent high-value orders:

```javascript

db.orders.find({

placedAt: { $gte: ISODate("2025-03-01"), $lt: ISODate("2025-04-01") },

total: { $gte: 100 }

})

```

A projection limits the fields returned:

```javascript

db.orders.find(

{ customerId: "C42" },

{ _id: 0, placedAt: 1, total: 1, status: 1 }

)

```

Projection reduces network transfer and application-side processing, although it does not automatically make every query index-only. Sorting and limiting are also part of the access pattern:

```javascript

db.orders.find({ status: "PAID" })

.sort({ placedAt: -1 })

.limit(20)

```

Array fields require particular care. A query such as `{ "items.sku": "KB7" }` identifies orders containing that product. When several conditions must apply to the same array element, an element-match operator is safer:

```javascript

db.orders.find({

items: { $elemMatch: { sku: "KB7", quantity: { $gte: 2 } } }

})

```

Without an appropriate array operator, separate conditions may be satisfied by different elements, producing an unintended result. This is an example of semantic correctness preceding optimization.

Query Models Across NoSQL Families

Document databases generally support predicate queries over nested objects and arrays, together with aggregation pipelines. Their flexibility is useful for content systems, profiles, catalogs, and event records. However, flexible structure does not eliminate the need for controlled validation and consistent field types.

Wide-column databases, such as systems based on partition and clustering keys, are commonly queried according to a predetermined primary-key design. An efficient query normally identifies a partition and then narrows or orders rows within it. Queries that require scanning unrelated partitions are often unsuitable for production workloads. The model is therefore query-driven: a separate table may be created for each important access pattern.

Key-value databases optimize direct lookup by key. They are excellent for sessions, caches, configuration values, and identity-based retrieval, but are not intended for arbitrary filtering across all values. Search requirements may need a separate search engine or a deliberately designed secondary index.

Graph databases represent entities as vertices and relationships as edges. A traversal can answer questions such as “Which suppliers are connected to a product through two or three relationship types?” The traversal begins at selected vertices and follows constrained edges rather than scanning unrelated records.

Aggregation Pipelines and Data Processing

A query returns matching records; an aggregation operation derives information from them. Document databases commonly implement aggregation as an ordered pipeline. Each stage receives the output of the previous stage. Typical stages filter, unwind arrays, group, sort, project, join related collections, or write results elsewhere.

The following pipeline calculates paid sales by city for a specified period:

```javascript

db.orders.aggregate([

{ $match: {

status: "PAID",

placedAt: { $gte: ISODate("2025-03-01"), $lt: ISODate("2025-04-01") }

}},

{ $group: {

_id: "$shipping.city",

orderCount: { $sum: 1 },

revenue: { $sum: "$total" }

}},

{ $sort: { revenue: -1 } },

{ $limit: 10 }

])

```

The early `$match` is strategically important. Filtering before grouping reduces the volume that later stages must process and may allow an index to be used. A pipeline that begins by expanding every array or joining every document may consume substantial memory and CPU.

To calculate product-level quantities, the array is unwound:

```javascript

db.orders.aggregate([

{ $match: { status: "PAID" } },

{ $unwind: "$items" },

{ $group: {

_id: "$items.sku",

units: { $sum: "$items.quantity" },

sales: { $sum: { $multiply: ["$items.quantity", "$items.price"] } }

}},

{ $sort: { sales: -1 } }

])

```

Aggregation has several design concerns. Grouping by high-cardinality values can create large intermediate results. Unbounded pipelines may exceed memory or timeout. Joining through a lookup stage can be useful, but repeated cross-collection joins may indicate that a read-optimized representation or controlled denormalization is preferable. Results intended for dashboards may be materialized periodically rather than recalculated on every request.

Data processing may be batch, streaming, or near real time. Batch aggregation is appropriate for daily financial summaries. Streaming processors are more suitable for fraud alerts or operational monitoring, where events must be evaluated shortly after arrival. In either case, idempotency, event ordering, duplicate handling, and late-arriving data must be addressed.

Graph Traversals and Search Operations

Graph queries are evaluated through paths. A conceptual Cypher query for recommending products might be:

```cypher

MATCH (u:User {id: 'U7'})-[:PURCHASED]->(p:Product)<-[:PURCHASED]-(other:User)-[:PURCHASED]->(recommendation:Product)

WHERE NOT (u)-[:PURCHASED]->(recommendation)

RETURN recommendation, count(other) AS support

ORDER BY support DESC

LIMIT 10

```

This query uses common purchasing behavior as evidence for a recommendation. Efficient graph design requires suitable starting points, selective labels or properties, bounded path lengths, and avoidance of uncontrolled variable-length traversals. A traversal that explores an entire highly connected social network can become computationally expensive and may expose more data than intended.

Full-text search differs from equality matching. A search engine usually analyzes text, applies tokenization and normalization, builds an inverted index, and ranks matching documents. Search features may include stemming, phrase matching, fuzzy matching, field weighting, synonyms, and highlighting. A document database’s basic text index may be sufficient for modest requirements, whereas multilingual, relevance-sensitive, or typo-tolerant applications often require a dedicated search service.

Geospatial queries operate on coordinates or shapes. A two-dimensional spherical index can support “near this point” or “within this polygon” requests. Data must use a consistent coordinate system and valid geometry. Location-based applications should also consider whether storing precise coordinates creates privacy risks.

Indexes: Structures That Trade Write Cost for Read Efficiency

An index is an auxiliary data structure that allows the database to locate candidate records without examining every stored record. The primary index is normally associated with the record’s key and is essential for direct identity lookup. Secondary indexes support other fields or expressions.

An index on `customerId` can accelerate equality queries:

```javascript

db.orders.createIndex({ customerId: 1 })

```

A compound index contains multiple fields and is often more useful than several unrelated indexes when queries combine conditions, sorting, or both:

```javascript

db.orders.createIndex({ customerId: 1, status: 1, placedAt: -1 })

```

The ordering of fields matters. The leftmost-prefix principle means that the index is generally most useful when a query constrains the leading fields. A compound index beginning with `customerId` is well suited to one customer’s orders, but less suited to a query filtering only on `status`. Equality fields are often placed before range fields, and a field supporting a required sort may influence the final ordering. These are guidelines, not mechanical rules; actual plans must be measured.

Multikey indexes support array fields, but arrays can increase index entries and create restrictions on compound designs. Sparse or partial indexes can avoid indexing documents that do not contain a field or do not satisfy a condition. A unique index enforces a business rule such as one account per email address, but uniqueness must be interpreted alongside normalization, case sensitivity, and distributed write behavior.

Geospatial indexes support spatial predicates. Text indexes or external search indexes support token-based retrieval. Hashed indexes distribute equality keys more evenly in some partitioning contexts, but they do not support ordinary range ordering. Each index type is therefore tied to particular query semantics.

Index selection should consider four properties:

• Selectivity: the proportion of records eliminated by a predicate.
• Cardinality: the number of distinct values in a field or combination of fields.
• Workload frequency: how often a query runs and how important its latency is.
• Mutation cost: the storage, memory, and write overhead of maintaining the index.

An index on a field containing only two values may have poor selectivity by itself. It can nevertheless be useful as part of a compound index when combined with a high-cardinality field or a required sort. Conversely, an index with excellent selectivity may not justify its cost if the query runs once per day.

Reading Query Plans and Measuring Cost

A query plan describes how the database intends to execute a query. Typical stages include an index scan, collection or table scan, fetch of complete records, filtering, sorting, grouping, and merging across partitions or shards. An explain facility exposes this information.

Important measures include:

• Keys examined: index entries inspected.
• Documents or rows examined: records read before filtering.
• Returned count: records delivered to the application.
• Execution time: elapsed execution cost under the test conditions.
• Index coverage: whether all required fields can be obtained from the index without fetching records.
• Selectivity ratio: a useful comparison between examined and returned records.

A collection scan is not automatically wrong. It may be appropriate for a small collection, a query returning most records, or a one-time administrative operation. It becomes a concern when a high-volume request repeatedly examines a large collection. Similarly, an index scan may still be expensive if it examines a large fraction of the index or requires a costly sort.

Plans should be tested with realistic data volume, distributions, concurrency, and cache state. A plan that is fast on one thousand records may be unacceptable on one hundred million. Query planners may select among candidate indexes, and data distribution can change the best choice over time. Production monitoring should therefore track slow queries, latency percentiles, resource consumption, and plan changes rather than relying only on development tests.

Index cost is multidimensional. Storage cost includes index pages and memory pressure. Write cost includes insertion, update, and deletion maintenance. Operational cost includes build time, replication traffic, rolling deployment complexity, and possible impact on availability. A practical design treats indexes as production resources, not free configuration switches.

An Illustrative Case: Regional Marketplace Search

Background

Suppose MarketBridge is an online marketplace operating in several cities. Its document store contains products, sellers, orders, and user search events. The initial system uses a single flexible products collection and only an index on product identity.

Situation and Problem

Product-detail requests are fast, but category pages take several seconds during peak periods. Users search by text, filter by category and price, sort by recent listing, and sometimes request products within a delivery radius. Analysts also need daily sales summaries. Engineers propose adding indexes to every frequently queried field.

Analysis

Uncontrolled indexing would increase write latency because listings change availability, price, and inventory. The team first records actual access patterns and query plans. Category pages commonly use equality on `categoryId`, a range on `price`, and descending order on `listedAt`. Search requests are text-oriented and require relevance ranking, which is not adequately addressed by a simple equality index. Geospatial requests require coordinate-aware indexing. The analytics workload scans many orders and should not compete with interactive requests.

Approach

The team adopts a compound index for the dominant category access pattern, after testing alternative field orders against realistic distributions. It creates a dedicated text-search index containing product title, description, and selected attributes, with field weights for relevance. A geospatial index supports radius queries, while a partial index covers active listings only because unavailable products are not displayed in normal browsing.

The order summary pipeline begins with a time and status filter, groups by region and product category, and writes results to a reporting collection on a schedule. The application requests only fields needed for card displays. Query plans are captured before and after deployment, and index usage is reviewed monthly. Product updates are tested because every additional index increases write work.

Outcome

The design reduces unnecessary record examination for category pages and separates relevance search from ordinary filtering. Reporting no longer performs large analytical operations during customer requests. The system accepts some additional storage and write cost in exchange for predictable interactive performance.

Learning Points

The case demonstrates that indexing is a workload decision, not a checklist. Search, geospatial filtering, browsing, and analytics are different workloads and may require different structures. Early filtering, projection, partial indexes, materialized summaries, and continuous plan observation work together. Most importantly, the team indexes documented queries rather than imagined future requirements.

Common Errors, Risks, and Governance Considerations

A frequent mistake is creating one index per field without examining compound query patterns. Another is assuming that an index guarantees fast execution even when a predicate has low selectivity or when sorting and filtering are incompatible with the index order. Developers may also overlook type inconsistency—for example, storing a price as both a string and a number—which can produce incorrect comparisons and unstable plans.

Unbounded pagination is another problem. Skipping a very large number of records becomes increasingly expensive in many systems. Range-based or seek pagination, using a stable ordered key, is usually more scalable. Aggregation pipelines should avoid unnecessary expansion, and graph traversals should impose depth and result limits.

Security and governance are integral to querying. Query parameters must be validated to prevent injection, unauthorized operators, or excessive resource consumption. Least-privilege database accounts should restrict sensitive collections and operations. Search indexes can replicate personal information into additional storage, increasing the impact of a breach. Location and behavioral data require careful retention, access control, consent, and anonymization policies where applicable. Audit logs should record sensitive access without exposing unnecessary data.

Indexes can also become a governance issue when they retain information beyond the business purpose or complicate deletion requests. Regulatory obligations concerning privacy, residency, retention, and explainability may apply to derived aggregates and search indexes as well as to the source data.

Automation, Observability, and Emerging Practice

Modern platforms increasingly automate index suggestions, workload profiling, query-plan analysis, and anomaly detection. These tools can identify slow queries or propose candidate indexes, but recommendations require human review. Automated tools may optimize average latency while overlooking write-heavy periods, storage budgets, privacy constraints, or rare but critical queries.

Operational practice is moving toward continuous performance engineering. Query fingerprints, distributed traces, execution statistics, and cost dashboards connect application behavior to database work. In distributed systems, query performance also depends on partition distribution, network transfer, replication, consistency level, and cross-region coordination. Vector search is an emerging capability for semantic retrieval using embeddings. It is useful for similarity-based discovery, but it introduces additional concerns: embedding quality, model drift, explainability, access control, and the cost of approximate nearest-neighbor indexing.

Best-Practice Decision Framework

A disciplined implementation can follow this sequence:

1. Document the business query and its latency, freshness, and consistency requirements.

2. Identify the data model and determine whether the query is a natural operation for that model.

3. Estimate selectivity, cardinality, frequency, result size, and growth.

4. Construct the simplest semantically correct query, with projection and limits where appropriate.

5. Design a primary or secondary index that supports the dominant predicate and required ordering.

6. Test alternative plans with representative data and concurrent workload.

7. Measure both read improvement and write, storage, memory, and replication costs.

8. Monitor production behavior and remove indexes that no longer justify their cost.

9. Apply authorization, validation, retention, and audit controls to the query path.

This framework prevents performance tuning from becoming isolated trial and error. It also reinforces the broader NoSQL principle that operational behavior must be designed together with data representation.

Chapter Summary

Efficient NoSQL querying begins with access patterns. Document queries support predicates, projections, arrays, sorting, and aggregation pipelines; graph systems answer relationship questions through bounded traversals; and search systems use analyzed indexes and ranking rather than simple equality matching. Primary, secondary, compound, multikey, geospatial, text, partial, and other indexes serve different purposes and impose different costs.

Index selection depends on selectivity, cardinality, workload frequency, ordering, data growth, and mutation overhead. Query plans reveal scans, examined records, sorting, fetching, and coverage, but they must be interpreted using realistic data and production-like workloads. Aggregation should filter early and control intermediate results. Governance requires validation, authorization, privacy protection, retention management, and auditing. The professional objective is not to maximize the number of indexes, but to achieve an evidence-based balance between latency, throughput, correctness, cost, and maintainability.

Key Terms

Access pattern: A recurring way in which an application reads, filters, sorts, or transforms data.

Aggregation pipeline: An ordered sequence of data-processing stages that transforms query results.

Cardinality: The number of distinct values represented by a field or field combination.

Collection scan: An execution method that examines records across a collection without using a selective index.

Compound index: An index containing two or more fields in a defined order.

Covered query: A query whose required predicates and returned fields can be satisfied entirely from an index.

Full-text search: Retrieval of text using tokenization, analysis, matching, and relevance ranking.

Geospatial index: An index designed for coordinate or geometric queries.

Index selectivity: The degree to which an index predicate narrows the candidate record set.

Key-value query: Direct retrieval of a value using its associated key.

Multikey index: An index capable of indexing values contained in array fields.

Query plan: The database’s chosen sequence of operations for executing a query.

Range query: A query using ordered comparisons such as greater than, less than, or between.

Secondary index: An index that supports access through attributes other than the primary key.

Seek pagination: Pagination that continues from a known ordered key rather than skipping many records.

Text analyzer: A component that transforms text into searchable terms through operations such as tokenization and normalization.

Traversal: Movement through connected vertices and edges in a graph database.

Write amplification: Additional write work caused by maintaining indexes, replicas, or derived data.

Review Questions

Short Answer Questions

1. What is meant by an access pattern in NoSQL database design?

2. Distinguish between cardinality and selectivity.

3. Why is the order of fields important in a compound index?

4. What is the purpose of an aggregation pipeline’s early filtering stage?

5. State two measures commonly examined in a query plan.

Long Answer Questions

1. Explain how document, wide-column, key-value, and graph databases differ in their approach to querying.

2. Discuss the benefits and costs of compound, multikey, partial, geospatial, and full-text indexes.

3. Describe a systematic method for selecting and validating indexes for a production workload.

4. Explain how aggregation pipeline design affects memory, latency, and scalability.

5. Analyze the relationship between query correctness, performance optimization, and governance.

Analytical / Application Questions

1. A query filters orders by `customerId`, `status`, and a date range, then sorts by date descending. Propose an index strategy and explain what evidence you would collect before deployment.

2. An application has fast text search but slow category browsing and high write latency after several indexes are added. Diagnose the likely causes and recommend a revised approach.

3. Design a query-processing strategy for a location-based marketplace that requires radius search, relevance-ranked product search, and daily regional sales reports. Justify the structures and workload separation you choose.

22. Multiple Choice Questions with Answers

Question 1

Which factor most directly describes how effectively an index narrows the candidate records?

A. Replication factor

B. Selectivity

C. Consistency level

D. Document size

Correct Answer: B. Selectivity

Question 2

Why is an early filtering stage generally preferred in an aggregation pipeline?

A. It guarantees a covered query

B. It reduces the data processed by later stages

C. It removes the need for indexes

D. It converts a graph into documents

Correct Answer: B. It reduces the data processed by later stages

Question 3

A compound index begins with customerId, followed by status and placedAt. Which query is most naturally aligned with its leftmost prefix?

A. A query filtering only on placedAt

B. A query filtering only on status

C. A query filtering on customerId and status

D. A query filtering only on total

Correct Answer: C. A query filtering on customerId and status

Question 4

Which query operation is most appropriate for conditions that must apply to the same element of an array?

A. Element-match semantics

B. Collection scanning only

C. Unrestricted grouping

D. Hash partitioning

Correct Answer: A. Element-match semantics

Question 5

What does a collection scan generally indicate?

A. Every result was obtained from an index

B. The database examined records without a selective index path

C. The query used a geospatial index

D. The query was automatically cached

Correct Answer: B. The database examined records without a selective index path

Question 6

Which workload most clearly requires a full-text search capability rather than a simple equality index?

A. Find an order by its unique identifier

B. Find products matching terms with relevance ranking

C. Retrieve a session by its key

D. Retrieve all orders for one customer

Correct Answer: B. Find products matching terms with relevance ranking

Question 7

Why can adding indexes increase write latency?

A. Each write may require maintenance of additional index structures

B. Indexes disable replication

C. Indexes remove primary keys

D. Each index converts documents into graph edges

Correct Answer: A. Each write may require maintenance of additional index structures

Question 8

Which design best limits the risk of an uncontrolled graph traversal?

A. Remove all vertex labels

B. Use bounded depth, selective starting points, and result limits

C. Traverse every edge before applying filters

D. Store every relationship as a text field

Correct Answer: B. Use bounded depth, selective starting points, and result limits

Question 9

What is the principal purpose of an explain facility?

A. To encrypt query parameters

B. To show the intended business meaning of a document

C. To expose the operations and costs of a query plan

D. To create a backup automatically

Correct Answer: C. To expose the operations and costs of a query plan

Question 10

Why might a reporting aggregation be written to a materialized summary collection?

A. To avoid all data validation

B. To separate repeated analytical work from interactive requests

C. To eliminate the need for source data

D. To make every query a key-value lookup

Correct Answer: B. To separate repeated analytical work from interactive requests