Web, Enterprise, and Distributed Search

Introduction

Information retrieval systems become substantially more difficult when the collection is the open Web, a large enterprise, or a geographically distributed set of repositories. In a small local collection, documents can often be indexed in one process and queried against one index. Web and enterprise systems must instead discover documents, respect policies, handle rapid change, remove duplicates, rank interconnected resources, distribute storage and computation, and enforce security for every result.

This chapter examines the principal architectures and algorithms used in these settings. It treats crawling as a controlled discovery process, link analysis as a source of authority and relevance evidence, duplicate detection and freshness as quality-maintenance problems, and distributed retrieval as a problem of partitioning, coordination, latency, and fault tolerance. It also considers federated search, in which a single user query is sent to several independent search services, and enterprise search, in which relevance must be combined with authorization.

The central engineering principle is that retrieval quality is not determined only by the ranking function. A search engine can rank documents elegantly and still fail if it has not discovered the right pages, indexed stale content, returned duplicates, violated access controls, or missed a shard during distributed execution.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain the architecture and operational logic of Web and enterprise crawlers.
2. Interpret robots protocols and distinguish policy compliance from security.
3. Describe link-analysis methods, including PageRank, and their limitations.
4. Design approaches to duplicate detection, canonicalization, and freshness management.
5. Compare sharding, replication, federated search, and centralized distributed retrieval.
6. Explain how access-controlled enterprise search prevents information leakage.
7. Evaluate distributed search architectures using quality, latency, scalability, and reliability criteria.
8. Identify governance, privacy, adversarial, and operational risks in search systems.

1. Search as a Collection-Maintenance and Retrieval Pipeline

A large-scale search system usually contains the following logical stages:

1. Discovery: obtaining URLs, repository records, feeds, APIs, or event notifications.
2. Fetching: retrieving content subject to politeness, authorization, and network constraints.
3. Parsing: extracting text, metadata, links, language, structured data, and security attributes.
4. Normalization: resolving redirects, canonical URLs, character encodings, and document versions.
5. Deduplication: identifying exact, near, and semantic duplicates.
6. Indexing: constructing inverted indexes and auxiliary structures.
7. Query processing: analyzing the query and selecting candidate documents.
8. Distributed retrieval: executing searches across partitions or collections.
9. Ranking and blending: combining textual, link, freshness, behavioral, and business signals.
10. Presentation and enforcement: filtering unauthorized results and presenting explanations, snippets, and facets.

These stages form a feedback system. Query logs may reveal missing content; crawl statistics may identify slow or unstable hosts; user reports may expose stale or incorrect results. Consequently, search engineering is iterative rather than a one-time indexing task.

2. Web Crawling

2.1 Crawler architecture

A Web crawler, or spider, starts with a seed set of URLs. It retrieves a page, extracts links, adds previously unseen URLs to a frontier, and repeats the process. A production crawler is normally composed of:

• a URL frontier, which stores discovered URLs and scheduling metadata;
• a scheduler, which selects the next eligible fetch;
• fetchers, which issue HTTP requests and handle timeouts, redirects, and content types;
• a parser, which extracts links and document fields;
• a repository, which stores fetched content and metadata;
• a deduplication service, which avoids repeated work;
• monitoring and policy components.

The frontier is not simply a queue. It commonly implements per-host politeness, priority, retry limits, recrawl times, and failure backoff. A host-based scheduler is important because unconstrained parallel requests can overload a site even when the crawler has sufficient computational capacity.

A simplified crawl-control model assigns each URL a priority score:

Here, P(u) is the priority of URL u; Q(u) is estimated page quality or importance; I(u) is expected information gain; F(u) is freshness need; C(u) is estimated fetch cost; and α, β, γ, and δ are non-negative engineering weights. This is a scheduling heuristic, not a universal law. The variables should be normalized to comparable scales before combining them, and the weights must be validated against crawl objectives.

2.2 Politeness and HTTP behavior

A responsible crawler identifies itself through a User-Agent string, observes host-level rate limits, handles Retry-After responses, and avoids unnecessary retrieval of large or irrelevant resources. Conditional requests reduce bandwidth and origin load. In HTTP, an ETag or Last-Modified value can be sent in a later request using If-None-Match or If-Modified-Since. A 304 Not Modified response indicates that the stored representation can be reused.

Crawler politeness has several dimensions:

• Concurrency: how many requests are issued to one host at once.
• Inter-request delay: time between requests to a host.
• Bandwidth: total data transferred from a host.
• Retry behavior: whether failures trigger immediate or delayed repetition.
• Scope: which paths, file types, or domains are eligible.

A crawler should also defend itself against traps such as infinite calendar URLs, session identifiers, URL-generated faceted navigation, and cyclic redirects. URL normalization may lower-case host names, remove default ports, resolve relative links, and normalize selected escape sequences. It must be conservative: changing a path or query parameter can change document identity.

2.3 Robots protocols

The Robots Exclusion Protocol allows a site administrator to express crawling preferences in a /robots.txt file. A crawler generally selects the user-agent group applicable to it and evaluates path rules. Directives can indicate disallowed paths, permitted paths, sitemap locations, and, in some implementations, crawl-delay preferences.

Robots directives are access preferences for compliant crawlers, not authentication. They do not prevent a malicious client from fetching a resource, and they should never be used to protect confidential data. Confidential information requires authentication, authorization, network controls, or removal from public exposure. A crawler should record the retrieved policy, its retrieval time, and the policy version used for decisions. Policy interpretation must account for redirects, unavailable policy files, caching, and changes in standards or implementation.

The noindex directive, when delivered in an HTML meta element or HTTP header, concerns indexing rather than crawling. A crawler may need to fetch a page to observe noindex; therefore, disallowing the page in robots.txt may prevent the crawler from seeing that directive. This distinction is operationally important.

3. Link Analysis and Web Authority

Textual relevance alone is often insufficient on the Web because pages vary greatly in quality and intent. Hyperlinks supply a graph signal: a link from one page to another may indicate endorsement, navigation, citation, or advertising. Link analysis exploits this graph, usually alongside textual evidence rather than as a replacement for it.

3.1 PageRank

PageRank models a random surfer who follows links with some probability and jumps to another page otherwise. In matrix form, a common formulation is:

where r is the vector of page scores, A is a column-stochastic transition matrix, d is the damping factor with 0 < d < 1, and v is a non-negative teleportation vector whose components sum to 1. Under standard conditions, including a valid stochastic construction and positive teleportation support, the equation has a stable stationary solution. Dangling pages with no outgoing links require special treatment, commonly by redistributing their probability according to v.

The score can be estimated by power iteration:

Iterations continue until the change between successive vectors is below a selected tolerance. PageRank is not a direct measure of truth, and it can be manipulated through link farms, purchased links, mutually reinforcing sites, or concentrated institutional structures.

3.2 Other graph signals

HITS distinguishes authorities, which contain valuable content, from hubs, which link to valuable authorities. Trust propagation methods attempt to reduce the influence of spam by assigning higher initial trust to selected seeds. Community detection can identify topical or organizational clusters. Anchor text, the visible text of an incoming link, can supply descriptive evidence even when the target page uses weak terminology.

Modern systems usually combine graph evidence with lexical relevance, semantic representations, quality classifiers, freshness, location, device context, and user intent. Link signals should be time-aware: an abandoned domain may retain historical authority while no longer providing reliable information.

4. Duplicate Detection and Content Identity

Duplicate documents waste storage, computation, and ranking space. They can also distort link and usage statistics. Duplication occurs at several levels:

• Exact duplicates: byte-identical representations.
• URL duplicates: different URLs resolve to the same resource.
• Template duplicates: pages differ only in navigation, advertisements, or boilerplate.
• Near duplicates: most substantive text is shared, but some sections differ.
• Version duplicates: successive snapshots represent the same document at different times.
• Semantic duplicates: independently written documents express substantially the same content.

A useful first step is canonicalization: follow redirects, normalize URLs carefully, inspect canonical-link metadata, and identify stable document identifiers where available. Canonical metadata is a signal, not absolute proof; a site can configure it incorrectly.

Exact duplicate detection can use a cryptographic or non-cryptographic content fingerprint. For near duplicates, shingling divides text into overlapping sequences of tokens. If S_A and S_B are sets of shingles for two documents, their Jaccard similarity is:

The value lies between 0 and 1 for finite sets. It equals 1 when the sets are identical and 0 when they have no common shingles. Computing every pair is expensive, so systems use MinHash signatures and locality-sensitive hashing to generate candidate pairs before exact comparison. Similarity thresholds must be calibrated by language, document type, boilerplate level, and the cost of false merges.

Deduplication should distinguish index collapsing from content deletion. A search engine may retain several URLs but display one representative, while preserving alternative URLs for navigation, provenance, or access differences. Over-aggressive deduplication can hide legitimate regional editions, language variants, legal versions, or personalized documents.

5. Freshness and Recrawling

Freshness is the degree to which an indexed representation reflects the current source. It has at least two components: the age of the indexed content and the importance of changes since the last fetch. A rapidly changing financial feed and a stable historical article should not receive the same recrawl schedule.

A practical scheduler estimates a document’s change rate from observations. If a page has changed k times over an observation interval of length T, a simple estimated change rate is:

Here, λ̂ has units of changes per unit time. This estimate assumes that observed changes are a useful approximation to future behavior; it is noisy when k is small and can be biased by outages or selective observation. More advanced systems use survival analysis, exponential smoothing, source-provided update times, or event-driven notifications.

Freshness policy also depends on value. A recrawl priority can combine estimated change rate, query demand, document importance, and fetch cost. For example, a page with high query demand but moderate change rate may deserve more attention than an obscure page that changes frequently. Conditional requests, feeds, sitemaps, webhooks, and repository change logs can reduce the need for full downloads.

Freshness has a cost-quality trade-off. Excessive crawling wastes bandwidth and may violate politeness; insufficient crawling produces stale answers. Operational dashboards should monitor age distributions, change-detection accuracy, fetch success, indexing delay, and freshness by content class rather than relying on a single average.

6. Enterprise Search and Access Control

Enterprise collections include file systems, document management platforms, email, wikis, ticketing systems, databases, source-code repositories, and cloud applications. Unlike the public Web, access rights are central to correctness. A relevant document that a user cannot access must not appear in the result list, snippet, facet, suggestion, cache, or query-dependent explanation.

6.1 Security models

Common authorization models include:

• Access-control lists (ACLs): permissions attached directly to objects.
• Role-based access control (RBAC): permissions assigned to roles, which users receive.
• Attribute-based access control (ABAC): decisions based on user, resource, action, environment, and policy attributes.
• Group and inheritance models: permissions propagate through organizational structures or folders.

An enterprise index may store security terms, such as group identifiers, with each document. At query time, the system computes the user’s effective authorization and filters candidates. Alternatively, it may create security-specific indexes or apply secure joins with an authorization service.

The fundamental invariant is:

Here, u is a user or service principal, q is a query, and Authorized(u) is the set of documents the user is permitted to access at decision time. This is a security requirement, not merely a ranking preference.

6.2 Security trimming and leakage risks

Security trimming may occur before ranking, during candidate generation, or after ranking. Late filtering can be efficient in some systems but creates risks: unauthorized documents may influence top-k selection, snippets may expose text, and cached result lists may be reused incorrectly. Secure caching must include authorization context or use a safe equivalence class of users.

Permission changes create a consistency problem. An index can contain a document whose source permission has just been revoked. Systems therefore combine indexed security metadata with authoritative checks for sensitive repositories, short permission-cache lifetimes, change notifications, or emergency invalidation. Encryption protects stored data but does not itself implement document-level authorization.

Search logs are also sensitive. Queries can reveal employee intentions, investigations, health information, or confidential projects. Governance should define retention, access, purpose limitation, auditing, and redaction policies.

7. Distributed Retrieval Architectures

Large collections exceed the capacity of one machine or require availability across regions. Distribution introduces partitioning, replication, network communication, partial failure, and consistency decisions.

7.1 Sharding

Document sharding assigns each document to one shard. A query is broadcast to all relevant shards, each shard returns local top candidates, and a coordinator merges them into a global ranking. This approach provides balanced storage and straightforward updates, but query fan-out grows with the number of shards.

Term-based sharding assigns terms or posting lists to different machines. It can reduce some work for selective queries, but multi-term queries require communication among term servers and can complicate ranking and document-at-a-time evaluation. Document sharding is therefore common in general-purpose web search.

For a top-k query, shard i can return local candidates with scores. The coordinator merges these candidates and retains the global k highest-scoring results. This is exact only when each shard returns enough candidates under the scoring and tie-breaking rules; approximate early termination can lower latency at the cost of recall.

The idealized query latency of a fan-out system is close to the slowest participating shard rather than the average shard. If shard latencies are L₁, L₂, …, L_n, the waiting component is approximately:

This is an engineering approximation. Queueing, retries, hedged requests, caching, and parallel overlap can change the observed value. It highlights the “straggler” problem: one slow shard can dominate user-visible latency.

7.2 Replication

Replication maintains multiple copies of a shard. It improves availability and read throughput, and it can distribute load geographically. A primary-replica design sends writes to a primary and propagates changes to replicas. A leaderless or quorum design can accept writes at several replicas but requires conflict and consistency policies.

Replication creates trade-offs among consistency, availability, latency, and operational complexity. Strong consistency can reduce the window in which a revoked permission or updated document is invisible, but may increase coordination cost. Eventual consistency improves availability and write performance but permits temporary divergence. Search systems often accept bounded staleness for ordinary content while requiring stronger controls for security metadata and deletion requests.

7.3 Centralized distributed search versus federated search

In a centralized distributed search system, one organization controls collection, indexing, ranking, and query coordination across internal shards. It can normalize schemas and calculate a common score.

In federated search, a broker sends a query to independent search services, such as a library catalogue, an enterprise repository, a patent database, and a news service. Each source may use a different syntax, vocabulary, ranking scale, access policy, and latency profile. The broker must translate the query, select sources, retrieve results, normalize scores or use rank-based merging, remove duplicates, and handle failures.

Score normalization is difficult because a score of 0.8 in one engine may not be comparable to 0.8 in another. Rank-based methods such as reciprocal rank fusion can combine result positions without assuming comparable score scales. One common form is:

Here, R(d) is the fused score for document d, m is the number of sources, rank_j(d) is the rank assigned by source j (with a convention for documents not returned), and c is a positive constant that reduces the effect of very high ranks. The formula is a heuristic for rank aggregation; it does not prove that sources are equally reliable or that their rankings measure the same construct.

Federated systems are valuable when data ownership, legal boundaries, or organizational autonomy prevent central indexing. Their weaknesses include incomplete coverage, inconsistent metadata, network failures, higher latency, and limited ability to perform global relevance learning.

8. Distributed Indexing and Query Execution

An inverted index maps terms to postings containing document identifiers and often term frequency, positions, and field information. Distributed indexing may use batch pipelines, streaming updates, or a hybrid architecture. A typical pipeline extracts documents, partitions them by shard key, builds immutable segments, merges segments, and publishes new index versions atomically.

Immutable segments simplify concurrent reads and recovery. Background merging reduces the number of segments but consumes CPU and I/O. Update systems must balance indexing freshness against query performance. Near-real-time indexing often makes a new segment searchable within seconds or minutes, not necessarily immediately.

Query execution can be:

• Document-at-a-time: process one candidate document through several query terms.
• Term-at-a-time: process postings for one term before moving to another.
• Two-phase: retrieve a broad candidate set, then apply expensive ranking or semantic models.
• Early-terminated: stop processing when remaining candidates cannot improve the result sufficiently.

Distributed systems commonly use a coordinator, shard replicas, result collectors, and monitoring services. They must handle timeouts, replica divergence, malformed documents, overloaded nodes, and network partitions. A partial-results policy should be explicit: returning fewer results quickly may be preferable to failing the entire query, but the interface should communicate degraded coverage when that matters.

9. Evaluation and Operational Metrics

Search quality must be evaluated at multiple layers.

For a binary relevance setting, precision at k is:

Recall is:

These measures require a defined relevance set and can be misleading when assessors disagree, when the total relevant set is unknown, or when users value novelty and task completion rather than document relevance alone. NDCG is useful when relevance is graded, while latency should normally be reported by percentiles such as p50, p95, and p99 rather than by an average alone.

10. Illustrative Case: A Global Manufacturing Knowledge Search

Background

A fictional manufacturer operates engineering repositories, maintenance records, safety procedures, supplier portals, and regional intranets. Employees complain that search returns duplicated manuals, obsolete procedures, and documents from the wrong region. A previous prototype indexed all sources centrally but occasionally displayed snippets from restricted projects.

Problem analysis

The organization identifies five interacting causes. First, regional URLs produce near-identical language editions. Second, maintenance pages change frequently but are crawled on a fixed monthly schedule. Third, repositories use different group and role identifiers. Fourth, the central index is divided into too many small shards, causing high tail latency. Fifth, supplier portals cannot permit central content export and must remain federated.

Approach and solution

The architecture adopts source connectors for controlled repositories and a compliant crawler for public supplier documentation. Documents receive stable source identifiers, version timestamps, language, region, and normalized security principals. Exact hashes remove byte-identical copies; shingling and candidate matching identify near duplicates, while regional and language variants remain separate when users need them.

A change-aware scheduler uses repository event feeds for maintenance records and conditional HTTP requests for supplier pages. High-demand safety procedures receive stricter freshness targets. The index uses document sharding with three replicas per shard. A coordinator broadcasts queries, applies security filters using normalized group membership, and merges local candidates. The supplier portal is accessed through a federated adapter, with results labelled by source and ranked through rank-based fusion rather than raw score comparison.

Permission changes are propagated through an event stream, and sensitive collections perform an authoritative check before displaying content. Query logs are minimized and access-controlled. Dashboards report p95 latency, stale-document age, duplicate-collapse rate, failed authorization checks, and source-level timeout rates.

Outcome

In this illustrative scenario, employees receive fewer duplicate results and maintenance content becomes substantially more current. Tail latency falls because shard sizes are balanced and slow replicas can be bypassed. The system does not claim that all content has one universal ranking: source provenance, update time, region, and authorization state remain visible. The principal improvement is architectural: discovery, content identity, freshness, security, and retrieval are treated as one connected system.

Learning points

1. Better ranking cannot compensate for missing or stale documents.
2. Deduplication should preserve meaningful variants rather than erase them.
3. Security must be enforced throughout retrieval and presentation.
4. Federated search is appropriate when ownership or legal boundaries prevent central indexing.
5. Tail latency and partial failure require explicit design, not after-the-fact tuning.

11. Challenges, Governance, and Contemporary Developments

Search systems face adversarial behavior, including crawler traps, spam links, cloaking, prompt-like document injection, poisoned metadata, and deliberate duplication. Machine-learning ranking can amplify historical popularity or organizational bias. Monitoring should therefore include representative quality evaluation, adversarial testing, explainability appropriate to the use case, and mechanisms for correction and appeal.

Privacy regulation and organizational policy may constrain collection, retention, profiling, cross-border transfer, and employee monitoring. Data minimization, purpose limitation, retention schedules, deletion workflows, encryption, audit trails, and separation of duties are relevant controls. A search index is often a new copy of sensitive data and must be governed accordingly.

Contemporary systems increasingly combine lexical retrieval with dense vector retrieval and reranking models. Hybrid retrieval can improve semantic matching, especially for paraphrases, but introduces model drift, embedding-update costs, explainability challenges, and possible leakage through vector stores. Access control must apply equally to lexical postings, vectors, caches, and model-generated summaries. Retrieval-augmented generation does not remove the need for reliable crawling, provenance, freshness, or authorization; it makes failures less visible if generated answers are not grounded and cited.

Best-Practice Checklist

• Define collection scope, ownership, and legal authority before crawling.
• Treat robots policies as compliance signals, never as confidentiality controls.
• Identify hosts, control concurrency, and implement backoff and conditional requests.
• Preserve provenance, version, language, region, and source timestamps.
• Use layered duplicate detection and validate thresholds by document class.
• Schedule recrawls using change evidence, demand, value, and fetch cost.
• Balance shards and replicate according to availability and consistency needs.
• Measure tail latency and design explicit timeout and partial-result behavior.
• Normalize federated metadata and avoid unjustified comparison of raw scores.
• Enforce authorization before exposing results, snippets, facets, or caches.
• Test deletion and permission-revocation paths as seriously as indexing paths.
• Evaluate quality, freshness, security, and operational performance together.

Chapter Summary

Web, enterprise, and distributed search are socio-technical systems rather than isolated ranking algorithms. Crawlers discover and refresh content through frontiers, politeness policies, HTTP controls, and robots directives. Link analysis supplies graph-based evidence but must be combined with textual, semantic, temporal, and quality signals. Canonicalization, hashing, shingling, and locality-sensitive candidate generation reduce duplicate content while preserving meaningful variants. Freshness scheduling balances change rate, demand, value, cost, and source constraints.

Enterprise search adds the non-negotiable requirement that retrieval results respect authorization. Security trimming must cover documents, snippets, facets, caches, suggestions, vectors, and generated answers. Distributed search uses sharding for scale, replication for availability, and coordinators for merging results, but faces stragglers, partial failures, consistency choices, and indexing delays. Federated search preserves source autonomy but requires query translation, metadata normalization, rank fusion, and careful handling of unavailable sources. Effective evaluation therefore combines relevance, coverage, freshness, latency, reliability, and security. Governance, privacy, adversarial robustness, and provenance are essential components of professional search engineering.

Key Terms

Crawler: A program that discovers and retrieves documents or resources.

URL frontier: A prioritized structure holding discovered URLs and scheduling metadata.

Robots Exclusion Protocol: A convention through which site owners express crawling preferences.

Politeness: Operational behavior that limits crawler impact on source systems.

Canonicalization: Normalization or selection of a preferred representation or URL.

Link analysis: Use of hyperlink or graph structure as retrieval evidence.

PageRank: A random-surfer-based method for estimating graph-based page importance.

Dangling node: A graph node with no outgoing links.

Duplicate detection: Identification of identical or substantially similar content.

Shingling: Representation of text as overlapping token sequences for similarity analysis.

MinHash: A compact signature technique for estimating set similarity.

Freshness: How closely indexed content reflects the current source.

Document sharding: Partitioning documents across independent index nodes.

Replication: Maintaining multiple copies of data or index partitions.

Federated search: Searching independent services through a coordinating broker.

Security trimming: Removing results that a particular user is not authorized to access.

ACL: An access-control list specifying permissions associated with an object.

Tail latency: High-percentile response time, commonly measured at p95 or p99.

Partial failure: A condition in which some distributed components fail while others continue operating.

Rank fusion: Combining ranked outputs from multiple retrieval systems.

Review Questions

Short Answer Questions

1. Why is a URL frontier more than a simple FIFO queue?
2. Explain why robots.txt is not a security mechanism.
3. Distinguish exact duplicates from near duplicates.
4. What is the purpose of replication in a distributed search system?
5. Why are raw relevance scores difficult to combine in federated search?

Long Answer Questions

1. Describe a complete Web crawling architecture and explain how politeness, retry behavior, URL normalization, and conditional requests interact.
2. Explain PageRank, including the meaning of the damping factor, teleportation vector, and dangling-node treatment. Discuss two limitations of link analysis.
3. Compare document sharding and term-based sharding in terms of query execution, communication, scalability, and operational complexity.
4. Develop an enterprise-search security design that supports ACLs, group inheritance, permission changes, secure caching, and auditability.
5. Explain how freshness scheduling and duplicate detection jointly influence search quality, storage, and crawl cost.

Analytical / Application Questions

1. An enterprise search service has excellent textual relevance but frequently displays stale maintenance procedures and duplicate regional manuals. Propose a data, crawl, indexing, and evaluation plan to address the problem.
2. A federated broker queries five independent databases, but one source is slow and another uses scores on a completely different scale. Design a query-processing and result-merging strategy that preserves useful coverage while controlling latency.
3. A permission is revoked immediately after a document is indexed. Analyze the security risks in early filtering, late filtering, snippets, caches, replicas, and generated summaries, and recommend controls for each risk.

Multiple Choice Questions with Answers

Question 1

Which statement best describes the role of robots.txt?

A. It encrypts restricted Web content.

B. It expresses crawling preferences for compliant crawlers.

C. It guarantees that a page will never be indexed.

D. It authenticates crawler identities to a server.

Correct Answer: B. It expresses crawling preferences for compliant crawlers.

Question 2

In the PageRank formulation r = dAr + (1 − d)v, what does v represent?

A. The vocabulary size

B. The vector of document lengths

C. The teleportation or personalization distribution

D. The number of dangling pages

Correct Answer: C. The teleportation or personalization distribution

Question 3

What is the Jaccard similarity of two identical shingle sets?

A. 0

B. 0.25

C. 0.5

D. 1

Correct Answer: D. 1

Question 4

Which technique is especially useful for generating candidate pairs of near-duplicate documents without comparing every pair?

A. Locality-sensitive hashing

B. Round-robin replication

C. Query expansion

D. HTTP redirection

Correct Answer: A. Locality-sensitive hashing

Question 5

What is the primary purpose of conditional HTTP requests in recrawling?

A. To increase the number of duplicate pages

B. To avoid transferring unchanged content unnecessarily

C. To bypass authorization controls

D. To convert URLs into vector embeddings

Correct Answer: B. To avoid transferring unchanged content unnecessarily

Question 6

In document-sharded retrieval, why does a coordinator contact multiple shards?

A. To reconstruct the source Web site

B. To obtain local candidates and merge them into a global result list

C. To change every document's access permissions

D. To eliminate the need for an inverted index

Correct Answer: B. To obtain local candidates and merge them into a global result list

Question 7

Which expression represents the essential enterprise-search security invariant?

A. Results are always larger than Authorized(u).

B. Results(u, q) is a subset of Authorized(u).

C. Authorized(u) is a subset of Results(u, q).

D. Results depend only on query length.

Correct Answer: B. Results(u, q) is a subset of Authorized(u).

Question 8

Why is rank-based fusion often useful in federated search?

A. It assumes all source scores use identical units.

B. It avoids the need to retrieve results from sources.

C. It can combine positions without directly comparing incompatible score scales.

D. It guarantees perfect recall from every source.

Correct Answer: C. It can combine positions without directly comparing incompatible score scales.

Question 9

If a source records 12 content changes over 6 months, the simple estimated change rate λ̂ = k/T is:

A. 0.5 changes per month

B. 2 changes per month

C. 6 changes per month

D. 72 changes per month

Correct Answer: B. 2 changes per month

Question 10

Which metric is specifically intended to describe high-percentile response time?

A. Recall

B. Jaccard similarity

C. p99 latency

D. PageRank

Correct Answer: C. p99 latency