Data Sources, Measurement and Governance

Introduction

Real estate market research depends on the ability to convert a complex, partly observable market into credible evidence. Properties are heterogeneous, transactions are infrequent and private, planning decisions alter future supply, and market participants often possess unequal information. Consequently, the researcher must do more than collect data. They must identify appropriate sources, understand how observations were generated, construct valid measures, document limitations, and govern access and use responsibly.

This chapter examines the principal sources used in real estate research: public records, transaction data, listings, leases, planning records, censuses, surveys, brokerage reports and alternative data. It then considers measurement, index construction, data quality, metadata, bias, privacy, consent, ownership and institutional governance. The central argument is that data quality is not an intrinsic property of a dataset. It is a relationship between the dataset, the research question, the measurement process and the decision for which the evidence is used.

Learning objectives

By the end of this chapter, students should be able to:

1. Evaluate the strengths, weaknesses and appropriate uses of major real estate data sources.
2. Distinguish concepts, variables, indicators, observations and measures.
3. Construct and interpret rates, ratios, price measures and market indices.
4. Diagnose coverage error, measurement error, missingness, duplication, selection bias and temporal problems.
5. Design metadata and data-governance arrangements for reproducible research.
6. Address privacy, consent, ownership, fairness, confidentiality and regulatory obligations.
7. Integrate conventional and alternative data without overstating the evidence.

1. The evidential structure of a real estate market

A market concept is an abstract idea such as affordability, vacancy, liquidity, development intensity or investment risk. A variable is a specified characteristic that can vary across properties, people, firms, locations or time. An observation is a recorded value of a variable for a defined unit. An indicator combines one or more variables to represent a broader condition.

For example, “housing affordability” is a concept. Monthly household income and monthly housing cost are variables. A household’s housing-cost-to-income ratio is an indicator. A dataset containing one record per household at a particular date contains observations.

A valid measurement process requires four decisions:

• Unit of analysis: property, parcel, transaction, lease, household, firm, neighbourhood or time period.
• Population: all units to which the conclusion is intended to apply.
• Reference time: transaction date, listing date, assessment date, survey date or observation period.
• Operational definition: the rule used to translate a concept into a variable.

These decisions prevent a common error: treating a readily available variable as though it were the concept itself. A listing price is not necessarily market value; an asking rent is not necessarily an achieved rent; and a planning approval is not necessarily a completed development.

2. Principal data sources

2.1 Public records and administrative data

Public records include land registries, property tax assessments, building permits, code-enforcement records, court filings, corporate registers and government-held planning records. They are valuable because they often provide geographic coverage, legal authority and historical continuity.

Their limitations are equally important. Assessment values may be updated infrequently or for taxation rather than valuation purposes. Parcel identifiers may change after subdivision. A recorded sale may omit concessions, side agreements or the condition of the property. Records can also reflect institutional practices: areas with stronger administrative capacity may appear to have more violations simply because enforcement is more complete.

Researchers should distinguish:

• Legal ownership from beneficial or economic ownership;
• Recorded consideration from total economic consideration;
• Permitted use from actual use;
• Planning approval from construction and occupation;
• Assessed value from market value.

2.2 Transaction databases

Transaction data describe completed sales, refinancing, mortgages, acquisitions or disposals. They are commonly used in price analysis, investment research, liquidity studies and hedonic models. Essential fields may include a unique property identifier, transaction date, price, property type, floor area, location, tenure and buyer or seller category.

A transaction price is an observed outcome of negotiation under particular financing, information and market conditions. It is not automatically comparable with every other price. Researchers must investigate arm’s-length status, portfolio sales, related-party transfers, distressed sales, bundled assets, currency, taxes and incentives.

A basic price-per-area measure is:

where P_i is the price of transaction i in currency units and A_i is the relevant area in square metres or another consistently defined unit. This measure is useful for exploratory comparison, but it can mislead when areas are measured differently or when property quality, tenure, location and building form vary substantially.

2.3 Listings and portals

Listings provide asking prices, advertised rents, availability dates, descriptions, amenities, photographs and sometimes listing histories. They are particularly useful where completed transaction data are delayed or inaccessible. Listing data can provide early signals of supply, seller expectations and market search conditions.

However, listings are offers, not completed exchanges. A property may be listed more than once, remain unsold, be withdrawn, or be advertised at a strategic price. Portal data may overrepresent digitally marketed properties, professionally managed stock and higher-value segments. A researcher should retain listing status and dates, identify relistings, and avoid treating the number of advertisements as the number of unique properties without deduplication.

2.4 Leases and rental data

Lease data can include contract rent, incentives, term, break clauses, renewal terms, indexation, service charges, tenant improvements and vacancy periods. For commercial property, the distinction between headline rent and effective rent is critical.

If a tenant receives a rent-free period, a simplified effective-rent calculation is:

where R_eff is average effective rent per period, R is contractual rent per paid period, N is the total lease periods, and C is the monetary value of rent-free concessions. The formula assumes that the concession can reasonably be valued in the same currency and spread over the stated lease term. It does not automatically account for discounting, stepped rents, tenant improvements or operating expenses.

2.5 Planning and development data

Planning applications, zoning maps, development approvals, building permits and completion certificates provide evidence about potential and realised supply. They support pipeline analysis, land-use studies, infrastructure assessment and scenario planning.

Planning data require a stage model. An application, approval, start, completion and occupation are distinct events. Counting all approvals as future supply produces an inflated pipeline. A robust dataset records the status, status date, expected capacity, probability of delivery where justified, and reasons for lapse or cancellation.

2.6 Census and official statistics

Censuses and official surveys provide population, household, employment, tenure, income, commuting and dwelling characteristics. They are essential for market segmentation and neighbourhood analysis because they provide a consistent population frame and often include small-area geography.

Limitations include infrequent collection, disclosure controls, boundary changes, non-response and differences between residence-based and workplace-based measures. Comparisons across years require harmonisation of definitions and geographic boundaries.

2.7 Surveys and interviews

Surveys measure attitudes, intentions, experiences and characteristics that administrative records do not capture. Interviews and focus groups can reveal motivations, constraints and institutional practices. A survey’s credibility depends on sampling, question design, response rates, weighting, mode effects and the handling of missing data.

A sample proportion is:

where x is the number of sampled units with the characteristic and n is the number of valid observations. This estimate describes the sample directly. Generalising it to a population requires an appropriate sampling design and assumptions about selection and non-response.

2.8 Brokerage, consultancy and industry reports

Brokerage reports often contain timely market intelligence, rent series, vacancy estimates, investment yields and expert interpretation. Their practical value can be high, particularly where the underlying market is opaque. Their incentives and methodology must nevertheless be examined. Definitions of “prime,” “vacant,” “available,” “under construction” and “market rent” may differ across firms. A report should be treated as an evidence source with a stated perspective, not as a neutral benchmark by default.

2.9 Alternative data

Alternative data may include satellite imagery, building sensors, mobile-device aggregates, web searches, footfall counts, energy use, geospatial traces, imagery-derived building attributes and platform activity. These data can improve temporal frequency or reveal phenomena not captured in official records.

They also create distinctive risks: opaque collection, uncertain representativeness, changing algorithms, vendor dependence, re-identification and unstable definitions. A footfall index may measure device presence rather than all visitors; a search index may measure interest among internet users rather than effective demand; and an automated building classification may systematically misclassify unusual structures.

3. Measurement and variable construction

3.1 Levels of measurement

Nominal variables classify categories without ordering, such as property type. Ordinal variables provide order but not necessarily equal intervals, such as condition ratings. Interval measures have meaningful differences but no meaningful zero in the relevant scale. Ratio measures have a meaningful zero, such as floor area or number of completed units.

The level of measurement determines which summaries and transformations are defensible. Averaging nominal codes is meaningless; treating an ordinal rating as a precise monetary value is also inappropriate without justification.

3.2 Rates, ratios and denominators

A rate requires a clearly defined numerator, denominator and time or population base. For example, a vacancy rate may be defined as:

where U is the number of vacant units and S is the total relevant stock. The denominator must be stated: total stock, lettable stock, completed stock or surveyed stock can produce different results. A rate is not comparable across markets unless definitions, reference dates and coverage are aligned.

Growth between an initial value and a final value can be expressed as:

where X₀ and X₁ are comparable values at the beginning and end of the period. If reported as a percentage, multiply g by 100. The calculation assumes consistent units and definitions and is not a substitute for a compound annual growth rate when the interval spans several years.

For a period of T years, the compound annual growth rate is:

It assumes positive values and describes a constant equivalent annual rate, not the actual path followed.

3.3 Price indices

A simple average price is sensitive to changes in the composition of transactions. If expensive properties make up a larger share of sales, the average may rise even when comparable property values do not. A median can reduce the influence of extreme observations but remains sensitive to the composition of the middle transaction.

A hedonic model controls statistically for observed characteristics. A simplified log-linear specification is:

where P_i is price, α is an intercept, X_ji are property or locational characteristics, β_j are associated coefficients, δ_t are time effects, and ε_i is the unexplained component. The model assumes that relevant characteristics are measured adequately, functional form is reasonable, observations are sufficiently comparable, and the error structure is handled appropriately. Coefficients should not automatically be interpreted as causal effects.

Repeat-sales methods compare the same property across transactions and therefore control for time-invariant observed and unobserved characteristics. They require reliable property matching and are vulnerable to renovation, selective resale and low transaction frequency. No index is universally best: the appropriate method depends on the research question, data coverage and quality.

3.4 Index construction and reweighting

An index expresses change relative to a base period. A basic price relative is:

where P_t is the value in period t and P₀ is the base-period value. The index equals 100 in the base period. In a composite market, weights should reflect a defensible objective, such as transaction volume, stock, floor area or expenditure. Changing weights can change the interpretation of the index, so the weighting rule and revision policy must be documented.

4. Data quality and metadata

Data quality is multidimensional. Important dimensions include:

Quality assurance should include range checks, date checks, geographic validation, duplicate detection, reconciliation to authoritative totals, visual inspection and sample verification against source documents. Missing values should be classified as unknown, not applicable, withheld, not yet available or not collected. Coding all missing values as zero creates serious measurement error.

Metadata are the information needed to interpret and reproduce a dataset. A professional metadata record should state the source owner, purpose, population, unit of analysis, field definitions, geography, time reference, collection method, update schedule, transformations, missing-value codes, known biases, access restrictions, licence, version and responsible custodian.

A useful distinction is between data validity and fitness for purpose. A correctly recorded asking rent may be valid as an asking rent but unsuitable for estimating achieved rent. Likewise, a dataset can be complete for one neighbourhood and unsuitable for city-wide inference.

5. Bias, uncertainty and integration

5.1 Common sources of bias

• Coverage bias: some properties, households or locations have no chance of appearing in the frame.
• Selection bias: observed cases differ systematically from unobserved cases.
• Survivorship bias: failed, vacant or withdrawn properties disappear from the dataset.
• Temporal bias: records refer to different dates or are updated with unequal delays.
• Measurement bias: definitions or instruments systematically overstate or understate values.
• Linkage bias: records are incorrectly matched or unmatched across systems.
• Publication bias: only favourable or commercially attractive results are reported.
• Algorithmic bias: automated systems perform differently across property types, places or demographic groups.

Triangulation is not simply combining more datasets. Sources must be compared for conceptual equivalence, time, geography, population and collection mechanism. Apparent disagreement may reflect different constructs rather than error. For example, asking rents, signed rents and effective rents may all be accurate measures of different stages of the rental process.

A data integration workflow should define a common property or area identifier, establish a master time reference, document matching confidence, preserve source values, and retain a transformation log. Analysts should not overwrite raw data with harmonised values.

6. Governance, privacy and responsible use

Data governance allocates authority and accountability across the data life cycle: collection, acquisition, storage, access, processing, publication, retention and deletion. A governance framework should specify the purpose of processing, lawful basis where applicable, data minimisation, role-based access, security controls, quality responsibilities, audit procedures and incident response.

Privacy risk exists even when names are removed. Location, transaction date, property characteristics and rare combinations can identify households or individuals. Small-area publication may enable inference about occupants, wealth or vulnerability. Researchers should use aggregation, suppression, perturbation, pseudonymisation and controlled access where appropriate. Pseudonymisation reduces direct identification but does not make data anonymous if re-identification remains reasonably possible.

Consent is context-dependent. Consent obtained for one purpose may not justify unrelated commercial profiling. Public availability does not automatically eliminate ethical duties. Researchers should consider reasonable expectations, proportionality, transparency and the potential harm of publication.

Ownership has several dimensions: legal rights in the source, contractual rights to use it, intellectual property in derived products, and stewardship responsibilities toward affected communities. A vendor licence may permit internal analysis but prohibit redistribution or model training. Every project should record permitted uses, prohibited uses, retention periods, attribution requirements and derivative-data rights.

Fairness and accountability are especially important when research informs lending, tenant screening, valuation, enforcement or investment allocation. A model that uses neighbourhood proxies may reproduce historical exclusion even without using protected characteristics directly. Governance should therefore include impact assessment, explainability appropriate to the decision, human review, contestability and monitoring for disparate error.

7. Technology and AI in market research

AI can assist with document extraction, geocoding, image classification, entity resolution, anomaly detection and forecasting. These applications are useful when human review is retained and the training data, confidence levels and error patterns are understood. Large language models may help classify planning documents or draft metadata, but they can produce plausible errors and should not be treated as authoritative sources.

Good practice includes maintaining a human-verified test sample, measuring precision and recall for classification tasks, recording model versions, monitoring drift and preserving the original document or image. Automated valuation or demand models should be stress-tested across locations, property types and market conditions. Automation increases the need for governance; it does not remove it.

8. Applied case: integrating evidence for a rental-market strategy

The following is an illustrative case.

Background

A metropolitan housing agency must assess whether a proposed rental development would address unmet demand in three districts. Available sources include property-tax records, portal listings, a quarterly brokerage report, census data, planning applications and a voluntary tenant survey.

Problem

The sources disagree. Listings suggest rapidly rising rents, the brokerage report shows stable effective rents, and the census indicates a large stock of under-occupied dwellings. The agency must decide whether the disagreement reflects data error or different market measures.

Analysis

The research team first defines the unit of analysis and reference period. It distinguishes dwelling stock, advertised units, signed leases and occupied households. Listings are deduplicated using address, floor area, photographs and listing history. Asking rents are kept separate from signed rents. Brokerage definitions are documented, including whether incentives are incorporated.

The team calculates district vacancy using a clearly stated denominator: units advertised as available during the reference month divided by estimated lettable stock. It does not call this an official vacancy rate. Census under-occupancy is treated as a household-use indicator, not evidence of immediately available supply. Planning applications are classified as application, approval, construction and completion, with no assumption that approval equals delivery.

Survey responses are weighted only where the sampling design supports weighting. Because the survey is voluntary, the report labels estimates as perceptions and reported experiences rather than population facts. The agency also conducts sensitivity analysis using alternative assumptions about listing coverage and delivery of approved projects.

Approach and solution

The final dashboard presents three linked but distinct measures: advertised rent, effective rent where available, and achieved rent from verified lease records. It reports ranges rather than a single precise demand estimate. A governance register records source owners, licences, update dates, personal-data risks and disclosure controls. The proposed development is evaluated under low, central and high delivery scenarios.

Outcome

The agency concludes that advertised rents rose because the composition of listed units shifted toward smaller, higher-quality dwellings, while effective rents remained comparatively stable due to concessions. Under-occupied stock indicates a long-term allocation issue but is not immediately mobilisable rental supply. The recommendation is to proceed conditionally, subject to monitoring signed rents, completions and household demand rather than relying on listings alone.

Learning points

The case demonstrates that disagreement among sources can be informative. Reliable research requires construct alignment, transparent denominators, stage-specific planning data, explicit uncertainty and governance proportionate to the consequences of the decision.

9. Professional best practices

A defensible real estate research project should:

1. Begin with the decision and construct, not the dataset that happens to be available.
2. Define the population, unit, time period, geography and denominator before analysis.
3. Preserve raw data and document every transformation.
4. Use a data dictionary and version-controlled metadata.
5. Separate observed facts, estimates, assumptions and forecasts.
6. Validate unusual observations rather than deleting them automatically.
7. Report missingness, coverage and likely direction of bias.
8. Triangulate sources while preserving their different meanings.
9. Apply data minimisation, access control and retention rules.
10. Make limitations prominent enough that a decision-maker cannot mistake precision for certainty.

Chapter Summary

Real estate market research is an evidential discipline in which source selection, measurement and governance are inseparable. Public records provide legal and administrative structure but may reflect institutional practices and outdated definitions. Transactions reveal completed exchanges but require attention to comparability, incentives and arm’s-length status. Listings are timely but measure asking behaviour rather than achieved outcomes. Leases, planning records, censuses, surveys, brokerage reports and alternative data each capture different parts of the market.

Sound analysis begins by defining the unit of analysis, population, reference time and operational definition. Rates and indices require explicit numerators, denominators, weights and assumptions. Hedonic and repeat-sales methods can improve comparability but do not eliminate selection, measurement or causal-inference problems. Data quality includes accuracy, completeness, timeliness, consistency, validity, uniqueness, lineage and accessibility.

Metadata, reproducible workflows and triangulation are essential for integration. Bias may arise from coverage, selection, survivorship, timing, measurement, linkage or algorithms. Privacy, consent, ownership, licensing, fairness and accountability must be managed throughout the data life cycle. AI can improve efficiency, but it requires validation, human oversight and monitoring. The professional standard is not to claim perfect data; it is to make the evidence, uncertainty and consequences transparent.

Key Terms

Alternative data: Non-traditional data such as satellite imagery, sensor records, web activity or aggregated mobility data.

Asking rent: Rent requested in an advertisement, before negotiation or concessions.

Bias: A systematic departure of an estimate or process from the target phenomenon.

CAGR: Compound annual growth rate representing an equivalent constant annual rate over an interval.

Coverage error: Error caused when the sampling or data frame excludes part of the target population.

Data lineage: A record of the origins, transformations and uses of data.

Effective rent: Rent adjusted for concessions and other relevant economic incentives.

Hedonic model: A model that relates price to observable property, locational and temporal characteristics.

Indicator: A measured variable or composite measure used to represent a broader condition.

Index: A relative measure used to compare a value with a base period or benchmark.

Metadata: Information describing the meaning, origin, structure, quality and permitted use of data.

Measurement error: Difference between a recorded value and the true value of the intended construct.

Population: The complete set of units to which a research conclusion is intended to apply.

Pseudonymisation: Replacing direct identifiers while retaining a separately controlled means of linkage.

Reference time: The date or period to which an observation relates.

Repeat-sales method: A price-index approach using multiple sales of the same property.

Selection bias: Bias arising when observed units differ systematically from unobserved units.

Unit of analysis: The entity about which an observation or conclusion is made.

Validity: The extent to which a measure represents the construct it is intended to measure.

Review Questions

Short Answer Questions

1. Why is a listing price not equivalent to an achieved transaction price?
2. Identify four dimensions of data quality.
3. What is the difference between a planning approval and a completed development?
4. Why must the denominator of a vacancy rate be stated?
5. What does metadata contribute to reproducible research?

Long Answer Questions

1. Compare public records, transaction data and listings for a city-wide residential price study.
2. Explain how hedonic and repeat-sales indices address, and fail to address, compositional change.
3. Discuss the principal sources of bias in voluntary tenant surveys.
4. Design a governance framework for a dataset combining property records with mobility data.
5. Evaluate the ethical and managerial implications of using AI to classify planning applications.

Analytical / Application Questions

1. A market contains 20,000 lettable units, of which 1,200 are advertised as available. Calculate the advertised-availability rate and explain why it should not automatically be called the official vacancy rate.
2. A commercial lease provides 12 monthly periods at 10,000 currency units per paid month and a two-month rent-free concession. Calculate the simplified average effective monthly rent and identify two factors omitted by the calculation.
3. A research team finds that portal asking rents rose 8%, signed rents rose 3% and effective rents rose 1%. Propose a structured investigation of the divergence, including data checks, possible explanations and governance safeguards.

Multiple Choice Questions with Answers

Question 1

Which source most directly records an asking price rather than a completed market exchange?

A. Land registry transfer record

B. Online property listing

C. Completed lease register

D. Property tax assessment

Correct Answer: B. Online property listing

Question 2

In the simplified effective-rent formula, what does the concession term represent?

A. The property’s floor area

B. The number of lease periods

C. The monetary value of incentives such as rent-free time

D. The annual growth rate of rent

Correct Answer: C. The monetary value of incentives such as rent-free time

Question 3

A vacancy rate is most sensitive to which specification choice?

A. The colour of the building

B. The denominator used for relevant stock

C. The spelling of the address

D. The number of report pages

Correct Answer: B. The denominator used for relevant stock

Question 4

What is a principal purpose of metadata?

A. To conceal the source of observations

B. To replace quality assurance

C. To describe meaning, origin, limitations and permitted use

D. To guarantee that all data are unbiased

Correct Answer: C. To describe meaning, origin, limitations and permitted use

Question 5

Which is an example of survivorship bias?

A. Recording the date of a transaction

B. Excluding withdrawn or failed properties from analysis

C. Using a consistent area unit

D. Publishing a data dictionary

Correct Answer: B. Excluding withdrawn or failed properties from analysis

Question 6

In a hedonic price model, time effects are commonly included to represent what?

A. Changes associated with market periods

B. The legal owner’s identity

C. The colour of listing photographs

D. The number of missing records

Correct Answer: A. Changes associated with market periods

Question 7

Why may a planning approval overstate future housing supply?

A. Every approval is necessarily occupied immediately

B. Approved projects may lapse, be delayed or never be built

C. Approvals contain no geographic information

D. Planning records always measure achieved rents

Correct Answer: B. Approved projects may lapse, be delayed or never be built

Question 8

Which practice best protects privacy when publishing small-area property research?

A. Publishing every address and date

B. Removing all quality checks

C. Using aggregation or suppression where re-identification risk exists

D. Assuming public data require no safeguards

Correct Answer: C. Using aggregation or suppression where re-identification risk exists

Question 9

If 1,200 of 20,000 units are advertised as available, the rate is:

A. 3%

B. 6%

C. 12%

D. 16.7%

Correct Answer: B. 6%

Question 10

Which statement about AI in real estate research is most defensible?

A. AI outputs are authoritative if they sound plausible

B. AI removes the need for metadata

C. AI should be validated, monitored and subject to human oversight

D. AI automatically eliminates selection bias

Correct Answer: C. AI should be validated, monitored and subject to human oversight