Measurement Systems and Instrument Characteristics

Introduction

Metrology is the science of measurement and its application. In mechanical engineering, measurements support design verification, manufacturing control, machine maintenance, research, safety, and conformity assessment. A measurement is not simply a number displayed by an instrument; it is the result of comparing an unknown quantity with a defined standard under stated conditions.

A practical measurement system includes the measurand, a sensing element, signal-conditioning components, a transmission or processing stage, an indicating or recording device, and an operator or control system. Each stage can introduce error. The engineer must therefore evaluate not only the nominal range and accuracy of an instrument, but also its resolution, sensitivity, linearity, hysteresis, drift, stability, response, loading, and environmental dependence.

The central engineering task is to select an instrument whose characteristics are appropriate for the quantity, required uncertainty, operating environment, speed, and cost of the measurement.

Learning Objectives

After studying this chapter, the reader should be able to:

• distinguish accuracy, precision, resolution, sensitivity, and uncertainty;
• evaluate static characteristics such as linearity, hysteresis, repeatability, drift, and stability;
• explain dynamic response and time-dependent measurement behavior;
• identify loading and environmental effects;
• select suitable instruments for dimensional, force, pressure, temperature, speed, and vibration measurements;
• interpret calibration information and apply correction factors;
• analyze an instrument-selection problem in a manufacturing context.

1. The Measurement System

The measurand is the quantity intended to be measured, such as shaft diameter, torque, temperature, or surface roughness. The input to the instrument is the physical quantity that produces a response. The output may be a pointer displacement, electrical voltage, digital value, or stored data record.

A generalized measurement chain is:

Figure: Functional elements of a generalized measurement system.

Standards and traceability

A measurement is traceable when its result can be related to a recognized reference through an unbroken chain of calibrations, each contributing to measurement uncertainty. Traceability does not mean that a result is error-free. It means that the calibration history, reference standards, methods, and uncertainties are documented.

A calibration compares an instrument with a standard and establishes the relationship between indication and reference value. Calibration may produce a certificate, a correction table, or an adjustment. Adjustment changes the instrument; calibration itself is the comparison and characterization process.

Accuracy, precision, error, and uncertainty

Accuracy describes closeness to the true or accepted reference value. Precision describes the closeness of repeated results to one another. An instrument can be precise but inaccurate if it has a systematic bias.

For a measured value x and reference value xᵣ, the measurement error is:

where E and x have the same units. Relative error is:

and percentage error is 100eᵣ percent. These expressions are meaningful when the reference value is sufficiently reliable and nonzero.

Uncertainty expresses the incomplete knowledge associated with a measurement result. It is not identical to error, because the exact error is generally unknown. A reported result may take the form x ± U, where U is an expanded uncertainty for a stated coverage condition.

2. Static Characteristics

Static characteristics describe instrument behavior when the input is constant or changes sufficiently slowly that dynamic effects are negligible.

Range, span, and threshold

The range is the interval between the minimum and maximum measurable input. The span is the algebraic difference between upper and lower range limits. The threshold is the minimum input that produces a detectable output change. A dead zone is an interval over which input changes produce no output change.

Resolution

Resolution is the smallest increment of input that can be distinguished under specified conditions. For a digital instrument, it is often the value of the least significant displayed digit, although noise and signal processing may make the effective resolution poorer. For an analog scale, resolution depends on scale spacing, pointer width, parallax, and reading technique.

Resolution is not accuracy. A digital micrometer displaying 0.001 mm increments may still have a ±0.003 mm calibration error.

Sensitivity

Sensitivity is the ratio of output change to input change. For a locally linear instrument:

where S is sensitivity, ∆y is output change, and ∆x is input change. Units depend on the output and input, such as mV/°C or V/mm. The differential sensitivity at a particular operating point is:

Sensitivity should not be confused with resolution. A highly sensitive instrument can still have poor resolution if its output is noisy.

Linearity

An ideal instrument has an output-input relationship that follows a specified straight line. Nonlinearity is the maximum deviation of the calibration curve from the selected reference line, commonly expressed as a percentage of full-scale span:

The result depends on whether the reference is a terminal-point line, zero-based line, or best-fit straight line. Therefore, a specification must identify the reference convention.

Hysteresis

Hysteresis occurs when the output at a given input depends on whether the input was approached from increasing or decreasing direction. It can result from friction, backlash, magnetic effects, elastic memory, or material changes. Hysteresis is commonly specified as the maximum separation between upscale and downscale indications at the same input, expressed as a percentage of span:

To evaluate hysteresis, measurements must be taken in both directions over the same input range and after allowing suitable stabilization.

Repeatability and reproducibility

Repeatability is closeness of agreement under the same conditions: same operator, instrument, method, location, and short time interval. Reproducibility concerns variation when conditions change, such as operator, instrument, laboratory, or day. Both are important in quality control. A stable mean with large scatter indicates poor repeatability; a consistent offset between operators may indicate a reproducibility problem.

Drift and stability

Drift is a gradual change in indication when the measurand is constant. Zero drift changes the output at zero input; sensitivity drift changes the slope of the calibration relationship. Causes include aging, temperature, humidity, contamination, mechanical relaxation, and electronic component changes.

Stability is the ability of an instrument to maintain its metrological characteristics over time. Stability is assessed by repeated calibration or control measurements over a defined period. Drift may be modeled approximately as:

where D is drift rate, ∆y is the change in indication at constant input, and ∆t is elapsed time. This is an empirical rate and should not be extrapolated beyond the conditions and period for which it was established.

Dead zone, backlash, and friction

A dead zone is an input interval producing no output response. Backlash is lost motion caused by clearance between mechanical parts, especially in gears and screw mechanisms. Friction can prevent a mechanism from responding to small changes and can produce direction-dependent readings. These effects are particularly important in dial indicators, mechanical scales, and machine-tool measurement systems.

3. Dynamic Characteristics

When the measurand changes with time, the instrument output may lag, attenuate, or distort the input. Dynamic performance is therefore essential for vibration, impact, engine testing, pressure pulsation, and rapid temperature measurements.

Response time and time constant

A first-order instrument can be represented by:

where x(t) is input, y(t) is output, K is static sensitivity, and τ is the time constant. This model assumes linear behavior, constant parameters, negligible higher-order effects, and operation within the instrument range.

For a step input x₀ applied at t = 0, with initial output zero:

At t = τ, the output reaches approximately 63.2% of its final value. Approximately 95% is reached at 3τ and 99.3% at 5τ. A smaller time constant gives a faster response.

Frequency response

For sinusoidally varying inputs, an instrument may have amplitude ratio and phase shift that depend on frequency. A first-order system has a break frequency:

where f_c is in hertz when τ is in seconds. Above this region, output amplitude decreases and phase lag increases. Dynamic specifications must therefore be matched to the bandwidth of the process being measured.

Dynamic error

Dynamic error is the difference between the actual input and the value inferred from an output that has not followed the input exactly. A slow temperature sensor may read below the true temperature during heating and above it during cooling. Increasing sensor speed may, however, increase noise or reduce mechanical robustness; instrument selection is a compromise.

4. Loading Effects

An instrument is not perfectly passive. Connecting it to a system can alter the quantity being measured. This is the loading effect.

For example, a voltmeter with finite input resistance connected to a circuit draws current and changes the voltage. In mechanical measurement, a contact probe may deflect a thin component, a pressure gauge may restrict flow, and a thermocouple may conduct heat away from a small specimen.

For a simple voltage divider with source voltage Vₛ, source resistance Rₛ, and voltmeter resistance Rᵥ, the indicated voltage is:

This expression assumes linear resistances and a steady direct-current circuit. Loading is reduced when Rᵥ is much greater than Rₛ.

Mechanical loading can be assessed by comparing the force applied by the instrument with the stiffness of the object. A contact force may cause an elastic displacement approximately given by:

where δ is displacement, F is contact force, and k is effective stiffness, assuming linear elastic behavior and negligible geometric complexity. Noncontact optical, laser, capacitive, or inductive sensors are preferred when contact force would distort the measurand.

5. Environmental Effects

Instrument performance depends on the environment in which it is used. Important influences include:

• temperature and temperature gradients;
• humidity and condensation;
• vibration and shock;
• electromagnetic and radio-frequency interference;
• dust, oil, corrosion, and contamination;
• supply-voltage variation;
• pressure and altitude;
• operator technique and illumination.

Temperature can change dimensions, electrical resistance, viscosity, spring modulus, and electronic offsets. For a component of length L and coefficient of linear thermal expansion α, a temperature change ∆T produces, under unconstrained conditions:

This assumes α is constant over the temperature interval and the component is free to expand. For a 100 mm steel workpiece with α = 11.5 × 10⁻⁶ K⁻¹ and a 10 K temperature rise:

Thus, thermal control is essential when tolerances are comparable with 0.01 mm. International dimensional measurements are commonly referenced to 20 °C; both workpiece and instrument should be brought to thermal equilibrium.

Environmental protection may involve shielding, grounding, vibration isolation, temperature compensation, sealed enclosures, regular cleaning, and controlled measurement procedures. Environmental specifications must be considered together with calibration conditions; a calibration performed in a laboratory does not automatically validate field performance.

6. Characteristics of Common Mechanical Measurement Instruments

Dimensional measurement

Vernier calipers provide versatile external, internal, depth, and step measurements. They are rapid but generally less accurate than micrometers and are sensitive to alignment, jaw wear, and operator force. Outside micrometers use a screw and spindle mechanism and can provide high resolution, but their performance depends on contact force, alignment, cleanliness, and temperature.

Dial indicators and electronic indicators measure relative displacement. They are suitable for runout, alignment, flatness comparisons, and setup verification. Their limitations include stylus alignment, cosine error, backlash, hysteresis, and limited travel.

Coordinate-measuring machines use probing or scanning to establish geometric features in three dimensions. They offer high capability but require environmental control, qualified probing procedures, temperature compensation, and appropriate datum strategies.

Force and torque

Load cells commonly use strain gauges arranged in a Wheatstone bridge. Their important characteristics include capacity, sensitivity, creep, hysteresis, repeatability, overload rating, and temperature coefficient. Torque transducers may use strain gauges, magnetoelastic effects, or optical techniques. Coupling misalignment and shaft bending can produce unwanted components.

Pressure

Bourdon-tube gauges are robust and inexpensive but can exhibit hysteresis, vibration sensitivity, and limited dynamic response. Diaphragm and bellows instruments are useful for lower pressures or differential measurements. Electronic pressure transducers provide signal integration but require attention to temperature effects, overpressure protection, and electrical noise.

Temperature

Resistance temperature detectors provide good stability and near-linear behavior; platinum RTDs are widely used for accurate industrial measurement. Thermocouples are rugged and cover broad temperature ranges but require reference-junction compensation and correct material selection. Infrared thermometers are noncontact, but emissivity, field of view, reflections, and surface condition strongly affect results.

Speed and vibration

Tachometers may be contact or noncontact. Optical tachometers can avoid mechanical loading but require suitable surface markings and line of sight. Accelerometers are used for vibration measurement; mounting stiffness, frequency range, sensitivity, transverse response, and signal conditioning are critical. A sensor with inadequate bandwidth can hide damaging high-frequency components.

7. Instrument Selection Framework

Instrument selection should begin with the measurement requirement rather than the instrument already available. The following sequence is effective:

1. Define the measurand, nominal value, range, tolerance, and required decision.
2. Establish required resolution and permissible measurement uncertainty.
3. Identify static and dynamic behavior, including rate of change and bandwidth.
4. Assess contact, loading, accessibility, geometry, and alignment.
5. Evaluate temperature, vibration, contamination, humidity, and electromagnetic conditions.
6. Check calibration status, traceability, correction data, and maintenance requirements.
7. Consider operator skill, data recording, safety, cost, and availability.
8. Verify the choice by a trial measurement or capability study.

A useful rule is that instrument resolution should be substantially finer than the product tolerance, but resolution alone does not guarantee capability. The instrument's uncertainty, repeatability, environmental sensitivity, and bias must also be suitable.

Worked Example: Selecting a micrometer

A shaft has a specified diameter of 25.000 ± 0.010 mm. The measurement must support acceptance decisions. A micrometer has a 0.001 mm display resolution and a stated accuracy of ±0.003 mm under controlled conditions. The instrument is operated at 23 °C, while the reference condition is 20 °C. The steel shaft has α = 11.5 × 10⁻⁶ K⁻¹ and is 25 mm in diameter.

The thermal expansion relative to 20 °C is:

The temperature effect is about 0.0009 mm, smaller than the micrometer accuracy but not negligible relative to a 0.010 mm tolerance. The micrometer resolution is one-tenth of the total tolerance band, and its stated accuracy is 30% of the ±0.010 mm unilateral limit. It is suitable only if calibration is current, the shaft and micrometer are thermally equilibrated, contact force is controlled, and alignment and cleanliness are maintained. If the measurement decision requires a guard band or uncertainty smaller than ±0.003 mm, a higher-capability instrument or a controlled dimensional laboratory may be required.

8. Calibration, Correction, and Measurement Quality

A calibration curve may be represented by:

where x is the reference input, y is the indicated output, a is zero offset, and b is sensitivity. If a and b are known and b ≠ 0, an input estimate can be obtained from:

This linear model is valid only over the range for which it was established. If nonlinearity is significant, a table, polynomial, piecewise model, or direct correction curve may be necessary. Correction should not be applied blindly: the calibration uncertainty, environmental difference, instrument drift, and model error must be included in the measurement uncertainty.

Measurement records should identify the instrument, serial number, calibration status, operator, date, environmental conditions, method, reference standard, readings, corrections, and decision rule. In regulated or safety-critical work, records must be protected from unauthorized alteration and retained according to the applicable quality system.

9. Illustrative Case: Inspection of a Precision Bearing Shaft

Background

A manufacturing cell produces bearing shafts with a diameter specification of 40.000 ± 0.008 mm. Production volume is high, and measurements are made near a grinding machine that generates vibration and heat.

Problem

Operators report inconsistent results. Some shafts pass on the shop floor but fail during laboratory inspection. The existing method uses a vernier caliper with 0.01 mm resolution.

Analysis

The caliper resolution is comparable to, or larger than, a significant part of the total tolerance width of 0.016 mm. Its contact geometry and operator force also make it unsuitable for reliable fine cylindrical measurement. The grinding machine creates temperature gradients, and the shaft may be warmer than the reference temperature. Vibration can affect readings from a dial comparator, while abrasive particles can prevent proper seating.

Approach and solution

The quality engineer introduces a calibrated outside micrometer with 0.001 mm resolution and controlled ratchet force. A master setting standard is checked at the beginning and end of each shift. Shafts are cleaned and allowed to stabilize on a temperature-controlled bench. Measurements are taken at two axial positions and at several angular orientations to detect taper and out-of-roundness. A digital indicator is added for a rapid comparative check, but final acceptance is based on the calibrated micrometer method.

A control chart of master readings is used to detect zero drift. Operators are trained to maintain alignment and avoid excessive force. The measurement method is periodically compared with a coordinate-measuring machine.

Outcome

The disagreement between shop-floor and laboratory results decreases substantially. Investigation shows that the original caliper method had insufficient resolution and that thermal differences contributed a smaller but measurable bias. The revised method improves repeatability and provides traceable records without imposing the cost of measuring every shaft on a coordinate-measuring machine.

Learning points

The most expensive or most sophisticated instrument is not automatically the best choice. Capability depends on the complete system: instrument characteristics, workpiece condition, environment, operator, method, and decision rule. Comparative instruments can be valuable for speed, but they must be linked to a traceable reference and supported by a suitable calibration plan.

10. Challenges, Technology, and Professional Practice

Modern instruments increasingly incorporate microprocessors, wireless communication, automated compensation, and machine-learning-based signal processing. These technologies can improve filtering, diagnostics, and trend detection, but they do not eliminate metrological responsibility. Digital filtering may suppress real defects as well as noise. An algorithm may introduce bias if it was trained under conditions unlike those in production. Software versions, data integrity, cybersecurity, and audit trails therefore matter.

Engineers should avoid reporting more digits than the measurement supports. They should distinguish an instrument's manufacturer specification from the uncertainty of the complete measurement process. Decisions near specification limits should use a documented rule, such as a guard band, rather than informal judgment.

Ethically, measurement results must not be altered to achieve production targets. Instruments known to be damaged, overdue for calibration, or used outside their environmental limits must be identified and controlled. Where measurements affect safety, legal compliance, or customer conformity, competence, traceability, and impartial review are essential.

Chapter Summary

A measurement system transforms a physical measurand into an indication or record through sensing, conditioning, transmission, and display or decision stages. Instrument selection requires more than checking nominal accuracy. Resolution is the smallest distinguishable increment; sensitivity is output change per input change; linearity describes departure from a reference straight line; hysteresis is direction-dependent indication; drift is time-dependent change; and stability is the ability to retain characteristics over time. Dynamic response determines whether changing inputs are followed without unacceptable lag or attenuation. Loading occurs when the instrument changes the system being measured, while environmental effects arise from temperature, vibration, contamination, humidity, electromagnetic fields, and other conditions. Calibration establishes a traceable relationship with a reference, but uncertainty remains. Sound practice combines suitable instrument characteristics, controlled method, competent operation, environmental management, documented calibration, and an appropriate decision rule.

Key Terms

Measurand: Quantity intended to be measured.

Calibration: Comparison of an instrument indication with a reference under specified conditions.

Traceability: Property of a result relating it to a reference through documented calibrations.

Accuracy: Closeness of agreement with a reference value.

Precision: Closeness of agreement among repeated results.

Resolution: Smallest distinguishable input increment.

Sensitivity: Change in output divided by change in input.

Linearity: Degree to which a calibration relationship follows a specified straight line.

Hysteresis: Dependence of indication on the direction of input change.

Repeatability: Agreement under the same measurement conditions.

Reproducibility: Agreement when specified measurement conditions change.

Drift: Gradual indication change at constant measurand.

Stability: Ability to maintain metrological characteristics over time.

Dead zone: Input interval producing no detectable output change.

Time constant: Parameter describing the speed of a first-order response.

Loading effect: Change in the measurand or system caused by the measuring instrument.

Uncertainty: Parameter characterizing dispersion of values reasonably attributable to a measurand.

Full-scale span: Difference between upper and lower range limits.

Environmental effect: Change in measurement performance caused by surroundings.

Guard band: Decision margin used to account for measurement uncertainty near a specification limit.

Review Questions

Short Answer Questions

1. Distinguish accuracy from precision and provide one example of a precise but inaccurate instrument.
2. Define resolution and explain why it is not equivalent to accuracy.
3. What is hysteresis, and how is it evaluated experimentally?
4. Explain the difference between calibration and adjustment.
5. Identify four environmental factors that can affect mechanical measurements.

Long Answer Questions

1. Explain the static characteristics of a measurement instrument, including sensitivity, linearity, hysteresis, repeatability, drift, and stability.
2. Derive the step response of a first-order measurement system and interpret the significance of its time constant.
3. Discuss loading effects in electrical and mechanical measurements, including methods for reducing them.
4. Compare micrometers, vernier calipers, dial indicators, and coordinate-measuring machines for dimensional inspection.
5. Develop a systematic procedure for selecting an instrument for a safety-critical mechanical measurement.

Analytical / Application Questions

1. A temperature sensor has a time constant of 4 s. Calculate the time required to reach approximately 95% of its final response, and explain whether it is suitable for a process changing significantly every 2 s.
2. A 150 mm steel component is measured 8 K above its reference temperature. Using α = 11.5 × 10⁻⁶ K⁻¹, calculate the dimensional change and discuss its significance for a ±0.010 mm tolerance.
3. A pressure transducer shows different readings during increasing and decreasing pressure cycles. Design a test to separate hysteresis, repeatability, and drift, and state what evidence would support replacing the instrument.

Multiple Choice Questions with Answers

Question 1

Which characteristic describes the smallest input increment that an instrument can distinguish?

A. Sensitivity

B. Resolution

C. Hysteresis

D. Drift

Correct Answer: B. Resolution

Question 2

An instrument gives closely grouped readings, but all are 0.5 mm above the reference value. The instrument is best described as:

A. Precise but biased

B. Inaccurate and imprecise

C. Sensitive but unstable

D. Linear but hysteretic

Correct Answer: A. Precise but biased

Question 3

For a first-order instrument, the output reaches approximately what percentage of its final step value after one time constant?

A. 37.0%

B. 50.0%

C. 63.2%

D. 95.0%

Correct Answer: C. 63.2%

Question 4

Which condition is most likely to produce hysteresis in a mechanical instrument?

A. A perfectly frictionless linkage

B. Backlash or elastic memory

C. A larger display resolution

D. A constant reference input

Correct Answer: B. Backlash or elastic memory

Question 5

A voltmeter has resistance Rᵥ and is connected to a source with resistance Rₛ. Which expression gives the indicated voltage for an ideal resistive divider?

A. Vₛ(Rₛ + Rᵥ)

B. VₛRₛ/Rᵥ

C. VₛRᵥ/(Rₛ + Rᵥ)

D. Vₛ/(RₛRᵥ)

Correct Answer: C. VₛRᵥ/(Rₛ + Rᵥ)

Question 6

Which practice most directly reduces thermal dimensional error in precision inspection?

A. Reading the instrument more quickly

B. Using a coarser scale

C. Allowing the workpiece and instrument to reach thermal equilibrium

D. Increasing contact force

Correct Answer: C. Allowing the workpiece and instrument to reach thermal equilibrium

Question 7

A calibration relationship is represented by y = a + bx. Which quantity represents the zero offset?

A. x

B. y

C. a

D. b

Correct Answer: C. a

Question 8

Which instrument is generally most appropriate for measuring shaft runout relative to a reference surface?

A. Dial indicator

B. Thermocouple

C. Bourdon pressure gauge

D. Load cell

Correct Answer: A. Dial indicator

Question 9

If the maximum calibration deviation is 0.02 V and the full-scale span is 10 V, the percentage nonlinearity is:

A. 0.02%

B. 0.2%

C. 2%

D. 20%

Correct Answer: B. 0.2%

Question 10

Which statement about traceability is correct?

A. It guarantees zero measurement error.

B. It means the result is related to a reference through documented calibrations.

C. It eliminates environmental effects.

D. It makes adjustment unnecessary.

Correct Answer: B. It means the result is related to a reference through documented calibrations.