Edge, Fog, Cloud, and IoT Data Platforms

Introduction and Context

An Internet of Things (IoT) system is not merely a collection of sensors connected to the internet. It is a data-processing system that senses physical conditions, communicates observations, applies computation, stores information, supports decisions, and often controls electrical or mechanical equipment. In a smart electrical system, examples include distribution automation, condition monitoring of transformers, smart buildings, photovoltaic plants, electric-vehicle charging, and demand-response control.

A central design question is where computation and data storage should occur. A fully centralized design sends measurements to a remote cloud. A distributed design performs some processing near the device, possibly at an edge gateway or fog node, and sends selected information to cloud platforms. The correct choice depends on latency, bandwidth, availability, privacy, security, computational capability, lifecycle cost, and operational risk.

This chapter develops an edge-to-cloud architecture for acquiring, filtering, storing, processing, visualizing, and governing IoT data. It also compares centralized and distributed approaches and applies the concepts to an illustrative electrical distribution case.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Distinguish device, edge, fog, cloud, and IoT data-platform functions.
2. Design an edge-to-cloud pipeline for electrical measurements and events.
3. Select filtering, buffering, storage, and stream-processing methods appropriately.
4. Compare architectures using latency, resilience, bandwidth, privacy, security, and lifecycle cost.
5. Explain device management, data governance, visualization, and digital services.
6. Evaluate a practical smart-electrical-system architecture and identify its limitations.

1. The Computing Continuum

The terms edge, fog, and cloud describe locations on a computing continuum rather than completely separate technologies.

• Device or endpoint layer: Sensors, meters, relays, actuators, protection equipment, and embedded controllers. Devices acquire measurements and may perform simple control.
• Edge layer: Computing located close to the physical process, such as an industrial gateway, substation computer, building controller, or intelligent electronic device (IED). Edge systems provide rapid response and local autonomy.
• Fog or regional layer: An intermediate layer connecting several edge sites. A utility control-centre server, regional data concentrator, or campus platform may aggregate and analyze data from many local systems.
• Cloud layer: Elastic remote computing and storage used for fleet-wide analytics, long-term records, model training, enterprise integration, and digital services.

These layers are logical roles. A commercial product may implement more than one role, and the boundary can change as hardware and networks evolve.

Figure: Logical edge-to-cloud roles in an IoT data platform.

2. From Physical Measurement to Usable Data

2.1 Acquisition and semantic meaning

A measurement is useful only when its meaning is preserved. A record should normally include a value, unit, timestamp, quality indicator, device identity, location, and relevant operating context. For example, “7.2” is ambiguous; “7.2 kW, feeder F12, 2025-03-08 10:15:00 UTC, quality=good” is interpretable.

Electrical measurements may include voltage, current, active power, reactive power, frequency, power factor, temperature, breaker state, harmonics, and energy. Sampling requirements vary. Protection and waveform applications may need high-rate synchronized samples, whereas energy dashboards may need one-minute or fifteen-minute intervals.

Time synchronization is essential when measurements from different devices are compared. Systems may use Network Time Protocol (NTP), Precision Time Protocol (PTP), or satellite timing. A timestamp should identify whether it represents measurement time, gateway receipt time, or cloud ingestion time.

2.2 Communication and protocols

At the device level, protocols may include Modbus, CAN, HART, OPC UA, IEC 61850, or proprietary interfaces. IoT messaging commonly uses MQTT or constrained application protocols. MQTT follows a publish/subscribe model: devices publish messages to topics, while authorized consumers subscribe to relevant topics. This decouples producers from applications but requires careful topic design and access control.

A gateway often translates legacy industrial protocols into a common data model. Translation must not silently change units, signs, scaling, timestamps, or quality flags. In electrical engineering, conventions such as leading or lagging reactive power and phase sequence must be documented.

2.3 Validation and preprocessing

Edge validation can reject impossible values, flag suspicious values, remove duplicates, and detect missing data. A simple range test accepts a value only when it lies within a technically justified interval. A rate-of-change test compares the current value with recent values. Such tests should flag rather than automatically delete data when the cause is uncertain.

Filtering reduces noise and data volume. A moving average of window length N is:

Here y_k is the kth sample, N is the number of samples, and ȳ_k is the filtered output. The formula assumes equally spaced samples and is useful for slowly changing quantities. It introduces delay and can obscure short faults, so it is unsuitable as the sole method for protection decisions.

For an exponentially weighted moving average:

The variables have the same meaning as above, s_k is the filtered estimate, and α is dimensionless. A larger α responds faster but smooths less. The initial value s₀ must be specified.

Other preprocessing includes resampling, feature extraction, compression, event detection, and calculation of derived quantities. For balanced three-phase active power, under sinusoidal steady-state assumptions:

Here V_L is line-to-line RMS voltage in volts, I_L is line RMS current in amperes, and φ is the voltage-current phase angle. The result is in watts. This relationship is not a substitute for waveform-based power measurement when the waveform is distorted or the system is unbalanced.

3. Data Storage and Information Models

An IoT platform commonly uses several storage types rather than one universal database.

A good design preserves raw or near-raw data for a defined period, while also creating validated and aggregated datasets. Downsampling should be reproducible. For example, a fifteen-minute energy value must specify whether it is an average power, an integral of power, or a meter register difference.

Data quality dimensions include completeness, validity, accuracy, consistency, timeliness, uniqueness, and lineage. A data catalogue should document source, owner, units, schema, retention, access classification, and transformation history.

4. Stream Processing and Event-Driven Operation

Batch processing analyzes accumulated data, whereas stream processing acts continuously as records arrive. Electrical applications often require both. A cloud service may train a transformer-health model from years of history, while an edge controller detects overheating within seconds.

A stream processor typically performs:

1. ingestion and authentication;
2. schema validation;
3. time ordering or windowing;
4. filtering and enrichment;
5. aggregation and feature calculation;
6. rule or model evaluation;
7. alert generation and downstream delivery.

Window types include tumbling windows, which do not overlap; sliding windows, which overlap; and session windows, which group activity separated by periods of inactivity. Late-arriving data requires a policy. Systems may use event time, watermarks, and bounded lateness, but the policy must be explicit because corrections can change previously published results.

A simple event rule might generate an alert when transformer temperature exceeds a threshold for a specified duration and the measurement quality is good. Thresholds should be based on equipment ratings, ambient conditions, loading, and engineering studies rather than arbitrary software defaults.

Machine learning can support anomaly detection, load forecasting, and predictive maintenance. It does not remove the need for sensor validation, representative training data, explainability, human review, and fail-safe control logic. Safety-critical actions should not depend exclusively on an unverified cloud model.

5. Visualization and Digital Services

Visualization converts data into decisions. Effective dashboards show context, not merely large numbers. A feeder dashboard may include a one-line diagram, live loading, voltage limits, alarm state, trend plots, asset identity, timestamp freshness, and data-quality status.

Color should not be the only indication of alarm state because users may have color-vision limitations. Alarm systems need priorities, acknowledgements, escalation rules, suppression of duplicate alarms, and an audit trail. A dashboard should distinguish measured values from estimates and stale values from current values.

Digital services built on the platform include:

• remote asset monitoring;
• outage and fault notification;
• energy benchmarking;
• demand-response coordination;
• maintenance scheduling;
• power-quality reporting;
• customer and operator portals;
• regulatory and sustainability reporting.

APIs should expose stable, documented interfaces. Services should apply role-based access, rate limits, versioning, and audit logging.

6. Device Management and Lifecycle Engineering

An IoT platform must manage devices throughout their lifecycle: commissioning, identity provisioning, configuration, monitoring, software updates, certificate renewal, replacement, retirement, and secure disposal.

Each device should have a unique identity and a controlled ownership record. Secure boot can verify that approved software runs at startup. Signed firmware updates, staged deployment, rollback capability, and update reporting reduce operational risk. An update that works in a laboratory may fail in a remote substation with limited bandwidth, so deployment should begin with a small cohort and expand only after verification.

Configuration management is as important as firmware management. Changes to sampling rates, control limits, network endpoints, and calibration values should be authorized, versioned, tested, and reversible. A device inventory that does not reflect actual field state is a major security and maintenance weakness.

7. Comparing Centralized and Distributed Architectures

A centralized architecture simplifies fleet-wide administration and can exploit large-scale cloud computation. However, every decision depending on a remote connection is vulnerable to communication delay, service interruption, and bandwidth costs.

A distributed architecture places time-critical processing near the equipment and sends summaries or selected events upstream. It improves local autonomy and can reduce exposure of raw data, but it increases the number of systems that must be secured, updated, monitored, and integrated.

A useful latency decomposition is:

The terms represent sensing, queueing, network transfer, computation, and actuation times. They are nonnegative and should be measured under realistic load; averages alone can hide dangerous tail latency. An edge decision usually reduces T_network and may reduce T_queue, but edge hardware may have greater T_compute for complex workloads.

For an approximate data-volume estimate:

Here n is the number of channels, r is samples per second, b is bytes per sample, and t is time in seconds. The result is bytes before protocol overhead, metadata, retransmissions, and compression. For 100 channels sampled at 10 samples/s with 8 bytes per sample for 86,400 s:

D = 100 × 10 × 8 × 86,400 = 691,200,000 bytes, approximately 691 MB per day before overhead. Edge aggregation can reduce this volume substantially, but raw data may still be retained selectively for fault investigation.

Lifecycle cost should include hardware, connectivity, cloud computation, storage, software licences, cybersecurity, engineering, field visits, training, energy consumption, and decommissioning. The cheapest initial architecture may not have the lowest total cost of ownership.

The most effective architecture is frequently hybrid: local control and safety at the edge, regional coordination in fog systems, and long-term analytics and enterprise services in the cloud.

8. Security, Privacy, Governance, and Regulation

IoT security must protect confidentiality, integrity, availability, safety, and accountability. A practical defence-in-depth design includes device identity, mutual authentication, encrypted communication, network segmentation, least privilege, secure configuration, vulnerability management, logging, backup, incident response, and recovery testing.

Security controls must recognize operational technology constraints. A patch that interrupts a protection relay or industrial process may create greater immediate risk than a delayed patch, so compensating controls and maintenance windows are necessary. Safety functions should remain operable during cloud or authentication-service failure where appropriate.

Privacy may arise from occupancy data, customer energy profiles, location, or employee activity. Data minimization means collecting only what is justified. Pseudonymization does not guarantee anonymity when datasets can be linked. Retention periods, access rights, lawful processing, and cross-border transfer rules depend on jurisdiction and organizational policy.

Governance assigns responsibility for data quality, access approval, retention, incident reporting, and model performance. Standards and regulatory obligations vary by sector and country; engineers must consult applicable electrical, cybersecurity, privacy, and critical-infrastructure requirements rather than assuming that a generic IoT platform is compliant.

9. Illustrative Case: Smart Distribution-Transformer Monitoring

This is a fictional illustrative case.

Background

A utility operates 600 distribution transformers. It wants to reduce failures and identify overloads. Each site has temperature, oil-level, current, voltage, and door-status sensors. Connectivity is intermittent at several rural locations.

Problem

Sending every raw reading to a cloud platform would increase cellular cost and would not guarantee timely local action. Conversely, storing all information only at the site would make fleet-wide comparison and maintenance planning difficult.

Analysis

Temperature and electrical loading require local plausibility checks. A high-temperature alarm must remain available during network loss. Long-term trends, peer comparison, and model training require centralized history. Waveforms are valuable during faults but do not need continuous transmission.

Approach

Each transformer gateway authenticates sensors and timestamps measurements using synchronized time. At the edge it:

• validates ranges and quality flags;
• calculates one-minute summaries;
• detects sustained overload and temperature events;
• stores seven days of local data in a circular buffer;
• sends alarms immediately when communication is available;
• transmits summaries periodically;
• uploads selected high-resolution records after an event.

A regional fog service aggregates transformer data, correlates alarms with feeder topology, and forwards normalized data to the cloud. The cloud stores multi-year history, supports maintenance dashboards, compares similar assets, and trains a condition-assessment model. The model produces a maintenance recommendation, but a qualified engineer approves work orders. Local protection and emergency shutdown functions remain independent of the cloud.

Outcome

The design reduces routine cellular traffic, preserves operation during communication outages, and enables fleet-wide analysis. It also introduces additional gateways, update procedures, certificate management, and model-governance responsibilities.

Learning points

1. Data reduction should occur after preserving information needed for safety and diagnosis.
2. Local autonomy must be explicit rather than assumed.
3. A cloud recommendation is not automatically a control command.
4. Device lifecycle management is part of system design, not an afterthought.
5. Architecture should be evaluated using measurable latency, availability, data quality, and cost requirements.

10. Design Method and Best Practices

A disciplined design process is:

1. Identify decisions, hazards, users, and required response times.
2. Classify measurements by rate, accuracy, criticality, and retention.
3. Define the data model, units, timestamps, quality flags, and asset identity.
4. Place control, filtering, storage, and analytics functions on the continuum.
5. Estimate bandwidth, latency, compute, storage, and lifecycle costs.
6. Design for communication loss, power loss, bad data, device compromise, and cloud outage.
7. Implement security and device management from commissioning onward.
8. Test with realistic traffic, delayed messages, duplicate messages, clock errors, and failure conditions.
9. Monitor service-level indicators such as freshness, completeness, alarm latency, update success, and false-alarm rate.
10. Review the architecture as equipment, regulations, and operational requirements change.

Important contemporary developments include containerized edge applications, private 5G, digital twins, event-driven serverless services, federated learning, and energy-aware computing. These technologies can be useful, but adoption should follow a verified operational need. Complexity, vendor dependence, opaque models, and new attack surfaces must be included in the decision.

Chapter Summary

IoT data platforms form a computing continuum from devices through edge and fog systems to cloud services. Devices acquire measurements; edge systems validate, filter, buffer, and control locally; fog systems coordinate sites and process regional streams; cloud systems provide scalable storage, fleet-wide analytics, visualization, and digital services. A successful pipeline preserves measurement meaning through units, timestamps, quality flags, identity, and lineage.

Centralized architectures simplify large-scale management and analytics but depend more heavily on communication and remote services. Distributed architectures improve latency, resilience, bandwidth efficiency, and data locality, but increase the number of components requiring security and lifecycle management. Hybrid architectures are often appropriate for smart electrical systems.

Engineering decisions should be based on measured latency, data volume, availability, privacy, safety, security, and total lifecycle cost. Filtering and machine learning must not eliminate information required for fault analysis or safe operation. Device identity, secure updates, governance, and failure-mode testing are essential parts of the architecture.

Key Terms

Edge computing: Processing performed near the physical source of data.

Fog computing: Intermediate distributed computing between edge sites and the cloud.

Cloud computing: Remote, scalable computing and storage delivered as a service.

IoT gateway: Device that connects local equipment and translates or aggregates protocols.

IED: Intelligent electronic device used for monitoring, protection, or control.

MQTT: Lightweight publish/subscribe messaging protocol commonly used in IoT.

Time-series database: Database optimized for timestamped observations.

Data lake: Scalable storage for raw and processed data in varied formats.

Stream processing: Continuous processing of data as events arrive.

Event time: Time at which a physical event occurred.

Watermark: Processing indication of how complete event-time data is expected to be.

Data lineage: Record of a datum’s source and transformations.

Device provisioning: Establishing identity, credentials, configuration, and ownership.

Secure boot: Startup verification that approved software is executed.

Digital twin: Digital representation linked to a physical asset and its data.

Latency: Time between an event and the corresponding response.

Resilience: Ability to continue or recover operation under faults.

Data minimization: Collecting and retaining only justified data.

Review Questions

Short Answer Questions

1. What functional roles do edge, fog, and cloud layers perform in an IoT system?
2. Why are timestamps and quality flags important for electrical measurements?
3. State one advantage and one limitation of moving filtering to the edge.
4. What is the difference between batch processing and stream processing?
5. Name four elements of a secure device-lifecycle process.

Long Answer Questions

1. Explain how a hybrid edge-to-cloud architecture can support both local control and fleet-wide analytics.
2. Compare time-series databases, relational databases, and object storage for an electrical IoT platform.
3. Discuss how latency, bandwidth, resilience, privacy, and lifecycle cost influence architectural placement.
4. Explain why visualization and alarm management require data quality and context, not merely attractive graphics.
5. Describe a cybersecurity and governance framework for remotely managed IoT gateways.

Analytical / Application Questions

1. A site has 200 channels sampled at 5 samples/s, with 4 bytes per sample, continuously for 24 hours. Calculate the raw daily data volume using D = n r b t, ignoring overhead. Explain two reasons the actual volume may differ.
2. Design an edge-to-cloud pipeline for a solar farm that must detect inverter faults locally but produce monthly performance reports centrally. Specify acquisition, edge processing, storage, communications, and cloud services.
3. A utility proposes sending all sensor data to the cloud and issuing protective commands from a cloud application. Critically evaluate the proposal using latency, outage resilience, safety, cybersecurity, and lifecycle considerations.

Multiple Choice Questions with Answers

Question 1

Which function is most appropriately assigned to an edge system in a smart electrical installation?

A. Long-term fleet-wide model training only

B. Local validation and rapid control during network loss

C. Global regulatory reporting only

D. Replacing all device sensors

Correct Answer: B. Local validation and rapid control during network loss

Question 2

What information most directly preserves the meaning of an electrical measurement?

A. Value only

B. Value and screen colour

C. Value, unit, timestamp, identity, and quality information

D. Cloud provider name only

Correct Answer: C. Value, unit, timestamp, identity, and quality information

Question 3

For a moving average, increasing the window length generally produces which effect?

A. More smoothing and potentially more delay

B. Less smoothing and zero delay

C. Higher sampling frequency

D. Automatic calibration

Correct Answer: A. More smoothing and potentially more delay

Question 4

Using D = n r b t, what raw data volume is produced by 100 channels sampled at 10 samples/s, with 8 bytes per sample, for one day?

A. 6.912 MB

B. 69.12 MB

C. 691.2 MB

D. 6.912 GB

Correct Answer: C. 691.2 MB

Question 5

Which storage type is generally best suited to high-volume timestamped power and temperature readings?

A. Time-series database

B. Word-processing file

C. Graphics editor project

D. Static firmware image

Correct Answer: A. Time-series database

Question 6

What is a principal advantage of stream processing?

A. It requires all data to be stored for years before use

B. It can evaluate events as they arrive

C. It eliminates the need for timestamps

D. It prevents all false alarms

Correct Answer: B. It can evaluate events as they arrive

Question 7

In the latency expression T_total = T_sense + T_queue + T_network + T_compute + T_actuate, which term is most directly reduced by moving processing closer to the equipment?

A. Sensing time only

B. Network transfer time

C. Actuator physical response time always

D. Equipment manufacturing time

Correct Answer: B. Network transfer time

Question 8

Why should a critical local protection function not depend exclusively on a cloud model?

A. Cloud models cannot process numerical data

B. Cloud dependence can introduce communication, availability, and safety risks

C. Edge devices cannot measure electricity

D. Models are never useful in maintenance

Correct Answer: B. Cloud dependence can introduce communication, availability, and safety risks

Question 9

Which practice best supports secure firmware deployment?

A. Unsigned updates from any source

B. Permanent shared passwords

C. Signed updates, staged rollout, and rollback capability

D. Disabling update records

Correct Answer: C. Signed updates, staged rollout, and rollback capability

Question 10

What is a major trade-off of a highly distributed architecture?

A. It always increases bandwidth use

B. It removes the need for cybersecurity

C. It improves local autonomy but increases management complexity

D. It prevents local operation

Correct Answer: C. It improves local autonomy but increases management complexity