Messaging and Application-Layer Protocols

Introduction

The application layer is where Internet of Things (IoT) data acquires meaning. Lower-layer protocols deliver frames and packets, but application-layer protocols define how a temperature reading, alarm, command, image, or equipment status is represented, exchanged, acknowledged, and secured. Selecting an appropriate protocol is therefore an architectural decision rather than a simple programming choice.

IoT systems commonly connect constrained sensors, embedded controllers, gateways, cloud platforms, enterprise applications, and human users. These participants differ in memory, processing power, bandwidth, energy supply, latency requirements, reliability needs, and security capabilities. No single protocol is optimal for every situation. MQTT is often effective for telemetry through a broker; CoAP suits constrained devices and REST-like interaction; HTTP/REST integrates naturally with web services; WebSocket supports continuous two-way communication; AMQP provides enterprise messaging features; DDS supports real-time, distributed systems; and OPC UA enables interoperable industrial automation.

This chapter compares these protocols and develops the patterns needed to apply them: request–response, publish–subscribe, brokers, topics, quality of service (QoS), and device discovery.

Learning Objectives

By the end of this chapter, the learner should be able to:

• explain the purpose of application-layer protocols in IoT;
• distinguish request–response from publish–subscribe communication;
• describe brokers, topics, queues, QoS, sessions, and device discovery;
• compare MQTT, MQTT-SN, CoAP, HTTP/REST, WebSocket, AMQP, DDS, and OPC UA;
• select a protocol according to constraints such as energy, latency, interoperability, and reliability;
• design a basic multi-protocol IoT architecture; and
• identify security, governance, and operational risks in messaging systems.

1. Application-Layer Communication in IoT

An IoT application-layer exchange has several logical elements:

1. Endpoint: a device, gateway, service, or client that sends or receives information.
2. Message: data and associated metadata such as content type, timestamp, identity, and correlation identifier.
3. Interaction model: the pattern governing communication.
4. Transport: commonly TCP, UDP, or a secured variant such as TLS or DTLS.
5. Naming model: a URL path, topic, queue, resource identifier, or data-domain name.
6. Reliability and delivery semantics: what happens when a packet, endpoint, or connection fails.

An application protocol should not be judged only by speed. A protocol may be lightweight but lack useful discovery or interoperability; another may provide excellent reliability but require too much memory for a small battery-powered node. The correct choice depends on the complete system.

1.1 Request–Response

In request–response communication, a client asks for an operation and a server returns a response. The response may contain data, an acknowledgement, an error, or the result of an action. HTTP and CoAP are the clearest examples.

A request may be expressed conceptually as:

GET /building/3/room/12/temperature

The response may include a value, unit, timestamp, status code, and representation format. This model is appropriate when the client knows which resource it needs and can tolerate waiting for a response. It supports reading a current value, updating configuration, or invoking a device operation.

Its limitations become visible when many clients need the same event. If 1,000 clients repeatedly request a temperature, the system creates repeated traffic and processing. Polling can also introduce delay between measurements.

1.2 Publish–Subscribe

In publish–subscribe (pub–sub), a publisher sends a message without addressing a particular recipient. Subscribers express interest in a subject, topic, or data key. A broker or middleware service matches publications to subscriptions.

For example:

• publisher: a room sensor;
• topic: building/3/room/12/temperature;
• subscribers: a dashboard, an alarm service, and a data-storage service.

The publisher need not know which subscribers exist. This decouples producers from consumers in space, time, and often processing rate. Pub–sub is well suited to telemetry, event notification, and fan-out distribution.

The trade-off is additional infrastructure and more complex reasoning about retained messages, duplicate delivery, ordering, subscription permissions, and offline clients.

1.3 Broker, Topic, Queue, and Session

A broker is an intermediary that accepts messages and applies routing, authentication, authorization, buffering, and delivery rules. MQTT brokers route messages using topics. AMQP brokers commonly route messages through exchanges into queues. Some DDS deployments use distributed discovery and data-centric middleware rather than one central broker.

A topic is a logical name used to classify messages. Topic design is part of data architecture. A good topic hierarchy is stable, meaningful, and authorization-friendly, for example:

tenant/site/device/measurement

A topic should not unnecessarily encode frequently changing information or confidential data. Wildcard subscriptions are powerful but must be controlled because a broad subscription can expose excessive information.

A queue stores messages for one or more consumers until they are processed. Queues are useful for work distribution, buffering, and consumer recovery. A topic generally describes what information is about; a queue generally represents where work waits.

A session contains protocol state such as subscriptions, authentication context, unacknowledged messages, and connection status. Persistent sessions can help intermittent devices, but they consume broker resources and require expiry policies.

1.4 Quality of Service

QoS is not one universal property. It may refer to delivery assurance, latency, ordering, durability, priority, deadline compliance, or availability. A system designer must state which property is required.

MQTT defines three familiar delivery levels:

• QoS 0: at most once; the message is delivered without a protocol-level retry acknowledgement.
• QoS 1: at least once; delivery is acknowledged, but duplicates are possible.
• QoS 2: exactly once at the MQTT protocol exchange level; it uses a more extensive handshake and incurs greater overhead.

“Exactly once” does not automatically mean exactly-once business action. If a device switches a motor each time it receives a command, an application-level command identifier and idempotency rule may still be required. A retry can be safe when the operation is idempotent, such as setting a target value to 20 °C, but unsafe when the operation is “increase speed by 10%.”

CoAP commonly uses confirmable and non-confirmable messages. AMQP uses acknowledgements, persistence, and broker features; DDS offers reliability, durability, deadlines, liveliness, and other policies; OPC UA supports monitored items, subscriptions, and historical access. These mechanisms are not directly equivalent and should not be compared by labels alone.

1.5 Device Discovery

Discovery enables a client or service to learn what devices exist, how they are named, what data they expose, and how they can be accessed. Discovery may be:

• static: identifiers and endpoints are configured manually;
• directory-based: a registry stores device metadata;
• protocol-based: the protocol provides discovery mechanisms;
• local: devices announce services on a local network;
• semantic: metadata describes capabilities, units, relationships, and meaning.

Discovery must be secured. An unauthenticated device catalogue can reveal building occupancy, industrial assets, or exploitable interfaces. Devices should expose only the metadata needed by authorized clients.

Figure: The two principal interaction patterns used by IoT application protocols.

2. MQTT and MQTT-SN

2.1 MQTT

MQTT is a lightweight publish–subscribe protocol designed for constrained devices and unreliable or low-bandwidth networks. It normally operates over a long-lived TCP connection, commonly protected by TLS. An MQTT client connects to a broker, publishes messages to topics, and subscribes to topics.

Important MQTT concepts include:

• CONNECT and CONNACK: establish and acknowledge a session;
• PUBLISH: carry application data;
• SUBSCRIBE and SUBACK: request and acknowledge subscriptions;
• PINGREQ and PINGRESP: maintain an apparently idle connection;
• Will message: notify subscribers when an unexpected disconnection is detected;
• retained message: store the latest value for a topic so a new subscriber can receive current state;
• shared subscription: distribute messages among consumers where supported by the implementation.

MQTT is particularly effective for telemetry and event distribution. Its topic hierarchy is easy to understand, and broker-based fan-out simplifies integration. However, MQTT does not itself define the meaning of a payload. JSON, SenML, CBOR, Protocol Buffers, or a proprietary encoding may be used, but the system must define schemas, units, timestamps, and versioning.

MQTT is not automatically suitable for hard real-time control. Broker failure, network delay, retransmission, and scheduling can make timing nondeterministic. Safety-critical control should normally remain local or use a technology with appropriate real-time guarantees.

2.2 MQTT-SN

MQTT-SN (MQTT for Sensor Networks) adapts MQTT concepts for networks where TCP is undesirable or unavailable, such as low-power wireless sensor networks. It can use transports such as UDP and supports a gateway between sensor-network clients and an MQTT broker.

A significant feature is topic identifiers. Instead of repeatedly transmitting long topic strings, a gateway can assign short identifiers to registered topics. MQTT-SN also supports sleeping clients, which can wake periodically, exchange messages, and return to a low-power state.

The gateway is a major architectural component. It translates between MQTT-SN and MQTT, maintains registrations and state, and may become a failure or security boundary. MQTT-SN should therefore be selected when constrained networking and sleeping behavior justify the additional gateway design.

3. CoAP

The Constrained Application Protocol (CoAP) is designed for constrained nodes and networks. It follows a REST-like resource model and commonly runs over UDP, with security provided by DTLS; CoAP deployments can also use other transports and security arrangements.

CoAP methods resemble HTTP methods: GET retrieves a resource, POST creates or triggers an operation, PUT creates or replaces a resource, and DELETE removes one. CoAP uses compact binary headers and supports confirmable messages. A confirmable message is acknowledged; a non-confirmable message is not individually acknowledged. CoAP also supports response codes, content negotiation, caching, block-wise transfer, observation, and resource discovery.

The CoAP Observe capability allows a client to register interest in a resource and receive notifications when it changes. This provides an efficient alternative to repeated polling and demonstrates that a REST-like protocol can also support event-driven behavior.

A standard discovery approach uses the resource directory at /.well-known/core, where a server can describe available resources using link-format metadata. Discovery does not by itself guarantee semantic interoperability: two devices may both expose a “temperature” resource while using different units or calibration assumptions.

CoAP is a strong choice for local constrained networks and direct device interaction. It is less naturally integrated with enterprise applications than HTTP, although gateways can translate CoAP resources into HTTP APIs or message-broker topics.

4. HTTP and REST

HTTP is the dominant application protocol of the Web. HTTP clients and servers exchange requests and responses using methods, status codes, headers, and representations. REST is an architectural style, not a wire protocol. A RESTful IoT service typically models devices, measurements, commands, and configurations as resources identified by URLs.

For example:

• GET /devices/thermostat-7/state
• PUT /devices/thermostat-7/setpoint
• POST /devices/thermostat-7/commands

HTTP provides mature tooling, proxies, caching, authentication integration, observability, and developer familiarity. It is therefore excellent for cloud APIs, dashboards, provisioning services, and northbound integration.

Its disadvantages for constrained devices include comparatively large headers, connection-management overhead, and a request-driven model. HTTP polling can be inefficient for frequent telemetry. Persistent connections, HTTP/2, server-sent events, or a gateway can improve performance, but the architecture should be evaluated rather than assumed to be efficient.

REST design should use meaningful resource boundaries, consistent status codes, idempotent operations where possible, pagination for collections, explicit content types, and versioning. A POST command should return a correlation identifier when processing is asynchronous. APIs should distinguish a request accepted for later execution from an operation already completed.

5. WebSocket

WebSocket begins with an HTTP-based handshake and then provides a persistent, full-duplex connection between client and server. Either side can send messages after the connection is established. This makes WebSocket useful for live dashboards, control panels, collaborative interfaces, alarms, and browser-to-gateway applications.

WebSocket is not itself a pub–sub broker protocol. A server may implement subscriptions over WebSocket, or a WebSocket gateway may bridge browser clients to MQTT, AMQP, or another messaging system. The application must define message types, authentication, authorization, heartbeats, reconnection behavior, and back-pressure handling.

WebSocket is generally less suitable than MQTT for large populations of intermittently connected sensors. Browsers and interactive applications, however, benefit from its native two-way communication and broad platform support.

6. AMQP

The Advanced Message Queuing Protocol (AMQP) is an enterprise messaging protocol with strong concepts for reliable transfer, routing, acknowledgements, queues, and transactions. Implementations commonly provide exchanges, queues, bindings, consumers, publisher confirmations, persistence, and dead-letter handling.

A publisher sends a message to an exchange. Routing rules determine which queues receive it. Consumers read messages from queues and acknowledge successful processing. If a consumer fails before acknowledgement, the broker may redeliver the message, depending on configuration.

AMQP is appropriate for backend integration, business workflows, task distribution, and systems where message durability and controlled processing are important. It generally requires more infrastructure and protocol sophistication than MQTT. It is not usually the first choice for a tiny battery-powered sensor, but an IoT gateway can publish sensor data to MQTT while forwarding selected events to an AMQP enterprise bus.

AMQP reliability features must be designed with storage and failure assumptions in mind. A message marked persistent may still be lost if the broker is not configured for durable storage and appropriate replication. “Reliable” is an end-to-end property, not merely a message flag.

7. DDS

The Data Distribution Service (DDS) is a data-centric publish–subscribe technology for distributed real-time systems. Instead of treating messages primarily as arbitrary topic payloads, DDS models strongly typed data and distributes it according to policies. It is common in robotics, autonomous systems, simulation, defence, aerospace, and industrial control.

DDS provides rich Quality of Service policies, including reliability, durability, deadline, latency budget, liveliness, history depth, ownership, and resource limits. The Data Distribution Service specification and the DDS Interoperability Wire Protocol are associated with the Object Management Group (OMG).

DDS may use decentralized discovery and can support communication without a central broker. This can reduce a single point of failure and support local real-time exchange, but it increases configuration and operational complexity. DDS is powerful when timing, data ownership, type consistency, and distributed control matter. It can be excessive for simple environmental telemetry.

A useful distinction is that MQTT usually gives the application a simple broker-and-topic abstraction, whereas DDS exposes a richer data-distribution model with policies that can be matched to system requirements.

8. OPC UA

OPC Unified Architecture (OPC UA) is an industrial interoperability framework rather than merely a lightweight messaging protocol. It represents information in an address space containing objects, variables, methods, references, types, and metadata. This enables clients to understand not only a value but also its engineering unit, relationships, permissions, and operations.

OPC UA supports client–server services such as browsing, reading, writing, method invocation, and monitored subscriptions. It also supports PubSub models for selected industrial and distributed use cases. Security features include application authentication, user authentication, signing, encryption, and certificate-based trust models.

OPC UA is valuable when interoperability across industrial equipment, supervisory systems, manufacturing execution systems, and enterprise applications is required. Its information model is richer than a simple MQTT topic. The cost is greater implementation complexity and resource demand, although profiles and embedded implementations support a range of devices.

OPC UA should not be confused with an assumption that all industrial data can safely be exposed to a corporate network. Segmentation, security zones, gateways, least privilege, and change control remain essential.

9. Comparative Evaluation

Protocol selection should be evidence-based. Ask:

1. Are devices severely constrained in energy, memory, or bandwidth?
2. Is communication primarily telemetry, command, resource access, or workflow processing?
3. Is a broker acceptable, or is decentralized communication required?
4. What delivery, ordering, durability, deadline, and offline requirements exist?
5. Is browser or enterprise integration a priority?
6. Are typed semantics, industrial models, or certification requirements important?
7. How will identities, keys, updates, logs, and failures be managed for the system lifetime?

10. Applied Case: A Smart Manufacturing Cell

Background

The following is an illustrative case. A manufacturing cell contains vibration sensors, a programmable logic controller (PLC), a robotic arm, an operator dashboard, a maintenance analytics service, and an enterprise work-order system.

Problem

The plant needs millisecond-sensitive local coordination, continuous vibration telemetry, browser-based status display, industrial asset semantics, and reliable work-order integration. A single protocol would either be unsuitable for some devices or create unnecessary coupling.

Analysis

Local robot and PLC coordination requires predictable behavior and must continue if the cloud connection fails. Vibration sensors may be resource constrained and generate frequent measurements. The dashboard needs two-way updates. Maintenance analytics can tolerate seconds of delay but requires historical data and scalable ingestion. Work orders require durable enterprise processing and traceability.

Approach and Solution

• DDS is used for selected real-time, typed data exchanges within the cell, with deadlines and liveliness policies suited to local control.
• OPC UA provides an industrial information model for equipment state, variables, methods, and standardized access by supervisory systems.
• MQTT carries normalized telemetry from an edge gateway to the analytics platform. Topics are organized by plant, cell, asset, and data class, while payload schemas define units and timestamps.
• WebSocket connects the operator dashboard to a dashboard service. The browser does not receive unrestricted direct access to the plant network.
• AMQP carries maintenance events and work-order tasks to enterprise services, where queues, acknowledgements, retries, and dead-letter handling are useful.
• HTTP/REST provides administrative APIs for asset registration, historical queries, and user-facing configuration.

The edge gateway performs protocol translation, filtering, buffering, and policy enforcement. It does not blindly forward every control message to the cloud. Local control remains local, and cloud services receive only the data necessary for their functions.

Outcome

The architecture separates time-critical control from analytical and enterprise traffic. It permits different teams to use suitable interfaces while preserving governance at the gateway and platform layers. Failure tests must verify that the cell enters a safe state when DDS participants, the gateway, the broker, or the enterprise network becomes unavailable.

Learning Points

1. Protocols should be selected by interaction requirements, not popularity.
2. Gateways can reduce device complexity but become critical assets requiring redundancy and security.
3. Industrial interoperability requires information models, not merely transport connectivity.
4. Delivery guarantees must be connected to application behavior and failure handling.
5. A cloud connection must not be treated as a substitute for local safety control.

11. Security, Governance, and Operational Practice

Application protocols should be deployed with authenticated identities, encrypted communication where appropriate, authorization rules, certificate or key lifecycle management, and audit logging. TLS protects TCP-based protocols such as MQTT, HTTP, WebSocket, and AMQP; DTLS can protect UDP-based exchanges such as CoAP. Security configuration must also address certificate validation, trust anchors, renewal, clock accuracy, and device replacement.

Authorization should be expressed in terms of least privilege. A temperature sensor may publish to a measurement topic but should not subscribe to commands or write firmware settings. Administrative APIs should separate read, configuration, and destructive operations.

Message payloads should include schema version, timestamp, source identity, quality indicator, and unit where relevant. Payloads should be validated to prevent injection, oversized messages, invalid ranges, and resource exhaustion. Sensitive information should not be placed casually in topic names because topic metadata may appear in logs and access-control systems.

Operational design should include:

• bounded queues and back-pressure;
• retry limits and exponential backoff;
• dead-letter or quarantine handling;
• duplicate detection and idempotent processing;
• monitoring of connection counts, delivery latency, rejected messages, and queue depth;
• broker and gateway redundancy where justified;
• tested backup and recovery procedures;
• firmware and protocol-library update mechanisms; and
• clear ownership of schemas, topics, certificates, and retention policies.

Privacy and governance are also relevant. Telemetry can reveal occupancy, worker activity, production rates, or equipment behaviour. Data minimization, retention limits, access review, and regulatory obligations should be addressed during design rather than after deployment.

12. Contemporary Developments and Best Practices

Current IoT systems increasingly combine protocols rather than selecting one globally. Edge computing reduces latency and bandwidth use; gateways normalize data and isolate legacy systems; digital twins provide a structured representation of assets; and event-driven architectures connect telemetry to automated workflows.

Artificial intelligence is relevant when sensor streams feed anomaly detection, predictive maintenance, or adaptive control. AI does not remove the need for sound messaging design. Models require trustworthy timestamps, consistent units, known sampling conditions, data lineage, and handling of missing or duplicated messages. An automated control action generated by an AI service should pass through explicit authorization and safety constraints.

Best practice is to begin with communication contracts. Define the data model, interaction pattern, failure semantics, security roles, and operational limits before selecting a library. Test under packet loss, delay, broker restart, duplicate delivery, clock error, credential expiry, and partial network failure. Document which guarantees are protocol guarantees and which are application guarantees.

Chapter Summary

Application-layer protocols determine how IoT systems represent, route, secure, and act upon information. Request–response is suitable for direct resource access, while publish–subscribe efficiently distributes events to multiple consumers. Brokers decouple producers and consumers; topics classify information; queues buffer work; sessions preserve protocol state; and QoS expresses delivery or timing requirements.

MQTT provides simple brokered telemetry, while MQTT-SN adapts the model for constrained sensor networks and gateways. CoAP offers compact REST-like interaction for constrained devices. HTTP/REST provides broad web and enterprise compatibility, and WebSocket supports persistent two-way application communication. AMQP is suited to reliable enterprise messaging and workflows. DDS offers data-centric, policy-rich real-time distribution, while OPC UA provides industrial information modelling and secure interoperability.

No protocol guarantee is sufficient by itself: application semantics, idempotency, security, monitoring, and failure handling determine system behaviour. Effective architectures often combine protocols through controlled gateways, preserving local safety and matching each interface to its operational purpose.

Key Terms

Application layer: The layer that defines application messages, services, representations, and interactions.

Broker: An intermediary that receives, routes, stores, and delivers messages.

CoAP: A constrained, REST-like application protocol designed for IoT devices and networks.

DDS: A data-centric publish–subscribe technology with rich real-time QoS policies.

Device discovery: The process of finding devices, services, capabilities, and access information.

HTTP: The Web’s request–response application protocol.

Idempotency: A property whereby repeating an operation has the same intended effect as performing it once.

MQTT: A lightweight brokered publish–subscribe protocol.

MQTT-SN: An MQTT adaptation for sensor networks, including constrained and sleeping devices.

OPC UA: An industrial interoperability framework with a structured information model.

Payload: The application data carried by a message.

Publish–subscribe: An interaction model in which publishers send data and subscribers receive matching data.

QoS: Quality-of-service properties governing delivery, timing, ordering, durability, or related behaviour.

Queue: A storage and delivery mechanism for messages awaiting consumer processing.

REST: An architectural style that models application functionality as resources and representations.

Session: Protocol state associated with a client and its connection or subscriptions.

Topic: A logical name used to classify and route publish–subscribe messages.

WebSocket: A persistent, full-duplex communication protocol initiated through an HTTP handshake.

Review Questions

Short Answer Questions

1. What is the difference between request–response and publish–subscribe communication?
2. Define a broker and explain its role in MQTT.
3. What are the three MQTT QoS levels?
4. Why is device discovery important in IoT systems?
5. State one major strength and one limitation of HTTP/REST for IoT.

Long Answer Questions

1. Explain MQTT architecture, including clients, brokers, topics, sessions, retained messages, and QoS.
2. Compare CoAP and HTTP/REST for constrained IoT devices.
3. Explain how AMQP, DDS, and OPC UA address requirements beyond simple telemetry.
4. Discuss why “exactly once” message delivery does not necessarily guarantee exactly-once business action.
5. Describe the security and operational controls required for a production IoT messaging platform.

Analytical / Application Questions

1. Select protocols for a battery-powered environmental sensor, a browser dashboard, and a factory robot controller. Justify each selection.
2. Design a topic and payload strategy for a multi-site IoT deployment, including identity, units, timestamps, schema versioning, and authorization.
3. Analyse the manufacturing-cell case and identify two failure modes, their likely effects, and appropriate mitigations.

Multiple Choice Questions with Answers

Question 1

Which interaction model allows a publisher to send data without knowing the identities of its consumers?

A. Request–response

B. Publish–subscribe

C. Circuit switching

D. File transfer

Correct Answer: B. Publish–subscribe

Question 2

Which MQTT QoS level provides at-least-once delivery and may produce duplicates?

A. QoS 0

B. QoS 1

C. QoS 2

D. QoS -1

Correct Answer: B. QoS 1

Question 3

What is the primary purpose of an MQTT broker?

A. To manufacture sensor hardware

B. To compile payload schemas

C. To route messages between clients

D. To replace all network security

Correct Answer: C. To route messages between clients

Question 4

Which protocol is designed specifically for constrained REST-like communication and commonly uses UDP?

A. CoAP

B. AMQP

C. WebSocket

D. OPC UA

Correct Answer: A. CoAP

Question 5

Which technology is most directly associated with persistent full-duplex communication for browser applications?

A. MQTT-SN

B. WebSocket

C. CoAP Observe

D. AMQP queue

Correct Answer: B. WebSocket

Question 6

Which protocol is especially associated with enterprise queues, acknowledgements, and routed workflows?

A. AMQP

B. MQTT-SN

C. DDS

D. HTTP caching

Correct Answer: A. AMQP

Question 7

Which technology provides rich data-centric QoS policies for distributed real-time systems?

A. HTTP/REST

B. DDS

C. WebSocket

D. MQTT-SN

Correct Answer: B. DDS

Question 8

What is a major distinguishing feature of OPC UA?

A. It provides an industrial information model

B. It only supports unencrypted UDP

C. It cannot represent methods or metadata

D. It is limited to browser dashboards

Correct Answer: A. It provides an industrial information model

Question 9

Why can a command such as “increase speed by 10%” be unsafe to retry automatically?

A. It is always read-only

B. It is not necessarily idempotent

C. It cannot contain a payload

D. It requires no device identity

Correct Answer: B. It is not necessarily idempotent

Question 10

In the illustrative manufacturing case, which technology was selected for enterprise maintenance work-order messaging?

A. AMQP

B. MQTT-SN

C. DDS only

D. CoAP Observe

Correct Answer: A. AMQP