Real-Time Operating Systems and Concurrency

Introduction

Internet of Things devices increasingly perform several activities at once. A sensor node may sample temperature, process a communication protocol, store measurements in non-volatile memory, update a display, and respond to an alarm input. These activities compete for a processor, memory, peripherals, and communication channels. The central challenge is not merely making each activity work, but making the complete system respond within required time limits and remain correct when activities overlap.

A real-time operating system (RTOS) provides a structured way to organize such systems. It manages tasks, scheduling, inter-task communication, synchronization, timing, and resource ownership. Unlike a desktop operating system, an RTOS is designed primarily for predictable timing and controlled resource use. “Real-time” does not necessarily mean “very fast.” It means that correctness depends partly on when an operation completes.

This chapter develops the concepts required to design small embedded IoT systems using an RTOS. It introduces real-time requirements, RTOS architecture, tasks, scheduling, queues, semaphores, mutexes, timers, shared-resource protection, deadlocks, memory management, and deterministic behavior. Examples use generic RTOS terminology similar to that found in FreeRTOS, Zephyr, ThreadX, and other embedded systems, but exact APIs differ between implementations.

Learning Objectives

After studying this chapter, you should be able to:

• distinguish hard, firm, and soft real-time requirements;
• describe the major components of an RTOS;
• create and structure tasks using suitable priorities;
• select queues, semaphores, mutexes, event flags, and timers appropriately;
• analyze pre-emption, blocking, priority inversion, and response time;
• protect shared resources and avoid race conditions and deadlocks;
• apply safe memory-management and stack-management practices; and
• design a deterministic RTOS-based IoT application.

1. Real-Time Requirements

1.1 What makes a system real-time?

A real-time system must produce a logically correct result at an acceptable time. A temperature reading that is numerically accurate but arrives several minutes late may be useless for controlling a process. Conversely, a weather-monitoring device may tolerate occasional delays of several seconds.

Important timing terms include:

• Deadline: the latest acceptable completion time for an operation.
• Period: the interval between successive releases of a periodic activity.
• Response time: the elapsed time between an event becoming ready and the corresponding response completing.
• Execution time: processor time required by an activity when it runs without interruption.
• Jitter: variation in the timing of an event or response.
• Latency: delay between an event and the beginning of its service.
• Worst-case execution time (WCET): a conservative upper bound on execution time under specified conditions.

Real-time behavior is commonly classified as follows:

A useful basic relationship is that an activity is feasible only if its available processing demand does not exceed processor capacity. For a set of independent periodic tasks, a necessary utilization condition is:

Here, Cᵢ is the execution time of task i and Tᵢ is its period; both must use the same time unit, such as milliseconds. U is processor utilization. This condition is necessary but not sufficient for every scheduling algorithm because task priorities, blocking, release overhead, and deadlines also matter. Interrupt processing and operating-system overhead must be included in a realistic budget.

For fixed-priority response-time analysis, a commonly used model is:

The sum is taken over higher-priority tasks j that can interrupt task i. Bᵢ is the maximum blocking time caused by lower-priority tasks holding required resources. Rᵢ is solved iteratively because it appears on both sides. The task is schedulable under the stated model if Rᵢ ≤ Dᵢ, where Dᵢ is its deadline. This equation assumes a pre-emptive fixed-priority scheduler and requires a carefully defined task and blocking model; it should not be applied blindly to interrupt-driven or non-pre-emptive systems.

1.2 Determinism and predictability

A deterministic system has bounded and sufficiently predictable behavior. Determinism does not require every operation to take exactly the same time. It requires important delays, resource usage, and scheduling effects to have known bounds. Dynamic memory allocation, unbounded loops, lengthy critical sections, uncontrolled logging, and blocking network calls can make timing difficult to predict.

A practical design begins with requirements. For each activity, document its trigger, period or maximum arrival rate, WCET estimate, deadline, priority, shared resources, and acceptable failure response. Timing measurements should be taken under realistic worst-case conditions rather than under an idle laboratory workload.

2. RTOS Architecture

An RTOS normally consists of a kernel and supporting services. The kernel manages the processor and provides the primitives used by application software.

2.1 Kernel components

The principal components are:

1. Scheduler: selects the task that may execute.
2. Task or thread manager: creates, suspends, resumes, and deletes tasks.
3. Context-switch mechanism: saves the current task state and restores another task’s state.
4. Time base: maintains system ticks or another timing source.
5. Inter-task communication (ITC): provides queues, mailboxes, pipes, and notifications.
6. Synchronization services: provides semaphores, mutexes, event flags, and condition-like mechanisms.
7. Memory services: manages heaps, fixed-size pools, and task stacks.
8. Interrupt integration: permits interrupt service routines (ISRs) to signal or wake tasks.

A task usually has a control block containing its current state, priority, stack pointer, scheduling information, and sometimes accounting data. A context switch saves processor registers for the interrupted task and restores registers for the selected task. Context switching consumes time and may disturb cache or memory locality, so it should not be treated as free.

2.2 Task states

A task is commonly in one of these states:

• Running: currently using the processor.
• Ready: able to run but waiting for processor selection.
• Blocked: waiting for time to pass or for an event, queue item, semaphore, or notification.
• Suspended: deliberately removed from scheduling until resumed.
• Terminated: no longer executing; its resources may require reclamation.

Blocking is generally preferable to busy-waiting. A blocked task consumes little or no processor time, whereas a polling loop repeatedly checks a condition and can waste energy and reduce responsiveness.

2.3 Scheduling policies

A pre-emptive priority scheduler runs the highest-priority ready task. If a higher-priority task becomes ready, it can interrupt a lower-priority task. In a cooperative scheduler, a running task continues until it yields, blocks, or completes a scheduling point. Some RTOSs combine priority scheduling with time slicing among equal-priority tasks.

Priority should represent urgency and timing impact, not software status or personal preference. A useful hierarchy might place emergency handling above control computation, control above sensor acquisition, sensor acquisition above data processing, and bulk telemetry below these activities. The exact order depends on deadlines and blocking relationships.

Priority alone cannot solve timing problems. A high-priority task that performs lengthy computation or waits on a poorly protected resource can still cause failure. Tasks should perform bounded work, block while waiting, and delegate lengthy non-critical operations to lower-priority workers.

3. Tasks and Timing Services

3.1 Task design

A task is an independently schedulable execution path. A robust task normally follows this pattern:

1. initialize local state;
2. wait for a release event, timer period, or message;
3. perform a bounded unit of work;
4. communicate results through an RTOS object; and
5. return to a blocked state.

A periodic task should use an absolute release schedule where the RTOS supports it. For example, a task can record its previous release time and delay until the next release rather than simply delaying after its work. Relative delays accumulate execution-time variation and cause drift.

An RTOS tick frequency determines the resolution of tick-based delays. If the tick period is 1 ms, a tick delay cannot generally express a 0.4 ms interval. A higher tick rate improves resolution but increases timer-interrupt overhead. Hardware timers or high-resolution timers may be required for very precise events, although their use must be balanced against power and complexity.

3.2 Software timers

A software timer causes a callback or notification after a specified interval. A one-shot timer expires once; an auto-reload timer repeats. Software timers are useful for timeouts, periodic housekeeping, communication retries, and sensor polling. They are not always appropriate for hard real-time control because timer callbacks may execute in a shared timer-service task and may experience queue or scheduling delays.

Timer callbacks should be short and non-blocking. A callback should normally signal a task rather than perform lengthy sensor access, network transmission, or file operations directly.

3.3 Interrupts and deferred processing

An ISR should perform only urgent, bounded work: capture hardware state, clear the interrupt condition, store a small data item, and notify a task. It should not call ordinary blocking RTOS functions or perform complex protocol processing unless the RTOS explicitly provides an ISR-safe variant.

The common pattern is interrupt-to-task handoff. The ISR places data in a buffer or gives a semaphore, then a task performs the substantial processing at an appropriate priority. This reduces interrupt latency and simplifies synchronization.

4. Inter-Task Communication and Synchronization

Different RTOS objects solve different problems. Selecting the wrong object often produces fragile designs.

4.1 Queues

A queue is a bounded first-in, first-out storage object. A producer places data into the queue; a consumer removes it. A producer may block when the queue is full, and a consumer may block when it is empty. Queue length is a design parameter, not a guarantee that data cannot be lost. If the arrival rate persistently exceeds the service rate, every finite queue eventually fills.

Queue messages should be compact. Passing a complete structure copies data and may be safe but expensive. Passing a pointer reduces copying but requires clear ownership rules: the receiver must know when it may modify or release the referenced object. A fixed-size memory pool is often safer than unrestricted dynamic allocation for pointer-based messages.

4.2 Semaphores

A binary semaphore has two logical states: available and unavailable. It is suitable for signaling that an event has occurred. For example, an ADC ISR may give a semaphore, and an acquisition task may take it.

A counting semaphore stores a non-negative count. Taking it decrements the count; giving it increments the count, subject to the implementation’s maximum. It can represent multiple available buffers, resource units, or accumulated events. A semaphore does not normally identify an owner, so it is not the best tool for protecting a critical section.

4.3 Mutexes

A mutex provides mutual exclusion: at most one task owns a protected resource at a time. A correct pattern is:

1. take the mutex with a bounded timeout;
2. access the shared resource;
3. release the mutex on every normal and error path.

Many RTOS mutexes implement priority inheritance. If a high-priority task blocks on a mutex owned by a low-priority task, the owner temporarily inherits the higher priority, reducing interference from medium-priority tasks. Priority inheritance limits a common form of priority inversion but does not eliminate deadlocks or excessive critical-section duration.

A mutex must not usually be taken in an ISR because ISRs cannot block and do not behave as task owners. ISR-to-task signaling should use ISR-safe queues, semaphores, or notifications.

4.4 Event flags and notifications

Event flags represent conditions such as “network connected,” “new configuration available,” or “fault detected.” A task may wait for any flag or for all flags, depending on the RTOS. Direct task notifications are often more memory-efficient than queues or semaphores when one task is the intended recipient.

5. Concurrency Hazards

Concurrency errors arise when the result depends on the order in which tasks execute. A race condition occurs when multiple execution paths access shared state and at least one modifies it without adequate synchronization. An operation such as incrementing a counter is not necessarily atomic: it may involve read, modify, and write instructions. A context switch between these steps can lose an update.

A critical section temporarily prevents interference while a short operation executes. Disabling interrupts can protect data shared with an ISR, but it increases interrupt latency and should be limited to a few instructions when possible. Suspending the scheduler prevents task switches but may not prevent interrupts. Mutexes are preferable for longer task-to-task resource access because they allow unrelated high-priority tasks to run.

5.1 Priority inversion

Priority inversion occurs when a high-priority task is indirectly delayed by a low-priority task holding a required resource. A medium-priority task can worsen the delay by pre-empting the low-priority owner. Use priority inheritance, priority-ceiling protocols where supported, short critical sections, and careful resource partitioning.

5.2 Deadlock

Deadlock is a state in which tasks wait indefinitely for resources held by one another. Four conditions traditionally permit deadlock:

• mutual exclusion;
• hold and wait;
• no pre-emption of a held resource; and
• circular wait.

A practical prevention method is global lock ordering. If a task may need both an I²C mutex and a file-system mutex, every task must acquire them in the same order. Other techniques include avoiding nested locks, using bounded timeouts, releasing resources before waiting for unrelated events, and designing ownership-transfer protocols.

A timeout is useful for fault recovery, but it is not by itself proof of correctness. When a timeout occurs, the system should record diagnostics, release what it safely owns, and move to a defined degraded or recovery state.

6. Memory and Resource Management

Memory errors are particularly serious in long-running IoT devices. A task stack must hold local variables, saved registers, nested call frames, and sometimes interrupt-related context. Stack overflow can corrupt neighboring data and produce failures that appear unrelated to the original cause.

Use stack-watermark or high-water-mark instrumentation during testing. Allocate stack sizes from measured worst-case call depth plus a safety margin. Avoid large automatic arrays in task functions; use static buffers or controlled pools when appropriate.

Dynamic allocation is convenient during initialization but can suffer from fragmentation and unpredictable allocation time during continuous operation. A common embedded policy is:

• allocate tasks, queues, and fixed buffers during startup;
• use fixed-size pools for runtime messages and packets;
• avoid allocation in ISRs and hard real-time paths; and
• define ownership and release responsibility for every buffer.

Resource budgets should include RAM for stacks, queue storage, control blocks, network buffers, and worst-case temporary objects. A design that fits in nominal memory may still fail if simultaneous peak usage is not considered.

7. Worked Scheduling Example

Consider three independent periodic tasks on one processor:

The processor utilization is:

Thus, the tasks require 26% of the processor under the stated execution estimates. This leaves nominal capacity for idle time, interrupts, kernel overhead, and execution-time variation. It does not prove that all deadlines will be met. For example, a long non-pre-emptive section, a blocking mutex, or an underestimated WCET could still cause a deadline miss.

Suppose the sensor task has the highest priority and the control task has the next highest priority. If telemetry holds a shared logging mutex for 8 ms, a control task needing that mutex could be blocked for up to 8 ms, even though its own computation is only 2 ms. The remedy is to reduce the protected region, copy data out of the shared resource quickly, use a dedicated logging task, or redesign the ownership model.

8. Applied Case: Environmental Monitoring Gateway

Background

This illustrative case concerns a battery-powered gateway that measures temperature, humidity, and air quality, stores readings locally, and transmits summaries over a wireless link. It must sample sensors every 1 second, detect an over-temperature condition within 100 ms, and upload data when connectivity is available.

Problem

An initial prototype used one loop that polled sensors, wrote a flash record, and transmitted data sequentially. Wireless retries sometimes delayed sensor sampling by several seconds. Flash writes also blocked alarm processing. Increasing the loop speed did not solve the problem because network operations had unpredictable duration.

Analysis

The requirements have different timing classes. Alarm monitoring is the most urgent activity. Sensor sampling is periodic and moderately time-sensitive. Flash storage is important but can be buffered. Wireless upload is soft real-time and may be delayed when the network is unavailable.

The prototype also had a shared I²C bus. Sensor drivers and a display task accessed it without mutual exclusion, producing occasional corrupted transactions. Finally, a dynamically allocated message was sometimes freed by both the producer and consumer, causing memory corruption.

Approach and solution

The redesigned system uses:

• a high-priority alarm task released by a GPIO interrupt or periodic safety check;
• a sensor task with a one-second absolute-period release;
• a storage task receiving measurement records through a bounded queue;
• a communication task receiving summarized records through another queue;
• an I²C mutex protecting bus transactions;
• a binary semaphore for sensor-conversion completion;
• a software timer for connection retry; and
• a fixed-size message pool with explicit producer-to-consumer ownership transfer.

The alarm ISR records the input state and notifies the alarm task. The task then disables or commands the relevant actuator within a bounded path. It does not write flash or contact the network. The sensor task starts a conversion, blocks on the completion semaphore, reads the result under the I²C mutex, and sends a value object to the storage queue. The storage task performs the slower flash operation. The communication task transmits only when connected and returns failed messages to a controlled retry queue.

The I²C mutex is never held while waiting for a sensor conversion or network response. All tasks acquire resources in the defined order, and queue-send operations use bounded timeouts. If the storage queue is full, the policy is to retain the newest alarm-related record and discard or compress older routine telemetry according to the application requirement.

Outcome

The redesign separates urgent control from slow operations. Sampling timing becomes measurable, wireless delays no longer block alarm processing, and bus transactions are serialized. Instrumentation records task execution time, queue occupancy, mutex wait time, missed deadlines, stack high-water marks, and watchdog resets. These measurements allow the team to compare actual behavior with the timing budget.

Learning points

The case demonstrates that concurrency is a design problem, not merely a software feature. Tasks should be separated by timing and responsibility; communication should be explicit; shared resources require ownership rules; and slow or unreliable services should not execute in urgent paths.

9. Best Practices and Limitations

A reliable RTOS design should:

• define deadlines and failure behavior before assigning priorities;
• keep ISRs short and defer work to tasks;
• block rather than poll whenever practical;
• use queues for data and mutexes for resource ownership;
• keep critical sections and mutex-held regions short;
• use priority inheritance or a priority-ceiling strategy where appropriate;
• establish and document a global lock order;
• avoid unbounded loops and unbounded blocking in critical tasks;
• measure WCET, response time, stack use, queue occupancy, and jitter;
• use watchdogs as recovery mechanisms, not as substitutes for analysis; and
• test overload, communication loss, sensor failure, power interruption, and simultaneous events.

An RTOS does not automatically make an application real-time. Poor priority assignment, inappropriate blocking, memory fragmentation, excessive logging, and inaccurate execution estimates can defeat the kernel’s mechanisms. Certification-oriented systems may require traceability, formal scheduling analysis, restricted language features, and independently verified software components.

Security is also relevant. A compromised network task must not be able to corrupt control data or exhaust all queues. Apply least privilege where supported, validate message lengths, isolate communication buffers, authenticate firmware, and define safe behavior when updates or credentials are invalid. For safety-related products, applicable national and industry standards must be identified by the responsible engineering organization; an RTOS feature alone is not a certification.

10. Contemporary Developments

Modern IoT RTOSs increasingly support multicore microcontrollers, memory protection, low-power tickless operation, secure boot, Bluetooth and wireless stacks, and container-like isolation at a small scale. Multicore systems introduce additional concerns: tasks may migrate between cores, shared caches affect timing, and mutual exclusion must protect data across cores. A single-core scheduling analysis cannot simply be reused for a multicore system.

Energy-aware scheduling is important for battery devices. Tickless idle suppresses periodic timer interrupts during long idle intervals, reducing energy use, but wake-up latency and timer accuracy must be considered. Edge artificial intelligence can also be hosted in RTOS tasks, but model inference must be bounded, memory use must be measured, and lower-priority analytics must not interfere with control functions.

Tracing tools and runtime verification increasingly record task switches, interrupt latency, queue states, and resource contention. These tools improve evidence-based optimization, but tracing itself consumes memory and processor time. Measurements should therefore be interpreted with instrumentation overhead in mind.

Chapter Summary

Real-time embedded systems must satisfy both functional and timing requirements. An RTOS supplies scheduling, task management, timing, communication, synchronization, and memory services, but predictable behavior still depends on sound application design. Tasks should have clear responsibilities, bounded work, and appropriate priorities. Queues transfer data, semaphores signal events or count resources, mutexes protect owned shared resources, event flags represent conditions, and timers support periodic or delayed actions.

Concurrency creates race conditions, priority inversion, and deadlock risks. These risks are controlled through suitable synchronization, short critical sections, priority inheritance, lock ordering, bounded waits, explicit memory ownership, and measured resource budgets. Utilization and response-time equations are useful analysis tools, but their assumptions must match the actual system. A successful IoT design separates urgent control from slow communication and storage, uses ISR-to-task handoff, and verifies behavior under overload and failure conditions.

Key Terms

Real-time system: A system whose correctness depends partly on producing results within specified timing constraints.

Deadline: Latest acceptable completion time for an operation.

Period: Time interval between releases of a periodic task.

Jitter: Variation in the timing of an event or response.

Latency: Delay between an event and the beginning of its service.

WCET: Conservative upper bound on the execution time of an operation.

RTOS: Operating system designed to provide controlled and predictable timing behavior.

Task: Independently schedulable execution path managed by the RTOS.

Pre-emption: Interrupting a running task to execute a more urgent ready task.

Queue: Bounded FIFO object used to transfer data between execution contexts.

Semaphore: Synchronization object used to signal events or count resources.

Mutex: Mutual-exclusion object with ownership, used to protect shared resources.

Priority inversion: Delay of a high-priority task because a lower-priority task holds a required resource.

Race condition: Incorrect behavior caused by uncontrolled timing among concurrent accesses.

Critical section: Short code region protected from interfering execution.

Deadlock: Indefinite waiting caused by cyclic resource dependence.

Priority inheritance: Protocol that temporarily raises a resource owner’s priority.

Software timer: Kernel-managed timer that triggers a callback or notification.

Determinism: Predictability and boundedness of important system behavior.

Memory pool: Fixed collection of reusable memory blocks with controlled allocation behavior.

Review Questions

Short Answer Questions

1. Distinguish hard real-time from soft real-time requirements.
2. What is the difference between a mutex and a binary semaphore?
3. Why should lengthy processing normally be deferred from an ISR to a task?
4. Define priority inversion.
5. State two causes of non-deterministic behavior in an embedded application.

Long Answer Questions

1. Explain the principal components of an RTOS and describe how they cooperate during a context switch.
2. Compare queues, counting semaphores, mutexes, event flags, and direct notifications, giving one suitable IoT use for each.
3. Explain the four conditions for deadlock and discuss practical prevention methods.
4. Describe how task priorities should be assigned in a mixed sensor, control, storage, and communication application.
5. Discuss the advantages and risks of dynamic memory allocation in a long-running IoT device.

Analytical / Application Questions

1. An IoT gateway has a 1 ms sensor task every 10 ms, a 2 ms control task every 20 ms, and a 3 ms telemetry task every 50 ms. Calculate processor utilization and explain why the result alone does not prove schedulability.
2. Design an RTOS communication structure for a sensor node in which an ISR signals completed conversions, a task processes samples, and another task uploads results over an unreliable network. Identify the required RTOS objects and ownership rules.
3. A high-priority control task waits for a mutex held by a low-priority logging task while a medium-priority computation task continues running. Analyze the scheduling problem and propose at least three design improvements.

Multiple Choice Questions with Answers

Question 1

Which statement best describes a hard real-time system?

A. Every operation must execute in exactly the same time

B. A missed deadline can constitute system failure or an unacceptable hazard

C. Deadlines are optional if the final result is accurate

D. The system must always use the fastest available processor

Correct Answer: B. A missed deadline can constitute system failure or an unacceptable hazard

Question 2

What is the primary purpose of a queue in an RTOS?

A. To protect a shared peripheral through ownership

B. To disable interrupts during data transfer

C. To transfer bounded data or messages between execution contexts

D. To increase the processor clock frequency

Correct Answer: C. To transfer bounded data or messages between execution contexts

Question 3

Which RTOS object is normally most appropriate for protecting a shared I²C bus between tasks?

A. Mutex

B. Counting semaphore

C. Periodic timer

D. Event flag

Correct Answer: A. Mutex

Question 4

Why should an ISR generally defer substantial processing to a task?

A. Tasks cannot be pre-empted

B. ISRs should remain short to limit interrupt latency and simplify timing analysis

C. Tasks always execute faster than ISRs

D. The RTOS cannot communicate with interrupt handlers

Correct Answer: B. ISRs should remain short to limit interrupt latency and simplify timing analysis

Question 5

For task utilizations of 1/10, 2/20, and 3/50, what is the total utilization?

A. 0.16

B. 0.20

C. 0.26

D. 0.36

Correct Answer: C. 0.26

Question 6

What is priority inversion?

A. A low-priority task permanently changes its priority by choice

B. A high-priority task is delayed because a lower-priority task holds a needed resource

C. All tasks are assigned the same priority

D. The scheduler runs tasks in reverse creation order

Correct Answer: B. A high-priority task is delayed because a lower-priority task holds a needed resource

Question 7

Which practice directly helps prevent deadlock when multiple mutexes may be needed?

A. Acquiring mutexes in a globally consistent order

B. Allowing every task to acquire locks in any order

C. Disabling all interrupts permanently

D. Replacing every mutex with a busy-wait loop

Correct Answer: A. Acquiring mutexes in a globally consistent order

Question 8

What is a major risk of unrestricted dynamic allocation during continuous runtime?

A. It prevents all context switches

B. It can cause fragmentation and make allocation timing unpredictable

C. It guarantees stack overflow

D. It eliminates the need for ownership rules

Correct Answer: B. It can cause fragmentation and make allocation timing unpredictable

Question 9

Which statement about priority inheritance is correct?

A. It prevents every possible deadlock

B. It temporarily raises the priority of a resource-owning task to reduce priority inversion

C. It removes the need for mutexes

D. It applies only to hardware interrupts

Correct Answer: B. It temporarily raises the priority of a resource-owning task to reduce priority inversion

Question 10

A software timer callback must perform a lengthy network transmission. What is generally the better design?

A. Perform the transmission while disabling interrupts

B. Ignore timer expiration until the transmission completes

C. Have the callback notify a task that performs the transmission

D. Call the blocking transmission function from the ISR

Correct Answer: C. Have the callback notify a task that performs the transmission