Embedded systems
Designing RTOS Tasks That Survive Real-World Faults
A practitioner's guide to structuring RTOS tasks so deadlines, priorities and fault recovery hold up under real-world conditions, with methods to verify timing and watchdog behaviour.

Why task design is a reliability decision
An RTOS gives you threads, priorities and a scheduler, but it does not give you a reliable system. Reliability comes from how tasks are structured, how they share resources, and what happens when something blocks longer than expected. Most field failures in embedded products are not exotic silicon faults; they are ordinary design mistakes such as priority inversion, unbounded blocking, stack overflow, or a watchdog that resets the device without leaving useful evidence.
This article focuses on one specific problem: designing RTOS tasks so that deadlines are met under normal and degraded conditions, and so that faults lead to a controlled recovery rather than a silent hang. The methods apply to FreeRTOS, Zephyr, ThreadX and similar kernels; the exact APIs differ, but the reasoning does not. This is educational guidance and does not replace a site-specific assessment for safety-relevant systems.
Start from timing requirements, not from tasks
A common failure mode is starting with a list of tasks and then hoping the timing works out. Instead, derive timing requirements first. For each function, ask how often it must run, how long it may take, and what happens if it is late.
Typical categories are periodic sampling, event-driven processing, communication, housekeeping and diagnostics. A sensor loop might need a fixed period, while a command handler only needs to respond within a bounded latency. Writing these down as a simple budget, with worst-case execution time estimates and jitter allowances, exposes conflicts early. If two functions both claim the same CPU budget, you have found a design problem before writing code.
Rules of thumb such as keeping interrupt handlers short and deferring work to tasks are useful, but the actual limits are system-dependent and must be measured on your target.
Priorities, blocking and the inversion trap
Priority assignment should reflect urgency, not importance in a product sense. A task that must respond within a few milliseconds needs a higher priority than a logging task, even if logging feels more important to the business.
- Keep the number of distinct priorities small; too many levels make reasoning difficult.
- Avoid sharing a mutex between a high-priority and a low-priority task without a priority-inheritance or ceiling protocol.
- Prefer message queues and event flags over shared globals, so blocking is explicit.
- Never call a blocking API from an interrupt context.
- Bound every wait with a timeout, even if you believe it will never expire.
Priority inversion occurs when a high-priority task waits on a resource held by a low-priority task, which is itself preempted by a medium-priority task. The classic Mars Pathfinder anomaly is a well-documented public example. Kernels that support priority inheritance mitigate this, but the mitigation only works if you actually use the provided mutex type rather than a binary semaphore as a lock.
Designing for fault recovery, not just fault detection
Detection without recovery is incomplete. A watchdog that resets the device is a blunt instrument; if the reset reason is not recorded, you learn nothing and may repeat the fault indefinitely.
A practical pattern is a supervisor task that monitors heartbeats from critical tasks. Each critical task updates a counter or timestamp at a known point in its loop. The supervisor checks these within a window and, if one is stale, attempts a staged response: first log the event with context, then try to restart the affected subsystem, and only then trigger a full reset.
/* Illustrative pattern, not kernel-specific */
void critical_task(void *arg) {
for (;;) {
heartbeat_update(TASK_ID_SENSOR);
if (sensor_read(&sample) != OK) {
fault_report(FAULT_SENSOR_READ);
recover_sensor();
}
process(sample);
vTaskDelayUntil(&last, PERIOD_TICKS);
}
}Two details matter. First, the heartbeat must be updated at a point that proves the task is making progress, not merely scheduled. Second, the recovery path must itself be bounded; a recovery routine that blocks forever is just a second failure mode.
Verifying timing and stack behaviour on target
Simulation and desk analysis are useful, but RTOS behaviour must be verified on the real hardware with the real interrupt load. Several practical techniques help.
- Toggle a GPIO pin at task entry and exit and observe it on an oscilloscope or logic analyser to measure period and jitter.
- Use the kernel's runtime statistics or a trace facility to see actual execution time per task.
- Fill task stacks with a known pattern at startup and check the high-water mark after soak testing.
- Inject delays and errors deliberately, for example by stubbing a sensor read to fail, and confirm the recovery path behaves as intended.
- Run a long soak test with logging enabled and disabled, because logging changes timing and can mask or create problems.
Be sceptical of single measurements. Jitter and worst-case latency appear under specific conditions, often when interrupts coincide or when a flash write blocks execution. Verification means reproducing those conditions, not avoiding them.
Common failure modes to design against
- Unbounded blocking: a task waits forever on a queue or semaphore and never recovers.
- Stack overflow: deep call chains or large local buffers corrupt adjacent memory; enable stack checking during development.
- Priority inversion: shared resources without inheritance or ceiling protocols.
- Watchdog starvation: a task kicks the watchdog from a low-priority context, so a hung high-priority task goes unnoticed.
- Tickless or low-power modes: timing assumptions break when the kernel suppresses ticks; verify wake-up latency.
- Shared state without synchronisation: intermittent corruption that is hard to reproduce.
- Reset loops: the device resets, restarts, and fails again because the root cause was never captured.
Each of these has a design response. The point is to choose the response deliberately rather than discovering it in the field.
A practical checklist
- Define timing requirements per function before assigning tasks.
- Keep interrupt handlers short and defer work to tasks.
- Assign priorities by urgency and keep the set small.
- Use priority-inheritance or ceiling mutexes for shared resources.
- Bound every blocking call with a timeout and handle the timeout path.
- Implement heartbeats at points that prove progress.
- Design a staged recovery path and bound the recovery itself.
- Record reset reason and fault context in non-volatile memory.
- Measure period, jitter and stack high-water marks on target.
- Inject faults deliberately and confirm recovery behaviour.
- Soak test with realistic interrupt and communication load.
- Review the design against the failure modes above before release.
Task design is where scheduling theory meets field reality. Treating it as a reliability discipline, with requirements, verification and recovery built in, is what separates a demo from a device that keeps working.
Explore Rootcastle engineering tools → · Public profiles and publications

