Multicore debugging challenges in Zephyr RTOS: Part 3 – Inter-core messaging
By Bea Ben-Ali
Introduction
In modern embedded systems, especially those with multiple cores, it’s common to divide tasks between different cores or threads to make the system faster, more responsive, and energy efficient.
For this to work well, the cores need a safe way to share data and stay in sync with each other. That’s where Inter-Process Communication (IPC) comes in. It allows different processes to exchange information and work together smoothly. IPC can be achieved through two main methods: shared memory and message passing, both of which are supported by the operating system to ensure efficient and reliable communication between processes.
In parts 1 and 2 of this blog series, we tackled debugging race conditions and cache coherency issues to ensure synchronization across cores and improve system reliability; both of which are part of the “shared memory model” of IPC. Now, let’s dive into another primary IPC mechanism: “message passing.” Message queues are another exciting debugging challenge to overcome.
(Editor’s note: For the purpose of this blog, we will focus on symmetric multiprocessing (SMP) systems rather than asymmetric multiprocessing (AMP) systems. Implementing message queues in SMP is much less complex. Our goal is to provide an introduction to inter-core messaging and explain how it works.)
Debugging inter-core messaging issues
Let’s take a quick look at how two cores exchange information using a message queue:
Here, Core 0 runs a producer task and uses k_msgq_put(). Core 1 runs a consumer task and retrieves messages via k_msgq_get().
Messages can be missed or corrupted if the receiving core is busy, interrupts occur, or the queue fills up. These issues lead to lost data, incorrect messages, or queue overflows.
Overcoming inter-core messaging issues on Zephyr RTOS systems with SystemView and Ozone
As in earlier parts of this series, we instrument firmware using SystemView to log events and inspect message flow across cores.
Set up Zephyr Configuration system (Kconfig)
To enable real-time debugging and performance monitoring in a multicore Zephyr setup, several key Kconfig options must first be configured:
CONFIG_SMP=y
These settings create a powerful environment for debugging multicore applications.
Integrate SystemView
Zephyr’s message queue API allows cores or threads to safely send/receive messages. We connect SystemView to trace queue behavior.
Code snippet:
#include <zephyr/kernel.h>
The K_MSGQ_DEFINE macro creates the message queue at compile time, defining its name, message size, max messages, and alignment.
Alternatively, you can initialize it manually:
char msg[MSGQ_MAX_MSGS * MSG_SIZE];
Event IDs for SystemView:
#define EVENT_ID_MSG_SENT 0x1001
Sender thread:
void core0_thread(void)
Receiver thread:
void core1_thread(void)
Queue-full instrumentation:
if (k_msgq_put(&devheads_msgq, &msg, K_NO_WAIT) != 0) {
#
SEGGER_SYSVIEW_Warn("Message dropped!");
#
}
Continuing the debugging task using Ozone
To catch queue overflow issues, we add conditional logic:
int ret = k_msgq_put(&devheads_msgq, &msg, K_NO_WAIT);
#
SEGGER_SYSVIEW_PrintfHost("Queue put failed: error %d", ret);
#
}
Editor’s Note:
Using Ozone watchpoints, you can monitor queue length and message values over time to detect latency, drops, or unexpected stalls. This gives deeper visibility into system behavior.
Closing
This concludes our debugging series. We explored practical challenges in multicore embedded systems using Zephyr RTOS and demonstrated how tools like SEGGER SystemView and Ozone enhance understanding of system behavior.
We hope you enjoyed the series — stay tuned for more hands-on tutorials and insights!