Real-Time Performance Optimization in HMI Design & Development for Embedded Systems

When it comes to embedded devices and real-time performance, for many there's no room for compromise. Whether the human-machine interface (HMI) is designed for an automotive dashboard, a medical device, or an industrial control panel, it must provide instantaneous feedback without any noticeable delay. As industries continue to push for faster, more responsive systems, real-time HMI performance becomes a key benchmark in the optimization efforts of user interface (UI) development software engineers and designers.

This article probes the challenges and the best practices in achieving real-time performance in HMI design and development, especially for the needs of embedded systems, and highlights how UI development tools, such as Crank’s Storyboard, can play an important role in that optimization.

Why Real-Time Performance Matters in HMI Design?

With real-time HMI performance, responsiveness is not just a feature but sometimes is directly linked to user safety and performance in operation. For example, with automotive displays, the system should respond to user inputs, such as pressing a button, within 100 milliseconds (0.1 seconds) to ensure a safe and seamless user experience. In the case of devices designed for healthcare, delays in HMI performance can have serious consequences for the patient. Even a brief delay can increase the risk of an operator error leading to potentially critical consequences.

Therefore, it is essential for embedded HMI designers to ensure their designs are not only aesthetically pleasing and functional but can also optimized for real-time interaction.

Key Drivers of Real-Time Performance

• User Experience: Immediate feedback is essential to deliver good interaction from the user end.
• Operational Safety: In mission-critical applications, reliability is mostly dependent on real-time performance.
• System Efficiency: A well-optimized HMI does not put much stress on the processor, thus making it a better system overall.

HMI Development Challenges for Real-Time Performance

Suboptimal use of memory resources is a known cause of latency and increase in response time in embedded systems^[1][2], especially in resource-constrained environments like HMI applications. In resource-constrained environments, response times can be significantly improved by reducing or minimizing wait states and applying effective caching and memory optimization techniques. Therefore, simply optimizing memory usage in HMI applications designed for microcontrollers can greatly enhance overall responsiveness and improve the user experience.

The example above is just one of many situations where an HMI in an embedded system may struggle to deliver true real-time responsiveness. Some of the key challenges include:

Latency in Rendering: Real-time HMI systems need to render graphical elements without significant delays. However, complex UIs place a heavy load on the processor. High-quality visuals, dynamic content, and real-time data demands can increase this load. If the CPU (and possible GPU) lack sufficient power, the UI can suffer from frame drops or stuttering, making interactions with the system more difficult.

Data Handling: In real-time systems, HMI interfaces continuously communicate with external hardware, such as sensors, actuators, and control units. The HMI must then process the data, react to it, and display results without delay. However, slow data retrievals or inefficient memory management can result in bottlenecks that negatively impact the response time of the information displayed on the screen.

Synchronization with Real-Time Systems: Many applications consist of larger real-time control systems in which HMIs are embedded into, such as manufacturing robots or HVAC controllers. Even minor asynchrony between the interface with backend processes can greatly impact system performance. In some cases, this asynchrony may not only delay the performance but also disrupt the expected behavior of the control system.

Processor and Memory Constraints: Embedded HMI systems deployed on low power MCUs that limited computational power and memory. As a result, even simple graphical user interfaces (GUIs) may require multiple cycles to render. Additionally, the desire to recreate smartphone-like interfaces, with complex animations and high-resolution images, places further strain on the embedded processor, resulting in subpar HMI experiences.

Optimizing Real-Time Performance in HMI Development

To conquer real-time performance issues, HMI development engineers must implement various optimization techniques. The techniques primarily aim to reduce latency, enhance system responsiveness, and maximize the effective utilization of hardware resources.

Effective Rendering Techniques

Automobile companies like Mercedes Benz utilize GPU-accelerated interfaces to render their sophisticated dashboards. By offloading rendering tasks to the GPU, they achieve smooth real-time feedback on their touchscreens with minimal latency. This approach prevents the UI rendering from burdening the central processing system, allowing it to focus on the critical driving tasks.

Optimization for rendering begins with the rendering pipeline in real-time HMIs and any pixel counts. Below are some of the techniques that can help minimize the processing overhead.

GPU Acceleration: GPUs are designed for parallel processing and excel at complex graphical rendering. By offloading the rendering tasks to the GPU, the processor can focus on real-time operations. In HMI applications, tasks such as vector-based graphics rendering, image scaling, and anti-aliasing can be offloaded to the GPU, allowing the CPU to focus on time-critical data processing.

Preloading Critical Assets: Commonly used graphical assets, like icons, fonts, and UI components, can be preloaded into the program. This ensures they are readily available when needed, avoiding delays when launching the application. Preloading reduces the need for dynamic loading during operation, which can be time-consuming. The end-result is smoother interactions and faster rendering.

Choosing Graphics Format: Using vector-based graphics (SVGs) in HMIs allows visual elements to scale without losing performance, making it easier to maintain high-quality, responsive UIs on devices with different resolutions. SVG images are already in a format that is understandable by rendering engines, eliminating the need for a separate decoding step.

Other formats, such as JPG, PNG, and BMP, have their own set of advantages and disadvantages when it comes to memory storage and runtime performance. While Raw file formats can be displayed directly, encoded formats require system resources for decoding before rendering.

This video can help guide you in making these decisions.

Latency Reduction in User Interaction

Input response latency can significantly impact delays. For example, in industrial control panels a latency of just 200 milliseconds (.02 seconds) in factory automation can lead to significant downtime and increased production costs. Below are some strategies in embedded HMI development to help reduce latency.

Event-Driven Architecture (EDA): Unlike traditional linear control flows, EDA lets HMI applications react in real time to events as they arise. This architecture allows for flexible handling to user interactions and system events while minimizing CPU overhead. By eliminating unnecessary polling for buffering, EDA can help ensure a more instantaneous response to user inputs.

Touch Optimization: Touch-based HMIs must respond to touch events without delay. It can be achieved by adjusting the touch sensitivity settings and implementing noise cancellation touch interfaces for environments with high electrical noise interference.

Additionally, using event prediction algorithms that will predict when a user touches the screen can be instrumental in reducing latency and improving user experiences. These algorithms predict user actions based on previous interactions, allowing the system to pre-render elements or pre-emptively load necessary assets in advance, thereby minimizing response times.

Power Management: Power management can influence system latency by changing clock speed, voltage, or powering peripheral components on and off.  Effective power management can significantly enhance HMI performance by reducing latency, extending battery life, and lowering heat dissipation. Techniques such as Dynamic Voltage and Frequency Scaling (DVFS), sleep modes, and adaptive power control (APC) can achieve an optimal balance between performance and power consumption.

Data Handling and Real-Time Synchronization

A real-time control system ensures that the HMI remains in sync with the data stored in the backend. For instance, the Boeing 787 Dreamliner cockpit utilizes a real-time HMI interaction system connected to thousands of sensors spread throughout the aircraft. Boeing engineers utilized the VxWorks RTOS to ensure that the interface remains in sync with the critical aircraft data in real time.

Optimizing data flow and real-time synchronization is critical, and the following measures can be implemented to achieve this.

Data Caching: Minimize the amount of data requests from external systems by caching data that is most often accessed. This can significantly reduce the amount of data requests and improve response times.

Efficient Data Parsing: Efficient data parsing is crucial in real-time HMI systems to minimize latency when processing input from sensors or external systems. Optimize data parsing by using lightweight protocols that facilitate the smooth handling of data and help avoid complicated parsing functions.

Synchronization with Real-Time Systems: A key requirement for real-time HMI environments is tight synchronization between the HMI and the backend control system to prevent latency and enhance performance. This synchronization is typically using real-time operating systems (RTOS), which are designed to manage high priority tasks while maintaining accurate timing. By employing RTOS options such as FreeRTOS, VxWorks, or Integrity RTOS, embedded control systems can effectively synchronize with the HMI, ensuring efficient data exchange and scheduling.

Additional Best Practices for Real-Time HMI Development Efficiency

To further optimize real-time HMI performance, consider these additional strategies:

Choose the Right Hardware: Selecting hardware that meets the desired performance benchmarks in terms of processing power, graphics rendering, memory, and storage is crucial for the address the demands that real-time HMI rendering and data processing place on it.

For instance, dual-core microcontrollers allow for the separation of UI tasks from processing actual time data, thereby enhancing interactivity. One core can be dedicated to handling UI-related functions, while the other handles time-critical operations such as sensor value processing or communication with other external systems. NXP's i.MX RT series offers high-performance variants that are optimized for real-time applications.

Memory Optimization: To reduce the requirements in resources-constrained environments, HMI should be optimized to use the minimum amount of memory possible. Techniques like object pooling and memory compression can help make maintain lightweight memory usage without sacrificing performance.

Even getting the best possible performance out of an embedded system frequently comes down to good choices about memory. Given the differences in access time among various memory types, it is advisable to store dynamic content, such as the image framebuffers, in RAM while keeping persistent data, such as the application’s code, in flash memory.

Optimize Interrupt Service Routine (ISR): Interrupt handling is a crucial technique for minimizing latency in real-time HMI systems. Interrupt ensures the timely execution of critical tasks by allowing the system to respond promptly to external events. The best performance of ISRs is achieved if they are short and efficient. Developers can create highly responsive and efficient HMI systems simply by managing interrupts based on priority and time-sensitiveness.

Task Scheduling: Improving real-time performance in HMI systems involves effectively organizing and prioritizing task execution. By assigning higher priority to critical tasks, such as user input and real-time data processing, the system can respond more quickly and minimize delays.

Multithreading and parallel execution allow tasks to be carried out simultaneously, so that one task does not block another. This is particularly beneficial for separating complex, resource-intensive functions from the user interface, ensuring that the HMI remains responsive even under heavy system loads.

Additionally, effective scheduling reduces the need for context switching (a source of overhead) and ensures that critical resources, such as CPU and memory, are allocated efficiently to avoids bottlenecks.

Testing Under Real-World Conditions: It is important to always test your HMI under real operating conditions. While simulated environments can be helpful, testing on actual hardware provides the best insight into real-time performance.  Thorough testing and debugging can help identify latency-inducing elements, such as malfunctioning parts or software bugs, so that they can be fixed to improve system performance.

Real-Time Performance Optimization Using Crank’s Storyboard

Storyboard is designed for applications that require uncompromising real-time performance. Its HMI design and development framework caters specifically to the need of the embedded engineers looking for a streamlined way to create and deploy high-performance interfaces.

For example, a medical HMI device manufacturer used Crank’s Storyboard and significantly reduced their interface's frame rendering time in a short period of time. This improvement enhanced the real-time response of critical life-support devices.

Another example is Stages Cycling, which chose Crank Software to develop an advanced HMI for their e-bike system. The interface needed to be interactive, responsive, and lightweight enough to run on a low-power system. By applying key optimization techniques, crank was able to create a smooth and responsive HMI for Stages that enabled riders to operate their ride effortlessly.

In an automotive display instrument cluster application, Autometer successfully developed a customizable, multi-function LCD gauge display by optimizing real-time performance. This ensured the HMI provided instant feedback for the user. Additionally, the team created a ruggedized, fast-responding, and graphically rich HMI for the next generation of NASCAR racing vehicles, meeting the stringent real-time performance benchmarks required or safety-critical applications without sacrificing time to market or requiring extreme environmental specifications.

Key Features of Storyboard Supporting Real-Time HMI Optimization

Offload Rendering to GPU: Offloading rendering tasks to the GPU results in silky-smooth animations and even quicker screen transitions without overloading the CPU. Storyboard’s hybrid rendering technology was recognized as the Most Innovative Software technology at Embedded World in 2020. Storyboard supports the following software render engines:

• ChromeART - 2D engine (DMA2D)
• DAVE2D - 2D engine
• G2D - 2D engine
• OpenGL - 3D Engine (multiple platforms supported)
• RGA - 2D engine
• SDL - 2D Engine
• VGLite - OpenVG engine (coming soon)
• Vulkan - 3D Vulkan Engine (coming soon)

Hybrid Rendering: The hybrid approach reduces graphics pipeline overhead by allowing for a mix of 2D and 3D elements in the UI application. This removes the need for an application to be comprised entirely of 2D or 3D graphics, enabling the HMI applications to dynamically leverage the most suitable GPU based on the content type.

Cross-Platform Optimization: Storyboard offers cross-platform optimization for both low-power microcontrollers and high-end embedded processors, ensuring real-time consistency across multiple platforms.

Prototyping and Real-Time Validation: With Storyboard's real-time prototyping and integrated GUI testing and validation functions, software engineers can efficiently build, test, and refine HMI applications that provide instantaneous feedback in real-life situations.

Final Thoughts on Optimizing HMI Performance

Optimizing real-time performance in HMI applications for embedded systems involves a combination of hardware selection, software techniques, and design practices. By focusing on efficient rendering, low-latency input processing, and real-time data synchronization, development teams can create HMIs that deliver immediate feedback, smooth operation, and reliable performance.

Tools like Crank Software's Storyboard offer significant advantage, enabling UI developers to create optimized design cycles for real-time performance. For the next generation of user interfaces in mission-critical applications, designers and developers must prioritize the use of these kinds of optimization techniques for embedded GUI application development.