An analysis of Storyboard GUI performance on Microcontrollers (MCU)

Customers often ask us about Storyboard's performance on specific hardware. What is the maximum number of frames per second that Storyboard can achieve? What is the CPU usage of the engine? And how much memory is required to run an application?

The answer that is given is that this is all configurable, which makes it sound like the questions are being avoided. At that point, the assumption is that the questions are being avoided because the numbers are bad. This is not the case at all. The goal is not to mislead potential customers into thinking that the solution they are getting can achieve 60 frames per second (FPS) on a target while using no CPU or memory.  The goal of that answer is to inform potential users of Storyboard that they have options available to them when it comes to achieving the performance that they need to obtain

This article explores several other applications to illustrate how specific design considerations affect engine performance and the ability to achieve particular numbers.

Our goal is to provide insight so that customers considering Storyboard as their UI solution know what they can accomplish with this solution.

All the application run used the following configuration on the NXP RT1050:

• Back buffer to front buffer rendering approach. This means Storyboard renders into the back buffer and copies to the front buffer, which the display reads
• The back buffer is stored in the data tightly coupled memory (DTCM) SRAM section on the board. The DTCM section is incredibly fast for access on the RT1050, making pixels operations quicker. When compositing a screen, a read and a write need to be performed on the pixel, so improving access times affects performance
• The board's pixel pipeline (PXP) chip is used to copy the back buffer to the front buffer. This offloads the transfer of the back buffer to the front buffer from the CPU
• The framebuffers are allocated as 480x272 using two bytes per pixel, which means that each framebuffer requires an allocation of 261120 bytes (255 KB) for storage. There are two framebuffers, the back buffer in DTCM as mentioned above, and the front buffer, which is in SDRAM. This brings the total number of bytes needed to store the framebuffers to 522240 bytes (510 KB). The calculation to calculate framebuffer memory is "width of the framebuffer" * "height of the framebuffer" * "bytes per pixel of the framebuffer." For the configuration above, this would be 480x272x2. This would give the size of one frame buffer. That value can then be multiplied by the number of framebuffers needed to provide the total size required by the framebuffers
• The FreeRTOS OS, with the FreeRTOS tick hertz was set to 1000. This allows for 1ms timer resolution
• The MCUXpresso SDK 2.12.0 version was used
• The Storyboard Runtime version was 7.2.0

BubbleMark

When a new port of Storyboard is brought up on a platform, Bubblemark is one of the first applications to run. This is because the Bubblemark application tests Lua execution and timers. Depending on your preference, it can run flat out, generating a new redraw event as soon as the previous one has been serviced or off a timer so that you can control the frame rate. The engine's default configuration is to run flat out, which limits the top FPS you can expect if the CPU is fully utilized. Here is a video of the application running on the target:

It took 1.66 megabytes (MB) to store this application in a flash. This is the size of the BSP, the runtime, and the assets required to run the application. The images for the application were stored uncompressed in a flash. This was done to cut down on the memory required to draw the image.

The upper limit on this board is around 99 FPS, consuming all the CPU. Here is the performance data recorded from running this application on the RT1050-EVKB for two minutes.

The legend for the tables used in this document are:

• MS Elapsed:  The number of milliseconds that the application has been running
• CPU Time:  The amount of time that the CPU was not idle
• CPU %:  The percentage of time that the CPU was not idle
• Frames Rendered:  The number of frames the Storyboard Engine rendered in 10 seconds
• Render Time (MS):  The number of milliseconds out of the 10 seconds the engine spent rendering
• Time per Frame (MS): The average time taken to render one frame
• FPS:  The frames per second that the engine achieved
• Memory Used (KB):  The number of kilobytes from the heap that the engine used

Looking at the numbers, they correspond to what the application was reporting for an FPS. When the engine runs flat on the board, it achieves an average FPS of 99. There are a couple of things to note about the data. The first is that the average render time per frame is six milliseconds (MS). If the system only performed rendering, that render time would provide 167 frames per second. This fits with the rendering time accounting for the 58% usage time for rendering. The rest, 42%, is being used to calculate hit detection, speed, and direction of the balls through Lua.

Now that the upper limit has been evaluated, the bubble mark application can be configured to draw every 16 MS. The storage size does not change. The following is a video of the bubble mark application running using a timer to drive the drawing of the frame every 16 MS:

Here is the data recorded from this run:

The CPU usage drops down to an average of 60.72%, which is expected, as the engine is no longer running flat out. It is now breaking to wait until a frame needs to be drawn every 16 MS. The ratio of render time to calculation time remains consistent at 58% to 42%. The system is now idle for 39% of the time as the CPU is not needed during that time, and the engine is dormant during that time, waiting for the timer to fire before it needs to draw again.

The next step after that is to see what the application looks like when it is throttled to draw at 30 FPS, which is achieved by setting the timer to fire every 33 MS. Here is a video of what the app looks like running at 30 FPS:

Here is the data recorded from this run:

Again, there is a drop in CPU usage, which is now 29.24% on average. Based on previous configurations of this application, we observed a 58% to 42% split between rendering and calculation code.

Now, with the timer approach to updating the positions of the controls, there are no frames to drop, which is why the movement looks slower when the timer interval is set to 33 MS. There is a way to configure the engine to provide smooth animations while using less CPU. This will be highlighted in the next section.

Hello World

The Hello World Storyboard application is a simple application that runs a couple of animations. It is an excellent test to ensure that the animation plugin works properly when running Storyboard on a new platform. Here is a video of this application running on the hardware:

The Hello World application takes up 2.45 MB of flash space. The following is a table that shows the metrics for this application while it was running for 2 minutes:

This application does an excellent job of highlighting one of the core strengths of the Storyboard engine. The Storyboard engine is event-driven. In the absence of events, the engine does nothing. The screen does not need to be rendered if no events occur, which means no data needs to be updated. Therefore, the FPS numbers in the table may seem low, but that is the exact number of updates to the screen required to achieve the animation effect that the Designer was hoping for. The following screenshots show the design of the animations.

The animation design shows large blocks without any data change. These are times when the engine will stay idle, giving other threads in the system a chance to use the CPU without compromising the smoothness of the animation. This can be verified by looking at the CPU utilization numbers while running the Hello World application. The CPU usage stays below 10% for running this application.

Infinite List

Smooth scrolling lists have been a requirement in user interfaces since Apple released its first iPhone. There is a sample application provided in Storyboard called Infinite List. This application demonstrates how to create an extensive list of items, in this case, 10000 items, and shows them in groups of 60 items at a time. The sample was created to show how to create a scrollable list in Storyboard that is smooth and responsive. Here is a video of the Infinite List sample application running on the RT1050-EVKB:

The Infinite List sample takes 1.96 MB of flash to store. The following table shows the metrics for the engine while scrolling through the list:

Few things to note about the metrics: The first is that drawing text is more intensive than drawing images. The reason is that new text needs to be resized in order to be positioned properly in the rendered area. Additionally, the text has an alpha blending component since the font engine provides glyphs as alpha values that determine which pixels require drawing and which do not. Each pixel in the glyph needs to be colored with the chosen text color. This can lead to higher CPU usage as the new text items are being loaded into the list.

Also of note is the last entry in the table. At this point in the execution of the application, the list was left alone for 10 seconds. The engine did not render anything at this point and was idle. However, the CPU had a usage percentage of 5.67%. This was due to the input thread that got information from the touch screen. The way the CPU usage was calculated during the running of the applications was to look at the time that the CPU was idle. This means that any thread running on the system would count as CPU usage. The touchscreen driver used in this case did not support an interrupt-driven approach to reading data, which meant that the device needed to be polled for data.

Polling for data suggests that the thread will wake up at a defined interval, check the device for data, and if there is none, go back to sleep for the defined amount of time. This polling interval is why there is a 5.67% CPU usage and is not tied to the Storyboard engine. Any graphics engine that required touch input using this method would see some CPU usage from the input thread, where the usage would be based on the polling interval. The Bubblemark and Hello World applications did not need the input thread, so the CPU usage numbers reported for those applications are strictly what the Storyboard engine needs.

CoffeeApp

The CoffeeApp demo provides a way to compare Lua scripting to C code implementations for creating dynamic responses to events. The C code callback action offers an action that will call a C code function when an event occurs. The C code API for Storyboard is a low memory option compared to Lua, but the Lua API is richer in terms of API functionality. Simply put, fewer convenience functions are defined for the C code API because those functions require more memory.

Here is a video of the CoffeeApp demo running with C callbacks:

To store this application on the RT1050-EVKB it took 4.51 MB of flash memory. Here are the metrics for running the CoffeeApp demo on the target:

As a comparison, here is a video of the application running using Lua callbacks instead of C callbacks:

To store the version of the CoffeeApp demo that uses Lua callbacks on the device, it took 4.68 MB of flash memory. Here are the metrics for running the CoffeeApp demo using Lua callbacks on the target:

When comparing the metrics from the CoffeeApp run with C callbacks and the CoffeeApp run with Lua callbacks, the numbers generally remain the same for CPU utilization and rendering time. Still, by using C callbacks, 58.93 KB of RAM was saved.

This demo application utilizes circles and alpha blending to achieve the graphical look and feel seen in the UI.

These rendering techniques require a little more CPU to accomplish the rendering.

Home Controls

The Home Controls demo application is designed to emulate a real-world UI for an application. The demo has multiple screens that provide users with data about a system.

Here is a video of this application running on the RT1050-EVKB:

To store this application on the RT1050-EVKB it took 9.13 MB of flash. Here are the metrics from the system while running the Home Controls demo:

There isn't anything notable about the Home Control application when it comes to the approach or design of the application. It was a demo created to act like a typical UI application. It uses Lua for the dynamic aspects of the UI, has multiple screens, uses animations to provide effects, etc.

Conclusion

As can be seen from the metrics from the different applications that have been run on the RT1050-EVKB, Storyboard has many ways to build up a UI.  There is the choice of C callbacks or Lua for the dynamic aspect of the UI, or whether to build an animation using the animation timeline in Designer or using a timer in the application.  These choices allow a user of Storyboard to chart their own course and make decisions on how the UI will be built up so that the result matches the look and feel that the graphical designer was intending.

Storyboard is event-driven, and this architecture allows for an engine that will do nothing unless told to.  This allows the engine to conserve resources on a target machine that may be limited.  The engine is plugin based, which means that features that are not used can be removed from the engine, which will save on storage.

These are the reasons that when a potential customer asks for the amount of CPU or memory that are required by the engine, the answer that it is configurable is given, because the choices that the engine offer allow the user to choose which course best suits them when building the UI.