Creating sophisticated GUIs for microcontrollers using NXP MCUXpresso

Today’s IoT and industrial device markets are changing the way we build graphical user interfaces (GUI). As users demand richer and more featured-packed experiences, embedded development teams are looking for new ways to deliver value on platforms that are both power efficient and connected.

The combination of the NXP MCUXpresso SDK and Crank Storyboard allows UX designers and embedded developers to work together to deliver amazing GUI experiences on microcontroller (MCU) platforms. By connecting the creativity of designers to the maximum capabilities of NXP MCUs, developers can build sophisticated GUI applications that meet the needs of consumers.

Three experts from NXP and Crank sat down to discuss how to get the most out of both platforms: Shelby Unger, Systems & Architecture Engineer, NXP Semiconductors; Nik Schultz, Field Application Engineer, Crank; and myself, Scott Snider.

You can watch the entire discussion by clicking the image below or by reading the highlights here.

How the NXP i.MX product family supports good user experience

Growing demand for enhanced user experience has really driven the growth of both MCUs and applications processors but a lot of use cases have emerged where neither one perfectly addresses the needs of the situation. The i.MX RT crossover processors were really the first solution that bridged the gap between MCUs and applications processors that allowed embedded designers to scale between the two easily. They're designed to support the next generation of IoT applications with a high level of integration and security that's still balanced with MCU-level usability.

Storyboard supports several devices within the RT family and each is well suited for embedded graphics applications. For example, the i.MX RT1060, in combination with Storyboard, supports:

• High-performance real-time processing, operating at core speeds of up to 600 MHz and with very fast responsiveness and an interrupt latency as low as 20 ns
• Product designs that include advanced multimedia for embedded GUIs, featuring an enhanced LCD interface as well as the pixel processing pipeline
• 2D graphics acceleration engine with functions such as color scaling, overlay with alpha blending, image rotation, and color space conversions

These features are also included on the RT1050 and 1064, and supported by our MCUXpresso Ecosystem. The heart of this ecosystem are the core technologies that NXP provides as a cohesive set of software and tools for designing with NXP MCUs. This includes the integrated development environment, software development kit, configuration tools, and secure provisioning tools.

How MCUXpresso helps GUI developers

The core MCUXpresso technologies are enhanced by enabling software and tools, including runtime software and middleware, partner IDEs such as IAR Embedded Workbench and Arm Keil, debug probes, and various development boards that work seamlessly together and help speed up the development process. The MCUXpresso SDK provides the software framework and drivers that are key to an efficient development experience, including optional support for FreeRTOS and various integrated stacks and middleware.

It also includes reference software that comes in the form of different driver, middleware, and FreeRTOS usage demos as well as various application demos. All the software is BSD licensed, Black Duck scanned and free to use, and coded against industry quality standards.

One of the major benefits of the MCUXpresso SDK is the enabling software technologies that we provide for a variety of applications including wired and wireless connectivity, graphics, motor control and many more. Some of these are developed in-house at NXP and others are developed by third parties, but they're all provided for free and give you a head start in developing your application.

Crank Storyboard is one of these enabling software technologies that comes fully integrated with the MCUXpresso SDK. It makes it a lot easier for developers to go in and evaluate this software in conjunction with our MCUXpresso Ecosystem as well as their NXP devices. Essentially, when building and downloading a custom SDK using the online SDK builder, there’s the option to include Storyboard as one of the software components.

Developers have access to several fully functional examples that can be immediately programmed onto the board to get a feel for the entire solution.

How Storyboard helps build better GUIs for microcontrollers

When it comes to developing GUIs for today's embedded products, things have become more challenging. User expectations have been elevated and consumers expect an intuitive and modern touch experience. Not only for the physical work but the quality and smoothness and the experience of that GUI. This is because we walk around with a touch screen in our hands every day, and expect a similar type of experience on every device regardless of form factor or what lies under the hood.

This is where Storyboard comes into play as it’s designed to remove the barriers between UX design and embedded development. The Storyboard architecture and workflow decouple the frontend GUI from the backend logic and microcontroller, enabling the designer and developer to work in a parallel process while they build the app, making it possible to get started early and easier to embrace the inevitable design iterations.

It's not a matter of if iterations are going to occur, it's a matter of when. Storyboard embraces that reality and reduces the overall complexity of developing GUIs for microcontrollers and ever-evolving product lines. For example:

• Development can begin within familiar GUI design tools such as Photoshop and Sketch and when ready, imported into Storyboard to create the building blocks of the application
• The GUI can be brought to life using Storyboard's integrated animation tools, allowing you to create, edit, and review animations directly within the workflow
• When design changes occur, they can start in Photoshop or Sketch and can be reimported into Storyboard without having to create an entirely new application, or overwriting prior development and testing efforts

Embedded GUIs that scale

Storyboard was also created to be platform agnostic, so that customers can easily de-risk their projects by introducing this framework that supports technology changes at any point during the development cycle. This helps scale the GUI from an MCU to MPU or from an MPU down to an MCU.

One of the customers that we've helped achieve this wanted to introduce their next-generation beverage dispenser with new technologies and GUI changes based on consumer feedback. They had existing products in the field that were using older, lower-powered embedded devices and didn't want to disrupt them. We worked with them to demonstrate how our optimized runtime footprint for microcontrollers, such as the i.MX RT1060, could easily port the GUI to new beverage dispensers.

Working with partners like NXP ensures our Storyboard runtime engines are optimized for the target platforms so developers don’t have to worry about re-compiling and doing all that heavy lifting.

From MCUXpresso to Storyboard in minutes

The process of going from a project in the NXP MCUXpresso SDK to Storyboard and the target is fairly straightforward.

Configure the SDK package

Just like Storyboard, MCUXpresso is supported on Mac, Windows, and Linux. By going into the MCUXpresso SDK builder and picking the board kit, dependencies, and middleware pieces you need, you’re able to create a build package for deployment to the target hardware. This includes selecting Crank Storyboard as the GUI middleware.

Import SDK example

When importing into the project, there is a section for Crank Storyboard examples alongside many other examples of how to get different components up and running on your development board. The Crank Software Hello World example is a minimal configuration used for small memory footprints and resource constrained devices.

Build the example project

If you just want to get up and running, all you need to do is build and deploy to the target using the functionality within the MCUXpresso IDE, using your connection method of choice. You can also debug the application using the built-in tools.

Update the UX design

Outside of the MCUXpresso IDE, if you have a Photoshop or Sketch file available, you can directly import it into the Storyboard workspace. As you're going through and iterating and developing, your designers can lay out content exactly as they want to see it and it will translate seamlessly to Storyboard. All the original image assets are maintained and all screens are laid out as they were in the original design. The Storyboard model hierarchy also follows the architecture that was laid out in Photoshop or Sketch.

Once the design file is in Storyboard, you can:

• Manipulate the size and position of elements to better fit the desired UX
• Create animations for buttons, menus, screens, and other GUI elements (the animation timeline tool is similar to those found in After Effects or Premiere)
• Connect to the backend application code using events — a Storyboard GUI is completely event driven and on crossover systems like the NXP RT series, you want to be diligent about CPU cycles and resources
• Perform dynamic behavior using the Lua scripting language

A key point in GUI for microcontroller development is that if your screen is idle, Storyboard is not rendering or taking up CPU time. You can have very efficient applications and complex user interfaces running on these platforms, helping lower your bill of sales or bill of materials costs.

Once you have updated and exported your Storyboard application, you can bring it into the MCUXpresso IDE for further testing and iteration.

Typical questions from developers

What file format does Storyboard import from Photoshop?

Storyboard imports and re-imports the PSD file directly from Photoshop, converting all the assets into rasterized images.

Does the Crank framework offer multi-language support?

Yes, as Storyboard renders content with the free-type rendering engine. With that, you can import TTF or OTF fonts. There are customers shipping with multiple languages in different regions today.

How do you connect the GUI to real hardware operation and external real-time threads?

Storyboard is completely event driven with a number of different handling options, including Lua scripting and C callbacks. There is also a library called Storyboard IO that allows for platform-agnostic cross-task or cross-process communication, very similar to message queues. This allows you to inject events into the system or receive events from the GUI.

With wearables and other devices, the trend is towards non-rectangular, circular displays. Is that supported?

Yes, as Storyboard renders to frame buffers and you simply update to a region. It's still a screen and you do all your manipulation there. NXP also has products coming out that are circular-display based.

Does Storyboard support swipe events?

Yes, there is a Storyboard plugin for gesture input that supports left, right, up, down, and diagonal swipes. You can also define custom gestures if you want to do a zigzag-type input.

Does Storyboard support ping-pong frame buffering in the foreground and the background?

Yes, that's generally the approach in Storyboard. That’s the methodology used in some Storyboard demos you can try and we have also leveraged PXT acceleration to help the buffer swap and transfer to get a little extra performance. You could configure for a single-frame buffer configuration as well but if you're looking to do full-screen blends or fades, you may not get the effect you're looking for. To avoid tearing and similar artifacts, you need a second-frame buffer.

See the actual MCUXpresso and Storyboard demonstration by watching this webinar now: