When building an embedded Linux GUI application, there are a number of considerations to be taken into account when using Crank Storyboard. In this blog and accompanying walk-through video, I detail the steps required to best prepare your embedded GUI for a Linux board, specifically:
- Exporting Your Embedded Linux GUI application
- Configuring Your Embedded GUI Application for Linux
- Connecting to Your Embedded GUI with a Serial Connection
- Knowing Your Embedded GUI Application Options
- Understanding Your Environment Variables
- What is Wayland?
- Choosing Your Touch Library for a Linux Environment
- Executing the Runtime for an Embedded GUI on the Target
- Running Your Embedded Linux GUI
Don't have time to read? Watch the video demonstration at this link.
Exporting Your Embedded Linux GUI Application
I used the NXP i.MX 8M Nano evaluation board, so if you're looking to reproduce the steps that I’ll walk through, there are sample Storyboard GUI demo images available on our website to download and play around with.
The first thing you need to understand is how to export your application for the embedded system by deciding on an export configuration. This is where you can start to define the configurations for different platforms, such as when evaluating a couple different boards and you need to set them up. We have a little button at the top of the Storyboard toolbar called Edit Export Configurations. This allows you to pick and choose your assets that you want to export and deploy, decide on the format, and build up multiple configurations.
Right now, we have a single default configuration that closely mirrors the Linux OS configuration we're going to use. We could use this if we wanted to, but we’ll create our own here. I might name it NXP iMX8 to match our hardware and then start manipulating the content. For now, I'm going to leave all of this setup. Because we are exporting to an operating system that has a file system, we don't need to do anything that might be more MCU-specific. But, you can also go through and pick some of the assets that are going to be coming with it. By default, Storyboard exports everything in our project but there is a view to include or exclude specific elements.
If you're ever wondering where something is or if you create a configuration with exports and content, and some images aren't showing up, you can always cross reference the content in the left-hand pane in this project versus what gets deployed to your embedded system.
Once we've created our resource configuration, we can go through and start the export side of things. When we’re launching our embedded Linux GUI application on the desktop environment with the Simulate icon, we’re executing our runtime engine built against the desktop platform. In my case, I’m using a Windows desktop and I have access to OpenGL acceleration and hardware. The tooling actually exports this application behind the scenes so that it is seamlessly set up for us. But if you're poking around and curious to see how it's configured on your desktop, this is where the information is going to show up. It will also roughly mirror the command line options we can use on the embedded hardware.
Configuring Your Embedded GUI Application for Linux
That being said, we're not looking to run it on our simulation Storyboard environment anymore, we want it on the embedded hardware. This is where we open up our Application Export Configurations Panel to set up how we're going to export and send the embedded GUI application. The default packaging method is the GAPP file format and that runs the exporter. We're going to convert the GDE file into an embedded-ready file interpreted by our engines, so I'll select that. Storyboard also has other options available including APK for Android and C++ and resource header for MCU packages. This is where we can select our resource configurations on the iMX8 and we can also choose if we want to transfer this somewhere else. In this case, I have my embedded hardware running beside me so I’ll choose the option of doing an SCP transfer or a direct file system transfer. If I were to stick with a file system, that would export the gap file into the project directory specified under the navigator tab.
When the demo application is running on embedded hardware, I can click around and navigate through it. There’s an IP address showing up so if your hardware has an Ethernet jack and is configured to run the Ethernet stack on your Linux OS GUI configuration, it'll auto-display the IP address which is helpful. If I were to hit the apply and run buttons on the dialog I had opened earlier, we're going to SCP the application we had onto the embedded hardware system. And from there, since it's SCP, we can then go into the system, launch it, and run it.
I want to now explain how to manually run it from the embedded hardware. This is where if you'd like to execute a script after launch, you can. The first thing you want to do is get a connection to your embedded hardware.Connecting to Your Embedded GUI with a Serial Connection
In this case, the first thing you want to do is open up a serial connection. Most boards these days will have serial. If you have Ethernet, Storyboard gives you the option to do a SSH network connection which is a little bit more usable in some cases. Each board has specifics around the baud rate and how they are exposed. You'll have to check the documentation around the surrounding hardware to understand what's going on in terms of the specifics but, in this case, for the iMX8, I know we have two comm ports that show up and we want to select the highest one. The lowest comm port is going to be a serial connection for the embedded MCU chip that's on the board.
If you have Ethernet, it also gives you the option to do a SSH network connection which is a little bit more usable in some cases.
Our baud rate or speed is 115,200 which is pretty standard these days with eight bits data. I'll select OK. And then if we hit enter, you can see we're greeted with a login prompt. The default login for most embedded boards is root. It might prompt you for a password which is likely root as well. Once we’re on our embedded hardware system, the first thing that I always like to do is run ifconfig. That simply prints out the Ethernet information. You will see our IP address matches what we were displaying on the display.
Knowing Your Embedded GUI Application Options
The second thing is that there's obviously something happening on the board. You plug it in, flash an SD card, and it auto boots to a nice Storyboard demo launcher with a couple of different app options. If you're curious to know what that looks like, look under user Crank - that's where we keep all of our content and assets. If I list the directory there, you can see we have an apps directory, a runtimes directory SCP and then the Storyboard_SCP launch script. This is what we use to remotely launch an application.
Understanding Your Environment Variables
We also have the Storyboard SCP script. The most important thing to understand from your end is that, when you're looking to configure your board for an embedded Linux system, we export a few environment variables. The engine itself is looking for SB_ plugins which is going to give us the path to any of the plugins, and then the LD_ library path. These are the two main environment variables.
Some systems might require other variables, for example, in a Wayland environment. In our case, in the Weston compositor, you'll see a variable like this. But what's really happening is we've exported those paths, we have our engine, and then the path to that is called out here. When we run our application, we call the engine under the bin directory, SP engine, and pass in a series of options. We simply tell it to run full-screen and provide the path to the application. If I were to launch that from the command line, we can call Storyboard SCP and then pass in a path to the application. Under Apps, we have OpenGL and the gap file is located there. We've now launched the application that was running on the board.
If I hit Ctrl-C, that kills the application in the process. As I mentioned, there's the Apps Directory which is where we have some subdirectories underneath and there's existing applications there that you can play with too. Each one of these folders contains a gap file as well as the GDE, so if you wanted to re-import that into your design environment, you could copy that off the board using something like SCP - or you could put a USB drive into the board if you have USB connection. Other options include configuring FTP, or mounting to a remote filesystem directly.
The other important directory is the Runtimes Directory. In this case, we only have one runtime that lives on this board, labeled Linux, MX8, Yocto, and OpenGL ES2 Wayland. The name of this directory determines the configuration of the runtime. You can use that naming to help understand which one you want to pick. In our case, we know a lot of that is determined by the hardware, so for instance, the first piece that you're going to see in that title is Linux, determining the operating system. If we look at our Installation Directory, you’ll see Storyboard ships a large number of runtimes, and these are all built and configured for common evaluation boards out on the market. We also have a runtime that makes use of just the raw OpenGL interface. If you don't want to run the Wayland system, that's totally fine.
What is Wayland?
Wayland is a screen managing stack. If you think of a desktop scenario where you have programs that are running in their own windows, Wayland is a system that helps you manage applications as their own screens and windows. You could run multiple Storyboard applications in windows and use the Wayland system to build up some management functionality. Usually Wayland is run with a compositor. Most of the embedded systems that have system images built are generally using the Weston compositor. There's a good chance that if you're on your target and start typing Weston, it'll auto-complete and show a bunch of other simple applications.
To test and understand if you're running Weston, try running the Weston-Flower app. If you see a nice little flower popping up on the display, drag it around. The configuration information for Weston is under the /etc directory.
Try running the Weston-Flower app. If you see a nice little flower popping up on the display, drag it around. That's one way to test and understand if you're running Weston.
If you’re running Weston, it does a lot of the work for you. It handles touch input and the display configuration. If you want to rotate the display, you can modify that config file and flip everything. All of these Runtime Directories contain the plugins, so one of our environment variables is simply pointing to this list of plugins. These are the features and functionality that your application will leverage at runtime.
Choosing Your Touch Library for a Linux Environment
Now, depending on what system you're using and what configurations you're using, the Linux OS generally has two main touch libraries: tslib and mtdev. Traditionally, tslib was a single touch input and mtdev is multi-touch. Tslib has newer versions where it supports multi-touch, but out of the box, mtdev is usually easier to configure to get up and running. Of course, it always depends on what libraries are available on the system.
mtdev is usually a little bit easier to configure to get up and running
Trick #1: Increase Verbosity
A good way to check without actually searching for the libraries, is to go back to our command prompt and look at the Storyboard launch script, where you can see some options. We can increase the verbosity and it'll show us the plugins that are loading. If any of them fail to load, we'll understand if libraries are missing or not.
If I were to modify that, using the vi editor, I can just drop down and add some levels of verbosity to the engine. Adding four Vs increases the diagnostic output and each V adds another level. This is generally a sweet spot if you're looking to understand what's happening to the engine at runtime. We can see that we don't have the libraries for tslib so that’s a pretty good clue that tslib isn't really an option on this system and its configuration.
In terms of mtdev, we don't see any errors so we could leverage it. At this point, we don't have much information about the touch device right off the bat, so we’d have to do a bit more work to hook that up. Firstly, detect your touch device. On our system, we have an HDMI monitor with a USB touch connection plugged in. Assuming you have a standard system and everything enumerates and displays properly, your device is going to show up under the Dev Directory.
Trick #2: Cat the Device
We want to look under Input to try and understand which device is actually yours. The best way to do that is a handy little trick where you just cat the device. I'm just reading this file output and then if I touch the screen, the binary data is being printed out onto the console. That tells me that there's data flowing out of it and it's a pretty good clue that this is going to be the device. You could go through and validate the other ones just to be sure. So if I were to cat /event0 and hit Enter, we don't see anything when I'm touching the screen right now. If we wanted to leverage that, all you'd need to do is add as a parameter to the engine options to go back to the user, Crank Storyboard SP. O-mtdev. This is simply saying that this is a plugin object with the name mtdev, mapping to the plugin itself. This is where I can say device equals, and we simply pass in the path to the device, /dev/input/touchscreen0. If this was a system that we weren't running Wayland on, I would add it to hook up the touch input.
A handy little trick is where you just cat dev/input/touchscreen0. I'm just reading this file output and then if I touch the screen, the binary data is being printed out onto the console. That tells me that there's data flowing out of it and it's a pretty good clue that this is going to be the device.
Those are some of the main things you need to understand for running and configuring on the embedded system. It’s a pretty good test if you're going through and dropping a runtime, you can just run it from the terminal. If I were to put that and I have SCP copied my runtime (under Runtime), go into the Linux directory. We can take the path for this and have a little clear terminal. I can then export some of these paths.
Executing the Runtime for an Embedded GUI on the Target
This is where I can say, export LD_library_path = /lib. I will do this for lib and do the same thing for the SP_* plugins. Since we've exported both of those, you'll want to execute the runtime. I'll jump back into the Runtime Directory and jump into bin. This is where the SP engine runtime is. If I were to ./SPengine and hit Enter, a usage message will come up. This is going to be looking for another environment variable, the STG information, but if it does fail, it'll give us some diagnostic information.
So, that's usually the first step you want to do. Creating a launch script is also helpful to start building up different options. You could even be a little bit more creative and query for different devices. Maybe it's a more genericized one but that comes down to how you want to configure your shell script and depending on your comfort in bash scripting.
The final thing you want to do once you're happy with your application is auto-boot. You don't necessarily want to jump onto the terminal every time you put up your board and execute your script, or export the environment variables again. We’ll use systemd to launch a service. There's a couple different ways you can do that. On some of the older Linux system configurations, there would also be the scripts found under /etc/rc.d. You would find a directory like rc.0 through rc.6. Again, the different launch states of your system, but in our case with systemd, you can specify a service and dependencies.
Using the Storyboard Launcher
When we launch the board, it automatically calls a script that we have under /etc/rc.d. Put the script into Storyboard Launcher. The script has a start function and stop function, so you can specify commands into that. The minimum needed for a service is starting and stopping. Sometimes it's helpful to add a restart command, so you could basically kill and reboot it.
The script has a start function and stop function, so you can specify commands into that. The minimum needed for a service is starting and stopping.
We now want to trigger the launcher app and call it. This gets called on boot, so if you're playing around with one of our demo images, this is where you can start and stop them. The system control is a main command that you can run to print out all the different services running and their dependencies.
The system control is a main command that you can run to print out all the different services running and their dependencies.
Running Your Embedded Linux GUI
If you run system control, you can then specify a command. In our case, we can say start and then stop the Storyboard service. That's running the Storyboard service that's mapped into systemd, launching our application again. If we were to just run the system control command, and type into rep, and search for Storyboard, you could see that our services are running. You’ll see uploaded active, running in a launch Storyboard service. You can also use the stop command to tear it down.
That brings me to the end of how to configure your embedded GUI system for your Crank Storyboard apps running on Linux. If you'd like to learn more about Storyboard on your Linux board, check out our videos in our Knowledge Base and detailed article in our Help Centre.