Storyboard and Qt integration

Storyboard was created to be a portable and platform agnostic solution.  This allows us to give the same experience across Linux, QNX, Green Hills and so on.  We often get asked how we deal with or compete with Qt.  We see Qt as a platform abstraction layer like all of our other systems, it just happens to have an optional graphics component.

 

Recently we have been looking into how Storyboard could integrate into or communicate with a Qt based application.  Storyboard gives UI developers and designers an extremely fast and efficient way to build modern HMI solutions (you can see an example here of how Storyboard differs from QML for UI Development).  This solution is cross platform and extremely small and efficient.  Users of Qt are also generally building cross platform applications and using QtGui may not be an option due to memory and performance concerns.  Storyboard can be used in conjunction with Qt in order to solve these problems and still give the developer a cross platform solution.

One option to integrate Storyboard and Qt is to run Storyboard as a Qt widget.  This allows Storyboard content to be displayed and interacted with onscreen with any other Qt based widget elements.

Another option allows users to remove the QtGui component and use Storyboard for the entire UI.  This will also have the benefit of saving memory and increased performance.  In this option Storyboard is run as the user interface and a non-GUI Qt application is written to preform backend logic.  The Qt application can then communicate with Storyboard using a Qt/Storyboard IO bridge allowing data and events to seamlessly be transfered between the 2 environments.

As can be seen Storyboard and Qt are not a mutually exclusive solution, they can work together.  I think Qt users will find that Storyboard gives them a rich visual design tool and will help speed their designs to market while still maintaining a cross platform application.

FreeRTOS gets Storyboard

Recently we have been working on expanding Storyboard’s support for smaller RTOS platforms. The latest addition is FreeRTOS. Storyboard brings a great development environment to smaller platforms allowing customers to develop and prototype the UI on their desktops and then seamlessly deploy to an RTOS target. We have been working with our friends at FDI on an NXP LPC1788 platform. Here is a quick video showing what is possible with Storyboard on these smaller platforms.

Loading JavaScriptCore extensions dynamically

When doing a WebKit port many people are looking to extend JavaScriptCore with custom functions which expose features of their particular hardware platform. Usually, they are also looking to load JavaScript extensions dynamically at runtime rather than being forced to recompile the WebKit libraries each time they want to add or change a function.

This blog posting looks at two ways in which WebKit can support dynamically loading JavaScript. JavaScriptCore can also be extended by compiling in extensions and writing IDL files, but that is a subject for a future post.

JavaScriptCore provides a C public interface, which can be found under Source/JavaScriptCore/API. There are JavaDoc style comments describing the individual functions, and some documentation is available on Apple’s site. Another blog posting provides a nice, although somewhat OS X specific, overview of the API.

A good strategy when creating your dynamic library is to have a single entry point which will create and return a JavaScript object containing your extension functions and objects. This function should have a signature something like this:

JSObject jsextension_load(JSContextRef ctx);

This function will use the supplied JSContextRef to create a class and make and return an object of that class. The class will contain static functions and possibly values that you want the extension to export and is defined through a JSClassDefinition structure instance.

The class definition is passed in as an argument to a call to JSClassCreate. You only need a single class instance for the lifetime of your  dynamic library. Once the class instance exists, an object can be made using the JSObjectMake call, and this object is the return value of your jsextension_load function.

The next step to to build a dynamic library which exports jsextension_load. Once this is complete, you then need a way of loading the extensions. This could involve specifying the name of the dynamic library to open, or simply scanning a specified directory for dynamic libraries, opening them and seeing if they export a jsextension_load symbol.

This post will look at two ways of loading extensions. The first is by using something similar to the CommonJS “require” function. The second is adding a method to the WebView to load an extension.

WebKit does not support the CommonJS require function, so this will have to be added to the JavaScript interpreter. An easy way of doing this is to add it to the set of static functions associated with the JavaScriptCore global object. These functions are defined in JSGlobalObjectFunctions.h and JSGlobalObjectFunctions.cpp which are found in JavaScriptCore/runtime, and the require function can be added to these files, along with a reference to it in the global object table found in JSGlobalObject.cpp.

The require function can be given a string as an argument and then attempt to dlopen a dynamic library corresponding to that string. The advantage of this approach as opposed to modifying WebView is that the extensions are available to the standalone JavaScript interpreter and the extensions to be loaded are controlled through JavaScript rather than through the WebView.

A second way of loading extensions is to modify the WebView API to support an extension loading function. The problem that occurs with this approach is to ensure you have a valid JavaScript context that can be used when loading the dynamic libraries. The current JavaScript context can be retrieved by calling

toGlobalRef(frame->script()->globalObject(mainThreadNormalWorld())->globalExec()

where frame is the current “main” frame in the WebView.

This is fine, unless you want to load an extension once and have it persist across page loads, as each page load will cause a new JavaScript context to be  created. Fortunately, there is a callback in the FrameLoaderClient which is called after a new global object is created, called dispatchDidClearWindowObject. This callback can be used to trigger the WebView to reload any extension into the new JavaScript context.

The advantage of managing extensions through the WebView rather than through JavaScriptCore is that is minimizes the amount of change to core WebKit code and makes the full interface provided by the WebView available to extensions.

Storyboard on Fujitsu Jade

We have been working closely with Fujitsu to support their graphics platforms. The first one to be released is the Jade platform. This is a great ARM platform which can be accelerated using the Fujitsu VO3 library. This library allows Storyboard to be fully accelerated on the Jade platform and give high quality animation effects and great framerates on a lower end platform. Our flexible achitecture allowed us to fully accelerate the graphics pipeline on the Jade. The next step is to integrate our 3D model rendering plugin included in Storyboard 2.0.

Here is a video clip of Storyboard running on the Jade platform using the QNX operating system (excuse the odd pulse of the LCD as the timings are not quite right).