It’s no surprise that the view inside the medical industry is at a time of unprecedented change, with new device and data requirements reshaping functionality and development needs. One clear trend that we see really manifesting this change is the exploding use of embedded GUIs in devices - with the medical user interface design now being a significant part of product differentiation. Identifying ways to address this layer of development has become a critical need to accelerate overall system development and ultimately, product success in the marketplace.
From understanding the medical device space to key considerations for hardware, operating systems, and touchscreens, we virtually sat down with 4 embedded experts to talk embedded UIs in medical devices.
Steven Olson is the North American FAE Manager at Blackberry QNX. Steven has been in the real-time operating systems (RTOS) space for over 25 years and specifically focuses on safety certifiable software.
Daniel Lang is the Chief Marketing Officer at Toradex. Daniel has been in the medical industry for more than 15 years. Toradex focuses on simplifying the ease of use of embedded computing in areas including medical, industrial automation, robotics, smart city, and test and measurement.
Chris Rommel is the Vice President at VDC Research and leads the IoT and embedded technology research. VDC is a market research consulting firm and has focused on the global engineering community for over 20 years.
Jason Clarke is the Co-Founder and VP of Sales and Marketing at Crank Software. Crank Software is an embedded GUI solutions provider, providing tools and services that make it easier and faster for designers and engineers to bring rich graphical products to market for embedded device manufacturers.
While we expect to see a very high growth rate of medical GUI integration, one thing that really stood out in research was that only 35% of users with a GUI in their design, reported using a design tool specifically focused at that layer. When you think about pain points in development, especially something as intuitively associated with end-product differentiation, this is clearly a gap more OEMs need to address. More specifically, given all the other challenges, development teams need to address its importance for OEMs and consider solutions geared at that layer up front, as they plan their overall device hardware and software stack into next year.
In the medical field, device failure is never an option, especially when thinking about safety-critical development. We shouldn't accept project execution failure either. However, somehow people have taken this dynamic and that potential outcome for granted: The industry as a whole has long suffered from delays and missed deadlines - with schedule adherence far worse for medical device development. Our research at VDC shows that 37% of projects are reported as completed behind schedule.
Noting this, we're clearly in a scenario where the status quo does not suffice a new technology, and platforms are needed to accelerate complex product development. Seeing research results in the medical sector that were certainly worse than the embedded industry as a whole, we wanted to try to figure out how we, as an industry, can address some of the common challenges and outcomes that we're seeing, at least from a schedule perspective.
When we asked about the reasons for delay, embedded development engineers overwhelmingly reported a common theme across their responses: The accelerating rate of change in system goals and complexity.
When we think about that theme, you can imagine that the internal development efforts and resources are simply too stressful for these organizations to keep up with. In today's economy, it's not likely that there's going to be some reinforcement racing in to give you more developers to throw at a challenge or deadline. And then, for many products and projects in the medical sector, you have to layer on the safety-critical certification considerations, to which many of these devices must align. This works to amplify some of the challenges that these organizations face.
Given the noted difficulties of managing change and the deleterious effect it has on project performance, it is critical to find ways to simplify elements of the bill of materials and technology stack. In essence, OEMs must set common technology foundations when they're making their design decisions in order to help accelerate their development. This will hopefully maximize IP reuse in the future.
Already, the long-fragmented embedded market has been consolidating around a smaller set of technology choices and ecosystem. While it's still quite diverse in the grand scheme of things, more OEMs are seeing the need to choose third-party embedded technology. Transitioning to third-party operating systems is something that is long underway in the industry, with this snapshot as an example of what that embedded operating system landscape looked like a dozen years ago:
As you can see, many organizations were still developing their own operating systems back then. Still more were developing devices that maybe had no formal operating system on their product at all. The evolution of system requirements has changed everything.
OEMs were looking for smarter functionality in their systems, recognizing the need for sophisticated OS platforms. In some cases, that meant adding their own and for others, that meant moving to a third-party solution as they couldn't scale their own efforts. Additionally, developers had the desire to focus some of those resources on other stack layers and tap into what the third-party ecosystem offered.
We saw a similar pattern in the processor landscape as well — an area that has been defragmenting over the years.
A great example of this is the change in the processor landscape over that same time period. It's really turned into a two-horse race between ARM and Intel x86. You can see ARM has actually doubled its share and reach while Intel has expanded its reach with other higher-performance solutions. Ultimately, these changes in standardization have helped engineering organizations balance some of their goals for more capabilities while enhancing flexibility for future designs.
Medical device OEMs are redefining the ways they’re selecting the building blocks for their products. A perfect example of this has been the way they’re selecting their hardware.
If you look at this list of the most important characteristics, you can see that software availability and compatibility is the number one hardware selection criterion. This puts us in a position where those tight integrations and cross-domain expertise from hardware to software are becoming the industry's expectation. Feeds and speeds are no longer the deciding factor when selecting hardware, you need to think about so much more.
Embedded development teams really need an ecosystem of easily-integrated solutions that accelerate development and address some of the “bigger picture” challenges they have to navigate when bringing medical devices to market.
Crank Software, Blackberry QNX, and Toradex help organizations find third-party embedded technology platforms, help teams accelerate development, and assist in hardware and software selection.
Very high up on a list of technical complexities is the lack of human resources and the availability of hardware. All of the complexities call for a simplification of hardware design. An excellent way to achieve this without sacrificing flexibility is the use of a system on modules (SOM).
A system on module abstracts the complexity of common hardware elements such as processing, high-speed DDRM, power management, flash memory via network connectivity, and more. Also, most of the software dependencies are encapsulated on the module.
This means you can profit off the work of people you already need to support the software on these modules. SOMs are typically a strong solution if you have project volumes from a few hundred to a 10,000 pieces per year. The module allows you to build a customized port if you're differentiating technology and it lowers the project risk and time to market significantly.
Taking a look at requirements and specification changes, some SOMs also provide advantages. For example, Toradex has three families of modules and within these families, the modules are pin-compatible. This means that if development teams find that they need more performance through network-accelerated ECC-protected RAM, this can be done without a complete redesign.
It’s important to make sure that when you are looking at multiple vendors, you check how pin-compatible the products are.
Moving on to the more important criteria in selection of hardware: The software. Almost 90% of Toradex engineering goes into software although we recognize that not even this covers all client needs. This is why we dedicate resources to create a valuable software ecosystem with our partners, like Blackberry QNX and Crank Software, to solve issues and optimize the user experience behind the curtain. This simplifies the device development and maintenance for clients.
Long product availability is essential in medical devices, as lifecycles are typically much longer. Ruggedization and product durability is also crucial as our safety can directly depend on it. Toradex uses industrial and automotive-grade components, designed for a much longer lifetime than commercial-qualified chips. For example, commercial products are typically qualified for five years but only run for 50% of the time. On the other extreme, industrial products (the majority of what we use) is around 10 years, on 24/7, and also with higher temperatures. Some of our modules include NXP i.MX8QM or i.MX8X application processors.
The SOMs also feature error-correcting memory that increases reliability and makes it easier to comply with functional safety standards. They are also manufactured in a 28 nanometer FD-SOI device process that creates a more resilient chip. It's also a good idea to check with your hardware vendor to see if they do any vibration and shock testing or other tests proving they can withstand your environmental requirements.
Some of the ways to achieve required safety by keeping development efforts include partitioning the system to keep the safety-critical parts and the non-critical parts separate. If you can do that, it's easier to develop and certify the parts that aren't safety critical and reuse components.
An overly-simplified example of the most common approach we see at Toradex is using a dedicated microcontroller, often featuring lockstep.
Lockstep means the code is executed twice and compared. The semantics of the system on modules takes care of the connectivity, provides a beautiful user interface, and can also manage storage.
But make no mistake, even in these use cases it is important that the OS running on the system is secure, especially if your device is connected.
Putting the microcontroller on the same line as the main processor, rather than externally, has gained popularity in the last year, allowing for more flexibility and new use cases. For example, on the NXP i.MX8QuadMax, you can do a CRC check on the frame buffer and there is also a domain controller to make sure different CPU cores can not interfere with each other.
The last way of partitioning the system is by using a virtualization hypervisor. You can split the main CPU cores in different partitions, providing the option to run many different operating systems. This approach has gained traction in the automotive industry. Also, some RTOS types have special hardware features added to support this use case.
Next is selecting a real-time operating system. During the selection process, it's really about finding the right balance. When you look at a system that needs to run, there's the operating system functionality but also all of your applications. Additionally, there's the need for both safety and security. And right in the middle you see that balance — between all three of these domains— making sure you have the right mix of both hardware and software.
In terms of safety, the first consideration is the foundational software: The real-time operating system. This includes the tools that you'll use or anything that touches the code that runs on the target, your compilers, your linkers, any code generators, and libraries of functionality. If there is a third-party library, you want to be able to bring those in to assess and understand what the restrictions or requirements are from the safety use case. You can then look towards the certification standards that the system software has gone through.
Another key consideration for medical devices is the IEC 62304 standard. Development teams should make sure that the software has gone through certification and is going to give you a solid safety basis. From there, use the safety constructs from the RTOS. A lot of these operating systems, and certainly QNX OSs, come with a safety manual that show you how to use the product safely. This includes deploying things like semaphores, mutexes, and message passing, and eliminating any race conditions from your software.
After that, protect code by separating it into both your code segments and your data segments. The term FuSa, or Functional Safety, is a very important concept here, and has to do with the "What ifs." For example:
You could have something that's mission critical, that if it fails, is catastrophic. The ability for the operating system to either repair itself or be able to act accordingly in a safe mode (not just proceed along with its functionality) is crucial.
Let’s compare two OSs: QNX and Microkernel (think of something like Linux in this space). Microkernel is anything where you've combined all the drivers and any of the kernel modules with no restrictions between them. The blue wheel on the outside of the diagram below is about the same between them, because there are protections for the applications.
That's really one of the key differences. If you look at spatial separation, being able to separate out almost everything - QNX does that. Almost every feature of the OS is outside the kernel, it's a very small microkernel that does very specific things. That allows you to achieve some of the safety that you need in the system.
With message passing, there needs to be balance. The safest method is to just do a message pass where you'll pass a message and a copy is made into the new data space. That's not necessarily the most efficient and if you have a high-bandwidth sensor, maybe you need to establish a shared memory region between two applications. You can do that, but make sure it's bounded and built into your safety case. That way your buffer can be shared between the two applications in a safe manner.
At this point, service recovery is a differentiator. If you think about a device driver failing in the system during operation, that could be catastrophic for the system, where you have a kernel panic and it goes down. With QNX, we can detect the failure, assess it, and with our high-availability manager, deconstruct that driver, reconstruct a new driver in its place, and bring it up to state much more quickly than you can in other operating systems. That mitigates any downtime that may be happening while the system is in operation.
In terms of security, we need to be asking questions like:
A lot of things can occur in terms of denial of service, repurposing the device, and access to data that might violate HIPAA compliance. There’s even been cases where access is blocked until a ransom is paid.
We need to make sure that we minimize that attack surface. QNX does this in a series of techniques, like Russian stacking dolls, where you could get through one but then there's the next level and the next level, and so on. That core is on trusted hardware.
Then, look at tamper protection or minimizing access to things that don't need to get access to do different things in the software itself. Using the MMU to its full capability both from a data segment as well as from a hardware perspective, and being able to protect that from an application security standpoint. This is where you can think about something like a motor controller.
The motor controller itself does not need access to the network, whereas other applications do. You can actually lock down certain applications to say "It can talk to these things, maybe it needs to get to the file system, to a set of parameters that it was used to tune the motor controller."
And that's fine, but we're not going to allow access to the internet directly. Then you can rely on the operating system to ensure the temporal and spatial control as part of its course. But make sure that you've minimized access and you don't allow a system to just run away with the processor, either in taking as much memory as it wants or taking as much time as it wants - that could have an adverse effect on the system.
Using a safety manual shows you how to design the software, keeping security in mind, and then doing full requirements testability from the beginning — not the end. This way, when schedules are late, your system is always tested to a known level and you've designed your system with specific goals in mind, not just throwing a bunch of different software around.
It really comes down to choosing the right hardware for your product in your portfolio You want to ensure that you have the right mix. And then match the hardware with the system software from a safety perspective, from a security perspective, and a functionality perspective.
From there, you can leverage these applications across the entire product portfolio.
Nowadays, adding touchscreens to general embedded devices is becoming much more expected. Yet the problem that everybody experiences as soon as they put a touchscreen on their device is the expectation that it should feel like their cellphone. A user expects to be able to scroll, get some animation feedback, and at least a few different types of rich experiences. At the same time, we're seeing this demand in the medical GUI field too, as there's an increased functionality and complexity that's coming in and users want to use them easily. This means transitioning devices that may have been single purpose before to multipurpose with multiple menu layers.
Beyond that, the medical world is now supporting devices for multiple users. It's no longer just trained technicians using these devices, they're actually moving towards home's telehealth, and getting into wearables too.
Because there's a much broader user base interacting with touchscreen embedded devices, the usability and user experience is really high as far as getting approval and being able to use them safely. Underneath all these expectations is the reliability of the system — always of the utmost importance.
Medical devices are not just a touchscreen for people to play games on. In 2015, a test was conducted on 900 software recalls in medical devices that happened between 2012 and 2015. 423 of them were related to the user interface. These could have been input issues, usability issues, or bad feedback. But a lot of that has to do with testing and how you get it to market.
Ventec Life Systems, a recent client of ours, is a primary example of mapping multiple functionalities with their ventilator system. They took five different pieces of a traditional ventilator and brought them together in a single device that's portable and usable by the end user. The challenge? Getting it to the usability of a cell phone because that's what everybody's experiencing today.
Just like other industries, embedded teams in the medical GUI space are getting their graphics from design and marketing teams, bringing them together with development, and testing/validating. The medical GUI space has an extra challenge where they need a deeper level of safety and reliability. When lives are at stake, it's even higher in importance.
Crank Software has been doing this for a long time, through our embedded technology and services. Storyboard is an embedded GUI design and development tool that makes it a lot easier for development teams to bring in graphical assets and work back and forth with their design teams.
It's also about speed and ease of use, because we need to make sure that we can satisfy both the design team and an engineering team. Everybody wants to make sure their branding, their look, and their feel is on point. Even with medical user interface design, animations at a small level need to be effective as this makes the UI clear and easier to use.
The main ways we help is by accelerating the development, dealing with iterations smoothly, and providing flexibility by managing and making it easy to use partners like Toradex and QNX.
Accelerating development is a top priority for us. We help individuals across teams work easier and collaborate better using Storyboard. One of the ways to do that is by taking the graphical assets and bringing them directly into Storyboard, then managing the back and forth between designers and developers. Storyboard pulls in a Photoshop or Sketch file, or any other graphics file from design tools. We also provide animation functionality on an animation timeline. Everything is consistent so both engineers and designers can understand and use the platform with hand offs between them.
For example, if designers gave you a Photoshop file with a whole hierarchy of screens and layers and naming, that is the exact hierarchy that we use inside Storyboard.
In the traditional medical user interface design world, you may not think that animations are an important part of the development process. However, when a user is going through usability feedback, people prefer having visual animations to confirm actions. So, instead of an up and down image as you hit a button, it's a little pop of animation. That's what you’d see on your cell phone experience and you immediately get familiar and rich feedback.
What this helps prevent is users not knowing whether they pressed the button or not. The animation provides just a little bit of feedback after your fingers are pulled away. These things do make a better usability experience right out of the box, just by involving the users and helping them understand where they're going next in the GUI
..
Storyboard also helps take a design and move to hardware as fast as possible. Because the faster you get to hardware, the faster you start seeing your solution. This leads to the next piece: As soon as you get to hardware, that's when the feedback loop between stakeholders starts.
Every market's user feedback is important, your team's feedback, and your experts’ feedback is critical. You want to get doctors, nurses, and other practitioners in front of the GUI as early as possible and make sure that usability is exactly where you need it to be.
Many times, when developers are writing a lot of code to build the user interface, it's very difficult to make changes, and very hard to do things quickly. Sometimes, the engineering team goes away and nobody actually sees the GUI for a long period of time until it's getting close to release, where they can actually start using it and give feedback. By then, so much time and effort has been put into making it that changes and iterations become costly. And we've all learned that the UI always changes.
Knowing this, Storyboard makes it simple to re-import updated design files from Photoshop, Sketch, etc. and does a merge and compare directly. Within minutes, you can update your GUI with a new design file and take in the new imagery, the new positions, and if there's new assets, bring them in to start using right away — without breaking existing code.
This type of rapid iteration makes it much faster to get feedback from all stakeholders and iterate through different changes. Storyboard offers a graphical compare and merge, where you can see full images of your device as they're going in and not have to look at lines of code to see what's changed.
“We continue to rely on Crank Software because it's so easy to update our user interfaces. We're constantly innovating and Storyboard lets us tweak the graphics and optimize the animations with great ease, with minimal impact on our product development cycles." - Ventec Life Systems
The faster you get your medical GUI in place, the more time you have to collect user feedback. If you get your user interface up early and ahead of it, you're protected in a way that you can leverage this feedback process. You can really take advantage of a lot of the pieces discussed here for better development.
To wrap up, the big things are to choose the right hardware, employ the right RTOS, and get to target quickly and validate the GUI on the device. There's so much feedback that comes back immediately, just seeing the performance and whether or not you need to update the application, the system architecture, or the processor.
Above all else, comes the understanding that your GUI is going to evolve and change and you need the right tools to adapt.