NXP Semiconductors MCU-SMHMI-UG User manual

MCU-SMHMI-UG
Smart HMI User Guide
Rev. 0 — 29 September 2022 User guide
Document information
Information Content
Keywords MCU-SMHMI-UG, SLN-TLHMI-IOT, Human Machine Interface (HMI), Smart
Home, IoT
Abstract This document describes the smart Human Machine Interface (HMI) solution,
and its associated out-of-box features. The SLN-TLHMI-IOT turnkey solution
provides OEMs with a fully integrated, self-contained, software, and hardware
solution.

NXP Semiconductors MCU-SMHMI-UG
Smart HMI User Guide
1 Introduction
The SLN-TLHMI-IOT development kit implements NXP’s EdgeReady turnkey solution
for smart Human Machine Interface (HMI) projects. A high-performance member of the
i.MX RT1170 microcontroller (MCU) family enables this solution. It runs on a Real-Time
Operating System (RTOS) under dual-core architecture. One is Arm Cortex-M7 at 1
GHz clock rates and the other is Cortex-M4 at 400 MHz, with 2 MB on-chip SRAM. The
smart HMI solution provides OEMs with a fully integrated, self-contained, software, and
hardware turnkey solution. It includes cross-platform framework, pre-integrated machine
learning vision, voice algorithm, a 2D graphics accelerator supported by Light and
Versatile Graphics Library (LVGL). The SLN-TLHMI-IOT also includes required drivers for
all necessary peripherals, including memories, cameras, mics, speaker, display, and Wi-
Fi + Bluetooth Low Energy combo chipset. A runtime license to this software solution is
bundled with the i.MX RT117H-numbered devices in the i.MX RT1170 family. The solution
enables developers to implement multi-modal smart interactions with Machine Learning
(ML) vision (face and gesture recognition), far-field voice control, and 2D graphical user
interface on their products. This functionality can be enabled by using only a single
high-performance NXP crossover MCU, simplifying the overall system design. NXP’s
cross-platform dual-core framework architecture empowers designers with the flexibility
to customize their products, making i.MX RT117H a versatile fit for numerous smart
HMI applications. These applications include but are not limited to both consumer and
industrial applications.
Target applications
Smart appliance, smart home, smart building, and smart industrial applications that
require smart, intuitive human machine interactions.
•Smart appliances
–Whitegoods (fridge, standing air conditioner, ovens, laundry, and so on)
–Counter-top (coffee machine, purifier food processor, and so on)
•Smart home
–Alarm and smart home control panels
–Thermostats
–Universal remote controls
–Home entertainment
•Smart building, smart industrial
–Elevator
–Industrial HMI
–Healthcare
–Transportation
–Access and Identification
•Remote conference system, and so on
1.1 Processor overview
The i.MX RT117H is an EdgeReady member of the i.MX RT1170 family of crossover
MCUs, targeting low-cost embedded intellectual HMI applications. It features advanced
implementation of the Arm Cortex-M7 core from NXP. The i.MX RT117H operates at
speed up to 1 GHz and has a power-efficient Cortex-M4 core up to 400 MHz to provide
high-performance and real-time response. The i.MX RT117H has available CPU and
memory resources and many integrated peripherals. These features offer customers
MCU-SMHMI-UG All information provided in this document is subject to legal disclaimers. © 2022 NXP B.V. All rights reserved.
User guide Rev. 0 — 29 September 2022
2 / 51

NXP Semiconductors MCU-SMHMI-UG
Smart HMI User Guide
the maximum flexibility to customize and define products for a multitude of different use
cases.
The i.MX RT117H MCU is licensed to run NXP’s facial biometric authentication library,
audio front-end runtime library, and ML speech engine runtime library, which includes:
•Unified cross-platform framework
•Camera drivers, image capture, and pre-processing
•Face detection, tracking, alignment, and recognition (with quantified results and
confidence measure)
•Voice seeker Audio Front-End (AFE) for Acoustic Echo Cancellation (AEC), Noise
Reduction (NR), beamforming, and barge-in
•Conversa for full duplex speakerphone
•Cyberon or Voice Intelligent Technology (VIT) for automatic speech recognition (wake
words and commands)
•Display drivers, 2D graphics accelerator supported by LVGL
•Built-in security, bootloader, and application validation
•All drivers, including Bluetooth Low Energy and Wi-Fi
•USB mass storage device updates
•Support secure firmware update over a wireless connection
•Factory automation scripts
•Supported by an MCUXpresso SDK, MCUXpresso IDE, and configuration tools
1.2 Voice library limitations
Voice seeker AFE and Cyberon ASR libraries are provided with our SLN-TLHMI-IOT
projects as evaluation versions.
Note: Voice seeker AFE stops working after 25 hours, while Cyberon ASR stops working
after 100 detections.
To have voice detection work again, a power cycle is required when either one of these
thresholds is met.
Production versions of the libraries can be shared upon request.
2 Recommended configuration
Modifying and debugging the SLN-TLHMI-IOT example projects requires an up-to-date
computer that can run MCUXpresso IDE version 11.5.0.
Note: Both the 11.5.1 and 11.6.0 versions have bugs preventing the SLN-TLHMI-IOT
development experience from working as intended.
Computer type OS version Terminal program
Apple MAC OS PuTTY
PC Windows 7/10/11 PuTTY/Tera Term
PC Linux PuTTY
Table 1. Recommended computer configuration
MCU-SMHMI-UG All information provided in this document is subject to legal disclaimers. © 2022 NXP B.V. All rights reserved.
User guide Rev. 0 — 29 September 2022
3 / 51

NXP Semiconductors MCU-SMHMI-UG
Smart HMI User Guide
3 Hardware
The SLN-TLHMI-IOT turnkey solution for combined face and voice recognition comes
with a cost and form-factor-optimized hardware reference designs and access to full
software source code. In addition to the i.MXRT117H MCU, the board employs different
sensors and peripherals to provide tools to help develop applications with a wide array
of functions. Many of the onboard sensors and peripherals can be seen in Figure 1 and
Figure 2.
Figure 1. SLN-TLHMI-IOT kit callout
Figure 2. SLN-TLHMI-IOT main board callout
CAUTION: Depending on the mode of operation, the SLN-TLHMI-IOT kit can emit highly
concentrated white or non-visible infrared light which can be hazardous to human eyes.
Products which incorporate these devices must follow the safety precautions given in IEC
60825-1 and IEC 62471.
MCU-SMHMI-UG All information provided in this document is subject to legal disclaimers. © 2022 NXP B.V. All rights reserved.
User guide Rev. 0 — 29 September 2022
4 / 51

NXP Semiconductors MCU-SMHMI-UG
Smart HMI User Guide
For more information about the hardware design of the SLN-TLHMI-IOT, see the Smart
HMI Hardware Development User Guide.
3.1 Screen
The SLN-TLHMI-IOT kit, by default, uses a 5.5 inch TFT display (Rocktech
RK055MHD091A0-CTG) with 720*1280 resolution, LED backlight, full viewing angle,
MIPI interface, and a capacitive touch panel.
Datasheet: Rocktech RK055MHD091A0-CTG
Purchase link: Rocktech RK055MHD091A0-CTG
3.2 Speaker
A PUI ASE02808MR-150-R enclosed 8 Ω 2 W speaker is embedded in the SLN-TLHMI-
IOT kit.
Datasheet: ASE02808MR-LW150-R
Purchase link: ASE02808MR-LW150-R
3.3 Cameras
The SLN-TLHMI-IOT kit uses a GC2145 camera sensor by default. The GC2145 is a
high-quality 2MP CMOS image sensor. It incorporates a 1616 V x 1232 H active pixel
array, on-chip 10-bit ADC, and an image signal processor. The camera sensor connects
to the kit via a Mobile Industry Processor Interface (MIPI) interface. Additionally, the SDK
associated with the SLN-TLHMI-IOT has software support for both the device driver and
Hardware Abstraction Layer (HAL) driver for the sensor.
The stock firmware utilizes the camera sensor to register face data and recognize faces.
In addition to the GC2145 camera sensor, alternative image sensors are available for
use, including the GC0308 sensor. The default firmware does not support the GC0308
sensor. Enabling this camera requires minor code modifications to enable the respective
HAL device driver in the firmware. The GC0308 sensor connects to the kit using a CSI-
DVP interface.
3.4 Wireless radios
Usage condition
The following information is provided per Article 10.8 of the Radio Equipment Directive
2014/53/EU:
(a) Frequency bands in which the equipment operates
(b) The maximum RF power transmitted
PN RF technology (a) Frequency range (b) Maximum
transmitted power
SLN-TLHMI-IOT Bluetooth 2402-2483 MHz 4 dBm
Table 2. Bluetooth/WLAN frequency and power
MCU-SMHMI-UG All information provided in this document is subject to legal disclaimers. © 2022 NXP B.V. All rights reserved.
User guide Rev. 0 — 29 September 2022
5 / 51

NXP Semiconductors MCU-SMHMI-UG
Smart HMI User Guide
PN RF technology (a) Frequency range (b) Maximum
transmitted power
WLAN 2.4 GHz ISM Bands 2.412-2.472
GHz
5.15-5.25 GHz (FCC UNII-low
band) for US/Canada and Europe
5.25-5.35 GHz (FCC UNII-middle
band) for US/Canada and Europe
5.47-5.725 GHz for Europe
5.725-5.825 GHz (FCC UNII-high
band) for US/Canada
18.5 dBm
Table 2. Bluetooth/WLAN frequency and power...continued
EUROPEAN DECLARATION OF CONFORMITY (Simplified DoC per Article 10.9 of
the Radio Equipment Directive 2014/53/EU)
This apparatus, namely SLN-TLHMI-IOT, conforms to the Radio Equipment Directive
2014/53/EU. The full EU Declaration of Conformity for this apparatus can be found at:
www.nxp.com/mcu-smhmi.
4 Bootloader
The SLN-TLHMI-IOT uses a multi-stage boot process to enhance the capabilities of the
out-of-box firmware (FW) to include additional functionality. Such as the ability to flash
new firmware images remotely and without a debugging probe.
The bootloader operates as the first stage of the boot process, determining which
available out-of-box demo applications (coffee machine and elevator) to use, if any.
The bootloader, in addition to choosing which of the default out-of-box applications to
use (Normal Boot mode), also supports an additional boot mode: Mass Storage Device
(MSD) update mode. The following sections describe each of these different boot modes
in more detail.
4.1 Boot modes
The bootloader supports several different boot-up methods which augment the boot-up
behavior. These methods include Normal Boot mode (default) and Mass Storage Device
(MSD) mode for drag-and-drop flashing of new firmware images. The following sections
describe the usage of each different boot mode and how to enable them.
A high-level overview of the purpose of each of these features is discussed below.
Information regarding their usage, creating update payloads, and more can be found in
the Smart HMI Software Development User Guide.
4.1.1 Normal boot
By default, if no boot flags are set during the boot phase, the Normal Boot mode is used.
During Normal Boot, the bootloader simply boots into the "main" out-of-box demo
application that is most recently used. For example, if the coffee machine application is
used most recently, the bootloader jumps to the flash address associated with the coffee
machine application. Similarly, if either the elevator or smart home panel application
is used most recently, the bootloader jumps to the flash address associated with the
elevator or smart home panel application. If powering the kit for the first time and no
MCU-SMHMI-UG All information provided in this document is subject to legal disclaimers. © 2022 NXP B.V. All rights reserved.
User guide Rev. 0 — 29 September 2022
6 / 51

NXP Semiconductors MCU-SMHMI-UG
Smart HMI User Guide
previous application has been selected, the bootloader requires an application to be
manually selected first.
The OoBE FW has a set of three different applications that can be booted into at startup.
To select an application for the bootloader to use, while the kit is being powering on,
press and hold one of the onboard push buttons on the back of the kit. The switch
corresponding to each application is shown below:
•SW1 – Coffee Machine
•SW2 – Elevator
4.1.2 Mass Storage Device (MSD)
The MSD feature allows the SLN-TLHMI-IOT to receive firmware updates without a
debugging probe like a SEGGER J-Link. Instead MSD uses USB to emulate an MSD
interface like the one that is used for USB flash drives.1 This feature can be especially
useful for marketers or engineers in the field without access to a dedicated debug probe
tool.
MSD mode is enabled by pressing and holding the SW0 pushbutton on the back of the kit
while the kit is powering on.
Once enabled, a blinking purple LED on the front of the kit which pulsates at an interval
of roughly 1 second indicates MSD mode.
1Currently only supported on Windows PC devices running Windows 7 or newer
MCU-SMHMI-UG All information provided in this document is subject to legal disclaimers. © 2022 NXP B.V. All rights reserved.
User guide Rev. 0 — 29 September 2022
7 / 51

NXP Semiconductors MCU-SMHMI-UG
Smart HMI User Guide
Figure 3. Purple LED indicating MSD mode
Additionally, the PC recognizes the USB interface, assuming your SLN-TLHMI-IOT is
connected to a Windows PC. It shows a new USB Drive corresponding to your SLN-
TLHMI-IOT kit.
Figure 4. New USB drive for SLN-TLHMI-IOT kit
To update the firmware image on your kit, drag and drop the new binary to the USB drive
corresponding to your kit. Assuming the binary is properly generated, the bootloader
automatically updates the firmware on your device. Flashing a new firmware image
results in a pop-up window identical to the one used when copying files to a real USB
flash drive device.
MCU-SMHMI-UG All information provided in this document is subject to legal disclaimers. © 2022 NXP B.V. All rights reserved.
User guide Rev. 0 — 29 September 2022
8 / 51

NXP Semiconductors MCU-SMHMI-UG
Smart HMI User Guide
Figure 5. Copying new binary to the USB drive
Note: Only the main application (for example, coffee machine or elevator) can be
updated using MSD. The bootloader cannot be updated using MSD and requires the use
of a SEGGER J-Link Debug Probe or the Factory Programming Flow. Instructions MSD,
J-Link-based updates, and the Factory Programming Flow can be found in the Smart
HMI Software Development User Guide.
For more information regarding MSD mode and its usage, as well as instructions on
generating FW binaries compatible with MSD mode, see the Smart HMI Software
Development User Guide.
5 Out-of-box demo applications
5.1 Overview
There are two distinct applications that come pre-flashed with your kit: Coffee Machine
and Elevator. These example applications are built to showcase the complex machine
vision, machine voice, and advanced graphical UI capabilities the kit supports. They also
provide a base software platform for rapid integration of unique customer applications
and requirements.
The Coffee Machine application presents an example of a potential smart HMI use
case. Here, an HMI device must operate using hands-free voice control capabilities while
providing custom user experiences using face recognition to recognize repeat users of
the machine.
The Elevator application presents similar capabilities to the coffee machine demo by
providing an example of a smart HMI device. This device operates using hands-free
voice control capabilities to interact with an elevator, selecting the floor number, opening
and closing the elevator doors, and so on. Additionally, the elevator demo application
automatically recognizes new users and asks to save the floor number they traveled to.
When powering on your kit, a brief startup screen appears which shows icons for each
available out-of-box demo application. The icon associated with the currently enabled
application is most prominently displayed using a slightly larger icon.
MCU-SMHMI-UG All information provided in this document is subject to legal disclaimers. © 2022 NXP B.V. All rights reserved.
User guide Rev. 0 — 29 September 2022
9 / 51

NXP Semiconductors MCU-SMHMI-UG
Smart HMI User Guide
Figure 6. Startup screen
Each of the out-of-box demo applications is described in more depth in the following
sections.
5.2 Coffee machine
5.2.1 Overview
The coffee machine application, preprogrammed in the SLN-TLHMI-IOT kit, presents
a demo application. This application integrates both face and multi-language speech
recognition while showcasing the advanced graphics capabilities of the RT117H’s 2D
GPU as well.
In the demo, users can select their coffee preferences using onscreen touch controls
and hands-free voice control. After selecting their preferred coffee, they can save their
order by anonymously associating it with their face. The next time that face is recognized,
they can either choose a new coffee selection or automatically use their saved coffee
selection.
The following sections describe the out-of-box features of the coffee machine demo and
their usage.
5.2.2 Standby screen
After exiting the startup screen, a standby screen is displayed, which acts as a screen
saver to indicate that the system is waiting for user interaction. The standby screen
is shown in a few different scenarios, including: after startup, after coffee has finished
brewing, and after a ~60 second timeout with no user interaction (voice commands/touch
controls) has been reached.
MCU-SMHMI-UG All information provided in this document is subject to legal disclaimers. © 2022 NXP B.V. All rights reserved.
User guide Rev. 0 — 29 September 2022
10 / 51