ST AEKD-AICAR1 User manual

Introduction
The AEKD-AICAR1 is a versatile deep learning system based on a long-short term memory (LSTM) recurrent neural network
(RNN). It is able to classify the car state:
•car parked
• car driving on a normal condition road
• car driving on a bumpy road
• car skidding or swerving
The main idea is to define an AI-car sensing node ECU with an embedded artificial intelligence processing.
The system hosts an SPC58EC Chorus 4M microcontroller, which is able to acquire discrete acceleration variations on a
three-axis reference system.
The AIS2DW12 motion sensor mounted on the AEK-CON-SENSOR1 board retrieves inertial data. The acquired data are
transmitted to the LSTM RNN, which classifies the car state. The classification result is shown on the AEK-LCD-DT028V1 LCD
touch display.
The LSTM RNN has been implemented and trained using the TensorFlow 2.4.0 framework (Keras) in the Google Colab
environment. The AI-SPC5Studio plug-in has been used to convert the resulting trained neural network into an optimized C
code library, which can run on an MCU with limited power computing resources.
Figure 1. AEKD-AICAR1 evaluation kit
The LSTM RNN training has been performed with several time-series acceleration waveforms recorded on a real vehicle in
motion. The resulting prediction accuracy, calculated by the confusion matrix, is about 93%. Field tests carried-out under all road
conditions with a sedan confirm the adherence of the computed results compared with the real road conditions.
Note: The AEKD-AICAR1 is an evaluation tool for R&D laboratory use only. It is not destined for use inside a vehicle.
Getting started with the AEKD-AICAR1 evaluation kit for car state classification
UM3053
User manual
UM3053 - Rev 1 - September 2022
For further information contact your local STMicroelectronics sales office. www.st.com

1Neural network basic principles
1.1 Artificial neural network
The artificial neural network works as the human brain neural network. Data are transferred to the neuron through
the inputs. Then, they are sent as an output after processing.
Figure 2. Artificial neural network
Artificial neural networks are built on layers of different neural units. Each unit consists of three parts:
• an input part that receives the data
• a hidden part that uses the neuron weight to calculate the result
• an output part that receives the calculation results and applies an eventual bias
The weight associated with each neuron determines its firing probability. The bias is the measure of assumption
made by the form of the output.
This architecture is typical for deep learning processes like data classification and pattern recognition neural
networks.
1.2 Long short-term memory recurrent neural network (LSTM RNN)
Traditional artificial neural networks keep no memory of what happened in the past. They take their decision only
on the data provided instant by instant. These architectures are well suited for data classification and pattern
recognition.
For other applications, like speech recognition, the proper classification requires a memory of the context, that is,
the prior words in the speech recognition application.
A recurrent neural network (RNN) is a class of artificial neural network that includes neurons connected in a loop.
UM3053
Neural network basic principles
UM3053 - Rev 1 page 2/39

Figure 3. Recurrent neural network
In most cases, the output values of an upper layer are used as the input for a lower level one. This
interconnection enables the use of one of the layers as the state memory and allows modeling a dynamic time
behavior dependent on the information previously received.
A typical recurrent neural network able to keep the information received is the long short-term memory (LSTM)
recurrent network. Unlike traditional artificial neural networks, LSTM has feedback connections. It can process not
only single data points but also entire sequences of data. For this reason, the LSTM neural network architecture is
the best candidate to model a deep learning system for an AI-car sensing node, using a collection of time-series
based on acceleration values.
UM3053
Long short-term memory recurrent neural network (LSTM RNN)
UM3053 - Rev 1 page 3/39

2Designing an AI-car sensing node
2.1 Tool-set introduction
An AI-car sensing node is an AI-deep learning system based on an LSTM recurring neural network, which can
provide a car state classification: parking, driving on a normal condition road, driving on a bumpy road, and car
skidding or swerving.
Figure 4. AI-car sensing node: car state classification
The LSTM RNN has been modeled with TensorFlow (Keras framework). This is an open-source software library
for machine learning, which provides optimized modules to implement AI algorithms related to the classification
problem.
A significant amount of computing power is required to implement sufficiently robust and efficient models. For
initial tests, we can rely on a standard machine. As the dataset size increases, the execution of complex training
algorithms becomes rapidly prohibitive.
To address this issue, there are several cloud services that offer computing power. Google Colab is an alternative
platform, which allows running the code directly on the cloud, even if with some limitations.
2.2 Creating a Google Colab notebook
To use the Google Colab platform, you need a Google account to login. After logging in, create a new “Colab
notebook” project file.
UM3053
Designing an AI-car sensing node
UM3053 - Rev 1 page 4/39

Figure 5. Project file
The document that you are creating is not a static web page. It is an interactive environment that lets you write
and execute your Python code to implement an LSTM RNN by using the TensorFlow framework.
2.3 Colab notebook setup and package importing
To implement and train an LSTM RNN, install the Tensorflow 2.4.0 framework on your Google Colab notebook by
using the packet index package (PIP) command.
%pip install tensorflow==2.4.0
Note: PIP is already installed on your Google Colab notebook.
Using the Python commands, import the following packages:
•Pandas
Open-source software library for data manipulation and analysis.
import pandas as pd
•Numpy
Open-source software library, adding support for large, multidimensional arrays and matrices, along with a
large collection of high-level mathematical functions to operate on these arrays.
import numpy as np
•Tensorflow
Open-source software library for machine learning, which provides optimized modules to implement artificial
intelligence (AI) algorithms.
import tensorflow as tf
•Sklearn
A package providing several common utility functions and transformer classes to change raw feature vectors
into a representation that is more suitable for the downstream estimators.
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.model_selection import train_test_split , StratifiedShuffleSplit
from sklearn.metrics import confusion_matrix
•OS
The Python language OS module has several useful functions to make the program interact with the
computer operating system.
import os
UM3053
Colab notebook setup and package importing
UM3053 - Rev 1 page 5/39

•Random
This module implements pseudo-random number generators for various distributions.
import random
•Seaborn
Seaborn is a Python data visualization library based on Matplotlib. It provides a high-level interface to draw
attractive and informative statistical graphics.
import seaborn as sn
•Matplot
Matplotlib is a comprehensive library to create static, animated, and interactive visualizations in Python.
import matplotlib.pyplot as plt
2.4 AI-car sensing node life cycle
The following steps define the life cycle of the AI-car sensing node implementation as a deep learning model for
classification:
1. Model definition
2. Model training
3. Model fitting and compilation
4. Model evaluation
2.4.1 Model definition
To define the model, we have chosen the topology of the LSTM network. The AI-car sensing node network is
based on the same network architecture of a speech recognition system. It consists of a neural convolution kernel
as an input layer, able to elaborate a convolution function with the input vector over a temporal dimension.
The input vector is a time sequence of TIMESERIES_LEN number of samples, which consist of discrete
acceleration variations on a three-axis (x, y, z) reference system: Δax, Δay, and Δaz.
TIMESERIES_LEN represents the minimum size of the temporal window for the car state classification. With an
acquisition sampling time equal to 100 msec, the value of TIMESERIES_LEN has been fixed to 50 samples (5
seconds per acquisition).
A dense function implements the output layer. This function can provide an output shape of four-dimensional
vectors (one for each expected status: parking, normal, bumpy, skid). The Softmax function activates this layer
and converts the output vector values to a probability distribution.
From an implementation perspective, this involves a model layer architecture built with a connection topology into
a cohesive model:
UM3053
AI-car sensing node life cycle
UM3053 - Rev 1 page 6/39

## Conv1D based model
model = tf.keras.models.Sequential([
tf.keras.layers.Conv1D(filters=16, kernel_size=3, activation='relu',
input_shape=(TIMESERIES_LEN, 3)),
tf.keras.layers.Conv1D(filters=8, kernel_size=3, activation='relu'),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(4, activation='softmax')
])
_________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv1d (Conv1D) (None, 48, 16) 160
conv1d_1 (Conv1D) (None, 46, 8) 392
dropout (Dropout) (None, 46, 8) 0
flatten (Flatten) (None, 368) 0
dense (Dense) (None, 64) 23616
dense_1 (Dense) (None, 4) 260
=================================================================
Total params: 24,428
Trainable params: 24,428
Non-trainable params: 0
As described in the above code block table, 24,428 trainable parameters define the AI-car sensing node network
model. These parameters are related to the LSTM RNN topology chosen.
2.4.2 Model training
The AI-car sensing node network training has been performed by acquiring datasets of acceleration variations in
the time domain:
•over a three-axis reference system (Δax, Δay, Δaz)
• with a frequency equal to 10 Hz
• a sample time of 100 msec
Each dataset targets a specific car state: parking, normal driving, bumpy driving or skidding.
The use of acceleration variations instead of raw acceleration data avoids misinterpretations of the car state. If
the car is parked on a road with a negative (or positive) slope (as shown in the figure below), the IMU registers
a nonzero X-axis acceleration that would not lead to the parking state. Instead, if we consider an acceleration
variation in the time domain, the result would be 0, leading to the correct parking state.
UM3053
AI-car sensing node life cycle
UM3053 - Rev 1 page 7/39

Figure 6. Acceleration variations
Car speed Vx = 0 Km/sec;
Car acceleration ax! = 0;
Temporal variation of acceleration Δax = 0
2.4.2.1 Training dataset
The acquisition of training datasets can be performed using a specific configuration of the AI-car sensing node
mock-up (acquisition mode). A proper source code #define enables the acquisition mode. In this mode, the
application sends each acceleration sample read from the sensor to a serial port with a baud rate set to 38400.
The serial data stream can be read through a common serial client (Tera Term or Putty). The data acquisition and
parsing in a CSV file are managed via a specific Python script (see Section Appendix C ).
The following figure shows an example of a time-based dataset generated with the Python script.
UM3053
AI-car sensing node life cycle
UM3053 - Rev 1 page 8/39

Figure 7. Time-based dataset example
The time column contains the recorded acceleration time samples. The other three columns contain the
acceleration values measured on the X, Y, and Z axis, respectively (without the gravity offset).
To prepare the file for the network training, you have to:
• Compute the acceleration variations (excel cell operation: B3 = B3–B2 for X axis).
UM3053
AI-car sensing node life cycle
UM3053 - Rev 1 page 9/39

• Add a status column for the car state for each specific acquisition (P = parking; N = normal; B = bumpy; S=
skid).
Figure 8. Adding the status column
• Create a CSV file for each car state through a specific acquisition task.
Figure 9. Creating CSV files
• Merge all CSV files by coping and pasting each row without changing the time column values.
• Add a new dummy row at the end of the file with the time value equal to 100.
• Add random values for Ax, Ay, and Ax status.
Note: A ready-to-use dataset (Diff_profile.csv) has been created to train the network for the car state classification.
2.4.2.2 Neural network training with Google Colab
Import the training dataset contained in the CSV file into the Colab notebook environment. Then, run a parser
function on the imported data to build an input vector compliant with the LSTM RNN.
The following code block shows the Google Colab script, which imports and loads the CSV file.
from google.colab import files
uploaded = files.upload()
db = pd.read_csv('Diff_profile.csv',sep=',')
Parse each column of the CSV file (status, Acc_x, Acc_y, Acc_z, time).
UM3053
AI-car sensing node life cycle
UM3053 - Rev 1 page 10/39

states = db['Status'].value_counts()
ts_status = db.Status
ts_diff_Ax = (db.Acc_x.to_numpy().reshape(-1,1))/9.81
ts_diff_Ay = (db.Acc_y.to_numpy().reshape(-1,1))/9.81
ts_diff_Az = (db.Acc_z.to_numpy().reshape(-1,1))/9.81
ts_time = db['Time']
Use a simple script to reshape the dataset to be compliant with the shape of the LSTM input vector (50 samples
of a three-dimensional vector). The number of LSTM input vectors is computed from the total number of samples
acquired in all the states, choosing window waveforms of 50 samples.
If for the parking state we have 350 samples, 350/50 = 7 window waveforms are generated.
TIMESERIES_LEN = 50
Y_LABELS={'P':0,'N':1,'B':2,'S':3}
def trip_framing(trip,label,frame_size,db_x,db_y):
a = np.array( trip )
for i in np.arange( 0, a.shape[0]-frame_size, frame_size ):
x = a[i:i+frame_size]
db_x.append( x )
db_y.append( Y_LABELS[label] )
rows = ts_status.shape[0]
db_x = []
db_y = []
for states_id in states.keys():
trip = []
cnt = 0
for i in range(rows):
if ts_time[i] == 100:
if len(trip) > 0:
trip_framing( trip, states_id, TIMESERIES_LEN, db_x, db_y)
trip=[]
if ts_status[i] == states_id and cnt < 7500:
trip.append( [ts_diff_Ax[i],ts_diff_Ay[i],ts_diff_Az[i]] )
cnt += 1
Note: The cnt variable contains the number of available window waveforms according to the dataset acquired during
each specific car status.
2.4.3 Model fitting and compilation
Firstly, for model fitting, select the training configuration:
•Training test percentage:
– Percentage of the window waveform available over the total, used to perform the training task.
• Batch size:
– Percentage of the window waveform available over the total, used to perform the estimation of the
network accuracy (model error).
• Number of epochs:
– Number of loops through the training dataset.
x_train, x_test, y_train, y_test = train_test_split(db_x, db_y, test_size=0.4,
random_state=21,stratify=db_y)
x_train = np.asarray(x_train)[:,:,:,0]
x_test = np.asarray(x_test)[:,:,:,0]
y_train = np.asarray( y_train )
y_test = np.asarray( y_test )
db_stats = pd.Series( y_test )
To train the model, an algorithm is required to minimize the accumulated errors in identifying the correct car state.
This algorithm consists of a loss function to be passed to the model.compile API to generate the complete
runnable code.
The chosen loss function is the sparse categorical cross-entropy able to correlate the state label (for
example, “P” with the neural network prediction).
UM3053
AI-car sensing node life cycle
UM3053 - Rev 1 page 11/39

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=1000)
Fitting and compilation for the RNN generate a complete model. This procedure is quite slow. It can take few
seconds up to days, depending on the model complexity and the training dataset size.
For the AI-car sensing node, set 1000 epochs and 60% of the available window waveform used for the training.
This results in a computation time of around 70 seconds.
Figure 10. Complete model generation
2.4.4 Model evaluation
To evaluate the model accuracy, choose a holdout window waveform. This should be based on data not used in
the training process, to get an unbiased estimation of the model performance when making a prediction on “new”
data.
The model evaluation speed is proportional to the amount of data used for it, but faster than the training phase.
A confusion matrix performs the model accuracy evaluation. This is a technique to summarize the algorithm
classification performance. As the datasets have different sizes, using a pure classification accuracy could be
misleading. Therefore, calculating a confusion matrix can give you a better idea of what your classification model
is getting right and what miss classification is obtaining.
Use the following code to build a confusion matrix.
Y_pred = model.predict(x_test)
y_pred = np.argmax(Y_pred, axis=1)
confusion_matrix = tf.math.confusion_matrix(y_test, y_pred)
plt.figure()
sns.heatmap(confusion_matrix,
annot=True,
xticklabels=Y_LABELS,
yticklabels=Y_LABELS,
cmap=plt.cm.Blues,
fmt='d', cbar=False)
plt.tight_layout()
plt.ylabel('True label')
plt.xlabel('Predicted label')
plt.show()
The confusion matrix for the AI-car sensing node is:
•26 actual parking window waveforms used for the evaluation have been predicted as parking (no miss
classifications).
UM3053
AI-car sensing node life cycle
UM3053 - Rev 1 page 12/39

• 36 actual normal window waveforms used for the evaluation have been predicted as normal, whereas four
have been classified as bumpy.
• 18 actual bumpy window waveforms used for the evaluation have been predicted as bumpy (no miss
classifications).
• Only one actual skid window waveform has been recognized as skid, whereas three have been confused
with bumpy.
Figure 11. Confusion matrix
The LSTM RNN accuracy for the AI-car sensing node is about 93%. The miss classification for the skid state
impact the calculated accuracy value, due to the very low number of window waveforms used for training (only
four).
Despite the above considerations, we consider the accuracy value acceptable and defer a new accuracy
calculation for the skid state to the field tests to perform on the sedan.
UM3053
AI-car sensing node life cycle
UM3053 - Rev 1 page 13/39

3AutoDevKit ecosystem
3.1 SPC5-STUDIO-AI plugin
The LSTM AI-car sensing node network implemented with the TensorFlow framework inside the Google Colab
environment is then converted in a C library compliant with SPC5-STUDIO.
On the Google Colab notebook, the LSTM AI-car sensing node network architecture is saved as an *.h5 file. This
file is then fed into SPC5-STUDIO-AI to obtain a fast and optimized version of the RNN.
model.save('model_car_sts.h5')
SPC5-STUDIO-AI is the artificial intelligence plug-in for SPC5-STUDIO able to generate automatically a
pretrained neural network into an efficient “ANSI C” library to be compiled, installed, and executed on SPC58
microcontrollers.
You can easily import pretrained neural networks in the SPC5-STUDIO-AI from the most widely used deep
learning frameworks, such as Keras, TensorFlow Lite, Lasagne, Caffe, ConvNetJS, and ONNX.
SPC5-STUDIO-AI provides validation and performance analysis features to validate and characterize the
converted neural network and measure key metrics such as validation error, memory requirements (flash memory
and RAM), and execution time.
This plugin is integrated within the SPC5-STUDIO (version 6.0.0 or higher) development environment.
To install the SPC5-STUDIO-AI, follow the procedure below.
Step 1. Select [Install new software] from [Help].
Figure 12. Installing the plugin
UM3053
AutoDevKit ecosystem
UM3053 - Rev 1 page 14/39

Step 2. Insert http://ai.spc5studio.com in the [Work with] field.
Figure 13. Install new software
Step 3. Click on [Select all] and [Next].
Step 4. Install SPC5-STUDIO-AI.
UM3053
SPC5-STUDIO-AI plugin
UM3053 - Rev 1 page 15/39

3.1.1 How to import SPC5-STUDIO-AI using the standard importing procedure
Step 1. Create a new SPC5-STUDIO application based on the SPC58ECxx platform.
Figure 14. Creating a new SPC5-STUDIO application
Step 2. Import the following components:
–AutodevKit Init Package
–SPC58ECxx Init Package Component RLA
Step 3. Select [SPC58ECxx Platform Component RLA].
Figure 15. Selecting the component
UM3053
SPC5-STUDIO-AI plugin
UM3053 - Rev 1 page 16/39

Step 4. Click on [+] to add a new component.
Figure 16. Adding a new component
Step 5. Select [SPC5 AI Component RLA].
Figure 17. Selecting [SPC5 AI Component RLA]
3.1.2 How to import the pretrained LSTM neural network
The following procedure shows how to import the pretrained LSTM neural network for the AI-car sensing node.
UM3053
SPC5-STUDIO-AI plugin
UM3053 - Rev 1 page 17/39

Step 1. Select the [SPC5 AI Component RLA] in the project explorer.
A new window is opened with the network list imported.
Figure 18. Network list imported
Step 2. Add a new network by clicking on [+].
Figure 19. Adding a new network
Step 3. Double-click on the new network preconfigured with the default option.
Step 4. Configure the parameters for the LSTM AI-car sensing node.
Figure 20. Configuring the parameters
Step 4a. Enable the network initialization data.
Step 4b. Put a valid neural network name.
Step 4c. Select the framework used for the training (Keras).
Step 4d. Select a compression equal to 1 (it indicates the expected global factor of compression
applied to dense layers).
Step 4e. Select a valid model file path that contains the *.h5 file.
UM3053
SPC5-STUDIO-AI plugin
UM3053 - Rev 1 page 18/39

3.1.3 How to analyze the pretrained LSTM neural network
To analyze the pretrained LSTM neural network for the AI-car sensing node, follow the procedure below.
Step 1. Select [Analyze] in the [Outline] tab.
Figure 21. Selecting [Analyze]
UM3053
SPC5-STUDIO-AI plugin
UM3053 - Rev 1 page 19/39

Step 2. Click on [Analyze] in the newly opened window.
Figure 22. Clicking on [Analyze]
If the importing procedure for the LSTM AI-sensing node is correct, a new report is generated. This
new report shows the architecture of the neural network and the ROM and RAM memory usage.
Neural Network Tools for STM32 v1.4.0 (AI tools v5.2.0)
-- Importing model
-- Importing model - done (elapsed time 0.574s)
-- Rendering model
-- Rendering model - done (elapsed time 0.077s)
Creating report file C:\SPC5Studio-6.0\workspace_AI\SPC58ECxx_RLA AI Car Sensing
Node\components\spc5_ai_component_rla\cfg\sensing_node_network_analyze_report.txt
Exec/report summary (analyze dur=0.65s err=0)
------------------------------------------------------------------------------------------------------------------------
model file : C:\SPC5Studio-6.0\workspace_AI\SPC58ECxx_RLA AI Car Sensing Node\source\model\model_car_sts.h5
type : keras (keras_dump) - tf.keras 2.4.0
c_name : sensing_node_network
compression : None
quantize : None
workspace dir : C:\SPC5Studio-6.0\workspace_AI\SPC58ECxx_RLA AI Car Sensing Node\components\stm32ai_ws
output dir : C:\SPC5Studio-6.0\workspace_AI\SPC58ECxx_RLA AI Car Sensing
Node\components\spc5_ai_component_rla\cfg
model_name : model_car_sts
model_hash : 10794f1c230799b2a1cda67171827db2
input : input_0 [150 items, 600 B, ai_float, FLOAT32, (50, 1, 3)]
inputs (total) : 600 B
output : dense_1_nl [4 items, 16 B, ai_float, FLOAT32, (1, 1, 4)]
outputs (total) : 16 B
params # : 24,428 items (95.42 KiB)
macc : 49,668
weights (ro) : 97,712 B (95.42 KiB)
activations (rw) : 3,136 B (3.06 KiB)
ram (total) : 3,752 B (3.66 KiB) = 3,136 + 600 + 16
------------------------------------------------------------------------------------------------------------------------
id layer (type) output shape param # connected to macc rom
------------------------------------------------------------------------------------------------------------------------
0 input_0 (Input) (50, 1, 3)
conv1d (Conv2D) (48, 1, 16) 160 input_0 7,696 640
conv1d_nl (Nonlinearity) (48, 1, 16) conv1d
------------------------------------------------------------------------------------------------------------------------
1 conv1d_1 (Conv2D) (46, 1, 8) 392 conv1d_nl 18,040 1,568
conv1d_1_nl (Nonlinearity) (46, 1, 8) conv1d_1
------------------------------------------------------------------------------------------------------------------------
3 flatten (Reshape) (368,) conv1d_1_nl
------------------------------------------------------------------------------------------------------------------------
4 dense (Dense) (64,) 23,616 flatten 23,552 94,464
dense_nl (Nonlinearity) (64,) dense 64
------------------------------------------------------------------------------------------------------------------------
5 dense_1 (Dense) (4,) 260 dense_nl 256 1,040
dense_1_nl (Nonlinearity) (4,) dense_1 60
------------------------------------------------------------------------------------------------------------------------
model_car_sts p=24428(95.42 KBytes) macc=49668 rom=95.42 KBytes ram=3.06 KiB io_ram=616 B
Complexity per-layer - macc=49,668 rom=97,712
------------------------------------------------------------------------------------------------------------------------
id layer (type) macc rom
------------------------------------------------------------------------------------------------------------------------
0 conv1d (Conv2D) |||||||||| 15.5% | 0.7%
1 conv1d_1 (Conv2D) ||||||||||||||||||||||| 36.3% | 1.6%
4 dense (Dense) ||||||||||||||||||||||||||||||| 47.4% ||||||||||||||||||||||||||||||| 96.7%
4 dense_nl (Nonlinearity) | 0.1% | 0.0%
5 dense_1 (Dense) | 0.5% | 1.1%
5 dense_1_nl (Nonlinearity) | 0.1% | 0.0%
------------------------------------------------------------------------------------------------------------------------
UM3053
SPC5-STUDIO-AI plugin
UM3053 - Rev 1 page 20/39