IDS NXT User manual

IDS NXT
IDS Imaging Development Systems GmbH
Dimbacher Str. 10
D-74182 Obersulm, Germany
T: [phone]
E: [e-mail]
W: http://www.ids-nxt.com
Vision apps – Object Detector

October 2021 2
IDS NXT: Vision apps – Object Detector
Content
1 Preface ............................................................................................................................................. 3
2 Symbolsand hints ............................................................................................................................ 5
3 Vision app "Object Detector" ............................................................................................................ 6
3.1 Overview ...................................................................................................................................... 7
3.2 Settings ...................................................................................................................................... 8
3.3 Files ........................................................................................................................................... 9
3.4 Result output ............................................................................................................................... 9
3.5 Website .................................................................................................................................... 10
4 Interface description vision app "Object Detector" ......................................................................... 11
4.1 /vapps/objectdetector .................................................................................................................. 11
4.2 /vapps/objectdetector/configurables .............................................................................................. 12
4.3 /vapps/objectdetector/files/cnnfile/data ......................................................................................... 13
4.4 /vapps/<%REST-NAME-API%>/ftpclient ....................................................................................... 14
4.5 /vapps/objectdetector/images/default ............................................................................................ 15
4.6 /vapps/objectdetector/resultsources/last ....................................................................................... 16
4.7 Error codes ................................................................................................................................ 17
Index ................................................................................................................................................. 18

October 2021 3
IDS NXT: Vision apps – Object Detector
1 Preface
Introduction
IDS Imaging Development Systems GmbH has taken every possible care in preparing this manual. We however
assume no liability for the content, completeness or quality of the information contained therein. The content of
this manual is regularly updated and adapted to reflect the current status of the software. We furthermore do
not guarantee that this product will function without errors, even if the stated specifications are adhered to.
Under no circumstances can we guarantee that a particular objective can be achieved with the purchase of this
product.
Insofar as permitted under statutory regulations, we assume no liability for direct damage, indirect damage or
damages suffered by third parties resulting from the purchase of this product. In no event shall any liability
exceed the purchase price of the product.
Please note that the content of this manual is neither part of any previous or existing agreement, promise,
representation or legal relationship, nor an alteration or amendment thereof. All obligations of IDS Imaging
Development Systems GmbH result from the respective contract of sale, which also includes the complete and
exclusively applicable warranty regulations. These contractual warranty regulations are neither extended nor
limited by the information contained in this manual. Should you require further information on this product, or
encounter specific problems that are not discussed in sufficient detail in the manual, please contact your local
dealer or system installer.
Trademarks
The IDS logo is a registered trademark of IDS Imaging Development Systems GmbH,
registered for U.S. (Reg.No. 4,513,138) and other countries.
IDS NXT and uEye are registered trademarks of IDS Imaging Development Systems GmbH. Microsoft and
Windows are trademarks or registered trademarks of Microsoft Corporation. All other products or company
names mentioned in this manual are used solely for purposes of identification or description and may be
trademarks or registered trademarks of the respective owners.
Copyright
© IDS Imaging Development Systems GmbH. All rights reserved. This manual may not be reproduced,
transmitted or translated to another language, either as a whole or in parts, without the prior written permission
of IDS Imaging Development Systems GmbH.
Status: October 2021

October 2021 4
IDS NXT: Vision apps – Object Detector
Contact
Visit our web site http://www.ids-nxt.com where you will find all the latest information about our software and
hardware products. Please contact your local IDS distributors for first level support in your language. For a list
of IDS distributors worldwide please go to our website http://www.ids-nxt.com.
Address
IDS Imaging Development Systems GmbH
Dimbacher Str. 10
D-74182 Obersulm, Germany
T
[phone]
E
[e-mail]
W
http://www.ids-nxt.com
Address in UK
IDS Imaging Development Systems Ltd.
Landmark House, Station Road
RG27 9HA Hook | United Kingdom
T
[phone]
E
[e-mail]
W
https://en.ids-imaging.com

October 2021 5
IDS NXT: Vision apps – Object Detector
2 Symbols and hints
This symbol indicates hints with useful information for better understanding and using features
and functions.
This symbol indicates important warnings for product safety to prevent damage.
This symbol indicates important warnings for personal safety to prevent injury.

October 2021 6
IDS NXT: Vision apps – Object Detector
3 Vision app "Object Detector"
For version 1.8.5 of the vision app Object Detector, at least IDS NXT OS version 1.3.0. or
higher must be installed on the camera.
Vision app Object Detector allows you to locate and classify several objects in an image using a CNN
(convolutional neural network). It is not necessary to know the position of the objects in the image in advance.
The detection results are displayed with information on the class and number of classes as well as the
probability and ROI per object.
To use CNNs for object detection, the IDS NXT camera must have installed IDS NXT OS
version 1.1 or higher.
The processing of CNNs by the deep ocean core is done exclusively for a single vision app. If
you want to start another vision app, that also requires the deep ocean core, you have to stop
the currently running vision app first.
Classification or object detection
Different vision apps are available for classification or object detection. There may be overlaps in marginal
cases where several points have to be considered whether a classification or an object detection should be
used.
Fig. 1:Classification and object detection
· Accuracy: Higher accuracies can be achieved within classification, as each ROI is individually evaluated by a
CNN. The single ROI is scaled to the input size of the CNN (e.g. 224x224 pixels), while object detection
scales the whole image to find objects with different sizes.

October 2021 7
IDS NXT: Vision apps – Object Detector
· Inference time: the classification requires generally shorter inference times than object detection for the
evaluation of a single ROI. However, the processing time in classification increases linearly with the number
of ROIs. The inference time for object detection is independent of the number of objects found in the image.
Depending on which CNNs are used, object detection is therefore faster from a certain number of ROIs. If
processing time is important, it may be worth using object detection instead of classification even if the
object positions are known.
· Training effort: In general, classification tasks are easier to train than detection tasks. For a classification
only one input per image has to be made and images can be labelled in batches, e.g. by uploading whole
folders to IDS NXT lighthouse. For object detection, images must be individually marked with several boxes,
which results in a higher effort.
3.1 Overview
Fig. 2: Object Detector
1. Selection of the vision app
2. Information about the vision app
3. Switching between vision app settings and the result display ("Highlight")
4. Vision app settings
5. Upload or delete CNN file in vision app
6. Status of the FTP server The use of the FTP server must be activated in the network settings of the camera,
see manual "IDS NXT cockpit".
7. Result of image analysis and selection of the image view (live image, image detail)

October 2021 8
IDS NXT: Vision apps – Object Detector
3.2 Settings
Fig. 3: Select CNN package
CNN
· Select a CNN for the evaluation. You must load the CNN onto the camera in advance, see Files.
Additional information is displayed for the selected CNN, such as name, architecture, etc.
Which display name is used for the CNN?
· If you do not make any changes, the display name will be the name that was used to
create the project in IDS NXT lighthouse or IDS NXT ferry.
· You can rename the saved CNN file before uploading to change the display name (requires
IDS NXT OS 1.1 or later and IDS NXT cockpit version 2.1 or later).
Detection threshold
Here, you set the value as of when a detection should be evaluated as a valid result. Note that a value too low
might lead to false detections. On the other hand, a high value can result in objects not being detected.
FTP transfer
Here, you specify which images (including XML file with information about ROIs in the image, such as the class
name or ROI position) are transferred to the FTP server. The images are automatically stored in a subdirectory
called "Objectdetector". You have the following choices:
· FTP Disabled: Transfer to the FTP server is disabled.

October 2021 9
IDS NXT: Vision apps – Object Detector
· Any Detection: All images with a detection are transferred.
· No Detection: Only images without a detection are transferred.
· "Class Name": All images with a detection of the selected class will be transferred.
If the camera cannot establish a connection to the FTP server, the transfer of images is interrupted. The
camera will try to connect to the FTP server again in 5 minute intervals. If no connection to the FTP server
could be established after 1 hour, the FTP function is deactivated.
3.3 Files
Fig. 4: Upload a CNN package
Here, you upload a CNN file (*.det) that you have trained previously, e.g. in IDS NXT lighthouse for object
detection. After successfully uploading the CNN file, you can select the CNN in the settings. Object Detector
can use CNNs for object detection created for IDS NXT OS 1.1 or higher
IDS NXT cockpit displays an error message if the CNN packet is large and uploading takes a
long time. The upload is still performed, however. After uploading, the CNN package is
installed, which causes the vision app to take a few seconds to run (as it did the first time the
vision app was started).
Click to delete the currently selected CNN on the device. After deletion, the first CNN from the list
is activated. If the last CNN in the list is deleted, evaluation is not performed any more and you must first
upload a CNN again.
3.4 Result output
Fig. 5: Result output
· Switch between overall image and result image
In the result image, all detected objects are displayed with their number, the detection result and the
corresponding probability. Only detections with a probability above the detection threshold are displayed.
Each class is assigned one of six colors, which are repeated if models are trained on more than six classes.
· Class: Displays the found class
· Count: Frequency of the found class

October 2021 10
IDS NXT: Vision apps – Object Detector
· Data: Shows for each ROI the found class, the probability and the size and position of the ROI.
· Inference Time: Indicates the inference time in milliseconds.
3.5 Website
You can open the vision app website directly from the IDS NXT cockpit (Vision app manager) or directly access
the corresponding URL (http://<IP address of the camera>/vapps/objectdetector/website).
The website will show you the current image with the corresponding detection results.
Fig. 6: Objektdetektion

October 2021 11
IDS NXT: Vision apps – Object Detector
4 Interface description vision app "Object Detector"
The description of the REST interface is limited to the parameters/values that are specific for the vision app. A
more detailed description of the IDS NXT REST interface can be found in the manual "IDS NXT - REST
interface".
REST queries are only possible for activated vision apps, as these are generated dynamically at runtime.
Depending on the action and parameters, the used HTTP methods are color-coded and assigned to the
permission of IDS NXT user profiles (see manual "IDS NXT - Operation with the IDS NXT cockpit").
4.1 /vapps/objectdetector
/vapps/objectdetector
· GET
· OPTIONS
Provides information about the "Object Detector" vision app.
Response format
The content of the HTTP response is transferred in JSON format (application/json).
Return values
· Activated (Boolean): Activation status of the vision app
· Brief (String): Short description of the vision app
· Description (String): Long description of the vision app
· Manufacturer (String): Manufacturer of the vision app
· Name (String): Name of the vision app
· Status (String): Runtime status of the vision app "NotRunning", "Starting", "Running", "Crashed",
"FailedToStart", "TimedOut", "ReadError", "WriteError", "UnknownError", "Incompatible",
"DeepOceanCoreInUse", "Debugging"
· Title (String): Vision app title
· Type (String): Vision app type

October 2021 12
IDS NXT: Vision apps – Object Detector
· Version (String): Version of the vision app
· Website (String): URL of the vision app website
Error response
See Error codes
Sample code
curl -X GET "http://<ip>/vapps/objectdetector" -i --user admin:<password>
4.2 /vapps/objectdetector/configurables
/vapps/objectdetector/configurables
· GET
· OPTIONS
· PATCH
Shows the active CNN package.
Response format
The content of the HTTP response is transferred in JSON format (application/json).
Return values
· a_cnns (String): Selection of the CNN package for object detection
· detectionthreshold (Integer): Returns the value or sets the value as of when a detection should be evaluated
as a valid result. Note that a value too low might lead to false detections. On the other hand, a high value can
result in objects not being detected.
· f_ftp_transfer (Enumeration): Defines which images (including XML file with information about ROIs in the
image such as the class name or ROI position) are transferred to the FTP server. The images are
automatically stored in a subdirectory named "Objectdetector" (see Settings).
Error response
See Error codes

October 2021 13
IDS NXT: Vision apps – Object Detector
Sample code for GET
curl -X GET "http://<ip>/vapps/objectdetector/configurables" -i --user
admin:<password>
Sample output for GET
{
"a_cnns": "PersonDetector",
"detectionthreshold": 29,
"showresult": true
}
Sample code for PATCH
curl -X PATCH "http://<ip>/vapps/objectdetector/configurables" -i --user
admin:<password>--data "a_cnns=<New package from OPTIONS-->Range>"
Sample code for OPTIONS
curl -X OPTIONS "http://<ip>/vapps/objectdetector/configurables" -i --user
admin:<password>
4.3 /vapps/objectdetector/files/cnnfile/data
/vapps/objectdetector/files/cnnfile/data
· OPTIONS
· PUT
Uploads a new CNN file for object detection (*.det) or deletes the currently selected CNN on the device. After
deletion, the first CNN from the list is activated. If the last CNN in the list is deleted, evaluation is not performed
any more and a CNN must first be uploaded again.
You can rename the CNN file to change the display name when uploading via the REST interface (requires IDS
NXT OS 1.1 or higher).
Response format
The content of the HTTP response is transferred in JSON format (application/json).

October 2021 14
IDS NXT: Vision apps – Object Detector
Return values
· HTTP code 200
Error response
See Error codes
Sample code
curl -X PUT "http://<ip>/vapps/objectdetector/files/cnnfile/data" -i --user
admin:<password>-T /<path>/<file name>.det -H "Content-Type: application/octet-
stream"
Sample code with renamed CNN file
curl -X PUT "http://<ip>/vapps/objectdetector/files/cnnfile/data" -i --user
admin:<password>-H "content-type: application/octet-stream" -H "content-
disposition: attachment; filename=myRenamedCNN.det" --data-binary
@myRenamedCNN.det -vvv
Sample code for deleting a CNN file
curl -X DELETE "http://<ip>/vapps/objectdetector/files/cnnfile/data" -i --user
admin:<password>
4.4 /vapps/<%REST-NAME-API%>/ftpclient
/vapps/objectdetector/ftpclient
· GET
· OPTIONS
Returns the status of the FTP client. The use of the FTP server must be activated in the network settings of the
camera, see manual "IDS NXT cockpit".
Response format
The content of the HTTP response is transferred in JSON format (application/json).

October 2021 15
IDS NXT: Vision apps – Object Detector
Return values
· hasFtpClient (Boolean): Status of the FTP client.
Error response
See Error codes
Sample code for GET
curl -X GET "http://<ip>/vapps/objectdetector/ftpclient" -i --user
admin:<password>
4.5 /vapps/objectdetector/images/default
/vapps/objectdetector/images/resultimage
· GET
· OPTIONS
Transfers the current ROI as image file.
Response format
The Accept Token is used to select the format in which the result image is transferred.
· image/jpeg: Transfers a JPEG image (default).
· image/bmp: Transfers a bitmap image.
· image/png: Transfers a PNG image.
Parameters
The result image can be further specified using additional parameters.
· MaxHeight (Integer): Maximum height of the result image in pixels: 1 ... x (max. sensor height).
· MaxWidth (Integer): Maximum width of the result image in pixels: 1 ... x (max. sensor width).
· Quality (Integer): JPEG quality of the result image in percent if the content type “image/jpeg” is set: 0…100
Return values
· message-body: Image data

October 2021 16
IDS NXT: Vision apps – Object Detector
Error response
See Error codes
Sample code
curl -X GET "http://<ip>/vapps/objectdetector/images/resultimage?quality=100" -i
--user admin:<password>-H Accept:image/jpeg --output result.jpg
4.6 /vapps/objectdetector/resultsources/last
/vapps/objectdetector/resultsources/last
· GET
· OPTIONS
Returns the result of the last image analysis.
Response format
The content of the HTTP response is transferred in JSON format (application/json).
Return values
· class (String): Found class
· count (String): Number of found classes
· data (String): Shows the found class, probability, size and position of the found ROI.
· highlight (String): Result and formatting for the "highlight" tab in the IDS NXT cockpit.
· inferecetime (Integer): Inference time in milliseconds
Error response
See Error codes
Sample code
curl -X GET "http://<ip>/vapps/objectdetector/resultsources/last" -i --user
admin:<password>

October 2021 17
IDS NXT: Vision apps – Object Detector
4.7 Error codes
For the REST queries, HTTP response codes are used as return values. The following HTTP codes are possible
return values of the REST queries, in general any HTTP code would be possible:
HTTP code
Description
200
Success
404
The requested object could not be found (e. g. wrong path)
405
The query must be made using other HTTP methods, e. g. OPTIONS instead of GET.
500
Internal device error
503
Result not available (e.g. no code read yet)

October 2021 18
IDS NXT: Vision apps – Object Detector
- C -
CNN
information 8
select 8, 12
switch 8
CNN file
delete 13
upload 9, 13
- E -
Error code 17
- F -
FTP
client 14
- I -
Image
analysis 9
Image analysis
last 16
result 16
- O -
Object Detector
GUI 7
- R -
REST
CNN 13
query 11
Result
display 8, 9, 12
website 10
ROI
image 15
transfer 15
- T -
threshold
set 8, 12
- V -
Vision app
Object Detector 6, 11
website 10
- W -
Website 10