MYNTAI MYNT EYE D-Series Operator's manual

MYNT EYE D SDK Documentation
Release 1.8.0
MYNTAI
Nov 07, 2019

CONTENTS
1 PRODUCT 1
1.1 Product Description ........................................... 1
1.2 Product Surface ............................................. 2
1.3 Product Specification ........................................... 2
1.4 Support Resolutions ........................................... 6
1.5 IMU Coordinata System ......................................... 7
2 SDK 9
2.1 SDK Description ............................................. 9
2.2 SDK Installation ............................................. 9
2.3 SDK Samples ............................................... 21
2.4 SDK Tools ................................................ 32
2.5 SDK Project Demos ........................................... 37
2.6 Change Log ............................................... 51
3 Android SDK 55
3.1 SDK Explain ............................................... 55
3.2 SDK Download .............................................. 55
3.3 SDK Install ................................................ 55
3.4 SDK Samples ............................................... 55
3.5 Update Log ................................................ 58
4 FIRMWARE 61
4.1 Firmware Update ............................................. 61
4.2 Change Log ............................................... 62
5 TOOLS SUPPORT 63
5.1 Calibration Tool Manual ......................................... 63
6 OPEN SOURCE SUPPORT 69
6.1 How To Use In VINS-Mono ....................................... 69
6.2 How to Use In ORB_SLAM2 ...................................... 70
6.3 How To Use In OKVIS .......................................... 71
6.4 How To Use In VIORB ......................................... 72
6.5 How To Use In VINS-Fusion ...................................... 73
7 API DOCS 77
7.1 Camera .................................................. 77
7.2 Device .................................................. 81
7.3 Enums .................................................. 83
7.4 Types ................................................... 87
i

7.5 Utils ................................................... 91
8 Android API DOCS 93
8.1 USBMonitor ............................................... 93
8.2 MYNTCamera .............................................. 93
8.3 ImuData Device with Imu ........................................ 98
8.4 FrameData ................................................ 98
9 TECHNICAL SUPPORT 103
9.1 FAQ .................................................... 103
9.2 Contact Us ................................................ 103
Index 105
ii

CHAPTER
ONE
PRODUCT
1.1 Product Description
The MYNT Depth utilizes the camera and the motion sensor to provide visually accurate SLAM results with higher
precision, lower cost, simpler layout, along with the ability to achieve face and object recognition. The concept of
combining binocular and IMU is the leading-edge technology in the current SLAM industry.The depth version of the
product has a built-in depth calculation chip that can output depth images without the host computer. At the same
time, the product is equipped with leading hardware solutions such as IR active light, IMU six-axis, hardware-level
frame synchronization, global shutter, etc., up to 720p/60fps (120°FOV version) of synchronous image information,
the recognition distance can reach 15m (50 °FOV version), accuracy up to millimeters (50° FOV version).
Using camera techniques such as frame synchronization, automatic exposure, and white balance control, the MYNT
EYE Depth can produce synchronized image sources with high-precision, which decreases the difficulty of algorithms
development, thus increasing efficiency. The Depth comes with six-axis sensor(IMU) and an infrared active light
detector (IR). Among them, the six-axis sensor(IMU) can provide complementarity and correction of data from the
visual positioning algorithms, and is suitable for visual inertial odometry(VIO) algorithms research to help improve
the positioning accuracy. The infrared active light detector (IR) can help solve the problem of identification of objects
such as indoor white walls and non-textured objects, as well as enhance the accuracy of image source recognition.The
Binocular+IMU scheme provides accurate six-axis complementary data for VSLAM applications and is more accurate
and robust than other single solutions. In addition, MYNT EYE Depth also provides a rich SDK interface and VSLAM
open source project support, which can help customers quickly integrate solutions, accelerate product development
process, and achieve rapid productization and implementation.
As a hardware product for in-depth research and development of stereo vision computing applications, MYNT EYE
Depth can be widely used in the field of visual positioning navigation (vSLAM), including visual real-time positioning
navigation system of driverless vehicle and robots, visual positioning system of UAV, obstacle avoidance navigation
system for driverless Vehicle, Augmented Reality (AR), Virtual Reality (VR), etc. At the same time, it can be used
in field of Visual recognition, including Stereoscopic face recognition, three-dimensional object recognition, space
motion tracking, three-dimensional gestures and somatosensory recognition. And of course, you can use it for mea-
surement which includes assisted driving system (ADAS), binocular volume calculation, industrial visual screening,
etc.
In order to ensure the quality of the output data of the camera products, we have calibrated the binocular and IMU.
The product has passed various hardware stability tests, such as high temperature and humidity continuous work
and operation, low-temperature dynamic aging, high-temperature operation, low-temperature storage, whole-machine
thermal shock, sinusoidal vibration and random vibration tests to ensure the stability and reliability of the product.
In addition to the research and development of products and technologies, it can also be directly applied to mass
production, accelerating the process from R&D to productization.
1

MYNT EYE D SDK Documentation, Release 1.8.0
1.2 Product Surface
Shell(mm) PCBA board(mm)
165x31.5x29.6 149x24
A. Camera:please pay attention to protect the camera sensor lenses, to avoid imaging quality degradation.
B. Infrared structured-light transmitter and outlet: the infrared structured-light can effectively solve the problem as-
sociated with the visual positioning calculations of white wall non-textured object(For non-IR version, the outlet is
reserved but there is no internal structured-light emitter).
C. USB Micro-B interface and set screw holes: during usage, plug in the USB Micro-B cable and secure it by fastening
the set screws to avoid damage to the interface and to ensure stability in connection.
D. ¼ inch standardized set screw hole: for fixing the stereo camera to tripods or other devices.
1.3 Product Specification
1.3.1 D1000-120/Color Specification
2 Chapter 1. PRODUCT

MYNT EYE D SDK Documentation, Release 1.8.0
Product parameters
Model D1000-IR-120/Color
Size 165x31.5x30.12mm
Frame Rate Up to 60FPS
Resolution 2560x720;1280x480
Depth Resolution On chip 1280x720 640x480
Pixel Size 3.75x3.75m
Baseline 120.0mm
Visual Angle D:121° H:105° V:58°
Focal Length 2.45mm
IR Support Yes
IR detectable range 3m
Color Mode Color
Working Distance 0.32-7m
Scanning Mode Global Shutter
Power 1.9~3.5W@5V DC from USB
Synchronization Precision <1ms (up to 0.01ms)
IMU Frequency 200Hz
Output data format YUYV/MJPG
Data transfer Interface USB2.0/3.0
Weight 184g
UVC MODE Yes
Software
Support system Windows 10Ubuntu 16.04/18.04ROS kinetic/melodicAndroid 5.x ~ Android 8.x
SDK http://www.myntai.com/dev/mynteye_depth
Support ORB_SLAM2OKVISVins-MonoVins-FusionVIORB
Environment
Operating Temperature -10°C~55°C
Storage Temperature -15°C~70°C
Humidity 10% to 80% non-condensing
Package
Content MYNT EYE x1 USB Micro-B Cable x1
Warranty
Warranty 12 Months Limited Manufacturer’s Warranty
1.3. Product Specification 3

MYNT EYE D SDK Documentation, Release 1.8.0
Precision
Depth accuracy The error doesn’t exceed 2%
1.3.2 D1000-50/Color Specification
Product parameters
Model D1000-50/Color
Size 165x31.5x29.85mm
Frame Rate Up to 60FPS
Resolution 2560x720;1280x480
Depth Resolution On chip 1280x720 640x480
Pixel Size 3.75x3.75m
Baseline 120.0mm
Visual Angle D:70° H:64° V:38°
Focal Length 3.9mm
IR Support NO
IR detectable range -
Color Mode Color
Working Distance 0.49-10m
Scanning Mode Global Shutter
Power 1.8W@5V DC from USB
Synchronization Precision <1ms (up to 0.01ms)
IMU Frequency 200Hz
Output data format YUYV/MJPG
Data transfer Interface USB2.0/3.0
Weight 152g
UVC MODE Yes
Software
Support system Windows 10Ubuntu 16.04/18.04ROS kinetic/melodicAndroid 5.x ~ Android 8.x
SDK http://www.myntai.com/dev/mynteye_depth
Support ORB_SLAM2OKVISVins-MonoVins-FusionVIORB
Environment
Operating Temperature -10°C~60°C
Storage Temperature -20°C~70°C
Humidity 10% to 80% non-condensing
Package
Content MYNT EYE x1 USB Micro-B Cable x1
4 Chapter 1. PRODUCT

MYNT EYE D SDK Documentation, Release 1.8.0
Warranty
Warranty 12 Months Limited Manufacturer’s Warranty
Precision
Depth accuracy The error doesn’t exceed 2%
1.3.3 D1200 Specification
Product parameters
Model D1200
Size 75.5x34.5x12.9mm
Frame Rate Up to 30fps
Resolution 2560*720;1280*480
Depth Resolution 1280*720; 640*480
Pixel Size 3.0*3.0m
Baseline 40.0mm
Visual Angle D:66° H:59° V:35°
Focal Length 3.3mm
IR Support YES
IR detectable range 2m
Color Mode Color
Working Distance 0.2-3m
Scanning Mode Rolling Shutter
Power 0.75-2.5W@5V DC from USB
Output data format YUYV/MJPG
Data transfer Interface Type-C/Micro USB2.0
Weight 44g
UVC MODE Yes
Software
Support system Android 5.x ~ Android 8.x
Environment
Operating Temperature -10°C~55°C
Storage Temperature -15°C~70°C
Humidity 10% to 80% non-condensing
1.3. Product Specification 5

MYNT EYE D SDK Documentation, Release 1.8.0
Package
Content MYNT EYE x1 USB Cable
Warranty
Warranty 12 Months Limited Manufacturer’s Warranty
Precision
Depth accuracy The error doesn’t exceed 1%
1.4 Support Resolutions
mode interface color resolution color fps depth resolution depth fps
L’+D USB3.0 1280x720 60/30/20/10 1280x720 60/30/20/10
L’+D USB3.0 640x480 60/30 640x480 60/30
L’+R’+D USB3.0 2560x720 30 1280x720 30
L’+R’+D USB3.0 1280x480 60/30 640x480 60/30
L+D USB3.0 1280x720 60/30/20/10 1280x720 60/30/20/10
L+D USB3.0 640x480 60/30 640x480 60/30
L+R+D USB3.0 2560x720 30 1280x720 30
L+R+D USB3.0 1280x480 60/30 640x480 60/30
L+R USB3.0 2560x720 30 not open null
L’+R’ USB3.0 2560x720 30 not open null
D USB3.0 not open null 1280x720 60/30
D USB3.0 not open null 640x480 60/30
L+R USB2.0 2560x720 5 not open null
L’+R’ USB2.0 2560x720 5 not open null
L+R USB2.0 1280x480 15 not open null
L’+R’ USB2.0 1280x480 15 not open null
L’+D USB2.0 1280x720 5 640x720 5
L’+D USB2.0 640x480 15 320x480 15
L+D USB2.0 1280x720 5 640x720 5
L+D USB2.0 640x480 15 320x480 15
L’ USB2.0 1280x720 5 not open null
L USB2.0 1280x720 5 not open null
D USB2.0 not open null 640x720 5
D USB2.0 not open null 320x480 15
L+R USB2.0/MJPG 2560x720 5 not open null
L+R USB2.0/MJPG 1280x480 15 not open null
L USB2.0/MJPG 1280x720 5 not open null
Note: L’=left rectify image, L=left image,R’=right rectify image, R=right image, D=depth image
6 Chapter 1. PRODUCT

MYNT EYE D SDK Documentation, Release 1.8.0
In IR Depth Only mode, framerate only support 15fps and 30fps.
1.5 IMU Coordinata System
IMU coordinate system is right-handed,the axis directions are as follows:
1.5. IMU Coordinata System 7

MYNT EYE D SDK Documentation, Release 1.8.0
8 Chapter 1. PRODUCT

CHAPTER
TWO
SDK
2.1 SDK Description
2.1.1 Supported Platforms
SDK is built on CMake and can be used cross multiple platforms such as Linux, WIndows,etc.We provide two instal-
lation modes:Download pack file and install, Compile and install from source code.
These are the platforms that can be used:
*Windows 10
*Ubuntu 18.04/16.04
*Jetson TX1 TX2 Xavier
*RK3399 firefly
*Raspberry Pi 3B
Tip: Ubuntu only support source installation mode. Only supports 64 bit systems.
Warning: Due to the requirement of hardware transmission rate, please use the USB 3 interface. In addition,
virtual machines have USB driver compatibility problems, thus they are not recommended.
2.2 SDK Installation
2.2.1 Ubuntu Source Installation
1. Install SDK dependencies
1.1 Install OpenCV
If you have installed opencv already or you want use it in ROS, you can skip this part.
1.1.1 Install OpenCV with apt or compile (Choose one)
9

MYNT EYE D SDK Documentation, Release 1.8.0
1.1.1.1 Install OpenCV with apt (Recommend)
sudo apt-get install libopencv-dev
1.1.1.2 Install OpenCV by Compile
To build and install Opencv, please refer to Installation in Linux
Tip: If you need to install ros, you can skip this step and use opencv in ros.
Alternatively, refer to the command below:
[compiler]sudo apt-get install build-essential
[required]sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev
˓→libavformat-dev libswscale-dev
[optional]sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-
˓→dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev
git clone https://github.com/opencv/opencv.git
cd opencv/
git checkout tags/3.4.3
cd opencv/
mkdir build
cd build/
cmake ..
make -j4
sudo make install
1.2 Install PCL for Point Cloud sample (Optional)
To build and install PCL, please refer to PCL Installation
Tip: If you need to install ros, you can skip this step and use pcl in ros.
sudo apt install -y libboost-all-dev libflann-dev libeigen3-dev libusb-1.0-0-dev
˓→libvtk6-dev libproj-dev
git clone https://github.com/PointCloudLibrary/pcl.git
cd pcl
git checkout pcl-1.7.2
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j2
sudo make -j2 install
10 Chapter 2. SDK

MYNT EYE D SDK Documentation, Release 1.8.0
1.3 Link libGL.so for TX1/TX2 compile bug (Optional)
sudo ln -sf /usr/lib/aarch64-linux-gnu/tegra/libGL.so /usr/lib/aarch64-linux-gnu/
˓→libGL.so
2. Build SDK
git clone https://github.com/slightech/MYNT-EYE-D-SDK.git
cd MYNT-EYE-D-SDK
2.1 Init SDK
Note: Because of the problem of device permissions, you must reinsert the camera device after the command is
executed and on the same computer, this operation only needs to be done once.
make init
2.2 Compile SDK
make all
3. Run Samples
Note: Open the rectified image by default (Run vio need to raw image, run depth or points cloud need to rectified
image.)
1) get_image shows the left camera image and colorful depthmap (compatible with USB2.0)
./samples/_output/bin/get_image
2) get_stereo_image shows the left camera image and colorful depthmap
./samples/_output/bin/get_stereo_image
3) get_depth shows the left camera image, 16UC1 depthmap and depth value(mm) on mouse pointed pixal
./samples/_output/bin/get_depth
4) get_points shows the left camera image, 16UC1 depthmap and point cloud view
./samples/_output/bin/get_points
5) get_imu shows motion datas
./samples/_output/bin/get_imu
2.2. SDK Installation 11

MYNT EYE D SDK Documentation, Release 1.8.0
6) get_img_params show camera intrinsics and save in file
./samples/_output/bin/get_img_params
7) get_imu_params show imu intrinsics and save in file
./samples/_output/bin/get_imu_params
8) get_from_callbacks show image and imu data by callback
./samples/_output/bin/get_from_callbacks
9) get_all_with_options open device with different options
./samples/_output/bin/get_all_with_options
10) get_depth_with_filter display filtered depth image
./samples/_output/bin/get_depth_with_filter
11) get_points_with_filter display filtered point cloud image
./samples/_output/bin/get_points_with_filter
4 Install With OpenCV ROS
If you won’t use ROS(The Robot Operating System), you can skip this part.
ROS installation and operation steps, refer to ROS Wrapper Installation ROS Wrapper Usage .
5. Package
If you wanna package with specified OpenCV version:
cd <sdk> # local path of SDK
make cleanall
export OpenCV_DIR=<install prefix>
export OpenCV_DIR=/usr/local
export OpenCV_DIR=$HOME/opencv-2.4.13.3
Packaging:
cd <sdk> # local path of SDK
make pkg
6. Clean
cd <sdk> # local path of SDK
make cleanall
make uninstall
12 Chapter 2. SDK

MYNT EYE D SDK Documentation, Release 1.8.0
2.2.2 Windows Source Installation
The following steps are how to install from source codes. If you wanna using prebuilt DLL, please see Windows EXE
Installation .
1. Install Build Tools
1.1 Install Visual Studio
Download Visual Studio 2017 from https://visualstudio.microsoft.com/zh-hans/vs/older-downloads/ and install, select
“Desktop development with C++” .
Tip: support Visual Studio 2015 and Visual Studio 2017.
1.2 Install CMake
Download CMake from https://cmake.org/ and install
1.3 Install MSYS2
1) Download MSYS2 from http://mirrors.ustc.edu.cn/msys2/distrib/x86_64/ and install
2) Add bin path to System PATH environment variable list (Add to the PATH on Windows 10)
C:\msys64\usr\bin
3) Install make , double click msys2.exe , input following command:
pacman -Syu
pacman -S make
Finally, the CMD (Command Prompt) can run the following command:
2.2. SDK Installation 13

MYNT EYE D SDK Documentation, Release 1.8.0
>make --version
GNU Make 4.2.1
2. Install SDK dependencies
2.1 Install OpenCV
2.1.1 Install OpenCV with Pre-built Libraries (Recommend)
*For more details you can reference OpenCV offical document *
1) Go to OpenCV Sourceforge page http://sourceforge.net/projects/opencvlibrary/files/opencv-win/
2) Choose a build you want to use and download it. For example 3.4.3/opencv-3.4.3-vc14_vc15.exe
3) Make sure you have admin rights. Unpack the self-extracting archive
4) To finalize the installation, go to set the OpenCV environment variable and add it to the systems path
2.1.2 Set up environment variable
1. Start up a command window as admin and enter following command to add OPENCV_DIR environment vari-
able:
Change the “D:OpenCV” to your opencv unpack path
setx -m OPENCV_DIR D:\OpenCV\Build\x64\vc14\lib (suggested for Visual Studio 2015
˓→- 64 bit Windows)
setx -m OPENCV_DIR D:\OpenCV\Build\x64\vc15\lib (suggested for Visual Studio 2017
˓→- 64 bit Windows)
Or referring to Add to the PATH on Windows 10 to add OPENCV_DIR environment variable manually.
D:\OpenCV\Build\x64\vc14\lib (suggested for Visual Studio 2015 - 64 bit Windows)
D:\OpenCV\Build\x64\vc15\lib (suggested for Visual Studio 2017 - 64 bit Windows)
2. Add OpenCV bin path to System PATH environment variable list
D:\OpenCV\Build\x64\vc14\bin (suggested for Visual Studio 2015 - 64 bit Windows)
D:\OpenCV\Build\x64\vc15\bin (suggested for Visual Studio 2017 - 64 bit Windows)
2.2 Install libjpeg-turbo
1) Download libjpeg-turbo from https://sourceforge.net/projects/libjpeg-turbo/files/ and install
2) Add bin path to System PATH environment variable list
C:\libjpeg-turbo64\bin
2.3 Install PCL for Point Cloud sample (Optional)
Download All-in-one installers (PCL + dependencies) from: https://github.com/PointCloudLibrary/pcl/releases
14 Chapter 2. SDK

MYNT EYE D SDK Documentation, Release 1.8.0
3. Build SDK
Open “x64 Native Tools Command Prompt for VS 2017”(suggested for Visual Studio 2017 - 64 bit Windows) com-
mand shell
git clone https://github.com/slightech/MYNT-EYE-D-SDK.git
cd MYNT-EYE-D-SDK
make all
Tip: Visual Studio Command Prompt can be opened from the Start menu,
You can also open it from the Visual Studio Tools menu.
However, if you do not have the Visual Studio 2015 Tools menu, you can add one yourself.
Open Tools’s External Tools. . . and Add the following:
2.2. SDK Installation 15

MYNT EYE D SDK Documentation, Release 1.8.0
Field Value
Title Visual Studio Command Prompt
Command C:\Windows\System32\cmd.exe
Arguments /k "C:\Program Files (x86)\Microsoft Visual Studio 14.
0\Common7\Tools\VsDevCmd.bat"
Initial Direc-
tory
$(SolutionDir)
4. Run Samples
Note: Open the rectified image by default (Run vio need to raw image, run depth or points cloud need to
rectified image.)
1) get_image shows the left camera image and colorful depthmap (compatible with USB2.0)
.\samples\_output\bin\get_image.bat
2) get_stereo_image shows the left camera image and colorful depthmap
./samples/_output/bin/get_stereo_image.bat
3) get_depth shows the left camera image, 16UC1 depthmap and depth value(mm) on mouse pointed pixel
.\samples\_output\bin\get_depth.bat
4) get_points shows the left camera image, 16UC1 depthmap and point cloud view
.\samples\_output\bin\get_points.bat
5) get_imu shows motion datas
.\samples\_output\bin\get_imu
6) get_img_params show camera intrinsics and save in file
.\samples\_output\bin\get_img_params
7) get_imu_params show imu intrinsics and save in file
.\samples\_output\bin\get_imu_params
8) get_from_callbacks show image and imu data by callback
.\samples\_output\bin\get_from_callbacks
9) get_all_with_options open device with different options
.\samples\_output\bin\get_all_with_options
10) get_depth_with_filter display filtered depth image
.\samples\_output\bin\get_depth_with_filter
11) get_points_with_filter display filtered point cloud image
16 Chapter 2. SDK