Intel RealSense D400 Series User manual
Intel®RealSense™ Depth Module
D400 Series Custom Calibration
Revision 1.1.0
January 2018
2
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL
OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND
CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED
WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A
PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death.
SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD
INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS
AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY,
ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION,
WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT
OR ANY OF ITS PARTS.
Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or
characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no
responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without
notice. Do not finalize a design with this information.
The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from
published specifications. Current characterized errata are available on request.
Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.
Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-
800-548-4725, or go to: http://www.intel.com/design/literature.htm.
Code names featured are used internally within Intel to identify products that are in development and not yet publicly announced for release.
Customers, licensees and other third parties are not authorized by Intel to use code names in advertising, promotion or marketing of any product
or services and any such use of Intel's internal code names is at the sole risk of the user.
Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries.
*Other names and brands may be claimed as the property of others.
Copyright © 2018, Intel Corporation. All rights reserved.
3
Contents
1Introduction ....................................................................................................... 8
1.1 Purpose and Scope of This Document ........................................................ 8
1.2 Organization .......................................................................................... 8
2Overview .......................................................................................................... 9
2.1 Calibration API and Calibration Data Read/Write/Restore ............................. 9
2.2 Calibration Parameters .......................................................................... 10
2.3Frame Formats Used in Custom Calibration .............................................. 11
2.4 Frame Sync.......................................................................................... 11
2.5 Accuracy.............................................................................................. 11
3Setup ............................................................................................................. 13
3.1 Hardware............................................................................................. 13
3.1.1 Device.................................................................................... 14
3.1.2 Target .................................................................................... 14
3.1.3 Tripod .................................................................................... 15
3.1.4 USB ....................................................................................... 17
3.1.5 PC ......................................................................................... 17
3.2 Software.............................................................................................. 17
3.2.1 Custom Calibration Sample Application....................................... 17
3.2.2 Intel® RealSense™ Calibration Tool and API ............................... 18
3.2.3 Intel® RealSense™ SDK........................................................... 19
3.2.4 OpenCV 3.3.0 ......................................................................... 19
3.2.5 Glut Library ............................................................................ 20
4Calibrating Device with Custom Calibration Sample Application.............................. 21
4.1 Process Overview.................................................................................. 21
4.2 Connect Device to Computer .................................................................. 21
4.3 Running Custom Calibration Sample Application ....................................... 21
4.3.1 Starting Application ................................................................. 21
4.3.2 Capturing Images from 6 Viewpoints.......................................... 22
4.4 Calibration Result.................................................................................. 35
4.5 Updating Results to Device..................................................................... 37
4.5.1 Depth Quality Check before Updating Calibration ......................... 38
4.5.2 Writing Optimized Calibration to Device ...................................... 38
4.5.3 Depth Quality Check after Updating Calibration ........................... 40
5Developing Custom Calibration Application.......................................................... 42
5.1 Sample Application Source Code and Compile........................................... 42
5.2 Calibration Mode Camera Configuration ................................................... 44
5.2.1 Emitter................................................................................... 44
5.2.2 Auto Exposure......................................................................... 44
5.2.3 Streaming Resolution and Format .............................................. 45
5.2.4 Image Captures....................................................................... 45
5.2.5 Demosaic Left/Right Images for ASR / PSR SKUs ......................... 45
5.3 Detecting the Chessboard in an Image with OpenCV ................................. 45
4
5.4 Calculating Depth Camera Calibration with OpenCV................................... 46
5.5 Calculating RGB Camera Calibration with OpenCV ..................................... 47
5.6 Calculating RGB Camera Calibration Extrinsics with OpenCV....................... 47
5.7 Writing Calibration Parameters ............................................................... 48
5
Tables
Table 2-1. Frame Formats Used in Custom Calibration.......................................... 11
Table 3-1. Intel® RealSense™ Calibration API Resources ..................................... 18
Table 3-2. Intel® RealSense™ SDK Resources .................................................... 19
Table 3-3. OpenCV 3.3.0 Resources ................................................................... 20
Table 3-4. OpenCV 3.3.0 Libraries Required for the Example................................. 20
6
List of Figures
Figure 2-1 Software Stack with Dynamic Calibration API and Calibration Apps........... 9
Figure 3-1 Hardware Setup ............................................................................... 13
Figure 3-2 D415 Device .................................................................................... 14
Figure 3-3 8x7 60x60 mm Checker Calibration Target .......................................... 15
Figure 3-4 Tripod............................................................................................. 16
Figure 4-1 Center Right Position ........................................................................ 25
Figure 4-2 Viewpoint #1 ................................................................................... 26
Figure 4-3 Center Right .................................................................................... 27
Figure 4-4 Viewpoint #2 ................................................................................... 28
Figure 4-5 Left ................................................................................................ 29
Figure 4-6 Viewpoint #3 ................................................................................... 29
Figure 4-7 Right .............................................................................................. 30
Figure 4-8 Viewpoint #4 ................................................................................... 31
Figure 4-9 Top Looking Down............................................................................ 32
Figure 4-10 Viewpoint #5 ................................................................................. 33
Figure 4-11 Bottom Looking Up ......................................................................... 34
Figure 4-12 Viewpoint #6 ................................................................................. 34
Figure 4-13 Depth Quality before Updating Calibration ......................................... 38
Figure 4-14 Updating Calibration Parameters to Device ........................................ 39
Figure 4-15 Calibration Parameter Change .......................................................... 40
Figure 5-1 General Process for custom Calibration ............................................... 42
7
Revision History
Document
Number
Revision
Number
Description
Revision Date
XXXXXX
1.0
Initial Release
01/2018
1.1
Update with Calibration API 2.5.2.0 release
01/2018
Introduction
8
1Introduction
1.1 Purpose and Scope of This Document
In order to operate Intel® RealSense™ D400 device efficiently and accurately, users need
to make sure the device is well calibrated. The Intel supplied calibration tools including
Intel® RealSense™ Dynamic Calibrator and OEM Calibration Tool for Intel® RealSense™
Technology are designed to calibrate the devices using Intel proprietary algorithms.
Some customers or developers may choose to use their own calibration algorithms and
update the device with their custom calibration data.
This document contains technical information to assist those developing custom
calibration solutions for Intel® RealSense™ D400 series modules. The primary goal is to
guide the user how to create a calibration application, using the D400 corresponding
APIs. This includes a description how to configure the device under calibration, define
the calibration parameters, and how to read/write these parameters from/into the
device. In addition, a simple sample application based on OpenCV algorithm is
provided as an example.
It is not in the scope of this document to discuss details of calibration algorithm or
accuracy. Developing custom calibration requires knowledge in computer vision as well
as good understanding of the RealSense device operating details. It is a complex topic
and intended only for those with expert level knowledge.
1.2 Organization
This document is organized into four main parts: overview, setup, calibrating a device
with custom calibration sample app, and developing custom solution:
Overview –brief overview of the Calibration API and parameters.
Setup –hardware and software setup for running the Custom Calibration
Sample App to calibrate a device and developing custom calibration solutions.
Calibrating Device with Custom Calibration Sample App –describes the
necessary hardware and software setup required running the Custom
Calibration Sample App and details steps to calibrate device.
Developing Custom Calibration Solution –uses the sample application as an
example to describe details of steps implementing a custom calibration
solution for Intel® RealSense™ D400 series modules.
Overview
9
2Overview
2.1 Calibration API and Calibration Data
Read/Write/Restore
Intel provides a software interface in Calibration API to enable user uploading those
calibration parameters to Intel® RealSense™ D400 devices and read the parameter
back from device:
WriteCustomCalibrationParameters –write parameters to device
ReadCalibrationParameters –read parameters from device
Figure 2-1 Software Stack with Dynamic Calibration API and Calibration Apps
An example tool CustomRW is also included in Calibration API to read calibration
parameters from XML file and write them to the device.
A user custom calibration app can choose one of the two approaches to update the
results to the device:
To link to the WriteCustomCalibrationParameters and
ReadCalibrationParameters and write directly to the device through the APIs.
Overview
10
To write the results into a parameter XML file and then use CustomRW to
read/write the parameters to the device.
In case user needs to restore the calibration data on the device,
ResetDeviceCalibration in Calibration API can be called to programmatically restore
the device calibration to gold settings.
ResetDeviceCalibration –restore device calibration to gold settings
The CustomRW example tool also supports restore through command line option.
2.2 Calibration Parameters
Calibration parameters includes INTRINSICS and EXTRINSICS. Assume left camera is
the reference camera and is located at world origin. RGB parameters only apply to
modules with RGB, e.g., D415 and D435.
Intrinsic includes
Focal length - specified as [fx; fy] in pixels for left, right, and RGB cameras
Principal point - specified as [px; py] in pixels for left, right, and RGB cameras
Distortion - specified as Brown's distortion model [k1; k2; p1; p2; k3] for left,
right, and RGB cameras
Extrinsic includes
RotationLeftRight - rotation from right camera coordinate system to left camera
coordinate system, specified as a 3x3 rotation matrix
TranslationLeftRight - translation from right camera coordinate system to left
camera coordinate system, specified as a 3x1 vector in millimeters
RotationLeftRGB - rotation from RGB camera coordinate system to left camera
coordinate system, specified as a 3x3 rotation matrix
TranslationLeftRGB - translation from RGB camera coordinate system to left
camera coordinate system, specified as a 3x1 vector in millimeters
The calibration data read/write API allows user to upload both INTRINSICS and
EXTRINSICS. The user custom algorithm is free to optimize all these parameters. The
sample in this document optimizes both intrinsic and extrinsic parameters.
The following table shows how the various calibration tools would impact calibration
parameters for Intel® RealSense™ D400 series depth cameras.
Factory
Calibration
OEM
Calibration
Technician
Calibration
User Custom
Calibration
Dynamic
Calibration
Overview
11
Intrinsic
x
x
x
x
Extrinsic
x
x
x
x
x
2.3 Frame Formats Used in Custom Calibration
The following unrectified calibration frame formats are available. The Custom
Calibration Sample App uses these formats.
Table 2-1. Frame Formats Used in Custom Calibration
Format
SKU
Used
Comment
Y16
(16-bit)
D400
Left and Right Sensors:
1920x1080 @ 15 FPS
Intel® RealSense™ Camera D400, D410, D415
D410
D420
Left and Right Sensors:
1280x800 @ 15 FPS
Intel® RealSense™ Camera D430 D420, D435
D430
YUY2
D415
RGB Sensor:
1920x1080 @ 15 FPS
Intel® RealSense™ Camera D415, D435
D435
2.4 Frame Sync
The calibration frames are not synced. To avoid possible motion blur, calibrate the
device with images captured at static positions. In this sample, a tripod is used to keep
the camera device at fixed positions while capturing the images for calibration. This is
same if you want to develop your own custom calibration app.
2.5 Accuracy
The calibration will not be accurate if the checkerboard doesn’t sufficiently cover the
field of view or the poses are not diverse enough between captures. The calibration will
Overview
12
also not be accurate if the camera or board is handheld (or not completely still) as there
is no framesync in calibration mode.
Setup
13
3Setup
This section describes the required hardware and software setup for running the
Custom Calibration Sample Application to calibrate a device and developing a Custom
Calibration Application.
3.1 Hardware
The hardware required including a calibration target, D400 series device to be
calibrated, USB cable, and a Windows 10 computer. To ensure the images are synced,
pictures are captured at static device positions, a tripod is used to support the device
during calibration.
Figure 3-1 Hardware Setup
Setup
14
3.1.1 Device
Intel® RealSense™ D415 device as shown below is used to show case the custom
calibration process. All D400 series devices can use the same process.
Figure 3-2 D415 Device
3.1.2 Target
The target used in this custom calibration example is an 8x7 checkerboard with a
60x60mm checker size. The target image pdf 540x480_60mm.pdf is included in the
software package described in 3.2.1 Custom Calibration Sample Application. The
target pdf is located under the target directory. Users must ensure that when printing
the target, the target is not scaled.
A developer may choose to use a different target, but will need to modify the sample
application accordingly and recompile.
Setup
15
Figure 3-3 8x7 60x60 mm Checker Calibration Target
3.1.3 Tripod
Any medium sized tripod should be sufficient. In this example setup, we used a
Manfrotto Compact Light Aluminum Tripod with ball head. The ball head makes
adjustment to device orientation easier.
Setup
16
Figure 3-4 Tripod
Setup
17
3.1.4 USB
A long USB type C cable to connect the device to the host computer where the custom
calibration sample application will run.
3.1.5 PC
A Windows 10 computer.
3.2 Software
Install the following to the windows computer.
3.2.1 Custom Calibration Sample Application
A CustomCalibrationSample.zip package is provided for the Custom Calibration
Sample Application. It includes the binary executable and a calibration target file. The
source code is included in Intel® RealSense™ Calibration API package.
Unzip the package:
CustomCalibrationSample
|
|---------Bin
| |-------- CustomCalibrationSampleApp.exe
|
|--------- Target
| |-------- 540x480_60mm.pdf
|
|--------- CustomResult
Bin folder contains the executable. CustomCalibrationSampleApp.exe is the
executable for the Custom Calibration Sample Application.
Target contains the target image pdf 540x480_60mm.pdf. User will need to print it
without scaling.
CustomResult contains all device calibration results.
Setup
18
3.2.2 Intel® RealSense™ Calibration Tool and API
The calibration example uses support functions provided by the Intel® RealSense™
Calibration API to store updated calibration parameters into the non-volatile storage
on the Intel® RealSense™ D400 series modules. Required functions were first
introduced in version 2.5.2.0 of the Dynamic Calibration API, so install the latest
version and ensure that it is version 2.5.2.0 or later. Table 3-1 lists where to
download the Intel® RealSense™ Calibration API software and associated
documentation.
Table 3-1. Intel® RealSense™ Calibration API Resources
Resource
URL
Intel®
RealSense™
Depth Module
D400 Series
Dynamic
Calibration
Software
Download
https://downloadcenter.intel.com/download/27415/?v=t
Intel®
RealSense™
Depth Module
D400 Series
Dynamic
Calibration User
Guide
https://cdrd.intel.com/v1/dl/getContent/574999.htm
Intel®
RealSense™
Depth Module
D400 Series
Dynamic
Calibration
Programmer
Guide
https://www.intel.com/content/www/us/en/support/articles/000026724.html
Find Calibration Tool and Calibration API installers and follow instruction to install.
The latest version 2.5.2.0. After installation, the directory structure should look like
below:
CalibrationToolAPI
|
|-------- 2.5.2.0
| |-------- bin
| | |-------- Intel.Realsense.CustomRW.exe
|
| |-------- Include
| |-------- attributions.txt
Setup
19
| |-------- release_notes.txt
| |-------- license.txt
| |-------- gimbal
| |-------- target
| |-------- examples
| | |-------- CustomCalibration
Intel.Realsense.CustomRW.exe under bin directory is the calibration data read/write
tool. We will be using it to update the calibration parameters to the device after
running the Custom Calibration Sample App and obtained optimized parameters for
the device.
CustomCalibration under examples directory is the Custom Calibration Sample App
source code and project solution files to compile it. We will go over it later in this
document.
3.2.3 Intel® RealSense™ SDK
Install the latest release of the Intel® RealSense™ SDK so your custom calibration
application can use LibRealSense to conveniently open and access Intel® RealSense™
D400 series modules. Table 3-2 contains pointers to the SDK homepage, GitHub*
repository where you can download the latest release, and the SDK documentation.
Table 3-2. Intel® RealSense™ SDK Resources
Resource
URL
Intel® RealSense™
SDK
Home Page
https://software.intel.com/en-us/realsense/sdk
LibRealSense GitHub*
https://github.com/IntelRealSense/librealsense
SDK Documentation
https://github.com/IntelRealSense/librealsense/tree/master/doc
3.2.4 OpenCV 3.3.0
The calibration example presented in this document uses several OpenCV 3.3.0
libraries for computations, so OpenCV 3.3.0 must be installed. Table 3-3 lists URLs
for downloading OpenCV 3.3.0, and Table 3-4 lists the specific OpenCV libraries that
are required by the example code.
Setup
20
Table 3-3. OpenCV 3.3.0 Resources
Resource
URL
OpenCV GitHub* Repository
https://github.com/opencv/opencv
OpenCV 3.3.0 Download
https://github.com/opencv/opencv/releases/tag/3.3.0
Table 3-4. OpenCV 3.3.0 Libraries Required for the Example
OpenCV 3.3.0 Library
ittnotify.lib
opencv_calib3d330.lib
opencv_core330.lib
opencv_features2d330.lib
opencv_flann330.lib
opencv_imgproc330.lib
zlib.lib
3.2.5 Glut Library
The FreeGLUT package is a convenient open source alternative to the OpenGL Utility
Toolkit for displaying graphical output to the operator of the custom calibration
application. You can download and install the FreeGLUT package at
http://freeglut.sourceforge.net/.
Table of contents
