Kinect Orientation

Finger and gesture recognition with

Microsoft Kinect

Daniel James Ryan

Department of Electrical and Computer Engineering

University of Stavanger

ii

Abstract

The Kinect is a cheap and wildly spread sensor array with some interesting

features, such as a depth sensor and full body skeleton tracking.

The Kinect SDK does not support nger tracking; We have therefore created

algorithms to nd nger positions from depth sensor data. With the detected

nger position we use dynamic time warping to record and recognize nger

gestures.

We have created an API that can record and recognize nger gestures. The

API was created with focus on ease of use and the possibility to customize

and change core algorithms. Even the Kinect can be changed for other

similar devices. This opens up a new eld of applications utilizing the Kinect

and nger gestures.

iv

Acknowledgements

I would like to thank professor Reggie Davidrajuh for providing valuable

feedback on writing this thesis.

I would also like to thank Glenn F. Henriksen at EVRY for providing con-

structive feedback on coding the API and for programming advice in general.

ii

Contents

List of Figures vii

Glossary ix

1 Introduction 1

1.1 Inspiration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Existing solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 The Kinect sensor 3

2.1 Kinect versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.1 Commercial and development usage . . . . . . . . . . . . . . . . 3

2.2 Specs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.3 Kinect for Windows SDK . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.3.1 Data streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.4 Possibilities with the Kinect . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Contour tracking 7

3.1 Existing algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3.2 Finding the initial pixel . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3.3 Tracking the contour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.3.1 Search directions . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.3.2 Single pixel lines . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.3.3 Backtracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.3.4 Track two hands . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.4 Termination states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

iii

CONTENTS

3.4.1 Termination state 1: A pixel was discovered twice. . . . . . . . . 13

3.4.2 Termination state 2: A xed number of pixels were discovered. . 13

4 Finger recognition 15

4.1 Related works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.2 Curve detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.3 Finger detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

5 Gesture Recognition 19

5.1 Related works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

5.2 Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

5.3 Recording a gesture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5.4 Recognizing a gesture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5.4.1 Step 1: Finding a candidate gesture . . . . . . . . . . . . . . . . 20

5.4.2 Step 2: Euclidean distance matrix . . . . . . . . . . . . . . . . . 21

5.4.3 Step 3: Accumulated distance matrix . . . . . . . . . . . . . . . . 21

5.4.4 Step 4: Lowest distance path . . . . . . . . . . . . . . . . . . . . 22

5.4.5 Optimizations and improvements . . . . . . . . . . . . . . . . . . 22

5.4.5.1 Euclidean distance . . . . . . . . . . . . . . . . . . . . . 23

5.4.5.2 Accumulated cost matrix . . . . . . . . . . . . . . . . . 23

5.4.5.3 Lowest distance path . . . . . . . . . . . . . . . . . . . 24

5.5 Store gestures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

5.6 Retrieve stored gestures . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

6 Auto depth interval detection 27

7 Enhancements 31

7.1