Making Microsoft Lync be able to see and hear Part 3/3

In part one of this series I discussed how you can use the Kinect V2 for Windows to make programs listen without having to enable the terrible Windows.Speech library.  In part two we used our hands as a mouse cursor to manipulate large screen buttons.  Part three will focus on gestures.  In our example there are 2 spots where using a gesture will come in handy, answer and hang-up.  We may not have time to say "answer" then "confirm" to get the call quick enough.  We may not have time to activate the pop out menu and move our hand over the answer button and click, as those real time operations need to happen in a split second.  Here is where the Kinect's ability to handle either coded or ML (machine learning) gestures comes in handy.  This post is going to use code to make simple gesture detection, as ML requires quite a bit of work with Kinect Studio and Gesture Builder, which would be preferable for a more complicated scenario.

To get started we can already use some of the code we are using to listen for speech and hands in the PHIZ.  I'll add some things in the window class for the gesture detection, the most notable being the BodyFrameReader.

        private BodyFrameReader bodyFrameReader = null;
        private Body[] bodies = null;
        private List<Tuple<JointType, JointType>> bones;

From here we will need to initialize some things in our window load.

 try
            {
                //body frame
                FrameDescription frameDescription = this.kinectSensor.DepthFrameSource.FrameDescription;
                this.bodyFrameReader = this.kinectSensor.BodyFrameSource.OpenReader();
                #region bones
                this.bones = new List<Tuple<JointType, JointType>>();
                // Torso
                this.bones.Add(new Tuple<JointType, JointType>(JointType.Head, JointType.Neck));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.Neck, JointType.SpineShoulder));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.SpineShoulder, JointType.SpineMid));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.SpineMid, JointType.SpineBase));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.SpineShoulder, JointType.ShoulderRight));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.SpineShoulder, JointType.ShoulderLeft));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.SpineBase, JointType.HipRight));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.SpineBase, JointType.HipLeft));
                // Right Arm
                this.bones.Add(new Tuple<JointType, JointType>(JointType.ShoulderRight, JointType.ElbowRight));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.ElbowRight, JointType.WristRight));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.WristRight, JointType.HandRight));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.HandRight, JointType.HandTipRight));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.WristRight, JointType.ThumbRight));
                // Left Arm
                this.bones.Add(new Tuple<JointType, JointType>(JointType.ShoulderLeft, JointType.ElbowLeft));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.ElbowLeft, JointType.WristLeft));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.WristLeft, JointType.HandLeft));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.HandLeft, JointType.HandTipLeft));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.WristLeft, JointType.ThumbLeft));
                // Right Leg
                this.bones.Add(new Tuple<JointType, JointType>(JointType.HipRight, JointType.KneeRight));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.KneeRight, JointType.AnkleRight));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.AnkleRight, JointType.FootRight));
                // Left Leg
                this.bones.Add(new Tuple<JointType, JointType>(JointType.HipLeft, JointType.KneeLeft));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.KneeLeft, JointType.AnkleLeft));
                this.bones.Add(new Tuple<JointType, JointType>(JointType.AnkleLeft, JointType.FootLeft));
                #endregion
            }
            catch { }


Then lastly we are going to want to be notified when the body frame reader gets a new frame.  Note the body frame reader is capable of seeing and identifying the movements of 6 different bodies simultaneously.


try
            {
                if (this.bodyFrameReader != null)
                {
                    this.bodyFrameReader.FrameArrived += this.Reader_FrameArrived;
                }
            }
            catch { }

Now that we have it all set up it's just a matter of handling the frames and seeing if either of our 2 conditions are met.

1.  Closed fist right hand over head - answer

2.  Closed fist left hand over head - hang-up


Why the closed fist?  Since our menu will pop out when we have an open hand facing the screen, the closed fist ensure no confusion between our menu and our gestures.


Before we code the frame read event we need to have a function to be able to tell if the body part in question is over the head.

 private bool IsHandOverHead(Body body, JointType jointType)
        {
            bool isDetected = false;
            var head = body.Joints[JointType.Head];
            var hand = body.Joints[jointType];
            if (hand.Position.Y > head.Position.Y)
                isDetected = true;
            return isDetected;
        }

So the final part of our gesture detection will be to process the frames, I made a couple of global variables to make sure I didn't try to hang-up or answer 30 times a second when I was already in the process of one of the those tasks, since there was a tremendous amount of frame data being send our way.

 private void Reader_FrameArrived(object sender, BodyFrameArrivedEventArgs e)
        {
            bool dataReceived = false;
            using (BodyFrame bodyFrame = e.FrameReference.AcquireFrame())
            {
                if (bodyFrame != null)
                {
                    if (this.bodies == null)
                    {
                        this.bodies = new Body[bodyFrame.BodyCount];
                    }
                    // The first time GetAndRefreshBodyData is called, Kinect will allocate each Body in the array.
                    // As long as those body objects are not disposed and not set to null in the array,
                    // those body objects will be re-used.
                    bodyFrame.GetAndRefreshBodyData(this.bodies);
                    dataReceived = true;
                }
            }
            if (dataReceived)
            {
                foreach (Body body in this.bodies)
                {
                    if (shouldAnswer == false && IsHandOverHead(body, JointType.HandRight))
                    {
                        if (body.HandRightState == HandState.Closed)
                        {
                            shouldAnswer = true;
                            Task.Factory.StartNew(() =>
                            {
                                //DO YOUR ANSWER CODE HERE
                                Thread.Sleep(1500);
                                shouldAnswer = false;
                            });
                        }
                    }

                    if (shouldHangup == false && IsHandOverHead(body, JointType.HandLeft))
                    {
                        if (body.HandLeftState == HandState.Closed)
                        {
                            shouldHangup = true;
                            Task.Factory.StartNew(() =>
                            {
                                //DO YOUR HANGUP CODE HERE
                                Thread.Sleep(1500);
                                shouldHangup = false;
                            });
                        }
                    }
                }
            }
        }

That will wrap it up for part 3.  Check back in a couple of weeks when we demonstrate the entire process in a 4k video production.

Doug Routledge, C# Lync, SQL, Exchange, UC Developer