Hand gesture using opencv

Hi ! In this post I will be describing the code for hand gesture recognition using OpenCV.The code is written in C on Dev C++.For installing the necessary libraries on Dev C++ you can check my previous post. So basically to start with I had to extract the hand region .Which can be done by many ways for eg                                                                                                                                                                                  1) you can segment the hand region using RGB values i.e.R G B values of hand will be different from background                                                                                                                                                         

OR

2) you can use edge detection

 OR

3) background subtraction.

     I have used background subtraction model. OpenCV provides us with different back ground subtraction models I choose codebook ( no specific reason).What it does is it calibrates for some time to be exact for some frames.In which for all the images it acquires; it calculates the average and deviation of each pixel and accordingly designates boxes. For more information please refer a book.

     So at this stage we have removed the background and in the foreground we only have our hand. For those who are new to CV it is like a black and white image with only the hand as white.

  

   In the next part what we intend to do is recognise the gesture. Here we use Convex Hull to find the finger tips.Convex hull is basically the convex set enclosing the hand region.

     The red line bounding the hand is convex hull .Basically it’s a convex set ; means if we take any two points inside the red region and join them to form a line then the line entirely lies inside the set.

     The yellow dot is the defect point and there will be many such defect points i,e every valley has a defect point. Now depending upon the number of defect points we can calculate the number of fingers unfolded.

summary :-

• The hand region extraction has been done using background substraction using codebook method.
• For Tip points i have used convex hull 2 and for depth points convexity defects.

The main code for extracting the contour and detecting the convexity points is in the function

void detect(IplImage* img_8uc1,IplImage* img_8uc3);

Place the camera in front of a steady background ; run the code ,wait for some time .Once the calibration has been done . U see the connected component image showing some disturbance.Bring your hand in cameras view . Enjoy .

VIDEOS:-

convex hull 2 usage

http://www.youtube.com/watch?v=OJ_u8nDNmvs&feature=related

hand gesture

http://www.youtube.com/watch?v=TFW6bMPXH4U&list=UUlCqqRGDRih0m-L0SQFgGxg&index=1&feature=plcp

CODES:-

Link 1 : Convex Hull2 usage

Link 2 : Hand gesture recognition

                    FOR OPENCV 2.4

 Background subtraction has been done using codebook.
My code has been written over the basic example available in the opencv examples for codebook.So all that i have written has been included in a new function named detect() .

void detect(IplImage* img_8uc1,IplImage* img_8uc3) {

//8uc1 is BW image with hand as white And 8uc3 is the original image

CvMemStorage* storage = cvCreateMemStorage();
CvSeq* first_contour = NULL;
CvSeq* maxitem=NULL;
double area=0,areamax=0;
int maxn=0;

//function to find the white objects in the image and return the object boundaries

int Nc = cvFindContours(
img_8uc1,
storage,
&first_contour,
sizeof(CvContour),
CV_RETR_LIST // Try all four values and see what happens
);

int n=0;
//printf( "Total Contours Detected: %d\n", Nc );

//Here we find the contour with maximum area

if(Nc>0)
{
for( CvSeq* c=first_contour; c!=NULL; c=c->h_next )
{
//cvCvtColor( img_8uc1, img_8uc3, CV_GRAY2BGR );
area=cvContourArea(c,CV_WHOLE_SEQ );
if(area>areamax)
{areamax=area;
maxitem=c;
maxn=n;
}

n++;
}


CvMemStorage* storage3 = cvCreateMemStorage(0);
//if (maxitem) maxitem = cvApproxPoly( maxitem, sizeof(maxitem), storage3, CV_POLY_APPROX_DP, 3, 1 );


if(areamax>5000) //check for area greater than certain value and find convex hull
{
maxitem = cvApproxPoly( maxitem, sizeof(CvContour), storage3, CV_POLY_APPROX_DP, 10, 1 );
CvPoint pt0;
CvMemStorage* storage1 = cvCreateMemStorage(0);
CvMemStorage* storage2 = cvCreateMemStorage(0);
CvSeq* ptseq = cvCreateSeq( CV_SEQ_KIND_GENERIC|CV_32SC2, sizeof(CvContour),
sizeof(CvPoint), storage1 );
CvSeq* hull;
CvSeq* defects;
for(int i = 0; i < maxitem->total; i++ )
{ CvPoint* p = CV_GET_SEQ_ELEM( CvPoint, maxitem, i );
pt0.x = p->x;
pt0.y = p->y;
cvSeqPush( ptseq, &pt0 );
}
hull = cvConvexHull2( ptseq, 0, CV_CLOCKWISE, 0 );
int hullcount = hull->total;
defects= cvConvexityDefects(ptseq,hull,storage2 );
//printf(" defect no %d \n",defects->total);

CvConvexityDefect* defectArray;
int j=0;
//int m_nomdef=0;
// This cycle marks all defects of convexity of current contours.
for(;defects;defects = defects->h_next)
{
int nomdef = defects->total; // defect amount
//outlet_float( m_nomdef, nomdef );
//printf(" defect no %d \n",nomdef);
if(nomdef == 0)
continue;
// Alloc memory for defect set.
//fprintf(stderr,"malloc\n");
defectArray = (CvConvexityDefect*)malloc(sizeof(CvConvexityDefect)*nomdef);
// Get defect set.
//fprintf(stderr,"cvCvtSeqToArray\n");
cvCvtSeqToArray(defects,defectArray, CV_WHOLE_SEQ);
// Draw marks for all defects.
for(int i=0; i
{ printf(" defect depth for defect %d %f \n",i,defectArray[i].depth);
cvLine(img_8uc3, *(defectArray[i].start), *(defectArray[i].depth_point),CV_RGB(255,255,0),1, CV_AA, 0 );
cvCircle( img_8uc3, *(defectArray[i].depth_point), 5, CV_RGB(0,0,164), 2, 8,0);
cvCircle( img_8uc3, *(defectArray[i].start), 5, CV_RGB(0,0,164), 2, 8,0);
cvLine(img_8uc3, *(defectArray[i].depth_point), *(defectArray[i].end),CV_RGB(255,255,0),1, CV_AA, 0 );
}
char txt[]="0";
txt[0]='0'+nomdef-1;
CvFont font;
cvInitFont(&font, CV_FONT_HERSHEY_SIMPLEX, 1.0, 1.0, 0, 5, CV_AA);
cvPutText(img_8uc3, txt, cvPoint(50, 50), &font, cvScalar(0, 0, 255, 0));
j++;
// Free memory.
free(defectArray);
}

cvReleaseMemStorage( &storage );
cvReleaseMemStorage( &storage1 );
cvReleaseMemStorage( &storage2 );
cvReleaseMemStorage( &storage3 );
//return 0;
}
}
}

thank you!! :)

Line Tracker with PID

LINE TRACKER

LINK TO THE CODES

https://docs.google.com/open?id=0B7lDtwez94H3ZWUxNDM0MjktNjE5Mi00ZDFhLWI2ZTAtZjI4MmUwZDcwMzFh

This was my first project and i did it when i was in my first year of graduation.Line tracker is a best way to put your hand into robotics.In this post i will teach you to make a line tracker along with the advanced PID controller.






Firstly we define a line tracker: It is a robot which follows a line.So we need to program a robot to track line. To do this we need to give some kind of input to the robot to let it know where the line is .This is where line sensors come in .And to drive the robot we need some kind of actuators(motors).

Materials required

Mechanical:
2 DC geared motors 100rpm Rs 125 each
2 L shaped clamps to hold motors Rs 15 each
1 Castor wheel Rs 15
some wood or acrylic or aluminium to build a chassi .( For my first bot i used a plastic box)

Electronics:

Dev board

1 Atmega16 microcontroller
1 40 pin mount
berg strips
connecting wires
paraller port connector DB25 male
330 ohms resistor
IC 7805
IC L293D
PCB

Sensor Board

8 pair of IR (Tx ,Rx)
330 ohms resistor
10k ohms resistor
PCB

Circuit Diagram


D ( 1 3 5 7 9 11) - tx ; D ( 2 4 6 8 10) - Rx ; R ( 1 3 5 7 9 11) - 330 ohms ; R ( 2 4 6 8 10) - 10Kohms ;
In the above circuit the first one is transmitter circuit and below we have receiver circuit.
The transmitter and receiver should be placed one below other.

Working:
A transmitter is a simple infrared led .It emits infrared light when forward biased.
While the receiver is a photo diode it is used in reverse biased state.when infra red light falls on it the resistance across the reverse biased diode decreases. This property is used to detect white and black surface.Now consider your sensor pair ( Tx and Rx) is on white line , in this case the IR light emitted by Led is completely reflected back by white surface and this light falls on Rx due to which the resistance across Rx decreases .And the output(lf,l2,l1,r1,r2,rf) vary i.e. under normal conditions the resistance of Rx is infinity therefore the voltage across 10k resistance may be close to 0V.And when IR light falls; it may increase as the diode resistance decreases drastically and becomes comparable to 10k .
With the above information we can fairly judge the o/p under white and black surfaces.
Black : o/p will be high
white : o/p will be low


Here are some pictures of sensor board.These pictures are of different sensor board.It has 8 sensor pair but there is never a need of 8 sensor 6 should suffice for us.





Ok ! Now we are done with sensor board its time to test it .How to test it ?
Its really simple take a mutimeter connect one end(black) to gnd other(Red) to sensor output (use multimeter in voltage measuring mode select voltage range below 10/20 V).
Now measure the value of output with your hands on the sensor(make sure your hands be just above the sensors and very close to them around 1cm above), lets call it V1.Now remove your hands completely and let the area above the sensors be open and take the reading, let it be V2.
The sensors work properly if there is a substantial difference between V1 and V2 .V1 sould always be greater than V2. And V1-V2 should be approx around 1 volt.But this may differ with the environment . You may get false readings if you test this in sunlight.Sunlight contains a lot of IR light .So if you are facing problems during day and everything works fine at night then sunlight is the only problem.The only solution is to cover the sensors.

The above board has 8 sensors but its not necessary we can use 4 or 6 of them and follow a line perfectly well.

Now moving to drive system .We will be needing a Motor driving Circuit to drive the 2 motors of the robot.
You can google for more info about the motor driver circuit. I used L293D as driving IC. It is a simple H bridge driving circuit.
The IC can drive 2 motors. and takes 4 inputs 2 for each . Which lets motor control in both directions.

And finally we have a micro controller which does the part of controlling the motor depending on the inputs from the Sensor board.