Machines learning what an X is

Today I worked on attempting to detect a user creating an 'X' with their hands/arms solely based on the depth data from the Kinect camera. I went through about 3 or 4 iterations till what I am at none, and unfortunately none of them have had much success :(.

Essentially I was feeding in positive and negative samples of an 'X', labeling them as such, and then trying to figure out a good way to classify these samples such that in the future a computer would be good at discovering them.

Examples (first 3 positive, last 3 are negative):

At first I was just doing general histograms of the value part (of the image in HSV format), without much success at all, as this had no relation to the location of the values. Several iterations later and I was breaking down the positive and negatives images into 3x3 grids and doing average values of value in relation to where they were located with some minor success...

Yah.... going to need to do something different.