More fun with the marvelous OpenCV library. While red is still a convenient color to track, I’m no longer using real vegetation. Instead, I found an old pair of gloves that do the job quite well.
(Get it for yourself at github.)
A brief explanation: I’m no longer using the library’s object-tracking algorithms. Instead, the mouse is just tied to the position of the “average” red pixel, with some trivial logic for throwing away outliers. Once the bounding box of the glove drops below a certain width, a click event is triggered. I simulate clicks and mouse motion with the XTest library.
(The interaction is not normally this laggy, but istanbul clobbers performance when compositing is enabled. Disable compositing, you say? I need my exposé: the window buttons on a taskbar would be too small to use reliably. The system’s not perfect. Yet.)
Some thoughts:
The click interaction really needs to be improved. There’s just enough “wobble” between the camera and an unsteady hand to make it very difficult to click without dragging, and perspective makes the rotate gesture almost impossible to execute when not dead center.
In the same vein, it would be good if I could get the program to even roughly judge the distance of the glove to set the threshold for click events dynamically.
Once I get the kinks ironed out of the current setup I’m going to try moving up to two gloves, but I think I have a way to go first. Glove-in-air interaction will never be a good model for desktop use, but I could see this being useful in applications (games in particular) that require only gross input.