LMNts’ third blog post and image are up on Metropolis Magazine’s POV blog where Scott Crawford shares his thoughts on a design technology Cabinet of Curiosities. Be sure to also checkout Confessions of a Design Technology Evangelical and Re-Upping on Design Technology if you haven’t already.
As we mentioned in the previous post, a complete code walk-through for the Kinect SDK is beyond the scope of this blog post. Microsoft has some great documentation and support for getting up and running with the SDK. That said, let’s cover the main event loop for the TouchGestures source code we are providing on top of Microsoft’s code…
The basic steps in our touch-as-depth software can be summarized by the flowchart at left. For each mode of operation (SurfaceCalibration, ExtentsCalibration, and MissionMode), the data flow is slightly different. The flowchart displayed is that of the typical MissionMode flow loop. Let’s go over these one by one…
Depth information is gathered frame-by-frame from the Kinect. These depth points are stored in the depth buffer. Next, the base interaction surface is found in relation to the location of the Kinect (as determined by the SurfaceCalibration routine). During MissionMode, if it’s the first pass through, all the depth data is adjusted relative to the calibration matrix. In all subsequent frames, this depth data is then compared relative to the surface to determine whether or not there is a “touch” event. The raw touch-from-depth data is then filtered using a custom mean filter that weeds out some of the extraneous noise…some, but not all. Connected 8 component labeling is used to disambiguate one finger from another. Particle analysis is then performed to figure out exactly which points are “touch” points and which are noise…at this point, the data should not be noisy. Now that we have the touch points, we need to make sure they line up with the spots in the real world. The raw points as seen by the Kinect are then transformed relative to the tracked space (as determined by the ExtentsCalibration routine). Finally, a filter is applied to the frame-to-frame behavior of the points to determined their mapped keyboard and mouse event. Rinse and repeat…as fast as possible. Now let’s go through each of the steps in more detail… (more…)
1.Kinect Sensor | 2.Raw Depth Image | 3.Segmented Depth Image | 4.After Lowpass Filter | 5.Connected Component Labeling | 6.Particle Analysis | 7n.Gesture Filters | 8.Keyboard/Mouse Events
In our previous post, we talked about our efforts to use the Microsoft Kinect to control a large-scale table-top interface running Grasshopper. In short: we used the Kinect to sense touch and gestures…which we then used to control the canvas via keyboard and mouse events. We’ve had a lot of fun building Kinect Multitouch Interactions but – being an architecture firm – we can only spend so much time developing the code. We think we’ve created a solid foundation and would like to share with the broader community to use, modify, and extend. Obviously, Grasshopper is only one possible application and we’d love to see how others will use this interaction. So, in the spirit of openness, we’re providing the complete source – as well as Visual Studio solution files – downloadable below. It’s our hope that those of you reading this will adapt and improve on what we’ve started.
BE WARNED: Before you jump in and try this at home, we encourage you to read this post (and the next one) to get a sense of how we went about this. Rest assured, downloading and compiling our sample code is not a terribly complex endeavor, provided you have some experience with linking and building applications. You’ll need to have some experience with C/C++ programming…so please read the detailed technical descriptions included in the SDK (and in these posts).
To get started, you’ll need the following…
Prerequisites/Dependencies
1. Microsoft Kinect for XBox with external power source and USB cables.
2. Microsoft Windows 7 or later.
3. Kinect for Windows PC SDK with drivers. We used the first public beta.
4. Microsoft .NET Framework 4.0.
5. VisualStudio 2010 IDE or similar. Express editions should work fine.
6. Our source code, which includes the Visual Studio Solution files…scroll down.
Optional Equipment
7. Projector (or similar) display. Our setup, while not expensive, is a bit exotic.
8. Kinect stabilizing mount (for walls, tables and ceilings).
Kinect Multitouch Interaction by LMNts is copyleft and licensed under the GNU General Public License.
Obligatory Disclaimer/Terms and Conditions of Use: By downloading, you agree to use this software at your own risk. Under no circumstances shall LMN or LMNts be liable for direct, indirect, special, incidental, or consequential damages resulting from the use, misuse, or inability to use this software. The software is provided “as is,” whereas we cannot provide a guarantee of support. LMNts does not guarantee that this software is bug-free or that it will solve all your problems…
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
Ok, let’s get into the fun details of how this works… (more…)
This Holiday Season, LMNers lit candles…LOTS of candles. The video was directed, shot, edited and produced by: cbaker, jbeaver, scrawford, cgrammens, kgregga, thenderson, ahunter, kkimura, amacdonald, and gshaw.
Guest blogging at Metropolis Magazine, dbelcher organizes his Field Notes. (Meanwhile, the Desk Liberation Front launches another photographic assault). For an pseudo-iconoclastic-interlude to all this computational fetishism, go read the Confessions of a Design Technology Evangelical.
