ARCADE and SAM are hosting an event on May 10 at the SAM where Pierluigi Serraino and John Marx will discuss their recently published book, Wandering the Garden of Technology and Passion.  Scott has been invited to be part of the discussion panel following the presentation.  More details located here.

Parametric modeling was not used during the design of the Veil, but the modeling work-flow was like a mental parametric model where the critical parameters and logic of the system were held in the minds of a few individuals instead of a Grasshopper definition.  The designer developing the study models for the Veil was very rigorous in forming the logic of the system and was capable of producing a large number of studies by continuously modifying or remodeling the Veil using linear arrays.  LMNts was asked to help develop a Building Information Model that could be used by the contractor to do more detailed quantity takeoffs as well as address a few issues that proved difficult to solve through the direct modeling process.

There are a number of similarities between the layout of this system of panels and the way that we developed the base geometry for the Cleveland Medical Mart.  Both approaches begin with a shell model that is broken down into a grid according to the width and height module of the base unit panel.  An issue that often arises with modular systems is what happens when the dimensions of a face aren’t a multiple of the module.  For example, the north and south elevations of the Medical Mart were 195′ wide and we were using a 10′ module.  This left us with a difference of 5′ to the closest module multiples of 190′ and 200′.  The difference was taken up in uniquely sized corner panels, a piece of the facade that was going to be unique no matter what.

The Veil is not a water tight enclosure so the layout flexibility of the system was taken up in the gap between panels which could range from 0.25″ to 0.50″.  This was originally causing problems in the direct modeling approach because even slight changes to the base geometry required the panel spacing to be recalculated for each face.  Once we had the Veil modeled in Grasshopper it was possible to calculate the number panels that needed to occur on each course and then adjust the gap to be within the acceptable range.  Adjusting the gap size also afforded us more flexibility when trying to align the panels with openings and structure.

The design team had established a variety of panel types and organization patterns for how the panels were arrayed across the building.  Here again there was a series of rules that the patterns were following, making it fairly straightforward to replicate the design.  The challenge with parametric modeling process is incorporating exceptions to rule.  For every exception there needs to be a system of logic defined to always pull out the exception and handle it accordingly.  These exceptions end up carrying over into the fabrication and construction process as well.  The more exceptions, the more room for potential errors to occur.

In the above image, there are a series of trapezoidal panels that occur along the top edge of the Veil. Each of these panels are unique and there are over 1300 of them.  The current fabrication plan for the project is to field cut full-size panels.  We’ve digitally modeled every one of those panels so there’s the potential that shop drawings could be provided for all of them.  To demonstrate this potential we put together a work-flow that writes to a spreadsheet the left and right edge height for each panel and these values are then used as a Design Table for an Autodesk Inventor sheet metal file.  We don’t often use Inventor but fabricators do so we thought the fabricator building the Veil might be interested in this information.  

Turns out we were wrong for a few reasons.  They were intrigued that we could provide them with that information, but at that time the fabricator was only responsible for installing the panels and another contractor was responsible for installation.  Therefore, the time investment in machine code setup, tagging and crating the 1300+ panels had no payoff for them.  From our point of view, shop cutting the panels seemed like it would offer more control and likely a better end product.  The increased setup time and effort involved with tracking the unique panels could potentially be offset through material and labor savings.

In the end, means and methods are not within our scope of responsibilities and we’re not responsible for selecting who builds what on the project so there are limits to how much we can influence the decision this time.  However, more and more we are developing models that contain a significant amount of information that could be useful in the fabrication process and we’re looking for opportunities to share that information rather than simply handing over 2D drawings and asking someone to take their best shot.  We’re now trying to make our consultants and collaborators more aware of the control we have over our models and the information we can extract from them with the hope that someone will take full advantage of this information.

LMNts’ most recent blog post for Metropolis Magazine’s POV blog has been published.  Sam Miller, a partner at LMN, shares his thoughts on Advancing a Data-Driven Approach to Design.  Be sure to also checkout Architectural Storytelling vs Public Relations, Cabinet of Curiosities, Confessions of a Design Technology Evangelical and Re-Upping on Design Technology if you haven’t already.

October 3, 8AM: The two diagrams show the same result for this time period.

The way the overshadowing diagram works can be shown in the images above. On the left, we are in shade because of the immediate shadows cast (we are the red dot in the center). On the right, we are in shade because the sun is contained in an overshadowed area.

The basic overshadowing diagram is effective for point based shadow studies, but not very intuitive. It’s difficult to understand and hard to explain to others. While generated three dimensionally, it must be studied in plan, and those unfamiliar with it often visualize the surfaces as a representation of literal building shadows rather than the abstraction of a shadow range. In short, the diagram is presented radially but wants to be interpreted linearly.

To remedy this, the Rorschach Shadow diagram translates the overshadowing diagram into a basic bar graph. Rather than just overlaying a sun path diagram onto an overshadowing diagram, the Rorschach considers the hybrid of these two metrics. To generate this, sun path curves are projected onto the overshadowing diagram. The curves are then split between areas of shade and areas of sunlight. These curves are unrolled to create a bar graph showing black and white representations of shade and sunlight. The user can then control period of time to study (seasonal, monthly, weekly, daily, etc.). Other parameters can be controlled at the same time. For example, a deciduous tree’s transparency can be flexed parametrically in accordance with the day of the year.

The original Rorschach test. What do you see?

By splitting our sun paths between areas of shade and sunlight, we are also able to quantify the percentage of daylight hours that a point of interest is in the sun. This allows us to compare multiple points on our site to determine areas of maximum and minimum shading. The Rorschach metric enables us to analyze the content of the overshadowing diagram while preserving the legibility of a bar graph.

The Rorschach diagram at contrasting latitudes shows different results for the same overshadowing diagram.

Nearly 100 years ago when San Antonio’s Municipal Auditorium was first built, it was among the city’s first major civic structures.  Over the last century, the city has experienced a significant evolution – both in terms of urban development and the cultural arts. The city recognized the need to respond to that evolution with a new performing arts center that is evocative of its new urban condition and community significance.

The city’s goals for the Tobin Center for the Performing Arts is to create a world-class home for the major performing arts organizations of San Antonio, and establishes the center as a vital part of the cultural life of San Antonio, Bexar County and all of South Texas.  The design integrates two new performance spaces with portions of the historic Municipal Auditorium to create a new center serving the symphony, opera, ballet and a rich diversity of theatrical programs.

A design concept was to create moments in the building that reflected the patterns and spectacle that can be found throughout the city, the riverwalk, festivals and regional culture.  Of particular interest was how these patterns could be made more dynamic through the use of lighting, either natural or artificial.  Our future posts will focus on the Veil (shown above) which forms the exterior cladding of LMN’s addition, and the North Wall that serves as the theater’s face to the lobby (shown below on the left).

Parts of the posts will build off of ideas and processes we developed while working on the Cleveland Medical Mart. However, we plan on sharing our experiences with a few new problems that we’d hadn’t dealt with before such as management of a very large number of components (18,000+ in the Veil) and the rationalization of a doubly curved surface. More to come…

LMNts’ fourth blog post is up on Metropolis Magazine’s POV blog where Adrian MacDonald of our Marketing department shares his thoughts on Architectural Storytelling vs. Public Relations.  The three previous posts include Cabinet of Curiosities, Confessions of a Design Technology Evangelical and Re-Upping on Design Technology.

LMNts’ third blog post and image are up on Metropolis Magazine’s POV blog where Scott Crawford shares his thoughts on a design technology Cabinet of Curiosities.  Be sure to also checkout Confessions of a Design Technology Evangelical and Re-Upping on Design Technology if you haven’t already.

The basic steps in our touch-as-depth software can be summarized by the flowchart at left.  For each mode of operation (SurfaceCalibration, ExtentsCalibration, and MissionMode), the data flow is slightly different.  The flowchart displayed is that of the typical MissionMode flow loop.   Let’s go over these one by one…

Depth information is gathered frame-by-frame from the Kinect.  These depth points are stored in the depth buffer.  Next, the base interaction surface is found in relation to the location of the Kinect (as determined by the SurfaceCalibration routine).  During MissionMode, if it’s the first pass through, all the depth data is adjusted relative to the calibration matrix.  In all subsequent frames, this depth data is then compared relative to the surface to determine whether or not there is a “touch” event.  The raw touch-from-depth data is then filtered using a custom mean filter that weeds out some of the extraneous noise…some, but not all.  Connected 8 component labeling is used to disambiguate one finger from another.  Particle analysis is then performed to figure out exactly which points are “touch” points and which are noise…at this point, the data should not be noisy.  Now that we have the touch points, we need to make sure they line up with the spots in the real world.  The raw points as seen by the Kinect are then transformed relative to the tracked space (as determined by the ExtentsCalibration routine).  Finally, a filter is applied to the frame-to-frame behavior of the points to determined their mapped keyboard and mouse event.  Rinse and repeat…as fast as possible.  Now let’s go through each of the steps in more detail…

SurfaceCalibration Mode

Once you have placed your Kinect relative to the surface (a table, a wall, a chair, etc.) you want to track touch on, you need to get some very accurate readings about depth.  This is a one-time calibration…unless you move your Kinect or change the surface, in which case you will need to re-calibrate.  The SurfaceCalibration mode is responsible for modeling the interaction surface over multiple frames and a fixed spatial resolution of 640×480 depth points.  The methodology of modeling the surface and extracting touch information that we use is explained very clearly by Andy Wilson of Microsoft Research.  Unfortunately, the process does not calculate depth information adaptively.  Hence, any objects placed on the interaction surface during calibration will be modeled as being part of the touch space.  It is important to keep the field-of-view devoid of interference from direct light sources and objects (even a few seconds of interference would yield result in errors of ~10% in calibration).

SurfaceCalibration mode involves a single step as soon as the depth buffer is ready and locked.  This step contains the routine (Gelib_DoSurfaceCalibration) that translates the depth values to millimeters and stores the cumulative sum over a period.  The number of frames to track the depth values (defined by DUMPFOR parameter) and the frame interval (defined by DUMPEVERY parameter) are specified in the header file TouchGestures.h.  The calibration routine shows a progress bar.  Once the depth values are gathered, the final depth data for each pixel (one of 640×480) are calculated by finding the average values (dividing each pixel accumulated value by DUMPFOR).  The average depth values are stored in the file DepthAverage.dump (our file format uses a semicolon-space-newline delimitation).  The SurfaceCalibration.exe application may be terminated as soon as this file is generated (or as soon as the progress bar indicates completion).  At this point, the Kinect should be calibrated relative to your surface and you are ready to tell the Kinect what the boundaries of your tracked space are…for that you switch to:

ExtentsCalibration Mode

During the ExtentsCalibration mode, the system works in partial Mission mode.  Touch gestures are processed but not translated and emulated as mouse events.  Instead, touch points are recorded for a certain time period and the corresponding co-ordinates are logged in a file.

Upon initialization and depth buffer filled event, the first step is to read the surface calibration depth values as an IMAGE structure from the stored file DepthAverage.dump. The IMAGE data structure is defined as a 2D unsigned short integer array with sizes specified as DWIDTH and DHEIGHT (640, 480).

The image analysis and processing routines are similar to that of the MissionMode and will be explained in that section. The additional function (CALIBRATION Gelib_CalibrateExtents( CALIBRATION C ) ) displays and logs the touch points in the sequence. The five points are iterated through a counter (cIteration) and the calibration mode swaps to regular mode (qswap = 0) once the calibration is completed…

The Calout data structure contains the calibration reference points.  These points are stored in the Calibration.dat file.  The data flow switches to touch display mode hence forth (displays transformed touch points and does not translate and emulate mouse events).

Transforming touch points: An affine transform of the touch points is performed to translate the touch points from the depth space to the table space…

The depth space is defined as the 2D region as seen by the Kinect depth camera (this is different from the 2D video space seen by the Kinect). This is expected to be a subset of the table space which is the physical 2D region of the glass top over the table. The transforms applied are:

1. Translation in X,Y direction:  T[C0(x,y)] towards & relative to the table space origin of C0.
2. Scale in X,Y direction: S[M(x,y))] towards the upper boundaries of the table space.

The above process determines factors to calculate the transformation offset and scale factors. The translation offset and scale factors are then applied to any incoming touch particle set over a loop…

The gestureCanvas image is necessary for debugging and visualizing the touch points in real time by assigning the gesture points to a square 5×5 super pixel.

Mission Mode

The Mission Mode is the run time mode that tracks touch gestures in real time and translates it to keyboard and mouse events for the Windows 7 OS.

Reading Surface Calibration Values
During the system boot-up (Frames = 0), after the global run time variables are initialized, the surface calibration values are read off of the DepthAverage.dump file…

gSurface is an image data structure that holds the average depth values of the table surface obtained during calibration. Following this, the calibration reference co-ordinates obtained from the ExtentsCalibration Mode is read back from the file Calibration.dat and stored in the structure Calout. For each calibration co-ordinate obtained from the file, an envelope bounding box (rectangular) is applied and the co-ordinates are redefined. This is done to adjust for any rotational shifts in the Z-axis during calibration (please note: this is not a transform, but a linear extrapolation).

Calculating Touch from Depth

Following the above operations of reading the calibration files and loading it in the run time memory variables, the conversion of depth image to touch is performed by calling:

The output of the Gelib_CalcTouchImage function is an image structure containing binary touch information (a 640×480 array of 0s and 255s). The function takes inputs of the buffer containing depth information passed on to the application from the SDK as a memory pointer, bounding box co-ordinates for the calibration points (this is required to exclude points outside the touch space), the constants DSURFACE_TOL and FINGER passed through the GUI.

The touch image is calculated by first reducing the depth buffer and extracting the masked depth values (by a logical bitwise &0x0ffff).  Again, following Andy Wilson’s work, the classification of depth values as touch is explained in detail here.

The values of Dmax and Dmin are calculated as shown in the code snippet above. Dmax is the maximum value of depth beyond which any value of depth is classified as not belonging to the touch range. Dmin is the minimum depth value below which depth values do not count as meaningful touch values. The small range of values between Dmax and Dmin is classified as a valid touch point. (Note: To avoid spurious calibration parameters to be counted in the process, any value of Depth equaling 0 or greater than 1200 is classified as a non-touch point). Dmax can be assumed to be the actual surface value as calculated during the surface calibration procedure. However, owing to temporal depth noise, a tolerance parameter (DSURFACE_TOL) is included in the calculation of Dmax. The thickness of your finger dictates the Dmin calculation. We include the parameter FINGER to determine the Dmin value.

2D Spatial Depth Filter

As recommended by A. Wilson, a 2D separable mean filter (without weights) is implemented to remove boundary noise during interactions…

The kernel size is fixed at 9×9 (although this is configurable using the GUI).  A binary threshold is applied following the two stages of filtering.  The threshold is adjustable from the MissionMode GUI.  A value of 96 to 128 worked well at the time of testing.

Connected Component Labeling

The process of identifying blobs of connected particles (in this case, separate one finger from another in the touch space) and labeling them progressively for each frame is known as Connected Component Labeling (CCL for short). The algorithm used here is 8 connected (Moore Neighborhood) component labeling. Check out the Wikipedia entry for more info.  Our algorithm (borrowing heavily from the standard) is implemented here:

The CCL algorithm is run in two passes.  The first pass labels each valid pixel incrementally depending on the 8-connectivity rule.  The second pass creates an equivalence table for labels that are associated with the same particle but have a different primary value (a primary value is the dominant value in the particle).  The connected-8 kernel is defined as…

First Pass: For each pixel (represented by the solid black cell), the surrounding connected pixels are compared with the current pixel. The smallest label above 0 in the connected 8 kernel is assigned as the new label for the current pixel. If the smallest label in the kernel cannot be found (probably connected by zeros), then the pixel value is assigned a new label which is defined as a counter.

Second pass: The first pass output is rerun to derive the equivalence table. The equivalence table associates boundary particles that may have a different label than the primary value of the particle. The same procedure of 8-connecting is applied and the table structure is formed based on the variance in pixel values between the current pixel and the surrounding pixels. Once the association is derived, the equivalence table is normalized and the frame is redrawn. The resulting output is a well-connected and a uniquely labeled blob set.

Particle Analysis

Particle analysis deals with performing morphology and intensity based parameter measurements on binary images.  Typical operations involve measuring areas, lengths, coordinates, chords and axes, shape equivalence and shape features.  Before particle analysis, Histogram-based thresholding and equalization is done to remove small particles and noise.  Currently, the threshold is a fixed parameter of 32, which means that any particle with less than 32 pixels area is ignored for gesture calculations.

We limit the particle analysis operations in this application to a “bounding-box and center point” approach.  More accurate measurements are unnecessary considering the errors in touch registration.  For each particle identified as a label, the bounding box and center values are found as the maximum and minimum values in X and Y direction.

The particle analysis structure P returns the measured values for each valid particle back to the application.

Detecting Gestures and Emulating I/O Device Actions

The particle structure is used as the input to the gesture filter switch-case structure.  Based on the number of particles detected (label count from the particle analysis routine), the gesture filter redirects I/O control (typically mouse actions or keyboard events) to the corresponding functions.  These functions process gestures temporally and spatially, hence the accuracy of gesture detection depends on well-trained coefficients (these coefficients are currently hard-wired in the source code.  These can be exposed as controls during the Surface Calibration process or as a completely different calibration/training process.  Coefficients maybe determined as OTP values (hard-wired), dynamic and linear values or intelligence based values such as using neural networks).  At the time of writing, up to 5 particles per frame are handled by the gesture filter (up to five fingers or five cases in the switch-case structure), although there is no limit to the number of particles (NUM_PARTICLES parameter) that the particle analysis method can process and the gesture filter can handle.

For multi-user interactions, it may be necessary to process multiple particles through deeper particle measurements (palm recognition etc.).  This would also require the Windows Multipoint SDK for multi-mouse interactions.  This should get easier in the future.

For the moment, our gesture detection mechanism is designed in a way to facilitate smooth hand off between gestures and gesture fade-in and fade-out.  As an example, there may be cases where a two-finger zoom gesture may transition and fade-out from 2 particles to 1 particle followed by 0.  These transitions have been accounted for in the gesture tracking mechanism.  Here’s a breakdown of how each gesture works…

Idle mode: During the idle mode, the gesture filter and class variables are initialized and the filter waits for a valid gesture particle set.  Any value of the number of particles above 1 is assumed to be a valid gesture. In addition, all emulation events are reset (mouse buttons are released).  In order to avoid spurious gesture resets to case 0 (noisy frames), a time out counter is set.  For any value of the time out beyond 3 (3 frames) the previous gesture is assumed to have completed.  Certain gestures like Select/Move have special time out values that are unique to the behavior of those interactions.  Every gesture case also keeps track of the previous gesture in the pGesture variable (which acts a lot like a state machine).

Select/Move mode: This mode (case 1) handles gestures such as Select/Move components and delete component as well as (left and right) mouse click gestures.

Handling Mouse Right Click:The right click event is handled based on a timed gesture response.  This is required to differentiate between gesture transitions as well as other single particle gestures.  The right click event is determined based on the dtTimeoutcounter value…

Component Select, Move and Delete:  The Gelib_SelectElement function processes all touch gestures that require translation to corresponding events on the Grasshopper canvas.

The select gesture is valid when the spatial difference between the current touch point and the previous touch point (the previous frame) is less than a tolerance value defined by SE_TOLERANCE.  The tolerance value is set to a strict 1 currently.  To distinguish between spurious or stray touch points, we track the gesture for stability.  If the current touch point is stable for 10 frames, the gesture is locked down by setting the mouse pointer position at the translated position and a “mouse left-click and hold” event is sent to the operating system.  For the 11th frame (special case), the mouse pointer is jittered for a pixel in the Y direction to ensure that the component information (usually achieved by mouse hover) is displayed.

The component move gesture is opposite in implementation to the move gesture.  In this case, we check if the difference in coordinates of the current and previous touch point is above a certain threshold (SE_TOLERANCE + 2) and proceed to send the “mouse left-click and hold” event to the OS.  The delete component gesture is also implemented under the same category wherein we check if the difference in current and previous touch coordinates is above the delete threshold of DE_TOLERANCE.  Once the threshold limit is hit, the delete gesture identifier proceed to a binary encoded style state machine that tracks for successive gestures (back and forth movement of the finger).  After 3 successive move gestures (back and forth), the gesture recognition loop proceeds to send the “Delete” key event to the OS.

Zoom: The two finger zoom gesture is similar in implementation to the Select/Move mode…

On function entry, the floating point distance between the two fingers is calculated.  This is required to differentiate between a zoom in and zoom out gesture. For each frame, the current distance between the fingers and the previous frame‟s distance between the fingers are compared.  If the difference is positive, the gesture is a zoom-out.  In this case, the mouse-scroll event is sent to the OS with a positive WHEEL_DELTA.  A zoom counter in each case is incremented to verify switching between one zoom-mode to the other.

Pan: Cases 4 and 5 are used in tandem for the pan mode.  The pan gesture is a 5 finger gesture, however there may be instances where two fingers combine to one particle and yield 4 particle gestures.  In those cases, the gesture filter operates in the same pan canvas mode.  The start of the gesture however requires 5 unique particles…

The actual mouse position is determined as the average X and Y position of each of the (4 or) 5 particles passed as the parameter to the Pan Canvas function.  The pan canvas gesture is easily susceptible to temporal noise owing to the inconsistency in the ability to determine a fixed mouse pointer coordinate.  To nullify this problem, a 2 point frame average of the coordinates is calculated as the final pan point.  Whenever the current touch point changes position, the movement delta values in X and Y directions are calculated.  If the delta values exceed a threshold (PAN_TOLERANCE), the pan gesture is simulated by a mouse right-click and hold event.

Drag-wire and add connection: This is a 3 finger gesture similar to the move gesture with the additional shift key event included with the mouse left click hold event.

It is important to identify the optimum relative positioning of the 3 fingers on the touch space.  For natural interaction the index finger, middle finger and the thumb are used.  The middle finger shall be used as the actual pointer position for the gesture.  To calculate the validity of this condition, a conditional check is performed on the 3 coordinates passed to the function.  Following the classification of the touch points, the final touch point (unfortunately, it’s the middle finger) is extracted and checked for motion across frames using the same method described for the move gesture.  The shift key is also held in addition to the “Left-Click-Hold” event.  The 3 finger hold gesture is implemented as well for the sake of uniformity in behavior.

Related Posts

Grasshopper Canvas with Kinect Interaction: Part 2

Grasshopper Canvas with Kinect Interaction: Part 1

Grasshopper Canvas Real Estate

1.Kinect Sensor | 2.Raw Depth Image | 3.Segmented Depth Image | 4.After Lowpass Filter | 5.Connected Component Labeling | 6.Particle Analysis | 7n.Gesture Filters | 8.Keyboard/Mouse Events

In our previous post, we talked about our efforts to use the Microsoft Kinect to control a large-scale table-top interface running Grasshopper.  In short: we used the Kinect to sense touch and gestures…which we then used to control the canvas via keyboard and mouse events.  We’ve had a lot of fun building Kinect Multitouch Interactions but – being an architecture firm – we can only spend so much time developing the code.  We think we’ve created a solid foundation and would like to share with the broader community to use, modify, and extend.  Obviously, Grasshopper is only one possible application and we’d love to see how others will use this interaction.  So, in the spirit of openness, we’re providing the complete source – as well as Visual Studio solution files – downloadable below.  It’s our hope that those of you reading this will adapt and improve on what we’ve started.

BE WARNED:  Before you jump in and try this at home, we  encourage you to read this post (and the next one) to get a sense of how we went about this.  Rest assured, downloading and compiling our sample code is not a terribly complex endeavor, provided you have some experience with linking and building applications.  You’ll need to have some experience with C/C++ programming…so please read the detailed technical descriptions included in the SDK (and in these posts).

To get started, you’ll need the following…

Prerequisites/Dependencies

1. Microsoft Kinect for XBox with external power source and USB cables.
2. Microsoft Windows 7 or later.
3. Kinect for Windows PC SDK with drivers.  We used the first public beta.
4. Microsoft .NET Framework 4.0.
5. VisualStudio 2010 IDE or similar.  Express editions should work fine.
6. Our source code, which includes the Visual Studio Solution files…scroll down.

Optional Equipment

7. Projector (or similar) display.  Our setup, while not expensive, is a bit exotic.
8. Kinect stabilizing mount (for walls, tables and ceilings).

Kinect Multitouch Interaction Source and Examples

Kinect Multitouch Interaction by LMNts is copyleft and licensed under the GNU General Public License.

Obligatory Disclaimer/Terms and Conditions of Use:  By downloading, you agree to use this software at your own risk.  Under no circumstances shall LMN or LMNts be liable for direct, indirect, special, incidental, or consequential damages resulting from the use, misuse, or inability to use this software.  The software is provided “as is,” whereas we cannot provide a guarantee of support.  LMNts does not guarantee that this software is bug-free or that it will solve all your problems…

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

Ok, let’s get into the fun details of how this works…

Hardware Setup

There are a great number of possible configurations and methods of display, from as simple as tracking touch on a standard LCD monitor to registering touch on the surface of a wall or a chair.  In our example, we used a large ground glass table and a giant mirror to project the interactive workspace and UI from beneath the table.  Our goal was to create a poor man’s Microsoft Surface.  As it’s unlikely that you will have a similar setup, let’s assume you want to project onto a standard table from above.  The table top should typically be capable of allowing light from the projector to be dispersed on the surface to facilitate viewing but not exhibit excessive specular behavior.  That’s a fancy way of saying: you will need a table with a dull surface finish and position the projector at an angle (adjusting the image keystone) so that it doesn’t bounce light directly back into the depth sensor.  This is important since the Kinect sensor operates by projecting infra-red light through the field of view; light bouncing off any reflective surface would be reproduced as noise or invalid depth data.  You don’t want that.

If you set this up on a table, the RGBD sensor (ie: Kinect) should be installed overhead with a field of view covering the table top.  While the minimum distance of the sensor (distance to table) is restricted by the Kinect’s capabilities (typically around 0.75m), the maximum distance to table determines the accuracy of touch calculations.  A typical depth value (and recommended) is 1.00 m for the specific table dimensions.  You will want to mount the Kinect in such a way as it cannot be moved or disturbed…even small vibrations cause noise in the depth data and could throw off the interaction.

If you set this up on a wall (or similar surface), the above rules still apply, but you will want to be careful to position the Kinect where its view of the surface won’t be occluded by users standing directly in the way.  Our touch implementation is based on line-of-sight…overhead mounting works best for wall interaction as well.

Software Setup

A complete walkthrough of the Microsoft Kinect PC SDK is well beyond the scope of this post.  Microsoft has some great documentation and support for getting up and running with the SDK.  For clarity, we’ve built our the demo interaction around Microsoft’s SkeletalViewer sample code (a popular place to start for good reason).  We have NOT included Microsoft’s source, so it will be necessary to link and recompile with the actual Microsoft SDK if you want to make major changes.  The VisualStudio Solution file is setup to link to the necessary dependencies, but you will have to copy the relevant sources from the SDK into the project folder of your choosing.

The source code is written in C++ and targeting the .NET framework 4.0.  We have included the three executable builds:

SurfaceCalibration.exe
ExtentsCalibration.exe
MissionMode.exe

Each of these builds is a modified version of the main TouchGestures sample code.  The sample included in our download builds the MissionMode executable by default.  (Should you like to rebuild SurfaceCalibration.exe or ExtentsCalibration.exe, the TouchGestures.h contains a globally defined switch that will trigger #ifndef code in the implementation…after building, simply rename the execs).

The executables should be run in the following order (one at a time):

1.  SurfaceCalibration.exe – necessary for depth estimation across the interaction surface.  This utility calibrates the Kinect relative to a surface in its field of view.   Once the executable completes, a dump file called DepthAverage.dump is created in the Release folder containing the depth information in millimeters for each of the 640×480 pixels.

2.  ExtentsCalibration.exe – is required to map the 2D table top space to the projected interaction space (in our case, the Grasshopper canvas).   Build and run this executable.  If you are using a projector, move the window directly over the area you wish to track.   This executable displays a square pixelated area on the display which acts as the target point to be mapped onto the 2D depth space (which is normally restricted to a subset of the Kinect’s field-of-view).  Once the square is displayed (starting bottom-right and moving clockwise for each corner, ending in the center),  you are expected to touch and hold the square for a few seconds until it moves to the next calibration location.  Watch carefully: the better the tracking, the quicker it moves.  This continues until all the 5 points are displayed and logged by the calibration utility.  At the end of the process, a data file called Calibration.dat is created in the Release directory containing the co-ordinates logged during the extents calibration process. 

The controls in the UI are required to adjust the touch parameters and thresholds during image processing: this section displays the frame rate, provides control sliders to modify the DSurface, Finger and Mean filter threshold value.  The filter kernel size can also be modified using the text control box (we’ll explain these terms in the next post).  The default values in the UI are usually sufficient for a typical use case but, depending on where you position the Kinect relative to the tracked area, you may need to adjust the parameters to remove some of the noise.

3.  MissionMode.exe – this is the sample touch and image processing executable that processes touch inputs, identifies and translates the touch inputs as gestures and mouse events on the Grasshopper canvas.  This executable should always be open and running to enable touch gestures tracking, but it does not need to be coincident with the tracked area.  The window displays the touch points at ~30 frames per second.   This exe should only be run once the calibration procedure is complete.   In our example, the Grasshopper canvas can be pushed to the display space (in our case: the table) and the fun begins.

So that’s the overview.  If you are testing this out with Grasshopper (which we recommend), the above gestures are implemented and should work fine without any changes to the code.  In our next post, we’ll unpack and walk through each of the code functions in greater detail…

Related Posts

Grasshopper Canvas with Kinect Interaction: Part 3

Grasshopper Canvas with Kinect Interaction: Part 1

Grasshopper Canvas Real Estate

This Holiday Season, LMNers lit candles…LOTS of candles.  The video was directed, shot, edited and produced by: cbaker, jbeaver, scrawford, cgrammens, kgregga, thenderson, ahunter, kkimura, amacdonald, and gshaw.