LMNts’ third blog post and image are up on Metropolis Magazine’s POV blog where Scott Crawford shares his thoughts on a design technology Cabinet of Curiosities.  Be sure to also checkout Confessions of a Design Technology Evangelical and Re-Upping on Design Technology if you haven’t already.

The basic steps in our touch-as-depth software can be summarized by the flowchart at left.  For each mode of operation (SurfaceCalibration, ExtentsCalibration, and MissionMode), the data flow is slightly different.  The flowchart displayed is that of the typical MissionMode flow loop.   Let’s go over these one by one…

Depth information is gathered frame-by-frame from the Kinect.  These depth points are stored in the depth buffer.  Next, the base interaction surface is found in relation to the location of the Kinect (as determined by the SurfaceCalibration routine).  During MissionMode, if it’s the first pass through, all the depth data is adjusted relative to the calibration matrix.  In all subsequent frames, this depth data is then compared relative to the surface to determine whether or not there is a “touch” event.  The raw touch-from-depth data is then filtered using a custom mean filter that weeds out some of the extraneous noise…some, but not all.  Connected 8 component labeling is used to disambiguate one finger from another.  Particle analysis is then performed to figure out exactly which points are “touch” points and which are noise…at this point, the data should not be noisy.  Now that we have the touch points, we need to make sure they line up with the spots in the real world.  The raw points as seen by the Kinect are then transformed relative to the tracked space (as determined by the ExtentsCalibration routine).  Finally, a filter is applied to the frame-to-frame behavior of the points to determined their mapped keyboard and mouse event.  Rinse and repeat…as fast as possible.  Now let’s go through each of the steps in more detail…

SurfaceCalibration Mode

Once you have placed your Kinect relative to the surface (a table, a wall, a chair, etc.) you want to track touch on, you need to get some very accurate readings about depth.  This is a one-time calibration…unless you move your Kinect or change the surface, in which case you will need to re-calibrate.  The SurfaceCalibration mode is responsible for modeling the interaction surface over multiple frames and a fixed spatial resolution of 640×480 depth points.  The methodology of modeling the surface and extracting touch information that we use is explained very clearly by Andy Wilson of Microsoft Research.  Unfortunately, the process does not calculate depth information adaptively.  Hence, any objects placed on the interaction surface during calibration will be modeled as being part of the touch space.  It is important to keep the field-of-view devoid of interference from direct light sources and objects (even a few seconds of interference would yield result in errors of ~10% in calibration).

SurfaceCalibration mode involves a single step as soon as the depth buffer is ready and locked.  This step contains the routine (Gelib_DoSurfaceCalibration) that translates the depth values to millimeters and stores the cumulative sum over a period.  The number of frames to track the depth values (defined by DUMPFOR parameter) and the frame interval (defined by DUMPEVERY parameter) are specified in the header file TouchGestures.h.  The calibration routine shows a progress bar.  Once the depth values are gathered, the final depth data for each pixel (one of 640×480) are calculated by finding the average values (dividing each pixel accumulated value by DUMPFOR).  The average depth values are stored in the file DepthAverage.dump (our file format uses a semicolon-space-newline delimitation).  The SurfaceCalibration.exe application may be terminated as soon as this file is generated (or as soon as the progress bar indicates completion).  At this point, the Kinect should be calibrated relative to your surface and you are ready to tell the Kinect what the boundaries of your tracked space are…for that you switch to:

ExtentsCalibration Mode

During the ExtentsCalibration mode, the system works in partial Mission mode.  Touch gestures are processed but not translated and emulated as mouse events.  Instead, touch points are recorded for a certain time period and the corresponding co-ordinates are logged in a file.

Upon initialization and depth buffer filled event, the first step is to read the surface calibration depth values as an IMAGE structure from the stored file DepthAverage.dump. The IMAGE data structure is defined as a 2D unsigned short integer array with sizes specified as DWIDTH and DHEIGHT (640, 480).

The image analysis and processing routines are similar to that of the MissionMode and will be explained in that section. The additional function (CALIBRATION Gelib_CalibrateExtents( CALIBRATION C ) ) displays and logs the touch points in the sequence. The five points are iterated through a counter (cIteration) and the calibration mode swaps to regular mode (qswap = 0) once the calibration is completed…

The Calout data structure contains the calibration reference points.  These points are stored in the Calibration.dat file.  The data flow switches to touch display mode hence forth (displays transformed touch points and does not translate and emulate mouse events).

Transforming touch points: An affine transform of the touch points is performed to translate the touch points from the depth space to the table space…

The depth space is defined as the 2D region as seen by the Kinect depth camera (this is different from the 2D video space seen by the Kinect). This is expected to be a subset of the table space which is the physical 2D region of the glass top over the table. The transforms applied are:

1. Translation in X,Y direction:  T[C0(x,y)] towards & relative to the table space origin of C0.
2. Scale in X,Y direction: S[M(x,y))] towards the upper boundaries of the table space.

The above process determines factors to calculate the transformation offset and scale factors. The translation offset and scale factors are then applied to any incoming touch particle set over a loop…

The gestureCanvas image is necessary for debugging and visualizing the touch points in real time by assigning the gesture points to a square 5×5 super pixel.

Mission Mode

The Mission Mode is the run time mode that tracks touch gestures in real time and translates it to keyboard and mouse events for the Windows 7 OS.

Reading Surface Calibration Values
During the system boot-up (Frames = 0), after the global run time variables are initialized, the surface calibration values are read off of the DepthAverage.dump file…

gSurface is an image data structure that holds the average depth values of the table surface obtained during calibration. Following this, the calibration reference co-ordinates obtained from the ExtentsCalibration Mode is read back from the file Calibration.dat and stored in the structure Calout. For each calibration co-ordinate obtained from the file, an envelope bounding box (rectangular) is applied and the co-ordinates are redefined. This is done to adjust for any rotational shifts in the Z-axis during calibration (please note: this is not a transform, but a linear extrapolation).

Calculating Touch from Depth

Following the above operations of reading the calibration files and loading it in the run time memory variables, the conversion of depth image to touch is performed by calling:

The output of the Gelib_CalcTouchImage function is an image structure containing binary touch information (a 640×480 array of 0s and 255s). The function takes inputs of the buffer containing depth information passed on to the application from the SDK as a memory pointer, bounding box co-ordinates for the calibration points (this is required to exclude points outside the touch space), the constants DSURFACE_TOL and FINGER passed through the GUI.

The touch image is calculated by first reducing the depth buffer and extracting the masked depth values (by a logical bitwise &0x0ffff).  Again, following Andy Wilson’s work, the classification of depth values as touch is explained in detail here.

The values of Dmax and Dmin are calculated as shown in the code snippet above. Dmax is the maximum value of depth beyond which any value of depth is classified as not belonging to the touch range. Dmin is the minimum depth value below which depth values do not count as meaningful touch values. The small range of values between Dmax and Dmin is classified as a valid touch point. (Note: To avoid spurious calibration parameters to be counted in the process, any value of Depth equaling 0 or greater than 1200 is classified as a non-touch point). Dmax can be assumed to be the actual surface value as calculated during the surface calibration procedure. However, owing to temporal depth noise, a tolerance parameter (DSURFACE_TOL) is included in the calculation of Dmax. The thickness of your finger dictates the Dmin calculation. We include the parameter FINGER to determine the Dmin value.

2D Spatial Depth Filter

As recommended by A. Wilson, a 2D separable mean filter (without weights) is implemented to remove boundary noise during interactions…

The kernel size is fixed at 9×9 (although this is configurable using the GUI).  A binary threshold is applied following the two stages of filtering.  The threshold is adjustable from the MissionMode GUI.  A value of 96 to 128 worked well at the time of testing.

Connected Component Labeling

The process of identifying blobs of connected particles (in this case, separate one finger from another in the touch space) and labeling them progressively for each frame is known as Connected Component Labeling (CCL for short). The algorithm used here is 8 connected (Moore Neighborhood) component labeling. Check out the Wikipedia entry for more info.  Our algorithm (borrowing heavily from the standard) is implemented here:

The CCL algorithm is run in two passes.  The first pass labels each valid pixel incrementally depending on the 8-connectivity rule.  The second pass creates an equivalence table for labels that are associated with the same particle but have a different primary value (a primary value is the dominant value in the particle).  The connected-8 kernel is defined as…

First Pass: For each pixel (represented by the solid black cell), the surrounding connected pixels are compared with the current pixel. The smallest label above 0 in the connected 8 kernel is assigned as the new label for the current pixel. If the smallest label in the kernel cannot be found (probably connected by zeros), then the pixel value is assigned a new label which is defined as a counter.

Second pass: The first pass output is rerun to derive the equivalence table. The equivalence table associates boundary particles that may have a different label than the primary value of the particle. The same procedure of 8-connecting is applied and the table structure is formed based on the variance in pixel values between the current pixel and the surrounding pixels. Once the association is derived, the equivalence table is normalized and the frame is redrawn. The resulting output is a well-connected and a uniquely labeled blob set.

Particle Analysis

Particle analysis deals with performing morphology and intensity based parameter measurements on binary images.  Typical operations involve measuring areas, lengths, coordinates, chords and axes, shape equivalence and shape features.  Before particle analysis, Histogram-based thresholding and equalization is done to remove small particles and noise.  Currently, the threshold is a fixed parameter of 32, which means that any particle with less than 32 pixels area is ignored for gesture calculations.

We limit the particle analysis operations in this application to a “bounding-box and center point” approach.  More accurate measurements are unnecessary considering the errors in touch registration.  For each particle identified as a label, the bounding box and center values are found as the maximum and minimum values in X and Y direction.

The particle analysis structure P returns the measured values for each valid particle back to the application.

Detecting Gestures and Emulating I/O Device Actions

The particle structure is used as the input to the gesture filter switch-case structure.  Based on the number of particles detected (label count from the particle analysis routine), the gesture filter redirects I/O control (typically mouse actions or keyboard events) to the corresponding functions.  These functions process gestures temporally and spatially, hence the accuracy of gesture detection depends on well-trained coefficients (these coefficients are currently hard-wired in the source code.  These can be exposed as controls during the Surface Calibration process or as a completely different calibration/training process.  Coefficients maybe determined as OTP values (hard-wired), dynamic and linear values or intelligence based values such as using neural networks).  At the time of writing, up to 5 particles per frame are handled by the gesture filter (up to five fingers or five cases in the switch-case structure), although there is no limit to the number of particles (NUM_PARTICLES parameter) that the particle analysis method can process and the gesture filter can handle.

For multi-user interactions, it may be necessary to process multiple particles through deeper particle measurements (palm recognition etc.).  This would also require the Windows Multipoint SDK for multi-mouse interactions.  This should get easier in the future.

For the moment, our gesture detection mechanism is designed in a way to facilitate smooth hand off between gestures and gesture fade-in and fade-out.  As an example, there may be cases where a two-finger zoom gesture may transition and fade-out from 2 particles to 1 particle followed by 0.  These transitions have been accounted for in the gesture tracking mechanism.  Here’s a breakdown of how each gesture works…

Idle mode: During the idle mode, the gesture filter and class variables are initialized and the filter waits for a valid gesture particle set.  Any value of the number of particles above 1 is assumed to be a valid gesture. In addition, all emulation events are reset (mouse buttons are released).  In order to avoid spurious gesture resets to case 0 (noisy frames), a time out counter is set.  For any value of the time out beyond 3 (3 frames) the previous gesture is assumed to have completed.  Certain gestures like Select/Move have special time out values that are unique to the behavior of those interactions.  Every gesture case also keeps track of the previous gesture in the pGesture variable (which acts a lot like a state machine).

Select/Move mode: This mode (case 1) handles gestures such as Select/Move components and delete component as well as (left and right) mouse click gestures.

Handling Mouse Right Click:The right click event is handled based on a timed gesture response.  This is required to differentiate between gesture transitions as well as other single particle gestures.  The right click event is determined based on the dtTimeoutcounter value…

Component Select, Move and Delete:  The Gelib_SelectElement function processes all touch gestures that require translation to corresponding events on the Grasshopper canvas.

The select gesture is valid when the spatial difference between the current touch point and the previous touch point (the previous frame) is less than a tolerance value defined by SE_TOLERANCE.  The tolerance value is set to a strict 1 currently.  To distinguish between spurious or stray touch points, we track the gesture for stability.  If the current touch point is stable for 10 frames, the gesture is locked down by setting the mouse pointer position at the translated position and a “mouse left-click and hold” event is sent to the operating system.  For the 11th frame (special case), the mouse pointer is jittered for a pixel in the Y direction to ensure that the component information (usually achieved by mouse hover) is displayed.

The component move gesture is opposite in implementation to the move gesture.  In this case, we check if the difference in coordinates of the current and previous touch point is above a certain threshold (SE_TOLERANCE + 2) and proceed to send the “mouse left-click and hold” event to the OS.  The delete component gesture is also implemented under the same category wherein we check if the difference in current and previous touch coordinates is above the delete threshold of DE_TOLERANCE.  Once the threshold limit is hit, the delete gesture identifier proceed to a binary encoded style state machine that tracks for successive gestures (back and forth movement of the finger).  After 3 successive move gestures (back and forth), the gesture recognition loop proceeds to send the “Delete” key event to the OS.

Zoom: The two finger zoom gesture is similar in implementation to the Select/Move mode…

On function entry, the floating point distance between the two fingers is calculated.  This is required to differentiate between a zoom in and zoom out gesture. For each frame, the current distance between the fingers and the previous frame‟s distance between the fingers are compared.  If the difference is positive, the gesture is a zoom-out.  In this case, the mouse-scroll event is sent to the OS with a positive WHEEL_DELTA.  A zoom counter in each case is incremented to verify switching between one zoom-mode to the other.

Pan: Cases 4 and 5 are used in tandem for the pan mode.  The pan gesture is a 5 finger gesture, however there may be instances where two fingers combine to one particle and yield 4 particle gestures.  In those cases, the gesture filter operates in the same pan canvas mode.  The start of the gesture however requires 5 unique particles…

The actual mouse position is determined as the average X and Y position of each of the (4 or) 5 particles passed as the parameter to the Pan Canvas function.  The pan canvas gesture is easily susceptible to temporal noise owing to the inconsistency in the ability to determine a fixed mouse pointer coordinate.  To nullify this problem, a 2 point frame average of the coordinates is calculated as the final pan point.  Whenever the current touch point changes position, the movement delta values in X and Y directions are calculated.  If the delta values exceed a threshold (PAN_TOLERANCE), the pan gesture is simulated by a mouse right-click and hold event.

Drag-wire and add connection: This is a 3 finger gesture similar to the move gesture with the additional shift key event included with the mouse left click hold event.

It is important to identify the optimum relative positioning of the 3 fingers on the touch space.  For natural interaction the index finger, middle finger and the thumb are used.  The middle finger shall be used as the actual pointer position for the gesture.  To calculate the validity of this condition, a conditional check is performed on the 3 coordinates passed to the function.  Following the classification of the touch points, the final touch point (unfortunately, it’s the middle finger) is extracted and checked for motion across frames using the same method described for the move gesture.  The shift key is also held in addition to the “Left-Click-Hold” event.  The 3 finger hold gesture is implemented as well for the sake of uniformity in behavior.

Related Posts

Grasshopper Canvas with Kinect Interaction: Part 2

Grasshopper Canvas with Kinect Interaction: Part 1

Grasshopper Canvas Real Estate

1.Kinect Sensor | 2.Raw Depth Image | 3.Segmented Depth Image | 4.After Lowpass Filter | 5.Connected Component Labeling | 6.Particle Analysis | 7n.Gesture Filters | 8.Keyboard/Mouse Events

In our previous post, we talked about our efforts to use the Microsoft Kinect to control a large-scale table-top interface running Grasshopper.  In short: we used the Kinect to sense touch and gestures…which we then used to control the canvas via keyboard and mouse events.  We’ve had a lot of fun building Kinect Multitouch Interactions but – being an architecture firm – we can only spend so much time developing the code.  We think we’ve created a solid foundation and would like to share with the broader community to use, modify, and extend.  Obviously, Grasshopper is only one possible application and we’d love to see how others will use this interaction.  So, in the spirit of openness, we’re providing the complete source – as well as Visual Studio solution files – downloadable below.  It’s our hope that those of you reading this will adapt and improve on what we’ve started.

BE WARNED:  Before you jump in and try this at home, we  encourage you to read this post (and the next one) to get a sense of how we went about this.  Rest assured, downloading and compiling our sample code is not a terribly complex endeavor, provided you have some experience with linking and building applications.  You’ll need to have some experience with C/C++ programming…so please read the detailed technical descriptions included in the SDK (and in these posts).

To get started, you’ll need the following…

Prerequisites/Dependencies

1. Microsoft Kinect for XBox with external power source and USB cables.
2. Microsoft Windows 7 or later.
3. Kinect for Windows PC SDK with drivers.  We used the first public beta.
4. Microsoft .NET Framework 4.0.
5. VisualStudio 2010 IDE or similar.  Express editions should work fine.
6. Our source code, which includes the Visual Studio Solution files…scroll down.

Optional Equipment

7. Projector (or similar) display.  Our setup, while not expensive, is a bit exotic.
8. Kinect stabilizing mount (for walls, tables and ceilings).

Kinect Multitouch Interaction Source and Examples

Kinect Multitouch Interaction by LMNts is copyleft and licensed under the GNU General Public License.

Obligatory Disclaimer/Terms and Conditions of Use:  By downloading, you agree to use this software at your own risk.  Under no circumstances shall LMN or LMNts be liable for direct, indirect, special, incidental, or consequential damages resulting from the use, misuse, or inability to use this software.  The software is provided “as is,” whereas we cannot provide a guarantee of support.  LMNts does not guarantee that this software is bug-free or that it will solve all your problems…

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

Ok, let’s get into the fun details of how this works…

Hardware Setup

There are a great number of possible configurations and methods of display, from as simple as tracking touch on a standard LCD monitor to registering touch on the surface of a wall or a chair.  In our example, we used a large ground glass table and a giant mirror to project the interactive workspace and UI from beneath the table.  Our goal was to create a poor man’s Microsoft Surface.  As it’s unlikely that you will have a similar setup, let’s assume you want to project onto a standard table from above.  The table top should typically be capable of allowing light from the projector to be dispersed on the surface to facilitate viewing but not exhibit excessive specular behavior.  That’s a fancy way of saying: you will need a table with a dull surface finish and position the projector at an angle (adjusting the image keystone) so that it doesn’t bounce light directly back into the depth sensor.  This is important since the Kinect sensor operates by projecting infra-red light through the field of view; light bouncing off any reflective surface would be reproduced as noise or invalid depth data.  You don’t want that.

If you set this up on a table, the RGBD sensor (ie: Kinect) should be installed overhead with a field of view covering the table top.  While the minimum distance of the sensor (distance to table) is restricted by the Kinect’s capabilities (typically around 0.75m), the maximum distance to table determines the accuracy of touch calculations.  A typical depth value (and recommended) is 1.00 m for the specific table dimensions.  You will want to mount the Kinect in such a way as it cannot be moved or disturbed…even small vibrations cause noise in the depth data and could throw off the interaction.

If you set this up on a wall (or similar surface), the above rules still apply, but you will want to be careful to position the Kinect where its view of the surface won’t be occluded by users standing directly in the way.  Our touch implementation is based on line-of-sight…overhead mounting works best for wall interaction as well.

Software Setup

A complete walkthrough of the Microsoft Kinect PC SDK is well beyond the scope of this post.  Microsoft has some great documentation and support for getting up and running with the SDK.  For clarity, we’ve built our the demo interaction around Microsoft’s SkeletalViewer sample code (a popular place to start for good reason).  We have NOT included Microsoft’s source, so it will be necessary to link and recompile with the actual Microsoft SDK if you want to make major changes.  The VisualStudio Solution file is setup to link to the necessary dependencies, but you will have to copy the relevant sources from the SDK into the project folder of your choosing.

The source code is written in C++ and targeting the .NET framework 4.0.  We have included the three executable builds:

SurfaceCalibration.exe
ExtentsCalibration.exe
MissionMode.exe

Each of these builds is a modified version of the main TouchGestures sample code.  The sample included in our download builds the MissionMode executable by default.  (Should you like to rebuild SurfaceCalibration.exe or ExtentsCalibration.exe, the TouchGestures.h contains a globally defined switch that will trigger #ifndef code in the implementation…after building, simply rename the execs).

The executables should be run in the following order (one at a time):

1.  SurfaceCalibration.exe – necessary for depth estimation across the interaction surface.  This utility calibrates the Kinect relative to a surface in its field of view.   Once the executable completes, a dump file called DepthAverage.dump is created in the Release folder containing the depth information in millimeters for each of the 640×480 pixels.

2.  ExtentsCalibration.exe – is required to map the 2D table top space to the projected interaction space (in our case, the Grasshopper canvas).   Build and run this executable.  If you are using a projector, move the window directly over the area you wish to track.   This executable displays a square pixelated area on the display which acts as the target point to be mapped onto the 2D depth space (which is normally restricted to a subset of the Kinect’s field-of-view).  Once the square is displayed (starting bottom-right and moving clockwise for each corner, ending in the center),  you are expected to touch and hold the square for a few seconds until it moves to the next calibration location.  Watch carefully: the better the tracking, the quicker it moves.  This continues until all the 5 points are displayed and logged by the calibration utility.  At the end of the process, a data file called Calibration.dat is created in the Release directory containing the co-ordinates logged during the extents calibration process. 

The controls in the UI are required to adjust the touch parameters and thresholds during image processing: this section displays the frame rate, provides control sliders to modify the DSurface, Finger and Mean filter threshold value.  The filter kernel size can also be modified using the text control box (we’ll explain these terms in the next post).  The default values in the UI are usually sufficient for a typical use case but, depending on where you position the Kinect relative to the tracked area, you may need to adjust the parameters to remove some of the noise.

3.  MissionMode.exe – this is the sample touch and image processing executable that processes touch inputs, identifies and translates the touch inputs as gestures and mouse events on the Grasshopper canvas.  This executable should always be open and running to enable touch gestures tracking, but it does not need to be coincident with the tracked area.  The window displays the touch points at ~30 frames per second.   This exe should only be run once the calibration procedure is complete.   In our example, the Grasshopper canvas can be pushed to the display space (in our case: the table) and the fun begins.

So that’s the overview.  If you are testing this out with Grasshopper (which we recommend), the above gestures are implemented and should work fine without any changes to the code.  In our next post, we’ll unpack and walk through each of the code functions in greater detail…

Related Posts

Grasshopper Canvas with Kinect Interaction: Part 3

Grasshopper Canvas with Kinect Interaction: Part 1

Grasshopper Canvas Real Estate

This Holiday Season, LMNers lit candles…LOTS of candles.  The video was directed, shot, edited and produced by: cbaker, jbeaver, scrawford, cgrammens, kgregga, thenderson, ahunter, kkimura, amacdonald, and gshaw.

Guest blogging at Metropolis Magazine, dbelcher organizes his Field Notes.  (Meanwhile, the Desk Liberation Front launches another photographic assault).  For an pseudo-iconoclastic-interlude to all this computational fetishism, go read the Confessions of a Design Technology Evangelical.

We have three major goals with this project: First – and key to why we went about this the way we did – we wanted to understand the underlying costs, infrastructure changes, and digital tools necessary for proper energy monitoring.  The first goal will also have the added benefit of allowing us to tinker with our setup as much as we like…something you may have noticed (if you read this blog regularly) that we enjoy doing. Second, we want to understand our current usage and gather a baseline before we make any changes to our behavior.  Finally, we want to quantify our progress as we begin to take Energy Conservation Measures (ECMs) – a fancy way of saying: “forming better habits”.   In this post, we’ll address the first two goals directly and come back to the third goal (ECMs) in a future post…

First, the hardware.  We shopped around a bit for electrical Energy Monitoring Units.   From the start, we were looking for an EMU that would allow us to learn as much about the process of monitoring as we could.   There are many commercial-grade units available on the market, but most are very expensive and worse: some require purchasing additional software (sometimes through subscription) just to view the data the EMUs record.  With these systems, there’s the considerable upfront cost for the equipment and installation, plus – to add insult to injury – the cost of the software to boot.  What we wanted was something purpose built, interoperable, and open.  After some shopping, we found the OptoEMU.  Opto22‘s open standards philosophy appealed to us: all their systems run on non-proprietary communication standards like Ethernet and IP.  This means that one could integrate their EMUs into an existing monitoring system, purchase a subscription to a data-logging service (such as Pulse Energy), or roll your own system at any scale you want.  We chose the latter, without precluding the option of scaling it up.  (Baby steps.  Baby steps).  Our office occupies two floors and has four quadrants fed by four electrical panels that roughly correspond to the quadrants (over the course of 30+ years in the building, there might be some cross wiring…but we’ll find out soon enough).  We decided to test out our proposed system on one quadrant of the office to make sure that we had all the software and hardware setup before we scaled the system up to the rest of the office.  So we bought one of the Opto22 EMUs and had a professional electrician install it in our electrical room.

We called up Opto22 and explained our electrical setup and they recommended starting with a single EMU (shown above left) and three current transformers (CTs).  Out of the box, the OptoEMU is straightforward.  The installation instructions were easy to follow so we went ahead and had our electrician install one EMU in our quadrant (thanks to Tim and Josh of North Star Electric for doing a quick and professional job of installing the EMUs and CTs.)  Once installed, we plugged the EMU into our network backbone and assigned it a static IP-address.  At that point, the EMU was acting a lot like your typical wireless router: you can use any web-browser to access the firmware on the device to configure its settings.  All we needed were the correct current ratios from the electrician.  Just a couple of fields and you’re up and running.  (The browser configuration tool is also where you can setup your data-logging service if you choose to use a third-party such as Pulse).  Rather than use a third-party logging service, we decided to do this ourselves.  This will take a bit of explaining…

Second, the software.  We downloaded the OptoEMU Sensor .NET Toolkit from Opto22′s website.  We used the toolkit to write our own custom polling and logging software that runs on a dedicated machine on our network.  The application polls each of the EMUs once per minute (even though the OptoEMUs can sample at a much faster rate) and records the current kW value and a time-stamp to a SQL database running on the same computer.  This monitoring software also handles the e-mail digest system (see below).  Once we had tested this setup with one EMU, scaling it to four EMUs was very easy.  The downside of this approach is that it requires a dedicated computer on the network to handle the logging.  However, this computer serves double-duty as the Database and the Webserver.  Once all the software was up and running, we were confident enough to purchase the remaining three OptoEMUs and monitor the rest of the office.

The data logging continues, ad infinitum, but each morning – at 10:10 am – the software sends an e-mail to LMN employees with the averages from the previous day.  The e-mail separates out averages for the day and for the night.  As we start to better understand our consumption, we plan to set up alerts for unusual demands or steep trends.  The e-mail is useful, but nowhere near as useful as the interactive chart we have setup to view the data…

The interactive chart allows any user to track variable time-scales (from one minute to one year) broken out by quadrant.  The user can highlight any point in the chart and get the kW reading at that minute.  The embeddable chart is a mash-up of html, pHp, jQuery and the Google Chart Tools API.  Before assembling this mix, we looked at Pachube, Pulse Energy, and Wattvision.  All three had their advantages and disadvantages, but none was exactly what we wanted.  In short: we just wanted to do this ourselves and we didn’t think it would be that difficult.  Even though Google had “retired” PowerMeter, they had left intact the graphing API that they used to construct all their various data visualization systems…from finance to web analytics.  The only piece that was a bit of a challenge was the Annotation feature…

Users can annotate the chart much like a stock-ticker records historical events relevant to spikes or troughs in the data.  A button at the bottom of the graph brings up a calendar (we would have liked this to be a mouse-click on the graph itself, but this proved difficult to do with the ChartTools) and a text-field.  The viewer can then enter in some information (a guess of end-use, a significant event, etc.) that is viewable to anyone who checks the graph.  We are using this feature to figure out which plug-loads are responsible for specific spikes…we also think it will be a great feature for recording intentional ECMs that we will implement in the future.

All four EMUs have been up and running for about a month now.  What are we learning?  As you can see from the chart above, our 5 West quadrant consumes considerably more energy compared to the other three quadrants.  This is because 5W is the home of our servers.  We know that the plotters and printers could probably go into sleep mode earlier in the evening than they do now.  We know that lights are responsible for a large chunk of our consumption.  However, all of this is low-hanging fruit.  What we are focused on now is collecting baseline data of our typical behavior so that we can better assess the effectiveness of future conservation measures.  We’ve learned that energy monitoring doesn’t have to be overly expensive or complex – that you can right-size a solution to an organization of any scale.  We are also learning that it’s difficult to resist the urge to make changes right away, especially when we have such high-resolution data at our fingertips.

What next?  We plan to continue collecting baseline data as long as we can (how long is up for debate within the office).  We plan to embed the interactive graph into our intranet.  We also plan to use our experience to get more monitoring and sub-metering into our built projects.  We plan to start playing around with more interesting ways to interact and consume near real-time data – perhaps hooking up the feed to Arduino to change the color of ambient lighting relative to consumption?  Maybe triggering an air-horn hidden near our SysAdmin’s desk if the servers are out-of-control energy hogs?  (Just kidding Chad).  Whatever we end up doing, we plan to use this data to change our habits and consume energy in a smarter way.

We’ll return to the EMUs in a future post and let you know how things are going…

DIVA stands for Design Iterate Validate Adapt which is similar to the approach that we take with parametric modeling.  We’ve found that we tend to learn more about a design problem when we look at a large number of iterations as opposed to just choosing the best of three.  With this approach it often is possible to tease out patterns of performance which can help in developing a better intuition about how to find an optimized solution to a problem.

One of DIVA’s capabilities is to run DAYSIM climate-based metrics, such as Useful Daylight Index(UDI) and Daylight Autonomy(DA), which use weather data to calculate lighting performance over the course of the entire year.  Looking at a design’s daylighting performance in this way can help to identify the overall effectiveness of a daylighting solution and give an idea of potential energy savings.  An example of this is a study we ran with DIVA that looked at the relationship between ceiling height, window size, percent glazing and a floorplate’s Useful Daylight Index(UDI).

The first thing you’ll notice from the above animation is that false color graphics are awesome.  Well, at least they are to us.  They never fail to get people excited even if the viewer has no idea what they’re depicting.  This is a problem.  Yes, we want people to look at what we show them, but it’s also important that they understand how to read the graphics.  The general rule is that bright colors correspond to higher illuminance levels or better daylight distribution, but we’ve also found that it can be useful to use different color gradients for different metrics so that it is obvious to the viewer that we’re showing something different.  It is important to get both clients and designers to look at these graphics as a tool for improving the design rather than just another form of documentation.*

The advantage of using a tool like Grasshopper is that we can automate most if not all of the simulation process.  Using techniques similar to those we used for exploring the panel geometry of the Cleveland Medical Mart, we were able to get the definition to iterate through each analysis on its own so that we could set it to run overnight.

A challenge that came out of this process was creating all of the annotations for each iteration.  Normally, we would do this manually in something like InDesign if we only had a handful of simulations to run.  In the above animation there were over 40 simulations which would’ve meant a lot of bookkeeping and data entry.  To get around this we collected text from various parts of the Grasshopper definition and then displayed that text in the viewport using our analysis key definition.  Initially, we were baking all of the geometry and text into Rhino and then rendering it, but later realized it was much faster to use the ViewCaptureToFile command.

The filename for each image became a record of the parameter settings for that iteration.  Each parameter is separated by a character, in this case “^”, and then there is a panel component in Grasshopper that records what each value relates to.  It’s definitely not the easiest thing to read, but it has proven helpful many times.  The results file that DIVA generates is also named using this same system.

The results of a simulation are returned as an output of the DIVA Analysis component, but it is also possible to use the Grasshopper Read File component to load the results into the definition.  This approach requires a few additional components to specify the file path and remove the unneeded data (Download importDIVAResults.ghx, 20.93 kB ).  Using this method makes it possible to quickly switch between the different results files.  A problem we’ve had with the UDI 100-2000lux results is that 100 lux is lower than we want our minimum to be while the 2000 lux is an acceptable maximum.  Recently, we created a custom range by subtracting the UDI>2000 results from the Daylight Autonomy results which allows us to specify the minimum through the Daylight Autonomy settings and then use the UDI>2000 results as our maximum.

Simulation analysis has the potential to become more prevalent in the design process as it is further integrated with the 3D modeling work flow especially in the case of parametric modeling.  It’s becoming more apparent how effective parametric modeling is for exploring a multitude of design iterations and with the integration of simulation analysis those iterations can be judged against yet another metric.  We’re looking forward to the release of DIVA 2.0 which will include thermal analysis capabilities through the use of EnergyPlus and seeing how that additional metric influences something like the relationships we explored above.

*CAUTION:  False color graphics are sometimes considered “eye candy” presumably because they look so sweet!  As with real candy, you risk making someone sick if you force them to consume too much.  Always pair with a healthy dose of explanation and annotation.

While preparing materials for the exhibition we decided to try and build a longer version of the flexible grid.  Our previous grid prints were sized to take up roughly half of the print bed (5″x7″).  The resolution of the meshes that we used for those prints were creating large .STL files (>200MB) so when we tried to create a print the size of the entire bed (8″x10″) the Objet Studio software would crash.

We revisited this problem and played around with different mesh resolutions that allowed us to still achieve the separation and movement between the parts but with much smaller file sizes.  After that we stacked three copies of the 8″x10″ print and then connected them together at alternating ends to create a 8″x30″ print.

The connection between the three layers had to be manually modeled in Rhino because of the rotation that the pieces have to go through when the grid is folded.  It took a few test prints to get the pieces to not fuse together.  The bigger issue arose when we tried to clean the 8″x30″ print and realized how delicate the vertically oriented rings became.
The majority of the rings are sliced horizontally which gives them good structural integrity, but the slices running through the rings that occur at the fold create the perfect weak spot for the ring to break along.  Unfortunately, we learned this after experiencing a number of fractures which reduced our final print to 8″x20″.  Still much bigger than earlier prints.

Like any other fabrication process, the use of a 3d printer requires an understanding of the underlying material properties which is something these studies are just beginning to inform us about.  We’ll have to see if we can find a use for that fragile cross-section in future prints.

Stephen Van Dyck and Scott Crawford of LMN Architects will be presenting some recent work at this year’s Build Boston conference on November 17 as part of a session about Generative Design.  The event is sponsored by the AIA’s Technology in Architectural Practice (TAP) group.

http://www.buildboston.com

http://network.aia.org/technologyinarchitecturalpractice/Home/