Charlie Robbins

Surface Multi-touch

As I mentioned in a previous post the Microsoft Surface implements multi-touch using infrared computer vision as opposed to a capacitive approach. This is interesting for a couple of reasons. First it allows the Surface to recognize a very large number of touches. Second, it allows ambitious developers to tap directly into this infrared image data to perform more complex operations (like shape recognition). Getting at this information however is not entirely straight forward though. To simply things I created an attached behavior.

Attached Behaviors

Attached behaviors are something that I’ve blogged about before. They are a very convenient way to add user interface behavior to controls without having to implement a proper subclass. This behavior also implements a custom attached event, which allows you to easily handle and process images captured from the Surface.

Using the Behavior

Using the behavior is simple; you just have to include the library in your solution, add the xmlns and set a couple of attached properties and an event handler:

<s:SurfaceWindow
  behaviors:SurfaceWindowRawImageCapture.IsEnabled="True"
  behaviors:SurfaceWindowRawImageCapture.RawImageCaptured="YourEventHandler">
  (...)
</s:SurfaceWindow>

Optional Parameters

This behavior is designed to be flexible. There are several parameters that allow you to take advantage of this flexibility.

UseExplicitCapture=(true | false)

Setting this property to true will disable the attached event from being raised. It is false by default. Use it if you want to capture images programmatically which is explained below.

SaveTo=”C:\path\to\captured\images

Often it may be necessary to save the images that you capture to a specific location. If you set this property all images captured by the behavior will be saved to the specified directory.

Capturing Images Programmatically

The one drawback to the way that the Surface SDK allows developers to capture raw images is that the ContactDown event must be handled on the instance of the SurfaceWindow itself. What that means is that your attached event handler has to also be on the SurfaceWindow itself. The solution to this limitation is to call the behavior programmatically from your own code:

SurfaceWindowRawImageCapture.CaptureRawImageAsync(
    Application.Current.MainWindow as SurfaceWindow,
    new Rect(100, 100, 100, 100),
    result =>
    {
        // Code to execute when the result has been captured
    });

The Rect that is passed to the method defines an area to crop the entire Surface image. This is useful if you are only interested in a small sub-area of the total Surface. The callback that is passed to the method will be executed after the image has been captured.

Let me know if you run into any issues with the behavior by emailing me: charlie [you know] robbins [obviously] gmail [you know] com. As always, all of this code is available under the MIT license on GitHub.

How it happened

It’s actually a very simple story. I’m on Facebook and I see the following image:

Now I start to think to myself, who is this person? A quick Google search reveals very simply that Gianna is a porn star. Gianna is also most likely not her real name (duh!). Now the rest of this discussion has nothing to do with porn (sorry to disappoint), but merely a thought experiment as to how whatever silly algorithm they have running at Facebook could have calculated that this was a relevant suggestion.

Now that I’m back from staycation I’ve been looking around at some miscellany that needs to be done at the Lab. One of those items used my open-source (MIT ftw)Custom Chrome Window library. In the process of using it I came across a little bug that’s discussed on the WPF Window Chrome site here.

Basically, even though the “Caption bar” (aka “Title bar,” aka “Window bar,” aka “Non-client Window area”) was hidden, the Window.Top and Window.Left calculations take it into account. After a colleague of mine did some serious Google-foo on Win32 we found this solution which has been added to the library on GitHub. The solution is outlined below, starting with including some Win32 API calls:

public static class SafeNativeMethods
{
    [DllImport("user32.dll")]
    public static extern int GetWindowLong(IntPtr hwnd, int index);

    [DllImport("user32.dll")]
    public static extern int SetWindowLong(IntPtr hwnd, int index, int newStyle);

    [DllImport("user32.dll")]
    public static extern bool SetWindowPos(IntPtr hwnd, IntPtr hwndInsertAfter, int x, int y, int width, int height, uint flags);

    [DllImport("user32.dll")]
    public static extern IntPtr SendMessage(IntPtr hwnd, uint msg, IntPtr wParam, IntPtr lParam);

    internal const int WS_CHILD = 0x40000000;
    internal const int WS_VISIBLE = 0x10000000;
    internal const int LBS_NOTIFY = 0x00000001;
    internal const int HOST_ID = 0x00000002;
    internal const int LISTBOX_ID = 0x00000001;
    internal const int WS_VSCROLL = 0x00200000;
    internal const int WS_BORDER = 0x00800000;


    public const int GWL_EXSTYLE = -20;
    public const int GWL_STYLE = -16;
    public const int WS_EX_DLGMODALFRAME = 0x0001;
    public const int SWP_NOSIZE = 0x0001;
    public const int SWP_NOMOVE = 0x0002;
    public const int SWP_NOZORDER = 0x0004;
    public const int SWP_FRAMECHANGED = 0x0020;
    public const uint WM_SETICON = 0x0080;
    public const int WS_DLGFRAME = 0x00400000;

    public const int WS_SYSMENU = 0x80000;
    public const int WS_MINIMIZEBOX = 0x20000;
    public const int WS_MAXIMIZEBOX = 0x10000;
}

After which we need to call into these methods in the CustomChromeWindow class:

protected override void OnSourceInitialized(EventArgs e)
{
    base.OnSourceInitialized(e);

    // Get this window's handle
    IntPtr hwnd = new WindowInteropHelper(this).Handle;

    // Change the extended window style to not show a window icon
    int extendedStyle = SafeNativeMethods.GetWindowLong(hwnd, SafeNativeMethods.GWL_EXSTYLE);
    int style = SafeNativeMethods.GetWindowLong(hwnd, SafeNativeMethods.GWL_STYLE);

    SafeNativeMethods.SetWindowLong(
        hwnd, 
        SafeNativeMethods.GWL_STYLE,
        style & ~SafeNativeMethods.WS_SYSMENU & ~SafeNativeMethods.WS_MINIMIZEBOX & ~SafeNativeMethods.WS_MAXIMIZEBOX & SafeNativeMethods.WS_DLGFRAME);

    // Update the window's non-client area to reflect the changes
    SafeNativeMethods.SetWindowPos(
        hwnd, 
        IntPtr.Zero, 
        0, 0, 0, 0,
        SafeNativeMethods.SWP_NOMOVE | SafeNativeMethods.SWP_NOSIZE | SafeNativeMethods.SWP_NOZORDER | SafeNativeMethods.SWP_FRAMECHANGED);
}

Hopefully you’ll never have to wallow in the pain of custom window chrome, but if you do take a look at the library on GitHub.

Top on the list this year was a little “nerd spring cleaning”: getting rid of old electronics and computer parts at the Lower East Side Ecology Center, organizing some files (maybe on DropBox), and finally upgrade from Win7 RC1 to Win7 RTM. Most developers have a lot of software they require to work and I thought that I would share mine for comparison:

Acrobat Reader
Adobe AIR
Adobe CS4
ANTLR
DropBox
F#
Flash 10
Fiddler2
GhostDoc
Git
Google Chrome
Google Talk
Git
IntelliJ
Java (JRE / JDK)
RubyMine
.NET
DirectX
Divx Codec
Blend3
Blend 3 SDK
FxCop
Microsoft Office 2007
Silverlight 3
Silverlight 3 SDK
Silverlight 3 Tools for VS2010
Oslo (aka SQL Modeling)
SQL Studio Express Edition
StyleCop
Microsoft Surface SDK
Visual Studio 2010
XNA 2.0
Firefox
MySQL
Paint.NET
PowerISO
Python 2.6.2
QuickTime
Regionerate
Rooler
Ruby 1.8.7
Skype
Songbird
TortoiseSVN
TweetDeck
VLC
WinMerge
WinRAR
Xvid

If I’m missing anything, let me know!

The M language and in particular MGrammar have caught my attention recently after completing Programming Languages and Translators this past semester at Columba. I didn’t take compilers as a part of my undergraduate studies at McGill and after completing Professor Aho’s course I am really intrigued by the possibilities offered by DSLs and DSVs.

After getting setup with .NET 4, the SQL Modelling November 2009 CTP, and Intellipad my first inclination was to see how I could implement some classic ideas in language theory with MGrammar. One such idea is “inherited attributes.” An inherited attribute is a value that is captured or calculated at a parent node in an abstract syntax tree and then passed down to child nodes for any calculations or translations that they may need to perform. A classic example of this would be the “with” keyword that is available in Javascript:

with (myObj) {
  x += 3;
  y += 3;
}

This would be equivalent to:

myObj.x += 3;
myObj.y += 3;

The Surface SDK comes in two flavors; WPF and XNA. All of the input events on the WPF Surface SDK are implemented as Attached Events, which are essentially just a clever use of standard Routed Events in WPF. I won’t delve into the all details here about what Attached Events are, but they were a primary focus for an implementation of ContactMultiTap on the Surface.

For those of you not familiar with the Surface, it’s convenient to think of it as a gigantic iPhone (although it’s been around in secret long before the iPhone) that can also recognize object tags (like 2D barcodes). This is because instead of using a capacitive multi-touch approach like the iPhone it implements a multi-camera computer vision system and interprets inputs from that. In this post, I discuss my implementation as well as the benefits of time-based multi-tap input on this “big a** table.”

Recently, I have been more involved with development on the Microsoft Surface as part of my work with the Computer Graphics and User Interfaces Lab. We also have a couple of Surfaces here at Lab49. The Surface is a very interesting device to me because of it’s potential not necessarily as a consumer device but in marketplace and enterprise scenarios. Furthermore, it’s SDK is implemented in WPF using very clever tricks like attached events.

One thing that frustrated me about the Surface though, was how closed off its development community had been. If Microsoft really wants this thing to take off (a topic that has been the subject of heated debate), then why not just let anyone who wants to download the tools and SDK to test it out? Well it seems that Microsoft has come to their senses and put all the Surface materials (technical resources, documentation, and downloads) up for public consumption. If you’re a UI enthusiast or just a WPF developer I highly suggest checking it out:

Download Surface SDK 1.0 (SP1)
Surface Documentation on MSDN
New Surface Forums

Happy Coding!

Patrick Smacchia has an interesting post on interesting findings in the diff between .NET 4.0 Beta1 and Beta2. I've been a little behind the times on updates, but this article brought me up to speed with some API changes quickly. Some interesting changes to System.Windows.Input to reflect the native Multi-Touch support for Window7. I wonder how the Surface SDK will change with the transition from Vista to Win7.

I came across a concept video of Microsoft Courier today and I was blown away. I have wanted a device like this for as long as I can remember; it is basically the same form factor as Penny’s computer “book” from Inspector Gadget. It was a folding computer book that had two separate touch screens.

There seems to have been some research done by Microsoft on the concept for several years. I came across this research paper on Courier that describes it as a “Collaborative Phone-Based File Exchange System.” There is more information about the MSR Courier Group here. A lot of the interactions in the demo video remind me of some ubiquitous computing paradigms; the hand writing of the URL into the web browser in particular.

Over the last several months there have been a number of tools, snippets, macros, and templates that I’ve used at Lab49 to streamline my .NET development and make things move more quickly. Many of these resources were created by my colleague Dan Simon. Here’s the list:

Tools

Visual Studio Team System (a version of Visual Studio that includes set of tools on top of the standard VS2008): http://msdn.microsoft.com/en-us/teamsystem/default.aspx

StyleCop (Code Style Analysis): http://code.msdn.microsoft.com/sourceanalysis

FxCop (Static Code Analysis, we actually use the “Static Analysis” tool in VS Team System, which uses FxCop under the covers): http://msdn.microsoft.com/en-us/library/bb429476%28VS.80%29.aspx

GhostDoc (Generate XML Comments automatically with some nice features): http://submain.com/products/ghostdoc.aspx

Regionerate (Automatically organize your code with regions): http://www.rauchy.net/regionerate/

Code Snippets and Templates

StyleCopClass.zip: A Visual Studio template for creating StyleCop compliant classes by default. To install copy to C:\Documents And Settings[Your User]\Visual Studio 2008\Templates\Item Tempaltes

Regionerate StyleCop Rules.xml: A set of rules for Regionerate that organizes code according to StyleCop guidelines. To install copy to C:\Program Files\Regionerate\My Code Layouts

StyleCopMacros.vb: A Visual Studio macro that executes a number of useful tools together for making code more StyleCop compliant.

cmdprop.snippet: A snippet for a command property in a ViewModel. To install copy to C:\Documents and Settings[Your User]\Documents\Visual Studio 2008\Code Snippets\Visual C#\My Code Snippets

filehead.snippet: A snippet for an XML file header including relevant information. To install copy to C:\Documents and Settings[Your User]\Documents\Visual Studio 2008\Code Snippets\Visual C#\My Code Snippets

notifyprop.snippet: A snippet for a property in a class that implements INotifyPropertyChanged. To install copy to C:\Documents and Settings[Your User]\Documents\Visual Studio 2008\Code Snippets\Visual C#\My Code Snippets

It’s important to note that some of these snippets and templates require a little editing for each project they are used on. The filehead.snippet and StyleCopClass.zip files have the name “MyCompany Inc.” in them. To change it just simply open the files in Notepad (or other text editor), and save the files in the appropriate directory outlined above. All of the resources are available on GitHub. Enjoy!