I’ve been poking around with Windows Phone more often lately. The system makes heavy use of multitouch gestures, and I thought it was a shame that it didn’t have some kind of multitouch emulation that came with its emulator. In the emulator all you get is the mouse, so you can’t pinch zoom or rotate or any other the other fun things MT can enable. I do not actually have a physical Windows Phone of my own, so this made me sad.

That was until I downloaded the Surface 2.0 SDK. It comes with a utility called ‘Input Simulator’ which is intended to allow you to emulate Surface-style interaction with an ordinary PC mouse.  It leverages the fundamental system multitouch support that Microsoft released with Windows 7.

The coolest side effect of this that I’ve seen is that it works perfectly with the Windows Phone Emulator. My friend Jobi put together this video to demonstrate.

Ta daaa

Why on earth is the standard numeric data type used throughout WPF and Silverlight a double?

97% of the time, when you’re working with numbers in any app, a float will do you  just fine. Particularly when you’re merely dealing with layout and transforms in WPF/SL, I can scarcely imagine ever needing anything more than a float.

“Why NOT doubles?” you may ask. After all, if you just use them everywhere then you never have to worry about switching between double and float.

If you do a lot of work with special animations, as I do, you wind up using calls like TransformToDescendant/TransformToVisual a lot. And you may notice, as I have, that you are really limited on the number of these calls you can make per frame without degrading your application’s performance; they are not cheap in the slightest. These calls involve a series of matrix multiplications that can really add up quickly if you’re not careful.

It might surprise you to learn that, if you switch to floats and strip away the inherent overhead of C#, matrix multiplication is almost trivial. It’s one of the more common functions a CPU will perform, and as such their designers have optimized it on hardware umpteen different ways. On most CPUs a matrix multiply using floats can be performed with merely 3 or 4 machine instructions using SIMD extensions. Even if you don’t have access to those, CPUs are inherently designed to perform math on floats much faster than on doubles.

So by using doubles everywhere, Microsoft has effectively thrown out all of the optimized paths CPUs offer to applications that need to do this sort of extremely common math. I’m too lazy at the moment to put together a quick benchmark comparison, but if I get around to it I’ll post it here. Bottom line is we can’t have as much fun with animation as we might otherwise be able to, and I, for one, have no idea why.

http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/download.aspx

As part of Microsoft’s SDK launch event, then held a 24-hour ‘code camp’ in which small teams had 24 hours to produce something cool with the Kinect. I was invited to participate, and made a simple app that turns the user into a cylon! It turned out quite well.

You can see the video currently up at http://channel9.msdn.com/live. Scroll to roughly 2:20:00 to see me bumble around on camera.

I also got a chance to wedge in a really basic Kinect-powered PowerPoint presentation viewer. It existed in OpenNI first… so not really in the spirit of the code camp, but what I thought was remarkable was that the Microsoft SDK is easy enough to get into that I was able to port it while sitting on the couch waiting to go on the air at the channel9 building. Well done Microsoft!

It doesn’t work that well because your non-cursor arm gets really tired moving up to reach the other hand, and then back down when you’re done. Particularly if you’re already reaching out pretty far with your cursor hand to reach a far corner of the screen, touching it with your other hand can be damn near impossible.

So, this idea fizzled. The search for something better than hover continues!

I’ll post a demo of it when I get the chance. The idea is just to easily gesture something that’s the equivalent of a mouse click. Pull it off and I’ll have saved people countless hours of hovering their hand in one place for ~1.5 seconds at a time.

Part 1: Intro
Part 2: Gestures
Part 3: Cursor
Part 4: Buttons
Part 5: Multiple Users

My thinking surrounding a lot of these issues has already evolved quite a bit, so some of it is outdated at this point, but I’ll stand by it as an accurate reflection of my thinking at the time, and update here as things progress.

For example, although I feel it is a crutch, I do find myself using hover-activated buttons quite a bit. Finding alternative interactions that actually work well is difficult. It’s rather easy to create an interaction that’s tuned to my particular way of gesturing, but other people do things differently, and properly identifying them all is a real challenge.

It’s not just getting the machine to recognize gestures right, however. Getting the user to understand how to do a particular gesture is the other half of this. It’s the easier half, to be sure. The solution is to play an animation displaying exactly how to perform the gesture. I’ve learned that plain English descriptions of a gesture are never enough to properly convey the right information. But I’m no artist so it’ll be a bit before I get around to those.

Although there’s always room for improvement, I feel like the original problem of not being focused on or properly understanding “the consumer” is fixed. Within the professional development community, it seems to be on our mind quite regularly these days. Good for us! I’m beginning to get a better picture of this new side of the fence, and although there are many wonderful things about it, there are a couple that bother me, which I wanted to throw out there. Maybe others feel the same, or think I’m full of crap–either way, I’m curious to find out.

First, the obvious observation–not everybody is a casual, ordinary consumer. Certainly not many people bothering to read this blog. We exist too, and our demands of our hardware and software are different. We prefer flexibility and performance over simplicity and smallness. We are impressed primarily by the raw power of the devices we hold because, unlike a “normal” consumer, we intend to use it all. Certainly we are in the minority, and that disparity in demand may necessarily lead to more expensive products for us. I’m OK with that… as long as we’re not completely forgotten in this “post-PC” era.

Why would I fear being forgotten? Well, perhaps it is somewhat irrational, but in another sense it has already happened. Modern smartphones and tablets generally run on iOS, Android, and WP7. All support a centralized distribution mechanism (app store) for the apps that users can download. It’s a great way to get apps, and it makes life a lot easier than downloading it on your PC and then syncing it to the device. But on iOS and WP7 (and with the pressure it’s faced, I would suspect eventually Android), this is the one and only mass distribution mechanism allowed. Every time you submit an app it has to be ‘approved’ by someone at Apple/Microsoft, which can take days to weeks and fail for any number of reasons.

So what if, as a developer, I wanted to create some interesting experimental apps. With the portability of these devices, and the cameras, GPS, accelerometers, and wifi at your disposal, there’s a ton of stuff you can do on them that you just can’t really pull off on a laptop. My apps wouldn’t be a big production, just me bashing away at some of the wackier ideas I’ve had. Ideally I’d like to be able to quickly publish new builds on a frequent basis… they’d surely be kinda buggy, and the UX pretty roughshod, but just getting the concept out there and generally working would satisfy me. Tons of popular programs have essentially started this way, yet on iOS and WP7, and probably eventually Android, this way of doing things is simply not allowed.

You may think “Evan, your idea sounds stupid. Why would ordinary users download your buggy, unfinished app?” Because, once again, we’re not all ordinary consumers!!  Us geeks still exist, we have smartphones too, and we’d love to try cool new stuff even if it’s really rough. It’s the same reason people sign up to beta test things. What would be wrong with having an App Store/Marketplace Beta Portal for such projects? It’d be special, sectioned off from the rest of the store, and would make you agree to some additional disclaimer when you enter. But once you did, you’d have access to a bunch of interesting little toys people are playing with.

It’s not just a matter of personal preference, either. This community of developers, experimenters, and their geeky fans are the origin of tons of technology that today has finally achieved “consumer-centric” status. Multitouch touchscreens, streaming video, voice recognition, wireless networks, motion tracking (the Kinect), GPS, hardware accelerated graphics, and even digital photography all began life as the often-broken, barely-functional toys of us geeks who loved them, and it was ultimately us being willing to put up with their eccentricity that allowed them to evolve into the more polished, friendly versions they exist as today. In the long run, stifling this community will stifle a huge source of new features for the almighty ordinary consumer.

It’s not all that bad yet, and hopefully I’m just overreacting, but I do see a dramatic change in the mentalities of device/OS developers from truly empowering and enabling users by giving them access to everything their machine is capable of, to locking it down for safety,  forcibly funneling developers down the same path to ensure a consistent user experience, and dumbing it down for people who aren’t really interested in the technology. Just remember that although ordinary consumers  are the large majority of users and they don’t care what’s going on under the hood or how it got there, they would not have hardly any of that power were it not for the people who do, and our crappy barely-functional toys.

But storyboard objects animate only with respect to the simple passage of time. This bothers me. What if I want to play an animation in response to mouse movement, or something scrolling? Here’s an example. Suppose I have an app that has a cool mechanical theme, with gears moving and things spinning like the app itself is some kind of machine. The app has a scrollable region, with a scrollbar. I want some gears attached to that scrollbar, and I want them to turn, not constantly over time, but in response to me adjusting the scrollbar’s value. That’d be much more interactive and fun.

Fundamentally, what I want is the ability to map an artibrary range of values (in this example, the range of my scrollbar) to a storyboard’s timeline, and then bind that value to the animation.

While I am a little disappointed that Microsoft doesn’t have any means of implementing your own animation clock directly (at least, not that I could find), it’s actually not that hard to set this up. Care must be taken, however, because normally animations occur on another thread (that seems to be lower priority than the UI thread). Thankfully Microsoft gives us a means to seek to a specific position along the timeline synchronously through Storyboard.SeekAlignedToLastTick.

My implementation defines 4 properties: Storyboard, Value (used in place of a clock), and Minimum/Maximum, which provide boundaries for the Value property. Most of the class is just plumbing, and you can download it using the link below. But the meat of it involves finding the duration of the storyboard, and seeking to the right timeline position when Value changes.

Finding the duration of a storyboard is strangely harder than it ought to be. Here’s what worked well for me:

Storyboard.Begin();
Storyboard.SeekAlignedToLastTick(TimeSpan.FromMilliseconds(
TimeSpan.MaxValue.TotalMilliseconds - 1));
double dur = Storyboard.GetCurrentTime().TotalMilliseconds;
Storyboard.Stop();
return dur;

Annoying. But, it works, so I’ll take it. Now, when Value changes, the storyboard timeline needs to be updated to reflect that. Here’s what that looks like:

Storyboard.SeekAlignedToLastTick(TimeSpan.FromMilliseconds(_sbDuration * (Value - Minimum) / (Maximum - Minimum)));

What’s it good for? In addition to binding to things like slider values, scrollbar positions, and progress bar values, you can use it to animate in response to user input. Maybe bind it to the position or velocity of the mouse. In my case, I intend to use it in response to a user’s movement from the Kinect. Animation like this provides feedback to the user that can be invaluable in providing discoverability to your application.

The source is available here.

• coefficients of a plane equation, that describe the plane of the display in world space
• a transform matrix that transforms from world space into screen/pixel space

So let’s get right to it. I realized that a plane in 3D space can be defined  by three points. Once you know those, getting the plane coefficients is simple. Also, if you know the relative pixel locations of each of those three points as well, you can easily construct the transform.

I’ll work backwards a little bit and assume we have these three points; their coordinates in both world space and in pixelspace. To find the plane, we’ll find its normal (coefficients , , and from before… we called it ) and then use it to find the last coefficient D. We use the rule that the cross product of two vectors is going to be normal to both of them. We can use the three points to construct two vectors that both lie on that plane. I’m going to call my three world space points , , and and my three pixel space vectors , , and . The math to get the normal of the display then looks like:

There is a minor detail here regarding which direction the normal goes. You could just as easily swap and and the normal would point in the opposite direction. But if you choose your three points carefully, this isn’t going to be a problem. Don’t choose three points at random; make sure you choose three that work well for the calibration process (more on getting the points later).

Now that you we have the normal, we need D. The plane equation again (using ) is , where is any point that lies on the plane. Just use one of our three points for (I’ll use , why not).

Piece of cake. Now for the transform! We know that the transform (which I’ll label ) is supposed to properly transform from to , and likewise for the other two. Turns out, if you have 4 of these sample transformations, it’s easy to derive the transform that successfully transforms them all (since A is a 4×4 matrix). Here’s how: I can write the transform of one vector as , but first I kinda have to redefine and . was 3D before. Now it’s 4D. The extra component should just be a 1, so the vector looks like <x, y, z, 1>. For , it used to be 2D but now it’s 4D as well. Use 0 for the z coordinate, so the vector looks like <x, y, 0, 1>.  is used as a horizontal vector in this previous equation, so suppose I ‘stacked’ it on top of , like this:

That’s just fine, according to the rules of matrix multiplication, right? If I do this with 4 points, this turns those ‘stacks’ of vectors into 4×4 matrices of their own, which we can work with to solve A! So now let’s do it with 4 points… I just need that 4th point first. A really easy way to get the 4th point is to just move one of my world points out from the screen along the normal. Because it’s moving directly away from the screen, I know it’s not actually changing the pixel position of the point. I’ll do this for so it looks nice. So here’s what it looks like now:

So now with a simple inverse, we can solve for A!! We did it!

All of this math hinges now on having 3 points, both in world space and pixel space. Finding these is the calibration step that the user must perform. You can get really creative with how this is done. I have this grand vision of using a special mirror that reflects visible light but is opaque in the IR spectrum. But I haven’t found that yet, and I don’t know squat about materials engineering, so I’m stuck there. What I did do, that works pretty well, is displayed my three points on the screen (using hard coded pixel locations), and asked the user to point at them. After pointing at them once, I ask the user to move around a bit and point at them again. This creates two lines per point, which I can find the intersection of to get the corresponding world coordinates for each point.

There may be other, better, ways to do the calibration that I haven’t thought of yet. I like the pointing approach because it doesn’t require any additional hardware, but naturally it’s limited by the imprecision of a human arm doing the pointing. Feel free to reply with any ideas.

To get an application to interpret the events directly, I implemented a focus model for controls very similar to keyboard focus. When you press a key on the keyboard, the system sends the events to the control with focus. It’s no different with Kinect gestures. To focus on a particular control, all you had to do was point at them (I’m using past tense on purpose… I’ll get to that story later).

So, one major issue was how to take the “raw” (relatively speaking) data of the user’s hand, elbow, and shoulder positions, and convert that into a specific pixel on the screen being pointed at.

There are three basic steps to this process:

1. Figure out whether the user is using their arm to point at something. If so, find a ray in world space that represents the pointing arm.
2. The display screen lies on a plane in world space. We must know this plane, and must intersect it with the ray we just computed to find a specific point on the display plane.
3. Once we have that point, a simple transform can convert it into screen space (a pixel coordinate). Of course, the trick is getting that transform in the first place.

Problem #1 is fairly simple. When a user is pointing at something, their shoulder, elbow, and hand are all basically collinear. To test this, I construct unit vectors representing the forearm and upper arm, and the use the property of dot products that states:

Where is the angle between the two vectors. We want that angle to be close to zero, so the cosine of it should be close to 1. Since the upper arm and forearm vectors are unit vectors already, their lengths are both 1, so those drop out of the equation. Realistically, it’s impossible to expect your hand, elbow, and shoulder to be precisely collinear, so we threshold it. In the end, the code looks like:

Vector3D upperarm = Elbow - Shoulder;
Vector3D forearm = Hand - Elbow;
upperarm.Normalize();
forearm.Normalize();
bool ispointing = Vector3D.DotProduct(upperarm, forearm) > 0.95;

So by checking whether the dot product is greater than 0.95, the system allows you to be ‘close enough’ without any trouble. Next, we need to define the ray that represents the pointing arm. I just use the vector that goes from the user’s shoulder to the hand, making our ray (which originates at going in direction ), and our solution to problem #1:

For problem #2, we need to know the plane that the display lies on. This requires a calibration step that I think I’ll save for my next blog post, since it’s kind of its own topic. So for now I’ll just assume I know these numbers magically. To define the plane, we use the traditional plane equation:

To intersect the ray with the plane, first realize that the ray breaks down to three different equations, one for each dimension in 3D space:

where . To find our point, we need to use substitution to factor out the x, y, and z variables, leaving u as the only unknown. Remember, and were all figured out in step 1 (they’re the components of and ). Substitution yields this nasty thing:

With a little bit of swapping things around and simplifying, we can figure out u:

Personally, I’m a big fan of vector calculus. It makes everything look so much cleaner! Did you know that in the plane equation, the coefficients A, B, and C actually represent the normal of the plane? Just for fun, if I let that same normal be represented as , the previous equation becomes

Ahhh… clean. Once you know u, you can calculate the point in space using the equations for the ray I listed earlier. Or,  just . Hee!

So that solves problem #2! We’re almost done. Problem #3 involves converting this point of ours into a pixel location on the screen. We do this using a 4×4 transform matrix. The difficulty with problem #3 is entirely in finding this matrix, which is part of the calibration step and therefore something for my next blog post. So, assuming we have it, just use it to transform the point P we just found. I set my matrix up so that the pixel locations are in the X and Y coordinates of the output vector. The Z should be zero, but if it doesn’t wind up that way… I don’t really care. I just want the pixel position.

So that’s it! After transforming P, we have an onscreen pixel (well, you’ll have to do some bounds checking to make sure of that) that the user is pointing at.

As an aside, the reason I originally referred to this method in the past tense is that it is fairly inaccurate, no matter what you do. The idea that I can pinpoint a precise pixel with my arm is a bit silly, and in practice you do indeed get a lot of jitter in the input, and the pixel that is computed winds up being offset from where you think it should be. This isn’t because of a defect in the calibration process, but in our own perceptions (which in my experience changes rather frequently, depending on subtle differences in your stance). I think in the future, when the sensors have better precision, and as long as your UI is either very simple or displayed on a VERY large screen, this technique might be useful. Lately I’ve opted to just take pointing as a rough suggestion, based simply on direction the user is pointing in. It doesn’t require calibration, the precision is much better, and the fact that you aren’t physically pointing at anything doesn’t seem to be a bother.

In the future I may try adjusting my algorithm in step #1, so that the pointing ray actually starts from the user’s eye, instead of the shoulder, but I’m really not sure that will have a huge impact on the overall quality of the experience. I’m happy with the simpler approach, but when it works well, being able to physically point at the screen and see it respond is very impressive.