Stupid Emulator Tricks

Posted: July 26, 2011 in Uncategorized

Or… trick. Just one.

I’ve been poking around with Windows Phone more often lately. The system makes heavy use of multitouch gestures, and I thought it was a shame that it didn’t have some kind of multitouch emulation that came with its emulator. In the emulator all you get is the mouse, so you can’t pinch zoom or rotate or any other the other fun things MT can enable. I do not actually have a physical Windows Phone of my own, so this made me sad.

That was until I downloaded the Surface 2.0 SDK. It comes with a utility called ‘Input Simulator’ which is intended to allow you to emulate Surface-style interaction with an ordinary PC mouse.  It leverages the fundamental system multitouch support that Microsoft released with Windows 7.

The coolest side effect of this that I’ve seen is that it works perfectly with the Windows Phone Emulator. My friend Jobi put together this video to demonstrate.

Ta daaa


Posted: June 23, 2011 in Graphics, Rant

I feel like a quick rant….

Why on earth is the standard numeric data type used throughout WPF and Silverlight a double?

97% of the time, when you’re working with numbers in any app, a float will do you  just fine. Particularly when you’re merely dealing with layout and transforms in WPF/SL, I can scarcely imagine ever needing anything more than a float.

“Why NOT doubles?” you may ask. After all, if you just use them everywhere then you never have to worry about switching between double and float.

If you do a lot of work with special animations, as I do, you wind up using calls like TransformToDescendant/TransformToVisual a lot. And you may notice, as I have, that you are really limited on the number of these calls you can make per frame without degrading your application’s performance; they are not cheap in the slightest. These calls involve a series of matrix multiplications that can really add up quickly if you’re not careful.

It might surprise you to learn that, if you switch to floats and strip away the inherent overhead of C#, matrix multiplication is almost trivial. It’s one of the more common functions a CPU will perform, and as such their designers have optimized it on hardware umpteen different ways. On most CPUs a matrix multiply using floats can be performed with merely 3 or 4 machine instructions using SIMD extensions. Even if you don’t have access to those, CPUs are inherently designed to perform math on floats much faster than on doubles.

So by using doubles everywhere, Microsoft has effectively thrown out all of the optimized paths CPUs offer to applications that need to do this sort of extremely common math. I’m too lazy at the moment to put together a quick benchmark comparison, but if I get around to it I’ll post it here. Bottom line is we can’t have as much fun with animation as we might otherwise be able to, and I, for one, have no idea why.

It’s here, in beta form!

As part of Microsoft’s SDK launch event, then held a 24-hour ‘code camp’ in which small teams had 24 hours to produce something cool with the Kinect. I was invited to participate, and made a simple app that turns the user into a cylon! It turned out quite well.

You can see the video currently up at Scroll to roughly 2:20:00 to see me bumble around on camera.

I also got a chance to wedge in a really basic Kinect-powered PowerPoint presentation viewer. It existed in OpenNI first… so not really in the spirit of the code camp, but what I thought was remarkable was that the Microsoft SDK is easy enough to get into that I was able to port it while sitting on the couch waiting to go on the air at the channel9 building. Well done Microsoft!

Well, that took entirely too long. Anyway, I tried out the idea I mentioned in my last post and it didn’t go very well. Oh well. Basically, the idea was to provide an analog of mouse down/mouse up events using the Kinect with a hand cursor by having the user touch their hands together. If both hands were together, consider that the equivalent of a ‘down’ event. When they came apart, consider it an ‘up’ event.

It doesn’t work that well because your non-cursor arm gets really tired moving up to reach the other hand, and then back down when you’re done. Particularly if you’re already reaching out pretty far with your cursor hand to reach a far corner of the screen, touching it with your other hand can be damn near impossible.

So, this idea fizzled. The search for something better than hover continues!

I’m excited! I don’t want to divulge details yet but I’ve thought of a way to interact with an application using Kinect that doesn’t involve hover-over, and if it works well it ought to have all the power and flexibility of a standard Apple mouse! (pretty low standard, I know… /rib /rib)

I’ll post a demo of it when I get the chance. The idea is just to easily gesture something that’s the equivalent of a mouse click. Pull it off and I’ll have saved people countless hours of hovering their hand in one place for ~1.5 seconds at a time. :)

In case you haven’ t seen, I have a series of blogs published on IdentityMine’s website regarding Kinect-based interaction and some of the thoughts and experiments I’ve done with it. You’ll find it here:

Part 1: Intro
Part 2: Gestures
Part 3: Cursor
Part 4: Buttons
Part 5: Multiple Users

My thinking surrounding a lot of these issues has already evolved quite a bit, so some of it is outdated at this point, but I’ll stand by it as an accurate reflection of my thinking at the time, and update here as things progress.

For example, although I feel it is a crutch, I do find myself using hover-activated buttons quite a bit. Finding alternative interactions that actually work well is difficult. It’s rather easy to create an interaction that’s tuned to my particular way of gesturing, but other people do things differently, and properly identifying them all is a real challenge.

It’s not just getting the machine to recognize gestures right, however. Getting the user to understand how to do a particular gesture is the other half of this. It’s the easier half, to be sure. The solution is to play an animation displaying exactly how to perform the gesture. I’ve learned that plain English descriptions of a gesture are never enough to properly convey the right information. But I’m no artist so it’ll be a bit before I get around to those. :)

Steve Jobs called this the “post-PC” era, where simpler smartphones and tablets dominate a market that used to be exclusive to desktops and laptops.  Even game consoles can do many of the basic services of a computer. The focus on the consumer has, in the past few years, taken on its primary role as the center of developer attention (and rightfully so!). We see it not only in the growth of the smartphone and tablet markets, but also in the growth of the casual game market, and the expansion of game consoles’ capabilities into space that used to be reserved for a standard PC.

Although there’s always room for improvement, I feel like the original problem of not being focused on or properly understanding “the consumer” is fixed. Within the professional development community, it seems to be on our mind quite regularly these days. Good for us! I’m beginning to get a better picture of this new side of the fence, and although there are many wonderful things about it, there are a couple that bother me, which I wanted to throw out there. Maybe others feel the same, or think I’m full of crap–either way, I’m curious to find out.

First, the obvious observation–not everybody is a casual, ordinary consumer. Certainly not many people bothering to read this blog. We exist too, and our demands of our hardware and software are different. We prefer flexibility and performance over simplicity and smallness. We are impressed primarily by the raw power of the devices we hold because, unlike a “normal” consumer, we intend to use it all. Certainly we are in the minority, and that disparity in demand may necessarily lead to more expensive products for us. I’m OK with that… as long as we’re not completely forgotten in this “post-PC” era.

Why would I fear being forgotten? Well, perhaps it is somewhat irrational, but in another sense it has already happened. Modern smartphones and tablets generally run on iOS, Android, and WP7. All support a centralized distribution mechanism (app store) for the apps that users can download. It’s a great way to get apps, and it makes life a lot easier than downloading it on your PC and then syncing it to the device. But on iOS and WP7 (and with the pressure it’s faced, I would suspect eventually Android), this is the one and only mass distribution mechanism allowed. Every time you submit an app it has to be ‘approved’ by someone at Apple/Microsoft, which can take days to weeks and fail for any number of reasons.

So what if, as a developer, I wanted to create some interesting experimental apps. With the portability of these devices, and the cameras, GPS, accelerometers, and wifi at your disposal, there’s a ton of stuff you can do on them that you just can’t really pull off on a laptop. My apps wouldn’t be a big production, just me bashing away at some of the wackier ideas I’ve had. Ideally I’d like to be able to quickly publish new builds on a frequent basis… they’d surely be kinda buggy, and the UX pretty roughshod, but just getting the concept out there and generally working would satisfy me. Tons of popular programs have essentially started this way, yet on iOS and WP7, and probably eventually Android, this way of doing things is simply not allowed.

You may think “Evan, your idea sounds stupid. Why would ordinary users download your buggy, unfinished app?” Because, once again, we’re not all ordinary consumers!!  Us geeks still exist, we have smartphones too, and we’d love to try cool new stuff even if it’s really rough. It’s the same reason people sign up to beta test things. What would be wrong with having an App Store/Marketplace Beta Portal for such projects? It’d be special, sectioned off from the rest of the store, and would make you agree to some additional disclaimer when you enter. But once you did, you’d have access to a bunch of interesting little toys people are playing with.

It’s not just a matter of personal preference, either. This community of developers, experimenters, and their geeky fans are the origin of tons of technology that today has finally achieved “consumer-centric” status. Multitouch touchscreens, streaming video, voice recognition, wireless networks, motion tracking (the Kinect), GPS, hardware accelerated graphics, and even digital photography all began life as the often-broken, barely-functional toys of us geeks who loved them, and it was ultimately us being willing to put up with their eccentricity that allowed them to evolve into the more polished, friendly versions they exist as today. In the long run, stifling this community will stifle a huge source of new features for the almighty ordinary consumer.

It’s not all that bad yet, and hopefully I’m just overreacting, but I do see a dramatic change in the mentalities of device/OS developers from truly empowering and enabling users by giving them access to everything their machine is capable of, to locking it down for safety,  forcibly funneling developers down the same path to ensure a consistent user experience, and dumbing it down for people who aren’t really interested in the technology. Just remember that although ordinary consumers  are the large majority of users and they don’t care what’s going on under the hood or how it got there, they would not have hardly any of that power were it not for the people who do, and our crappy barely-functional toys.