VBandi's blog

Kinect Interactions with(out) WPF – Part III: Demystifying the Interaction Stream

2013-05-03T19:43:20Z

In this part of my Kinect Interaction blog post series, we go deep into the rabbit hole, and examine the foundation of Kinect Interactions – the InteractionStream, upon which the entire library is built. This is a risky ride – with no official documentation, we can only count on our trusty reflector, the source code of the Kinect Interaction SDK and careful exploration.

You only need to access the treasures of InteractionStream, if you want to go beyond what the KinectRegion and other controls provide. For example, you want to create your own KinectRegion, you want to zoom a map by gripping it with two hands, or want to build your entirely new interaction model, using two hands along with the press and grip gestures.

Initializing the InteractionStream

Initializing the InteractionStream is much like initializing the DepthStream or the SkeletonStream. If you have a KinectSensor object, all you need are the next two lines of code:

   1:  _interactionStream = new InteractionStream(_sensor, new DummyInteractionClient());

   2:  _interactionStream.InteractionFrameReady += InteractionStreamOnInteractionFrameReady;

What is the DummyInteractionClient that we pass in the constructor? Turns out, that the InteractionStream needs an object that implements the IInteractionClient interface to work. The interface defines a single method:

   1:  public interface IInteractionClient

   2:  {

   3:    InteractionInfo GetInteractionInfoAtLocation(

   4:        int skeletonTrackingId,

   5:        InteractionHandType handType,

   6:        double x,

   7:        double y);

   8:  }

X and Y are obviously coordinates, skeletonTrackingID is the identifier of the user. The InteractionHandType type is a simple enum with the values of “None”, “Left”, “Right”, identifying the hand of the user. The InteractionInfo class has the following members:

   1:  public sealed class InteractionInfo

   2:  {

   3:    public bool IsPressTarget { get; set; }

   4:    public int PressTargetControlId { get; set; }

   5:    public double PressAttractionPointX { get; set; }

   6:    public double PressAttractionPointY { get; set; }

   7:    public bool IsGripTarget { get; set; }

   8:  }

At this point, my Spider-sense tingles, and I think I have guessed the purpose of the IInteractionClient interface: it’s GetInteractionInfoAtLocation method is called to determine that when the hand cursor of a certain user’s certain hand is at a certain position, is that position pressable and / or grippable. So, I created a dummy implementation of the IInteractionClient interface that keeps saying YES to all these questions.

   1:  public class DummyInteractionClient : IInteractionClient

   2:  {

   3:      public InteractionInfo GetInteractionInfoAtLocation(

   4:          int skeletonTrackingId,

   5:          InteractionHandType handType,

   6:          double x,

   7:          double y)

   8:      {

   9:          var result = new InteractionInfo();

  10:          result.IsGripTarget = true;

  11:          result.IsPressTarget = true;

  12:          result.PressAttractionPointX = 0.5;

  13:          result.PressAttractionPointY = 0.5;

  14:          result.PressTargetControlId = 1;

15:

  16:          return result;

  17:      }

  18:  }

It seems like we are on track. Just pass an initialized KinectSensor object and this DummyInteractionClient to the constructort of the InteractionStream, and we should be all set, right? Well, not quite. The InteractionFrameReady event does not fire.

Interaction Needs Skeleton and Depth

It turns out, that for the InteractionStream to work, it needs to process the data from both the depth and the skeleton streams. So, we need to initiate all three of the streams. This is what the entire OnLoaded method (which you have to wire up either in XAML or in the constructor of the page) looks like:

   1:  private KinectSensor _sensor;  //The Kinect Sensor the application will use

   2:  private InteractionStream _interactionStream;

3:

   4:  private Skeleton[] _skeletons; //the skeletons

   5:  private UserInfo[] _userInfos; //the information about the interactive users

6:

   7:  private void OnLoaded(object sender, RoutedEventArgs routedEventArgs)

   8:  {

   9:      if (DesignerProperties.GetIsInDesignMode(this))

  10:          return;

11:

  12:      // this is just a test, so it only works with one Kinect, and quits if that is not available.

  13:      _sensor = KinectSensor.KinectSensors.FirstOrDefault();

  14:      if (_sensor == null)

  15:      {

  16:          MessageBox.Show("No Kinect Sensor detected!");

  17:          Close();

  18:          return;

  19:      }

20:

  21:      _skeletons = new Skeleton[_sensor.SkeletonStream.FrameSkeletonArrayLength];

  22:      _userInfos = new UserInfo[InteractionFrame.UserInfoArrayLength];

23:

24:

  25:      _sensor.DepthStream.Range = DepthRange.Near;

  26:      _sensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);

27:

  28:      _sensor.SkeletonStream.TrackingMode = SkeletonTrackingMode.Seated;

  29:      _sensor.SkeletonStream.EnableTrackingInNearRange = true;

  30:      _sensor.SkeletonStream.Enable();

31:

  32:      _interactionStream = new InteractionStream(_sensor, new DummyInteractionClient());

  33:      _interactionStream.InteractionFrameReady += InteractionStreamOnInteractionFrameReady;

34:

  35:      _sensor.DepthFrameReady += SensorOnDepthFrameReady;

  36:      _sensor.SkeletonFrameReady += SensorOnSkeletonFrameReady;

37:

  38:      _sensor.Start();

  39:  }

It doesn’t seem so simple now, so let's walk through each line of code and see what they do. The first five lines define local variables that hold a reference to the Kinect sensor the application will use, the InteractionStream itself, the skeletons identified by the SkeletonStream, and the hand position information about the users as determined by the InteractionStream.

Line 9 makes sure that we don't do anything when running in the designer. Lines 12 through 19 initialize the first available KinectSensor. Note, that for simplicity's sake, we are not using the KinectSensorChooser introduced in the first part of the series.

After that, the _skeletons array is initialized to the maximum amount of skeletons the SkeletonStream can handle. Similarly, the _userInfos array (which will store the information we get from the InteractionStream) is initialized to the maximum number of users the InteractionStream can work with. Both of these are 6 at the moment, but it is safer to use this initialization method to be future-proof. The rest is fairly standard initialization for the Kinect SDK. Note that we are using than the near mode of the Kinect for Windows sensor, which may not be available if you are using the Kinect for Xbox sensor. In this case, you may want turn these off in lines 25 and 29.

The event handlers for the SkeletonFrameReady and DepthFrameReady events are fairly simple, but there’s a little bit of error handling that makes them longer. I emphasized the key parts below:

   1:  private void SensorOnSkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs skeletonFrameReadyEventArgs)

   2:  {

   3:      using (SkeletonFrame skeletonFrame = skeletonFrameReadyEventArgs.OpenSkeletonFrame())

   4:      {

   5:          if (skeletonFrame == null)

   6:              return;

7:

   8:          try

   9:          {

  10:              skeletonFrame.CopySkeletonDataTo(_skeletons);

  11:              var accelerometerReading = _sensor.AccelerometerGetCurrentReading();

  12:              _interactionStream.ProcessSkeleton(_skeletons, accelerometerReading, skeletonFrame.Timestamp);

  13:          }

  14:          catch (InvalidOperationException)

  15:          {

  16:              // SkeletonFrame functions may throw when the sensor gets

  17:              // into a bad state.  Ignore the frame in that case.

  18:          }

  19:      }

  20:  }

Essentially, we acquire a SkeletonFrame, and pass it to the ProcessSkeleton method of the InteractionStream, along with the accelerometer reading coming from the Kinect sensor and the timestamp.

The DepthFrameReady event handler is pretty similar:

   1:  private void SensorOnDepthFrameReady(object sender, DepthImageFrameReadyEventArgs depthImageFrameReadyEventArgs)

   2:  {

   3:      using (DepthImageFrame depthFrame = depthImageFrameReadyEventArgs.OpenDepthImageFrame())

   4:      {

   5:          if (depthFrame == null)

   6:              return;

7:

   8:          try

   9:          {

  10:              _interactionStream.ProcessDepth(depthFrame.GetRawPixelData(), depthFrame.Timestamp);

  11:          }

  12:          catch (InvalidOperationException)

  13:          {

  14:              // DepthFrame functions may throw when the sensor gets

  15:              // into a bad state.  Ignore the frame in that case.

  16:          }

  17:      }

  18:  }

We acquire a DepthFrame, get its raw pixel data, and pass it over for processing to the InteractionStream, along with a timestamp. And finally, the InteractionFrameReady event gets called. Yay!

Dissecting InteractionFrameReadyEventArgs

So, now that we have finally made the InteractionStream call the InteractionFrameReady event, we can start analyzing what kind of data we can get to. The first step is to get our hand on an InteractionFrame, and the interaction data it contains:

   1:  using (var iaf = args.OpenInteractionFrame()) //dispose as soon as possible

   2:  {

   3:      if (iaf == null)

   4:          return;

5:

   6:      iaf.CopyInteractionDataTo(_userInfos);

   7:  }

Remember – the _userInfos array is an array of UserInfo objects. A UserInfo object contains all the interaction data related to that specific user. More specifically it has two public properties:

int SkeletonTrackingId, which is the id of the user corresponding to the user ID in the SkeletonStream. If the skeletonTrackingID property equals 0, its means that the UserInfo object does not contain a valid user.
ReadOnlyCollection<InteractionHandPointer> HandPointers, which contains the hands of the specified user (this collection probably has no more than two items most of the time).

Let's dig deeper! The next class to examine is the InteractionHandPointer class. This class represents one hand of the user. I have used a small WPF application along with the Kinect Studio to understand the properties, and displayed all properties of the InteractionHandPointer class in a nifty little window. Here is the output from my Kinect, with me sitting in front of my computer:

On the right is the depth stream, and the super-cool 3D Viewer from Kinect Studio. I have both of my hands in the air, with the left hand closed, and the right one open. The keen eyed ones among you may realize that the two pictures seem to be not just from a different point of view, but also mirrored – it looks like my right hand is closed instead of my left. It is just how the display is.

On the left side of the screenshot, you can see the output of my little program, showing the properties of the InteractionHandPointer class (and a little more). I will discuss all of these in detail below.

InteractionHandType HandType

Indicates whether the hand that belongs to the InteractionHandPointer is the left hand (Left), right hand (Right), or neither (None). Which sounds a bit funny, but the enumeration still has this option.

bool IsActive

Indicates whether this hand is active. A hand is considered active if it is raised and is in front of the user.

InteractionHandEventType HandEventType

This is another enumeration, showing the grip event that has happened to the hand in this frame. Which means that if you have performed a Grip gesture (closed your hand), the value of HandEventType will only be Grip or GripRelease for one frame! It would be very useful to have a boolean property for this event, since now we have to track the last event that is not None to know whether a hand is open or closed.

InteractionHandEventType LastHandEventType

This is unfortunately not part of the InteractionHandPointer class, but solves the abovementioned problem of storing the last grip event for each hand. You can find a sample code for tracking the last HandEventType for every user and both hands in the attached solution.

A bit of good news though: it seems like when the Kinect detects a user, it correctly sets the HandEventType property in the first frame. This means that even if the user comes into the play area with his fist closed, you will know and you will be able to react accordingly.

bool IsPrimaryForUser

Every user has one primary hand when it comes to Kinect Interactions, although you can ignore this when using the InteractionStream directly. The primary hand is the one which the user has raised first. If the user raises his / her other hand, the second hand is not considered active as long as the first hand is still raised. If the user lowers his first hand, then the second hand becomes primary (assuming it is still raised). In the picture above you can see that I have lifted my right hand first.

bool IsInteractive

To be honest, I have no idea what this property indicates. It seems like you can have both hands interactive at the same time, but it is possible to have a hand active, but not interactive. If you know what this property is about, please let me know in the comments, and I will update the post.

float PressExtent and bool IsPressed

Apart from the grip gesture, another gesture the InteractionStream can help you with is the “Press” gesture. You can tell that a lot of consideration and thought went into the implementation of this seemingly simple gesture. You can press towards the screen, you can press forward in front of you, and it will still detect the pressing of the button fairly well. You can start from an arm close to your body or one which is almost entirely extended. You can perform the gesture quickly or with moderate speed. However, if you perform it slowly, the gesture will not be recognized, since it amounts to just pushing the PhIZ (Physical Interaction Zone – the area where your hand is considered as a pointer and mapped to the screen) a bit further from your body.

PressExtent is proportional to how far along the press gesture the hand is. It is used to fill the hand cursor more and more to indicate that you are getting close to touching the button. When PressExtent is 1 or higher, IsPressed becomes true.

Note: I wish that the grip gesture would also have an IsGripping or IsHandClosed boolean for consistency’s sake. Or events, indicating the change of these states. Maybe in an upcoming SDK…

bool IsTracked

The name of the property indicates that this property shows whether the Kinect sensor can directly see the hand. The Skeleton engine has this information, but I have not been able to hide my hand so much that the IsTracked turned to false.

float RawX, RawY, RawZ, X, Y

The raw positions of the hand, relative to the Kinect. For the RawZ coordinate, the 0 point is a little bit in front of your body, and the numbers increase as you extend your hand towards the sensor. The X and Y coordinates have the same values as the RawX and RawY coordinates. These coordinates seem to cover the PhIZ in front of the user. For X, the numbers increase to the user’s right, and for Y the numbers increase downwards. The origin moves with the user. The PhIZ seems to be between the raw coordinates (0,0,0) and (1,1,1). I am saying “seems to be”, since this is a bit gray area, where some documentation would be very useful.

Show Me the Codez!

Finally, here is the entire code of the InteractionFrameReady event handler. Of course, the entire solution is also downloadable.

   1:  private Dictionary<int, InteractionHandEventType> _lastLeftHandEvents = new Dictionary<int, InteractionHandEventType>();

   2:  private Dictionary<int, InteractionHandEventType> _lastRightHandEvents = new Dictionary<int, InteractionHandEventType>();

3:

   4:  private void InteractionStreamOnInteractionFrameReady(object sender, InteractionFrameReadyEventArgs args)

   5:  {

   6:      using (var iaf = args.OpenInteractionFrame()) //dispose as soon as possible

   7:      {

   8:          if (iaf == null)

   9:              return;

10:

  11:          iaf.CopyInteractionDataTo(_userInfos);

  12:      }

13:

  14:      StringBuilder dump = new StringBuilder();

15:

  16:      var hasUser = false;

  17:      foreach (var userInfo in _userInfos)

  18:      {

  19:          var userID = userInfo.SkeletonTrackingId;

  20:          if (userID == 0)

  21:              continue;

22:

  23:          hasUser = true;

  24:          dump.AppendLine("User ID = " + userID);

  25:          dump.AppendLine("  Hands: ");

  26:          var hands = userInfo.HandPointers;

  27:          if (hands.Count == 0)

  28:              dump.AppendLine("    No hands");

  29:          else

  30:          {

  31:              foreach (var hand in hands)

  32:              {

  33:                  var lastHandEvents = hand.HandType == InteractionHandType.Left

  34:                                              ? _lastLeftHandEvents

  35:                                              : _lastRightHandEvents;

36:

  37:                  if (hand.HandEventType != InteractionHandEventType.None)

  38:                      lastHandEvents[userID] = hand.HandEventType;

39:

  40:                  var lastHandEvent = lastHandEvents.ContainsKey(userID)

  41:                                          ? lastHandEvents[userID]

  42:                                          : InteractionHandEventType.None;

43:

  44:                  dump.AppendLine();

  45:                  dump.AppendLine("    HandType: " + hand.HandType);

  46:                  dump.AppendLine("    HandEventType: " + hand.HandEventType);

  47:                  dump.AppendLine("    LastHandEventType: " + lastHandEvent);

  48:                  dump.AppendLine("    IsActive: " + hand.IsActive);

  49:                  dump.AppendLine("    IsPrimaryForUser: " + hand.IsPrimaryForUser);

  50:                  dump.AppendLine("    IsInteractive: " + hand.IsInteractive);

  51:                  dump.AppendLine("    PressExtent: " + hand.PressExtent.ToString("N3"));

  52:                  dump.AppendLine("    IsPressed: " + hand.IsPressed);

  53:                  dump.AppendLine("    IsTracked: " + hand.IsTracked);

  54:                  dump.AppendLine("    X: " + hand.X.ToString("N3"));

  55:                  dump.AppendLine("    Y: " + hand.Y.ToString("N3"));

  56:                  dump.AppendLine("    RawX: " + hand.RawX.ToString("N3"));

  57:                  dump.AppendLine("    RawY: " + hand.RawY.ToString("N3"));

  58:                  dump.AppendLine("    RawZ: " + hand.RawZ.ToString("N3"));

  59:              }

  60:          }

61:

  62:          tb.Text = dump.ToString();

  63:      }

64:

  65:      if (!hasUser)

  66:          tb.Text = "No user detected.";

  67:  }

And the MainWindow is simply a TextBox inside a ViewBox so that the text size follows the window’s size:

   1:  <Window x:Class="InteractionStreamTest.MainWindow"

   2:          xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"

   3:          xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"

   4:          Title="MainWindow" Height="350" Width="525">

   5:      <Grid>

   6:          <Viewbox>

   7:              <TextBlock Name="tb" FontFamily="Lucida Console" Text="Initializing..."/>

   8:          </Viewbox>

   9:      </Grid>

  10:  </Window>

Summary

I’ve shown you how you can get started with the InteractionStream and acquire information about the hands of the users, detect active hands, pressing (IsPressed and PressExtent), and open / closed hands (HandEventType and our custom LastHandEventType). What I’ve shown here does not require WPF – you can use this from any .NET application, be it XNA, Windows Forms or even Command Line.

Please share your thoughts, feedback, etc. in the comments!

Kinect Interaction with WPF Part II: Getting Started Easier

2013-04-19T14:27:33Z

In the previous post of this series, I have introduced the Kinect for Windows SDK 1.7, and the key controls and concepts within its Kinect Interactions toolkit. If you have followed through that post, you have seen all the work and code you need to get the basics running. Most of the initialization code was biolerplate, and code that you can easily copy and paste into your own project.

I have packaged up the initialization code into a UserControl called KinectChooserEx, and wired up a couple of dependency properties.

Note: I could have inherited from KinectChooser instead of packaging it into a user control. But I ran into some serious strong name issues, and wanted more consistent naming for the properties, so it was better to hide some internal workings.

The NearMode property is a boolean value, and can be used to turn Near Mode on and off. Changing Near Mode also sets the EnableTrackingInNearRange of the SkeletonStream. You will probably want to keep NearMode as false if you are using a Kinect for XBox sensor instead of a Kinect for Windows one. Otherwise the interaction engine works better at close range, so I have set NearMode to be true by default.

The other property is KinectSensor, which contains null if no sensor is initialized, or a KinectSensor object if one is.

Both of these properties are bindable. With KinectChooserEx, you can finally get started with your own Kinect project purely in Blend, without writing a single line of code (at least for the interaction part). Here is a step by step tutorial on how to do this:

Create a New Project and Add KinectChooserEx.xaml and KinectChooserEx.xaml.cs

Create a new WPF project, and add the necessary references as described in my previous post. Drag and drop the KinectChooserEx.xaml and KinectChooserEx.xaml.cs files from Explorer into the project. You will also want to add the KinectInteraction170_32.dll and KinectInteraction170_64.dll files at this point (again, see my previous post).

Make sure that the project compiles.

Add KinectChooserEx to the MainPage

You should be able to find the KinectChooserEx user control in Blend now. Press CTRL+. or the >> button on the left toolbar to bring up the Assets panel. If you start typing “Kinect”, you should see the KinectChooserEx control:

Drag and drop it onto the design surface.

Position it so that it looks a bit better. I am setting the HorizontalAlignment and VerticalAlignment properties so that the control will have adequate space to expand (remember, if you move the mouse over it, KinectChooser shows error status and explanation).

You should also see a new Property Category called “Kinect”, with a lonely NearMode property, set to true by default:

Adding a KinectRegion

As I showed in the last post, the KinectRegion control is the heart of the entire Kinect Interaction experience. So, let’s add one of these to our project. Again, just add it from the Assets panel, and place it so that it occupies most of the screen:

To make the KinectRegion work, it needs a KinectSensor object. This is what KinectChooserEx can provide, so let’s wire up this binding. Select the KinectRegion control, find its KinectSensor property, and click the little property peg:

Select Bind to Element and click on the KinectChooserEx control in either the designer area or the Object tree. Blend automatically chooses the KinectChooserEx’s KinectSensor property, since it has the right type. All we have to do is press the OK button.

Adding a KinectUserViewer control

Similarly, we can add a KinectUserViewer control to the application. I have added it above the KinectRegion, so that the hand cursor will be in front of the user’s silhouette – but otherwise I allowed to take up the entire screen. It is up to you – you may want to put it into the top, or one of the corners.

You have to bind the KinectUserViewer control’s KinectRegion property to the KinectRegion we created earlier. Binding the KinectSensor property will not work. Unfortunately, it seems like the KinectRegion property is not shown in Blend, so we have to do this part manually in XAML.

First, let’s add a name to the KinectRegion control. I called it “region” for now.

Next, select the KinectUserViewer control, and open the split view. Type the following within the XAML tag:

k:KinectRegion.KinectRegion="{Binding ElementName=region}"

So, the entire XAML tag looks like this:

Let’s See it Work!

Press F5, and Blend should launch the application. If all is well, you should see something like this:

If the application does not start or crash, please load the solution up in Visual Studio and check the error message. The first post in this series has some troubleshooting info for the most common issues.

Using the Near Mode

For fun, we can add a checkbox that controls the Near Mode of the Kinect Sensor. All you have to do is bind its IsChecked property to the NearMode property of the KinectChooserEx control, and make sure the binding is set to be a TwoWay binding:

Now you can control whether you want near mode on or off.

As you can see above, when near mode is off, it cannot detect my left hand anymore even though I moved it back a bit between the two screenshots.

Summary

I hope you will find this little Blend + Kinect tutorial and the KinectChooserEx control useful. Feel free to enhance it, add more properties, features, bugfixes, etc. You can download the code from here (it also contains the project from the previous article). Please keep the feedback coming!

Kinect Interactions with WPF - Part I: Getting Started

2013-03-25T14:24:22Z

Introduction

Microsoft has recently released V1.7 of the Kinect for Windows SDK. And they immediately made a mistake – there are so many new and awesome stuff in it, that it should be called 2.0 :)

The 1.7 SDK focuses on two key new features: Kinect Fusion and Kinect Interactions. Kinect Fusion is essentially a 3D scanner that enabled developers to create accurate 3D renderings in real time.

Kinect Fusion. Source: Kinect for Windows blog

While Kinect Fusion is beyond awesome, this blog post series focuses on the other big new feature of the Kinect for Windows (K4W) SDK 1.7: Kinect Interactions. Kinect Interactions allow developers to create interactive applications with a new, well-designed and ergonomically friendlier interaction language than before. The Kinect SDK can now detect a “push towards the screen” gesture to activate buttons, and this is the recommended “activate” gesture now instead of the old “hover, wait until the circle fills and its pressed” gesture.

Browsing lists larger than a dozen or so items has been even more difficult using Kinect. The recommended solution was to implement a “swipe from side” gesture, which paged through items on the screen. Here is such a list, straight from my XBox:

You can see that the list has 358 games, and 5 of them are visible at the same time. To get through even half of this list, you would have to perform the swipe gesture 36 times! This is extremely slow, frustrating and physically exhausting.

The latest K4W SDK solves this issue by differentiating between an open and a closed hand. When you close your hand, you “grip” whatever’s underneath it (such as a scrollviewer), and then you can move your hand to scroll. If you release the scroller while your hand is still moving, the scrolling keeps happening – similarly to the inertia scroll you are already used to on touch devices. You can close your hand again at any time, and stop the inertia – again, as you expected.

In this post, I will help you to get started with Kinect Interactions by creating a Hello World WPF app that uses the features mentioned above. In later posts, I will dive deeper into the Kinect Interactions SDK, show you how to customize the existing controls, create new ones and go deep into the lower levels of the InteractionStream to invent new ways of interacting with your app. But first, let’s see how you can get started with Kinect Interactions today!

Note: I am assuming that you are familiar with C# and WPF for this article series.

First Things First – Download and Install the bits

Assuming you already have a Kinect for Windows device (or at least a Kinect for XBox one, which doesn’t support Near Mode, but can be legally used for development, but not “going live”), the next step is to install the SDK. Head over to Kinect Developer Downloads and download and install the SDK, and then the Kinect for Windows Developer Toolkit. After everything is done, launch the Kinect Developer Toolkit Browser from your start menu, and verify that the computer sees your Kinect by launching the “Kinect Explorer – WPF” sample. If all’s good, you can play a little with the samples called “Controls Basics – WPF” and “Interaction Gallery – WPF” to experience Kinect Interactions for yourself.

Create the Project

First, create a new WPF Application project called “GettingStarted”.

Add the following references:

From the C:\Program Files\Microsoft SDKs\Kinect\v1.7\Assemblies\ folder, add Microsoft.Kinect.dll.

From the C:\Program Files\Microsoft SDKs\Kinect\Developer Toolkit v1.7.0\Assemblies\ folder, add Kinect.Toolkit.dll, Kinect.Toolkit.Controls.dll and Kinect.Toolkit.Interaction.dll

Initializing the Kinect Sensor with KinectSensorChooser

The first control we will use from the Microsoft.Kinect.Toolkit.Controls namespace is the KinectSensorChooserUI. It is a fairly boring control, but it is nonetheless important: it indicates the status of the Attached Kinect Sensor, and can inform the user if the sensor is unplugged, plugged into the wrong USB port, etc.

In the MainView.xaml, add the following namespace:

xmlns:k=http://schemas.microsoft.com/kinect/2013

Then add the KinectSensorChooserUI control to the main grid:

<Grid>
    <k:KinectSensorChooserUI HorizontalAlignment="Center" VerticalAlignment="Top" Name="sensorChooserUi" />
</Grid>

We also need to initialize the KinectSensorChooser from the codebehind with some boilerplate code (I took these from the K4W samples).

Add a private field to the MainWindow codebehind:

private KinectSensorChooser sensorChooser;

Set up an event handler for the OnLoaded event in the MainWindow constructor (or in XAML, as you wish):

public MainWindow()
{
    InitializeComponent();
    Loaded += OnLoaded;
}

And create the OnLoaded event handler:

private void OnLoaded(object sender, RoutedEventArgs routedEventArgs) 
{ 
    this.sensorChooser = new KinectSensorChooser(); 
    this.sensorChooser.KinectChanged += SensorChooserOnKinectChanged; 
    this.sensorChooserUi.KinectSensorChooser = this.sensorChooser; 
    this.sensorChooser.Start(); 
}

If a sensor is chosen or got initialized, the code above will invoke the SensorChooserOnKinectChanged event handler. Let’s keep this one simple for now, just add a MessageBox to indicate things happened:

private void SensorChooserOnKinectChanged(object sender, KinectChangedEventArgs args) 
{ 
    MessageBox.Show(args.NewSensor == null ? "No Kinect" : args.NewSensor.Status.ToString()); 
}

Now we can run the app. First, let’s see what happens if there are no Kinects connected to the computer:

Also, sensorChooser.Kinect is null in this case.

If the mouse is moved over the little box above, it expands, and shows a more detailed help on the required action. Nice!

The help link takes you to a Microsoft web page about Kinect for Windows.

If you connect the Kinect Sensor now, the KinectSensorChooserUI will switch to an “Initializing” mode:

With the mouse hovering over it:

… and after a while, the Kinect sensor will be initialized, and the event handler with the MessageBox will get called:

After this, the Kinect chooser UI disappears until there’s a problem.

The KinectChooserUI control can also warn the user to move the Kinect to another USB port, check the power supply, etc. Here are the possible statuses:

public enum KinectStatus 
{ 
  Undefined, 
  Disconnected, 
  Connected, 
  Initializing, 
  Error, 
  NotPowered, 
  NotReady, 
  DeviceNotGenuine, 
  DeviceNotSupported, 
  InsufficientBandwidth, 
}

Setting up the Kinect for Interactions

Now that we made sure that we have a properly connected Kinect sensor, it’s time to get it to work with some more boilerplate code. Let’s replace the temporary SensorChooserOnKinectChanged event handler with the real stuff:

   1:  private void SensorChooserOnKinectChanged(object sender, KinectChangedEventArgs args)

   2:  {

   3:      bool error = false;

   4:      if (args.OldSensor != null)

   5:      {

   6:          try

   7:          {

   8:              args.OldSensor.DepthStream.Range = DepthRange.Default;

   9:              args.OldSensor.SkeletonStream.EnableTrackingInNearRange = false;

  10:              args.OldSensor.DepthStream.Disable();

  11:              args.OldSensor.SkeletonStream.Disable();

  12:          }

  13:          catch (InvalidOperationException)

  14:          {

  15:              // KinectSensor might enter an invalid state while enabling/disabling streams or stream features.

  16:              // E.g.: sensor might be abruptly unplugged.

  17:              error = true;

  18:          }

  19:      }

20:

  21:      if (args.NewSensor != null)

  22:      {

  23:          try

  24:          {

  25:              args.NewSensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);

  26:              args.NewSensor.SkeletonStream.Enable();

27:

  28:              try

  29:              {

  30:                  args.NewSensor.DepthStream.Range = DepthRange.Near;

  31:                  args.NewSensor.SkeletonStream.EnableTrackingInNearRange = true;

  32:                  args.NewSensor.SkeletonStream.TrackingMode = SkeletonTrackingMode.Seated;

  33:              }

  34:              catch (InvalidOperationException)

  35:              {

  36:                  // Non Kinect for Windows devices do not support Near mode, so reset back to default mode.

  37:                  args.NewSensor.DepthStream.Range = DepthRange.Default;

  38:                  args.NewSensor.SkeletonStream.EnableTrackingInNearRange = false;

  39:                  error = true;

  40:              }

  41:          }

  42:          catch (InvalidOperationException)

  43:          {

  44:              error = true;

  45:              // KinectSensor might enter an invalid state while enabling/disabling streams or stream features.

  46:              // E.g.: sensor might be abruptly unplugged.

  47:          }

  48:      }

  49:  }

This code handles properly Shutting down the old Kinect Sensor (you can have multiple sensors connected to the PC) and starts up the Depth- and Skeleton streams for the new one. In Lines 31 and 32, near mode and seated skeleton tracking are turned on – these are optional, but I found them useful for my little developer corner.

The Kinect Region

The last step before we can see something from Kinect Interactions is to add a KinectRegion to the XAML. KinectRegion is the key element of Kinect Interactions on WPF: it is the screen area where interactive elements are placed and can be manipulated. KinectRegion is also responsible for displaying and moving the hand cursor. An application can have multiple KinectRegions, but they cannot be nested. Each KinectRegion can have their own respective Kinect sensor.

Let’s add a KinectRegion to the main Grid of the MainWindow, thus making it stretch the entire window:

<k:KinectRegion Name="kinectRegion"> 
</k:KinectRegion>

Also, let’s make sure that the KinectSensor property of this KinectRegion is set to the actual Kinect by adding the following at the end of the SensorChooserOnKinectChanged event:

if (!error) 
    kinectRegion.KinectSensor = args.NewSensor;

If you run the app now, you get one more error:

“Unable to invoke library 'KinectInteraction170_32.dll'.”

This is a common problem with Kinect beginners. The dll mentioned can be a bit tricky to find. The simplest way is to install the Controls Basics – WPF source code from the Developer Toolkit Browser (press the Install button) – or just download the sample project for this article (as long as you are targeting the same, 1.7 Kinect SDK).

Go to the AnyCPU.Debug folder of the sample’s installation directory and copy the KinectInteraction170_32.dll and KinectInteraction170_64.dll files to your project’s output directory. Now the app should run. Note: you may have to wait as much as 20 seconds for the Kinect Sensor to initialize. During this process, your application will not respond and will show a wait cursor if you move the mouse over it.

If all is OK, you should be able to raise one of your hands in front of you, and see a Hand Cursor representing it in your Main Window. You may have to wiggle around a bit so that the Kinect Sensor can detect you as a user.

For the screenshot above, I have also launched Kinect Studio. The Depth Viewer and Kinect Studio windows are not part of our application. Kinect Studio helps monitor the data our application receives. It can also record the Kinect’s output and allows you to play back the same user actions later. You can learn more about Kinect Studio on MSDN.

User Viewer

The next control we discuss is the Microsoft.Kinect.Toolkit.Controls.KinectUserViewer. Let’s add it to the main Grid:

<k:KinectUserViewer VerticalAlignment="Top" HorizontalAlignment="Center"

k:KinectRegion.KinectRegion="{Binding ElementName=kinectRegion}" Height="100" />

The KinectUserViewer displays the depth pixels that belong to an identified user or users. You can use the KinectUserViewer control to make sure that the Kinect SDK “sees” you as a human and is able to separate you from the surroundings. If you see nothing in the KinectUserViewer (after the Kinect initialization has finished), it means that the Kinect is not tracking you, it hasn’t identified you as a person yet. Try to move around, and remember that you need to be at least a meter (or even more) away from the sensor for it to detect you.

Interactive Controls

So far we managed to notify the user of the Kinect Sensor’s state, put a hand cursor on the screen that followed the user’s hand and to show an outline of the user. It’s time for the hand to actually do something! The first such something is going to be pressing a button.

The KinectTileButton

The KinectTileButton is one of the simplest controls available. It resembles a Windows 8-style tile with a template-able label and a Content that is just like the content of the regular Button control. Here is a very simple declaration of the KinectTileButton that should be put inside the main Grid of MainWindow.xaml:

<k:KinectTileButton Label="Press me!" Click="ButtonOnClick"></k:KinectTileButton>

To handle the button’s Click event, let’s just do a simple MessageBox:

private void ButtonOnClick(object sender, RoutedEventArgs e) 
{ 
    MessageBox.Show("Well done!"); 
}

The KinectButton control has a nice Kinect-purple color by default. Here’s what it looks like: (Note: I’ve changed window height and width to 600x500 for more space)

If you move the hand cursor over the KinectButton it grows a little to indicate the “hover” state. The hand cursor also changes subtly.

If you start to move your hand towards the screen (attempting to perform the “press” gesture), the hand will begin to fill up with a purple circular pattern.

When you actually press the button, the button will play some visual effects, and the cursor’s fill color will change again. Then, our ugly little MessageBox will show up, indicating that the button has indeed been pressed, and the proper event handler was called. Nice!

The KinectCircleButton

The next button type works a lot like the KinectTileButton. However, in order to show it, we first need to change the existing xaml a little bit. The KinectRegion is a ContentControl, which means it can only have one child. Luckily, this Child can be a control that in turn can contain multiple children – such as a Grid or a StackPanel. For now, we will need to surround the existing KinectTileButton with a Grid. I’ve also moved the KinectTileButton to the left side to leave some space for the KinectCircleButton we are about to add. The button has a content (which, again can be anything, including an image or a much more complex element), and a Label. The KinectCircleButton resembles the RoundButton of Windows Phone. Let’s add it to the Grid we just added inside the Kinectregion:

<k:KinectCircleButton Label="Circle" HorizontalAlignment="Right" Height="200" Click="ButtonOnClick">Hi</k:KinectCircleButton>

This is what the application looks like now:

Moving the hand above the KinectCircleButton, we can see that it also has an “over” state, and changes the look of the hand just like the KinectTileButton did.

Moving the hand towards the screen:

… and finally pressing it:

It is good to know that these interactive elements also work with the mouse. This way you can test the app itself without using Kinect. Kinect initialization can takes 15 seconds or more, so to test your app’s logic, UI and flow, you may even want to disconnect the Kinect to get the app started faster.

They See Me Scrollin’

As I said in the Introduction, browsing a big list of items and selecting the one you want has always been difficult with Kinect – but with the latest Kinect SDK, this situation is vastly improved. The key is the third (and last) out-of-the-box control of the 1.7 SDK: the KinectScrollViewer. Let’s place a KinectScrollViewer at the bottom of our window by pasting the code below into the Grid inside the KinectRegion:

<k:KinectScrollViewer VerticalScrollBarVisibility="Disabled" HorizontalScrollBarVisibility="Auto" VerticalAlignment="Bottom">
   <StackPanel Orientation="Horizontal" Name="scrollContent" />
</k:KinectScrollViewer>

This looks exactly like a standard WPF ScrollViewer. It even has a StackPanel called “scrollContent”, and these two controls are set up to scroll horizontally. Let’s fill up the StackPanel with some KinectCircleButtons. For this, add the following lines to the OnLoaded event handler in the code behind:

//fill scroll content
for (int i = 1; i < 20; i++)
{
    var button = new KinectCircleButton
                        {
                            Content = i,
                            Height = 200
                        };

    int i1 = i;
    button.Click +=
        (o, args) => MessageBox.Show("You clicked button #" + i1);

    scrollContent.Children.Add(button);
}

If we run the application now, this is what it looks like:

Moving the hand over the scrollable area, it changes color:

The trick now is to close your hand and make a fist, as if you were grabbing something. The hand cursor will change, indicating that you indeed grabbed onto the KinectScrollViewer:

Now you can move your hand, and the KinectScrollViewer will “stick to it”, much like when you scroll a list on a touch device, such as a phone or a tablet. If you open your hand, you will release the KinectScrollViewer. If you push your hand forward, you can select easily one of the items in the list and get the corresponding MessageBox:

Download

You can download the demo project for this article here.

Summary

In this post, I’ve introduced you to the basics to get started with a Kinect Interactions application – getting the Kinect sensor up and running, defining a KinectRegion, adding buttons and scrollable areas to it.

In the next posts of this series, I will dive deeper into the world of Kinect Interactions. We will dive deeper into the – unfortunately very sparsely documented – controls I discussed above, and see the customization options they offer. And finally, going deeper into the rabbit hole, we will examine the low level Interaction Stream.

Windows Phone Performance – Part 8 of many: Lumia 620 performance

2013-03-16T20:43:58Z

In this post, I’ll take a look at the performance of the Nokia Lumia 620 from a developer’s point of view. This post is part of my Windows Phone Performance series.

HUGE thanks to Joost von Schaik for performing the tests in this article. Since I don’t have access to a Lumia 620, without him, this post would not have been possible.

The Nokia Lumia 620

The 620 is the almost-smallest member of Nokia’s Windows Phone 8 family yet. The 520 is smaller, but not yet available. The 620 is a great phone, and provides a good Windows Phone 8 experience for a fraction of the price of its bigger brothers, such as the 820 and the 920 (both of which we have tested earlier).

The 620 has a dual core Snapdragon S4 processor. This is not the same as the ones found in the 920 (MSM8960T) and the 820 (MSM8960) though (both of which run at 1.5GHz). The 620 has an MSM8227, running at 1 GHz, and an Adreno 305 GPU. The screen has 800x480 pixels (same as the 820, but lower than the 920). Another notable difference is that the 620 has 512 MBytes of RAM onboard. While this may affect multitasking and other end-user features, our tests were not impacted by the lower amount of memory.

Because of its low price and attractive design, the 620 is expected to sell very well in countries such as India, Chine and Eastern Europe (just like the 610 did).

Application Loading / CPU Performance

For this test, I am using the same test I used for the other phones, from the second part of this series: It’s Full of Stars!

Number of Stars	Lumia 620	Optimus 7	Lumia 800	Lumia 920	Lumia 820
100	310	399	347	164	497
1000	727	1133	846	449	539
2000	1119	1882	1357	711	723
5000	2425	4258	3007	1557	1465
10000	4569	8348	5792	2913	2899
15000	6887	12183	9057	4404	4349
20000	9206		11646	5903	5774
25000	11205			7328	7283
30000	13235			8861	8756
35000	15546			10314	10131
40000	17929			11677	11589

We get the expected result – the CPU clock difference between the other WP8 phones (920 and 820) clearly means slower startup and computational times. In fact, if we compensate for the CPU clock, the numbers are pretty much exactly the same. The 620 has no problem beating second generation Windows Phone 7 devices (such as the Lumia 800). The 800 has a single-core and outdated CPU, but runs at a 40% higher speed of 1.4 GHz. If we compare the 620 to the first generation WP7 devices, such as the LG Optimus 7, we get almost 100% performance gain. Moore’s Law in action here…

GPU Performance

To measure the raw CPU performance, once again I used the “Fill Rate Test Sample” from the MSDN article Performance Considerations in Applications for Windows Phone. This test turned out some surprising results!

Here are the numbers in table and chart format:

Rectangles	Fill Rate	Lumia 620 FPS	Lumia 800 FPS	Optimus 7 FPS	Lumia 920 FPS	Lumia 820 FPS
0	0	60	59	50	59	59
1	1.03	60	59	50	59	59
5	1.53	60	59	50	59	59
10	2.15	60	59	50	59	59
15	2.78	60	59	48	59	59
20	3.47	60	59	38	59	59
25	4.02	60	59	34	59	59
30	4.65	60	59	30	59	59
35	5.28	60	59	26	55	59
40	5.9	60	59	23	50	59
45	6.53	60	59	21	45	59
50	7.15	60	59	19	42	59
55	7.78	60	53	17	40	59
60	8.4	60	49	16	36	59
65	0.03	60	45	15	33	59
70	9.65	60	43	14	31	59
75	10.28	60	42	14	30	59
80	10.9	60	40	13	29	59
85	11.53	60	37	12	26	59
90	12.15	58	34	11	25	57
95	12.775	53	33	11	24	54
100	13.4	49	32	10	23	51
105	14.03	45	30	10	22	48
110	14.65	44	30	9	21	47
120	15.9	40	27	9	20	44
130	17.15	40	26	8	18	41
140	18.4	40	24	7	17	39
150	19.65	36	23	7	16	36
170	22.15	33
200	25.78	29

Barring some measurement inaccuracies, the 620 performed just as good as our current champion, the 820 – even though the SoC runs at 66% speed! I’ve tried to find a reason for this difference, but I couldn’t find a definite answer. It seems like the bottleneck is elsewhere – perhaps the memory bandwidth, the graphics driver or some other architectural component. If you have an idea of why the 620 performs this much better than expected, please share it in the comments.

Nevertheless, it seems like developers we won’t have to worry about Fill Rates with the 620. It keeps an impressive 60 fps up until the Fill Rate of 12, which should be more than enough for any application. Even at a Fill Rate of 24, the frame rate is still above the red zone of 30.

Summary

From a developer’s point of view, the 620 is noticeably slower than the other WP8 devices when it comes to raw computing power. However, this speed difference shouldn’t cause much problems if you are sticking with the performance guidelines outlined in the previous parts of this series. Even better, when it comes to Fill Rate, you have nothing to worry about. For Fill Rate optimization, you should still use the Lumia 920. If you are also targeting WP7 devices, the Lumia 610 should be your baseline. The first generation phones are finally becoming rare enough to ignore minor FPS problems on. Which is good news, since it’s often extremely hard to keep the Fill Rate above their FPS drop point of 2.5.

DataContextChanged event for WinRT

2013-01-23T10:22:41Z

Unlike WPF, WinRT (or Windows Store App API) does not have a DataContextChanged event handler. This may cause problems if you are doing framework-y stuff, or even for some controls, where you are .

Luckily, Jeremy Likness wrote a solution for the exact same problem for Silverlight, back in 2009. Silverlight now has the DataContextChanged event implemented, but his technique is still useful for WinRT. There is a small change to do though: the way you perform callback from an attached property is a tiny bit different.

So, here is the complete code that compiles and works perfectly on WinRT. Again, all credit goes to Jeremy for this solution!

public static class DataContextChangedHelper<T> where T : FrameworkElement, IDataContextChangedHandler<T>
{
    private const string INTERNAL_CONTEXT = "InternalDataContext";

    public static readonly DependencyProperty InternalDataContextProperty =
        DependencyProperty.Register(INTERNAL_CONTEXT,
                                    typeof(Object),
                                    typeof(T),
                                    new PropertyMetadata(null, _DataContextChanged));

    private static void _DataContextChanged(object sender, DependencyPropertyChangedEventArgs e)
    {
        T control = (T)sender;
        control.DataContextChanged(control, e);
    }

    public static void Bind(T control)
    {
        control.SetBinding(InternalDataContextProperty, new Binding());
    }
}

public interface IDataContextChangedHandler<T> where T : FrameworkElement
{
    void DataContextChanged(T sender, DependencyPropertyChangedEventArgs e);
}

To actually have the DataContextChanged event fire for your control (FrameworkElement), you have to add one more line in your control’s constructor:

DataContextChangedHelper<MyControl>.Bind(this);

That’s it!

Windows Phone Performance – Part 7 of many: Lumia 820 performance

2013-01-10T11:17:54Z

In this post, I’ll take a look at the performance of the Nokia Lumia 820 from a developer’s point of view. This post is part of my Windows Phone performance series.

HUGE thanks to Alex Sorokoletov for performing the tests in this article. Since I didn’t have access to a Lumia 820, without him, this post would not have been possible.

The Nokia Lumia 820

The little brother of the flagship Nokia Lumia 920, the 820 has nothing to be ashamed of when it comes to performance. Actually, the 820 has the same dual-core Snapdragon S4 processor as the 920, running at the same 1.5GHz speed. From a developer’s perspective, the big difference is the screen – while the 920 has 1280x768 pixels, the 820 keeps the modest 800x480 pixel resolution of Windows Phone 7. Most of the differences between the 920 and the 820 have to do with how the 920 has to work with 2.5 as much pixels as the 820.

It is worth noting that the ~~820 only has~~ minimum requirement for 800x480 Windows Phone devices is 512MByte RAM, while the 920 has twice as much. This also boils down to the difference in screen pixels – apps on the the 920 need more RAM simply to decode the high resolution bitmaps and keep them in memory.
Update: the Lumia 820 has 1G RAM

Let’s see how the 820 performs in our little tests!

Application Loading / CPU performance

For this one, I am using the old artificial test from the second part of this series (It’s Full of Stars!).

Number of Stars	Lumia 610	Optimus 7	Lumia 800	Lumia 920	Lumia 820
100	624	399	347	164	497
1000	1325	1133	846	449	539
2000	2106	1882	1357	711	723
5000	4303	4258	3007	1557	1465
10000	8430	8348	5792	2913	2899
15000	12401	12183	9057	4404	4349
20000	17205		11646	5903	5774
25000				7328	7283
30000				8861	8756
35000				10314	10131
40000				11677	11589

With Compile in the Cloud being a general Windows Phone 8 feature, the 820 should start applications just as fast as the 920. The first two launches show a bit different picture though, the reasons of which is not entirely clear. Alex performed multiple measurements for each pass, but the 100 star test consistently returned a slower startup time. Regardless of this difference, you can see that the 820 performed roughly as well as the 920 did (almost completely hiding the 920’s line in the chart), even though the later one had to render much more physical pixels.

GPU Performance

To measure the raw CPU performance, once again I used the “Fill Rate Test Sample” from the MSDN article Performance Considerations in Applications for Windows Phone. Here are the numbers in table and chart format:

Rectangles	Fill Rate	Lumia 610 FPS	Lumia 800 FPS	Optimus 7 FPS	Lumia 920 FPS	Lumia 820 FPS
0	0.9	59	59	50	59	59
1	1.05	59	59	50	59	59
5	1.56	59	59	50	59	59
10	2.19	59	59	50	59	59
15	2.83	59	59	48	59	59
20	3.47	59	59	38	59	59
25	4.1	54	59	34	59	59
30	4.74	48	59	30	59	59
35	5.37	45	59	26	55	59
40	6.01	41	59	23	50	59
45	6.65	38	59	21	45	59
50	7.28	35	59	19	42	59
55	7.92	32	53	17	40	59
60	8.56	29	49	16	36	59
65	9.19	28	45	15	33	59
70	9.83	27	43	14	31	59
75	10.46	25	42	14	30	59
80	11.1	24	40	13	29	59
85	11.74	22	37	12	26	59
90	12.37	21	34	11	25	57
95	13	20	33	11	24	54
100	13.6	19	32	10	23	51
105	14.28	18	30	10	22	48
110	14.92	17	30	9	21	47
120	16.19	16	27	9	20	44
130	17.46	15	26	8	18	41
140	18.73	14	24	7	17	39
150	20	13	23	7	16	36

The Lumia 820 is clearly the top performer here – no wonder, since it has much less pixels to work with than the 920. The 820 keeps a solid 60 FPS up until a Fill Rate of 12, and still doesn’t drop below an acceptable 30 with a Fill Rate of 150. This really shows how the Snapdragon S4’s GPU may even be a bit too powerful for the phone (at least when it comes to Silverlight applications). It can compose 2.5 – 3 times as many full-screen layers as on the 920 before the first signs of lags even show. Not surprisingly, this is the same multiplier as the number of pixels there are between the two phones. Fill Rate is about number of pixels after all.

Summary

In CPU, the two Windows Phone 8 Lumia phones perform similarly. However, when it comes to the GPU, the 820 feels like a race car engine in a common vehicle.

When developing Windows Phone 8 applications and testing on the Lumia 820, you have to keep in mind that high-resolution phones do not tolerate Fill Rates above 4.5 or so. Even though your applications may keep a consistently smooth animation at a Fill Rate of 6 on the 820, other WP phones will struggle. So, keep your Frame Rate Counters turned on. Fortunately, it is not hard to keep the Fill Rate below 5 – we are in a much better situation than what we had with the first generation Windows Phones, where the a value of 2.5 already caused problems, and a lot of time went into making sure we kept the Fill Rate below that level.

Windows Phone performance – Part 6 of many: Nokia Lumia 920 performance

2012-12-27T12:37:17Z

In this post, I’ll take a look at the performance of the Nokia Lumia 920 from a developer’s point of view. This post is part of my Windows Phone performance series.

The Nokia Lumia 920

As Nokia’s flagship Windows Phone device, the Lumia 920 is a beast. Both in size and performance, the phone is great. In fact, the very first thing I noticed as a Windows Phone 7 user (apart from the new Start Screen) was how quickly the applications launched and how well they performed.

The 920 has a screen resolution of 1280x768 (983,040 pixels). This is more than 2.5 times as many as the standard resolution of the Windows Phone 7 devices – 800x480 (384,000 pixels). Other Windows Phone 8 devices can have a resolution of 1280x720 (almost the same as the 920, but a different aspect ratio) or 800x480. The additional pixel count requires a faster CPU to render the textures and a faster GPU to move the textures around. Fortunately, both of these are taken care of with the dual-core 1.5GHz Snapdragon S4 SOC. Also, since images in your application should contain a bigger number of pixels to remain sharp, the screen pixels mean that more memory is used by the applications. Because of this, you won’t see an 1280p WIndows Phone 8 device with less than 1G RAM.

In the previous post of this series, I compared the LG Optimus 7 (a first generation WP7 device), the Nokia Lumia 800 (a second generation WP7 device) and the Lumia 610 (a low-end second generation phone). Let’s add the 920 to the mix, and see how it fares!

Application Loading / CPU performance

With Windows Phone 7, application load times were the result of two factors – the Just-In-Time Compilation of the dlls, and the calculations needed to display the first screen of your app. With the apps growing bigger as more and bigger dlls were included in the XAP, whatever you did, a lot of them had to be JIT-ted just to get the first line of your actual code executed.

Things have dramatically changed with Windows Phone 8. Thanks to the faster, dual-core processor, but most importantly, to the cloud compilation infrastructure, Silverlight apps can now start up almost as fast as the first party native apps. Cloud Compilation happens when you deploy your app into the Store, and makes 99% of the work of the Just-in-time Compiler unnecessary. With Windows Phone 7, application load times were mostly the result of the apps growing bigger as more and bigger dlls were included in the XAP. Whatever you did, a lot of them had to be JIT-ted just to get the first screen up. You can learn more details about the Cloud Compilation in the Build talk “Deep Dive into the Kernel of .NET on Windows Phone 8”.

The magic of Compile in the Cloud – rowi startup time vs the People Hub.

With the JIT compilation out of the way, let’s see how the phone deals with a lot of calculations at startup. For this, I will summon my old artificial test from the second part of this series (It’s Full of Stars!). Please click on the link to refresh your memory, I’ll wait here.

Let’s look at the numbers. The first column of the table shows how many stars (small ellipses) are drawn, and the other columns show the number of milliseconds it took to launch the application. The stars are drawn on the UI thread at startup, so this drawing delays the startup time of the app.

Number of Stars	Lumia 610	Optimus 7	Lumia 800	Lumia 920
100	624	399	347	164
1000	1325	1133	846	449
2000	2106	1882	1357	711
5000	4303	4258	3007	1557
10000	8430	8348	5792	2913
15000	12401	12183	9057	4404
20000	17205		11646	5903
25000				7328
30000				8861
35000				10314
40000				11677

Here is it in a chart form:

As you can see, the 920 kicks the rear end of every WP7 phone – even the fastest Lumia 800 is beat by a factor of 2, despite the much larger number of pixels to deal with. Not bad!

You will also experience this difference when navigating between pages of the application. The layout and render cycle is much faster, resulting in a better navigation experience.

GPU Performance

To measure the raw CPU performance, I used the “Fill Rate Test Sample” from the MSDN article Performance Considerations in Applications for Windows Phone. This does not test things like shaders, perspective transformations, etc – but it is a good first indicator. I gradually increased the number of rectangles on the screen, which in turn increased the fill rate. As we saw in part 4 of this series, The Power of the GPU, Fill Rate is an important measurement for how much work the GPU has to do, and a high Fill Rate makes the FPS (frame per seconds) drop. Without further ado, here are the results in table and chart format:

Rectangles	Fill Rate	Lumia 610 FPS	Lumia 800 FPS	Optimus 7 FPS	Lumia 920 FPS
0	0.9	59	59	50	59
1	1.05	59	59	50	59
5	1.56	59	59	50	59
10	2.19	59	59	50	59
15	2.83	59	59	48	59
20	3.47	59	59	38	59
25	4.1	54	59	34	59
30	4.74	48	59	30	59
35	5.37	45	59	26	55
40	6.01	41	59	23	50
45	6.65	38	59	21	45
50	7.28	35	59	19	42
55	7.92	32	53	17	40
60	8.56	29	49	16	36
65	9.19	28	45	15	33
70	9.83	27	43	14	31
75	10.46	25	42	14	30
80	11.1	24	40	13	29
85	11.74	22	37	12	26
90	12.37	21	34	11	25
95	13	20	33	11	24
100	13.6	19	32	10	23
105	14.28	18	30	10	22
110	14.92	17	30	9	21
120	16.19	16	27	9	20
130	17.46	15	26	8	18
140	18.73	14	24	7	17
150	20	13	23	7	16

In this test, the Lumia 800 is still better than the 920. The reason for that is the sheer number of pixels the GPU has to deal with – the 920 has 2.5 times as many pixels as the Lumia 800. On the 920, the frame rates start to drop around a fill rate of 5, while the 800 can still push pixels at 60 fps up until a fill rate of 7.5. So, what does it mean for your application? It means that as long as you have less than 5 (screen-sized) layers in your app (e.g. background picture, darkening layer, lots of text over other images, etc), you will be fine, the scrolling will not become jumpy. Frankly, it is hard to think how one would exhaust the Fill Rate of 4 in a normal Silverlight application. The low-end 610 can handle a Fill Rate of 4 easily, only the first generation devices will have problems with it.

Summary

The Lumia 920 launches apps and navigates between screens much faster, and performs as smooth as we became used to with Windows Phone – despite the much higher pixel count. Overall, applications feel much more snappier, and finally performance parity between native apps and Silverlight apps can be acheved. I expect similar performance from the other HD phones, such as the HTC 8X or the Samsung Ativ. 800x480 devices may perform even better in the Fill Rate test, but I couldn’t measure it – if I get my hands on such a phone, I will write another blog post.

Adding and Editing WinRT Behaviors with Blend

2012-09-14T09:06:00Z

I am a huuuuuuge fan of Behaviors (and Actions and Triggers) for Silverlight and WPF. I have been very disappointed when WinRT turned out not to support them.

Luckily, Joost van Schaik created WinRTBehaviors on CodePlex (also available on NuGet). But his solution still lacks any Blend support. I have not been able to reproduce the full Blend behavior editing experience, but I managed to take the first baby-step that at least saves you from creating Behaviors from code.

Here is how to use the updated WinRTBehaviors with Blend.

First, let’s create a simple Behavior for illustration purposes. It can be attached to a Rectangle, and changes the Rectangle’s color when it is pressed. Told you it was simple

using WinRtBehaviors;
using Windows.UI;
using Windows.UI.Xaml;
using Windows.UI.Xaml.Media;
using Windows.UI.Xaml.Shapes;

namespace WinRTBehaviorTest
{
    public class ChangeColorBehavior : Behavior<Rectangle>
    {
        protected override void OnAttached()
        {
            base.OnAttached();
            AssociatedObject.PointerPressed += (sender, args) => AssociatedObject.Fill = new SolidColorBrush(PressedColor);
        }

        public Color PressedColor
        {
            get { return (Color)GetValue(PressedColorProperty); }
            set { SetValue(PressedColorProperty, value); }
        }

        // Using a DependencyProperty as the backing store for PressedColor.  This enables animation, styling, binding, etc...
        public static readonly DependencyProperty PressedColorProperty =
            DependencyProperty.Register("PressedColor", typeof(Color), typeof(ChangeColorBehavior), 
            new PropertyMetadata(Colors.Red));
    }
}

Now, let’s create a simple Metro Windows Store app in Blend, and add a Rectangle:

To add our behavior, first select the Rectangle, then look for the Interaction.Behaviors attached property in the Properties panel:

Press the … button:

Select <Other Type> from the Drop down:

Notice that Blend has already filtered all the types in our application and shown as those that inherit from Behavior<T>. You may want to do a search here, but we don’t need that since this is a simple project. Select ChangeColorBehavior here, and press OK, then press the Add button.

Our behavior is now listed and editable! However, since the Behavior class is a descendant of the FrameWorkElement class (in order to allow bindings), there are a lot of unused properties here. I’ll try to address this issue later, but for now, let’s be happy that the PressedColor property is visible, and also editable:

When you’re done, press OK, and let’s see how the application runs.

Then press on the rectangle:

Yaay!

By the way, thanks to the shared XAML visual editor in Visual Studio 2012 and Blend 5, the above also works in Visual Studio:

I’ll dig deeper into Behaviors and Blend and I’ll try to make them even easier to use for a non-developer without touching XAML. This is just the first step, stay tuned!

Note: the above code has already been checked in to CodeProject. Next, Joost will release a new version and update the NuGet package. Until that happens, you have to download the source code of WinRTBehaviors. Update: Joost has updated the NuGet package, all cool

Using NUnit to test Windows Phone 7 applications Part 3: ViewModels and MVVM Light Toolkit

2012-08-17T00:40:59Z

In the previous posts of this series, we managed to get NUnit up and running to test WP7 code outside of the Windows Phone Emulator, in test runners such as NUnit, ReSharper and NCrunch, but paid the price by running into assembly load issues. We created a TestBase class to be the base class for our classes, and added convenience functions to it such as setting a global General.IsTesting flag, comparing arrays and verifying whether INotifyPropertyCollection calls have been performed properly.

In this post, I am going to concentrate on testing ViewModels that use Laurent Bugnion’s MVVM Light Toolkit.

Adding MVVM Light to the Project

Let’s start by adding MVVM Light from Nuget (I am using only the libraries for now):

Now we can change our Calculator class to inherit from ViewModelBase. Which also means that we can get rid of our existing PropertyChanged event and its RaisePropertyChanged invoker method as well, since they are implemented in one of our base classes.

Note: to compile the project, you’ll also have to add the MVVM Light dlls to the Test project.

All our tests still pass, proving that MVVM Light’s RaisePropertyChanged method works just like ours did. Phew.

Verifying Design Mode

The ViewModelBase class in MVVM Light has a very useful IsInDesignMode property that allows us to determine if the code is running in a design tool, and do things like creating fake test data for the designer to use instead of empty TextBoxes and Lists. This property can also be used to make sure that you are not trying to access a database from within Blend by making sure you only connect to the database or cloud service when the app is running “for real”.

Unfortunately, as soon as you access the IsInDesignMode property, the unit tests crash:

This looks awfully like the crash we experienced in the previous part with ObservableCollections. The actual reason is that the setter of this property accesses the DesignerProperties.IsInDesignTool property, which causes an assembly load failure. The solution is pretty similar: we have to make sure the IsInDesignMode never gets called. We can do this by first checking for our General.IsTesting property:

public Calculator()
{
    if (!General.IsTesting && IsInDesignMode)
    {
        //do design time setup

    }
    else
    {
        //do normal initialization
    }
}

And the tests pass again. Alternatively, if Laurent provides us with a way to set the IsInDesignMode property from our tests, we could simply set that and the code would never have to access DesignerProperties.IsInDesignTool.

Verifying if Message is sent

Messaging has to be one the my favorite features of MVVM Light Toolkit. It allows loose coupling of ViewModels, sometimes even Views without having to worry about memory leaks.

When testing, you probably want to make sure that your ViewModel has sent (or hasn’t sent) a Message. Luckily, Laurent made the Messaging infrastructure extensible for exactly this reason. We can create a MessengerSpy class, which logs every message that has ever been sent through the Messenger.

public class MessengerSpy : Messenger
{
    public List<object> SentMessages = new List<object>();

    public bool HasSentMessage<T>()
    {
        return SentMessages.Any(o => o is T);
    }


    public override void Send<TMessage>(TMessage message)
    {
        SentMessages.Add(message);
    } 
}

Note that the message doesn’t actually get sent, only stored. The reason is that we would run into another one of those pesky assembly load failures.

In the TestBaseSetup method, we use this spy class and set it up so that a new spy is created for every test:

//set up message logging
messengerSpy= new MessengerSpy();
Messenger.OverrideDefault(_msngr);

So, how do we use this? Suppose we want to send a message whenever X changes in the Calculator ViewModel. We have an XChangedMessage for this purpose:

public class XChangedMessage
{
    public int NewX { get; set; }
}

Since TestBase.messengerSpy is declared protected, it can be accessed from our tests. To make sure the message has been sent, we can simply use the HasSentMessage method:

[Test]
public void XChangedMessageShouldBeSentAfterXChanges()
{
    _calc.X = 10;
    Assert.IsTrue(base.messengerSpy.HasSentMessage<XChangedMessage>());
}

This fails until we append the following line to the setter of the Calculator.X property:

Messenger.Default.Send(new XChangedMessage());

If we want to go one step further and verify that the NewX value is set properly, then we can do the following:

[Test]
public void XChangedNewXShouldBeSentAfterXChanges()
{
    _calc.X = 10;
    Assert.IsTrue(messengerSpy.SentMessages.Any(m => ((XChangedMessage)m).NewX == 10));
}

And of course, to make this one pass, we have to set the value of NewX in the X setter when the message is created:

Messenger.Default.Send(new XChangedMessage {NewX = 10});

Testing RelayCommands

Fortunately, testing RelayCommands is straightforward. Let’s create a ResetXToZero command (which should not be executable when X is already 0) like this:

 ResetXToZero = new RelayCommand(() => X = 0, () => X != 0);

We can easily test it using the funky Sequential test attribute for zero, positive and negative X values:

[Test, Sequential]
public void VerifyExecuteResetXCanExecute(
    [Values(0, 1, -1)] int x,
    [Values(false, true, true)] bool expected
    )
{
    _calc.X = x;
    Assert.AreEqual(expected, _calc.ResetXToZero.CanExecute(null));
}

Summary

This concludes my short blog series on using NUnit to test Windows Phone applications. As you can see, the method outlined here has quite a few issues, especially when you want to load assemblies beyond the basics. However, this technique was extremely useful for me during our last project, where complex calculations had to be tested that did not rely on any platform-specific functionality – but still took advantage of the Math functions and a huge amount of Linq expressions, IQueriables, etc.

Source Code

You can download the source code for this little project here.

Using NUnit to test Windows Phone 7 applications – Part 2

2012-08-12T16:11:44Z

In the previous post of this series, we managed to get NUnit up and running to test WP7 code outside the Windows Phone emulator, in test runners such as NUnit, Resharper or Ncrunch. In this post, we pay the price for that as we move on to more complicated tests. I’ll focus on ObservableCollections and the INotifyPropertyChanged in this post.

ObservableCollection gotchas

Suppose we want to extend our calculator with a simple function that returns all positive integers until a given number. We start implementing the tests:

[Test]
public void PositivesUntil0ShouldBeEmpty()
{
    _calc.X = 0;
    Assert.AreEqual(0, _calc.PositiveIntsUntilX.Count);
}

Of course, this doesn’t compile yet, we need the X and the PositiveIntsUntilX properties. Since later we want the Calculator to act as a ViewModel, and bind directly to it, the PositiveIntsUntilX should be an ObservableCollection, to be recalculated whenever X changes. But we’re not there yet, just trying to do the simplest thing that will make the test pass. This should be it:

public int X { get; set; }

private ObservableCollection<int> _positivesUntilX = new ObservableCollection<int>();

public ObservableCollection<int> PositiveIntsUntilX
{
    get { return _positivesUntilX; }
}

And the test fails! More interestingly, it fails when the ObservableCollection is being created. But why?

The error message says: SetUp : System.IO.FileNotFoundException : Could not load file or assembly 'System.Windows, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e' or one of its dependencies. The system cannot find the file specified.

The reason for this error is that the unit test runner cannot load the Windows Phone System.Windows dll, and thus throws errors as soon as that dll is referenced.

Now, you may want to follow Andriy Buday’s hints, and create a copy of your project, linking files to it, and use the real Silverlight runtime at this point. I will examine a different approach – less powerful, but simpler.

NUnit is confused since it cannot load the dll hosting the ObservableCollection class. But do we really need the ObservableCollection just to make sure the list has the right items? Can’t we just use a simple List<int> object here for testing, and an ObservableCollection in production?

Turns out, we can. Both ObservableCollection and List implement the IList interface. So, let’s change our code:

private IList<int> _positivesUntilX = General.IsTesting ? (IList<int>) new List<int>() : new ObservableCollection<int>();

public IList<int> PositiveIntsUntilX
{
     get { return _positivesUntilX; }
}

We will also need to create a new global class called General, that lets our code know if it is running in a test environment. This is not nice, and smells, but we’ll take care of that later.

public class General
{
    public static bool IsTesting = false;
}

The Windows Phone Silverlight runtime will still update the ListBox bound to PositiveIntsUntilX, since it detects that there is an ObservableCollection (more precisely, an object implementing the INotifyCollectionChanged interface) behind the ILIst property. And our unit tests never need to create an ObservableCollection, so they pass.

Comparing Lists

Next, we want to make sure that the PositiveIntsUntilX list has the right elements for different X values. No special WP7 issues here, but I thought it would be nice to have this here as well… It is useful to have a CompareLists<T> method for these tests. I put the CompareLists method in a TestBase class (which will also be useful later):

public class TestBase
{
    public void CompareLists<T>(IList<T> expected, IList<T> actual)
    {
        Assert.AreEqual(expected.Count, actual.Count);
        for (int i = 0; i < expected.Count; i++)
            Assert.AreEqual(expected[i], actual[i]);
    }
}

Here is the test:

[Test]
public void VerifyPositivesUntil11()
{
    _calc.X = 11;
    CompareLists(new List<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}, _calc.PositiveIntsUntilX);
}

And of course, we have to change the implementations of our Calculator to make the test pass:

public int X
{
    get { return _x; }
    set
    {
        _x = value;
        CalculatePositivesUntilX();
    }
}

private IList<int> _positivesUntilX = General.IsTesting ? (IList<int>) new List<int>() : new ObservableCollection<int>();
private int _x;

public IList<int> PositiveIntsUntilX
{
    get { return _positivesUntilX; }
}

private void CalculatePositivesUntilX()
{
    _positivesUntilX.Clear();
    for (int i = 1; i <= X; i++)
        _positivesUntilX.Add(i);
}

All green!

Verifying that INotifypropertyChanged gets called

Most of the time you will have a central, proven mechanism for handling INotifyPropertyChanged in your properties. But sometimes you want to make sure in your tests that these actually get called at the right time. In our little example, we want to make sure that after setting X, the PropertyChanged even is raised for X, but not or PostitiveIntsUntilX – we went through all that trouble making it an ObservableCollection after all…

Here is the PropertyChanged notification watcher for the TestBase:

[SetUp]
public void TestBaseSetup()
{
    General.IsTesting = true;
    if (_propertyNotifications != null)
        foreach (var key in _propertyNotifications.Keys)
            StopMonitoringINotifyPropertyChanged(key);

    _propertyNotifications = new Dictionary<INotifyPropertyChanged, List<string>>();
}

protected Dictionary<INotifyPropertyChanged, List<string>> _propertyNotifications;

public void MonitorINotifyPropertyChanged(INotifyPropertyChanged vm)
{
    vm.PropertyChanged += OnVMOnPropertyChanged;
}

public void StopMonitoringINotifyPropertyChanged(INotifyPropertyChanged vm)
{
    vm.PropertyChanged -= OnVMOnPropertyChanged;
}

private void OnVMOnPropertyChanged(object sender, PropertyChangedEventArgs args)
{
    var key = (INotifyPropertyChanged) sender;
    if (!_propertyNotifications.ContainsKey(key))
        _propertyNotifications.Add(key, new List<string>());

    _propertyNotifications[key].Add(args.PropertyName);
}

public bool HasPropertyChangedBeenRaised(INotifyPropertyChanged vm, string propertyName)
{
    return _propertyNotifications.ContainsKey(vm) && _propertyNotifications[vm].Contains(propertyName);
}

The NUnit test runner executes the SetUp of the TestBase class first, so all is taken care of by the time we get to running the actual tests. Here is how to test if a PropertyChanged has been raised for our calculator:

[Test]
public void PropertyChangedNotRaisedForX()
{
    MonitorINotifyPropertyChanged(_calc);
    _calc.X = 10;
    Assert.IsTrue(HasPropertyChangedBeenRaised(_calc, "X"));
}

Of course the above test passes until you actually add the code to raise the event. I’ll leave that to you so that we can focus on the testing now.

If you want even more detailed verification, you can access all the events in the order they occurred by accessing the _propertyNotifications[_calc] list. This contains all the property names in the order their respective PropertyChanged even has been raised.

In the next post, I’ll look at the challenges of using Laurent Bugnion’s MVVM Light Toolkit with this Unit testing approach. I’ll give directions on how to handle his IsInDesignMode and Messenger while testing, and also a look at how some of the code smells above can be decreased by using the SimpleIOC container in the Toolkit.

Using NUnit to test Windows Phone 7 applications – Part 1

2012-08-08T08:33:17Z

Unit testing phone applications is not common. Partly, because these applications tend to be UI-focused and there is not too much logic to talk about in the first place. Partly, because the lack of tooling support. I am not a big unit tester myself, since I’ve mostly focused on UX recently.

However, our latest Windows Phone project simply cried out for unit testing. It has an extremely complicated inner logic with many complexity layers. We had a reference iOS app that we could turn to for verifying the results of the difficult calculations, but not much more. Most new new features increased the difficulty of these calculations by a whole new dimension. We had tons of calculations we need to be sure stayed valid as the code evolved. So, I began to look at Windows Phone unit testing.

At first, I stumbled upon Jeff Wilcox’s Silverlight Unit Test Frameworks. This is a test harness that runs on a Windows Phone device or the emulator. Here is a screenshot from Jesse Liberty’s intro blog post on the topic.

There are quite a few pros and cons for this framework. The big pro is that it runs on the actual environment, on the same .NET Compact Framework, etc. as the actual application. This didn’t help me much, since the #1 thing I wanted to test was the calculation logic, which is UI agnostic. The big con is that it doesn’t run in a regular unit test runner. Resharper won’t execute it, you can barely hack it into the build process. But the big turn-off for me was that you can’t double click on a failed test and be taken to the right line of code in Visual Studio. You have to do a lot of code navigation and emulator / phone tapping to run a single test in debug mode to figure out what’s wrong.

Let’s create a new Windows Phone Class Library, this will be our code being tested, called TestMe:

Then, another one, which will contain our tests, called TestMe.Test:

Both of them should be Windows Phone 7.1.

Now, if you want to add the nunit.framework dll to the TestMe.Test project, it will fail.

Here is the Message Box text so that search engines can find it:

---------------------------
Incompatible reference
---------------------------
"nunit.framework, Version=2.6.1.12217, Culture=neutral, PublicKeyToken=96d09a1eb7f44a77" is incompatible with Windows Phone 7.1

In order to add it you should to change the project's target to a compatible framework first.

There are a couple of solutions out there for this one. Most include manually editing the project files and other trickery. However, there is a much simpler solution – use NuGet! Instead of adding the nunit.framework library as a reference, just right click on the project, select Manage NuGet Packages:

Then search for NUnit and install it.

We don’t get any error messages complaining about incompatible runtimes. I don’t know if this is a bug or feature in the NuGet package manager, but I’m happy it works

Reference the TestMe project from the TestMe.Test project, and setup is done.

Let’s write a little bit of test to make sure everything is OK. In the TestMe project, create a Calculator Class:

namespace TestMe
{
    public class Calculator
    {
        public int Add(int x, int y)
        {
            return -1;
        }
    }
}

And in the TestMe.Test project, let’s write our first test:

using NUnit.Framework;

namespace TestMe.Test
{
    [TestFixture]
    public class CalculatorTests
    {
        private Calculator _calc;
        [SetUp]
        public void InitTest()
        {
            _calc = new Calculator();
        }

        [Test]
        public void AddingZerosShouldReturnZero()
        {
            int result = _calc.Add(0, 0);
            Assert.AreEqual(0, result);
        }

    }
}

Obviously, this test should fail. But what kind of Test Runners can we use to verify that?

We can use the Nunit test runner, but that hasn’t been installed with the NuGet that we performed above. But we can also use Resharper, which flawlessly recognizes the tests:

And we can also use NCrunch, my new favorite test runner tool, which runs the tests as you type, and provides you almost instantaneous feedback.

And of course, we can just select a test, debug it and stop on any breakpoint:

Note: it is not the goal of this post to teach you how to do unit testing in general. There are much better sources for that

Now that we have the tooling out of the way, we can start writing Windows Phone unit tests just like we do for the desktop framework… Until… well, until we run into a few gotchas with this method. In the next parts of the series, I will look into how you can handle assembly load failures (these are tough), test ViewModels, and tweak Laurent Bugnion’s MVVM Light Toolkit so that it works well with unit testing scenarios.

MVP Again… 5 years and counting!

2012-07-09T23:28:03Z

I’ve waited for my MVP badge to arrive before announcing this on my blog: I am an MVP again, for the 5th year in a row!

The novelty of receiving the famous “Congratulations! We are pleased to present you with the 2012 Microsoft® MVP Award! ” email may fade by the fifth time, but the feeling of being honored by the Award, being humbled by the amazing fellow MVPs I can be associated with, and the whole-day grin on my face does not fade at all.

Starting my fifth MVP year feels special. Five years is a lot! My son turned five yesterday, and oh boy, has he changed. So has technology. My first MVP award was in the Client App Development category, mostly for my community work with WPF and generally around User Experience. Then along came Silverlight, and I found myself being in the first wave of Silverlight MVPs, and part of such an amazing group that it makes me incredibly proud and humbled at the same time to be among them – and even call some of them friends.

Last year was a blur (my MVP year is from April to the end of March, with the awards arriving in July). While Silverlight’s best days may be behind it, the spirit lives on. Ever since I heard the first details, I was focused on Windows Phone. I jumped right into it, absorbing the – Silverlight based – development model along with the amazing User Experience it provided. I talked at tons of events about the phone, development, design - including the Hungarian launch of Windows Phone 7, and later the Nokia devices. But the biggest speaker achievement of this last year was that – along with Bálint Orosz, the designer of Cocktail Flow – I could not only go to Mix’11, but also speak there about SurfCube (our Innovation Award winner browser for Windows Phone 7). Over 800 “outsider” talks were submitted, 200 put up to vote, and merely 12 made it to the conference. Wow.

I also got into Kinect. I love the green field UX experiments this amazing piece of technology allows us to perform. To see some of our Kinect work, visit our homepage at www.response.hu. I also got invited to a number of Kinect talks, including a “tour” during the Baltic Tech Days.

All in all, I gave almost 30 talks last year, many of them abroad. Our Silverlight User Group was transformed to a vibrant trend-watching community, vNext.hu, which is the most active vNext group – not bad from a small country of only 10 million people . I also kept blogging here, posted 23 articles about Kinect, Windows Phone and User Experience. But the writing didn’t stop there: I wrote an e-book called Windows Phone book for Silverlight Developers, the performance chapter of the first Hungarian Windows Phone book, and also a chapter for a Wrox book which was later pulled due to scheduling conflicts. And these are just the highlights…

What’s next for this year? I will keep following my passion and interest in technology. That’s the only way to do it. I’ll keep working with Windows Phone 7, while looking forward to Windows Phone 8. Windows 8 will definitely be a big part of my activities, the new tablet form factor and the Metro design language are very exciting. Kinect is also something that I plan to work with and share my thoughts on – both here, on Twitter and also on conferences as a speaker.

Here’s to another five years!

Windows Phone Performance - Part 5 of many: A Developer’s look at the Nokia Lumia 610

2012-05-31T12:31:40Z

The plan for part 5 was to continue my WP Performance series with a discussion of List performance. However – as it usually happens - Life has came and changed the plan. First, Oren Nachman published a great article on The Windows Phone Developer Blog called “Listbox, Why Art Thou Blanking”, and I don't really have too much to add to what he wrote. But also, I happened to get a Nokia Lumia 610 phone to test some of the applications we are working on. The 610 (and other, so-called Tango devices) will have a big impact on the lives of Windows Phone developers, but most of have never seen one first hand. I cannot give you one, but I am trying to do the next best thing: I will share experiences with the device and the results of some measurements that I performed so that you can get a feel for the performance characteristics.

The Nokia Lumia 610

The Lumia 610 is the first “Tango” device. Tango is the codename for the Windows Phone 7.5 Refresh (build 7.10.8773.98), but you will not catch a Microsoftee call it Tango officially. As an OS update, Tango only brings a handful of small features to Windows Phone. The big change is that Tango has been optimized for phones that have as little as 256 MBytes of RAM available. And that is exactly what the Lumia 610 has. The other notable difference (from a developer’s point of view) is that the 610 has an 800 MHz CPU (the first generation devices had 1GHz, and most of the second generation phones have a 1.4-1.5 GHz CPU).

The Nokia Lumia 610

The Lumia 610 is a very good-looking device – especially the black-white combination that you can see above. It looks better than the much more capable Lumia 710. The Tango OS also performs nicely – you can barely say that the OS runs on a low-specs device, as long as you are using the native features of the operating system. The looks and low price, great in-store performance (where you mostly play with the OS) will probably make it a hit among those who are looking to enter the world of smartphones.

Things change dramatically when you start third party applications. At this point, you will notice that some applications take forever to load, and for more complicated apps, you will experience glitchy scrollings, blankings, slow response time and even occasional freezes for a second.

Application Loading / CPU performance

In the second part of this series (It's Full of Stars!), I dealt with application startup times. Let’s see how three phones measure up in this test: the Lumia 610, the LG Optimus 7 and the Lumia 800.

I tested the star drawing app on all devices. The LG Optimus 7 is a decent first generation Windows Phone, with 1GHz CPU. The Lumia 800 is a second generation device, with 1.4GHz CPU. Both of these have 512 MBytes of RAM. As I mentioned before, the Lumia 610 has an 800 MHz CPU with 256Mbyte RAM. Just for kicks, I also included the values measured on the Emulator.

The first column of the table shows how many stars (small ellipses) are drawn, and the other columns show the number of milliseconds it took to launch the application. The stars are drawn on the UI thread at startup, so this drawing delays the startup time of the app.

Silverlight Layout and Ellipse rendering performance on the Nokia Lumia 610, Nokia Lumia 800 and the LG Optimus 7.

The results are surprising, but clearly show how efficient the CPU in the Lumia 610 is. Despite the 20% lower CPU clock, the Lumia 610 could easily match the performance of the LG Optimus 7. It could even draw 20,000 stars, which the Optimus 7 couldn’t finish in 20 seconds, so the OS killed the app!

This is of course by no means a comprehensive test of the CPU. But it is a fairly good indicator on what you can expect from the Lumia 610 in Silverlight – the test app exercises the layout and rendering subsystems of Silverlight, which are key to any application.

GPU Performance

Fill Rate vs FPS on the Nokia Lumia 610, Nokia Lumia 800 and the LG Optimus 7

As you can see, the Lumia 610 not just matches, but seriously outperforms the first generation Optimus 7, delivering almost twice as many frames per second under heavy load! The Lumia 800 has 75% higher CPU and GPU speed than the 610, and it performs accordingly.

The FPS starts to drop at the Fill Rate of only 2.5 for the Optimus 7 (which is tuned to a maximum frame rate of 50 instead of the usual 60). At this point, the list scrolling becomes somewhat choppy – it begins to jump a little bit every second or so, while remaining completely smooth in between. This is a very low value – it barely gives you enough space to put a background image and a full screen of information on top of it. E.g. if you have a Panorama app where you want to put a semi-transparent background behind your content, you are already at the edge. The first cutoff points are much further for the 610 – at a Fill Rate of 4. This means that the 610 can manipulate 1.5 more screens worth of pixels without any visible glitch. Of course, the Lumia 800 wins here again, which can almost go up until a Fill Rate of 8 before the first signs of strain start to show.

The pain point (where the frame rate drops below 30 FPS) comes way too early on the Optimus 7 – at a Fill Rate of 5, we are already in the red zone. The 610 can handle 70% more pixels before dropping below 30 FPS, at a Fill Rate of 8.5, while the 800 can soar up to a Fill Rate of 15.

Memory

So, the Lumia 610 CPU is on par with first generation devices, and the GPU is much better. Why did I say then that the performance of the Lumia 610 is not perfect? It all comes down to lack of memory.

Microsoft has done an awesome job shoehorning the Windows Phone 7.5 Mango OS into 256Mbytes. The only feature they had to remove are the background tasks, and some media codecs (see the details at Handle Feature Reductions). But while apps on 512Mbyte phones have a lot of memory to play with, apps on 256 MByte phones only get 60 MBytes. Tango improved virtual memory (paging), which means that your application still can use up to 90 MBytes of memory, only the data above 60 MBytes gets swapped out. When this happens between two screens, all the user sees is some delay between the screens. But when it happens while you are still on one screen, the app seems to freeze and does not react to user input. This usually takes about a second, which is more than enough to cause frustration in the user.

There are some great posts by Mike Battista on Optimizing Apps for Lower Cost Devices that cover what you can do to help this situation. You can get to a fairly decently performing app following his directions. However, if you are using things like the WebBrowserControl or the BingMaps control (and can’t replace them with a call to the appropriate OS task), you are out of luck. No matter what you do, these controls take up too much memory. For example, a simple app only with only a WebBrowserControl can easily consume as much as 50-52 MBytes of RAM, which doesn’t leave much space until the swapping starts to happen.

Lumia 610 loading a complicated web page in a simple app, occupying 56 MBytes of RAM (which later goes down to 42) out of the available 60.

To see what the low memory means for a more complex app that uses the WebBrowserControl, check out this video. Note, that I do not load anything beyond a basic text file into the browser – if I did, the 610 would load up unbearably slow.

Launching SurfCube on LG Optimus 7 and Nokia Lumia 610

Summary

The Lumia 610 is a surprisingly powerful device that easily matches the first generation Windows Phones and even outperforms them when it comes to graphics. However, the lack of memory kills the joy – a slightly more complicated app can easily find itself being swapped out, resulting in slow app launches, responsivity issues and temporary freezes due to paging. Despite the fairly powerful GPU and CPU, it takes significant developer effort to match the user experience of apps when compared to first generation devices – if at all possible.

Zoomcity Budapest – Controlling a 7.2 Gigapixel panorama with Kinect

2012-03-28T16:36:21Z

ZoomCity Budapest is our third Kinect project, and the first that used Microsoft’s Official Kinect for Windows SDK (Although we have upgraded our Kinect Quiz app to the official SDK, too).

The software was created in cooperation with 360World (the guys who created the world recorder 70 Gigapixel panorama of Budapest) and Aera. My company worked on the Kinect controls, the Silverlight Deep Zoom engine and the user experience. The panning is controlled by the left or the right hand, while zooming is performed by stepping forward (closer to the screen) for zooming in, and stepping backwards for zooming out.

The application is very easy to learn by following the on-screen instructions, and tuned for an exhibition scenario – it has a narrow “play area” and robust user tracking among other things. We have also created little “step here” markers to help the users.

Here is the end result:

Thoughts?

Windows Phone Start Screen Wastes Tons of Screen Real Estate. Or does it?

2012-03-23T15:17:00Z

One of the things people say when looking at the Windows Phone Start Screen is that it wastes a lot of Screen Real Estate on the right side. Here is what I am talking about:

See? The right side is occupied by just a black space with an arrow. “That space could and should be used to display useful information!”

One of the reasons for the space on the right side is that if you move the tiles up, the clock doesn’t get obscured:

But still, I hear you say… “There is so much wasted space there!” OK, I’ll play along. What if we enlarge the tiles and don’t care about the clock?

See? Compared to the original version, we actually have less information on the screen, despite starting a bit higher up. If I remove the margin entirely, things are even worse:

No whitespace, just a lot of valuable content. Right? But where did the Photo Hub’s tile go? Barely visible, just a few pixels here. And how much value did we get? We can put a few more characters in the tiles without changing the font size, but nothing to call home about.

OK, but if we do utilize the entire screen, we may be able to put three columns of tiles there, right?

Maybe… these tiles are 152 pixels wide, maybe 10% drop from the current 173. But the fonts on the tiles are getting too small to read easily if you are beyond your twenties… What do you think? Does being able to see the extra 5-6 tile worth it? I personally prefer the current, two column layout, three columns just feel crowded. Maybe if I had a bigger phone and bigger resolution, additional columns would make sense…