Our lab is buzzing with activity as the team prepares for the National Retail Federation’s 102nd Annual Convention & EXPO in New York. On display will be the latest iteration of Razorfish 5D— the world’s first cross-device, cross-OS, connected retail platform. Launched at last year’s NRF convention, 5D has already been launched in several markets and was used to create Audi City London, a one-of-a-kind immersive virtual showroom. This year we’re showing how our platform can power customized, personalized and seamlessly synchronized shopping experiences. We threw in some augmented reality and a bunch of transparent displays as well.

Our team will be demonstrating the 5D experience in booth #1005 on Level 3 of the Jacob K. Javits Convention Center. If you can’t make the show, be sure to follow us on Twitter to get the latest updates.

With Microsoft’s Build 2012 over and Surface now available for purchase, it seems like the only thing Microsoft developers can talk about is building Windows 8 store apps. Creating Windows apps that can be monetized using the same wildly successful revenue sharing model that Apple pioneered is compelling. However, in the rush to cash in on this new frontier it’s easy for developers to lose sight of the fact that Windows 8 is far more than the addition of a new shell and a new application model.

As a product, Windows has been continuously developed for over fifteen years (which is far longer than Apple has been using the revenue sharing monetization model), but this time Microsoft has built some great technologies that have received far less attention.

The first that comes to mind is DirectX. When most developers think of DirectX they typically think of 3D applications and games, and while DirectX is a great platform for building 3D applications and games, it is capable of doing much more. In Windows 7, Microsoft added Direct2D to the DirectX family of technologies, and as you would expect Direct2D finally adds the ability to execute drawing commands against a 2D surface. In many ways Direct2D is being setup to take the place of GDI, and because it is built on top of Direct3D and DXGI it is also hardware-accelerated and runs on the GPU. Microsoft has also added DirectWrite, which now provides developers a way to layout and render high quality text while making full use of the GPU. Before DirectWrite developers either had to use GDI or rig their own system to render text; now with DirectWrite, developers have access to a rich API that supports layout, international text, and sub-pixel anti-aliasing that integrated easily into the rest of their application whether they are using GDI or DirectX.

Even though DirectX is an amazing API and has only grown in capabilities over the years, there are still numerous applications that don’t use DirectX. Most of these applications use GDI (a much older technology ) to paint to the screen, and yet other applications use other rendering libraries like WPF. It used to be the case that once a developer chooses a core rendering technology to build an app, it was impractical to leverage any other rendering technology. Fortunately Microsoft has built a new technology into Windows 8 called Direct Composition that does away with this limitation. At its core, Direct Composition is simply a bitmap compositing engine. By using Direct Composition it is now possible to use WPF to build the bulk of your application, and to sprinkle in some DirectX code to give your app that extra sparkle that would otherwise be too difficult or which would run too slowly if attempted using WPF. Additionally, because Direct Composition is baked deep into Windows 8 it is possible to compose applications and effects that are generated from code running in separate processes, which opens up a whole new front in software engineering.

While certainly not the last hidden technology in Windows 8, the last I’ll cover is Direct Manipulation. Most developers are extremely comfortable in the world of mouse driven user interfaces—we all understand the concepts of click, hover, right click, move, etc. But the world of touch driven user interfaces is largely uncharted by many developers and most will find that it is far more complicated and difficult than the mouse driven world. In a touch user interface, there may be one, two, five, or no touches on the screen. The user may be pinching to zoom or he or she may just want to move an object but by using two fingers instead of one. Of course it would be possible for developers to build state machines that were able to process and interpret these user gestures and to continue to build their touch based application, but it turns out that building these state machines is not an easy task. Fortunately Microsoft has done the hard work and has included the Direct Manipulation technology in Windows 8. Direct Manipulation is essentially just a touch input state machine that frees developers from the details of interpreting user input. Instead, by using Direct Manipulation, developers can be notified when a user is engaging in a common gesture like pinching, sliding, or rotating. In fact much of the Direct Manipulation API is the same as what you would find in WPF or in WinRT Xaml, and it is my guess that the WinRT Xaml stack is actually built on top of Direct Manipulation.

There are of course many other  new technologies that Microsoft has created for Windows 8 and still more existing technologies that Microsoft has improved. The Windows Store app model opens up entirely new markets for Microsoft developers, but let’s not forget that Windows 8 is still a great platform for building desktop apps.

Ever since Microsoft started leaking details about its upcoming version of their flagship product, Windows 8, there has been firestorm of controversy among Microsoft’s faithful.  Many Silverlight application developers and publishers feel like they have been willfully misled into investing in a technology that Microsoft is now apparently abandoning. Many IT Pros dislike and fear the retraining efforts they will have to make with new Start Screen and other Windows shell changes. Finally, many ASP.NET web developers don’t see how Windows 8 relates to them despite the fact that Microsoft is adding “WinJS”, a runtime that allows Web developers to leverage their existing skills to build native applications.  On the ground, it may seem like things are going badly for Windows 8, but with a little developer ingenuity and a lot more communication and documentation from Microsoft, Windows 8 could be the product that saves Microsoft from being a victim of their own success.

Take the Start Screen for example—In order to finally enable OEMs to build devices that can truly be considered a “Tablet PC”, Microsoft has to provide a way for users to launch applications.  One might be tempted to think that the Start Menu in Windows 7 could be adapted to serve this purpose, but fingers are just not good at tapping on small icons or icons that are densely packed.  Making the Start Menu a full screen experience is really the only way to get enough space to create a truly usable touch-optimized experience.  We in the Emerging Experience group have known this for years as practically every single touch based application that we have built has been a full screen app.  On top of this, Microsoft’s Start Screen’s animations are extremely fluid and natural, and so to us it seems like a natural platform from which to launch our showcase applications.

To give a little background about ourselves and our applications, we are a technology agile group, which means that we use the technology that creates the best experience for our customers. Many of our apps are built using WPF but we also have apps that are built using Flash.  Obviously attempting to port our applications would not be a good strategy for the Flash apps, but even after a brief investigation, I quickly decided that attempting to port all of our WPF applications was a non-starter.  The Metro APIs are far too different and who knows if, after porting the WPF applications, I would even end up with an app that worked?  The solution, it was decided, was to leave the existing showcase applications as they were but to simply create live tiles for them so that they could be invoked.

The problem with this solution is that it is not possible to really take advantage of the Live Tile infrastructure from a Win32 app.  In a Metro-style (WinRT) application you can supply different resolution images for the tiles by altering the AppX manifest, but Win32 Applications don’t have AppX manifests.  It might seem trivial to simply create a WinRT application that upon launch invokes one of our showcase applications, and to use the WinRT app’s AppX manifest to customize the Live Tile, but unfortunately the relationship between WinRT and Win32 is significantly more complex than that.  First of all, WinRT applications can call some Win32 APIs, but it explicitly cannot create new processes—this is part Microsoft security model for WinRT apps.  On top of that, even though WinRT apps can call many Win32 APIs, many of those calls either fail outright or fail to have the desired result.  Clearly this is an area where Microsoft can do a much better job in providing documentation. 

To work around these limitations, I decided to create a WPF application that lives in the System Tray as a notification icon.  The entire purpose of this WPF application is to listen for network calls and then launch and activate the requested application.  At this point our WinRT “launcher” application was simply responsible for initiating the network call and then close itself down.

While this worked beautifully in the debugger, I was surprised to find that it did not work once the applications were freed from the debugger.  Sure, the Launcher application still made a network call to the WPF application and the WPF application still launched the showcase application, but the showcase application was never displayed.  The problem, it turns out, is that the Win32 function “SetForegroundWindow” on which my WPF application was indirectly relying behaves differently if the calling application is being debugged.  Clearly the Windows shell makes use of a facility to show the desktop when the user clicks on the Desktop tile in the Start Screen, but when I asked Microsoft about this and SetForegroundWindow,  I was essentially told that this was by design and that only the end user should control which window has focus.  I understand the wisdom of this decision, but this answer didn’t get me any closer to being able to launch our showcase applications from nice looking Live Tiles.

While I wouldn’t propose that developers do this in production applications, Windows 8 isn’t a production OS itself—and I still hold out hope that will make this whole endeavor moot by they release Windows 8.  With the disclaimer in effect, the way that I solved this problem was to create a third Windows Forms application whose sole responsibility is to run CDB, the command-line debugger, and automate it to launch and attach to the WPF application.  Because the WPF application has a debugger attached, it is now able use the SetForegroundWindow API and the entire system works as expected.  In fact, by not creating a Window in the Windows Forms application and launching CDB without a console window the entire hack is invisible to the user and everything transparently works as expected.

While there probably are other ways to achieve the same effect, this really should highlight noteworthy holes in Microsoft’s messaging and documentation.  First of all, I can understand that Metro-style apps won’t be backwards compatible on Windows 7, but this is no reason not to provide a facility to make Win32 apps forward compatible the Windows 8 Start Screen’s Live Tiles.  Also, while I understand that, generally speaking, it is not good design to allow application developers to control which application is active, at the same time there are always exceptions and this functionality should be enabled/disabled in registry settings or group policy and not be decided in a one-size-fits-all fashion.  But with this said, Windows 8 is actually proving to be a natural evolution of Windows 7 and a good platform on which to build touch-based experiences.  With all the talk Silverlight, Javascript, and strategy, it’s easy to forget that applications are only valued for creating good user experiences and that Windows 8 delivers a great user experience for a pre-Beta operating system.

This year, Razorfish sent several of our people to MIX 11, the annual Microsoft sponsored conference in Las Vegas for developers and designers.

So much happened during our week at MIX  that it is difficult to summarize it all thematically.   There were announcements and sessions on several major topics: IE9, HTML5, ASP.NET MVC 3, Silverlight 5, Windows Phone Mango release, and the Kinect SDK. In addition, there were also appearances from MS Surface v2, Windows Azure, oData and Sharepoint as well as a remarkable set of UX presentations.

The word on everyone’s lips seemed to be fragmentation, whether in reference to the expected HTML5 compatibility issues between future browsers (which the emphasis on the IE9 “native” browser experience only exacerbated) or to the greater array of Microsoft development technologies fighting for developers’ attentions.

What the four Razorfish attendees at MIX saw, on the contrary, were patterns of evolution.  The much ballyhooed struggles between the Windows Team and the Development Team inside Microsoft for the future of HTML5 and Silverlight indicate to us that Microsoft can still respond to a rapidly changing worldwide technology ecosystem.  When a product is struggling in the niche it was doing fine in a year ago, it can be refitted to survive in a new niche. Such is the case with Silverlight, originally intended as a Flash-killer.  Silverlight developers never truly adopted the original Flash-killer strategy and instead used Silverlight to develop more sophisticated and interesting line-of-business applications.  The problem is that LOB applications do not really belong on the web.  They belong behind firewalls.  The lack of casual games written in Silverlight likely affected the ability of Silverlight to force downloads and gain browser share.  So instead, the strengths of Silverlight are being moved to the desktop as well as specialized platforms such as Windows Phone, the XBOX (?), and possibly Windows vNext.

WPF, which was once the pre-eminent desktop development platform, is in turn becoming a specialized tool for NUI development for multi-touch, Surface and Kinect.  The announcement of the Kinect SDK itself demonstrates Microsoft’s continuing ability to innovate and surprise.  It is, in the best sense of the term, a fortuitous mutation.

This all leaves HTML5 as the preferred technology for the web.  We of course see the early signs of browser compatibility issues. At the same time, though, we have each been through this before and survived. The extra gyrations developers will have to go through will, in the end, provide the illusion consumers desire – that the same application can run similarly on any operating system and any device.  As one MIX speaker put it, “The technology you use impresses no one.  The experience you create with it is everything.”

Speaking of devices, we are excited to see that the WP7 team is not only going for parity with other smart phones but is firing warning shots across their bows with the much touted Mango release.  Features we’re used to like multitasking are being expanded beyond current implementations with updating live tiles and “Live Agents” which allow for more full-featured multitasking.

There was naturally some complaining about the placement of various keynotes and sessions.  With the multiple announcements and cross-blocking sessions, isn’t there a danger that individual messages will get drowned out in the general cacophony?  We find that the panoply of conflicting viewpoints is one of the chief charms of MIX. Microsoft is not Apple.  To borrow from Isaiah Berlin’s famous title, Apple is the hedgehog that does one thing well; Microsoft is the fox that explores all avenues and experiences.  The great strength of Microsoft is its ability to challenge developers and create new harmonies out of these encounters. Should MIX ever be split up into different web, Silverlight, Windows Phone and UX conferences, we would all be poorer for it since all we would ever get would be our own opinions reflected back on ourselves – an echo chamber effect that will only serve to make us all deaf.

The overall quality of all sessions and boot camps were extremely high this year.  In the past, we have been happy with a 60% success rate on talks.  This year roughly 85% of the talks rang our internal bells. Certain sessions deserve a special shout out, however.

While all the UX lightning talks were extraordinary,  August de los Reyes’s 21st Century Design (10’ 45”) talk took it to a different level.  In the live session, the slide deck itself was the star with the brilliant August narrating it much as Peter Jones was the voice of the book in the old Hitchhiker’s Guide to the Galaxy television series.

Despite its inauspicious title, Ivan Tashev’s talk Audio for Kinect revealed what a truly remarkable device the Kinect really is. We honestly didn’t understand half of the technical stuff and we became queasy when formulas started flying across the screen. What we learned, though, was that only a fragment of the Kinect’s full audio capability is currently being used.  Dr. Tashev demonstrated the ability of the Kinect’s audio algorithms to pick out two separate speakers, one reading Chinese and the other reading Korean, and separate them into different channels.  All of this cool functionality will, moreover, be handed over to developers when the Kinect SDK beta is released at the end of spring.

Finally, we cannot say enough good things about Luis Cabrera and his willingness to demonstrate the Surface 2 at work in A Whole NUI World. Razorfish, of course, has a special affinity for anything Surface. What was outstanding in this presentation was not only the beauty and power of the new Surface devices but also the amount of thought that has gone into the tooling. Kudos to the Surface team, they’re reaching for a goal that is more than just a new technology but a new way for people to interact with computers and each other.

By the end of MIX, we were all quite exhausted mentally and physically. It may take us a full year – until the next MIX – to finish ingesting everything that we learned and experienced at MIX11.

So long, Microsoft, and thanks for all the Kinects.