Microsoft Kinect


Mar 25, 2014 by in Kinect, Microsoft Kinect, News, Projection, Technology, Travel

As passengers of San Francisco International Airport (SFO) walk into the newly renovated Terminal 3E they are greeted by a 16-foot wide glowing data visualization that hovers above six multi-touch screens, the Flight Deck. The Flight Deck highlights SFO’s global reach, services, amenities, and museum exhibits. We want to share the backstory about the development of the interactive data visualization component of the Flight Deck.


The Projection

Our goal was to create a visual beacon in an airport that stood out from other digital displays and had the ability to be seen on both sides. We envisioned a uniquely shaped floating image without bezels that was constantly moving to capture the attention of passengers whether they are arriving or departing. To meet this challenge, we experimented with several concepts but decided on attempting a 24/7 rear-projection in a bright airport terminal. Rear-projections in backlit environments are difficult since projectors don’t project black. Initial tests (images below) showed that a projection in the space would be possible given the proper projector, proper dark grey rear-projection film, and window treatment. We worked with the architects to specify a frit for the windows immediately behind the projection. We used a 20,000-lumen projector, the brightest rear-projection film we could find, and optimized the placement of the glass and projector for the visitor approach. Visitors approach the Flight Deck parallel to the projector, so that they experience the brightest portion of the projection when walking toward the Flight Deck.


The Experience

We wanted to create an ever-changing experience where a Kinect camera would use people’s presence to alter the background, and real-time data from flight paths would highlight the global reach of SFO from moment to moment. Key rewards were also synchronized with the touch screens so that visitors are able to take over the entire projection if they collect hidden rewards within the touch screen experience. The software was developed using Cinder, and used OSC for network communication. An early decision was to heavily thread our architecture so that we could maintain a minimum of 60 fps for low interaction latency and smooth animation. Unfortunately several components in Cinder were not properly written for heavy multithreading such as the Console or the Kinect class, so we added locks and properly threaded components as needed. Early interaction prototypes tested the limits of our frame rate and latency to determine areas for optimization.


Kinect Interaction

As curious visitors approach the Flight Deck a Kinect camera captures their form and creates triangles and shapes that play with their form using an attraction based physics system, as seen in the image above. As visitors move left to right the triangles leave trails of triangles on the screen that orbit the space, creating constantly changing visuals. Visitors can also use their arms to attract and control the shapes around them. A rear mounted Kinect camera captures approaching visitors and uses computer vision algorithms to convert the depth image to blobs. The computer vision allows for tracking of any number of people, vs two people for skeleton tracking. In addition, we also use depth data from a depth range that is normally unreliable for skeleton tracking. We modified the Kinect API for Cinder to use an extended depth range to capture people who are up to 16 feet away and to see people within a 15 foot wide area.

Data Visualization

Live flight data is visualized on screen as well as inbound and outbound activity from SFO in the last 24 hours. Positional data from flights is received in a polar coordinate system and is converted to a Cartesian coordinate system on the GPU. Our data visualization uses the positional data from flights so that flight paths can be visualized with bends and waviness vs perfectly smoothed flight paths. The challenge with positional data was that live flight data is not always available in all positions of a flight so some interpolation had to be mixed with the actual data. SLERP was utilized for interpolating positions and real data was combined with interpolated data using a gated smoothing algorithm. The approach allowed for us to properly visualize flights, for example flights to Asia, where positions are occasionally lost but picked up at various locations.

Looking Forward

The interactive data visualization represents only one aspect of the Flight Deck. The Flight Deck also consists of a multi-touch experience and a mobile experience and was a collaboration of creative and technical talent. Since the launch, we’ve heard a lot of feedback where people are comparing the Flight Deck to their favorite sci-fi shows and movies. We hope that we can continue to create forward thinking experiences for the public.


Audi City Berlin Launched

Feb 05, 2014 by in 5D, Experience Design, Microsoft Kinect, Mobile, Multi-touch, News, Portfolio, Retail, Technology, Touchscreen

First London, then Beijing and now Berlin: Audi City has transformed the dealership experience in ways never before seen. It’s delivered by one of the most technologically advanced retail environments ever created and features a variety of multi-touch displays for configuring your Audi from millions of possible combinations. Personalize your Audi then toss it onto one of the floor-to-ceiling digital “powerwalls” to visualize and explore your configuration at a true 1:1 scale. Audi City is a true dealership of the future and an effort we are proud to be part of.






Big Ideas Come in Small Packages

Jan 08, 2014 by in 5D, Experience Design, Microsoft Kinect, News, Retail, Technology, Touchscreen

NRF Container

Our container arrived at New York’s Javits Center this morning where our team is prepping for next week’s National Retail Federation annual convention and expo. What’s in the container? Stay tuned.

Tags: , ,

We’re Building the Next Big Thing

Jan 07, 2014 by in 5D, Experience Design, Kinect, Microsoft Kinect, Multi-touch, Retail, Technology, Touchscreen

Stay tuned to find out what it is.

KinectiChord: Touch Technology Like Never Before

Jun 18, 2013 by in Experience Design, Kinect, Microsoft Kinect, Technology

This week at the Cannes Lions International Festival of Creativity, we debuted our latest creation—KinectiChord: a multiuser, multisensory experience that blends physical and digital in an unexpected and delightful way. On display in the Microsoft Advertising Beach Club, this experience allows multiple users to see, hear and feel technology like never before.

Bringing 5D to Life at NRF

Jan 14, 2013 by in 5D, Augmented Reality, Experience Design, Kinect, Microsoft Kinect, Microsoft Surface, Multi-touch, Near Field, News, Portfolio, Retail, Technology, Touchscreen

Get your hands on the 5D experience by embarking on a unique shopping journey that utilizes a variety of platforms and technologies, including a first of it’s kind, seamlessly-synchronized transparent interactive display wall. It’s located in the Microsoft booth (1005) on Level 3. And to see more of 5D in action, head on over to


The Presence of Technology

Nov 07, 2012 by in Microsoft Kinect, News, Technology


At the same time the //Build/ conference was going down in Redmond, Washington, I was next door in Seattle for the Seattle Interactive Conference (SIC://). Besides a fondness for forward slashes, these two conferences shared a common interest in the future of technology. //Build approached this topic from the software side while SIC:// did it from the design and agency side. The Kinect for Windows technology, interestingly, was present at both events.

I was invited to SIC:// in order to represent EE on a panel about Natural User Interfaces. It was an amazing panel that included David Kung from Oblong, Matt von Trott from Assembly Ltd, Scott Snibbe from Snibbe Interactive and John Gaeta of FLOAT Hybrid. Our conversation about what NUI means today was preceded by an amazing fifteen minute talk by Oscar Murillo that showed off many K4W techniques in a holodeck-like demo. You can read more about the panel here and here. It was expertly moderated by Steve Clayton of Microsoft.

What made the event fascinating for me was the time I got to spend with the other panelists before our talk and after. There was a clear trajectory in our backgrounds. John is involved in the motion picture industry and helped design many of the futuristic movies (like The Matrix) that have inspired the rest of us to work with bleeding-edge interface technology. Dave’s company brought forward advanced academic research to actually realize Minority Report (one of Oblong’s founders helped design the gestural interface Tom Cruise uses in the movie). Microsoft turned gestural interfaces into a consumer technology. Matt, Scott and I are using it for retail and marketing which will help fund and expand the proliferation of gestural sensors. Our collective goal is to create technology that anticipates and responds to our desires rather than simply frustrating us on most days.

We want to use technology, when it comes down to it, to hide the presence of technology in our everyday lives.

5D at Oracle OpenWorld

Oct 30, 2012 by in 5D, Microsoft Kinect, Microsoft Surface, Touchscreen

In early October, the Emerging Experiences practice’s San Francisco office brought our Razorfish 5D retail platform to Oracle OpenWorld. Within this global event was the first ever Customer Experience Summit. This event gathered industry leaders together to discuss strategies for driving customer-centric initiatives while interacting with some of the most future-forward experiences and minds.

Emerging Experiences set up our Razorfish 5D retail experience in beautiful Union Square park. We demonstrated how a seamless customer journey can cross over touch tables, gestural sensors, digital screens, tablets and mobile apps to transform the retail experience.

The 5D installation for Oracle CX showed how each element of the contemporary brick-and-mortar store can be enhanced and streamlined. Digital displays, smartphones and HD touch tables communicated with each other to provide infinite shelves as well as an immersive experience to tell the stories behind the store brands.

Tablet software provided store associates with the opportunity to not only help shoppers select items, but even interact with their customer’s smartphones. The 5D retail experience also demonstrated how virtual dressing rooms with augmented reality can enhance the retail experience. Each of these touch-points in turn generates massive amounts of data about the sales process.

Sharing our retail story with the attendees at the Oracle Customer Experience Summit was both extremely rewarding and entertaining. We look forward to returning next year.

Audi City: Inventing the Dealership of the Future

Jul 19, 2012 by in 5D, Experience Design, Kinect, Microsoft Kinect, Mobile, Multi-touch, News, Portfolio, Retail, Technology, Touchscreen

We’re excited by the launch of a revolutionary showroom experience for a premiere automotive brand. After a year of collaboration between Audi and a wide range of partners, Audi City has launched near Piccadilly Circus in London, ahead of the 2012 Olympics.

Piccadilly Circus in London

Audi City London is a groundbreaking dealership experience delivered by one of the most technologically advanced retail environments ever created. The digital environment features multi-touch displays for configuring your Audi vehicle from millions of possible combinations. Your personalized car is visualized in photorealistic 3D using real-time render technology, making the Audi City vehicle configurator the most advanced in the world. After personalizing your Audi, you can toss your vehicle onto one of the floor-to-ceiling digital “powerwalls” to visualize your car configuration in life-size scale. From here, you can use gestures to interact with your personalized vehicle, exploring every angle and detail in high resolution using Kinect technology.

credit: Audi

A purely digital showroom can’t deliver on the tactile experience of buying a car. Therefore, a store associate can save your configuration on a RFID-enabled USB stick and guide you into a personal consultation area that features a variety of tactile objects. These objects help the customers get hands-on with the materials of the vehicle including car exterior color and finish options and interior upholstery options. Each of these tangible objects are digitally-tagged through RFID technology. You can bring bring any of these physical objects over to the configurator experience and the corresponding exterior paint finishes and interior options will automatically update your vehicle configuration.

credit: Audi

When purchasing a car, the customer journey occurs across multiple channels. In order to integrate and simplify the car buying process, we’ve allowed customers to retrieve their online car configurations in the showroom environment. In addition, any car configuration made in the showroom is synchronized to your personal USB stick. Simply pop in the USB stick at home and the web-based configurator is automatically launched with the exact car configuration you created in the showroom. This allows Audi to deliver a “start anywhere, end anywhere” buying cycle for the customer, which has proven elusive for retailers.

Not only is Audi City a premier showroom environment, the dealership concept represents a fundamental shift in retail strategy for the brand. This new small-footprint retail format brings Audi closer to their customers, not only geographically but also emotionally. The smaller-footprint concept will launch in metropolitan environments and reach a younger urban and digitally-enabled demographic. After hours, the environment will serve as a cultural center in the larger community by playing host to readings, round-table discussions and art exhibitions.

credit: Audi

“Audi City combines the best of two worlds – digital product presentation and personal contact with the dealer” says Peter Schwarzenbauer, Member of the Board of Management at Audi. “People are placing greater emphasis than ever before on a direct and personal bond of trust with their vehicle brand – especially in respect of the increasing variety of products and available information. Thus, with Audi City we are creating a one-stop-shop for experiencing our brand. It is right in the midst of our customers’ lives, yet seamlessly connected to the online range offered by the four rings.”

Audi announced at the London launch that 20 showrooms in other major international cities will follow by 2015.


DeltaZone @ Madison Square Garden

Apr 15, 2012 by in Microsoft Kinect, Mobile, Multi-touch, Portfolio, Touchscreen

Anticipation has been building for years.

The expectation has always been that our lives will be transformed by new technologies. Everything from travel to sports and entertainment would be made new again…redefined.

And now, thanks to Delta and Madison Square Garden in partnership with Razorfish, that time has finally arrived.

Delta Air Lines’ Touch the Future of Travel and a newly refreshed yet still iconic Madison Square Garden is here.

In addition to the 11,000 square foot lounge which features select menus, multi-screen event coverage, and a clear view of professional athletes entering the arena through a glass hallway, we’ve created a unique experience for VIPs.

A personalized, curated way for travelers to discover new destinations, collecting content from around the globe and enjoying fantastic vistas that transport them into the magic of destination travel and discovery.

Delta’s Touch the Future of Travel is about unique inspiration, easier access to what you want, when you want it, and sharing travel ideas with friends…and Razorfish with Delta is making it all happen.