As you might know, Microsoft just posted a guide for HoloLens developers. I looked through the whole thing, and as it turns out, there is tons of new and detailed information in there regarding the user experience in HoloLens. So I wrote this overview of what it will be like to use HoloLens:
TL;DR-version:
>> HoloLens allows you to fill your apartment and office ("spaces") with traditional Windows 10 apps in the form of virtual screens. The device remembers what your "spaces" look like and where you left your apps last time you used them. When you close an app, it collapses into a tile.
>> Actual 3D apps (such as games etc.) are launched via launch points (little 3D objects) that you can pin to your room alongside the app tiles. Launching such an app makes all your 2D screens disappear.
>> Holograms that look black in those videos look transparent to the user. There are no black or dark holograms.
>> HoloLens supports a variety of input methods, including motion sensors in the "HoloLens Clicker", and Bluetooth keyboards.
YOUR ROOM IS YOUR "START SCREEN"
You may have seen the holographic Start screen - the little window in front of the user's face - yes, it's there, but that's not really where your apps live. Your apps sit on your tables and walls or where ever you left them last time, ready to be used every time you turn on the device. Some of them look like windows, some of them look like tiles, and some are little 3D objects representing a holographic app.
Here's how it works: The first thing that HoloLens does when you use it for the first time, is scan the room you're in. You see a room scanning animation running over your walls and furniture as you move your head around so the device can scan all the corners of your room. When you think the room is scanned sufficiently for your needs, you confirm with an "air tap". Now you can bring up the Start menu by doing the "blooming" hand gesture and select an app. This doesn't open the app yet - it just gives you a little tile or 3D object that serves as a launch point for that app, and you put it anywhere you want in your room. From now on that app sits there permanently, until you move it to another spot or remove it. (Sadly, those tiles don't support live tile functionality at this point). That way you can fill your room, or your entire apartment, with apps, and you can put multiple instances of the same app on different spots within your place - so you can have a video player in your bedroom AND your kitchen. Not only that, but you can create multiple "spaces", each with their own sets of apps. What that means is, when you take your HoloLens to work and use it there, it will recognize it as a new place (because the available WiFi networks are different than the ones at your home) and define it as a new "space", which you can again scan and fill with apps. All those spaces are stored on the device separately, along with the information on room geometry and where you put your apps. The device automatically recognizes which one of your spaces you're currently in (by recognizing the WiFi networks).
TWO COMPLETELY DIFFERENT KINDS OF EXPERIENCES
There are basically two separate modes or ways of using apps on HoloLens:
One way is running Windows 10 apps on multiple virtual screens within your "space", which means that you're in your apartment or office that you decorated with app launch points, and by opening them they expand into flat windows with the familiar Windows 10 app UIs. Any existing Windows 10 universal app can be used in this way. On the top of the virtual screens, there's a bar with access to window management (resize, move or close window) as well as interaction tools (zoom, scroll, pan) and a back button. Unfortunately there is no true multi tasking in this version of HoloLens, which means that only the app you're interacting with is active, and all the other opened windows just show frozen screenshots from the last time you used them. (They can play audio in the background though, and perform various background tasks). Also, although you can place multiple launch points for the same app, you can only have it running once, so for example you can't have two separate messaging conversations going on on two separate screens.
Another way of using apps on HoloLens is by opening a launch point of a true 3D app (as opposed to flat universal apps), which automatically makes all the launch points and windows in your space disappear - 3D apps take over the whole experience, you can't have a 2D window running while using a 3D app, and you can't run multiple 3D apps at the same time. Even system notifications get replaced by audio notifications (by Cortana) while using a 3D app. Those apps can be anything from a Minecraft game to Holographic Skype.
Some apps might be a combination of 2D and 3D experiences, with a traditional windowed interface from which you can launch a 3D experience, so the app switches between the two "modes" described above.
HOLOGRAMS DON'T LOOK LIKE THEY DO IN THOSE VIDEOS
I would argue they look even cooler than that. Those shot-on-HoloLens videos are made by digitally combining video footage from the camera on the front of the device with the rendered virtual images. The image that the user sees, on the other hand, is a combination of the real world image and light added by the device to create the holographic images. So the device can only add light, not block it, which means that black colors appear as fully transparent to the user. The Start menu for example, which has a black background in the videos, is actually transparent (with a blue-ish glow around the cursor that follows the user's gaze). White accents are used In order to add contrast to black or dark objects. All this makes holograms looks kind of ghost-like, and probably more like the fake promotional images than the videos captured using HoloLens.
These are some other limitations to the realism of the holograms: the lenses produce virtual images with an optical focal point that is 2m from the eyes, which results in an uncomfortable accommodation-convergence-conflict when looking at holograms that are much closer than that. It's one of the reasons why holograms are clipped off when they get within arm's length. And of course there's the small field of view, which takes away from the realism.
THERE'S MORE THAN JUST THE "AIR TAP"
We all know the slightly awkward looking "air tap" gesture for HoloLens, but in fact the device supports multiple input methods.
First of all, there's the "gaze". No, there's no eye tracking in HoloLens. Think of "gaze" as a laser pointer glued to your forehead, serving as a cursor that is always visible while using HoloLens (except while watching videos etc.). Just like the light of a laser pointer, that cursor appears right on the surface of the (real or virtual) object you're looking at. The appearance of the cursor (usually a circle) can change to indicate when an object is actionable or supports a voice command etc. Depending on the type of app, the cursor can "magnetically" stick to buttons or objects to make it easier for the user to select things, and the often jittery movement of the cursor can be smoothened by the app to make it easier to follow.
There are multiple ways to "click" on a button or object: While placing the cursor on top of a button or object you can either do the "air tap" gesture, click the Bluetooth "Clicker" that comes with the device, or say "select". Or, when you see a button within the system UI (window bars, start menu, keyboard, …) you can simply say the voice command that is written on the button, without having to move the cursor to it.
Apps can have their own specific voice commands for all kinds of actions (US English only at this point). "Select" and "Hey, Cortana …" work system-wide though. You can tell Cortana to launch or close apps, or to adjust image brightness and sound volume, among other things.
In order to move objects and sliders, scroll and pan through content, or zoom in and out, you either move your hand through space as indicated by arrows that appear in front of you, or instead you tilt the Clicker up/down/sideways, which does the same thing and is less tiring to the arms. The Clicker has a motion sensor built in to enable that feature.
Of course you can also freely manipulate objects (draw, move objects, …) by moving your hand through space.
Currently there is no way of directly interacting with virtual objects with your hands - your hands (or the Clicker) serve as remote motion controls. Also, mouse input is supported by the OS, but currently not implemented in the system UI.
For text input you can either use the holographic keyboard by gazing at letters and "air tapping" them, or select the voice input button on the keyboard. (When you need to type inside a 3D app, the app will temporarily disappear and the keyboard shows up, because the keyboard is a 2D app and can not run alongside a 3D app). You can also pair any Bluetooth keyboard with the HoloLens and use that.
HOLOLENS ALLOWS YOU TO SEE THROUGH WALLS
HoloLens has one really cool capability that has never been mentioned before: Since the device knows your space even when it can't see all parts of it from your current angle, it can potentially show you - or hint at - real or virtual objects that are behind a wall in another room, behind a closed door, or under a desk, by providing you with an x-ray-vision type of view. That way you could not only hear, but also see a notification going off in another room, or see enemies approaching around the corner in a game.
HOLOLENS IS MADE FOR INDOORS
Under certain conditions, either the visuals or spatial mapping can be affected. Those conditions include low light environments, strong sunlight, and empty spaces. Also, you need WiFi to create a "space" to use HoloLens in (you don't necessarily need WiFi to use it in an existing "space"). Mirrors can lead to false spatial maps. Moving objects can be handled well though.
Source: https://dev.windows.com/en-US/holographic/
TL;DR-version:
>> HoloLens allows you to fill your apartment and office ("spaces") with traditional Windows 10 apps in the form of virtual screens. The device remembers what your "spaces" look like and where you left your apps last time you used them. When you close an app, it collapses into a tile.
>> Actual 3D apps (such as games etc.) are launched via launch points (little 3D objects) that you can pin to your room alongside the app tiles. Launching such an app makes all your 2D screens disappear.
>> Holograms that look black in those videos look transparent to the user. There are no black or dark holograms.
>> HoloLens supports a variety of input methods, including motion sensors in the "HoloLens Clicker", and Bluetooth keyboards.
YOUR ROOM IS YOUR "START SCREEN"
You may have seen the holographic Start screen - the little window in front of the user's face - yes, it's there, but that's not really where your apps live. Your apps sit on your tables and walls or where ever you left them last time, ready to be used every time you turn on the device. Some of them look like windows, some of them look like tiles, and some are little 3D objects representing a holographic app.
Here's how it works: The first thing that HoloLens does when you use it for the first time, is scan the room you're in. You see a room scanning animation running over your walls and furniture as you move your head around so the device can scan all the corners of your room. When you think the room is scanned sufficiently for your needs, you confirm with an "air tap". Now you can bring up the Start menu by doing the "blooming" hand gesture and select an app. This doesn't open the app yet - it just gives you a little tile or 3D object that serves as a launch point for that app, and you put it anywhere you want in your room. From now on that app sits there permanently, until you move it to another spot or remove it. (Sadly, those tiles don't support live tile functionality at this point). That way you can fill your room, or your entire apartment, with apps, and you can put multiple instances of the same app on different spots within your place - so you can have a video player in your bedroom AND your kitchen. Not only that, but you can create multiple "spaces", each with their own sets of apps. What that means is, when you take your HoloLens to work and use it there, it will recognize it as a new place (because the available WiFi networks are different than the ones at your home) and define it as a new "space", which you can again scan and fill with apps. All those spaces are stored on the device separately, along with the information on room geometry and where you put your apps. The device automatically recognizes which one of your spaces you're currently in (by recognizing the WiFi networks).
TWO COMPLETELY DIFFERENT KINDS OF EXPERIENCES
There are basically two separate modes or ways of using apps on HoloLens:
One way is running Windows 10 apps on multiple virtual screens within your "space", which means that you're in your apartment or office that you decorated with app launch points, and by opening them they expand into flat windows with the familiar Windows 10 app UIs. Any existing Windows 10 universal app can be used in this way. On the top of the virtual screens, there's a bar with access to window management (resize, move or close window) as well as interaction tools (zoom, scroll, pan) and a back button. Unfortunately there is no true multi tasking in this version of HoloLens, which means that only the app you're interacting with is active, and all the other opened windows just show frozen screenshots from the last time you used them. (They can play audio in the background though, and perform various background tasks). Also, although you can place multiple launch points for the same app, you can only have it running once, so for example you can't have two separate messaging conversations going on on two separate screens.
Another way of using apps on HoloLens is by opening a launch point of a true 3D app (as opposed to flat universal apps), which automatically makes all the launch points and windows in your space disappear - 3D apps take over the whole experience, you can't have a 2D window running while using a 3D app, and you can't run multiple 3D apps at the same time. Even system notifications get replaced by audio notifications (by Cortana) while using a 3D app. Those apps can be anything from a Minecraft game to Holographic Skype.
Some apps might be a combination of 2D and 3D experiences, with a traditional windowed interface from which you can launch a 3D experience, so the app switches between the two "modes" described above.
HOLOGRAMS DON'T LOOK LIKE THEY DO IN THOSE VIDEOS
I would argue they look even cooler than that. Those shot-on-HoloLens videos are made by digitally combining video footage from the camera on the front of the device with the rendered virtual images. The image that the user sees, on the other hand, is a combination of the real world image and light added by the device to create the holographic images. So the device can only add light, not block it, which means that black colors appear as fully transparent to the user. The Start menu for example, which has a black background in the videos, is actually transparent (with a blue-ish glow around the cursor that follows the user's gaze). White accents are used In order to add contrast to black or dark objects. All this makes holograms looks kind of ghost-like, and probably more like the fake promotional images than the videos captured using HoloLens.
These are some other limitations to the realism of the holograms: the lenses produce virtual images with an optical focal point that is 2m from the eyes, which results in an uncomfortable accommodation-convergence-conflict when looking at holograms that are much closer than that. It's one of the reasons why holograms are clipped off when they get within arm's length. And of course there's the small field of view, which takes away from the realism.
THERE'S MORE THAN JUST THE "AIR TAP"
We all know the slightly awkward looking "air tap" gesture for HoloLens, but in fact the device supports multiple input methods.
First of all, there's the "gaze". No, there's no eye tracking in HoloLens. Think of "gaze" as a laser pointer glued to your forehead, serving as a cursor that is always visible while using HoloLens (except while watching videos etc.). Just like the light of a laser pointer, that cursor appears right on the surface of the (real or virtual) object you're looking at. The appearance of the cursor (usually a circle) can change to indicate when an object is actionable or supports a voice command etc. Depending on the type of app, the cursor can "magnetically" stick to buttons or objects to make it easier for the user to select things, and the often jittery movement of the cursor can be smoothened by the app to make it easier to follow.
There are multiple ways to "click" on a button or object: While placing the cursor on top of a button or object you can either do the "air tap" gesture, click the Bluetooth "Clicker" that comes with the device, or say "select". Or, when you see a button within the system UI (window bars, start menu, keyboard, …) you can simply say the voice command that is written on the button, without having to move the cursor to it.
Apps can have their own specific voice commands for all kinds of actions (US English only at this point). "Select" and "Hey, Cortana …" work system-wide though. You can tell Cortana to launch or close apps, or to adjust image brightness and sound volume, among other things.
In order to move objects and sliders, scroll and pan through content, or zoom in and out, you either move your hand through space as indicated by arrows that appear in front of you, or instead you tilt the Clicker up/down/sideways, which does the same thing and is less tiring to the arms. The Clicker has a motion sensor built in to enable that feature.
Of course you can also freely manipulate objects (draw, move objects, …) by moving your hand through space.
Currently there is no way of directly interacting with virtual objects with your hands - your hands (or the Clicker) serve as remote motion controls. Also, mouse input is supported by the OS, but currently not implemented in the system UI.
For text input you can either use the holographic keyboard by gazing at letters and "air tapping" them, or select the voice input button on the keyboard. (When you need to type inside a 3D app, the app will temporarily disappear and the keyboard shows up, because the keyboard is a 2D app and can not run alongside a 3D app). You can also pair any Bluetooth keyboard with the HoloLens and use that.
HOLOLENS ALLOWS YOU TO SEE THROUGH WALLS
HoloLens has one really cool capability that has never been mentioned before: Since the device knows your space even when it can't see all parts of it from your current angle, it can potentially show you - or hint at - real or virtual objects that are behind a wall in another room, behind a closed door, or under a desk, by providing you with an x-ray-vision type of view. That way you could not only hear, but also see a notification going off in another room, or see enemies approaching around the corner in a game.
HOLOLENS IS MADE FOR INDOORS
Under certain conditions, either the visuals or spatial mapping can be affected. Those conditions include low light environments, strong sunlight, and empty spaces. Also, you need WiFi to create a "space" to use HoloLens in (you don't necessarily need WiFi to use it in an existing "space"). Mirrors can lead to false spatial maps. Moving objects can be handled well though.
Source: https://dev.windows.com/en-US/holographic/
Last edited:
Facebook
Twitter
Instagram