What exactly is SensorCore? Answers inside...

[a5cent]

I've noticed a lot of confusion floating around as to what SensorCore actually is. I've been there too. It wasn't long ago that I felt every article I read about SensorCore contradicted the one I read before it:

• is SensorCore a hardware feature?
• is it software?
• is it based on the Qualcomm SoC (system on a chip) in our devices, or does it require special hardware from Nokia?
• why do only some Nokia devices support it?

The above questions are just as confusing to most of the tech press as they are to us. If you've ever wondered about any of this, then this thread is for you, but be warned, I'm the type of person that get's into the details. Here we go...

What does SensorCore do exactly?

Once enabled, SensorCore keeps a set of your smartphone's sensors running at all times, even when your device is in stand-by. Specifically, these sensors are:

• compass/magnetometer *
• accelerometer *
• gyroscope *
• radios that passively scan for WiFi hotspots and cell tower IDs in your vicinity, and use that data to roughly estimate your location.
• low power DSP (digital signal processor) with the ability to match ambient noise with pre-recorded phrases **

*	these three sensors are collectively known as motion sensors
**	SensorCore support for this sensor was announced at IFA 2014 in Berlin. This is the technology behind the "Hey Cortana" feature, scheduled to arrive with the Lumia Denim firmware update.

SensorCore monitors these various always-on sensors, interprets what they report, and for most sensors, records that data for the last ten days (although the user can wipe that data at any time). This all functions completely independently of any app, but as users, we need a SensorCore enabled app to visualize this data and process it in ways that makes it useful to us.

Note:

although SensorCore deletes data that is older than ten days, SensorCore enabled apps can make their own copies of whatever data they choose and store it for as long as they want

SensorCore is most often mentioned in the context of health and fitness apps. However, those are just typical examples of what can be done with that kind of data. Hey Cortana, or Geofencing are further examples of features enabled by SensorCore.

The take away from this section is that SensorCore is all about monitoring low-power and always-on sensors, and not about any specific type of app.

How do I get SensorCore running on my device?

Prerequisites:

1. SensorCore functionality is only available on the Nokia Lumia 1520, Icon, 930, 630, 635, 730, 735 and 830.
2. SensorCore is only available on devices running Windows Phone 8.1 and Nokia Cyan
3. SensoreCore requires that you've got "Location Services" and "Motion Data Collection" enabled. Both can be enabled in your device's Settings.

At that point you'll have it running, but to use it you also need a SensorCore enabled app. Bing Health and Fitness is one example.

How does SensorCore work and is it hardware or software?

SensorCore is both hardware and software. Roughly, these are the individual components that go into making a SensorCore enabled smartphone:

• the smartphone OEM must include the necessary sensors on their device. Without the sensors there is nothing for SensorCore to record. Not all sensors are mandatory however.
• all these sensors must be hooked up to Qualcomm's sensor engine (in some documents this is referred to as the sensor hub). This is a hardware feature included on some of Qualcomm's newer Snapdragon chips. It's purpose is to handle low-level sensor monitoring, basically determining if a sensor is reporting anything interesting at all, or if the device is just laying motionless on the table in a silent room. The point of the sensor engine is to do this while using almost no power. If a sensor does report something, the sensor engine will wake up the corresponding subsystem to process the incoming signals. Most often this is the DSP, but it could also be a modem or a CPU core.
• the smartphone firmware must be designed to interact with Qualcomm's sensor engine. For example, the firmware is the piece of software that does the recording and keeps that data available for 10 days.
• the OS must also provide a facility by which apps can access this recorded SensorCore data. This bit of software was also created by Nokia and is known as the SensorCore SDK.

Why does SensorCore not work on every Lumia device?

Either the device doesn't include all the required sensors (like the Lumia 530) or the SoC used in the device doesn't include Qualcomm's sensor engine (like the Lumia 520, 920, 925, etc).

Why does "Hey Cortana" only work on devices with the Snapdragon 80x chip?

Because only the Snapdragon 80x chips include a sensor engine with an integrated low-power always on DSP that is capable of continuous voice recognition.

Will "Hey Cortana" and other SensorCore features come to other non Nokia/MS/Lumia devices, like the HTC One for Windows?

Good question. If MS hadn't bought Nokia the answer would definitely be no, as SensorCore isn't native to WP, but rather an extension to the OS that resides in Nokia's firmware. Now that Nokia belongs to MS, my guess would be that it will, but it will come to Lumia devices first. But again, that is only my guess.

 

[Chris_Kez]

Excellent write up. How does the power consumption compare to Bluetooth LE, and how does Sensorcore fit with BT-LE and/or NFC as part of a holistic context-awareness system?
This really makes me want to upgrade and excites me more than anything else about the next generation of devices. But what kind of limitations/restrictions are there? I know for example that NFC was severely restricted in WP until recently, and there are still lots of things you can't do. So what might we reasonably look forward to right away besides a step counter?

 

[RumoredNow]

Nice break down. Very cogent and well written. Kudos.

I'm already enjoying how SensorCore is helping Bing Fitness track my steps and look forward to blurting out, "Hey, Cortana," at socially awkward moments.

 

[anon(123856)]

Very well done. Thank you.

 

[a5cent]

Excellent write up. How does the power consumption compare to Bluetooth LE, and how does Sensorcore fit with BT-LE and/or NFC as part of a holistic context-awareness system?
This really makes me want to upgrade and excites me more than anything else about the next generation of devices. But what kind of limitations/restrictions are there? I know for example that NFC was severely restricted in WP until recently, and there are still lots of things you can't do. So what might we reasonably look forward to right away besides a step counter?

Those are good questions, to which I wish I had all the answers.

Power consumption

Qualcomm doesn't specify power consumption specs for their sensor engine. They say only that it's affect on battery life is negligible. Based on sensor specifications, I suspect that all sensors + sensor engine together draws less than 2 mA. If those were the only things in our smartphones, a moderately sized 1500 mAh battery could power them for a month without recharging. BT-LE is rated at > 0.01 mA and < 0.5 mA depending on use case.

how does Sensorcore fit with BT-LE and/or NFC

At a hardware level, BT-LE is just another low-power always on sensor. As such it makes complete sense to have Qualcomm's sensor engine also do the monitoring, but I do not know if that is actually how BT-LE is setup in the newer Lumia devices, I just suspect it is, because that is how Qualcomm expects it to be.

On the software side things get much more complicated however. I have no idea what WP's current BT-LE software stack looks like, nor where it is headed. This is the thing: BT-LE serves a completely different purpose compared to the technologies discussed so far. SensorCore allows a device to sense its location, orientation, movements, and "hear" the sounds around it. It's about the device becoming "aware" of itself and its surroundings. BT-LE isn't about any of that. BT-LE is about sensing the proximity to other BT enabled devices, establishing connections to and communicating with those devices, and allowing OTHERS to sense YOUR precise location. That doesn't match up with the purpose of SensorCore. That is why I suspect those features will be exposed through software APIs unrelated to SensorCore, but again, I'm just guessing.

Still, it will be cool when our smartphones can communicate with our key-chains and wallets and remind us when we leave the house without them, or tell us where we left them after a night out frequenting the local pubs. The BT-LE feature I'm least enthusiastic about is location based advertising, but this is where BT-LE is expected to be used most. Gimbal is a company (partially?) owned by Qualcomm working on such technology, allowing retailers to detect when you enter the store, how much time you spend in front of which displays, and when you leave. Obviously, sending advertising to your phone will also be part of that.

I can't really think of how NFC fits into this. At least detecting an NFC tag isn't something that can be done passively, so it doesn't qualify as a low-power always-on application.

But what kind of limitations/restrictions are there? I know for example that NFC was severely restricted in WP until recently

SensorCore isn't a standard. It's just the result of engineering efforts by Qualcomm and Nokia. Lacking a standards specification, there isn't anything that we could hold it up against and judge the extent to which that standard has been correctly and fully implemented.

So what might we reasonably look forward to right away besides a step counter?

These are examples of how SensorCore data is being used in apps right now, but considering how new this is, we're really just getting started. Step counting is really just the simplest thing you could most quickly hammer out a SensorCore enabled app for, which explains why we see so many of them.

Currently, SensorCore records four different types of information:

1. Step Counter: Logs the users steps and distinguishes between running and walking.
2. Activities: Logs when a user was walking, running, driving (car, bike, whatever), carrying the device but rather motionless (watching TV), etc
3. Places: Logs the geographical places you frequent most often. Based on this data SensorCore can automatically deduce where you live and where you work or go to school.
4. Tracks: Logs the routes you've travelled. Basically the smartphone is able to deduce along which paths you prefer to get from A to B.

What will come of that all is only limited by developer's imaginations.

 

[Chris_Germany]

Is the SD400 as energy efficient as the SD8xx when i activate the whole motion sensor action?
This arcticle sounds like it?s written for SD800 (you mention DSP which isnt available for SD400, for example). So is my 830 with its SD400 also capable of all these features (yes, apart from the sound)?

Crap im too tired, dont know if you understand me. I dont understand myself in the moment but i try and press "reply"

 

[a5cent]

Is the SD400 as energy efficient as the SD8xx when i activate the whole motion sensor action?
This arcticle sounds like it?s written for SD800 (you mention DSP which isnt available for SD400, for example). So is my 830 with its SD400 also capable of all these features (yes, apart from the sound)?

Crap im too tired, dont know if you understand me. I dont understand myself in the moment but i try and press "reply"

This thread is not specific to the sd800. I need to fix something if that is what it sounds like. I just don't know what.

This applies to any device that supports Nokia's SensorCore SDK. As far as Qualcomm's sensor engine is concerned, the sd400 is just as efficient as the sd800, if not more so.

Apart from always-on listening, the sd400 sensor engine is also just as capable as the sd800. Any differences will be a result of Nokia omitting sensors on some devices. For the L830 this is not the case, as it comes with a full set of sensors equivalent to those used on the L930.

 

[Chris_Germany]

Ok thank you