Cool idea? The live transparent start screen

[Joe920]

So now that we have seen the nice parallax effect on 'transparent tiles' in WP8.1, I have an idea for a cool (but battery depleting) start screen effect that will make the phone look transparent.

Step one: record a live image with the rear facing camera. Part of this will be used as a live tile background.
Step two: process the front facing camera image to determine where the user's eyes are. This gives angle and distance (from eye spacing) to the phone.

Based on the user distance and angle, you can determine what part of the surroundings (rear cam image) the user would see if the phone was not in the way.

This should be enough to live update the tile background to make the phone look transparent. Cool! Creepy side effect: it will make your hand look transparent as well.

Any programmers in here that can slap this together?


Edit: example image where it's pretty obvious how you would fill in the tiles to make the phone look transparent. Bonus points for blurring the RFC image to match the user's depth of focus. Double bonus points for using the FFC to get pupil size to get the correct depth of focus.

 

[poiman]

I've thought about this before, but you said it, it would kill you battery too fast.

 

[Joe920]

I've thought about this before, but you said it, it would kill you battery too fast.

Maybe, although users don't spend that much time on the start screen, so the battery burn may be acceptable. Are you up for the challenge of coding this? Shouldn't take more than three years.

 

[vitor.rf]

Maybe, although users don't spend that much time on the start screen. Are you up for the challenge of coding this? Shouldn't take more than three years.

Sorry, I'm confused. You first say it's uninteresting because "users don't spend that much time on the start screen". Then you ask if he's up for the challenge AND add it shouldn't take more than three years.
Are you trying to talk him in or out of it?

Anyway, it would look "cool". But it would be like "oh, look at what my phone can do" "hey, nice". That's all

 

[Joe920]

Sorry, I'm confused. You first say it's uninteresting because "users don't spend that much time on the start screen".

I never said it's uninteresting. I think it would be awesome. I was just saying that the battery burn would be limited since people don't stay on the start screen for very long.

 

[Im_Q]

The great thing about software is that there are plenty of ways to cheat or take shortcuts to get a desired effect. The biggest battery drain would probably be the cameras running and the image processing. Why not drop the front-facing camera used for point-of-view processing and use the accelerometer? This was how the Zune HD did a subtle 3D tilt effect and then later on iOS 7 to a greater degree. This is the easy way of APPROXIMATING our relative vision and is used in many scenarios. The 3DS game system uses this in at least one of its game.

That would help greatly to make this work by mitigating some excess energy consumption, but let's talk about why it would still not work. The rear camera would need a pretty wide field of vision since we're going to crop away the video then panning it based on our relative vision. More high end cameras have the field of vision to do this, but the majority don't and that's the population that is the most important. There's also other issues with the fact of the video feed being capture from a 2D camera from fixed point. Even if the video feed is panned left/right/up/down to simulate our point of view changing relative to the tiles, the video isn't changing perspective. We're merely sliding the footage around under the tiles so the effect still won't be that convincing even if everything else worked as plan.

 

[Joe920]

Very thoughtful comments. Agreed on everything except maybe the field of view and perspective. Both play a role, but it might be relatively minor. Reason: you mostly use your phone looking at it within maybe plus or minus 15deg from the surface normal. That's plenty small angle to be captured by the RFC. With some artificial 'simulated depth of field' blurring I don't think at such small angles slight errors in perspective would kill the effect. I haven't tried it though, so I can't say for sure.