I see Meta are releasing an object occlusion for their passthrough. I was wondering if it would be possible to see your hands (etc) in front of the video when using the passthrough mode? The immersion levels will go through the roof if the model doesn't obscure your hands at least. Hope that makes sense.

I hope they can make it happen 🤝

I imagine this is possible with apple vision so lets see

It should literally be the number 1 focus! Be good to hear from someone with a vision pro where this will look amazing. But hopefully we can at least get metas rough version in some way. Everyone should be asking about this!

@"Repaelleinad"#p85383 for sure 👍

@"fenderwq"#p85394 we do have depth maps. Let's see how we utilise these

@"fenderwq"#p85394 @"fenderwq"#p85394 I had a fear it would be different on a video 😩

@"doublevr"#p85398 I'm looking forward to this please keep us updated

Occlusion Passthrough Update

fenderwq

It would be really cool but it will require the video to have depth information to work. Probably will need to be generated with AI if it's to work on all videos and then streamed along with the video as real time depth estimation for the entire video is not feasible in real time at the moment on a quest.

So, not a small request but I agree, definitely a worthwhile venture. Especially as that depth information can also be used to create proper auto focus.

doublevr

fenderwq we do have depth maps. Let's see how we utilise these

Repaelleinad

fenderwq

fenderwq I had a fear it would be different on a video 😩

doublevr

Repaelleinad for sure 👍

SexisAmazing

doublevr I'm looking forward to this please keep us updated

speedyrulz8

I agree with this. It's also the top feature I would be looking forward to.

Repaelleinad

I saw a review of 'stranger things' on quest. It has a mixed reality section, in the video you could see his actual hands over VR assets, seems like it basically just used the hand tracking for position to cut out the hands. We must be close to this...... Surely! It's next level stuff.

doublevr

Repaelleinad it's coming 🔥

iddqd

Quick research conclusion:

We might use dynamic occlusions to render video behind user's hands
It cannot be used to move your hand in the space of the video (let's say you want to move your hand behind one girl, in front of another) - that requires 3D objects, not stereo image. Requires deep research to figure out such usage. We might come back to this idea later.

Integration:

Requires heavy modification of the image rendering
Most likely will affect performance (FPS drop, can't predict how much)
Makes more sense for 3D objects and volumetric scenes with "real" depth

We will try to make simple integration with hands in front of the rendering mesh first and see where it can lead us 🙂
Stay tuned!

Vrsumo2017

So how will that work while fapping, the occlusion should be an option, turn on turn off. Toggle switch.

iddqd

Vrsumo2017 yeah, that's a good question 🙂
We will share experimental builds if we manage to make it not that intrusive

iddqd

Check this out. This is what can be done (potentially).
Let's see if we have enough interest to implement it.

Keep in mind:

Rendering performance drops
It's not really easy to improve quality (flickering on the edges, etc) without losing more performance
This only works on Q3 at the moment

DevAnimate

iddqd Perfect! btw can it be optional too?

BigTittyLicker

iddqd This is awesome and when can we expect this?

LOLBRUH

iddqd When can we expect this update to drop? It looks very promising

Hairsational

iddqd You already have a player with really bad performance and this gimmicky feature will only make it worse

fenderwq

Nice, already looks cool! Curious to try it out. It might not be perfect yet but tech has to start somewhere, meta will keep improving the algorithm, hardware will keep getting better, etc. So, it's a yes for me. Make it toggable though and also assignable to a hot key for quick switching.

Vrsumo2017

Looka good, will th8s be an option?

Vrsumo2017

On the real, that looks badass! 🥂🎉