emister6

I'm a bit late to the discussion, so apologies if you all already discussed this, but are there not multiple software packages available that can chroma key out the background of a human subject even without a green screen? Hell, even videoconferencing apps can do it realtime now. And isn't the objective of passthrough in SLR to almost always key out just the background and allow users to experience the human actors in their room?? So then why not just run videos through keying software first, embed the key layer into the video file, and then the passthrough toggle in SLR can toggle it on or off? Then rather than having all these sliders to fiddle with, you just toggle ON/OFF, and then maybe there is an edge width and strength sliders, like in the Touchly app. This seems totally possible, and if SLR was the first to start doing this post-processing step, rather than the studios themselves, then it would give users an incentive to use SLR over other services.

I've been a long-time passthrough enthusiast, and while the initial integration was cool, and fun to play with, it already seems majorly outdated. Its too much work for the user to dial in a decent setting every time, and it feels time to integrate a proper passthrough toggle that ensures that you are viewing a clean and professionally done video effect.

I dug into this a bit more last night and it appears there IS already an example available for an AR player like this available on Sidequest. It is currently limited to flat MP4 playback, and keying for only black, blue and green color values, but It works really well!

https://sidequestvr.com/app/5982/hologram-vr-show

@doublevr I have some ideas around this implementation as I have played with it a bit, and tried a few things.

First, I think it would be great to have a toggle in the settings, that would switch passthrough mode ON/OFF for the main menu background. Much like you can toggle this on the Quest home environment. This would create more spatial awareness while browsing, and generally be just a cool video watching environment on its own.

For the player, I think there is a cool way to do it, that should be fairly straightforward to implement. In the future I'm sure the studios can handle this on the production end, run software to key out backgrounds and embed occluder masks and everything will be perfect. In the meantime I think there are some achievable effects that would be worth having in the app. So ideally you would want to start with source footage that was shot on a single color background for use in AR (there may also be algorithms that can do this in Unity without a clean background) and then use chroma keying to create an occluder mask to view the passthrough. So a relatively low tech way of doing this would be a color picker to choose a color value from the video, and then a key slider to adjust the amount of keying, and finally maybe even an edge blur to soften the edges a bit. When combined with the contrast and saturation sliders, I think this kind of filtering would provide a lot of functionality and a very cool first proof-of-concept for AR video that I havent seen anyone else tackle yet. Like I mentioned though, if there is a decent way to recognize and key out human figures, rather than just color values, this may be the better approach. Check this video example HERE to see a bit of what I'm talking about.