• Apps

  • How to watch AI Passthrough vids with HereSphere or locally?

As a subscriber I'd like to have them as well.

I'm 99% sure there's no technical reason we can't download the AI passthrough files, it's just that they're security locked to the DeoVR app. I went digging around in the DeoVR cache files on the VR headset and found the URLs of the files (called 'pov_mask.asvr'), but a signature variable prevents one from downloading them in a regular browser.

Although HereSphere hasn't been programmed to make use of them yet, it might in the future, and I'd like to have them on file when the feature goes live for 'offline' viewing, as I prefer HereSphere to DeoVR.

I suspect this is a business strategy, and I don't like either. More of the "You'll own nothing and be happy" economy.

The new AI passthrough videos (not AR videos, those are different) are just like the old regular ones; it's just that the AI filter is applied in real-time on the headset itself. Most of them are even without alpha masking, so kudos to the team for making it perform so well even on the mobile chipsets and up to 8K/60fps resolution.

The issue for me is that the passthrough for the local videos doesn't render properly. I get either some semi-transparent picture, or letters of the studio that made the video for the entire length of the video, or simply nothing at all. Either some metadata (like a JSON file instructing the app on how to play videos) is left on the server, or the app just intentionally (or in a buggy way) handles the same video streams differently. I don't see what else it could be.

A similar thing was happening with the old videos where RGB masking/passthrough was applied on the fly. First, the videos needed to be manually renamed with "_alpha" appended to the filename, and then all alpha color parameters needed to be set like in those streaming videos (that were already predefined). Luckily, we have to set these only once, so the app caches it and uses it the next time. Just be careful with the "clear browser cache" option within the app; it's pretty aggressive if you ask me, and it could wipe those color settings too.

EDIT: correction on "the letters of the studio that made the video" mentioned above - it's actually this message:
"Watch the video with SLR app or DeoVR player", plus the filename and some other stuff that's hard to read for me.
Well, I'm using DeoVR ?! And I downloaded the video with my SLR account. So why the app is refusing to play it ?!

    BursVR I don't think it's true what you are saying about AI PT being generated on your headset for downloaded videos. I think what you did is activating alpha PT in DeoVR and what you are seeing is a result of that. You can hover with your pointer over the PT icon in the player UI to confirm that it's alpha, not AI PT.

        SchnuppiLilac Sorry, but where did I say about AI PT being "generated" on my headset for downloaded videos? On the contrary, I'm unable to make that work, and that's actually the issue I was describing in my initial post. I know I wrote a lot of things, but maybe I was misunderstood.

        I'm aware of the PT icon, and the video I downloaded clearly has "AI" beside the icon. When I'm streaming, it works, but when I download the video, it doesn't (and it says I should use the DeoVR app, which I am using to play it). The other ones that have "+" beside the icon are packed with alpha channel, and the old ones (call it 1st-gen if you will) are chroma-keyed, and both of these are working just fine with downloaded videos. Just not these new ones with AI.

        The AI filter works in real-time on the headset for sure (there are a few posts around), so I don't see the reason why it can't work with the downloaded videos too. I'm quite sure it's the AI videos I'm having issues with, because they are still not perfect, and have less precision cutting out the background than the ones with packed alpha channel, so it can be often obvious which is which.

            BursVR I understood your problem is that (you think) you have downloaded an AI PT video and it turns out that AI PT does not work when you play it. Did I get this wrong? What I was trying to say is that right now you cannot download the AI PT part of the videos with the AI PT icon. What you are downloading is just the regular VR video but not the mask that makes it AI PT which can only be streamed. And if you try to play these regular videos locally and you are getting the semi-transparent look or "Watch the video with SLR app or DeoVR player", it is because you switched on alpha PT in DeoVR which is not intended for regular VR videos.

            It is probably possible to make AI PT work for local videos but for this you will need to download the mask separately and this is not possible at the moment.

                Ok, I get it, what you say might very well be true. I thought this AI mask works the same way as alpha-packed MP4 files. But then again, when the AI model gets smarter, that would mean the videos should be reuploaded. And that would be, well, very inconvenient to say the least, considering the sheer amount of SLR content. And believing the headset itself did this heavy lifting with AI filtering was a little bit of naive thinking for me. So, it's probably that separate AI mask stream/file that I'm missing in my downloads folder. It makes sense. Now, let's find a way how to download the "offline" AI mask 🙂

                BursVR To clarify, I mean downloading the mask. I'm not sure if DeoVR allows local PT even if you have the mask files though.

                  BursVR : The AI passthrough mask is contained in a separate video file, and SLR appears to have put security in place to prevent subscribers from downloading that file. Maybe there is a technical reason the headset can't combine a local file with a streamed file in the DeoVR app...perhaps harder to get them to sync. But I would think it should be possible while maintaining the security. If both files were local, my guess is combining them would be easier than syncing the 2 streams. If you jump around the cuepoints in a AI passthrough video, you can see the streams go out of sync for a moment.

                      sunkenlust They should release the AI Passthrough tech to other studios on licence or even to users so they can create their own masks of previously downloaded files.

                      I have so many scenes that would benefit from a passthrough mask and i know a 'Drive' is coming, but that will have too many restrictions due to the size of uploads and ongoing costs involved, server space and the usual price increases with this sort of thing.

                          SchnuppiLilac The chances are that it is against the ToS, but that has never stopped people from consuming content how they want to. Take YouTube Vanced, for example.

                            Silly-Sausage : Here's another idea...I put my SLR credentials into HereSphere and then I get access to all the AI passthrough matte streams, which the HereSphere combines with the videos. Win-win, SLR gets their subscription money for the tech they've developed, and I get to use the VR player I want.

                            Of course the HereSphere dev would have to build this in to his app, because it doesn't have this feature yet.

                                Silly-Sausage : Maybe. AI background removal seems to be everywhere right now, getting incorporated into smart phone photo software and desktop editing software. Maybe SLR is even using one of the free open source tools out there. If SLR won't let subscribers download their matte files, maybe we can start making our own because the tech will be so ubiquitous. HereSphere would still need development work to make use of them in the future.

                                    sunkenlust If SLR won't let subscribers download their matte files, maybe we can start making our own because the tech will be so ubiquitous

                                    Too be honest i'm too lazy to be bothered with all that stuff these days and if somebody does something well then i'm happy to pay them for it, for now

                                        Silly-Sausage : I thought I'd run some tests using an AI powered background removal After Effects plugin. On the lowest (least accurate) setting 1 hour of 4K 60fps footage takes about 6 hours of rendering. This is on an M1 MacBook, still considered a fairly fast computer (even though 4 years old). So it takes a bloody long time to make these mattes. Now I know why it's taken some time for SLR to go through their library; they probably have a small farm of computers working on this. Not so easy to do it yourself.

                                            sunkenlust I don' t think it's worth it for me, if i was in my 20's and excited to be involved in tech when it was genuinely exciting then maybe i would look into it, but i'm now at the age where i understand life and getting away from tech is important, so i'll let others waste their life and get paid for their time.