• SLR

  • [Updated videos] SLR Depthmaps in the making

Hi! I’m reseacher from SLRLabs team and we want to improve the VR experience for you. While we look at the VR videos, you may notice some artifacts making the VR experience uncomfortable. Most of artifacts are caused by inconsistent position of stereoscopic camera objectives and eyes. Here I’ll explain the types of that artifacts, where it comes from and how we work on fixing it

  1. “I have to close one eye in closeup scenes!”. On close-ups the faces are really closes to camera. Our eyes are comfortable to track the objects and if something is close to your nose, usually the eyes are crossed and not looking straight ahead, so the interested objects are located at center of each eye field of view. The stereoscopic VR cameras have larger field of view but no moving parts, so the image remain static.

As a result, person instinctively crosses the eyes when something goes close to nose, but the image remains unchanged from stable cameras. Mind is not able to process that images, so usually people feel eyes strain and tend to close one eye. The panoramic stereocameras usually have larger field of view, than eye, so it is possible to fix that issue by rotation of the projected image during runtime. The rotation rate should correspond to eye position and can be taken either from eye tracking device (most of headsets do not have that ability for now) or calculated from the depth of point at central area of sight and interpapillary distance (IPD) using simple geometry.

  1. “When I shake my head, the image goes wild!” In reality when people shake their head, one eye becomes accidently lower, than other one, so the visible objects are going up or down depending on the distance to them. Video from camera objectives remains static.

    The solution to fix headtilt is straitforward motion of left eye and righteye videos up and down at the rate dependent on depth and head tilt angle.

  2. “I cannot look at sides!”. The sides problem is the biggest one from the described ones. If you rotate your head, the eye position in virtual world will significantly differ from camera objective positions. If you look at far objects, the distortions are neglectable, but not for close objects.

    The angular size for close objects may vary a lot for left and right image. The eyes do not see such difference in real world. So the scaling (in general, more complex deformation) of image should be applied depending on distance to eyes.
    Each of corrections require the information about the distance to the objects, so the depthmaps could be precomputed and streamed among with video or calculated on a fly during videoplay. Than we can correct each frame depending on the depth information and head position. So, the streaming with depth information will be helpfull to remove annoying artifacts. Check it out with a test SLR app build for the Quest with the following videos:
    https://insights.sexlikereal.com/videos/SLR_ASM2_vgg19_MIDAS_decoded_MKX200.mp4
    https://insights.sexlikereal.com/videos/SLR_LAUNDRY_vgg19_MIDAS_decoded_MKX200.mp4
    https://insights.sexlikereal.com/videos/SLR_HAUNTED2_vgg19_MIDAS_decoded_MKX200.mp4

Here you can download testing app with such corrections applied and play with correction rate for them. Here is some videos with depth maps streamed using Test build SLR compatible format. The rate of corrections is in options tab.

What else can we do with depthmaps?
Despite a lot of research dedicated to implicit encoding of 3D scenes (mainly form NVIDIA), most of such methods are unacceptable for dynamic scene streaming for now. I believe that explicit encodings including depth maps are the vital step for 3D video scene reconstruction and streaming the scene in walkable (in some degree) format. We also work in that direction with some progress. Looking for response abot what else can be fixed for better VR experience

    doublevr changed the title to SLR Depthmaps in the making .

    Very interesting, I guess one question is how reliable it is to make depthmaps out of pure 2D images, at least as far as I know there are now stereoscopic cameras with TOF sensors, and Lytro is not existing anymore as well as far as I know.

    Besides that the biggest problem I see is when looking to the side, but in a lying/leaning position. Then the eyes are basically on top of each other vertically, while the camera lenses recorded the scenes obviously have been horizontally. This is actually an even bigger problem then the mentioned point 3. But I am not sure if depth maps are enough to counteract it, as there basically is no horizontal parallax information in the images. I guess you´d have to record with 4 lenses and blend between the images according to the head position.

      I want to contribute my grain of sand waiting for someone to serve you. I used to have focus issues when the camera was too close, double vision, and other issues you mention. I thought it was a problem with the videos until a few months ago I tried hereSphere on my Quest 2, and I can say that it is a huge change. It's like jumping from gearvr to Quest 2, I don't understand how a video player can make such a difference. now I can focus perfectly even when the girl is too close, in 8k videos I can clearly see the details of the iris in the eye, it's absolutely crazy. I totally recommend it!!

        Smilyy This is because Heresphere gauges the depth of what is in the center of your view. Then based on the lens distortion profile it knows how to uncross your eyes.
        This works really well and makes close objects, object to the far left/right perfectly watchable. It also corrects the crosseye effect you get from tilting your head and I noticed that it often makes the scène look sharper / beter focussed overall.
        You do get the occasional glitch but this is due to the lack of eye tracking (it estimates the depth based on the center of your view).

        The above is what made me switch over to that player too (and that it is way faster in seeking).

        That being said. I'm really curious how the depth maps will work out and how it compares to the heresphere method. Very cool to see al the initiatives and innovation going on at SLR!

        (By the way, please don't make this a streaming only option. That would be very dissapointing)

          ibins For sure the directly measured depth can be useful. But not all of studios has that equipement. As for the DMs from pure 2d - that is not wery reliable, though there is some success in that field. But in case of stereoscopic depth estimation the problem boils down to stereomatching -how to find the regions on one image that corresponds to the other image regions. At stereomathcing the precision is quiet high. Some of big self-driving car companies are even shifted from separate depth sensors to estimation of depth based on several cameras since higher reliablilty and lower cost.
          As for non-standard head position the corrections with DMs can help to fix some of inconsistency, but evidently stereoscopic effect will be lost. Problem of free camera motion can possibly be solved using on-device rendering of reconstructed (in some degree) scene and DM could be helpful for that

              AlexSLRLabs

              AlexSLRLabs Some of big self-driving car companies are even shifted from separate depth sensors to estimation of depth based on several cameras since higher reliablilty and lower cost.

              True but they do it only for the lower cost, they still can not get the reliability of dedicated LIDAR sensors.
              But I guess if it is good enough for self driving cars, it should be good enough for port too

                  doublevr changed the title to [Updated videos] SLR Depthmaps in the making .

                  Thx AlexSLRLabs
                  Great job on that 🔥🔥🔥

                  Currently depth might not exactly be obvious, but

                  1. it makes it easier for eyes to focus. Many videos were so fad you couldn't watch it. That should be fixed
                  2. it removes distortions where things are doubling.

                  Once again it's a huge ongoing process. We just started. There's a lot more to come

                    I'm really impressed with the new videos.

                    What we should do is automated switch to mono feature where users can set a value when the object gets too close to the camera. It's going to be a beta just to explore how things work.

                    We definitely can bring things to the next level. It's only beginning

                      doublevr

                      I am not sure about that, at least for me looking at mono becomes very stressful in close ranges, way worse then the cross-eyed thing you get with stereo which seems to be much more natural

                        Btw the best way to see the difference is to pause a video where a girls gets right into the camera and turn auto focus on and off. We will add a biding to controllers mapping with the next release.

                        I don’t know if this is depth map related, but if you zoom all the way in or all the way out, it becomes apparent that the video image is being distorted on the inner surface of a sphere to simulate a “zoom” rather than actually just translating the image closer or father. Would this effort also address this type of distortion from zooming? If not, I have a request to be able to zoom in an out without such distortion.

                        Vrsumo2017 Idk quest cannot see this file, installing from sidequest has a error "A task failed. Check the tasks screen for more info." help 🙂

                          This is beta version of SLR App or just for testing depthmaps effect? Can this function apply for other video?

                            RTHK is depth maps demo.
                            Currently we generate dm for each video, working to get it automated in the player with real time rendering.

                                RTHK right now it's for those download videos posted above. We will feature some few streaming videos with the next update.
                                The focus it to make the player create depth maps on fly for every video, both downloads and streaming