Feedback on VR from seasoned video tech engineer

doublevr

The guy is pro in all sorts of conventional video over the past few decades.

Introducing him to SLR tech efforts. His feedback was priceless:
I am new to VR and viewers so I came in with no visual expectations but also with no "built-up" tolerance for whatever issues, discomforts and problems there are. It also means that my experience is limited to the Quest Pro, the only viewer I have. But I have lots and lots of experience with the technical aspects of video and a little bit of experience with video production from a distant past.
I watched a few hours of content, and here are my thoughts:
First and foremost, the resolution of the Quest Pro is very low. It sucks. If this is the best in class right now, then there is a lot of ground to go before VR can take hold.
It also means that whatever is shot, edited and streamed needs to match only that resolution. Additional resolution will only have negative effects (extra store requirements, slow editing, more data needed to stream, etc) and no benefits as it gets scaled down at the viewer. Obviously some additional resolution is needed for future-proofing, etc, but this is a separate topic.
I don't really know the FPS that the Quest Pro drives. But most content, SLR originals and Czech included, feel like 15 fps. They may be shot at 60 fps or higher (I don't really know yet), but somehow the presentation is happening at a much lower frame rate. The good ones (I bumped into just a few) feel like 45 fps, the really bad ones (majority) feel like 15 or even lower.
There are visual issues, at least with how I see it. Maybe because I am new to headsets and my eyes still think it is the real world out there.
Often I get broken 3D. As soon as I look to the side or sometimes not even very far to the side.
The biggest problem that I see is that the Z position of parts of the image don't correspond to what the eye wants it to be. I am not sure how to describe it, but I think it has to do with the angle that the eyes expect to make to focus on close objects. It feels to me that the camera shoots with parallel "eyes", but my brain commands the eyes to converge. The result is discomfort, the sensation of "this is wrong", and break of reality and seeing things out of proportion.
I think that visual quality could improve a lot if, somehow, it was possible to make movies with the correct convergence for the point of attention.
Another big issue for me is focus. A lot of the time the focus is off the main subjects, on and off through the movie. Various scenes have the focus plane away from where the focus distance should be. Granted the depth of field is enormous with these fish eye lenses, but somehow it misses a lot.
This is an issue with most movies from everybody (that I watched), but this is something that the Czech guys either pay more attention to or get lucky with.
Lighting seems to be an issue too. Apparently shooting in VR requires a lot of light. Well lit scenes look much better than scenes with less light.
Distortion is a big problem too.
A lot of the POV have girls way too close. I understand that what is in the screen is the reality because the camera is where it should be and the math is right. But something goes really wrong when the main object is too close.
There is distortion. The body of the model gets wide-angle distortion - faces become like elves with long noses, arms bend in unlikely directions, stomachs become bigger, etc.
Things appear too close - somehow they look closer than reality. Maybe because if they were really close the eyes would be converging, but if the eyes converge (and many times I had to make an effort to focus on infinity in spite of the brain wanting to get an angle) then the object appears very big. Maybe the brain uses information about the eye angle to determine perceived scale.
Also it is more comfortable, cinematically, to have a little bit more distance so you can see more of the scene without having to move the neck back and forth. So maybe some people use the scale control on the viewer, and then distortion really kicks in.
The orientation of the camera during some parts of the shooting is not ideal.
I understand that is how we see things in real life. But many scenes call for not looking straight. But it sure is more comfortable if the action spot is moved a bit towards the center of view. A few movies do so, and in my opinion that makes for a much better experience. Even though my neck is not looking down as much, the feeling is much better because there is no fighting with problems of the virtual environment. So tilt the camera down a little bit on the cowboy.
The video compression of many of the movies is not too great. I have not really analyzed the bitstreams, but it seems to me that many of the pieces could benefit from higher bit rate and better quality encoders.
If the stream is 30 fps, then 30 Mbps yields an average of 125 KBytes per frame (half of that for a 60 fps stream). So encoding quality matters a lot.
Mono 4K 30 fps content need 20 to 25 Mbps or more look good on a big screen. So 30 Mbps for stereo VR content is probably not enough and/or the quality of the encoding is not optimal.
The result is visible as "pasty" details, quantization artifacts, blocking, smearing and solarization. It is all around.
Unless I am looking in the wrong place or doing something wrong, the passthru chroma keyer is horrible. makes everything unusable. Is it possible to actually encode and send alpha layers?
Thoughts, anybody?

mirdumar

doublevr
Agree with spacepirate almost all of this have been stated before, i've almost never had problems with FPS, 60 always feels like 60 if it's shot in 60FPS.

The video compression of many of the movies is not too great. I have not really analyzed the bitstreams, but it seems to me that many of the pieces could benefit from higher bit rate and better quality encoders.
If the stream is 30 fps, then 30 Mbps yields an average of 125 KBytes per frame (half of that for a 60 fps stream). So encoding quality matters a lot.
Mono 4K 30 fps content need 20 to 25 Mbps or more look good on a big screen. So 30 Mbps for stereo VR content is probably not enough and/or the quality of the encoding is not optimal.
The result is visible as "pasty" details, quantization artifacts, blocking, smearing and solarization. It is all around.

With this i agree 100%, and people have been telling you that for years.

someoneX

doublevr The guy is pro in all sorts of conventional video over the past few decades.
doublevr The video compression of many of the movies is not too great. I have not really analyzed the bitstreams, but it seems to me that many of the pieces could benefit from higher bit rate

We told you the same thing many times. Maybe you believe a "pro".

Of course there has to be some compromise for file size and streaming, but 10-20Mbps more could help a lot.

doublevr

My reply:

While there are some inherent distortions at current level tech VR is definitely making it’s way and things are greatly improving over time. That’s the whole point of this discussion to make things right. We really want to have a clear idea what is already good and what has to be worked out.
Figuring convergence plane is a big deal indeed. Curious what would be the right way to approach it.
We want to approach “too close to the camera” issue with computer vision tools by changing the focus. It already works, yet some app performance issues are being fixed to make it into public release.
Also I’m very critical of current production as camera guy, director and post production teams aren’t synced on tech aspects. Actually no one is really taking wholistic approach, but rather dealing with separate matters. Something we really want to get fixed.

spacepirate

none of his observations were new tbh.. like we know the resolution is low even on the best headsets, we know looking anywhere but the center takes the images out of alignment, we know there are scale problems and distortion in general (and btw last time i used deo it had more distortion than heresphere), we know that scenes by czech studios have better focus (because they dont use the canon camera), we know well lit scenes look better, we know some producers still havent worked out optimal camera positions, and we know that video bitrates are too low (particularly on slr).

he mustve been doing something wrong to get such low framerates... maybe was watching 30 fps/interpolated scenes, using 60hz, or missing codecs?

doublevr Also I’m very critical of current production as camera guy, director and post production teams aren’t synced on tech aspects. Actually no one is really taking wholistic approach, but rather dealing with separate matters. Something we really want to get fixed.

this 👆 is exactly what slr originals needs

bacula8630

I like that you guys are seeking out opinions and observations from seasoned professionals, but I question some of his conclusions.

First off, he states any resolution beyond the native resolution of the headset isn’t beneficial. Now, this may be true for a fixed 2D panel. Where the viewer can see the entire screen in his FOV. But with a VR headset, we are only seeing a part of the entire image at any given time. Depending on the FOV of the headset. In the case of the Quest Pro, FOV is approximately 105 degrees.

So if a scene were to match the native resolution of headset. And any over is wasted and possibly detrimental, as the gentleman suggests. Then according to him, the optimal resolution of someone viewing on a Quest Pro should have a horizontal resolution of 1920 pixels per eye. But after that is projected on the inside of a hemispherical dome, we are only seeing 105 degrees of the that because of the FOV of the headset. Which is considerably less than the native resolution of the headset.

So increased resolution does have benefits to image sharpness. Of course, as resolution increases, there are limits to how beneficial it is. And at a certain point. Compression and bitrate outweigh any resolution gains. But I still see benefits with scenes up to 8K in resolution.

Secondly, I’m not sure about his observations about FPS. Was he using the headset standalone? Was he tethered to a PC? Sounds like he was using a PC with inadequate hardware. If he thought scenes felt like 15FPS. IDK.

His observations about double vision when looking off to the side isn’t really new. The focus feature on Heresphere somewhat compensates for that. I think SLR is working on a similar feature already.

Also, I don’t what scenes he watched. But his comments about camera distances seem to be be more about camera positioning for a particular scene. Versus a global issue with the camera set too close. And I don’t think filming at distances farther than reality is going to make for better scenes. I think that would just make for less realistic and engaging scenes.

Same thing goes with framing things towards the center. Regardless of how they subject would be framed in real life. Not sure if he used the tilt feature. But sounds like he doesn’t know about it.

I’m not questioning this gentleman’s expertise. But it sounds like he’s very new to VR. And I question many of his conclusions. I also believe a lot of his complaints can be reduced with more knowledge about the playback controls at his disposal.

Just my opinion.

BobF

About the resolution, of course I meant observable resolution via headset at any given time. Forr example (and I don't know the real numbers) if the headset displays 2 Mpix per eye and that corresponds to 50% of the total possible field of view, then of course the material needs to contain 4 Mpix to be optimal in optical resolution.
Compression is yet another factor. There is a penalty in bit rate to pay just for increasing the resolution. It will take more bits to encode pictures containing more pixels even if the pixels are visually the same, just because it is the way the codec works.

About the FPS, I was using the Quset Pro alone, directly connected to wifi network and fiber. I don't know what causes the low FPS, but something does and a lot of SLR and non-SLR content have "bumpy" motion. Interestingly enough, some content have smooth motion.

Although I am very experienced with non-VR video, I am indeed very new to VR. I am tryig to be useful by describing what I see. I would love to know more "about the playback controls at his disposal". Can you point me in the right direction?

I did feel weird when objects, particularly human objects, get distorted past some point. It just breaks reality for me. The issue of not being able to have parallel and angled convergence at the same time (so that viewers can see very near and far objects without double vision) is indeed an interesting problem that should be addressed somehow. Maybe the producer can minimize the occurrence of both very near and far objects in the same scene or somehow visually make the viewer's attention go where it should be. I don't really know. I do know that all these little things seem to add up to break the magic. Maybe shooting in VR just requires much more control than shooting 2D, in the sense that one needs to always mitigate the effects of users looking elsewhere, ultra wide depth of field and distortions that go with it, focal plane location, limitiations of positioning of the camera, viwers zooming at the headset and making distortion even worse, etc.

doublevr

BobF About the FPS, I was using the Quset Pro alone, directly connected to wifi network and fiber. I don't know what causes the low FPS, but something does and a lot of SLR and non-SLR content have "bumpy" motion. Interestingly enough, some content have smooth motion.

try switching resolution. 8K might be dropping frames indeed. the next lower should be fine

bacula8630

Hi Bob,

I hope you didn’t take any offense from my comment. I certainly wasn’t trying to insult you.

This may sound like a stupid question, but have you taken the time to properly adjust the IPD on the headset? You spoke of double vision a few times. But when properly adjusted, you shouldn’t see double vision looking straight ahead unless the object is within a few inches from the cameras. Anything from a few inches out to infinity should be clear.

Off to the side is different story of course. When looking straight ahead anything off to the side out towards the edge of your peripheral vision can appear with double vision.

Also, you said you were using the headset stand alone. I find that using it connected to a PC with an adequate graphics card yields significantly better image quality. If you have a PC available, I recommend giving it a try.

As far as “playback controls”, the tilt function is probably one of the tools I use the most. If the scene is framed in a way that the subject is too high or low, you can tilt the image to center it in your FOV. I don’t think I could live without it. I believe it is default to the thumbstick on the controllers. If it isn’t, you can program the button by clicking on the gear icon in the menu.

Speaking of the menu, when you bring up the shuttle controls, there should be an icon floating off to the right for picture controls. If you click it it bring up another window with image adjustments. But also, in the bottom left corner of the window it will show your current FPS. You should make sure you’re approximately 90FPS. If you’re not, something isn’t right.

Anyways, hope this helps.

spanker

so basically saying waiting for the new apple 8k headset might be a waste...just get the quest 3 ?

Padarak

spanker 8k resolution is just marketing bs. It has the same resolution as the hp reverb g2.

metichemsi

someoneX hopefully they didnt pay for that "advice"

Padarak

I am surprised that you do not know that.

The biggest issue basically all studios do is making vr porn traditional way.
What does it mean? For example the actor should not have tattoos at all. He should be as average as possible. So not fat and not super skinny. Also not super muscular looking like on steroids. The cock should be around average: ₁₆ cm / ₆,3 inch. No 10 inch cocks.

Resolution is fairly decent, quest pro is on the lower end but shouldn't look much worse than the top tiers. I am using VR since DK1, so I am used to it over the years.
Close up shots SLR can't do at all. It is like no one is viewing them, it is way too close. Virtualrealporn and CzechVR are leading here.