Thanks for all the great input. I've downloaded a few download managers to try out.
why do my downloads not complete. vids cut off halfway through.
- Edited
Video encoding is not just about the images. For starters, the keyframe image header metadata is different with streaming optimized encoded video.
Have you ever encountered, even with a video on your local machine, you could not skip to a certain position, it would always skip to a couple seconds before or after?
You need far smaller chunks when skipping through a video when it's optimized for streaming. Decreasing the delay when skipping forward or backwards. For non-streaming optimized video, every time you skip forward or backward, you would need to download more data before the part you skip to would start playing.
This in turn changes the way the video data can be transferred. You can more easily take advantage of parallel downloading instead of sequential downloading. In many cases (it's not a rule of thumb) sequential downloads are much easier to deal with for both your system and browser. You're less likely to 'choke', less likely to run into connection balancing issues (ie windows auto-tune I mentioned above.) Downloading larger 'chunks' is easier for the system to reconstruct from cache.
And so on...
You might sometimes have noticed that after downloading large files with a browser, when the download is finished (or almost finished) there is a copying job being done. This is moving and reconstructing from cache, to the download destination folder. A download manager (much like a torrent client) provisions space directly in the target folder instead of going through the system (and browser) cache. Some download managers even let you pre-allocate space in the destination folder. This will make it much less likely for your connection or network adapter to 'choke' as it writes the data away immediately instead of caching it.
'Choking' you can see as the network cache being full, without having any complete chunks to pass on. Maybe counter-intuitive, this happens much more easily with smaller chunks as the amount of simultaneous open download connections is increased. (parallel downloading). Hence download managers also have settings for max simultaneous connections.
Some websites (NAVR) don't even allow you to use download managers because they are not set up to handle parallel downloading, and it hammers their servers too much. Which I can attest to. One day after taking out an annual sub with them, they blocked my IP for using a download manager.
For what it's worth, I even asked ChatGPT4 to review the accuracy of my statements, and both validate and invalidate the statements. https://chat.openai.com/share/d4e97591-252b-455b-9a18-d9cf0fcbefc7
I'm not here to make excuses. I believe when people understand how something works, or why something is the way it is, they are more likely be able to know how to handle issues. Instead of thinking it is something we have to fix.
'Fixing' this download issue some people have, would render the streaming service inoperable.
Simultaneously, for legal reasons, I can not recommend the usage of 3rd party apps.
- Edited
I'm not sure if this image makes it easier or harder to understand, but I'll try to explain the difference in encoding.
- The upper half of the illustration is streaming encoded.
- As you can see, the green frames are all neatly following each other. 1 2 3 4 5 6 ...
- 1 2 3 is a 'chunk', 4 5 6 is a 'chunk' (a packet of data that's sent from the server to you).
- Each key frame also all have their own metadata header containing the information of the frame. These basically hold the instructions on how to generate an image from the data.
- Because each keyframe has it's own metadata, the playback of the video can be jumped to very easily, and doesn't require downloading an entire GOP (see next part)
- The lower half of the image is not encoded for streaming.
- The first thing you'll notice is that they are 'smaller' because there is no metadata for each frame.
- Another thing you might notice is that here the frames are not neatly following each other, now we have 1 4 7 2 5 8 ... (3 6 9).
- This means that when you want to jump to that position in the video, you will need to download the entire GOP, all 3 packets, so the video player can put them in the correct order again (1 2 3 4 5 ...)
- The trade-off is that you don't need extra data for the frame headers. Here one metadata header is used for the entire group of pictures 1 to 9. Making the whole video file-size smaller.
This is a simplified representation.
A GOP of 250 is usually the default used, not 9. GOP literally means 'group of pictures'.
There is nothing we can do when it comes to download management on server side, that changes the construction of the video files. The most common issue for SLR users is network caching 'choke' of being full of incomplete packages. If you still run in to this issue while using a download manager, lower the amount of parallel connections in the settings.
There are still other factors too! ie buffered vs unbuffered drives, Shingled drives vs Conventional drives. But these would be edge cases.
Rakly3 Are the very large files for SLRO videos that are provided only for downloading encoded for streaming as well? This is not self-evident because I understand you cannot stream them even if you wanted.
Why not just encode a different "download" version of the videos and have the download link point to that?
Rakly3 Some websites (NAVR) don't even allow you to use download managers
This is not entirely true. When using their mirror server it works fine for me with a download manager. But they locked my account twice during a period in which my internet connection was unstable for having too many different IP addresses, so I know the feeling.
This is not entirely true. When using their mirror server it works fine for me with a download manager.
All I know is that I lost access within 24h haha
Are the very large files for SLRO videos that are provided only for downloading encoded for streaming as well? This is not self-evident because I understand you cannot stream them even if you wanted.
They are the same files from the same servers. There are just more choices for streaming.
- Edited
again you are talking a lot about video encoding and gops when that has nothing to do with issues experienced when downloading a video as a file in a browser or download manager (which is what this thread is about).
those two posts read like you have some understanding of a few concepts but its very evident that you dont really understand what you are talking about because most of what you said there is straight up wrong. i was tempted to correct some of the things you said i honestly dont know where to start.
Rakly3 The most common issue for SLR users is network caching 'choke' of being full of incomplete packages.
Is this not true? I don't know anything about computers but this at least made some sense of why it always stops downloading without a DLM.
rerun119 that sentence is nonsensical
downloading large files is less reliable and slower in browsers compared to download managers because browsers arent specialized for that specific task. browsers are designed to be efficient and "network friendly" so they only use a single connection for downloads by default. if anything goes wrong with that connection (e.g. because your internet provider or network equipment are garbage or the server sucks) your download will fail and the browser will give up. download managers dont care about being "network friendly" so they will flood servers with multiple connections and retry failed connections aggressively.
- Edited
spacepirate that sentence is nonsensical
You're saying pretty much the same as me however.
Browsers have a hard time dealing with this when downloading these files in such large sizes.
I'm trying to explain why users have this problem only on SLR. Downloading from a streaming server is different from downloading from a fileserver. Like you just said, browsers are not equipped to handle large segmented files. Why are they so much more segmented than from other non-streaming sites? What makes it possible to send the data much more "random" than sequential with more simultaneous parallel connections?
When using a video player to stream the files, you will get sent the data within a certain timeframe of the file. When downloading the entire file, you're getting sent parts from all over the file, Making it much more likely to run in to caching issues.
The reason I started explaining specifically about encoding is bc you said this
spacepirate dunno why youre talking about how videos are encoded when that has nothing to do with download issues.
While all i had said up to that point was
It isn't. Our streaming service would stop working if we 'fixed' this. There are many different ways in how videos can be encoded. SLR videos are encoded specifically for streaming. Browsers have a hard time dealing with this when downloading these files in such large sizes. It's a very broad topic involving even your network card and router, but it basically comes down to how the chunks are managed and cached. Download managers handle these completely differently.
The point being, that people are downloading from a streaming server. And what the difference is, because the streaming service would not work if we would not use streaming optimized encodes.
You make it sound as if I said that it's the encoding itself that causes the issue. - Which I didn't.
Rakly3 I'm trying to explain why users have this problem only on SLR. Downloading from a streaming server is different from downloading from a fileserver.
Rakly3 The point being, that people are downloading from a streaming server. And what the difference is, because the streaming service would not work if we would not use streaming optimized encodes.
when you say "streaming server" are you referring to a cdn (content delivery network) where the served files may not be fully cached?
Rakly3 You make it sound as if I said that it's the encoding itself that causes the issue. - Which I didn't.
not at all. im just not sure why you were even mentioning encoding in the first place. how a file is encoded is of no concern to a browser or download manager that is downloading the file. the encoding has no impact on downloading and cannot cause download issues.