Subtitles for Videos?

dudder

Hi, I was just thinking that it would be nice to have subtitles added for videos. There are lots of JAV videos that look interesting but they're all in Japanese so it's hard to follow. Having the dirty talk and plot would make it hot!

One tool that could automate this is OpenAI's Whisper. It's like it was designed for translating porn, haha. It can translate to different languages and output vtt/srt for subtitles.

I actually got rolling on doing this for some old videos that I had on my computer. They were in German and I always wondered what they were about. The content was good but they became 1000x better when the subtitles were added in.

The translated results were okay. Sometimes they would do little German language goofs like "tail" vs "cock" or "puddle" vs "pussy". The models probably weren't trained on porn videos. It might be worth tweaking it with reinforcement learning or something to get better results.

Optimizing aside, it seems like this could be low hanging fruit so excited to see what you all think!

ZENRA

dudder I believe there is in some internal discussion about this. From my opinion as someone who has subtitled over a thousand movies and counting, the auto-solutions look a bit promising for extremely simple movies like VR since usually only one person talks and the audio is crystal clear. Still, the steps to produce auto-generated captions are a bit cumbersome and when you factor in necessary proofreading for both accuracy and compliance reasons along with ensuring all lines were actually captioned, you're still looking at a minimum of 2-3 times the length of the movie in human resources. This is a big issue because these types of simple VR movies can probably get extremely accurate and professional-quality subtitles in the same or less amount of time. I think this is the reason huge sites like CrunchyRoll employ many (sadly underpaid) subtitlers rather than letting machines handle it. They did the math and both financials and QC point to humans.

dudder

That's a good point, thanks for the detailed response! I think you're right. Consistently getting production quality is still probably out of reach for a ML system. Whisper is definitely a step forward but yeah it's probably still not quite there yet.

In the whitepaper for the project, they mentioned that Whisper reached human level accuracy. I'm not so sure about that but it is pretty close. It's the best open source model so far for sure.

It could be better than nothing if most of the translations are okay. Like you mentioned, the main danger is that if the quality is really bad on some videos (like pokemon green level translations), then it could make the overall experience feel unprofessional. No idea what the QC process would look like for that at scale...

Maybe it's still worth testing it out just to see whether it's totally terrible? I'll try running through a couple videos to sample it and post back the vtt files. Hopefully it's not a total lemon. 🤞

ZENRA

dudder From my viewpoint, subtitles should help with immersion and this goes triple with VR where full immersion is the name of the game. Suddenly getting a weird translation, seeing weird timing, or totally missed lines may totally kick the user out of the fantasy land they erected. Thus, less is more: a potential auto-generation option for users who really want it combined with significantly less but professionally-translated captions for more popular movies.

dudder

Wait, that's probably a bad idea to post the actual vtt files since it would break copyright on the video. I'll try running it locally and say how it went in general I guess 😅

doublevr

Subtitles are coming to SLR. Not sure what's the best way to get it from. Will do some auto-generated stuff with an option to upload your own

Bucket619

doublevr do you mean like automated vs a human doing translations?

Sandi_SLR

Yea, we just added single subtitle for local files option, you are free to test it, we will follow up with support for streaming videos and multiple subtitles.

dudder

Thanks for the update that sounds great! I'll try it out and see how it goes.

Just using Whisper alone gave so-so results. It wasn't very good quality.

I tried running MDX-NET first to pull the voice out and normalize the audio before passing it into Whisper. It still had vocab issues but it was a lot better. Not professional quality but okay for personal use.

@ZENRA I totally agree that the final subtitles should be bug free. Popular videos are probably worth humans helping with the subtitles. For less popular videos there might be some kind of middle ground where the first pass gets auto-generated and then QC people can tag and fix the problem spots. I don't know, it's hard to make it 100% human or 100% machine generated. Looking forward to proper subtitles though!

spanker

seen a few good ones over the years with JAV stuff...sometimes only needs a couple well-placed subtitles and it makes the clip.