Using a translator to translate 2 millions rows

exec said:

Not even. Multi-threading will add a lot of over-head due to resources allocations by each Thread. Making API calls is pure I/O operation locks, and you can leverage on asynchronicity to perform concurrent operation while the GIL lock (awaiting a request-response) happens.

Click to expand...

If I was making only api calls it would be way faster. That's why I did at first.

Then I switched to scrapping the web interface of deepl with puppeteer (I'm using nodejs btw).

It's a bit slow bc I need to open and close a web browser after each translation in order to use a different proxy each time.

There's better way to do that though, for example having the same browser open all the time, and use a proxy per page, instead of a proxy per browser, but I went for the quickest way to implement to me.

The best would be either :
- making api calls only (don't know any way to do that without having an api key)
- self host the translation algorithm (then the question becomes, which algorithm is at least as good as deepl to self host)

MKSKS said:

Exactly that why I said to use multiple servers which will also increase the expenses.
I prefer to do it this way but I don't have requirements for that many words or characters.
I still use it to generate content for few of my autoblogs.

Click to expand...

What's the maximum total characters you needed to translate ?

moonlighsunligh said:

It is best to download fairseq, and run it on a runpod server. It is around 0.3 usd per hour for RTX 3090.

https://huggingface.co/facebook/m2m100-12B-avg-5-ckpt
Not sure what is the speed but you can check the quality there.

It is probably slow so maybe better to run this one: https://huggingface.co/facebook/m2m100_1.2B

Click to expand...

I will give that a try today.

You can make a http request directly to the inference api for prototyping they say, which is cool.

Elvlin said:

With that spec, I think you can use Open source LLM model and run it locally on your PC, to do the translation. You need to test and trial which model works best tough/ or just fine-tune one for your use. then your cost will only be electricity.

Click to expand...

Yeah, that's what @moonlighsunligh said.

I will compare the translation quality VS deepl.

AndroidOnAppleDevice said:

What's the maximum total characters you needed to translate ?

Click to expand...

Max 2k words maybe a bit more.

My use case is autoblogs so I hardly touch the max it's around 1000 words average.