Using a translator to translate 2 millions rows

[AndroidOnAppleDevice]

As the title said, any of you already used a translator to translate 2 millions rows from a database ?

Text to translate range from 0 to 500 characters more or less.

For now, I'm using rotating proxies to translate from a web interface, but it's so slow. I haven't found a better idea.

I have 20 browser open that use a translator with a new ip for each translation.

Would appreciate any help, thank you

 

[AndroidOnAppleDevice]

Up

 

[AndroidOnAppleDevice]

Up

 

[Elvlin]

just use gpt api to translate it than.

 

[AndroidOnAppleDevice]

just use gpt api to translate it than.

great idea, i haven't thought about that

it seems the translation quality is matching the one from deepl

thanks for the hint !

 

[Juhku]

You can also try Google Translate API.
They give $300 of trial funds, I believe, which should be more than enough for your use case.

 

[AndroidOnAppleDevice]

You can also try Google Translate API.
They give $300 of trial funds, I believe, which should be more than enough for your use case.

Thanks for the tip.

I already tried the google translate api though, and it doesnt really fit my use case.

The first reason is, I have 2 millions rows I want to translate, the free tier of google translate offers max 500 000 characters to translate.
If I was paying for the api, I would have a bill on average ~4 000€, which I don't want at all.

I set up a script to translate everything using a pool of rotating proxies, which worked pretty well, and only paid for the proxy (30€).

So far so good.

But what made me switched to deepl is the following reason, while I was translating my content, i logged every translation on my console, and I saw a few bad ones.

That's why I ended up with deepl.

 

[MKSKS]

Just get a bot coded that uses g translate with proxies.

Only think is there's a limit of 5k characters in one go.

 

[AndroidOnAppleDevice]

Just get a bot coded that uses g translate with proxies.

Only think is there's a limit of 5k characters in one go.

Yeah, I wrote one myself, but translation on google trad sucked a little bit.

That's why I did the same thing for deepl.

The main problem is, translating 2 millions rows takes too much time using a bot and the web interface, that's why I wanted to know if people had a better alternative

 

[MKSKS]

Yeah, I wrote one myself, but translation on google trad sucked a little bit.

That's why I did the same thing for deepl.

The main problem is, translating 2 millions rows takes too much time using a bot and the web interface, that's why I wanted to know if people had a better alternative

Multithreds on a powerful server or maybe on multiple servers can be the way to achieve that.

This will increase your expenses though

 

[sendlerad]

You can use a translation API like this one - https://rapidapi.com/gofitech/api/nlp-translation/

 

[AndroidOnAppleDevice]

Multithreds on a powerful server or maybe on multiple servers can be the way to achieve that.

This will increase your expenses though

Thats a great solution.

I do have a pretty solid computer imo :
- 16Go RAM
- CPU : AMD Ryzen 7 6800HS (8 cores/16 threads)
- GPU : Nvidia GeForce RTX 3060

I'm planning to use it, as I don't want to spend too much before earning anything though.

The answer of @Juhku seems good.

I found a free version of chatgpt on github (from blackhatworld), but from my understanding, the dude who created the repo use api keys of other companies.

The problem I see is, openai rate limit its api. So if i'm rotating between 8 000 proxies as i'm doing right now, I would probably make the api key rate limited.

 

[AndroidOnAppleDevice]

You can use a translation API like this one - https://rapidapi.com/gofitech/api/nlp-translation/

Thank you, I will give it a try, its cheap enough.

Edit:
I wont as I read a dude got f***. But thanks for the idea, I havent thought about using rapidapi.

https://rapidapi.com/gofitech/api/nlp-translation/discussions/91155

 

[Deleted member 1635563]

As the title said, any of you already used a translator to translate 2 millions rows from a database ?

Text to translate range from 0 to 500 characters more or less.

For now, I'm using rotating proxies to translate from a web interface, but it's so slow. I haven't found a better idea.

I have 20 browser open that use a translator with a new ip for each translation.

Would appreciate any help, thank you

2 millions is a lot

 

[Deleted member 1635563]

Maybe try a service that offer API and you do it in parts.

 

[AndroidOnAppleDevice]

2 millions is a lot

Yeah, compared to a normal usage its a lot, but we're on blackhatworld, i read people handle more content than that

 

[exec]

Multithreds on a powerful server or maybe on multiple servers can be the way to achieve that.

This will increase your expenses though

Not even. Multi-threading will add a lot of over-head due to resources allocations by each Thread. Making API calls is pure I/O operation locks, and you can leverage on asynchronicity to perform concurrent operation while the GIL lock (awaiting a request-response) happens.

 

[MKSKS]

Not even. Multi-threading will add a lot of over-head due to resources allocations by each Thread. Making API calls is pure I/O operation locks, and you can leverage on asynchronicity to perform concurrent operation while the GIL lock (awaiting a request-response) happens.

Exactly that why I said to use multiple servers which will also increase the expenses.
I prefer to do it this way but I don't have requirements for that many words or characters.
I still use it to generate content for few of my autoblogs.

 

[moonlighsunligh]

It is best to download fairseq, and run it on a runpod server. It is around 0.3 usd per hour for RTX 3090.

https://huggingface.co/facebook/m2m100-12B-avg-5-ckpt
Not sure what is the speed but you can check the quality there.

It is probably slow so maybe better to run this one: https://huggingface.co/facebook/m2m100_1.2B

 

[Elvlin]

Thats a great solution.

I do have a pretty solid computer imo :
- 16Go RAM
- CPU : AMD Ryzen 7 6800HS (8 cores/16 threads)
- GPU : Nvidia GeForce RTX 3060

I'm planning to use it, as I don't want to spend too much before earning anything though.

The answer of @Juhku seems good.

I found a free version of chatgpt on github (from blackhatworld), but from my understanding, the dude who created the repo use api keys of other companies.

The problem I see is, openai rate limit its api. So if i'm rotating between 8 000 proxies as i'm doing right now, I would probably make the api key rate limited.

With that spec, I think you can use Open source LLM model and run it locally on your PC, to do the translation. You need to test and trial which model works best tough/ or just fine-tune one for your use. then your cost will only be electricity.