I need help with scraping phone numbers with python

[ekuin0x]

I wrote a script in python to get phone numbers of people by country, city, job/keyword, the problem is: it's taking an hour to scrape only 150 numbers and I have no idea what's a legitimate size of a phone number lists.
I am new to this and I'm hoping to sell the data to someone who would be interested in buying it, I would like to know an average list's size to see if I should buy proxies to fasten the process or should I just wait few days to collect the data.

 

[Dmaxxy]

I wrote a script in python to get phone numbers of people by country, city, job/keyword, the problem is: it's taking an hour to scrape only 150 numbers and I have no idea what's a legitimate size of a phone number lists.
I am new to this and I'm hoping to sell the data to someone who would be interested in buying it, I would like to know an average list's size to see if I should buy proxies to fasten the process or should I just wait few days to collect the data.

did you use multiple threads or just single thread?

 

[Deleted member 1701249]

I wrote a script in python to get phone numbers of people by country, city, job/keyword, the problem is: it's taking an hour to scrape only 150 numbers and I have no idea what's a legitimate size of a phone number lists.
I am new to this and I'm hoping to sell the data to someone who would be interested in buying it, I would like to know an average list's size to see if I should buy proxies to fasten the process or should I just wait few days to collect the data.

In order to turn any profit you need thousands of numbers , multiple tens of thousands , if i was you i would use multiple threads and find more sites to scrape numbers from and run the script 24/7

 

[MgicalaEarner]

Use headless browser and multithread.

 

[Dunston]

I'd recommend python's async/await feature since there will be a lot of time waiting for HTTP requests to return.

 

[ekuin0x]

did you use multiple threads or just single thread?

I'm using single threads cause I don't have proxies, If I use multiple threads my pc will be detected and google will ban my IP

 

[ekuin0x]

In order to turn any profit you need thousands of numbers , multiple tens of thousands , if i was you i would use multiple threads and find more sites to scrape numbers from and run the script 24/7

That was very helpful, I'll give it a try

 

[ekuin0x]

Use headless browser and multithread.

I'm using headless chrome but I'll give multithreads a try, thanks

 

[ReverseEngineering]

I'm using headless chrome but I'll give multithreads a try, thanks

Don't use any browser emulation, that's not required and takes a lot of resources.

Use pure HTTP requests instead.

 

[ekuin0x]

Don't use any browser emulation, that's not required and takes a lot of resources.

Use pure HTTP requests instead.

Scraping Google is not possible with pure http requests, it returns javascript code even when using the right request headers

 

[ReverseEngineering]

Scraping Google is not possible with pure http requests, it returns javascript code even when using the right request headers

And what prevents you from parsing JavaScript code without an oversized browser emulation?

 

[ekuin0x]

And what prevents you from parsing JavaScript code without an oversized browser emulation?

What do you mean by parsing javascript? , I usually get the html code and parse it to scrape the data.

 

[ReverseEngineering]

What do you mean by parsing javascript? , I usually get the html code and parse it to scrape the data.

You asked about the JavaScript that Google returns above:

https://www.blackhatworld.com/seo/i...one-numbers-with-python.1563098/post-17222452

So I answered.

 

[Lipstick Spoiler]

I wrote a script in python to get phone numbers of people by country, city, job/keyword, the problem is: it's taking an hour to scrape only 150 numbers and I have no idea what's a legitimate size of a phone number lists.
I am new to this and I'm hoping to sell the data to someone who would be interested in buying it, I would like to know an average list's size to see if I should buy proxies to fasten the process or should I just wait few days to collect the data.

Where do u wish to sell it?

 

[ekuin0x]

Where do u wish to sell it?

Many people online showed interest in my lLinkedIn phone lists, I'm willing to sell to them