christian99
Newbie
- Dec 15, 2017
- 1
- 0
I need a two step scrape.
1. Iterate through thousands of sequential sites
like
mainclass?studioid=0001
mainclass?studioid=0002
...
mainclass?studioid=7676
...
mainclass?studioid=28695
Each result will be
Active - meaning they have an active subscription
Deactivated (no longer subscribed)
Invalid /mainclass?studioid=250027 redirects to website/?errno=60
Action:
A. capture the studioID, Active Status
B. Capture the Linked URL (it will be on the page studioid=28695)
Easy Peasy - the main site runs on classic ASP (!) and has no cloud flare protection or other protection. We did test scans of 1000 per hour with no issues, its a headache and i'd rather pay.
About 70% of the sites are invalid. Out of 0001-9999 i expect about 700 valid results.
Part 2:
for all the captured URLs, like bobsburgersdotWhatever go to the site and pull phone number and emailaddress.
leave it all in a csv with a few columns, divided into blocks of 250 valid results (we will call them as well as email)
I'd like a quote for either # of site IDs scanned (there's about 500,000 but we know they have 50,000 active clients) or # of emails collected.
I won't do 500,000 at once but if it goes well will scan 20,000 per month (or 2,000 active sites) or if it goes very well i will do them all at once and will have ongoing needs.
I can provide an AWS server if needed.
I can pay in crypto or USD, paypal, whatever. The project is on the Up and Up -
1. Iterate through thousands of sequential sites
like
mainclass?studioid=0001
mainclass?studioid=0002
...
mainclass?studioid=7676
...
mainclass?studioid=28695
Each result will be
Active - meaning they have an active subscription
Deactivated (no longer subscribed)
Invalid /mainclass?studioid=250027 redirects to website/?errno=60
Action:
A. capture the studioID, Active Status
B. Capture the Linked URL (it will be on the page studioid=28695)
Easy Peasy - the main site runs on classic ASP (!) and has no cloud flare protection or other protection. We did test scans of 1000 per hour with no issues, its a headache and i'd rather pay.
About 70% of the sites are invalid. Out of 0001-9999 i expect about 700 valid results.
Part 2:
for all the captured URLs, like bobsburgersdotWhatever go to the site and pull phone number and emailaddress.
leave it all in a csv with a few columns, divided into blocks of 250 valid results (we will call them as well as email)
I'd like a quote for either # of site IDs scanned (there's about 500,000 but we know they have 50,000 active clients) or # of emails collected.
I won't do 500,000 at once but if it goes well will scan 20,000 per month (or 2,000 active sites) or if it goes very well i will do them all at once and will have ongoing needs.
I can provide an AWS server if needed.
I can pay in crypto or USD, paypal, whatever. The project is on the Up and Up -
