I tried scrapping JustDial.com through various means but was not successful at all. I even used Cloud Scraping by OctoParse that rotates new IPs every 30s.
The problem is that it seems like they do not have a sitemap containing list of all businesses.
But there are links like this which are available on Google searches, which means they do have a sitemap -
https://www.justdial.com/Mumbai/Chemists/nct-10096237 ("Top 100 Chemists in Mumbai - Best Medical Drug ... - Justdial")
I want to find a way to scrape all business information, for which I will need the links to all pages hosted on Justdial.com. PLEASE HELP.
The problem is that it seems like they do not have a sitemap containing list of all businesses.
But there are links like this which are available on Google searches, which means they do have a sitemap -
https://www.justdial.com/Mumbai/Chemists/nct-10096237 ("Top 100 Chemists in Mumbai - Best Medical Drug ... - Justdial")
I want to find a way to scrape all business information, for which I will need the links to all pages hosted on Justdial.com. PLEASE HELP.