dummerfranz
Newbie
- Aug 26, 2016
- 16
- 2
Im in need of a setup for web crawling
BHW Community | I need your help
https://www.google.com/shopping/product/15441182589886828859/online?q=nike&newwindow=1&rlz=1C1CHBF_deDE891DE891&sxsrf=ALeKk02B3VGibi_0xDWt4Dr11faGzzaxEg:1588968660842&biw=1176&bih=959&prds=paur:ClkAsKraX7UAvI_UK5daydTsvDGc8wLd2Uvr85i_-09PHWfJvRTFZ2vZAj2qcvV9mtYCt8Npbt2VyWLbLBrHJ2ryY0cZku1zDllJ_wzgAQbtk1K27bmYv7bZJxIZAFPVH71iVUznMdZnEQ-Eye5no0_qLHo5Ew,scoring:tp&sa=X&ved=0ahUKEwj88-LGiaXpAhVLRBoKHW9qAVUQ0ykIPg
this is a sample link of a random product from google shopping.
Its a pricing table which is showing data about the competition. For my project i need to get info about the best price (lowest endprice with shipping costs). Furthermore i want to scrape thousands of these google shopping price tables, so i need a tool which is able to work multi-threaded and with proxies. On my journey i couldnt find any tool which is able to do this. therefore im asking this community to help me out. Would be great if there is any program which is able to parse the information directly into an exel or csv, etc. file.
im aware that google proxies will may cost and that free proxies may be blocked. So in the first step im interested in a working tool that can output the information i need in a fast and reliable way. It should check if the price table is working -> block check or some way of checking if data is fetchable.
I tried to setup scrapy or other python based programs but all of them where not able to work for me or werent able to work multi threaded or have proxy support...
thanks in advance for your help!
BHW Community | I need your help
https://www.google.com/shopping/product/15441182589886828859/online?q=nike&newwindow=1&rlz=1C1CHBF_deDE891DE891&sxsrf=ALeKk02B3VGibi_0xDWt4Dr11faGzzaxEg:1588968660842&biw=1176&bih=959&prds=paur:ClkAsKraX7UAvI_UK5daydTsvDGc8wLd2Uvr85i_-09PHWfJvRTFZ2vZAj2qcvV9mtYCt8Npbt2VyWLbLBrHJ2ryY0cZku1zDllJ_wzgAQbtk1K27bmYv7bZJxIZAFPVH71iVUznMdZnEQ-Eye5no0_qLHo5Ew,scoring:tp&sa=X&ved=0ahUKEwj88-LGiaXpAhVLRBoKHW9qAVUQ0ykIPg
this is a sample link of a random product from google shopping.
Its a pricing table which is showing data about the competition. For my project i need to get info about the best price (lowest endprice with shipping costs). Furthermore i want to scrape thousands of these google shopping price tables, so i need a tool which is able to work multi-threaded and with proxies. On my journey i couldnt find any tool which is able to do this. therefore im asking this community to help me out. Would be great if there is any program which is able to parse the information directly into an exel or csv, etc. file.
im aware that google proxies will may cost and that free proxies may be blocked. So in the first step im interested in a working tool that can output the information i need in a fast and reliable way. It should check if the price table is working -> block check or some way of checking if data is fetchable.
I tried to setup scrapy or other python based programs but all of them where not able to work for me or werent able to work multi threaded or have proxy support...
thanks in advance for your help!