Scrapebox Custom Data Grabber error 200 ?

iktheavus said:

Hello all.

I've written a custom Data Grabber and after running it some of the entries return weird error 200 (please see image attached).

Running the grabber several times on the same list of urls returns inconsistent results - sometimes it works correctly, sometimes it returns error 200 for the same url, so the issue is, I am not getting correct results for every run.

I am running it via 10 shared proxies I've bought from buyproxies. However same issues observed when I run it just from my ISP IP - works 80% of the time, but sometimes doesn't.

Any clue? maybe @loopline grace me with his knowledge?

Click to expand...

loopline said:

Never heard of this. Are you using V2.1.0.0

If not try to update and try it.

You could try lowering connections to 1 as a test and try it.

If it happens without proxies then its not the proxies, so it could be like security software, DNS, router etc..

You could try using google free public DNS (just google it) thats what I use. often times ISPs DNS is poor. Then post back and let me know if its still not working

Click to expand...

loopline said:

Sure. I mean the problem is that your using too many threads for some threshold. Either Etsy or some security software/router and thus the errors

So basically try it with proxies and slowly turn up the connections to 2 or 5 etc.. and keep going until you start getting the errors and then you know.

Also you can whitelist the entire scrapebox folder in all security software.

Click to expand...

iktheavus said:

Appreciate the help!

Click to expand...

DeLaGarde said:

Jumping on this thread as I am having the same error 200 with Custom Data Grabber, hoping you could chime in @loopline cause you are the master at this and I have been binging on your Youtube videos where I learnt how to use the Custom Data Grabber.

I tried the suggestions to use one Connection which I was doing anyway and whitelisted the Scrapebox folder in Windows Defender and Firewall, no other security software running. I also changed my DNS to use googles 8.8.8.8 and 8.8.4.4. I am using Scrapebox V2.1.0.0

I know Regex and HTML and don't think I am confused about what I am supposed to be doing, but after a whole day of watching tutorials and messing about I am no closer to solving the mystery.

This is an example of a page on website I am trying to scrape https://www.skroutz.gr/m/973/Samarasae
Basically just trying to grab the link under the button with the text 'Website καταστήματος'

After trying a bunch of variations for the mask I have ended up with this simple mask that should be able to grab any URL - at this point trying to get it working at any basic level of scraping anything at all from that page with the Custom Data Grabber, will make it more specific if that works:

Code:

before_after=href="|">

Here is the whole code around the link:

DeLaGarde said:

Jumping on this thread as I am having the same error 200 with Custom Data Grabber, hoping you could chime in @loopline cause you are the master at this and I have been binging on your Youtube videos where I learnt how to use the Custom Data Grabber.

I tried the suggestions to use one Connection which I was doing anyway and whitelisted the Scrapebox folder in Windows Defender and Firewall, no other security software running. I also changed my DNS to use googles 8.8.8.8 and 8.8.4.4. I am using Scrapebox V2.1.0.0

I know Regex and HTML and don't think I am confused about what I am supposed to be doing, but after a whole day of watching tutorials and messing about I am no closer to solving the mystery.

This is an example of a page on website I am trying to scrape https://www.skroutz.gr/m/973/Samarasae
Basically just trying to grab the link under the button with the text 'Website καταστήματος'

After trying a bunch of variations for the mask I have ended up with this simple mask that should be able to grab any URL - at this point trying to get it working at any basic level of scraping anything at all from that page with the Custom Data Grabber, will make it more specific if that works:

Code:

before_after=href="|">

Here is the whole code around the link:

Code:

<a rel="nofollow external" class="btn-transparent icon shop-web" target="_blank" href="https://www.samarasae.gr">Website καταστήματος</a>

The link to be extracted is https://www.samarasae.gr/ which I also entered into the 'Examples of expected data, but it makes no difference either way.

The link extractor addon is able to pull that URL fine, but I don't want to use the link extractor for this cause it pulls a whole bunch of other links on the page that I don't want.

The only thing that makes sense to me at this point is that my Scrapebox is not working properly and I suspect it is not using the proxies I have entered (my own proxies one of which is my home resi IP which is working fine and can access the site and scrape it using any other tool), I think Scrapebox is trying to go direct. The reason I think that is cause Scrapebox is on my server which cannot access that site I wanna scrape directly (I think I got the IP banned by them not long ago lol). But like I said the ScrapeBoxs link extractor works fine going through my proxies. I tried to use the 'Get links by crawling a site' function on this page just to see what happens, and there is also zero results with that as with the Custom Data Grabber, as if the site does not exist.

It's doing my head in! Any help would be much appreciated, just want to exhaust the possibilities before I conclude that it is Scrapebox that is not working.

Click to expand...

Code:

<a rel="nofollow external" class="btn-transparent icon shop-web" target="_blank" href="https://www.samarasae.gr">Website καταστήματος</a>

The link to be extracted is https://www.samarasae.gr/ which I also entered into the 'Examples of expected data, but it makes no difference either way.

The link extractor addon is able to pull that URL fine, but I don't want to use the link extractor for this cause it pulls a whole bunch of other links on the page that I don't want.

The only thing that makes sense to me at this point is that my Scrapebox is not working properly and I suspect it is not using the proxies I have entered (my own proxies one of which is my home resi IP which is working fine and can access the site and scrape it using any other tool), I think Scrapebox is trying to go direct. The reason I think that is cause Scrapebox is on my server which cannot access that site I wanna scrape directly (I think I got the IP banned by them not long ago lol). But like I said the ScrapeBoxs link extractor works fine going through my proxies. I tried to use the 'Get links by crawling a site' function on this page just to see what happens, and there is also zero results with that as with the Custom Data Grabber, as if the site does not exist.

It's doing my head in! Any help would be much appreciated, just want to exhaust the possibilities before I conclude that it is Scrapebox that is not working.

Click to expand...

DeLaGarde said:

That's very kind of you, thanks!

Click to expand...

DeLaGarde said:

Thanks a lot for looking into it @loopline, really appreciate that. I seem to have a special skill, more like a curse, for uncovering bugs in software. Well at least I know now that I am not doing something wrong. I guess there is nothing else to do than wait it out for that update.

Click to expand...