HOW 2 SCRAPE THE WEB WITHOUT GSA SER

[metalfingersdoom]

Hey everyone,

I'm trying to collect as much information as I can about any website I can find on the web, with a special emphasis on finding websites with specific urls and html forms.

I'm aware GSA SER has certain plugins that may help me accomplish the task, but my concern is GSA not being the "perfectly equipped" tool for this task, as in it was written with spamming as its main emphasis. Does anyone know of such tools?

 

[Deleted member 731332]

You can use scrapebox to scrape using keywords and footprints. Footprints are made using words you commonly find on the page like “first name” “last name” and search operators like +, AND, inurl:, or intext:

so for a contact form a good footprint for scraping could be:

inurl:/contact-us/ “first name” “last name” “submit”

Merge that with a list of 100k keywords inside of scrapebox and you’ll find a lot of contact forms.

then you can use scrapebox page scanner to ensure html element form is in the source or whatever specifics you need in the source code, and filter out any document that doesn’t contain the necessary elements.

 

[metalfingersdoom]

You can use scrapebox to scrape using keywords and footprints. Footprints are made using words you commonly find on the page like “first name” “last name” and search operators like +, AND, inurl:, or intext:

so for a contact form a good footprint for scraping could be:

inurl:/contact-us/ “first name” “last name” “submit”

Merge that with a list of 100k keywords inside of scrapebox and you’ll find a lot of contact forms.

then you can use scrapebox page scanner to ensure html element form is in the source or whatever specifics you need in the source code, and filter out any document that doesn’t contain the necessary elements.

Thanks man! Would you happen to know the algorithm of this particular thing? I have dork scanners installed to do the inurl: thing and would be very sad if scrapebox just does the exact same thing these dork scanners do?

Does it, say follow through different links in websites and ends up gathering more than what dorking would do?

One way to tell it would be to look at the timelapse of the entire thing and see if it stops collecting new links after some time. WAs this or has it been the case for you?

 

[Deleted member 731332]

Thanks man! Would you happen to know the algorithm of this particular thing? I have dork scanners installed to do the inurl: thing and would be very sad if scrapebox just does the exact same thing these dork scanners do?

Does it, say follow through different links in websites and ends up gathering more than what dorking would do?

One way to tell it would be to look at the timelapse of the entire thing and see if it stops collecting new links after some time. WAs this or has it been the case for you?

You can follow links on websites whether internal or external by using a different scrapebox addon. It is the link extractor addon. Scrapebox does not do that when using the harvester (which is the main scraping interface). Harvester only harvests urls from search.

I don’t know if I can help about the dork scanner questions. I just know how to scrape and find what I want using scrapebox. I don’t use SER for that, ever.

@loopline can probably help you better with specific questions. He’s much more experienced than me with the inner workings of scrapebox software.

 

[metalfingersdoom]

You can follow links on websites whether internal or external by using a different scrapebox addon. It is the link extractor addon. Scrapebox does not do that when using the harvester (which is the main scraping interface). Harvester only harvests urls from search.

I don’t know if I can help about the dork scanner questions. I just know how to scrape and find what I want using scrapebox. I don’t use SER for that, ever.

@loopline can probably help you better with specific questions. He’s much more experienced than me with the inner workings of scrapebox software.

@loopline, come here man! Need you rn lol

 

[Deleted member 731332]

I just saw your last question if scrapebox stops collecting links. It does stop collecting links if the search engine does not return any results for the footprint you’re scraping (then it moves to the next in the list), if it has run out of queries to scrape for, or if the proxy is banned. It doesn’t just stop collecting links for no reason, no.

I don’t know if you’re using the the default footprints that come with SER, but if you are: the default footprints don’t return many results in google. You need to make your own footprints if you’re going to scrape google and get many results. Otherwise you need to be scraping different search engines. Bing would be a better choice by far with the default footprints in SER.

 

[metalfingersdoom]

just saw your last question if scrapebox stops collecting links. It does stop collecting links if the search engine does not return any results for the footprint you’re scraping (then it moves to the next in the list), if it has run out of queries to scrape for, or if the proxy is banned. It doesn’t just stop collecting links for no reason, no.

I don’t know if you’re using the the default footprints that come with SER, but if you are: the default footprints don’t return many results in google. You need to make your own footprints if you’re going to scra

Nah making my own footprints anyway. Does scrapebox only search google btw? It'd be awesome if I could use it.

Since we've come this far into the convo, how many threads per proxy do you recommend, and can you recommend proxies

 

[Deleted member 731332]

Nah making my own footprints anyway. Does scrapebox only search google btw? It'd be awesome if I could use it.

Since we've come this far into the convo, how many threads per proxy do you recommend, and can you recommend proxies

yes scrapebox can scrape a ton of search engines not just google.

I use public proxies to scrape so I don’t have to worry about threads. I crank it to 3500 threads (which is max). I get a lot of results fast. Scraping with private proxies, unless you have 200+ is going to be slow.

1 thread per 10 proxies with 30-60 second pause is an okay configuration for scraping google with private proxies. Other search engines aren’t as strict as google.

I am using a service here in BHW that I’ve used for a long time for port scanned public proxies. The service is not anywhere near as good as it was years ago.. I really don’t want to recommend it and you get pissed that only 10% of the proxies on the list work every list.

Red-proxy.net. There’s a BST for them in the marketplace. You’ll see my complaints and the lack of support there in the thread. They make excuses, make you jump through hoops then try to gaslight you into believing you don’t know how to test a proxy. I’m pretty disappointed.

 

[metalfingersdoom]

yes scrapebox can scrape a ton of search engines not just google.

I use public proxies to scrape so I don’t have to worry about threads. I crank it to 3500 threads (which is max). I get a lot of results fast. Scraping with private proxies, unless you have 200+ is going to be slow.

1 thread per 10 proxies with 30-60 second pause is an okay configuration for scraping with private proxies.

I am using a service here in BHW that I’ve used for a long time for port scanned public proxies. The service is not anywhere near as good as it was years ago.. I really don’t want to recommend it and you get pissed that only 10% of the proxies on the list work every list.

Red-proxy.net. There’s a BST for them in the marketplace. You’ll see my complaints and the lack of support there in the thread. They make excuses, make you jump through hoops then try to gaslight you into believing you don’t know how to test a proxy. I’m pretty disappointed.

Alright so would you say going with the service you're using is still my best bet? I have katana scanner installed specifically for its proxy scraping capabilities, but again it's not a service so no way it'll be better. May I ask what the name is and how much do you pay for it monthly?

Also, my scanning doesn't have to be THAT fast. It's just that I need it to go on a consistent basis. I'm researching TOR rn, I think they don't get banned, heard something about it back in the day. Its about 10 times slower than regular internet speed, just to give you an idea of the speed I need. I just need consistency.

Would you suggest I use your service or something else?

 

[Deleted member 731332]

I don’t recommend that you buy this service if you’re only going to scrape google. I’m only getting about 20-40 google passed proxies per list and about 200 anonymous. 8k+ proxies just straight up don’t work as soon as I get the list every list.

I’m probably going to file a shit list on this service. So no I don’t recommend it because you’re not getting what you’re paying for as is advertised. It’s $32.00 a month. Should be more like $10 per month.

The thread is titled “scrapebox google passed proxies.” Read their reply to me. They said don’t use scrapebox to test the proxies LOL. Garbage.

https://www.blackhatworld.com/seo/s...ng-proxies-gscraper-gsa-red-proxy-net.633545/

 

[metalfingersdoom]

lol 9 years of google updates probably didn't treat them well, not buying it thnx. I'll go have a look at tor

 

[Deleted member 731332]

The only thing I can come up with is port scanning on my own with nmap or Gsa proxy scraper but I really don’t have time to learn that right now. Pretty frustrating. I am getting decent results using that service scraping bing, yahoo, and search.com. Millions of urls over a period of hours depending on the footprints I am using.

still not worth $32.00 a month imo.

good luck figuring out a solution

 

[metalfingersdoom]

well I'll do scrapebox 4 sure, at least I learned that. Seems I'll have to figure the proxy thing on my own. I'll try the free ones for a start

 

[Deleted member 731332]

well I'll do scrapebox 4 sure, at least I learned that. Seems I'll have to figure the proxy thing on my own. I'll try the free ones for a start

Make sure you check the BST in the marketplace for the BHW coupon. Scrapebox is $97 on their website but you can get it for much less using the BHW coupon. I think I paid $57 for it 10 years ago. One of the best purchases I’ve ever made. If you’re creative you’ll find way more uses for scrapebox than just scraping. I use it everyday for all kind of stuff not related to scraping at all. Even just dealing with massive lists like splitting them up into smaller lists or randomizing lists etc.

 

[metalfingersdoom]

Make sure you check the BST in the marketplace for the BHW coupon. Scrapebox is $97 on their website but you can get it for much less using the BHW coupon. I think I paid $57 for it 10 years ago. One of the best purchases I’ve ever made. If you’re creative you’ll find way more uses for scrapebox than just scraping. I use it everyday for all kind of stuff not related to scraping at all. Even just dealing with massive lists like splitting them up into smaller lists or randomizing lists etc.

I don't know if it's a banned word on this forum so let me put it this way: I prefer "free" versions of paid software.
I live in a shit country and don't have 50$ rn that I can spare for this task. I'll be looking out for some free versions

 

[Deleted member 731332]

You got to do what you got to do. I don’t make the rules and I certainly don’t judge. Best of luck, as always.

 

[710fla]

Alright so would you say going with the service you're using is still my best bet? I have katana scanner installed specifically for its proxy scraping capabilities, but again it's not a service so no way it'll be better. May I ask what the name is and how much do you pay for it monthly?

Also, my scanning doesn't have to be THAT fast. It's just that I need it to go on a consistent basis. I'm researching TOR rn, I think they don't get banned, heard something about it back in the day. Its about 10 times slower than regular internet speed, just to give you an idea of the speed I need. I just need consistency.

Would you suggest I use your service or something else?

Depending how fast you want to scrape you can use private proxies, use the Detailed Harvester and set a high delay depending on the amount of proxies. You'll have to play around with the delay until you see your proxies aren't getting banned and just let it do it's thing.

Rotating proxies are also a thing but some providers eventually end up having their whole IP pool Google temp banned so it's hit and miss