softtouch2009
Senior Member
- Dec 2, 2009
- 1,064
- 257
It is currently developed on a Windows 10 64 bit PC, so yes, it works just fine.
Permanently Closed Marketplace Sales Thread
- Thread starter Sweetfunny
- Start date
- Status
It is currently developed on a Windows 10 64 bit PC, so yes, it works just fine.
minustheartist
Registered Member
- Nov 21, 2009
- 85
- 9
Epic! Thanks. Ordering now...
- Jan 25, 2009
- 6,378
- 3,774
Your issue is your using public proxies. Don't use public proxies on alive checks. As a general rule you don't need proxies at all on alive checks, unless your alive checking a massive amount of urls from the same domain. Then you could use private proxies only.
But when you encounter a public proxy that has a 404 thats all the alive checker can see is a 404. It can't tell if the 404 was from a dead proxy or from the end site, and its pointless to try and have it tell that anyway. Just don't use proxies on alive checking.
In fact, just for sake of reference, pretty much the only thing you should use public proxies for is for harvesting results from search engines or search engine related services like the google competition finder or the google image grabber etc..
It work fine for me, it shows as alive in my alive checker, which means it would be red with your setup in your alive checker.
Try it without proxies, does it still show this?
Not to mention using public proxies is going to slow the whole process from the speed of a jet plane to the speed of turtle.
Last edited:
NobelNerd
Senior Member
- Feb 21, 2013
- 948
- 490
Hi I did it without proxies still it is showing live domains as 404
http://prntscr.com/auw4vi
http://prntscr.com/auw5t2
- Jan 25, 2009
- 6,378
- 3,774
Well that site is the exception not the rule. That site is indeed returning a 404. For example, try it in header check tool:
http://www.webconfs.com/http-header-check.php
Some urls will return a header one way but a GET request another way. The alive checker does first a head check and then a GET request. When searching for alive urls it looks for the HEAD and if its a 404 does a GET and the site returns a 200.
But your using the alive checker backwards, your using it as a dead checker. So when it finds a HEAD of 404, which you have set to mark as alive, it then stops and doesn't try a GET because it always has what you have set it to look for.
Some sites out there do "Funky" (how do you like my technical term) things based on user agents or HEAD vs GET or scripts etc..
What I suggest you do is use the alive checker as an alive checker. Check for 200 and 301 etc.. Then take the dead urls and run them against your reverse alive checker to find the 404 ones.
Or if you tell me what your end game is I might be able to tell you a better way to go about the whole thing anyway. Like if your looking for expired domains then just skip the alive checker.
If your looking to find broken links the broken link checker works better etc..
But Scrapebox can't change the fact that some sites don't follow convention and do funky things.
NobelNerd
Senior Member
- Feb 21, 2013
- 948
- 490
Well that site is the exception not the rule. That site is indeed returning a 404. For example, try it in header check tool:
http://www.webconfs.com/http-header-check.php
Some urls will return a header one way but a GET request another way. The alive checker does first a head check and then a GET request. When searching for alive urls it looks for the HEAD and if its a 404 does a GET and the site returns a 200.
But your using the alive checker backwards, your using it as a dead checker. So when it finds a HEAD of 404, which you have set to mark as alive, it then stops and doesn't try a GET because it always has what you have set it to look for.
Some sites out there do "Funky" (how do you like my technical term) things based on user agents or HEAD vs GET or scripts etc..
What I suggest you do is use the alive checker as an alive checker. Check for 200 and 301 etc.. Then take the dead urls and run them against your reverse alive checker to find the 404 ones.
Or if you tell me what your end game is I might be able to tell you a better way to go about the whole thing anyway. Like if your looking for expired domains then just skip the alive checker.
If your looking to find broken links the broken link checker works better etc..
But Scrapebox can't change the fact that some sites don't follow convention and do funky things.
thanks for your response I am looking for some good expired domains to start PBN for my website
- Jan 25, 2009
- 6,378
- 3,774
If thats the case, I would run the alive checker with your setup looking for 404 as being alive. Just like you do. Then I would just mass export all the ones that shows as "alive" aka a 404 and run them thru the domain availability checker (as thats your ultimate "its really available") anyway. So basically your using the alive checker to quickly trim your list down and then use the domain availability checker to get the final say of X is available and Y is not. Because even if a site comes up as 404 that doesn't mean that the domain is available. Also you should keep in mind that doing it your way is looking for a 404. You only get a 404 if there is hosting. If a domain is registered but the name servers don't point at any hosting your not getting a 404. So its not going to show as "yes" in your setup and you will miss it. So having thought that thru out loud, I would personally scrap your setup of the alive checker and just run the alive checker like its meant to be, looking for whats alive with a 200 3xx code setup. Then I would take all the dead and run them thru the domain availability checker. It might sound dumb but I own many domains that aren't setup with hosting. For example I have the .com of my kids name for all of my kids, so that when they get older if they want it they have it. More pointedly my brother owns a tree service type company in a small town. I did some research and was trying to find some nice expired domains with authority links from his comunity. I found a few and registered 1 or 2 of them, but then got side tracked and never did anything with them. Best of intentions, but 1 of them is up for renewal so Ive had it for a year now. LOL Point being even with expired domains those would never show up with your setup, Id guess your missing most of your possible options by forcing a 404 for it to qualify. My 2 cents anyway. Edit: Well I have formatted that, but BHW keeps messing it up, getting other errors also, sorry about the mass paragraph.
notvfanymore
BANNED
- Nov 26, 2014
- 66
- 22
Can you add do-folow checkbox on link extractor addon alongwith external/internal ?
I had earlier asked a way to get only do-folow links where Matt had advised to use data graber.But its too much resource consuming and lot of filtration is required since regex will also get lot of scrape along with "Do follow URL's".
If theirs any other way to get only do follow links from bunch of target urls then your help is greatly appreciated .....
I had earlier asked a way to get only do-folow links where Matt had advised to use data graber.But its too much resource consuming and lot of filtration is required since regex will also get lot of scrape along with "Do follow URL's".
If theirs any other way to get only do follow links from bunch of target urls then your help is greatly appreciated .....
Joseph Lich
BANNED
- Nov 25, 2015
- 401
- 79
If don't care meta nofollow, "Comment Grab Platform Configuration" might be a choice.
- Jan 25, 2009
- 6,378
- 3,774
You can post to all the urls and then link check them, just use a bogus url. Then run them all thru the ******** test plugin using the bogus url to find which are ********. Then go back and post to the ones you want with your actual links. That or you could probably just skip the link checker as the ******** test addon can do this in 1 step.
For use with Scrapebox? Are you talking about for spinning using the article scraper plugin? If so you can just upload spun articles into the spinner and it can learn and build database from that on its own, or you can do it 1 word at a time, all from within the GUI.
notvfanymore
BANNED
- Nov 26, 2014
- 66
- 22
Thanks Matt as always..
But I think you misinterpreted my question.Suppose I've some target url then I want to extract only external do-follow links.(I don't want them for commenting but for authoritative links to build PBN).
What happen is along with authoritative do-follow links (in main content) like from Forbes some No-follow links like comment links or forum links(which are non editorial,Spamy Chinese) get into the list.
This can be avoided if on the link extractor itself has do-follow/no-follow filter/checkbox.
Also data grabber will require a lot of filtration since regex will also get lot of scrape along with "Do follow URL's" like title,class etc and is resource heavy.
Joseph Lich
BANNED
- Nov 25, 2015
- 401
- 79
Custom Data Grabber can do this.
Filtrations doesn't have to be set when grab data.
Then process the grabbed data. Including remove internal links, and keep the external links.
Filtrations doesn't have to be set when grab data.
Then process the grabbed data. Including remove internal links, and keep the external links.
- Jan 25, 2009
- 6,378
- 3,774
Mine got data today just fine when I ran it. Most likley your ip or proxies are blocked. So try new proxies if your using them or if your using proxies try it without proxies and with a delay of 60.
Ahh. I think it could use more resources of course to include ******** scanning in the link extractor, but that would be up to sweetfunny.
You want to mail support {at) scrapebox [dot) com and ask them to add the ******** check to the link extractor as a request. The only other way I can think of is as mentioned by using the custom data grabber.
Yes I have a 8.1 machine running Scrapebox just fine. It works from windows xp to windows 10 and server 2003 forward.
Joseph Lich
BANNED
- Nov 25, 2015
- 401
- 79
Matt, do you think what he want is building backlinks to his PBN?
- Jan 25, 2009
- 6,378
- 3,774
No it works for me just now. Typically when you get a 403 that is coming from the proxies. Moz doesn't usually return a 403.
Try it without proxies.
Failing that it could be your firewall etc...
Maybe, hard to say. There is a lot of bad info out there about ******** links, so some people leave good links on the table without testing, while others have done their tests and know exactly what they need. Its hard to guess where it falls.
~~~~~~
New Video
Safely Scraping Google And Other Engines - Scrapebox 2.0
Last edited by a moderator:
- Status