Domain Hunter Gatherer, newbie teething problems.

[TheAlmightyDada]

Hi all,

I'm having some trouble with DHG and I'm wondering if anybody could give me a hand.

I have a list of around 30 Keywords and while searching expired domains, I am getting 1 result that is not of good enough quality. Is this normal, or have I messed up a setting somewhere?

I am using 200 Squid Proxy, proxies.

As a test, I searched one very broad keyword. "Car" - below is the result.

Any help from those in the know, would be very much appreciated.



@jimbobo2779 @Nargil

Many Thanks.

 

[jimbobo2779]

I would assume that you have an issue (or potentially issues) related to your proxies.

You mention you are using 200 squid proxies but the software is showing 209 proxies, this would suggest you have either not IP authenticated your proxies or you have mixed in some bad proxies with the paid for proxies.

From here I would suggest the following:

1. Remove any proxies that are public / not paid for
2. Run a proxy test within DHG to ensure that you are only using the proxies that have a checkmark in the "Working" column
3. Lower number of searching threads to 5 (just for testing)
4. Lower number of "Crawling / Availability Threads" to 50 (just for testing)

Once you are happy with the results coming in you can begin gradually increasing the number of search threads and then also the Crawling / Availability threads. The searching threads are only used for searching so will speed up searching but have no effect on the crawling phase of the process and the crawling threads setting only affects how quickly the tool can crawl pages and check availability of domains.

I have just run a quick test with that keyword and DHG found 5600 total search results (compared to your 464) and by the end of the run found 917 available expired domains (compared to your 2). Below is a SS of this quick run.

 

[TheAlmightyDada]

I would assume that you have an issue (or potentially issues) related to your proxies.

You mention you are using 200 squid proxies but the software is showing 209 proxies, this would suggest you have either not IP authenticated your proxies or you have mixed in some bad proxies with the paid for proxies.

From here I would suggest the following:

1. Remove any proxies that are public / not paid for
2. Run a proxy test within DHG to ensure that you are only using the proxies that have a checkmark in the "Working" column
3. Lower number of searching threads to 5 (just for testing)
4. Lower number of "Crawling / Availability Threads" to 50 (just for testing)

Once you are happy with the results coming in you can begin gradually increasing the number of search threads and then also the Crawling / Availability threads. The searching threads are only used for searching so will speed up searching but have no effect on the crawling phase of the process and the crawling threads setting only affects how quickly the tool can crawl pages and check availability of domains.

I have just run a quick test with that keyword and DHG found 5600 total search results (compared to your 464) and by the end of the run found 917 available expired domains (compared to your 2). Below is a SS of this quick run.

Life saver.

I'm not entirely sure what I did to get 209, but I went and got them again and ran the test (one didn't work).

While my results are not identical to yours (see below) - could this just be standard deviation?

Thanks a lot for your help - problem solved.

 

[jimbobo2779]

The difference is likely due to the difference in results of the search process. You can see in my SS I got 5600 total results where you got 5100. The 500 results difference is likely from Google which is more likely to be unique results.

I thought maybe it is that squid proxies are not so good these days so I have started another test run with my 100 squid proxies (shared) I have and right away I have gotten 7657 total search results so the issue is not that you are using Squid proxies, they are definitely good proxies, it is probably just down to the IP lottery.

When renting proxies you never really know who has used them in the past or what they have been used for so it is very possible that a portion of them will not be usable for searching. Search engines are pretty quick to block or limit an IP and start returning no results or fewer results for the queries so it is possible / likely that some of the IPs are not ideal for searching, they are highly likely to be fine for crawling / availability checking but not so great for searching.

I will post back when this current test is finishing to show the results from my 100 squid proxies.

 

[TheAlmightyDada]

Hi @jimbobo2779

Thanks for putting in all of this effort, I really appreciate it. I have one last question and as you will be coming back - I may as well ask it here in case anybody else ever has the same question.

I have the Pro version of DHG, which comes bundled with 50K DomDetailer credits.

That 50K doesn't refresh every month, does it?

Can I use the DomDetailer credits to look at the stats of non-expired domains? If so, how would I go about doing that? As my login details for DHG do not work with the DomDetailer website.

Many thanks for your time and attention.

 

[jimbobo2779]

So my all squid proxies test finished with 863 domains, this was with the 96 out of 100 working squid proxies:

So then I thought let's try with the 96 squid proxies for searching and the 25 other proxies for crawling / avail checking (something I think is only possible in DHG) just to see how that affects things and this is how it turned out

As you can see the results will vary from run to run depending on how you have your proxies setup. A lot of this is to do with how the proxies are being run, i.e. if they have never been used for searching the search result numbers will be good but if they have been continually used they may not be returning nearly as many results.

I will be running a bunch of tests soon to see how different proxy setups with different providers affect the running of the tool.

Re DomDetailer credits: Yes you do get 50k credits for DomDetailer every month.

You can import any domain into DHG by clicking on the "Load / Import Domains" tab header and then "Import From File" button and choose to import domains from txt, csv, xml (DHG save file) or straight from clipboard. From here you can use the controls in the Analyse Domains tab to get the different stats for the domains you have in the domain list. One piece of advice I would give is to only check for the stats you are absolutely interested in, don't just select all of the stats and openly check all of them as you will be wasting time / API credits and/or getting your IPs blocked in the search engines.

Cheers

 

[jerkijum]

I would assume that you have an issue (or potentially issues) related to your proxies.

You mention you are using 200 squid proxies but the software is showing 209 proxies, this would suggest you have either not IP authenticated your proxies or you have mixed in some bad proxies with the paid for proxies.

From here I would suggest the following:

1. Remove any proxies that are public / not paid for
2. Run a proxy test within DHG to ensure that you are only using the proxies that have a checkmark in the "Working" column
3. Lower number of searching threads to 5 (just for testing)
4. Lower number of "Crawling / Availability Threads" to 50 (just for testing)

Once you are happy with the results coming in you can begin gradually increasing the number of search threads and then also the Crawling / Availability threads. The searching threads are only used for searching so will speed up searching but have no effect on the crawling phase of the process and the crawling threads setting only affects how quickly the tool can crawl pages and check availability of domains.

I have just run a quick test with that keyword and DHG found 5600 total search results (compared to your 464) and by the end of the run found 917 available expired domains (compared to your 2). Below is a SS of this quick run.

what the functions on proxies?
i tried search several keywords but only show same one results domain.
whats wrong here?
i dont have any private proxy, how to get like you 25 proxies? is it private? how much?

 

[jimbobo2779]

what the functions on proxies?
i tried search several keywords but only show same one results domain.
whats wrong here?
i dont have any private proxy, how to get like you 25 proxies? is it private? how much?

Proxies will allow you to do more reliable searches as your IP will not be getting blocked, also the more proxies you have the more searches will be possible at any one time so the searching will be faster as well. If you are looking to crawl a single site deeply then again proxies will prevent you from getting your IP limited or blocked on these sites as well.

There are many benefits to running with proxies that even running with 10 private proxies is far more advisable than running without proxies, more is generally better (providing they are all private / shared and not public) however.

If you are doing lots of searching and getting few to no results then the proxies you are using are trash and you should get rid of them. Public proxies can actually be more damaging to a run's results than running without proxies so I wouldn't advise using public proxies in any capacity with a tool like Domain Hunter Gatherer.

There are many providers of proxies, if you want to get some you could try the marketplace on this website as there are lots of providers to choose from and you can look at recent posts on the thread(s) to see the experience of other users before purchasing.