Is BHW allowed auto robot's sign up ?
- Thread starter LinkWorld
- Start date
Google bots.
staypositive
Supreme Member
- Jul 28, 2015
- 1,433
- 225
many vendors have crawlers keep crawling bhw for business opportunities.
if u post a thread " i need a content writer " and i am sure u will get many PM within one hour. cox they got auto notification
- Nov 29, 2012
- 46,013
- 13,844
Thread moved to whitehat SEO as the discussion seems to have changed topics.
Crawlers are pieces of software that will visit random pages on the internet to try and sort the information on the page into a database so it can be called later on.
So for example Googlebot will visit this thread, see that people are discussing crawlers and add it to the Google database so when people search for crawlers this page comes up in the results.
They will litterally visit this page, add all the links to a list of links to visit later, get the information they need then follow one of the links to a new page and repeat.
Other crawlers look for other information, they might be SEO tools like Majestic or Ahrefs that want to just index links to external sites or they could be bots to alert users when certain words (such as their brand name) is mentioned.
Big sites like BHW with millions of pages will have 100's of bots crawling at any one time.
Crawlers are pieces of software that will visit random pages on the internet to try and sort the information on the page into a database so it can be called later on.
So for example Googlebot will visit this thread, see that people are discussing crawlers and add it to the Google database so when people search for crawlers this page comes up in the results.
They will litterally visit this page, add all the links to a list of links to visit later, get the information they need then follow one of the links to a new page and repeat.
Other crawlers look for other information, they might be SEO tools like Majestic or Ahrefs that want to just index links to external sites or they could be bots to alert users when certain words (such as their brand name) is mentioned.
Big sites like BHW with millions of pages will have 100's of bots crawling at any one time.