1. This website uses cookies to improve service and provide a tailored user experience. By using this site, you agree to this use. See our Cookie Policy.
    Dismiss Notice

crawlers

  1. marcuus
    Thread

    How to analyze backlinks after blocking robots

    Hello, I've been interested in SEO for some time and have one question. If I block Ahrefs, Majestic etc robots in htaccess file, how can I...
    Thread by: marcuus, Jan 20, 2019 at 7:37 PM, 0 replies, in forum: Black Hat SEO
  2. Better Know
    Thread

    Writing a crawler

    I need to write a crawler for a project of mine that needs to be extremely scalable and effecient. I was advised to use a lower level language,...
    Thread by: Better Know, Jan 14, 2019, 10 replies, in forum: General Programming Chat
  3. KM1
    Thread

    Block majestic, ahrefs, moz, etc

    Anybody has a robots.txt file to block out these crawlers? Majestic Moz Ahrefs Other major crawlers.. I can make one, but it takes time. If...
    Thread by: KM1, Jul 27, 2017, 2 replies, in forum: Black Hat SEO
  4. chrismartin4u
    Thread

    Google crawl rate and indexing related issue

    Hello Friends, I need help regarding google crawl rate and indexing related issue. Is it possible that google crawler start indexing old content...
    Thread by: chrismartin4u, May 10, 2017, 0 replies, in forum: White Hat SEO
  5. tinyboxer
    Thread

    looking for a tool that filters usernames / IDs by gender - anyone knows?

    hey, i'm looking for a tool that knows to distinguish between male and female instagram users. Tooligram claims to be able to do that, but i cant...
    Thread by: tinyboxer, Jan 16, 2016, 0 replies, in forum: Instagram
  6. alexel
    Thread

    What HTML Tags Are Significant to Search Engines

    I know that obviously crawlers and spiders index sites based on their meta, H1, H2, alt, title, and all the basic ones. But do search engines...
    Thread by: alexel, Sep 24, 2015, 1 replies, in forum: Black Hat SEO
  7. rodzzlessa93
    Thread

    Developing social media webbots

    Hello I'm interested in developing advanced web-bots for social media websites like twitter, Instagram, YouTube, and Pintrest. To give you guys a...
    Thread by: rodzzlessa93, Jan 3, 2014, 6 replies, in forum: Other Languages
  8. howard310
    Thread

    What is the best scraper our right now?

    Looking for fresh company data across the US.
    Thread by: howard310, Sep 27, 2010, 0 replies, in forum: Black Hat SEO Tools
Showing results 1 to 8 of 8