How do you block crawlers from your Pbn?

I know about using robots.txt, but for some reason I still believe that some of them is crawling my sites.

Also... Is there a comprehensive list of all seo spiders that should be blocked?

Is it necessary to block their ips? Where can I get a list of their ips?

If you are talking about WordPress, then in the free plugin library there are special plugins for blocking unnecessary bots.

I have not used any of them yet. But I will need them in the near future. If someone has experience of using - tell us please.

Mops said:

If you are talking about WordPress, then in the free plugin library there are special plugins for blocking unnecessary bots.

I have not used any of them yet. But I will need them in the near future. If someone has experience of using - tell us please.

Click to expand...

I'm not using wp, but i might take a look at that plugin to see what exactly they are blocking

They don't necessarily follow robots.txt rules, you should deny them using your htaccess file based on user agent and/or ip address

I'm no SEO expert. But, if they blocked crawlers I don't think you're going to be able to find out what the PBN links look like unless you have access to a crawler that isn't blocked in their robots / .htaccess. If you think they used some brand anchors, you can always use some basic Googlefu skills and search for "site name," "author name," and "sitename.com." You might find a few of the sites... and could then try to reverse engineer / find a footprint / provider... but any half decent PBN isn't going to have obvious footprints.