It is possible to know from which site the bot/crawler is coming when following a link?

Guybrushthepirate

Guybrushthepirate

Regular Member
Joined
Dec 23, 2015
Messages
281
Reaction score
169
Pretty much what I am asking in the title. I'd like to know if there is any form to know from which source a crawler is coming when reaching a URL, linked to another.

Example: in site1.com there is a link pointing to site2.com.

Is there a chance for the owner of site2.com, to know that the crawler is reaching its site following the link in site1.com?

Of course the crawler will find and index site2.com from many different sources, but I'd like to know if there is a way to know, every time a crawler visit a URL because of a link pointing to that specific URL in another site, the origin/source from which a bot comes from.
 
Advertise on BHW
If they are crawling with a full browser then you might get lucky and have the referral header set, otherwise no if I understand your question correctly
 
Entirely depends on the bot. I'm not sure what kind of bot you're thinking about but the answer is probably not.
 
We might be able to help more if you explain why you want to know this :-)
 
Guybrushthepirate said:
Pretty much what I am asking in the title. I'd like to know if there is any form to know from which source a crawler is coming when reaching a URL, linked to another.

Example: in site1.com there is a link pointing to site2.com.

Is there a chance for the owner of site2.com, to know that the crawler is reaching its site following the link in site1.com?

Of course the crawler will find and index site2.com from many different sources, but I'd like to know if there is a way to know, every time a crawler visit a URL because of a link pointing to that specific URL in another site, the origin/source from which a bot comes from.
Click to expand...

Yes, Guybrushthepirate, in WP under heading Tool > Redirection > Log It will give you the source URL and User-Agent (where the bot/Crawler came from and ip address.)
 

Attachments

  • Screen Shot 2019-11-20 at 1.32.44 PM.png
    Screen Shot 2019-11-20 at 1.32.44 PM.png
    267.5 KB · Views: 2
jamie3000 said:
If they are crawling with a full browser then you might get lucky and have the referral header set, otherwise no if I understand your question correctly
Click to expand...

jamie3000 said:
We might be able to help more if you explain why you want to know this :)
Click to expand...

Thanks jamie3000, I am afraid I can't express myself properly as English is not my native language, but I'll try my best. Basically I'd like to be able to set some sort of rule that says: if Google bot is coming to my URL from that specific link, then I'd like to [ACTION TO BE DEFINED]. In order to do that, I need to know how (from which linking URL) the bot is finding my destination URL.

SEOMadHatter said:
Entirely depends on the bot. I'm not sure what kind of bot you're thinking about but the answer is probably not.
Click to expand...

Thanks for taking time to answer. I guess, as your says, that probably it is not possible. I am referring to Google bot anyway.
 
I just realized that I didn't provide probably important information. This type of verification should be done in real time, so that whenever the Google bot is trying to visit a specific URL of my website, coming from a link placed in the www.example.com website, I can decide whether to let it crawl the URL or not.
 
Guybrushthepirate said:
Can you provide a little more information, please? How would you do that?
Click to expand...

No, he can't. The Google bot does not have the referer header.

If I remember rightly (it's been a while) you can technically see this by removing the page and looking at the crawl errors since Google will tell you the pages linking to that page. It does mean breaking that page for a while but I can't think of another way unless backlink spiders have found the link as well.

In general, most bots won't include the referer unless they're trying to pretend to be a real browser.
 
SEOMadHatter said:
No, he can't. The Google bot does not have the referer header.

If I remember rightly (it's been a while) you can technically see this by removing the page and looking at the crawl errors since Google will tell you the pages linking to that page. It does mean breaking that page for a while but I can't think of another way unless backlink spiders have found the link as well.

In general, most bots won't include the referer unless they're trying to pretend to be a real browser.
Click to expand...

Thanks SEOMadHatter, I'll do the test you suggest, but it makes sense that bots won't include header referer. I thought it would be difficult to do what I had in mind.
 
Unless you control the link on example.com adding a custom header X-something could identify what you need other than that is a bad use case like other mentioned you cannot trust external headers
 
Yee said:
Unless you control the link on example.com adding a custom header X-something could identify what you need other than that is a bad use case like other mentioned you cannot trust external headers
Click to expand...

Is not the case. Example.com is a site that I don't control directly. "Simply" I'm trying to figure out if there's a way to identify the cases in which Google bot tries to access my URL, coming through a link on example.com.
 
If you're goal is to identify GoogleBot you should use DNS
 
Well it's a fundamental part how the internet works but it's not something that would work out on a short base
 
Yee said:
Well it's a fundamental part how the internet works but it's not something that would work out on a short base
Click to expand...

I know what DNS are, I was referring on how that would apply to what I am trying to achieve.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Networking Downloads

Marketplace

Account Selling Content / Copywriting Hosting Images, Logos & Videos Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Misc

Making Money

Affiliate Programs Hire a Freelancer Making Money Pay Per Click Site Flipping

BlackHatWorld

BHW Merch Forum Suggestions & Feedback Introductions Lounge Dispute Resolution

Other

Domain Name Forum IM Journeys Web Hosting Newsletter
Back
Top