Referrer Tutorial - How to Check Referrer Wireshark & HTTP Fox

 

Results 1 to 10 of 10
There is a great thread on here about 'faking' and 'blanking' the referrer. READ THREAD ...
  1. #1
    Lutherblissett's Avatar
    Lutherblissett is offline Regular Member
    Join Date
    Feb 2008
    Posts
    474
    Thanks
    46
    Thanked 165 Times in 109 Posts

    Default Referrer Tutorial - How to Check Referrer Wireshark & HTTP Fox

    There is a great thread on here about 'faking' and 'blanking' the referrer. READ THREAD HERE. In fact there are a lot of great threads here on techniques to use with the referrer from DMR, to nested frames, . . and the list goes on. However, I feel like a lot of people here have no idea how to check a referrer on the fly.

    Why would YOU want to learn this?
    -To check other's claims about a method they swear 'doesn't leak'
    -To see if the software/script you stole that is 3 years old, really still works.
    -To check wild configurations that you are uncertain about
    -To spy on a competitor and see what he is actually passing to the sponsor or site as his referrer.
    -A shitload of other reasons i'm not going to post. . its endless.

    Method 1 - HTTP fox
    Http fox is a FF add on (yes it works with ff 9.0). Download it here.
    Once installed goto your add on manager.
    Check box 'always open in new window' (you don't have to do this, however i find it a lot easier to actually see the headers as they fly by with a new window full screen).
    Goto the bottom right corner and find the 'http fox' button.
    Click it to open HTTP fox in a new window.
    Now that its open it looks like this:



    Go back to your original FF window and visit google.
    Enter 'Buy Tools' and hit enter.
    You should get results like this :



    Go to the HTTP fox window
    Click clear to clear any headers there, and then click start.
    Go back to the SERPS and click on an ad. I used Lowe's (if your not from the states your going to get different results. . doesn't matter its just an example).
    Now go back to your HTTP fox window. Use the scroll bar to scroll back up to the top.
    It should look like this:



    Click on the first result.
    When you do it will open the header info. in the bottom section. Use the up and down arrows to go up and down each different request. In this example I am going to inspect the second GET (request).



    In this case i'm going to examine what referrer lowe's gets when that google sends them a paid link. We can see a lot of great information here. I can see in the top portion that i'm looking at the get request that is a 302 redirect to lowes. In the bottom section there is a lot more goodies. Here we can see the referrer is a massive google link that is unique each time (i hate them for this). We can also see the user agent info that was passed (firefox in this case). Also we can see the request line was to GET an interior redirect link, and the 'host' is some interior redirecting system. If you go down to the next header packet, you will see that the host is lowes, and from here on out . . its all communications with the lowes page.

    In this example i can tell for sure that google passed that wild insanely long redirect link as the referrer to lowes. The point of this is to get you acquainted with the http header and what the raw packet actually is sending. Now you can use this to check any of your pages and see what referrer you are actually passing along.

    THIS IS IMPORTANT because a lot of people say a lot of shit that isn't true. Now you can check for yourself, well at least on firefox. . . which leads to How do we check this on all other browsers one might use. That is a great question.

    To do that I use wireshark. Its a freeware, download it here

    Wireshark is a lot more advanced and measures every imaginable packet that goes across your computer and network, Its crazy, and to be honest I really don't understand most of it, its over my head. Install the software then open it up. It should look like this. .



    IN THIS EXAMPLE WE ARE ONLY USING IT TO SNIFF GET REQUESTS ON YOUR MACHINE. This is highly simplified, so just follow along.
    Click on Capture in the nav bar.
    Select Capture filters.
    Click New.
    Now select the filter string and enter the following code:
    port 80 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420

    Label the name 'GET request'


    It should look like this:



    Click ok. .

    Now go back to Capture in the nav. bar. and select 'Options'.
    Click the right drop down on the interface option and select 'Microsoft / blahblahb' or whatever you have for your computer.
    Now click the button Capture Filter, it will open a pop up box.
    Select your newly created 'GET request' filter we just made. It should look like this:



    Click start.

    Now you will see a big blank area that is going to fill up fast, once you start browsing on any browser. Try it out by surfing around the web. Before you know it there will be a lot of packets to look at. Now at any time you can just press CTRL+R to reset and clear all packets. When you get to the point of where you want to examine a referrer, press CTRL+R, and then click the link you want to check in your browser. Then Go back to wireshark and Stop or pause the stream and you will have a much more manageable list of packets to look at.

    Examine the packets as i told you earlier. HTTP headers are more or less the same, and pass the same info around in slightly different ways.

    Now you should have a good way to see what is actually going across your HTTP headers as you surf the net and check links. You are going to see some funny stuff, that you never knew was happening, especially as you check other browsers. The amount of requests that go to google is disturbing. Hope you enjoy the Tut. . .I don't write them a lot so excuse the long winded descriptions and any missed statemetents or mistakes.

  2. The Following 11 Users Say Thank You to Lutherblissett For This Useful Post:

    angelas111 (01-28-2012), britcpa (03-08-2012), gtreeoutsourcing (12-11-2012), incognito876 (03-01-2012), lotar (02-29-2012), Ms. Jukey (02-25-2012), navi4305 (04-02-2013), Rasputin78 (02-29-2012), sudirboi (03-08-2012), THUNDERELVI (01-29-2012), wawawiwa (03-06-2012)




  3. #2
    Lutherblissett's Avatar
    Lutherblissett is offline Regular Member
    Join Date
    Feb 2008
    Posts
    474
    Thanks
    46
    Thanked 165 Times in 109 Posts

    Default Re: Referrer Tutorial - How to Check Referrer Wireshark & HTTP Fox

    over 50 views and not a peep. . . wonder if I did something wrong.

  4. #3
    angelas111's Avatar
    angelas111 is offline Jr. VIP
    Join Date
    Jan 2009
    Location
    ohio
    Posts
    1,523
    Thanks
    814
    Thanked 974 Times in 481 Posts

    Default Re: Referrer Tutorial - How to Check Referrer Wireshark & HTTP Fox

    i think folks are trying to comprehend it.

  5. #4
    lotar is offline Newbies
    Join Date
    Aug 2011
    Posts
    33
    Thanks
    10
    Thanked 11 Times in 9 Posts

    Default Re: Referrer Tutorial - How to Check Referrer Wireshark & HTTP Fox

    I've noticed that technically insightful posts are lost on this forum... so don't take it personally. This was good information, and you enabled anyone who reads it to find the truth out for themselves... which is a better gift for any aspiring SEO than the other posts who are ultimately trying to sell something.

    This post teaches men to fish, I wish there were more posts like these or a section where the people who want to talk intellectually about this kind of stuff can converse.

  6. The Following User Says Thank You to lotar For This Useful Post:

    Lutherblissett (02-29-2012)

  7. #5
    Lutherblissett's Avatar
    Lutherblissett is offline Regular Member
    Join Date
    Feb 2008
    Posts
    474
    Thanks
    46
    Thanked 165 Times in 109 Posts

    Default Re: Referrer Tutorial - How to Check Referrer Wireshark & HTTP Fox

    Quote Originally Posted by lotar View Post
    I've noticed that technically insightful posts are lost on this forum... so don't take it personally. This was good information, and you enabled anyone who reads it to find the truth out for themselves... which is a better gift for any aspiring SEO than the other posts who are ultimately trying to sell something.
    Thanks man. . yeah people shut off on the length of it, but I can't tell you how valuable wireshark has been to me. Its just shows me so much stuff that was hidden behind the curtain for so long. It exposes all the dirty little tricks done, even by the big boys. I guess i should have made a post that said,

    SEE what GOOGLE SEES while you surf

  8. #6
    lotar is offline Newbies
    Join Date
    Aug 2011
    Posts
    33
    Thanks
    10
    Thanked 11 Times in 9 Posts

    Default Re: Referrer Tutorial - How to Check Referrer Wireshark & HTTP Fox

    Just in case anyone else is reading... you want to make sure that "Enable network name resolution" is checked under Capture->Options so you can see something other than the IP address on the output.

    Btw, if you're running this in Linux, make sure you run as root (after you compile the source)... I'm ashamed to say it, but that got me for a sec.

    This info can help you find out what products your competitors are promoting too.

    Interesting bit.. I went to yahoo there were about 8 requests for a server called "yw-in-f148.1e100.net"... and when you enter it into your url, it redirects straight to google's home page.

    ... this is likely what the Collusion does for you if you've been reading them on HN Daily.

  9. #7
    britcpa is offline Regular Member
    Join Date
    Mar 2010
    Posts
    464
    Thanks
    461
    Thanked 1,351 Times in 224 Posts

    Default Re: Referrer Tutorial - How to Check Referrer Wireshark & HTTP Fox

    FAO: Luther Blisset...

    i used to have a dog called Bill. a great big thing he was. a belgian cattle dog. a bouvier de flanders...

    fiercely independant, totally self-willed, bright as a button and always ready to play. when we walked through epping forest and i saw a squirrel runnning up a tree, i'd whistle his attention & point my finger at the squirrel and Bill would tense up knowing something was happening but he didnt know exactly what, so he'd freeze and stare AT my pointing finger, but not at where my finger was pointing TO.

    i guess he was expecting me to be throw something for him to chase, not realising i was pointing out something much more fun for him than any stick or ball.

    reading your excellent tutorial makes me realise that actually, in the scheme of things, im not much different to Bill: i know something important has been brought to my attention, but i cant quite see it yet!

    ive given you rep x 3 for the information as its clearly well written and knowledgable, even though its beyond my experience and abilities.

    in theory, i'd like to come back to your tutorial and investigate it some more, but whether or not that will happen any time soon is another matter - not when there are so many other sticks being thrown around for me to chase ;- )

    if you'd be interested in offering some more examples of how you use the information youre extracting with wireshark and explain a little bit more about what you use it for on a practical level, you'd certainly be helping me out and maybe some of the other residents here too.

    im not sure whether youre initial motivation in using that software was for 'defensive' purposes - i.e. to discover what info YOU are giving away to the search engines (wasnt sure if you were concerned about any other bodies/organisations than search engines), or whether it was to acquire marketing intelligence (but what would you do with it)?

    i got the impression it was actually for both purposes, but as you can see, im stuck in the fog....

    either way, thanks for taking the time to help raise the educational standards on bhw.

  10. #8
    Neo240's Avatar
    Neo240 is offline Power Member
    Join Date
    Sep 2011
    Location
    Vacation
    Posts
    540
    Thanks
    900
    Thanked 312 Times in 200 Posts

    Default Re: Referrer Tutorial - How to Check Referrer Wireshark & HTTP Fox

    Just downloaded Wireshark. Going to try this out. The documentation is blooody 294 pages long though. Anyway thanks for this excellent post. Its truly a rareity on BHW for such a techy post.

  11. #9
    Lutherblissett's Avatar
    Lutherblissett is offline Regular Member
    Join Date
    Feb 2008
    Posts
    474
    Thanks
    46
    Thanked 165 Times in 109 Posts

    Default Re: Referrer Tutorial - How to Check Referrer Wireshark & HTTP Fox

    Quote Originally Posted by britcpa View Post
    FAO: Luther Blisset...

    if you'd be interested in offering some more examples of how you use the information youre extracting with wireshark and explain a little bit more about what you use it for on a practical level, you'd certainly be helping me out and maybe some of the other residents here too.

    im not sure whether youre initial motivation in using that software was for 'defensive' purposes - i.e. to discover what info YOU are giving away to the search engines (wasnt sure if you were concerned about any other bodies/organisations than search engines), or whether it was to acquire marketing intelligence (but what would you do with it)?
    My initial work on it was to check the referrer. Many people on this forum blank or fake the referrer using a variety of techniques. Some work, some work sometimes, some work only on some browsers, etc. That is why i got into wireshark.

    There are a lot of ways to make money online, although if the sponsor sees the referrer they may ban you for TOS violations. Others once they see how your doing something (if its large enough) will steal your idea and cut you out. That is happening A LOT in many different ways today.

    Once I started looking at it, i started noticing some funny things. I started seeing massive google GET requests, then other requests for yahoo and alexa. The funny thing is I don't have these services on my browser (or so i thought)

    The web big web businesses of today Google, Facebook, Firefox, etc. are doing massive data capture. They come off like your buddy, and you have control, etc. . . but in reality a lot of stuff is behind the scenes, and unless you really read the TOS and understand the legal ease it's written in. . you don't see what they actually claim. I think a lot of data is tracked online in general, but when i saw what requests were going out, I was shocked.

  12. #10
    britcpa is offline Regular Member
    Join Date
    Mar 2010
    Posts
    464
    Thanks
    461
    Thanked 1,351 Times in 224 Posts

    Default Re: Referrer Tutorial - How to Check Referrer Wireshark & HTTP Fox

    Quote Originally Posted by Lutherblissett View Post
    My initial work on it was to check the referrer. Many people on this forum blank or fake the referrer using a variety of techniques. Some work, some work sometimes, some work only on some browsers, etc. That is why i got into wireshark.

    There are a lot of ways to make money online, although if the sponsor sees the referrer they may ban you for TOS violations. Others once they see how your doing something (if its large enough) will steal your idea and cut you out. That is happening A LOT in many different ways today.

    Once I started looking at it, i started noticing some funny things. I started seeing massive google GET requests, then other requests for yahoo and alexa. The funny thing is I don't have these services on my browser (or so i thought)

    The web big web businesses of today Google, Facebook, Firefox, etc. are doing massive data capture. They come off like your buddy, and you have control, etc. . . but in reality a lot of stuff is behind the scenes, and unless you really read the TOS and understand the legal ease it's written in. . you don't see what they actually claim. I think a lot of data is tracked online in general, but when i saw what requests were going out, I was shocked.
    thanks for clearing that up - on a separate note and following up on your comments regards firefox collecting data along with google and facebook, i remember reading that google donate $50m (cant recall whether thats per year or so far) to help fund firefox which is the metamorphis of netscape.

    at the time of reading that, it occurred to me that was a strategic move by google to block total dominance of the browser market by microsofts I.E. ahead of them developing their own competitive browser (chrome). its also questionable as to what level of data sharing there is between the two browsers (if any)....


Similar Threads

  1. StellaArtois Chrome Extension Tutorial!
    By StellaArtois in forum FaceBook
    Replies: 35
    Last Post: 03-30-2012, 09:52 PM
  2. Is there a way to check blanking the referrer is working?
    By BlackxHat in forum Cloaking and Content Generators
    Replies: 1
    Last Post: 01-28-2011, 08:55 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  




BlackHatWorld on Twitter BlackHatWorld on FaceBook


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108