1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Who's clever and knows how to stop geo redirect when scraping google?

Discussion in 'BlackHat Lounge' started by jamie3000, Feb 20, 2017.

  1. jamie3000

    jamie3000 Supreme Member

    Joined:
    Jun 30, 2014
    Messages:
    1,414
    Likes Received:
    655
    Occupation:
    Finance coder looking for semi-retirement
    Location:
    uk
    Who's clever and knows how to stop geographical redirect when scraping google?

    This is scraping so can't just put google.com/ncr in. It has to be a URL parameter.

    Tried setting hl=gb-us etc and does't seem to work just 301's all my scrape requests and redirects me to google.co.uk.
     
  2. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,787
    Likes Received:
    11,434
    Occupation:
    COINZ
    Location:
    BUYAH
    Home Page:
  3. jamie3000

    jamie3000 Supreme Member

    Joined:
    Jun 30, 2014
    Messages:
    1,414
    Likes Received:
    655
    Occupation:
    Finance coder looking for semi-retirement
    Location:
    uk
    this is for URL's programmatically called so has to be a url parameter in the query, I'm pretty sure its possible still.
     
  4. Brian Alexander

    Brian Alexander Regular Member UnGagged Attendee

    Joined:
    Aug 12, 2016
    Messages:
    205
    Likes Received:
    116
    Gender:
    Male
    You're writing a scraper but don't know how to send a cookie?


    Save the cookie you get from the /ncr request and include it in any future requests.
     
  5. Dhiraj Pandey

    Dhiraj Pandey Newbie

    Joined:
    Feb 20, 2017
    Messages:
    0
    Likes Received:
    0
    Gender:
    Male
    Have you tried "gl" parameter in search url? &gl=US
     
  6. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,787
    Likes Received:
    11,434
    Occupation:
    COINZ
    Location:
    BUYAH
    Home Page:
    Brian Alexander has given you the solution.

    First you crawl the ncr link, G will send you a cookie. You send that cookie back, it'll contain the code to avoid redirection. Any scraping tool will have a default cookie jar - you shouldn't even bother with the cookies, just crawl the ncr link once and your scraper won't be redirected.
     
  7. jamie3000

    jamie3000 Supreme Member

    Joined:
    Jun 30, 2014
    Messages:
    1,414
    Likes Received:
    655
    Occupation:
    Finance coder looking for semi-retirement
    Location:
    uk
    I've found it. in case anyone else ever needs this.

    &gfe_rd=cr&gws_rd=cr
     
    • Thanks Thanks x 1