Scraping Japanese, Chinese with HRefer

Discussion in 'Black Hat SEO' started by jb2008, Jan 12, 2011.

  1. jb2008

    jb2008 Senior Member

    Joined:
    Jul 15, 2010
    Messages:
    1,158
    Likes Received:
    972
    Occupation:
    Scraping, Harvesting in the Corn Fields
    Location:
    On my VPS servers
    For some reason I can't seem to scrape chinese/japanese/russian or any UNICODE characters with hrefer. It either queries nonsense strings or it doesn't read anything at all. I thought hrefer could scrape all languages?
    I even tried changing the language settings but it doesn't make a difference either.

    I can scrape fine with normal characters, but anything special like russian, japanese and it's impossible. How do I scrape these?
     
  2. mondaytuesday

    mondaytuesday Newbie

    Joined:
    Jun 8, 2011
    Messages:
    1
    Likes Received:
    0
    Long time reader first time poster here at BHW. Just to answer your question there is a hrefer wordlist encoder out there I saw a few days ago. If any one has it I would love a copy.
     
  3. symss

    symss Regular Member

    Joined:
    Feb 14, 2009
    Messages:
    217
    Likes Received:
    206
    Search on google : "hrefer-word-list-encoder"