1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scraping Japanese, Chinese with HRefer

Discussion in 'Black Hat SEO' started by jb2008, Jan 12, 2011.

  1. jb2008

    jb2008 Senior Member

    Joined:
    Jul 15, 2010
    Messages:
    1,158
    Likes Received:
    972
    Occupation:
    Scraping, Harvesting in the Corn Fields
    Location:
    On my VPS servers
    For some reason I can't seem to scrape chinese/japanese/russian or any UNICODE characters with hrefer. It either queries nonsense strings or it doesn't read anything at all. I thought hrefer could scrape all languages?
    I even tried changing the language settings but it doesn't make a difference either.

    I can scrape fine with normal characters, but anything special like russian, japanese and it's impossible. How do I scrape these?
     
  2. mondaytuesday

    mondaytuesday Newbie

    Joined:
    Jun 8, 2011
    Messages:
    1
    Likes Received:
    0
    Long time reader first time poster here at BHW. Just to answer your question there is a hrefer wordlist encoder out there I saw a few days ago. If any one has it I would love a copy.
     
  3. symss

    symss Regular Member

    Joined:
    Feb 14, 2009
    Messages:
    216
    Likes Received:
    206
    Search on google : "hrefer-word-list-encoder"