1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Any idea how to scrape wikis using Scrapebox?

Discussion in 'Black Hat SEO' started by mqs1000, Sep 8, 2013.

  1. mqs1000

    mqs1000 Power Member

    Joined:
    May 4, 2013
    Messages:
    533
    Likes Received:
    56
    Trying to make my own wiki list. I've tried adding the footprint inurl:"wiki" to the top and then adding niche related keywords in the box, but most of the URLs that get scraped don't contain wiki in the URL. Am I doing something wrong?
     
  2. mirajkadam

    mirajkadam Regular Member

    Joined:
    Jul 18, 2013
    Messages:
    245
    Likes Received:
    38
    Occupation:
    SEO
    Location:
    Mumbai
    It was posted on some scrapebox thread forgot it u can find it!
     
  3. mqs1000

    mqs1000 Power Member

    Joined:
    May 4, 2013
    Messages:
    533
    Likes Received:
    56
    I did a search already can't really find it
     
  4. Sephirot_90

    Sephirot_90 Registered Member

    Joined:
    May 20, 2011
    Messages:
    97
    Likes Received:
    36
    Location:
    Belgrade
    This is how i do it. I merge this footprint with keywords I want to target or a generic ones and just hit scrape. With this you will get a very diverse profile of wiki backlinks.

    Code:
    "Powered by TikiWiki""Theme: Fluid Index by Your Index"
    "Thank you for installing TikiWiki!"
    "Make sure to whitelist this domain to prevent registration emails being canned by your spam filter!"
    inurl:"tiki-register.php"
    inurl:"tiki-login.php"
    inurl:"tiki-forums.php"
    "Theme: Eatlon"
    "Theme: Fivealive - Kiwi"
    "Tema: Fivealive"
    "Tema: Fivealive - Lemon"
    "Theme: Coelesce"
    "Theme: Fivealive"
    "Theme: Feb12"
    "Theme: Jqui"
    "Theme: Ohia"
    "Thème: Strasa - Mono"
    "Powered by Tikiwiki CMS/Groupware"
    inurl:"tiki-index.php"
    "Powered by TikiWiki CMS/Groupware v2"
    "Theme: Strasa - Mono" wiki
    "Powered by WikkaWiki"
    Categories PageIndex Recent Changes Recently Commented "Login/Register" "[Source]" "[History]"
    "You will find more useful pages in the Wiki category or in the PageIndex"
    "Thanks for installing Wikka! This wiki runs on version"
    "Recently commented pages" "CategoryWiki"
    inurl:"/RecentlyCommented"
    inurl:"CategoryWiki"
    "There are no comments on this page." "Your hostname is" "Valid XHTML" "Valid CSS"
    "This is an alphabetical list of pages you can read on this server." "Login/Register"
    inurl:"/wikka/UserSettings"
    inurl:"/wikka.php?wakka=UserSettings"
    inurl:"wiki/RecentlyCommented"
    "pageindex" "recentchanges" "recentlycommented"
    "you only need to fill in when" categorywiki
    "dokuwiki.txt"
    inurl:/wiki/dokuwiki
    "wiki is licensed under"
    "Driven by DokuWiki"
    "DokuWiki supports some simple markup language"
    "The wiki, blog, calendar, and mailing list"
    "Log in to my page" "wikis"
    inurl:groups "log in to my page"
    "updates" "wikis" "blogs" "calendar" "mail"
    "Mac OS X Server - Wikis"
    "first" "prev" "1-20 of" "next" inurl:groups
    "What’s Hot" "Recent Changes"
    "What’s Hot" "Recent Changes" "Upcoming Events"
    "What’s Hot" "Recent Changes" "Upcoming Events" "Tags"
    "What’s Hot" "Recent Changes" "Upcoming Events" "Tags" "Edited"
    intitle:"Mac OS X Server"
    "Collaborate with online document creation, editing, and comments."
    "Deze pagina is het laatst bewerkt op" "Deze pagina is" "Aanmelden / registreren"
    "Diese Seite wurde zuletzt am" "Diese Seite wurde bisher" "Anmelden / Benutzerkonto erstellen"
    "This page was last modified on" "wiki"
    "This page was last modified" "This page has been accessed" "Log in / create account"
    "Wiki:About"
    inurl:"%C4%90%E1%BA%B7c_bi%E1%BB%87t:%C4%90%C4%83ng_nh%E1%BA%ADp" wiki
    inurl:"%D0%A3%D1%87%D0%B0%D1%81%D1%82%D0%BD%D0%B8%D0%BA:" wiki
    inurl:"%E4%BD%BF%E7%94%A8%E8%80%85:" wiki
    inurl:"%E7%89%B9%E5%88%A5:%E3%83%AD%E3%82%B0%E3%82%A4%E3%83%B3" wiki
    inurl:"%ED%8A%B9%EC%88%98%EA%B8%B0%EB%8A%A5:%EB%A1%9C%EA%B7%B8%EC%9D%B8" wiki
    inurl:"/mediawiki/index.php"
    inurl:"/wiki/index.php"
    inurl:":UserLogin"
    inurl:"Especial:Entrar" wiki
    inurl:"Especial:Registre_i_entrada" wiki
    inurl:"Especial:Userlogin" wiki
    inurl:"http://mediawiki."
    inurl:"http://wiki."
    inurl:"http://wikka."
    inurl:"Istimewa:Masuk_log" wiki
    inurl:"Pengguna:" wiki
    inurl:"Sp%C3%A9cial:Connexion" wiki
    inurl:"Speci%C3%A1lis:Bel%C3%A9p%C3%A9s" wiki
    inurl:"Speci%C3%A1ln%C3%AD:P%C5%99ihl%C3%A1sit" wiki
    inurl:"Speciaal:Aanmelden" wiki
    inurl:"Special:Inloggning" wiki
    inurl:"Special:UserLogin" wiki
    inurl:"Speciale:Entra" wiki
    inurl:"Speciel:Log_p%C3%A5" wiki
    inurl:"Specjalna:Zaloguj" wiki
    inurl:"Spezial:Anmelden" wiki
    inurl:"title=%CE%A3%CF%85%CE%B6%CE%AE%CF%84%CE%B7%CF%83%CE%B7_%CF%87%CF%81%CE%AE%CF%83%CF%84%CE%B7:"
    inurl:"title=%D0%9E%D0%B1%D0%B3%D0%BE%D0%B2%D0%BE%D1%80%D0%B5%D0%BD%D0%BD%D1%8F_%D0%BA%D0%BE%D1%80%D0%B8%D1%81%D1%82%D1%83%D0%B2%D0%B0%D1%87%D0%B0:"
    inurl:"title=%D0%9E%D0%B1%D1%81%D1%83%D0%B6%D0%B4%D0%B5%D0%BD%D0%B8%D0%B5_%D1%83%D1%87%D0%B0%D1%81%D1%82%D0%BD%D0%B8%D0%BA%D0%B0:"
    inurl:"title=%D0%A0%D0%B0%D0%B7%D0%B3%D0%BE%D0%B2%D0%BE%D1%80_%D1%81%D0%B0_%D0%BA%D0%BE%D1%80%D0%B8%D1%81%D0%BD%D0%B8%D0%BA%D0%BE%D0%BC:"
    inurl:"title=%D7%A9%D7%99%D7%97%D7%AA_%D7%9E%D7%A9%D7%AA%D7%9E%D7%A9:"
    inurl:"title=%D8%A8%D8%AD%D8%AB_%DA%A9%D8%A7%D8%B1%D8%A8%D8%B1:"
    inurl:"title=%E0%A4%B8%E0%A4%A6%E0%A4%B8%E0%A5%8D%E0%A4%AF_%E0%A4%B5%E0%A4%BE%E0%A4%B0%E0%A5%8D%E0%A4%A4%E0%A4%BE:"
    inurl:"title=%E0%A6%AC%E0%A7%8D%E0%A6%AF%E0%A6%AC%E0%A6%B9%E0%A6%BE%E0%A6%B0%E0%A6%95%E0%A6%BE%E0%A6%B0%E0%A7%80_%E0%A6%86%E0%A6%B2%E0%A6%BE%E0%A6%AA:"
    inurl:"title=%E0%B4%89%E0%B4%AA%E0%B4%AF%E0%B5%8B%E0%B4%95%E0%B5%8D%E0%B4%A4%E0%B4%BE%E0%B4%B5%E0%B4%BF%E0%B4%A8%E0%B5%8D%E0%B4%B1%E0%B5%86_%E0%B4%B8%E0%B4%82%E0%B4%B5%E0%B4%BE%E0%B4%A6%E0%B4%82:"
    inurl:"title=%E0%B8%84%E0%B8%B8%E0%B8%A2%E0%B8%81%E0%B8%B1%E0%B8%9A%E0%B8%9C%E0%B8%B9%E0%B9%89%E0%B9%83%E0%B8%8A%E0%B9%89:"
    inurl:"title=%E0%B8%84%E0%B8%B8%E0%B8%A2%E0%B9%80%E0%B8%81%E0%B8%B5%E0%B9%88%E0%B8%A2%E0%B8%A7%E0%B8%81%E0%B8%B1%E0%B8%9A%E0%B8%9C%E0%B8%B9%E0%B9%89%E0%B9%83%E0%B8%8A%E0%B9%89:"
    inurl:"title=%E4%BD%BF%E7%94%A8%E8%80%85%E8%A8%8E%E8%AB%96:"
    inurl:"title=%E5%88%A9%E7%94%A8%E8%80%85%E2%80%90%E4%BC%9A%E8%A9%B1:"
    inurl:"title=%E5%88%A9%E7%94%A8%E8%80%85%E3%83%BB%E3%83%88%E3%83%BC%E3%82%AF:"
    inurl:"title=%EC%82%AC%EC%9A%A9%EC%9E%90%ED%86%A0%EB%A1%A0:"
    inurl:"title=Anv%C3%A4ndardiskussion:"
    inurl:"title=Benutzer_Diskussion:"
    inurl:"title=Bruger_diskussion:"
    inurl:"title=Brugerdiskussion:"
    inurl:"title=Brukerdiskusjon:"
    inurl:"title=Discussion_utilisateur:"
    inurl:"title=Discussioni_utente:"
    inurl:"title=Diskuse_s_u%C5%BEivatelem:"
    inurl:"title=Diskuse_s_wikistou:"
    inurl:"title=Diskusia_s_redaktorom:"
    inurl:"title=Dyskusja_u%C5%BCytkownika:"
    inurl:"title=Kasutaja_arutelu:"
    inurl:"title=Lietot%C4%81ja_diskusija:"
    inurl:"title=Overleg_gebruiker:"
    inurl:"title=Pembicaraan_Pengguna:"
    inurl:"title=Special:UserLogin"
    inurl:"title=Szerkeszt%C5%91vita:"
    inurl:"title=Th%E1%BA%A3o_lu%E1%BA%ADn_Th%C3%A0nh_vi%C3%AAn:"
    inurl:"title=U%C5%BEivatel_diskuse:"
    inurl:"title=User:" wiki
    inurl:"title=Usu%C3%A1rio_Discuss%C3%A3o:"
    inurl:"title=Usuari_Discussi%C3%B3:"
    inurl:"title=Usuario:"
    inurl:"title=Usuario_discusi%C3%B3n:"
    inurl:"title=Utilizador_Discuss%C3%A3o:"
    inurl:"U%C5%BCytkownik:" wiki
    inurl:"User_talk:" wiki
    inurl:"Utente:" wiki
    inurl:"Utilisateur:" wiki
    inurl:"Utilizador:" wiki
    inurl:"wiki/index.php?title=" wiki
    "There is currently no text in this page, you can search for this page title in other pages or edit this page."
    inurl:"Special:Whatlinkshere"
    "what links here" "related changes" "special pages"
    "Powered By MediaWiki" inurl:wiki
    "MoinMoin Powered"
    "MoinMoin Powered" "Python Powered"
    "MoinMoin Powered" "Valid HTML 4.01"
    "MoinMoin Powered" "GPL licensed"
    "recentchanges" "findpage" "helpcontents"
    "Welcome to MoinMoin. You will find here the help pages for the wiki system itself."
    FrontPage RecentChanges FindPage Help Contents
    "Immutable Page" Info Attachments
    
    
    
     
    • Thanks Thanks x 2
  5. watsonovedades

    watsonovedades Senior Member

    Joined:
    Mar 10, 2011
    Messages:
    922
    Likes Received:
    233
    Location:
    Mexico
    what software do you use to post? GSA?
     
  6. mqs1000

    mqs1000 Power Member

    Joined:
    May 4, 2013
    Messages:
    533
    Likes Received:
    56
    Thanks for that. So you just type in keywords in the box, put that code in a text file and then merge them all together?

    Also, do you remove duplicate domains or do any other type of filtering once the scrape is finished?
     
  7. mqs1000

    mqs1000 Power Member

    Joined:
    May 4, 2013
    Messages:
    533
    Likes Received:
    56
    I use SeNuke XCR