Twitter Web Scraper?

Discussion in 'Twitter' started by ringtail, Dec 30, 2014.

  1. ringtail

    ringtail Registered Member

    Joined:
    Dec 29, 2013
    Messages:
    60
    Likes Received:
    20
    Hey guys,

    I'm looking for a different kind of twitter scraper, would any of you be able to point me in the right direction?

    Needed - Scrape the Name and Username of a user based on a search without having to have an account logged in or API credentials. Proxy rotation/decaptcha modules are a plus (just incase).

    Ex. Scrape all Scott's and their twitter usernames from a url - https://twitter.com/search?q=scott&src=typd&mode=users

    I want to be able to feed the URL so I can change the q='s param

    Let me know if you know where to find one. All I can find are bots and other scrapers that require logging in or API credentials and that wont work due to high volume.

    Thanks BHW!
     
  2. uncce

    uncce Junior Member

    Joined:
    Dec 24, 2014
    Messages:
    166
    Likes Received:
    27
    If you are ready to spend some $$ PM me
     
  3. Tobbe co

    Tobbe co Junior Member

    Joined:
    Sep 29, 2014
    Messages:
    171
    Likes Received:
    139
    If you have a budget I could create a custom bot for you, PM me with your skype details.
    If you're looking for a free solution I actually don't know. There's lots of twitter bots out there and maybe some premium bots have this feature, but I doubt any free one do.
     
  4. divok

    divok Senior Member

    Joined:
    Jul 21, 2010
    Messages:
    1,068
    Likes Received:
    648
    Location:
    .IN
    in your chrome install scraper plugin.

    then enter the xpath of the elements you want to extract on a particular page.

    Since twitter uses dom to load results , to scrape more scroll to bottom of the page before using the scraper plugin.
     
  5. ringtail

    ringtail Registered Member

    Joined:
    Dec 29, 2013
    Messages:
    60
    Likes Received:
    20
    Thanks for the responses guys. @DIVOK: thanks dude, I looked into the extensions but they didn't seem to suit my need for this project.

    Again, not looking for a bot (I have 'em) or a paid solution (I can hire a web scraper/data extractor off odesk). I ended up going with a one off solution using import.io. It's slow as FFFF but may get the job done as a proof of concept. If anyone else knows any other solutions, please let me know.