1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Twitter Web Scraper?

Discussion in 'Twitter' started by ringtail, Dec 30, 2014.

  1. ringtail

    ringtail Registered Member

    Joined:
    Dec 29, 2013
    Messages:
    60
    Likes Received:
    20
    Hey guys,

    I'm looking for a different kind of twitter scraper, would any of you be able to point me in the right direction?

    Needed - Scrape the Name and Username of a user based on a search without having to have an account logged in or API credentials. Proxy rotation/decaptcha modules are a plus (just incase).

    Ex. Scrape all Scott's and their twitter usernames from a url - https://twitter.com/search?q=scott&src=typd&mode=users

    I want to be able to feed the URL so I can change the q='s param

    Let me know if you know where to find one. All I can find are bots and other scrapers that require logging in or API credentials and that wont work due to high volume.

    Thanks BHW!
     
  2. uncce

    uncce Junior Member

    Joined:
    Dec 24, 2014
    Messages:
    166
    Likes Received:
    27
    If you are ready to spend some $$ PM me
     
  3. Tobbe co

    Tobbe co Junior Member

    Joined:
    Sep 29, 2014
    Messages:
    171
    Likes Received:
    139
    If you have a budget I could create a custom bot for you, PM me with your skype details.
    If you're looking for a free solution I actually don't know. There's lots of twitter bots out there and maybe some premium bots have this feature, but I doubt any free one do.
     
  4. divok

    divok Senior Member

    Joined:
    Jul 21, 2010
    Messages:
    1,056
    Likes Received:
    644
    Location:
    .IN
    in your chrome install scraper plugin.

    then enter the xpath of the elements you want to extract on a particular page.

    Since twitter uses dom to load results , to scrape more scroll to bottom of the page before using the scraper plugin.
     
  5. ringtail

    ringtail Registered Member

    Joined:
    Dec 29, 2013
    Messages:
    60
    Likes Received:
    20
    Thanks for the responses guys. @DIVOK: thanks dude, I looked into the extensions but they didn't seem to suit my need for this project.

    Again, not looking for a bot (I have 'em) or a paid solution (I can hire a web scraper/data extractor off odesk). I ended up going with a one off solution using import.io. It's slow as FFFF but may get the job done as a proof of concept. If anyone else knows any other solutions, please let me know.