C#: How can I scrape Twitter usernames?
- Thread starter kytro360
- Start date
xcubic
Regular Member
- Apr 24, 2008
- 444
- 588
I was wondering if anyone knows the code to scrape the usernames of of Twitter search. Any help would be appreicated, thanks
You can scrap someone's friends and the friends of the friends...
darshan1994
BANNED
- Oct 9, 2009
- 654
- 320
http://www.codeproject.com/KB/cs/TagBasedHtmlParser.aspx < Could work if you find out tags used by twitter for usernames.
supernour09
Regular Member
- Jul 20, 2011
- 383
- 62
i can help with imacro for 10 buck
scriptomania
Junior Member
- Dec 28, 2010
- 126
- 251
This might help you out: http://www.blackhatworld.com/blackhat-seo/black-hat-seo-tools/345255-free-python-codng-available.html
Something rather simple written by me in python, uses Twitters API (so you are restricted to a certain number of calls per hour). But feel free to improve on it if you like. I'll probably publish a new version pretty soon, just so busy with other projects right now.
Anyhow, if you wanna do it without the use of an API (correct me if I'm wrong guys, maybe twitter changed some stuff around.... idk), you'll need to use C#'s httpwebrequest class to send a simple "GET" request to download the desired webpage. Then parse it using regex.
Just look at the source code of the page and try to figure out a pattern. For example, suppose that usernames can be found between <span> tags. Your regex command would look something like:
Resources:
http requests: http://www.csharp-station.com/HowTo/HttpWebFetch.aspx
regex: http://msdn.microsoft.com/en-us/library/ms228595(v=vs.80).aspx#Y347
Something rather simple written by me in python, uses Twitters API (so you are restricted to a certain number of calls per hour). But feel free to improve on it if you like. I'll probably publish a new version pretty soon, just so busy with other projects right now.
Anyhow, if you wanna do it without the use of an API (correct me if I'm wrong guys, maybe twitter changed some stuff around.... idk), you'll need to use C#'s httpwebrequest class to send a simple "GET" request to download the desired webpage. Then parse it using regex.
Just look at the source code of the page and try to figure out a pattern. For example, suppose that usernames can be found between <span> tags. Your regex command would look something like:
Code:
<span>(.*?)</span>Resources:
http requests: http://www.csharp-station.com/HowTo/HttpWebFetch.aspx
regex: http://msdn.microsoft.com/en-us/library/ms228595(v=vs.80).aspx#Y347
xenon2010
Regular Member
- Apr 27, 2010
- 233
- 49
you can achieve that with any language...
if you are looking for scripting language go for PHP..
use cURL to download the page then use either regex or tags parser to extract the usernames..
personally I prefer to use regex (supported by all languages) its harder but more advanced.
if you are looking for desktop applications go for C# or VB. (although C# is better with syntax).. use webclient class or webpage control to download the page. then use regex or html tags parser to extract the data from the page..
now go and learn something useful
and rep me up
ashishthakkar
Junior Member
- Nov 4, 2011
- 127
- 10
I have done exactly this using visual basic 6.
Dont have the require tools to do it in c# or would code it.
~ASHISH
Dont have the require tools to do it in c# or would code it.
~ASHISH
kokoloko75
Elite Member
- Jan 1, 2011
- 1,627
- 1,950
Old thread...
The method has been shared by Xenon2010 :
1. Downlaod the page with GET.
2. Parse the code with regular epxression to extract username.
3. Put each username in list box.
Use the feed search from Twitter to get optimized datas, example :
Then, extract string like :
And remove "<uri>http://twitter.com/" and "</uri>" in the string.
Beny
The method has been shared by Xenon2010 :
1. Downlaod the page with GET.
2. Parse the code with regular epxression to extract username.
3. Put each username in list box.
Use the feed search from Twitter to get optimized datas, example :
Code:
http://search.twitter.com/search.atom?q=loveThen, extract string like :
Code:
<uri>http://twitter.com/IamRainey89</uri>And remove "<uri>http://twitter.com/" and "</uri>" in the string.
Beny
If you use the RSS method above, here is a regex expression to scrape out everything:
To use it, get the webpage with httpwebrequest or some other method of getting the source code, add:
to the top of your code.
Where you want to do the scraping use:
Code:
(?<=<uri>http://twitter.com/).*?(?=</uri>)To use it, get the webpage with httpwebrequest or some other method of getting the source code, add:
Code:
using System.Text.RegularExpressions;to the top of your code.
Where you want to do the scraping use:
Code:
Regex userRegex = new Regex("(?<=<uri>http://twitter.com/).*?(?=</uri>)");
foreach(Match item in userRegex.Matches(sourceCodeStringHere))
{
String matchedText = item.Value;
//Your code here
}