Help: Twitter scraping using python

C

Chris B

Newbie
Joined
Jun 29, 2017
Messages
26
Reaction score
2
I am trying to do something extremely simple in python yet somehow it's very difficult. All I want to do is write a python script that records the number of people a Twitter user is following, and the number of its followers. That's it.

Can anyone point me to a good package to do this? preferably not beautiful soup as that is overly complicated for what I want to do. I just want something like

[user: example_user, followers:9019, following:217]

The account I want to scrape is public. I have signed up for the API and I am little confused with it, but I do have the four codes I need for authorization.

Any help is appreciated.
 
Advertise on BHW
This is a little late, but I don't want to walk away from a post that has no answers. This might help someone in the future.


You do not need the API to do this, that is overly-complicated. You will need Selenium.


1. Go ahead and run "pip install Selenium" in command prompt, this will install Selenium for you.
2. Download the appropriate chromedriver for your computer. I'm not allowed to post links yet, but you should be able to find it easily.
3. There are tutorials for installing Selenium when you get stuck, it took me a while.
4. set up the driver in Python by importing selenium and creating a driver instance. (Many online tutorials for this)
5. Go straight to the twitter page of the person you want with driver.get(<Url of person's page goes here>)
6. Google how to find elements by xpathing. Will take a bit to learn but is extremely easy once you get the hang of it. Inspect the xpath of the elements you want, grab them, and done.


Hope this helps someone in the future,
cheers.
 
You can no longer do this in the API, as the API is no longer public and is now closed and you need to request access which is unlikely to be given.

Also, you don't need selenium. that is overkill

Scrape the page (requests is a good python library), then load into an HTML DOM parser (beautiful soup) or just regex (not ideal) the data you want out.

For using beautiiful soup you need to understand XPATH, css selectors. But that is fair straight forward. If you are wanting to move forward with python scraping, learn the basics of beautiful soup
 
I have been scraping twitter using Selenium. Is it possible to do it w/ requests w/o reverse engineering their Javascript? Twitter's source-code is just the same 40 lines no matter what search you request from it:

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>Twitter</title>
<style>
body {
background-color: #ffffff;
font-family: sans-serif;
}
a {
color: #1da1f2;
}
svg {
color: #1da1f2;
display: block;
fill: currentcolor;
height: 21px;
margin: 13px auto;
width: 24px;
}
</style>
</head>
<body>
<noscript>
<center>If you’re not redirected soon, please <a href="/">use this link</a>.</center>
</noscript>
<script nonce="QE1AL+vpGK3Yp8I+8YYpKg==">
document.cookie = "app_shell_visited=1;path=/;max-age=5";
location.replace(location.href.split("#")[0]);
</script>
</body>
</html>
 
Arc717 said:
I have been scraping twitter using Selenium. Is it possible to do it w/ requests w/o reverse engineering their Javascript? Twitter's source-code is just the same 40 lines no matter what search you request from it:
Click to expand...

You are making the requests wrong, the first pull you get those 40 lines, but a cookie is set and you are redirect to the correct page. Then all other requests after that just pull the correct page. No javascript required. The data you want, may be in the source as JSON, or may just be written into the page, I cannot remember. But I rewrote our twitter API bot to 100% use the website after they killed the API to the public without too much difficulty (some searches were awkward IIRC) so is definitely possible.
 
Working parser
Code: 
import re
from grab import Grab

g=Grab(timeout=30);
url='https://twitter.com/luulubuu';
g.go(url);

data={}
if g.doc.code==200:   
   try:
       data['User']=url;
       data['Following']=g.doc.select('//li[@class="ProfileNav-item ProfileNav-item--following"]').select('.//span[@class="ProfileNav-value"]').text();
       data['Following']=''.join(re.findall(r'\d+',data['Following']));
       data['Followers']=g.doc.select('//li[@class="ProfileNav-item ProfileNav-item--followers"]').select('.//span[@class="ProfileNav-value"]').text();
       data['Followers']=''.join(re.findall(r'\d+',data['Followers']));
       print data;
   except:
       print "Parse Error";
 
Arc717 said:
This is a little late, but I don't want to walk away from a post that has no answers. This might help someone in the future.


You do not need the API to do this, that is overly-complicated. You will need Selenium.


1. Go ahead and run "pip install Selenium" in command prompt, this will install Selenium for you.
2. Download the appropriate chromedriver for your computer. I'm not allowed to post links yet, but you should be able to find it easily.
3. There are tutorials for installing Selenium when you get stuck, it took me a while.
4. set up the driver in Python by importing selenium and creating a driver instance. (Many online tutorials for this)
5. Go straight to the twitter page of the person you want with driver.get(<Url of person's page goes here>)
6. Google how to find elements by xpathing. Will take a bit to learn but is extremely easy once you get the hang of it. Inspect the xpath of the elements you want, grab them, and done.


Hope this helps someone in the future,
cheers.
Click to expand...
wouldnt bs be faster than selenium?
 
Zetar2 said:
Selenium work via browser like chome
Click to expand...
Yes I know that, but OP only needs a basic scraping tool not a browser, using beautiful soup would surely be more efficient. Tbh I don't know much about this, but my friend makes reddit bots and they were all created using beautiful soup.
 
Zetar2 said:
Try python grab
Click to expand...

Thank you! Holy Cow, grab took me 10 minutes to get what I want, I've been looking for hours. Here's what I got in Python:

Code: 
from grab import Grab
g = Grab(timeout=30)
url = "I cant post urls on BHW yet"
resp = g.go(url)
what_you_want = resp.unicode_body()

I tried to do that with requests and sessions, but the entire doc would not load. Looks like grab is the way to go.

For anyone curious as to what this is, I'm trying to load the first 500 tweets of a certian search phrase, in this case "hello". After that I want to measure how often the phrase is tweeted, and how recently, basically getting timestamp of a tweet. All that's left now is to figure out how to scroll down twitter's page with grab.

Thanks again.
 
Zetar2 said:
Working parser
Code: 
import re
from grab import Grab

g=Grab(timeout=30);
url='https://twitter.com/luulubuu';
g.go(url);

data={}
if g.doc.code==200:  
   try:
       data['User']=url;
       data['Following']=g.doc.select('//li[@class="ProfileNav-item ProfileNav-item--following"]').select('.//span[@class="ProfileNav-value"]').text();
       data['Following']=''.join(re.findall(r'\d+',data['Following']));
       data['Followers']=g.doc.select('//li[@class="ProfileNav-item ProfileNav-item--followers"]').select('.//span[@class="ProfileNav-value"]').text();
       data['Followers']=''.join(re.findall(r'\d+',data['Followers']));
       print data;
   except:
       print "Parse Error";
Click to expand...


very nice, I don't do much in python any more, but next time I do, will check it out.

Is it mainly a parser with some http_request functionality, or has full support for making requests? post / get / json / xml, changing useragent etc
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top