Twitter: Scraping Accounts

C

Cam Newton

Junior Member

Oct 4, 2017

103

31

• Nov 27, 2017

• #1

Hey guys,

I want to scrape a lot of accounts, for followers, very quickly. I know there is a rate limit using the API of 15 requests per 15 minutes. That would be 1440 per day. If I use 100 API keys, I could scrape 144,000 accounts per day. But I am using the same IP address. Does twitter check, or care, what IP address is sending API requests? Is this possible? Would my IP address get banned?

Or would it be better to scrape the accounts using scraping bots, without APIs?

Thanks,
Cam

 

141244

Senior Member

Oct 6, 2010

1,119

224

• Nov 27, 2017

• #2

check the java tool https://www.blackhatworld.com/seo/get-mass-twitter-id-scraper.707481/

you can scrape up to ~1,3m in 3-6h

 

Last edited: Nov 27, 2017

C

Cam Newton

Junior Member

Oct 4, 2017

103

31

• Nov 27, 2017
• Thread Starter Thread Starter

• #3

Thanks for the reply. I actually want the followers and followings of accounts. I know I will have to run a script to accomplish this. But I am just concerned about getting IP banned if I use 100+ API keys to do this.

 

ankit03

BANNED

Apr 3, 2016

2,737

277

• Nov 27, 2017

• #4

Cannot it be done without api just with browser based bot.... It will work according to me.

 

C

Cam Newton

Junior Member

Oct 4, 2017

103

31

• Nov 27, 2017
• Thread Starter Thread Starter

• #5

I assume twitter has anti-scraping features, like Instagram. Does twitter allow infinite scrolling through lists of followers? I believe this can’t be done properly in Instagram. I haven’t tried twitter yet though.

 

141244

Senior Member

Oct 6, 2010

1,119

224

• Nov 27, 2017

• #6

with the java tool you can scrape the max amount (~1,3 m) with only one api key

 

HoNeYBiRD

Elite Member

May 1, 2009

10,187

12,580

• Nov 27, 2017

• #7

﴾͡๏̯͡๏﴿.tk said:

with the java tool you can scrape the max amount (~1,3 m) with only one api key

Click to expand...

Yea, but if i can remember well, that tool gives you user ids instead of usernames and pretty much every bot require usernames to work with and can't do anything with user ids. It's true that user ids can be converted to usernames, but it's a rather slow process.