ScrapeBox question

NeVeREire · Sep 21, 2018

Hi,

I'm trying to build/setup a custom grabber to do the following

1) Collect all the followers from a twitter account
2) Check each of those accounts and scrape there location

I've looked at Looplines tutorial about customer grabber, but can't get it working.

I can't figure out to even scrape the followers of an account.

I tested the grabber to try get the location of random accounts i added using the following
before_after=location":"|","url

but doesn't work.

Any help would be great and if Loopline reads this please help

thanks.

SiddharthW · Sep 21, 2018

@loopline

NeVeREire · Sep 21, 2018

SiddharthW said:
@loopline

Thanks for tagging him. Didn't think of that.

loopline · Sep 22, 2018

Let me start by saying that I think its not possible because I recall helping someone else with scraping twitter users and we found they couldnt do it. But thats been a minute and I don't remember why, so lets try.

Can you give a few of the exact twitter urls you are after?

Having a look at
https://twitter.com/garyvee/followers

before_after=data-screen-name="|"

will get a handful of followers. The remainder is loaded by a script not by page numbers (unless you can find a url structure I didn't see), in which case you can't get them. So you might get 10 or less followers, but thats it. Scrapebox doesn't execute scripts.

For location
before_after="location":"|"

should work.

NeVeREire · Sep 24, 2018

loopline said:
Let me start by saying that I think its not possible because I recall helping someone else with scraping twitter users and we found they couldnt do it. But thats been a minute and I don't remember why, so lets try.

Can you give a few of the exact twitter urls you are after?

Having a look at

before_after=data-screen-name="|"

will get a handful of followers. The remainder is loaded by a script not by page numbers (unless you can find a url structure I didn't see), in which case you can't get them. So you might get 10 or less followers, but thats it. Scrapebox doesn't execute scripts.

For location
before_after="location":"|"

should work.

Hi Loopline and thanks for the reply

In this case i'm looking at SEMRush followers ( can't add a link )

Lets say I had a list of all the [@] twitter-names - Could I load them in a custom grabber and then run the location before_after ?

It's just getting those followers which will be the problem now.

loopline said:
Let me start by saying that I think its not possible because I recall helping someone else with scraping twitter users and we found they couldnt do it. But thats been a minute and I don't remember why, so lets try.

Can you give a few of the exact twitter urls you are after?

before_after=data-screen-name="|"

will get a handful of followers. The remainder is loaded by a script not by page numbers (unless you can find a url structure I didn't see), in which case you can't get them. So you might get 10 or less followers, but thats it. Scrapebox doesn't execute scripts.

For location
before_after="location":"|"

should work.

I tried what you said for screen name and i got a Error 500 in the status.

Same error with the location one.

I added about URL's to different accounts and added the before_after="location":"|" you said.. but got Error 500

I'm beginning to think Twitter have blocked this type of stuff.

Anything i try run on it doesn't work.

Even grabbing links by crawling site - grabber doesn't work

But it does work on other social media sites i just checked.

loopline · Sep 25, 2018

NeVeREire said:
Hi Loopline and thanks for the reply

In this case i'm looking at SEMRush followers ( can't add a link )

Lets say I had a list of all the [@] twitter-names - Could I load them in a custom grabber and then run the location before_after ?

It's just getting those followers which will be the problem now.

You need to load in urls, so probably twitter has a format like

twitter.com/user

or something that you could fabricate all the urls.

NeVeREire said:
Same error with the location one.

I added about URL's to different accounts and added the before_after="location":"|" you said.. but got Error 500

Are you using proxies? If so it could be the proxies, but otherwise yes its probably twitter blocking it.

You can try using other user agents, like the one from your browser is a good start.

NeVeREire · Sep 25, 2018

I am using proxies.

I am also using the full URLs

Is it possible for you to do a test and see if you get any errors ? .. Maybe screenshot the mask so I can see how you set it up. I'm running the latest scrapebox from a VPS with about 30 clear proxies.

loopline · Sep 26, 2018

NeVeREire said:
I am using proxies.

I am also using the full URLs

Is it possible for you to do a test and see if you get any errors ? .. Maybe screenshot the mask so I can see how you set it up. I'm running the latest scrapebox from a VPS with about 30 clear proxies.

Im slightly still lost on what you are doing. So lets take it basic.

you want the listed on peoples twitter url, correct? Can you PM me some sample urls that are not working so I can try them?

Also can you clarify, is that what you want, because I don't understand how SEM rush ties into this. Im just missing 1 piece somewhere and it doesn't make sense.

NeVeREire · Sep 26, 2018

loopline said:
Im slightly still lost on what you are doing. So lets take it basic.

you want the listed on peoples twitter url, correct? Can you PM me some sample urls that are not working so I can try them?

Also can you clarify, is that what you want, because I don't understand how SEM rush ties into this. Im just missing 1 piece somewhere and it doesn't make sense.

Sorry I can't PM for some reason .

I will try explain as best I can
1) It doesn't have to be SEMRush's twitter account. I picked these as an example. It can be any account - As you said we can't scrape followers so i'm not looking to scrape followers anymore.

2) I do want to scrape a users Location - So what I was doing is loading in any random twitter accounts ( I don't want to post links here so just select 10/20 at random ) using the full Twitter URL twitter.com/username

So lets say I get a list from where with 10000 twitter URL's I'd like to be able to scrape each of these URL's and have the location collected. Maybe I need more as when the report is done i'd like to be able to see 1) the URL i scrapped / 2) There Location

This way I can find the locations Id like and open there accounts and contact them. I'm using this to try hire people in certain counties . I picked SEMRush as the person's i'm looking to hire should be interested in SEO

I hope that helps and thank you.

loopline · Sep 26, 2018

NeVeREire said:
Sorry I can't PM for some reason .

I will try explain as best I can
1) It doesn't have to be SEMRush's twitter account. I picked these as an example. It can be any account - As you said we can't scrape followers so i'm not looking to scrape followers anymore.

2) I do want to scrape a users Location - So what I was doing is loading in any random twitter accounts ( I don't want to post links here so just select 10/20 at random ) using the full Twitter URL twitter.com/username

So lets say I get a list from where with 10000 twitter URL's I'd like to be able to scrape each of these URL's and have the location collected. Maybe I need more as when the report is done i'd like to be able to see 1) the URL i scrapped / 2) There Location

This way I can find the locations Id like and open there accounts and contact them. I'm using this to try hire people in certain counties . I picked SEMRush as the person's i'm looking to hire should be interested in SEO

I hope that helps and thank you.

Ok, thanks I understand now.

This is it
before_after=<div class="location">|</div>

Based on the default user agent that the custom grabber is using, twitter redirects to the mobile version. Mobile version code is different, so thats it. I use a program called http debugger pro. You can google, it has a 14 day trial. It lets you see the actual responses so you see the same code as scrapebox and you can see all the request etc...

NeVeREire · Sep 27, 2018

loopline said:
Ok, thanks I understand now.

This is it
before_after=<div class="location">|</div>

Based on the default user agent that the custom grabber is using, twitter redirects to the mobile version. Mobile version code is different, so thats it. I use a program called http debugger pro. You can google, it has a 14 day trial. It lets you see the actual responses so you see the same code as scrapebox and you can see all the request etc...

Thanks i've added that and tested it and i'm still getting Error 500 - Will that debugger pro tool stop the Errors ? Is it needed. ?

I've attached 2 screen shots. Showing how i've setup the grabber. Can you have a look and let me know what looks wrong.

Thanks.

loopline · Sep 28, 2018

500 is a server error. Either twitter is blocking your IP or your proxies are having issues. are you using proxies? Try without proxies if so, else if your not using proxies try with proxies. your setup looks identical to me and mine works.

I wouldnt recommend http debugger in this case, it won't help you.

NeVeREire · Sep 28, 2018

loopline said:
500 is a server error. Either twitter is blocking your IP or your proxies are having issues. are you using proxies? Try without proxies if so, else if your not using proxies try with proxies. your setup looks identical to me and mine works.

I wouldnt recommend http debugger in this case, it won't help you.

IT worked !! thank you .. it was my proxies .. I did have 30 and they appears ok when i tested them.. I turned them off and it worked !

thank you so much.

blackcars · Sep 28, 2018

Proxies are very important in this case. Without the proxy there will be no further steps will be done.

loopline · Sep 29, 2018

NeVeREire said:
IT worked !! thank you .. it was my proxies .. I did have 30 and they appears ok when i tested them.. I turned them off and it worked !

thank you so much.

Great, glad its working for you!

ScrapeBox question

NeVeREire

Registered Member

SiddharthW

Power Member

NeVeREire

Registered Member

loopline

Elite Member

NeVeREire

Registered Member

loopline

Elite Member

NeVeREire

Registered Member

loopline

Elite Member

NeVeREire

Registered Member

loopline

Elite Member

NeVeREire

Registered Member

Attachments

loopline

Elite Member

NeVeREire

Registered Member

blackcars

Registered Member

loopline

Elite Member

Main Menu

Marketplace

Making Money

BlackHat World