Simple Email Scraper - Mac OS X 10.4 and greater

[oldenstylehats]

Okay - it ran great for about an hour and then no more e-mails started showing up. So I tried to close the program and it stopped responding so I forced quitted.

Then now whenever I open it back up I can't get any e-mails showing in the box at all after I kick a search off.

What Time Delay settings were you running it at?

 

[HowNowBlackHat]

5 Seconds time delay - since then tried messing about with 'em all.

Even 30 second option. Only I'll wait half an hour and nothing will show now.

 

[HowNowBlackHat]

And please note I'm not getting a Google CAPTCHA message - I just tested.

Though perhaps building in proxy support would be a good idea on a future version? I'm new to scraping (you guys really got me interested though) and I'm excited about the possibilities, hence my... uh... "eagerness" to start.

 

[oldenstylehats]

And please note I'm not getting a Google CAPTCHA message - I just tested.

Though perhaps building in proxy support would be a good idea on a future version? I'm new to scraping (you guys really got me interested though) and I'm excited about the possibilities, hence my... uh... "eagerness" to start.

That is truly bizarre. You aren't by chance running it through a VPN, are you? When we were testing we had errors with some but not all VPNs. I don't think anyone has been hipped to the footprint. We just ran a pretty sizable test and had very good results.

Let me know via PM or email if you continue to have issues and we'll figure something out.

Edit: Also, try giving the queries a break for a few hours and see what happens.

 

[HowNowBlackHat]

Okay - I'll try again tomorrow. And no, definitely not a VPN.

Must be an issue with my config/setup/ISP or something. I'll drop you a PM if things don't improve.

It was running great up and the lil' blue bar got about 25% of the way through with about a thousand emails harvested and then it just... well. It just stopped.

I'll give it some beauty sleep though. Thanks for the prompt responses!

 

[oldenstylehats]

Okay - I'll try again tomorrow. And no, definitely not a VPN.

Must be an issue with my config/setup/ISP or something. I'll drop you a PM if things don't improve.

It was running great up and the lil' blue bar got about 25% of the way through with about a thousand emails harvested and then it just... well. It just stopped.

I'll give it some beauty sleep though. Thanks for the prompt responses!

No problem. We aim to please. When you test tomorrow, try switching around your keyword order and plurality too. Let me know how it goes.

 

[HowNowBlackHat]

Thanks mate! I've dropped you a PM with a few comments. Hope that's cool with you guys.

Off to bed - will post improvements if they materialize tomorrow.

 

[antx16]

Why not try to get it running with Tor that would certainly get rid of the ip tracking and footprint problem from googl

 

[oldenstylehats]

Why not try to get it running with Tor that would certainly get rid of the ip tracking and footprint problem from googl

Yeah, we're working on a back-end proxy rotator, but maybe a Tor implementation would be more effective. I'll definitely look into it. Thanks.

The footprint itself is pretty well randomized and I doubt G will be easily able to normalize without someone looking manually. From what HowNowBlackHat told me via PM, it may be his ISP filtering certain kinds of automated processes. He said he'd had issues in the past. We'll see. We're testing the waters of the Mac software world right now and definitely learning a lot as we're going on. We know the market is small, but we're definitely looking for more people to test and play.

 

[HowNowBlackHat]

Right - I've successfully ran the program three time... Funnily enough, getting to about 1.5k in the list before it stops dead and then won't work again for another 5 or 6 hours.

In other words - I can run it at this rate successfully four times a day before it stops. Hmph. Hope that made sense. If no-one else is having this then I'm left to presume that it MUST be my ISP - Virgin/NTLworld.

 

[HowNowBlackHat]

Um okay it's working today - and again. The same problem. Works and always gets to roughly the same amount of mails before stopping. And then won't rework for 4-6 hours. I'm almost certain it's an ISP thing.

Will find out if I can find a WiFi network to bunnyhop onto just for tonight and test.

Perhaps another feature suggestion... Make it possible for us to have multiple instances of SES running at once? Like multiple windows, each building individual lists for individual niches. So I could (once ISP is changed) kick it off before I go sleep and wake up to a few hundred thousand e-mails.

Now that'd be cool.

Oh... and a quick Q, I've done some testing and there seems to be no difference in having it set to "No time delay" as opposed to 5, 15 or even 30. What's the point of the feature? To throttle network traffic/supposedly make it harder to detect automated activity?

- David

 

[oldenstylehats]

Um okay it's working today - and again. The same problem. Works and always gets to roughly the same amount of mails before stopping. And then won't rework for 4-6 hours. I'm almost certain it's an ISP thing.

Will find out if I can find a WiFi network to bunnyhop onto just for tonight and test.

Perhaps another feature suggestion... Make it possible for us to have multiple instances of SES running at once? Like multiple windows, each building individual lists for individual niches. So I could (once ISP is changed) kick it off before I go sleep and wake up to a few hundred thousand e-mails.

Now that'd be cool.

Oh... and a quick Q, I've done some testing and there seems to be no difference in having it set to "No time delay" as opposed to 5, 15 or even 30. What's the point of the feature? To throttle network traffic/supposedly make it harder to detect automated activity?

- David

We've talked about allowing the option to run multiple campaigns at once, but haven't coded anything yet.

Also, the time delay is more of a suggestion than a set rule. Setting the time delay to 30 increases the statistical likelihood that a delay of 30 seconds will occur, but it is randomized so as not to set off any flags.

 

[HowNowBlackHat]

Roger that - thanks for clearing things up.

And at least I know multiple campaigns is at least being discussed then, along with proxy rotators.

Gee-whiz. If Adobe only gave this kind of support for their software. It's not as if they're lacking in manpower or anything. Keep it up guys!

 

[HowNowBlackHat]

Heya all... Long time no post.

Anyhoo, just thought I'd let you know that I'm still having cutout issues, even when using different ISPs. Tried wired ethernet connections and wireless. Just can't understand why the hell it's happening, if no-one else is having the issue.

Any ideas guys?

 

[ellie]

Yeah, we're working on a back-end proxy rotator, but maybe a Tor implementation would be more effective. I'll definitely look into it. Thanks.

The footprint itself is pretty well randomized and I doubt G will be easily able to normalize without someone looking manually. From what HowNowBlackHat told me via PM, it may be his ISP filtering certain kinds of automated processes. He said he'd had issues in the past. We'll see. We're testing the waters of the Mac software world right now and definitely learning a lot as we're going on. We know the market is small, but we're definitely looking for more people to test and play.

You'll find the google blockage is purely ip-related. Just like how SEO Elite when misconfigured without a proxy will get you a daddy smack from google queries.

I have a mac sitting around my office, if you want me to test anything for you, just give me a shout (or pm which would get my attention).

Cheers

 

[HowNowBlackHat]

Hey Blackhatzen dudes,

How's the TOR rotator going? I'm desperate for anything which (quite simply) will let me scrape for lengthy periods of time without the whole process grinding to a hault. Reason being, I'd love to kick off the program and come back a few hours later to find a few hundred thousand e-mails waiting. From what I've seen (10k in 10 mins) this should be more than possible, if only the whole thing didn't stop.

Keep up the great work!

- David

 

[HowNowBlackHat]

Long time bump. This hasn't been working in 6 months... Any plans to fix it?