Permanently Closed Marketplace Sales Thread

[CFT]

Is it safe to use without proxies

 

[Buzzika]

Is it safe to use without proxies

I would say depends on the site you're scraping. But that's not an official statement. I am not from the Crawlie team.

As long as the website you're scraping doesn't ban your IP, I don't see why not.

 

[Pruvia]

is the free trial still on? if so please count me in.. thanks

The free trial is still available!

Are there any discount coupons? I just signed up for the trial and if I like it, I will be upgrading tot he paid plan very soon.

I've send you a discount code for 20% OFF your first 2 months.

Unable to edit my previous post so will have to double post...

Few suggestions:

* Add option to test proxies
* Is this multithreaded? How can we control thread count? If it isn't multi-threaded this would be disappointing.

Also, I tried crawling avvo.com from my UK based VPS and it says crawl finished after what seems like 3-5 seconds only and the "Pages crawled" remains at zero.
It is crawling the other sites just fine. I wonder why that is.

Thanks for the suggestions.

* Add option to test proxies
This is a good idea, I'll look into adding this.

At the moment you can't change the amount of threads, I plan on adding this in the future.

Is it safe to use without proxies

As Buzzika said, it's possible to use Crawlerie without using proxies and have no problems (most of my testing was done without proxies and never ran into any problems where I was blocked). Some sites can block you if they detect you making too many requests, if that's the case just add some proxies and run the crawler again.

 

[johan22]

Is the free trial still available?

 

[ErossplayLa]

Works for web 2.0 ?

 

[Eddamoun]

The free trial signup page is not working!