Building Online URL Scraper - Need Testers...

[SpamHat]

Hey

Writing an AJAX app that scrapes Google URLs and have gone over it a few times but I'm probably missing something.

Anyone who can be bothered can you check this out:
http://gScrape.com
(Needs FireFox 3.5)

Looking for bugs etc

Cheers

 

[SpamHat]

Yes, but with Goog et al you would search, copy and paste each url to a text file, go to page 2, copy and paste each..... etc

With this it's a few clicks and you can have thousands of URLs.

You seem pretty new (as you say) so here's some examples:


Someone needs content for a website.
They enter a google footprint into gScrape like this:
site:goarticles.com bingo
And that will return a nice big list of article URLs on goarticles.
You can then use this list to scrape the articles with other tools.



If you've got some seo software like autopligg or whatever and are looking for pligg urls (or any other site urls) to plug into the software you can do something like this:
"powered by pligg"


Or you want a big list of do-follow blogs to comment on:
"powered by wordpress" +********


and so on.

It's a flexible way of powering other software and squeezing some of the goodness out of Google for other purposes.




btw: how did you get on with the actual functioning of the app? Are you using FF 3.5+?

 

[SpamHat]

Thanks for testing it out

The URLs that are returned are similar to Google's so to get better results you just need to tune your footprint.

For example, to get articles for AC you could use this footprint:
site:associatedcontent.com inurl:article intitle:"YOUR KEYWORD(S) HERE"

That gives me an idea though... a nice one.
Thanks again

 

[iglow]

didnt work too godo for me
nothing scraped - ive got newest fff

 

[SpamHat]

@scudder: It only needs 3.5+. I'm running 3.5.2 like you.

@iglow: any JS errors? You did click the LoadUrls button after it finished, right?

 

[redsasy]

Works great with FF 3.5.1

 

[mrsmirf]

I have FF 3.5.3 . Your website didn't work. It just says "Scraping: 'Keyword' " and nothing happens afterwords.

 

[fatboy]

Working fine here mate - cool site
No probs, everything worked as it should of for me.

 

[acp0rnstar]

Same thing here. It just hangs. I tried with FF 3.5.3 and Chrome 4.0xx

I have FF 3.5.3 . Your website didn't work. It just says "Scraping: 'Keyword' " and nothing happens afterwords.

 

[Velvet]

working fine for me with FF 3.5.3
registered for the huge amount of urls

 

[Mesach]

I tested on 3.5.3 and it works great, however i did 2 queries, and when i ran the 2nd one, the results from the first completely unrelated query came up at the beginning of the results.

 

[SpamHat]

I tested on 3.5.3 and it works great, however i did 2 queries, and when i ran the 2nd one, the results from the first completely unrelated query came up at the beginning of the results.

Yeah it's built so it remembers the URLs from ALL queries so you can download them at the end.

To clear the cache just refresh the page.

 

[iPwnJ00]

I'm using FF 3.7a1pre, otherwise known as Minefield and it says that I need FF 3.5....

But I can just spoof it to 3.5.

Why does it need FF3.5? What function are you taking from FF to make it FF-only?

 

[iPwnJ00]

Looks good, works well.

Found a bug though. If you re-search for the same keyword, it doesn't check whether the URL is already in memory and therefore will just repeat the same URLs.

 

[SpamHat]

I'm using FF 3.7a1pre, otherwise known as Minefield and it says that I need FF 3.5....

But I can just spoof it to 3.5.

Why does it need FF3.5? What function are you taking from FF to make it FF-only?

I'll take a look at the useragent verification - it should be letting you through. It could technically be used with IE8 as well but would need some changes - I'm planning to do this sometime.

I needed a way to manage all the urls in the users browser without the browser crashing so am using something only modern browsers have

The best thing about it is that once the page has loaded in your browser it doesn't talk to the server - all the scraping is done client side with JS.

Looks good, works well.

Found a bug though. If you re-search for the same keyword, it doesn't check whether the URL is already in memory and therefore will just repeat the same URLs.

hmmmmmmmm

I just searched for "cat" without quotes, twice and didn't get any dupes.

It will give you multiple URLs from the same domain if it finds them, but shouldn't give dupes.

What keyword were you using?

 

[submitoke]

my FF is not on 3.5 but ive use this...

http://goohackle.com/scripts/google_parser.php

 

[HenryHavoc]

After you make a search and clear the list, a research adds to the number of URLS scraped. Idk if that's on purpose. AMAZING overall! Bookmarking now

 

[SpamHat]

my FF is not on 3.5 but ive use this...

http://goohackle.com/scripts/google_parser.php

There are ton's of small scripts like that - it requires a captha and only gives a few results. I wanted to make something that was fast, didn't get banned (ever) and gives unlimited results.

After you make a search and clear the list, a research adds to the number of URLS scraped. Idk if that's on purpose. AMAZING overall! Bookmarking now

Yeah, if I'm understanding you correctly, yes it's on purpose.

Thanks for the feedback guys :cool2:

 

[shadowpwner]

You, sir, win a couple internets.

Awesome script, works fine. However, signing up, i had to try twice. This is probably aweber's fault.

Some advice: the screen froze up when i was trying to get 2000 keywords. Could you output it as a downloadable file, instead of on browser text?

 

[dizz]

If you need anymore testers, we would like to try it out.

Thanks