Building Online URL Scraper - Need Testers...

SpamHat

Junior Member

Apr 27, 2009

151

67

• Sep 4, 2009

• #1

Hey

Writing an AJAX app that scrapes Google URLs and have gone over it a few times but I'm probably missing something.

Anyone who can be bothered can you check this out:
http://gScrape.com
(Needs FireFox 3.5)

Looking for bugs etc

Cheers

 

SpamHat

Junior Member

Apr 27, 2009

151

67

• Sep 4, 2009
• Thread Starter Thread Starter

• #2

Yes, but with Goog et al you would search, copy and paste each url to a text file, go to page 2, copy and paste each..... etc

With this it's a few clicks and you can have thousands of URLs.

You seem pretty new (as you say) so here's some examples:


Someone needs content for a website.
They enter a google footprint into gScrape like this:
site:goarticles.com bingo
And that will return a nice big list of article URLs on goarticles.
You can then use this list to scrape the articles with other tools.



If you've got some seo software like autopligg or whatever and are looking for pligg urls (or any other site urls) to plug into the software you can do something like this:
"powered by pligg"


Or you want a big list of do-follow blogs to comment on:
"powered by wordpress" +********


and so on.

It's a flexible way of powering other software and squeezing some of the goodness out of Google for other purposes.




btw: how did you get on with the actual functioning of the app? Are you using FF 3.5+?

 

SpamHat

Junior Member

Apr 27, 2009

151

67

• Sep 5, 2009
• Thread Starter Thread Starter

• #3

Thanks for testing it out

The URLs that are returned are similar to Google's so to get better results you just need to tune your footprint.

For example, to get articles for AC you could use this footprint:
site:associatedcontent.com inurl:article intitle:"YOUR KEYWORD(S) HERE"

That gives me an idea though... a nice one.
Thanks again

 

iglow

Elite Member

Feb 20, 2009

2,254

870

• Sep 5, 2009

• #4

didnt work too godo for me
nothing scraped - ive got newest fff

 

SpamHat

Junior Member

Apr 27, 2009

151

67

• Sep 5, 2009
• Thread Starter Thread Starter

• #5

@scudder: It only needs 3.5+. I'm running 3.5.2 like you.

@iglow: any JS errors? You did click the LoadUrls button after it finished, right?

 

R

redsasy

Newbie

Feb 16, 2009

44

80

• Sep 11, 2009

• #6

Works great with FF 3.5.1

 

mrsmirf

Regular Member

Aug 2, 2008

231

169

• Sep 21, 2009

• #7

I have FF 3.5.3 . Your website didn't work. It just says "Scraping: 'Keyword' " and nothing happens afterwords.

 

fatboy

Elite Member

Aug 13, 2008

1,615

3,262

• Sep 21, 2009

• #8

Working fine here mate - cool site
No probs, everything worked as it should of for me.

 

acp0rnstar

Junior Member

Sep 29, 2008

168

155

• Sep 23, 2009

• #9

Same thing here. It just hangs. I tried with FF 3.5.3 and Chrome 4.0xx

mrsmirf said:

I have FF 3.5.3 . Your website didn't work. It just says "Scraping: 'Keyword' " and nothing happens afterwords.

Click to expand...

 

Velvet

Junior Member

Feb 18, 2009

199

148

• Sep 23, 2009

• #10

working fine for me with FF 3.5.3
registered for the huge amount of urls

 

M

Mesach

Junior Member

Oct 1, 2009

108

40

• Oct 1, 2009

• #11

I tested on 3.5.3 and it works great, however i did 2 queries, and when i ran the 2nd one, the results from the first completely unrelated query came up at the beginning of the results.

 

SpamHat

Junior Member

Apr 27, 2009

151

67

• Oct 4, 2009
• Thread Starter Thread Starter

• #12

Mesach said:

I tested on 3.5.3 and it works great, however i did 2 queries, and when i ran the 2nd one, the results from the first completely unrelated query came up at the beginning of the results.

Click to expand...

Yeah it's built so it remembers the URLs from ALL queries so you can download them at the end.

To clear the cache just refresh the page.

 

I

iPwnJ00

Junior Member

Mar 10, 2009

133

22

• Oct 4, 2009

• #13

I'm using FF 3.7a1pre, otherwise known as Minefield and it says that I need FF 3.5....

But I can just spoof it to 3.5.

Why does it need FF3.5? What function are you taking from FF to make it FF-only?

 

I

iPwnJ00

Junior Member

Mar 10, 2009

133

22

• Oct 4, 2009

• #14

Looks good, works well.

Found a bug though. If you re-search for the same keyword, it doesn't check whether the URL is already in memory and therefore will just repeat the same URLs.

 

SpamHat

Junior Member

Apr 27, 2009

151

67

• Oct 4, 2009
• Thread Starter Thread Starter

• #15

iPwnJ00 said:

I'm using FF 3.7a1pre, otherwise known as Minefield and it says that I need FF 3.5....

But I can just spoof it to 3.5.

Why does it need FF3.5? What function are you taking from FF to make it FF-only?

Click to expand...

I'll take a look at the useragent verification - it should be letting you through. It could technically be used with IE8 as well but would need some changes - I'm planning to do this sometime.

I needed a way to manage all the urls in the users browser without the browser crashing so am using something only modern browsers have

The best thing about it is that once the page has loaded in your browser it doesn't talk to the server - all the scraping is done client side with JS.

iPwnJ00 said:

Looks good, works well.

Found a bug though. If you re-search for the same keyword, it doesn't check whether the URL is already in memory and therefore will just repeat the same URLs.

Click to expand...

hmmmmmmmm

I just searched for "cat" without quotes, twice and didn't get any dupes.

It will give you multiple URLs from the same domain if it finds them, but shouldn't give dupes.

What keyword were you using?

 

S

submitoke

Newbie

Dec 6, 2008

16

0

• Oct 5, 2009

• #16

my FF is not on 3.5 but ive use this...

Code:

http://goohackle.com/scripts/google_parser.php

 

HenryHavoc

Senior Member

Mar 24, 2008

1,166

2,215

• Oct 6, 2009

• #17

After you make a search and clear the list, a research adds to the number of URLS scraped. Idk if that's on purpose. AMAZING overall! Bookmarking now

 

SpamHat

Junior Member

Apr 27, 2009

151

67

• Oct 6, 2009
• Thread Starter Thread Starter

• #18

submitoke said:

my FF is not on 3.5 but ive use this...

Code:

http://goohackle.com/scripts/google_parser.php

Click to expand...

There are ton's of small scripts like that - it requires a captha and only gives a few results. I wanted to make something that was fast, didn't get banned (ever) and gives unlimited results.

HenryHavoc said:

After you make a search and clear the list, a research adds to the number of URLS scraped. Idk if that's on purpose. AMAZING overall! Bookmarking now

Click to expand...

Yeah, if I'm understanding you correctly, yes it's on purpose.

Thanks for the feedback guys :cool2:

 

S

shadowpwner

Regular Member

Apr 19, 2009

299

73

• Oct 11, 2009

• #19

You, sir, win a couple internets.

Awesome script, works fine. However, signing up, i had to try twice. This is probably aweber's fault.

Some advice: the screen froze up when i was trying to get 2000 keywords. Could you output it as a downloadable file, instead of on browser text?

 

Last edited: Oct 11, 2009

dizz

Elite Member

May 19, 2009

2,069

1,799

• Oct 11, 2009

• #20

If you need anymore testers, we would like to try it out.

Thanks