I need to scrape 500,000 pages

[valser29]

Hello there,
I need to scrape Alexa top 500,000 sites and select only ones that have checkout pages.
If anyone is interested, please reply here or send me a pm with prices.
Thanks!

 

[blackhat777]

I believe I can do it. Will you be providing the site lists?
Add me on Skype.

 

[tounsi7orr]

Yes! I can do this very well.
I'm python programmer and I use selenium librarie to make scraping bots that works perfectly. I want to tell you that the script I will code scrap a single page in 5 seconds.
This is my skype: akermiy.yassine

 

[MrBlue]

...I want to tell you that the script I will code scrap a single page in 5 seconds...

That's not very fast at all.

 

[tounsi7orr]

That's not very fast at all.

If you run the script 5 times in a free aws vps. It gonna be 1 second.
And believe me you'll never find a script that scrap less than two seconds.

 

[MikeTO2]

Yes! I can do this very well.
I'm python programmer and I use selenium librarie to make scraping bots that works perfectly. I want to tell you that the script I will code scrap a single page in 5 seconds.
This is my skype: akermiy.yassine

selenium is not the best choice when it comes to scraping content

 

[Ren]

If you run the script 5 times in a free aws vps. It gonna be 1 second.
And believe me you'll never find a script that scrap less than two seconds.

Hey give me ur Skype pls

 

[chrisyoungsd]

Check out scrape box. Great tool that you can use for all kinds of things.

 

[Omoruyiik]

i am sure you'd find paigham bot (in bst) useful, they even have videos here

 

[Deleted member 657006]

Info do it your self very easy.

http://www.the-art-of-web.com/php/parse-links/


<?php

// Original PHP code by Chirp Internet: www.chirp.com.au
// Please acknowledge use of this code by including this header.

$url ="http://www.example.net/somepage.html";
$input =@file_get_contents($url)ordie("Could not access file: $url");
$regexp ="<a\s[^>]*href=(\"??)([^\" >]*?)\\1[^>]*>(.*)<\/a>";
if(preg_match_all("/$regexp/siU", $input, $matches)){
// $matches[2] = array of link addresses
// $matches[3] = array of link text - including HTML code
}

?>


Easy peasy

Or

/*
Function to get all links on a certain url using the DomDocument
*/

function get_links($link)
{
//return array
$ret = array();
/*** a new dom object ***/
$dom = new domDocument;
/*** get the HTML (suppress errors) ***/
@$dom->loadHTML(file_get_contents($link));
/*** remove silly white space ***/
$dom->preserveWhiteSpace = false;
/*** get the links from the HTML ***/
$links = $dom->getElementsByTagName('a');
/*** loop over the links ***/
foreach ($links as $tag)
{
$ret[$tag->getAttribute('href')] = $tag->childNodes->item(0)->nodeValue;
}
return $ret;
}


//Link to open and search for links
$link = "http://www.php.net";

/*** get the links ***/
$urls = get_links($link);

/*** check for results ***/
if(sizeof($urls) > 0)
{
foreach($urls as $key=>$value)
{
echo $key . ' - '. $value . '<br >';
}
}
else
{
echo "No links found at $link";
}

 

[tounsi7orr]

Hey give me ur Skype pls

This : akermiy.yassine

 

[tounsi7orr]

selenium is not the best choice when it comes to scraping content

I know, it's problem is that it waits until the page fully load to start scaping and that takes some time. But, I always disable images and javascript and then page loading speed become very fast.

 

[MrBlue]

...I always disable images and javascript...

LOL. What's the point of using a headless browser if you are disabling JS.

 

[JustChillin]

Top 500,000 or 500? Cause I think Alexa doesn't show the top 500,000 pages on their website. It's limited to 500 for me.

 

[Pakal]

Alexa gives you free access to the top 1 million sites. The link below should help you get it directly from Alexa:

https://support.alexa.com/hc/en-us/articles/200461990-Can-I-get-a-list-of-top-sites-from-an-API-

Also I posted a few days ago a thread for a huge list of unique domains ready for download which could help you out. This list was harvested and generated with a reverse ip technique which generated roughly about 42 million unique domains

http://www.blackhatworld.com/seo/get-42-million-unique-domains-good-for-seo.862822/

Hope it helps

 

[turkeypockets]

I can do this. I have a list of the top 200,000 Alex's ranked sites already, I could target checkout pages and scrape the web assets, localize all the links, and get past robot restrictions. You realize how big this data is gonna be though right? We're talking an easy 50 gigs on 500,000 html pages.

 

[SEMWORLD]

Hello buddy. If you are still looking for a person to do the scraping, then I am available to help you out. Kindly just reach me through the inbox so that we can discuss the details of the project and see how best we can help each other. Looking forward to hear from you.

 

[tounsi7orr]

LOL. What's the point of using a headless browser if you are disabling JS.

Because, I only need HTML

 

[turkeypockets]

I can scrape html pages only and use filters on page size, localize assets or not, add necessary head info charset/meta for original source, append file extension, etc. I can't have it done instantly and it might take a few dry runs, but all in all my scraper rips html assets probably 10 pages/second it's stupid fast

 

[valser29]

Thanks everyone for the responses! Did not expect to receive so many
Sorry, I am kind of busy and obviously cannot respond to everyone. I am currently negotiating with one provider.