help creating PHP web crawler

Hi all , I have this idea in mind long time ago , since first time I know about php and curl ,
any way I found this sitemap generator thats crawl entire site and literally grape all it pages links ,
I run it on one site an it grape ~8k pages ling (it was big site ) , this script name is Simple site crawler (Google it ) , it create sitemap.xml for the site you targeting , I modify the code to get ride of extra (.xml tags) leaving only urls .
here is the code (use it on your own ) https://pastebin.com/55Mf5AuB
save it as `sitemap.php` and run it like this

php sitemap.php

Click to expand...

the script on it state are useless since it only saves site url , while our goal is to scrape the whole site and save it data , I know there is some other php scraping script but all of them are complicated slow and not easy to get data out of it ,
I can add simple function to save each page the script crawled to separate file but that will create a lot of mess and huge number of files ,
So what solution you suggest to temporally save these page to process it data later ,
Should I use DB to store pages contents ( e.g SQLite or MySQL ) or put the whole contents on septate .xml file ( and extend it contents in each time the script crewel new page )

Don't use PHP for this. Learn how to script in Python and look up for Scrapy framework.
PHP is just not suitable for mass-scraping a lot of websites.
Also, you should consider using some proxies if you don't wanna get banned while scrapping certain site.

C#, golang, C they'd be the best languages for this. PHP will do it. Just not very efficiently.

Lamarr said:

Don't use PHP for this

Click to expand...

jamie3000 said:

PHP will do it. Just not very efficiently

Click to expand...

Oh man I was about to put note at the end but I forgot (I don't care about languages performance at all ) even if 1 hour task takes a week ,
Honesty I don't have time t learn either C# or Go Lang , I know how to coding in php and that enough to me ,
The only minor problem I faced right now is temporary storage of crawled pages they mostly be ~10k page each project ,
what is simplest and easies way to do it , I'm thinking using single .xml file and open it add each new page complete contents at the end of the file ,
but I'm afraid that (file open and file close command) 10k time will be slow process ,
and I don't know how fast php could open the final file I think it will be about 60MB size at least . so have you experience such situation !
BTW I'm trying to avoid MySQL as much I can since it consumes a lot of rescuers (my VPS have 512MB RAM ) and it barley enough for sitemap script .
SQLite I read it's didn't accept synchronized connection (at the same time ) .
Last I'm still working on the script I may test all these methods and tell which one works .

It's not about the language as you are saying you need to scrape 10k links. Just use the one you are most comfortable with.

If you try to scrape 10k links at once, your process may become very large, and might make your server unstable.

Try using batch processing instead. Make a mysql table and put your links there. Then have a last scraped column.

Once you do that, you can put a limit on how many links are scraped in each call, by limiting the sql and ordering by last scraped (the latest scraped link gets least significance).

If you are only scraping once, you can put a scraped flag instead of timestamp.

Edit: for making the bot even faster, try multi threading the batch runner script (open 10 links in parallel, for example). This is where languages like python and perl come handy. The last time I checked, php did not have proper multi threading support.

Hope that helps.

Gogol said:

It's not about the language as you are saying you need to scrape 10k links. Just use the one you are most comfortable with.
If you try to scrape 10k links at once, your process may become very large, and might make your server unstable.
Try using batch processing instead. Make a mysql table and put your links there. Then have a last scraped column.
Once you do that, you can put a limit on how many links are scraped in each call, by limiting the sql and ordering by last scraped (the latest scraped link gets least significance).
If you are only scraping once, you can put a scraped flag instead of timestamp.
Edit: for making the bot even faster, try multi threading the batch runner script (open 10 links in parallel, for example). This is where languages like python and perl come handy. The last time I checked, php did not have proper multi threading support.
Hope that helps.

Click to expand...

Thank you for the valuable reply , it's really help me to have bigger picture for the script , any I wish you test the script first (at least read it complete codes ) , I know this may be boring to anyone who have other projects to finish
In short that script visiting the page and scape all it contents (BTW it finish th 10K links on ~hour ) with this line :-
` $html = GetUrl ($url);`
that will bring entire page contains via curl , the script writer made it only filter urls , if they are from the same site he will them to array (and save them into .xml file ) .
my issue is very simple beside page links I want to store all page contests (that being scraped on the same loop ) , and because this script consume almost all RAM on my VPS 512MB I can't use MySQL as temporary storage of 10k pages , I;ll test it using another .xml file and will give you results .
For multi-threads thing its still to early for me to do it . I need to make it work and then improve it .

mainceaft said:

Thank you for the valuable reply , it's really help me to have bigger picture for the script , any I wish you test the script first (at least read it complete codes ) , I know this may be boring to anyone who have other projects to finish
In short that script visiting the page and scape all it contents (BTW it finish th 10K links on ~hour ) with this line :-
` $html = GetUrl ($url);`
that will bring entire page contains via curl , the script writer made it only filter urls , if they are from the same site he will them to array (and save them into .xml file ) .
my issue is very simple beside page links I want to store all page contests (that being scraped on the same loop ) , and because this script consume almost all RAM on my VPS 512MB I can't use MySQL as temporary storage of 10k pages , I;ll test it using another .xml file and will give you results .
For multi-threads thing its still to early for me to do it . I need to make it work and then improve it .

Click to expand...

A few things:

1) I don't think your RAM has anything to do with the MySQL tables, because these kind of data is stored on the disk; not your RAM. That being said, how much record you process per run does matter. Don't scrap thousands at once. Scrap a handful, like 10-20 to begin with (keep experimenting until you hit the sweet spot).
2) You can store large HTML/XML pages in your MySQL table. Look into MEDIUMBLOB or LONGBLOB depending on your requirement. Alternatively, you can store the files in a folder, and then keep the name of the file in the db. That should work just fine.
3) What is your script scraping? HTML, or XML? If HTML, why would you store the data in XML?
4) As for finishing 10K links in an hour, it really depends on a few things like your script's performance and the network speed of your server.
5) What stops you from upgrading your server from 512MB RAM to let's say, 2GB RAM?
6) Multi threading is easier than you might think, given that you use python instead of PHP. There's a lot of tutorials available on the internet. Just copy pasting would usually work (except for some cases like if you want to kill the script with CTRL + C, it might cause problem if the script is not done properly).

Edit: Oh I see that you have a pastebin link. I will check it and update this thread later.