[GET] guide to web scraping PHP

Contents
xiii
Foreword xvii
Chapter 1 � Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 1
Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
How to Read This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Web Scraping Defined . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Applications of Web Scraping . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Appropriate Use of Web Scraping . . . . . . . . . . . . . . . . . . . . . . . . . 4
Legality of Web Scraping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Topics Covered . . . . . . . . . . . 4
Chapter 2 � HTTP . . . . . . . . . . . . . . . . . . . . . . . . . 7
Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
GET Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Anatomy of a URL . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Query Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
POST Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
HEAD Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Headers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Cookies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Redirection . . . . . 16
.
.
.
.
Chapter 3 � HTTP Streams Wrapper . . . . . . . . . . . . . . . . . . . . . 27
Simple Request and Response Handling . . . . . . . . . . . . . . . . . . . . . 28
Stream Contexts and POST Requests . . . . . . . . . . . . . . . . . . . . . . . 29
Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
HTTP Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
A Few More Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Wrap-Up . . . . . . . . . . . . . . . . . . . 33
Chapter 4 � cURL Extension . . . . . . . . . . . . . . . . . . . . . 35
Simple Request and Response Handling . . . . . . . . . . . . . . . . . . . . . 36
Contrasting GET and POST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Setting Multiple Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Handling Headers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Cookies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
HTTP Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Redirection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Referers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Content Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
User Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Byte Ranges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
DNS Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Timeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Request Pooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Wrap-Up . . . . . . . . . . . . . . . . . . . 46
Referring URLs . . . . . . . .
Persistent Connections . . . .
Content Caching . . . . . . .
User Agents . . . . . . . . . .
Ranges . . . . . . . . . . . . .
Basic HTTP Authentication .
Digest HTTP Authentication
Wrap-Up . . . . . . . . . . . . . . .
Chapter 5 � pecl_http PECL Extension
GET Requests . . . . . . . . . . . . . .
POST Requests . . . . . . . . . . . . .
Handling Headers . . . . . . . . . . . .
Debugging . . . . . . . . . . . . . . . .
Timeouts . . . . . . . . . . . . . . . . .
Content Encoding . . . . . . . . . . .
Cookies . . . . . . . . . . . . . . . . . .
HTTP Authentication . . . . . . . . . .
Redirection and Referers . . . . . . . .
Content Caching . . . . . . . . . . . .
User Agents . . . . . . . . . . . . . . .
Byte Ranges . . . . . . . . . . . . . . .
Request Pooling . . . . . . . . . . . . .
Wrap-Up . . . . . . . . . . . . . . . . .
Chapter 6 � PEAR::HTTP_Client . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Requests and Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Juggling Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Wrangling Headers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Using the Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Observing Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Wrap-Up . . . . . . . . . . . . . 68
Chapter 7 � Zend_Http_Client . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Basic Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
URL Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Custom Headers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Cookies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Redirection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
User Agents . . . . . . . . . . 77
HTTP Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Wrap-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chapter 9 � Tidy Extension
Validation . . . . . . . . .
Tidy . . . . . . . . . . . . .
Input . . . . . . . . . . . .
Configuration . . . . . . .
Options . . . . . . . . . . .
Debugging . . . . . . . . .
Output . . . . . . . . . . .
Wrap-Up . . . . . . . . . .

Chapter 10 � DOM Extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Types of Parsers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
Loading Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
Tree Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Elements and Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Locating Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
XPath and DOMXPath . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Absolute Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Relative Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Addressing Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Unions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Resources . . . . . . . . . . 109
Chapter 8 � Rolling Your Own
Sending Requests . . . . . . .
Parsing Responses . . . . . .
Transfer Encoding . . . . . .
Content Encoding . . . . . .
Timing . . . . . . . . . . . . .

Chapter 12 � XMLReader Extension . . . . . . . . . . . . . . . . . . . . . . . . . 121
Loading a Document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Elements and Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
DOM Interoperation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Closing Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Wrap-Up . . . . . . . . . . . . . . . 127
Chapter 13 � CSS Selector Libraries
Reason to Use Them . . . . . . . . . .
Basics . . . . . . . . . . . . . . . . . . .
Hierarchical Selectors . . . . . . . . .
Basic Filters . . . . . . . . . . . . . . .
Content Filters . . . . . . . . . . . . . .
Attribute Filters . . . . . . . . . . . . .
Child Filters . . . . . . . . . . . . . . .
Form Filters . . . . . . . . . . . . . . .
Libraries . . . . . . . . . . . . . . . . .
PHP Simple HTML DOM Parser
Zend_Dom_Query . . . . . . . .
phpQuery . . . . . . . . . . . . .
DOMQuery . . . . . . . . . . . . .
Wrap-Up . . . . . . . . . . . . . . . . .

Chapter 11 � SimpleXML Extension
Loading a Document . . . . . . . .
Accessing Elements . . . . . . . . .
Accessing Attributes . . . . . . . .
Comparing Nodes . . . . . . . . . .
DOM Interoperability . . . . . . .
XPath . . . . . . . . . . . . . . . . .
Wrap-Up . . . . . . . . . . . . . . .
Chapter 15 � Tips and Tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Batch Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Parallel Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Crawlers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
That�s All Folks . . . . . . . . 162
Appendix A � Legality of Web Scraping 165
Chapter B � Multiprocessing 169


little messed up, but u get the point


Detection ratio: 0 / 43

Might find some tips in this one. I think my methods are pretty solid, but there is always room for improvements.

Where did you find this? Though, i think i would prefer downloading a compiled program already LOL

RidiculeMe said:

a compiled program

Click to expand...

you do know what PHP is right ?

RidiculeMe said:

Where did you find this? Though, i think i would prefer downloading a compiled program already LOL

Click to expand...

Excellent share! After a quick scan through I can say it offers some great detailed info on xPath and using cURL with proxies and setting referer! Hard to find this info on the php manual site.
Thanks Given

What would be the use for this script?

paincake said:

Click to expand...

hehe but what has that gotta do with this thread?

You can use this to program scripts to grab data from other websites. Perhaps you want to pinch the information from a popular online directory or scrape Google for link prospects.

The trick is to appear as legit as possible. Otherwise you'll get banned. This is why a lot of scripts like this will require proxies.

andee said:

you do know what PHP is right ?

Click to expand...

maybe he wants a hiphoped version

Thanks for this. I have been using php to scrape urls from Google for different footprints (don't have scrapebox), scraping twitter, downloading/uploading youtube videos on server. And I now want to port the youtube view bot into php. I think this book would be great help for some quick curl examples lookup.

Can you please reupload? File was already deleted

Welcome to 2007. Scraping others content won't work unless you are superior creative.

file has already deleted, please reupload

Link is dead, so thread is closed temporarily. If anyone has a mirror, PM me and I 'll update the post and re-open the thread.

OP has provided a new link, thread has been reopened.

but forgot the link

Any good web scraping book could use a section on using Wireshark - the only way to truly see how legit you look is to analyze the network traffic yourself and compare to legitimate traffic. Also a section on JavaScript emulation is near essential these days for some sites, especially if you're sending data back to servers.

RidiculeMe said:

Where did you find this? Though, i think i would prefer downloading a compiled program already LOL

Click to expand...

You will usually find that each web scraping job you do is different, but you can reuse code from previous scraping projects so a all in one web scraping program is not possible.

I'd say the best thing this guide could use is video tutorials and samples.