Part 3. Porn star page.
Now we know how to scrape data from video page URL. It's time to parse those URLs. Let's start with porn star page.
PHP:
$client = new Client;
$crawler = $client->request('GET', 'dirtysite/pornstar/madison-ivy');
$urls = $crawler->filter('.videos .videoblock .title a')->each(function($node) {
return $node->link()->getUri();
});
var_dump($urls);
Done. We will get URLs from first page. Now we can loop through them and create video object for each URL to get needed data.
PHP:
foreach ($urls as $url) {
$video = new Video($client, $url);
echo $video->getTitle() . PHP_EOL;
}
What was easy. But now we will get few errors.
First one:
Code:
PHP Notice: Undefined index: mp4 in ../script.php
This means that our regular expression not working every time. Let's fix it.
As you remember our pattent was: var player_quality_720p = 'OUR LINK', but sometimes there will be no 720p quality variable. By looking in source code you could find "var player_quality_240p" or "var player_quality_480p" or something else.
There a few options here.
1. Find first URL and return it.
2. Find all URLs and return array.
3. Find all URLs and return best quality URL.
For the sake of example I will use first option.
Let's change our regular expression.
PHP:
preg_match('/var player_quality_\d+p = \'(?<mp4>.*?)\'/', $this->crawler->html(), $matches);
\d+ means: one or more digit. This pattern will match "player_quality_720p", "player_quality_1p", "player_quality_1337p", "player_quality_1234567890p" and so on.
Now you can run script again and see that's everything works fine. Next step to create a class, just to make life easyer.
PHP:
class Pornstar {
protected $client;
protected $url;
protected $crawler;
public function __construct($client, $url)
{
$this->client = $client;
$this->url = $url;
$this->crawler = $this->client->request('GET', $url);
}
public function getUrl()
{
return $this->url;
}
public function getVideoUrls()
{
return $this->crawler->filter('.videos .videoblock .title a')->each(function(Crawler $node) {
return $node->link()->getUri();
});
}
}
So now we have URLs from first page, what about other pages?
Our scraper should check if there are "Next" page link and if there is, go there.
Let's create method that will check if there is "Next" page link.
PHP:
public function hasNextPage()
{
return $this->crawler->filter('.pagination3 .page_next')->count() > 0 ? true : false;
}
Cool. Next method will go to next page.
PHP:
public function goToNextPage()
{
$link = $this->crawler->filter('.pagination3 .page_next a')->link();
$this->crawler = $this->client->click($link);
}
Done. Let's check if everything works fine.
PHP:
$client = new Client;
$pornstar = new Pornstar($client, 'dirtysite/pornstar/madison-ivy');
var_dump($pornstar->getVideoUrls());
$pornstar->goToNextPage();
var_dump($pornstar->getVideoUrls());
Run script and you should see 2 different arrays. Cool. So how do we scrape all URLs? Easy.
PHP:
$client = new Client;
$pornstar = new Pornstar($client, 'dirtysite/pornstar/madison-ivy');
var_dump($pornstar->getVideoUrls());
while ($pornstar->hasNextPage()) {
$pornstar->goToNextPage();
var_dump($pornstar->getVideoUrls());
}
Simple. But let's make it even simplier. Let's create method that will return all URLs so we will not need to create loops ourselves.
PHP:
public function getAllVideoUrls()
{
$urls = $this->getVideoUrls();
while ($this->hasNextPage()) {
$this->goToNextPage();
$urls = array_merge($urls, $this->getVideoUrls());
}
return $urls;
}
Now test our method.
PHP:
$client = new Client;
$pornstar = new Pornstar($client, 'dirtysite/pornstar/madison-ivy');
var_dump($pornstar->getAllVideoUrls());
This looks better.
Now we can scrape data from video pages like this.
PHP:
$client = new Client;
$pornstar = new Pornstar($client, 'dirtysite/pornstar/madison-ivy');
foreach ($pornstar->getAllVideoUrls() as $url) {
$video = new Video($client, $url);
echo $video->getTitle() . PHP_EOL;
}
Everything works great, but sometimes we get strange error.
Code:
PHP Fatal error: Uncaught exception 'InvalidArgumentException' with message 'The current node list is empty.'
It means that sometimes we are trying to work with unexpected HTML. Before diving into code let's think a little bit. When you make request with real browser you are sending User Agent string that contains information about your browser. We are not using real browser so what do we send in User Agent string? "Symfony2 BrowserKit". Let's change that to a normal User Agent string.
PHP:
$client = new Client;
$client->setHeader('User-Agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/44.0.2403.89 Chrome/44.0.2403.89 Safari/537.36');
Sometimes it's really important to look like real browser... or google bot
Also it's important to pause between requests. Right now we will use "sleep" function. Not the best way and we will improve that later.
Next step it to add some sugar.
Look at this code:
PHP:
foreach ($pornstar->getAllVideoUrls() as $url) {
$video = new Video($client, $url);
echo $video->getTitle() . PHP_EOL;
}
Can we make it better? Yes we can! I really want to use my scraper like this:
PHP:
foreach ($pornstar->getVideos() as $video) {
echo $video->getTitle() . PHP_EOL;
}
Well it's kinda easy to implement but we will have a little problem there.
For example porn star will have 10k of videos (really hardworking person). We are scraping each video page. So we get mp4 URL from first page. It's important to tell that this URL will not work forever. Adult tubes have temporary URLs for video streaming. You can't scrape it and put it on your website. It may work for some time, but eventually URL will die and you will have tube with dead videos. Just imagine that you are scraping 1000th video page and URL from first already died.
Few options here.
1. Do not make request in constructor. We could have Video object initialized but data about video would not be available. When we will ask for it, scraping will happen. So now we could create array of Video objects and then start to loop through them having fresh data.
2. Make some callback functionality. Callback will be fired when new video page scraped. Example:
PHP:
$pornstar->eachVideo(function($video) {
echo $video->getTitle();
});
I will go with first option.
Let's change our Video class.
Modify constructor:
PHP:
public function __construct(Client $client, $url)
{
$this->client = $client;
$this->url = $url;
}
Add new method:
PHP:
protected function parseIfNeeded()
{
if ($this->parsed) return;
$this->crawler = $this->client->request('GET', $this->getUrl());
$this->parseTitle();
$this->parsePornstars();
$this->parseCategories();
$this->parseTags();
$this->parseMp4();
$this->parsed = true;
}
Modify getTitle method:
PHP:
public function getTitle()
{
$this->parseIfNeeded();
return $this->title;
}
Same with all get* methods.
Now we can create Video object and no request will be triggered. Later we will ask for data (title for example) and everything will be scraped.
Our Video class will look like this:
PHP:
class Video {
protected $client;
protected $url;
protected $crawler;
protected $title;
protected $pornstars;
protected $categories;
protected $tags;
protected $mp4;
protected $parsed = false;
public function __construct(Client $client, $url)
{
$this->client = $client;
$this->url = $url;
}
public function getUrl()
{
return $this->url;
}
public function getTitle()
{
$this->parseIfNeeded();
return $this->title;
}
public function getPornstars()
{
$this->parseIfNeeded();
return $this->pornstars;
}
public function getCategories()
{
$this->parseIfNeeded();
return $this->categories;
}
public function getTags()
{
$this->parseIfNeeded();
return $this->tags;
}
public function getMp4()
{
$this->parseIfNeeded();
return $this->mp4;
}
public function toArray()
{
return [
'title' => $this->getTitle(),
'pornstars' => $this->getPornstars(),
'categories' => $this->getCategories(),
'tags' => $this->getTags(),
'mp4' => $this->getMp4()
];
}
public function toJson()
{
return json_encode($this->toArray());
}
protected function parseIfNeeded()
{
if ($this->parsed) return;
$this->crawler = $this->client->request('GET', $this->getUrl());
$this->parseTitle();
$this->parsePornstars();
$this->parseCategories();
$this->parseTags();
$this->parseMp4();
$this->parsed = true;
}
protected function parseTitle()
{
$this->title = trim($this->crawler->filter('.video-wrapper .title-container .title')->first()->text());
}
protected function parsePornstars()
{
$this->pornstars = $this->crawler->filter('.video-info-row:contains("Pornstars:") a:not(:contains("Suggest"))')->each(function(Crawler $node) {
return trim($node->text());
});
}
protected function parseCategories()
{
$this->categories = $this->crawler->filter('.video-info-row:contains("Categories:") a:not(:contains("Suggest"))')->each(function(Crawler $node) {
return trim($node->text());
});
}
protected function parseTags()
{
$this->tags = $this->crawler->filter('.video-info-row:contains("Tags:") a:not(:contains("Suggest"))')->each(function(Crawler $node) {
return trim($node->text());
});
}
protected function parseMp4()
{
preg_match('/var player_quality_\d+p = \'(?<mp4>.*?)\'/', $this->crawler->html(), $matches);
$this->mp4 = $matches['mp4'];
}
}
Let's add "getVideos" and "getAllVideos" to Pornstar class.
PHP:
public function getVideos()
{
return array_map(function($url) {
return new Video($this->client, $url);
}, $this->getVideoUrls());
}
public function getAllVideos()
{
return array_map(function($url) {
return new Video($this->client, $url);
}, $this->getAllVideoUrls());
}
Let's test how it works:
PHP:
$client = new Client;
$client->setHeader('User-Agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/44.0.2403.89 Chrome/44.0.2403.89 Safari/537.36');
$pornstar = new Pornstar($client, 'dirtysite/pornstar/madison-ivy');
foreach ($pornstar->getAllVideos() as $video) {
echo $video->getTitle() . PHP_EOL;
}
Cool. Sometimes you will get error. Just use "sleep" inside a function.
PHP:
foreach ($pornstar->getAllVideos() as $video) {
echo $video->getTitle() . PHP_EOL;
sleep(1);
}
Later we will improve this.
P.S. Github repo updated.
