Scrape Load More page

shaymiller · Jun 4, 2020

I am using php to scrape pages. it seem to work well. There is one site that has Load more. But I can't scrape that

http://dramaonline.pk/hum-tv-latest-dramas-episodes/

krychaj5 · Jun 4, 2020

1. Open devtools then go to Network -> XHR
2. Click the "load more" button
3. Find request made by this click (load-more-channels.php)
4. Curl it with relevant headers to get response with "load more" data
5. Enjoy

shaymiller · Jun 4, 2020

krychaj5 said:
1. Open devtools then go to Network -> XHR
2. Click the "load more" button
3. Find request made by this click (load-more-channels.php)
4. Curl it with relevant headers to get response with "load more" data
5. Enjoy

before posting this question, I already knew all steps up to # 3

not sure about # 4?

Thanks

krychaj5 · Jun 4, 2020

CURL this url in PHP with headers You see in Your devtools at this request which appears after clicking button..

shaymiller · Jun 4, 2020

krychaj5 said:
CURL this url in PHP with headers You see in Your devtools at this request which appears after clicking button..

right, I do see that URL, whats next? what should I do? scrape that http://dramaonline.pk/contents/load-more-channels.php?channel=hum-tv

NorbGier · Jun 4, 2020

Check in browser url, headers and data that is being sent with this request when you press the button. Then in your program post request to the url, spoof headers, insert data to request and send request. In response you will receive html with content. Repeat this step till no load mroe button is found. I checked it and requests only differ in data. I suggest googling tutorials, you lack basic knowledge.

Thousand Bytes · Jun 4, 2020

They dont have spesific header. Dont forget setting Content-type: application/x-www-form-urlencoded and default headers.

POST to http://dramaonline.pk/contents/load-more-channels.php
with BODY var_post=Hum TV/Pakistan Tonight/120

shaymiller · Jun 4, 2020

jova said:
They dont have spesific header. Dont forget setting Content-type: application/x-www-form-urlencoded and default headers.

POST to http://dramaonline.pk/contents/load-more-channels.php
with BODY var_post=Hum TV/Pakistan Tonight/120

Not quiet get it. not that super coder like you.

I tried this code from the net

[code/

<?php
// From URL to get webpage contents.
$url = "https://www.geeksforgeeks.org/";
// Initialize a CURL session.
$ch = curl_init();
// Return Page contents.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
//grab URL and pass it to the variable.
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
echo $result;
?>

[/code]

NorbGier · Jun 4, 2020

Check this tutorial:http://subscription.packtpub.com/bo...c13/submitting-a-form-using-curl-intermediate Check POST request example.
I am not expers on PHP, but in every language you have to do the same thing so I will try to help you. You need to send POST request to this URL http://dramaonline.pk/contents/load-more-channels.php
With data Hum TV/Pakistan Tonight/30
You should receive as a response HTML document with more content. You can scrape it for Repeatmore content. Next POST request this time with /60 next with /120 next try with higher numbers to get more content.
You really lack basic knowledge. Ready about http requests. Find in google tutorial about web scraping. That one I linked seems to be decent.

shaymiller · Jun 5, 2020

jova said:
They dont have spesific header. Dont forget setting Content-type: application/x-www-form-urlencoded and default headers.

POST to http://dramaonline.pk/contents/load-more-channels.php
with BODY var_post=Hum TV/Pakistan Tonight/120

Code:

 <?php 
 
// From URL to get webpage contents. 
$url = "http://dramaonline.pk/contents/load-more-channels.php"; 
 
// Initialize a CURL session. 
$ch = curl_init(); 
 
// Return Page contents. 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
 
//grab URL and pass it to the variable. 
curl_setopt($ch, CURLOPT_URL, $url); 
 
$result = curl_exec($ch); 
 
echo $result; 
 
?>

@NorbGier

@krychaj5

krychaj5 · Jun 5, 2020

Whats the response? ($result)

You definitely should set more curl options, google "how to set curl headers" and put these headers from devtools you found in xhr (I'm writing it 3rd time in this topic..)

NorbGier · Jun 5, 2020

You seem to lack fundamentals. First of all learn how to use google. Answers for all your qiestions are on first google page if you search for : "php we bscraping", "php http requests", "php POST web scarping". I found complete video tutorials on how to scrape content that you need with 1 simple google search.

shaymiller · Jun 5, 2020

NorbGier said:
You seem to lack fundamentals. First of all learn how to use google. Answers for all your qiestions are on first google page if you search for : "php we bscraping", "php http requests", "php POST web scarping". I found complete video tutorials on how to scrape content that you need with 1 simple google search.

Let’s face the facts, under your username, it’s labeled “ Newbie” which means your are new/baby who lack fundamentals. Even BHW says so!

I had a question so I ask. If I knew the answer to my question, why would I ask? get it? or is it too much to for your newbie brain. Not everything can be answered by google. if so, maybe you should marry google and not another person.

I use this forum to learn & teach. If you did find something on google, just share it. if you are not here to help, just exit the door!, if you need help with that I can help you exit!!!

peace out!

NorbGier · Jun 5, 2020

You got the answer 3 times from me and other people. And you still ask the same question. And now you are trying to offend me? Here 4th time comes.
make POST request to http://dramaonline.pk/contents/load-more-channels.php
with BODY var_post=Hum TV/Pakistan Tonight/30
Are you waiting for us to write complete code for you?

Kuku-Nav · Jun 5, 2020

NorbGier said:
You seem to lack fundamentals. First of all learn how to use google. Answers for all your qiestions are on first google page if you search for : "php we bscraping", "php http requests", "php POST web scarping". I found complete video tutorials on how to scrape content that you need with 1 simple google search.

I think for learning purpose google is the best..yes BHW also helps to clarify your queries with the help of its family member

shaymiller · Jun 6, 2020

NorbGier said:
You got the answer 3 times from me and other people. And you still ask the same question. And now you are trying to offend me? Here 4th time comes.
make POST request to http://dramaonline.pk/contents/load-more-channels.php
with BODY var_post=Hum TV/Pakistan Tonight/30

Are you waiting for us to write complete code for you?

yes, if its not too much to ask for?

POST page

$playlisturl = mysite.php?curl.php?var_post=Hum%20TV/Pakistan%20Tonight/30

Code:

 <?php 

// From URL to get webpage contents. 

$playlisturl = $_GET['var_post'];



$url = "http://dramaonline.pk/contents/load-more-channels.php?var_post=$playlisturl"; 

echo $playlisturl;

// Initialize a CURL session. 
$ch = curl_init(); 

// Return Page contents. 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 

//grab URL and pass it to the variable. 
curl_setopt($ch, CURLOPT_URL, $url); 

$result = curl_exec($ch); 

echo $result; 

?>

$result is

Bad Request
Your browser sent a request that this server could not understand.

shaymiller · Jun 10, 2020

^ bump

Scrape Load More page

shaymiller

Regular Member

krychaj5

Junior Member

shaymiller

Regular Member

krychaj5

Junior Member

shaymiller

Regular Member

NorbGier

Junior Member

Thousand Bytes

Regular Member

shaymiller

Regular Member

NorbGier

Junior Member

shaymiller

Regular Member

krychaj5

Junior Member

NorbGier

Junior Member

shaymiller

Regular Member

NorbGier

Junior Member

Kuku-Nav

BANNED

shaymiller

Regular Member

shaymiller

Regular Member

Main Menu

Marketplace

Making Money

BlackHat World