Any one have idea how web scraping works the easy way?

Advertise on BHW
Steptoe said:
That interest rate is output via Javascript, so for ease you'd need to find which Javascript function/API is supplying that data and scrape that instead. I did the grunt work for you, it's this one:
https://www.binance.com/bapi/earn/v...sion=3&simpleEarnType=ALL&assetLeftMatch=USDT
Something like WP Scraper or WPAutomatic should be able to scrape that for you.
Click to expand...
Awesome stuff thanks very much! How did u find the JS url above? Was it like inspect element and then check?

I see the out put via api is coming as a decimal value for the % APY example right now APY on normal page shows as 13.41% while in api it shows as 0.13419501 which means it same value it is refering to but in decimal format

Any idea how I can convert it using WPAutomatic?
 
Sorry, I'm not that much of a WP user, but I believe WPAutomatic has a regex option you could try, maybe as two separate values you then concatenate.
 
Steptoe said:
Sorry, I'm not that much of a WP user, but I believe WPAutomatic has a regex option you could try, maybe as two separate values you then concatenate.
Click to expand...
What should I type as regex value to get apy data daily since apy value changes all the time from this line;
"productId":"USDT001","asset":"USDT","highestApy":"0.12794436"

Actually I don't think using Regular experession of ap automatic can extract that data since it is live updating

so maybe using the other option given as css id/class or xpath?
Not certain if it will also work since the url you gave is plain text right?
 
Last edited:
Going off-the-top-of-my-head, these two regexes should get the values you need:
Code: 
highestApy":"0\.(\d\d)
highestApy":"0\.\d\d(\d\d)
The first will get the 12, the second will get the 79, then you need to join the two with a . to get your 12.79%
 
Steptoe said:
Going off-the-top-of-my-head, these two regexes should get the values you need:
Code: 
highestApy":"0\.(\d\d)
highestApy":"0\.\d\d(\d\d)
The first will get the 12, the second will get the 79, then you need to join the two with a . to get your 12.79%
Click to expand...
what would be the regex value If we want to get the 0.12794436 directly instead of %
 
Code: 
"highestApy":"([^"]+)"
Again, off-the-top-of-my-head, tested on regex101 but not WPAuto
 
Steptoe said:
Code: 
"highestApy":"([^"]+)"
Again, off-the-top-of-my-head, tested on regex101 but not WPAuto
Click to expand...
Works as expected thank you

Also can you tell me this?
How did u find the JS url above? Was it like inspect element and then check?
 
Nw_Work said:
Works as expected thank you

Also can you tell me this?
How did u find the JS url above? Was it like inspect element and then check?
Click to expand...
I've been scraping web content since the early 2000s in one way or another - I have a sixth sense for what request in dev tools is likely to be supplying the relevant data. Though if there's a tool that tells you where exactly the content from an inspected element comes from if it's dynamically generated, I'd be all on that!
 
I haven't used Binance API in a long time now (maybe 2 years or more) but the docs should guide you. I'm pretty sure this information can be obtained via official API call ( free to use) instead of scraping some webpage. When there's an official API, it's better to use that than an unofficial API that they use internally, as with these unofficial things the likelihood of change is higher.

One minute later on Google:
https://binance-docs.github.io/apidocs/spot/en/#simple-earn-endpoints
 
Steptoe said:
I've been scraping web content since the early 2000s in one way or another - I have a sixth sense for what request in dev tools is likely to be supplying the relevant data. Though if there's a tool that tells you where exactly the content from an inspected element comes from if it's dynamically generated, I'd be all on that!
Click to expand...
Random Interviewer asking Steptoe
"How did you become so rich, what did you do to make your money?"
Steptoe:scraping web content since the early 2000s:cool:

Btw your head is pretty good at web scraping and related queries no doubt

Potential for a dev to create a new free tool for finding where exactly the content from an inspected element comes from if it's dynamically generated in exchange for customer email submit or ads monetization
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Networking Downloads

Marketplace

Content / Copywriting Hosting Images, Logos & Videos Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design Misc

Making Money

Affiliate Programs Hire a Freelancer Making Money Pay Per Click Site Flipping

BlackHatWorld

BHW Merch Forum Suggestions & Feedback Introductions Lounge Dispute Resolution

Other

Domain Name Forum IM Journeys Web Hosting Newsletter
Back
Top