Can any programmer help me on this selector?
- Thread starter chamith
- Start date
- Sep 10, 2010
- 11,806
- 26,657
document.querySelector('.cmc-link p.iworPT')
OR
document.querySelector('.cmc-link p')
However, I am almost sure this will change. There's a better way to scrape cmc. Look in the source page for script id __NEXT_DATA__ (it is made with next.js by the way). Then take that object and parse it for what you need.
OR
document.querySelector('.cmc-link p')
However, I am almost sure this will change. There's a better way to scrape cmc. Look in the source page for script id __NEXT_DATA__ (it is made with next.js by the way). Then take that object and parse it for what you need.
document.querySelector('.cmc-link p.iworPT')
OR
document.querySelector('.cmc-link p')
However, I am almost sure this will change. There's a better way to scrape cmc. Look in the source page for script id __NEXT_DATA__ (it is made with next.js by the way). Then take that object and parse it for what you need.
- Sep 10, 2010
- 11,806
- 26,657
What I gave you is JS code. For your tool (not sure what it is), just use .cmc-link p
But then again.. you will waste a lot of time with this with no proper result (the non js version only loads like 10 results, then nextJS loads the rest of the results with JS, and that data is taken from the __NEXT_DATA__ object). I have already done this lol.
Use the underlying __NEXT_DATA__ instead of this. That object has everything you will need.
- Sep 10, 2010
- 11,806
- 26,657
Not sure what to write on your tool, as I don't use it. But with JS/python, you will want to first understand the structure of that object, then have a foreach/for/while loop to take the data you need. This isn't the simplest site to scrape nevertheless. Be it with the dom, or with that object.
- Sep 10, 2010
- 11,806
- 26,657
Sorry for the double post as it is too late to edit..
Here's how I would parse that object (tested, works fine in my setup)..
json is the object json that you found inside __NEXT_DATA__, parsed into an object with JSON.parse();
output is the variable that has the exact data you need (in this case, I need).
Here's how I would parse that object (tested, works fine in my setup)..
json is the object json that you found inside __NEXT_DATA__, parsed into an object with JSON.parse();
output is the variable that has the exact data you need (in this case, I need).
Code:
var keys = json.props.initialState.cryptocurrency.listingLatest.data.shift().keysArr;
var data = json.props.initialState.cryptocurrency.listingLatest.data;
var output = [];
var keymap = [];
for(var i in keys){
if(!keys[i].includes('.')){
keymap.push([
i,
keys[i]
])
} else {
if(keys[i].startsWith('quote.USD')){
var n = keys[i].lastIndexOf('.');
var substr = keys[i].substring(n + 1);
if(substr != "name"){
keymap.push([
i,
substr
])
}
}
}
}
data.forEach(item => {
var temp = [];
for(var i in keymap){
temp[keymap[i][1]] = item[keymap[i][0]];
}
output.push(temp)
});
console.log(output)
Last edited:
Thank you for your help but my tool won't accept all that text
it's this chrome extension https://chrome.google.com/webstore/...b-scra/jnhgnonknehpejjnehehllkliplmbmhn?hl=en
it will say select and give me the option to select what i want to scram
i can maybe use coingecko.com if it's easier than coinmarketcap to scrap from
- Apr 3, 2019
- 600
- 768
Always hit the smallest API endpoint with your bot, if it exists, ideally one that returns data in JSON (implementation dependent). Parsing HTML is a pain in the ass, wastes a lot of bandwidth and should only be done in last resort. The underlying API is also much less likely to change long-term, than their frontend HTML.
- Sep 10, 2010
- 11,806
- 26,657
What you are doing is tough to achieve with a readymade plugin to be honest. You need a proper custom bot for the job. I already provided you the "tough part" of the code, e.g. parsing the data into desired format. All your bot needs to do is visit cmc site, get the __NEXT_DATA__ source, run it through the parser I provided you, and save in your desired format.
Also like @oldguyjoe pointed out, scraping html can be a pita, and it is almost never a "cheap" way to bot.
- Jun 10, 2010
- 2,401
- 1,676
Are you trying to scrape CoinMarketCap? They have an API you can use to fetch data, why dick around with screen scraping when you can call their endpoints?
I'm not that level yet