Scrapper for an API

[SirLouen]

Has anyone done a scrapper on demand that executes on API calls?

For example: I execute an API REST call and the scrapper, does the scrapping job, picking the info accordingly and presenting it to me in a JSON format (for example).

I'm not experienced on web scrapping and I'm not 100% confident on the viability of this (for example, since the scrap may fail I may have a timeout fallback or something, not 100% sure)

I would like to see an example code in github in case there exist something like this

 

[polecat]

Yes I have done before a few times or reprogrammed other people stuff to make then run better before.

 

[SirLouen]

Yes I have done before a few times or reprogrammed other people stuff to make then run better before.

Which combination of technologies have you used to accomplish this?

 

[polecat]

Which combination of technologies have you used to accomplish this?

I did a few Javascript before and also bit of Python and Linux as well.

 

[693840]

@Gogol Any idea what a scrapper is? I only know scraper.

 

[polecat]

If you want further help or advice you need to get a ferret and prove it on a camera to show you one of us plus make updated videos showing you with at least one ferret same hob or jill then I will teach you more things.

 

[Gogol]

@Gogol Any idea what a scrapper is?

Sure, It's a project that is destined to fail; afaik (i.e. the project will be scrapped soon).

Coming to the topic itself (and considering OP means a Scraper), there are lots of ways to do this. For example:

Selenium + python
Puppeteer + Node
Cheerio (if you are a jquery fan, you will like it) + node and so on.

Not sure what your exact question is though. You could do a search for something like
"puppeteer example site:github.com" if you want some example.

 

[SirLouen]

Not sure what your exact question is though. You could do a search for something like

Imagine you have this API endpoint

https://example.com/fetch/balance/usd
This may go to a specific site, scraPPe it (or blackhat lift it), parse the info and select just a specific number in a column within that site, that you know it always has the same format.

Then you give the result to the API query something like

{balance: 32.52}

Probably the difference here, is that the scraPPing may take like, lets say 5 or 6 seconds (a full onLoad before you pick the result, parse and present into the JSON format).
+ There is a probability that if the site to be scraPPed might be dead, or the structure could change at some point, without me noticing this. So, after a while (lets say 10 seconds timeout), the API endpoint query may fail and send an exception.

I've never handled any of these scenarios. I understand that this is doable, but not 100% confident on how to start building this.

 

[IYASS]

Check: https://scraperapi.com/

 

[SirLouen]

Check: https://scraperapi.com/

Looks wonderful, but I have to do this by myself or looking at their pricing, it might cost me a shit ton of $.

Anyway im going to test it to get a bare approximation

EDIT: This is a rip off. They do something simple AF. They just handle all the captcha/proxy and all that but from the whole project this is by far the most minimal part... completely useless and extremely expensive from what they are offering. Not worthy at least for me that I can manage to handle this part.

I'm more lost in the other part (scraping the site itself on demand and managing the waiting)

 

[Gogol]

Imagine you have this API endpoint

https://example.com/fetch/balance/usd
This may go to a specific site, scraPPe it (or blackhat lift it), parse the info and select just a specific number in a column within that site, that you know it always has the same format.

Then you give the result to the API query something like

{balance: 32.52}

Probably the difference here, is that the scraPPing may take like, lets say 5 or 6 seconds (a full onLoad before you pick the result, parse and present into the JSON format).
+ There is a probability that if the site to be scraPPed might be dead, or the structure could change at some point, without me noticing this. So, after a while (lets say 10 seconds timeout), the API endpoint query may fail and send an exception.

I've never handled any of these scenarios. I understand that this is doable, but not 100% confident on how to start building this.

It is easy to get started. All you need is npm installed. Look for a tutorial on youtube for inspiration.

For example: https://www.youtube.com/results?search_query=node+cheerio+tutorial

Some sites won't work with cheerios (as per my observation. Perhaps someone else can make it work). If you have the same problem, look for https://www.youtube.com/results?search_query=node+puppeteer+tutorial

If you can't find an element with id or class somehow, all you need is the xpath (which you can get by inspecting the element -> right clicking on the node inside the inspect element panel -> copy -> copy full xpath). For fetching an element by xpath, check this SOF answer out.

For returning json, you need JSON.stringify(object)

If you want the script to be an api, use express js instead of using node directly.

If you do not know python, don't try selenium + python. You can do pretty much the same using puppeteer, but JS is a bit less messier (atleast for me).

 

[NulledCode]

You can look into serverless functions for small/quick jobs or amazon SQS to execute long running processes if you want to roll your own. Something you need to be mindful of is how long the scraping process will take to execute.

Another service you can have a look at is APIFY.

 

[SirLouen]

If you want the script to be an api, use express js instead of using node directly.

Another service you can have a look at is APIFY.

Will check this.

 

[Gogol]

P.s. I forgot to mention about data caching/saving. Like you said, a result might need 5-6 seconds to render. However, if you get a mongodb/sqlite/mysql database, and get another node bot to update data in that db, your api interface can then fetch data from the db instead of having to first fetch from the website your are scraping.

The first api call will "enqueue" such a job and assign a node bot to it. Depending on the logic, the bot will keep updating the data in intervals.

 

[TheNumbersGuy]

can easily use flask as your own endpoint api, then have requests / selenium whatever you want to scrape with. Post / get request to your flask endpoint to initiate the script to run on that vm / local machine / vps --> return the data your parsing in JSON.

 

[Gogol]

can easily use flask as your own endpoint api, then have requests / selenium whatever you want to scrape with. Post / get request to your flask endpoint to initiate the script to run on that vm / local machine / vps --> return the data your parsing in JSON.

Flask will work, however; it is not the easiest thing for someone without proper experience in python. I would rather suggest something like strapi, which provides UI interface for tables and fields, and also provides api out of the box (even user login.. file upload etc by default). It's much easier to bootstrap your backend using strapi. Then have a node bot that updates the data to the backend. Should be much easier to implement, as many already know JS.

 

[MasonF]

I've done that on amazon, I used requests.

 

[TheNumbersGuy]

Flask will work, however; it is not the easiest thing for someone without proper experience in python. I would rather suggest something like strapi, which provides UI interface for tables and fields, and also provides api out of the box (even user login.. file upload etc by default). It's much easier to bootstrap your backend using strapi. Then have a node bot that updates the data to the backend. Should be much easier to implement, as many already know JS.

True, not sure if OP is interesting in learning a new library (or even python as a whole) . But it's not that hard to watch a youtube video and learn how to host a flask server themselves.

 

[NulledCode]

You should have a look at Gearman. I used it to accept orders on a website for restoring websites from web.archive.org. When an order was placed, I sent a request to the job server (using Amazon SQS) which was then picked up by a worker that executed my script. Once the script was done, I updated my website's database with the completed order details. The job server and worker were on my local PC doing all of the work! Your approach will be similar if not the same as what I did in that you want to send requests to a worker who does the work. This approach works well because if you send multiple requests to the job server and there are no workers ready to work (still scraping from the last request) then the messages will just sit in the queue waiting to be picked up.

You need to know what you are doing but the above approach is a good learning exercise. If you think it's over your head then I would just go with a serverless approach. Amazon lambda was not available for me at the time otherwise I would have used that over gearman.

Read these about using serverless functions to accomplish the same thing. Also, Flask and Zappa make deploying serverless functions a breeze!

https://aws.amazon.com/blogs/architecture/scaling-up-a-serverless-web-crawler-and-search-engine/https://aws.amazon.com/blogs/architecture/serverless-architecture-for-a-web-scraping-solution/

 

[daibu]

Quick list of technologies you should know/research (in order):

1) JavaScript
2) Node.js
3) Express (search for "node express")
4) Puppeteer (search for "node puppeteer")

There are other stacks that will get the job done, but this is the most standard modern approach.