Looking for info on Web scraping

[TrumanB]

I'm looking to start a site that displays data scraped from another site. How would I go about scraping data and displaying it on another page. What should I expect to pay to have this page made for me?

Thanks

 

[ResetUA]

scrape into db, or what?

 

[TrumanB]

scrape into db, or what?

Yes, I imagine it needs to use a DB. I want to scrape data of something that is released every few minutes, then look up more info on each created item and publish it to a page. Then keep checking back every few minutes for changes to the data and updating my page in real time..

The data to be scraped is in a well organised structured but complicated format.

Once I get to that point, I want tk add filter and data point checks tk be used in a traffic light system on my page to show the shit from the gems

 

[ResetUA]

I still don't understand question.

If you need a tool, in 99% it will be a custom solution, that checks site and fire scraping on new items.

 

[TrumanB]

I still don't understand question.

If you need a tool, in 99% it will be a custom solution, that checks site and fire scraping on new items.

Yes, a custom solution is needed, I don't believe there are any off the shelf tools for what I want to do. I knows there's scrapers, but I don't know how contomisable the data they can scrape is without having custom tools made. How much would I expect to pay to have this coded?

I also would have no idea how to display the output of the scraper in a page on the web

 

[WebGenix]

I think it's best to start a thread in the HAF section, and you'll get contacted by developers who can do the job for you.

 

[krychaj5]

It should be coded in Python/NodeJS easily

 

[Bots-Scrapers]

I'm looking to start a site that displays data scraped from another site. How would I go about scraping data and displaying it on another page. What should I expect to pay to have this page made for me?

Thanks

You may require real time data feeding using data feeding using API.
The main site will have contact with API and API will transport the data to mainsite by scraping in the backend / from backend where ugly codes will remain.

 

[TheAfrikan]

Try octoparse...you can customize tasks which i think can give you the flexibility you want

 

[MountProxies]

I'm looking to start a site that displays data scraped from another site. How would I go about scraping data and displaying it on another page. What should I expect to pay to have this page made for me?

Thanks

What kind of data are you scraping? Is this a static source from which you're scraping the data? Websites that have layers of security are difficult to scrape, which raises the bar in terms of development protocols. So when you say "how much to pay" - you need to think about the dependencies.