crawler

Its an interesting and profitable way to earn for website owners. but I think bots content caching would be a great problem here. Still its a great initiative by cloudflare to facilitate the webmaster. Lets see how it ends. More here: https://blog.cloudflare.com/introducing-pay-per-crawl/

Good morning all! I want to ask a friend here Does anyone know any tricks or ways to get a one-page website, for example, like HEYLINK.ME or LINKTR.EE, to be quickly indexed by the Google search engine, because these sites cannot be verified by the Google search console, I hope someone knows the...

I’m looking for someone skilled to build a custom scraper to pull data from multiple platforms. The goal is to extract leads and job-related info from sites like hotel booking sites and job sites. What I Need: Scraping tools that can grab data like job listings, company details, contact info...

Hello! I have a project and I need help with gathering data from the internet in the electric energy field (especially the ones who sell/design photovoltaics, storage, batteries, inverters and hydrogen). You'll have to extract these infos from available electric energy companies (all over the...

I'm trying to crawl data from https://www.expireddomains.net/ using Python They require an account to see all data, so signed up and used Chrome to inspect what headers were sent to their server. I copied all that headers to my Python script including Cookie. I even put a random 3 to 5 minutes...

Hello! I have a project and I need help with gathering data from the internet in the electric energy field. You'll have to extract these infos from available electric energy companies: - The company name - The company site - Contact email - Contact phone - A short description of the company...

Send me a dm of the site you wish to have scraped and the data ya want. - cheers

Hello there, i hope that you can help me to solve my problem... I am trying from a lot of time to scrape pages from 2 specific websites (with text and images) and post them into my wordpress: https://www.ticketone.it/city/roma-216/ and https://culture.roma.it/musica/feed i tried many...

Greetings BHW community, I hope that you all are doing great! A question or actually two if I may: 1- How to find out which crawler bots visit my website? Like crawlers from google, bing, yahoo and... 2- Have you ever experienced slow website caused by too many active crawlers? Can my host...

Sales Thread Update ⚠️ This sales thread has been permanently closed!⚠️ Original sales content and contact information have been removed to protect our members and maintain the quality of the Marketplace. ✋ Click here to find similar active and tested services! ✋ Check out our Marketplace...

Hi as title said I want to develop Crawler that search the web and hunt good old domains that have BL's from big Authority sites , the crawler will give you warning whenever any of these domains is about to expire son to hunt it , This will be my first to do something on this scale , as you can...

Hi there, I am looking to develop or find a crawler / script /bot that scans 120 social networks Like you see here on https://www.spokeo.com/social/profile?loaded=1&q=sw-PJJZbnKYmOO1i9z3kK7ZUnpM5QlMvWlup5CGRE2h_Yxqn1ZH5CgsLA example. ( I don`t work for them or promote them) I want to be...

Hello, I'm working on a news website that publishes about 30 articles per day, in a 9-10 hour interval. Despite the site is quite old and it should have no issues with indexing (I've shared more details about the website here), Google crawls it quite rare for a news website, which is not that...

Hello, I have a custom PHP based solution for scraping Amazon (EU locales). Im looking for a developer with proper knowledge and experience with Amazon: The software works with backconnect proxies and can handle/sync up to 500.000 ASINs per day. Actually, we are facing an issue with...

Hello, I have a custom PHP based solution for scraping Amazon (EU locales). Im looking for a developer with proper knowledge and experience with Amazon: The software works with backconnect proxies and can handle /sync up to 500.000 ASINs daily but sometime it need some optimizations and...

Pretty much what I am asking in the title. I'd like to know if there is any form to know from which source a crawler is coming when reaching a URL, linked to another. Example: in site1.com there is a link pointing to site2.com. Is there a chance for the owner of site2.com, to know that the...

Few days back I made few backlinks at promodj.com then today I searched all my made promodj.com backlinks in google and lucky few of them have got crawled (appearing in the search result) but those crawled links are not showing in my webmaster tools in the links section while all my old...

Hello everyone, When I looked at my website's statistics on Google Analytics, there was a search record "site:crawlersite". When I visited this site, I saw that they had duplicated my website's mainpage. Whatever I add my website, they are immediately updating their content or any changes I have...

ok guys I need your help, looked everywhere and couldn't find any answers and I'm not a noob here I manage/create multiple websites and have a lot of experience managing servers. But this is something new.. I had an old website made in Magento, I erased everything and installed a Wordpress...

Hey, Trying to find tool that can find top content from X profile? With some options like: Filter last 24h, last month and so on.. Option to crawl/download filtered content.. Thanks