crawler

Hello there, i hope that you can help me to solve my problem... I am trying from a lot of time to scrape pages from 2 specific websites (with text and images) and post them into my wordpress: https://www.ticketone.it/city/roma-216/ and https://culture.roma.it/musica/feed i tried many...

Greetings BHW community, I hope that you all are doing great! A question or actually two if I may: 1- How to find out which crawler bots visit my website? Like crawlers from google, bing, yahoo and... 2- Have you ever experienced slow website caused by too many active crawlers? Can my host...

PLEASE REMAIN ALL THE SUPPORT AND QUESTIONS IN THIS THREAD. We will create videos or update guides to answer everybody in this thread. All our pricing and product model is now built-in into a SUITE. So all the extra tools we launch will be inside your membership. IF you get a LIFETIME license...

Hi as title said I want to develop Crawler that search the web and hunt good old domains that have BL's from big Authority sites , the crawler will give you warning whenever any of these domains is about to expire son to hunt it , This will be my first to do something on this scale , as you can...

Hi there, I am looking to develop or find a crawler / script /bot that scans 120 social networks Like you see here on https://www.spokeo.com/social/profile?loaded=1&q=sw-PJJZbnKYmOO1i9z3kK7ZUnpM5QlMvWlup5CGRE2h_Yxqn1ZH5CgsLA example. ( I don`t work for them or promote them) I want to be...

Hello, I'm working on a news website that publishes about 30 articles per day, in a 9-10 hour interval. Despite the site is quite old and it should have no issues with indexing (I've shared more details about the website here), Google crawls it quite rare for a news website, which is not that...

Hello, I have a custom PHP based solution for scraping Amazon (EU locales). Im looking for a developer with proper knowledge and experience with Amazon: The software works with backconnect proxies and can handle/sync up to 500.000 ASINs per day. Actually, we are facing an issue with...

Hello, I have a custom PHP based solution for scraping Amazon (EU locales). Im looking for a developer with proper knowledge and experience with Amazon: The software works with backconnect proxies and can handle /sync up to 500.000 ASINs daily but sometime it need some optimizations and...

Pretty much what I am asking in the title. I'd like to know if there is any form to know from which source a crawler is coming when reaching a URL, linked to another. Example: in site1.com there is a link pointing to site2.com. Is there a chance for the owner of site2.com, to know that the...

Few days back I made few backlinks at promodj.com then today I searched all my made promodj.com backlinks in google and lucky few of them have got crawled (appearing in the search result) but those crawled links are not showing in my webmaster tools in the links section while all my old...

Hello everyone, When I looked at my website's statistics on Google Analytics, there was a search record "site:crawlersite". When I visited this site, I saw that they had duplicated my website's mainpage. Whatever I add my website, they are immediately updating their content or any changes I have...

ok guys I need your help, looked everywhere and couldn't find any answers and I'm not a noob here I manage/create multiple websites and have a lot of experience managing servers. But this is something new.. I had an old website made in Magento, I erased everything and installed a Wordpress...

Hey, Trying to find tool that can find top content from X profile? With some options like: Filter last 24h, last month and so on.. Option to crawl/download filtered content.. Thanks

Hello Can you tell me which code should i use please to block all web crawlers and only allow search engine to find my site. I have already found some answers here but most of it were old. Is it a good idea only to allow search engine to crawl my site and block all the rest?

I know about using robots.txt, but for some reason I still believe that some of them is crawling my sites. Also... Is there a comprehensive list of all seo spiders that should be blocked? Is it necessary to block their ips? Where can I get a list of their ips?

Hi guys, did you notice, that Majestic recently indexed lots of PBN-Links although the bot should be blocked via htaccess? Does anyone have information about that? It seems they've changed the name of their crawler or sth.

I need a webcrawler that will go from site to site and create a list of sites where the registration Form contains only mail or mail+password and that dont have captcha.

The last roughly 3 days the Google Submit URL linked below has not been working for me. I have tried tons of different URLs from different domains and NONE have been indexed. I have been using this tool for years when in a rush for indexing a page on Google. 99% of the time after adding the URL...

Where can I find an updated list of search engines and SEO crawlers to block my money sites from? Tks

Is there a tool or script which can pull up the embed code from a post in instagram and save it somewhere?