crawler

Hello Can you tell me which code should i use please to block all web crawlers and only allow search engine to find my site. I have already found some answers here but most of it were old. Is it a good idea only to allow search engine to crawl my site and block all the rest?

I know about using robots.txt, but for some reason I still believe that some of them is crawling my sites. Also... Is there a comprehensive list of all seo spiders that should be blocked? Is it necessary to block their ips? Where can I get a list of their ips?

Hi guys, did you notice, that Majestic recently indexed lots of PBN-Links although the bot should be blocked via htaccess? Does anyone have information about that? It seems they've changed the name of their crawler or sth.

I need a webcrawler that will go from site to site and create a list of sites where the registration Form contains only mail or mail+password and that dont have captcha.

The last roughly 3 days the Google Submit URL linked below has not been working for me. I have tried tons of different URLs from different domains and NONE have been indexed. I have been using this tool for years when in a rush for indexing a page on Google. 99% of the time after adding the URL...

Where can I find an updated list of search engines and SEO crawlers to block my money sites from? Tks

Is there a tool or script which can pull up the embed code from a post in instagram and save it somewhere?

I am not new to this form but i never sign up to this form ( usually i never sign up to any website/form). So why you do sign up here you ask? Well i am planning to build a script to crawl IMDB movie pages. But i need your help to figure out what to crawl. I have come up with a list to crawl...

Hello all I restart my new journey in this thread as I posted it initially in "Making Money" by mistake. Goals My objective is to launch an open-source bot framework, to allow peope to develop more easily customized bots and progressively to build a repository. Planned features: Proxies...

He everyone, i have a site in which you can get nutritional value on foods (compare them with one another). Right now the URL is like that : https://compareyourfood.com/?Bananas=09040&Apples-with-skin=09003 I'm wondering if google crawler will be able to index this page differently from my...

Hello, i've seen some websites with a board of live instagram Hastags. Like a website with a board pushing all photos posted with hashtag #TravelBrasil or something like it. Anyone how how i can do it? Is it a crawler? I rlly want to learn about this + learn about crawlers, any idea where i can...

Hello blackhatworld users i need a little help about getting python code instagram crawler if there is a existing one so i can modify a little for my needs, am not able to do it from beginning because i dont have that much experience and too lazy.. please if you have any source where i can find...

Hi, I have a list of urls from an online retail store, they are all API product pages and display available product information (like title, description, sizes available) in simple text format. I need a tool that can check these pages and look for specific keywords/brand names so that I can...

Who can help me with a bot that can do the following: 1. crawl Google and find Magento CMS sites in a specific country . 2. if found, than check which Magento version the site is using, 3. than you check if the Magento site is outdated 4. check which security patches these sites need 5. if the...

What is your favorite SEO Crawling tool? Do you use a desktop based tool or Cloud base tool?

Sales Thread Update ⚠️ This sales thread has been permanently closed!⚠️ Original sales content and contact information have been removed to protect our members and maintain the quality of the Marketplace. ✋ Click here to find similar active and tested services! ✋ Check out our Marketplace...

hi everyone, so my developer managed to make the perfect crawler.. that can convert a profile into an email. Yes. its possible via emailhunter, etc.. only 1 problem .. it doesnt work with new accounts that have 0 networking connections. and the only accounts we have to test with are our...

Hello everyone, here I am again desperately seeking for help. I have list of leads here with their company url. I wanted to find the social media accounts of the contact person/marketing persons of these companies. Is there any software that could do this? Your suggestion is a big help.

Hello experts! I have 1000 website urls. I wanted to pull out their marketing persons' information so that I could save it in an excel file. Is there a scraping tool that could do this? Or is this even sounds possible? Thanks I am already tired looking for some answers. Your reply is...

Hello! I am desperate, I wanted to build a list of leads by their social media accounts. I have here the basic information, the names and email address. I want a software (maybe they call it a crawler) that could generate the Social Media Accounts such as facebook, twitter, linkedin...