crawlers

Hi Everyone, We've been struggling with a homepage indexing issue for the past two months. Interestingly, other pages on our site are appearing in Google search results without any problems. Last week, we fixed a Google Rich Results "Rendering" issue, hoping it would resolve the homepage...

Looking for someone to scrape thousands of channels (particular channels) You'll need to deliver: YT Channel's Name: Sub Count: Business Email: Send me your rates and how quick you can get it done

Hello, I've been interested in SEO for some time and have one question. If I block Ahrefs, Majestic etc robots in htaccess file, how can I analyze the incoming links to my site and how can I check the indexing of new links?

I need to write a crawler for a project of mine that needs to be extremely scalable and effecient. I was advised to use a lower level language, specifically C++ Anyone with experience doing something similar, I would love to hear from you

Anybody has a robots.txt file to block out these crawlers? Majestic Moz Ahrefs Other major crawlers.. I can make one, but it takes time. If anyone has one already, it will be much appreciated. Thanks! /K

Hello Friends, I need help regarding google crawl rate and indexing related issue. Is it possible that google crawler start indexing old content of a page even after updating on regular basis. We are not even using any kind of server side caching system anymore. But still search engine is...

hey, i'm looking for a tool that knows to distinguish between male and female instagram users. Tooligram claims to be able to do that, but i cant seem to be able to install it without the antivirus deleting it. My best guess is it's based on some kind of facebook connection cos there's no gender...

I know that obviously crawlers and spiders index sites based on their meta, H1, H2, alt, title, and all the basic ones. But do search engines prefer sites with proper tags for site's. So for instance, lets say I don't use the tag but only for my navigation bar? Or maybe if my classes are named...

Hello I'm interested in developing advanced web-bots for social media websites like twitter, Instagram, YouTube, and Pintrest. To give you guys a bit of a background of myself I started off my programming adventure very simple building HTML and CSS static websites. About a month into that I...

Looking for fresh company data across the US.