crawler

  1. A

    Help for a web crawler/scraper/automated poster on wordpress plugin

    Hello there, i hope that you can help me to solve my problem... I am trying from a lot of time to scrape pages from 2 specific websites (with text and images) and post them into my wordpress: https://www.ticketone.it/city/roma-216/ and https://culture.roma.it/musica/feed i tried many...
  2. 4ry4n

    [Question] How to find out that which crawlers visited my website?

    Greetings BHW community, I hope that you all are doing great! A question or actually two if I may: 1- How to find out which crawler bots visit my website? Like crawlers from google, bing, yahoo and... 2- Have you ever experienced slow website caused by too many active crawlers? Can my host...
  3. dreadpixel

    LINK&GROW: Lead Generation & Sales Software Suite - Scrape ANY Network for Emails/Phones - Verify- Enrich and Mass SMTP, SMS, etc. + EXT...

    PLEASE REMAIN ALL THE SUPPORT AND QUESTIONS IN THIS THREAD. We will create videos or update guides to answer everybody in this thread. All our pricing and product model is now built-in into a SUITE. So all the extra tools we launch will be inside your membership. IF you get a LIFETIME license...
  4. mainceaft

    Developing domains hunter need partner

    Hi as title said I want to develop Crawler that search the web and hunt good old domains that have BL's from big Authority sites , the crawler will give you warning whenever any of these domains is about to expire son to hunt it , This will be my first to do something on this scale , as you can...
  5. K

    Looking for a crawler for multi social networks scraping

    Hi there, I am looking to develop or find a crawler / script /bot that scans 120 social networks Like you see here on https://www.spokeo.com/social/profile?loaded=1&q=sw-PJJZbnKYmOO1i9z3kK7ZUnpM5QlMvWlup5CGRE2h_Yxqn1ZH5CgsLA example. ( I don`t work for them or promote them) I want to be...
  6. R

    Can constant manual indexing requests improve the automated indexing speed?

    Hello, I'm working on a news website that publishes about 30 articles per day, in a 9-10 hour interval. Despite the site is quite old and it should have no issues with indexing (I've shared more details about the website here), Google crawls it quite rare for a news website, which is not that...
  7. Andrew696

    PHP based scraper/parser: expert needed

    Hello, I have a custom PHP based solution for scraping Amazon (EU locales). Im looking for a developer with proper knowledge and experience with Amazon: The software works with backconnect proxies and can handle/sync up to 500.000 ASINs per day. Actually, we are facing an issue with...
  8. Andrew696

    Looking for Amazon scraper expert

    Hello, I have a custom PHP based solution for scraping Amazon (EU locales). Im looking for a developer with proper knowledge and experience with Amazon: The software works with backconnect proxies and can handle /sync up to 500.000 ASINs daily but sometime it need some optimizations and...
  9. Guybrushthepirate

    It is possible to know from which site the bot/crawler is coming when following a link?

    Pretty much what I am asking in the title. I'd like to know if there is any form to know from which source a crawler is coming when reaching a URL, linked to another. Example: in site1.com there is a link pointing to site2.com. Is there a chance for the owner of site2.com, to know that the...
  10. Mymusic

    How long does Google takes to show the crawled link in the google webmaster tools?

    Few days back I made few backlinks at promodj.com then today I searched all my made promodj.com backlinks in google and lucky few of them have got crawled (appearing in the search result) but those crawled links are not showing in my webmaster tools in the links section while all my old...
  11. adsensell

    Copied my home page

    Hello everyone, When I looked at my website's statistics on Google Analytics, there was a search record "site:crawlersite". When I visited this site, I saw that they had duplicated my website's mainpage. Whatever I add my website, they are immediately updating their content or any changes I have...
  12. mark14141414

    HELP!! Facebook crawler spamming my website

    ok guys I need your help, looked everywhere and couldn't find any answers and I'm not a noob here I manage/create multiple websites and have a lot of experience managing servers. But this is something new.. I had an old website made in Magento, I erased everything and installed a Wordpress...
  13. GodLy

    Find/Filter top conetnt of X profile and download?

    Hey, Trying to find tool that can find top content from X profile? With some options like: Filter last 24h, last month and so on.. Option to crawl/download filtered content.. Thanks
  14. R

    Block Semrush,ahref,moz

    Hello Can you tell me which code should i use please to block all web crawlers and only allow search engine to find my site. I have already found some answers here but most of it were old. Is it a good idea only to allow search engine to crawl my site and block all the rest?
  15. Jinyus

    How do you block crawlers from your Pbn?

    I know about using robots.txt, but for some reason I still believe that some of them is crawling my sites. Also... Is there a comprehensive list of all seo spiders that should be blocked? Is it necessary to block their ips? Where can I get a list of their ips?
  16. kleinerwauwau

    Majestic indexed lots of PBN-Links recently. Wtf?!

    Hi guys, did you notice, that Majestic recently indexed lots of PBN-Links although the bot should be blocked via htaccess? Does anyone have information about that? It seems they've changed the name of their crawler or sth.
  17. D

    Site Web Crawler

    I need a webcrawler that will go from site to site and create a list of sites where the registration Form contains only mail or mail+password and that dont have captcha.
  18. G

    Google's Submit URL tool not working last couple days ?

    The last roughly 3 days the Google Submit URL linked below has not been working for me. I have tried tons of different URLs from different domains and NONE have been indexed. I have been using this tool for years when in a rush for indexing a page on Google. 99% of the time after adding the URL...
  19. msrougi

    Updated list of crawler bots

    Where can I find an updated list of search engines and SEO crawlers to block my money sites from? Tks
  20. Ecodor

    Instagram crawler [embed code]

    Is there a tool or script which can pull up the embed code from a post in instagram and save it somewhere?
Back
Top