crawler

  1. BullseyeZ

    Hiring Web Crawler/Scraper Specialist

    Hello! I have a project and I need help with gathering data from the internet in the electric energy field (especially the ones who sell/design photovoltaics, storage, batteries, inverters and hydrogen). You'll have to extract these infos from available electric energy companies (all over the...
  2. D

    Crawl data from expireddomains.net with Python

    I'm trying to crawl data from https://www.expireddomains.net/ using Python They require an account to see all data, so signed up and used Chrome to inspect what headers were sent to their server. I copied all that headers to my Python script including Cookie. I even put a random 3 to 5 minutes...
  3. BullseyeZ

    Hiring Web Crawler/Scraper Specialist

    Hello! I have a project and I need help with gathering data from the internet in the electric energy field. You'll have to extract these infos from available electric energy companies: - The company name - The company site - Contact email - Contact phone - A short description of the company...
  4. yellowcat

    Free Web scraping any site

    Send me a dm of the site you wish to have scraped and the data ya want. - cheers
  5. A

    Help for a web crawler/scraper/automated poster on wordpress plugin

    Hello there, i hope that you can help me to solve my problem... I am trying from a lot of time to scrape pages from 2 specific websites (with text and images) and post them into my wordpress: https://www.ticketone.it/city/roma-216/ and https://culture.roma.it/musica/feed i tried many...
  6. 4ry4n

    [Question] How to find out that which crawlers visited my website?

    Greetings BHW community, I hope that you all are doing great! A question or actually two if I may: 1- How to find out which crawler bots visit my website? Like crawlers from google, bing, yahoo and... 2- Have you ever experienced slow website caused by too many active crawlers? Can my host...
  7. dreadpixel

    LINK&GROW: Lead Generation & Sales Software Suite - Scrape ANY Network for Emails/Phones - Verify- Enrich and Mass SMTP, SMS, etc. + EXT...

    PLEASE REMAIN ALL THE SUPPORT AND QUESTIONS IN THIS THREAD. We will create videos or update guides to answer everybody in this thread. All our pricing and product model is now built-in into a SUITE. So all the extra tools we launch will be inside your membership. IF you get a LIFETIME license...
  8. mainceaft

    Developing domains hunter need partner

    Hi as title said I want to develop Crawler that search the web and hunt good old domains that have BL's from big Authority sites , the crawler will give you warning whenever any of these domains is about to expire son to hunt it , This will be my first to do something on this scale , as you can...
  9. K

    Looking for a crawler for multi social networks scraping

    Hi there, I am looking to develop or find a crawler / script /bot that scans 120 social networks Like you see here on https://www.spokeo.com/social/profile?loaded=1&q=sw-PJJZbnKYmOO1i9z3kK7ZUnpM5QlMvWlup5CGRE2h_Yxqn1ZH5CgsLA example. ( I don`t work for them or promote them) I want to be...
  10. R

    Can constant manual indexing requests improve the automated indexing speed?

    Hello, I'm working on a news website that publishes about 30 articles per day, in a 9-10 hour interval. Despite the site is quite old and it should have no issues with indexing (I've shared more details about the website here), Google crawls it quite rare for a news website, which is not that...
  11. Andrew696

    PHP based scraper/parser: expert needed

    Hello, I have a custom PHP based solution for scraping Amazon (EU locales). Im looking for a developer with proper knowledge and experience with Amazon: The software works with backconnect proxies and can handle/sync up to 500.000 ASINs per day. Actually, we are facing an issue with...
  12. Andrew696

    Looking for Amazon scraper expert

    Hello, I have a custom PHP based solution for scraping Amazon (EU locales). Im looking for a developer with proper knowledge and experience with Amazon: The software works with backconnect proxies and can handle /sync up to 500.000 ASINs daily but sometime it need some optimizations and...
  13. Guybrushthepirate

    It is possible to know from which site the bot/crawler is coming when following a link?

    Pretty much what I am asking in the title. I'd like to know if there is any form to know from which source a crawler is coming when reaching a URL, linked to another. Example: in site1.com there is a link pointing to site2.com. Is there a chance for the owner of site2.com, to know that the...
  14. Mymusic

    How long does Google takes to show the crawled link in the google webmaster tools?

    Few days back I made few backlinks at promodj.com then today I searched all my made promodj.com backlinks in google and lucky few of them have got crawled (appearing in the search result) but those crawled links are not showing in my webmaster tools in the links section while all my old...
  15. adsensell

    Copied my home page

    Hello everyone, When I looked at my website's statistics on Google Analytics, there was a search record "site:crawlersite". When I visited this site, I saw that they had duplicated my website's mainpage. Whatever I add my website, they are immediately updating their content or any changes I have...
  16. mark14141414

    HELP!! Facebook crawler spamming my website

    ok guys I need your help, looked everywhere and couldn't find any answers and I'm not a noob here I manage/create multiple websites and have a lot of experience managing servers. But this is something new.. I had an old website made in Magento, I erased everything and installed a Wordpress...
  17. GodLy

    Find/Filter top conetnt of X profile and download?

    Hey, Trying to find tool that can find top content from X profile? With some options like: Filter last 24h, last month and so on.. Option to crawl/download filtered content.. Thanks
  18. R

    Block Semrush,ahref,moz

    Hello Can you tell me which code should i use please to block all web crawlers and only allow search engine to find my site. I have already found some answers here but most of it were old. Is it a good idea only to allow search engine to crawl my site and block all the rest?
  19. Jinyus

    How do you block crawlers from your Pbn?

    I know about using robots.txt, but for some reason I still believe that some of them is crawling my sites. Also... Is there a comprehensive list of all seo spiders that should be blocked? Is it necessary to block their ips? Where can I get a list of their ips?
  20. kleinerwauwau

    Majestic indexed lots of PBN-Links recently. Wtf?!

    Hi guys, did you notice, that Majestic recently indexed lots of PBN-Links although the bot should be blocked via htaccess? Does anyone have information about that? It seems they've changed the name of their crawler or sth.
Back
Top
AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features and essential functions on BlackHatWorld and other forums. These functions are unrelated to ads, such as internal links and images. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock