Checklist to avoid bot detection and getting banned

Discussion in 'BlackHat Lounge' started by jackbenimble, May 29, 2016.

  1. jackbenimble

    jackbenimble Newbie

    May 29, 2016
    Likes Received:
    I have a website that I scrape. I have an account there and got real lazy after never having issues for years. Logged in on my bot laptop and they called me and said, "hey, stop that shit". So I did, its been a while, but want to get back to scraping. Would like feedback / advice on what I need to do to minimize detection. I don;t want them to know that 1) it is ME scraping or 2) it is anyone with my geo-location town scraping.

    forgive my rookie-level knowledge please, thx. been researching a lot trying to learn.

    Current plan:
    > recycle router IP - done
    > Buy refurb windows 7 laptop that has never been 'seen' by the website - done
    > install virtual machine, refresh it daily - I'm about to get VMWare or equivalent
    > connect with IPVanish, and use anti-leak settings etc (already have account with them)
    > block FIOS in/out ports via firewall rules (have had trouble making this work properly)
    > proxy in IE just in case (I have to use IE with this bot)
    > run VPNwatch or equivalent and kill bot upon disconnect
    > never go to that website on that machine or any device via the router it runs through
    > open new alias account for site account activity related to scraping
    > turn off bot at night, use all new queries versus b4 they warned me

    what else should i do?
    should i use openvpn? not sure yet what that is all about
    any IE settings that i should change within the virtual machine? (i have to use IE with this bot)
    > (settings: cookies, tracking, geo-location, other?)
    damn what if FIOS ports change? do they change?
    anything else i can/should do versus geo-location detection?
    anything else i'm missing?

    I have 2 routers. my vanilla machines doing vanilla stuff are on 1 router. the old and new scraping laptops are currently on other router, wireless connection.

    i have 50 dedicated proxies available for whatever. i use them in other software

    i realize i may have some overkill going on, but i was such a dumbass getting the warning and don;t want to screw up and get caught again. and you never know what part of your setup will fail or how the game will change over time.

    i know this websites basic strategy versus botting. have seen a video detailing it! i know they go as far as user agent tracking. but they seem to either be unable or not bothering to block IPs automatically. they blocked IPs automatically maybe a year or 2 ago, but that is no longer an issue. they use captcha but its not an issue for me. i was doing basically nothing to avoid detection for a year other than not logging into site on that machine. and i was not getting blocked. amazingly i only got the warning once i committed to lowest level of stupidity by logging via the bot laptop. even with that, I must have logged in 50 times over a couple months before they came up with a match. or perhaps there was something else about my account activity that made them take a look at me; or they had a crackdown; or who the heck knows.