Selenium script in python to get redirection url using free proxies

DrEinstein10 · Apr 3, 2021

Hi everyone,

I'm trying to build a python script to follow the redirection of some urls that cannot be obtained using regular http calls like curl. (I apologize if the terminology is wrong, I don't really know the technical terms for all of this)

Here's what the script does so far:
- It gets a free proxy ip: port using the pubproxpy library.
- Adds the proxy to chrome options
- Opens chrome browser using the undetected_chromedriver library
- Opens the target url and checks until redirection occurs
- Saves the final url after redirection

The script actually works, however, after a while I started getting errors and warning messages from Cloudflare firewalls, and then at some point I just get a blank page with errors like 429 or 503. I guess there are some footprints that are being leaked, but I'm still learning about all of this and I don't know how to avoid being flagged.

Does anyone have suggestions on how to solve this issue, or how to improve the script to hide all the footprints? If you see anything wrong with my code please let me know.
Here's the code:

Python:

import os
import random
import time
import undetected_chromedriver as uc
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from pubproxpy import Level, Protocol, ProxyFetcher

executable_path = "chromedriver.exe"
os.environ["webdriver.chrome.driver"] = executable_path
chrome_options = Options()
chrome_options.add_argument('--disable-notifications')

chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', False)
chrome_options.add_argument('--disable-blink-features=AutomationControlled')

chrome_options.add_argument("window-size=1280,800")
chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36")

preferences = {
    "webrtc.ip_handling_policy" : "disable_non_proxied_udp",
    "webrtc.multiple_routes_enabled": False,
    "webrtc.nonproxied_udp_enabled" : False
}
chrome_options.add_experimental_option("prefs", preferences)

https_pf = ProxyFetcher(protocol=Protocol.HTTP, https=True, level=Level.ELITE, time_to_connect=20, countries=["US"])
PROXYlist = https_pf.get(1)
PROXY = PROXYlist[0]
print(PROXY)
chrome_options.add_argument('--proxy-server=%s' % PROXY)
      
driver = uc.Chrome(executable_path=executable_path, options=chrome_options)

driver.get('https://httpbin.org/ip')
time.sleep(2)

#These are just example websites:
urlsList=['http://transportr.io/redirect/5ade3fee-82ba-40ea-a394-1232289eab9a','http://www.123haitao.com/redirect/id-aHR0cHM6Ly91cy5wdW1hLmNvbS9lbi91cy9wZC9jYWxpLXNwb3J0LXdvbWVucy1zbmVha2Vycy8zNzEyMDIuaHRtbA%3D%3D__uid-28976','https://www.joyk.com/dig/redirect/1611772549201233','https://www.lootdunia.in/redirect?content=D6i9dlpzNqJy2VTiDZNXKpNgCOaC2o5k7s3MZRSdeiaykn3lIGiBZiLFU0sT%2BAaorBiDN9JsjLZxd3R8DEeHAuoZ24gdLYyxNHQLqn7i5X6K%2FBBzV8KP','https://www.goingpremium.com/en/Redirect/S/26428']
redirectedUrls=[]

for x in urlsList:
    driver.get(x)
    attempts=0
    while (x == driver.current_url and attempts<10):
        time.sleep(5)
        #print(driver.current_url)
        attempts=attempts+1
    if x != driver.current_url:
        redirectedUrls.append(driver.current_url)
    time.sleep(random.randint(5, 20))
driver.quit()
print(redirectedUrls)

thetrustedzone · Apr 3, 2021

stackoverflow.com is better place to ask programming questions, however you could achieve Same thing with http request without third party like selenium , check this :

https://stackoverflow.com/questions/36070821/how-to-get-redirect-url-using-python-requests

DrEinstein10 · Apr 3, 2021

thetrustedzone said:
stackoverflow.com is better place to ask programming questions, however you could achieve Same thing with http request without third party like selenium , check this :

https://stackoverflow.com/questions/36070821/how-to-get-redirect-url-using-python-requests

Thanks, I thought about stackoverflow but I have seen that questions about scrapping or shady practices usually have answers with moral reasons explaining why you shouldn't be doing it instead of an actual solution. Here people are more open minded that's why I wanted to try my luck here. I might try asking this in stackoverflow anyway, thanks for the suggestion.

I have already tried with the http requests but I only get in return the same initial url, it looks like the url has to be opened in a browser in order for the redirection to be triggered, that's why I'm trying with selenium.

Selenium script in python to get redirection url using free proxies

DrEinstein10

Junior Member

thetrustedzone

Elite Member

DrEinstein10

Junior Member

Main Menu

Marketplace

Making Money

BlackHat World