ASK ME ANYTHING about web automation | web scraping

courses_retailer said:
undetectable it will work man,
you may be just did something wrong
Click to expand...
did you try it this year?
I saw so many videos and docs and all of them for the Past years and it was working with them
but now it don't
here is screen with the problem
 

Attachments

  • MnjWd.png
    MnjWd.png
    30.2 KB · Views: 77
Advertise on BHW
middeen2004 said:
did you try it this year?
I saw so many videos and docs and all of them for the Past years and it was working with them
but now it don't
here is screen with the problem
Click to expand...
I am working on several projects using undetectable browser and it works fine

could you show me the imports and you code that lunch the browser
 
courses_retailer said:
if Cloudflare stops you just use undetectable browser
+ For social media it is important to put random wait times and don't put a static wait time
+ If you find a way to spoof the MAC address
then you are king nothing will stop you
Click to expand...
Thank you for the share :)
 
SocialTuber said:
What do you mean with post-paid? Thanks for the tip btw, it helps me out a lot!
Click to expand...
it means that you can add a card that has $1 on it, do $10000000 worth of searches, download the data and be done with the site. At least that's how it was back when I was using it. I did do like $50 of searches without paying for them, end of the story
 
courses_retailer said:
I am working on several projects using undetectable browser and it works fine

could you show me the imports and you code that lunch the browser
Click to expand...
I was working on it from weeks ago and edit the code a lot but I think this is the last version I made for undetected
from selenium import webdriver from selenium.webdriver.common.keys import Keys from undetected_chromedriver import Chrome from selenium.webdriver.common.by import By #from selenium_stealth import stealth # Start a new instance of the Chrome driver options = webdriver.ChromeOptions() #options.add_argument('--disable-extensions') options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36") options.add_argument('--profile-directory=Default') options.add_argument("--password-store=basic") #options.add_argument("--incognito") options.add_argument("--disable-plugins-discovery") options.add_argument("--start-maximized") options.add_argument("--disable-notifications") options.add_argument("--disable-infobars") #options.add_argument("--disable-extensions") #options.add_argument("--disable-web-security") options.add_argument('--disable-blink-features=AutomationControlled') driver = Chrome(options=options) # stealth(driver, # languages=["en-US", "en"], # vendor="Google Inc.", # platform="Win32", # webgl_vendor="Intel Inc.", # renderer="Intel Iris OpenGL Engine", # fix_hairline=True, # ) # Navigate to the Google login page driver.get("https://accounts.google.com/") # Find the email field and enter your email address email_field = driver.find_element(By.XPATH, '//*[@id="identifierId"]') email_field.send_keys("[email protected]") email_field.send_keys(Keys.RETURN) # Wait for the password field to load, then enter your password # (Note: you may need to wait a bit longer than this depending on your network speed) driver.implicitly_wait(10) password_field = driver.find_element(By.XPATH, '//*[@id="password"]/div[1]/div/div[1]/input') password_field.send_keys("password") password_field.send_keys(Keys.RETURN) driver.implicitly_wait(20) # You should now be logged in to Google
if you can't see the problem could you share your code for login in google
 
What is the best mobile / residential proxy for scraping?

GB sales system where you can create many ports and use by bandwidth?
Mobile proxies? As private devices ports?

Give me some ideas what would be your perfect scheme for web scraping proxy
 
courses_retailer said:
1. such tasks can be done but tbh I never think of these such tasks :)
2. captcha try 2captcha API for proxies I use oxylabs as they provide a high user experience but they are expensive a little bit
3. it is the easiest task on selenium
here is an ex using java
public static void main(String[] args) { System.setProperty("webdriver.chrome.driver", "chromedriver.exe"); ChromeOptions options = new ChromeOptions(); options.addArguments("here you will put your fake user agent"); WebDriver driver = new ChromeDriver(options); }
and here is an ex using python
options = Options() options.add_argument("--user-agent= her you will add your fake user agent") driver = webdriver.Chrome(executable_path="chromedriver.exe", options=options)
4. The most complicated task is the dynamically loaded page but you can bypass this with Selenium with ease
Click to expand...
I was expecting more detailed answer from experience for number 1, and 4. but ok. I mean I'm a developer such an answer can be googled or ChatGPTed.

Thank you,
 
middeen2004 said:
I was working on it from weeks ago and edit the code a lot but I think this is the last version I made for undetected
from selenium import webdriver from selenium.webdriver.common.keys import Keys from undetected_chromedriver import Chrome from selenium.webdriver.common.by import By #from selenium_stealth import stealth # Start a new instance of the Chrome driver options = webdriver.ChromeOptions() #options.add_argument('--disable-extensions') options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36") options.add_argument('--profile-directory=Default') options.add_argument("--password-store=basic") #options.add_argument("--incognito") options.add_argument("--disable-plugins-discovery") options.add_argument("--start-maximized") options.add_argument("--disable-notifications") options.add_argument("--disable-infobars") #options.add_argument("--disable-extensions") #options.add_argument("--disable-web-security") options.add_argument('--disable-blink-features=AutomationControlled') driver = Chrome(options=options) # stealth(driver, # languages=["en-US", "en"], # vendor="Google Inc.", # platform="Win32", # webgl_vendor="Intel Inc.", # renderer="Intel Iris OpenGL Engine", # fix_hairline=True, # ) # Navigate to the Google login page driver.get("https://accounts.google.com/") # Find the email field and enter your email address email_field = driver.find_element(By.XPATH, '//*[@id="identifierId"]') email_field.send_keys("[email protected]") email_field.send_keys(Keys.RETURN) # Wait for the password field to load, then enter your password # (Note: you may need to wait a bit longer than this depending on your network speed) driver.implicitly_wait(10) password_field = driver.find_element(By.XPATH, '//*[@id="password"]/div[1]/div/div[1]/input') password_field.send_keys("password") password_field.send_keys(Keys.RETURN) driver.implicitly_wait(20) # You should now be logged in to Google
if you can't see the problem could you share your code for login in google
Click to expand...
delete all this and write this
import undetected_chromedriver as uc
driver = uc.chrome()
# your code here
and that's set you are don't if you run this code a new Chrome will open and you can log in with it on Google :)

BeyondDigital said:
How do we make small automated softwares for our daily work with chat gpt?
Click to expand...
like what ??
Anne Hemingway said:
What is the best mobile / residential proxy for scraping?

GB sales system where you can create many ports and use by bandwidth?
Mobile proxies? As private devices ports?

Give me some ideas what would be your perfect scheme for web scraping proxy
Click to expand...
I am always using oxylabs for proxies, especially for scraping

devilmoldova said:
I was expecting more detailed answer from experience for number 1, and 4. but ok. I mean I'm a developer such an answer can be googled or ChatGPTed.

Thank you,
Click to expand...

what details do you want to know :)
jadecoder said:
how to use proxy with authentication in python selenium?
Click to expand...
what do you mean by this, could you explain more
 
jadecoder said:
I want to run python selenium using proxy that has username and password authentication
Click to expand...
hmm ok
here is the code that you need
USERNAME = "YOUR USERNAME"
PASSWORD = "TOUR PASSWORD"
ENDPOINT = "nz-pr.oxylabs.io:10000" # you need to replace it with your endpoint that you want

def chrome_proxy(user: str, password: str, endpoint: str) -> dict:
wire_options = {
"proxy": {
"http": f"http://{user}:{password}@{endpoint}",
"https": f"http://{user}:{password}@{endpoint}",
}
}

return wire_options

this will return to you a selenium wire that will change IP for every session
 
courses_retailer said:
hmm ok
here is the code that you need
USERNAME = "YOUR USERNAME"
PASSWORD = "TOUR PASSWORD"
ENDPOINT = "nz-pr.oxylabs.io:10000" # you need to replace it with your endpoint that you want

def chrome_proxy(user: str, password: str, endpoint: str) -> dict:
wire_options = {
"proxy": {
"http": f"http://{user}:{password}@{endpoint}",
"https": f"http://{user}:{password}@{endpoint}",
}
}

return wire_options

this will return to you a selenium wire that will change IP for every session
Click to expand...
thanks, i can't use auth proxy in undetectable chrome driver right?
 
What are some best practices for web automation and web scraping, such as avoiding legal or ethical concerns?
 
courses_retailer said:
no there is a smart way than that
go manually and config the search link that you want to give to the scraper by opening the advanced setting on your Google and put infinite scroll
then get the link and put it on your scraper
Click to expand...
I've been messing with puppeteer and Im trying to use it to go to youtube and switch channels with the help of ChatGPT, but man.. I dont know what the hell to do to make it work, here's my latest attempt code:
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const fetch = require('node-fetch');
puppeteer.use(StealthPlugin());
(async () => {
// Get the port and wsEndpoint values from the API
const response = await fetch('http://localhost:3001/v1.0/browser_profiles/profile_id/start?automation=1');
const data = await response.json();
const port = data.automation.port;
const wsEndpoint = data.automation.wsEndpoint;
// Connect to the browser
const browser = await puppeteer.connect({
browserWSEndpoint: `ws://127.0.0.1:${port}${wsEndpoint}`
});
// Navigate to YouTube
const page = await browser.newPage();
await page.setViewport({ width: 1280, height: 720 });
await page.goto('https://www.youtube.com/');
// Click on the avatar button to open the menu
await page.click('button#avatar-btn');
// Click on the "Switch account" link in the menu
await switchAccount(page);
// Function to switch YouTube accounts
async function switchAccount(page) {
// Wait for the "Switch account" link to appear
await page.waitForSelector('ytd-compact-link-renderer #content-icon yt-icon path');
// Click on the "Switch account" link
await page.click('//*[@id="items"]/ytd-compact-link-renderer[3]');
// Wait for the accounts menu to appear
await page.waitForSelector('tp-yt-iron-dropdown tp-yt-paper-listbox');
// Click on "Your channel" in the accounts menu
await page.click('tp-yt-iron-dropdown tp-yt-paper-listbox ytd-compact-link-renderer:nth-child(2)');
// Wait for the page to reload
await page.waitForNavigation();
console.log('Switched to "Your channel".');
}
// Close the browser
await browser.close();
})();
Click to expand...

do you know how can I make it click the "Switch Account" to be able to switch channels?
 
untaxed said:
I've been messing with puppeteer and Im trying to use it to go to youtube and switch channels with the help of ChatGPT, but man.. I dont know what the hell to do to make it work, here's my latest attempt code:


do you know how can I make it click the "Switch Account" to be able to switch channels?
Click to expand...
I am sorry mate but I never used puppeteer with JS
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top