ASK ME ANYTHING about web automation | web scraping

[Alfon]

Do You really think that extra-stealth-plugin is enough?
For me it stopped working on one website at some point (datadome blocking me), and I got to move to another library.

No, it's not enough, and anyone saying that it is is lying or has no idea what he's doing, lol.
Your WebGL, WebRTC, Canvas, fonts...etc fingerprints will be unique across all your accounts/sessions and you'll be detected easily.

 

[SeasonedCode]

No, it's not enough, and anyone saying that it is is lying or has no idea what he's doing, lol.
Your WebGL, WebRTC, Canvas, fonts...etc fingerprints will be unique across all your accounts/sessions and you'll be detected easily.

Exactly I agree 100%

 

[SocialTuber]

no there is a smart way than that
go manually and config the search link that you want to give to the scraper by opening the advanced setting on your Google and put infinite scroll
then get the link and put it on your scraper

Could you check the following code and maybe give me improvements?

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

driver = webdriver.Chrome()

driver.get("https://www.google.com/")

search_box = driver.find_element_by_name("q")
search_box.send_keys("your search query")
search_box.send_keys(Keys.RETURN)

time.sleep(5)


search_results = driver.find_elements_by_xpath("//div[@class='r']/a")
for result in search_results[:100]:
url = result.get_attribute("href")
driver.get(url)
time.sleep(2)

emails = driver.find_elements_by_xpath("//a[contains(@href, 'mailto:')]")
for email in emails:
print(email.get_attribute("href")[7:])


driver.quit()

 

[SeasonedCode]

Could you check the following code and maybe give me improvements?

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

driver = webdriver.Chrome()

driver.get("https://www.google.com/")

search_box = driver.find_element_by_name("q")
search_box.send_keys("your search query")
search_box.send_keys(Keys.RETURN)

time.sleep(5)


search_results = driver.find_elements_by_xpath("//div[@class='r']/a")
for result in search_results[:100]:
url = result.get_attribute("href")
driver.get(url)
time.sleep(2)

emails = driver.find_elements_by_xpath("//a[contains(@href, 'mailto:')]")
for email in emails:
print(email.get_attribute("href")[7:])


driver.quit()

Is this code working for you

I don't think so

 

[Stagged]

How would you make a scraper that scrapes the internet, to find websites in a certain niche with contact information? Any google language. What would such a scraper cost?

 

[SocialTuber]

Is this code working for you

I don't think so

No it’s not, I basically don’t have much coding experience, but do know some things in python, and this is the result of 70% ChatGPT and 30% me, I just don’t see why it’s not working, but that’s also because I am very limited in my knowledge

 

[SeasonedCode]

How would you make a scraper that scrapes the internet, to find websites in a certain niche with contact information? Any google language. What would such a scraper cost?

you can not be done this using traditional coding without AI
you need an AI mode here that is capable to understand HTML code

No it’s not, I basically don’t have much coding experience, but do know some things in python, and this is the result of 70% ChatGPT and 30% me, I just don’t see why it’s not working, but that’s also because I am very limited in my knowledge

yeah just forgot about this code man, it won't work
and here are some key factors to improve your code
first, you can use this
driver.get(f"https://www.google.com/search?q={your search query}")

instant of this

driver.get("[URL]https://www.google.com/[/URL]") search_box = driver.find_element_by_name("q") search_box.send_keys("your search query") search_box.send_keys(Keys.RETURN)
this is an outdated selenium find method
driver.find_elements_by_xpath("//div[@class='r']/a")
you need to use this
driver.find_element(By.XPATH, "your XPATH")

like I have said before this won't work
for result in search_results[:100]:
url = result.get_attribute("href")
driver.get(url)
time.sleep(2)

emails = driver.find_elements_by_xpath("//a[contains(@href, 'mailto:')]")
for email in emails:
print(email.get_attribute("href")[7:])

you need to put your SERP requirements on the Google URL so you can go directly

 

[benis123]

thanks! I was already using the extra strealth plugin, do you where the directory paths for each browser profile is located?

I want to specify the profile and use my exported cookie. The goal is to auto post youtube videos to my channel and then scale that.

just ps aux | grep Anty to find out.

on mac it's /Users/FFFF/Library/Application Support/dolphin_anty/browser_profiles/1323133/data_dir

 

[benis123]

How to make a webscraper that scrapes email adresses from google search results.
For example, I search: mechanics in London and that I can choose from how many results I want the email adresses scraped, should I choose 100 for example, then from result 1-100 the email adresses will be scraped and exported to a csv file.

outscraper does this btw, and it's post-paid, so you can just use it for free and never pay

 

[SocialTuber]

outscraper does this btw, and it's post-paid, so you can just use it for free and never pay

What do you mean with post-paid? Thanks for the tip btw, it helps me out a lot!

 

[mark jarve]

I have a stupid question, but I have to ask it. It's been on my mind for a month. How can I search for a large number of WordPress sites that use plugins that have a certain vulnerability? How can I find and scrape such a thing.?

 

[mark jarve]

Another question. I have a freelance site. The idea of the site is that the owner of the project comes and publishes an offer. For example, he wants a WordPress website designer, but I cannot communicate with this person except when I submit an offer to him. The image of the owner of the project and his full name appear. What I specifically want is to know that person without submitting any request. For work, as they allow part of the name of the project owner to be shown without his image, any ideas

 

[SeasonedCode]

I have a stupid question, but I have to ask it. It's been on my mind for a month. How can I search for a large number of WordPress sites that use plugins that have a certain vulnerability? How can I find and scrape such a thing.?

you can use google dork for this to get the SERP page that has such websites
then you need to fetch every website source code to look at the plugin that you want

Another question. I have a freelance site. The idea of the site is that the owner of the project comes and publishes an offer. For example, he wants a WordPress website designer, but I cannot communicate with this person except when I submit an offer to him. The image of the owner of the project and his full name appear. What I specifically want is to know that person without submitting any request. For work, as they allow part of the name of the project owner to be shown without his image, any ideas

I don't have any idea about this, you need to discuss this with a backend dev

 

[neutrall007]

I have no particular question i just want to thank you OP for helping the community

 

[TheToolbox]

What you use to hide fingerprints and bypass bot detection systems ?

 

[SeasonedCode]

What you use to hide fingerprints and bypass bot detection systems ?

if Cloudflare stops you just use undetectable browser
+ For social media it is important to put random wait times and don't put a static wait time
+ If you find a way to spoof the MAC address
then you are king nothing will stop you

 

[middeen2004]

do you know how to login with google by email and password
I tried different webdriver but all give the same problem when try to login
your browser isn't secure

 

[SeasonedCode]

do you know how to login with google by email and password
I tried different webdriver but all give the same problem when try to login
your browser isn't secure

if you are using selenium you can by pass this with 2 methods
1. use a webdriverr called undetectable webdriver
2. use Chrome portable profile

 

[middeen2004]

if you are using selenium you can by pass this with 2 methods
1. use a webdriverr called undetectable webdriver
2. use Chrome portable profile

I tried undetectable before but it give the same problem
I used firfox portable before also didn't work but will it be different for Chrome portable driver?

 

[SeasonedCode]

I tried undetectable before but it give the same problem
I used firfox portable before also didn't work but will it be different for Chrome portable driver?

undetectable it will work man,
you may be just did something wrong