1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

I want to scrape an ajax website, but detects my bot (Ubot professional stealth edition).

Discussion in 'General Programming Chat' started by loadsofthem, Sep 16, 2016.

  1. loadsofthem

    loadsofthem Newbie

    Joined:
    Aug 1, 2016
    Messages:
    14
    Likes Received:
    2
    I'm creating a mailing bot with ubot, but i have a problem when gathering my friends, because there is a button that has to be clicked, so there appear more friends. the probem is that ajax detects that i'm a robot, and the the button that loads more friends, begins to stay frozen for longer periods of time each time is clicked. An then, when the bot tries to click the button, it shows and error about jar and i can only scrape 415 users, and this when making it wat 20 seconds for the button, if i make the button wait 60 secons it gathers 715 users before crashing. when i open the page in a browser and i click by myself it lods after 5 seconds all the times.

    I already use the javascript:

    run javascript("window.scrollTo(0,document.body.scrollHeight);")

    but it it just scrolls the page to the button.

    I found atutorial in google that uses this phython code to scrape pages:

    run python("import mechanize
    import urllib2
    import time
    from selenium import webdriver
    from selenium common exceptions import NoSuchAttributeException
    from selenium.webdriver.common.keys import Keys
    import os
    import re
    br.set_handle_robots(false)
    br.set_handle_refresh(false)
    br.addheaders=[(\'user-agent\'),(\'chrome\')]")


    But when i try to run the scripts, it shows me errors all the time.

    • Does somebody know an alternative way to prevent to load more button from doing this? (I does it provably because it detects a bot)
    • How can i do to insert python code in ubot so it does not have errors?
     
    • Thanks Thanks x 1
  2. ekapek

    ekapek Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 2, 2010
    Messages:
    273
    Likes Received:
    48
    Home Page:
    mechanize from python is a headless webbrowser so it doesn't have js engine
    You can use selenium or you can use javascript to click on that button with click() method
     
  3. SethTurin

    SethTurin Jr. VIP Jr. VIP

    Joined:
    Nov 19, 2011
    Messages:
    282
    Likes Received:
    147
    Occupation:
    President and CEO of Seth Turin Media, Inc. - Crea
    Home Page:
    Hey there,

    Have you asked on our forum? There may be a better response! Also, are you in our skype group ?
     
  4. Google Prince

    Google Prince Jr. VIP Jr. VIP

    Joined:
    Dec 24, 2015
    Messages:
    162
    Likes Received:
    93
    Location:
    Google's Search Engine
    If you still need help with this, pm me I can help. I'm pretty experienced with selenium using python.
     
  5. Iamhere123

    Iamhere123 Newbie

    Joined:
    Jul 28, 2017
    Messages:
    13
    Likes Received:
    0
    Gender:
    Male
    Your only option is to scrape with Selenium as you can choose your browser or get Exbrowser plugin. Ubot browser is too easily detectable. Though Selenium is quite easy to learn. I am still learning it and can't yet scrape pages but can do a lot of other stuff I don't need