1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Anyone good with selenium (python)?

Discussion in 'Programming' started by bk071, Jul 18, 2017.

  1. bk071

    bk071 Jr. Executive VIP Jr. VIP

    Joined:
    Nov 24, 2010
    Messages:
    3,223
    Likes Received:
    8,005
    I'm stuck on something. Will reveal more details in PM/skype.
    Hit me up if you feel like helping a bro out.

    Virtual hugs in return.
     
    • Thanks Thanks x 1
  2. Al Ali

    Al Ali Newbie

    Joined:
    Jul 28, 2017
    Messages:
    9
    Likes Received:
    0
    I need your help with something man. Please write back to me.
     
  3. Brian Alexander

    Brian Alexander Junior Member UnGagged Attendee

    Joined:
    Aug 12, 2016
    Messages:
    149
    Likes Received:
    69
    Gender:
    Male
    I've been using Selenium Webdriver with .NET for years.
    PM me the issue - I can probably help you out.
     
  4. bk071

    bk071 Jr. Executive VIP Jr. VIP

    Joined:
    Nov 24, 2010
    Messages:
    3,223
    Likes Received:
    8,005
  5. bk071

    bk071 Jr. Executive VIP Jr. VIP

    Joined:
    Nov 24, 2010
    Messages:
    3,223
    Likes Received:
    8,005
    What do you need help with?
     
  6. Al Ali

    Al Ali Newbie

    Joined:
    Jul 28, 2017
    Messages:
    9
    Likes Received:
    0
    Is there a way we can talk in particular on this website?
     
  7. Iamhere123

    Iamhere123 Newbie

    Joined:
    Jul 28, 2017
    Messages:
    13
    Likes Received:
    0
    Gender:
    Male
    Speaking of helping a brother out, anyone know how to scrape into an excel sheet. I've learnt a lot about selenium except for the only thing I wanted which was how to scrape into excel lmao xD
     
  8. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,041
    Likes Received:
    10,830
    Occupation:
    WHEREZ MA
    Location:
    BITCOINS AT?
    Home Page:
    Scrape it into CSV and then import into excel.
     
    • Thanks Thanks x 2
  9. Iamhere123

    Iamhere123 Newbie

    Joined:
    Jul 28, 2017
    Messages:
    13
    Likes Received:
    0
    Gender:
    Male
    Yeah haha except when I do it it just gives me a blank spreadsheet xD. Do you have an example script that works? I'd kill for that..... lmao.
     
  10. GoDesain

    GoDesain Regular Member

    Joined:
    Feb 26, 2011
    Messages:
    444
    Likes Received:
    131
    Home Page:
    • Thanks Thanks x 1
  11. Iamhere123

    Iamhere123 Newbie

    Joined:
    Jul 28, 2017
    Messages:
    13
    Likes Received:
    0
    Gender:
    Male
    Scrapy is good but it can't replace an actual browser. Selenium gets the job done where you want to emulate a browser and have no fingerprinting. I thing apache is not working on my pc for some reason . Maybe that's why I keep getting blank excels. Who knows *shrugs*. Time for 5 hours of bug fixing... Scrapy is good if you have lots of pages but selenium is great where you want to maintain a good relationship with the webpage and act as a user instead of just an api.
     
  12. luckystomp

    luckystomp Newbie

    Joined:
    Nov 1, 2012
    Messages:
    14
    Likes Received:
    0
    @Iamhere123

    selenium is great if you need to emulate a human

    as for excel - check stackoverflow there are lots of examples
    and most of them working good
    you can use csv.writer or DictWriter
    depends on your needs
     
  13. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,041
    Likes Received:
    10,830
    Occupation:
    WHEREZ MA
    Location:
    BITCOINS AT?
    Home Page:
    There's probably a loose comma somewhere breaking the CSV. Don't kill for this, it's a very trivial script ;)
     
    • Thanks Thanks x 1
  14. Iamhere123

    Iamhere123 Newbie

    Joined:
    Jul 28, 2017
    Messages:
    13
    Likes Received:
    0
    Gender:
    Male
    Hahaha yeah probably a bit overkill huh? I can scrape the data well enough but any excel actions don't seem to work. I have installed apache a few times :S. I wonder if there's a way to verifiy the install through windows :S. I'll be looking at lots of stackoverflow q's to see if I can't copy someone else's code and hope that works or just try and do a fresh install of apache. I'm sure it's probably something simple af I've overlooked though xD
     
  15. Iamhere123

    Iamhere123 Newbie

    Joined:
    Jul 28, 2017
    Messages:
    13
    Likes Received:
    0
    Gender:
    Male
    Turns out I was missing a few apache files so it wasn't working properly. Now I'm going to have to dig through some stackoverflow q's and see if I can't write to excel with the extracted data :D.
     
  16. bk071

    bk071 Jr. Executive VIP Jr. VIP

    Joined:
    Nov 24, 2010
    Messages:
    3,223
    Likes Received:
    8,005
    Sounds simple. Write to a text file and use semicolon (;) as the delimiter. Then import into excel by going to Data tab and clicking the "From Text" option.
    [​IMG]
    It will ask you about the delimiter, choose semicolon. All done.

    Edit: Of course you can use the CSV library. But the above approach is what I've tried and it worked perfectly. Didn't feel the need to use the csv lib.

    Shoot me a PM if you want me to take a look.
     
    • Thanks Thanks x 1
  17. rafark

    rafark Regular Member

    Joined:
    Jan 15, 2013
    Messages:
    403
    Likes Received:
    189
    Gender:
    Male
    Occupation:
    Moderador
    Location:
    Noble and Heroic MC
    I'm no python programmer but I recently parsed a large spreadsheet fule. What I can say is that don't read it at once, it's better to stream it line by line. An even more optimized way is to convert it to csv and work it from that as Bart Simpson suggested.
     
  18. akashasnani

    akashasnani Regular Member

    Joined:
    Mar 30, 2011
    Messages:
    242
    Likes Received:
    112
    Home Page:
    using csv library u can do it very easily.
     
  19. Iamhere123

    Iamhere123 Newbie

    Joined:
    Jul 28, 2017
    Messages:
    13
    Likes Received:
    0
    Gender:
    Male
    So getting extracted data and putting it straight into excel is slow where there is a a large quantity of data?
     
  20. Iamhere123

    Iamhere123 Newbie

    Joined:
    Jul 28, 2017
    Messages:
    13
    Likes Received:
    0
    Gender:
    Male
    Sounds interesting :). So the delimiter allows you to use the text file and keep the data separate in the text file. Any good examples on this? II'll see if stack overflow has some examples and try it out.