1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Hello to the people who love automating their life.

Discussion in 'Introductions' started by scrapingpundit, Mar 13, 2015.

  1. scrapingpundit

    scrapingpundit Newbie

    Joined:
    Mar 13, 2015
    Messages:
    6
    Likes Received:
    0
    Location:
    States
    Hey all,

    I am a Rails-Python programmer who loves to automate almost all business aspects for my clients (and me). I am deep down into data mining (including web scraping), API integrations, online automata, custom CLI/server scripts for product promotion, etc.

    I apologize for this promotion, but a kid's gotta eat. I must tell people who I am, and what they can expect from me, when I'm to introduce myself. Isn't that the goal? So, there's that. :)

    I wanted to say hi to all the members here on BHW, and I can say this (with confidence) that I will be a loyal and intriguing friend to peeps who:
    - know what CLI means, and can run CLI apps on their servers.
    - prefer ever-updating web based interfaces to scripts rather than clunky non-updating desktop UIs.
    - love to talk about programming, esp. in Ruby or Python and their aesthetics.
    - want to learn programming with a sincere dedication - I'll help however I can.
    - love to automate their online business, wherever and however, possible in order to grow revenue.

    I will be providing the BHW community with various data sets, scripts, web applications for their use, in a desperate attempt to have this profile get some gravitas.

    For starters, I created a simple script (for this specific thread) to get the keywords from Amazon's best sellers index (took me around 30 mins). I took a random sample of 4000 categories from that scraping, sorted it and uploaded it here: paste.bin.c.o.m/KHrwdD6E for anyone looking for it. You should correct that url with common sense, since I am not allowed to post urls at the moment.

    Let me know, if you need help on processing this list.
     
    Last edited: Mar 13, 2015
  2. jamie3000

    jamie3000 Supreme Member

    Joined:
    Jun 30, 2014
    Messages:
    1,311
    Likes Received:
    586
    Occupation:
    Finance coder looking for semi-retirement
    Location:
    uk
    Good to see a fellow experienced coder joining :)
     
    • Thanks Thanks x 1
  3. Wiseman05

    Wiseman05 Newbie

    Joined:
    Mar 10, 2015
    Messages:
    34
    Likes Received:
    4
    Gender:
    Male
    Occupation:
    Self Employed.
    Location:
    At Home
    Well hey there. All of that actually sounds really kind of interesting, I'm not going to lie. I can't say that I follow well your list of criteria for getting along. But, I do know that this one fits me pretty well " - want to learn programming with a sincere dedication - I'll help however I can." . I come from a background of Business, Finance, and Accounting, or as much as they're willing to teach in college, but I'm always interested in expanding my repertoire of skills. Do you have a recommended book, or website for programming newbs? I actually am genuinely interested in learning.

    Also welcome to the community! I've been lurking for the last several days before officially joining (just joined earlier today) and I've already learned more than my previous 2 semesters at school.
     
    • Thanks Thanks x 1
  4. nanavlad

    nanavlad Jr. VIP Jr. VIP

    Joined:
    Dec 2, 2009
    Messages:
    2,488
    Likes Received:
    921
    Gender:
    Male
    Occupation:
    SEO Consultant
    Location:
    Proxy Central
    [h=1]Whoops.[/h]
    [COLOR=rgba(0, 0, 0, 0.247059)]We seem to have missed the gist of that gist you were looking for.[/COLOR]​
     
    • Thanks Thanks x 1
  5. scrapingpundit

    scrapingpundit Newbie

    Joined:
    Mar 13, 2015
    Messages:
    6
    Likes Received:
    0
    Location:
    States
    Updated URL. Seems like the Gist got deleted as someone flagged it or something. Uploaded on Pastebin now.
    paste.bin.c.o.m/KHrwdD6E
     
  6. scrapingpundit

    scrapingpundit Newbie

    Joined:
    Mar 13, 2015
    Messages:
    6
    Likes Received:
    0
    Location:
    States
    Thanks for the welcome. I sure know of several programming books or websites that you can tap into for learning purposes. But, rather than suggesting you something half-arsed, I would like to know what specifically you are aiming for, e.g. automating things, designing websites, developing custom apps/solutions, freelancing or simply, a fair understanding of a language.

    Needless to say, Ruby and Python will be my top pick for programming languages at the moment for almost all the above aims. Google Michael Hartl for a detailed getting-started tutorial on Ruby On Rails.
     
  7. darkrache

    darkrache Junior Member Premium Member

    Joined:
    Mar 27, 2011
    Messages:
    148
    Likes Received:
    162
    It is such a pleasure to have you here! Your coding skills will help you vastly in the future in internet marketing. I am a coder too and the number of things I've been able to automate/make easier has saved me so much time! It's also great that you're dedicating time to help benefit the BHW community! Great to have you here!

    If you ever need help or want to talk, feel free to PM me - and again, welcome!
     
    • Thanks Thanks x 1
  8. Gintoki

    Gintoki Regular Member

    Joined:
    Mar 24, 2012
    Messages:
    215
    Likes Received:
    152
    Occupation:
    Freelancer
    Welcome! I've been trying to get into automation lately. I tried to create a script with autoit3 that logs in to a browser based game and does stuff in my place, all that without any kind of browser simulation (so basically POST / GET requests ) to keep resource usage to a minimum +keep it fast and of course so I can run it in the background on my work computer :) But sadly I can't seem to get the captcha image this way as it is dynamically loaded in an iframe, so I can't extract it from the source code without it being displayed in an actual browser window... So I'm stuck, but we'll see how it goes. Do you have any kind of suggestion for this problem? Maybe I should open a browser window and hide it... Or use another language maybe?
    Thank you for your time, I hope you will enjoy your stay here on BHW!
     
    • Thanks Thanks x 1
  9. angelas111

    angelas111 Jr. VIP Jr. VIP

    Joined:
    Jan 4, 2009
    Messages:
    1,591
    Likes Received:
    1,026
    Location:
    ohio
    Hi, Do you have skype?
     
  10. krozbonek

    krozbonek Newbie

    Joined:
    Jul 15, 2012
    Messages:
    30
    Likes Received:
    1
    Location:
    Indonesia
    Home Page:
    hi scrapingpundit, welcome!
     
    • Thanks Thanks x 1
  11. baokydev

    baokydev BANNED BANNED

    Joined:
    Mar 11, 2014
    Messages:
    401
    Likes Received:
    150
    Hello.

    I fit few of your criteria

    Love to automate business. Yes my business run on automation.

    Love to use cli. Definitely because its the only or rather best option for automation and resource optimize


    But sorry i dont use python(that much) or ruby(totally no)

    I run automation with php hahaha.. but well as long its works ;)
     
    • Thanks Thanks x 1
  12. iwillnbd

    iwillnbd Regular Member

    Joined:
    Mar 13, 2012
    Messages:
    424
    Likes Received:
    374
    Occupation:
    ....
    Location:
    C:
    Hi, welcome aboard!
    Your skill sets will definitely help you a lot in this line.

    What would you suggest for a programming newbie, with some very basic knowledge of c/c++, wanting to get into web-automation(scraping,auto-posting to sites etc.) ?
     
  13. laur.laurix

    laur.laurix Power Member

    Joined:
    May 8, 2013
    Messages:
    592
    Likes Received:
    222
    Occupation:
    Reverse Engineering Maniac
    Location:
    Mars
    Show us the goods....
     
  14. scrapingpundit

    scrapingpundit Newbie

    Joined:
    Mar 13, 2015
    Messages:
    6
    Likes Received:
    0
    Location:
    States

    Thanks for the welcome :) Regarding the PM, I sure will take you up on that. :)


    Yes, just added to the profile.

    I would really ask you to look into Python, mechanize and scrapy. Or, Ruby and mechanize.

    Well, I am more into automating things on the web end - as I must have said above, but you can always use a captcha breaking service for this. Regarding the problem with actually extracting the image source for the captcha, please first make sure that you can reload the captcha without it being refreshed. If you can do so, one easy way would be to use a small headless html parser to load the dom and parse it to extract the img source, or you can simply use regex/grep etc. I am not sure if autoit3 would allow you to do so (not a windows user myself), but if it allows some good support for string manipulation, you only need to extract the captcha image, and then break it. Let me know, if that works.


    If you were to do this in Python, or Ruby, I would use a headless browser like Watir or capybara webkit to simulate this. It's way easy with that. Out of curiosity, which game?

    Wow, Glad to meet you. What kind of things are we automating here? Promotions? Moreover, I did not mean that automating stuff works only with Python or Ruby. I like PHP. I like the way its easy. But, I just prefer Python/Ruby more because of cleanliness. Let me know, if I can help out with some of the tasks. And, once again, glad to meet you :)



    PS: BHW moderation is way too much. Can't really say anything.
     
    Last edited: Mar 14, 2015
  15. scrapingpundit

    scrapingpundit Newbie

    Joined:
    Mar 13, 2015
    Messages:
    6
    Likes Received:
    0
    Location:
    States
    ........merged above.....
     
    Last edited: Mar 14, 2015
  16. fun4uoc

    fun4uoc Supreme Member

    Joined:
    Dec 25, 2007
    Messages:
    1,200
    Likes Received:
    1,042
    Location:
    ☆ ♔ ☆ ♔ ☆ ♔ ☆
    Brand new to ruby and am issue with uploading.

    I'm on a Mac and on a regular site I can upload with no issues with the following code.

    Code:
    filename = '/path/to/file.jpg'
      file = File.join(Dir.pwd, filename)
    
      driver.get 'http://site.com/upload'
      driver.find_element(id: 'file-upload').send_keys file
      driver.find_element(id: 'file-submit').click
    The site I'm having an issue with is using js for the upload and I'm not sure where to go. I can click on the link which opens the finder window but I'm unable to choose the image and click Open.

    I'd rather not go with SikuliX if at all possible.

    Thanks
     
  17. ThreadKiller

    ThreadKiller Power Member

    Joined:
    Jan 31, 2012
    Messages:
    627
    Likes Received:
    305
    Location:
    Never too long in one place.
    Good introduction. I am way into automation and scraping as well. Usually use iMacros for it. I don't quite have the brain to learn languages.

    Anyway, gonna read your posts with interest. Welcome to BHW.
     
  18. 12040

    12040 Junior Member

    Joined:
    Dec 16, 2011
    Messages:
    112
    Likes Received:
    60
    i clicked the link because i saw "automating" :D
    welcome
     
  19. scrapingpundit

    scrapingpundit Newbie

    Joined:
    Mar 13, 2015
    Messages:
    6
    Likes Received:
    0
    Location:
    States
    Without actually being able to see the site in question, all I can recommend is that you can use a headless javascript based driver for filling the form/selecting the file. This should work most probably. Example driver is PhantomJS based poltergeist.
     
  20. jamie3000

    jamie3000 Supreme Member

    Joined:
    Jun 30, 2014
    Messages:
    1,311
    Likes Received:
    586
    Occupation:
    Finance coder looking for semi-retirement
    Location:
    uk
    If you know the parameters expected can you not make the call to the server that iframe does yourself?