Hello to the people who love automating their life.

scrapingpundit · Mar 13, 2015

Hey all,

I am a Rails-Python programmer who loves to automate almost all business aspects for my clients (and me). I am deep down into data mining (including web scraping), API integrations, online automata, custom CLI/server scripts for product promotion, etc.

I apologize for this promotion, but a kid's gotta eat. I must tell people who I am, and what they can expect from me, when I'm to introduce myself. Isn't that the goal? So, there's that.

I wanted to say hi to all the members here on BHW, and I can say this (with confidence) that I will be a loyal and intriguing friend to peeps who:
- know what CLI means, and can run CLI apps on their servers.
- prefer ever-updating web based interfaces to scripts rather than clunky non-updating desktop UIs.
- love to talk about programming, esp. in Ruby or Python and their aesthetics.
- want to learn programming with a sincere dedication - I'll help however I can.
- love to automate their online business, wherever and however, possible in order to grow revenue.

I will be providing the BHW community with various data sets, scripts, web applications for their use, in a desperate attempt to have this profile get some gravitas.

For starters, I created a simple script (for this specific thread) to get the keywords from Amazon's best sellers index (took me around 30 mins). I took a random sample of 4000 categories from that scraping, sorted it and uploaded it here: paste.bin.c.o.m/KHrwdD6E for anyone looking for it. You should correct that url with common sense, since I am not allowed to post urls at the moment.

Let me know, if you need help on processing this list.

jamie3000 · Mar 13, 2015

Good to see a fellow experienced coder joining

Wiseman05 · Mar 13, 2015

Well hey there. All of that actually sounds really kind of interesting, I'm not going to lie. I can't say that I follow well your list of criteria for getting along. But, I do know that this one fits me pretty well " - want to learn programming with a sincere dedication - I'll help however I can." . I come from a background of Business, Finance, and Accounting, or as much as they're willing to teach in college, but I'm always interested in expanding my repertoire of skills. Do you have a recommended book, or website for programming newbs? I actually am genuinely interested in learning.

Also welcome to the community! I've been lurking for the last several days before officially joining (just joined earlier today) and I've already learned more than my previous 2 semesters at school.

nanavlad · Mar 13, 2015

[h=1]Whoops.[/h]

[COLOR=rgba(0, 0, 0, 0.247059)]We seem to have missed the gist of that gist you were looking for.[/COLOR]

scrapingpundit · Mar 13, 2015

Updated URL. Seems like the Gist got deleted as someone flagged it or something. Uploaded on Pastebin now.
paste.bin.c.o.m/KHrwdD6E

scrapingpundit · Mar 13, 2015

Wiseman05 said:
Well hey there. All of that actually sounds really kind of interesting, I'm not going to lie. I can't say that I follow well your list of criteria for getting along. But, I do know that this one fits me pretty well " - want to learn programming with a sincere dedication - I'll help however I can." . I come from a background of Business, Finance, and Accounting, or as much as they're willing to teach in college, but I'm always interested in expanding my repertoire of skills. Do you have a recommended book, or website for programming newbs? I actually am genuinely interested in learning.

Also welcome to the community! I've been lurking for the last several days before officially joining (just joined earlier today) and I've already learned more than my previous 2 semesters at school.

Thanks for the welcome. I sure know of several programming books or websites that you can tap into for learning purposes. But, rather than suggesting you something half-arsed, I would like to know what specifically you are aiming for, e.g. automating things, designing websites, developing custom apps/solutions, freelancing or simply, a fair understanding of a language.

Needless to say, Ruby and Python will be my top pick for programming languages at the moment for almost all the above aims. Google Michael Hartl for a detailed getting-started tutorial on Ruby On Rails.

darkrache · Mar 13, 2015

It is such a pleasure to have you here! Your coding skills will help you vastly in the future in internet marketing. I am a coder too and the number of things I've been able to automate/make easier has saved me so much time! It's also great that you're dedicating time to help benefit the BHW community! Great to have you here!

If you ever need help or want to talk, feel free to PM me - and again, welcome!

Gintoki · Mar 13, 2015

Welcome! I've been trying to get into automation lately. I tried to create a script with autoit3 that logs in to a browser based game and does stuff in my place, all that without any kind of browser simulation (so basically POST / GET requests ) to keep resource usage to a minimum +keep it fast and of course so I can run it in the background on my work computer

But sadly I can't seem to get the captcha image this way as it is dynamically loaded in an iframe, so I can't extract it from the source code without it being displayed in an actual browser window... So I'm stuck, but we'll see how it goes. Do you have any kind of suggestion for this problem? Maybe I should open a browser window and hide it... Or use another language maybe?
Thank you for your time, I hope you will enjoy your stay here on BHW!

angelas111 · Mar 14, 2015

Hi, Do you have skype?

krozbonek · Mar 14, 2015

hi scrapingpundit, welcome!

baokydev · Mar 14, 2015

Hello.

I fit few of your criteria

Love to automate business. Yes my business run on automation.

Love to use cli. Definitely because its the only or rather best option for automation and resource optimize

But sorry i dont use python(that much) or ruby(totally no)

I run automation with php hahaha.. but well as long its works

iwillnbd · Mar 14, 2015

Hi, welcome aboard!
Your skill sets will definitely help you a lot in this line.

What would you suggest for a programming newbie, with some very basic knowledge of c/c++, wanting to get into web-automation(scraping,auto-posting to sites etc.) ?

laur.laurix · Mar 14, 2015

Show us the goods....

scrapingpundit · Mar 14, 2015

darkrache said:
It is such a pleasure to have you here! Your coding skills will help you vastly in the future in internet marketing. I am a coder too and the number of things I've been able to automate/make easier has saved me so much time! It's also great that you're dedicating time to help benefit the BHW community! Great to have you here!

If you ever need help or want to talk, feel free to PM me - and again, welcome!

Thanks for the welcome

Regarding the PM, I sure will take you up on that.

angelas111 said:
Hi, Do you have skype?

Yes, just added to the profile.

iwillnbd said:
Hi, welcome aboard!
Your skill sets will definitely help you a lot in this line.

What would you suggest for a programming newbie, with some very basic knowledge of c/c++, wanting to get into web-automat.ion(scraping,auto-posting to sites etc.) ?

I would really ask you to look into Python, mechanize and scrapy. Or, Ruby and mechanize.

Gintoki said:
Welcome! I've been trying to get into automat.ion lately. I tried to create a script with autoit3 that logs in to a browser based game and does stuff in my place, all that without any kind of browser simulation (so basically POST / GET requests ) to keep resource usage to a minimum +keep it fast and of course so I can run it in the background on my work computer But sadly I can't seem to get the captcha image this way as it is dynamically loaded in an iframe, so I can't extract it from the source code without it being displayed in an actual browser window... So I'm stuck, but we'll see how it goes. Do you have any kind of suggestion for this problem? Maybe I should open a browser window and hide it... Or use another language maybe?
Thank you for your time, I hope you will enjoy your stay here on BHW!

Well, I am more into automating things on the web end - as I must have said above, but you can always use a captcha breaking service for this. Regarding the problem with actually extracting the image source for the captcha, please first make sure that you can reload the captcha without it being refreshed. If you can do so, one easy way would be to use a small headless html parser to load the dom and parse it to extract the img source, or you can simply use regex/grep etc. I am not sure if autoit3 would allow you to do so (not a windows user myself), but if it allows some good support for string manipulation, you only need to extract the captcha image, and then break it. Let me know, if that works.

If you were to do this in Python, or Ruby, I would use a headless browser like Watir or capybara webkit to simulate this. It's way easy with that. Out of curiosity, which game?

baokydev said:
Hello.

I fit few of your criteria

Love to automat.e business. Yes my business run on automat.ion.

Love to use cli. Definitely because its the only or rather best option for automat.ion and resource optimize

But sorry i dont use python(that much) or ruby(totally no)

I run automat.ion with php hahaha.. but well as long its works

Wow, Glad to meet you. What kind of things are we automating here? Promotions? Moreover, I did not mean that automating stuff works only with Python or Ruby. I like PHP. I like the way its easy. But, I just prefer Python/Ruby more because of cleanliness. Let me know, if I can help out with some of the tasks. And, once again, glad to meet you

PS: BHW moderation is way too much. Can't really say anything.

scrapingpundit · Mar 14, 2015

........merged above.....

fun4uoc · Mar 16, 2015

Brand new to ruby and am issue with uploading.

I'm on a Mac and on a regular site I can upload with no issues with the following code.

Code:

filename = '/path/to/file.jpg'
  file = File.join(Dir.pwd, filename)

  driver.get 'http://site.com/upload'
  driver.find_element(id: 'file-upload').send_keys file
  driver.find_element(id: 'file-submit').click

The site I'm having an issue with is using js for the upload and I'm not sure where to go. I can click on the link which opens the finder window but I'm unable to choose the image and click Open.

I'd rather not go with SikuliX if at all possible.

Thanks

ThreadKiller · Mar 16, 2015

Good introduction. I am way into automation and scraping as well. Usually use iMacros for it. I don't quite have the brain to learn languages.

Anyway, gonna read your posts with interest. Welcome to BHW.

12040 · Mar 16, 2015

i clicked the link because i saw "automating"

welcome

scrapingpundit · Mar 18, 2015

fun4uoc said:
Brand new to ruby and am issue with uploading.....

Without actually being able to see the site in question, all I can recommend is that you can use a headless javascript based driver for filling the form/selecting the file. This should work most probably. Example driver is PhantomJS based poltergeist.

jamie3000 · Mar 21, 2015

Gintoki said:
Welcome! I've been trying to get into automation lately. I tried to create a script with autoit3 that logs in to a browser based game and does stuff in my place, all that without any kind of browser simulation (so basically POST / GET requests ) to keep resource usage to a minimum +keep it fast and of course so I can run it in the background on my work computer But sadly I can't seem to get the captcha image this way as it is dynamically loaded in an iframe, so I can't extract it from the source code without it being displayed in an actual browser window... So I'm stuck, but we'll see how it goes. Do you have any kind of suggestion for this problem? Maybe I should open a browser window and hide it... Or use another language maybe?
Thank you for your time, I hope you will enjoy your stay here on BHW!

If you know the parameters expected can you not make the call to the server that iframe does yourself?

Hello to the people who love automating their life.

Newbie

Elite Member

Newbie

Elite Member

Newbie

Newbie

Junior Member

Regular Member

Elite Member

Newbie

Regular Member

Power Member

Elite Member

Newbie

Newbie

Supreme Member

Power Member

Junior Member

Newbie

Elite Member

Main Menu

Marketplace

Making Money

BlackHatWorld

Other