[JOURNEY] Web scraping with scrapy

[n3s7or]

Hello, I am n3s7or.

This is my first journey and probably my very first thread here in the forum. I've been browsing the forum since almost a year next May.

A bit about me:
I am a 26 years old guy who likes to improve skills and learn new ones. So far I can code in python, C/C++ (kinda rusty these days), php, JS. I know a lil bit of Symfony 4 and among other things I've been studying reactjs.

This journey
I found out that I like to create web-scrapers and I have made a little few of them for personal use. So I decided to start this journey in first place to improve my coding skills and, secondly, to find out some ideas of what to scrap.

The tool I will be mainly using is scrapy.

If you have in mind something you would like to scrape please let me know.

Best regards.

PS: Sorry if there is any typo or bad English, I am not native. You can suggest corrections as well

 

[Seoanytime]

For such a journey it's best you utilize the "Joint Ventures" or "Scripting" section..

 

[Swagmonster]

Wishing you the best luck! You can make big $$ scraping sites and creating a good scraping service. If I were you I would focus on creating scrapers for Google and LinkedIn. Those are the two mainly requested sources to be scraped! Good luck!

 

[n3s7or]

Wishing you the best luck! You can make big $$ scraping sites and creating a good scraping service. If I were you I would focus on creating scrapers for Google and LinkedIn. Those are the two mainly requested sources to be scraped! Good luck!

Thanks! I will do my best to improve my python and scrapy skills, I am not worrying about the money part so far.

For such a journey it's best you utilize the "Joint Ventures" or "Scripting" section..

Well, I think I can't move the thread, so if a mod could move it to the "Joint Ventures" section that's Ok for me.

Thanks for your comment.

 

[Ash1]

Hey buddy,
I'm glad to see someone trying that which I was trying from days.

I was trying to scrape email contacts using scrapy.

But, I got a lot of unwanted results.

I'm facing issues in manipulating the regex expression / re.findall() function.
Also, I don't know how I can apply to reject method to avoid unnecessary outcomes.

Let me know if you wanna try it.

@Ash1

 

[Seven4]

Wishing you lots of luck and success with this. Be back soon to check-in. You can do it!

 

[Purush]

I am very much interested in scraping. following your journey. You are giving scraping as service or you scrape the data and using the data for other product.

 

[Mr Schneebly]

Pretty easy nowadays to scrape anything with BeautifulSoup. If you already know python, it’ll take you a day to learn and master scraping

 

[n3s7or]

I am very much interested in scraping. following your journey. You are giving scraping as service or you scrape the data and using the data for other product.

Thanks, I will do my best to keep the thread updated,

Pretty easy nowadays to scrape anything with BeautifulSoup. If you already know python, it’ll take you a day to learn and master scraping

Python is easy to learn, my goal is to master python and learn also web scraping with scrapy. I am aiming to maybe someday to get hired in scrapinhub, lol


Thanks all for your comments

 

[lazguy]

scraping is interesting.. good luck with your journey!

 

[n3s7or]

Not too much to report so far. I have been studying and configuring the basic template for my scraping projects, messing around with selenium to scrape google results, and after gather the results perform other actions.
I'm coding a broken link detector, or expired domain finder, it's almost the same. Then I will code a image scraper from some sites. Let's see what happen.

 

[n3s7or]

Update:

I tested the expired domain finder today, it still follow in diapers but it works. I ran it for about a minute and guess what??? It found TWO expired domains!!!!
I know it sounds pretty basic, maybe easy, and it is, but when you start building something from the ground you feel excited even from "minor successes".

There is a long way to go but I feel it is the right one.

Next steps will be related to being careful from the big G, he hates being scraped.
Ideas about it would be appreciated.

 

[JayDee1984]

What are your intentions with the scrapped data?

 

[bkenn]

Nice post . I happened to have played around with scrapy once it was interesting , but ended up reverting to using selenium. Cant remember why. Good Luck .

 

[n3s7or]

What are your intentions with the scrapped data?

I accepted two giveaway jobs, so I am working on it and learning at the same time.

Nice post . I happened to have played around with scrapy once it was interesting , but ended up reverting to using selenium. Cant remember why. Good Luck .

I am playing with selenium until I can start developing using splash.

 

[n3s7or]

Hey buddy,
I'm glad to see someone trying that which I was trying from days.

I was trying to scrape email contacts using scrapy.

But, I got a lot of unwanted results.

I'm facing issues in manipulating the regex expression / re.findall() function.
Also, I don't know how I can apply to reject method to avoid unnecessary outcomes.

Let me know if you wanna try it.

@Ash1

PM me the site you tried to scrape, perhaps I can gather what you want or at least I would learn something.

 

[Ash1]

It's not about the particular site.

You scrape websites and then you crawl them to pull out email addresses.

Now all the trick lies in the regex function you use to validate email ids & how you manage to filter the trash that comes with it.

 

[n3s7or]

UPDATE:
I've been busy these days plus I suffered from third molars extraction on the right side, the good point was that it didn't hurt, too much.

Anyway, I spent lot of time figuring out why my spider crawled just one keyword from a long list of keywords. Finally I found out it was due to a tiny bug on my code when iterating over the file containing the keywords.

Side note: When iterating over a file, the iterator blocks the tell() function, which return the current stream position. (in Python)

The hurdles from now on remains on

(...)being careful from the big G, he hates being scraped.

The steps are clear on how to avoid getting blacklisted, for example in my case I need to:

• Set a proxy rotator routine
• Set a user-agent rotator routine

I also need a ban checking routine, so I can detect when google redirects me to a captcha.

When I was writing this post I found a service provider that allows to use his service for free up to 1000 API calls, I am thinking to create a branch on my current project to use their API to avoid deal with selenium. Let's see what happen.

 

[SirTop]

Looks like a great read. OP would you be generous enough to share these tools on the download section after been successfully been created ? You def growing more audience using this approach.

I will be glad to test out some of your compiled .exe tools.

 

[n3s7or]

UPDATE
It have been few days since last update, well, I have a job and this is a hobby

I decided to split the scrapper in two:

1. Google result scraper, and
2. Link checker

Why? Modularization

As the name says:

The first one yields a list of scrapped domain from google, performing all the required steps to crawl and scrape google results (proxies and js rendering).

The second one filters for duplication and checks if expired or not​

To do list:

• Sequentially run the scripts (piece of cake, not even need to mention it here)
• Optimize google search for better results or broad results
• Look for a better way to check if domain is expired (the one I am running in my opinion is not the best - if someone could show me the light)
• Gather data from archive.org about the domain
  
• Look where and how can I (automatically) gather SEO data about every domain (study case)
• Find out if there is a way I can run Selenium + Chrome (chromedriver) on a VPS, and if there is start to think in a deploy test
• Modularize even more the first script. It means split my current custom middleware for Selenium and proxies in two middlewares one for each job. That will provide code re-utilization for further projects

I will focus to work on the previous list. If you have any idea or hint to help me, please let me know.

| Second Test |
|
| Yesterday I tested the project using about ten keywords. It crawled almost a thousand links and after checking for expiracy it found out 6 expired domain for the given Web 2.0 platform domain. Then I looked for a free tool to check domain authority and three of them showed the following results:

• DA 18 - PA 31 *
  
• DA 9 - PA 16 *
  
• DA 26 - PA 58

I am not sure how good is it for SEO, for the others links: PA and DA were almost cero. The starred ones are in the same niche.

That's all for today