[JOURNEY] Web scraping with scrapy

hi, where you from ? sorry for offtopic

UPDATE
So, this week....I managed myself to run my scraper for longer time, I bulk tested the script for about one and a half hour. It crawled more than 4000 pages and scraped 36 expired domains for almost 100 keywords (Note: My internet connection is slow and kind of expensive for the average people).

I just checked with free tools (the first I could get from google results) the quality of those links, nothing too interesting.

I found one with:
DR 55
Backlinks 3570
Referring domains 330

The rest were way below that. In my opinion there are lots of people running crawlers looking for high quality expired web 2.0 with better resources than I do. To be honest I've learned a lot but I am loosing interest.

I like scraping, but I can't find many usecases for it, other than selling it as a service - which I sometimes do. Seems like you have the same problem.

Good luck in your journey

n3s7or said:

If you have in mind something you would like to scrape please let me know.

Click to expand...

Hey mate, let’s get in touch, I have something in mind.

Well, this week wasn't as productive as it could be. I was busy with my job, but I created a small spider to crawl advertiser information from an e-commerce site, as a giveaway. The next steps for this spider depends on how interested that person is to go further. Whatever happen it already feels like a win because it was a successful project.

coldshowers said:

I like scraping, but I can't find many usecases for it, other than selling it as a service - which I sometimes do. Seems like you have the same problem.

Click to expand...

The deal with scraping is to gather data, behind that in my opinion there are two exciting parts about it:

• What to do with the data, and
• How to manage yourself to create the scraper avoiding being banned from the given site and handling JS

Thanks for your comment.

SEO Maker said:

Good luck in your journey

Click to expand...

Thanks man!

UPDATE: (just studying)

It have been two weeks since last update but I haven't quit the scraping journey. It happens I am a busy guy with a regular job and in my free time I do several things, study, work, gf, family, etc, as everybody does.

I made a scraping task, for my brother so no payment, he needed to gather a contest result data from a table, easy task.

Other project I have in mind is to create an API or an app for a communication service here in my country.

I've been wondering, I could scrape "easily" data from local e-commerces, what could I do with the scraped data?

I have found out how to scrape from pages with lazy loading and using javascript for data loading. I have yet explore the login portion for member content.

Good luck mate i love to know a bout scrapping a movies site to another

Good luck on your journey!
The last time I was trying to scrape google I got blacklisted very quickly (despite using proxies), so I just ended up using a third-party api that returns results from the serps (serpwow).

Good luck on your journey. Do you use a VPS or your local machine?

Good luck with your journey, i love scrapping too, but i only do with php, interesting to learn python in the future

Loving this journey. I scrape on daily basis and have a few VPSs and proxies ready to go in case you need to test some of your scrapers, I could also provide feeback or report bugs.

In the good old days, I loved making websites via scrapping other info for fun.

unfortunately, duplication came to town.

php , mysql , ajax , html

ACMillions said:

I have found out how to scrape from pages with lazy loading and using javascript for data loading. I have yet explore the login portion for member content.

Click to expand...

Yeah, JavaScript is often a pain in the ass. The login step is not easy either, Scrapy uses a LoginForm spider that emulates a user sending auth data, I haven't really test it yet but I will when I begin to code and app I have in mind.

litotoyou said:

Good luck mate i love to know a bout scrapping a movies site to another

Click to expand...

Thanks, so far I have only scraped text data, but it is not so much different than files, though movies are really large files for my current bandwidth.

Stop said:

Good luck on your journey!
The last time I was trying to scrape google I got blacklisted very quickly (despite using proxies), so I just ended up using a third-party api that returns results from the serps (serpwow).

Click to expand...

Thanks, Google doesn't like being scraped, to be blacklisted is just for a moment I guess, it shows captchas, right?
Whenever I need to request multiple times I use the free plan from a third party api, 1000 api calls, you probably know which one it is.

ZeroDefects said:

Good luck on your journey. Do you use a VPS or your local machine?

Click to expand...

Thanks, most of the time I use my pc from work, and rarely my own.

reddflix said:

Good luck with your journey, i love scrapping too, but i only do with php, interesting to learn python in the future

Click to expand...

Thanks, I suggest you to learn Python, it is very easy to learn and it's
a powerful language. If you already know a programming language, you already know Python.

mylastvacation said:

Loving this journey. I scrape on daily basis and have a few VPSs and proxies ready to go in case you need to test some of your scrapers, I could also provide feeback or report bugs.

Click to expand...

Thanks for your support, man!

So far my biggest problem is what to do with the knowledge. I am not an expert but I can handle my self and do some stuff.
For example:
I could scrape data from local e-commerces like I said before I'm my previous post, but then what!? Perhap someone experienced like you could help me with some guidelines or tips.

Anyways, thanks.

redarrow said:

In the good old days, I loved making websites via scrapping other info for fun.

unfortunately, duplication came to town.

php , mysql , ajax , html

Click to expand...

Thanks for your comment. I have thought about that, not the duplication itself, but to gather data from other sources and compile it to a blog.

↑
Loving this journey. I scrape on daily basis and have a few VPSs and proxies ready to go in case you need to test some of your scrapers, I could also provide feeback or report bugs.
Thanks for your support, man!

So far my biggest problem is what to do with the knowledge. I am not an expert but I can handle my self and do some stuff.
For example:
I could scrape data from local e-commerces like I said before I'm my previous post, but then what!? Perhap someone experienced like you could help me with some guidelines or tips.

Anyways, thanks.


Of course I could help. I have no technical experience to build scrapers but I have bought 4 scrappers before and got some success doing lead generation. Collecting leads for small businesses, and I've gotten between 1 cent to 10 cents per lead sometimes, so it's a good business.

When you finish building your scraper, you can let me know what kind of things you can scrape and I'll try to find some clients for them. You know there's a market for everything.

Hi all, it´ve been a long time since my last update. I didn´t abandoned the thread, what happened? Well, when I was browsing internet to find learning material I discovered some hiring companies and apply to some of them in order to change my current job for a better paid one.

So, one day after, a few weeks, one of them contacted me and made me go through interviews and a test, and here I am starting from zero in a new job. So far I still being "the new hire", tomorrow ..... who knows LOL.

Well anyways now I will close this thread, perhaps in a future I will open a new one.

Lets see how this goes. Good luck to you!