tool that can scrape any site from internal links

omartaheri2005

Newbie

Dec 17, 2020

5

0

• Jul 3, 2022

• #1

I want to get all udemy courses links for my small project.
The number of courses is more than 250k, and I have not found any tool that can do this task
Is there any tool that can scrape all udemy internal links? I hope you can help me

 

PHPInjected

Super Moderator

Moderator

Executive VIP

Jr. VIP

Apr 25, 2014

6,911

12,273

• Jul 3, 2022

• #2

If Scrapebox cannot do this, you could probably just pay a programmer to write you a Python/iMacros script to do this for you.


Python is perfect for this.

 

omartaheri2005

Newbie

Dec 17, 2020

5

0

• Jul 3, 2022

• #3

PHPInjected said:

If Scrapebox cannot do this, you could probably just pay a programmer to write you a Python/iMacros script to do this for you.


Python is perfect for this.

Click to expand...

I'm a Python programmer, but I'm looking for an easier solution. it's difficult to impossible because a lot of bot protection in udemy.

 

terrycody

Elite Member

Sep 29, 2012

6,102

3,165

• Jul 3, 2022

• #4

Maybe try some old tools like Xenu or Screaming Frog?

 

BlackShield

Elite Member

Jr. VIP

Jun 25, 2015

3,397

767

• Jul 4, 2022

• #5

+1 for Xenu or Screaming Frog tools.

 

omartaheri2005

Newbie

Dec 17, 2020

5

0

• Jul 5, 2022

• #6

BlackShield said:

+1 for Xenu or Screaming Frog tools.

Click to expand...

I test Screaming Frog today but it doesn't work

 

BlogPro

Elite Member

Jr. Executive VIP

Jr. VIP

Apr 23, 2012

2,577

5,402

• Jul 5, 2022

• #7

omartaheri2005 said:

I'm a Python programmer, but I'm looking for an easier solution. it's difficult to impossible because a lot of bot protection in udemy.

Click to expand...

There is nothing easier. If you need to scrape, you need to figure out a way to evade bot protection.

Use proxies, use selenium-stealth, run a multi-threaded approach.

What all data are you looking to scrape?

 

I

Ithetona

Compromised Account

Nov 30, 2021

80

16

• Jul 5, 2022

• #8

omartaheri2005 said:

I test Screaming Frog today but it doesn't work

Click to expand...

I have used Scrapebox and Phantombuster. I recommend these tools.

 

G

graham25s

Power Member

May 8, 2010

755

186

• Jul 5, 2022

• #9

Could only get these using my own tool, 250k courses? that seems a lot.

 

Attachments

• LinksInternal.txt

  643.7 KB · Views: 8

T

Thecodder

Junior Member

Aug 14, 2015

143

91

• Jul 5, 2022

• #10

I scraped udemy 2 years ago for free courses, with nothing but a simple php scrapper wrote by me, is very easy!
They use a public api between backend and frontend and you can get everything you want from that api(course, ratings, instructor s,price, total duration, etc)
If you know a bit of php i can give you my old code if you'll be lucky and i will find it )