Screaming frog is not crawling my website - Need Help

arijitseth53

Newbie

Oct 6, 2021

19

6

• Oct 6, 2021

• #1

Hi any screaming frog expert is there? My website is now fully crawling. Can any body give some idea?
Website is skillsz dot com

 

rambhavlekar

Junior Member

Jr. VIP

Sep 27, 2021

119

15

• Oct 6, 2021

• #2

Check your Robot.txt

 

arijitseth53

Newbie

Oct 6, 2021

19

6

• Oct 6, 2021
• Thread Starter Thread Starter

• #3

User-agent: *
allow: /
disallow: /dashboard

sitemap: https://skillsz.com/sitemap.xml this is ok i think

 

TZ2011

Senior Member

Jun 26, 2011

1,095

1,131

• Oct 6, 2021

• #4

Under Spider Configuration you have option to crawl specific sitemap, copy link there, enable it manually.

 

arijitseth53

Newbie

Oct 6, 2021

19

6

• Oct 7, 2021
• Thread Starter Thread Starter

• #5

Yes its working. You are genius.