1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robot file !!! It is urgent !!!!!!!!

Discussion in 'Clickbank' started by Mehsal, Jan 1, 2017.

  1. Mehsal

    Mehsal Newbie

    Joined:
    Dec 11, 2016
    Messages:
    37
    Likes Received:
    0
    Hello everybody , I have a problem concerning the robot file , actually this robot file is blocking my urls and it prevents the appearance of our page in Google research results !!! and when i try to unlock it it displayed that the URL is in redirection state !!!! What should i do To unlock this robot file ???
     
  2. Heisenberg

    Heisenberg Jr. VIP Jr. VIP

    Joined:
    Sep 11, 2014
    Messages:
    718
    Likes Received:
    373
    Occupation:
    Freelancer
    Location:
    Croatia
    Make a blank text file with notepad and call it robots, upload it to your website folder. Or just edit current one, if you posted contents of your robots.txt we could give you more info.
     
  3. Mehsal

    Mehsal Newbie

    Joined:
    Dec 11, 2016
    Messages:
    37
    Likes Received:
    0
    What is the next step after uploading the blank text file ??
     
  4. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,492
    Likes Received:
    11,190
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    A blank robots.txt file is not recommended. Just delete it if you want it blank. Better return 404 than a blank robots.txt
     
  5. Mehsal

    Mehsal Newbie

    Joined:
    Dec 11, 2016
    Messages:
    37
    Likes Received:
    0
    How i will delete it ? ?
     
  6. SunnyLeon

    SunnyLeon Jr. VIP Jr. VIP

    Joined:
    Oct 13, 2013
    Messages:
    518
    Likes Received:
    178
    Gender:
    Male
    Location:
    Outside the Matrix
    If you don`t want to upload it blank, just google search for a basic robots.txt .. most of mine have no more than 7 rows and you won't need more.
     
  7. SunnyLeon

    SunnyLeon Jr. VIP Jr. VIP

    Joined:
    Oct 13, 2013
    Messages:
    518
    Likes Received:
    178
    Gender:
    Male
    Location:
    Outside the Matrix
    Do you have cpanel or ftp access to your webhosting account ? because that`s where you find the robots.txt file, inside your website's files
     
  8. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,492
    Likes Received:
    11,190
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    Do you have cpanel or FTP? In either one it's just a click.
     
  9. Mehsal

    Mehsal Newbie

    Joined:
    Dec 11, 2016
    Messages:
    37
    Likes Received:
    0
    I didn't understand exactly the main idea !!!! Can you clarify more please ??
     
  10. SunnyLeon

    SunnyLeon Jr. VIP Jr. VIP

    Joined:
    Oct 13, 2013
    Messages:
    518
    Likes Received:
    178
    Gender:
    Male
    Location:
    Outside the Matrix
    Yes, your website is hosted in a hosting account with a hosting company. What type of hosting do you have? How did you build your website? where did you forward the domain ? You have a hosting account and if that hosting account has access the the website's files in either cpanel or ftp, you can modify, add and delete files from there.
     
  11. naskootbg

    naskootbg Power Member

    Joined:
    Nov 8, 2010
    Messages:
    652
    Likes Received:
    221
    Home Page:
    robots.txt tells search engines robots what to do with your website.
    Strange issue. Very strange!

    When you try to open it, it redirects correct?
    Check the redirects in the host panel.
    If have not redirection in the cPanel, than they are in .httaccess file . If second the problem is bit complex.

    To fix robots.txt :
    1. Login your cPanel and go to File Manager or use FTP to access domain's root directory
    2. Find robots.txt and edit
    3. Search g00gle "robots txt generator" and generate code for your new robots.txt file
     
    • Thanks Thanks x 1
  12. Mehsal

    Mehsal Newbie

    Joined:
    Dec 11, 2016
    Messages:
    37
    Likes Received:
    0
    I deleted the robot file but when I launched the exploration of Google it showed me that the urls are in redirection! !! Please explain me what is this redirection and why they are in the redirection state ??
     
  13. Mehsal

    Mehsal Newbie

    Joined:
    Dec 11, 2016
    Messages:
    37
    Likes Received:
    0
    I deleted the robot file but when I launched the exploration of Google it showed me that the urls are in redirection! !! Please explain to me what is this redirection and why they are in the redirection state ??
     
  14. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,492
    Likes Received:
    11,190
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    Try ending your urls in / instead of a word and see if they still redirect.

    For example /blabla/page-1

    try /blabla/page-1/
     
  15. Mehsal

    Mehsal Newbie

    Joined:
    Dec 11, 2016
    Messages:
    37
    Likes Received:
    0
    Please tell me , can I send the backlinks of the website even if google didn't crawl my urls ??
     
  16. naskootbg

    naskootbg Power Member

    Joined:
    Nov 8, 2010
    Messages:
    652
    Likes Received:
    221
    Home Page:
    Can't get this. You want to send backlinks and hide them from g00gle?
    robots.txt is only for the domain - non related to its backlinks. You can restrict bots like ahrefs and majesticseo from robots.txt so they will never crawl the backlinks to your pages (most likely).
     
  17. Mehsal

    Mehsal Newbie

    Joined:
    Dec 11, 2016
    Messages:
    37
    Likes Received:
    0
    But me I want to send the backlinks and I want to know if it is possible to do this action even if google didn't crawl my urls yet ? ??? ! !! ( and I am waiting for the crawling because I have already send the indexation request in order to allow To Google to crawl my website )
     
  18. RayBay

    RayBay Junior Member

    Joined:
    Sep 4, 2013
    Messages:
    106
    Likes Received:
    16
    Location:
    online
    Test your site. Add in browser domain.com/robots.txt (replace domain.com with your domain name) and see if you actually have a robots.txt file. If its there and you don't like it - simply delete it via your cpanel.
    For those who do not plan block search engines this file is of no use. Crawlers should be blocked via .htaccess.
     
  19. Mehsal

    Mehsal Newbie

    Joined:
    Dec 11, 2016
    Messages:
    37
    Likes Received:
    0
    I did all what you have said but what I want to know is , can I send the backlinks even if crawling my website still pending ? ?
     
  20. RayBay

    RayBay Junior Member

    Joined:
    Sep 4, 2013
    Messages:
    106
    Likes Received:
    16
    Location:
    online
    Not sure if I understand. Do you mean that your site is not indexed and you want to start backlinking? If thats the case, than start linking as it will help to index your site faster.

    Google - imtalk website submitter. Add your url and it will post it to 100 - 1000 sites. Not much of a link value, but will help you index it faster.
     
    • Thanks Thanks x 1