Weebly Support

When user-agent and allow or disallow are together, it’s regarded as a solitary entry in a file where in fact the action will simply be employed to the user agent that is specified. If you wish to direct this to numerous user-agents, list an asterisk (*).

You will then must make sure that your particular robots.txt file is conserved. Be sure that you perform some following in order for internet crawlers should be able to find and determine your file.

  • Save the robots.txt file as a text file.
  • Position the file inside the highest-level directory associated with site ( or in the main regarding the domain).
  • The file needs to be called robots.txt.
    • Example: a robots.txt file that is conserved during the reason behind test.com utilizing the Address of http://www.sample.com/robots.txt is discoverable by internet crawlers, if the robots.txt file is situated at a URL like http://www.sample.com/not_root/robots.txt it’ll perhaps not have the ability to be located by internet crawlers.
  • www.weeblywebsitebuilder.com

There is certainly a tool that is testing for robots.txt, and it surely will explain to you in the event that file is effectively blocking Bing’s internet crawlers from accessing particular links on your internet site. Read More