What Is the Robots.txt File for My Zibster Website?
Overview
Every launched Zibster website automatically includes a robots.txt file. This file gives search engines and other automated crawlers instructions about which areas of your website they may access.
Zibster creates, hosts, and manages your robots.txt file automatically. You do not need to configure it, and you cannot edit, replace, or remove it from your Zibster account.
This guide explains how robots.txt works on Zibster websites, what the included instructions mean, why some system URLs are intentionally restricted, and what to check if an SEO tool reports a blocked URL.
What Is a Robots.txt File?
A robots.txt file is a plain-text file located at the top level of a website. Search engines and other automated services may review this file before crawling your website.
The file contains instructions for different crawlers, which may also be called bots or user agents. These instructions can:
- Allow crawlers to access specific website resources
- Restrict crawlers from selected URL paths
- Apply different instructions to specific crawlers
- Identify the location of your XML sitemap
- Help manage automated crawler traffic
Robots.txt primarily controls crawling. It does not control your website navigation, change your page content, or provide security for private information.
How Robots.txt Works on Zibster Websites
Zibster manages robots.txt at the platform level. Launched Zibster websites use a standardized configuration designed to allow search engines to access normal public website content while restricting selected administrative and system-generated areas.
Your robots.txt file may include instructions that:
- Allow access to resources needed to display your website
- Allow supported search engine and social media crawlers
- Restrict selected automated bots
- Restrict administrative and system-generated paths
- Restrict selected cart, checkout, billing, or account paths
- Identify your XML sitemap
- Provide a crawl-delay instruction for crawlers that support it
These restrictions help crawlers focus on public website content instead of internal Zibster system areas that are not intended to appear in search results.
You do not need to activate or maintain your robots.txt file.
How to View Your Robots.txt File
You can view the robots.txt file for your launched website by adding /robots.txt to the end of your domain name.
For example:
https://www.yourdomain.com/robots.txt
Replace yourdomain.com with your website's domain.
The file will open as a plain-text page in your browser.
Your website must be launched on a domain before its domain-specific robots.txt file is available.
What Do the Robots.txt Instructions Mean?
User-agent
A User-agent identifies the crawler that should follow a group of instructions.
Some instructions apply to a specific crawler. The asterisk in:
User-agent: *
represents crawlers that do not have a more specific set of instructions.
Allow
An Allow instruction identifies a resource or URL path that a crawler may access.
Zibster may use Allow instructions for website resources, such as CSS and JavaScript files, that supported crawlers need when processing and displaying your website.
Disallow
A Disallow instruction identifies a URL path that the specified crawler should not access.
Zibster uses these instructions to restrict selected bots and system-related areas of the platform.
A disallowed system path does not mean your entire website is blocked from search engines.
A blank Disallow instruction means that no paths are restricted for that crawler.
Sitemap
The Sitemap instruction identifies the location of your website's XML sitemap.
You can also view your sitemap by adding /sitemap.xml to your domain:
https://www.yourdomain.com/sitemap.xml
Zibster automatically creates and maintains your sitemap as you update your website.
For more information, see How to Find Your XML Sitemap in Zibster.
Crawl-delay
The Zibster robots.txt file includes:
Crawl-delay: 60
This asks crawlers that support the instruction to wait between requests to your website.
Google does not support the crawl-delay instruction and ignores it. Google determines its own crawling frequency, so this setting does not prevent Google from crawling your public website or directly affect your Google search rankings.
Other crawlers may recognize and follow the crawl-delay instruction.
Zibster System Paths vs. Public Website Pages
When reviewing a Disallow rule, pay attention to the complete URL path, including trailing slashes.
A Zibster system directory can use a word that is also used as the URL for a normal Website Builder page. That does not necessarily mean the public page is blocked.
For example, if a robots.txt rule contains:
Disallow: /portfolio/
the rule applies to URLs inside that directory, such as:
https://www.yourdomain.com/portfolio/client-name
It does not apply to the standalone URL:
https://www.yourdomain.com/portfolio
The trailing slash is important.
This means a restricted Zibster system directory does not automatically prevent a similarly named public Website Builder page from being crawled.
If an SEO tool reports a blocked URL, always compare the complete URL in the report with the complete path shown in robots.txt.
You can learn more about how robots.txt rules are interpreted in Google's official Robots.txt Crawling Infrastructure Guide.
Robots.txt and Search Engine Indexing
Crawling and indexing are related, but they are not the same thing.
Crawling occurs when a search engine visits and reviews a URL.
Indexing occurs when a search engine evaluates a URL for possible inclusion in search results.
Robots.txt primarily controls whether compliant crawlers can access a URL. It does not guarantee that a restricted URL will never appear in search results. Search engines may discover URLs through links, previously collected information, or other sources.
If you have a normal public Website Builder page that you do not want included in search results, use a noindex tag instead.
See How to Add a No Index Tag to a Page in Zibster.
For content that should not be publicly accessible, use the appropriate privacy settings, password protection, disable the page, delete the page, or remove the content from your public website. Robots.txt should not be used as a security or privacy feature.
Can I Edit My Zibster Robots.txt File?
No. Zibster manages robots.txt at the platform level.
You cannot:
- Edit the existing robots.txt file
- Upload a replacement robots.txt file
- Remove the file
- Add instructions for a specific crawler
- Add custom rules for an individual page or URL path
Zibster automatically maintains the file as part of your hosted website.
Troubleshooting and FAQs
Does the Zibster robots.txt file block Google?
No. The Zibster robots.txt file does not block Googlebot from your normal public website content.
Selected system paths may be restricted, but those restrictions do not mean your entire website is blocked.
Why does an SEO audit report blocked URLs?
SEO auditing tools may discover administrative, cart, checkout, billing, account, or other Zibster system URLs that are intentionally restricted from crawling.
A blocked system URL generally does not indicate a problem with your public website or its search visibility.
Check the complete URL reported by the auditing tool. If the report identifies a normal public Website Builder page that you expect search engines to crawl, contact Zibster Support and provide the complete URL so our team can review it.
Why does robots.txt show a path that looks like one of my website pages?
Zibster system paths may use words that are also commonly used for public website pages.
The complete path matters. For example, /portfolio/ represents a directory and is different from the standalone /portfolio URL.
A restriction on a system directory does not automatically restrict a similarly named public Website Builder page.
Does the 60-second crawl delay hurt my SEO?
No. Google does not support the crawl-delay instruction and ignores it.
Other crawlers may follow the instruction, but it does not directly affect your Google rankings or prevent Google from crawling your public website.
Can I add a custom robots.txt rule for one page?
No. Custom robots.txt rules are not available for individual Zibster websites.
If you have a normal public Website Builder page that you do not want included in search results, use a noindex tag instead.
Can robots.txt protect private pages, Client Galleries, or images?
No. Robots.txt provides instructions to automated crawlers. It does not prevent someone from visiting a publicly accessible URL, and not every automated bot is required to follow robots.txt instructions.
Use the appropriate privacy or password-protection settings for content that should only be available to selected visitors.
Do I need to submit my robots.txt file to Google?
No. Search engines automatically check for a robots.txt file at the standard /robots.txt location on your domain.
You do not need to upload, submit, or maintain the file yourself.
Does robots.txt include my XML sitemap?
Yes. Your robots.txt file includes a Sitemap instruction that identifies the XML sitemap associated with your domain.
Zibster automatically creates and updates your sitemap as you make changes to your website.
Can removing a page from my sitemap prevent it from being indexed?
No. Removing a URL from your sitemap does not prevent search engines from discovering it through other sources.
If you have a normal public Website Builder page that should not appear in search results, use a noindex tag. If the page should no longer be publicly accessible, disable or delete the page.
Related Articles
Want Help With Your Website's SEO?
If you would like professional help improving your website's search visibility, Zibster offers SEO Go services. Our SEO team can help optimize your website and develop an ongoing strategy for improving your presence in search results.
Need Help?
Support Hours: Monday to Friday, 9am to 6pm EST
Phone: 844.353.3412
Support Channels: Phone, Live Chat, Support Ticket
