Manual robots.txt Manager.

What the plugin does

From version 6.7.1 on Shopware delivers the robots.txt itself, with the same default rules for every domain. The plugin replaces this output with your own text per sales channel. If a single domain needs something different, it gets its own version.

If nothing is stored for a domain or the version is switched off, Shopware delivers its default as usual.

Getting started

1. Install and activate the plugin, then clear the cache.
2. Open the robots.txt Manager under Extensions, Stone & Water - Plugin Suite.
3. In the Overview tab, check whether a red notice is shown.
4. Click "Edit channel default" for a sales channel.
5. Adjust the suggested content or insert a template.
6. Save. The change is live right away, the cache is purged selectively.
7. Use "Live check" to confirm the domain delivers the new version.

Channel default and own version

The channel default applies to all domains of a sales channel. Select a single domain in the editor to create its own version.

A robots.txt always applies to a whole host. Domains with a path, such as shop.de/en, therefore share the robots.txt of shop.de.

Delivery order:

1. Active version of the domain
2. Active channel default
3. Shopware default

Editor settings

• Deliver custom robots.txt: Off means the domain falls back to the channel default or the Shopware default. The content stays stored.
• Add sitemap entries automatically: adds a sitemap line for every domain on the host. Lines already in the text are not repeated.
• Template: replaces the content with a template after confirmation.

Templates

• Shopware default - the rules Shopware 6.7 delivers itself, including access to theme files, media and thumbnails.
• Shopware default, block AI training - additionally blocks GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent and Bytespider. Search assistants stay allowed.
• Shopware default, Google unrestricted - Googlebot and Googlebot-Image may do anything, everyone else gets the default.
• Allow everything - no restrictions.
• Block everything - blocks the whole site, for staging environments for instance.

Preview and hints

The preview shows exactly the text that will be delivered, including the sitemap lines. Below it the plugin points at typical mistakes:

• Disallow: / for all crawlers blocks the whole site
• lines without a colon or unknown directives
• rules before the first user-agent line
• paths not starting with / or *
• sitemap without a full address
• noindex and crawl-delay, which Google ignores
• files over 500 KiB, Google reads no more

The hints do not block saving.

Live check

"Live check" fetches the domain's robots.txt from the server and compares it with the stored version. Possible results:

• Matches the stored version - everything is fine.
• Cache serves an older version - clear the cache or wait a moment.
• Different robots.txt - usually a file in the public folder or a web server rule.
• Shopware default - no own version is active for this domain.
• Not reachable - the server cannot reach itself, open the address in a browser.

From the editor the fetched content can be used as a starting point with "Copy into editor". Sitemap lines are removed if automatic completion is switched on.

Notices in the overview

• Physical robots.txt in the public folder - the web server serves this file directly, Shopware never gets a turn. "Disable file" renames it and keeps it as a backup.
• robots rule in the .htaccess - a redirect prevents delivery by Shopware. Remove the rule by hand.
• Staging mode active - Shopware then blocks all domains. Your own content takes effect only after switching it off.

Uninstalling

Uninstalling without "keep data" removes the table with all content. After that Shopware delivers its default again.

Support

We answer questions directly. Write to us via the contact form.