A/B testing manual.

From setup to decision: create the experiment, split variants and traffic, estimate the runtime, read the results correctly and publish the winner safely.

System requirements

Shopware 6.7 with Storefront and shopping experiences, PHP 8.2 or newer. Meaningful results need enough visitors - the runtime estimate tells you how many upfront.

Installation

Upload the ZIP in the admin under Extensions, install and activate it. Then enter the licence token; it is bound to your shop domain.

Creating an experiment

Under A/B experiments you create a new experiment: name, description, page type and the original shopping experience (A). Then add the variants - each with its own layout and traffic share.

Rules: at least one variant is required, every variant needs its own layout, and none may be the same as the original. The sum of shares must not exceed one hundred percent.

Audience and measurement

An experiment can be limited to one sales channel; without a selection it applies to all. Bots and crawlers should stay excluded, otherwise search engines count as visitors.

The confidence level defines from which certainty a result counts as significant. On top you can require a minimum number of impressions per variant.

Estimating the runtime

Enter the expected conversion rate of the control group and the smallest difference you want to detect. From that the plugin estimates the sample size needed per variant.

Both values are optional and only used for the estimate - they do not influence the measurement. But they answer the decisive question: is my traffic enough for this test?

Start, pause, complete

On start, visitors are randomly assigned to a variant and stay with it. A running experiment can be paused and resumed later.

Completing determines the winner. Publishing happens on your command - or automatically if you enabled that setting.

Reading the results

Reported per variant are conversion rate, revenue per visitor and the path through the funnel.

Patience: as long as the minimum sample is not reached, a lead means little. Early swings almost always even out.

Publishing the winner

Publishing overwrites the original page with the winning variant. A rollback restores the original, so the decision is reversible.

Cleaning up

Variants create copies of the shopping experience, marked in the name. After the test the cleanup function removes all clones that no longer belong to an experiment.

Troubleshooting

No data: the experiment has to be started and the tested page needs visitors. Nothing is collected before the start.
Result keeps flipping: a sign of small numbers. Raise the minimum sample and let the test run longer.
Licence not recognised: the token is bound to the shop domain. Without a domain in the sales channel the check fails.

Support

Questions come straight to us - support is from the developer. You can also request a feature through the Plugin Suite in the admin.

Availability

Directly from Stone & Water

Available now

Monthly19.99 €
Half-yearly99.99 €17 % cheaper
Best priceYearly179.99 €25 % cheaper

All prices net, plus VAT.

You buy the licence directly from us and receive an invoice from us. Installation by upload in the admin without SSH, updates through our licence platform, support straight from the developer.

Get in touch +49 2555 9997342

Shopware Store

In preparation

The plugin will be listed in the official Shopware Community Store. Installation and automatic updates then run through the plugin manager in the backend, billing through your Shopware account. We will link it here as soon as it is available.