Shelf Test Methodology
The Shelf Test is designed to study purchasing preferences through a realistic shopping experience.
Shelf Metrics visualization:
- Shelf Heatmap displays the density of clicks using 2d kernel density estimate. There are 9 layers—the bottom layer represents an area where we expect to find 99% of the clicks, and the top layer 50%. Note: a click in this context is a literal click on the location on the screen, whether to select a product for purchase, or just to view its details. Product location randomization isn’t considered—the heatmap illustrates coordinates of the clicks on whatever product a particular respondent happened to see on that position.
- Product Heatmap visualizes total product quantities with a color scale ranging from the minimum (green) to the maximum (red) quantity.
- Clicks view shows individual clicks made by respondent. A click in this context is a literal click on the location on the screen, whether to select a product for purchase, or just to view its details. Product location randomization isn’t considered - the heatmap illustrates coordinates of the clicks on whatever product a particular respondent happened to see on that position.
- Shelf view has no additional overlays so that the original shelf setup can be viewed without obstructions. If product randomization was involved, displayed image is one of the possible realizations of the algorithm, and individual experience of a respondent may have varied.
Total summary statistics:
- Average cart spend is the geometric mean of respondent-level sum of money spent.
Base: respondents who purchased at least one product. - Average cart size is the simple mean of the respondent-level quantity of purchased products.
Base: respondents who purchased at least one product. - Average number of unique products in cart is the simple mean of the respondent-level number of unique purchased product.
Base: respondents who purchased at least one product. - Average time at shelf is the geometric mean of time spent on the question.
Base: respondents who saw the question. - Average attention span per product is the simple mean across product-level Attention Span aggregates. NA option isn’t considered.
- Total products purchased is the sum of all products purchased by all respondents throughout the survey duration. NA option isn’t considered.
- Total product spend is the sum of money spent on products purchased by all respondents. NA option isn’t considered.
Product metrics:
- Purchase selection represents proportion of respondents who purchased the product.
Base: respondents who saw the question. - Purchase quantity is the simple mean of the quantities of that product purchased by respondents who decided to purchase that product.
Base: respondents who purchased at least one unit of that product. - Purchase amount is the geometric mean of the amount of money spent of that product by respondents who decided to purchase that product.
Base: respondents who purchased at least one unit of that product. - Attention span is the geometric mean of the amount of time spent viewing that product's detail view.
Base: respondents who purchased at least one unit of that product. - Time to cart is the geometric mean of the amount of time between starting the task and selecting that product.
Base: respondents who purchased at least one unit of that product.
Geometric Mean
When analyzing the results of a Shelf Test, the geometric mean is often used as a way to account for outliers in the data. The geometric mean is most frequently used in economics and finance but can also be used when dealing with values that don't have a finite range, such as money. Unlike the standard arithmetic mean, the geometric mean does a better job at trying to account for and not be as influenced by extreme outliers.
In the case of the Shelf Test - because there could be so much behavioral variation where some respondents might fill up their carts - rather than removing them as outliers altogether, the geometric means acknowledges this could be real life behavior but does not let it skew the results.
For example, in the shelf test, average cart spend is calculated using geometric mean because it is a more robust metric that avoids the extreme influence of outliers.