Problem
This product was designed and built for an e-commerce company with several websites and dozens of product types. We found that users consistently wanted a quicker way to scan and compare product information than reading entire pages.
An ideal version would be flexible across website brands and the products, which span the fitness, wellness, nutrition and sleep industries to accommodate their varying product information and attributes. It also needed to integrate to the existing product information inventory system, which housed product details, specs, and custom review data--all relevant for customers' purchase journeys.
​
Crucial to its success was pinning down the right product information to include for each respective product.
UX & Business Objectives
We wanted to create a tool that was simple to use yet comprehensive in the product information it displayed to users. It was also important that it was optimized for mobile, which comprised ~65% of users.
​
From a business and revenue perspective, it was important to maintain click through rate (CTR) on one or more products and eventual conversion. We wanted to be careful not to discourage users from clicking by giving an overwhelming amount of information or specs that would easily disqualify them from purchasing.
​​
WHAT USERS SAID
This product idea arose from user feedback heard during generative UX studies where we showed one of the e-commerce sites to users in the market for that particular product.
Users complained about the length of pages and the density of the information; some expressed the desire for a table to compare products and glean all the product specs needed.
The Approach
Just before the exploratory user study on the landing page and overall UX, we had run an AB test putting a simple comparison table structure at the top of the page (replacing another element) and seen a 34% drop to CTR. But hearing the strong user feedback around a more comprehensive table kept us aimed at getting the design and interaction right instead of abandoning the project.
​
We tapped the Design team to make early, lightweight aesthetic tweaks, and wove in visual elements that properly focused user attention, which we then validated with user studies and AB tests that saw high engagement on the new elements.
AB Testing & UX Research Cycle
AB Test #1
Placed a basic comparison table top of page, replacing a link list of the same products. This test was the idea of a separate team and occurred at the same time as the exploratory study detailed below.
​
Results: Loss - 34% drop to click-through rate (CTR)
Next step: Take a step back and do a more exploratory user study around what mattress attributes users are hoping to find on a product page and in a comparison table.
User Study #1
Exploratory user study to better understand trends in what product information consumers are looking for when shopping for a mattress. This was the first of many studies around this particular topic. Results were used to guide which product information should be included in the next iteration of the comparison table.
​
Results: Most users said firmness was an important attribute; a smaller percentage passionately advocated for cooling as very important to them. Users were adamant that a tool like this would help them in their search and help them to purchase.
Next steps: Add coolness and firmness attributes into table and retest.
AB Test #2
Added coolness and firmness columns to table; replaced price with dollar signs to avoid discouraging click through from sticker shock.
​
Results: Loss - 35% drop to CTR
Next steps: Hearing users so adamantly push for a comparison table, we strongly believed there were signs of life in this product. So instead of giving up, we spent some design resources sprucing it up aesthetically and took it back to users for an evaluative study.
User Study #2
Evaluative user study showing users two different comparison tables: our lightly redesigned table, and a competitor's much larger and flashier version. The goals were to confirm which product attributes should be present and which visual elements were engaged with and preferred.
​
Results: Users noticed and appreciated the design tweaks and felt that the table had all attributes they would need when shopping for a mattress. They did like that the competitor's version had images, brand icons, and fit more information in a single view. They also asked questions about deals on different models and thought that would be good information to include. Overall, users once again passionately advocated for the comparison table, voicing that this type of element is especially important in the mattress shopping journey.
Next steps: Weave in positively viewed elements from competitor's table, implement positively viewed design tweaks, expand attributes listed, add deal language. We also added back in the link list, the element that the table had replaced in AB tests #1 and #2 to check if this variable was causing some dip in performance.
AB Test #3
Added icons and CTA buttons, removed review link, added deal language, shaded every other row for better legibility.
​
Results: Loss - 4% drop to CTR
Next steps: Though a loss, this version showed two things: the link list that the table had previously replaced was an important asset to keep on page; and this was a promising page component. With 2 user studies now demonstrating that consumers expected and liked comparison tables, the decision was to run another AB test weaving in some of the other findings from User Study #2.
AB Test #4
Kept many of the changes from AB Test #3. Additionally: made all views visible in a single view, changed Sleeper Type to more subjective Superlative.
​
Results: Flat - 0.13% drop to CTR (no statistical significance)
Next steps: At this point we were doing concurrent work to overhaul the entire landing page template. So we rolled in this element to the new v1 template and the user study done on the larger redesign project.
User Study #3
We decided to pare down the number of product attributes displayed to focus user attention and reduce real estate used by this element. After hearing in other studies that users want to see the price of the mattress, we replaced the dollar sign indicators with actual starting price amount. Importantly, we used mattress imagery rather than brand logos (another piece of feedback heard from users). We also included star ratings - another element users had mentioned as helpful when shopping and comparing.
​
Results: Of the whole page, users consistently pointed to the comparison table as a favorite component and one that would be useful (even critical) for their shopping process. They liked the star ratings, price being listed, and how succinct and straightforward the table was.
Next steps: User Study #3 gave us confidence to keep the table pared down, so we moved forward with design. To determine which product information to include, we revisited User Study #2, which was the most focused on attributes users wanted to see. With a few other design tweaks, we opted for another AB test to validate the conclusions from the previous research and testing done on this tool.
AB Test #5
Used a smaller list of attributes based on user feedback: Star rating, firmness, material, cooling and sleep position. Made CTA color a bright green which had won in a separately run AB test. Made mattress brand/model clickable links to give more opportunity for users to click through.
​
A note on price: though we had heard from users that they wanted to see true price listed, we decided to remove this field entirely as we had also observed users in studies not clicking through once they had seen price. The business model of this page depends on healthy click through for proper attribution, so we chose to balance business needs with user expectations.
​
Results: Win - 5.1% lift to CTR
LEARNINGS AND FINAL NOTES
This was the first mixed methods product that I had managed on this team. It was a perfect example of weaving in user feedback insights with the behavioral results seen in split tests. Some of the guiding principles that kept informing this product and more general UX and product development after the last AB test in this sprint:
​
-
Users like to see product imagery, even if not perfect captures
-
Star ratings are a winner
-
For this brand, the green CTA became table stakes (nicknamed "money green")​​
-
These product attributes are a safe bet for top 4 across varying shopper types
​
From a build perspective, we wanted there to be flexibility for different product attributes to be tested in the future (including price), so we made sure that all product fields from our product database were possible to include for testing purposes.
​
This tool was consistently a huge revenue driver across the company, and was extended to other brands and 14 different product categories with minimal additional testing. Meaning the testing and research we did accounted for a lot of what we needed to know to build a great and flexible product.