surefoot saves 5 hours a day using Equals

“Moving from R to Equals has truly made us more efficient. We’re easily saving 5 hours a day. Getting the equivalent of half a person's time back each week is huge.” – Case White, Director of Client Services & Implementation

Discovering the opportunity

surefoot helps clients improve their digital customer experience to catalyze growth. Companies like Hydro Flask, Brooklinen and Solawave rely on their services, which include data analysis based on a mix of user research, experience design, and A/B testing. Since most of their work requires clear, metrics-driven deliverables, having a seamless way of extracting this data is critical to their operations.

In the early days, the team at surefoot was using the Google Analytics interface to monitor A/B tests. They quickly ran into the limitations on this platform: often taking 30 minutes to check the performance of a test and download a report. Unsatisfied, the team moved over to R as a way of running inflight monitoring reports. This still required an analyst to manually run a script every time results were checked for all live tests –which would take about 10 minutes per live test. That is significant considering they have 30-50 tests live at any given time. Case said, “Everyone felt the pain of the old process when the analyst running the report, who had proficiency in R, left surefoot. Maintaining those scripts was a struggle. Switching to Equals and being able to use JavaScript has been a huge help for us and has democratized the results-gathering, since many of us are familiar with JavaScript.”

Implementing changes

surefoot has automated the daily monitoring of live A/B tests using Equals. For each test variant, they track various metrics in Google Analytics and calculate the lift in performance relative to the baseline site experience (and its statistical significance). Having everything updated automatically has allowed the team to track their data on a daily basis, instead of their prior cadence of a few times per week. “When we were using R we had to maintain R projects in R Studio. This required not only a knowledge of the R language, but also the R ecosystem. In Equals, anyone that knows JavaScript can update our reports and doesn't need to fool with project setup or environment maintenance.”

Favorite feature: “I love the core feature of being able to write JavaScript directly in the import script interface. No longer having to rely on an external environment and setting things to run on a scheduled basis has allowed us to monitor our metrics daily.”

Unlocking growth

Case mentions that Equals has been a noticeable time-saver for the company. “With our prior automation, there were still some manual steps in pulling an in-flight monitoring report, and it would take us about 10 minutes per live test per day. With Equals, we spend about 10 minutes setting up a test initially (i.e. only one time!), then it runs automatically each day. So we're saving at least 10 minutes per live test each day. This translates to almost 25 hours saved per week.” Talk about getting off on the right foot!