How Sky and RS Group built testing and experimentations programs that actually deliver

Testing and experimentation are cornerstones to building a great customer experience—but how do you make sure you’re testing the right things? We received some robust insights at a panel discussion hosted by Optimizely at a recent CX circle London event. 

The panel consisted of two testing and experimentation experts—Natasha Senior, Senior Digital Manager at Sky and Stewart Ehoff, Head of Experimentation at the leading provider of industrial and electronic solutions, RS Group.

During the session, the duo filled us in on how they get ideas for testing, what tools they use and the biggest item on their testing agenda for 2023. 

Here’s a round-up of Natasha and Stewart’s expert insights and best advice for anyone looking to take their testing game to the next level in the new year.  

How has experimentation evolved in the last couple years? 

Natasha from Sky has seen experimentation expand from a purely marketing-focused initiative to a cross-functional tactic. At Sky, they use experimentation across the business to reduce contact center calls, improve web accessibility, and even optimize their offline customer experience. 

“Experimentation is becoming the heart of everything we do. People think it’s just one department that owns it—like product, tech or marketing—but it’s not. It’s our whole business culture.” — Natasha Senior, Senior Digital Manager at Sky

And Stewart from RS Group agreed, highlighting how experimentation continues to grow as an organization-wide practice. 

“Over the last three years, experimentation has been more widely adopted, so there’s a lot more talent in the space,” said Stewart. “I hope that one day, experimentation will have the same level of importance as SEO—where you simply wouldn’t build products, services or solutions without it.”

Where do you get your test ideas from?

For Stewart, it’s important to involve as many key stakeholders as possible to build a strong and effective experimentation roadmap. That’s why RS Group is on a mission to expand experimentation ideation horizontally across the business. 

If people don’t understand the power of experimentation, it can limit the number of good testing ideas produced—because those ideas truly can come from any department. 

“There needs to be an outreach process in place, where you take your stakeholders through the value of experimentation and educate them on the possibilities,” said Stewart. “This will give you more ideas to test and learn from, which is great for experimentation!”

For Sky, the best experimentation ideas come from outside the experimentation team—specifically, from their contact center, which has a front-row seat for how the business’ tech affects its customers.

“We try to visit the contact center once or twice a year to sit with the advisors. They often give us great testing ideas because they’ve sat talking to the customer all day, every day,” said Natasha. “Their insights are invaluable.”

How have Optimizely products helped you build a culture of experimentation? 

Experimentation teams are great at creating (and—as we’ve seen—sourcing) ideas for experiments and understanding which of these are worth pursuing; but they tend not to have the deep tech expertise required to integrate these experiments into a business’ tech stack.  

Optimizely’s Feature Experimentation product has helped RS Group embed experimentation across the business and ensure experiments are built, not by their experimentation team, but by their engineers. Instead, Stewart’s experimentation team is set up to empower other teams to use their testing tools and methodologies to drive their own outcomes.

“You can’t build a culture of experimentation by simply sticking a web snippet on a page and having a few people run some tests. It has to be deeper than that. Testing has to be fundamentally ingrained into your tech stack.” — Stewart Ehoff, Head of Experimentation at RS Group

Most of RS Group’s experiments are delivered server-side, built by their engineers. 

The same goes for Natasha’s team at Sky, who have a long-standing partnership with Optimizely.

“Learning how to use Optimizely Full Stack is part of every developer’s training when they join us,” said Natasha. Just like RS Group, Sky’s experimentation team isn’t involved in building experiments, instead, it’s ingrained in the development team’s way of working. 

What tools are you using to increase your experimentation capabilities?  

Alongside Optimizely, Sky uses Contentsquare’s digital experience analytics platform to create problem statements and build hypotheses for experiments. 

“Your problem statement can’t go anywhere without data,” said Natasha. Contentsquare’s digital experience analytics cloud platform provides Sky with unique customer behavior insights to help build a pipeline of robust, data-driven experiments that the entire company can get behind. 

Contentsquare’s integration with Qualtrics, a Voice of Customer (VoC) feedback platform, enables Sky to validate errors flagged in customer feedback by rewatching affected sessions in Session Replay.

“A/B testing tells you whether a test has worked, but not why. Contentsquare and Qualtrics give us the ‘why’ we need to effectively iterate our testing. — Natasha Senior, Senior Digital Manager at Sky.

For RS Group, the process of improving experimentation practices is still in its early stages. “One of our biggest challenges is around how to increase our understanding of our customers and the problems they’re having to ensure our testing is rooted in strong data and hypotheses,” said Stewart.

Currently, RS Group uses VoC surveys, feedback from customer services and Adobe Analytics to obtain the quantitative data they need to understand their customers. They also work closely with their user experience research team, which is a “wealth of knowledge and information,” said Stewart. “That’s why bringing experimentation and user experience research together is so powerful.” 

What’s next on Sky and RS Group’s experimentation roadmap for 2023?

Sky plans to install Optimizely into their telephonic systems using Optimizely Agent. “We want to start doing A/B tests end-to-end,” said Natasha. “If someone lands on sky.com, uses the bot and then calls in, I want to track and experiment on the entire journey.” 

Contentsquare will help by providing rich, contextual insights into how customers feel at each stage of that digital journey—from start to finish, page by page. This will help the team at Sky make informed decisions about which tests will have the biggest impact on improving the digital experience of their customers.

Alternately, RS Group is focusing on process and automation to get experimentation velocity to the next level. Stewart’s team has spent this year building an experimentation program management system in Airtable to help align stakeholders from different business units. “The program will help us expand our testing roadmap and begin automating our experiments,” said Stewart. 

“To scale properly, we need to reduce the manual time spent analyzing experiments, writing emails, responding to Slacks and building experiment scorecards.” So, for RS Group, it’s all about building repeatable, scalable and automated experiments that save time.

Watch the session on-demand 

Feeling inspired? There’s more where that came from. Watch Natasha’s and Stewart’s session from CX circle London on Contentsquare’s on-demand page to get more inspiration for your experimentation strategy.

And if you’d like to learn more about how Contentsquare can help you skyrocket your experimentation program, get in touch with us today. We’d love to show you how our digital experience analytics platform gives you all the tools you need to build informed tests that truly deliver.