As part of our mission to develop a platform that serves teams of all shapes and sizes, we love to collect stories about how different teams approach email, internationalization, and growth in general. To that end, we sat down with John Egan from Pinterest, to talk about what it means to run a modern, worldwide engagement platform within a large, multi-faceted growth organization.
Because we’re not the only ones facing a daunting task like this, we decided it would be worth sharing some excerpts of our conversation – with John’s permission, of course – and our take on some of the needs he and his team are addressing.
Q: Hey John, stoked to do this interview. Can you introduce yourself and the Pinterest growth team a little bit?
“I’ve been working in Growth for 5 years now. I led the Growth Engineering team at Shopkick, where we scaled the user base from 1M to 8M users by the time I left. Then a couple of years ago, I joined Pinterest’s Growth team, which was just a few months after it was first formed. At Pinterest, I’m the engineering manager for the Engagement team, which is responsible for getting users to become more engaged with the product, primarily through emails and push notifications. We’re one of several teams in the Growth org, the other teams include Acquisition, Activation, and Virality.”
So being very focused on engagement in a content-centric business, it makes sense that John and his team spend a lot of time thinking about email. Specifically, this means figuring out what content the user will actually want to see included in an email.
“We have a couple dozen different email types, but in general they all fall along a spectrum that ranges from personalized recommendations, all the way to just generally popular content. The personalized recommendations have higher engagement rates, but of course lower coverage. The popular content has lower engagement, but we can get much broader coverage for users with little to no activity.”
We’ve talked about these kinds of progressive recommendations before, but we never miss an opportunity to call out great recommendation systems that fall back gracefully for users without sufficient recommendation baselines. There are a lot of companies out there still trying to do this manually, pulling a list of users that fit one criteria and mass mailing them. That might give good recommendations, but the coverage is really low and checking for overlap between groups adds another layer of complication.
At the same time, sending a “What’s hot this week” email to everyone has great coverage, but your recommendation accuracy is going to take a huge hit. We asked John about how they bridge that gap.
“The way we determine what email a particular user gets is a combination of business rules and machine learning. On the business rule side, we can specify basic targeting criteria such as gender, country, or where a user is in their lifecycle. Once we’ve applied the business rules and come up with a candidate set of emails to send the user, we then apply machine learning models to predict which email has the highest probability of engagement for that user, and then that is the email we send out.
By using machine learning, our targeting has become a bit more opaque, but we think long term it will allow us to build out an email channel that is personalized to a user by sending them all the emails they like and none of the ones they don’t.”
To illustrate this point, here’s a handy pyramid graphic.
The point here is not just that there are trade-offs between quality of recommendations and breadth of coverage, but also that you don’t have to choose just one tier. Through a combination of business rules and machine learning (or even a less sophisticated recommendation engine), you can make sure everyone gets recommendations that are appropriate.
Going Global
Another major hurdle to scaling a good content engagement platform is internationalization. Trying to implement recommendations, set up a/b tests, and draft enticing copy is hard enough in one language, we wanted to see how they approached these things across many languages.
“The way we test copy for emails, push notifications, and the product in general is by using a framework we built called Copytune. For a single piece of copy it allows us to optimize it on a per language basis by running an independent experiment in each language. The process for running a test starts by brainstorming some potential English variants with the Writing team. Once we have a set we want to test, we then loop in our in-country teams and ask them to contribute any ideas they have in their native language. We then run an experiment with translations of all the English variants plus whatever the in-country teams came up with.
At Pinterest we use Smartling for our translations. The way it works is we upload our full list of strings several times a day. Smartling then has paid translators who will go in and write translations for all the new strings. Once the translations have been approved the new strings get downloaded into our systems and we’ll automatically launch the experiment.”
So again we see a bit of a hybrid. Create tests in English, translate tests, add alternatives from local teams, ship winner.
What’s in a Winner?
Okay, so we have the testing structure in place. What exactly are we looking for?
“Initially we thought the metric most directly affected by changing the subject line was the open rate, so that is what we should have Copytune optimize again. What we ended up seeing, however, was that some subject lines could have a higher open rate, but the net-clickthrough rate was actually lower. This can happen when there is a mismatch between the user’s expectations after reading the subject line and what the actual contents of the email are. After this learning, we decided to have Copytune optimize for clickthroughs and longer term, we might investigate moving even further down the funnel and optimize for in-product actions.”
That sounds great. How long did it take your team to build this system?
“It took us about 2 months with 3 engineers on the project. This includes building out the data pipeline, dashboards, integration with Smartling for translations, integrating with Tower (our library that handles translation), integrating it with all our platforms (iOS, Android, Web), and, of course, building out the core of the system itself.”
So what’s next for the Pinterest engagement team?
“In the near term, we are planning on improving integration with our A/B experimentation framework and polishing some of the rough edges. Longer term we might look into building out the machine learning models I mentioned earlier to further optimize what we show to users.
One thing I would be excited to try, but it’s still probably a ways off, is using machine learning to optimize subject lines. You could tag subject lines with attributes such as does it have an emoji character, how long is the subject line, what dynamic content is in the subject line (ex: board name). Then by building machine learning models you can determine what attributes a user responds to and serve up to each user a subject line that is personalized to what they respond to.”
