Friday, November 9, 2018

How one simple experiment raised conversion in an app by 15%

Product development is like building a one-story summer house. You start with the groundwork and everything seems so well thought-out and logical. Then you get cocky and think to yourself: hey, why not build a second floor? What could possibly go wrong? So you do it. And if the structure still holds up, you also build an adjacent garage, a sauna, a greenhouse, and a terrace. That's when your lovely summer house turns into a lovely little mess that looks incoherent and could even break down.

Building an app works in a similar way. You start out with an MVP and with each new iteration you pile on more and more features, the navigation becomes confusing, the design — cluttered. At that point, most people realize that's not the way to go and something's gotta give. Some teams even go as far as to scrap everything and go back to the drawing board.

Taken from the Evernote corporate blog and their post about redesigning the app

But the problem is not just with the design.

If you have a freemium app it's more than likely that your subscription-only feature set is that messy summer house, a legacy of days gone by that needs to be reevaluated

At least that's how it turned out for us.

What's a Freemium?

Freemium is a form of monetization and distribution where basic services are provided free of charge while additional features, services or virtual goods are only available for those paying a premium. There are 3 kinds of freemium limitations out there:

  • functional. A premium unlocks new features. Examples: YouTube Premium (offers ad-free viewing, offline watching, watching videos in the background, etc.), Tinder (the swiping is unlimited, and you also get superlikes, boosts, and other neat things).
  • quota limitation. You can use the product for free up until you reach a certain quota. Examples: Dropbox (storage space quota), Slack (limited number of integrations, messages), CleverPay (MAU limit).
  • support limitation. A rare format for the b2c sector, it's more of an enterprise-level thing, often associated with OSS-projects (nginx, SphinX, MongoDB, RHEL, elastic, и т.д.). The software is distributed for free, the premium gets you extensive support from the team of developers.

In this article, we'll focus on the first two kinds of limitations.

Is the Freemium model right for you?

What do the successful freemium-based businesses have in common? The majority of their user base can go for years without having paid a single cent for the service. Personally speaking, I've been using Dropbox since 2009 and I've never paid for it. Notion, Google Drive, Evernote — none of those ever got my credit card details, even though I rely on them every day.

So how come these services are even successful? Well, a user who's been loyal to the product for years is more likely to become a paying customer. Admittedly, it does seem like a long shot. However, if the product is good, your paying client base is bound to grow exponentially, because people are eager to recommend something they get great use out of to everyone they know and their mother.

If you're considering freemium, it's better if your app meets the following criteria:

  • it's a truly great service. Recommendations will be your bread and butter, so you better be worthy of telling a friend about;
  • you can afford having customers use the app for free. With 95% of "free" users and only 5% paying ones, can you still turn a profit?
  • there's a decently sized market for your service. Recommendations don't work as well (or at all) for a "niche" offering;
  • your product implies frequent use. If it's something one needs once in a blue moon, they're not going to run into the limitations and you...you'll go broke.

What should you charge for?

Ah, that is the question, isn't it? And the wrong answer can ruin everything: your revenue, your referral program, even your product. Nobody knows the right answer and if anyone tells you they do — they're lying.

It's a guessing game, just like the one you play when picking the price of the product: you pick the one that best fits its worth to you and then tweak it based on the market's reaction.

There are, however, certain things to consider if you're still in the development phase:

  • your goal is to maximize retention and service usage. If you have a large user base and plenty of sessions, monetizing it shouldn't be a problem. Just try not to choke the user with the limitations;
  • it's much more painless to make a "premium" feature available for free than to take a free feature away;
  • get ready to experiment a lot with feature availability.

How our team screwed up the free feature set

Let me start by saying that we've made yet another expense tracker app. We thought that it would be best to go the functional limitation route. At its very start, the app was designed to track the expenses only. As time went by, we've added income tracking, debt tracking, reports, a shared budget, and other stuff. All of the new features were hidden behind the paywall, as we thought that expense tracking should be more than enough for the majority of users.

We were wrong. As new features for the Premium subscription kept piling up and the free feature set remained the same, our retention suffered. We could see the problem as we compared the retention rate among paying customers versus the free users, but we had no idea what to do about it.

Having exhausted all of the conventional attempts at increasing retention, we've decided to switch one of the paid features over to the free plan. It was a leap of faith: we took the most used feature (the one that correlated with the highest retention rate, the one the users always asked for) and made it free.

This feature was basically welded into the subscription, so the switch required rewriting a part of the app, and the oldest one at that. We spent over a week doing it (from formulating the task, to writing code, to testing, to releasing…), but the newest build had the updated subscription set-up.

We've also launched an experiment, directing two thirds of our users to the old version of the product (with the paid income tracker) and one third — to the version where it was free. We were prepared to see the revenue take a dip, but what we actually saw was something completely unexpected.

Blue line — test group; orange and green lines — control groups. Retention calculated based on the event of a new transaction.

That's what the 30-day retention chart looked like. Two control groups — it was an АА-test — performed well. The test group went ahead starting from Day 1. In a week's time, we've stopped adding new users to the experiment as the outcome was pretty transparent. The test group had a 50% higher retention rate by the end of the week.


Now, in-app purchases are another story. If you're using a subscription model, conversions alone don't tell the whole story, you have to look out for subscription renewals. Renewals are the litmus test of how valuable your product is. We've waited for 2 months since that experiment. In the meantime, we realized that closed it too soon, making the results a little less reliable. Still, it was quite illustrative.

We took a feature out of the premium subscription feature set, which technically makes it less valuable. And yet, the conversion rate increased and we managed to retain the same number of renewals. On top of that, since our retention rate grew, we now have a longer tail, which means if we were to hold a sale down the road, more users would likely take advantage of it.

At that point, our referral program was in its MVP infancy, so I can't show you the stats on new referrals, but something tells me you'd be seeing an uptick there too.

To sum things up, a single experiment raised our conversion are by 15% and monthly retention rate — by 60%.

What's next?

After the experiment with income tracking, we've reevaluated our paid subscription and held 12 more experiments. To launch those faster and get the developers involved as little as possible, we turned to feature toggling, only slightly modified.

Our app had 10 major features we wanted to test. We taught the app not to look for a subscription from a user, but instead to listen to a server that would decide whether a user had access to a particular feature.

This approach gave us a lot of room for experimentation. Here are some of the more interesting ones we've held:

  1. Should we use a quota limitation? We've opened up all of our features, but limited the usage to 100 created entities;
  2. Dividing the subscription into several tiers. On top of the regular Premium on a discount we've added a SuperPremium for a higher price, offering several of our niche features exclusively;
  3. Does feature X availability affect revenue and retention? The same experiment as our first one, only for different features;
  4. What if we should abandon Subscriptions altogether? We offered the users the option to unlock a feature through a one-time purchase;
  5. Unlocking the features awarded for referring a friend. Our referral program rewards users for inviting their friends by offering them certain exclusive features. We've tried giving those away for free.

The most exciting part of the whole process was being able to launch an experiment without getting the developers involved and regardless of the release cycle. We would just come up with a hypothesis, design the experiment and launch it, while the developers remained focused on their work on the app.

Not every app is destined to become the next Dropbox or Tinder, but we can all aim for it. If you're not a 100% sure that you're charging your users for the right things, it's worth checking that hunch. Don't be afraid to make a hypothesis, get the reasoning behind it (like relevant support tickets, stats, an educated guess — whatever it may be) and get to work, experimenting in search of the right model.

Do you experiment with your pricing? Share your experience in the comments.


DataTau published first on DataTau

No comments:

Post a Comment