2022 W44 - The Dip
Rows is now "Building in Public": every week I'll post about one thing that happened!
Read last week's article on doing hiring interviews.
---
On Monday we opened our Rows dashboard to discover that our North star metric (W3 active users) had gone down. This comes after a streak of several weeks in activity growth.
Some background: Our dashboard covers our 2 North star metrics, W3 actives and Viral sign-ups. We picked W3s to be our main metric because we found this is the most honest metric for activity.
Weekly Active users (WAUs) includes every user that does something relevant on the platform. It includes new users and old users. This metric is very sensitive to campaigns and social media and can be gamed easily.
W1 Actives are users that came back any week after converting for the first time (W0s). This is a great way to count all of the people who are active and had an "a-ha moment". If you came back the any week after the first, it's because something interesting happened on your first week!
W3 Actives are those who came back 3 weeks or more after the first, and it's a proxy for "users who formed a habit". This is the number we want to grow!
W12 Actives are those who have been coming back for 12 weeks or more, and it counts "long term retention." It's even better as a north star metric, but it lags too much: any change we do will take 1 quarter (12 weeks) to be seen in metrics. This is why we stayed with W3s and not W12s.
On the Dashboard we show these 2 metrics and then the drill down to other supporting metrics. You can check and duplicate our template Dashboard spreadsheet here and adapt it for your own business (that template has fake data).
So, with the W3s number going down, the next two things to do are: a) to learn about why the drop happened, b) to do something about it if there's something to fix!
For the first question, our Dashboard already has a basic breakdown of that metric into 3 components:
W3s who've been here for long (W3s AND W12s).
W3s who've not been here long term (W3s AND NOT W12s)
Automation users. In this last group are users who come back infrequently but who have an automation connected to messages that reach humans - say, you're repeating an alert on a slack message.
Looking at that breakdown we know that our numbers are still healthy: Out of the W3s, 70% of W3s are long-term, 20% are habit-users, 10% relevant automation users. This proportion has remained stable. In addition, we found that this dip affected newer users more than old users.
We drilled down further into activity, to notice that power spreadsheets and active spreadsheets are still strong. The supporting metrics of opens and views are also healthy. (Note: the dips in the green charts are weekends).
All in all, it seems like we had less W3s as a normal fluctuation on engagement.
What can we do about it? Well, our job.
The main remedy is to keep working with users and to ship more product. Just this past week we released easy duplication of Tables, Charts and Pages, among other fixes that were high on the requested list. A week in fluctuation per se isn't abnormal.
The second thing is to be aware.
This week we will look at the fresh numbers again! ☝️ If numbers go down and there's a trend, when we will again dig deep into activity, and eventually if it persists we will go into errors rates in the platform and what actions we did. At the deepest level, if numbers don't explain what's going on, we will interview every user who didn't come back to find out.
