Shipping, Analyzing and Improving our OpenAI integration

This is the eight post in our PLG Series, where we write deep dives on the experiments, strategies and tactics that we used at Rows to accelerate our product-led growth motion.

Make sure to check out the previous articles on 3 Matrices for building products with AI and Building a Loginless Experience for 1B people.

In December of 2022, as Q4 drew to a close, one of our engineers took the last few days of the quarter to work on a new integration. This time, with OpenAI. The goal was to add OpenAI’s GPT-3 model to Rows and become the first spreadsheet that combined a grid interface, spreadsheet formulas, and the advanced capabilities of LLMs.

=ASK_OPENAI was the first function of the OpenAI integration

We started with a single function - ASK_OPENAI() - which was a wrapper of the GPT-3 model. It was clear, even back then, that this type of functionality was going to be adopted by every productivity tool, and we were already getting multiple customers telling us that they had built Google Apps Scripts/VBA Macros to build similar (albeit clunky) wrappers.

After a silent release at the end of December, on January 3rd, we announced it publicly.

The Early Days: Initial Success

The initial launch of the OpenAI integration was met with enthusiasm and curiosity. Thousands of people from diverse backgrounds - from data analysts to marketing professionals - began experimenting with the new capabilities. They used AI to clean up company names, do sentiment analysis, translate text and transform how they worked with data. 

However, after the initial buzz, a trend emerged that caught our attention. The growth in weekly active users of the OpenAI integration began to plateau. Every week we were getting hundreds of new people trying the integration, but few returned. 

So we dug into the data.

Cumulative number of OpenAI users in first 6 months

Going after the root cause

The real litmus test for any feature is its retention. Looking at daily retention of the new integration, it was clear that the feature did not correspond to user expectations. In fact, only 5% of people used the integration in the week following its first use (week 1 retention).

This said a lot about the usefulness of the feature, especially given that connecting the Integration required adding your own API key. This already filtered out many ‘general enthusiasts’ and people who had - to use a buzzword from the SaaS world - ‘low intent’.

We knew the retention was low. But as always, it's the why that matters.

Weekly retention of the OpenAI integration

Our analysis revealed a primary reason for the low retention. High error rates in the functions and issues with the UX of the integration:

• 43% of people that tried the OpenAI functions got at least 1 error related to rate limits/authentication issues with the API.

• >40% of people who experienced an error on their first day, churned.

Here’s why: the most common use for the Integration was to automate a column of a table such as tagging a list of customer support tickets across hundreds of rows in a spreadsheet. Given the stringent rate limits of the API (enforced by OpenAI),  when users dragged their formulas down to the column, they would inevitably get errors in the result of the functions. This was a core bottleneck in the experience.

The other issue was a mix of a blank canvas problem and the UX of the OpenAI functions. In the former, many people who connected the integration didn’t have a clear use case in mind, and ended up not finding what to do with the integration. 

The latter is a common conundrum with the spreadsheet interface. The OpenAI integration was served via formulas, which contain parameters where the user adds text, numbers of reference cells (e.g. A1).

Customer wants to build their prompts with a combination of text and cell references (e.g. to tell the model tag cell A2 with a list of tags), they’d have to use a string-reference concatenation in the prompt parameter, like: =ASK_OPENAI(“Classify the sentiment of the following text between Positive, Negative and Neutral. Text “&A2).

While this was fine for spreadsheet experts, this is not something that we can expect a generic spreadsheet user to quickly understand.

This meant that we needed a simpler way for people to use a combination of prompts and cell references, and to surface simple tasks that are relevant to their use case. 

To confirm these assumptions, we sent an email to more than 1k people who had connected the integration and churned. We did across in two batches. This was the copy of the email:

Email asking for feedback on the integration

Addressing the root causes

Armed with this feedback, our team worked on 4 big changes to the Integration:

1. A technical rewrite: This was the biggest change in the UX. We re-engineered the integration to be keyless, removing the need for an API key and rate limiting. This change was significant - it meant that users could now use the integration on a much larger scale without encountering frustrating limitations. To enable it, we moved the OpenAI integration to a Premium integration and offered 50 free uses of the integration for people in the Free plan.

2. Expanded catalogue of functions: We went from the initial 1 function to 16 functions of the integration. Each solved a particular problem (e.g. =SENTIMENT_ANALYSIS_OPENAI) and was very straightforward to use. In parallel, we added new templates that showcase the OpenAI use cases, new guides on our docs, and tweaked the email onboarding sequence to show the value of using OpenAI in the first 24 hours.

3. A Persona-picker: To help people navigate through the new functions, we added a step in the Action wizard where people could pick a role - e.g. ‘Marketing’ - and see a list of recommended OpenAI actions.

4. A show demo experience: To address the blank canvas problem, we added a “Show example” functionality in the OpenAI functions that provided live, hands-on examples by loading a demo table right in the spreadsheet

We shipped these 4 changes between October and November. Then, we looked at the results.

Results

Now onto the results. We break down the KPIs with a before and after period. Before contains the data between July and September (Q3), and the after concerns Oct-Dec (Q4). 

In the middle of June, with the release of the AI Analyst✨ and Instant Rows⚡ we had a big acceleration in acquisition (10-20x), which meant that taking the Before period starting at the start of the year would not be representative of the number of people coming in, and their intent. Finally, we looked at Paid customers specifically, because OpenAI is a Premium feature, and the paid customer experience is obviously vitally important to us.

Acquisition

KPI: % New workspaces to use OpenAI in first day of activity.

Trend: -7% of workspaces try OpenAI vs before period; 19% of paid customers use OpenAI in their first day of activity.

% New workspaces to use OpenAI in first day - Before vs After

We have a slightly lower % of people using the OpenAI integration on the first day of activity. This is explained by the ‘anxiety’ associated with trying a paid feature, even if under a free trial. With the changes we introduced in the last days of 2023 and a new onboarding version coming in early 2024, we’re confident we can double it by Q1 2024.

Errors

KPI: % of users who get at least one function error related to API issues

Trend: 85% decrease in % of users who have function errors.

% of Users who get errors related to the API - Before vs After

This is perhaps the most significant KPI. The experience of using OpenAI in Rows is dramatically better than before. It is a testament to the work done by our engineering team. Anyone can now use the integration to tag, extract, translate and transform data in thousands of spreadsheets rows in one go, without a hiccup.

Retention

KPI: Week 1 retention of OpenAI integration.

Trend: 42% improvement in week 1 retention for OpenAI users (1.7% vs 1.2%). Week 3 retention of OpenAI integration for paid customers is 27%.

Weekly retention of OpenAI integration - Before vs After

It’s good to see improvements on the overall retention. We know that there are a lot of people who try it out of curiosity and without a real use case for it, but with the changes we’re introducing in Q1 we’re confident we’ll show value faster to anyone who tries it for the first time.

The retention of OpenAI users who are on the paid plans is strong, a testament to the value of the integration for the paid accounts.

Monetization

We released the new Pricing Plans in the first week of October, and two months in, we’re happy to see that the OpenAI integration is a strong driver of upgrades. More than 50% of paid workspaces use the OpenAI integration, which is a strong indication of value.

It also tells us that working on the onboarding experience and showing more people how to use and benefit from the OpenAI integration in Rows is a big lever to increasing ARR.

Wrap up

One year in, our journey with the OpenAI integration is just getting started. We have a packed Q1 with new onboarding experiments, OpenAI improvements and big upgrades to the AI Analyst✨ (our second big AI experience)

We’re building the easiest way to import, transform and share data in a spreadsheet. And AI is a big part of it.

If you haven’t yet, head out to rows.com and see what the future of spreadsheets looks like.