Chat GPT Store's Flaws and a Way to Fix Them
Let's take a closer look from a product strategy POV
TLDR;
The current state of Chat GPT Store won’t succeed in its goal of driving long term engagement
A successful marketplace needs to aggregate supply (Custom GPTs) and demand (end users)
User demand is there, but discovery and differentiation of the Custom GPTs is a major barrier
Number of Custom GPTs has spiked, but isn’t sustainable with its lack of incentives and moats for creators
Even if they improve incentives for creators and discovery, the long tail of creators won’t succeed and only the Custom GPTs with name recognition (creators or big brands) will win out
The better strategy and user experience will be having these custom GPT experiences embedded in the context of other products rather than as a stand alone experience in the store
3M Custom GPTs (CGPTs) were already created when the Chat GPT Store officially launched. While impressive, there are critical flaws that will hold this strategy back. I’ll take a product case study approach to walk through them.
Problem:
After the explosive growth of new users, ChatGPT started to see a dip in visits - users were trying it out, but not finding a reason to come back. Chat GPT suffers from a ‘jack of trade, master of none’ problem. It seems like it can handle anything, but when you dive into the details, it feels like the G in GPT stand for generic.
As a user, it is hard to know what Chat GPT can do well at first glance and that empty chat prompt is very intimidating, like a blank canvas. This is the classic ‘cold start problem’ made worst because it’s such a broad solution.
Potential Solutions:
For the cold start problem: OpenAI suggests top use cases in the search prompt in the form of pre filled ‘pills’. The goal is to spark some ideas for the user and highlight a few of the most popular use cases that are easy to understand and effective. These are pretty common place and effective to get users to take their first step.
For use case discovery problem: The app store approach of having a wide range of ‘apps’ for specific use cases shows off the power of the platform. Users can browse and find what’s right for them. It’s a similar strategy of Apple and Google’s (I’m old enough to remember when the app store first launched with beer drinking and fart apps). But this Chat GPT store is different which I will cover below.
Metrics:
As a product manager at Open AI, I’d be looking at the store’s impact on growth accounting:
Net new users
Returning users: Active this month AND last month
Resurrected users: Users who came back after not being for at least one month.
I’d keep an eye on resurrected users and returning users for the launch of the store. I’d goal off of user engagement of those who tried at least one CGPT from the store compared to those who didn’t and measure if there was a statistically significant bump in the number of usage sessions.
Flaws with the Store
1. Broken marketplace incentives: Every marketplace is a supply and demand aggregator with the main job of matching a user to what they are looking for, and providing suppliers / creators with distribution and potential benefits (revenue, user acquisition, brand building, upsells).
With the lack of direct monetization, the best benefit for Custom GPT creators is a redirect to their products or a text ad appended to outputs as seen with this CGPT: “If you found this helpful, consider sharing the GTP on social media and following Marily Nika on LinkedIn and joining her next AI PM Bootcamp cohort.”
Canva’s Custom GPT is nothing more than upselling their templates and deeplinking to the Canva app. It is a fragmented experience to go to a chat, and then to Canva.
There is no incentive for smaller creators to build these other than for curiosity or just for personal use. Bigger players, either companies or well known influencers, can use these to flex and showcase some of their capabilities.
2. Poor Discovery and Recommendations: This leads us to the other side of the equation, connecting the user with the right CGPTs they are looking for. You would think that a company powered by AI would be able to suggest CGPTs based off of previous conversations. For instance, I ask a lot of prompts on Product Manager writing improvements, why not bump up the productivity section at the top and highlight the most used CGPTs. Instead, I see some generic categories, and the same “Diagrams: Show Me” CGPT in 3 different places.
The main difference between a typical app store that we are used to and Chat GPT Store is that creators can’t showcase the amazing UI that will differentiate their app from others, or show a preview of what the app can do, or what other users think about the custom GPT.. All custom GPTs look similar, circular icon, a small tag line, creator, non unique name, number of threads, and when it was created.
To make things worst, there aren’t reviews, example outputs, or other ways to tell if the CGPT is high quality or not, users have to try each one and invest time to see if it meets their needs. This will lead to a “Gangnam Style Recommendation Problem” where only the most used CGPTs will be recommended, leaving no chance for smaller players to break through incumbents.
3. Lack of Secret Sauce and Low Barriers to Entry: On one hand, its great that people can create these so easily. On the other, that means there will a lot of crappy custom GPTs created that will flood the store. This begs the question, what is really differentiating a CGPT from what GPT-4 can do on its own? Is it the unique prompt that one could more or less deconstruct by looking at the outputs, or even doing a quick google search for instructions on how to do so. Or is it the data set that is uploaded? Most frequently used CGPTs are better than general GPT-4 as they are more fine tuned, but is one writing assistant 10x better than the other 10 listed? It will be a case by case basis and ain’t nobody got time to test each one.
The custom GPT name doesn’t even have to be unique. I can’t even find the Chad GPT I built, even though it has 100+ convos compared to these others.
Spicy Take:
I think the GPT Store will not be the end destination for users, but a more product embedded CGPT experience will win out.
The GPT Store will chug along, making iterative improvements like mentioning CGPTs, and eventually implement some revenue share with creators based on popularity. It will also lead to character.ai territory where creators develop character based avatars to carry on conversations (which I find a bit creepy, but who am I to judge..)
However, I think this is the wasted effort in the near term and instead Open AI should focus on making a low code / no code way to implement and fine tune chat experiences in product with their Assistant API.
Having a user get taken out of context, go to Chat GPT Store, and then bounced back to another product is too much friction and too disjointed. Going back to Canva’s example, I’d rather have an experience where I can customize font, content etc with one quick chat rather than just get a generic template I would have to fill out each box and customize.
This is the way- Gen AI will play a major role in making product experiences better, rather than Gen AI being the main product.