What to do if ChatGPT is getting dumber?

Subscribe | AI Tools | Sponsor | Past Issues

Today I’m answering the hot question being debated all over X (Twitter), Reddit, and Discord: is ChatGPT getting dumber? - and if it is, what can you do about it?

The short answer is: it’s complicated…

Like most things in AI, the situation is gray and not strictly black or white. Let me explain why and what you should do about it.

🙃 Is ChatGPT getting dumber?

Conversations on social networks have been happening for some time now on ChatGPT, and if it’s losing its edge. This is not new. But a recent research paper titled “How Is ChatGPT’s Behavior Changing Over Time?” poured fuel on the fire and struck controversy by claiming that GPT-3.5 and GPT-4 (language models that power ChatGPT) have been getting worse over time at performing the same task. Interestingly, no one is talking about the example they provided that showed the two models improving with time.

4 AI concepts help us better understand whether or not ChatGPT is getting dumber:

1. Large Language Models learn and evolve 📝

By design, large language models (LLMs) evolve with time. They learn from usage, feedback, and additional training. So it is fully expected that the model will get better with time, but a side effect is that it could also get worse in some areas. As someone using an LLM-powered AI (e.g. ChatGPT, Bard, Claude, etc) it’s important to keep this issue in mind as you rely more heavily on AI for your work and business.

Unfortunately, LLM developers have no easy answer to know which way the model will go for which type of task.

2. Prompt drift 🌀

Another element to keep in mind is the concept of Prompt Drift. Prompt Drift happens when an AI, over the course of a long conversation, starts to lose focus of the initial task or topic. This can lead to the AI appearing to become "confused" or off-topic.

This tendency is mainly due to the inherent limitations of how LLMs like ChatGPT operate. They have a maximum input length (measured in tokens or characters), and when a conversation exceeds this length, the AI typically summarizes previous parts of the conversation to retain context while staying within the token limit.

However, this summarization process can lead to the loss of certain details over time, especially when conversations are long and complex, which results in the phenomenon known as Prompt Drift.

3. We missed things before 🪹

OpenAI’s Product Vice President Peter Welinder joined the conversation and said that GPT-4 is not getting “dumber,” but he has a theory that by using GPT-4 more heavily, we are starting to uncover issues we didn’t see before.

4. Our standards are higher now 💎

Remember the first time you asked ChatGPT a question, and it gave a cogent response? Or when it wrote a song for you in an instant based on a completely random topic?

When we first started using ChatGPT, we were generally more forgiving because of our awe of its remarkable capabilities. 🤩 But now, what was extraordinary just a few months ago is the new normal. So, if we don’t get the 'perfect' answer from ChatGPT, we start to think that it’s becoming dumber than before.

🤨 So, what do we do about all of this?

For many of us who are starting to rely on ChatGPT (or other AIs like Bard or Claude) for our business and work, this can become a real issue. But fear not, there are actions you can take to mitigate these problems.

Here are some of the ways that you can address issues with ChatGPT losing its edge.

1. Maintain a prompt library 🗂

Have a list of your go-to prompts organized in a prompts library and review it periodically. I’ll have an entire article on prompt libraries soon. For now:

• If working with a team, use Google Sheets or Docs to make it easy to share your prompts library.

• Review your prompt library periodically to see if your ChatGPT’s response to your prompts is getting better or worse.

• Categorize prompts by topic/use case.

• Track the date tested and which AI model you used (if you use more than ChatGPT).

• Rate prompts on the accuracy/usefulness of the response.

• Flag prompts that need to be retested soon.

2. Use ChatGPT custom instructions ⚙️

Custom Instructions are a way for you to “teach” ChatGPT additional things about you, your business, and your needs.

You can use your custom instructions to make ChatGPT smarter in the areas that it might not be doing great in right now.

For more on this topic, read my intro to custom instructions and the top 50 use cases I compiled.

3. Use Prompt Chaining 🔗

Don’t rely on AI to summarize critical key points from a previous chat forward. Help it by adding your own summary to your prompt. This way, you can make sure that the key points you want are carried forward.

4. Keep your prompts explicit 🎯

Like this section, keep prompts short, and don’t ramble.

5. Lower the temperature 🔽

Temperature settings that allow you to control how creative the AI should be in its response can help control drift.

This setting is not available in ChatGPT yet, it’s only available in the API, but I’m pretty sure it will be added to ChatGPT soon enough.

Less than a year ago, we didn’t think most of what we are seeing happening with AI was going to be possible any time soon. We knew it would eventually come to fruition, but no one knew when exactly.

AIs will get smarter or dumber, get better or worse. We will figure all of that out. As long as you are well informed on what’s happening and have safeties in place to make sure you (your business and teams) are using ChatGPT, Bard, or any other AI properly, there are tons of benefits from using these tools.

The key is understanding their limitations and being prepared for them.

Best,
~ Hussein Yahfoufi ✌️

🌟 Spotlight your business to 5,000+ business leaders, founders, and entrepreneurs

Grab a GPT Hacks sponsorship spot and put your brand in front of 5,000+ startup founders and business leaders like you!