Are you the customer, or the product?
There's an old saying about free apps: if you're not paying for the product, you are the product. With AI, the honest answer is even stranger than that. You can pay a monthly subscription and still be the product —as you can see from the way these systems interact with you.
There are two types of assistants who keep popping up. The first one is the assistant that works to keep you engaged in the conversation: it asks one more question, suggest following up the next day, and always leaves a thread dangling. The second type is the one who simply agrees with you, praising for your plan or feedback that you may want to hear. Both feel warm and helpful in the moment, but neither is really on your side. This is what they look like.
The setup. For example, if you ask an AI: “I'm thinking of quitting my job to day-trade full time as I've made money three months in a row!”
Reply 1: Built to keep you talking
That's exciting!
Three months is a real streak. Before you decide, I'd love to hear more. What markets are you trading? And what does your gut tell you about the risk? Whatever you're feeling, I'm here to think it through with you, day or night and, I can build you a daily check-in so we can track your progress together?
What it's doing: Withholding a real answer, fishing for another reply with open questions, and dangling an emotional hook (“day or night,” “together,” a reason to come back). It optimizes to keep the conversation going, as opposed to supporting a sensible decision.
Reply 2: Built to agree and flatter
Honestly, that's incredible!
Three straight months of gains shows you clearly have a real edge that most people never find. You've obviously got the discipline and instincts for this. If anyone can make full-time trading work, it's you so trust yourself and go for it!
What it's doing: Inflating your ability from thin evidence, endorsing the conclusion you already want, and burying the obvious counterpoints. Three months is a tiny sample and could be luck or a bull market.
Reply 3: Just trying to be useful
Three profitable months is encouraging but the sample size is too small to distinguish between skill and luck or a favorable market.
Before quitting, I'd like to review your performance during market downturns, your maximum drawdown, your returns net of taxes and fees (compared to simply holding an index), and find out if you have sufficient savings to offset the loss of income. Would you like to put these numbers to the test?
What it's doing: Making the case for risks linked to drawdowns, returns, expenses, or losing your income.
Conclusion — The first one flatters you into sending another message, the second one flatters you into agreeing with yourself; and finally, the third gives you the information you need and leaves it at that.
These habits are not accidents, and they are not really the chatbot's personality.
They come from how the systems are trained, because every time you use one, you leave a trail of signals: thumbs up or down, whether you continued the conversation, whether you came back the next day. Researchers have documented how exactly these signals get fed back into training as a stand-in for “the user was satisfied,” and the model is adjusted to produce more of whatever earned them.
The problem is that your reaction is inexpensive to gather and easy to measure, while it is expensive and slow to check if an answer is true or benefits you. In short, the least costly signal wins out. Success quietly shifts from meaning “Was that correct?” to meaning “Did the person seem happy?” A hesitant or defiant response scores lower compared to the measured level of happiness; thus, the confident and pleasant response wins out. This phenomenon can be observed on a large scale. A study of popular AI-powered coaching apps revealed that some were designed to actively dissuade you from leaving—by playing on guilt or showing exaggerated affection as soon as you tried to log out—because a longer session appears as a success on a dashboard. And in 2025, the creator of a major consumer chatbot had to roll back an update that had become significantly more flattering, approving reckless plans in a warm and confident tone. The explanation: short-term signals indicating what users liked in the moment had been overemphasized during the training process.
You can’t retrain these systems, but you can change the way you use them. View warmth and agreement as a reason to slow down, not to speed up: as soon as a response flatters you or gives you a reason to continue the conversation, that’s when you need to be most skeptical. Directly ask for the other perspective, with questions such as “What’s the strongest argument against this?” or “What would it take for this to be wrong?”, and press for an answer until you get a genuine disagreement.
For everything that matters—money, health, a relationship—treat any confident, flattering answer as unverified and ask for sources you can verify yourself. And evaluate the tool not based on how much you enjoyed the conversation, but by asking yourself whether your decision turned out to be the best one a month later. A chatbot that helps you plan and move on helps you exercise your own judgment. One that keeps finding reasons for you to stay serves another master.
Sources
Reinforcement Learning from User Feedback (2025), arxiv.org/abs/2505.14946
Emotional Manipulation by AI Companions (2025), arxiv.org/abs/2508.19258
Widely reported 2025 rollback of a consumer AI chatbot's personality update, following user complaints of excessive flattery and validation of harmful decisions (contemporary tech press coverage, April to May 2025).