How Your AI Chatbot Changes ProdOps

A few weeks ago, Pendo released their new Agent Analytics tool. If your product has an embedded AI assistant or chatbot, then this tool assists ProdOps in several ways, like:

• Categorizing user prompts into discrete, analyzable use cases
• Identifying "rage prompts" (new user frustration metric)
• Comparing workflow completion rates between the AI and pre-AI versions

The big question on our minds: in a world where prompts/chats are just as important to product experience as views/clicks, does "product management" as we know it have to be reborn?

Welcome to Beyond The Click by Balboa Solutions. In today's issue we're broaching the complexities of PM in the AI era, including:

1. The "why" isn't changing 📌
2. But the "how" is changing fast 🛣️
3. And Product teams aren't keeping up 🏃
4. So here's some ways to get on board now 🚂

Let's dive in.

The "why" isn't changing 📌

First, let's address the elephant in the room. Page views and feature clicks aren't going to evaporate from software products. There's a fundamental human need to be involved in automated processes (think George Jetson complaining about clicking six buttons instead of three during a busy workday in 2062). And there are good reasons for this human trait:

• To learn and "grok" the process
• To keep quality assurance on the final product
• To retain ownership of the outcomes

So, traditional software UI elements are here to stay. But even if page views and clicks did disappear entirely, they've never been the point of product management anyway. Anyone solely focused on these surface-level metrics is missing the point.

What truly matters is aligning around the primary objective for your user. What's the value of the experience you're building for them? Great product work requires looking beyond singular interactions to develop a solid strategy that correlates to that end objective. This brings us back to Marty Cagan's essential triad for product discovery and design:

1. Value (determined by the product manager)
2. Feasibility (determined by the engineer)
3. Usability (determined by the UX designer)

Whether interfaces evolve into agentic chat, voice commands, or something else we haven't imagined yet, these principles remain constant. We still have to deliver valuable outcomes for customers and users. We still have to make solutions technically feasible. And we absolutely still have to make the interaction experience usable across the entire problem we're trying to solve.

But while the fundamentals of good PM are still there, the details of how we achieve them are another story. 👇

The "how" is changing fast 🛣️

With AI capabilities expanding rapidly, product teams will soon have more opportunity than ever to accidentally ship seemingly cool features that users actually hate. Extreme automation combined with hyper-personalization can be an amazing productivity hack to one person, but creepy spyware to another.

Product managers will need to become much more attuned to accessibility, security, privacy, and what's socially acceptable from an end-user perspective. It's not just about regulations anymore. It's about empathy and understanding what crosses the line into "creepy territory" for your specific users.

The metrics that matter are also evolving. User behavior analytics will increasingly give way to product behavior analytics. Today, PMs spend significant time analyzing visitor metrics, but soon the focus will shift dramatically toward action metrics:

• How many actions got completed?
• Which use cases did those actions serve?

These actions will draw from all modalities (visitor clicks, agent clicks, AI API calls, etc.), and from both on-platform and off-platform sources (for more on off-platform product actions, check out our recent piece on MCP for ProdOps).

For example, consider ordering coffee on the Starbucks app. Whether you log into the app and select your drink from a dropdown menu, or you speak a natural language prompt to an AI assistant, there's still fundamentally just one action taking place: ordering a coffee. The modality changes, but the outcome remains the same.

This shift also has major implications for how products are monetized. Don't forget that PMs are the ones who cultivate value. As the visitors ➡️ actions transition occurs, more products will need to move toward consumption-based and outcome-based pricing models rather than traditional user seat-based pricing. This properly aligns price to actual user value.

As an example, look at Intercom's Fin customer support agent (#notanad), which charges for solved cases rather than user seats or messages sent. This is pure outcome-based pricing, and it aligns with the value that the user (a customer support specialist) needs.

Product managers must start thinking strategically about if their monetization model should be AI-first and outcome-focused, because there are real differences in how you build products to support these different approaches.

These are real, tactical changes to the product management playbook, and they're going to take some work to adopt. 👇 

Product teams aren't keeping up 🏃

If great product management requires value, feasibility, and usability regardless of the underlying technology, and if the practical execution is changing so dramatically, where does that leave most Product orgs today?

Most product teams are thinking about AI, but they're overwhelmed by the gravity of this shift. They're focused internally, asking "how do we do our jobs better?" and "how do we run our team more efficiently?" They're using AI to speed up development processes and reduce team sizes. And, credit where credit is due, these operational gains from AI are real.

But here's the gap: most teams are so swamped with these internal transformations that they don't have the bandwidth to think deeply about the customer perspective. They're not yet asking "how is AI impacting our users while they interact with our product?" Some are trying to sprinkle a little ✨AI magic✨ into their existing products so they can say they're on the AI train, but it's a much smaller sample that's building truly AI-first products.

Even just starting with the customer value prop proves challenging for many organizations. When pressed to describe how they bring products to market, teams often default to listing features and functions rather than articulating clear customer outcomes. This problem isn't new, but it's more severe in the AI era.

But we always try to offer ProdOps leaders some practical advice on how they can address the challenges we're seeing in the industry today. 👇

Here's some ways to get on board now 🚂

If you want to manage your product experience for an era where prompts may soon overtake views and clicks, here's 3 steps you can start working toward:

1. Quantify 📐

Start by understanding the scale of this shift within your own product. What percentage of user actions today are AI-driven versus traditional UI interactions? If you don't currently have efficient ways to measure this, tools like Pendo Agent Analytics can provide visibility into how AI agents are being used in your product. You can't manage what you don't measure.

2. Prepare 📋

Identify all the AI features in your product roadmap. Then forecast how the AI action percentage you quantified above will increase over the next 12, 24, and 36 months. This projection will help you understand whether you're facing an incremental shift or a fundamental transformation in how users interact with your product.

3. Strengthen 🏋️

Take every planned AI feature and rigorously assess it against Cagan's triad: Value, Feasibility, and Usability. Pay special attention to Usability. Ask honestly and repeatedly: "How might a user find this unsettling?" Test your assumptions about what feels helpful versus invasive. You can also use this step to assess if your products need a Value re-alignment via outcome-based pricing. 

The rules of AI-first product are changing rapidly. But the teams that ground themselves in timeless principles while thoughtfully adapting their execution will be the ones that ship products users actually love. The "why" behind great product management isn't changing. But if you're not actively preparing for how the "how" is evolving, you're already behind.