For the last few years, most professional conversations about generative AI have been organized around outputs. Can it write the email, summarize the meeting, create the presentation, analyze the spreadsheet, generate the image, draft the proposal? That was a reasonable way to understand the technology when most people encountered it through a chat window and the interaction was relatively simple: a person entered a prompt, the model produced an answer, and the person decided whether the answer was useful.
That is becoming an increasingly narrow description of how AI is being deployed.
I was recently listening to an episode of AI & I featuring Quinten Farmer and Eliot Peper from Portola, the company behind the AI companion Tolan. Their product sits well outside the enterprise and event workflows I spend most of my time working with, which is precisely why the conversation caught my attention. They are solving a very different problem, yet many of the design decisions underneath their product resemble the challenges companies are beginning to encounter as they move from isolated AI use into persistent workflows, agents, memory systems and connected tools.
Tolan is designed to develop an ongoing relationship with a user. To make that experience work, Portola has to manage memory, personality, conversational context, storytelling, model selection, response speed, evaluation, user preferences and the accumulation of previous interactions. They cannot prewrite every response because they cannot know what every user will say. They also cannot simply give a model unlimited freedom and assume that a capable language model will consistently produce the experience they want.
Their job has become designing the environment in which the model operates.
That is the transition I think organizations need to understand.
We are moving from using AI to produce isolated pieces of work toward building systems in which AI participates in the work itself. Once that happens, prompt quality remains useful, but it becomes only one part of a much larger operating structure.
Prompting is becoming one layer inside a larger system
We have spent several years teaching people to evaluate AI one answer at a time. Write a better prompt, receive a better result. Add context, clarify the task, specify the format, refine the instructions. All of that remains useful when a person is directly interacting with a model and reviewing every response.
Now consider what happens when an AI-enabled workflow has access to company files, email, calendars, project-management systems, databases and external research. The system may remember previous interactions, retrieve information from several sources, select a tool, perform multiple actions, pass the result to another model for evaluation and surface only the final output to a person for approval. In that environment, there may be no single prompt capable of explaining why the system behaved as it did.
The operating model starts to look more like context, rules, memory, tools, action, evaluation and feedback continuously influencing one another.
This changes what competent AI use requires. A person designing that workflow needs to know which information the system should receive, which sources take precedence when information conflicts, how much authority the model has, which actions require human approval, how exceptions are handled and how the system should respond when the situation falls outside the conditions it was designed to handle. A brilliant prompt does not compensate for a workflow built on outdated files, ambiguous authority, poorly defined approval rules or missing failure conditions.
That is why I increasingly see prompt engineering as one skill inside workflow design rather than the professional capability itself.
The human role moves upstream
One of the most useful parts of the Portola conversation came from Eliot Peper describing how their storytelling approach changed. The team originally tried to create more structured narrative experiences with clear plans and branching paths, something closer to a choose-your-own-adventure model. The approach became difficult for the models to navigate and produced interactions that felt too rigid.
They eventually moved in the opposite direction. Instead of scripting an entire story, they began giving the model situations, memories, character information, world context and narrative hooks, then allowing the model to respond more like an improv actor.
Peper described his role in a way that stayed with me: he is no longer writing every story the character tells. His work is teaching the system how to tell a good story in the moment.
Take that out of the storytelling context and place it inside an organization.
The same shift appears when a team moves from asking AI to draft a document toward allowing AI to participate in a repeatable workflow. The person is no longer responsible only for producing the final artifact. Someone has to design the conditions under which acceptable work can be produced repeatedly, including when the input changes.
In event production, for example, "manage this event" is almost meaningless as an instruction for an agent. The word manage contains dozens of authorities, dependencies, exceptions and professional judgments that experienced producers navigate without consciously documenting every one of them. Can the system change room assignments? Can it alter equipment quantities? Can it email speakers? Can it modify a labor call? Which version of the agenda controls the workflow? What happens when the signed proposal conflicts with a newer client request? Which change requires approval from the producer, the venue or the client?
A better starting point is much narrower: when a revised agenda arrives, compare it against the approved run of show, room assignments and technical requirements, identify every downstream document potentially affected, surface conflicts, and prepare a change report for human review.
That is not a glamorous vision of autonomous AI, but it is how reliable systems are built. Responsibility is bounded, authoritative inputs are identified, the expected output is known, and the human review point is explicit. If that performs consistently, the system can gradually receive more responsibility.
The design skill is deciding where those boundaries belong.
Memory creates a curation problem before it creates an intelligence advantage
Portola also described memory in a way that translates directly into enterprise AI. The challenge is not simply storing everything the user has ever said. Their system has to decide which memories should enter the context of a specific conversation because sending every available memory into every interaction would be inefficient, noisy and often counterproductive.
The question becomes: what does the model need to know right now in order to respond appropriately?
Companies are beginning to run into the same problem. Giving an AI agent access to an entire project repository sounds powerful until the repository contains six agenda versions, outdated proposals, old budgets, duplicate speaker documents, conflicting client comments and a dozen files with names like FINAL, FINAL2 and FINAL_USE_THIS_ONE.
Access does not create clarity.
An experienced employee often survives that environment because the logic lives in their head. They know which spreadsheet is current, which email changed the scope, which client request superseded an earlier decision and which document everyone else should stop using. The organization may never have formally documented those rules because the person carrying them has compensated for the disorder.
An AI system cannot reliably compensate for information chaos through intuition. Someone has to define which sources are authoritative, how recency is determined, which information should be ignored, how conflicts are resolved and when uncertainty requires escalation.
The more AI organizations connect to internal knowledge, the more visible their information architecture becomes. Companies that have relied on tribal knowledge, inconsistent file structures and undocumented exceptions are going to discover that AI can retrieve their information without necessarily understanding which parts deserve trust.
Memory architecture is therefore not only a technical storage problem. It is an operational curation problem.
Expertise becomes the evaluation layer
Another part of the Portola discussion focused on how they evaluate the responses their AI characters produce. They use other AI models as judges, but they found that generic evaluation does not work particularly well. Asking a model whether an output is "good" produces broad, agreeable feedback because the model has not been given a sufficiently precise definition of quality.
Their team builds rubrics, collects examples, labels outputs, studies user reactions and brings in people whose expertise matches the kind of interaction being evaluated. For some decisions, they evaluate at the level of individual sentences and compare examples of stronger and weaker responses.
That process exposes something I think the AI labor conversation often misses. Human judgment does not disappear when more of the execution is automated. Experienced people become responsible for defining the standards the automated system is expected to meet.
An experienced technical producer can look at a run of show and immediately recognize a transition that will fail. The cue may be logically sequenced and perfectly formatted, but there may not be enough physical time for the speaker change. The microphone handoff may be impossible from the location of the stage manager. The video playback may assume an input that does not exist at front of house. The room turn may require labor that was never scheduled. The client request may violate a venue restriction that appeared three meetings ago and never made it into the latest document.
A language model can produce a beautifully organized run of show without recognizing any of those conditions unless those patterns have been captured somewhere in the system.
This is where experienced professionals become extremely valuable in AI-supported environments. Their expertise can be used to identify failure conditions, create evaluation criteria, define thresholds, label examples and determine where an automated result is technically correct but operationally unacceptable.
The challenge is that much of professional expertise is tacit. People often know something is wrong before they can explain why. If organizations want AI systems to operate at a higher level, they will need to invest time in extracting that judgment from the people who have spent years developing it.
That is slower than buying an enterprise AI license. It is also much closer to the work required to build a dependable system.
One successful output proves very little
Portola's team repeatedly described the amount of manual testing, evaluation and iteration required to move from an interesting AI interaction to a product people consistently enjoy using. That distinction should be familiar to anyone who has built an automation or agent that looked impressive during the first demonstration and then failed as soon as real users introduced real-world variability.
A successful AI result is easy to overinterpret.
If a model correctly converts one agenda into one run of show, you have demonstrated that the task is possible under those conditions. You have not demonstrated that the workflow can handle missing fields, inconsistent session names, overlapping rooms, speaker changes, conflicting versions, last-minute agenda revisions or a document format the system has never seen before.
Moving from "the model did it" to "we can depend on this process" introduces an entirely different body of work: testing, edge cases, permissions, source management, exception handling, human review, documentation, monitoring and maintenance.
Then another problem appears. The underlying technology changes.
A model is upgraded. A connector behaves differently. A source system changes its schema. A vendor deprecates an API. An internal process changes. A person who understood the workflow leaves. A new compliance requirement is introduced. The original automation still exists, but the environment around it has moved.
This is where I think many organizations are going to discover the cost hidden inside AI adoption. Building the first version of an agent may become increasingly easy. Maintaining a growing ecosystem of agents, workflows, connectors, evaluation systems and permissions will require ownership, documentation and ongoing technical attention.
An organization can accumulate dozens of successful pilots and still end up with a fragile operating environment if nobody has been assigned responsibility for what happens after deployment.
More reasoning can produce a worse system
Portola shared one experiment that I think should be required reading for anyone evaluating AI products. They added another reasoning step to improve the quality of the companion's response. The system would reflect on what it planned to say, check the response against memory and then produce the final message.
The additional process added roughly half a second.
Their metrics deteriorated.
The AI may have been performing more reasoning internally, but users experienced a conversation that felt slower and less natural. A technically stronger process produced a weaker product.
That example should complicate how organizations think about AI performance because model capability is only one variable in an operational system. Response time, cost, reliability, interface friction, approval burden and the conditions under which the work is being performed all influence whether the system is useful.
In live event environments, this is obvious. A technically perfect answer delivered after the cue has passed is worthless. An AI system that requires twelve confirmations before completing a routine action may satisfy a theoretical risk model while making the workflow unusable. A live demonstration that pauses for fifteen seconds after every interaction may produce sophisticated answers while losing the room.
The same applies outside events. A detailed analysis available next week may be inferior to a directional answer available during the decision meeting. A highly autonomous workflow that creates constant exceptions may consume more staff time than the manual process it replaced. A model with superior benchmark scores may perform worse inside a particular business process because it is too slow, too expensive or poorly integrated with the systems surrounding it.
AI should be evaluated inside the conditions where the work actually happens.
The model is becoming a component, not the strategy
Portola also discussed using models from several AI companies across different parts of the product. They do not assume that one model should perform every task. Creative generation, low-latency conversation, memory processing and evaluation can each have different technical requirements.
That resembles how I increasingly think organizations should approach their own AI environments.
The question "Which model is best?" assumes there is one answer across every workflow. There rarely is. One model may provide stronger reasoning for a particular task while another returns results faster. Another may be significantly less expensive at scale. Another may have better access to a company's existing software environment. Another may be more useful for long-context analysis or multimodal work.
Once AI becomes embedded inside workflows, the model becomes one component in a larger architecture.
This also reduces the value of building an entire organizational AI strategy around loyalty to one vendor or interface. Companies should understand what capabilities they need, where those capabilities sit inside the workflow, what happens if a provider changes and how difficult it would be to replace one component without rebuilding everything around it.
The workflow should survive longer than the current favorite model.
AI capability is advancing faster than work is being redesigned
Toward the end of the episode, Farmer described a "capability overhang," the gap between what AI systems can already do and what ordinary users understand them to be capable of doing. I see that gap constantly.
Many professionals still use AI primarily to summarize, brainstorm, rewrite and draft. Those are legitimate uses. At the same time, AI systems can retrieve information from connected environments, compare multiple sources, execute sequences, create structured deliverables, monitor conditions, call tools and evaluate other models.
The organizational response has largely been to add these capabilities to existing jobs.
Employees are told to use AI. Managers are told to encourage adoption. Leaders buy platforms and request productivity gains. Yet the roles, approval structures, operating procedures, incentives and workload expectations surrounding the employee often remain untouched.
The result is predictable. AI becomes another layer of work to learn, supervise, troubleshoot and maintain while the original job continues underneath it.
That is why I think the adoption conversation is already too shallow for where the technology is going. The difficult question is no longer whether employees will use AI. Many already do. The difficult question is how organizations redesign work once AI can perform portions of that work with varying degrees of autonomy.
Who owns the workflow? Who evaluates the system? Which tasks disappear? Which tasks become supervisory? Which responsibilities move to people with deeper expertise? What new maintenance work is being created? What should the organization stop doing because AI has changed the economics of the process? Where should the organization deliberately refuse automation because the human involvement is itself valuable?
Those decisions cannot be delegated to the model.
The next phase of AI will be an operating design problem
The idea I keep returning to from the Portola conversation has little to do with alien companions.
Their team cannot control every conversation their AI will have, so they have learned to design the conditions around the conversation: the context the system receives, the memories it can retrieve, the boundaries it operates within, the situations it encounters, the models assigned to different functions and the evaluation systems used to determine whether the experience is performing as intended.
Businesses are moving toward the same problem.
Once AI participates inside a workflow rather than simply producing an isolated answer, the central questions change. What role does the system have? What information can it trust? What authority has it been given? What happens when sources conflict? How will failure be recognized? Which decisions require human approval? Who defines acceptable performance? Who owns maintenance when the environment changes?
Those are not prompting questions. They are questions of operational design, governance, expertise and organizational responsibility.
The companies that answer them well will not necessarily be the companies with the largest number of AI tools or the greatest number of internal agents. They will be the ones that know where AI belongs inside the work, where it does not, what conditions allow it to perform reliably and which human judgment remains too valuable to leave undefined.
That is the AI capability I want more leadership teams preparing for now.
