Pattern Is Not Understanding

The clock looked right

One of the Week 3 examples that stayed with me was surprisingly simple: an AI-generated watch.

The image looked convincing. The watch face was polished, the proportions looked right, and nothing immediately suggested that the model had failed. Yet the hands repeatedly appeared around 10:10 - a time commonly associated with watch advertising and product photography.

What mattered to me was not whether 10:10 is a valid time. Of course it is. The point was that the model appeared to have learned a very strong visual pattern for what a watch is supposed to look like. It could reproduce that familiar pattern remarkably well, but the image alone did not demonstrate that the system understood the underlying concept of time.

That small example gave me a much larger question to carry into professional work. A financial analysis can look polished. An accounting memo can sound technically sophisticated. A dashboard can look executive-ready. But a familiar professional form does not tell me whether the underlying population was complete, whether the right evidence was used, whether exceptions were considered, or whether the conclusion can be defended.

Week 3 gave me clearer language for something I had already begun to sense: pattern is not understanding, and capability is not reliability.

What a language model is actually doing

The technical lessons helped me understand why that distinction matters. Large language models process text as tokens and generate a response sequentially. At each step, the model uses the prompt and the tokens already generated to estimate which token could reasonably come next, selects a continuation, and repeats the process until the response is complete.

In one Python exercise in a Jupyter Notebook, we inspected log probabilities and the alternative tokens the model considered for a particular position. Seeing those probabilities changed the way I looked at the finished answer on the screen. A polished paragraph can feel as though the model first reached a complete conclusion and then simply presented it to me. The exercise made that impression much harder to maintain. I could see that the response was being constructed one token at a time from possible continuations.

That did not make the technology less impressive to me. It made me more precise about what a fluent answer does - and does not - prove.

The course also explored temperature, which controls how much freedom the model has when choosing among possible next tokens in systems that expose that setting. In plain language, a lower temperature tends to keep the model closer to the choices it already considers most likely, so responses are usually more consistent and predictable. A higher temperature allows the model to reach more often for less likely alternatives, creating more variation and, in many creative tasks, more originality or surprise. Neither setting creates truth. A model can repeat the same wrong answer very consistently, and a more creative answer can still be well supported or poorly supported depending on the task and the evidence.

That distinction changed the way I think about reproducibility. If the same prompt can produce different outputs, one successful run is weak evidence for a consequential business process. Even when outputs are highly repeatable, repeatability alone does not establish that the answer is complete, supported, or appropriate. Consistency is useful; reliability requires more.

A convincing output can still be the wrong kind of evidence

This is where the technical lesson turned into a familiar accounting instinct for me.

In my work, I would never accept a number merely because it arrived in the correct format. I would want to know where it came from, what assumptions produced it, what evidence supports it, whether the population is complete, and whether I can reproduce the result. I should not apply a lower standard simply because an AI answer is articulate or confident.

An AI system can produce something that resembles a forecast without validating the assumptions behind the forecast. It can summarize a contract without recognizing that an amendment changes the conclusion. It can draft a reconciliation explanation without confirming that the underlying population is complete. The output may fit the expected form while the underlying work remains incomplete.

Accountants already know this problem in another form. Presentation is not evidence. A workpaper does not become reliable because it is well formatted. A reconciliation does not become complete because the ending balance agrees. A journal entry can balance and still belong to the wrong period, entity, account, or population.

That is why the same professional skepticism we apply to financial work belongs in AI-assisted work. Familiar form should never substitute for support.

The test itself can distort what we think we measured

From there, Week 3 complicated the reliability question in a way I found especially interesting: sometimes the problem is not only the model. The way we test the model can influence the result too.

The course introduced research by Columbia Business School professors Melanie Brucks and Olivier Toubia. In their 2025 PLOS ONE paper, they examined what they call prompt architecture - the seemingly small design choices built into a prompt, such as the order of response options, the labels attached to those options, framing, and whether the model is asked to justify its answer. Their experiments found that some of these choices can systematically affect LLM responses. In other words, an evaluation can partly measure sensitivity to the test design rather than only the capability we intended to measure.

That finding immediately reminded me of control testing. If the design of the test can influence the result, then we need to evaluate not only the AI system but also the method we used to test it. A test can look rigorous and still be measuring, in part, an artifact of its own design.

For business teams, the implication is practical: do not treat one prompt, one run, or one conveniently successful scenario as decisive evidence. Where the risk warrants it, vary the test conditions, include normal and adverse cases, use known-answer cases where possible, and ask whether the evaluation itself could be steering the result.

The lesson is not that AI cannot be evaluated. It is that the evaluation deserves the same skepticism we apply to the output. That set up the most important question of the week for me: even if a system is intelligent, is it reliable enough to delegate work to?

Intelligence is not reliability

The most important shift for me came during the discussion of the reliability gap in real-world AI.

The analogy that stayed with me was not really about technology. It was about hiring a new employee. Intelligence matters, of course. But a manager also needs someone dependable enough to delegate to - someone who will complete the assignment, recognize when something is unclear, disclose uncertainty, and come back with a question instead of quietly filling in missing information.

I have worked with enough processes, deadlines, and reviews to know that I can work with a person who does not know everything. What I need is a person who knows when they do not know, and who knows when to ask for help.

That is reliability.

AI can invert that relationship. A model may have access to more information than a junior employee and may perform extraordinarily well on a difficult benchmark - a standardized test used to measure model performance - yet still be less dependable when the situation is ambiguous, the evidence conflicts, or the correct next step is to stop and escalate.

This is why I now separate capability from reliability. Capability shows that an AI system can complete a task. Reliability determines whether management can safely delegate the task.

The distinction also reminds me of control testing. A successful demonstration reminds me of a walkthrough: it shows that a process can work under selected conditions. It does not establish that the process operated consistently over time, handled exceptions correctly, preserved authorization boundaries, or escalated when evidence was insufficient.

When AI can act, the control question changes

Once AI moves from answering questions to taking actions, the reliability issue becomes much more consequential.

A chatbot that gives me a bad answer is one problem. An agent that can change a file, send an email, update a record, initiate a transaction, or continue a workflow is a different problem entirely. The same reasoning error that once produced an incorrect paragraph can become an operational event.

This was where my accounting brain became much louder.

We would not intentionally design a sensitive financial process in which the same person prepares a transaction, approves it, executes it, records it, and then independently verifies that everything was done correctly. Yet with agentic AI, it is surprisingly easy to create the technological equivalent of exactly that if we focus only on what the system is capable of doing.

The issue is not merely whether an agent has been told to follow policy. The stronger question is whether the surrounding system makes prohibited behavior technically possible - and whether the agent has more authority than the task actually requires.

An AI agent should not serve simultaneously as preparer, approver, executor, recorder, and sole reviewer of its own consequential activity. Segregation of duties still matters when the actor is software.

The Reliability Ladder

That reasoning led me to a simple framework for thinking about delegation: the Reliability Ladder.

Before delegating a business process to AI, I want to know not only whether the system can perform the task, but what level of authority the organization is actually giving it.

Retrieve means find the relevant information. Draft means generate a preliminary output. Recommend means evaluate alternatives and suggest an action. Decide means select an outcome with business consequences. Execute means carry out the action in a real system.

A reconciliation example makes the progression concrete. At Retrieve, an AI assistant could locate the reconciliation policy, transaction history, and supporting documents. At Draft, it could prepare a preliminary explanation of an open item. At Recommend, it could suggest whether the item should be investigated, corrected, or escalated. At Decide, it would select a defined disposition within approved rules when that level of authority is appropriate for the use case. At Execute, it could carry out the approved action in the real system - for example, posting an entry, clearing an item, or changing a record. The technology may be capable of moving through all five levels. The governance question is whether it should be allowed to.

The important point is not that every system should climb to the top. Quite the opposite. For many accounting and finance processes, the appropriate design may intentionally stop at Retrieve, Draft, or Recommend. More autonomy is not automatically better governance.

As authority increases, I would expect the evidence and controls to become stronger: better testing, clearer permissions, defined human approval points, stronger exception handling, independent logging, monitoring, reversibility, and explicit accountability. The question is not how autonomous the system can become. It is whether the evidence and control environment justify the authority being granted.

Responsible AI has to become operating discipline

The conversation with Paul Dongha of NatWest helped move these ideas from theory into the environment of a regulated financial institution. What stood out to me was that responsible AI cannot remain a collection of admirable principles. If the principles matter, they eventually have to become operating requirements.

That feels familiar from an internal-control perspective. 'Be fair,' 'protect privacy,' or 'keep a human involved' are useful intentions, but they are not controls by themselves. A principle becomes operational when someone owns it, when requirements are defined, when approval or testing occurs, when exceptions have a path, and when evidence exists that the process actually operated as intended.

The same principle applies when technology is purchased rather than built. A vendor may provide the model or platform, but management still owns how the system is configured, what data it receives, what authority it has, and what decisions the organization makes from its output. Outsourcing the technology does not outsource responsibility.

That is also why I find the National Institute of Standards and Technology, or NIST, useful as a reference point. NIST is an agency of the U.S. Department of Commerce. Its Artificial Intelligence Risk Management Framework, commonly called the AI RMF, is a voluntary and non-sector-specific framework designed to help organizations manage AI risks across the design, development, use, and evaluation of AI systems. From an accounting perspective, I think of it as a structured way to ask what could go wrong, how trustworthiness will be evaluated, and who is responsible for managing the risk.

NIST discusses characteristics such as validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy, and fairness. One short NIST statement captures the connection to my own thinking particularly well: 'Trustworthy AI depends upon accountability.' That is not an accounting rule, but the principle feels familiar. We can automate parts of the work, use outside providers, and delegate tasks to a model. Responsibility for the outcome still has to live somewhere human and identifiable.

Reliability begins before the model answers

The Python and data-analysis portion of Week 3 broadened the reliability question for me: sometimes the AI is not the first place to look when an answer goes wrong.

What interested me was not memorizing pandas commands. It was seeing how much analytical reliability depends on decisions made before the final calculation: What does one row represent? What is the population? Which fields are identifiers? Are dates and amounts being interpreted correctly? What happens to unmatched records when datasets are joined? Could duplicate keys multiply transactions? Could a filter silently remove the very exceptions we need to investigate?

Those questions are deeply familiar in accounting. A join can change completeness. A duplicate mapping can overstate a population. A missing value can mean zero, not applicable, unknown, or simply missing - and those are not interchangeable conclusions.

The broader insight is straightforward: reliable analytics requires disciplined definitions, transformations, reconciliations, and validation before a polished output ever reaches management.

Python and generative AI can work together, but they are not the same thing. Python is a programming language. It can be used for data analysis, exact calculations, automation, and classical machine-learning models. Generative AI systems, by contrast, generate new content and interpret natural-language instructions. In a well-designed finance workflow, Python might perform an exact calculation or apply a defined rule while a generative AI model helps interpret, summarize, draft, or communicate the result. Professional judgment still determines whether the data, method, evidence, and conclusion are appropriate for the decision.

What the CPA contributes

Week 3 did not leave me more skeptical of AI in the sense of wanting to use it less. The opposite may be closer to the truth. The more I learn about these systems, the more interested I become in what they can do. But I am becoming much more precise about what I mean when I say I trust one.

A CPA's contribution is not simply checking the last number. It is asking whether the population is complete, whether the evidence is authoritative, whether assumptions are visible, whether exceptions were handled correctly, whether the process can be reproduced, whether the result is defensible, and who remains accountable when other people rely on it.

Those habits become more valuable as AI becomes more capable because capability increases the scale and speed of both good decisions and bad ones. A single unsupported human assumption may affect one transaction. An automated system can repeat the same assumption across thousands.

That turns reliability from a productivity question into a governance question.

The workplace does not merely reward the ability to answer. It depends on the judgment to know when not to answer - and when to ask for help.

Pattern is not understanding

I still think about the watch.

What interests me now is not whether a model can make a beautiful image or produce an articulate explanation. Those capabilities are already useful. The professional question begins after that: Is the system using the right information? Does the result survive exceptions? Does it recognize uncertainty? Is its authority appropriately limited? Can the work be traced and challenged? Who remains accountable for the outcome?

That is the intelligence-reliability gap I now see much more clearly.

AI can help us retrieve, draft, recommend, and in some contexts decide or act. The future of AI in accounting and finance will depend not only on capability, but on designing processes that define where the system's authority should stop and what evidence is required before it moves further.

Capability tells us what AI can do. Reliability and control determine what we should allow it to do.

Reader resources

Reliability Ladder at a glance

Key takeaways

• A polished output is not evidence of accuracy or completeness; reliable AI-assisted work still depends on sound data, evidence, and professional judgment.
• Repeatability is not reliability, and evaluation design itself can influence what appears reliable.
• As AI moves from retrieving information toward deciding and executing, the required evidence, authorization, monitoring, and independent review should increase.
• Human accountability does not disappear when a model or vendor performs part of the work.

Questions I am still exploring

• What evidence should management require before moving an AI use case from one Reliability Ladder level to the next?
• Which finance activities should deliberately remain at Retrieve, Draft, or Recommend even when technology could technically execute them?
• How should organizations test an AI system’s ability to recognize insufficient evidence and escalate rather than guess?
• How should segregation of duties be redesigned when one AI system can interact with many tools and workflows?
• What should be retained as an audit trail when AI contributed to a recommendation or action?
• How should organizations reassess reliability as models, prompts, data, integrations, and business conditions change?

Further reading from my Week 3 learning

One part of the Week 3 conversation that I wanted to carry forward was the reading list. I bought these books myself because I wanted to continue learning beyond the course, and I am including them here for readers who may be on a similar path:

• AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference — Arvind Narayanan and Sayash Kapoor
• Human + Machine: Reimagining Work in the Age of AI — Paul R. Daugherty and H. James Wilson
• The Alignment Problem: Machine Learning and Human Values — Brian Christian
• System Error: Where Big Tech Went Wrong and How We Can Reboot — Rob Reich, Mehran Sahami, and Jeremy M. Weinstein

These titles were recommended during my Week 3 conversation with Paul Dongha, and I added them to my own reading list as part of continuing the learning beyond the course.

Selected public sources

1. Brucks, Melanie, and Olivier Toubia. “Prompt architecture induces methodological artifacts in large language models.” PLOS ONE 20(4), 2025. Read the original research
2. National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0), 2023. Official NIST publication
3. NIST AI Resource Center. “AI Risks and Trustworthiness,” including the accountability discussion cited in this article. Read the NIST trustworthiness section
4. NIST. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, 2024. Official NIST GenAI profile

Continue the conversation

Explore The Reliability Ladder, then use Practice Guide No. 003 - The AI Delegation Review to document the maximum level of authority an AI use case is ready to receive. Related future topics include AI agents and segregation of duties, testing AI like a control, and responsible AI as an operating system.

About the author

Valentina DuPont, CPA is an accounting professional specializing in financial reporting, internal controls, and the practical application of emerging technologies in finance. Through the Knowledge Hub, she explores how professional judgment, governance, and AI intersect in modern business.

© 2026 Valentina DuPont, CPA. All rights reserved.

Related Knowledge Hub Materials