A Finance Professional’s Guide to Large Language Models

Large language models — the AI systems behind tools such as ChatGPT, Claude and Microsoft Copilot — are the technology at the heart of most of the AI a finance professional will now encounter. You do not need the technical detail to use them well, but a basic, conceptual understanding of what these systems are, why they behave as they do and what their capabilities and limitations mean in practice helps you use them sensibly and avoid the common pitfalls. This guide provides that understanding, written for finance professionals rather than technologists.

It covers what large language models are, how they work at a conceptual level (including tokens, training and the context window), what that means for their capabilities and limitations, why they hallucinate, how retrieval, reasoning and agents change the picture, and what all of this means for finance work, governance and skills. It deliberately does not repeat our other AI guides: for choosing a product, see Claude vs Copilot vs ChatGPT for finance; for writing better instructions, prompt engineering for finance; for confidential data, data security when using AI; and for rules of use, our AI usage policy for finance teams.

What Large Language Models Are

A large language model is an AI system trained to work with language — to understand and generate text — by learning patterns from enormous amounts of written material. From that training it has learned the patterns of language and much of the information that language contains, which lets it answer questions, summarise, explain, draft and perform many other text tasks. When you use ChatGPT, Claude or Copilot, you give a model text (a prompt) and it generates text in response, based on those learned patterns.

The key point is that these are fundamentally language systems, generating plausible, well-formed text in response to what they are given. That is what makes them so capable with language, and it is also the source of their limitations. A large language model is not a database of facts, not a calculator and not a reasoning engine in the way a person reasons; it is a system that generates plausible text from learned patterns. Grasping that distinction is the foundation for everything else in this guide.

How They Work at a Conceptual Level

At a conceptual level, a large language model generates text that is a plausible continuation of, or response to, the text it is given. In effect it predicts what should come next, one small piece at a time, based on everything in front of it and the patterns it learned in training. It is producing what is plausible rather than retrieving a stored fact or reasoning from first principles. Plausible text is very often correct, which is why these tools are so useful — but it is generated for plausibility, not guaranteed for accuracy.

That one idea explains a great deal. It explains the fluency, because the model is producing well-formed text. It explains the breadth, because it learned from material spanning almost every subject. And it explains why the model can be confidently wrong: an incorrect answer can be just as plausible-sounding as a correct one, and it is produced with the same fluency. A few further concepts fill in the picture.

Tokens

Models do not read words or digits the way we do. Text is broken into “tokens” — chunks that may be a whole word, part of a word or a few characters. A common rule of thumb is that a token is roughly three-quarters of an English word, though it varies by model and language. Numbers are often split into several tokens in ways that do not respect place value, which is one reason a model can mangle a long figure or slip on arithmetic that a spreadsheet would never get wrong. Tool limits, speed and cost are usually measured in tokens.

Training

Training happens in broad stages. In pre-training, the model learns to predict text across a vast body of written material, which is where its general knowledge and language ability come from. It is then refined — through further training on examples and on human and automated feedback — to follow instructions, hold a conversation and decline unsafe requests. Two practical consequences follow. First, a model’s built-in knowledge stops at its training cut-off, so it may not know about a recent tax change, accounting amendment or regulatory deadline unless it is given that information. Second, the model does not remember your organisation’s data from its training; anything specific to your business has to be supplied in the conversation or through a connected system.

The context window

The context window is the amount of text a model can consider at once: your instructions, any documents you attach, the conversation so far and its own reply. Think of it as the model’s working desk. Context windows in current leading tools can be large — often enough to hold a full annual report — but they vary by product and plan, and models can attend less carefully to material buried in the middle of a very long input. When a conversation runs long, earlier material may be summarised or dropped. For finance work, the practical rule is to give the model the specific documents it needs, point it at the relevant sections, and start a fresh conversation for a new task.

Why the same question can get different answers

Generation involves an element of chance: at each step the model chooses among likely next tokens rather than always taking the single most likely one. That makes the output more natural, but it also means two runs of the same prompt can produce different wording — and occasionally different conclusions. For anything you rely on, that variability is another reason to verify rather than trust a single answer.

What This Means for Their Capabilities

The way large language models work gives them genuine capabilities that are useful in finance. Because they are trained on vast text and generate fluent language, they are good at language tasks — drafting, summarising, explaining, rephrasing and working with written material — and much of finance work involves text. Because they learned from material spanning many subjects, they are broadly knowledgeable. And because they respond to instructions in natural language, they are flexible: the same tool can draft variance commentary, summarise a lease, explain a technical point to a non-finance manager or suggest the structure of a board paper.

In practice, that makes them strong at the text-and-language side of finance — first drafts of variance commentary and management reporting narrative, summaries of contracts and policies, explanations and checklists — and useful as a thinking partner when structuring analysis. The capabilities are real and flow directly from how the systems work.

What It Means for Their Limitations

The same mechanics create limitations that matter a great deal in finance. The most important is that models can be confidently wrong: because they generate plausible text rather than guaranteed facts, they can produce output that is fluent and confident but incorrect, sometimes fabricating information that sounds right. This is a structural feature, not an occasional glitch, and it means output cannot be trusted uncritically — especially where accuracy matters, which in finance is almost everywhere.

Other limitations follow. Models do not understand or reason in the way a person does, which limits their reliability on complex judgement. They can be inconsistent, as described above. Their knowledge may be incomplete, out of date or wrong on specifics. They are weak at precise arithmetic unless they use a calculation tool. And they generate an answer even when they lack a sound basis for one. Taken together, these mean a large language model is a useful tool with significant constraints — an assistant whose work must be checked, not an oracle.

Why They Hallucinate

“Hallucination” is the common term for confident output that is not supported by fact or by the source provided. It happens because the model’s job is to produce a plausible continuation, and a plausible-looking citation, figure or paragraph reference is exactly the kind of thing it can generate when the real answer is not in front of it. Training rewards helpful, complete-looking answers, so a model may fill a gap rather than say “I don’t know”. Developers have reduced hallucination considerably, and tools increasingly cite sources, but it has not been eliminated.

In finance, the classic patterns are invented figures, wrong paragraph or section references to accounting standards and tax legislation, case law or guidance that does not exist, out-of-date thresholds stated as current, and misread tables. Hallucination is most likely when you ask from memory about niche or recent topics, and least likely when you supply the document and ask the model to answer only from it. Our guide to AI-assisted financial statement analysis shows how this plays out on a real set of accounts.

Retrieval: Grounding Answers in Your Own Documents

Retrieval — often called retrieval-augmented generation, or RAG — is the main technique for making a model answer from the right information. Instead of relying on what it absorbed in training, the system first searches a defined set of documents (your policies, contracts, the finance manual, last year’s audit file) or the web, places the most relevant passages into the context window, and asks the model to answer from them. When a tool shows you citations to the documents it used, retrieval is usually what is happening.

Retrieval reduces hallucination but does not remove it. The search can miss the relevant passage, retrieve an out-of-date version, or pull a passage that looks relevant but is not; and the model can still misread what it is given. The quality of the answer depends heavily on the quality and currency of the documents behind it — which is why data quality in the finance function matters more, not less, once AI is involved. Connecting a model to finance systems also raises access-control questions; see connecting AI to finance data safely.

Reasoning Models

Many current tools offer a “reasoning” or “thinking” mode. These models are trained to work through a problem in intermediate steps before giving a final answer, spending more computation on harder questions. In practice they tend to perform better on multi-step tasks — reconciling a set of figures, working through a technical accounting question, checking a model’s logic — than the same tools answering instantly.

Two cautions. First, better is not infallible: reasoning models still make errors, and a long, confident chain of steps can make a wrong conclusion more persuasive. Second, the step-by-step explanation a model displays is not necessarily a faithful record of how it reached its answer, so it should not be treated as an audit trail. Use reasoning modes for genuinely multi-step work, and verify the conclusion against the source as you would any other output.

Agents and Tool Use

An AI agent is a model that can use tools and take steps towards a goal rather than simply replying in text: running code to do a calculation, searching the web, reading and writing files, querying a database or acting in a business system. The model decides which tool to call, looks at the result and decides what to do next, sometimes over many steps. Tool use addresses some weaknesses directly — a model that runs code to calculate is far more reliable on arithmetic than one that generates the numbers as text.

Agents also change the risk profile. A chat assistant that drafts a wrong paragraph wastes a few minutes; an agent that posts a journal, sends a payment file or emails a supplier can cause real loss. The control principles finance already uses — segregation of duties, authorisation limits, review before posting, audit trails — apply directly. Our guide to human-in-the-loop AI controls covers how to design them, and AI automation for management accountants looks at where automation is already practical.

What the Mechanics Mean for Finance Work

Putting the concepts together gives a practical map of where large language models fit in a finance function. It is a guide to fit, not a guarantee — capabilities differ between tools and are changing quickly.

How to Use Them Well

Understanding how large language models work leads directly to using them well. Use them for what they are good at — drafting, summarising, explaining and broad-knowledge assistance — and not for what they cannot reliably do. Always verify their output, particularly anything factual or consequential. Keep your own judgement, understanding and accountability firmly in place: the model is an assistant, not a replacement.

Using them well also means interacting effectively — clear instructions, the right context and documents, and iteration — which our prompt engineering guide and finance prompt library cover in detail. The mechanics above add a few rules of their own: give the model the source rather than asking from memory; ask it to say when the answer is not in the material; use a calculation tool or spreadsheet for numbers; start a new conversation for a new task; and choose a reasoning mode for multi-step problems. A finance professional who uses these tools as capable but fallible assistants captures their value safely; one who trusts them uncritically courts the errors their limitations produce. For a broader view, see our guide on what’s real and what’s hype in AI.

Why the Fluency Can Be Misleading

The fluency of large language models can create an impression of reliability and understanding that is not always warranted. We are used to associating fluent, confident expression with competence in people, so polished output naturally feels trustworthy. With a model, fluency reflects its skill at generating well-formed text, not the accuracy of the content: it can present wrong information just as fluently as right information.

So do not let the fluency lull you into trusting the output. Confident, articulate presentation is not evidence of accuracy, and a neatly formatted table is not evidence that the numbers in it are right. A finance professional who understands this is less likely to over-trust the output and more likely to apply the verification the content actually needs. The fluency is impressive, but it is a feature of how the system writes, not a signal of reliability.

Governance: What Finance Leaders Need in Place

Understanding the technology makes governance more practical, because each mechanism maps to a control. Hallucination calls for review and verification standards; training cut-offs call for grounding in current sources; context and retrieval call for data classification and access control; agents call for authorisation and audit trails. A proportionate framework for a finance function usually covers:

• Approved tools and data rules — which tools may be used, with what classes of data, set out in a written AI usage policy.
• Accountability — a named owner for each AI-assisted process, and the principle that the person who signs off owns the output.
• Review standards — what must be checked, by whom, before AI-assisted work is relied on or leaves the team.
• Records — enough documentation to show auditors and boards what was AI-assisted and how it was checked; see what auditors ask about AI and what boards are asking about AI in finance.
• Data protection — where personal data is involved, UK GDPR applies; the ICO’s guidance on AI and data protection is the reference point.

The UK has so far taken a principles-based approach, relying on existing regulators rather than a single AI statute, as set out in the government’s AI regulation white paper; the position may change, so check for current developments. For FCA-regulated firms, the FCA has said it does not plan AI-specific rules and will rely on existing frameworks, notably the Consumer Duty and the Senior Managers and Certification Regime — see AI and the FCA: our approach. In practice, that means a senior manager remains accountable for how AI is used in their area. Firms needing senior compliance or SMF appointments to oversee this can speak to our sister firm SMF Capital.

Skills Finance Professionals Need

Nobody in a finance team needs to build a model, but the most effective finance professionals now combine traditional technical strength with a working understanding of these tools. The skills employers increasingly ask about are:

• Conceptual literacy — knowing, in the terms of this guide, why a model might be wrong and what kind of task it suits.
• Verification discipline — checking outputs against source data efficiently and knowing when a check is not optional.
• Problem framing — breaking work into steps a tool can help with and supplying the right context.
• Data judgement — knowing what data can go where under the firm’s policy.
• Process and control design — redesigning workflows around AI without weakening controls, which is increasingly part of the financial controller’s role.
• Core technical expertise — still the foundation, because you can only verify what you understand.

The professional bodies are building resources in this area; the ICAEW’s generative AI guide is a useful starting point. For employers, our AI-ready finance recruitment page explains how we assess these skills, and how AI is changing what employers look for in an FC covers the shift at controller level.

Frequently Asked Questions

Do large language models look up answers in a database?

Not by default. A model generates text from patterns learned in training. Some tools add retrieval — searching documents or the web and feeding the results to the model — which helps ground answers, but the model can still misread or misreport what it retrieves.

Why are LLMs bad at arithmetic?

Because they generate numbers as text, and numbers are split into tokens in ways that do not follow place value. Tools that run code or call a calculator are much more reliable. For finance, treat any number produced in a chat reply as unverified until a spreadsheet or the source agrees.

Will hallucination be solved?

It has reduced considerably and continues to improve, particularly with retrieval and citations, but it is a consequence of how the technology generates text, so it is prudent to assume some residual risk and keep verification in place for anything that matters.

Does an AI tool learn from the data I put into it?

It depends on the product and the terms of your licence. Some consumer services may use inputs to improve models unless you opt out; business and enterprise agreements commonly exclude this. Check your organisation’s approved tools and our guide to AI data security in finance before entering confidential data.

What is the difference between a chatbot and an agent?

A chatbot replies in text and leaves action to you. An agent can use tools — run code, read files, query or update systems — and take several steps on its own. Agents can save more time but need stronger controls, because their mistakes can become actions.

Do finance professionals need to learn to code to use AI?

No. Conceptual understanding, verification discipline and strong technical accounting matter more. Some familiarity with spreadsheets, data and how tools connect to systems is increasingly useful, particularly in FP&A and systems-facing roles.