I have written software professionally for more than twenty years, through several rounds of technology that was going to change everything. This is an account of why I remain skeptical of AI as it is currently sold, and why that skepticism is the reason we are comfortable implementing it.
Two definitions
Two ideas underpin everything that follows.
Feeling foolish is the result of a hidden assumption. You reach for a hook that someone has moved, and your keys land on the floor. You fill a diesel engine with gasoline because nobody told you it was diesel. In both cases the cost is caused by something you did not know was there. In software, the expensive version of this is deploying a system whose behavior you have not actually understood.
A system works only if it runs without supervision. Every machine needs maintenance, but maintenance is a known, scheduled action: change the oil, rotate the keys, patch the server. If an application needs to be nudged, restarted, corrected or babysat to do its ordinary job, it is not a working application. It is a broken one with an operator attached.
These two definitions are the standard against which I judge any tool, including this one.
What a computer is
A computer is a device that gathers, stores, processes and transmits information. Processing means executing a precise sequence of instructions. That precision is the entire value of the machine.
Programming, properly done, is an exact discipline. I can predict what a given instruction will do. I can state the reliability of a system within defined bounds. There are established methods for fault tolerance, for real-time guarantees in aircraft and telecom equipment, and for the arithmetic in banking systems that computes balances and fees so precisely that nobody notices it. A computer can repeat a task indefinitely without deviation, and that is why we built it. The word "computer" once described the people who did arithmetic by hand for aerospace and military programs. We replaced them because the machine does not drift.
What a language model adds
Under the product names, a large language model is matrix arithmetic with controlled randomness introduced into the output. That randomness is what lets it recombine language in useful ways. It is also a deliberate departure from the definition above. A computer is supposed to return an exact answer. A model returns a probable one.
This is not a criticism of the technology. It is a description of it, and it has consequences that no amount of investment removes:
- No model can guarantee a correct answer. That limits its use wherever correctness is the requirement rather than a preference.
- A model-driven system needs care. It must be checked, corrected and constrained, which means that by the second definition above it is not, on its own, a working system.
- The output is fluent whether it is right or wrong, so the errors are harder to see than the errors of a conventional program.
The practical risk follows directly. A single misread command in a generated script can delete a working directory or an entire filesystem. Generated code reaches production unreviewed because it looked plausible and the deadline was real. In banking and payments, where I do most of my work, the people responsible for the ledger understand this instinctively. When someone suggests putting a model on the money path, they do not need the argument explained to them.
Where the technology is genuinely useful
Skepticism about the pitch is not dismissal of the tool. There are tasks where a probabilistic answer is exactly what is wanted.
- Large-scale pattern work. Predictive models over big datasets, weather being the obvious example, produce more nuanced probabilities than we had before. They are still probabilities.
- Text transformation. Rewriting for tone, changing register, producing a first-pass translation that a person then corrects. These tasks were never exact, and the model does them far better than a global find-and-replace.
- Reading at volume. Summarizing, classifying and extracting from documents that a person previously had to read one by one.
What these have in common is that a nearly right answer is valuable and a person checks the result. Where a nearly right answer is worthless, such as a settlement file, a balance, or a command executed with elevated privileges, the model does not belong in the loop without deterministic code and a person around it.
The human cost
The area where over-reliance does the most damage is anything involving people. Screening candidates is the clearest case. With thousands of applications per opening, some automation is unavoidable. But every rejection a model issues has a reason that nobody examined. The system will admit a candidate who interviews well on paper and reject an excellent engineer who does not happen to match a pattern. The only remedy is the one that has always worked: a person talks to the candidate.
The balance
The claim in the market is that AI is a gold mine with no downside. The reality is that it is another technology that must be studied, understood and applied with judgment. Ignore it and you will be slow. Apply it everywhere and you will spend your time debugging plausible nonsense. The balance is specific to each task, and finding it is engineering work, not a purchasing decision.
That is the position this firm implements from. We expect the model to be wrong, we design so that it does not matter when it is, and we say plainly when a task does not need a model at all. Before accepting any vendor's claim that AI must be everywhere, ask to see the sales figures and the earnings of the companies making it.