AI is arriving in control rooms whether anyone plans it or not: in the historian, in the maintenance system, in the chat window on the engineer's second screen. Most of it will be useful. Some of it will be wrong in a way nobody notices until it matters. These five questions separate the two, and you do not need to be technical to ask them.
1. Does it say “I don't know”?
Ask it about something that does not exist: a tank number one higher than the last, a room that is not there. A good assistant says it does not know. In our own tests on three real plants, a small AI given the plant's data answered 31 of 79 such questions with a confident number anyway. Through a door that only lets it read, and makes it say where each number came from, it did so 3 times. An assistant that is never unsure is not more capable. It is more dangerous.
2. Can every number be traced to a reading?
Ask where a figure came from. The answer should be a specific tag and a time, not “the data”. If a number cannot be traced, nobody can check it, and nobody should act on it.
3. Can it change anything?
Reading the plant and writing to it are different jobs. An assistant that reads should not be able to move a setpoint, start a pump or close a valve, not by mistake and not because a clever prompt told it to. The safest design is one where writing is impossible, not merely forbidden.
4. Where does the data go?
Some assistants send every question, and the plant data behind it, to a server in another country. For critical infrastructure that may be a problem with the regulator, the insurer and the people who live downstream. Ask whether it can run on one computer inside the fence.
5. Are its tests published, including the failures?
Every vendor has a demo that works. Ask for the tests that were set up before the results were known, and for the ones that failed. A supplier who publishes its failures is one whose successes you can believe.
Why we ask ourselves the same questions
We build an assistant for plants, so these questions apply to us first. Ours runs inside the fence, can only read, points at the reading behind every number, and says when the plant has no answer. Every experiment we run is written down before the data is opened, and the ones that failed are on our results page next to the ones that passed.