What does the release of ChatGPT have in common with the Chernobyl plant disaster?
Dr Alex Mikhalev, Managing Partner of Zestic AI
At Zestic AI, we believe that AI is a powerful force, which, if used correctly and ethically, can and will change our lives as well as our planet for the better. However, now is the time to start differentiating between ‘Good AI’ and ‘Bad AI’. The earlier we put the guardrails in place, the better. Consider the Chernobyl plant disaster - could a wide adoption of AI trigger a comparable disaster? You might think this comparison is far-fetched, however, having worked on AI/ML for the past 20 years, we don’t think so…
As many of us who have tried ChatGPT / GPT4 can attest, it is an impressive and exciting tool. If run locally, it is safe. During its R&D phase, it has largely remained in the information world without touching reality.
However, now things are different.
We are in an extremely dangerous situation, as Large Language Model (LLM) - based AI systems are being deployed into production with no constraints. No additional capabilities are required to cause an industrial catastrophe on a large scale: with humans in the loop, AI doesn’t need to escape from the box — it has already done so when it was plugged into Bing.
One of the likely paths would be humans following instructions generated by AI, which they have not properly verified.
When given shortcuts and prompts, we often follow them without necessary sense checks. It ranges from blindly following our car navigators to, on some occasions, pilots executing wrong instructions from an autopilot or a beacon system, and performing a controlled flight into terrain (CFIT).
So, unlike in disaster movies, AI doesn’t need to devise a cunning plot to take over a large piece of infrastructure, for instance, a nuclear power plant — humans could create an accident themselves, while following incorrect or not properly verified (AI-generated) instructions. Currently we have a stable system, but a wider use of ChatGPT and LLMs increase the risks to the existing infrastructure.
And as we all know, people do stupid things all the time, even while operating safety-critical systems.
For example, 5,000 pilots were investigated for allegedly misinforming the regulator (FAA) about their health conditions, according to a recent New York Post article. Sixty pilots “posed a clear danger to aviation safety” and were ordered to cease flying while their records were reviewed. ChatGPT and other LLMs make such things easier, eagerly coming up with suggestions on what to say and what not to say. They can simulate a medical commission interview for you as many times as you wish until you feel safe to pass it with flying colours, even if you are not qualified.
This behaviour has also been observed in job interviews [1], [2]. The Glider.ai report says that in 2019, 21% of candidates engaged in various forms of cheating,” and the situation worsens as the economic climate goes down and competition on the market increases.
Consider the cause of the Chernobyl plant disaster:
People ran a performance acceptance test when they disabled the reactor’s power-regulating system, disabled the safety mechanism and failed to re-enable both mechanisms. A compounding of human mistakes has led to a total meltdown. We are still living with grim consequences of the disaster.
ChatGPT/Open AI-based Bing and hundreds of ‘me too’ LLM clones can potentially create a similar scenario. Just imagine the following - an LLM only needs to:
Convince a nuclear plant management to run productivity improvement experiments
Generate an experiment schedule and actions
Generate safety procedures for the critical infrastructure
Such LLMs already have all the capabilities required, however, it has been shown that the current versions of LLMs have a major flaw of being able to ‘make stuff up', or ‘hallucinate’.
Such systems have already been exposed to hundreds of millions of people in a live environment without proper guardrails.
We don’t track how the released LLMs are influencing people.
Many professionals across all levels of seniority and industries have been experimenting with using LLMs, in particular ChatGPT and Bard - with mixed results.
The case of nuclear plants, something we all can imagine, can be replaced with any other critical infrastructure - there are myriads of potentially disastrous errors accumulating in different areas thanks to irresponsible vendors and marketing hype.
Unguarded AI is extremely dangerous when applied to decision making, as it can deceive or unduly influence hundreds of millions of people.
Most importantly, this type of AI does not need additional capabilities to cause harm:
Within Bing, its LLM is now connected to the largest knowledge graph in the world. It doesn’t need to model real-world concepts — these are given to AI by humans.
Humans interacting with chat are now part of the reinforcement bidirectional learning loop, where on one side, humans teach AI. However, on the other side, AI can apply reinforcement learning to many people, removing their agency — “free will” at a large scale. All misinformation (hallucinations) can be magnified on an enormous scale with no monitoring or controls attached.
It is clear that we do need to act now to prevent such dramatic scenarios from happening, and the role of regulators is key to deploying AI safely and responsibly.
At Zestic AI, we design and implement ‘ethical AI’ solutions - the AI you can trust - privacy-first, secure and tailored to your needs. All without breaking a bank. Please get in contact with us to find out more.