Phalanx and the AI quandary
Thread Starter
Join Date: Feb 2006
Location: Hanging off the end of a thread
Posts: 33,047
Received 2,920 Likes
on
1,249 Posts
Phalanx and the AI quandary
This is getting a farce, we might as well just surrender now…
https://ukdefencejournal.org.uk/uk-f...dgetail-order/
The session included testimony from Professor Mariarosaria Taddeo, Dr. Alexander Blanchard, and Verity Coyle, who examined the implications of AI in defence and security.
Professor Taddeo highlighted three main issues with the implementation of AI in weapons systems, stating, “We need to take a step back here, because it is important to understand that, when we talk about artificial intelligence, we are not just talking about a new tool like any other digital technology. It is a form of agency.”
She emphasised concerns regarding the limited predictability of outcomes, difficulty attributing responsibility, and the potential for AI systems to perpetrate mistakes more effectively than humans. Taddeo argued that the unpredictability issue is intrinsic to the technology itself and unlikely to be resolved.
Verity Coyle, a Senior Campaigner/Adviser at Amnesty International, emphasized the potential human rights concerns raised by autonomous weapons systems (AWS), saying, “The use of AWS, whether in armed conflict or in peacetime, implicates and threatens to undermine fundamental elements of international human rights law, including the right to life, the right to remedy, and the principle of human dignity.”
She argued that without meaningful human control over the use of force, AWS cannot be used in compliance with international humanitarian law (IHL) and international human rights law (IHRL).
During the session, Verity Coyle provided an example of an existing AWS, the Kargu-2 drones deployed by Turkey, which have autonomous functions that can be switched on and off. She warned that, “We are on a razor’s edge in terms of how close we are to these systems being operational and deadly.”
In response to questions about existing AI-driven defence systems, such as the Phalanx used by the Royal Navy, Coyle stated, “If it is targeting humans, yes,” indicating that any system targeting humans should be banned.
The experts recommended the establishment of a legally binding instrument that mandates meaningful human control over the use of force and prohibits certain systems, particularly those that target human beings.
https://ukdefencejournal.org.uk/uk-f...dgetail-order/
In a recent AI in Weapons Systems Committee session, experts debated the ethical, legal, and technical concerns of AI in weapons like the Royal Navy’s Phalanx system, discussing potential bans on specific autonomous systems.
The Artificial Intelligence in Weapons Systems Committee recently held a public evidence session, inviting experts to discuss ethical and legal concerns surrounding the use of AI in weaponry.The session included testimony from Professor Mariarosaria Taddeo, Dr. Alexander Blanchard, and Verity Coyle, who examined the implications of AI in defence and security.
Professor Taddeo highlighted three main issues with the implementation of AI in weapons systems, stating, “We need to take a step back here, because it is important to understand that, when we talk about artificial intelligence, we are not just talking about a new tool like any other digital technology. It is a form of agency.”
She emphasised concerns regarding the limited predictability of outcomes, difficulty attributing responsibility, and the potential for AI systems to perpetrate mistakes more effectively than humans. Taddeo argued that the unpredictability issue is intrinsic to the technology itself and unlikely to be resolved.
Verity Coyle, a Senior Campaigner/Adviser at Amnesty International, emphasized the potential human rights concerns raised by autonomous weapons systems (AWS), saying, “The use of AWS, whether in armed conflict or in peacetime, implicates and threatens to undermine fundamental elements of international human rights law, including the right to life, the right to remedy, and the principle of human dignity.”
She argued that without meaningful human control over the use of force, AWS cannot be used in compliance with international humanitarian law (IHL) and international human rights law (IHRL).
During the session, Verity Coyle provided an example of an existing AWS, the Kargu-2 drones deployed by Turkey, which have autonomous functions that can be switched on and off. She warned that, “We are on a razor’s edge in terms of how close we are to these systems being operational and deadly.”
In response to questions about existing AI-driven defence systems, such as the Phalanx used by the Royal Navy, Coyle stated, “If it is targeting humans, yes,” indicating that any system targeting humans should be banned.
The experts recommended the establishment of a legally binding instrument that mandates meaningful human control over the use of force and prohibits certain systems, particularly those that target human beings.
this has been an issue since Asimov wrote I Robot. How much can you trust the machines?
More worrying was a story - I think it might have been Eric Frank Russell - about some poor sod who gots caught in a completely machine driven problem over a missing murder mystery book which the machines gradually escalate, quite logically, to finding him guilty of murder and having him executed....................
This is getting a farce, we might as well just surrender now…
https://ukdefencejournal.org.uk/uk-f...dgetail-order/
https://ukdefencejournal.org.uk/uk-f...dgetail-order/
Last edited by beardy; 6th May 2023 at 19:35.
Doesn't Phalanx have an Off-Switch? Operated by a human?
You mean like loitering munitions or mines?
Basically, like 'self driving cars' someone still has to be held liable when there is a crash. Who is it? hopefully there's a person behind the wheel (as you must be ready to take back control at any time....) or the manufacturer if its operating in full self drive mode?
A cynic might say the self drive is programmed to hand back control of the car before the crash so that the driver is responsible for it...
Same for any other autonomous system - who owns the risk of it going wrong?
A cynic might say the self drive is programmed to hand back control of the car before the crash so that the driver is responsible for it...
Same for any other autonomous system - who owns the risk of it going wrong?
I suppose something like the CBU-97 would fit the bill. Deploys 10 sub-munitions which each deploy 4 individually self-targeting sub-sub-munitions; "skeets". Each skeet searches using a laser sensor and height/contour algorithms to detect a "target", and attacks with an Explosively Formed Penetrator. Apparently it has controversies of its own, but largely based on its characteristics involving the term "cluster".
I'm pretty sure this isn't the only dog of war which is designed to detect, select and attack a target after being let-slip by a human.
I'm pretty sure this isn't the only dog of war which is designed to detect, select and attack a target after being let-slip by a human.
For a determinate system i.e. you know what it will do (very nearly) every time you turn it on (1e-9 failure rate - like systems in an airliner) then sure, for the most part that is fine. Problem is, AI isn't a determinate system as we understand it. There's nothing to say it will do as it is expected to do. Every time it is switched on it has the opportunity to give a different answer. So leaving it alone to determine what it should attempt to kill is a pretty risky idea. We already have enough problems with algorithms trying to understand traffic on smart motorways, lets not give them guns.
When an AI system does something unexpected, its difficult to troubleshoot - maybe even impossible. If the outcome of that unexpected action is fatal, expensive or even just embarrassing the finger pointing starts. Someone needs to be the risk holder. I know for a fact I'd never take that on.
The whole point of the AI system is that it can be left alone to do its job - potentially faster and 'better' than the human it was supposed to replace. If you require a human to nanny it, then its value proposition tanks.
Slightly off topic, but it is my understanding that a report last year by the Law Commission of England, Wales and Scotchland, suggested that the legal responsibility for accidents caused by self-driving vehicles should rest not with the person in the driver’s seat, but with the company or body that obtained authorization for the self-driving features used by the vehicle. To add to the confusion, there is currently no legal definition in the UK for what constitutes a 'Self Driving Vehicle'.
Back on topic. Twenty years ago I went to see 'Terminator 3, Rise of the Machines' and I immediately thought the opening sequence was where the world was heading and my view has not changed since.
Back on topic. Twenty years ago I went to see 'Terminator 3, Rise of the Machines' and I immediately thought the opening sequence was where the world was heading and my view has not changed since.
It's a good start, however the person turning it on is likely to be the least liable - they're likely to know the least about how the system works and potentially have the least say in its operation.
For a determinate system i.e. you know what it will do (very nearly) every time you turn it on (1e-9 failure rate - like systems in an airliner) then sure, for the most part that is fine. Problem is, AI isn't a determinate system as we understand it. There's nothing to say it will do as it is expected to do. Every time it is switched on it has the opportunity to give a different answer. So leaving it alone to determine what it should attempt to kill is a pretty risky idea. We already have enough problems with algorithms trying to understand traffic on smart motorways, lets not give them guns.
When an AI system does something unexpected, its difficult to troubleshoot - maybe even impossible. If the outcome of that unexpected action is fatal, expensive or even just embarrassing the finger pointing starts. Someone needs to be the risk holder. I know for a fact I'd never take that on.
The whole point of the AI system is that it can be left alone to do its job - potentially faster and 'better' than the human it was supposed to replace. If you require a human to nanny it, then its value proposition tanks.
For a determinate system i.e. you know what it will do (very nearly) every time you turn it on (1e-9 failure rate - like systems in an airliner) then sure, for the most part that is fine. Problem is, AI isn't a determinate system as we understand it. There's nothing to say it will do as it is expected to do. Every time it is switched on it has the opportunity to give a different answer. So leaving it alone to determine what it should attempt to kill is a pretty risky idea. We already have enough problems with algorithms trying to understand traffic on smart motorways, lets not give them guns.
When an AI system does something unexpected, its difficult to troubleshoot - maybe even impossible. If the outcome of that unexpected action is fatal, expensive or even just embarrassing the finger pointing starts. Someone needs to be the risk holder. I know for a fact I'd never take that on.
The whole point of the AI system is that it can be left alone to do its job - potentially faster and 'better' than the human it was supposed to replace. If you require a human to nanny it, then its value proposition tanks.
Yes of course it is. Dumb munitions don't possess a self generated intent, they do what they are designed and built for, their ROE are designed in and they are deployed knowing their capabilities and limitations. AI develops its own intent and ROE, that's the point of AI. The difficulty at the moment is to determining how it develops and deploys itself, the process is opaque and could be too fast to be overridden by a human. Of course you could have a supervisory AI monitoring the operational AI.
Basically, like 'self driving cars' someone still has to be held liable when there is a crash. Who is it? hopefully there's a person behind the wheel (as you must be ready to take back control at any time....) or the manufacturer if its operating in full self drive mode?
A cynic might say the self drive is programmed to hand back control of the car before the crash so that the driver is responsible for it...
Same for any other autonomous system - who owns the risk of it going wrong?
A cynic might say the self drive is programmed to hand back control of the car before the crash so that the driver is responsible for it...
Same for any other autonomous system - who owns the risk of it going wrong?