With the rapid rise of AI systems in our everyday lives, the question of ethics and transparency is becoming an increasingly important point of discussion. We as consumers really don’t have a clear understanding of the digital products and services that we use.
But a more alarming fact is that sometimes the developers of these systems themselves don’t fully understand how their systems work. This is because some advanced AI models are so called “Black Boxes”.
What is a Black Box AI?
A black box AI model is a system where the inputs and outputs are visible, but the internal mathematical process is ‘opaque’ and undecipherable to human logic. It essentially provides a final answer without providing any explanation for how it arrived at that conclusion, making it nearly impossible to audit or explain.
There are two different ways that black box AI models arise as stated in the IBM article on the subject. Either the developers make the models black boxes on purpose, or they become black boxes organically because of their complex training process.
Why would developers make black boxes on purpose? The main reason for this is to protect intellectual property. The creators of these systems know how they work, but choose to keep the source code and logic a trade secret.
The other way for an advanced AI system to become a black box is through machine learning. IBM labels these as so called “organic black boxes”. In these cases, deep learning systems (like complex neural networks) become so convoluted and vast during their training that the human creators themselves no longer understand the inner workings and logical steps of these systems.
The Real-World Impact
These black box AI systems can have immediate and negative impacts on people’s lives. Let’s take, for example, a university student who is applying for a master’s degree or a highly competitive job. What if they were perfectly qualified for the position, but a screening algorithm simply said “no”. If that algorithm was a black box system, the applicant would have no way of getting an explanation for that decision. In the database, they are just labeled as “unfit” for the position, with no way of effectively appealing the model’s decision.
This kind of example perfectly fits the EU’s AI acts “High-risk” category, which was created to prevent this exact scenario. As highlighted in the European Commission’s regulatory framework, these kinds of risks are not hypothetical sci-fi scenarios but realistic societal problems that must be tackled and regulated right now.
Towards a Transparent Future
While the current era of AI has been largely defined by the black box model of algorithms, the future we are moving towards seems to be more trasparrent and explainable.
One technical solution to the opaqueness of these algorithms is explainable AI (XAI). This approach, advocated by tech giants like IBM, focuses on creating systems that provide clear, understandable reasoning for their outputs. This would inspire more trust in these systems and allow humans to check for accuracy, accountability, and auditability.
The other crucial solution is AI governance and legislation. A good example is the newly established EU AI Act, which provides clear guidelines on the use of these systems. For instance, Article 86 of the AI Act would directly affect the earlier example of the student applying for work or studies. The article gives the student the legal right to demand a clear explanation from the deployer regarding the decision made by the automated system.
While the black box issue still persists and hasn’t been completely solved, we are moving towards a brighter future thanks to emerging transparent technologies (XAI) and more robust governance and legislation in the field.
