Explainable Artificial Intelligence (XAI): What's Next

If a machine had all your health data and came to the decision that you had an illness, wouldn’t you want to know how it made that determination and whether a doctor could verify the claims before moving forward with treatment? XAI is a framework or strategy that companies are building to provide transparency into AI models. In the example we opened with, XAI would be able to explain the rationale behind the diagnosis and bring the relevant information for the doctor to review. 

When AI is used in sectors such as medicine, military defense, or data privacy, AI must produce transparent and understandable results that reflect the ethical standards of our society. XAI helps us understand how right, how fair, and how just an AI’s output, outcome, and impact are. This capability is transforming machine learning (ML) to build trust, clarity, and transparency into an AI’s decisions and the factors that produce those decisions. 

With traditional modeling techniques (such as regression models and surrogate decision tree-based models), the relationship between inputted data and outputted decision models is traceable and explainable. These are called white box models. More complex modeling techniques like deep learning and ensemble models promise increases in model accuracy, but often at the expense of model explainability. These are called black box models because it is much harder to observe and understand the relationship between the model input data and output decisions. Sources of bias, such as unbalanced training data, can impact how reliable these outputs are. Compared to white box models, black box models introduce risk because there is no way to observe what is occurring in a black box model on its own. This is where the value of XAI comes in.

With a suite of emerging techniques, XAI explains decisions for five main purposes:

• Analyst Insight: Helping analysts handle high-volume alerts and augmenting their performance with guidance around where to focus their attention and when
• Bias Detection & Mitigation: Investigating the presence of harmful bias and its potential effects on an algorithm's decision process in order to help mitigate throughout the entire AI development process 
• Continuous Improvement: Fostering continuous optimization and building a symbiotic relationship between human and machine, with a recognition that human understanding of processes can lead to imperfect decisions
  
• Quality Assurance: Providing consistent, easy-to-read, explainable insights for the interpretation of data and decision outputs of machines
• Regulatory Transparency: Documenting decision factors and actions to ensure confidence that the machine is doing what it’s supposed to do

This challenge of transparent and interpretable AI impacts every sector, from deploying critical government missions to setting policy for social media, which is why monitoring the development of XAI will be crucial in the coming years. What should you and your organization be thinking about as you advance your AI capabilities and processes? Here are key considerations for Federal Government leaders.

1. The future of AI is centered around ethics, requiring new tools and governance. Ethical AI is not just a buzzword. Standout organizations will continue to develop AI rooted in deep scientific and technological rigor with built-in mechanisms for tracking data, model accountability, and auditability.
   
2. The XAI market is growing at high velocity. Every notable AI company is developing or marketing a component of XAI; however, there are very few companies that focus solely on providing XAI models. While the XAI startup market is still small with fewer than 50 companies identified, some are developing XAI as an additional feature of their pre-existing products and platforms. Monitor this growing field so your organization can stay current on emerging capabilities and use cases. 
3. Tomorrow’s AI algorithms must work for the humans they impact. Technology leaders should focus on building AI teams that are meaningfully diverse and inclusive to have a wide range of perspectives and experiences to draw on. The future of XAI and ethical AI will depend on humans holding each other, and their models, accountable.