About ten years ago, deep-learning models started beating humans at a variety of tasks.
For example, they beat the world champions at board games and were better at finding breast cancer than doctors.
How do we know if AI is working the way we want it to?
Artificial neural networks, which were initially suggested in the 1940s and have since become a common kind of machine learning, are often the foundation of these potent deep-learning models.
A computer learns to analyze input by having layers of linked nodes, or neurons, that look like the human brain.
Artificial neural networks have expanded with machine learning Significantly.
Deep-learning models are now often made up of millions or billions of linked nodes arranged in several layers and trained on enormous quantities of data to perform detection or classification tasks.
But even the academics who create the models don't completely comprehend how they operate since the models are so incredibly complicated.
This makes it difficult to determine whether they are operating properly.
For instance, a model created to aid doctors in patient diagnosis could have accurately predicted that a skin lesion was malignant, but it may have done so by concentrating on an unrelated mark that commonly appears in photos of diseased tissue rather than the cancerous tissue itself.
The forecast was accurate by the model, but for the incorrect reason. It could lead to a missed diagnosis in a real-world clinical situation where the mark does not show up in photos showing cancer.
How can one figure out what is happening within these so-called "black-box" models when there is so much ambiguity around them?
This conundrum has given rise to a fresh and quickly expanding field of research where scientists create and test explanation techniques (also known as interpretability methods) that aim to explain how black-box machine-learning algorithms produce predictions.
How do you explain things?
Methods of explanation are either global or local at their most fundamental level.
While global explanations aim to describe an entire model's overall behavior, local explanation methods concentrate on describing how the model arrived at a single prediction.
Most of the time, this is done by making a different, smaller model that matches the larger, black-box model and should be easy to understand.
But since deep learning models function in fundamentally nonlinear and complicated ways, it is especially difficult to create a global explanation model that is successful.
Yilun Zhou, a graduate student at the Computer Science and Artificial Intelligence Laboratory's Interactive Robotics Group who researches models, methods, and assessments in interpretable machine learning, says that this has led researchers to focus more on local explanation methods in recent years.
The most common ways to explain something in a local context can be put into three main groups.
The very earliest and most popular kind of explanation technique is called "feature attribution."
The characteristics that the model prioritized when it made a certain choice are shown through feature attribution techniques.
A machine-learning model uses features as its input variables, which are then utilized to make predictions. Features are taken from a dataset's columns when the data is tabular (they are transformed using a variety of techniques so the model can process the raw data). On the other hand, for jobs involving image processing, each pixel in a picture is a feature.
The feature attribution technique would draw attention to the pixels in that particular X-ray that were most crucial for the model's prediction, saying that an X-ray picture reveals malignancy.
Basically, feature attribution techniques show what the model pays attention to when it makes a prediction.
Using this feature attribution explanation, you can determine whether a false association is cause for concern.
instance, it will display whether the pixels in a watermark or a real tumor are highlighted.
A counterfactual explanation is the second kind of explanation technique.
These techniques demonstrate how to alter an input such that it belongs to a different class, given the input and the forecast of a model.
For instance, the counterfactual explanation reveals what has to happen for a borrower's loan application to be approved if a machine-learning algorithm predicts that she will be refused credit.
Perhaps she has to have a better credit score or income, two factors that the algorithm utilizes to make its forecast, in order to be accepted.
The benefit of this explanation style is that it clearly explains how to alter the input in order to reverse the choice, which may be useful in real-world situations.
This explanation would explain to someone who applied for a mortgage but didn't obtain it what they needed to do to attain the result they wanted, he claims.
Sample importance explanations are the third group of explanation techniques. This approach needs access to the data that was used to train the model, unlike the others.
A sample significance explanation will highlight the training sample that a model most often uses to make a certain prediction; ideally, this sample will be the one that is closest to the input data.
This kind of justification is especially helpful when one notices an apparent irrationality in a forecast.
A specific sample that was used to train the model may have been impacted by a data input mistake.
With this information, the sample could be fixed, and the model could be retrained to function more accurately.
What explanatory techniques are employed?
To do quality control and troubleshoot the model is one reason for creating these explanations.
For example, with a better knowledge of how features affect a model's decision, one may see when a model is operating incorrectly and take action to change it, discard the model, and begin over.
Exploring the use of machine-learning algorithms to find scientific patterns that humans haven't yet found is another, more recent topic of inquiry.
For instance, a cancer diagnosis model that performs better than clinicians may be flawed, or it may be detecting hidden patterns in an X-ray image that represent an early pathological pathway for cancer that were either overlooked by human physicians or believed to be unimportant, according to Zhou.
That field of study is still in its infancy, however.
A word of caution
End-users should exercise caution when attempting to use explanation methods in practice, warns Marzyeh Ghassemi, an assistant professor and the leader of the Healthy ML Group in CSAIL.
While explanation methods can occasionally be helpful for machine-learning practitioners when they are trying to catch bugs in their models or understand the inner workings of a system,
Explanation techniques are being used to assist decision makers better comprehend a model's predictions so they can know when to trust the model and employ its advice in practice.
Machine learning has been embraced in many fields, from health care to education. Ghassemi cautions against using these techniques in that manner nonetheless.
"We have discovered that explanations lead to overconfidence in a certain recommendation system's competence or counsel among individuals, including experts and nonexperts.
I believe it is crucial for people to keep using their internal circuitry to say, "Let me question the advice that is being offered," she adds.
She continues, citing some recent studies conducted by Microsoft researchers, "Scientists know explanations make individuals overconfident based on other recent studies."
Methods of explanation are far from perfect and are not without flaws.
First of all, Ghassemi's recent study shows that explanation techniques might make people more prejudiced and give people from underrepresented groups worse results.
Another drawback of explanation techniques is that it is sometimes hard to determine whether they are true in the first place.
It would be necessary to compare the explanations to the real model, but Zhou claims that this is circular reasoning since the user doesn't understand how the model operates.
Zhou says that even the best explanation should be taken with a grain of salt because he and other researchers are trying to make explanation techniques better match the predictions of the real model.
Additionally, we are prone to generalizing too much and believe that these models make decisions in a manner similar to that of humans.
To genuinely ensure that the generalized model of knowledge they construct from these local explanations is balanced, we need to calm folks down and hold them back, the expert continues.
That is what Zhou's most recent study aims to achieve.
What is the future of machine-learning explanation methods?
Rather than concentrating on explanations, Ghassemi believes that more research should be done to explore how information is presented to decision makers so that they comprehend it, and that more regulation should be put in place to guarantee machine-learning algorithms are utilized responsibly in practice.
Better explaining techniques alone will not be enough.
I've been encouraged to see that there is a growing realization, even within the industry, that we can't just take this data and create a lovely dashboard and expect people to perform better as a result.
"You need quantifiable gains in action, and I'm hoping that leads to genuine standards for changing the way we convey information in these very complex sectors, such as medicine," she adds.
In addition to the new work on enhancing explanations, Zhou anticipates increased study on explanation techniques for particular use cases such as model debugging, scientific discovery, fairness audits, and safety assurance.
Researchers could come up with a theory that matches explanations to specific situations. This could help solve some of the problems that come up when using explanations in the real world by identifying the fine-grained properties of explanation techniques and the needs of different use cases.