Machine learning Image recognition potential for AFM
Submitted by Dalia Yablon Ph.D on 07 January, 2020.
Machine learning 2: image recognition potential for AFM
As we continue to explore the potential of machine learning for AFM, we will intersperse basic concepts and jargon of ML into the blog to make sure everyone is on the same wavelength. For example, although we are here using the terms AI and ML interchangeably, it should be noted that machine learning falls under the broad umbrella of artificial intelligence and that deep learning is a type of machine learning.
In this blog we want to focus on one of the possible uses of machine learning in microscopy: image analysis. This can take a lot of forms; we will start with the very popular application of image recognition. Image recognition plays a role in many processes that take advantage of AI/ML: self-driving cars, secure logins to iphones, and other security applications like screening in airports or even urban areas. Image recognition also follows a basic rule of thumb in machine learning where an algorithm is created for a simple task that takes a human only a couple of seconds to do: recognizing something or someone based on their visual presentation is exactly that.
Image recognition takes advantage of concepts from computer vision and is generally done with multiple layers of neural networks. They are a complex nonlinear network of interconnected layers (called hidden layers). If you want to fit a neural network image classifier that identifies pictures of dogs, you will train a model on several dog and non-dog pictures. The “pictures” here are an array of numbers since all pictures are matrices in the red, blue and green channel. Given a picture (i.e. an array of numbers), the neural network will be able to predict whether the image is a dog or not.
One way to evaluate the success of such a dog-classifier is by plotting a ‘confusion matrix’ as shown below. This is a way to evaluate how well a model performs on several unseen pictures. A confusion matrix is a matrix where each column is the actual label of a picture, and each row is the label predicted by the ML model. Ideally, this matrix should have all no-diagonal terms as zero, if the model was 100% accurate. In this test set, the classifier predicts 50 dog pictures correctly but predicts 2 dog pictures as non-dog. Machine learning models are tweaked to achieve some target performance levels evaluated by measures like the ‘confusion matrix’.
In the case of AFM images, instead of the red, green, and blue, the channels are height, phase, amplitude etc. Note that neural network models can get really complex and consume massive computational power. That is when we start calling it deep learning! In the simple example of the dog image classifier shown here, a single image is an array with more than 12,000 entries. Think of how much computation power it will take to train a multi-layer network with thousands of such images.
Recent work done by researchers at Oak Ridge National lab uses very high-power computational facility at UT Austin’s Texas Advanced Computing Center (TACC). Using similar method of identifying dog pictures, one can build networks for more impactful applications (not that dogs are not important!). Another study uses machine learning to identify cancerous from non-cancerous cells from microscopy images.
A straightforward extension of the dog image recognition example above would be to train the computer to recognize AFM images based on the material being imaged. This has been a successful application for us through a neural net where the computer has learned to distinguish AFM images of different material blends, no matter the resolution or scan size of the image!
As we pursue more practical applications, machine learning can be used to perform sophisticated image analysis. Often image analysis in AFM can be challenging as backgrounds in AFM images can be very heterogeneous. ML offers a perfect tool to do what our eye does most naturally: identify a feature out of a complicated background. We have been working specifically on tricky particle analysis applications and results to follow soon!
Finally, AFM has an additional level of sophistication over other microscopies that can be taken advantage of in machine learning: multiple correlated channels. In an image recognition problem, a whole image serves as one training data point. However, we can build different machine learning models for AFM data where the height, phase, and amplitude output (or even more channels) of a single location can all together act as a single data point.
As we mentioned at the outset, the theme of ML in AFM imaging is a multi-part blog where we introduce concepts and jargons as well as discuss some early results of using ML to analyze AFM images. Stay tuned for our next blog where we will start to discuss some of our results!
Dalia Yablon and Ishita Chakraborty