Information Theory as a tool for Machine Learning

Koichi Harada, Hiroshima University

Seeking for optimum points is one of the main topics in the research of Engineering and Science. The definition of the optimum point varies in each research field but the efficiency in approaching the goal is crucial in every topic. When we select the optimum path to the goal, an evaluation function is required that might give the best decision for the path selection. The evaluation function determines whether or not the optimization problem (in the sense of high quality solution be obtained) be successfully solved.

In the talk, we first summarize a typical image processing research topic. The topic relates to medical image analysis. More specifically, the analysis of the blood vessel is demonstrated as an example of an image processing research. Two types of laser-oriented imaging: LSCI (Laser Spackle Contrast Imaging) and LASCA (Laser Speckle Contrast Analysis) are used, and then design filters that can reduce the noise contained in the original images. In the noise reduction step (De-noising), various optimization techniques are examined to obtain the best hybrid filters that are created by combining existing Wiener, Nonlinear, and Wavelet filters. Another approach is on the blood vessel in Sclera-conjunctive image. The local noise in this image hampers high quality blood vessel shape pattern classification. The optimum setting of the threshold to remove the local noise is the key to solve this problem. The optimum threshold setting is another example that could be viewed as the selection of the best path to the goal.

The evaluation function (or error function) should be reduced for better path selection to the goal. Historically, the evaluation function reduction is proposed as Hopfield’s weight adjustment approach. We know that appropriate adjustment of the weight could bring us to the better position to the goal, but what weight we should select and/or how much should change the weight is not clear. If we keep changing the weights in the direction of reducing the evaluation function value for various inputs, a better solution could be expected after many attempts (Machine leaning approach).

The best selection of the weight can be viewed as the decision problem. A best decision tree should work that offers best selection of the weights. To construct better decision tree, the information theory is very useful.

We consider the traditional information theory (or similar quantity) could serve as the guide to generate better weights (and hence the better path to the goal) in the widely used optimization problem.

Subscribe to ICTer News