Using entropy to discover visually salient features

• elpidiovaldez5
In summary: I think may be done by the neural network itself, basing its decision on some sort of entropy calculation. In summary, visually salient features may be discovered by unsupervised learning by looking for patterns that are less likely to occur by chance.
elpidiovaldez5
I am trying to understand how visually salient features could be discovered by unsupervised learning. I do not want to assume that we already have edge detectors or convolutional neural networks, rather I am trying to imagine how these could be discovered by observing the world.

Imagine that a camera captures grey scale images, and that these are thresholded, so that half the pixels are zero (black) and half the pixels are 1 (white). Now look at each pixel position and extract the 9 pixels in the 3x3 grid centred at that position. There are 2^9 (512) possibilities. So if all pixels were independent, the probability of observing a particular pattern would be 1/512. We could easily work out the real world probability by counting occurrences of each pattern in a large number of real world images.

Now consider a single image in which a horizontal line comprising 20 consecutive white pixels is present. We can calculate the probability of this occurring, if the pixels are independent, or using real world data. Either way it seems very unlikely that this would occur by chance, but we have to be quite careful, because ANY specific pattern of 20 pixels is very unlikely. What is special about a LINE of pixels ? (it is analagous to the unintuitive fact that a winning lottery combination of 9,999,999,999 seems very unlikely, but is exactly as likely as 6,936,125,674). It seems to me that although this arrangement is equally unlikely as many other random arrangements, the pixels in a line has lower entropy. The size of the image is also significant because it might be unlikely to find a line of 20 pixels in a 640x480 image, but much less so in a 4096x2048 image.

This is where my poorly remembered maths and physics let's me down. Is there any mathematical technique which would detect the line as a surprising anomaly ? I am thinking of Hypothesis Testing, or some kind of entropy calculation. In entropic considerations it seems like I need to distinguish 'microstates' (the pixel patterns) from 'macrostates' (a CONTIGUOUS sequence of the same state extended over a certain range). So a macrostate would define exactly which pattern of pixels are being predicted.

Am I missing something here ? I'd like to see some mathematical rigour brought to the idea. If it makes sense for lines, the same principles could be brought to bear to discover other visual features.

elpidiovaldez5 said:
I am trying to understand how visually salient features could be discovered by unsupervised learning
Do you mean so-called deep learning methods used to train neural networks? If so, then I recommend this free online book. http://neuralnetworksanddeeplearning.com/

The book uses handwritten character recognition as it's working example, that relates to visually salient features.

berkeman
The book looks great, but here I am actually I am interested in a more physics based view of saliency. Neural nets, particularly Convolutional neural nets (and more recently Transformers) are well suited to detecting visual features like edges, but I am trying to get at the reason for their success, which I think may be based on entropy.

elpidiovaldez5 said:
but I am trying to get at the reason for their success, which I think may be based on entropy.
I would characterize them more as curve fitting. Cousins of this:
https://en.wikipedia.org/wiki/Least_squares

Edit: They can also be viewed as an optimization problem. Define a function that says how well the neural net performs, then tweak all the knobs to get the best that you can.

Last edited:
elpidiovaldez5 said:
If it makes sense for lines, the same principles could be brought to bear to discover other visual features.
Are you looking for a particular (known) figure embedded in the picture (where's waldo?) or asking identification of unknown but unusual patterns automatically? Please elaborate

I used a line as an example, but I am really interested in detecting any unusual/unlikely patterns. I am not even sure it is possible without pre-judging the types of interesting patterns, but I think it may be. e.g. the components that make up a line (edge segments) probably occur more frequently in real scenes than would be expected from a scene of random pixels. If these edge segments also occur in a continuous run of, say 20, then this is also unlikely. Any specific pattern of 20 edge segments would be equally surprising, but continuous, directed runs occur frequently in natural scenes, whereas other specific patterns would be observed rarely. How could I design an algorithm to discover these anomalies ?

These are all very good questions but not one of them is simple. I have seen estimates that 20% of our "brainpower" is devoted to these kinds of questions so that should be an indicator.
The adjudication of what is "unusual" is possibly very difficult. My suggestion would be to first concentrate on the straight (edges?) of a minimum length. This seems like a reasonable research project. What is the most efficient way to flag very linear features? One immediate plus is that a linear feature viewed at an angle (without optical aberration) is still linear.

You do understand that this is a more than lifetime of work?

elpidiovaldez5 said:
I used a line as an example, but I am really interested in detecting any unusual/unlikely patterns. I am not even sure it is possible without pre-judging the types of interesting patterns, but I think it may be. e.g. the components that make up a line (edge segments) probably occur more frequently in real scenes than would be expected from a scene of random pixels.

I think you must judge what patterns show lines in order to implement that idea. And you have pre-judged that such patterns are significant and that they should be lumped together as the event "image contains a line".

Regarding the question in the title of your post, Shannon entropy is a property of probability distributions. In your example, a single image is a one outcome of a probability distribution. The outcome has a probability but it doesn't have a defined entropy. Similarly, the occurrence of some event defined by a set of outcomes (e.g. outcomes that show a line) is an event that has a probability. If we want to say the event "has an entropy", we need to extend the definition of "entropy".

If an event has probability ##p##, we could assign it the number ##p \ln(p)## and call that number the entropy of the event. That would not be conventional terminology. However, I've see such a thing done. In the publication

The maximum entropy formalism : a conference held at the Massachusetts Institute of Technology on May 2-4, 1978​

(which I haven't found online and I don't have my copy handy) there is a paper about finding where to drill for oil. The author used the distribution of various types of strata to define an entropy value for wells that had already been drilled. He then plotted "entropy contours" as a guide where to drill next.

hutchphd and Jarvis323

What is entropy and how is it relevant to discovering visually salient features?

Entropy is a measure of disorder or randomness in a system. In the context of discovering visually salient features, entropy can be used to quantify the amount of information contained in an image and identify regions with high visual contrast or variability.

How is entropy calculated and what are its units?

Entropy is typically calculated using the Shannon entropy formula, which takes into account the probability of each pixel intensity value in an image. The units of entropy are bits (or nats), which represent the amount of information in a system.

What are some applications of using entropy to discover visually salient features?

One application is in computer vision, where entropy can be used to identify important or interesting regions in an image for tasks such as object detection or image segmentation. It can also be used in image compression algorithms to prioritize important image information for preservation.

What are some limitations of using entropy for discovering visually salient features?

One limitation is that entropy is a statistical measure and may not always capture human perception of saliency. It can also be affected by image noise or artifacts, leading to false detections. Additionally, entropy may not be useful for identifying salient features in images with uniform or repetitive patterns.

Are there any alternative methods for discovering visually salient features?

Yes, there are other methods such as edge detection, feature detection, and saliency mapping that can be used to identify visually salient features in an image. These methods may complement or be used in conjunction with entropy-based approaches for more accurate results.

• General Discussion
Replies
2
Views
6K
Replies
2
Views
1K
• Sci-Fi Writing and World Building
Replies
21
Views
1K
• General Discussion
Replies
3
Views
4K
• Cosmology
Replies
1
Views
2K
• Classical Physics
Replies
1
Views
2K
• Special and General Relativity
Replies
8
Views
2K
• Cosmology
Replies
17
Views
7K
• General Discussion
Replies
10
Views
2K
• Biology and Medical
Replies
2
Views
3K