Brain: How Do Visual and Auditory Systems Interact?

  • Medical
  • Thread starter translation31
  • Start date
  • Tags
    Brain Visual
In summary, a discussion on brain topics in the Physics Forums thanked frequent contributors such as atyy, Moonbear, neurocomp2003, somasimple, and zoobyshoe. The topic of the brain is popular in the Medical Sciences section and there have been discussions on the differences between visual and auditory systems in terms of neuronal activity and processing. However, it seems that the differences lie more in the cytoarchitecture and organization of processing modules rather than the activity of neurons. Additionally, our experience of the world is a mental construction and there is still much to learn about the brain's role in this. The visual and auditory systems have separate pathways in the brain, with visual information traveling to the occipital lobe and auditory information
  • #1
translation31
2
0
First I'd like to thank some frequent contributors to brain topics in the Physics Forums:

(in unbiased alphabetic order)

atyy
Moonbear
neurocomp2003
somasimple
zoobyshoe

Some great discussions on brain topics here! Based on frequency of the word "brain" in subject headings I would say it's the most popular topic in the Medical Sciences section. Does anyone know if serious collaboration ever originated in Physics Forums? It would be pretty cool!

getting to the point

I find it remarkable that vision and sound are such vastly different conscious experiences, yet both are described by neuronal communication.

I'm expecting the consensus is "No one knows why this is", so it would be helpful if anyone had links/references regarding differences between neuronal activity in visual vs. auditory systems (e.g. firing patterns, chemical properties, etc).

Just a first guess is perception is vastly different because signaling in the visual system occurs much faster than in the auditory system (since cells are interpreting light frequencies vs. sound frequencies). But that's strictly intuitive argument hence the request for more sound research. Any help would be greatly appreciated.

Thank you.
 
Biology news on Phys.org
  • #2
We tried a "mind and brain" forum but no one posted to it, so it was scrapped.

Medical problems would go in Medical. Discussions of the brain and/or neurology should go in biology.
 
  • #3
translation31 said:
Just a first guess is perception is vastly different because signaling in the visual system occurs much faster than in the auditory system (since cells are interpreting light frequencies vs. sound frequencies).
That doesn't sound right. The signal comes either from a nerve that is located as to be triggered by sound in a range of wavelengths, or located as to be triggered by light in a range of colours. But the frequency of the neural signal (I guess you would mean how often a neuron fires) has nothing to do with the wave-frequency that the sense corresponds with. It is nothing like a digital oscilloscope (which actually does monitor amplitudes with fine time-resolution).

translation31 said:
Does anyone know if serious collaboration ever originated in Physics Forums? It would be pretty cool!
Dunno, but PF generally forbid discussion of unpublished research, frown on highly active threads, and usually lock any topic the admins don't already understand, so it's probably only conducive to collaboration among the admins. By design PF is never more than it was explicitly intended to be.
 
Last edited:
  • #4
translation31 said:
I find it remarkable that vision and sound are such vastly different conscious experiences, yet both are described by neuronal communication.

I'm expecting the consensus is "No one knows why this is", so it would be helpful if anyone had links/references regarding differences between neuronal activity in visual vs. auditory systems (e.g. firing patterns, chemical properties, etc).
The inportant difference doesn't seem to be neuronal activity so much as cytoarchitecture. Brodeman found the organization of the cells was different in different dedicated areas:
http://en.wikipedia.org/wiki/Brodmann_area

So, they have, at least, found that the "architecture" of the cells plays an important role in how the signal is experienced.
 
  • #5
As said, in general, all brain cells react much the same and so the qualitative differences must be down to the organisation of the processing. Visual processing is retinotopic - the mapping of what the eyes see to the primary visual cortex preserves the spatial information. Aural processing is tonotopic - the way the cochlea breaks down response into frequency is preserved, so that high tones are dealt with at one end, low at the other.

So neurons are general components, and the processing in the brain physically preserves some of the felt structure of the stimulus, which is at least a start on explaining why similar components can generate different experiences.

Then each stream of sensory processing actually involves hierarchies of "processing modules". For vision, you have at least 30 doing things like representing motion, colour, etc. For hearing, there are far fewer. And so vision feels like a richer experience than hearing. There is more kinds of analysis going on.

A final point would be that what we actually experience is a mental construction. Colour for example is just an invention, not some literal transduction of the stimulus. So even though the general spatial and temporal structure of the world is being preserved in the processing, in the end, we still have a long way to go to be able to say why red IS red, or the chime of a bell is how we experience it.

We can look at the brain circuits and predict something about the nature of the experience. This is what people are trying to do right now with things like bat echolocation - go from the neural architecture to some sense of what it is like to be a bat seeing via sound. But it does not seem likely that we can get a complete sense of explanation from this bottom-up (from neurons to experience) kind of account.
 
  • #6
Processing of the two sensory systems happens in completely different parts of the brain.

The eyes are actually developed from the nervous system itself. The optic nerves (carrying the visual part of the information from the eye) synapse in a part of the thalamus called the lateral geniculate nuclei. The post-synaptic neurons follow a few different pathways, but the primary one for visual information extends along what are called the optic radiations to the occipital lobe of the brain, which is the lobe of the cortex all the way at the back of the brain (the part of information about light that is involved in things like pupil reflexes and synchronizing circadian rhythms follows other paths, but I don't think these are the ones you're asking about).

Auditory information travels along a different cranial nerve that first enters the brainstem in the cochlear nucleus (in the pons). It has a more indirect pathway than the visual system, so does a few hops, skips and jumps through synapses of neurons in the superior olive (near the junction of pons and medulla) to the inferior colliculus in the midbrain, then up to the medial genicular bodies of the thalamus (next to the lateral genicular bodies where visual information first enters), and then ends up being processed in the temporal lobe of the cortex, which is along the side and bottom of the brain.

You can readily find pictures of the lobes of the cortex all over the internet to see where the occipital and temporal lobes are. The part of the temporal lobe that receives auditory information is right under the parietal lobe if you look it up, tucked into what you'll find labeled as the lateral fissure. As for your question about collaborations among PFers, those of us who are actually doing research in neuroscience are all studying vastly different subjects, and others are not actually neuroscientists, but have a keen interest in the subject and have read a lot but are not actively doing any type of research in it. So, no, we're not collaborating on any projects, we just enjoy picking each others' brains from time to time. :wink:
 
  • #7
I understand that the op mentions differences in the experiences of sight and hearing resulting from external stimuli, but since I mentioned it recently (post #176 of the Synesthesia Thread in Medical Sciences), I understand the McGurk effect may be an example of visual stimuli affecting heard experiences.

http://www.nature.com/nature/journal/v264/n5588/abs/264746a0.html

http://www.youtube.com/watch?v=jtsfidRq2tw&feature=related
(picked this version because it had an amount of explanation, regardless of what else is said)
 
  • #8
Some very rough, exaggerated ideas, just to illustrate the range of possibilities:

1) The brain is all the same, but the relationship between "objects in the real world" (things that you interact with via actions) and the way they can be inferred via visual or auditory signals on the retina or cochlea is different. If you wired visual signals to the auditory cortex, it would end up like visual cortex.
http://www.ncbi.nlm.nih.gov/pubmed/10786793
http://www.ncbi.nlm.nih.gov/pubmed/10786784

I like this line of thinking about how we use audition to infer what an "object" is http://www.staff.ncl.ac.uk/t.d.griffiths/Griffiths_NRN_2004.pdf [Broken]. You can try the same sort of logic on images to see what you get, this is the sort of thing people in computer vision do http://iris.usc.edu/Vision-Notes/bibliography/contents.html.

2) Or maybe the brain areas really are different, so even if you sent the "same" signals to those areas, you would get different experiences.
http://www.ncbi.nlm.nih.gov/pubmed/11694887 (this paper compares audition and touch, not audition and vision, but the idea is the same)
 
Last edited by a moderator:
  • #9
atyy said:
Some very rough, exaggerated ideas, just to illustrate the range of possibilities:

1) The brain is all the same, but the relationship between "objects in the real world" (things that you interact with via actions) and the way they can be inferred via visual or auditory signals on the retina or cochlea is different. If you wired visual signals to the auditory cortex, it would end up like visual cortex.
http://www.ncbi.nlm.nih.gov/pubmed/10786793
http://www.ncbi.nlm.nih.gov/pubmed/10786784

Hehehehe. The ferrets looked like they were having a visual experience.
 
  • #10
zoobyshoe said:
Hehehehe. The ferrets looked like they were having a visual experience.

So what is it like to be a ferret? :smile:

http://www.uwalumni.com/home/onwisconsin/archives/spring2007/balancingact.aspx
http://www.engr.wisc.edu/bme/newsletter/2007/in_memoriam.html [Broken]
 
Last edited by a moderator:
  • #11
atyy said:
Some very rough, exaggerated ideas, just to illustrate the range of possibilities:

(snip)

I like this line of thinking about how we use audition to infer what an "object" is http://www.staff.ncl.ac.uk/t.d.griffiths/Griffiths_NRN_2004.pdf [Broken]. You can try the same sort of logic on images to see what you get, this is the sort of thing people in computer vision do http://iris.usc.edu/Vision-Notes/bibliography/contents.html.


Nice.
 
Last edited by a moderator:
  • #12
atyy said:
So what is it like to be a ferret? :smile:
What's it like, or what does it look like?

http://www.uwalumni.com/home/onwisconsin/archives/spring2007/balancingact.aspx
http://www.engr.wisc.edu/bme/newsletter/2007/in_memoriam.html [Broken]
Very interesting and mysterious. One of the people I met by accident at the cafe happens to be a neurologist specializing in vestibular disorders. Next time he comes in I'm going to ask if he's heard about this.
 
Last edited by a moderator:
  • #13
zoobyshoe said:
What's it like, or what does it look like?

What's it feel like - ie, did the visual stimuli look visual, or did they sound like sounds? Of course we can't know, but presumably the verbal responses of people who've experienced sensory substitution would be informative.
 
Last edited:
  • #14
translation31 said:
First I'd like to thank some frequent contributors to brain topics in the Physics Forums:

(in unbiased alphabetic order)

atyy
Moonbear
neurocomp2003
somasimple
zoobyshoe

Some great discussions on brain topics here! Based on frequency of the word "brain" in subject headings I would say it's the most popular topic in the Medical Sciences section. Does anyone know if serious collaboration ever originated in Physics Forums? It would be pretty cool!

getting to the point

I find it remarkable that vision and sound are such vastly different conscious experiences, yet both are described by neuronal communication.

I'm expecting the consensus is "No one knows why this is", so it would be helpful if anyone had links/references regarding differences between neuronal activity in visual vs. auditory systems (e.g. firing patterns, chemical properties, etc).

Just a first guess is perception is vastly different because signaling in the visual system occurs much faster than in the auditory system (since cells are interpreting light frequencies vs. sound frequencies). But that's strictly intuitive argument hence the request for more sound research. Any help would be greatly appreciated.

Thank you.

One field you might want to study in depth is neuroplasticity and how the visual and auditory cortexes will compensate for one another in the event of blindness or deafness.

Here's a link to the Neurosciences division of the U of O where all the primary research has been done with regard to neuroplasticity:

http://bdl.purpled.com/Publications/publications.php

I've given you the publications page and recommend this PDF on that page...

Neville, H. and Sur, M. (2009). Neuroplasticity. In M. Gazzaniga (ed), The Cognitive Neurosciences IV, MIT Press, Cambridge, pp. 89-90. [pdf]
 
  • #15
atyy said:
What's it feel like - ie, did the visual stimuli look visual, or did they sound like sounds? Of course we can't know, but presumably the verbal responses of people who've experienced sensory substitution would be informative.
Yes, but in the case of the vestibular disorders the people's brains weren't literally rewired like the ferrets. Learning to use sensations on the tongue to keep your balance would, I think, be like any indirect control: watching a speedometer for example. The article mentions the analogy of having someone following you with a finger touching the center of the top of your head. If you tilt you can feel the position of the finger touch change and correct to bring it back to center. I didn't get the impression from the article that normal balance had returned, just that, for some mysterious reason, the tongue-balance effect lasted after the device was removed.
 
  • #16
baywax said:
One field you might want to study in depth is neuroplasticity and how the visual and auditory cortexes will compensate for one another in the event of blindness or deafness.

Here's a link to the Neurosciences division of the U of O where all the primary research has been done with regard to neuroplasticity:

http://bdl.purpled.com/Publications/publications.php

I've given you the publications page and recommend this PDF on that page...

Neville, H. and Sur, M. (2009). Neuroplasticity. In M. Gazzaniga (ed), The Cognitive Neurosciences IV, MIT Press, Cambridge, pp. 89-90. [pdf]
One thing to note here is that the apparent use of the visual cortex by blind people, for example, which will show up on a brain scan, isn't necessarily for auditory purposes. Sacks reports in Musicophilia that blind people actually have masses of unformed, elementary visual experiences. They can't see, but they have an abstract visual world, rather like synesthesia. Likewise, but also contrariwise, people who go deaf often experience musical hallucinations, the non-stop experience of vividly hearing music. Not an abstract world of sounds, but repetitious, stereotyped hallucinations. Then there is the phantom limb phenomenon, where the parts of the cortex formerly responsible for the movement and sensation of the missing limbs continue to behave as if the limb were still there, such that the person, for all intents and purposes, feels that it is, contradicted only by the sight of it being missing and the failure of external objects to react to actions made by the phantom.

So, it seems that dedicated brain areas actually resist performing non-dedicated functions. Plasticity is most evident when a dedicated area becomes much better at its dedicated function, as seen in the amazing hearing of people with William's Syndrome, who are born with underdeveloped occipital lobes. There's a hardware cap on their visual abilities, but their hearing can become almost superhuman. Most of them suffer from hyperacusis, anyway. Scans show exceptionally active temporal lobes.
 
  • #17
I understand humans have mulitsensed perceptions with benefits like added affirmation and precision. I’ve read the gustatory and olfactory sensations are difficult to distinguish between and are particularly old senses both with important benefits in the detection, and interpretation of quality, of food. Benefits of audio-visual sensory integration would involve precise location of stimulus.

This is book seems a good over-view from 2004. Chapter 2, on page 27 is about audio-visual perception in particular-

http://books.google.co.uk/books?id=...A#v=onepage&q=stein cell multisensory&f=false

Parts of the book describe cross-modality, hetromodality, multisensory neurons, synaesthesia, etc.. This is an example of the papers mentioned-
http://www.jneurosci.org/cgi/content/abstract/22/13/5749

I had mentioned the McGurk effect before here, (again-
) and understand that the youtube I presented described a typical response, but that reactions vary and may hypothetically depend on individual modal strengths and weaknesses. I don’t have that response (and can think of a reason why that may be the case).
 
Last edited by a moderator:
  • #18
zoobyshoe said:
Yes, but in the case of the vestibular disorders the people's brains weren't literally rewired like the ferrets. Learning to use sensations on the tongue to keep your balance would, I think, be like any indirect control: watching a speedometer for example. The article mentions the analogy of having someone following you with a finger touching the center of the top of your head. If you tilt you can feel the position of the finger touch change and correct to bring it back to center. I didn't get the impression from the article that normal balance had returned, just that, for some mysterious reason, the tongue-balance effect lasted after the device was removed.

Yes, it is indirect control. But what does it "feel" like? After all, the best musicians surely feel that their instruments are part of themselves, else how could they truly "speak" through them?
 
  • #19
atyy said:
What's it feel like - ie, did the visual stimuli look visual, or did they sound like sounds? Of course we can't know, but presumably the verbal responses of people who've experienced sensory substitution would be informative.
zoobyshoe said:
Yes, but in the case of the vestibular disorders the people's brains weren't literally rewired like the ferrets. Learning to use sensations on the tongue to keep your balance would, I think, be like any indirect control: watching a speedometer for example. The article mentions the analogy of having someone following you with a finger touching the center of the top of your head. If you tilt you can feel the position of the finger touch change and correct to bring it back to center. I didn't get the impression from the article that normal balance had returned, just that, for some mysterious reason, the tongue-balance effect lasted after the device was removed.
atyy said:
Yes, it is indirect control. But what does it "feel" like? After all, the best musicians surely feel that their instruments are part of themselves, else how could they truly "speak" through them?
I don't know where you're going with this. The OP is wondering why some populations of neurons present an "auditory" experience to consciousness and others present a "visual" experience. Becoming proficient with a musical instrument, or any similar skill, doesn't change that, that I'm aware of.
 
  • #20
zoobyshoe said:
I don't know where you're going with this. The OP is wondering why some populations of neurons present an "auditory" experience to consciousness and others present a "visual" experience. Becoming proficient with a musical instrument, or any similar skill, doesn't change that, that I'm aware of.

The analogy being:

What makes a thing feel "visual" or "auditory"?

What makes a thing feel "external" or "part of you"?
 
  • #21
atyy said:
The analogy being:

What makes a thing feel "visual" or "auditory"?

What makes a thing feel "external" or "part of you"?

First question may get answered in this paper...

Abstract

Our brain integrates the information provided by the different sensory modalities into a coherent percept, and recent studies suggest that this process is not restricted to higher association areas. Here we evaluate the hypothesis that auditory cortical fields are involved in cross-modal processing by probing individual neurons for audiovisual interactions. We find that visual stimuli modulate auditory processing both at the level of field potentials and single-unit activity and already in primary and secondary auditory fields. These interactions strongly depend on a stimulus’ efficacy in driving the neurons but occur independently of stimulus category and for naturalistic as well as artificial stimuli. In addition, interactions are sensitive to the relative timing of audiovisual stimuli and are strongest when visual stimuli lead by 20–80 msec. Exploring the underlying mechanisms, we find that enhancement correlates with the resetting of slow (10 Hz) oscillations to a phase angle of optimal excitability. These results demonstrate that visual stimuli can modulate the firing of neurons in auditory cortex in a manner that depends on stimulus efficacy and timing. These neurons thus meet the criteria for sensory integration and provide the auditory modality with multisensory contextual information about co-occurring environmental events.

http://cercor.oxfordjournals.org/cgi/content/full/18/7/1560
 

1. How does the brain process visual information compared to auditory information?

Visual information is processed through the visual cortex, located in the occipital lobe of the brain. This part of the brain receives and interprets signals from the eyes, allowing us to see objects, colors, and movement. Auditory information, on the other hand, is processed through the auditory cortex, located in the temporal lobe. This part of the brain receives and interprets signals from the ears, allowing us to hear sounds and speech.

2. Which sense is more dominant in the brain, visual or auditory?

The visual sense is typically considered to be more dominant in the brain compared to the auditory sense. This is because the visual cortex is larger and more complex than the auditory cortex, and our brains are highly dependent on visual information for survival, such as identifying potential threats and navigating our surroundings.

3. Can visual and auditory information be processed simultaneously?

Yes, visual and auditory information can be processed simultaneously by the brain. In fact, our brains are constantly integrating information from all of our senses to create a complete and cohesive perception of the world around us. However, some studies have shown that when two senses conflict, our brains may prioritize one over the other.

4. How do visual and auditory information affect our memory?

Visual information tends to be more memorable than auditory information, as our brains are better at retaining visual images rather than auditory sounds. This is due to the fact that the visual cortex has a stronger connection to the hippocampus, which is responsible for long-term memory formation. However, our brains are still able to remember auditory information through repetition and association with visual cues.

5. Can brain damage affect our visual and auditory processing?

Yes, brain damage can greatly affect the brain's ability to process visual and auditory information. Damage to the visual cortex can result in vision loss or impairment, while damage to the auditory cortex can result in hearing loss or difficulty understanding speech. In some cases, damage to one area of the brain can also affect the processing of the other sense, as both visual and auditory information can be processed in multiple areas of the brain.

Similar threads

Replies
3
Views
2K
  • Biology and Medical
Replies
5
Views
2K
  • Special and General Relativity
Replies
1
Views
992
  • Biology and Medical
Replies
8
Views
7K
  • General Discussion
3
Replies
72
Views
11K
  • General Discussion
Replies
4
Views
604
Replies
5
Views
2K
  • Sci-Fi Writing and World Building
Replies
9
Views
4K
  • Biology and Medical
Replies
2
Views
11K
Back
Top