Cross-modal processing

Question 1

what is modular processing?

Answer 1

• Fodor (1983) proposed that cognition is a collection of independent information-processing modules.
• Although this has been a very influential theory and there are many virtues, e.g. different areas in the brain are specialised for different tasks to optimise speed and efficiency, there is evidence that modules (1) can change in specificity, (2) may be correlated in some tasks and (3) can communicate with each other.

brain = app ownjobs notalking “apps” sometimesadapt, collaborate shareda

Question 2

what modules can change?

Answer 2

• Ferrets in which retinal projections are redirected neonatally to the auditory thalamus have visually responsive cells in auditory thalamus and cortex, form a retinotopic map in auditory cortex and have visual receptive field properties in auditory cortex that are typical of cells in visual cortex.
• von Melchner et al. (2000) reported that this cross-modal projection in the auditory cortex can mediate visual behaviour: When light stimuli are presented in the portion of the visual field that is ‘seen’ only by this projection, ‘rewired’ ferrets respond as though they perceive the stimuli to be visual rather than auditory.

Question 3

can modules be correlated?

Answer 3

• One example of this is in speech processing where visual and audio information are processed together.
• In a noisy environment, understanding someone’s speech can be facilitated by reading their lip movement. In fact, Spence (2002) demonstrated that visual cues can improve speech perception by up to 15-20dB.
• This facilitation is possible because visual and auditory cues are normally correlated. So a particular sound will be accompanied by a specific mouth shape. If the sound is unclear, you can rely on the visual, and vice versa.

Question 4

how do modules communicate with each other?

Answer 4

• Emergence highlights the brain’s ability to create new precepts by deeply integrating sensory inputs, fundamental for complex tasks like navigating environments or understanding social cues.

Question 5

what is the McGurk effect?

Answer 5

• When audio stimulus is /b/, but the visual stimulus is /g/, the actual perception becomes one of /d/. When the participants shut their eyes, they can hear the /b/, but when they see the incongruent /g/ lip movement, they hear /d/.
• This demonstrates that neither visual or audio takes precedence and both streams of information work together to allow a new coherent percept.

Question 6

what is cross modal cueing?

Answer 6

• Two pathways or information channels is better than one.
• When participants are required to detect a faint visual target on the left or right of fixation, if they are also given an additional auditory or tactile cue on the side of the stimuli, reaction times are faster (Butter et el., 1989; Spence et al., 2009).

Question 7

what did McDonald et al. 2000 study?

Answer 7

• McDonald et al. (2000) asked participants to detect a weak visual stimulus that was presented to the left or right of a fixation point.
• The visual target was preceded by an audio cue, either on the same side of the visual target (valid) or on the other side (invalid).
• McDonald et al. (2000) measured two variables:
  - Low-level: Perceptual detectability measured as d’
  - High-level: Decision criterion, called bias or β
• The found that d’ was significantly higher when audio was presented on the same side, but there were inconsistent effects on bias β.
• The results indicated that the cross-modal cueing does involve some modulation of relatively low-level sensory processing, but there was no evidence of high-level involvement in this study.

left ear = preped -> more sensitive but judgement style doesn’t change

Question 8

what is dominance?

Answer 8

• This relates to which sense is prioritised when there is conflicting information. Accordingly, which type of sensory dominance (visual or auditory) emerges may be dependent on the conditions with which the stimuli are presented.

Question 9

what is the colavita effect?

Answer 9

• Colavita (1974) presented suprathreshold auditory and visual targets to participants and asked them to press one button for auditory targets and another button for visual targets.
• On some trials, both the light and sound were presented at the same time. These trials were interleaved into the sequence of unimodal target trials.
• The results found that most participants perceived the visual on the light-sound trials. Some reported being unaware that the sound component. This demonstrates visual dominance over auditory stimuli.

braintrusts eyes morethan ears when both senses give input at same time

Question 10

what is the modality appropriateness hypothesis?

Answer 10

• Certain properties of stimuli are better processed by a specific sense, leading to a particular modality dominating.
• Visual modality is superior at processing spatial information (e.g. Posner et al., 1976).
• Auditory modality is superior for temporal information (e.g. Shams, Kamitani, & Shimojo,2002).

Question 11

what is the biased/integrated competition hypothesis?

Answer 11

• Spence, Parise, and Chen (2012) suggest that brain systems dedicated to the different sensory systems may compete. They suggest that given that a large proportion of the brain is dedicated to visual processing (Sereno et al.,1995), visual stimuli and the visual system should be more likely to dominate and inhibit processing in other sensory systems.

Question 12

what is visual dominance?

Answer 12

• This is when the apparent location of a sensory stimulus can be influenced by visual information.
• During a ventriloquist’s act the mouth of the puppet does not correspond to the location of the sound source, yet observers perceive the puppet’s speech come from its mouth.
• Rubber hand illusion (Botvinick & Cohen, 1998): When participants placed an arm underneath a table and a false arm was stroked at the same time as a real arm and participants perceived strokes from the brush on an artificial arm and even felt that was their own.

Question 13

what is the sound-induced flash illusion?

Answer 13

• Shams et al., (2000) presented a visual illusion that is induced by sound. When a single flash is accompanied by two bursts of noise, you may perceive two flashes instead of one.
• Shams et al., (2000) showed how a visual stimulus can be altered by another modality even when the visual stimulus is not ambiguous. This demonstrates that although vision is usually very dominant, like other modalities, it is also malleable by other senses.

Question 14

how does one sense enhance another?

Answer 14

• Smell and colour: The strawberry smell of a liquid appears stronger when the liquid is coloured red (Sellner & Kautz, 1990).
• Sound and light: Auditory noise presented with light tends to be perceived as louder than noise presented alone (Odgaard et al, 2004).
• Light and touch: Tactile discrimination thresholds are lower during visual observation (Kennett et al., 2001).
• Sound and touch: Perturbing the sound made as hands are rubbed together can alter the perception of the skin’s texture (Jousmaki & Hari, 1998; Guest et al., 2002).

Question 15

what is the neural basis to multimodal processing?

Answer 15

• Convergence zones in the brain have been suggested for multimodal processing (see Driver & Noesselt, 2008). These are higher level areas, such as the superior temporal sulcus, the parietal cortex, the prefrontal cortex, and the subcortical nuclei.
• Some neurons in the subcortical nuclei show superadditive responses to multimodal stimulation, where stimulation from multiple senses is greater than the sum of response from the difference senses in isolation (Alvarado et al, 2007).
• Some researchers have also found multimodal responses in previously considered modality-specific areas, such as the visual cortex and auditory cortex.
• There is increased evidence that signals can travel laterally from one unimodal area to another, but there is also some downward travel from the higher order multimodal/polysensory cortex (STP).

Question 16

what is the Bouba-Kiki effect?

Answer 16

• Köhler(1929) showed two forms and asked readers which shape was called “takete” and which was called “maluma”. More recently, Ramachandranand Hubbard (2001) repeated the experiment with the words “kiki” and “bouba”, and asked American college undergraduates andTamilspeakers in India, “Which of these shapes is bouba and which is kiki?” In both groups, 95% to 98% selected the curvy shape as “bouba” and the jagged one as “kiki”, suggesting that the human brain somehow attaches abstract meanings to the shapes and sounds consistently.
• Ćwiek et al. (2022) found the bouba/kiki effect is across-cultural phenomenon where 917 participants speaking 25 different languages maintain a consistency in bouba/kiki identification, regardless of their native language.

Question 17

what is synaesthesia?

Answer 17

• This is the multisensory phenomenon in which stimulation of one sense causes sensory experience in another. People who experience this are calledsynaesthetes.
• One of the most common synaesthesia is the colour-grapheme association where words/letters/numbers have particular colours.

Question 18

what did Simner 2019 find?

Answer 18

• Simner (2019) reported 128 different types of synaesthesias and the number of people with synaesthesia amounted to just 4.4 per cent of the population.
• Synaesthesia is a multi-variant condition:
  - Lexical-gustatory synaesthesia: Words have different tastes
  - Sound-colour synaesthesia: Sounds have different colours
  - Flavour-colour synaesthesia: Tastes have different colours
  - Touch-colour synaesthesia: Textures have different colours
  - Visual-auditory synaesthesia: Silent moving objects produce sounds

Question 19

what are the four important features of synaesthesia?

Answer 19

• Four important features:
  - The experiences are elicited by particular stimuli that would not evoke such experiences in most members of the population. The triggering stimuli could be perceptual or conceptual.
  
  • The experiences are automatic and are extremely difficult to suppress.
  • The nature of the synaesthetic experience itself is a conscious perceptual event.
  • The synaesthetic experience is consistent over time, although this is still debated.

Question 20

what are the bottom up factors?

Answer 20

• Witthoft and Winawer (2006) showed that for some grapheme-colour synesthetes, altering the fonts and cases of letters can affect the saturation of synaesthetic colours. In their study, the colour category of the synaesthetic colours was not strongly affected by these manipulations, only the saturation.

Question 21

what did Dixon et al. 2006 study?

Answer 21

• presented a case study with a synaesthete called J.
• presented J with ambiguous graphemes (e.g., a grapheme that could either be interpreted as ‘S’ or ‘5’) in biasing contexts (e.g., 34_67 or MU_IC), and followed this by a Stroop-like task.
• This was followed this by a Stroop-like task where J had to name the coloured of the target. The colour of the target could match the synaesthetic colour for number 5 (pink) or for the letter S (green).
• found that the colour of the ambiguous grapheme was determined by the context.
• J was faster at naming pink when they saw
• J was faster at naming green when they saw
• The researchers suggested that it is the identity of a grapheme (top-down feature), and not just its physical appearance (bottom-up feature), that determines synaesthetic colour.

Question 22

what is the correlation between culture and synaesthesia?

Answer 22

• Colour-grapheme synaesthesia is common in aural-oral society and visual-literate one.
• Howes (2006) suggested that culture organises the senses which descend via the psychological to the physiological level of the brain. They suggested that drawing up an inventory of the range of cultural practices and technologies can help us understand different sensory combinations and types of synaesthesias across different cultures and historical periods.

Question 23

what is said about people hearing smells?

Answer 23

• Stevenson and Boakes (2004)’s claim that “Odor display taste properties but do not elicit auditory or visual sensations” may be true for western societies but there is evidence of audio-olfactory synasthesia in other cultures.
• For example, people in Papua-New Guinea and Trobriand Islands talk about hearing smells.
• This audio-olfactory synasthesia may have arisen because in these locations, most communication takes place face-to-face (i.e. within olfactory range of the other) and odoriferous substances (e.g. anointing the body with oil, chewing ginger) are regularly used to augment the power of a person’s presence and words (Howes, 2003).

Question 24

what is said about the Dogon culture and hearing smells?

Answer 24

• Hearing-smell associations are also common in various African languages, such as Dogon. To those who speak Dogon, speech has material properties that despite its invisible nature are more than just sounds. It has an odour because sound and odour both arise from vibrations (Calame-Griaule, 1986).
• Dogon also classify words by smell: good speech smells sweet, and bad or impetuous speech smells rotten. A Dogon expression states that “the mouth is too ready to speak is likened to the rectum”.
• Consistent with the intrinsic connection between sound and odour: The nose rings are believed to have the super-additive function of modulating both her hearing and her speech, by (1) promoting the reception and utterances of “good-smelling words” and by (2) deflecting or repressing of the bad-smelling ones.

Question 25

what did Wassily Kandinsky study?

Answer 25

- Kandinsky believed his art could evoke the sound of music and used the terms “improvisations” and “compositions” to describe his paintings. Composition VII—with its swirling vortex of colour and symbols—was often described as “operatic.” - Kandinsky’s use of circles, grids, semicircles, triangles, and other mathematical forms reflect his belief that shape and colour alone could communicate emotion and sounds. - According to Kandinsky, black signified silence while orange indicated the alto range. Yellow, forming a halo around the blue and red circles, represents explosions of sounds, such as loud trumpets and fanfares.