Table of Contents
Developmental Research: Language Acquisition Mechanisms and Deficits
Wang, X., Wang, S. Fan, Y., Huang, D., & Zhang, Y. (2017). Speech-specific categorical perception deficit in autism: An Event-Related Potential study of lexical tone processing in Mandarin-speaking children. Scientific Reports, 7, 43254. Recent studies reveal that tonal language speakers with autism have enhanced neural sensitivity to pitch changes in nonspeech stimuli but not to lexical tone contrasts in their native language. The present ERP study demonstrated that the distinct pitch processing pattern for speech and nonspeech stimuli in autism was due to a speech-specific deficit in categorical perception of lexical tones. In view of the important role of lexical tones in acquiring a tonal language, the results point to the necessity of early intervention for the individuals with autism who show such a speech-specific categorical perception deficit.
Yu, L., Fan, Y., Deng, Z., Huang, D., Wang, S., & Zhang, Y. (2015). Pitch processing in tonal-language-speaking children with autism: An event-related potential study. Journal of Autism and Developmental Disorders, 45(11), 3656-3667. The present study investigated pitch processing in Mandarin-speaking children with autism using event-related potential measures. Two experiments were designed to test how acoustic, phonetic and semantic properties of the stimuli contributed to the neural responses for pitch change detection and involuntary attentional orienting. The results indicate domain specificity of enhanced pitch processing in autism, which may interfere with lexical tone acquisition and language development for children who speak a tonal language.
Zhang, Y., Zhang, L., Shu, H. Xi, J., Zhang, Y., & Li, P. (2012). Universality of categorical perception deficit in developmental dyslexia: An investigation of Mandarin Chinese tones. Journal of Child Psychology and Psychiatry, 53, 874–882. This study investigated categorical perception deficit for Mandarin lexical tones in Chinese children with dyslexia. Both behavioral and electrophysiological data indicate impaired categorical perception of lexical tones in the dyslexic children. The findings support the hypothesis that dyslexic children have a general deficit in categorical perception of speech, which extends to non-alphabetic languages at the suprasegmental level.
Zhang, Y., Koerner, T., Miller, S., Grice-Patil, Z., Svec, A., Akbari, D., Tusler, L., & Carney, E. (2011). Neural coding of formant-exaggerated speech in the infant brain. Developmental Science, 14, 566–581. This event-related potential (ERP) study investigated neural coding of formant-exaggerated speech in 6-12 months old infants. ERP waveform analysis showed significantly enhanced N250 for formant exaggeration, which was more prominent in the right hemisphere than the left. Time-frequency analysis indicated increased neural synchronization for processing formant-exaggerated speech in the delta band at frontal-central-parietal electrode sites as well as in the theta band at frontal-central sites. Minimum norm estimates further revealed a bilateral temporal-parietal-frontal neural network in the infant brain sensitive to formant exaggeration. Collectively, these results provide the first evidence that formant expansion in infant-directed speech enhances neural activities for phonetic encoding and language learning.
Imada, T., Zhang, Y., Cheour, M., Taulu, S., Ahonen, A., & Kuhl, P. K. (2006). Infant speech perception activates Broca's area: a developmental magnetoencephalography study. Neuroreport, 17, 957-962. The activation patterns observed in the superior temporal and inferior frontal regions provide initial evidence for the developmental emergence early in life of a perceptual-motor link for speech perception that may depend on experience.
Kuhl, P. K., Tsao, F. M., Liu, H. M., Zhang, Y., & De Boer, B. (2001). Language/culture/mind/brain: Progress at the margins between disciplines. Annals of New York Academy of Sciences, 935, 136-174. At the forefront of research on language are new data demonstrating infants' strategies in the early acquisition of language. The data show that infants perceptually “map” critical aspects of ambient language in the first year of life before they can speak. Statistical and abstract properties of speech are picked up through exposure to ambient language. Moreover, linguistic experience alters infants' perception of speech, warping perception in a way that enhances native-language speech processing. Infants' strategies are unexpected and unpredicted by historical views. At the same time, research in three additional disciplines is contributing to our understanding of language and its acquisition by children. Cultural anthropologists are demonstrating the universality of adult speech behavior when addressing infants and children across cultures, and this is creating a new view of the role adult speakers play in bringing about language in the child. Neuroscientists, using the techniques of modern brain imaging, are revealing the temporal and structural aspects of language processing by the brain and suggesting new views of the critical period for language. Computer scientists, modeling the computational aspects of childrens' language acquisition, are meeting success using biologically inspired neural networks. Although a consilient view cannot yet be offered, the cross-disciplinary interaction now seen among scientists pursuing one of humans' greatest achievements, language, is quite promising.
Cross-linguistic Research: Neural Commitment and Brain Plasticity
Zhang, Y. (2016). Categorical Perception. In Sybesma, R., Behr, W., Gu, Y., Handel, Z., Huang, J., Myers, J. (Eds.), Encyclopedia of Chinese Language and Linguistics, Leiden, Netherlands: Brill. First online in 2015. Categorical perception provides an account for how human symbolic thinking is grounded in perception and action. The study of Chinese lexical tones offers a unique venue to investigate the origin and development of CP in the brain and its importance in language and cognition.
Zhang, L., Li, Y., Wu, H., Li, X., Shu, H., Zhang, Y., & Li, P. (2016). Effects of semantic context and fundamental frequency contours on Mandarin speech recognition by second language learners. Frontiers in Psychology, 7, 908.
Cheng, B., & Zhang, Y. (2015). Syllable structure universals and native language interference in second language perception and production: Positional asymmetry and perceptual links to accentedness. Frontiers in Psychology, 6, 1801. The present study investigated how syllable structure differences between the first Language (L1) and the second language (L2) affect L2 consonant perception and production at syllable-initial and syllable-final positions. Consistent with previous studies, significant positional asymmetry effects were found across speech sound categories in terms of voicing, places of articulation, and manner of articulation. Furthermore, significant correlations between perception and accentedness ratings were found at the syllable onset position but not for the coda. Many exceptions were also found, which could not be solely accounted for by differences in L1-L2 syllabic structures. The results show a strong effect of language experience at the syllable level, which joins force with acoustic, phonetic, and phonemic properties of individual consonants in influencing positional asymmetry in both domains of L2 segmental perception and production. The complexities and exceptions call for further systematic studies on the interactions between syllable structure universals and native-language interference with refined theoretical models to specify the links between perception and production in second language acquisition.
Wu, H., Ma, X., Zhang, L., Liu, Y., Zhang, Y., & Shu, H. (2015). Musical experience modulates categorical perception of lexical tones in native Chinese speakers. Frontiers in Psychology, 6, 436. Using both identification and discrimination tasks, this study compared Chinese-speaking musicians and non-musicians in their CP of a lexical tone continuum (from the high level tone, Tone1 to the high falling tone, Tone4). While the identification functions showed similar steepness and boundary location between the two subject groups, the discrimination results revealed superior performance in the musicians for discriminating within-category stimuli pairs but not for between-category stimuli. These findings suggest that musical training can enhance sensitivity to subtle pitch differences between within-category sounds in the presence of robust mental representations in service of CP of lexical tonal contrasts.
Zhang, Y. & Cheng, B. (2011). Brain plasticity and phonetic training for English-as-a-second-language learners. In David J. Alonso (Ed.). English as a Second Language, pp. 1-50. Hauppauge, New York: Nova Science Publishers.
English as a second language (ESL) education has gained an increasingly important role in career development in science, business, and industry on the global stage. One great challenge for adult ESL learners is to reduce or eliminate "foreign accent" in their English pronunciation. Decades of behavioral and brain research have shown that language experience early in life leaves an indelible mark on speech perception and production. This chapter examines the underlying causes for the foreign accent problem in second language learning and discusses the approaches to promote brain plasticity through phonetic training. The first section provides a summary of the main research findings to illustrate the role of phonetic knowledge in language learning, the neural mechanisms of speech perception, and the relationship between perception and production. The second section outlines a theoretical framework of brain plasticity for phonetic learning, featuring quantifiable measures to test relevant hypotheses about second language acquisition in terms of neural sensitivity, neural efficiency, neural specificity, and neural connectivity and their behavioral correlates.The third section introduces a synergistic Speech Assessment and Training (SAT) software program, to overcome first-language interference. The chapter concludes with a discussion of the implications for second-language education and future research.
Xi, J., Zhang, L.J., Shu, H., Zhang, Y., & Li, P. (2010). Categorical perception of lexical tones in Chinese revealed by mismatch negativity. Neuroscience, 170, 223–231.
The present study demonstrates that acoustic and phonological information is processed in parallel within the MMN time window for the perception of lexical tones. Moreover, homologous nonspeech stimuli elicited similar MMN patterns in native speakers of Chinese only, indicating that lexical tone knowledge influences the perception of nonspeech signals.
Cheng, B. & Zhang, Y. (2009). Modern foreign language education in the light of brain mechanisms of language acquisition (母语习得的脑神经机制研究及对外语教学的启示). Journal of Xi’an Jiaotong University (Social Sciences) (西安交通大学学报(社科版)), 3, 98-104. Download PDF in Chinese (作者：程冰1; 张旸2）
Zhang, Y., Kuhl, P. K., Imada, T., Iverson, P., Pruitt, J., Stevens, E., Kawakatsu, M., Tohkura, Y., & Nemoto, I. (2009). Neural signatures of phonetic learning in adulthood: A magnetoencephalography study. Neuroimage, 46, 226-240.
The present study provides corroborating evidence that substantial neural plasticity for second-language learning in adulthood can be induced with adaptive and enriched linguistic exposure. Like the MMF, the ECD cluster and duration measures are sensitive neural markers of phonetic learning.
Zhang, Y., & Wang, Y. (2007). Neural plasticity in speech learning and acquisition. Bilingualism: Language and cognition, 10(2), 147-160.
In this selective review, we discuss the role of phonetic learning in language acquisition, the “critical period” of learning, the agents of neural plasticity, and the distinctiveness of linguistic systems in the brain. In particular, we argue for the necessity to look at brain–behavior connections using modern brain imaging techniques, seek explanations based on measures of neural sensitivity, neural efficiency, neural specificity and neural connectivity at the cortical level, and point out some key factors that may facilitate or limit second language learning.
Zhang, Y., Kuhl, P. K., Imada, T., Kotani, M., & Tohkura, Y. (2005). Effects of language experience: neural commitment to language-specific auditory patterns. Neuroimage, 26(3), 703-720.
The behavioral and neuromagnetic results showed that Japanese listeners were less sensitive to the phonemic /r–l/ difference than American listeners. Furthermore, processing non-native speech sounds recruited significantly greater brain resources in both hemispheres and required a significantly longer period of brain activation in two regions, the superior temporal area and the inferior parietal area. We argue that early exposure to a particular language produces a neural commitment to the acoustic properties of that language and that this neural commitment interferes with foreign language processing, making it less efficient.
Zhang, Y., Kuhl, P. K., Imada, T., Iverson, P., Pruitt, J., Stevens, E., Kotani, M., & Tohkura, Y. (2000). Neural plasticity revealed in perceptual training of a Japanese adult listener to learn American /l-r/ contrast: A whole-head magnetoencephalography study. Proceedings of the 6th International Conference on Spoken Language Processing, 3, 953-956.
Zhang, Y., Kuhl, P. K., Imada, T., Kotani, M. & Stevens, E. B. (2000). MMFs to native & nonnative syllables show an effect of linguistic experience. In: Nenonen, J., Ilmoniemi, R. J. & Katila, T. (Eds.) Proceedings of the 12th International Conference on Biomagnetism, 379-382. Helsinki University of Technology, Espoo.
Hearing in Adversity: Hearing Aid Users, Cochlear Implant Users, and Coping with Noisy Environments
Koerner, T. K., Zhang, Y., Nelson, P., Wang, B., & Zou, H. (2017). Neural indices of phonemic discrimination and sentence-level speech intelligibility in quiet and noise: A P3 study. Hearing Research, 350, 58-67.
Jiang, W., Li, Y., Shu, H., Zhang, L., & Zhang, Y. (2017). Use of semantic context and F0 contours by older listeners during Mandarin speech recognition in quiet and single-talker interference conditions. Journal of the Acoustical Society of America, 141, EL338-EL344.
Koerner, T. K., & Zhang, Y. (2017). Application of linear mixed-effects models in human neuroscience research: A comparison with Pearson correlation in two auditory electrophysiology studies. Brain Sciences, 7, 26.
Yu, L., Rao, A., Zhang, Y., Burton, P.C., Rishiq, D., & Abrams, H. (2017). Neuromodulatory effects of auditory training and hearing aid use on audiovisual speech perception in elderly individuals. Frontiers in Aging Neuroscience, 9, 30.
Rao, A., Rishiq, D., Yu, L., Zhang, Y., & Abrams, H. (2017). Neural correlates of selective attention with hearing aid use followed by ReadMyQuips auditory training program. Ear and Hearing, 38, 28–41.
Koerner, T. K., Zhang, Y., Nelson, P., Wang, B., & Zou, H. (2016). Neural indices of phonemic discrimination and sentence-level speech intelligibility in quiet and noise: A mismatch negativity study. Hearing Research, 339, 40-49.
Miller, S., Zhang, Y., & Nelson, P. (2016). Neural correlates of phonetic learning in postlingually deafened cochlear implant listeners. Ear and Hearing, 37, 514-528.
Koerner, T. K., & Zhang, Y. (2015). Effects of background noise on inter-trial phase coherence and auditory N1-P2 responses to speech stimuli. Hearing Research, 328, 113-119.
Miller, S., Zhang, Y., & Nelson, P. (2015). Efficacy of multi-talker phonetic identification training in postlingually deafened cochlear implant listeners. Journal of Speech, Language, and Hearing Research, 59, 90-98. This study implemented a pretest-intervention-posttest design to examine whether multiple-talker identification training enhanced phonetic perception of the /ba/-/da/ and /wa/-/ja/ contrasts in adult postlingually deafened cochlear implant (CI) listeners. The data provide initial evidence for the efficacy of the multiple-talker identification training paradigm for postlingually deafened CI users. This pattern of results is consistent with enhanced phonemic categorization of the trained speech sounds.
Nie, Y., Zhang, Y., & Nelson, P. (2014). Auditory stream segregation using bandpass noises: evidence from event-related potentials. Frontiers in Neuroscience, 8, 277. This study examined MMN and P3 responses that underlie auditory stream segregation of noise stimuli with or without clear spectral contrast. The results suggest that attention facilitates auditory streaming in conditions of weak spectral contrast and reliable ERP measures can be measured as indirect indicators for auditory stream segregation in both passive listening and active listening conditions. The findings have important implications for cochlear implant (CI) studies – as spectral information available through a CI device or simulation is substantially degraded, it may require more attention to achieve stream segregation.
Miller, S., & Zhang, Y. (2014). Validation of the cochlear implant artifact correction tool for auditory electrophysiology. Neuroscience Letters, 577, 51-55. Auditory event-related potentials (ERPs) collected from cochlear implant (CI) users are often contaminated by large electrical device-related artifacts. The purpose of this study was to perform an independent validation of the Cochlear Implant Artifact Correction (Viola et al., 2012) algorithm. We further assessed whether the ERP responses were stable over the course of one year when analyzed manually with the conventional ICA (independent component analysis) approach or the semi-automatic CIAC approach.
Miller, S., & Zhang, Y. (2014). Neural coding of phonemic fricative contrast with and without hearing aid. Ear and Hearing, 35, e122-e133. This ERP paper describes how normal-hearing adult listeners neurally code the phonemic fricative contrast (‘s’ and ‘sh’) in two listening conditions (with or without a hearing aid). While the aided listening condition increased and delayed the N1 and Acoustic Change Complex (ACC) responses, significant differences in the P1-N1-P2 and ACC components were still observed that would support fricative contrast perception at the cortical level. Thus, despite significant alterations in the ERP responses by the aided condition, normal-hearing adult listeners showed distinct neural coding patterns for the voiceless fricative contrast with or without a hearing aid.
Wang, J., Shu, H., Zhang, L., Liu, Z., & Zhang, Y. (2013). The roles of fundamental frequency contours and sentence context in Mandarin Chinese speech intelligibility. Journal of the Acoustical Society of America, 134, EL91-EL97. Previous studies show that flattening the fundamental frequency contours (F0) of Mandarin Chinese sentences reduces their intelligibility in noise but not in quiet. In this study, speech intelligibility was evaluated when participants listened to sentences and word lists with or without F0 variations in quiet and noise. The results showed that sentence context partially explained the unchanged intelligibility of monotonous Chinese sentences. Moreover, F0 variations and sentence context act in concert during speech comprehension.
Sense and Sensibility I: Neural Coding Mechanisms and Computational Models
Rao, A., Koerner, T., Madsen, B., & Zhang, Y. (Submitted). Interactions between cortical mechanisms and medial olivocochlear efferents in auditory perception.
Zhang, Y., Cheng, B., Koerner, T. K., Schlauch, R. S., Tanaka, K., Kawakatsu, M., Nemoto, I., & Imada, T. (2016). Perceptual temporal asymmetry associated with distinct ON and OFF responses to time-varying sounds with rising versus falling intensity: A magnetoencephalography study. Brain Sciences, 6, 27.
Zhang, Y. (2015). Categorical Perception. In Sybesma, R., Behr, W., Gu, Y., Handel, Z., Huang, J., Myers, J. (Eds.), Encyclopedia of Chinese Language and Linguistics, Leiden, Netherlands: Brill. Categorical perception provides an account for how human symbolic thinking is grounded in perception and action. The study of Chinese lexical tones offers a unique venue to investigate the origin and development of CP in the brain and its importance in language and cognition.
Zhang, L., Yue, Q., Zhang, Y., Shu, H., & Li, P. (2015). Task-dependent modulation of regions in the left temporal cortex during auditory sentence comprehension. Neuroscience Letters, 584. 351-355. In this fMRI study, we addressed task-dependent functional specialization of the left temporal cortex in auditory sentence comprehension by applying both independent component analysis (ICA) and general linear model (GLM) methods. Consistent with previous studies, intelligible sentences elicited greater activity in the left lateral temporal cortex relative to unintelligible sentences. Moreover, responses to intelligibility in the sub-regions were differentially modulated by task demands. The anterior and posterior superior temporal sulcus and middle temporal gyrus (STS/MTG) were equally activated during both passive and active tasks, whereas the middle STS/MTG was selectively activated only during the active task. Our results not only confirm the critical role of the left lateral temporal cortex in auditory sentence comprehension but further demonstrate that task demands modulate functional specialization of the anterior-middle-posterior temporal sub-areas.
Zhang, Y., Zhang, W., Reynoso-Alcántara, V., & Silva-Pereyra, J. (2014). Magnetoencefalografía: Mapeo de la dinámica espaciotemporal de la actividad neuronal (Magnetoencephalography:: Mapping spatiotemporal dynamics of neuronal activities.) Suma Psicológica, 21, 45-53. (MEG review paper published in Spanish). Magnetoencephalography (MEG) is a noninvasive imaging technique that measures the magnetic fields outside the human head produced by neuronal currents in brain regions with high temporal accuracy. It is extremely useful in basic and clinical research because MEG can additionally locate the sources of neural activity on an individual basis. In this review, we presented basic aspects of the biophysics of the method and its major findings are in research on speech perception, auditory attention and integration of visual-auditory information. The benefits, limitations and new trends in research with MEG are also illustrated.
Rao, A., Zhang, Y., & Miller, S. (2010). Selective listening of concurrent auditory stimuli: An Event-Related Potential study. Hearing Research, 268, 123-132.
The results show that the essential ERP components for the same compound auditory stimuli are modulated by listeners’ focus on specific aspects of information in the stimuli. Thus changes in ERP components during selective listening reflect domain-general processing mechanisms not specific to dimensional processing for human speech.
Sense and Sensibility II: Language and Social Communication
Diamond, E., & Zhang, Y. (2016). Cortical processing of phonetic and emotional information in speech: A cross-modal priming study. Neuropsychologia, 82, 110-122. This study employed behavioral and electrophysiological measures to investigate the timing, localization, and neural oscillation characteristics of cortical activities associated with phonetic and emotional information processing of speech. Behavioral results showed a congruency effect for both percent correct and reaction time. Two ERP responses, the N400 and late positive response (LPR), were identified in both conditions. Source localization and inter-trial phase coherence of the N400 and LPR components further revealed different cortical contributions and neural oscillation patterns for selective processing of phonetic and emotional information in. speech. The results provide corroborating evidence for the necessity of differentiating brain mechanisms underlying the representation and processing of co-existing linguistic and paralinguistic information in spoken language, which has important implications for theoretical models of speech recognition as well as clinical studies on the neural bases of language and social communication deficits.
Stevens, J., & Zhang, Y. (2014). Brain mechanisms for processing co-speech gesture: A cross-language study of spatial demonstratives. Journal of Neurolinguistics, 30, 27-47. In this ERP study, we examine linguistic relativity in terms of how gesture modulates congruency effect for spatial demonstrative use. Reaction times show similar congruency effects for English and Japanese speakers. Both subject groups show N400 responses to trials with co-speech pointing gesture. An English-specific P600 congruency effect is found in the no-gesture trials. Therefore, brain processing of co-speech gesture is shaped by the language system.
Stevens, J., & Zhang, Y. (2013). Relative distance and gaze in the use of entity-referring spatial demonstratives: An event-related potential study. Journal of Neurolinguistics, 26, 31-45. This study examined how linguistic expressions such as spatial demonstratives (this vs. that) are bound by nonverbal and contextual factors in speech communication. Joint gaze was found to be a primary factor in determining the use of spatial demonstratives. An N400-like effect was observed in the hearer-associated spatial context. The effect was localized in left temporal and bilateral superior parietal areas. The results provide the first neural evidence that the use of spatial demonstratives in English is obligatorily influenced by two factors: (1) shared gaze of speaker and hearer, and (2) the relative distance of the object to the speaker and hearer.