Search

Cochlear implants have been the most successful neural prosthesis, with one million users globally. Researchers used the source-filter model and speech vocoder to design the modern multi-channel implants, allowing implantees to achieve 70%–80% correct sentence recognition in quiet, on average. Researchers also used the cochlear implant to help understand basic mechanisms underlying loudness, pitch, and cortical plasticity. While front-end processing advances improved speech recognition in noise, the unilateral implant speech recognition in quiet has plateaued since the early 1990s. This lack of progress calls for action on re-designing the cochlear stimulating interface and collaboration with the general neurotechnology community.

© 2015 Acoustical Society of America. Long-term loudness perception of a sound has been presumed to depend on the spatial distribution of activated auditory nerve fibers as well as their temporal firing pattern. The relative contributions of those two factors were investigated by measuring loudness adaptation to sinusoidally amplitude-modulated 12-kHz tones. The tones had a total duration of 180s and were either unmodulated or 100%-modulated at one of three frequencies (4, 20, or 100Hz), and additionally varied in modulation depth from 0% to 100% at the 4-Hz frequency only. Every 30 s, normal-hearing subjects estimated the loudness of one of the stimuli played at 15dB above threshold in random order. Without any amplitude modulation, the loudness of the unmodulated tone after 180s was only 20% of the loudness at the onset of the stimulus. Amplitude modulation systematically reduced the amount of loudness adaptation, with the 100%-modulated stimuli, regardless of modulation frequency, maintaining on average 55%-80% of the loudness at onset after 180s. Because the present low-frequency amplitude modulation produced minimal changes in long-term spectral cues affecting the spatial distribution of excitation produced by a 12-kHz pure tone, the present result indicates that neural synchronization is critical to maintaining loudness perception over time.

Across bilateral cochlear implants, contralateral threshold shift has been investigated as a function of electrode difference between the masking and probe electrodes. For contralateral electric masking, maximum threshold elevations occurred when the position of the masker and probe electrode was approximately place-matched across ears. The amount of masking diminished with increasing masker-probe electrode separation. Place-dependent masking occurred in both sequentially implanted ears, and was not affected by the masker intensity or the time delay from the masker onset. When compared to previous contralateral masking results in normal hearing, the similarities between place-dependent central masking patterns suggest comparable mechanisms of overlapping excitation in the central auditory nervous system.

The addition of low-passed (LP) speech or even a tone following the fundamental frequency (F0) of speech has been shown to benefit speech recognition for cochlear implant (CI) users with residual acoustic hearing. The mechanisms underlying this benefit are still unclear. In this study, eight bimodal subjects (CI users with acoustic hearing in the non-implanted ear) and eight simulated bimodal subjects (using vocoded and LP speech) were tested on vowel and consonant recognition to determine the relative contributions of acoustic and phonetic cues, including F0, to the bimodal benefit. Several listening conditions were tested (CI/Vocoder, LP, T(F0-env), CI/Vocoder + LP, CI/Vocoder + T(F0-env)). Compared with CI/Vocoder performance, LP significantly enhanced both consonant and vowel perception, whereas a tone following the F0 contour of target speech and modulated with an amplitude envelope of the maximum frequency of the F0 contour (T(F0-env)) enhanced only consonant perception. Information transfer analysis revealed a dual mechanism in the bimodal benefit: The tone representing F0 provided voicing and manner information, whereas LP provided additional manner, place, and vowel formant information. The data in actual bimodal subjects also showed that the degree of the bimodal benefit depended on the cutoff and slope of residual acoustic hearing.

At present commercially available bilateral cochlear implants (CIs) improve their users' speech understanding in noise but they employ two independent speech processors that cannot provide accurate and appropriate interaural level and time differences as seen binaurally in normal hearing (NH) listeners. Previous work suggests that binaural cues are accessible to bilateral CI users when presented to single pairs of pitch-matched electrodes, but the scope was limited and the mechanisms remained unclear. In this study, binaural masking level differences (BMLDs) were measured in five bilateral Nucleus-24 CI users over multiple pairs of pitch-matched electrodes. Average BMLD was 4.6+/-4.9 dB, but large individual variability prevented significance (p=0.09). Considering just the 125 Hz condition, as in previous work, phase (N0S0 vs N0Spi) and electrode effects were significant. Compared with simulated bilateral CI users, actual bilateral CI users had proportionally higher thresholds for N0Spi than N0S0. Together the present results suggest that the performance gap in BMLDs between CI and NH listeners is not due to a lack of sufficient acoustic cues in the temporal envelope domain but to a true binaural deficit related to a central mechanism in deprived binaural processing.

The possibility that "dead regions" or "spectral holes" can account for some differences in performance between bilateral cochlear implant (CI) users and normal-hearing listeners was explored. Using a 20-band noise-excited vocoder to simulate CI processing, this study examined effects of spectral holes on speech reception thresholds (SRTs) and spatial release from masking (SRM) in difficult listening conditions. Prior to processing, stimuli were convolved through head-related transfer-functions to provide listeners with free-field directional cues. Processed stimuli were presented over headphones under binaural or monaural (right ear) conditions. Using Greenwood's [(1990). J. Acoust. Soc. Am. 87, 2592-2605] frequency-position function and assuming a cochlear length of 35 mm, spectral holes were created for variable sizes (6 and 10 mm) and locations (base, middle, and apex). Results show that middle-frequency spectral holes were the most disruptive to SRTs, whereas high-frequency spectral holes were the most disruptive to SRM. Spectral holes generally reduced binaural advantages in difficult listening conditions. These results suggest the importance of measuring dead regions in CI users. It is possible that customized programming for bilateral CI processors based on knowledge about dead regions can enhance performance in adverse listening situations.

A cochlear implant vocoder was used to evaluate relative contributions of spectral and binaural temporal fine-structure cues to speech intelligibility. In Study I, stimuli were vocoded, and then convolved through head related transfer functions (HRTFs) to remove speech temporal fine structure but preserve the binaural temporal fine-structure cues. In Study II, the order of processing was reversed to remove both speech and binaural temporal fine-structure cues. Speech reception thresholds (SRTs) were measured adaptively in quiet, and with interfering speech, for unprocessed and vocoded speech (16, 8, and 4 frequency bands), under binaural or monaural (right-ear) conditions. Under binaural conditions, as the number of bands decreased, SRTs increased. With decreasing number of frequency bands, greater benefit from spatial separation of target and interferer was observed, especially in the 8-band condition. The present results demonstrate a strong role of the binaural cues in spectrally degraded speech, when the target and interfering speech are more likely to be confused. The nearly normal binaural benefits under present simulation conditions and the lack of order of processing effect further suggest that preservation of binaural cues is likely to improve performance in bilaterally implanted recipients.

The present study evaluated auditory-visual speech perception in cochlear-implant users as well as normal-hearing and simulated-implant controls to delineate relative contributions between sensory experience and cues. Auditory-only, visual-only, or auditory-visual speech perception was examined in the context of categorical perception, in which an animated face mouthing /ba/, /da/, or /ga/ was paired with synthesized phonemes from an 11-token auditory continuum. A 3-alternative, forced-choice, method was used to yield percent identification scores. Normal-hearing listeners showed sharp phoneme boundaries and strong reliance on the auditory cue, whereas actual and simulated implant listeners showed much weaker categorical perception but stronger dependence on the visual cue. The implant users were able to integrate both congruent and incongruent acoustic and optical cues to derive relatively weak but significant auditory-visual integration. This auditory-visual integration was correlated with the duration of the implant experience but not the duration of deafness. Compared with the actual implant performance, acoustic simulations of the cochlear implant could predict the auditory-only performance but not the auditory-visual integration. These results suggest that both altered sensory experience and improvised acoustic cues contribute to the auditory-visual speech perception in cochlear-implant users.

Cochlear-implant users perform far below normal-hearing subjects in background noise. Speech recognition with varying numbers of competing female, male, and child talkers was evaluated in normal-hearing subjects, cochlear-implant users, and normal-hearing subjects utilizing an eight-channel sine-carrier cochlear-implant simulation. Target sentences were spoken by a male. Normal-hearing subjects obtained considerably better speech reception thresholds than cochlear-implant subjects; the largest discrepancy was 24 dB with a female masker. Evaluation of one implant subject with normal hearing in the contralateral ear suggested that this difference is not caused by age-related disparities between the subject groups. Normal-hearing subjects showed a significant advantage with fewer competing talkers, obtaining release from masking with up to three talker maskers. Cochlear-implant and simulation subjects showed little such effect, although there was a substantial difference between the implant and simulation results with talker maskers. All three groups benefited from a voice pitch difference between target and masker, with the female talker providing significantly less masking than the male. Child talkers produced more masking than expected, given their fundamental frequency, syllabic rate, and temporal modulation characteristics. Neither a simulation nor testing in steady-state noise predicts the difficulties cochlear-implant users experience in real-life noisy situations.

Natural spoken language processing includes not only speech recognition but also identification of the speaker's gender, age, emotional, and social status. Our purpose in this study is to evaluate whether temporal cues are sufficient to support both speech and speaker recognition. Ten cochlear-implant and six normal-hearing subjects were presented with vowel tokens spoken by three men, three women, two boys, and two girls. In one condition, the subject was asked to recognize the vowel. In the other condition, the subject was asked to identify the speaker. Extensive training was provided for the speaker recognition task. Normal-hearing subjects achieved nearly perfect performance in both tasks. Cochlear-implant subjects achieved good performance in vowel recognition but poor performance in speaker recognition. The level of the cochlear implant performance was functionally equivalent to normal performance with eight spectral bands for vowel recognition but only to one band for speaker recognition. These results show a disassociation between speech and speaker recognition with primarily temporal cues, highlighting the limitation of current speech processing strategies in cochlear implants. Several methods, including explicit encoding of fundamental frequency and frequency modulation, are proposed to improve speaker recognition for current cochlear implant users.

Speech recognition in noise and music perception is especially challenging for current cochlear implant users. The present study utilizes the residual acoustic hearing in the nonimplanted ear in five cochlear implant users to elucidate the role of temporal fine structure at low frequencies in auditory perception and to test the hypothesis that combined acoustic and electric hearing produces better performance than either mode alone. The first experiment measured speech recognition in the presence of competing noise. It was found that, although the residual low-frequency (

When instructed to speak clearly for people with hearing loss, a talker can effectively enhance the intelligibility of his/her speech by producing "clear" speech. We analyzed global acoustic properties of clear and conversational speech from two talkers and measured their speech intelligibility over a wide range of signal-to-noise ratios in acoustic and electric hearing. Consistent with previous studies, we found that clear speech had a slower overall rate, higher temporal amplitude modulations, and also produced higher intelligibility than conversational speech. To delineate the role of temporal amplitude modulations in clear speech, we extracted the temporal envelope from a number of frequency bands and replaced speech fine-structure with noise fine-structure to simulate cochlear implants. Although both simulated and actual cochlear-implant listeners required higher signal-to-noise ratios to achieve normal performance, a 3-4 dB difference in speech reception threshold was preserved between clear and conversational speech for all experimental conditions. These results suggest that while temporal fine structure is important for speech recognition in noise in general, the temporal envelope carries acoustic cues that contribute to the clear speech intelligibility advantage.

Frequency modulation (FM) detection was investigated in acoustic and electric hearing to characterize cochlear-implant subjects' ability to detect dynamic frequency changes and to assess the relative contributions of temporal and spectral cues to frequency processing. Difference limens were measured for frequency upward sweeps, downward sweeps, and sinusoidal FM as a function of standard frequency and modulation rate. In electric hearing, factors including electrode position and stimulation level were also studied. Electric hearing data showed that the difference limen increased monotonically as a function of standard frequency regardless of the modulation type, the modulation rate, the electrode position, and the stimulation level. In contrast, acoustic hearing data showed that the difference limen was nearly a constant as a function of standard frequency. This difference was interpreted to mean that temporal cues are used only at low standard frequencies and at low modulation rates. At higher standard frequencies and modulation rates, the reliance on the place cue is increased, accounting for the better performance in acoustic hearing than for electric hearing with single-electrode stimulation. The present data suggest a speech processing strategy that encodes slow frequency changes using lower stimulation rates than those typically employed by contemporary cochlear-implant speech processors.

It is important to know what cues the sensory system extracts from natural stimuli and how the brain uses them to form perception. To explore this issue, Smith, Delgutte, and Oxenham [Nature (London) 416, 87–90 (2002)] mixed one sound’s temporal envelope with another sound’s fine temporal structure to produce auditory chimaeras and found that “the perceptual importance of the envelope increases with the number of frequency bands, while that of the fine structure diminishes.” This study addressed two technical issues related to natural cochlear filtering and artificial filter ringing in the chimaerizing algorithm. In addition, this study found that the dichotomy in auditory perception revealed by auditory chimaeras is an epiphenomenon of the classic dichotomy between low- and high-frequency processing. Finally, this study found that the temporal envelope determines sound location as long as the interaural level difference cue is present. The present result reinforces the original hypothesis that the temporal envelope is critical for speech perception whereas temporal fine structure is critical for pitch perception, but does not support the assertion regarding the temporal envelope and fine structure as the acoustic basis for the “what” and “where” mechanisms.

Auditory neuropathy affects synaptic encoding or neural conduction of signals in the cochlea or the auditory nerve. Subjects with auditory neuropathy poorly recognize speech in noise which correlates with poor temporal processing. The integrity of temporal processes in the auditory system can be assessed with detection of just-noticeable differences in gap between sounds. Disorder in the auditory periphery appears to alter the precise timing or latency of synchronous neural discharges important for temporal coding. However, the relative contribution of auditory nerve activities to central temporal processing is unknown. Auditory neuropathy produced significantly worse than normal gap detection within a frequency but normal gap detection between different frequencies. No correlation between same- and different-frequency gap detection supports two temporal processes: a peripheral mechanism dependent on overlapping nerve fibers mediating same-frequency gaps and a central mechanism dependent on cross-correlat...

Traditional loudness models have been based on the average energy and the critical band analysis of steady-state sounds. However, most environmental sounds, including speech, are dynamic stimuli, in which the average level [e.g., the root-mean-square (rms) level] does not account for the large temporal fluctuations. The question addressed here was whether two stimuli of the same rms level but different peak levels would produce an equal loudness sensation. A modern adaptive procedure was used to replicate two classic experiments demonstrating that the sensation of "beats" in a two- or three-tone complex resulted in a louder sensation [E. Zwicker and H. Fastl, Psychoacoustics-Facts and Models (Springer-Verlag, Berlin, 1990)]. Two additional experiments were conducted to study exclusively the effects of the temporal envelope on the loudness sensation of dynamic stimuli. Loudness balance was performed by normal-hearing listeners between a white noise and a sinusoidally amplitude-modulated noise in one experiment, and by cochlear implant listeners between two harmonic stimuli of the same magnitude spectra, but different phase spectra, in the other experiment. The results from both experiments showed that, for two stimuli of the same rms level, the stimulus with greater temporal fluctuations sometimes produced a significantly louder sensation, depending on the temporal frequency and overall stimulus level. In normal-hearing listeners, the louder sensation was produced for the amplitude-modulated stimuli with modulation frequencies lower than 400 Hz, and gradually disappeared above 400 Hz, resulting in a low-pass filtering characteristic which bore some similarity to the temporal modulation transfer function. The extent to which loudness was greater was a nonmonotonic function of level in acoustic hearing and a monotonically increasingly function in electric hearing. These results suggest that the loudness sensation of a dynamic stimulus is not limited to a 100-ms temporal integration process, and may be determined jointly by a compression process in the cochlea and an expansion process in the brain. A level-dependent compression scheme that may better restore normal loudness of dynamic stimuli in hearing aids and cochlear implants is proposed.

Cochlear implants produce highly unnatural temporal and spatial (tonotopic) patterns of neural activation in the auditory system, and the patterns may vary considerably from patient to patient. In spite of these abnormal patterns, psychophysical measures of temporal processing are relatively normal in patients with cochlear implants and brainstem implants. A high level of speech recognition is possible with only three bands of modulated noise [Shannon et al., Science 270, 303–304 (1995)]. What parameters of the peripheral activity pattern are critical for the transmission of speech pattern information? Four‐band speech processors were constructed which reduced the spectral information in speech to four, amplitude‐modulated noise bands. Variations were constructed that altered the location, spectral extent, overlap, and amplitude compression of the four noise bands relative to the original speech. Manipulations of the band overlap and the cut‐off frequencies dividing the bands had relatively little effect ...

The growth of loudness of a tone burst following an intense forward masker was measured as a function of the tone level. The level of the forward-masked tone was adjusted to balance the loudness of a standard tone presented without a forward masker, using a 2AFC, double-staircase, tracking procedure. The forward masker was a 90-dB SPL, 100-ms, 1000-Hz pure tone. The standard tone and the masked tone were both 25-ms, 1000-Hz pure tones. The forward masker and the masked tone were always presented in the first interval. With a 100-ms delay between them, there was little or no threshold elevation for the masked tone. However, the masker caused the masked tone to sound louder than it would if it had not been masked, a phenomenon termed "loudness enhancement" [Irwin and Zwislocki, Percept. Psychophys. 10, 189-192 (1971); Galambos et al., J. Acoust. Soc. Am. 52, 1127-1130 (1972)]. In addition, the present results show a nonmonotonic enhancement function that the forward masker introduced a 10-16-dB enhancement effect for tones of 40-65 dB SPL and no significant effect for the 30 and 90 dB SPL tones. The loudness variability in forward masking was estimated from the upper and lower sequences tracking the 21% and the 79% louder response levels on the psychometric function, respectively. The variability demonstrated a similar nonmonotonic function. In forward masking loudness grows more steeply at low-medium sensation levels, and merges with normal growth at high levels.(ABSTRACT TRUNCATED AT 250 WORDS)

The cochlear implant is a product of deliberate systems engineering, starting with a clear design goal by physicians to restore human hearing by means of electrical stimulation. The engineering development of the cochlear implant has been a dynamic interplay between a perfectionist's approach attempting to replicate the intricate sensing and processing in a normal biological system and a reductionist's approach building an artificial system with minimal components and complexity. The reductionist's approach won the initial battle as the first FDA-approved single-channel cochlear implant consisted of simply a microphone, an amplifier, and a pair of coils. Borrowing techniques and knowhow’s from diverse engineering areas in acoustics, aerospace, and electronics, modern multi-channel cochlear implants have shifted towards the perfectionist's approach. Acoustic researchers, particularly speech scientists, have played an important role in the engineering development of the cochlear implant. From an engineering perspective, I will highlight development milestones, evaluate their contributions to actual implant performance, and delineate the complicated relationships between artificial and natural hearing.

In 1990, Bertram Scharf and I discussed about me doing a post doc in his laboratory. The opportunity to work with Bertram did not materialize, but his work in loudness, efferents, and attention has been a continuing inspiration not only for my own research but also for auditory perception, physiology and audio engineering in general. For example, a Google Scholar search of "Bertram Scharf" on January 3, 2013 produced 1526 citations for his top 10 papers, with 6 on loudness and critical bands, 2 on efferents and 2 on attention. Here I highlight several recent studies that have been inspired by Scharf's work. The first study showed significant loudness adaptation in patients with auditory neuropathy, particularly those with otoferlin deficits. This result directly supports Scharf's proposition that simple loudness adaptation is due to limited excitation in the sensory process, which in this case can be pinned down to transmitter release and replenishment in the hair cell and nerve synapse. The second study extended Scharf's theoretical work in efferents and attention to improving feedback control in cochlear implant users and tinnitus sufferers. This line of work could help improve cochlear implant speech performance and reduce internal gain to alleviate tinnitus.

A unique characteristic of tonal language processing is the use of multiple acoustic cues that are distributed in both time and frequency domains to achieve robust recognition of tones. The temporal cues can be represented by simple acoustic differences in duration or amplitude contours that are correlated to tonal patterns. The spectral cues can be represented directly by fundamental frequency or indirectly by its harmonics. Because contemporary cochlear implants process only the less salient temporal cues, their users have generally poor tonal processing abilities. The present study takes advantage of residual low-frequency acoustic hearing (e.g., < 500 Hz) that is often present in the cochlear implant users. Although this low-frequency hearing contributes little to speech intelligibility directly, it is sufficient to convey tonal information via fundamental frequency. The present study measures word recognition in noise in a group of Mandarin-speaking cochlear implant users who have significant residua...

Cochlear implant performance is improved by extracting pitch information and encoding such pitch information into the processor of a cochlear implant. One embodiment of the invention is to explicitly extract the pitch and deliver it to the cochlear implant by co-varying the stimulate site and rate. Another embodiment of the invention is to implicitly encode the pitch information via a code book that serves as the carrier of stimulation in the cochlear implant.