Search

Sound, as one the most natural way of human communication, has become a ubiquitous modality for human-machine-environment interactions. Despite many environmental sensing capabilities enabled by microphones, sound sensing systems have limitations, such as weak source separation when multiple speakers are present, being prone to replay attacks, and reduced performance under interference and noise. On the other hand, thanks to the availability of next generation communication systems and miniaturized radars, mmWave has become an emerging sensing modality in the recent years. Mobile phones and smart hubs include mmWave radars for environment sensing. To extend the sensing capabilities of these devices, and overcome limitations of microphones, we explore sound sensing and its applications by mmWave radars. In this dissertation, we first explore how and to what extent ambient sound and sound induced vibration could be sensed by mmWave-based sensing. We first establish fundamentals to sense sound from ambient objects, such as a piece of aluminum foil, or active speaker surfaces. We show that, unlike microphones, which sense the sound at the sensor location, radars can sense sound remotely (e.g. from the environment), and robustly. We conduct a variety of experiments to understand the limitations of sound sensing from passive objects. After establishing the fundamentals of sound sensing from the environment, we propose RadioMic, a system that can detect and localize the source of a sound robustly, and enhance the noisy radar signals via deep learning methods. Extensive experiments show how our proposal outperforms existing work and enables sound sensing in challenging conditions, such as through-wall and through-soundproof objects. Furthermore, RadioMic can extract individual sound streams when multiple sources are present. Last, we illustrate how RadioMic can detect whether a source is a live source or an inanimate source, mitigating the vulnerability of microphones again