Recently, novel applications in the space of artificial intelligence (AI) such as solving constraint optimization problems, probabilistic inferencing, contextual adaptation, and continual learning from noisy data are gaining momentum to address relevant real-world problems. A majority of these tasks are compute and/or memory intensive. While traditional deep learning has been fueled by the utilization of graphic processing units (GPUs) to accelerate algorithms primarily in the cloud, today we see a surge in the development of application/domain-specific integrated circuits and systems that aim at providing an order of magnitude improvement over traditional GPU-based approaches in terms of energy efficiency and latency. This growing branch of research taps into the realms of neuronal dynamics, collective computing using dynamical systems, harnessing stochasticity to enable probabilistic computing, and even draws inspiration from quantum computing. We envision such specialized application/domain-specific systems to perform complex tasks such as solving NP-hard optimization problems, performing reasoning and cognition in the presence of uncertainty with superior energy-efficiency (and/or area and latency improvements) compared to conventional GPU-based approaches and von Neumann computing using traditional silicon-based devices, circuits, and architectures. Of special interest is to utilize such nontraditional computing approaches to reduce the time to obtain solutions for computationally challenging problems that otherwise tend to grow exponentially with problem size. To support this vision, there needs to be fundamental advances in both nontraditional devices and circuits/architectures. Recent works have shown that novel circuit topologies and architectures involving non-Boolean, oscillatory, spiking, probabilistic, or quantum-inspired computing are more suited toward tackling applications such as solving constraint optimization problems, performing energy-based learning, performing Bayesian learning and inference, lifelong continual learning, and solving quantum-inspired applications such as Quantum Monte Carlo. A flurry of current research highlights that compared to traditional silicon-based devices, emerging nanodevices utilizing novel quantum materials such as complex oxides, ferroelectric materials, and spintronic materials can allow the realization of these novel circuits and architectures with lower foot-print area, higher energy efficiency, and lower latency.