Generalized quantum measurements identifying nonorthogonal states without ambiguity often play an indispensable role in various quantum applications. For such an unambiguous state-discrimination scenario, we have a finite probability of obtaining inconclusive results and minimizing the probability of inconclusive results is of particular importance. In this paper, we experimentally demonstrate an adaptive generalized measurement that can unambiguously discriminate the quaternary phase-shift-keying coherent states with a near-optimal performance. Our scheme is composed of displacement operations, single-photon detections and adaptive control of the displacements dependent on a history of photon-detection outcomes. Our experimental results show a clear improvement of both a probability of conclusive results and a ratio of erroneous decision caused by unavoidable experimental imperfections over conventional static generalized measurements.POPULAR SUMMARYIn our everyday life, we are usually able to tell two similar but different things apart by choosing the right way of observing them. Even “identical” twins will have different personalities or perhaps birthmarks in different places. This is no longer the case when looking at quantum objects. Quantum states that are nonorthogonal, i.e., have a certain overlap, cannot be perfectly distinguished, no matter what measurement you attempt. This is a central property of quantum mechanics.Quantum state discrimination is the task of figuring out which quantum state you have at hand, typically out of a small set of candidates. When you cannot distinguish the states perfectly, you will need to decide what to give up on: You can give up trying to know for certain which state you have. Alternatively, you can give up trying to get a conclusive answer every time. Depending on what you are trying to accomplish, either of those strategies may be advantageous.In this work, we study the latter approach, unambiguous state discrimination in the technologically relevant context of distinguishing four very weak laser pulses with distinct phases. We introduce and demonstrate a measurement scheme, based on photon detection and fast feedback to the laser pulse, which significantly improves on the best existing schemes. We obtain near-optimal probabilities of obtaining a conclusive answer and low error probabilities. We expect that quantum key distribution as well as classical optical communication, which is ultimately limited by quantum noise, may benefit from our newly developed technique.