Search

Vision Language Models (VLMs) have recently been adopted in robotics for their capability in common sense reasoning and generalizability. Existing work has applied VLMs to generate task and motion planning from natural language instructions and simulate training data for robot learning. In this work, we explore using VLM to interpret human demonstration videos and generate robot task planning. Our method integrates keyframe selection, visual perception, and VLM reasoning into a pipeline. We named it SeeDo because it enables the VLM to ''see'' human demonstrations and explain the corresponding plans to the robot for it to ''do''. To validate our approach, we collected a set of long-horizon human videos demonstrating pick-and-place tasks in three diverse categories and designed a set of metrics to comprehensively benchmark SeeDo against several baselines, including state-of-the-art video-input VLMs. The experiments demonstrate SeeDo's superior performance. We further deployed the generated task plans in both a simulation environment and on a real robot arm.

In this study, we introduce "CosmoAgent," an innovative artificial intelligence framework utilizing Large Language Models (LLMs) to simulate complex interactions between human and extraterrestrial civilizations, with a special emphasis on Stephen Hawking's cautionary advice about not sending radio signals haphazardly into the universe. The goal is to assess the feasibility of peaceful coexistence while considering potential risks that could threaten well-intentioned civilizations. Employing mathematical models and state transition matrices, our approach quantitatively evaluates the development trajectories of civilizations, offering insights into future decision-making at critical points of growth and saturation. Furthermore, the paper acknowledges the vast diversity in potential living conditions across the universe, which could foster unique cosmologies, ethical codes, and worldviews among various civilizations. Recognizing the Earth-centric bias inherent in current LLM designs, we propose the novel concept of using LLMs with diverse ethical paradigms and simulating interactions between entities with distinct moral principles. This innovative research provides a new way to understand complex inter-civilizational dynamics, expanding our perspective while pioneering novel strategies for conflict resolution, which are crucial for preventing interstellar conflicts. We have also released the code and datasets to enable further academic investigation into this interesting area of research. The code is available at https://github.com/MingyuJ666/Simulating-Alien-Civilizations-with-LLM-based-Agents.

Optical frequency division (OFD) has revolutionized microwave and mmWave generation and set spectral purity records owing to its unique capability to transfer high fractional stability from optical to electronic frequencies. Recently, rapid developments in integrated optical reference cavities and microresonator-based optical frequency combs (microcombs) have created a path to transform OFD technology to chip scale. Here, we demonstrate an ultra-low phase noise mmWave oscillator by leveraging integrated photonic components and Kerr optical frequency division. The oscillator derives its stability from an integrated CMOS-compatible SiN coil cavity, and the optical frequency division is achieved spontaneously through Kerr interaction between the injected reference lasers and soliton microcombs in the integrated SiN microresonator. Besides achieving record-low phase noise for integrated mmWave oscillators, our demonstration greatly simplifies the implementation of integrated OFD oscillators and could be useful in applications of Radar, spectroscopy, and astronomy., Comment: 8 pages, 5 figures

The generation of ultra-low noise microwave and mmWave in miniaturized, chip-based platforms can transform communication, radar, and sensing systems. Optical frequency division that leverages optical references and optical frequency combs has emerged as a powerful technique to generate microwaves with superior spectral purity than any other approaches. We demonstrate a miniaturized optical frequency division system that can potentially transfer the approach to a CMOS-compatible integrated photonic platform. Phase stability is provided by a large-mode-volume, planar-waveguide-based optical reference coil cavity and is divided down from optical to mmWave frequency by using soliton microcombs generated in a waveguide-coupled microresonator. Besides achieving record-low phase noise for integrated photonic microwave/mmWave oscillators, these devices can be heterogeneously integrated with semiconductor lasers, amplifiers, and photodiodes, holding the potential of large-volume, low-cost manufacturing for fundamental and mass-market applications., Comment: 8 pages, 4 figures

Millimeter-wave (mmWave) technology continues to draw large interest due to its broad applications in wireless communications, radar, and spectroscopy. Compared to pure electronic solutions, photonic-based mmWave generation provides wide bandwidth, low power dissipation, and remoting through low-loss fiber. However, at high frequencies, two major challenges exist for the photonic system: the power roll-off of the photodiode, and the large signal linewidth derived directly from the lasers. Here, we demonstrate a new photonic mmWave platform by combining integrated microresonator solitons and high-speed photodiodes to address the challenges in both power and coherence. The solitons, being inherently mode-locked, are measured to provide 5.8 dB additional gain through constructive interference among mmWave beatnotes, and the absolute mmWave power approaches the theoretical limit of conventional heterodyne detection at 100 GHz. In our free-running system, the soliton is capable of reducing the mmWave linewidth by two orders of magnitude from that of the pump laser. Our work leverages microresonator solitons and high-speed modified uni-traveling carrier photodiodes to provide a viable path to chip-scale high-power, low-noise, high-frequency sources for mmWave applications., Comment: 9 pages, 5 figures