Fires in university laboratories often lead to serious casualties and property damage, and traditional sensor-based fire detection techniques suffer from fire warning delays. Current deep learning algorithms based on convolutional neural networks have the advantages of high accuracy, low cost, and high speeds in processing image-based data, but their ability to process the relationship between visual elements and objects is inferior to Transformer. Therefore, this paper proposes an improved YOLOX target detection algorithm combining Swin Transformer architecture, the CBAM attention mechanism, and a Slim Neck structure applied to flame smoke detection in laboratory fires. The experimental results verify that the improved YOLOX algorithm has higher detection accuracy and more accurate position recognition for flame smoke in complex situations, with APs of 92.78% and 92.46% for flame and smoke, respectively, and an mAP value of 92.26%, compared with the original YOLOX algorithm, SSD, Faster R-CNN, YOLOv4, and YOLOv5. The detection accuracy is improved, which proves the effectiveness and superiority of this improved YOLOX target detection algorithm in fire detection.