The construction industry is a high-risk and high accident rate industry, and it is crucial to conduct safety inspections on construction sites. Therefore, the study introduces an improved YOLOX algorithm and performs lightweight processing such as replacing the backbone network and pruning channels. At the same time, the optimized YOLOX algorithm will be applied to the construction of a model for safety detection in intelligent building construction sites. Results showed that the improved model proposed in the study had the best inference speed and average accuracy, with an average accuracy of 95.01%. In the experimental analysis under different detection categories, the model proposed in the study had the highest detection accuracy for whether to wear a safety helmet, with an accuracy rate of 96.39%, which was 10.05% higher than the YOLOX model. At the same time, the accuracy of the model in detecting whether to wear welding masks, masks, and welding gloves was as high as 92.37%, 94.49%, and 94.61%, respectively. In addition, the recall rate of the model proposed by the research institute in helmet wearing detection was as high as 95.48%, which was 12.57% higher than the YOLOX model. The improved model proposed by the research institute has performed well in safety inspection of construction sites, not only possessing high-speed processing capabilities but also high-precision detection performance, providing reliable technical support for real-time monitoring and early warning of intelligent building construction.