The development of computer and digital technology has accelerated the reform in the field of architectural design. For enhancing the conceptual expression of architectural design and enhance the interactive experience of design, this study explores architectural design methods on the ground of image deep learning image recognition technology and augmented reality technology. The experiment demonstrates that when the accuracy of the improved You Only Look Once version 4 (YOLOv4) model is 0.9, the recall rate is 0.98, and the curve area is 0.93. The model loss function curve converges to the minimum value of 0.04, with the fastest convergence speed and the highest model recognition efficiency. Its time consumption has decreased by as much as 70.06%, indicating better overall performance. Meanwhile, the clustering strategy design of the model is relatively optimal, with the highest values of purity, standard mutual information, and Rand coefficient reaching 0.944, 0.931, and 0.942, respectively. In practical analysis of architectural design, the improved YOLOv4 model's average accuracy and intersection to union ratio values have confirmed the excellent detection performance of this method. The application of virtual reality technology in building information models has significantly improved the delay rate of visualization, and the user's subjective evaluation level is relatively high. The combination of visible image recognition and augmented reality can achieve intelligent processing and application of drawing information, improve design efficiency and quality, and optimize design experience.