Abstract Aim To develop and validate a deep learning-based AI system for the dynamic, real-time differentiation of benign and malignant gastric ulcers during endoscopy, with the goal of enhancing diagnostic precision. Methods This was a multicenter, retrospective study collecting endoscopic images and videos from four tertiary hospitals in China. An improved YOLOv8 model, incorporating an illumination attention module, was developed for real-time instance segmentation and classification. The dataset comprised 9,820 benign ulcer images, 1,727 malignant ulcer images, and 15,791 normal mucosa images, split into training, testing, and validation sets at an 8:1:1 ratio. Performance was evaluated based on precision, recall, specificity, and processing latency. Results On the validation set, the AI model achieved an overall precision, recall, and specificity of 0.91, 0.91, and 0.95, respectively. For malignant ulcer recognition specifically, the precision, recall, and specificity were 0.90, 0.91, and 0.99. The model demonstrated strong real-time performance with a latency of 8.84 ms per frame and a processing speed of 113 frames per second. Conclusion The developed AI model enables accurate, real-time discrimination between benign and malignant gastric ulcers during endoscopy. It holds potential to augment clinical decision-making, standardize diagnostic quality, and optimize biopsy strategies.