Abstract: This study addresses the technical challenges of coal gangue detection algorithms during separation in coal mines, such as low recognition accuracy, missed detections and false positives in complex conditions involving uneven brightness distribution, motion blur, similar appearances between coal and gangue, and mutual occlusion. These challenges are addressed using a coal gangue dataset collected from actual coal mines in the Taiyuan region. We propose an improved intelligent detection model based on YOLOv11n, called SDSE-YOLO. The SDSE-YOLO model introduces three innovative structural optimisations for complex conditions. First, the SCINet module, a low-light image enhancement network, is integrated before the YOLOv11n backbone neural network. This module adaptively corrects uneven illumination in images, providing high-quality input for feature extraction and improving detection accuracy in unevenly lit scenes. Secondly, the variable-shape convolution DCNV2 module replaces the traditional CSP (Cross-Stage Partial) module within the backbone neural network. Variable-shape convolution kernels are introduced in the final two C3k2 layers to enable adaptive adjustment of convolution sampling points and enhance the capture of target geometric deformations. This ensures the effective extraction of features from coal and coal gangue, even when there is motion blur or significant geometric distortion, thereby improving the accuracy of detection when coal and coal gangue exhibit motion blur or visual similarity. Thirdly, the SEAM module is introduced at the end of the model to enhance recognition accuracy in scenarios where coal and coal gangue occlude each other. Results on the self-built gangue dataset demonstrate that, compared to the YOLOv11n model, the SDSE-YOLO module achieves improvements of 3.5% in precision (P), 3.6% in recall (R) and 3.8% in mean average precision (mAP). When benchmarked against popular detection algorithms currently in use, the SDSE-YOLO model demonstrates superior accuracy, offering a reliable solution for detecting coal and gangue in complex operational conditions.