Abstract: Aiming at the unfavorable environmental factors such as low brightness of artificial light source and dense distribution of coal gangue in coal mine underground, which lead to the problems of feature extraction difficulty and low localization accuracy of existing coal gangue detection algorithms, a real-time segmentation method of coal gangue based on adaptive low illumination image enhancement is proposed, which consists of CG-IENet (Coal Gangue Image Enhancement Network) and CG-TSNet (Coal Gangue Target Segmentation Network). The CG-IENet is based on the CycleGAN network model framework, which optimizes the design of the encoder, feature extraction network and decoder in the generator network through the attention mechanism, and adopts PatchGAN as the discriminator network to achieve adaptive picture quality enhancement of low illumination coal gangue images, so as to better retain the surface texture information of the coal gangue. The CG-TSNet is based on the DeepLabV3 + network model framework, which introduces the lightweight backbone network MobileNetV2 and the neck network SPS-ASPP module in the encoder, and combines the feature fusion module, the attention mechanism, and the dynamic sampling in the decoder, which improves the model segmentation accuracy and reduces the model complexity effectively at the same time, it enables faster processing speeds and lower energy consumption for easy deployment of edge-based computing devices. The experimental results show that CG-IENet improves 47.86%, 61.20%, 14.30%, 31.77% and 23.68% on average in the peak signal-to-noise ratio, structural similarity index, information fidelity criterion, entropy and gray mean, respectively, compared with the other low illumination image enhancement models, and the mean chromaticity error is as low as 1.18. The three-dimensional gray scale distribution graph and gray scale histogram are used to visually compare and analyze the various models, which finally shows that CG-IENet has the best enhancement effect, which can better improve the brightness of the image to avoid color distortion while retaining the detailed information of the image. CG-TSNet has the highest mean intersection over union, mean pixel accuracy, pixel accuracy, and F1-Score (F₁) metrics, 91.76%, 95.84%, 96.85%, and 95.04%, respectively, and the best overall detection performance when compared to the other four semantic segmentation models, such as HRNet, UNet, and PSPNet, on the basis of the guaranteed model memory of 4.567 MB. The model can be adapted to the complex working conditions in underground coal mines, thus realizing accurate and efficient sorting of coal gangue.