Abstract: The weak interfacial transition zone (ITZ) significantly affects the macroscopic flexural strength of coal gangue concrete (CGC). To investigate the influence of soda residue on the pore structure of the coal gangue ITZ and its relationship with CGC flexural strength, this study used CGC designed for floor hardening. The effect of soda residue content on the flexural strength was tested, and backscattered electron (BSE) imaging was employed to analyze the ITZ pore structure characteristics in terms of porosity and the distribution frequency of different pore sizes. Through grey relational analysis and linear regression between pore surface area ranges and flexural strength, the critical pore size boundaries for beneficial pores, less harmful pores, and more harmful pores within the coal gangue ITZ were determined. A quantitative model relating CGC flexural strength to the multi-harm porosity and less harmful porosity of the coal gangue ITZ was established. The results show that with increasing soda residue content from 0 to 12%, the CGC flexural strength first increases and then decreases, while the coal gangue ITZ porosity first decreases and then increases, revealing a negative correlation between flexural strength and ITZ porosity. Larger pore sizes in the ITZ correspond to lower distribution frequencies; pores in the 0.25−1.75 μm range exhibit the highest frequency, and the soda residue content most significantly affects the distribution of pores in the 0.25−0.75 μm range. Using 0.75 and 3.57 μm as critical thresholds, the ITZ pores are classified into beneficial pores, less harmful pores, and more harmful pores. At a soda residue content of 6%, the distribution frequencies of more harmful pores and less harmful pores decrease by 47.06% and 42.02%, respectively. The established quantitative model of soda residue content–coal gangue ITZ pore structure–CGC flexural strength demonstrates high accuracy, with the prediction errors of the validated model for multi-harm porosity and less harmful porosity not exceeding 4.35% and 3.88%, respectively. An appropriate dosage of soda residue can effectively reduce the distribution frequencies of more harmful and less harmful pores, thereby significantly enhancing the compactness of the coal gangue ITZ and the flexural strength of CGC. The research findings can provide theoretical guidance for the resource utilization of solid wastes such as gangue and soda residue, as well as for the performance regulation of coal gangue concrete.