To solve the problem of video image detection in underground coal mine, an improved ViBe algorithm is proposed. First, to reduce the impact of calculation and environmental noise, we delineate the ROI area and perform image conversion and image smoothing. Second, the“ghosting”problem of the original ViBe algorithm and poor detection effect caused by background disturbance can be solved by improving the background modeling initialization method and adaptive threshold. Finally, the relative area of gangue is calculated and compared with the alert value to determine whether there is a large gangue in the picture. The experiment proves that the algorithm proposed in this paper can meet the real-time performance and accurately detect the large gangue appearing in the video image, and the algorithm recognition rate can reach 96.06%.