In the obstacle detection system, a great challenge is the perception of the surrounding environment due to the inherent limitation of the sensor. In this paper, a novel fusion methodology is proposed, which can effectively improve the accuracy of obstacle detection compared with the vision-based system and laser sensor system. This fusion methodology builds a sport model based on the type of obstacle and adopts a decentralized Kalman filter with a two-layer structure to fuse the information of LiDAR and vision sensor. We also put forward a new obstacles-tracking strategy to match the new detection with the previous one. We conducted a series of simulation experiments to calculate the performance of our algorithm and compared it with other algorithms. The results show that our algorithm has no obvious advantage when all the sensors are faultless. However, if some sensors fail, our algorithm can evidently outperform others, which can prove the effectiveness of our algorithm with higher accuracy and robustness.