Track stiffness abrupt change caused by track substructure defects can lead to rather large dynamic irregularity when the train goes through it. Thus the dynamic action between wheel and rail is increasing, which will influence the safety and comfort of the train. In this paper, the dynamic responses of vehicle and track are analyzed based on vehicle track coupling model in which two kinds of defects of track substructure are simulated. To reduce the maintenance cost, a recognition algorithm of track substructure defects using vehicle dynamic responses is proposed based on Support Vector Machines. Finally, a case has been studied based on the algorithm proposed in this paper, and the simulation results show that the method is effective to recognize the track substructure defects such as unsupported sleepers, loosened fasteners, and the accuracy of recognition can be reached to 90%.