This paper proposes a novel graph based learning approach to classify agricultural datasets, in which both labeled and unlabelled data are applied to the classification procedure. In order to capture the complex distribution of data, we propose a similarity refinement approach to improve the robustness of traditional label propagation. Then the refined affinity matrix is applied to label propagation. Thus, the traditional pair-wise similarity is updated with scores using median filter of its neighbors in manifold space. And the proposed classification approach can propagate the labels from the labeled data to the whole dataset. The experiments over agricultural datasets have shown that embedding information fusion approach in manifold space is beneficial in classification.