Accurately estimating 3D optical flow in computer-generated holography poses a challenge due to the scrambling of 3D scene information during hologram acquisition. Therefore, to estimate the scene motion between consecutive frames, the scene geometry should be recovered first. Recent studies have demonstrated that a 3D RGB-D representation can be extracted from an input hologram with relatively low error under well-chosen numerical reconstruction parameters. However, limited attention has been given to how the produced error can impact the flow estimation algorithms. Therefore, in this study, we evaluate different learning/non-learning methodologies for recovering 3D scene geometry. Next, we analyze the types of distortions produced by these methods and attempt to minimize estimation error using spatial and temporal constraints. Finally, we compare the performance of several state-of-the-art methods to estimate the 3D optical flow vectors on the recovered sequence of RGB-D images.