Video-based vibration measurement is a cost-effective way for remote monitoring the health conditions of transportation and other civil structures, especially for situations where accessibility is restricted and does not allow installation of conventional monitoring devices. Besides, video-based system is global measurement. The technical basis of video-based remote vibration measurement system is digital image analysis. Comparison of the images allow the field of motion to be accurately delineated. Such information is important to understand the structure behaviors including the motion and strain distribution. This paper presents system and analyses to monitor the vibration velocity and displacement field. The performance is demonstrated on a testbed of model building. Three different methods (i.e., frame difference method, particle image velocimetry, and optical Flow Method) are utilized to analyze the image sequences to extract the feature of motion. The Performance is validated using accelerometer data. The results indicate that all three methods can estimate the velocity field of the model building, although the results can be affected by factors such as background noise and environmental interference. Optical flow method achieved the best performance among these three methods studied. With further refinement of system hardware and image processing software, it will be developed into a remote video based monitoring system for structural health monitoring of transportation infrastructure to assist the diagnoses of its health conditions.