Gas membrane separation process is highly unpredictable due to interacting non-ideal factors, such as composition/pressure-dependent permeabilities and real gas behavior. Although molecular dynamic (MD) simulation can mimic those complex effects, it cannot precisely predict bulk properties due to scale limitations of calculation algorithm. This work proposes a method for modeling a membrane separation process for volatile organic compounds by combining the MD simulation with the free volume theory. This method can avoid the scale-up problems of the MD method and accurately simulate the performance of membranes. Small scale MD simulation and pure gas permeation data are employed to correlate pressure-irrelevant parameters for the free volume theory; by this approach, the microscopic effects can be directly linked to bulk properties (non-ideal permeability), instead of being fitted by a statistical approach. A lab-scale hollow fiber membrane module was prepared for the model validation and evaluation. The comparison of model predictions with experimental results shows that the deviations of product purity are reduced from 10% to less than 1%, and the deviations of the permeate and residue flow rates are significantly reduced from 40% to 4%, indicating the reliability of the model. The proposed method provides an efficient tool for process engineering to simulate the membrane recovery process.