A quantitative energy leakage model was developed based on the thermography image data measured for both external and internal building surfaces. The infrared thermography images of both surfaces of doors, windows, and walls of an office building in the Hengqin Campus of University of Macao were taken at various times in a day for four seasons. The transient heat flux for sample units were obtained based on measurements of the seasonal transient local temperature differences and calculations of the effective thermal conductivity from the multiple-layer porous medium conduction model. Effects of construction unit types, orientations, and seasons were quantitatively investigated with unit transient orientation index factors. The corresponding electric energy consumption was calculated based on the air conditioning system coefficient of performance of heat pump and refrigerator cycles for different seasons. The model was validated by comparing to the electric meter records of energy consumption of the air conditioning system. The uncertainties of the predicted total building energy leakage are about 14.7%, 12.8%, 12.4%, and 15.8% for the four seasons, respectively. The differences between the predicted electric consumption and meter values are less than 13.4% and 5.4% for summer and winter, respectively. The typical daily thermal energy leakage value in winter is the highest among the four seasons. However, the daily electric energy consumption by the air conditioning system in summer and autumn is higher than that in winter. The present decomposition model for energy leakage is expected to provide a practical tool for quantitative analysis of energy leakage of buildings.