Damage is defined as changes to the material and/or geometric properties of a structural system, comprising changes to the boundary conditions and system connectivity, adversely affecting the system’s performance. Inspecting the elements of structures, particularly critical components, is vital to evaluate the structural lifespan and safety. In this study, an optimization-based method for joint damage identification of moment frames using the time-domain responses is introduced. The beam-to-column connection in a metallic moment frame structure is modeled by a zero-length rotational spring at both ends of the beam element. For each connection, an end-fixity factor is specified, which changes between 0 and 1. Then, the problem of joint damage identification is converted to a standard optimization problem. An objective function is defined using the nodal point accelerations extracted from the damaged structure and an analytical model of the structure in which the nodal accelerations are obtained using the Newmark procedure. The optimization problem is solved by an improved differential evolution algorithm (IDEA) for identifying the location and severity of the damage. To assess the capability of the proposed method, two numerical examples via different damage scenarios are considered. Then, a comparison between the proposed method and the existing damage identification method is provided. The outcomes reveal the high efficiency of the proposed method for finding the severity and location of joint damage considering noise effects.