Digital video is an important data format in multimedia information systems. Traditional video representation is just a time sequence——video stream, thus it is difficult for computer to recognize or perceive video in the content level. To efficiently access and utilize video information, a suitable organization of video data is critical. This paper proposes a video organization scheme, which arranges video into four layers： video program, episode, shot and image frame. This hierarchical structure provides a compact and meaningful video catalogue, which can be easily used for non-linear browsing and content-based retrieval of video data. To achieve such an organization, it needs not only detect the boundary of shots and episodes, but also extract the key frames of shots and select the representative shots and frames for episodes. This paper proposes a number of suitable criteria and techniques for video segmentation and organization, and integrates these techniques into a prototype system. Some organization results using real video data are presented, which show the effectiveness of this organization scheme.