CHI 97 Electronic Publications: Late-Breaking/Interactive Posters
Synchronized Retrieval of Recorded Multimedia Data
Yukihiro Kawamata, Kimiya Yamaashi, Masayasu Futakawa
Hitachi Research Laboratory, Hitachi Ltd.
7-1-1 Omika-cho, Hitachi-shi, Ibaraki-ken, 319-12 Japan
Email: kawamata@hrl.hitachi.co.jp
ABSTRACT
This paper describes techniques for the retrieval of recorded multimedia data for supervisory control systems. Currently these systems operators can only retrieve recorded data individually. We developed new techniques to access all recorded data is synchronization. The techniques enable users to retrieve multimedia data such as sensor data and videos simultaneously, and also enable users to obtain the desired related data, including objects in videos, by ÒDrag and DropÓ operation. All these techniques allow operators to exactly and quickly analyze phenomena in the systems based on the recorded multimedia data.
Keywords
Data retrieval, multimedia, video, drag and drop
© 1997 Copyright on this material is held by the authors.
INTRODUCTION
Users cannot refer to recorded data in synchronization with current tools. In supervisory control systems, sensor values and alarms are recorded to analyze phenomena in the systems in times of trouble. However, users can only refer to recorded data individually, so that it was difficult to recognize the global condition of the phenomena. People in the systems desire a tool that allows them to see data and videos in synchronization to obtain the real conditions of systems. Furthermore they need to refer to related data by analyzing the phenomena quickly.
Some people have discussed data retrieval of recorded data, but we cannot retrieve the recorded data in synchronization. For example Karon et al. [1] describes video logging tools which enable users to correlate their personal notes or keywords with a videotape. The system allows users to access a video based on data, but they cannot support playback with synchronization between the recorded video and data. However, in supervisory systems, users want to refer to the playback of various recorded data in synchronization in order to analyze phenomena in the systems.
In supervisory systems, users want to access various data according to their interests, but most systems predefine links between data. Tani et al. [2] proposed a technique of retrieving a related data by direct manipulation of the supervisory systems. The technique predefines the relationship between a live video and data based on the positions of objects in the video.
However in analyzing phenomena in the systems, users want to refer to various relationships among data, including videos. We also need to support on-line userÕs definition of relationship among data, including objects in video.
To fulfill the above requirements for analyzing recorded data, we developed the following techniques.
To fulfill the above needs for analyzing recorded data, we developed the following techniques.
- The technique plays back multiple continuous data in synchronization.
- The technique allows users to obtain the desired data directly by ÒDrag and DropÓ operation.
All these techniques allow users to analyze phenomena exactly and quickly.
SYNCHRONIZATION
In analyzing phenomena in the system, users want to play back multiple recorded data in synchronization to find any correlations between the data.
To synchronize the time of recorded data, we record time codes in all the recorded data, including sensor data and video data. When the system plays back recorded data, the system checks the time codes periodically to synchronize the data.
For example if users specify the time on a graph of sensor data with a cursor, users can refer to the video image at a specified time (see Fig.1) as shown by Karon[1]. And if users play the video back, the graph also is updated in synchronization with the video.
Fig.1 Synchronization with time-codes
RETRIEVAL of RELATED DATA
Retrieving by 'Drag & Drop'
Our technique allows users to directly retrieve desired data related to the displayed data. A screen displays various types of data in the supervisory systems, but the data displayed are limited due to the size of screen. Users need to retrieve the desired data according to their interests.
To retrieve the desired data quickly, our technique employs a ÒDrag and DropÓ operation. Our system assumes that the data at an origin of a drag-and-drop operation is the key data to be retrieved, and the destination of the operation indicates the type of the data to be retrieved. With this technique, users can retrieve the desired data related to the key data. The time range for the retrieved data is automatically synchronized with the time of the key data.
Figure 2 shows an example of a screen image. The screen shows 3 types of recorded data; sensor data, video data and alarm data. When a user specifies an alarm text in the alarm frame and drags it to the sensor data frame, the user can obtain the sensor data related to the alarm text in the sensor data frame. Furthermore, the time range of the sensor data frame is adjusted according to the time of the alarm text.
This technique also allows users to specify an object in the video as a key. When a user specifies an object in a video, the technique clips the objectÕs area of the video image. The user can drag the clipped image to the desired area. For example, when users specify an object (Backup Roll) in a video and drag the clipped Backup RollÕs image to the sensor data area , the Backup RollÕs sensor data are shown in the sensor data area. (see Fig.2).
Fig.2 Retrieving with clipped video image
PROTOTYPING
We made a prototype system to test the usability of the technique. This prototype system supports synchronization among multiple data and retrieval of desired data by the ÒDrag and DropÓ technique. Fig.3 shows a screen example of this prototype for a steel work. The upper frame shows sensor data. The lower-left frame shows the structure of a steel mill and a controller for playback of the recorded data. The lower-middle frame shows recorded camera video data. The lower-right frame shows alarm data.
To synchronize the video data and the sensor data, users click video and sensor data with the right mouse button, and Òplay buttonÓ of the controller. The system play back the specified video and sensor data in synchronization. Users can refer to both the sensor data and the video data simultaneously.
To retrieve the related sensor data with ÒWorkRollÓ from video data, users specify the ÒWorkRollÓ object in the video and drag it to the sensor data frame and drop it (see Fig.3). Then the data related to ÒWorkRollÓ appears in the sensor data frame.
We demonstrated this prototype system to people of the steel works. They told us the above techniques allowed them to understand the exact situation of the system.
Fig.3 Prototype system
CONCLUSION
We propose new techniques for the synchronization and retrieval of recorded multimedia data for supervisory control systems, and test usability, and come to the following conclusions.
- The technique for synchronization enables users to see multiple recorded data simultaneously. Users thereby analyze phenomena in the systems exactly.
- The technique for retrieval enable users to see desired data at a desired time. Users can analyze phenomena in the systems quickly.
REFERENCES
- Karon, W. and Alex P. Marquee: A Tool For Real-Time Video Logging. in CHI'94 Conference Proceedings. 24-28, April 1994. Boston, Massachusetts., pp.58-64
- Tani, Masayuki. et al. : Object-Oriented Video: Interaction with Real-World Objects Through Live Video. in CHI'92 Conference Proceedings. 3-7, May 1992. Monterey, California., pp.593-598
CHI 97 Electronic Publications: Late-Breaking/Interactive Posters
