DEVELOPMENT OF ADAPTIVE VIDEO ANALYTIC FRAMEWORK FOR REAL-TIME HUMAN IDENTITY AND ACTIVITY RECOGNITION

Abstract

Emerging smart visual surveillance systems are requiring adaptive video analytic frameworks for automatic detection and tracking of moving objects within a dynamic environment. This is required in order to study the behaviors of such objects and also recognize when an action has occurred and register such action for later forensic purposes. This class of systems is also requiring the identification and recognition of objects within the scene and the prediction of the actions being performed by the detected objects. There are challenging issues especially in real time environment. This thesis presents a computational framework for the automatic detection and recognition of human beings and prediction of some kinds of human actions from video cameras via smart visual systems that automatically sense and correctly recognize human identities and actions by means of machine vision and machine learning techniques. Statistical machine learning algorithms necessary for detecting and recognizing individuals and their actions are developed. Initial human detection is performed by background detection techniques using parallel–processed Kernel Density Estimation (KDE). Temporal tracking of the objects’ trajectories is performed by employing a spatial body tracking system designed as a multi-part colour histogram- based tracking scheme. In classification, five object classification algorithms are developed using machine learning techniques to classify moving objects into vehicle or human. In identity recognition, Principal Component Analysis (PCA) is employed for face and gait recognition. In addition to PCA for gait recognition, Hidden Markov Model is also developed for gait recognition. In case of action segmentation and recognition, Hidden Markov Models (HMMs) are used to model individual key action and a threshold model. The threshold HMM model serves as the adaptive cut-off point for each action class. The algorithms for learning the parameters of these models from available datasets as well as for performing inferences with those models are developed and described. The implementation of the framework is done using C++ with OpenCV and LIBSvm. A website is developed using hypertext markup language (HTML) for easy access and querying of the activities occurring within the system. The experimental studies of the algorithms are carried out. These include automatic human detection and tracking, face detection, human gait recognition, human action recognition, event logging and querying and raw video data transmission. The performance analysis of the algorithms in terms of execution times and recognition rates are carried out. An experiment on raw video transmission has been carried out between the Computer Science Post-Graduate laboratory and Computer Resource Centre (CRC) within Federal University of Technology, Akure (FUTA) and the system performs excellently although with little transmission delay. The proposed object detection algorithm has been tested in both indoor and outdoor environments. The results obtained when compared to frame difference show a high detection rate and has a low average detection time of 50ms which is good for real time surveillance system. The object tracking algorithm is able to track objects successfully within the scene at an average of 0.85 microseconds (ms) per frame. The case of tracking multiple objects that are more than 3 in number still poses a challenge. A particle filter algorithm used as comparison shows an average tracking time of 3000 ms. The proposed genetically trained neural network model for object classification shows classification performance of 99.152% which is the same with the state of art Support Vector Machine. Other object classification algorithms used are K-Means with classification accuracy of 98.79%, Artificial Neural Network with accuracy of 98.55% and K-Nearest Neighbor which has the lowest accuracy of 95.195%. The committee of these classifiers shows a recognition accuracy of 99.70%. The proposed face and gait human identity recognition algorithms achieve overall recognition rate of 60 % using ‘AND’ fusion and 100% using ‘OR’ fusion. This shows that gait and face can be combined to recognize a person in videos. Depending on the application scenario, the result of the human identity recognition can be used in cascade mode or in parallel mode. For instance, in an access control facility, the cascade mode can be used where a person is given access to a facility after face recognition is performed followed by gait recognition. In a suspect detection system however, any of the modes can be used to detect the suspect. Finally, the result of the proposed human activity recognition shows that the proposed threshold model is capable of recognizing actions in videos. A recognition accuracy of 93.16% is achieved using six actions being performed by eight people. The adaptability of the framework is achieved by using adaptive thresholds both in the foreground detection algorithms and activity recognition algorithms. The adaptability of the recognition schemes made them recognized objects and actions irrespective of their spatial and temporal variances. The results show that the proposed system performs excellently on the test datasets. Thus, this framework is good for adaptive recognition of human identities and activities in real-time videos.