Project Title: Mining, Indexing, and Retrieval of Spatio-Temporal
Patterns in Video Databases of Human Motion
National Science Foundation Award Number: IIS-0308213 (09/15/03 --
09/14/06)
Stan Sclaroff, PI
George Kollios and Margrit Betke, Co-PI's
Department of Computer Science
Boston University
111 Cummington St., Boston, MA 02215, U.S.A.
Phone: (617) 353-8919, Fax: (617) 353-6457
E-mail: {sclaroff,gkollios,betke}@cs.bu.edu
URL: http://www.cs.bu.edu/fac/sclaroff
URL: http://www.cs.bu.edu/fac/gkollios
URL: http://www.cs.bu.edu/fac/betke
Keywords:
spatio-temporal data mining, motion time series data, video database
methods
Project Summary:
The aim of this research project is to develop and test methods for
indexing, retrieval, and data mining of human motion trajectories in
video databases. Computer vision techniques are being devised for
automatic extraction of human motion time series data from video. Data
mining algorithms are being developed that can be used to discover
clusters and other patterns in the extracted motion time-series data.
One promising direction being explored is to model the observed motion
time series sequences with a finite mixture of Hidden Markov Models
(HMMs). Use of the HMM representation presents certain advantages with
regard to modeling; however, it presents important challenges for the
design of efficient clustering, indexing, and retrieval algorithms. Thus
more efficient, sampling-based and embedding-based methods must be
formulated. The resulting ideas are evaluated in a prototype video
retrieval system, with real-world video datasets that depict human body
motion. Synthetic sequences (e.g., generated via computer graphics) are
used in quantitative performance experiments where ground truth
information is required.
Publications and Products:
There is nothing to report yet, since this is a new award.
Project Impact:
Data mining and indexing methods for video databases of human motion
would enable numerous applications that are valuable to society, for
instance:
- Video indexing, mining, and
retrieval of archival footage. A large percentage of video
transmission depicts human activities: sports, news, entertainment,
education, etc.
- Assistive environments. Video
data mining of human motion could be useful in homecare for the
handicapped, infirm, elderly; e.g., a person's day to day motion
patterns could be analyzed for illness or decline and alert caregivers.
Such an approach could also prove useful in occupational safety.
- Homeland Security.
Particularly useful would be the ability to cluster and retrieve common
motion patterns, and to employ the models gained in data mining for
outlier detection thereby alerting database analysts to anomolous
activities discovered in the video data streams.
- Video-based analysis of human
motion. Data mining methods could enable automated studies of
human movement. Such motion analysis should prove to be a very useful
tool in occupational safety and therapy, as well as in training for
sports and dance.
Goals, Objectives, and Targeted Activities:
This effort will focus on indexing and data mining methods for video
databases of human motion. Such data has a spatio-temporal askect that
must be dealt with, and therefore a major focus of the project will be
on developing efficient methods for indexing, clustering and mining
databases of motion time-series data. Another important part of
the effort will focus on methods that can automatically extract and
analyze motion time-series from video databases.
Area Background:
Up until very recently, data mining of video databases of human motion
received only a small amount of attention [1], while there has been a
great deal of related work in video-based surveillance and activity
monitoring (too numerous to mention here) . Recently there have
been two workshops organized to explore potential directions for
research on this topic, for instance see [2,3]. On the other
hand, a variety of methods for clustering time-series data have been
proposed recently, particularly in the statistics and data mining
communities. For example, the Fourier transform has been
successfully applied in clustering certain types of time-series data
[4], splines have been used to cluster functional data [5]. In
other work, Gaffney, et al. [6] proposed trajectory clustering with a
mixture of regression models, where the regression models can be either
parametric (e.g., linear, quadratic) or non-parametric (e.g., Gaussian
Kernels). Cadez, et al, [7] gave a general EM-based probabilistic
framework for clustering non-vector data. A number of HMM-based
clustering methods have also been proposed ([8,9.10] to name a few).
Area References
[1] G. Kollios, S. Sclaroff, and M. Betke. Motion mining:
Discovering spatio-temporal patterns in databases of human motion. In Proc. ACM-SIGMOD Int. Workshop on Data
Mining and Knowledge Discovery (DMKD), May 2001.
[2] A. Rosenfeld, D. DeMenthon, and D. Doermann, eds. Proc. DIMACS Workshop on Video Mining,
2002.
[3] I. Haritaoglu, and T. Syeda-Mahmood, eds., Proc. IEEE Workshop on Event Mining:
Detection and Recognition of Events in Video, 2003.
[4] R. Agrawal, C. Faloutsos, and A. Swami. Efficient similarity
search in sequence databases, in Proc.
Int. Conf. on Foundations fo Data Organization and Algorithms,
pp. 69-84, 1993.
[5] G. James and C. Sugar, Clustering
for sparsely sampled functional data, Technical Report, Dept. of
Information and Operations Management, USC, 2002.
[6] S. Gaffney and P. Smyth. Trajectory clustering with mixtures of
regression models. In Knowledge
Discovery and Data Mining, pp. 63-72, ACM Press, 1999.
[7] I.V. Cadez, S. Gaffney, and P. Smyth. A general probablistic
framework for clustering individuals and objects. In Knowledge Discovery and Data Mining, pp.
140-149, ACM Press, 2000.
[8] P. Smyth. Clustering sequences with hidden Markov models. In Advances in Neural Information Processing
Systems (NIPS), vol. 9, pp. 648-654, MIT Press, 1997.
[9] C. Li and G. Biswas. A Bayesian approach to temporal data
clustering using hidden Markov models. In Proc. International Conf. on Machine
Learning, pp. 599-608, 2000.
[10] T. Oates, L. Firoiu, and P.R. Cohen. Using dynamic time warping to
bootstrap HMM-based clustering of time series. In Sequence Learning: Paradigms, Algorithms, and
Applications. Springer, 2000.
Potential Related Projects:
These will be posted here later.
Project Web site URL:
http://www.cs.bu.edu/groups/ivc/MotionMining/
Online software:
There is nothing to report yet, since this is a new award.
Online resources:
There is nothing to report yet, since this is a new award.