Real-time algorithms are needed to compare and analyze digital videos of machines and processes. New video analysis techniques, for computationally efficient dimensionality-reduction, for determination of accurate motion-information, and for fast video comparison, will enable new approaches to system monitoring and control. We define the video alignment path (VAP) as the sequence of local time-and-space transformations required to optimally register two video clips. We develop an algorithm, dynamic time and space warping (DTSW), which calculates the VAP. Measures of video similarity, and therefore system similarity, are estimated based on properties of the VAP. These measures of similarity are then monitored over time and used for decision-making and process control. We describe the performance, structure, and computational complexity of a DTSW implementation, which is parallelizable and which can achieve the processing rates necessary for many video-based industrial monitoring applications. We describe two case studies of unsupervised monitoring for mechanical wear and for fault detection. Results suggest opportunities for boarder applications of video-based instrumentation for real-time feedback control, wear and defect detection, or statistical process control.

References

1.
Sagiroglu
,
S.
, and
Sinanc
,
D.
,
2013
, “
Big Data: A Review
,”
International Conference on Collaboration Technologies and Systems
(
CTS
), San Diego, CA, May 20–24, pp.
42
47
.
2.
Rao
,
P. K.
,
Liu
,
J. P.
,
Roberson
,
D.
,
Kong
,
Z. J.
, and
Williams
,
C.
,
2015
, “
Online Real-Time Quality Monitoring in Additive Manufacturing Processes Using Heterogeneous Sensors
,”
ASME J. Manuf. Sci. Eng.
,
137
(
6
), p.
061007
.
3.
Huang
,
Y.
,
Leu
,
M. C.
,
Mazumder
,
J.
, and
Donmez
,
A.
,
2015
, “
Additive Manufacturing: Current State, Future Potential, Gaps and Needs, and Recommendations
,”
ASME J. Manuf. Sci. Eng.
,
137
(
1
), p.
014001
.
4.
Lin
,
P. P.
,
Guo
,
Q.
, and
Li
,
X.
,
2006
, “
Online Inspection for Glass Fiber Forming
,”
ASME J. Manuf. Sci. Eng.
,
129
(
1
), pp.
164
171
.
5.
Grasso
,
M.
,
Laguzza
,
V.
,
Semeraro
,
Q.
, and
Colosimo
,
B. M.
,
2016
, “
In-Process Monitoring of Selective Laser Melting: Spatial Detection of Defects Via Image Data Analysis
,”
ASME J. Manuf. Sci. Eng.
,
139
(
5
), p.
051001
.
6.
Dutta
,
S.
,
Pal
,
S. K.
, and
Sen
,
R.
,
2015
, “
Tool Condition Monitoring in Turning by Applying Machine Vision
,”
ASME J. Manuf. Sci. Eng.
,
138
(
5
), p. 051008.
7.
Shaban
,
Y.
,
Yacout
,
S.
, and
Balazinski
,
M.
,
2015
, “
Tool Wear Monitoring and Alarm System Based on Pattern Recognition With Logical Analysis of Data
,”
ASME J. Manuf. Sci. Eng.
,
137
(
4
), p. 41004.
8.
Yang
,
S.
,
Bagheri
,
B.
,
Kao
,
H.-A.
, and
Lee
,
J.
,
2015
, “
A Unified Framework and Platform for Designing of Cloud-Based Machine Health Monitoring and Manufacturing Systems
,”
ASME J. Manuf. Sci. Eng.
,
137
(
4
), p. 40914.
9.
Xu
,
W.
,
Yu
,
J.
,
Zhou
,
Z.
,
Xie
,
Y.
,
Pham
,
D. T.
, and
Ji
,
C.
,
2015
, “
Dynamic Modeling of Manufacturing Equipment Capability Using Condition Information in Cloud Manufacturing
,”
ASME J. Manuf. Sci. Eng.
,
137
(
4
), p. 40907.
10.
Zhang
,
Z.
,
Liu
,
G.
,
Jiang
,
Z.
, and
Chen
,
Y.
,
2015
, “
A Cloud-Based Framework for Lean Maintenance, Repair, and Overhaul of Complex Equipment
,”
ASME J. Manuf. Sci. Eng.
,
137
(
4
), p. 40908.
11.
Ren
,
L.
,
Cui
,
J.
,
Li
,
N.
,
Wu
,
Q.
,
Ma
,
C.
,
Teng
,
D.
, and
Zhang
,
L.
,
2015
, “
Cloud-Based Intelligent User Interface for Cloud Manufacturing: Model, Technology, and Application
,”
ASME J. Manuf. Sci. Eng.
,
137
(
4
), p. 40910.
12.
Tapoglou
,
N.
,
Mehnen
,
J.
,
Vlachou
,
A.
,
Doukas
,
M.
,
Milas
,
N.
, and
Mourtzis
,
D.
,
2015
, “
Cloud-Based Platform for Optimal Machining Parameter Selection Based on Function Blocks and Real-Time Monitoring
,”
ASME J. Manuf. Sci. Eng.
,
137
(
4
), p. 40909.
13.
Buckholtz
,
B.
,
Ragai
,
I.
, and
Wang
,
L.
,
2015
, “
Cloud Manufacturing: Current Trends and Future Implementations
,”
ASME J. Manuf. Sci. Eng.
,
137
(
4
), p.
040902
.
14.
Wang
,
P.
,
Gao
,
R. X.
, and
Fan
,
Z.
,
2015
, “
Cloud Computing for Cloud Manufacturing: Benefits and Limitations
,”
ASME J. Manuf. Sci. Eng.
,
137
(
4
), p.
040901
.
15.
Caspi
,
Y.
, and
Irani
,
M.
,
2002
, “
Aligning Non-Overlapping Sequences
,”
Int. J. Comput. Vision
,
48
(1), pp. 39–51.
16.
Cao
,
X.
,
Wua
,
L.
,
Xiao
,
J.
,
Foroosh
,
H.
,
Zhu
,
J.
, and
Li
,
X.
,
2010
, “
Video Synchronization and Its Application to Object Transfer
,”
J. Image Vision Comput.
,
28
(1), pp. 92–100.
17.
Zelnik-Manor
,
L.
, and
Irani
,
M.
,
2001
, “
Event-Based Analysis of Video
,”
IEEE Conference on Computer Vision and Pattern Recognition
(
CVPR
), Kauai, HI, Dec. 8–14, pp. II-123–II-130.
18.
Padua
,
F.
,
Carceroni
,
R.
,
Santos
,
G.
, and
Kutulakos
,
K.
,
2010
, “
Linear Sequence-to-Sequence Alignment
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
32
(2), pp. 304-320.
19.
Gao
,
Z.
,
Lu
,
G.
, and
Yan
,
P.
,
2016
, “
Averaging Video Sequences to Improve Action Recognition
,”
Ninth International Congress on Image and Signal Processing, BioMedical Engineering and Informatics
(
CISP-BMEI
), Datong, China, Oct. 15–17, pp.
89
93
.
20.
Gritai
,
A.
,
Sheikh
,
Y.
,
Rao
,
C.
, and
Shah
,
M.
,
2009
, “
Matching Trajectories of Anatomical Landmarks Under Viewpoint, Anthropometric, and Temporal Transforms
,”
Int. J. Comput. Vision
,
84
(3), pp. 325–343.
21.
Rao
,
C.
,
Gritai
,
A.
,
Shah
,
M.
, and
Syeda-Mahmood
,
T.
,
2003
, “
View-Invariant Alignment and Matching of Video Sequences
,”
Ninth IEEE International Conference on Computer Vision
(
ICCV
), Nice, France, Oct. 13–16, pp. 939–945.
22.
Freeman
,
I.
,
Wieschollek
,
P.
, and
Lensch
,
H. P. A.
,
2016
, “
Robust Large-Scale Video Synchronization Without Annotations
,” preprint
arXiv:1610.05985
.https://arxiv.org/abs/1610.05985
23.
Karpathy
,
A.
,
Toderici
,
G.
,
Shetty
,
S.
,
Leung
,
T.
,
Sukthankar
,
R.
, and
Fei-Fei
,
L.
,
2014
, “
Large-Scale Video Classification With Convolutional Neural Networks
,”
IEEE Conference on Computer Vision and Pattern Recognition
(
CVRP
), Columbus, OH, June 23–28, pp.
1725
1732
.
24.
Yamato
,
J.
,
Ohya
,
J.
, and
Ishii
,
K.
,
1992
, “
Recognizing Human Action in Time-Sequential Images Using Hidden Markov Model
,”
IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(
CVPR
), Champaign, IL, June 15–18, pp. 379–385.
25.
Fleet
,
D. J.
,
Black
,
M. J.
, and
Jepson
,
A. D.
,
1998
, “
Motion Feature Detection Using Steerable Flow Fields
,”
IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(
CVPR
), Santa Barbara, CA, June 23–25, pp. 274–281.
26.
Psarrou
,
A.
,
Gong
,
S.
, and
Walter
,
M.
,
2002
, “
Recognition of Human Gestures and Behaviour Based on Motion Trajectories
,”
Image Vision Comput.
,
20
(5), pp. 349–358.
27.
BenAbdelkader
,
C.
,
Cutler
,
R.
,
Nanda
,
H.
, and
Davis
,
L. S.
,
2001
, “
Eigengait: Motion-Based Recognition of People Using Image Self-Similarity
,”
International Conference on Audio and Video-Based Person Authentication
(
AVBPA
), Halmstad, Sweden, June 6–8, pp. 284–294.
28.
Laptev
,
I.
, and
Lindeberg
,
T.
,
2004
, “
Velocity Adaptation of Space-Time Interest Points
,”
17th International Conference on Pattern Recognition
(
ICPR
), Cambridge, UK, Aug. 23–26, pp. 52–56.
29.
Zhao
,
T.
, and
Nevatia
,
R.
,
2004
, “
Tracking Multiple Humans in Complex Situations
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
26
(9), pp. 1208–1221.
30.
Wang
,
F.
,
Sun
,
Z.
,
Jiang
,
Y. G.
, and
Ngo
,
C. W.
,
2014
, “
Video Event Detection Using Motion Relativity and Feature Selection
,”
IEEE Trans. Multimedia
,
16
(
5
), pp.
1303
1315
.
31.
Cutler
,
R.
, and
Davis
,
L. S.
,
2000
, “
Robust Real-Time Periodic Motion Detection, Analysis, and Applications
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
22
(8), pp. 781–796.
32.
McKenna
,
S.
, and
Gong
,
S.
,
1996
, “
Tracking Faces
,”
Second International Conference on Automatic Face and Gesture Recognition
(
AFGR
), Killington, VT, Oct. 13–16, pp. 271–276.
33.
Blank
,
M.
,
Gorelick
,
L.
,
Shechtman
,
E.
,
Irani
,
M.
, and
Basri
,
R.
,
2005
, “
Actions as Space-Time Shapes
,”
Tenth IEEE International Conference on Computer Vision
(
ICCV
), Beijing, China, Oct. 17–21, pp. 1395–1402.
34.
Ke
,
Y.
,
Sukthankar
,
R.
, and
Hebert
,
M.
,
2005
, “
Efficient Visual Event Detection Using Volumetric Features
,”
Tenth IEEE International Conference on Computer Vision
(
ICCV
), Beijing, China, Oct. 17–21, pp. 166–173.
35.
Ngo
,
C.-W.
,
Pong
,
T.-C.
, and
Zhang
,
H.-J.
,
2003
, “
Motion Analysis and Segmentation Through Spatio-Temporal Slices Processing
,”
IEEE Trans. Image Process.
,
12
(3), pp. 341–355.
36.
Allmen
,
M. C.
,
1991
, “
Image Sequence Description Using Spatiotemporal Flow Curves: Toward Motion-Based Recognition
,”
Ph.D. thesis
, University of Wisconsin, Madison, WI.http://dl.acm.org/citation.cfm?id=143909
37.
Polana
,
R.
, and
Nelson
,
R.
,
1997
, “
Detection and Recognition of Periodic, Nonrigid Motion
,”
Int. J. Comput. Vision
,
23
(3), pp. 261–282.
38.
Ma
,
Y.-F.
, and
Zhang
,
H.-J.
,
2002
, “
Motion Texture: A New Motion Based Video Representation
,”
16th International Conference on Pattern Recognition
(
ICPR
), Quebec City, QC, Canada, Aug. 11–15, pp. 548–551.
39.
Ma
,
Y.-F.
, and
Zhang
,
H.-J.
,
2003
, “
Motion Pattern Based Video Classification and Retrieval
,”
EURASIP J. Appl. Signal Process.
,
2003
(2), pp. 199–208.https://asp-eurasipjournals.springeropen.com/articles/10.1155/S1110865703211021
40.
Lee
,
S.-L.
,
Chun
,
S.-J.
,
Kim
,
D.-H.
,
Lee
,
J.-H.
, and
Chung
,
C.-W.
,
2000
, “
Similarity Search for Multidimensional Data Sequences
,”
IEEE 16th International Conference on Data Engineering
(
ICDE
), San Diego, CA, Feb. 28–Mar. 3, pp. 599–608.
41.
Vlachos
,
M.
,
Kollios
,
G.
, and
Gunopulos
,
G.
,
2002
, “
Discovering Similar Multidimensional Trajectories
,”
18th International Conference on Data Engineering
(
ICDE
), San Jose, CA, Feb. 26–Mar. 1, pp. 673–684.
42.
Bashir
,
F. I.
,
Khokhar
,
A.
, and
Schonfeld
,
D.
,
2004
, “
Real-Time Motion Trajectory-Based Indexing and Retrieval of Video Sequences
,”
IEEE Trans. Multimedia
,
9
(1), pp. 58–65.
43.
Porikli
,
F.
, and
Haga
,
T.
,
2004
, “
Event Detection by Eigenvector Decomposition Using Object and Frame Features
,”
Conference on Computer Vision and Pattern Recognition Workshop
(
CVPR
), Washington, DC, June 27–July 2, p. 114.
44.
Fischler
,
M. A.
, and
Elschlager
,
R. A.
,
1973
, “
The Representation and Matching of Pictorial Structures
,”
IEEE Trans. Comput.
,
C-22
(1), pp. 67–92.
45.
Felzenszwalb
,
P. F.
, and
Huttenlocher
,
D. P.
,
2005
, “
Pictorial Structures for Object Recognition
,”
Int. J. Comput. Vision
,
61
(1), pp. 55–79.
46.
Black
,
M. J.
,
1999
, “
Explaining Optical Flow Events With Parameterized Spatio-Temporal Models
,”
IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(
CVPR
), Fort Collins, CO, June 23–25, p. 332.
47.
Isard
,
M.
, and
Blake
,
A.
,
1998
, “
Condensation–Conditional Density Propagation for Visual Tracking
,”
Int. J. Comput. Vision
,
29
(1), pp. 5–28.
48.
Black
,
M. J.
, and
Jepson
,
A. D.
,
1998
, “
Recognizing Temporal Trajectories Using the Condensation Algorithm
,”
Third IEEE International Conference on Automatic Face and Gesture Recognition
(
AFGR
), Nara, Japan, Apr. 14–16, pp. 16–21.
49.
Sand
,
P.
, and
Teller
,
S.
,
2004
, “
Video Matching
,”
ACM Trans. Graphics
,
23
(3), pp. 592–599.
50.
Diego
,
F.
,
Serrat
,
J.
, and
Lopez
,
A. M.
,
2013
, “
Joint Spatio-Temporal Alignment of Sequences
,”
IEEE Trans. Multimedia
,
15
(
6
), pp.
1377
1387
.
51.
Evangelidis
,
G. D.
, and
Bauckhage
,
C.
,
2013
, “
Efficient Subframe Video Alignment Using Short Descriptors
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
35
(
10
), pp.
2371
2386
.
52.
Ukrainitz
,
Y.
, and
Irani
,
M.
,
2006
, “
Aligning Sequences and Actions by Maximizing Space-Time Correlations
,”
European Conference on Computer Vision
(
ECCV
), Graz, Austria, May 7–13, pp. 538–550.
53.
Wang
,
L.
,
Bao
,
Y.
,
Li
,
H.
,
Fan
,
X.
, and
Luo
,
Z.
,
2017
,
Compact CNN Based Video Representation for Efficient Video Copy Detection
,
Springer International Publishing
,
Cham, Switzerland
, pp.
576
587
.
54.
Wu
,
Z.
, and
Aizawa
,
K.
,
2014
, “
Self-Similarity-Based Partial Near-Duplicate Video Retrieval and Alignment
,”
Int. J. Multimedia Inf. Retr.
,
3
(
1
), pp.
1
14
.
55.
Kordopatis-Zilos
,
G.
,
Papadopoulos
,
S.
,
Patras
,
I.
, and
Kompatsiaris
,
Y.
,
2017
,
Near-Duplicate Video Retrieval by Aggregating Intermediate CNN Layers
,
Springer International Publishing
,
Cham, Switzerland
, pp.
251
263
.
56.
Liu
,
J.
,
Huang
,
Z.
,
Cai
,
H.
,
Shen
,
H. T.
,
Ngo
,
C. W.
, and
Wang
,
W.
,
2013
, “
Near-Duplicate Video Retrieval: Current Research and Future Trends
,”
ACM Comput. Surv.
,
45
(
4
), p. 44.
57.
Chou
,
C. L.
,
Chen
,
H. T.
, and
Lee
,
S. Y.
,
2015
, “
Pattern-Based Near-Duplicate Video Retrieval and Localization on Web-Scale Videos
,”
IEEE Trans. Multimedia
,
17
(
3
), pp.
382
395
.
58.
Song
,
J.
,
Yang
,
Y.
,
Huang
,
Z.
,
Shen
,
H. T.
, and
Luo
,
J.
,
2013
, “
Effective Multiple Feature Hashing for Large-Scale Near-Duplicate Video Retrieval
,”
IEEE Trans. Multimedia
,
15
(
8
), pp.
1997
2008
.
59.
Alon
,
J.
,
Athitsos
,
V.
,
Yuan
,
Q.
, and
Sclaroff
,
S.
,
2005
, “
Simultaneous Localization and Recognition of Dynamic Hand Gestures
,”
Seventh IEEE Workshops on Application of Computer Vision WACV/MOTIONS
(
ACVMOT
), Breckenridge, CO, Jan. 5–7, pp. 254–260.
60.
Lu
,
C.
, and
Mandal
,
M.
,
2013
, “
A Robust Technique for Motion-Based Video Sequences Temporal Alignment
,”
IEEE Trans. Multimedia
,
15
(
1
), pp.
70
82
.
61.
Hohne
,
H. D.
,
Coker
,
C.
,
Levinson
,
S. E.
, and
Rabiner
,
L. R.
,
1983
, “
On Temporal Alignment of Sentences of Natural and Synthetic Speech
,”
IEEE Trans. Acoust., Speech, Signal Process.
,
31
(4), pp. 807–813.
62.
Myers
,
C.
,
Rabiner
,
L. R.
, and
Rosenberg
,
A. E.
,
1980
, “
Performance Tradeoffs in Dynamic Time Warping Algorithms for Isolated Word Recognition
,”
IEEE Trans. Acoust., Speech, Signal Process.
,
28
(6), pp. 623–635.
63.
Ratanamahatana
,
C. A.
, and
Keogh
,
E.
,
2004
, “
Everything You Know About Dynamic Time Warping is Wrong
,”
Third Workshop on Mining Temporal and Sequential Data, in Conjunction With the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
(
KDD
), Seattle, WA, Aug. 22–24, pp. 22–25.https://www.semanticscholar.org/paper/Everything-you-know-about-Dynamic-Time-Warping-is-Ratanamahatana-Keogh/42157b64537a5ce4415177ab6ccb69480c2120c6
64.
Keogh
,
E.
,
Palpanas
,
T.
,
Zordan
,
V.
, and
Gunopulos
,
D.
,
2004
, “
Indexing Large Human-Motion Databases
,”
International Conference on Very Large Databases
(
VLDB
), Toronto, ON, Canada, Aug. 31–Sept. 3, pp. 780–791.http://dl.acm.org/citation.cfm?id=1316689.1316757
65.
Anthony
,
B. W.
,
2006
, “
Video Based System Monitoring
,”
Ph.D. thesis
, Massachusetts Institute of Technology, Cambridge, MA.https://dspace.mit.edu/handle/1721.1/38263
66.
Chu
,
S.
,
Keogh
,
E.
,
Hart
,
D.
, and
Pazzani
,
M.
,
2002
, “
Iterative Deepening Dynamic Time Warping
,”
SIAM International Conference on Data Mining
, Arlington, VA, Apr. 11–13.
67.
Salvador
,
S.
, and
Chan
,
P.
,
2004
, “
Toward Accurate Dynamic Time Warping in Linear Time and Space
,”
KDD Workshop on Mining Temporal and Sequential Data
, Seattle, WA, Aug. 22.https://pdfs.semanticscholar.org/33d1/a0f2bfdbce75eb2949c5bae7a2ef78994353.pdf
68.
Kassidas
,
A.
,
MacGregor
,
J. F.
, and
Taylor
,
P. A.
,
1998
, “
Synchronization of Batch Trajectories Using Dynamic Time Warping
,”
AIChE J.
,
44
(4), pp. 864–875.
69.
Keogh
,
E.
, and
Pazzani
,
M.
,
2001
, “
Dynamic Time Warping With Higher Order Features. Derivative Dynamic Time Warping
,”
First SIAM International Conference on Data Mining
(
SDM
), Chicago, IL, Apr. 5–7.
70.
Gil
,
J.
, and
Werman
,
M.
,
1993
, “
Computing 2-D Min, Median, and Max Filters
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
15
(5), pp. 504–507.
71.
Gevorkian
,
D. Z.
,
Astola
,
J. T.
, and
Atourian
,
S. M.
,
1997
, “
Improving Gil-Werman Algorithm for Running Min and Max Filters
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
19
(5), pp. 526–529.
72.
Benhimane
,
S.
,
Ladikos
,
A.
,
Lepetit
,
V.
, and
Navab
,
N.
,
2007
, “
Linear and Quadratic Subsets for Template-Based Tracking
,”
IEEE Conference on Computer Vision and Pattern Recognition
(
CVPR
), Minneapolis, MN, June 17–22, pp. 1–6.
73.
la Torre
,
F. D.
, and
Black
,
M. J.
,
2001
, “
Robust Principal Component Analysis for Computer Vision
,”
Eighth International Conference on Computer Vision
(
ICCV
), Vancouver, BC, Canada, July 7–14, p. 362.http://doi.ieeecomputersociety.org/10.1109/ICCV.2001.10084
74.
Sakoe
,
H.
, and
Chiba
,
S.
,
1978
, “
Dynamic Programming Algorithm Optimization for Spoken Word Recognition
,”
IEEE Trans. Acoust., Speech, Signal Process.
,
26
(1), pp. 43–49.
75.
Itakura
,
F.
,
1975
, “
Minimum Prediction Residual Principle Applied to Speech Recognition
,”
IEEE Trans. Acoust., Speech, Signal Process.
,
23
(1), pp. 67–72.
76.
Fu
,
A. W.-C.
,
Keogh
,
E.
,
Lau
,
L. Y. H.
, and
Ratanamahatana
,
C. A.
,
2005
, “
Scaling and Time Warping in Time Series Querying
,”
VLDB
,
17(
4), pp. 899–921.
You do not currently have access to this content.