Human actions can be represented by the trajectories of skeleton joints. Traditional methods generally model the spatial structure and temporal dynamics of human skeleton with hand-crafted features and recognize human actions by well- designed classifiers. For this project, considering that recurrent neural network (RNN) can model the long-term contextual information of temporal sequences well, we aim to explore an end-to-end hierarchical RNN for skeleton based action recognition. A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle. This creates an internal state of the network which allows it to exhibit dynamic temporal behavior. Unlike feedforward neural networks, RNNs can use their internal memory to process arbitrary sequences of inputs. This makes them applicable to tasks such as unsegmented connected dynamic action recognition such as handwriting recognition, where they have achieved the best known results.
In this project, the main challenge is to explore how to design a suitable RNN framework for human joints data. (That is, how to represent the body by parts, how many layers of NN to use, what kind of layers(BRNN or RNN) to use and how to combine them and so on) We will evaluate our model on three benchmark datasets: MSR Action3D Dataset, Berkeley Multimodal Human Action Dataset (Berkeley MHAD), and Motion Capture Dataset HDM05.
- Literature on RNN and RNN for human action recognition (http://www.cvfoundation.org/openaccess/content_cvpr_2015/papers/Du_Hierarchical_Recurrent_Neural_2015_CVPR_paper.pdf http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf,)
- Explore and implement suitable structure of RNN for skeleton based action recognition
- Evaluate your algorithm on benchmark datasets
- Bonus: Adapt your algorithm into GPU implementation for real-time testing
- Good C or python skills,
- Good knowledge in machine learning and computer vision
- Highly motivated and independent
Internal supervisor(s): Jie Song, email@example.com, Otmar Hilliges, firstname.lastname@example.org
CLS Student Project (MPG ETH CLS)
Information, Computing and Communication Sciences