Publication and Technical Report Details
NAPA: Neural Art Human Pose Amplifier
This is the project report for CSCI-GA.2271-001. We target human pose estimation in artistic images. For this goal, we design an end-to-end system that uses neural style transfer for pose regression. We collect a 277-style set for arbitrary style transfer and build an artistic 281-image test set. We directly run pose regression on the test set and show promising results. For pose regression, we propose a 2d-induced bone map from which pose is lifted. To help such a lifting, we additionally annotate the pseudo 3d labels of the full in-the-wild MPII dataset. Further, we append another style transfer as self supervision to improve 2d. We perform extensive ablation studies to analyze the introduced features. We also compare end-to-end with per-style training and allude to the tradeoff between style transfer and pose regression. Lastly, we generalize our model to the real-world human dataset and show its potentiality as a generic pose model. We explain the theoretical foundation in Appendix. We release code at https://github.com/ strawberryfg/NAPA-NST-HPE, data, and video.
This is the project report for CSCI-GA.2271-001. We target human pose estimation in artistic images. For this goal, we design an end-to-end system that uses neural style transfer for pose regression. We collect a 277-style set for arbitrary style transfer and build an artistic 281-image test set. We directly run pose regression on the test set and show promising results. For pose regression, we propose a 2d-induced bone map from which pose is lifted. To help such a lifting, we additionally annotate the pseudo 3d labels of the full in-the-wild MPII dataset. Further, we append another style transfer as self supervision to improve 2d. We perform extensive ablation studies to analyze the introduced features. We also compare end-to-end with per-style training and allude to the tradeoff between style transfer and pose regression. Lastly, we generalize our model to the real-world human dataset and show its potentiality as a generic pose model. We explain the theoretical foundation in Appendix. We release code at https://github.com/ strawberryfg/NAPA-NST-HPE, data, and video.
Patch-based 3D Human Pose Refinement
State-of-the-art 3D human pose estimation approaches typically estimate pose from the entire RGB image in a single forward run. In this paper, we develop a post-processing step to refine 3D human pose estimation from body part patches. Using local patches as input has two advantages. First, the fine details around body parts are zoomed in to high resolution for preciser 3D pose prediction. Second, it enables the part appearance to be shared between poses to benefit rare poses. In order to acquire informative representation of patches, we explore different input modalities and validate the superiority of fusing predicted segmentation with RGB. We show that our method consistently boosts the accuracy of state-of-the-art 3D human pose methods.
State-of-the-art 3D human pose estimation approaches typically estimate pose from the entire RGB image in a single forward run. In this paper, we develop a post-processing step to refine 3D human pose estimation from body part patches. Using local patches as input has two advantages. First, the fine details around body parts are zoomed in to high resolution for preciser 3D pose prediction. Second, it enables the part appearance to be shared between poses to benefit rare poses. In order to acquire informative representation of patches, we explore different input modalities and validate the superiority of fusing predicted segmentation with RGB. We show that our method consistently boosts the accuracy of state-of-the-art 3D human pose methods.
DeepSkeleton: Skeleton Map for 3D Human Pose Regression
We present a novel intermediate feature representation named skeleton map for regression. It distills structural context from irrelavant properties of RGB image e.g. illumination and texture. It is simple, clean and can be easily generated via deconvolution network. For the first time, we show that training regression network from skeleton map alone is capable of meeting the performance of state-of-the-art 3D human pose estimation works. We further exploit the power of multiple 3D hypothesis generation to obtain reasonbale 3D pose in consistent with 2D pose detection. The effectiveness of our approach is validated on challenging in-the-wild dataset MPII and indoor dataset Human3.6M.
Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals
We strive to answer two questions: What is the current state of 3D hand pose estimation? And, what are the next challenges that need to be tackled? Following the successful Hands In the Million Challenge (HIM2017), we investigate 11 state-of-the-art methods on three tasks: single frame 3D pose estimation, 3D hand tracking, and hand pose estimation during object interaction. We analyze the performance of different CNN structures with regard to hand shape, joint visibility, view point and articulation distributions.
Model-based Deep Hand Pose Estimation
We propose a model based deep learning approach that adopts a forward kinematics based layer to ensure the geometric validity of estimated poses. For the first time, we show that embedding such a non-linear generative process in deep learning is feasible for hand pose estimation. Our approach is verified on challenging public datasets NYU and ICVL and achieves state-of-the-art performance.
--------------------------------------------------This is the separation line----------------------------------------------
Figure 2. The wall close to Norman Y. Mineta San Jose International Airport.
Figure 3. Human skeleton exhibition in UC Berkeley Art Museum and Pacific Film Archive (BAMPFA).
Videos
The Bat Bread I made
Music by "The Dark Knight Rises - Rise - Hans Zimmer"
Produced by Adobe Premiere Pro
Produced by Adobe Premiere Pro
Summertime Magic Bread Journey
Music by Childish Gambino - Summertime Magic (Instrumental) and Ray LaMontagne - Part Two - In My Own Way.
Credit to Beatzdaily for the remake.
Produced by Adobe Premiere Pro
Credit to Beatzdaily for the remake.
Produced by Adobe Premiere Pro
Fun facts
1. Did you know that after 1880, more and more of Monet’s paintings eliminate the human figure; later they even cut out such solid reminders of human existence as bridges, houses and boats. One wonders what part the early death of his wife Camille, his favorite model, played in this renunciation.
2. Gaugin complained to Pissarro in 1881 about the repetitive nature of the landscape Impressionists' subjects: "The way Durand-Ruel
is operating with Sisley and Monet who are churning out pictures at top speed, he will soon have four hundred paints which he will not be able to get rid of. Hmm.
3.
2. Gaugin complained to Pissarro in 1881 about the repetitive nature of the landscape Impressionists' subjects: "The way Durand-Ruel
is operating with Sisley and Monet who are churning out pictures at top speed, he will soon have four hundred paints which he will not be able to get rid of. Hmm.
3.
Mood Booster
The Cinematic Orchestra - A Caged Bird/Imitations of Life
Señorita
If I Killed Someone For You
LES INDES GALANTES
Señorita
If I Killed Someone For You
LES INDES GALANTES
Places in Shanghai
Pier 39 (Jing'an Kerry Centre Shopping Mall)
SHAKE SHACK (several locations)
Peet's (personally recommend the IAPM one)
Fascino (W Nanjing Rd)
81BAKERY (Daxue Rd, near Fudan Univ.)
%Arabica
T Plus
茶巭道
SHAKE SHACK (several locations)
Peet's (personally recommend the IAPM one)
Fascino (W Nanjing Rd)
81BAKERY (Daxue Rd, near Fudan Univ.)
%Arabica
T Plus
茶巭道
Places in New York
Riverbank State Park (Ice Skating)
Candlelight
Bobst
Strand
Candlelight
Bobst
Strand