Hao Jiang


		Hao Jiang Research Scientist Facebook

I am a Research Scientist at Facebook Reality Labs. I received PhD in computer science from Simon Fraser University in 2006. I spent a year from 2006 to 2007 as a Postdoctoral Research Fellow at the University of British Columbia. From 2007 to 2017, I was an Assistant Professor and then a Tenured Associate Professor in the Computer Science Department at Boston College. My research spans human pose, tracking, action understanding, 3D computer vision, egocentric vision and deep learning. I developed the people tracking and object association algorithm for Amazon Go, when I was a Senior Research Scientist at Amazon in 2013-2014. In 2016, I led the Computer Vision Team at AutoX.ai at Silicon Valley working on pure vision based next generation self-driving cars. From 2017 to 2020, I was a Principal Researcher at Microsoft Cloud and AI working on real-time 4D computer vision.

Research Interests

My research interests are in computer vision with the focus on image matching, object detection, tracking, pose and action recognition, self-driving technology and deep learning. Please see my list of publications for more details.

Research Projects

4D Vision. The news coverage about our project. We demoed the real-time 4D technology in CVPR 2018 and CVPR 2019.
RGBD
- Region Matching (CVPR'15)
- Finding Convex Shapes (ECCV'14)
- Finding Cuboids (CVPR'13)
Object Matching
Object Segmentation
- Topology Constrained Scale Invariant Global Figure Ground Separation (CVPR'12).
Motion Estimation and Object Tracking
- Long range motion estimation using convex programming and detail preserving PDE (CVPR'04).
- Appearance adaptive object tracking (EMMCVPR'05).
- Optimizing multiple object tracking (CVPR'07, TMM 2007).
- Shadow resistant object tracking (ICME'03, Pattern Recognition 2007).
Human Pose Estimation
- Finding People Using Scale, Rotation and Articulation Invariant Matching (ECCV'12).
- Consistent max-covering for human pose estimation (ICCV'09, TPAMI 2011).
- Concurrently finding human poses and foreground segmentation in videos (ACCV'10).
- 3D human pose estimation using millions of exemplars (ICPR'10).
- Pose estimation using non-tree structures (CVPR'08).
Action Detection
- Finding actions using shape flows (ECCV'08).
- Clustering actions in images (CVPR'06).
- Action detection using convex shape matching (CVPR'06, AMFG'05, ICME'05, ICME'06, IEEE Multimedia 2007, TCSVT 2010).
Image and Video Processing
- Videorama (ICPR'10).
- PCA based camera color auto-adjustment (ICIP'03, CIC'03).
- Detail preserving image interpolation (ICIP'02).
- Edge adaptive error concealment (ICIP'02).
Multimedia
- TV program classification and segmentation based on audio and video ( Patent, ICME'00, ACM-MM'01, TSAP 2002).

Award and Funding

NSF Award "RI: Small: Convex Architecture for Human Movement Understanding", 2010-2015.
Governor General's Gold Medal, 2007.

Code

Convex Matching: a fast global matching method (Should work on most Linux boxes. Tesed on Fedora and recently on Linux Mint 17.1 Cinnamon 64-bit.)
Scale and Rotation Invariant Linear Matching (Updated on Dec 11, 2012. Now runs on Fedora 17 64bit. Also tested on Linux Mint 17.1 Cinnamon 64-bit.)
3D Pose Estimation Using Millions of Exemplars (Updated on Dec 11, 2012. Now runs on Fedora 17 64bit)
Pose Estimation with Maxcovering
Linear Multiple Object Tracker
Scale and Rotation Invariant Matching Using Linearly Augmented Trees
Linear method for Cuboid Detection in RGBD images: The code has been updated in Jan 2014 to improve the efficiency. See README in the zip files for how to run the demo. The code was tested on Fedora, Ubuntu and MAC OSX (Mountain Lion) 64-bit machines. The clusters in kmeans (n_cluster_new.m) is set to be M=6. If you want more boxes detected, change it to M=10. We received a lot of questions about what the rgbd.mat is. It is generated from NYU kinect V2 dataset. If you want to download all the mat files, download the data here; the code in the directory is for the ECCV2014 paper.
The code for Approximate Convex Shapes in RGBD images (ECCV 2014) has been released. It can be used to segment a scene into near-object-level regions. Here is the code and a bigger zip file contains both code and test data.

Journal Publications

	Scale and Rotation Invariant Matching Using Linearly Augmented Trees Hao Jiang, Tai-Peng Tian and Stan Sclaroff IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 12, December 2015. [PDF]
	Human Pose Estimation Using Consistent Max-Covering Hao Jiang IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 9, 2011. [PDF] [Videos & Dataset].
	Linear Scale and Rotation Invariant Matching Hao Jiang, Stella X. Yu and David R. Martin IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no.7, 2011 [PDF] [Demo Code, Videos and Data].
	Action Detection in Cluttered Video with Successive Convex Matching Hao Jiang, Mark S. Drew and Ze-Nian Li IEEE Transactions on Circuits and Systems for Video Technology. 2010. [PDF]
	Optimizing Multiple Object Tracking and Best View Video Synthesis Hao Jiang, Sidney Fels and James J. Little IEEE Transactions on Multimedia, vol.10, no.6, October 2008. [PDF] [Videos]
	Matching by Linear Programming and Successive Convexification Hao Jiang, Mark S. Drew and Ze-Nian Li IEEE Transactions on Pattern Analysis and Machine Intelligence, June, 2007. [PDF] [Demo Code]
	Shadow Resistant Object Tracking using Inertia Constraints Hao Jiang and Mark S. Drew Pattern Recognition, July, 2007. [PDF]
	Content Analysis for Audio Classification and Segmentation Lie Lu, Hong-Jiang Zhang, and Hao Jiang IEEE Transactions on Speech and Audio Processing, volume: 10 Issue: 7 , Oct 2002, Page(s): 504 -516. [PDF]

Patent

	4D tracking Hao Jiang, Quanzeng You, Zhengyou Zhang US Patent 20190279382
	Determining an item involved in an event Hao Jiang, Yasser Baseer Asmi, Nishitkumar Ashokkumar Desai, Emilio Ian Maldonado, Ammar Chinoy, Daniel Bibireata, Sudarshan Narasimha Raghavan US Patent 10438277
	Audio Segmentation and Classification Hao Jiang and Hong-Jiang Zhang US Patent 7328149

Conferences and Archives

	Chat2Map: Efficient Scene Mapping From Multi-Ego Conversations Sagnik Majumder, Hao Jiang, Pierre Moulon, Ethan Henderson, Paul Calamia, Kristen Grauman, Vamsi Krishna Ithapu CVPR2023.
	Egocentric Auditory Attention Localization in Conversations Fiona Ryan, Hao Jiang, Abhinav Shukla, James M. Rehg, Vamsi Krishna Ithapu CVPR2023.
	Ego4D: Around the World in 3,000 Hours of Egocentric Video CVPR2022.
	Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization Hao Jiang, Calvin Murdock, Vamsi Krishna Ithapu CVPR2022.
	Egocentric Pose Estimation From Human Vision Span Hao Jiang, Vamsi Krishna Ithapu ICCV 2021. [PDF]
	Real-time 3D Deep Multi-Camera Tracking Quanzeng You and Hao Jiang arXiv:2003.11753. [PDF]
	Action4D: Online Action Recognition in the Crowd and Clutter Quanzeng You and Hao Jiang CVPR 2019. [PDF]
	Real-time Multiple People Hand Localization in 4D Point Clouds Hao Jiang, Quanzeng You arXiv:1903.01695
	Action4D: Real-time Action Recognition in the Crowd and Clutter Quanzeng You, Hao Jiang arXiv:1806.02424 [Project Page]
	Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Video Hao Jiang and Kristen Grauman IEEE Conference on Computer Vision and Pattern Recognition 2017 (CVPR'17). Spotlight. [PDF] [Project Page] [Code]
	Detangling People: Individuating Multiple Close People and Their Body Parts via Region Assembly Hao Jiang and Kristen Grauman IEEE Conference on Computer Vision and Pattern Recognition 2017 (CVPR'17). Oral. [PDF]
	Human's Scene Sketch Understanding Yuxiang Ye, Yijuan Lu, Hao Jiang ACM on International Conference on Multimedia Retrieval 2016 (ICMR'16) [PDF]
	A Bayesian Part-based Approach to 3D Human Pose and Camera Estimation Ernesto Brau, Hao Jiang ICPR'16
	3D Human Pose Estimation via Deep Learning from 2D Annotations Ernesto Brau, Hao Jiang 3DV'16 [PDF].
	Matching Bags of Regions in RGBD images Hao Jiang IEEE Conference on Computer Vision and Pattern Recognition 2015 (CVPR'15). [PDF].
	Finding Approximate Convex Shapes in RGBD Images Hao Jiang European Conference on Computer Vision 2014 (ECCV'14). [PDF] [Code] [Code and Test Data]
	A Linear Approach to Matching Cuboids in RGBD Images Hao Jiang and Jianxiong Xiao IEEE Conference on Computer Vision and Pattern Recognition 2013 (CVPR'13). [PDF] [Code]
	Human Movement Summarization and Depiction from Videos Yijuan Lu and Hao Jiang IEEE International Conference on Multimedia and Expo 2013 (ICME'13). [PDF] [Slides] (Best paper in ICME2013)
	A Scale and Rotation Invariant Approach to Tracking Human Body Part Regions in Videos Yihang Bo and Hao Jiang CVPR 2013 Workshop: Computer Vision in Sports (CVSports'13). [PDF] [Slides]
	Finding People Using Scale, Rotation and Articulation Invariant Matching Hao Jiang European Conference on Computer Vision 2012 (ECCV'12). [PDF]
	Linear Solution to Scale Invariant Global Figure Ground Separation Hao Jiang IEEE Conference on Computer Vision and Pattern Recognition 2012 (CVPR'12). [PDF][Videos]
	Scale Resilient, Rotation Invariant Articulated Object Matching Hao Jiang, Tai-Peng Tian, Kun He and Stan Sclaroff IEEE Conference on Computer Vision and Pattern Recognition 2012 (CVPR'12). [PDF][Videos]
	Scale and Rotation Invariant Matching Using Linearly Augmented Trees Hao Jiang, Tai-Peng Tian and Stan Sclaroff IEEE Conference on Computer Vision and Pattern Recognition 2011 (CVPR'11). (Oral) . [PDF][Videos, Code and Slides]
	Finding Human Poses in Videos Using Concurrent Matching and Segmentation Hao Jiang ACCV 2010 (oral). [PDF]. [Videos].
	Building a Videorama with Shallow Depth of Field Soonmin Bae and Hao Jiang IEEE International Conference on Pattern Recognition (ICPR 2010). [PDF]
	3D Human Pose Reconstruction Using Millions of Exemplars Hao Jiang IEEE International Conference on Pattern Recognition (ICPR 2010). [PDF] [Code]
	Human Pose Estimation Using Consistent Max-Covering Hao Jiang IEEE International Conference on Computer Vision 2009 (ICCV'09). [PDF]
	Linear Solution to Scale and Rotation Invariant Object Matching Hao Jiang and Stella X. Yu IEEE Conference on Computer Vision and Pattern Recognition 2009 (CVPR'09). (Oral) [PDF][Slides][CODE] [Data]. and videos (bear book magazine butterfly bee fish)
	Learning from Disagreements: Discriminative Performance Evaluation Christina Pavlopoulou, David Martin, Stella X. Yu and Hao Jiang The 11th IEEE International Workshop on Performance Evaluation of Tracking and Surveillance (PETS'09), 2009. [PDF]
	Finding Actions Using Shape Flows Hao Jiang and David R. Martin European Conference on Computer Vision 2008 (ECCV'08). [PDF]
	Global Pose Estimation Using Non-Tree Models Hao Jiang and David R. Martin IEEE Conference on Computer Vision and Pattern Recognition 2008 (CVPR'08). [PDF]
	A Linear Programming Approach for Multiple Object Tracking Hao Jiang, Sidney Fels and James J. Little IEEE Conference on Computer Vision and Pattern Recognition 2007 (CVPR'07) . [PDF]
	Successive Convex Matching for Action Detection Hao Jiang, Mark S. Drew and Ze-Nian Li IEEE Conference on Computer Vision and Pattern Recognition 2006 (CVPR'06). [PDF]
	Unsupervised Discovery of Action Classes Yang Wang, Hao Jiang, Mark S. Drew, Ze-Nian Li, Greg Mori IEEE Conference on Computer Vision and Pattern Recognition 2006 (CVPR'06) [PDF]
	Convex Quadratic Programming for Object Localization Hao Jiang, Mark S. Drew, Ze-Nian Li International Conference on Pattern Recognition 2006 (ICPR'06). [PDF]
	Detecting Human Action in Active Video Hao Jiang, Ze-Nian Li and Mark S. Drew International Conference on Multimedia and Expo 2006 (ICME'06). [PDF]
	Linear Programming Matching and Appearance-adaptive Object Tracking Hao Jiang, Mark S. Drew and Ze-Nian Li Energy Minimization Methods in Computer Vision and Pattern Recognition 2005 (EMMCVPR'05). [PDF]
	Linear Programming for Matching in Human Body Gesture Recognition Hao Jiang, Ze-Nian Li and Mark S. Drew Analysis and Modeling of Faces and Gestures 2005 (AMFG'05). [PDF]
	Human Posture Recognition with Convex Programming Hao Jiang, Ze-Nian Li and Mark S. Drew International Conference on Multimedia and Expo 2005 (ICME'05) [PDF]
	Optimizing Motion Estimation with Linear Programming and Detail-Preserving Variational Method Hao Jiang, Ze-Nian Li and Mark S. Drew IEEE Conference on Computer Vision and Pattern Recognition 2004 (CVPR'04). [PDF]
	Shadow-Resistant Tracking in Video Hao Jiang and Mark S. Drew IEEE International Conference on Multimedia and Expo 2003 (ICME'03). [PDF]
	Non-Diagonal Color Correction Brian Funt and Hao Jiang IEEE International Conference on Image Processing 2003 (ICIP'03). [PDF]
	Predicting Cone Quantum Catches Under Illumination Change Graham Finlayson, Brian Funt and Hao Jiang Color Imaging Conference 2003. [PDF]
	A New Direction Adaptive Scheme for Image Interpolation Hao Jiang and Cecilia Moloney International Conference on Image Processing 2002 (ICIP'02), Volume: 3, Page(s): 369 -372. [PDF]
	Error Concealment Using a Diffusion Based Method Hao Jiang and Cecilia Moloney International Conference on Image Processing 2002 (ICIP'02), Volume: 1 , Page(s): 832 -835. [PDF]
	A Predictive Contour Inertia Snake Model for General Video Tracking Hao Jiang and Mark S. Drew International Conference on Image Processing 2002 (ICIP'02), Volume: 3 , Page(s): 413 -416. [PDF]
	A Robust Audio Classification and Segmentation Method Lie Lu, Hao Jiang and Hong-Jiang Zhang ACM Multimedia 2001 (ACM-MM'01):203-211. [PDF]
	Video Segmentation with the Assistance of Audio Content Analysis Hao Jiang, Tong Lin and Hong-Jiang Zhang IEEE International Conference on Multimedia and Expo 2000 (ICME'00). [PDF]

PhD Thesis

Successive Convexification For Consistent Labeling

Invited Talks

Microsoft TechFest, 2018
CVPR 2017.
Princeton University, 2016.
Microsoft Research, 2013.
Northeastern University, 2013.
Harvard University, GVI group, 2013.
MIT, AI Lab, 2013.
Intel Labs, 2013.
Amazon, 2013.
MERL, 2012.
SRI Sarnoff, 2012.
ObjectVideo, 2012.
MIT, AI Lab, 2011.
IBM Research, 2011.
Boston University, 2011.
CVPR 2011.
CVPR 2009.
University of Texas in Austin, 2009
Texas State University, 2009.
Microsoft Live Labs, 2007.
Boston University, 2007.
University of British Columbia, 2006.

Reviewer and Program Committee

IEEE Computer vision and Pattern Recognition (CVPR), 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2019.
International Conference on Computer Vision (ICCV), 2009, 2011, 2013, 2015, 2017, 2019.
European Conference on Computer Vision (ECCV), 2012, 2014.
Internation Conference on Face and Gesture (FG), 2015.
British Machine Vision Conference (BMVC), 2011, 2012, 2013, 2014, 2015, 2016, 2017.
Canadian Conference on Computer and Robot Vision (CRV), 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019.
NSF funding committee, 2011, 2014.