PUBLICATION

Preprint

(* indicates corresponding author)

RoboFactory: Exploring Embodied Agent Collaboration with Compositional Constraints
Yiran Qin, Li Kang, Xiufeng Song, Zhenfei Yin*, Xiaohong Liu, Xihui Liu, Ruimao Zhang*, Lei Bai*
arXiv preprint arXiv:2503.16408 (2025) 【PDF】
( We propose the concept of compositional constraints for embodied multi-agent systems, addressing the challenges arising from collaboration among embodied agents.)




Ensuring Force Safety in Vision-Guided Robotic Manipulation via Implicit Tactile Calibration
Lai Wei^, Jiahua Ma^, Yibo Hu, Ruimao Zhang*
arXiv preprint arXiv:2412.10349 (2024) 【PDF】
( We introduce a novel state diffusion termed SafeDiff to generate a prospective state sequence from the visual context observation while incorporating real-time tactile feedback to refine the sequence.)




WorldSimBench: Towards Video Generation Models as World Simulators
Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao*, Lei Bai*, Wanli Ouyang, Ruimao Zhang*
arXiv preprint arXiv:2410.18072v1 (2024) 【PDF】
( We take the initial step in evaluating Predictive Generative Models up to the S3 stage by introducing both Explicit Perceptual Evaluation and Implicit Manipulative Evaluation)




MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control
Enshen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang, Ruimao Zhang*, Lu Sheng*, Yu Qiao, Jing Shao
arXiv preprint arXiv:2403.12037 (2024) 【PDF】
( We employ a Chain-of-Imagination (CoI) mechanism to envision the step-by-step process of executing instructions and translating imaginations into more precise visual prompts tailored to the current state )

Newly Accepted Articles

(* indicates corresponding author)

1. Jie Yang, Ailing Zeng, Tianhe Ren, Shilong Liu, Feng Li, Ruimao Zhang*, Lei Zhang, "ED-Pose++: Enhanced Explicit Box Detection for Conventional and Interactive Multi-Object Keypoint Detection“, IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2025 ( The extension version of our CVPR and ICCV papers. ED-Pose for Multi-Object Keypoint Detection.) 【PDF】


2. Shunlin Lu, Jingbo Wang*, Zeyu Lu, Ling-Hao Chen, Wenxun Dai, Junting Dong, Zhiyang Dou, Bo Dai*, Ruimao Zhang*, "ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model“, Proc. of IEEE International Conference on Computer Vision and Pattern Recognition( CVPR ), 2025 ( We introduce a scalable motion generation framework and observe the scaling behavior in autoregressive motion generation model for the first time. ) 【PDF】【Code】


3. Yiran Qin, Ao Sun, Hong Yuze, Benyou Wang, Ruimao Zhang*, "NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistants“, Proc. of IEEE International Conference on Robotics and Automation( ICRA ), 2025 ( A large vision-language model with a diffusion network, named NavigateDiff, continuously predicts the agent’s potential observations to assist robots in generating robust actions.) 【PDF】


4. Chaoqun Wang, Jie Yang, Xiaobin Hong, Ruimao Zhang*, "Unlock the Power of Unlabeled Data in Language Driving Model“, Proc. of IEEE International Conference on Robotics and Automation( ICRA ), 2025 ( This work aims to overcome the barrier of large models' extreme dependence on costly large-scale high-quality annotated data in self-driving scenarios.) 【PDF】


5. Chaoqun Wang, Xiaobin Hong, Ruimao Zhang*, "Semantic-Supervised Spatial-Temporal Fusion for LiDAR-based 3D Object Detection“, Proc. of IEEE International Conference on Robotics and Automation( ICRA ), 2025 ( A novel fusion module to address spatial misalignment caused by object motion over time.) 【PDF】


6. Ziye Wang, Yiran Qin, Lin Zeng, Ruimao Zhang*, "High-Dynamic Radar Sequence Prediction for Weather Nowcasting Using Spatiotemporal Coherent Gaussian Representation“, Proc. of International Conference on Learning Representations( ICLR ), 2025 (Oral) ( A newly proposed Spatio-Temporal Coherent Gaussian Representation (STC-GS) for dynamic scene prediction, offering a promising way for advancing 4D world model! ) 【PDF】【Project Page】


7. Chaoqun Wang, Yiran Qin, Zijian Kang, Ningning Ma, Yukai Shi, Zhen Li, Ruimao Zhang*, "Boosting 3D Object Detection via Self-distilling Introspective Data“, IEEE Transactions on Intelligent Transportation Systems ( T-ITS ), 2025 ( A novel self-distilling paradigm termed SID to boost the accuracy of 3D object detection in both LiDAR-based and LiDAR-Camera-based scenarios.) 【PDF】


8. Jie Yang, Wang Zeng, Sheng Jin, Lumin Xu, Wentao Liu, Chen Qian, Ruimao Zhang*, "KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension“, Proc. of Conference on Neural Information Processing Systems ( NeurIPS ), 2024 ( KptLLM introduces a novel challenge of Semantic Keypoint Comprehension, which aims to understand keypoints within different human-AI interaction contexts. ) 【PDF】


9. Jie Yang, Xuesong Niu, Nan Jiang, Ruimao Zhang*, Siyuan Huang, "F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactions“, Proc. of European Conference on Computer Vision ( ECCV ), 2024 ( F-HOI aims to unify human-object interaction's understanding, reasoning, and generation into a general framework by learning HOI representations in 2D, 3D, and linguistic spaces. ) 【PDF】


10. Jie Yang, Ailing Zeng*, Ruimao Zhang*, Lei Zhang, "X-Pose: Detecting Any Keypoints“, Proc. of European Conference on Computer Vision ( ECCV ), 2024 ( UniPose is proposed to detect keypoints of any articulated, rigid, and soft objects via visual or textual prompts for fine-grained vision understanding and manipulation. ) 【PDF】【Code】

Recent Selected Publications ( See Full List )

(* indicates corresponding author)

ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model
Shunlin Lu, Jingbo Wang*, Zeyu Lu, Ling-Hao Chen, Wenxun Dai, Junting Dong, Zhiyang Dou, Bo Dai*, Ruimao Zhang*
Proc. of IEEE International Conference on Computer Vision and Pattern Recognition( CVPR ), 2025 【PDF】【Code】
( Scaling behavior in autoregressive motion generation model for the first time. )




High-Dynamic Radar Sequence Prediction for Weather Nowcasting Using Spatiotemporal Coherent Gaussian Representation
Ziye Wang, Yiran Qin, Lin Zeng, Ruimao Zhang*
Proc. of International Conference on Learning Representations( ICLR ), 2025 (Oral) 【Project】
( A newly proposed Spatio-Temporal Coherent Gaussian Representation (STC-GS) for dynamic scene prediction, offering a promising way for advancing 4D world model! )




HumanTOMATO: Text-aligned Whole-body Motion Generation
Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang* , Lei Zhang, Heung-Yeung Shum*
Proc. of International Conference on Machine Learning ( ICML ), 2024【PDF】【Code】
( A novel text-aligned whole-body motion generation framework that can generate high-quality, diverse, and coherent whole body motions. )




Open-World Human-Object Interaction Detection via Multi-modal Prompts
Jie Yang, Bingliang Li, Ailing Zeng, Lei Zhang, Ruimao Zhang*
Proc. of IEEE International Conference on Computer Vision and Pattern Recognition ( CVPR ), 2024 【PDF】 【Project】
( A novel prompt-based HOI detector designed to leverage both textual descriptions for open-set generalization and visual exemplars for handling high ambiguity in descriptions. )





MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception
Yiran Qin, Enshen Zhou, Qichang Liu, Zhenfei Yin, Lu Sheng*, Ruimao Zhang*, Yu Qiao, Jing Shao
Proc. of IEEE International Conference on Computer Vision and Pattern Recognition ( CVPR ), 2024 【PDF】 【Project】 【Youtube】【Bilibili】
( MP5 is an open-ended multimodal embodied system that can conduct situation-aware plans and perform embodied action control via active perception scheme. )





FreeMan: Towards Benchmarking 3D Human Pose Estimation under Real-World Conditions
Jiong Wang, Fengyu Yang, Wenbo Gou, Bingliang Li, Danqi Yan, Ailing Zeng, Yijun Gao, Junle Wang, Ruimao Zhang*
Proc. of IEEE International Conference on Computer Vision and Pattern Recognition ( CVPR ), 2024 【PDF】【Project】






Toward Accurate Camera-based 3D Object Detection via Cascade Depth Estimation and Calibration
Chaoqun Wang, Yiran Qin, Zijian Kang, Ningning Ma, Ruimao Zhang*
Proc. of IEEE International Conference on Robotics and Automation ( ICRA ), 2024
【PDF】





Enhancing Human-AI Collaboration Through Logic-Guided Reasoning
Chengzhi Cao, Yinghao Fu, Sheng Xu, Ruimao Zhang, Shuang Li,
Proc. of International Conference on Learning Representations ( ICLR ), 2024
【PDF】





Neural Interactive Keypoint Detection
Jie Yang, Ailing Zeng*, Feng Li, Shilong Liu, Ruimao Zhang*, Lei Zhang
Proc. of IEEE International Conference on Computer Vision ( ICCV ), 2023
【PDF】 【Code】





SupFusion: Supervised LiDAR-Camera Fusion for 3D Object Detection
Yiran Qin, Chaoqun Wang, Zijian Kang, Ningning Ma, Zhen Li, Ruimao Zhang*
Proc. of IEEE International Conference on Computer Vision ( ICCV ), 2023
【PDF】 【Code】





Semantic Human Parsing via Scalable Semantic Transfer over Multiple Label Domains
Jie Yang, Chaoqun Wang, Zhen Li, Junle Wang, Ruimao Zhang*
Proc. of IEEE International Conference on Computer Vision and Pattern Recognition ( CVPR ), 2023 【PDF】【Code】





Inherent Consistent Learning for Accurate Semi-supervised Medical Image Segmentation
Ye Zhu, Jie Yang, Siqi Liu, Ruimao Zhang*
Proc. of Conference on Medical Imaging with Deep Learning( MIDL ), 2023 ( Oral )
【PDF】【Code】





Explicit Box Detection Unifies End-to-End Multi-Person Pose Estimation
Jie Yang, Ailing Zeng*, Shilong Liu, Feng Li, Ruimao Zhang*, Lei Zhang
Proc. of International Conference on Learning Representations( ICLR ), 2023
【PDF】【Code】





AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
Yuanfeng Ji, Haotian Bai, Jie Yang, Chongjian Ge, Ye Zhu, Ruimao Zhang*, et al.
Proc. of Conference on Neural Information Processing Systems ( NeurIPS ), 2022 ( Oral )
【PDF】【AMOS Challenge】





Weakly Supervised Object Localization via Transformer with Implicit Spatial Calibration
Haotian Bai, Ruimao Zhang*, Jiong Wang, Xiang Wan
Proc. of Europe Conference on Computer Vision( ECCV ), 2022
【PDF】 【Code】 【Youtube】





Switchable Normalization for Learning-to-Normalize Deep Representation
Ping Luo, Ruimao Zhang*, Jiamin Ren, Zhanglin Peng, Jingyu Li
IEEE Transactions on Pattern Analysis and Machine Intelligence ( T-PAMI ), 43(2):712-728, 2021
【PDF】 【Code】





Exemplar Normalization for Learning Deep Representation
Ruimao Zhang, Zhanglin Peng, Lingyun Wu, Zhen Li, Ping Luo
Proc. of IEEE International Conference on Computer Vision and Pattern Recognition ( CVPR ), 2020 【PDF】 【Supp】





Hierarchical Scene Parsing by Weakly Supervised Learning with Image Descriptions
Ruimao Zhang, Liang Lin, Guangrun Wang, Meng Wang, Wangmeng Zuo
IEEE Transactions on Pattern Analysis and Machine Intelligence ( T-PAMI ), 41(3):596 - 610, 2019
【PDF】





SCAN: Self-and-Collaborative Attention Network for Video Person Re-identification
Ruimao Zhang, Jingyu Li, Hongbin Sun, Yuying Ge, Ping Luo, Xiaogang Wang, Liang Lin
IEEE Transactions on Image Processing ( T-IP ), 28(10):4870-4882, 2019
【PDF】【Code】

MEMBER

Ph.D. Students

Chaoqun Wang

Ph.D., since 2021, CUHK-SZ

Scene Understanding, Video Analysis, Multimodal Learning

M.S.: Nanjing Univ. of Sci. & Tech.

B.S.: Huazhong Univ. of Sci. & Tech.

Yiran Qin

Ph.D., since 2021, CUHK-SZ

Scene Understanding, Embodied AI, Large Visual Language Model

M.S.: not applicable

B.E.: Shandong University (Top 10%)

Jie Yang

Ph.D., since 2021, CUHK-SZ

Human Centric Visual Perception and Generation

M.S.: not applicable

B.E.: Harbin Engineering Univ. (Top 1%)

Shunlin Lu

Ph.D., since 2023, CUHK-SZ

Human Centric Understanding and Generation, Multi-modal Learning

M.S.: University of Southern California

B.E.: Wuhan University of Technology

Jiahua Ma

Ph.D., since 2025, Sun Yat-sen University

Embodied AI, Vision-Language-Action Model, Policy Learning

M.S.: Shanghai Jiao Tong University

B.E.: Xidian University (Top 5%)

Research Assistants

Ziye Wang

M.S., Harbin Institute of Technology

Architecture Design, Generative Model, Gaussian Splatting

Xuechen Xiong

M.S., Huazhong Univ. of Sci. & Tech.

Human Motion Generation, Multi-modal Learning

Bingqi Liu

M.S., Imperial College London

Human Motion Generation, Multi-modal Learning

Yuyu Sun

B.S., South China Normal Univ. (Top 1%)

Motion Generation, Humanoid Robot Simulation

Alumni

TEACHING