Lingyu Zhang 张凌宇

PhD Student
Duke University
lingyu.zhang@duke.edu

I am a PhD student at the General Robotics Lab @ Duke University, advised by Prof. Boyuan Chen .

Previously, I graduated from Columbia with an MS EE degree. At Columbia, I worked on robust computer vision models with Prof. Junfeng Yang and Prof. Carl Vondrick, and worked closely with PhD student Chengzhi Mao. I was also a research assistant in Prof. Shih-Fu Chang's DVMM lab, where I worked on multimodal learning, under the supervision of Dr. Mingyang Zhou.

Prior to Columbia, I earned my Bachelor's degree at Nanjing University, China, where I did my undergraduate thesis on neural image compression, advised by Prof. Qiu Shen.

CV  /  Scholar  /  Linkedin  /  Twitter  /  Github

[11/2024] Our paper CREW is accepted to TMLR 11/2024

[9/2024] Our paper GUIDE is accepted at Neurips 2024

[9/2024] Our paper HUMAC is released on Arxiv

[8/2024] Our platform for Human-AI teaming CREW is released

I am broadly interested in machine learning for decision making and perception. In particular, advancing the real-world agents by improving the robustness and generalization of ML models.

Real-time human-guided RL with dense continuous rewards and a learned feedback model to reduce human input and enable continual training. We also provide insights on what makes a good human trainer for agents.

We introduce a platform for Human-AI teaming research. CREW offers extensible environment design, enables real-time human-AI communication, supports hybrid Human-AI teaming, parallel sessions, multimodal feedback, and physiological data collection, and features ML community-friendly algorithm design.

Enables multi-robot collaboration from single-human guidance. Inspired by the human theory-of-mind, HUMAC leverages human-robot interface that allows a single human to guide multiple robots simultaneously, through which collaborative behavior can be learned.

We introduce a framework that uses the dense intrinsic constraints in natural images to robustify inference, allowing the model to adjust dynamically to each individual image's unique and potentially novel characteristics at inference time.

We find that adversarial attacks generated for fooling video classifiers also collaterally corrupt motion. We propose to defend against attacks at test time by restoring disrupted motion.

We improve and optimize the ZNCC stereo matching algorithm for three-dimensional eye localization. We improve operation logic of the matching and optimize the scanning strategy based on the application scenarios. algorithm

We propose to use noise as masks for masked image modeling. While randomized patch masking has yielded decent results in self-supervised learning, it is not at all obvious that it is the optimal design. We show that theoretically inspired semantically-guided noise masks can be a potentially well-performing alternative.

We propose to use deterministic encoding along with actual quantization on latents, rendering the IB problem a source compression. By doing so, finite non-trivial mutual information can be estimated.

We modeled mixed sources of audio signals by sinusoidal modeling with Short-Time Fourier Transforms. Based on selected spectral peaks of sinusoidal parameters, we constructed a similarity function between time and frequency components, and applied spectral clustering to globally partition the data.

Learned image compression has surpassed the rate-distortion performance of hand-crafted traditional image codecs in recent years. However, they are not yet practical because of their significantly slower decoding speed than classical algorithms. We investigated the possibility of directly performing vision tasks in the latent space and found that using a multi-scale encoder helped preserve semantic meaning in latent codes while maintaining state-of-the-art compression rates

Implemented an accelerated stereo matching algorithm for video sequences, utilizing a dynamic disparity range search based on temporal correlation between frames, saving 21% of computational time with minimal accuracy loss. Designed a Divided Section cost function, preserving more information than Census cost, achieving 18% better matching accuracy while trading off computational complexity.

Webpage template from Jon Barron