Skip to main content

jiancong wang

Followers

3

Following

1

Public Views

Hyung Jin Chang

David Orlando Delgado Fuentes

matthieu perrot

New York University

University of North Carolina at Chapel Hill

Interests

Uploads

Papers by jiancong wang

NODEO: A Neural Ordinary Differential Equation Based Optimization Framework for Deformable Image Registration

arXiv (Cornell University), Aug 7, 2021

Figure 1. 2D image registration using our framework. The rows show (J) the moving images, (I) fix... more Figure 1. 2D image registration using our framework. The rows show (J) the moving images, (I) fixed images, (J ψ ) warped moving images, (|D ψ |) Jacobian determinants of the transformation ψ, and visualization of ψ respectively. The columns (b)(d)(f)(h)(j) incorporate a fixed boundary constraint, as indicated by the box with solid red line, while (a)(c)(e)(g)(i) do not (box with dotted red line).

Enhanced generative adversarial network for 3D brain MRI super-resolution

arXiv (Cornell University), Jul 10, 2019

Single image super-resolution (SISR) reconstruction for magnetic resonance imaging (MRI) has gene... more Single image super-resolution (SISR) reconstruction for magnetic resonance imaging (MRI) has generated significant interest because of its potential to not only speed up imaging but to improve quantitative processing and analysis of available image data. Generative Adversarial Networks (GAN) have proven to perform well in recovering image texture detail, and many variants have therefore been proposed for SISR. In this work, we develop an enhancement to tackle GAN-based 3D SISR by introducing a new residual-in-residual dense block (RRDG) generator that is both memory efficient and achieves state-of-the-art performance in terms of PSNR (Peak Signal to Noise Ratio), SSIM (Structural Similarity) and NRMSE (Normalized Root Mean Squared Error) metrics. We also introduce a patch GAN discriminator with improved convergence behavior to better model brain image texture. We proposed a novel the anatomical fidelity evaluation of the results using a pre-trained brain parcellation network. Finally, these developments are combined through a simple and efficient method to balance between image and texture quality in the final output.

Nested Scale Editing for Conditional Image Synthesis

arXiv (Cornell University), Jun 3, 2020

Figure : Our approach enables scale-specific visual editings in conditional image synthesis. We c... more

Regional Deep Atrophy: a Self-Supervised Learning Method to Automatically Identify Regions Associated With Alzheimer's Disease Progression From Longitudinal MRI

arXiv (Cornell University), Apr 10, 2023

Longitudinal assessment of brain atrophy, particularly in the hippocampus, is a well-studied biom... more Longitudinal assessment of brain atrophy, particularly in the hippocampus, is a well-studied biomarker for neurodegenerative diseases, such as Alzheimer's disease (AD). In clinical trials, estimation of brain progressive rates can be applied to track therapeutic efficacy of disease modifying treatments. However, most state-of-the-art measurements calculate changes directly by segmentation and/or deformable registration of MRI images, and may misreport head motion or MRI artifacts as neurodegeneration, impacting their accuracy. In our previous study, we developed a deep learning method DeepAtrophy that uses a convolutional neural network to quantify differences between longitudinal MRI scan pairs that are associated with time. DeepAtrophy has high accuracy in inferring temporal information from longitudinal MRI scans, such as temporal order or relative inter-scan interval. DeepAtrophy also provides an overall atrophy score that was shown to perform well as a potential biomarker of disease progression and treatment efficacy. However, DeepAtrophy is not interpretable, and it is unclear what changes in the MRI contribute to progression measurements. In this paper, we propose Regional Deep Atrophy (RDA), which combines the temporal inference approach from DeepAtrophy with a deformable registration neural network and attention mechanism that highlights regions in the MRI image where longitudinal changes are contributing to temporal * Data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at: content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf inference. RDA has similar prediction accuracy as DeepAtrophy, but its additional interpretability makes it more acceptable for use in clinical settings, and may lead to more sensitive biomarkers for disease monitoring in clinical trials of early AD.

Sensitive biomarkers to Alzheimer's disease progression in longitudinal MRI from deep learning

Alzheimer's & Dementia, 2020

BackgroundIn clinical trials of disease‐modifying treatments of Alzheimer’s disease (AD), the mea... more BackgroundIn clinical trials of disease‐modifying treatments of Alzheimer’s disease (AD), the measure of hippocampus volume change from longitudinal MRI is a well‐studied biomarker of disease progression. However, even the most sensitive longitudinal MRI measures can be confounded by the presence of motion and susceptibility artifacts. We hypothesize that deep learning can be used to differentiate MRI changes due to biological factors (tissue loss) from changes due to artifact and other non‐biological factors, leading to novel biomarkers that are more sensitive to longitudinal change that can accurately evaluate disease progression.Method357 subjects (142 training, 215 test) with 3‐8 longitudinal T1‐weighted MRI scans (0.5 to 5.5 years) were selected from ADNI2/GO (Table 1), including cognitively normal (CN), amyloid negative (A‐) older adults, preclinical AD (A+ controls), and A+ patients with early and late mild cognitive impairment (eMCI, lMCI). For each subject, pairs of scans w...

Three-dimensional mapping of neurofibrillary tangle burden in the human medial temporal lobe

Brain, 2021

Tau protein neurofibrillary tangles are closely linked to neuronal/synaptic loss and cognitive de... more Tau protein neurofibrillary tangles are closely linked to neuronal/synaptic loss and cognitive decline in Alzheimer’s disease and related dementias. Our knowledge of the pattern of neurofibrillary tangle progression in the human brain, critical to the development of imaging biomarkers and interpretation of in vivo imaging studies in Alzheimer’s disease, is based on conventional two-dimensional histology studies that only sample the brain sparsely. To address this limitation, ex vivo MRI and dense serial histological imaging in 18 human medial temporal lobe specimens (age 75.3 ± 11.4 years, range 45 to 93) were used to construct three-dimensional quantitative maps of neurofibrillary tangle burden in the medial temporal lobe at individual and group levels. Group-level maps were obtained in the space of an in vivo brain template, and neurofibrillary tangles were measured in specific anatomical regions defined in this template. Three-dimensional maps of neurofibrillary tangle burden rev...

DeepAtrophy: Teaching a neural network to detect progressive changes in longitudinal MRI of the hippocampal region in Alzheimer's disease

NeuroImage, 2021

Investigators represent the names responsible for leadership and infrastructure. Site Investigato... more Investigators represent the names responsible for leadership and infrastructure. Site Investigators represent the names of individuals at each recruiting site. All papers, including methodological papers, should have an acknowledgement list that consists of Infrastructure Investigators plus the FULL list.

Leveraging Label-Specific Discriminant Mapping Features for Multi-Label Learning

ACM Transactions on Knowledge Discovery from Data, 2019

As an important machine learning task, multi-label learning deals with the problem where each sam... more As an important machine learning task, multi-label learning deals with the problem where each sample instance (feature vector) is associated with multiple labels simultaneously. Most existing approaches focus on manipulating the label space, such as exploiting correlations between labels and reducing label space dimension, with identical feature space in the process of classification. One potential drawback of this traditional strategy is that each label might have its own specific characteristics and using identical features for all label cannot lead to optimized performance. In this article, we propose an effective algorithm named LSDM, i.e., leveraging label-specific discriminant mapping features for multi-label learning , to overcome the drawback. LSDM sets diverse ratio parameter values to conduct cluster analysis on the positive and negative instances of identical label. It reconstructs label-specific feature space which includes distance information and spatial topology infor...

A Natural Occurring Mouse Model with Adgrv1 Mutation of Usher Syndrome 2C and Characterization of its Recombinant Inbred Strains

Cellular physiology and biochemistry : international journal of experimental cellular physiology, biochemistry, and pharmacology, Jan 29, 2018

Our laboratory discovered a Kunming mouse with enormous electroretinogram (ERG) defects. Its audi... more Our laboratory discovered a Kunming mouse with enormous electroretinogram (ERG) defects. Its auditory brainstem response (ABR) threshold was significantly elevated and closely resembled the features of Usher syndrome (USH). This study sought to cross these USH-like mice (named KMush/ush mice) with CBA/CaJ mice to establish recombinant inbred strains and identify their phenotypes and genotypes. KMush/ush mice were crossed with CBA/CaJ mice to establish inbred strains by sibling mating. ERG, ABR, ocular fundus morphology, histological examinations of the retina and inner ear, quantitative real-time polymerase chain reaction, western blotting, and exon sequencing were performed to assess the phenotypes and genotypes of the offspring strains. The F1 hybrids from crossing KMush/ush and CBA/CaJ mice had normal ERG and ABR responses. The F2 offspring from intercrossing the F1 mice showed a segregation of the retinitis pigmentosa (RP) and hearing loss phenotypes. The CBA-1ush/ush mice had a...

Multimodal Image Outpainting With Regularized Normalized Diversification

arXiv (Cornell University), Oct 24, 2019

Figure 1: Given only a small foreground region, our model can learn to outpaint a set of diverse ... more

Barnes-Hut Approximation for Point SetGeodesic Shooting

arXiv (Cornell University), Jul 10, 2019

Geodesic shooting has been successfully applied to diffeomorphic registration of point sets. Exac... more Geodesic shooting has been successfully applied to diffeomorphic registration of point sets. Exact computation of the geodesic shooting between point sets, however, requires O(N 2) calculations at each time step on the number of points in the point set. We propose an approximation approach based on the Barnes-Hut algorithm to speed up point set geodesic shooting. This approximation can reduce the algorithm complexity to O(N b + N logN). The evaluation of the proposed method in both simulated images and the medial temporal lobe thickness analysis demonstrates a comparable accuracy to the exact point set geodesic shooting while offering up to 3-fold speed up. This improvement opens up a range of clinical research studies and practical problems to which the method can be effectively applied.

NODEO: A Neural Ordinary Differential Equation Based Optimization Framework for Deformable Image Registration

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

2D image registration using our framework. The rows show (J) the moving images, (I) fixed images,... more 2D image registration using our framework. The rows show (J) the moving images, (I) fixed images, (J ψ) warped moving images, (|D ψ |) Jacobian determinants of the transformation ψ, and visualization of ψ respectively. The columns (b)(d)(f)(h)(j) incorporate a fixed boundary constraints while (a)(c)(e)(g)(i) do not.

Inpaint2Learn: A Self-Supervised Framework for Affordance Learning

2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

Figure 1: Our proposed Inpaint2Learn framework allows us to train models that can predict afforda... more Figure 1: Our proposed Inpaint2Learn framework allows us to train models that can predict affordances in various scenarios. In panel a, our model can predict plausible and diverse human poses in an in-the-wild image. In panel b, our model can predict what type of objects to insert at a given location and how large it should be. In panel c, our model can predict the 6D pose (3D translation + 3D rotation) required to place an object into a target scene.

Improving Multi-atlas Segmentation by Convolutional Neural Network Based Patch Error Estimation

Lecture Notes in Computer Science, 2019

Multi-atlas segmentation (MAS) is widely used in automatically labeling medical images. The perfo... more Multi-atlas segmentation (MAS) is widely used in automatically labeling medical images. The performance of patch-based MAS approaches relies on accurate estimation of local patch similarity, a proxy of the probability that an atlas patch provides the same label as the target patch. Learning-based image patch embedding techniques were recently proposed to transform raw intensity to feature maps and yield promising improvements compared to traditional raw intensity or hand-crafted features. In this study, we present a different approach in which the probability of atlas patch generating an erroneous vote, i.e. having a different label from the target patch, is directly estimated from the patches using a convolutional neural network (CNN). Experiments demonstrate that CNN-based estimates improve the segmentation accuracy of popular patch-based MAS techniques, i.e. spatially varying weighted voting and joint label fusion, in the context of segmenting medial temporal lobe subregions in T1-weighted MRI.

Deep Label Fusion: A 3D End-to-End Hybrid Multi-Atlas Segmentation and Deep Learning Pipeline

Deep learning (DL) is the state-of-the-art methodology in various medical image segmentation task... more Deep learning (DL) is the state-of-the-art methodology in various medical image segmentation tasks. However, it requires relatively large amounts of manually labeled training data, which may be infeasible to generate in some applications. In addition, DL methods have relatively poor generalizability to outof-sample data. Multi-atlas segmentation (MAS), on the other hand, has promising performance using limited amounts of training data and good generalizability. A hybrid method that integrates the high accuracy of DL and good generalizability of MAS is highly desired and could play an important role in segmentation problems where manually labeled data is hard to generate. Most of the prior work focuses on improving single components of MAS using DL rather than directly optimizing the final segmentation accuracy via an end-to-end pipeline. Only one study explored this idea in binary segmentation of 2D images, but it remains unknown whether it generalizes well to multi-class 3D segment...

Learning Object Placement by Inpainting for Compositional Data Augmentation

Computer Vision – ECCV 2020, 2020

We study the problem of common sense placement of visual objects in an image. This involves multi... more We study the problem of common sense placement of visual objects in an image. This involves multiple aspects of visual recognition: the instance segmentation of the scene, 3D layout, and common knowledge of how objects are placed and where objects are moving in the 3D scene. This seemingly simple task is difficult for current learningbased approaches because of the lack of labeled training pair of foreground objects paired with cleaned background scenes. We propose a self-learning framework that automatically generates the necessary training data without any manual labeling by detecting, cutting, and inpainting objects from an image. We propose a PlaceNet that predicts a diverse distribution of common sense locations when given a foreground object and a background scene. We show one practical use of our object placement network for augmenting training datasets by recomposition of object-scene with a key property of contextual relationship preservation. We demonstrate improvement of object detection and instance segmentation performance on both Cityscape [4] and KITTI [9] datasets. We also show that the learned representation of our PlaceNet displays strong discriminative power in image retrieval and classification.

Multimodal Image Outpainting with Regularized Normalized Diversification

2020 IEEE Winter Conference on Applications of Computer Vision (WACV), 2020

Figure 1: Given only a small foreground region, our model can learn to outpaint a set of diverse ... more

Diffeomorphic Medial Modeling

Lecture Notes in Computer Science, 2019

Deformable shape modeling approaches that describe objects in terms of their medial axis geometry... more Deformable shape modeling approaches that describe objects in terms of their medial axis geometry (e.g., m-reps [10]) yield rich geometrical features that can be useful for analyzing the shape of sheet-like biological structures, such as the myocardium. We present a novel shape analysis approach that combines the benefits of medial shape modeling and diffeomorphometry. Our algorithm is formulated as a problem of matching shapes using diffeomorphic flows under constraints that approximately preserve medial axis geometry during deformation. As the result, correspondence between the medial axes of similar shapes is maintained. The approach is evaluated in the context of modeling the shape of the left ventricular wall from 3D echocardiography images. 1

Characterizing Anatomical Variability and Alzheimer’s Disease Related Cortical Thinning in the Medial Temporal Lobe Using Graph-Based Groupwise Registration and Point Set Geodesic Shooting

Shape in Medical Imaging, 2018

Hard to quantify due to the existence of discrete anatomical variants, defined by the folding and... more

Combining Deep Learning and Multi-atlas Label Fusion for Automated Placenta Segmentation from 3DUS

Data Driven Treatment Response Assessment and Preterm, Perinatal, and Paediatric Image Analysis, 2018

In recent years there is growing interest in studying the placenta in vivo. However, 3D ultrasoun... more In recent years there is growing interest in studying the placenta in vivo. However, 3D ultrasound images (3DUS) are typically very noisy, and the placenta shape and position are highly variable. As such, placental segmentation efforts to date have focused on interactive methods that require considerable user input, or automated methods with relatively low performance and various limitations. We propose a novel algorithm using a combination of deep learning and multi-atlas joint label fusion (JLF) methods for automated segmentation of the placenta in 3DUS images. We extract 2D cross-sections of the ultrasound cone beam with a variety of orientations from the 3DUS images and train a convolutional neural network (CNN) on these slices. We use the prediction by the CNN to initialize the multi-atlas JLF algorithm. The posteriors obtained by the CNN and JLF models are combined to enhance the overall segmentation performance. The method is evaluated on a dataset of 47 patients in the first trimester. We perform 4-fold cross-validation and achieve a mean Dice coefficient of \(86.3 \pm 5.3\) for the test folds. This is a substantial increase in accuracy compared to existing automated methods and is comparable to the performance of semi-automated methods currently considered the bronze standard in placenta segmentation.

NODEO: A Neural Ordinary Differential Equation Based Optimization Framework for Deformable Image Registration

arXiv (Cornell University), Aug 7, 2021

Figure 1. 2D image registration using our framework. The rows show (J) the moving images, (I) fix... more Figure 1. 2D image registration using our framework. The rows show (J) the moving images, (I) fixed images, (J ψ ) warped moving images, (|D ψ |) Jacobian determinants of the transformation ψ, and visualization of ψ respectively. The columns (b)(d)(f)(h)(j) incorporate a fixed boundary constraint, as indicated by the box with solid red line, while (a)(c)(e)(g)(i) do not (box with dotted red line).

Enhanced generative adversarial network for 3D brain MRI super-resolution

arXiv (Cornell University), Jul 10, 2019

Single image super-resolution (SISR) reconstruction for magnetic resonance imaging (MRI) has gene... more Single image super-resolution (SISR) reconstruction for magnetic resonance imaging (MRI) has generated significant interest because of its potential to not only speed up imaging but to improve quantitative processing and analysis of available image data. Generative Adversarial Networks (GAN) have proven to perform well in recovering image texture detail, and many variants have therefore been proposed for SISR. In this work, we develop an enhancement to tackle GAN-based 3D SISR by introducing a new residual-in-residual dense block (RRDG) generator that is both memory efficient and achieves state-of-the-art performance in terms of PSNR (Peak Signal to Noise Ratio), SSIM (Structural Similarity) and NRMSE (Normalized Root Mean Squared Error) metrics. We also introduce a patch GAN discriminator with improved convergence behavior to better model brain image texture. We proposed a novel the anatomical fidelity evaluation of the results using a pre-trained brain parcellation network. Finally, these developments are combined through a simple and efficient method to balance between image and texture quality in the final output.

Nested Scale Editing for Conditional Image Synthesis

arXiv (Cornell University), Jun 3, 2020

Figure : Our approach enables scale-specific visual editings in conditional image synthesis. We c... more

Regional Deep Atrophy: a Self-Supervised Learning Method to Automatically Identify Regions Associated With Alzheimer's Disease Progression From Longitudinal MRI

arXiv (Cornell University), Apr 10, 2023

Longitudinal assessment of brain atrophy, particularly in the hippocampus, is a well-studied biom... more Longitudinal assessment of brain atrophy, particularly in the hippocampus, is a well-studied biomarker for neurodegenerative diseases, such as Alzheimer's disease (AD). In clinical trials, estimation of brain progressive rates can be applied to track therapeutic efficacy of disease modifying treatments. However, most state-of-the-art measurements calculate changes directly by segmentation and/or deformable registration of MRI images, and may misreport head motion or MRI artifacts as neurodegeneration, impacting their accuracy. In our previous study, we developed a deep learning method DeepAtrophy that uses a convolutional neural network to quantify differences between longitudinal MRI scan pairs that are associated with time. DeepAtrophy has high accuracy in inferring temporal information from longitudinal MRI scans, such as temporal order or relative inter-scan interval. DeepAtrophy also provides an overall atrophy score that was shown to perform well as a potential biomarker of disease progression and treatment efficacy. However, DeepAtrophy is not interpretable, and it is unclear what changes in the MRI contribute to progression measurements. In this paper, we propose Regional Deep Atrophy (RDA), which combines the temporal inference approach from DeepAtrophy with a deformable registration neural network and attention mechanism that highlights regions in the MRI image where longitudinal changes are contributing to temporal * Data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at: content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf inference. RDA has similar prediction accuracy as DeepAtrophy, but its additional interpretability makes it more acceptable for use in clinical settings, and may lead to more sensitive biomarkers for disease monitoring in clinical trials of early AD.

Sensitive biomarkers to Alzheimer's disease progression in longitudinal MRI from deep learning

Alzheimer's & Dementia, 2020

BackgroundIn clinical trials of disease‐modifying treatments of Alzheimer’s disease (AD), the mea... more BackgroundIn clinical trials of disease‐modifying treatments of Alzheimer’s disease (AD), the measure of hippocampus volume change from longitudinal MRI is a well‐studied biomarker of disease progression. However, even the most sensitive longitudinal MRI measures can be confounded by the presence of motion and susceptibility artifacts. We hypothesize that deep learning can be used to differentiate MRI changes due to biological factors (tissue loss) from changes due to artifact and other non‐biological factors, leading to novel biomarkers that are more sensitive to longitudinal change that can accurately evaluate disease progression.Method357 subjects (142 training, 215 test) with 3‐8 longitudinal T1‐weighted MRI scans (0.5 to 5.5 years) were selected from ADNI2/GO (Table 1), including cognitively normal (CN), amyloid negative (A‐) older adults, preclinical AD (A+ controls), and A+ patients with early and late mild cognitive impairment (eMCI, lMCI). For each subject, pairs of scans w...

Three-dimensional mapping of neurofibrillary tangle burden in the human medial temporal lobe

Brain, 2021

Tau protein neurofibrillary tangles are closely linked to neuronal/synaptic loss and cognitive de... more Tau protein neurofibrillary tangles are closely linked to neuronal/synaptic loss and cognitive decline in Alzheimer’s disease and related dementias. Our knowledge of the pattern of neurofibrillary tangle progression in the human brain, critical to the development of imaging biomarkers and interpretation of in vivo imaging studies in Alzheimer’s disease, is based on conventional two-dimensional histology studies that only sample the brain sparsely. To address this limitation, ex vivo MRI and dense serial histological imaging in 18 human medial temporal lobe specimens (age 75.3 ± 11.4 years, range 45 to 93) were used to construct three-dimensional quantitative maps of neurofibrillary tangle burden in the medial temporal lobe at individual and group levels. Group-level maps were obtained in the space of an in vivo brain template, and neurofibrillary tangles were measured in specific anatomical regions defined in this template. Three-dimensional maps of neurofibrillary tangle burden rev...

DeepAtrophy: Teaching a neural network to detect progressive changes in longitudinal MRI of the hippocampal region in Alzheimer's disease

NeuroImage, 2021

Investigators represent the names responsible for leadership and infrastructure. Site Investigato... more Investigators represent the names responsible for leadership and infrastructure. Site Investigators represent the names of individuals at each recruiting site. All papers, including methodological papers, should have an acknowledgement list that consists of Infrastructure Investigators plus the FULL list.

Leveraging Label-Specific Discriminant Mapping Features for Multi-Label Learning

ACM Transactions on Knowledge Discovery from Data, 2019

As an important machine learning task, multi-label learning deals with the problem where each sam... more As an important machine learning task, multi-label learning deals with the problem where each sample instance (feature vector) is associated with multiple labels simultaneously. Most existing approaches focus on manipulating the label space, such as exploiting correlations between labels and reducing label space dimension, with identical feature space in the process of classification. One potential drawback of this traditional strategy is that each label might have its own specific characteristics and using identical features for all label cannot lead to optimized performance. In this article, we propose an effective algorithm named LSDM, i.e., leveraging label-specific discriminant mapping features for multi-label learning , to overcome the drawback. LSDM sets diverse ratio parameter values to conduct cluster analysis on the positive and negative instances of identical label. It reconstructs label-specific feature space which includes distance information and spatial topology infor...

A Natural Occurring Mouse Model with Adgrv1 Mutation of Usher Syndrome 2C and Characterization of its Recombinant Inbred Strains

Cellular physiology and biochemistry : international journal of experimental cellular physiology, biochemistry, and pharmacology, Jan 29, 2018

Our laboratory discovered a Kunming mouse with enormous electroretinogram (ERG) defects. Its audi... more Our laboratory discovered a Kunming mouse with enormous electroretinogram (ERG) defects. Its auditory brainstem response (ABR) threshold was significantly elevated and closely resembled the features of Usher syndrome (USH). This study sought to cross these USH-like mice (named KMush/ush mice) with CBA/CaJ mice to establish recombinant inbred strains and identify their phenotypes and genotypes. KMush/ush mice were crossed with CBA/CaJ mice to establish inbred strains by sibling mating. ERG, ABR, ocular fundus morphology, histological examinations of the retina and inner ear, quantitative real-time polymerase chain reaction, western blotting, and exon sequencing were performed to assess the phenotypes and genotypes of the offspring strains. The F1 hybrids from crossing KMush/ush and CBA/CaJ mice had normal ERG and ABR responses. The F2 offspring from intercrossing the F1 mice showed a segregation of the retinitis pigmentosa (RP) and hearing loss phenotypes. The CBA-1ush/ush mice had a...

Multimodal Image Outpainting With Regularized Normalized Diversification

arXiv (Cornell University), Oct 24, 2019

Figure 1: Given only a small foreground region, our model can learn to outpaint a set of diverse ... more

Barnes-Hut Approximation for Point SetGeodesic Shooting

arXiv (Cornell University), Jul 10, 2019

Geodesic shooting has been successfully applied to diffeomorphic registration of point sets. Exac... more Geodesic shooting has been successfully applied to diffeomorphic registration of point sets. Exact computation of the geodesic shooting between point sets, however, requires O(N 2) calculations at each time step on the number of points in the point set. We propose an approximation approach based on the Barnes-Hut algorithm to speed up point set geodesic shooting. This approximation can reduce the algorithm complexity to O(N b + N logN). The evaluation of the proposed method in both simulated images and the medial temporal lobe thickness analysis demonstrates a comparable accuracy to the exact point set geodesic shooting while offering up to 3-fold speed up. This improvement opens up a range of clinical research studies and practical problems to which the method can be effectively applied.

NODEO: A Neural Ordinary Differential Equation Based Optimization Framework for Deformable Image Registration

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

2D image registration using our framework. The rows show (J) the moving images, (I) fixed images,... more 2D image registration using our framework. The rows show (J) the moving images, (I) fixed images, (J ψ) warped moving images, (|D ψ |) Jacobian determinants of the transformation ψ, and visualization of ψ respectively. The columns (b)(d)(f)(h)(j) incorporate a fixed boundary constraints while (a)(c)(e)(g)(i) do not.

Inpaint2Learn: A Self-Supervised Framework for Affordance Learning

2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

Figure 1: Our proposed Inpaint2Learn framework allows us to train models that can predict afforda... more Figure 1: Our proposed Inpaint2Learn framework allows us to train models that can predict affordances in various scenarios. In panel a, our model can predict plausible and diverse human poses in an in-the-wild image. In panel b, our model can predict what type of objects to insert at a given location and how large it should be. In panel c, our model can predict the 6D pose (3D translation + 3D rotation) required to place an object into a target scene.

Improving Multi-atlas Segmentation by Convolutional Neural Network Based Patch Error Estimation

Lecture Notes in Computer Science, 2019

Multi-atlas segmentation (MAS) is widely used in automatically labeling medical images. The perfo... more Multi-atlas segmentation (MAS) is widely used in automatically labeling medical images. The performance of patch-based MAS approaches relies on accurate estimation of local patch similarity, a proxy of the probability that an atlas patch provides the same label as the target patch. Learning-based image patch embedding techniques were recently proposed to transform raw intensity to feature maps and yield promising improvements compared to traditional raw intensity or hand-crafted features. In this study, we present a different approach in which the probability of atlas patch generating an erroneous vote, i.e. having a different label from the target patch, is directly estimated from the patches using a convolutional neural network (CNN). Experiments demonstrate that CNN-based estimates improve the segmentation accuracy of popular patch-based MAS techniques, i.e. spatially varying weighted voting and joint label fusion, in the context of segmenting medial temporal lobe subregions in T1-weighted MRI.

Deep Label Fusion: A 3D End-to-End Hybrid Multi-Atlas Segmentation and Deep Learning Pipeline

Deep learning (DL) is the state-of-the-art methodology in various medical image segmentation task... more Deep learning (DL) is the state-of-the-art methodology in various medical image segmentation tasks. However, it requires relatively large amounts of manually labeled training data, which may be infeasible to generate in some applications. In addition, DL methods have relatively poor generalizability to outof-sample data. Multi-atlas segmentation (MAS), on the other hand, has promising performance using limited amounts of training data and good generalizability. A hybrid method that integrates the high accuracy of DL and good generalizability of MAS is highly desired and could play an important role in segmentation problems where manually labeled data is hard to generate. Most of the prior work focuses on improving single components of MAS using DL rather than directly optimizing the final segmentation accuracy via an end-to-end pipeline. Only one study explored this idea in binary segmentation of 2D images, but it remains unknown whether it generalizes well to multi-class 3D segment...

Learning Object Placement by Inpainting for Compositional Data Augmentation

Computer Vision – ECCV 2020, 2020

We study the problem of common sense placement of visual objects in an image. This involves multi... more We study the problem of common sense placement of visual objects in an image. This involves multiple aspects of visual recognition: the instance segmentation of the scene, 3D layout, and common knowledge of how objects are placed and where objects are moving in the 3D scene. This seemingly simple task is difficult for current learningbased approaches because of the lack of labeled training pair of foreground objects paired with cleaned background scenes. We propose a self-learning framework that automatically generates the necessary training data without any manual labeling by detecting, cutting, and inpainting objects from an image. We propose a PlaceNet that predicts a diverse distribution of common sense locations when given a foreground object and a background scene. We show one practical use of our object placement network for augmenting training datasets by recomposition of object-scene with a key property of contextual relationship preservation. We demonstrate improvement of object detection and instance segmentation performance on both Cityscape [4] and KITTI [9] datasets. We also show that the learned representation of our PlaceNet displays strong discriminative power in image retrieval and classification.

Multimodal Image Outpainting with Regularized Normalized Diversification

2020 IEEE Winter Conference on Applications of Computer Vision (WACV), 2020

Figure 1: Given only a small foreground region, our model can learn to outpaint a set of diverse ... more

Diffeomorphic Medial Modeling

Lecture Notes in Computer Science, 2019

Deformable shape modeling approaches that describe objects in terms of their medial axis geometry... more Deformable shape modeling approaches that describe objects in terms of their medial axis geometry (e.g., m-reps [10]) yield rich geometrical features that can be useful for analyzing the shape of sheet-like biological structures, such as the myocardium. We present a novel shape analysis approach that combines the benefits of medial shape modeling and diffeomorphometry. Our algorithm is formulated as a problem of matching shapes using diffeomorphic flows under constraints that approximately preserve medial axis geometry during deformation. As the result, correspondence between the medial axes of similar shapes is maintained. The approach is evaluated in the context of modeling the shape of the left ventricular wall from 3D echocardiography images. 1

Characterizing Anatomical Variability and Alzheimer’s Disease Related Cortical Thinning in the Medial Temporal Lobe Using Graph-Based Groupwise Registration and Point Set Geodesic Shooting

Shape in Medical Imaging, 2018

Hard to quantify due to the existence of discrete anatomical variants, defined by the folding and... more

Combining Deep Learning and Multi-atlas Label Fusion for Automated Placenta Segmentation from 3DUS

Data Driven Treatment Response Assessment and Preterm, Perinatal, and Paediatric Image Analysis, 2018

In recent years there is growing interest in studying the placenta in vivo. However, 3D ultrasoun... more In recent years there is growing interest in studying the placenta in vivo. However, 3D ultrasound images (3DUS) are typically very noisy, and the placenta shape and position are highly variable. As such, placental segmentation efforts to date have focused on interactive methods that require considerable user input, or automated methods with relatively low performance and various limitations. We propose a novel algorithm using a combination of deep learning and multi-atlas joint label fusion (JLF) methods for automated segmentation of the placenta in 3DUS images. We extract 2D cross-sections of the ultrasound cone beam with a variety of orientations from the 3DUS images and train a convolutional neural network (CNN) on these slices. We use the prediction by the CNN to initialize the multi-atlas JLF algorithm. The posteriors obtained by the CNN and JLF models are combined to enhance the overall segmentation performance. The method is evaluated on a dataset of 47 patients in the first trimester. We perform 4-fold cross-validation and achieve a mean Dice coefficient of \(86.3 \pm 5.3\) for the test folds. This is a substantial increase in accuracy compared to existing automated methods and is comparable to the performance of semi-automated methods currently considered the bronze standard in placenta segmentation.