Journal of Medical Internet Research, May 12, 2021
Background: Millions of individuals with visual impairment use vision assistance apps to help wit... more Background: Millions of individuals with visual impairment use vision assistance apps to help with their daily activities. The most widely used vision assistance apps are magnifier apps. It is still largely unknown what the apps are used for. Lack of insight into the visual needs of individuals with visual impairment is a hurdle for the development of more effective assistive technologies. Objective: This study aimed to investigate how needs for visual aids may vary with social activities, by observing the changes in the usage of a smartphone magnifier app when many users take breaks from work. Methods: The number of launches of the SuperVision Magnifier app was determined retrospectively from 2018 to 2020 from among active users worldwide. The fluctuation in app usage was examined by comparing weekday vs weekend periods, Christmas and new year vs nonholiday seasons, and COVID-19 lockdowns vs the easing of restriction during the pandemic. Results: On average, the app was used 262,466 times by 38,237 users each month in 2020 worldwide. There were two major trough points on the timeline of weekly app usage, one aligned with the COVID-19 lockdowns in April 2020 and another aligned with the Christmas and new year week in 2018 and 2019. The app launches declined by 6947 (11% decline; P<.001) during the lockdown and by 5212 (9% decline; P=.001) during the holiday weeks. There was no significant decline during March to May 2019. App usage compensated for seasonal changes was 8.6% less during weekends than during weekdays (P<.001). Conclusions: The need for vision assistance technology was slightly lower during breaks and lockdowns, probably because the activities at home were different and less visually demanding. Nevertheless, for the entire user population, the needs for visual aids are still substantial.
Tracking head movement in outdoor activities is more challenging than in controlled indoor lab en... more Tracking head movement in outdoor activities is more challenging than in controlled indoor lab environments. Large-magnitude head scanning is common under natural conditions. Compensatory gaze (head and eye) scanning while walking may be critical for people with visual field loss. We compared the accuracy of two outdoor head tracking methods: differential inertial measurement units (IMU) and simultaneous localization and mapping (SLAM). At a fixed location experiment, a gaze aiming test showed that SLAM outperforms IMU in terms of error (IMU: 9.6°, SLAM: 4.47°). In an urban street walking experiment conducted with five patients with hemifield loss, the IMU drift, quantified by root-mean-square deviation, was as high as 68.1°, while the drift of SLAM was only 5.3°. However, the SLAM method suffered from data loss due to tracking failure (~10% overall, and ~ 18% when crossing streets). Our results show that the SLAM and IMU methods have complementary properties. Because of no data gaps, the differential IMU method may be desirable as compared to SLAM in settings where the signal drift can be removed in post-processing and small gaze estimation errors can be tolerated. Keywords Head scanning • IMU • Outdoor navigation • Visual SLAM People's mobility is affected by where they look, which in turn requires both head and eye movements. While saccade amplitude can be quite large, the distribution of saccade amplitudes in real-world circumstances are largely below 15° (Bahill, Adler, & Stark, 1975), making head tracking a necessity to understand real-world gaze (eye + head) (Bahill, Adler, & Stark, 1975; Einhäuser et al., 2007; Rothkopf & Pelz, 2004). To compensate for their visual field loss, visually impaired people while walking would presumably need to scan the environment more than the normally sighted. Numerous studies have analyzed the eye and head movements for people with normal vision and with vision loss in controlled laboratory setups (
Background Millions of individuals with visual impairment use vision assistance apps to help with... more Background Millions of individuals with visual impairment use vision assistance apps to help with their daily activities. The most widely used vision assistance apps are magnifier apps. It is still largely unknown what the apps are used for. Lack of insight into the visual needs of individuals with visual impairment is a hurdle for the development of more effective assistive technologies. Objective This study aimed to investigate how needs for visual aids may vary with social activities, by observing the changes in the usage of a smartphone magnifier app when many users take breaks from work. Methods The number of launches of the SuperVision Magnifier app was determined retrospectively from 2018 to 2020 from among active users worldwide. The fluctuation in app usage was examined by comparing weekday vs weekend periods, Christmas and new year vs nonholiday seasons, and COVID-19 lockdowns vs the easing of restriction during the pandemic. Results On average, the app was used 262,466 ti...
A normally sighted person can see a grating of 30 cycles per degree or higher, but spatial freque... more A normally sighted person can see a grating of 30 cycles per degree or higher, but spatial frequencies needed for motion perception are much lower than that. It is unknown for natural images with a wide spectrum how all the visible spatial frequencies contribute to motion speed perception. In this work, we studied the effect of spatial frequency content on motion speed estimation for sequences of natural and stochastic pixel images by simulating different visual conditions, including normal vision, low vision (low-pass filtering), and complementary vision (high-pass filtering at the same cutoff frequencies of the corresponding low-vision conditions) conditions. Speed was computed using a biological motion energy-based computational model. In natural sequences, there was no difference in speed estimation error between normal vision and low vision conditions, but it was significantly higher for complementary vision conditions (containing only high-frequency components) at higher speeds. In stochastic sequences that had a flat frequency distribution, the error in normal vision condition was significantly larger compared with low vision conditions at high speeds. On the contrary, such a detrimental effect on speed estimation accuracy was not found for low spatial frequencies. The simulation results were consistent with the motion direction detection task performed by human observers viewing stochastic sequences. Together, these results (i) reiterate the importance of low frequencies in motion perception, and (ii) indicate that high frequencies may be detrimental for speed estimation when low frequency content is weak or not present.
Translational Vision Science & Technology, 2020
The purpose of this study was to investigate the telescope use behaviors in natural daily driving... more The purpose of this study was to investigate the telescope use behaviors in natural daily driving of people with reduced visual acuity licensed to drive with a bioptic (a small spectacle-mounted telescope). Methods: A large dataset (477 hours) of naturalistic driving was collected from 19 bioptic drivers (visual acuity 20/60 to 20/160 without the telescope). To reduce the data volume, a multiloss 50-layer deep residual neural network (ResNet-50) was used to detect potential bioptic telescope use events. Then, a total of 120 hours of selected video clips were reviewed and annotated in detail. Results: The frequency of looking through their telescopes ranged from 4 to 308 times per hour (median: 27, interquartile range [IQR], 19-75), with each bioptic use lasting median 1.4 seconds (IQR, 1.2-1.8). Thus, participants spent only 1.6% (IQR, 0.7%-3.5%) driving time with their telescopes aiding their vision. Bioptic telescopes were used most often for checking the road ahead (84.8%), followed by looking at traffic lights (5.3%), and reading road signs (4.6%). Conclusions: In daily driving, the bioptic drivers mostly (>98% of driving time) drove under low visual acuity conditions. The bioptic telescope was mainly used for observing road and traffic conditions in the distance for situational awareness. Only a small portion of usage was for road sign reading. Translational Relevance: This study provides new insights into how the vision rehabilitation device-bioptic telescopes are used in daily driving. The findings may be helpful for designing bioptic driving training programs.
PURPOSEThere are many visually impaired people who can drive legally with bioptic telescope. Draw... more PURPOSEThere are many visually impaired people who can drive legally with bioptic telescope. Drawing on the experience of drivers with reduced vision, this study investigated the role of motion perception and visual acuity in driving, under simulated low visual acuity.METHODSTwenty normally sighted participants took part in a driving hazard perception (HP) test, in four different conditions: with/without motion interruption and with/without simulated low visual acuity. In interrupted motion conditions a mask frame was inserted between every frame of the driving videos. In simulated low visual acuity conditions, participants wore glasses with diffusing filters that lowered their visual acuity to 20/120 on average. Participants’ response time, hazard detection rates, and HP scores, which combined response time and detection rate, were compared.RESULTSRepeated measure ANOVA revealed that the HP scores significantly declined from 20.46 to 16.82 due to the motion mask (F(1,19) = 9.343, p...
Speed perception is an important task performed by our visual system in various daily life tasks.... more Speed perception is an important task performed by our visual system in various daily life tasks. In various psychophysical tests, relationship between spatial frequency, temporal frequency, and speed has been examined in human subjects. The role of vision impairment in speed perception has also been previously examined. In this work, we examine the inter-relationship between speed, spatial frequency, low vision conditions, and the type of input motion stimuli in motion perception accuracy. For this purpose, we propose a computational model for speed perception and evaluate it in custom generated natural and stochastic sequences by simulating low-vision conditions (low pass filtering at different cutoff frequencies) as well as complementary vision conditions (high pass versions at the same cutoff frequencies). Our results show that low frequency components are critical for accurate speed perception, whereas high frequencies do not play any important role in speed estimation. Since p...
This work proposes a hardware-friendly, dense optical flow-based Time-to-Collision (TTC) estimati... more This work proposes a hardware-friendly, dense optical flow-based Time-to-Collision (TTC) estimation algorithm intended to be deployed on smart video sensors for collision avoidance. The algorithm optimized for hardware first extracts biological visual motion features (motion energies), and then utilizes a Random Forests regressor to predict robust and dense optical flow. Finally, TTC is reliably estimated from the divergence of the optical flow field. This algorithm involves only feed-forward data flows with simple pixel-level operations, and hence has inherent parallelism for hardware acceleration. The algorithm offers good scalability, allowing for flexible tradeoffs among estimation accuracy, processing speed and hardware resource. Experimental evaluation shows that the accuracy of the optical flow estimation is improved due to the use of Random Forests compared to existing voting-based approaches. Furthermore, results show that estimated TTC values by the algorithm closely follo...
IEEE transactions on circuits and systems. II, Express briefs : a publication of the IEEE Circuits and Systems Society, 2018
This paper proposes a modified Eulerian Video Magnification (EVM) algorithm and a hardware implem... more This paper proposes a modified Eulerian Video Magnification (EVM) algorithm and a hardware implementation of a motion magnification core for smart image sensors. Compared to the original EVM algorithm, we perform the pixel-wise temporal bandpass filtering only once rather than multiple times on all scale layers, to reduce the memory and multiplier requirement for hardware implementation. A pixel stream processing architecture with pipelined blocks is proposed for the magnification core, enabling it to readily fit common image sensing components with streaming pixel output, while achieving higher performance with lower system cost. We implemented an FPGA-based prototype that is able to process up to 90M pixels per second and magnify subtle motion. The motion magnification results are comparable to the original algorithm running on PC.
Analyzing naturalistic driving behavior recorded with in-car cameras is an ecologically valid met... more Analyzing naturalistic driving behavior recorded with in-car cameras is an ecologically valid method for measuring driving errors, but it is time intensive and not easily applied on a large scale. This study validated a semi-automated, computerized method using archival naturalistic driving data collected for drivers with mild Alzheimer's disease (AD; = 44) and age-matched healthy controls (HC; = 16). The computerized method flagged driving situations where safety concerns are most likely to occur (i.e., rapid stops, lane deviations, turns, and intersections). These driving epochs were manually reviewed and rated for error type and severity, if present. Ratings were made with a standardized scoring system adapted from DriveCam. The top eight error types were applied as features to train a logistic model tree classifier to predict diagnostic group. The sensitivity and specificity were compared among the event-based method, on-road test, and composite ratings of two weeks of recor...
International journal of pattern recognition and artificial intelligence, 2018
Lane changes are important behaviors to study in driving research. Automated detection of lane-ch... more Lane changes are important behaviors to study in driving research. Automated detection of lane-change events is required to address the need for data reduction of a vast amount of naturalistic driving videos. This paper presents a method to deal with weak lane-marker patterns as small as a couple of pixels wide. The proposed method is novel in its approach to detecting lane-change events by accumulating lane-marker candidates over time. Since the proposed method tracks lane markers in temporal domain, it is robust to low resolution and many different kinds of interferences. The proposed technique was tested using 490 h of naturalistic driving videos collected from 63 drivers. The lane-change events in a 10-h video set were first manually coded and compared with the outcome of the automated method. The method's sensitivity was 94.8% and the data reduction rate was 93.6%. The automated procedure was further evaluated using the remaining 480-h driving videos. The data reduction rat...
Gaze-contingent displays have been widely used in vision research and virtual reality application... more Gaze-contingent displays have been widely used in vision research and virtual reality applications. Due to data transmission, image processing, and display preparation, the time delay between the eye tracker and the monitor update may lead to a misalignment between the eye position and the image manipulation during eye movements. We propose a method to reduce the misalignment using a Taylor series to predict the saccadic eye movement. The proposed method was evaluated using two large datasets including 219,335 human saccades (collected with an EyeLink 1000 system, 95% range from 1° to 32°) and 21,844 monkey saccades (collected with a scleral search coil, 95% range from 1° to 9°). When assuming a 10-ms time delay, the prediction of saccade movements using the proposed method could reduce the misalignment greater than the state-of-the-art methods. The average error was about 0.93° for human saccades and 0.26° for monkey saccades. Our results suggest that this proposed saccade predicti...
IEEE Transactions on Circuits and Systems for Video Technology, 2016
This paper proposes a bio-inspired visual motion estimation algorithm based on motion energy, alo... more This paper proposes a bio-inspired visual motion estimation algorithm based on motion energy, along with its compact very-large-scale integration (VLSI) architecture using low-cost embedded systems. The algorithm mimics motion perception functions of retina, V1, and MT neurons in a primate visual system. It involves operations of ternary edge extraction, spatiotemporal filtering, motion energy extraction, and velocity integration. Moreover, we propose the concept of confidence map to indicate the reliability of estimation results on each probing location. Our algorithm involves only additions and multiplications during runtime, which is suitable for lowcost hardware implementation. The proposed VLSI architecture employs multiple (frame, pixel, and operation) levels of pipeline and massively parallel processing arrays to boost the system performance. The array unit circuits are optimized to minimize hardware resource consumption. We have prototyped the proposed architecture on a low-cost field-programmable gate array platform (Zynq 7020) running at 53-MHz clock frequency. It achieved 30-frame/s real-time performance for velocity estimation on 160 × 120 probing locations. A comprehensive evaluation experiment showed that the estimated velocity by our prototype has relatively small errors (average endpoint error < 0.5 pixel and angular error < 10°) for most motion cases.
Investigative Opthalmology & Visual Science, 2015
PURPOSE. Although the impact of homonymous visual field defects (HFDs) on mobility has been inves... more PURPOSE. Although the impact of homonymous visual field defects (HFDs) on mobility has been investigated previously, the emphasis has been on obstacle detection. Relatively little is known about HFD patients' ability to judge collisions once an obstacle is detected. We investigated this using a walking simulator. METHODS. Patients with HFDs (n ¼ 29) and subjects with normal vision (NV; n ¼ 21) were seated in front of a large screen on which a visual simulation of walking was displayed. They made collision judgments for a human figure that appeared for 1 second at lateral offsets from the virtual walking path. A perceived-collision threshold was calculated for right and left sides. RESULTS. Symmetrical collision thresholds (same on left and right sides) were measured for participants with NV (n ¼ 21), and right (n ¼ 9) and left (n ¼ 7) HFD without hemispatial neglect. Participants with left neglect (n ¼ 10) showed significant asymmetry with thresholds smaller (compared to the NV group and other HFD groups) on the blind (P < 0.001) and larger on the seeing (P ¼ 0.05) sides. Despite the asymmetry, the overall width of the zone of perceived collision risk was not different, suggesting a relatively uniform rightward deviation in judgments of the left neglect group. CONCLUSIONS. Left neglect was associated with rightward asymmetry in collision judgments, which may cause collisions on the left side even when an obstacle is detected. These behaviors may represent the spatial misperceptions in body midline described previously in patients with left neglect.
Gaze-contingent display paradigms play an important role in vision research. The time delay due t... more Gaze-contingent display paradigms play an important role in vision research. The time delay due to data transmission from eye tracker to monitor may lead to a misalignment between the gaze direction and image manipulation during eye movements, and therefore compromise the contingency. We present a method to reduce this misalignment by using a compressed exponential function to model the trajectories of saccadic eye movements. Our algorithm was evaluated using experimental data from 1,212 saccades ranging from 38 to 308, which were collected with an EyeLink 1000 and a Dual-Purkinje Image (DPI) eye tracker. The model fits eye displacement with a high agreement (R 2. 0.96). When assuming a 10-millisecond time delay, prediction of 2D saccade trajectories using our model could reduce the misalignment by 30% to 60% with the EyeLink tracker and 20% to 40% with the DPI tracker for saccades larger than 88. Because a certain number of samples are required for model fitting, the prediction did not offer improvement for most small saccades and the early stages of large saccades. Evaluation was also performed for a simulated 100-Hz gaze-contingent display using the prerecorded saccade data. With prediction, the percentage of misalignment larger than 28 dropped from 45% to 20% for EyeLink and 42% to 26% for DPI data. These results suggest that the saccadeprediction algorithm may help create more accurate gaze-contingent displays.
Eye movements while watching video: comparisons across viewer groups Identifying the area of inte... more Eye movements while watching video: comparisons across viewer groups Identifying the area of interest (AOI) on a video frame may be necessary: (1) to develop a television magnifying aid for people with low vision; (2) to implement some data compression schemes; and (3) to transform images for rendering on devices with small display areas. We determined the AOI in video frames by recording and analyzing the eye movements of 4 groups (Young Male-YM, Young Female-YF, Older Male-OM and Older Female-OF) of 5 subjects each while they watched 4-8 minute video segments of 6 movies. Within-group AOI (temporal and spatial coincidence) was marked on a frame if valid eye position data was available for at least 4 of the 5 subjects and the bivariate contour ellipse area (BVCA) (k=1, P=63.2%) was < 9 deg 2 (the screen was 27x15 deg). Within group, temporal coincidence occurred for a minimum of 58% of frames in all movies. Within-group 9 deg 2 AOI rates ranged from 39% (YF) to 72% (OM) of all video frames (chance coincidences ranged from 11% to 32%). Between age and gender groups, the AOIs were < 3 deg apart at least 65% of the time. This was not due solely to the AOIs being clustered around the center of the screen, since they were within 3 deg of the center of the screen an average of 54% of the time. For a ≤ 25 deg 2 BVCA, the AOIs were within 5 deg of the screen center at least 91% of the time, accounting for almost all of the time that the AOIs were within 5 deg of each other. Conclusion: The high level of AOI coincidence within and across age groups, even away from the screen's center, suggests that a single AOI might be appropriate for varied audiences in many applications. Analysis of the frames where group AOIs differ might be of interest in determining what type of visual or content categories account for difference between gender and age group AOIs.
IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2010
A bioptic telescope is a visual aid used by people with impaired vision when driving in many US s... more A bioptic telescope is a visual aid used by people with impaired vision when driving in many US states, though bioptic driving remains controversial. Objective data on how and when bioptic drivers use the telescope and what they look at with it are crucial to understanding the bioptic telescope's effect on driving. A video-based technique to track the telescope's aiming point is presented in this paper. With three infrared retro-reflective markers pasted on the bioptic spectacles' frame, its movement is recorded using an infrared camera unit with infrared LED illuminators. The angles formed by the three markers are used to calculate the telescope's aiming points, which are registered with road scene images recorded by another camera. The calculation is based on a novel one-time calibration method, in which the light spot from a head-mounted laser pointer projected on a wall while the head is scanning is recorded by the scene camera, in synchronization with the infrared camera. Interpolation is performed within small local regions where no samples were taken. Thus, non-linear interpolation error can be minimized, even for wide-range tracking. Experiments demonstrated that the average error over a 70°×48° field was only 0.86°, with lateral head movement allowed.
MultiMedia Modeling : MMM ... : proceedings. International Conference on Multi-Media Modeling
Local features are widely used in visual tracking to improve robustness in cases of partial occlu... more Local features are widely used in visual tracking to improve robustness in cases of partial occlusion, deformation and rotation. This paper proposes a local fragment-based object tracking algorithm. Unlike many existing fragment-based algorithms that allocate the weights to each fragment, this method firstly defines discrimination and uniqueness for local fragment, and builds an automatic pre-selection of useful fragments for tracking. Then, a Harris-SIFT filter is used to choose the current valid fragments, excluding occluded or highly deformed fragments. Based on those valid fragments, fragment-based color histogram provides a structured and effective description for the object. Finally, the object is tracked using a valid fragment template combining the displacement constraint and similarity of each valid fragment. The object template is updated by fusing feature similarity and valid fragments, which is scale-adaptive and robust to partial occlusion. The experimental results show...
Ophthalmic & physiological optics : the journal of the British College of Ophthalmic Opticians (Optometrists), Jan 10, 2015
There is a need for automated retinal optical coherence tomography (OCT) image analysis tools for... more There is a need for automated retinal optical coherence tomography (OCT) image analysis tools for quantitative measurements in small animals. Some image processing techniques for retinal layer analysis have been developed, but reports about how useful those techniques are in actual animal studies are rare. This paper presents the use of a retinal layer detection method we developed in an actual mouse study that involves wild type and mutated mice carrying photoreceptor degeneration. Spectral domain OCT scanning was performed by four experimenters over 12 months on 45 mouse eyes that were wild-type, deficient for ephrin-A2 and ephrin-A3, deficient for rhodopsin, or deficient for rhodopsin, ephrin-A2 and ephrin-A3. The thickness of photoreceptor complex between the outer plexiform layer and retinal pigment epithelium was measured on two sides of the optic disc as the biomarker of retinal degeneration. All the layer detection results were visually confirmed. Overall, 96% (8519 out of 9...
Journal of Medical Internet Research, May 12, 2021
Background: Millions of individuals with visual impairment use vision assistance apps to help wit... more Background: Millions of individuals with visual impairment use vision assistance apps to help with their daily activities. The most widely used vision assistance apps are magnifier apps. It is still largely unknown what the apps are used for. Lack of insight into the visual needs of individuals with visual impairment is a hurdle for the development of more effective assistive technologies. Objective: This study aimed to investigate how needs for visual aids may vary with social activities, by observing the changes in the usage of a smartphone magnifier app when many users take breaks from work. Methods: The number of launches of the SuperVision Magnifier app was determined retrospectively from 2018 to 2020 from among active users worldwide. The fluctuation in app usage was examined by comparing weekday vs weekend periods, Christmas and new year vs nonholiday seasons, and COVID-19 lockdowns vs the easing of restriction during the pandemic. Results: On average, the app was used 262,466 times by 38,237 users each month in 2020 worldwide. There were two major trough points on the timeline of weekly app usage, one aligned with the COVID-19 lockdowns in April 2020 and another aligned with the Christmas and new year week in 2018 and 2019. The app launches declined by 6947 (11% decline; P<.001) during the lockdown and by 5212 (9% decline; P=.001) during the holiday weeks. There was no significant decline during March to May 2019. App usage compensated for seasonal changes was 8.6% less during weekends than during weekdays (P<.001). Conclusions: The need for vision assistance technology was slightly lower during breaks and lockdowns, probably because the activities at home were different and less visually demanding. Nevertheless, for the entire user population, the needs for visual aids are still substantial.
Tracking head movement in outdoor activities is more challenging than in controlled indoor lab en... more Tracking head movement in outdoor activities is more challenging than in controlled indoor lab environments. Large-magnitude head scanning is common under natural conditions. Compensatory gaze (head and eye) scanning while walking may be critical for people with visual field loss. We compared the accuracy of two outdoor head tracking methods: differential inertial measurement units (IMU) and simultaneous localization and mapping (SLAM). At a fixed location experiment, a gaze aiming test showed that SLAM outperforms IMU in terms of error (IMU: 9.6°, SLAM: 4.47°). In an urban street walking experiment conducted with five patients with hemifield loss, the IMU drift, quantified by root-mean-square deviation, was as high as 68.1°, while the drift of SLAM was only 5.3°. However, the SLAM method suffered from data loss due to tracking failure (~10% overall, and ~ 18% when crossing streets). Our results show that the SLAM and IMU methods have complementary properties. Because of no data gaps, the differential IMU method may be desirable as compared to SLAM in settings where the signal drift can be removed in post-processing and small gaze estimation errors can be tolerated. Keywords Head scanning • IMU • Outdoor navigation • Visual SLAM People's mobility is affected by where they look, which in turn requires both head and eye movements. While saccade amplitude can be quite large, the distribution of saccade amplitudes in real-world circumstances are largely below 15° (Bahill, Adler, & Stark, 1975), making head tracking a necessity to understand real-world gaze (eye + head) (Bahill, Adler, & Stark, 1975; Einhäuser et al., 2007; Rothkopf & Pelz, 2004). To compensate for their visual field loss, visually impaired people while walking would presumably need to scan the environment more than the normally sighted. Numerous studies have analyzed the eye and head movements for people with normal vision and with vision loss in controlled laboratory setups (
Background Millions of individuals with visual impairment use vision assistance apps to help with... more Background Millions of individuals with visual impairment use vision assistance apps to help with their daily activities. The most widely used vision assistance apps are magnifier apps. It is still largely unknown what the apps are used for. Lack of insight into the visual needs of individuals with visual impairment is a hurdle for the development of more effective assistive technologies. Objective This study aimed to investigate how needs for visual aids may vary with social activities, by observing the changes in the usage of a smartphone magnifier app when many users take breaks from work. Methods The number of launches of the SuperVision Magnifier app was determined retrospectively from 2018 to 2020 from among active users worldwide. The fluctuation in app usage was examined by comparing weekday vs weekend periods, Christmas and new year vs nonholiday seasons, and COVID-19 lockdowns vs the easing of restriction during the pandemic. Results On average, the app was used 262,466 ti...
A normally sighted person can see a grating of 30 cycles per degree or higher, but spatial freque... more A normally sighted person can see a grating of 30 cycles per degree or higher, but spatial frequencies needed for motion perception are much lower than that. It is unknown for natural images with a wide spectrum how all the visible spatial frequencies contribute to motion speed perception. In this work, we studied the effect of spatial frequency content on motion speed estimation for sequences of natural and stochastic pixel images by simulating different visual conditions, including normal vision, low vision (low-pass filtering), and complementary vision (high-pass filtering at the same cutoff frequencies of the corresponding low-vision conditions) conditions. Speed was computed using a biological motion energy-based computational model. In natural sequences, there was no difference in speed estimation error between normal vision and low vision conditions, but it was significantly higher for complementary vision conditions (containing only high-frequency components) at higher speeds. In stochastic sequences that had a flat frequency distribution, the error in normal vision condition was significantly larger compared with low vision conditions at high speeds. On the contrary, such a detrimental effect on speed estimation accuracy was not found for low spatial frequencies. The simulation results were consistent with the motion direction detection task performed by human observers viewing stochastic sequences. Together, these results (i) reiterate the importance of low frequencies in motion perception, and (ii) indicate that high frequencies may be detrimental for speed estimation when low frequency content is weak or not present.
Translational Vision Science & Technology, 2020
The purpose of this study was to investigate the telescope use behaviors in natural daily driving... more The purpose of this study was to investigate the telescope use behaviors in natural daily driving of people with reduced visual acuity licensed to drive with a bioptic (a small spectacle-mounted telescope). Methods: A large dataset (477 hours) of naturalistic driving was collected from 19 bioptic drivers (visual acuity 20/60 to 20/160 without the telescope). To reduce the data volume, a multiloss 50-layer deep residual neural network (ResNet-50) was used to detect potential bioptic telescope use events. Then, a total of 120 hours of selected video clips were reviewed and annotated in detail. Results: The frequency of looking through their telescopes ranged from 4 to 308 times per hour (median: 27, interquartile range [IQR], 19-75), with each bioptic use lasting median 1.4 seconds (IQR, 1.2-1.8). Thus, participants spent only 1.6% (IQR, 0.7%-3.5%) driving time with their telescopes aiding their vision. Bioptic telescopes were used most often for checking the road ahead (84.8%), followed by looking at traffic lights (5.3%), and reading road signs (4.6%). Conclusions: In daily driving, the bioptic drivers mostly (>98% of driving time) drove under low visual acuity conditions. The bioptic telescope was mainly used for observing road and traffic conditions in the distance for situational awareness. Only a small portion of usage was for road sign reading. Translational Relevance: This study provides new insights into how the vision rehabilitation device-bioptic telescopes are used in daily driving. The findings may be helpful for designing bioptic driving training programs.
PURPOSEThere are many visually impaired people who can drive legally with bioptic telescope. Draw... more PURPOSEThere are many visually impaired people who can drive legally with bioptic telescope. Drawing on the experience of drivers with reduced vision, this study investigated the role of motion perception and visual acuity in driving, under simulated low visual acuity.METHODSTwenty normally sighted participants took part in a driving hazard perception (HP) test, in four different conditions: with/without motion interruption and with/without simulated low visual acuity. In interrupted motion conditions a mask frame was inserted between every frame of the driving videos. In simulated low visual acuity conditions, participants wore glasses with diffusing filters that lowered their visual acuity to 20/120 on average. Participants’ response time, hazard detection rates, and HP scores, which combined response time and detection rate, were compared.RESULTSRepeated measure ANOVA revealed that the HP scores significantly declined from 20.46 to 16.82 due to the motion mask (F(1,19) = 9.343, p...
Speed perception is an important task performed by our visual system in various daily life tasks.... more Speed perception is an important task performed by our visual system in various daily life tasks. In various psychophysical tests, relationship between spatial frequency, temporal frequency, and speed has been examined in human subjects. The role of vision impairment in speed perception has also been previously examined. In this work, we examine the inter-relationship between speed, spatial frequency, low vision conditions, and the type of input motion stimuli in motion perception accuracy. For this purpose, we propose a computational model for speed perception and evaluate it in custom generated natural and stochastic sequences by simulating low-vision conditions (low pass filtering at different cutoff frequencies) as well as complementary vision conditions (high pass versions at the same cutoff frequencies). Our results show that low frequency components are critical for accurate speed perception, whereas high frequencies do not play any important role in speed estimation. Since p...
This work proposes a hardware-friendly, dense optical flow-based Time-to-Collision (TTC) estimati... more This work proposes a hardware-friendly, dense optical flow-based Time-to-Collision (TTC) estimation algorithm intended to be deployed on smart video sensors for collision avoidance. The algorithm optimized for hardware first extracts biological visual motion features (motion energies), and then utilizes a Random Forests regressor to predict robust and dense optical flow. Finally, TTC is reliably estimated from the divergence of the optical flow field. This algorithm involves only feed-forward data flows with simple pixel-level operations, and hence has inherent parallelism for hardware acceleration. The algorithm offers good scalability, allowing for flexible tradeoffs among estimation accuracy, processing speed and hardware resource. Experimental evaluation shows that the accuracy of the optical flow estimation is improved due to the use of Random Forests compared to existing voting-based approaches. Furthermore, results show that estimated TTC values by the algorithm closely follo...
IEEE transactions on circuits and systems. II, Express briefs : a publication of the IEEE Circuits and Systems Society, 2018
This paper proposes a modified Eulerian Video Magnification (EVM) algorithm and a hardware implem... more This paper proposes a modified Eulerian Video Magnification (EVM) algorithm and a hardware implementation of a motion magnification core for smart image sensors. Compared to the original EVM algorithm, we perform the pixel-wise temporal bandpass filtering only once rather than multiple times on all scale layers, to reduce the memory and multiplier requirement for hardware implementation. A pixel stream processing architecture with pipelined blocks is proposed for the magnification core, enabling it to readily fit common image sensing components with streaming pixel output, while achieving higher performance with lower system cost. We implemented an FPGA-based prototype that is able to process up to 90M pixels per second and magnify subtle motion. The motion magnification results are comparable to the original algorithm running on PC.
Analyzing naturalistic driving behavior recorded with in-car cameras is an ecologically valid met... more Analyzing naturalistic driving behavior recorded with in-car cameras is an ecologically valid method for measuring driving errors, but it is time intensive and not easily applied on a large scale. This study validated a semi-automated, computerized method using archival naturalistic driving data collected for drivers with mild Alzheimer's disease (AD; = 44) and age-matched healthy controls (HC; = 16). The computerized method flagged driving situations where safety concerns are most likely to occur (i.e., rapid stops, lane deviations, turns, and intersections). These driving epochs were manually reviewed and rated for error type and severity, if present. Ratings were made with a standardized scoring system adapted from DriveCam. The top eight error types were applied as features to train a logistic model tree classifier to predict diagnostic group. The sensitivity and specificity were compared among the event-based method, on-road test, and composite ratings of two weeks of recor...
International journal of pattern recognition and artificial intelligence, 2018
Lane changes are important behaviors to study in driving research. Automated detection of lane-ch... more Lane changes are important behaviors to study in driving research. Automated detection of lane-change events is required to address the need for data reduction of a vast amount of naturalistic driving videos. This paper presents a method to deal with weak lane-marker patterns as small as a couple of pixels wide. The proposed method is novel in its approach to detecting lane-change events by accumulating lane-marker candidates over time. Since the proposed method tracks lane markers in temporal domain, it is robust to low resolution and many different kinds of interferences. The proposed technique was tested using 490 h of naturalistic driving videos collected from 63 drivers. The lane-change events in a 10-h video set were first manually coded and compared with the outcome of the automated method. The method's sensitivity was 94.8% and the data reduction rate was 93.6%. The automated procedure was further evaluated using the remaining 480-h driving videos. The data reduction rat...
Gaze-contingent displays have been widely used in vision research and virtual reality application... more Gaze-contingent displays have been widely used in vision research and virtual reality applications. Due to data transmission, image processing, and display preparation, the time delay between the eye tracker and the monitor update may lead to a misalignment between the eye position and the image manipulation during eye movements. We propose a method to reduce the misalignment using a Taylor series to predict the saccadic eye movement. The proposed method was evaluated using two large datasets including 219,335 human saccades (collected with an EyeLink 1000 system, 95% range from 1° to 32°) and 21,844 monkey saccades (collected with a scleral search coil, 95% range from 1° to 9°). When assuming a 10-ms time delay, the prediction of saccade movements using the proposed method could reduce the misalignment greater than the state-of-the-art methods. The average error was about 0.93° for human saccades and 0.26° for monkey saccades. Our results suggest that this proposed saccade predicti...
IEEE Transactions on Circuits and Systems for Video Technology, 2016
This paper proposes a bio-inspired visual motion estimation algorithm based on motion energy, alo... more This paper proposes a bio-inspired visual motion estimation algorithm based on motion energy, along with its compact very-large-scale integration (VLSI) architecture using low-cost embedded systems. The algorithm mimics motion perception functions of retina, V1, and MT neurons in a primate visual system. It involves operations of ternary edge extraction, spatiotemporal filtering, motion energy extraction, and velocity integration. Moreover, we propose the concept of confidence map to indicate the reliability of estimation results on each probing location. Our algorithm involves only additions and multiplications during runtime, which is suitable for lowcost hardware implementation. The proposed VLSI architecture employs multiple (frame, pixel, and operation) levels of pipeline and massively parallel processing arrays to boost the system performance. The array unit circuits are optimized to minimize hardware resource consumption. We have prototyped the proposed architecture on a low-cost field-programmable gate array platform (Zynq 7020) running at 53-MHz clock frequency. It achieved 30-frame/s real-time performance for velocity estimation on 160 × 120 probing locations. A comprehensive evaluation experiment showed that the estimated velocity by our prototype has relatively small errors (average endpoint error < 0.5 pixel and angular error < 10°) for most motion cases.
Investigative Opthalmology & Visual Science, 2015
PURPOSE. Although the impact of homonymous visual field defects (HFDs) on mobility has been inves... more PURPOSE. Although the impact of homonymous visual field defects (HFDs) on mobility has been investigated previously, the emphasis has been on obstacle detection. Relatively little is known about HFD patients' ability to judge collisions once an obstacle is detected. We investigated this using a walking simulator. METHODS. Patients with HFDs (n ¼ 29) and subjects with normal vision (NV; n ¼ 21) were seated in front of a large screen on which a visual simulation of walking was displayed. They made collision judgments for a human figure that appeared for 1 second at lateral offsets from the virtual walking path. A perceived-collision threshold was calculated for right and left sides. RESULTS. Symmetrical collision thresholds (same on left and right sides) were measured for participants with NV (n ¼ 21), and right (n ¼ 9) and left (n ¼ 7) HFD without hemispatial neglect. Participants with left neglect (n ¼ 10) showed significant asymmetry with thresholds smaller (compared to the NV group and other HFD groups) on the blind (P < 0.001) and larger on the seeing (P ¼ 0.05) sides. Despite the asymmetry, the overall width of the zone of perceived collision risk was not different, suggesting a relatively uniform rightward deviation in judgments of the left neglect group. CONCLUSIONS. Left neglect was associated with rightward asymmetry in collision judgments, which may cause collisions on the left side even when an obstacle is detected. These behaviors may represent the spatial misperceptions in body midline described previously in patients with left neglect.
Gaze-contingent display paradigms play an important role in vision research. The time delay due t... more Gaze-contingent display paradigms play an important role in vision research. The time delay due to data transmission from eye tracker to monitor may lead to a misalignment between the gaze direction and image manipulation during eye movements, and therefore compromise the contingency. We present a method to reduce this misalignment by using a compressed exponential function to model the trajectories of saccadic eye movements. Our algorithm was evaluated using experimental data from 1,212 saccades ranging from 38 to 308, which were collected with an EyeLink 1000 and a Dual-Purkinje Image (DPI) eye tracker. The model fits eye displacement with a high agreement (R 2. 0.96). When assuming a 10-millisecond time delay, prediction of 2D saccade trajectories using our model could reduce the misalignment by 30% to 60% with the EyeLink tracker and 20% to 40% with the DPI tracker for saccades larger than 88. Because a certain number of samples are required for model fitting, the prediction did not offer improvement for most small saccades and the early stages of large saccades. Evaluation was also performed for a simulated 100-Hz gaze-contingent display using the prerecorded saccade data. With prediction, the percentage of misalignment larger than 28 dropped from 45% to 20% for EyeLink and 42% to 26% for DPI data. These results suggest that the saccadeprediction algorithm may help create more accurate gaze-contingent displays.
Eye movements while watching video: comparisons across viewer groups Identifying the area of inte... more Eye movements while watching video: comparisons across viewer groups Identifying the area of interest (AOI) on a video frame may be necessary: (1) to develop a television magnifying aid for people with low vision; (2) to implement some data compression schemes; and (3) to transform images for rendering on devices with small display areas. We determined the AOI in video frames by recording and analyzing the eye movements of 4 groups (Young Male-YM, Young Female-YF, Older Male-OM and Older Female-OF) of 5 subjects each while they watched 4-8 minute video segments of 6 movies. Within-group AOI (temporal and spatial coincidence) was marked on a frame if valid eye position data was available for at least 4 of the 5 subjects and the bivariate contour ellipse area (BVCA) (k=1, P=63.2%) was < 9 deg 2 (the screen was 27x15 deg). Within group, temporal coincidence occurred for a minimum of 58% of frames in all movies. Within-group 9 deg 2 AOI rates ranged from 39% (YF) to 72% (OM) of all video frames (chance coincidences ranged from 11% to 32%). Between age and gender groups, the AOIs were < 3 deg apart at least 65% of the time. This was not due solely to the AOIs being clustered around the center of the screen, since they were within 3 deg of the center of the screen an average of 54% of the time. For a ≤ 25 deg 2 BVCA, the AOIs were within 5 deg of the screen center at least 91% of the time, accounting for almost all of the time that the AOIs were within 5 deg of each other. Conclusion: The high level of AOI coincidence within and across age groups, even away from the screen's center, suggests that a single AOI might be appropriate for varied audiences in many applications. Analysis of the frames where group AOIs differ might be of interest in determining what type of visual or content categories account for difference between gender and age group AOIs.
IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2010
A bioptic telescope is a visual aid used by people with impaired vision when driving in many US s... more A bioptic telescope is a visual aid used by people with impaired vision when driving in many US states, though bioptic driving remains controversial. Objective data on how and when bioptic drivers use the telescope and what they look at with it are crucial to understanding the bioptic telescope's effect on driving. A video-based technique to track the telescope's aiming point is presented in this paper. With three infrared retro-reflective markers pasted on the bioptic spectacles' frame, its movement is recorded using an infrared camera unit with infrared LED illuminators. The angles formed by the three markers are used to calculate the telescope's aiming points, which are registered with road scene images recorded by another camera. The calculation is based on a novel one-time calibration method, in which the light spot from a head-mounted laser pointer projected on a wall while the head is scanning is recorded by the scene camera, in synchronization with the infrared camera. Interpolation is performed within small local regions where no samples were taken. Thus, non-linear interpolation error can be minimized, even for wide-range tracking. Experiments demonstrated that the average error over a 70°×48° field was only 0.86°, with lateral head movement allowed.
MultiMedia Modeling : MMM ... : proceedings. International Conference on Multi-Media Modeling
Local features are widely used in visual tracking to improve robustness in cases of partial occlu... more Local features are widely used in visual tracking to improve robustness in cases of partial occlusion, deformation and rotation. This paper proposes a local fragment-based object tracking algorithm. Unlike many existing fragment-based algorithms that allocate the weights to each fragment, this method firstly defines discrimination and uniqueness for local fragment, and builds an automatic pre-selection of useful fragments for tracking. Then, a Harris-SIFT filter is used to choose the current valid fragments, excluding occluded or highly deformed fragments. Based on those valid fragments, fragment-based color histogram provides a structured and effective description for the object. Finally, the object is tracked using a valid fragment template combining the displacement constraint and similarity of each valid fragment. The object template is updated by fusing feature similarity and valid fragments, which is scale-adaptive and robust to partial occlusion. The experimental results show...
Ophthalmic & physiological optics : the journal of the British College of Ophthalmic Opticians (Optometrists), Jan 10, 2015
There is a need for automated retinal optical coherence tomography (OCT) image analysis tools for... more There is a need for automated retinal optical coherence tomography (OCT) image analysis tools for quantitative measurements in small animals. Some image processing techniques for retinal layer analysis have been developed, but reports about how useful those techniques are in actual animal studies are rare. This paper presents the use of a retinal layer detection method we developed in an actual mouse study that involves wild type and mutated mice carrying photoreceptor degeneration. Spectral domain OCT scanning was performed by four experimenters over 12 months on 45 mouse eyes that were wild-type, deficient for ephrin-A2 and ephrin-A3, deficient for rhodopsin, or deficient for rhodopsin, ephrin-A2 and ephrin-A3. The thickness of photoreceptor complex between the outer plexiform layer and retinal pigment epithelium was measured on two sides of the optic disc as the biomarker of retinal degeneration. All the layer detection results were visually confirmed. Overall, 96% (8519 out of 9...
Uploads
Papers by Gang Luo