US8433110

USOO8433 11 OB2
(12) United States Patent (10) Patent No.: US 8.433,110 B2

Kropp et al. (45) Date of Patent: Apr. 30, 2013
(54) PULSE-RATE DETECTION USINGA (56) References Cited
FNGERPRINT SENSOR
U.S. PATENT DOCUMENTS
(75) Inventors: Ronald A. Kropp, West Palm Beach, FL 5,978.495 A 1 1/1999 Thomopoulos et al.
(US); Richard Irving, Palm Beach 6.459,804 B2 * 10/2002 Mainguet ...................... 382,124
6,885.439 B2 * 4/2005 Fujieda ........................... 356/71
E. th & Raist. Schmitt,
alm Beach Gardens, FL (US)
2006/0204062 A1 9/2006 Shigeta
2007/0177772 A1* 8/2007 Fujii et al. ................ 382,115
2008/0273768 A1 11/2008 Dennis et al.
(73) Assignee: Sonavation, Inc., Palm Beach Gardens, 2009 OO69668 A1 3/2009 Stemmer
FL (US) 2009/0092.290 A1 4/2009 Rowe
OTHER PUBLICATIONS
(*) Notice: Subject to any disclaimer, the term of this Search Report and Written Opinion for International Application No.
patent is extended or adjusted under 35 PCT/US 10/60073 mailed Feb. 11, 2011, 17 pages.
U.S.C. 154(b) by 307 days.
* cited by examiner
(21) Appl. No.: 12/966,328 Primary Examiner — Stephen R Koziol
Assistant Examiner — Raphael Schwartz
(22) Filed: Dec. 13, 2010 (74) Attorney, Agent, or Firm — Kenneth J. Lukacher Law
Group
(65) Prior Publication Data (57) ABSTRACT
US 2011 FO170750 A1 Jul. 14, 2011 Provided is a fingerprint sensor configured for detecting a
pulse of a human or other mammal. A time series of finger
print images is analyzed to identify time-based variations in
Related U.S. Application Data spatial location and/or pressure from ridges of a finger, where
the variations result from time-based variations in the flow of
(60) Provisional application No. 61/285,793, filed on Dec. blood. Based on the analysis, a determination is made if the
11, 2009. time-based variations are indicative of a pulse in a digit of the
person, and if so, a determination is made as to the pulse rate
(51) Int. Cl. based on the variations. Imaging sensitivity of the digit at the
G06K 9/00 (2006.01) fingerprint sensor is optimized for determination of the pulse.
(52) U.S. Cl. The time series of fingerprint images is also analyzed to
USPC .......................................................... 382/124 determine if there is bulk motion of the digit, in which case
(58) Field of Classification Search ........................ None pulse rate analysis is temporarily Suspended.
See application file for complete search history. 30 Claims, 10 Drawing Sheets
106a
Fingerprint Ridges at low pulse (diastole)

U.S. Patent Apr. 30, 2013 Sheet 1 of 10 US 8,433,110 B2
110
y 100a
Fingerprint Ridges at low pulse (diastole)
FIG. 1A
Fingerprint Ridges at high pulse (systole)

FIG. 1B
U.S. Patent Apr. 30, 2013 Sheet 2 of 10
FIG. 1C
FIG. 1D
200a
220
225
2OOb
11 Ob
215
22O
D
225
--------------------
---------------------
FIG. 2B
0||||_^
G09
— Processing System 225
Processor 404
Main Memory 408
Display Interface 402 Display 215
Secondary
Memory 410
Hard Disk Drive
412
( ) Removable Removable
Storage Drive H + -- Storage Unit
Communication 414 418
Infrastructure
406
Removable
Interface 420 - Storage Unit
422
) Timer/Clock 436
K ) Platen interface 432 - - - Fingerprint
Paten 11 O
K ) Control Interface 434

Controls 220
Communications
Interface 424
500
Frame 501 Subtracted

from frame 504
O
E.
E. s
OOOOOOOOO
DDDDDDDD
O OOOOOOOOOO
"""
DOD DOOOOOOOO
OOO
E. 511
.
E.
O EOOOO
al EEO
O OOOOO
EE
FIG. 5B
Adjust sensor sensitivity and contrast for pulse detection. 610
Adjust frame rate for pulse detection. Y\ , 615
Begin (continue) obtaining time-series of images of digit

i.e., obtain time-slices of digit.
-Y-
Perform time-delayed background image subtraction on time 625
series of images to obtain series of Difference Frames.
Determine coupling stability between digit and platen. NO

630
635
Coupling stable enough for

pulse rate detection?
Process pressure/location fluctuations (Difference Frames) to Y\, 640

determine instantaneous Fluctuation intensity for each time slice
Calculate Pulse Rate based on series of Y\ 645

instantaneous Fuctuation intensities
650
Display, Store, and/or Transmit Pulse Rate (optionally filter
based on last N rates being within X% of each other)
Exemplary Method For Detecting

A Pulse With Fingerprint Sensor FIG. 6
O start D/V 705 640

ldentify Difference Frame associated with Current time slice -
EE 715
Copy pixel grayScale values to One-dimensional array
Sort array from highest to lowest grayScale values
725
Retain selected percentage of highest and lowest grayscale
values in array, disposing of remaining grayScale values, Creating
upper/lower grayScale values array
Calculate Standard deviation of the population of the array, i.e.,

calculate instantaneous Fluctuation intensity
Insert Instantaneous Fluctuation intensity for current time slice

into Pulse Rate Waveform Array
Identify next time slice
Exemplary Method To Process Pressure/location

Fluctuations To Determine Instantaneous
Fluctuation intensity For Each Time Slice
FIG. 7
645
Exemplary Method To Calculate Pulse Rate
805
810
Set StartEvent timestamp = current time
815
Identify/update peak Instantaneous Fluctuation intensity among the time slices
in the Pulse Rate Waveform Array
y 82O
Determine an End-Event Threshold value equal to a designated percentage of
the peak instantaneous Fluctuation intensity
(e.g., 20% of the peak Instantaneous Fluctuation Intensity)
825
Calculate a Current Slope of the pulse rate waveform based on the
Instantaneous Fluctuation intensity of the current time slice and a designated
number (e.g., 3) of the immediately preceding time slices in the Pulse Rate No
Waveform Array
S. 830
Current Slope > End-Event Threshold?
Yes 835
Set End Event timestamp = current time
840
Rising Edge Elapsed Seconds F EndEvent - StartEvent
845
Calculate Pulse:
Beats Per Minute = 60 * (1 / Rising Edge Elapsed Seconds)
850
NO
Beats Per Minute in Allowed Range?
Yes 855
Add Beats Per Minute to buffer of last N qualified pulse rates (e.g., N=5)
860
Set StartEvent = Current time
FIG. 8
US 8,433,110 B2
1. 2
PULSE-RATE DETECTION USINGA Sor, earlier and later images from the time series of images of
FNGERPRINT SENSOR the fingerprint are compared. Differences in the time series of
images are identified.
CROSS-REFERENCE TO RELATED The present invention determines whether the differences
APPLICATIONS in images are so large as to be the result of bulk motion of a
finger or other digit in relation to the biometric sensor, in
This application claims benefit to U.S. Provisional Appli which case the images are not used to obtain heart rate data. If,
cation No. 61/285,793 filed on Dec. 11, 2009, which is incor in bulk aspect, the digit is Substantially stationary relative to
porated by reference herein in its entirety. the biometric sensor, then Smaller, more Subtle changes in the
10
BACKGROUND OF THE INVENTION
fingerprint over time are detected. These changes in the fin
gerprint are induced by the pulse of blood through the digit,
1. Field of the Invention and may be either Small changes in the pressure of ridges from
The present invention generally relates to biometrics. the digit, or Small changes in the spatial location of the ridges
2. Background Art 15 relative to pixels or coordinate locations on the biometric
A variety of well known techniques exist for sensing, mea sensor, or both.
Suring, and identifying biometric characteristics. These tech The changes in the fingerprint are analyzed to determine a
niques focus on unique characteristics associated with struc substantially cyclical pattern which is substantially stable
tures that form the biometrics, and which can therefore over a short period of time (for example, over several sec
uniquely identify an individual. By way of one example, onds), and which falls within a typical range for a pulse. Such
fingerprints, defined by ridges and Valleys in a finger, are one a Substantially cyclical pattern of the change in the finger
such biometric. print, within an acceptable range and of acceptable stability, is
Similarly, a variety of techniques exist for sensing and identified as being a result of the pulse. The identified pulse
measuring physiological functions. For example, a pulse rate, may then be displayed to a user, stored in data storage, trans
defined as the number of beats-per-minute of the heart, is a 25 mitted, etc.
measurable physiological function. Additional features and advantages of the invention will be
Typically, the sensors used to measure biometric informa set forth in the description that follows, and in part will be
tion are different in structure and function from those used to apparent from the description, or may be learned by practice
measure physiological activities. For example, the optical of the invention. The advantages of the invention will be
imaging, capacitative, and piezoelectric devices typically 30 realized and attained by the structure and particularly pointed
used to detect fingerprints are quite different in structure, out in the written description and claims hereof, as well as the
design and operation from the plethysmographic or infrared appended drawings.
devices used to measure a pulse. It is to be understood that both the foregoing general
In Some instances, however, it may be convenient or useful description and the following detailed description are exem
to obtain both biometric information and physiological activ 35 plary and explanatory and are intended to provide further
ity data via a single sensor. For example, in a fingerprint explanation of the invention as claimed.
sensor, it may be useful both to forman image of a fingerprint,
and also to determine that the fingerprint is actually from a BRIEF DESCRIPTION OF THE
live finger. Therefore, combining both fingerprint detection DRAWINGS/FIGURES
and pulse detection in a single device may be of practical 40
benefit for security purposes. The accompanying drawings illustrate the present inven
In other cases. Such a combination may prove convenient. tion and, together with the description, further serve to
For example, a person carrying a personal digital Support explain the principles of the invention and to enable one
device, a medical device, or a communication device may skilled in the pertinent art to make and use the invention. In
wish to identify themselves to the device via biometric iden 45 the drawings, like reference numbers indicate identical or
tification. At other times, the same person may also wish to functionally similar elements. Additionally, the left most digit
use the device to monitor a physiological parameter Such as or digits of a reference number identifies the drawing in which
pulse, for example for medical monitoring purposes. the reference number first appears. Additionally, Some ele
What is needed, therefore, are systems and methods that ments may be identified or further identified by a reference
enable pulse detection in devices conventionally configured 50 number followed by a letter of the alphabet, for example
for fingerprint detection. “110a,” “110b.” In these cases the number by itself (e.g.,
“110) may identify a generic class or generic instance of an
BRIEF SUMMARY OF THE INVENTION element, while the letter of the alphabet (“a,” “b,' etc.) may
identify a more specific instance or embodiment of the ele
In one embodiment, the present invention includes a fin 55 ment. In some cases only specific instances or embodiments
gerprint sensor, which may be a standalone fingerprint sensor are illustrated (e.g., “102a,” “102b,” “102c,” etc.), it being
or a fingerprint sensor coupled with or integrated into another understood that the number when used by itself (e.g., "102)
technology (for example, a cell phone or a computer). The may refer to either a single or multiple specific instances,
fingerprint sensor includes a biometric sensing element for depending on the context.
the fingerprint detection, and can include, or be coupled to, a 60 FIG. 1A is an illustration of an exemplary fingerprint dur
processor. The system of fingerprint sensor and processor ing a period of diastolic pressure (low pressure).
may also contain, or be coupled to, a memory configured to FIG. 1B is an illustration of the exemplary fingerprint of
store image data and/or to store instructions for processing FIG. 1A during a period of systolic pressure (high pressure).
image data. FIG. 1C is an illustration of another exemplary fingerprint
The biometric sensing element is configured to obtain, and 65 during a period of diastolic pressure (low pressure).
to send to the processor and/or memory, a time series of FIG. 1D is an illustration of the exemplary fingerprint of
images of at least a portion of a fingerprint. Using the proces FIG. 1C during a period of systolic pressure (high pressure).
US 8,433,110 B2
3 4
FIG. 2A is an illustration of an exemplary fingerprint sen FIGS. 1C and 1D illustrate a complete fingerprint 106 of a
SO. digit of a mammal (illustrated in two views as fingerprints
FIG. 2B is an illustration of another exemplary fingerprint 106a and 106b).
SSO. FIG. 1A is distinguished from FIG. 1B in that FIG. 1A
FIG.3 is an illustration of an exemplary platen of an exem represents the fingerprint tissue during an interval of diastolic
plary fingerprint sensor. pressure (low blood pressure) when the heart is at rest during
FIG. 4 is a system diagram of an exemplary processing a cardiac cycle. FIG. 1B represents the fingerprint tissue
system which is part of an exemplary fingerprint sensor. during an interval of systolic pressure (high blood pressure)
FIG. 5A illustrates an exemplary time series of images of a when the heart is pumping blood at maximum pressure during
fingerprint or a portion of a fingerprint as captured by an 10 a cardiac cycle. Similarly, FIGS. 1C and 1D are distinguished
exemplary platen of an exemplary fingerprint sensor. by representing the fingerprint during diastolic pressure and
systolic pressure, respectively.
FIG. 5B illustrates subtracting a first, earlier fingerprint The present invention is based on variations in the finger
image frame from a second, later fingerprint image frame to print between the diastolic and systolic points of the cardiac
create a difference frame. 15 cycle. During the diastolic interval or low blood pressure
FIG. 6 is a flowchart of an exemplary method of detecting interval, the ridges 102 of fingerprint 106 will be closer
a pulse with a fingerprint sensor. together, and fingerprint 106 will exert less pressure on platen
FIG. 7 is a detailed flowchart of one of the steps from the 110. During an interval of systolic pressure or high blood
exemplary method of the flowchart of FIG. 6, specifically, a pressure, the ridges 102 offingerprint 106 will expand further
flowchart of an exemplary method to process fluctuations in a from each other and ridges 102 will place higher pressure and
fingerprint to determine an instantaneous fluctuation inten be broader in their signature against platen 110.
sity. FIG. 1A compared to FIG. 1B illustrates the difference in
FIG. 8 is a detailed flowchart of one of the steps from the the fingerprint structure 100 between the diastolic and sys
exemplary method of the flowchart of FIG. 6, specifically a tolic intervals of a cardiac cycle. Similarly, FIGS. 1C and 1D
flowchart of an exemplary method to calculate a pulse rate 25 illustrates the change in the fingerprint 106 between the dias
based on a time series of instantaneous fluctuation intensities. tolic and systolic intervals of the cardiac cycle. In each case,
the area of contact between the ridges 102 and platen 110 may
DETAILED DESCRIPTION OF THE INVENTION be increased during the systole as compared to the diastole,
and the fingerprint structure 100 or fingerprint as a whole 106
The following detailed description of the present invention 30 may be distended during systole as compared with diastole.
refers to the accompanying drawings that illustrate exemplary It should be noted that for purposes of illustration, the
embodiments consistent with this invention. Other embodi differences in fingerprint structure 100 between the diastolic
ments are possible, and modifications may be made to the interval and the systolic interval have been artificially exag
embodiments within the spirit and scope of the invention. gerated in FIG. 1A as compared to FIG. 1B. Similarly, the
Therefore, the following detailed description is not meant to 35 differences in fingerprint 106 have been greatly exaggerated
limit the invention. Rather, the scope of the invention is between FIG. 1C and FIG. 1D. Persons skilled in the relevant
defined by the appended claims. art will recognize that the system and method disclosed
It will be apparent to one skilled in the art that the present herein will be capable of distinguishing much Smaller, more
invention, as described below, may be implemented in many subtle variations in the fingerprint between the diastolic and
different embodiments. Any actual software code implement 40 systolic intervals, which may be detected by the fingerprint
ing the present invention is not limiting of the present inven SSO.
tion. Thus, the operational behavior of the present invention The ridges 102 and valleys 104 of fingerprint 106 can be
will be described with the understanding that modifications sensed, measured, and identified based upon a number of
and variations of the embodiments are possible, given the different modalities. These modalities sense differences
level of detail presented herein. 45 between the fingerprint ridges and Valleys with respect to
Exemplary Fingerprints and Fingerprint Sensors dielectric permittivity, thermal conductivity, acoustic imped
As known to those of skill in the art, fingerprints are defined ance, optical index, or other sensing modalities.
by unique structures on the Surface of the finger called ridges For example, Some fingerprint measurement modalities
and Valleys. rely on density values associated with the ridges and Valleys.
FIGS. 1A and 1B are illustrations of the basic structure 100 50 Acoustic impediography, employing for example piezoelec
of a fingerprint (illustrated in two views as 100a and 100b). In tric sensing elements, is one means to distinguish the density
FIGS. 1A and 1B, the basic fingerprint structure 100 includes variations between ridges and Valleys.
ridges 102 and valleys 104 (illustrated as ridges 102a, 102b Other modalities rely on dielectric permittivity as mea
and valleys 104a, 104b, respectively), which combine to form Sured when an electric current is passed through the ridges
an entire fingerprint (elements 106a, 106b of FIGS. 1C and 55 and valleys. With respect to dielectric permittivity for
1D, discussed below). Ridges 102 are fingerprint tissue, while example, the permittivity of a ridge (i.e., tissue), is different
Valleys 104 are regions or empty space (or, typically air) from the permittivity of a valley (i.e., air between the ridges).
between the ridges. Capacitive sensing is one technique that can be used to detect
As will be discussed further below, in one embodiment the changes in permittivity. With capacitive sensing, capacitance
invention is directed towards a system and method for detect 60 values generated when a sensor plate (electrode) touches a
ing the pulse of a mammal. Such as a person, using a sensor ridge are different than those generated when the sensor is
which is conventionally configured as a fingerprint sensor. exposed to a valley.
In FIGS. 1A and 1B, fingerprint structure 100 is shown Yet another modality is thermal conductance which is a
pressed against a platen 110 (i.e. a sensing Surface of a fin measure of the temperature differences between the ridges
gerprint sensor). Exemplary fingerprint sensors, including 65 and valleys. Optics are yet another modality. Optical tech
platen 110, are described in further detail below in conjunc niques rely on an optical index of refractive and reflective
tions with FIGS. 2A, 2B, 3, and 4. changes between the ridges and the valleys.
US 8,433,110 B2
5 6
Although the modalities differ, each approach seeks to used by fingerprint sensor 200a. For example, control panel
accurately distinguish ridges from Valleys in order to image 220 can include controls to initiate a scan of a fingerprint, and
the fingerprint or otherwise capture an impression of the can include controls to initiate a pulse detection using finger
fingerprint. An "impression of a fingerprint’ is any data which print scanner 200a.
may be captured by platen 110, and which is descriptive of or 5 Fingerprint sensor 200a also includes a processing system
otherwise captures or is indicative of the structure of the 225, shown in FIG. 2A with a dotted outline. The dotted line
fingerprint, which may include the shape of the fingerprint (as indicates that processing system 225 is typically internal to
defined by the ridges and valleys), the pressure exerted by the fingerprint sensor 200a. Processing system 225 includes a
fingerprint, and/or other structural aspects of the fingerprint. variety of elements necessary to process fingerprint Scanning
Such data may include, for example and without limitation, 10 information, and also to detect and process a cardiac pulse.
fingerprint image data, fingerprint pressure data, dielectric The details of processing system 225 are discussed below in
permittivity data associated with the fingerprint, capacitive greater detail in conjunction with FIG. 4. As will be discussed
data associated with the fingerprint, thermal conductance in detail below, processing system 225 is configured to detect
data associated with the fingerprint, and/or optical index data a variation over time of fingerprint 106, or in a portion of a
associated with the fingerprint. 15 fingerprint 106, returned by platen 110. In particular, and as
FIG. 2A illustrates an exemplary fingerprint sensor 200a. It will also be discussed in further detail below, processing
should be noted that exemplary fingerprint sensor 200a may system 225 is configured to detect the variation over time in
be a standalone fingerprint sensor, or can be combined with or fingerprint 106 by comparing images or other fingerprint
integrated into another electronic device. For example, fin impressions from a time series of images, or time series of
gerprint sensor 200a can be part of a portable communica impressions, returned by platen 110.
tions device such as a cellphone or personal digital assistant. FIG. 2B is a view of another exemplary fingerprint sensor
Fingerprint sensor 200a can also be part of a personal com 200b. Most of the elements of exemplary fingerprint sensor
puter or laptop computer. In these configurations, elements of 200b are the same as elements of exemplary fingerprint sen
fingerprint sensor 200a can serve additional purposes other sor 200a discussed above, and the details of these elements
than those specifically indicated herein. For example, a dis 25 will not be repeated here. One distinguishing feature between
play element 215, discussed further below, can provide dis fingerprint sensor 200b and 200a is a sensor platen 110b.
play functions not only for fingerprint sensor 200a, but for the Sensor platen 110b is configured to be both tall enough and
cell phone, personal digital assistant, laptop computer, or wide enough to receive an entire fingerprint. Therefore, in
other similar device. Fingerprint sensor 200a can be inte conjunction with sensor 200b and platen 110b, a user would
grated into these other devices, in part or in whole because 30 not be required to move or Swipe their finger across platen
fingerprint sensor 200a can serve to provide biometric iden 110b.
tification and security for the device. It should be noted that exemplary fingerprint sensors 200a
Fingerprint sensor 200a includes platen or sensorpad 110a and 200b may include other elements not specifically illus
(discussed above in conjunction with FIG. 1), a display 215, trated or discussed. Such as for example power elements,
a control panel 220, and a processing system 225. 35 various ports for communications with other technologies
Platen 110a can also be referred to as a sensorpad, a sensor (and which may serve as additional control elements), and
plate, or as a biometric sensing element. Platen 110a is con other elements as well.
figured to receive a digit of a person or other mammal, such as FIG.3 illustrates a detailed view of exemplary platen 110,
a finger of a person. Platen 110a is configured with sensing which may correspond for example to platen 110 discussed
elements, discussed further below, which are capable of 40 above in conjunction with FIGS. 1A and 1B, platens 110a or
detecting distinguishing elements of fingerprint 106. In an 110b discussed above in conjunction with FIGS. 2A and 2B,
embodiment, platen 110a is configured to detect ridges 102 or to similar platens configured for fingerprint sensing. As
and valleys 104 of fingerprint 106. The platen 110a is suffi discussed above in conjunction with FIGS. 2A and 2B, platen
ciently wide to receive a full width offingerprint 106, but can 110 is the fingerprint sensing element of exemplary finger
be narrow enough in height to only receive a portion of a 45 print sensors 200a, 200b. Platen 110 is configured to receive
vertical section offingerprint 106. Therefore, fingerprintsen a digit of a mammal, such as a person, in order to detect the
sor 110a can require that to detect fingerprint 106, a user must fingerprint of the person. Additionally, platen 110 is config
swipe or scroll their finger across platen 110a. ured to detect a pulse associated with the received digit based
In an embodiment illustrated in FIG. 1, platen 110a is upon variations in the fingerprint.
configured to return an image offingerprint 106 consisting of 50 As discussed above, platen 110 may employ a variety of
pixel elements as discussed further below in conjunction with technologies in order to return data descriptive offingerprint
FIG. 3. Platen 110a is also configured to return a time series 106. Platen 110 returns data descriptive offingerprint 106 by
of images of fingerprint 106. In alternative embodiments, detecting the ridges 102 and valleys 104 of fingerprint 106.
platen 110a can be configured to return data other than image These detection technologies can, for example, be capacitive,
data which is descriptive of the structure of fingerprint 106. 55 dielectric, optical, thermal, or acoustic. Other technologies
Such data may be in alternative to, or in addition to, image can be employed as well. No matter which sensing technol
data offingerprint 106. As noted above, the data captured by ogy is employed, platen 110 will typically be divided into
platen 110a and which is descriptive of the structure 100 of discrete pixels 305.
fingerprint 106 is generically referred to as an impression of Pixels 305 can constitute either discrete physical sensing
fingerprint 106. 60 elements or logical image elements extracted from a finger
Sensor 200a includes display element 215. Display ele print image or other fingerprint impression. In FIG. 3, each
ment 215 can be used to display a variety of information pixel is represented by a small square, while circular black
pertinent to the use of fingerprint sensor 200a. For example, dots provide a representative illustration of a large plurality of
display 215 can display instructions for the use offingerprint pixels (larger than the number of black dots shown) which
sensor 200a. Display 215 can also display a result of a fin 65 would too numerous to fit in the space of the illustration.
gerprint detection, for example, a biometric determination of By way of example, platen 110 can be divided into eight
a user. Control panel 220 can include a variety of controls rows 315 of pixels, and 192 columns 310 of pixels 305. In
US 8,433,110 B2
7 8
alternative embodiments, different numbers of rows 315 and Processing system 225 can include a platen interface 432
columns 310 can be employed. that receives biometric data and possibly other data from
Each pixel 305 is configured to return an image density fingerprint platen 110, and that may also be configured to
which is indicative of a presence of a ridge 102 or valley 104 send configuration or control data to fingerprint platen 110.
of a fingerprint 106. A pixel 305 fully in contact with a ridge Processing system 225 can include a control interface 434
102 will return a fully darkened color or least magnitude that receives control data and control instructions from fin
signal, while a pixel 305 in contact with a valley 104 may gerprint sensor controls 220, and which may also send data to
return fully light, fully white, maximum intensity signal or controls 220 (for example, state controls, status display sig
image intensity. In an alternative embodiment, a pixel 305 in nals, etc.).
contact with a ridge 102, will return a maximum image inten 10 Processing system 225 can include timer/clock element
sity while a pixel 305 in contact with a valley 104 may return 436. Timer/clock element 436 may be configured to maintain
a minimum image intensity or signal intensity. a current system date and time, and may also be configured to
A pixel 305 partly in contact with a ridge, and partly in determine or report the time interval between two or more
contact with or in proximity to a valley, will return an inter designated events. Timer/clock element 436 may also be con
mediate value image intensity or signal. Similarly, a pixel 305 15 figured for related timing determinations, such as determina
which is in contact with a ridge, but where the ridge exerts a tion of frequencies of designated events. In an embodiment,
less than full pressure on platen 110 will also return an inter timer/clock 436 may be a separate element from processor
mediate image intensity or signal intensity. 404. In an alternative embodiment, timer/clock may be inte
A pixel 305 is typically configured to return a signal or grated into processor 404. In an alternative embodiment,
image intensity in a predetermined range. For example, a timer/clock 436 may be an external element which delivers a
pixel 305 can be configured to return an 8-bit value resulting clock signal and timing signals or timing data to processing
in image intensities ranging from Zero to 255 or one to 256. system 225, for example via communications interface 424 or
The pattern of image intensities or signal intensities on pixels another interface. In an alternative embodiment, timer/clock
305 is used by processing system 225 to identify a fingerprint 436 may be part of processing system 225, but may be supple
ofa mammal. In addition, and as discussed further below with 25 mented by external synchronization or timing data.
respect to FIGS. 5 through 8, the pattern of pixels 305 on Processing system 225 also includes a main memory 408,
platen 110 can also be used by processing system 225 to preferably random access memory (RAM), and may also
detect the pulse of the mammal. include a secondary memory 410. The secondary memory
Exemplary Processing System of a Fingerprint Sensor 410 may include, for example, a hard disk drive 412 and/or a
In one embodiment, the invention is directed toward one or 30 removable storage drive 414, representing a floppy disk drive,
more processing systems capable of carrying out the func a magnetic tape drive, an optical disk drive (CD/DVD/Blu
tionality described herein. FIG. 4 illustrates an exemplary eRay drive), USB slot, etc. The removable storage drive 414
processing system 225 which can be part of a fingerprint reads from and/or writes to a removable storage unit 418 in a
sensor 200. Processing system 225 can be configured, well known manner. Removable storage unit 418 represents a
through Suitable instructions stored in memory or hard coded 35 floppy disk, magnetic tape, optical disk (CD/DVD/BlueRay,
into firmware, to detect a fingerprint of a mammal. Processing etc.), flash drive, etc. which is read by and written to by
system 225 may also be configured, through Suitable instruc removable storage drive 414. As will be appreciated, the
tions stored in memory or hard coded into firmware, and as removable storage unit 418 includes a computer usable Stor
described in further detail below, to detect a pulse of a mam age medium having stored therein computer software and/or
maland determine the pulse rate of the mammal. If fingerprint 40 data.
sensor 200 is combined with or integrated into another tech In alternative embodiments, secondary memory 410 may
nology, for example a cellphone, a personal digital assistant, include other similar devices for allowing computer programs
or a computer, processing system 225 may be configured to or other instructions to be loaded into processing system 225.
perform additional processing tasks as well. Such devices may include, for example, a removable storage
Processing system 225 includes one or more processors, 45 unit 422 and an interface 420. Examples of such may include
which may be a microprocessor, Such as processor 404. In an a program cartridge and cartridge interface (such as that
embodiment of the present system and method, processor 404 found in video game devices), a removable memory chip
is configured to detect a variation overtime offingerprint 106, (such as an erasable programmable read only memory
or in a portion of a fingerprint 106, returned by platen 110. In (EPROM), or programmable read only memory (PROM))
one embodiment of the present system and method, processor 50 and associated Socket, and other removable storage units 422
404 is configured to detect the variation over time in finger and interfaces 420, which allow software and data to be
print 106 by comparing images or other fingerprint-descrip transferred from the removable storage unit 422 to processing
tive data from a time series of images, or time series of system 225.
descriptive data, returned by platen 110. Processing system 225 may also include a communications
Processor 404 is connected to a communication infrastruc 55 interface 424. Communications interface 424 allows software
ture 406 (e.g., a communications bus, cross over bar, or net and data to be transferred between processing system 225 and
work). The algorithms described herein for detecting a pulse external devices. Examples of communications interface 424
are typically implemented in various Software embodiments, may include a modem, a network interface (such as an Eth
which may be executed by exemplary processing system 225 ernet card), a communications port, a Personal Computer
using processor 404. After reading this description, it will 60 Memory Card International Association (PCMCIA) slot and
become apparent to a person skilled in the relevant art(s) how card, etc. Software and data transferred via communications
to implement the invention using other processing systems interface 424 are in the form of signals 428 which may be
and/or architectures. electronic, electromagnetic, optical or other signals capable
Processing system 225 can include a display interface 402 of being received by communications interface 424. These
that forwards graphics, text, and other data from the commu 65 signals 428 are provided to communications interface 424 via
nication infrastructure 406 (or from a frame buffer not shown) a communications path (e.g., channel) not shown. This chan
for display on the display unit 215. nel carries signals 428 and may be implemented using wire or
US 8,433,110 B2
10
cable, fiber optics, a telephone line, a cellular link, an radio However, in alternative embodiments higher frame rates
frequency (RF) link and other communications channels. can be used. Persons skilled in the relevant arts will recognize
In this document, the terms “computer readable physical that the system and method described below may be
program medium.” “computer usable physical storage employed with higher frame rates, provided Suitable adjust
medium.” “computer readable physical storage medium, and 5 ments are made to the frame-subtraction process and pulse
similar terms are used to generally refer to media Such as a rate-waveform-array (both described in further detail below).
hard disk installed in hard disk drive 412, removable storage For example, for non-compact, non-portable systems, a frame
drive 414, interface 420, and the physical removable storage rate of 100 slices per second might be preferred.
media 418, 422 such as CDs, DVDs, tape drives, flash Embodiments described immediately below are based on
memory, program cartridge, PCMCIA card, and similar 10
an exemplary frame rate of 20 slices per second, it being
physical media which may be inserted or installed into stor understood that the system and method may be suitably
age drive 414 or connected or coupled to interface 420. These adapted for higher frame rates.
physical computer program products provide Software to pro
cessing system 225. The invention is directed in part to Such Series of images 500 is a series over eleven consecutive
computer program products. 15 time slices of a fingerprint, numbered 501 through 511, with
Computer programs (also referred to as computer control some time slices (506 through 510) deliberately omitted. It
logic) are stored in main memory 408 and/or secondary can be seen in successive images, starting from 501 to 502,
memory 410. Computer programs may also be received via 502 to 503, and so forth, that as a result of variations in the
communications interface 424. Such computer programs, blood pressure in a person’s finger, the image of the finger
when executed, enable the processing system 225 to perform print varies from frame to frame. Image areas corresponding
the features of the present invention, as discussed herein. In to ridges and valleys have been labeled as 592 (ridges) and
particular, the computer programs, when executed, enable the 594 (valleys). (It will be appreciated that reference numbers
processor 404 to perform the features of the present invention 592 and 594 therefore do not refer to time slices.) It will be
directed towards detecting a pulse of a mammal and related further appreciated that slices 501 is representative of a fin
functions. Accordingly, such computer programs represent 25 gerprint image at or near diastole, and slice 505 is represen
controllers of the processing system 225. tative of a fingerprint image at or near systole. Slice 511 is
In an embodiment where the invention is implemented representative of a fingerprint image during a period of pres
using software, the Software may be stored in a computer Sure transition marking a return from systole to diastole.
program product and loaded into processing system 225 Location variations and/or intensity variations are cyclical
using removable storage drive 414, hard drive 412, interface 30 in nature, with the cycle corresponding to the pulse rate. So,
420, removable storage units 418, 422, or communications for example, in series 500, it can be seen that slice 511
interface 424. The control logic (software), when executed by corresponds to a period of return to diastole following systole.
the processor 404, causes the processor 404 to perform the It will be understood that time series 500 is representative
functions of the invention to detect a pulse of a mammal and only, and may not conform in all respects to an actual image
related functions as described herein. 35 of a fingerprint or section of a fingerprint. For example, the
In another embodiment, the invention is implemented pri number of ridges shown and the spacing between ridges is
marily inhardware using, for example, hardware components intended for purposes of illustration only, and may vary from
Such as application specific integrated circuits (ASICs). that of a real fingerprint. Similarly, in the time series of
Implementation of the hardware state machine so as to per images 500 shown in FIG. 5, the variation in ridges 102/592
form the functions described herein will be apparent to per 40 and valleys 104/594 overtime, due to blood pressure changes,
sons skilled in the relevant art(s). has been exaggerated for purposes of illustration.
In yet another embodiment, the invention is implemented It may be seen that from slice to slice, the fingerprint image
using a combination of both hardware and Software. can vary both in the location of a ridge 102 (as shown by the
Exemplary Methods for Detecting a Pulse location of ridge images 512), or valley 104 (as shown by the
FIG.5A is an exemplary series of images 500 from a sensor 45 location of valley images 514). The image can also vary in the
platen 110 offingerprint sensor 200. The series of images 500 intensity of a pixel 305 under a ridge 102. Both location
is a time series of images of a fingerprint 106. In an embodi variations and intensity variations are induced by the changes
ment, the images are taken at a frame rate of 20 slices per in the blood pressure as a result of the pulse. In an embodi
second or 20 frames per second, that is, once every 50 milli ment of the present invention, and as discussed in further
seconds. 50 detail below, the first four images are received into a frame
Through experimentation, a frame rate of approximately buffer which may for example be stored in memory 408.
20 slices per second, or 20 frames per second, has been found By way of example, beginning with a fourth image (e.g.,
to be an effective rate for detecting pulse-induced variations image 504), the third preceding image in the buffer (i.e., three
in the fingerprint image. At the same time, a frame rate of 20 frames back) is subtracted from the present image. In this
frames per second is sufficient to enable detection of gross 55 way, variations in the fingerprint image over time are
movement or bulk movement of a person’s finger, which detected. For example, from frame 504, frame 501 would be
interferes with the pulse detection process, as discussed fur subtracted. From frame 505, frame 502 would be subtracted.
ther below. And from frame 506, frame 503 would be subtracted.
It is noted here that a frame rate of 20 slices per second is, In alternative embodiments, the number of images received
approximately, the lowest or slowest frame acquisition rate 60 into the frame buffer, and the interval between the first frame
that will produce reliable results for detecting a pulse. An and second frame (where the first frame is subtracted from the
advantage of embodiments employing this lowest frame rate second, later frame) can vary, for example, five frames, six
of approximately 20 slices per second (or equivalently, frames, etc. In alternative embodiments with higher frame
twenty frames per second) is that it requires less processing rates (for example, 100 frames per second), the interval
and few frame buffers than higher frame rates. Especially for 65 between the first frame and the second frame may be greater
portable devices, where power and memory may be limited, still, for example, 10 frames or 20 frames, or some other
the conservation of power and memory may be advantageous. number as may be determined through testing.
US 8,433,110 B2
11 12
As shown in FIG. 5B, subtraction of a first frame from a Put another way, the sensor is adjusted to have a partially
second frame entails subtracting, on a basis of corresponding saturated image so that valleys 104 appear white and ridges
pixels, the pixel value of a first frame from the pixel value of 102 appear black.
a second frame. Frame 5XX refers to an image from a generic In step 615, the frame rate is adjusted for pulse detection.
time slice (at some frame count XX), and frame 5XX-N Typically, the sensor 200 is capable of a frame rate greater
refers to an image from a time slice N frames earlier. For than 20 frames per second. As indicated above, in one exem
example, 5XX-3 refers to an image slice or image frame 3 plary embodiment a frame rate of 20 slices per second is
frames earlier than frame 5XX. Sufficient to detect a pulse. In alternative embodiments, a
Regarding image subtraction, for two different frames or framerate greater than 20 slices per second may be employed.
image slices (for example, frame 5XX and frame 5XX-3), 10 Theoretically, the only upper limit to the sensor frame rate is
and for a selected pixel (for example, the upper-left-most imposed by (i) The memory required for a large array of frame
pixel), the pixel value of frame 5XX-3 is subtracted from the buffers required to create a high pass filter for every sensor
pixel, with a recovery time of approximately 0.2 seconds, and
pixel value of frame 5XX. The resulting difference is stored in (ii) the processing power required to calculate the instanta
a difference frame 551 in memory. This process would be 15 neous fluctuation intensities, for each frame, at a given frame
repeated for each pixel, subtracting the pixel value of frame rate.
5XX-3 (the earlierframe) from the pixel value of frame 5XX. The embodiment described here is based on a framerate of
A complete difference frame 551 represents the difference 20 frames per second, but this is exemplary only, and higher
between a first frame 5XX-3 and a second, later frame, 5XX. frame rates may be employed. In an embodiment, if sensor
Only one difference frame is illustrated in FIG. 5B. How noise becomes an issue at the 20 frame per second frame rate,
ever, and as discussed in further detail below, embodiments of then the framerate can be doubled and frame averaging can be
the present invention entail determining a time series of Such used to reduce the noise as an element of the signal process
difference frames 551, and then comparing the difference 1ng.
frames to determine a pulse rate. Each difference frame in the Step 620 entails obtaining a time series of images of a digit,
time series is determined based on a difference between a 25 that is, obtaining a time series of slices of a fingerprint. The
current frame, represented symbolically as 5XX, and a frame process either begins at Step 20 or, if the process is continuing,
several frames earlier, for example frame 5XX-3. it continues at step 620. An exemplary time series of finger
As discussed above, processing system 225, and processor print images 500 is discussed above in conjunction with
404 in particular, are configured to detect a variation overtime FIGS 5A and 5B.
offingerprint 106, or in a portion of a fingerprint 106, returned 30 In step 625, time delayed background image subtraction is
by platen 110. done on the time series of images. As indicated above (see
In particular, processing system 225 is configured to detect discussion of FIGS. 5A and 5B), this time delayed image
the variation over time in fingerprint 106 by comparing fin Subtraction is typically performed by Subtracting from each
gerprint-descriptive data from a time series of fingerprint frame the frame which preceded it by three frames earlier.
descriptive data returned by platen 110. By comparing
35 Thus, the fourth frame would have subtracted from it the first
frame, the fifth frame would have subtracted the second
descriptive data from discrete time slices of the time series, frame, the sixth frame would have subtracted from it the third
processing system 225 generates a series of data values which frame. The resulting series of images, which are a series of
are descriptive of fluctuations in fingerprint 106 over time. difference frames 551, are stored in a difference frame buffer,
This series of data values, which represent fluctuations in 40 for example in memory 408, for further processing.
fingerprint 106 over time, are further analyzed by processing An optional step, not shown in the figures, entails a nor
system 225 to determine the pulse of a person or other mam malization of the difference. For example, if a pixel has a
mal associated with fingerprint 106. same value in a later time slice than an earlier time slice,
In an exemplary embodiment of the present system and rather than the difference pixel value being Zero, the differ
method, described in detail in FIGS. 6, 7, and 8, the finger 45 ence value for that pixel may be normalized to an intermedi
print descriptive data obtained from platen 110 is image data ate value (for example, 140 on a scale from 0 to 255). With a
5XX of fingerprint 106. By comparing successive images 0-normalization to 140, a positive difference (that is, a posi
5XX, a series of values are generated which are descriptive of tive result of subtracting the earlier pixel from the later pixel)
differences between the images 5XX. The values are referred will result in a difference pixel value in a range from 141 to
to as “instantaneous fluctuation intensities.” Further analysis 50 255. Similarly, with a 0-normalization to 140, a negative
of the series of instantaneous fluctuation intensities yields the difference (that is, a negative result of subtracting the earlier
pulse. pixel from the later pixel) will result in a difference pixel
FIG. 6 is a flowchart of an exemplary method 600 for value in a range from 0 to 139. Persons skilled in the art will
detecting a pulse with a fingerprint sensor. The method 600 recognize that other 0-normalization values (e.g., other than
begins at step 605. 55 140) may be employed as well.
At step 610, the sensor is adjusted for image sensitivity and It should be noted that if there were no change at all in the
contrast, which is optimized for pulse detection. Typically, for fingerprint over time, the result of subtracting an earlier slice
example, the sensor circuitry is adjusted so that the back from a later slice would be an entirely uniform difference
ground image corresponding to regions of air (valleys 104) is frame 551. Because the pulse causes variation in the finger
represented as white (valley images 594). For example, for an 60 print over time, the result of the subtraction is a non-uniform
eight-bit sensor, the background image is adjusted to have a difference frame 551, which indicates the difference between
grayscale value of 255. The circuitry is also adjusted for high the later frame 5XX and the earlier frame SXX-3.
sensitivity and high contrast so that the image is approxi In step 630, a determination is made based on the series of
mately 20% binarized, meaning that approximately 20% of difference frames 551 as to the coupling stability between the
the fingerprint image is either completely white or completely 65 digit of the mammal, or the finger of the mammal, and the
black. This maximizes the sensitivity, but leaves enough gray platen 110. The details of step 630 are discussed in further
scale content so that pressure fluctuations can be measured. detail below. However, the method requires that the mamma
US 8,433,110 B2
13 14
lian digit in bulk, that is, the digit taken as a whole, be processing purposes. In an alternative embodiment, the pixel
Substantially stable and not in motion, that is, that the digit be values may be copied to along-term storage, for example hard
substantially static in relation to platen 110. While there will drive 412 or other long-term storage. The exact order in which
be movement of individual ridges 102 and valleys 104, the the pixels are copied from the current difference frame 551 to
digit taken as a whole needs to be in a Substantially fixed the temporary one-dimensional storage array does not matter.
position in relation to platen 110. Therefore, in step 630, the After the pixel grayscale values are copied, step 720 fol
coupling stability between the digit and the platen is deter lows. In step 720, the array of pixel values is sorted from
mined. highest to lowest grayscale values. The sorted values may be
In step 635, a decision is made as to whether the coupling stored in the same array, overwriting the unsorted values, or
is stable enough for pulse rate detection. If not (that is, the 10 may be stored in separate array storage.
digit is in bulk movement in relation to the platen), pulse rate In step 725, the midrange of the grayscale values are
detection ceases and the method returns to step 620 of obtain deleted from the sorted array. What is retained in the sorted
ing a time series of images of the digit. If in step 635, it is array is a selected percentage of highest and lowest grayscale
determined that the coupling between the digit and the platen values in the array. This creates an upper/lower-grayscale
110 is sufficiently stable for pulse rate detection (that is, the 15 values-array for the difference frame for the current time
digit in bulk is stationary in relation to the platen), then the slice. The values in the upper/lower-grayscale-values-array
method continues at step 640. are the highest pixel values and the lowest pixel values for the
In step 640, signal processing is undertaken to process the current difference frame 551, and so are representative of the
pressure and/or location fluctuations from one difference difference between fingerprint image for the current time
frame 551 to the next difference frame 551, to determine slice, and the fingerprint image for a time slice from several
instantaneous fluctuation intensity for each time slice. This frames earlier.
method of determining instantaneous fluctuation intensity for For example, in an embodiment the retained values in the
each time slice is discussed in further detail below in con upper/lower-grayscale-values-array are representative of the
junction with FIG. 7. difference between the current slice or current fingerprint
Following step 640, the method continues with step 645. In 25 image, and a slice taken four slices earlier. The highest and
step 645, a calculation of the pulse rate is made based on the lowest grayscale values retained in the upper/lower-gray
series of instantaneous fluctuation intensities. Step 645 is scale-values-array may as little as two percent of the highest
discussed in further detail below, in conjunction with FIG.8. grayscale values and two percent of the lowest grayscale
Inbrief, however, a variation of the instantaneous fluctuation values. In an alternative embodiment, the retained percentage
intensities is monitored to determine a cyclic pattern in the 30 may be as much as ten percent of the highest grayscale values,
instantaneous fluctuation intensities over the time series of and ten percent of the lowest grayscale values. Higher per
slices. Based on the pattern of repetition of instantaneous centages may be employed as well.
fluctuation intensities, the pulse is determined. In step 730 of method 640, a calculation is made as to the
Step 645 continues with step 650, which entails displaying, standard deviation of the pixel values population of the upper/
storing, and/or transmitting the pulse rate data. Step 650 35 lower-grayscale-values-array for the current time slice. In
either continues with step 620, which is obtaining further time other words, the standard deviation of the highest and lowest
series of images of digits, or step 650 continues with step 655, grayscale values is calculated. Once the standard deviation of
which entails stopping the method. the grayscale values is calculated, this standard deviation of
It will be understood by persons skilled in the relevant arts the population of the upper/lower-grayscale-values-array is
that the steps shown as part of method 600 in FIG. 6 are 40 considered to be an “instantaneous fluctuation intensity.”
exemplary only. Additional steps may be employed and some The instantaneous fluctuation intensity is a numerical
steps may be optional. For example, adjusting sensor sensi value embodying or representative of the difference between
tivity and contrast and/or adjusting frame rate may be the fingerprint image of the current slice and the fingerprint
optional steps. In addition, it will be understood that while the image of the preceding time slice that was subtracted from it.
steps are shown in a particular order, and occurring sequen 45 In step 735, the instantaneous fluctuation intensity for the
tially, that the order of Some steps may be changed, and in current time slice is inserted into an array structure which is
addition, some steps may occur in parallel through parallel referred to as the “pulse-rate-waveform-array.” The pulse
processing. rate-waveform-array stores or maintains a series of consecu
FIG. 7 is a flowchart of an exemplary method 640 to pro tive instantaneous fluctuation intensities, representative of a
cess pressure/location fluctuations in a fingerprint 106 to 50 series of changes in the fingerprint 106 over time. The pulse
determine instantaneous fluctuation intensity for each time rate-waveform-array may be stored in main memory 408,
slice. FIG. 6 (above) provided an exemplary method 600 for hard disk drive 412, or other secondary memory 410.
determining a pulse with a fingerprint sensor 200. In particu In step 740 of method 640, the next time slice is identified.
lar, in step 625 of method 600, a time delayed background The method then continues with step 710, which as noted
image Subtraction was performed to obtain a difference frame 55 above identifies the difference frame 551 associated with the
551, and by repeating step 625, a series of difference frames current time slice. The method 640 continues in this manner,
551 is obtained. In step 640 of method 600, the difference looping from step 710 through step 740, thus populating the
frames 551 are analyzed to determine a numerical value, pulse-rate-waveform-array with a time series of instanta
known as the “instantaneous fluctuation intensity.” for each neous fluctuation intensities for the fingerprint.
time slice. FIG. 7 presents in detail the method of step 640. 60 It will be understood by persons skilled in the relevant arts
The method 640 begins at step 705. that the steps shown as part of method 640 in FIG. 7 are
At step 710 the difference frame 551 associated with the exemplary only. Additional steps may be employed and some
current time slice (that is, associated with the current frame) steps may be optional. In addition, it will be understood that
is identified. while the steps are shown in a particular order, and occurring
In step 715, the pixel grayscale values from the current 65 sequentially, that the order of some steps may be changed, and
difference frame 551 are copied to a one dimensional array in in addition some steps may occur in parallel through parallel
memory 408, which may be a temporary array for storage and processing.
US 8,433,110 B2
15 16
FIG. 8 is a flowchart of an exemplary method 645 to cal beats-per-minute=60* (1/rising-edge-elapsed-seconds)
culate a pulse rate based upon an array of instantaneous In step 850, a determination is made as to whether the
fluctuation intensities. FIG. 6 above included a method step beats-per-minutes falls within an allowed range. For
640 to process pressure/location fluctuations, resulting in dif example, it is known that the human pulse generally falls in a
ference frames 551 to determine an instantaneous fluctuation range between 40 beats-per-minutes and, for example, 120
intensity for each time slice. Step 640, discussed in greater beats-per-minutes, though persons skilled in the relevant art
detail in FIG. 7, yielded a pulse-rate-waveform-array contain will recognize other upper and lower bounds may be set. If the
ing a series of instantaneous fluctuation intensities 551. Step pulse (beats-per-minute) that is determined in step 845 is
645 entails calculating the pulse rate based on the pulse-rate 10 determined to fall outside of the allowed beats-per-minutes
waveform-array. The method 645 of FIG. 8 provides further range, then the method continues with step 815 of identifying
detailed steps to calculate pulse rate according to an exem or updating the peak instantaneous fluctuation intensity.
plary method. If the beats-per-minutes falls within the allowed range,
Method 645 starts with step 805. then in step 855 the beats-per-minutes is added to a buffer of
In step 810, a start-event-timestamp is set which corre 15 the last several qualified pulse rates (for example, the last five
sponds to the current time. This time information and other qualified pulse rates).
timestamp information required by the method 645 can be In step 860 a new start-event-timestamp is set equal to the
obtained from timer clock 436 of processing system 225, current time. The method then returns to step 815, of identi
already discussed above in conjunction with FIG. 4. fying or updating the peak instantaneous fluctuation intensity
Step 815 follows step 810. In step 815, the pulse-rate among the time slices.
waveform-array is Surveyed to identify, or if necessary It will be understood by persons skilled in the relevant arts
update, a maximum value from among all the instantaneous that the steps shown as part of method 800 in FIG. 8 are
fluctuation intensity values for all the time slices. exemplary only. Additional steps may be employed and some
In step 820, an end-event-threshold-value is calculated. steps may be optional. In addition, it will be understood that
The end-event-threshold-value is equal to a designated per 25
while the steps are shown in a particular order, and occurring
centage of the peak instantaneous fluctuation intensity iden sequentially, that the order of some steps may be changed, and
tified in step 815. For example, in an embodiment the end in addition, Some steps may occur in parallel through parallel
event-threshold-value may be equal to 20% of the maximum processing.
instantaneous fluctuation intensity identified in step 815. In Exemplary Methods for Determining Bulk Finger Movement
an alternative embodiment, Some percentage other than 20% 30
may be employed, for example, 10%. As noted above, the present method for detecting a pulse
In step 825, a calculation is made of the current slope of the using a fingerprint sensor depends on the digit of the mammal
pulse-rate-waveform (where the pulse-rate-waveform is a taken as a whole (that is, in bulk) being Substantially station
graph or plot resulting from plotting the values in the pulse ary relative to platen 110.
rate-waveform-array). The current slope is calculated based 35 Presented here is one exemplary method for determining if
on the instantaneous fluctuation intensity in the pulse-rate this digit is in bulk motion relative to platen 110.
waveform-array for the current time slice, and a designated As each time slice 5XX is taken, then for platen 110 the
number of the immediately preceding time slices in the pulse sensor's pixel grayscales are averaged, grouped by column
rate-waveform-array. For example, the current time slice and 310. The averages are copied into an averaged-columns array.
the three preceding time slices may be used to determine the 40 The averaged-columns array represents the grayscale profile
current slope of the pulse-rate-waveform. In an alternative across the X-axis of the platen 110.
embodiment, some other number, for example 4, 5, 10 or The averaged-columns array is populated with multiple
other values, may be used for the total number of time slices sequentially averaged columnarrays, for multiple time slices.
preceding the current time slice, which are used to calculate The regression-line of the averaged-columns array is calcu
the current slope. Whatever number of slices are selected, a 45
lated to produce an X-axis slope. If the slope is too far from
current slope is calculated for the pulse-rate-waveform-array. Zero (+/-) then the finger is moving or the coupling between
In step 830, a determination as to whether or not the current the finger and platen 110 is unstable.
slope is greater than the end-event-threshold-value. If not If the regression-line slope is out-of-range, then the current
(meaning the current slope is not greater than the end-event slice image is disqualified and any further processing is dis
threshold), then the method returns to step 815, which iden 50
abled until the finger coupling is stabilized.
tifies or updates a new peak instantaneous fluctuation inten
sity among the time slices. Additional Detail and Alternative Embodiments
If in step 830, the current slope of the pulse-rate-waveform
is greater than the current value of the end-event-threshold, "Plethysmography’ is a method of measuring variations in
the method continues with step 835, where an end-event 55
timestamp is set equal to the current time. the size of an organ or body part on the basis of the amount of
In step 840, following step 835, a time interval is identified blood passing through or present in the part, while a “plethys
based on the difference between the end-event-timestamp and mograph' is a device configured to measure variations in the
the currently set start-event-timestamp. This time difference, size of an organ or body part on the basis of the amount of
the end-event-time minus the start-event-time, may be 60 blood passing through or present in the part. The exemplary
assigned to a variable known as the rising-edge-elapsed-sec systems disclosed herein may be viewed as plethysmographs,
onds. The rising-edge-elapsed-seconds is indicative of the and the exemplary methods disclosed herein may be viewed
time between a most recent pulse of the heart and the imme as methods of plethysmography.
diately preceding pulse of the heart. The methods disclosed herein for analyzing fingerprint
In step 845, a pulse calculation is made based on the rising 65 data may be viewed in part or whole as methods of digital
edge-elapsed-seconds. The formula for calculating the pulse signal processing for determining a pulse based on fingerprint
1S data, and the processing systems disclosed herein may be
US 8,433,110 B2
17 18
viewed in part or whole as digital signal processing systems In an alternative embodiment, platen 110 may be config
configured for determining the pulse based on fingerprint ured to return data descriptive of a fingerprint 106 other than
data. image data of the fingerprint 106. Processing system 225, and
As discussed above in conjunction with various embodi processor 404 in particular, may then be configured to deter
ments, a background image (that is, an older time slices mine an alternative difference between time slices of the
5XX-N) is constantly captured from a (N+1)-frame delay fingerprint 106 other than a difference between images. Fur
buffer (for example, a four frame delay buffer), and sub ther, processing system 225, and processor 404 in particular,
tracted from current fingerprint image 5XX to normalize the may be configured to determine the pulse of the mammal
based on a variation over time in the alternative difference
current slice image to a midlevel number of grayscales (for 10 between time slices.
example, 140 grayscales). The subtraction yields difference
frame 551. If a person or other mammal places a digit on the CONCLUSION
platen 110, and no bulk movement or blood-pressure-induced
fluctuations take place, then after N slice frames (for example, While various embodiments of the present invention have
four frames) the difference frame 551 would appear as a 15 been described above, it should be understood that they have
uniform field of pixels at the midlevel grayscale (for example, been presented by way of example only, and not limitation. It
140 grayscales). An exemplary target grayscale of 140 is will be understood by those skilled in the relevant art(s) that
selected so that the grayscale deviations are easier to see on various changes in form and details may be made therein
the difference frames 551. With an exemplary midlevel gray without departing from the spirit and scope of the invention as
scale normalization to 140, the typical peak grayscale fluc defined in the appended claims. Accordingly, the breadth and
tuation during pulse-rate acquisition is +/-30 grayscales (that scope of the present invention should not be limited by any of
is, approximately 110 to 170 grayscales on a scale of 0 to the above-described exemplary embodiments, but should be
255). defined only in accordance with the following claims and
In exemplary embodiments discussed above, typically 20 their equivalents.
time slices may be taken per second. Combined with the 25 What is claimed is:
background Subtraction algorithm already discussed above 1. A method of biometric sensing comprising: capturing at
(for example, step 625 of exemplary method 600), the back a biometric sensing element a time series of at least one of
ground subtraction algorithm effectively creates a high pass ridges or valleys of at least a portion of a fingerprint; receiving
filter for every sensor pixel, with a recovery time at 0.200 at a processor coupled to the biometric sensing element the
seconds. 30 time series of at least one of ridges or valleys of the at least the
This recovery time of 0.200 seconds is relatively short, portion of the fingerprint; detecting in the time series a varia
when compared with pulse rates on the order of beats-per tion over time in the at least one of ridges or valleys of the at
minute. The relatively short recovery time is indicated due to least the portion of the fingerprint; and determining a pulse of
the quality and stability of the coupling between the mamma a mammal associated with the fingerprint based on the varia
lian digit and platen 110. If a user takes a deep breath, 35 tion over time.
squirms, or shifts position, then the DC offset of the pressure 2. The method of claim 1, wherein; the time series of at
waveform (and the position of the fluctuation Zones on platen least one of ridges or valleys comprises a series of images of
110) will vary. The skin's pulse-rate-induced pressure fluc the at least the portion of the fingerprint; and detecting the
tuations are fast, causing rapid rise-times, so 0.200 seconds is variation over time in the time series comprises comparing a
considered a good compromise between over-attenuation of 40 second image in the time series to a first image in the time
the pulse signal and pulse-rate signal stability. series, wherein the first image and the second image are from
In embodiments of the present invention disclosed above, a different times in the time series.
first, earlier fingerprint 106 image 5XX-N is compared to a 3. The method of claim 2, wherein comparing the second
second, later fingerprint 106 image 5XX to obtain a differ image to the first image yields a data value representative of
ence between the two images. (See for example step 625 of 45 fluctuations in the at least the portion of the fingerprint from
method 600.) In the embodiments disclosed above, the com the first image to the second image.
parison is performed by Subtracting corresponding pixel val 4. The method of claim 2, wherein comparing the second
ues from the first and second fingerprint 106 images. image to the first image comprises Subtracting the first image
In an alternative embodiment, mathematical forms of com from the second image.
parison between two image slices 5XX.5XX-N other than an 50 5. The method of claim 2, wherein determining based on
operation of Subtraction may be used to determine a repre the variation over time in the time series the pulse of the
sentative difference value between two time slices or frames mammal comprises: comparing a second plurality of images
of the fingerprint 106. In an alternative embodiment, math from the time series to a first plurality of images from the time
ematical forms of comparison between two image slices series, wherein each respective image from the second plu
5XX.5XX-N may include subtraction of pixel values, but not 55 rality is compared to a respective earlier image from the first
necessarily Subtraction between corresponding pixels of the plurality; obtaining from the comparing a plurality data val
first frame and second frame. In an alternative embodiment, ues representative of fluctuations in the fingerprint over time;
mathematical forms of comparison between two image slices and analyzing the plurality of data values representative of
5XX. 5XX-N may include subtraction of pixel values, but fluctuations to determine the pulse of the mammal.
may also include additional operations beyond Subtraction to 60 6. The method of claim 5, wherein determining based on
determine the representative difference value between two the variation over time the pulse of the mammal comprises
time slices or frames of the fingerprint 106. determining a peak in the pulse based on the plurality of data
In an alternative embodiment, multiple calculations may values representative of the fluctuations.
be employed to determine more than one value or a plurality 7. The method of claim 6, wherein determining based on
of values, each different calculated value being indicative in a 65 the variation over time a pulse of the mammal further com
different way of the difference between a give first time slice prises: calculating a pulse based on a time interval between a
and a given second time slice. first peak in the pulse and a second peak in the pulse.
US 8,433,110 B2
19 20
8. The method of claim 1, wherein: the time series of at 18. The biometric sensor of claim 11, wherein: the biomet
least one of ridges or valleys comprises a series of images of ric sensing element is configured to capture a time series of
the at least the portion of the fingerprint; and capturing at the images of the at least the portion of the fingerprint; and the
biometric sensing element the time series of at least one of biometric sensing element and/or the processor are config
ridges or valleys of at least a portion of a fingerprint comprises 5 ured for at least one of an image capture sensitivity Substan
at least one of setting at the biometric sensing element an tially higher than an image capture sensitivity used for fin
image capture sensitivity Substantially higher than an image gerprint identification; and an image contrast Substantially
capture sensitivity used for fingerprint identification; and set higher than an image contrast used for fingerprint identifica
tion.
ting at the biometric sensing element an image contrast Sub
stantially higher than an image contrast used for fingerprint 10 19. The biometric sensor of claim 11, wherein the proces
identification. sor is further configured to determine that a digit associated
9. The method of claim 1, further comprising determining with said at least the portion of the fingerprint is, in bulk
aspect, Substantially stationary relative to the biometric sens
that a digit associated with said at least the portion of the ing element.
fingerprint is, in bulk aspect, Substantially stationary relative 15 20. The biometric sensor of claim 19, wherein: the biomet
to the biometric sensing element. ric sensing element is configured to capture a time series of
10. The method of claim 9, wherein: the time series of at images of the at least the portion of the fingerprint; and the
least one of ridges or valleys comprises a series of images of processor is configured to determine that the digit of the
the at least the portion of the fingerprint; and determining that mammal is, in bulk aspect, Substantially stationary relative to
the digit of the mammal is, in bulk aspect, Substantially sta the biometric sensing element by comparing at least a first
tionary relative to the biometric sensing element comprises image and a second image from the time series.
comparison of at least a first image and a second image from 21. A non-transitory computer readable medium having
the time series. control logic stored thereon, the control logic comprising
11. A biometric sensor, comprising: a biometric sensing machine executable code which when executed by a proces
element configured to capture a time series of at least one of 25 Sor causes the processor to: receive from a biometric sensing
ridges or valleys of at least a portion of a fingerprint; and a element a time series of at least one of ridges or valleys of at
processor coupled to the biometric sensing element, the pro least a portion of a fingerprint; detect in the time series a
cessor configured to: receive from the biometric sensing ele variation overtime in the at least one of ridges or valleys of the
ment the time series of at least one of ridges or valleys of the at least the portion of the fingerprint; and determine based on
at least the portion of the fingerprint; detect in the time series 30 the variation over time a pulse of a mammal associated with
a variation over time in the at least one of ridges or Valleys of the fingerprint.
the at least the portion of the fingerprint; and determine based 22. The non-transitory computer readable medium of claim
on the variation over time a pulse of a mammal associated 21, further comprising machine executable code which when
with the fingerprint. executed by the processor causes the processor to: receive
12. The biometric sensor of claim 11, wherein: the biomet 35 from the biometric sensing element a time series of images of
ric sensing element is configured to capture a time series of the at least the portion of the fingerprint; and detect the varia
images of the at least the portion of the fingerprint; and the tion over time in the time series by comparing a second image
processor is configured to detect in the time series of images in the time series to a first image in the time series, wherein the
the variation over time by comparing a second image in the first image and the second image are from different times in
time series to a first image in the time series, wherein the first 40 the time series.
image and the second image are from different times in the 23. The non-transitory computer readable medium of claim
time series. 22, further comprising machine executable code which when
13. The biometric sensor of claim 12, wherein the proces executed by the processor causes the processor to: compare
sor is further configured to compare the second image to the the second image to the first image to obtain a data value
first image to obtain a data value representative of fluctuations 45 representative of fluctuations in the at least the portion of the
in the at least the portion of the fingerprint from the first image fingerprint from the first image to the second image.
to the second image. 24. The non-transitory computer readable medium of claim
14. The biometric sensor of claim 12, wherein the proces 22, further comprising machine executable code which when
sor is configured to compare the second image to the first executed by the processor causes the processor to: compare
image by Subtracting the first image from the second image. 50 the second image to the first image by Subtracting the first
15. The biometric sensor of claim 12, wherein the proces image from the second image.
sor is further configured to: compare a second plurality of 25. The non-transitory computer readable medium of claim
images from the time series to a first plurality of images from 22, further comprising machine executable code which when
the time series, wherein each respective image from the sec executed by the processor causes the processor to: compare a
ond plurality is compared to a respective earlier image from 55 second plurality of images from the time series to a first
the first plurality, obtain from the comparing a plurality data plurality of images from the time series, wherein each respec
values representative of fluctuations in the fingerprint over tive image from the second plurality is compared to a respec
time; and analyze the plurality of data values representative of tive earlier image from the first plurality: obtain from the
fluctuations to determine the pulse of the mammal. comparing a plurality data values representative of fluctua
16. The biometric sensor of claim 15, wherein the proces 60 tions in the fingerprint over time; and analyze the plurality of
sor is further configured to determine a peak in the pulse data values representative of fluctuations to determine the
based on the plurality of data values representative of the pulse of the mammal.
fluctuations. 26. The non-transitory computer readable medium of claim
17. The biometric sensor of claim 16, wherein the proces 25, further comprising machine executable code which when
sor is further configured to calculate a pulse based on a time 65 executed by the processor causes the processor to: determine
interval between a first peak in the pulse and a second peak in a peak in the pulse based on the plurality of data values
the pulse. representative of the fluctuations.
US 8,433,110 B2
21 22
27. The non-transitory computer readable medium of claim
26, further comprising machine executable code which when
executed by the processor causes the processor to: calculate a
pulse based on a time interval between a first peak in the pulse
and a second peak in the pulse.
executed by the processor causes the processor to establish in
a time series of images of the fingerprint obtained via the
biometric sensing element at least one of an image capture 10
sensitivity Substantially higher than an image capture sensi
tivity used for fingerprint identification; and an image con
trast Substantially higher than an image contrast used for
fingerprint identification.
29. The non-transitory computer readable medium of claim 15
executed by the processor causes the processor to: determine
that a digit associated with the at least the portion of the
fingerprint is, in bulk aspect, Substantially stationary relative
to the biometric sensing element.
executed by the processor causes the processor to: determine
that the digit of the mammal is, in bulk aspect, Substantially
stationary relative to the biometric sensing element by com 25
paring at least a first image and a second image from a time
series of images of the least the portion of the fingerprint
received from the biometric sensing element.
k k k k k

US8433110

Uploaded by

Copyright:

Available Formats

US8433110

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

US8433110

Uploaded by

Copyright:

Available Formats

USOO8433 11 OB2

(12) United States Patent (10) Patent No.: US 8.433,110 B2

Fingerprint Ridges at low pulse (diastole)

Fingerprint Ridges at high pulse (systole)

— Processing System 225

Main Memory 408

Display Interface 402 Display 215

K ) Control Interface 434

Frame 501 Subtracted

Adjust sensor sensitivity and contrast for pulse detection. 610

Adjust frame rate for pulse detection. Y\ , 615

Begin (continue) obtaining time-series of images of digit

Determine coupling stability between digit and platen. NO

Coupling stable enough for

Process pressure/location fluctuations (Difference Frames) to Y\, 640

Calculate Pulse Rate based on series of Y\ 645

Exemplary Method For Detecting

O start D/V 705 640

Sort array from highest to lowest grayScale values

Calculate Standard deviation of the population of the array, i.e.,

Insert Instantaneous Fluctuation intensity for current time slice

Identify next time slice

Exemplary Method To Process Pressure/location

You might also like