Computers and Electronics in Agriculture 69 (2009) 12–18

Contents lists available at ScienceDirect

Computers and Electronics in Agriculture

journal homepage: www.elsevier.com/locate/compag

Hardware-based image processing for high-speed inspection of grains

Tom Pearson ∗
USDA-ARS-GMPRC, 1515 College Ave, Manhattan, KS 66502, United States

a r t i c l e i n f o a b s t r a c t

Article history: A high-speed, low-cost, image-based sorting device was developed to detect and separate grains
Received 3 February 2009 having slight color differences or small defects. The device directly combines a complementary
Received in revised form 10 June 2009 metal–oxide–semiconductor (CMOS) color image sensor with a field-programmable gate array (FPGA)
Accepted 12 June 2009
which was programmed to execute image processing in real-time, without the need of an external com-
puter. Spatial resolution of the imaging system is approximately 16 pixels/mm. The system includes three
Keywords:
image sensor/FPGA combinations placed around the perimeter of a single-file stream of kernels, so that
FPGA
most of the surface of each kernel is be inspected. A vibratory feeder feeds kernels onto an inclined chute
CMOS
Popcorn
that kernels slide down in a single-file manner. Kernels are imaged immediately after dropping off the end
Wheat of the chute and are diverted by activating an air valve. The system has a throughput rate of approximately
Corn 75 kernels/s per channel which is much higher than previously developed image inspection systems. This
Smart camera throughput rate corresponds to an inspection rate of approximately 8 kg/h of wheat and 40 kg/h of pop-
Machine vision corn. The system was initially developed to separate white wheat from red wheat, and to remove popcorn
having blue-eye damage, which is indicated by a small blue discoloration in the germ of a popcorn kernel.
Testing of the system resulted in accuracies of 88% for red wheat and 91% for white wheat. For popcorn,
the system achieved 74% accuracy when removing popcorn with blue-eye damage and 91% accuracy at
recognizing good popcorn. The sorter should find uses for removing other defects found in grain, such
as insect-damaged grain, scab-damaged wheat, and bunted wheat. Parts for the system cost less than
$2000, suggesting that it may be economical to run several systems in parallel to keep up with processing
plant rates.
Published by Elsevier B.V.

1. Introduction tations in parallel, and that they execute all commands in hardware
so they are ideal for real-time systems. Additionally, FPGAs are able
Field-programmable gate arrays (FPGAs) are semiconductor to perform computation on images as they are transferred to the
devices that are comprised of interconnected logic elements, mem- device, before the complete image has been loaded. In contrast,
ory, and digital signal processing hardware on a single chip. The processing of images on a PC usually requires the images to be
configuration of the interconnections, and therefore the function completely loaded into memory, then read out from memory and
of the device, is determined by compiled programs loaded onto the processed. This causes a delay in processing the image which can be
chip. These devices do not need any operating systems to function significant for high-speed sorting operations (Pearson et al., 2008)
and can handle data throughputs over 200 MB/s, which is at least Charge-coupled devices (CCD) have traditionally been the most
an order of magnitude higher than what can be achieved by micro- commonly used image sensors. While these devices produce
controllers. FPGAs are currently used in a large variety of hardware high-quality images, they require significant support electronics
where low cost and high data throughput rates are required, such (Yamada, 2006). However, this need for elaborate support electron-
as digital cameras, cell phones, speech recognition, and image pro- ics has recently been alleviated by the now-widespread production
cessing (Maxfield, 2004). Many traditional image frame-grabber of complementary metal–oxide–semiconductor (CMOS) image
boards use FPGAs to pre-process incoming images and direct the sensors (Takayanagi, 2006). CMOS sensors are less expensive than
data to other hardware for further processing and storage. Advan- equivalent CCD sensors, and the required support electronics are
tages of FPGAs over micro-controllers and personal computers for integrated onto the sensor chip. The integration of amplifiers,
image processing functions are that they can perform many compu- analog-to-digital converters, and signal processing capabilities onto
the CMOS image sensors drastically simplifies camera design and
further lowers cost. This makes interfacing CMOS chips with
∗ Tel.: +1 785 776 2729; fax: +1 785 537 5550. user-programmable processing hardware, such as FPGAs, relatively
E-mail address: [email protected]. simple.

0168-1699/$ – see front matter. Published by Elsevier B.V.

doi:10.1016/j.compag.2009.06.007
 T. Pearson / Computers and Electronics in Agriculture 69 (2009) 12–18 13

A direct CMOS-FPGA link is now widely used in many cellular to delayed drying of the kernels after harvest. Since popcorn quality
phones and digital cameras (Maxfield, 2004). The FPGAs in these is very sensitive to moisture, the corn cannot be dried as quickly as
systems are used as “glue logic” between the image sensor, micro- field corn. However, this can cause infection by fungi that lead to
processor, and memory. While FPGAs may perform some digital blue-eye damage, which appears as a small, dark blue spot on the
image processing such as white balancing or exposure compensa- germ. The problem is important to the popcorn industry as these
tion, they do not perform image feature extraction or classification. kernels can have a poor flavor when popped. The visual damage to
“Smart Camera” is an industry term for a camera that has an image the kernels is small and therefore not detectable by any commer-
sensor and processor that are directly linked, and that has capa- cially available sorting equipment. Automated separation of red and
bility for an end user to do some programming of the processor. white wheat is needed by wheat breeders developing white wheat
These devices are gaining acceptance for automated inspection of varieties that have the baking properties of red wheat. Upon harvest
machine parts and self-guided robots. However, currently these of field ploat with both red and white wheat, the white wheat needs
cameras are still very expensive, use digital signal processors rather to be separated from red kernels so that it can be propagated again.
than FPGA’s, and an end user usually has limited ability to mod- Commercial color sorters can distinguish red and white wheat with
ify the imaging software. Thus, these devices have not yet found approximately 80% accuracy after several passes through the sorter
uses for the inspection of agricultural products since agricultural (Pasikatan and Dowell, 2003), which may not always be accurate
applications typically deal with irregular product sizes, and specific enough for some breeding lines with small amounts of white wheat.
separation requirements. FPGA’s have gained computational power Additionally, the cost of these sorters is high and they are designed
by integrating more logic elements, fast on-chip memory, and dig- to handle large volumes rather than the smaller samples of 1 kg or
ital signal processing capabilities such as hardware to compute less that breeders generally work with.
convolutions. These enhancements make FPGA’s more cost effec- The purpose of this research was to directly link a low-cost CMOS
tive than DSP’s for some signal processing applications (Maxfield, image sensor with a low-cost FPGA, and to program the FPGA to
2004). With the price of CMOS imaging chips and FPGAs already extract simple features and classify objects in the image. The test
being very low, and with the simplicity of linking these two tech- objects were popcorn with and without blue-eye damage and red
nologies together, it is possible to assemble a CMOS image sensor and white wheat classes.
and FPGA together to make a “smart camera” for less than $100,
with a user having full access to program the camera, extract image
features, and make classifications for sorting operations. 2. Materials and methods
Most commercial high-speed sorting machines used for agri-
cultural products either have no spatial resolution or do not fully 2.1. Image sensor—FPGA design
utilize the spatial resolution produced by their sensors. For those
sorters with some spatial resolution, the only image processing per- A CMOS image sensor (KAC-9628, Eastman Kodak Company,
formed is thresholding and pixel counting. These sorters are useful Rochester, NY) was mounted onto a custom-designed printed
for detecting larger, high contrast, spots on products. Sorters hav- circuit board with all support electronics recommended by the
ing no spatial resolution may have multiple sensors coupled by manufacturer. Fig. 1 displays a schematic of the image sensor,
beam splitters to measure light at two or three specific spectral support electronics, and interface to the FPGA. The FPGA, with
bands. These types of sorters are effective at distinguishing prod- its necessary support electronics, was purchased pre-mounted
ucts with large color or chemical constituent differences by using onto a circuit board of its own (Pluto-II, KNJN-LLC-fpga4fun.com,
near infrared bands. Some newer sorters might have two or more Freemont, CA). The FPGA board has an electronically erasable pro-
sensor arrays coupled with beam splitters to get very low spatial grammable read only memory (EEPROM), or otherwise known as
resolution images at two or three spectral bands. However, for many non-volatile memory, that stores and loads the program the FPGA
products, certain defects are difficult to detect and remove using the is to run upon powering up. In addition, the board also has a serial
most advanced, currently available, commercial sorters. Products communications port for loading new programs and transferring
with slight color differences or small, low contrast spots or blem- data, a 25 MHz clock, an LED, a 3.3 V voltage regulator for powering
ishes are difficult or impossible to sort with commercial sorters. the FPGA and connected electronics, and a power socket to sup-
Shriveled and Fusarium Head Blight (scab) damaged wheat kernels ply external power. The FPGA used on the Pluto-II board is made
are a case in point. The efficacy of using a limited spatial resolu- by Altera Corporation (Model #Cyclone EP1C3) and the free Altera
tion (∼0.5 mm) commercial dual-band (one near infrared (NIR), one Quartus II web edition version 7.2 was used to develop and compile
visible) sorter for removal of scab-damaged kernels has been stud- programs for the FPGA. The FPGA board was connected directly to
ied (Delwiche et al., 2005). Only 50% of the scab-damaged kernels the image sensor board via header pins. Fig. 2 displays images of
were removed, while about 5% of the undamaged kernels were also the image board and attached FPGA board.
rejected. Preliminary studies have shown that the use of simple The circuit board built for the image sensor and interface for the
histograms of low resolution color images enables scab-damaged FPGA board was a four-layer board to simplify grounding and power
kernels to be distinguished from sound kernels with over 90% accu- distribution and to help shield electrical noise. A 470 ␮F electrolytic
racy (Pearson, 2006). This is an application that an FPGA linked to capacitor was used on the image board to stabilize the power from
an image sensor could perform very economically. low-frequency variation while several 0.1 ␮F capacitors were used
There have been many developments relevant to the inspection on the image board near the image sensor to dampen higher fre-
of agricultural products using imaging, such as the inspection of quency power fluctuations.
apples (Bennedsen and Peterson, 2004), rice Kumar and Bal, 2007), The FPGA board was also connected to a rotary hex DIP switch
and wheat (Wang et al., 2004). However, few of these developments (94HAB16RAT, Grayhill, Inc., LaGrange, IL) which sends a number
are able to be implemented as high-speed sorting applications at between 0 and 15 to the FPGA that adjusts a sensitivity thresh-
an economically feasible cost. old for classification (discussed later). For diverting grain, the FPGA
Popcorn infected with blue-eye damage and separation of red outputs a digital signal which triggers a solid state relay (D0061B,
and white wheat classes are two sorting applications that com- Crydom, San Diego, CA) and fires an electronic solenoid-activated
mercial sorters are not able to adequately separate and current air valve (36A–AAA–JDBA–1BA, Mac Valves, Inc., Wixom, MI). The
imaging technology is too slow and expensive for viable implemen- air valve sends a burst of air for 5 ms to a wide air nozzle (31875K41,
tation. Blue-eye damage in popcorn is a re-occurring problem due McMaster-Carr Co., Elmhurst, IL) to ensure proper rejection.
14 T. Pearson / Computers and Electronics in Agriculture 69 (2009) 12–18

The 25 MHz clock on the FPGA circuit board was wired to the main
clock input of the image sensor, so the pixel clock rate was 12.5 MHz.
The region of interest was limited to the two lines in the center
of the image sensor. This essentially made the two-dimensional
sensor work as a color linescan sensor. The image sensor used has
either a red, green or blue color filter over every pixel arranged in a
Bayer pattern, typical of most two-dimensional image sensors. One
line consists of red and green pixels and the next line consists of
green and blue pixels. Most two-dimensional color images are con-
structed by interpolating the colored pixels so that each pixel would
appear to have a red, green, and blue value. However, this was not
done in this application in order to reduce computations. Thus, the
red and blue image data was one quarter the full scale pixel resolu-
tion and the green image data was one half the full scale raw pixel
resolution. The analog image gain was set to a level of 128, which is
the middle of the amplification range on this sensor. The 12.5 MHz
clock rate produced images of popcorn and wheat kernels of the
correct aspect ratio when the imaged lines were reconstructed to
form a two-dimensional image.
The image sensor continuously sent pixel data to the FPGA.
Image acquisition was initiated at the first pixel of the red-green
image sensor line after any pixel data above an intensity of 15 was
detected by the FPGA. Self-triggering of image acquisition simpli-
fies the design and is made possible by running the image sensor in
linescan mode with direct connection to the FPGA. After triggering,
the FPGA would acquire 125 complete lines, which resulted in an
image 640 pixels wide by 125 pixels tall. Thus, image acquisition
time was approximately 6.4 ms. Both popcorn and wheat kernels
would range from approximately 100 to 120 lines tall in the result-
ing images. Images were not color interpolated by the FPGA, in order
to reduce complexity of the FPGA program. The red, green and blue
pixels were simply processed separately. Images from the image
sensor were also captured by a logic analyzer (LAP-16128U, Zero-
plus, Chung Ho City, Taiwan) and stored on a personal computer
for inspection of lighting and focus, and for off-line development of
image processing and classification schemes.
Three FPGA/image sensor cameras were mounted to a steel cir-
cular ring 30.5 cm in diameter. The cameras were placed 120◦ apart
on the ring so that almost the entire surface of kernels could be
Fig. 1. Schematic of image sensor—FPGA interface with supporting electronics for inspected as they dropped off the end of the chute. Preliminary
the image sensor. Not shown are the EEPROM, serial interface, 25 MHz clock, voltage experiments showed that a small spot on a popcorn kernel made
regulator, and support electronics for the FPGA which are supplied on the Pluto-II
by an ink pen was visible approximately 99% of the time. Con-
board.
versely, when only two cameras were used in an opposed manner,
the spot was visible in only 70% of the images. Thus, a three cam-
All non-default image sensor settings were sent from the FPGA era system was used as shown in Fig. 3 to ensure that the system
to the image sensor using an inter-integrated-circuit (I2 C) serial bus would be able to detect small spots or blemishes, such as blue-eye
communications protocol (Phillips Semiconductor, 2000) as spec- damaged popcorn. Illumination for each camera came from two
ified by the image sensor manufacturer. The default image sensor narrow-spot-beam 20 W halogen reflector bulbs (#1003116, Ushio
settings that were changed by the FPGA were the pixel clock divi- America, Cypress, CA) for a total of six bulbs for the entire sys-
sion (from four to two), the region of interest, and the analog gain. tem. The bulbs were placed such that they did not point directly

Fig. 2. Photo of the image sensor and FPGA boards connected together. The image sensor and lens are on the opposite side from the FPGA board.
 T. Pearson / Computers and Electronics in Agriculture 69 (2009) 12–18 15

2. At least three pixels between pixels 10 and 14 of the 24 pixel set

must have an intensity level less than a preset threshold level.
3. A rise in red pixel intensity of at least 15 intensity levels across
the width of six pixels occurring in the last 10 pixels of the 24
pixel set.

The rotary DIP switch was used to adjust the pixel intensity
threshold level between values of 60 and 90 in steps of 2 inten-
sity levels. This was needed to adjust for slightly different lighting
for each of the three cameras. In each image, the FPGA counted
each occurrence of the blue-eye conditions outlined above, with
only one count allowed per image line to avoid double-counting of
the same blemish on any given line. Blue-eye damage would usu-
ally have 10 or more occurrences while good kernels would usually
have 0 occurrences. The air valve was activated if more than ten
occurrences of the blue-eye damage criteria were met.
All of the image processing for each pixel was performed before
Fig. 3. End view of the sorter sensing system showing all three cameras, six light the next pixel data arrived, so there was no further processing, or
bulbs, air nozzle, and chute. delay, after the complete image was acquired. Once the image was
acquired, the FPGA would be ready to start processing a new image
at cameras on the opposite side of the ring. The lenses used on as soon as the next kernel came into the field of view of the camera.
each camera were miniature type with 12 mm × 0.5 mm thread and While the FPGA also controlled the opening and closing of the air
16 mm focal length, f1.8 (V-4316-2.0, Marshall Electronics, Inc., El valve, this was done in parallel with image acquisition to have the
Segundo, CA). Lenses were mounted to the image sensor circuit highest kernel throughput possible.
board with a threaded lens mount (LH-4, Marshall Electronics, Inc.,
El Segundo, CA). Aluminum tubes, 4 cm long, were press fit around 2.3. Signal processing—red and white wheat
each lens to block stray light from adjacent bulbs that would other-
wise affect the images. The feeder chute was aligned in the center Pearson et al. (2008) showed that the standard deviation of the
of the ring so that all cameras had similar spatial resolution. Fig. 3 red pixels and the number of blue pixels below a set threshold are
displays an end view of the complete sorting system. two good features for distinguishing red wheat from white wheat
when using color images. Red wheat tends to have higher stan-
2.2. Signal processing and classification—popcorn dard deviations of pixel intensities as they tend to have darker areas
accompanied by lighter, almost white, areas at the beard end. Also,
Fig. 4 displays images of a popcorn kernel with blue-eye damage weathering tends to create light areas on red wheat kernels. The
as well as an undamaged kernel (these are high-resolution images, combination of darker red and lighter white areas drives the pixel
not taken by the image sensor/FPGA camera used for sorting). Below intensity standard deviation higher than more consistently colored
each image is a gray image displaying only the red pixels. It can white kernels. Red wheat also has higher counts of blue pixels with
be seen that the blue-eye damage (darker spot on the germ) has dark intensity levels. This is due to the red kernel pigment absorb-
slightly better contrast in the mono-chrome red pixel image than ing blue light. In contrast, white kernels have lower counts of low
in the color image. Furthermore, the brightness of the endosperm is intensity blue pixels, since the white pigment reflects high amounts
closer to that of the germ in the red pixel image. Below the images of of blue light. The FPGA was programmed to compute these two fea-
the red pixels, an intensity profile plot across each kernel is shown. tures and classify kernels based on them. The variance of the red
Blue-eye damage is approximately 15 pixels wide by 40 pixels long pixels was computed by keeping a running tally of the sum and
in images taken by cameras used on the sorting machine. The chute sum squared of the red pixel intensities above a threshold level
used to feed the kernels had a round bottom groove approximately of 15, which segmented the kernel from the background. The low
9.0 mm wide so that kernels would usually be oriented so that their threshold level for counting blue pixels was also 15, but the upper
long axis was parallel with their direction of travel. This orientation threshold limit was set by the rotary DIP switch, which ranged from
caused the blue-eye damage to appear from top to bottom in the 60 to 121 in steps of 4 intensity levels. By trial and error, kernels
images. Blue-eye cannot be accurately identified by simple image were classified as red if the variance was greater than 2500 and the
thresholding, as the spot is very small and dark areas on the sides of count of red pixels with low intensities was greater than 50. The
the kernels have similar intensities to blue-eye areas. Profile plots sorter was calibrated by adjusting the rotary DIP switch until an
across blue-eye damaged portions of a kernel appear as a deep val- accurate sort was achieved.
ley having steep slopes from a high intensity level of the white germ
to low intensity levels of the darker blue-eye damaged area. 2.4. Sample source and sorter testing
Upon receipt of new red pixel data, the FPGA was programmed
to place it into a memory buffer of 32 pixels in length. Pixel intensity Popcorn samples were supplied by a major popcorn processor,
slopes were computed with a six pixel gap. Blue-eye damage was and were pulled from two storage bins known to have high inci-
detected when a set of pixels had a sharp drop in intensity, followed dences of blue-eye damage. One fifty-pound sack was pulled from
by at least three pixels having an intensity below a threshold level, each bin. Subsamples from these sacks were inspected by hand
then a sharp rise in pixel intensity, all happening within 24 pixels and 100 blue-eye damaged kernels and 1000 good kernels were
in one line. Specifically, a set of 24 pixels were considered to be part removed for sorter testing. After the FPGA’s were programmed, the
of blue-eye damage when the following conditions were met: good kernels were fed through the sorter with a lens cap over two
of the cameras. The rotary DIP switch on the camera without the
1. A drop in red pixel intensity exceeding 15 intensity levels across lens cap was adjusted until very few good kernels were rejected.
a width of six pixels. This must occur in the first 10 pixels of the The procedure was then applied to the other cameras. This was
24 pixel set. done to account for slightly different lighting conditions for each
16 T. Pearson / Computers and Electronics in Agriculture 69 (2009) 12–18

Fig. 4. Images of blue-eye damaged (left) and good (right) popcorn kernels. The top images are normal color images while the lower images are of the red pixels only. Note
that the blue-eye damage in the germ area has somewhat better contrast from the germ and kernel in the red pixel image than the color image. The blue-eye damage is
highlighted in the pixel intensity profile plots (bottom), causing a deep “valley” in the plot.

camera. After setting the DIP switches, the 100 blue-eye damaged the sorter and the rotary DIP switches were adjusted to mini-
kernels and 1000 good kernels were mixed and run through the mize rejection of red wheat. This was performed one camera at
system at a rate of approximately 75 per second. The fractions of a time using the same procedure as with the popcorn. Next, a
blue-eye and good kernels in the reject and accept streams were 200 g mixture of 90% red and 10% white wheat was run through
then counted. the sorter at approximately 75 kernels/s. The amount of red and
The red and white wheat were of the Jagger and Blanca Grande white kernels in each of the accept and reject streams were then
varieties, respectively. A sample of the red wheat was run through weighed.
 T. Pearson / Computers and Electronics in Agriculture 69 (2009) 12–18 17

3. Results and discussion process, however. A standard reference sample could be used to
perform this adjustment, and it probably would not need to be
3.1. Popcorn performed more than once per week (Pearson et al., 2008).
This work has demonstrated that use of a color image sensor
The sorter was able to separate 74% of the popcorn with blue-eye directly linked to an FPGA can facilitate sorting accuracies that are
damage while also rejecting, erroneously, 9% of the good popcorn. much higher than what can be accomplished with commercial color
Thus, it was 91% accurate on the good popcorn. The 9% false posi- sorting machines. Some defects found on grains, such as blue-eye
tive rate is acceptable since blue-eye damage usually occurs in a low damage on popcorn, cannot be detected by color sorting machines
percentage of storage bins every year. Therefore, only 9% of a small but can be detected with reasonable accuracy by use of simple
fraction of the entire popcorn stored would be downgraded into image processing. Likewise, the process of separating red and white
the lower valued blue-eye class. Inspection of the good kernels that wheat can be made more accurate than what can be accomplished
were rejected by the sorter revealed that approximately 20% (1.8% by color sorters, by use of both the red and blue spectral bands
of the total good kernels) had at least one small dark spot elsewhere and very simple image processing to compute the standard devia-
on the kernel not associated with blue-eye. It may be desirable for a tion and counts of certain colored pixels within specified intensity
popcorn processor to remove these kernels in order for their prod- ranges.
uct to have an appealing appearance. Since it takes 6.4 ms to acquire While this study demonstrated the use of this imaging hardware,
an image, process it and classify it, the sorter should have a theo- more work is needed to automatically account for lighting fluctua-
retical throughput of 156 kernels/s. However, this assumes that the tions, to extract more features to achieve higher sorting accuracies,
kernels would be perfectly spaced apart. If two kernels are touching to use more elaborate classification techniques, and to make the
as they enter the camera field of view, then both may be rejected system physically more robust. It has been shown that accura-
if one of them is classified as blue-eye damaged. It was found by cies for separating red and white wheat can be increased to about
trial and error that these types of errors could be minimized if the 96% when three image histogram features are extracted from color
kernel throughput did not exceed 75 kernels/s, approximately half images and discriminant functions are used to perform classifica-
of the theoretical maximum throughput rate. A throughput rate tions. While extracting more features can be easily performed on
of 75 popcorn kernels/s is approximately equivalent to 40 kg/h (or the FPGA, there is currently no easy way to enter new discriminant
1.5 bushels/h). At this rate, one sorter could process a 10,000 bushel function parameters. This could be overcome by adding the capa-
bin in approximately 278 days if it runs continuously for 24 h a day. bility to store images to removable flash memory so that they can
Since blue-eye damage does not occur in every storage bin, a typical be easily transferred to a personal computer, where new discrimi-
large popcorn processor may need to sort only two of three storage nant functions can be developed and their parameters loaded back
bins per year. The cost of all of the parts for one sorting system is to the flash memory that the FPGA reads at start-up. Accuracy for
approximately $2000, so having more than one sorter running in popcorn could likely be improved by more elaborate image pro-
parallel may be a viable option. cessing and by using other colored pixels as well. The FPGA chip
used in this study has approximately 2900 logic elements that can
3.2. Wheat
be programmed. The popcorn and wheat programs used 880 and
The accuracy achieved by the system was 88% for red wheat 950 logic elements, respectively. Thus, the FPGA has the capability
and 91% for white wheat. These accuracies are more than 10–20% to perform more elaborate computations, especially since minimal
above what can be accomplished after passing wheat through a effort was made to make efficient use of available logic elements.
commercial color sorter several times (Pasikatan and Dowell, 2003). Other FPGAs with much higher amounts of programmable logic
However, the accuracies are about 5% below what has been accom- elements are also available.
plished using three similar features extracted from color images Grain throughput of the sorting system could be greatly
using a traditional camera and personal computer to do the image improved for some products by re-configuration of the cameras
processing (Pearson et al., 2008). Classification on that system was so that only two are used, one above and one below a “water-
made using a discriminant function. However, the system using the fall” of grain sliding down several closely spaced parallel chutes.
personal computer has a lower throughput rate of approximately The images acquired in this study were 640 pixels wide, yet each
30 kernels/s, a higher initial cost, and is probably not as physically popcorn kernel was no more than 90 pixels wide. Thus, five or six
robust. streams of grain could be inspected, lowering the overall cost and
space requirements of the sorters. As discussed earlier, if only a few
3.3. General discussion simple image features are extracted, the FPGA chip used should
have enough logic elements to process several kernels in parallel.
The throughput of 75 kernels/s per channel approximates that This technique may work well for kernels having an overall color
of high-speed commercial color sorters and is substantially higher difference, such as red and white wheat. However this configuration
than what has been developed so far using traditional cameras con- would not inspect the entire perimeter of the kernels, so accuracy
nected to personal computers that perform the image processing in the detection of small localized defects, such as blue-eye damage,
(Pearson et al., 2008). Traditional cameras may output images of would suffer. Further research is needed to quantify the trade-off
similar resolution at rates of 60 frames per second, but inspection of higher throughput vs. accuracy, as well as lower cost vs. accu-
rates are about half (30 kernels/s) due to kernel feeding limitations. racy. Nevertheless, a sorter in this configuration might find many
Since the camera is directly linked to an FPGA, no personal com- other applications such as segregating mixed species of crops, such
puter is needed other than for compiling the FPGA software and as wheat and soybeans, weed seeds, broken/damaged corn kernels,
loading it onto the non-volatile memory on the FPGA board. The durum kernels with black-tip, durum virtuousness, and off color
FPGA then reads this memory at power up. This should make the sorghum due to fungal damage or weathering. These applications
FPGA/image sensor combination more robust and better able to will be a basis for future research.
withstand processing plant environments that can be hot, humid,
and dusty. The FPGA and camera do not require any ventilation for 4. Conclusion
cooling, so they can be enclosed in a sealed case. It is anticipated
that the rotary DIP switches would need to be adjusted periodically Simple image processing can be executed in hardware on FPGA
to account for lighting fluctuations. This is not a time-consuming chips directly linked to image sensors. This combination makes
18 T. Pearson / Computers and Electronics in Agriculture 69 (2009) 12–18

an economical system for the inspection of agricultural products, References

which until now has not been reported. While commercial color
sorters can do an excellent job of removing highly discolored prod- Bennedsen, B.S., Peterson, D.L., 2004. Identification of apple stem and calyx using
unsupervised feature extraction. Transactions of the ASAE 47 (3), 889–894.
uct, their accuracy may not be desirable when the defect is very Delwiche, S.R., Pearson, T.C., Brabec, D.L., 2005. High-speed optical sorting of soft
small or the color difference is slight. This study showed that the wheat for reduction of deoxynivalenol. Plant Disease 89 (11), 1214–1219.
image sensor/FPGA combination can improve sorting accuracy of Kumar, P.A., Bal, S., 2007. Automatic unhulled rice grain crack detection by X-ray
imaging. Transactions of the ASABE 50 (5), 1907–1911.
these types of defects or color differences, as illustrated with pop- Maxfield, C.M., 2004. The Design Warrior’s Guide to FPGAs. Newnes Press, Burling-
corn damaged with blue-eye and separation of red and white wheat. ton, MA.
Since all processing is done in hardware where many image pro- Pasikatan, M.C., Dowell, F.E., 2003. Evaluation of a high-speed color sorter for seg-
regation of red and white wheat. Applied Engineering in Agriculture 19 (1),
cessing steps are performed in parallel, kernel throughput rates are 71–76.
nearly double from what has been accomplished so far using tradi- Pearson, T.C., 2006. Low-cost bi-chromatic image sorting device for grains. ASABE
tional cameras with processing performed on personal computers. Paper No. 063085. ASABE, St. Joseph, MI.
Pearson, T.C., Brabec, D.L., Haley, S., 2008. Color image based sorter for separating
Parts for the system are lower in cost and physically more robust
red and white wheat. Sensing and Instrumentation for Food Quality and Safety
than systems using personal computers, so they might be more 2008 (2), 280–288.
suitable for processing plant environments. Phillips Semiconductor, 2000. The I2 C-Bus Specification. Document # 9398-393-
40011, Phillips Semiconductor Inc., Amsterdam, the Netherlands.
Takayanagi, I., 2006. CMOS image sensors. In: Image Sensors and Signal Processing
Acknowledgements for Digital Still Cameras. Taylor and Francis Group, Boca Raton, FL, pp. 143–178.
Wang, N., Zhang, N., Dowell, F.E., Pearson, T.C., 2004. Determination of durum wheat
Mention of trade names or commercial products in this publi- vitreousness using transmissive and reflective images. Transactions of the ASAE
48 (1), 219–222.
cation is solely for the purpose of providing specific information Yamada, T., 2006. CCD image sensors. In: Image Sensors and Signal Processing for
and does not imply recommendation or endorsement by the U.S. Digital Still Cameras. Taylor and Francis Group, Boca Raton, FL, pp. 95–142.
Department of Agriculture.