The Next Photography Revolution

Here comes a digital-camera chip that could change everything

"It's easy to have a complicated idea," Carver Mead used to tell his students at Caltech. "It's very, very hard to have a
simple idea."
The genius of Carver Mead is that over the past 40 years, he has had many simple ideas. More than 50 of them
have been granted patents, and many involved him in the start-up of at least 20 companies, including Intel. Without
the special transistors he invented, cell phones, fiber-optic networks, and satellite communications would not be
ubiquitous. Last year, high-tech high priest George Gilder called him "the most important practical scientist of the
late 20th century."
"Nobody," Bill Gates once said, "ignores Carver Mead."

Digital cameras have relied on image sensors that can't do what color film does: record all three 
primary colors of light at each point in the image. Instead, each light­sensitive point in the sensor 
measures just one color—blue, green, or red—and complicated software in the camera calculates the 
missing colors. Foveon's breakthrough X3 chip solves the problem with a three­layer design that 
captures red, blue, and green light at each point. To demonstrate quality differences, the monarch 
butterfly on this page was photographed with three cameras: an $1,800 Sigma SD9 with an X3 chip; a 
$300 Nikon Coolpix 2500; and a $2,300 Nikon 35 mm F5 film camera. Insets show magnified detail from 
each camera's image.

And now one of Mead's simplest ideas—a digital camera should see color the way the human eye does—is poised
to change everything about photography. Its first embodiment is a sensor—called the X3—that produces images as
good as or better than what can be achieved with film. That would make the X3 the most important advance in
photography in nearly 70 years, but the long-term implications are even richer. In a year or two, you will be able to
pack a true hybrid camera on vacation. It will take high-resolution stills, or upon the flip of a switch, it will take full-
length, full-motion video far exceeding the capabilities of present-day hybrid cameras. In the long run, X3
technology could even make cell-phone video sharp enough to project onto a big-screen TV, which would make
dandy travelogues to send back to the folks at home, or enhance collision-avoidance systems in automobiles, or
improve robot vision.
X3 is the latest and most innovative product from Foveon Inc., the Silicon Valley digital-imaging company that
Mead, 68, founded in 1997. Named for the fovea centralis—the part of the human retina where vision is sharpest
and most color perception is located—Foveon took as its mission another radically simple idea Mead loves: "Use all
the light."
Don't cameras already use all the light that enters the lens? Film cameras do, but digital cameras, with few
exceptions, don't. As Mead puts it, "They throw away two-thirds of the light." That makes sense only if you
understand how a typical image sensor works. It's basically a rectangle of silicon on which millions of microscopic
light-sensitive pixels (technically they're not pixels, but that's what these light-sensing points have come to be called
in the digital-camera business) are arranged in a grid. Pixels can't sense color. So a checkerboard of tiny red, green,
or blue filters must be bonded to the surface of the sensor so that each pixel lets in one of the three primary colors of
light. In so doing, it blocks out the other two.
By comparing each pixel's single-color reading with that of its neighbors, software can derive the values of the
two missing colors at each site. That takes approximately 100 calculations per pixel. In a four-mega-pixel camera, a
size commonly available today, that adds up to a lot of number crunching. The process is called interpolation, and
Mead has a less kind name for it.
"It's a hack," he says. "They have to do all this guesswork to figure out what they threw away. They end up with a
lot of data, but two-thirds of it is made up. We end up with the same amount of data, except ours is real."
That is because X3 does what until now only film has been able to do: in one exposure, on one image plane,
measure all three primary colors of light at every point on the picture. By doing so, it does away with the bugaboo of
so-called mosaic sensors, which often guess wrong, especially at the edges of complicated patterns, introducing
moiré effects and jagged color errors called artifacts.
Sensing all three colors at each pixel sounds simple, but more than one industry analyst has described it as "the
holy grail" of digital photography. "Engineers have been trying to solve this since the earliest days of digital
imaging," says Alexis Gerard, publisher of The Future Image Report. Phil Askey, whose exacting equipment tests on
his Web site, dpreview.com, are must reading in the trade, says, "This could be the first sensor to truly surpass film."

The only camera to contain an X3 sensor now is called the Sigma SD9, a single-lens reflex with a price tag of
$1,800 (not including the lens). But about this time next year, point-and-shoot cameras should be available from
other manufacturers with X3 technology. They will have chips with slightly less than half as many pixels as the chip
in the Sigma and sell for about $500. To be sure, Foveon will not find it easy to elbow its way into a market heavily
committed to existing technology. But it has some influential advocates, including Microsoft's Gates.
X3 is based on a well-known property of silicon: It absorbs different wavelengths, or colors, of light at different
depths from its surface. A standard wafer of pure crystal silicon—the polished
disc, five to eight inches in diameter, on which most microchips are made—is
about 1/25 of an inch thick. The absorption of visible light takes place within
1/10,000 of an inch of the surface. If you think of that 1/25 of an inch of silicon
as if it were a place in the ocean where the water is 1,000 feet deep, then all the
light absorption would be taking place within two or three feet of the surface. At
that scale, a human hair would be about 50 feet thick.
What Foveon has done is imbed a sandwich of three light sensors within that
first 1/10,000 of an inch. How they do it is a guarded trade secret, but the
principle is basic physics. Blue light, which has the shortest visible wavelength,
about 1/50,000 of an inch, is absorbed closest to the surface. Green light, which
has a longer wavelength, penetrates a little deeper. Red light, with the longest
wavelength, about 3/100,000 of an inch, burrows down farther before it is
absorbed. As the photons strike the silicon atoms, electrons are released. These
create electrical charges that the sensors measure.
It took film almost a century to figure out the best way to do color. But once
Carver Mead says, "The eye itself has 
Kodachrome was perfected in 1935, competing schemes largely faded away. By
taught us that remarkable things can 
devising the simplest and most reliable solution to the problem—a three-layer be accomplished by building 
emulsion—Kodak won the color war. Since then, color film has undergone intelligence into the image plane. An 
many refinements, and other companies have grabbed significant market share. intelligent image plane gives you 
But the three-layer emulsion is still gospel. higher quality photography with less 
"Exactly the same thing is going to happen with electronic capture," says demanded of the user."
Mead. "X3 is going to be the surviving image-capture technology. There's no
question about it."
If the rest of the industry isn't quite ready to throw in the towel, it's because the technology that Mead is up
against has been king of the hill, despite all its flaws, almost since it was invented.
If you have ever bought a video or digital camera, you have probably seen the letters CCD stamped on the box.
They stand for charge-coupled device. Versions of the CCD were invented independently at Bell Labs and Philips
Electronics about 33 years ago. They produced much better images than could be obtained with other kinds of solid-
state sensors, and they took off. Specialized long-exposure CCDs have long proved invaluable in astronomy, and the
CCD put video cameras and digital still cameras under countless Christmas trees.
If the CCD was the hare of the digital-imaging race, something called CMOS, for complementary metal-oxide
semiconductor, was the tortoise. CMOS is a sophisticated process developed in the 1960s that produces chips with
many transistors. Without it, X3 would not have happened. Although CMOS chips didn't make good images at first,
they made terrific microcircuits and became the backbone of the computer revolution. CMOS was the technology
Gordon Moore, the first president of Intel, had in mind when he made his famous 1965 prediction—subsequently
known as Moore's law—that transistor density on a chip would double every year. Before he refined his prediction,
changing it to doubling every 18 months, Moore consulted with an expert in circuit miniaturization—brilliant young
Caltech professor Carver Mead.

Mead has been fascinated by electricity since he was a child in Big Creek, California. His father worked in a
hydroelectric plant that supplied power to Los Angeles, so Mead got to watch as new generators were brought
online. He was awestruck by the immensity of the machinery and the force that turned it. He earned his bachelor's,
master's, and Ph.D. degrees at Caltech, and became a renowned professor of engineering and applied science. With
Caltech's Richard Feynman, the Nobel laureate, and biophysicist John Hopfield, he designed a course in 1986 called
The Physics of Computation, which became an instant classic, and its principles are still taught by its creators'
academic descendants.
In 1969, Mead took Moore's law to a new level. Miniaturization, he said,
would make it possible to build very-large-scale integrated (VLSI) circuits. They
weren't large in size—in fact, they became smaller and smaller—but they
incorporated more and more transistors on a single chip as the transistors
themselves shrank. Mead predicted that transistors would eventually get to be as
small as six-millionths of an inch across, or .15 micron. (Coincidentally, that is
the size of the transistors in today's state-of-the-art Pentium 4 microprocessor,
made by Intel. Foveon's X3 is right behind it, at .18 micron—the smallest
transistor in an image sensor.) Mead laid out an innovative set of concepts for
designing VLSI circuits, which became the industry standard as his students
went out into the work world. Dick Lyon, now Foveon's chief scientist, was one
of those students. "Carver taught a generation of engineers how to cope with the
complexities of millions of transistors on a chip before anyone believed there
would ever be millions of transistors on a chip," Lyon says.
In the 1980s, Mead turned his attention to an even more complex and Most sensors in point­and­shoot 
ingenious kind of circuitry: the human nervous system. It is clearly the most digital cameras are smaller than a 
successful computing system of all time, and it is an analog, rather than a digital, fingernail, but the X3 chip is closer to 
system. That part intrigued Mead the most. Instead of registering information in the size of a frame of 35 mm film. So 
digital strings of ones and zeros, an analog system, such as a retina or an ear, are the sensors in other top­of­the­
measures a continuum of values. Digital systems in the technological world line digitals, but the X3 is complex to 
were quick and complex; analog systems were slow and simple. Mead thought manufacture because of its three 
he could learn a lot by studying—and attempting to model in silicon—the layers and its transistor density. "If 
natural masterpieces of human analog design. you were to buy a Foveon chip," says 

This was such a new field, it had no name. So he gave it one: neuromorphic Chris Joyce, director of process 
technology at National 
electronics. And it grew into businesses. His study of the cochlea was
Semiconductor, "you could 
instrumental in the founding of Sonic Innovations, a digital hearing aid
company. Synaptics, which he cofounded in 1986, created the laptop TouchPad. deprocess it and figure out what 
we've done. But you wouldn't be able 
Foveon was spun off from Synaptics and National Semiconductor, a chip
manufacturer, to pursue the ideal of a digital image sensor that would see in full to figure out how we've done it."
color, like the human eye.
A slender, energetic man with a trim Vandyke beard, Mead has a disarming, gentle speaking voice and a face that
recalls both David Carradine in Kung Fu and Richard Kiley in Man of La Mancha. He rarely raises his voice, but
when something gets under his skin, you'll know it. "The more I learned about human vision," he says, "the more it
was clear that what these mosaic sensors were doing was introducing artifacts into the image. It was one of those
things that becomes so massively annoying that after a while you think you ought to go do something about it. It was
clear that the way image sensors worked was brain-dead. I talked to a lot of people, and nobody got it. So I finally
said: 'That's not a problem. That's an opportunity.'"
Among Mead's many talents is putting together a brain trust. From National Semiconductor, he recruited an
iconoclastic and fertile-minded inventor and electrical engineer named Dick Merrill to be his design wizard. His
choice for chief scientist, Dick Lyon, had worked with him on several neuromorphic projects. Mead's artificial-retina
project at Caltech had benefited greatly from ideas Lyon had developed in inventing the optical mouse at Xerox's
Palo Alto Research Center in 1980.
At the outset, X3 wasn't even a pipe dream. Foveon's first cameras were based on patented prisms that split
incoming light into its primary colors, directing the light into three separate sensors. The system produced
extraordinary images but was cumbersome (the camera had to be attached to a laptop) and costly. So one of Lyon's
first tasks was to sift through a pile of ideas that Merrill had written and see if any of them seemed promising.

One did. It proposed a way to build a sensor by taking advantage of the light-absorbing qualities of silicon.
Merrill had hit upon this method by accident. Back at National, he had built a sensor that didn't separate color the
way he expected it to. "So, not knowing much about sensors, I decided to look into why that was the case," Merrill
recalls.
Then he pretty much forgot about it, until the day Lyon came by asking if he really thought it could work. Merrill
thought it could, so they took it to Mead, who was skeptical. It turned out Mead had experimented with exactly that
concept at Caltech and found the color separation that silicon achieved was too indistinct to be useful for
photography.
Merrill and Lyon "massaged it around," as Lyon says, and brought a revised design back to Mead, who was
encouraged. They conducted simulations, but the color separation was still fuzzy. "We were tearing our hair out
about where to go with that," Lyon recalls, "when Merrill came up with a much more radical structure for how to
make the thing."
"Once we worked through the science of it, and Merrill and Lyon convinced
me it was fundamentally sound, I've never doubted," Mead says. "We basically
bet the company on it."
Now the ball was in Lyon's court. When he got to Foveon, he realized that the
one expertise the company was missing was optics and color theory. "So I
basically taught myself from books what I needed to know," he says. Lyon's
colleagues marvel at his polymathic abilities, not to mention his large collection
of slide rules, including a working seven-foot-long Pickett that he hung above
the bookshelves in the company library.
Lyon believed he could find a way to use the soft color separation in silicon
advantageously. The reason he thought so, as he reminded the others, is that "the
eye itself does it that way. The color sensitivities of the cones in the retina are
not sharply defined. On a graph, they would look like smooth overlapping
curves."
In clean­room bunny suits, chief 
scientist Dick Lyon (left) and chief 
In Merrill's new architecture, the top, or blue, sensor would pick up a little green designer Dick Merrill play with early 
and red light as those rays of light passed through. The middle, or green, sensor Foveon cameras. Merrill sees his job 
would pick up the leftover blue and a little red, and the bottom, or red, sensor as "making a list of all the reasons 
would pick up the leftover green and the last vestiges of blue. All digital people don't already own a digital 
cameras transform the raw measurements that come off the sensor into a set of camera and knocking them down one 
numbers called a matrix, which allows a computer monitor to display them as by one."
images.
The X3 needed much bigger matrix numbers than most sensors, and that made highlights and shadow areas harder
to render. "Even with film, it's a challenge to produce good color in highlights and shadows," Lyon says.
"Professional photographers know how to light a scene to work around that. But the issue for us was getting good
pictures when shot by amateurs, where the lighting is really challenging. That's what kept me up at night."
Mead's goal of "making the image plane more intelligent" hinged on more than measuring all the colors. The
vision was of a sensor that could be used to create video as well as stills—and, unlike current cameras, do both well
—and eventually take on tasks such as focus and exposure, now handled by expensive supplementary chips.
There is no way a CCD can do all that. It has its hands full merely schlepping the electrical charges from pixel to
pixel in a process that is usually described as a "bucket brigade." By applying a positive voltage to the negatively
charged electrons in the pixel, the CCD attracts the electrons and essentially hands them off from one pixel to the
next. There are different patterns, but typically all the pixels in the bottom row of the array are shuttled off, then the
charges from the line above drop down, something like a row of blocks in a game of Tetris. It's not only a relatively
slow process, but applying all those successive voltages also makes CCDs power hogs, which is one reason users of
digital cameras have to replace or recharge their batteries so often.
 So this is where the tortoise re-enters the story. Only about half the surface area of any given chip is actually
devoted to light sensing. The rest contains supporting circuitry, which in a CCD mans the bucket brigade. By using
state-of-the-art CMOS technology, Foveon was able to pack more transistors into each pixel, which enables the chip
to do more things. Equally important, CMOS architecture allows the camera software to communicate directly with
the pixels and read them out in parallel—meaning all at once—rather than serially, one after the other, as in a CCD.

The greatest of all parallel processors is the human nervous system. The signals generated by the rods and cones
in the eye's retina, for example, don't stand in line waiting to get into the optic nerve like cars backed up at a tunnel.
"They're actually processed rather extensively right in the retina," says Mead. The brain samples the signals in
parallel, taking information as it comes.
"The nervous system always operates on partial information," Mead
explained in an interview with the technical journal EE Times. "Its basic
assumption is that you start with no information, and anything beyond
that is something, and it is a heck of a lot better to have something
rather than nothing. But what the digital paradigm is based on is, 'I've
got to have all my arguments before I start any operation'—the opposite
end of the spectrum!" The significance of X3 is that it advances the art
of parallel processing and brings us closer to an age of truly smart
machines.
CCDs are dinosaurs, and it isn't only Carver Mead who thinks so. A
2002 report by the market research firm In-Stat/MDR projected that
production of CMOS sensors would draw even with CCDs in 2004 and
pull away thereafter. Much of the growth will be in the lower end of the This photograph, taken with the Sigma SD9 
market—in cell phones, Webcams, and toys, which are already popular using a Foveon chip, attains levels of 
in Japan. But Foveon was encouraged last year by Canon's introduction sharpness and color accuracy usually seen 
of two high-end CMOS cameras, both of which use a filter mosaic. only in medium­format cameras that use 120 
"They've made the world safe for CMOS," quips Lyon. mm film, which has a resolution about twice 
Dinosaurs, of course, ruled the earth for a long time. CCD makers that of 35 mm film.
have managed to develop a number of ingenious processing strategies to
compensate for the system's inherent weaknesses. And they've managed to bring down prices steadily while
increasing pixel count. Even though there's more to producing good images than stuffing greater numbers of smaller
and smaller pixels into a camera, smart marketing has established the mega-pixel as the sexy equivalent of
horsepower.
Mead has another paradigm in mind: "Whenever a radically new technology has developed, a new major
company has come out of it. When the transistor came along, we got Texas Instruments. When the integrated circuit
came along, we got Intel. When we got microprocessors and personal computers, we got Microsoft.
"That's the way I see Foveon. It doesn't mean we're going to put the others out of business. We have no intention
of doing that. They're becoming our customers. We're forming alliances.
"We're not going to be an Apple," he adds. "We're not going to turn ourselves into an island. We're going to be
more like a Microsoft or an Intel." It's amazing, come to think of it, how much Mead does look like Don Quixote.