Segmentation of Images of Lead Free Solder

Segmentation of Images of Lead Free Solder Matthias Scheller Lichtenauer, Silvania Avelar, and Grzegorz Toporek Swiss Federal Laboratories for Materials Testing and Research (EMPA) Laboratory of Media Technology Ueberlandstrasse 129, 8600 Duebendorf, Switzerland {matthias.scheller,silvania.avelar}@empa.ch Abstract. We present two approaches to segment metallic phases in images of lead free solder joints. We compare the results of a method without user interaction with another one extrapolating information from a relatively small set of user labeled pixels. The segmented images provide statistical data of spatial characteristics of phases to serve as input in numerical models of solder joints. 1 Introduction Lead has been replaced in the last decade as a joining material in soldering due to its toxic properties. The relatively new lead free soldering can still be improved. Some properties sought for are low melting point and mechanical properties like resistance to tear stress and tensile strength. Data about the size, form and spatial distribution of regions of homogeneous materials (phases) is needed for numerical models that predict mechanical properties at the scale of solder balls (100 − 1000
m) in contemporary electronic devices [1]. Methods to segment images of solder joints in regions containing specific materials are required in order to create statistical data of spatial distribution of phases. In materials science, related work has recently appeared. Sidhu and Chawla [2] described the use of numerical models to calculate solder deformation behavior. Erinc et al. [3] analyzed fatigue fracture of solder joints using images. Kang et al. [4] investigated soldering as a process in which size and form of phases can vary widely with cooling rates. Suitable segmentation methods for images of lead free solder have to be adaptive to cope with the variety of size and form of phases. There are three major challenges in detecting phases in images of solder. First, the appearance of a phase in the probe depends on probe preparation. Second, the imaging system may map the probe unequally to an image, due to depth of focus or to planarity of the probe. Third, methods to determine the chemical composition at a given spot have limitations in space, be it resolution or field of view, or limitations in the time needed to sample the probe. There is the risk to be inaccurate focussing on regions that are not representative enough or because two phases have the same look in an image, although they are chemically different. We aim to segment images of lead free solder joints such that segmented regions correspond to phases as human experts identify them. We combine light microscope images with different illumination and imaging methods. A. Elmoataz et al. (Eds.): ICISP 2010, LNCS 6134, pp. 165–172, 2010. c Springer-Verlag Berlin Heidelberg 2010  166 M.S. Lichtenauer, S. Avelar, and G. Toporek We apply two segmentation approaches and compare their results, namely hierarchical segmentation, as Ibrahim et al. [5] use, and the learning methodology of support vector machines (SVM) [6,7]. Hierarchical or decision tree methods partition the set of pixels to be segmented with each step, starting either with the pixels that can be reliably segmented or reducing the input size of computationally expensive steps to regions of interest. The SVM method maps the data to a higher dimensional feature space and constructs an optimal separation hyperplane between two classes in this space [6]. Both approaches are used for classifying the image data into categories, which lead to the segmentation of phase regions in the image [8]. We present the image capturing process of solder probes in the next section. Section 3 discusses the segmentation approaches. A comparison of the segmentation results is shown in section 4. We give our conclusions in section 5. 2 Image Capturing We have used Leica DC500 camera on Leica DMRX light microscope to take images of solder probes. The parameters of this system are listed in Table 1. We corrected the evenness of illumination with the built-in function of Leica Firecam Version 1.7.1 on Mac, which was also used for image capturing. The different resolution modes of the camera and magnification factors of the light microscope objectives allowed us to choose between alternative configurations with approximately the same pixel sampling interval. Lateral resolution of the optical system (λ/2NA) is calculated at λ = 380nm. The respective depth of focus δz is estimated at the same wavelength [9]. The structures we visualize are in the scale of 1
m, requiring at least objective 10× to achieve the Nyquist sampling frequency limit. We took microscopic images from metallic probes constituted of Cu joined with Sn solder that was reinforced with 4w.t.% Ag and 0.5w.t.% Cu. We will refer to this composition as SAC405. The knowledge on how phases look like is taken by scanning electron microscopy (SEM) and energy-dispersive spectroscopy (EDS/EDX). In most of the light microscope images human experts were able to visually distinguish five categories, namely copper, eutectic, βSn, void, and intermetallics (Cu6 Sn5 , Cu3 Sn and Ag3 Sn). Table 1. Areas which are visible in optical microscope and pixel sampling interval on the probe for the resolution modes of the camera. NA stands for numerical aperture. Objective 10× 20× 40× 50× 100× NA 0.20 0.45 0.60 0.86 0.90 min δz 4.75
m 0.88
m 0.47
m 0.19
m 0.16
m λ/2NA 1.80
m 0.81
m 0.61
m 0.44
m 0.42
m Field of View 1.360 × 1.057mm 0.680 × 0.529mm 0.340 × 0.270mm 0.272 × 0.215mm 0.136 × 0.108mm 3900 px 352nm/px 176nm/px 89nm/px 70nm/px 35nm/px 2600 px 528nm/px 264nm/px 133nm/px 106nm/px 52nm/px 1300 px 1.06
m/px 528nm/px 266nm/px 212nm/px 104nm/px Segmentation of Images of Lead Free Solder A raw probe, illumination not corrected 167 A polished probe, corrected illumination Fig. 1. Visible differences due to polishing, etching and illumination correction 2 1 0 −1 −2 Height Profile (micrometer) Depth of field objective 10x (NA=0.20) Depth of field objective 20x (NA=0.45) −3 −4 −5 0 250 500 750 1000 Fig. 2. Surface height profile compared to depth of focus of the light microscope. Both dimensions are in
m. With objective 20x or higher, some parts of the image will be out of focus even if the sample is optimally aligned. To improve visibility of phases, samples were polished and then etched for a few seconds. Etching creates a relief. The edges are visible in dark field illumination, but height differences increase depth of focus needed. There are known tradeoffs between magnification and depth of focus with light microscopy. Figure 1 shows on the right side an example of an image in bright field illumination taken with objective 50×, covering a visible field of 272 × 215
m. The image is focussed on the interface between the soldered copper and the solder, while the rest of the solder seems blurred. Surface line profile measurements of two probes using Ambios XP-1 profilometer at different locations revealed that the step height between the copper and the solder is about 1 to 2
m after polishing and etching, while the solder surface height varies in the range of ±1
m, which is more than the light microscope can keep in focus with objectives magnifying larger than 10× (Figure 2). We spectrally measured reflectance of all different phases in a probe with Leica MPV-SP microspectrometer mounted on Leica DMRX optical microscope. Except for copper, the phases do almost not show differences in hue or saturation, some not even in brightness. The reflectance of an X-Rite white standard is 168 M.S. Lichtenauer, S. Avelar, and G. Toporek White Sn Eutectic Cu Sn 10000 6 Reflection spectra of phases 5 Cu Ag Sn 3 5000 0 400 450 500 550 600 Wavelength 650 700 Fig. 3. Reflection measured with Leica MPV-SP microspectrometer. Intensity values are in relative units of this microspectrometer, as reflectance of the phases exceeds reflectance of the X-Rite white standard. way below that of all phases, supporting our assumption that the reflection of the phases was mostly specular and surfaces were mostly oriented parallel to the image plane (Figure 3). In optical microscopy using dark field illumination, smooth surfaces parallel to the image plane appear dark, whereas rough surfaces or slopes appear bright. We compared bright field and dark field images from the same field of view with regard to surface relief. Although the reflectance spectra of eutectic and Ag3 Sn intermetallic phase did not much differ, they can be visually distinguished in images by their different surface relief in dark field illumination and their brightness variation in bright field illumination. Hence, we provided paired bright field and dark field images (Figure 4). 3 Image Segmentation In order to generate good statistical data, we had to segment enough unconnected components of each phase. For users, interactively labeling each of the components is tedious and error prone. So we looked for segmentation approaches able to identify phases with minimal user interaction, and able to extrapolate information from a small samples set, even to several different images of a probe. Bright field illumination Dark field illumination Fig. 4. Paired light microscope images of a SAC405 probe Segmentation of Images of Lead Free Solder 169 As input for our segmentation methods, we provided two paired images in bright and dark field illumination showing exactly the same field of view sampled with 2600×2060 pixels. For comparison of results, we also provided the positions of 250 pixels to be labeled by the two methods. We also gathered the labels of the selected pixels for evaluation of results. To gather those 250 ground truth points, we used an image of a probe taken with bright field illumination which was displayed on a computer screen. Human experts labeled pixels by choosing one of five phases and clicking on its pixels. Some phases were quite rare, hence we discarded to randomly select pixels to be labeled and took care that there was an equal number of sample points for all phases (Figure 5). The decision tree approach calculates by thresholding methods a set of masks for flatness (gray level of dark field image), redness (difference of red and blue channels), and for brightness (gray level of bright field image). The combination of these masks is used to segment the phases. For the support vector machine, local features of the image are extracted from individual image channels by means of a sliding window. The brightness histogram [10] and the output of a bank of Gabor filters [11] are calculated at each window position to extract texture information. Features are gained by computing the mean of the output in the frequency domain and filtering the image texture with 4 orientations and 4 scales. We used the SVM with a radial basis function (RBF) kernel in our experiment. For training the SVM, we have provided other 250 labeled pixels of the five phases. The image in bright field illumination was tested to look for patterns. 4 Results The outputs of the two segmentation approaches on the images were compared to the ground truth. The result of the hierarchical approach based on pixelwise thresholding technique has hit rates between 72 and 96%. The recognition of the copper substrate is reliable enough to estimate the form and thickness of the interfacial intermetallic layer with the help of a distance transform. Using SVM, the prediction of phases for our ground truth samples has 83.2% accuracy (208/250 points). Misclassified phases were mostly near the boundary of different phases and in some areas of similar texture from copper and βSn. SVM performed better in distinguishing the eutectic and intermetallic phase, but worse in recognition of copper (Figure 6). We make here some runtime considerations. We assume a pixel resolution of 1
m2 . The decision tree classifier is able to segment more than 2600 × 2060 pixels a minute on a 2.8GHz Intel Core2 Duo. The extraction of image features to the SVM classification is more computationally intensive than the method itself. It takes about five minutes to take the two pictures in dark field and bright field once the probe is etched and polished. Using SEM, the same polishing and etching technique could be used, so time for probe preparation would not differ. But it would take at least about 100 hours to sample the same surface 170 M.S. Lichtenauer, S. Avelar, and G. Toporek + • ∗ ⋄  copper βSn eutectic intermetallic Legend copper eutectic intermetallics βSn void void Fig. 5. Ground truth. Zoomed examples are adjusted to increase visibility in print. Decision tree Ground truth Result Cu Eu Im Vo copper Cu 45 2 3 0 eutectic Eu 0 36 0 1 intermetallic Im 0 3 38 0 void Vo 0 0 2 48 βSn Sn 0 0 3 0 Total 45 41 46 49 SVM Sn 0 13 9 0 47 69 Hit (%) 90 72 76 96 94 Ground truth copper eutectic intermetallic void βSn Total Cu Eu Im Vo Sn Result Cu Eu Im Vo Sn 32 1 5 5 7 1 45 0 3 1 1 2 43 2 2 0 2 0 48 0 6 0 3 1 40 40 50 51 59 50 Hit (%) 64 90 86 96 80 Fig. 6. Phase-confusion resulting from the two segmentation approaches at one sample per
m2 with SEM using energy dispersive spectroscopy (EDS), although EDS technique would yield more reliable results. Using EDS to check phase attribution is feasible in acceptable time, since the areas to check are smaller. Segmentation of Images of Lead Free Solder 5 171 Conclusions and Future Research We presented two approaches to segment phases in solder joints images taken with light microscopy. These segmentation methods can be used to generate statistical data as input for numerical methods limited by the accuracy and resolution needed. The results can complement information of scanning electron microscopy using energy dispersive spectroscopy. Scanning electron and atomic force microscopy (AFM) could also be combined as imaging methods. The SVM method was mostly based on texture features. If texture can not be captured by the imaging methods due to focus problems or probe preparation, the prediction accuracy decreases. Ground truth points have to be chosen in regions with precise phases and near to the boundary of different phases. The use of new combinations of features, as well as increasing training data size, needs to be further investigated, in order to improve the SVM prediction accuracy for different images of a probe. The decision tree method used the color information contained in bright field as well as the flatness information available in dark field, but no texture information as SVM. A combination of both feature sets could lead to better results. Since SEM images are grayscale and the feature set extracted for use in SVM relies on grayscale information of the light microscope images only, a combination of SEM with light microscope images could also profit from the additional information of color and flatness. In order to get spatial characteristics of phases, we aim also to construct a three-dimensional model of phases. It is possible to gain a 3D model by a sequence of 2D cuts or by estimating it from one single 2D cut. Regardless which method we choose, the quality of the results depends on the quality of the segmentation. Future work includes the combination of image data with spatially correlated topographical measurements of a probe as well as the construction of threedimensional structures from two-dimensional cuts. References 1. Sivasubramaniam, V., Galli, M., Cugnoni, J., Janczak-Rusch, J., Botsis, J.: A study of the shear response of a lead-free composite solder by experimental and homogenization techniques. Journal of Electronic Materials 38(10), 2122–2131 (2009) 2. Sidhu, R.S., Chawla, N.: Three-dimensional (3D) visualization and microstructurebased modeling of deformation in a Sn-rich solder. Scripta Materialia 54(9), 1627–1631 (2006) 3. Erinc, M., Assman, T., Schreurs, P., Geers, M.: Fatigue fracture of SnAgCu solder joints by microstructural modeling. International Journal of Fracture 152(1), 37–49 (2008) 4. Kang, S.K., Choi, W.K., Shih, D., Henderson, D.W., Gosselin, T., Sarkhel, A., Goldsmith, C., Puttlitz, K.J.: Ag3Sn plate formation in the solidification of nearternary eutectic Sn-Ag-Cu. JOM Journal of the Minerals, Metals and Materials Society 55(6), 61–65 (2003) 172 M.S. Lichtenauer, S. Avelar, and G. Toporek 5. Ibrahim, A., Tominaga, S., Horiuchi, T.: Material classification for printed circuit boards by spectral imaging system. In: Trémeau, A., Schettini, R., Tominaga, S. (eds.) CCIW’09. LNCS, vol. 5646, pp. 216–225. Springer, Heidelberg (2009) 6. Vapnik, V.N.: The Nature of Statistical Learning Theory. Springer, New York (1995) 7. Bruzzone, L., Carlin, L., Melgani, F.: A Multilevel Hierarchical Approach to Classification of High Spatial Resolution Images with Support Vector Machines. In: Proceedings of IGARSS, IEEE International Geoscience and Remote Sensing Symposium, pp. 540–543 (2004) 8. Jain, A.K.: Fundamentals of digital image processing. Prentice-Hall, Englewood Cliffs (1989) 9. Young, I.T., Zagers, R., van Vliet, L.J., Mullikin, J., Boddeke, F., Netten, H.: Depth-of-Focus in Microscopy. In: Proc. 8th Scandinavian Conference on Image Analysis, SCIA, Tromso, Norway, pp. 493–498 (1993) 10. Chapelle, O., Haffner, P., Vapnik, V.N.: Support Vector Machines for HistogramBased Image Classfication. IEEE Transactions on Neural Networks 10(5) (1999) 11. Jian, M., Guo, H., Liu, L.: Texture image classification using visual perception texture features and Gabor wavelet features. Journal of Computers 4(8), 763–770 (2009)