Glass Patterns and Artistic Imaging

Glass Patterns and Artistic Imaging Giuseppe Papari and Nicolai Petkov Institute of Mathematics and Computing Science, University of Groningen [email protected], [email protected] Abstract. The theory of Glass patterns naturally combines three essential aspects of painterly artworks: perception, randomness, and geometric structure. Therefore, it seems a suitable framework for the development of mathematical models of the visual properties that distinguish paintings from photographic images. With this contribution, we introduce a simple mathematical operator which transfers the microstructure of a Glass pattern to an input image, and we show that its output is perceptually similar to a painting. An efficient implementation is presented. Unlike most of the existing techniques for unsupervised painterly rendering, the proposed approach does not introduce ’magic numbers’ and has a nice and compact mathematical description, which makes it suitable for further theoretical analysis. Experimental results on a broad range of input images validate the effectiveness of the proposed method in terms of lack of undesired artifacts, which are present with other existing methods, and easy interpretability of the input parameters. 1 Introduction Computer aided generation of painting-like images is an interesting subfield of image processing and non-photorealistic rendering (NPR). The classic approach consists in generating a set of possibly overlapping brush strokes, which are rendered in a certain order on a white or canvas-textured background. There is a large variety of available painterly rendering algorithms (PRA), both unsupervised [1, 2, 3] and interactive [4, 5, 6]. Much effort has been made in order to model different painting styles [7, 8, 9] and to design efficient interactive user interfaces [5, 10], some of which deploy special-purpose hardware [10]. For a survey of these techniques we refer to [11]. One limitation of the aforementioned PRA is that they are not derived from general principles [9], due to the lack of a universally accepted mathematical model of the visual properties that distinguish paintings from photographic images. Consequently, many ’ad hoc’ computational steps are deployed, and several ’magic numbers’ need to be introduced (such as, for instance, the threshold of the Sobel edge detector deployed in [1] or the mixture parameter introduced in [12]). Moreover, as admitted in [6], these PRA may fail as soon as one of the intermediate operations does not perform well for some particular input image. In contrast, considerable improvement can be gained if one abstracts from the imitation of classical tools and focuses on the visual properties of the desired T. Wada, F. Huang, and S. Lin (Eds.): PSIVT 2009, LNCS 5414, pp. 1034–1045, 2009. c Springer-Verlag Berlin Heidelberg 2009  Glass Patterns and Artistic Imaging 1035 output. In this paper we consider three important aspects of painterly artworks: perception, randomness and geometric structure. The former stems from the fact that artists put in their paintings their personal interpretation of a given natural scene [13, 14]. Randomness is important because artists never make two identical paintings of the same scene. Geometric structure plays a central role in visual arts because people perceive geometric dispositions of objects in the space [15]. Moreover, artists may deliberately align brush strokes according to the principle of good continuation in order to enhance the perception of forms and to make their paintings more vibrant. A mathematical framework on which these concepts meet each other naturally is the theory of Glass Patterns (GP). A GP is defined as the superposition of two random point sets, one of which is obtained from the other by means of a small geometric transformation. Examples of such patterns are shown in Fig. 1. GP have drawn the attention of many researchers from different disciplines. Direct measurement of the neural activity in the V1 and V2 areas of the brain of primates indicates that, when GP are presented, neurons of the visual cortex strongly respond to the local orientation of dot pairs [16, 17]. Such responses are then processed by means of association fields, thus extracting long chains of collinear segments [18, 19]; such chains determine the geometrical structure perceived in GP. Circular and spiral structures in GP are the most salient geometries [20, 21, 22], indicating that the contour integration process that is performed by the visual system is more sensitive to closed structures [23]. This might be the basis of the gestalt principles of closure and praegnanz [23, 24]. Several computational models of the perception of GP have been proposed [23, 25, 26]. GP have also been studied from a purely mathematical point of view in terms of their macro and microstructure. The former concerns local dot density, which turns out to be lower in the center of the pattern. The related phenomenology has been exhaustively studied [27, 28], and it can be explained by the general theory of moiré effects [29]. On the other hand, microstructure concerns the orientation field that is induced by the strong correlation between the two superposed point sets that compose a GP. Microstructural properties of GP can be studied naturally in the framework of the dynamical systems theory [30, 31] and algorithms able to synthesize any microstructure have been provided [31]. Fig. 1. Glass patterns obtained by several geometric transformations. From left to right: rotation, isotropic scaling, expansion and compression in the horizontal and vertical directions respectively, combination of rotation and isotropic scaling, and translation. Note that translational GP are the least salient. 1036 G. Papari and N. Petkov In this paper, we propose a simple mathematical operator which transfers the microstructure of a GP to a natural image, thus resulting in a nice artistic effect. The method is validated by a wide range of experimental results and comparisons with well-estabilished PRA. 2 Mathematical Formalism In this section we review the mathematical formalism related to classic discrete GP (Section 2.1), then we extend it to the continuous case (Section 2.2), and we show how to transfer their microstructure to a photographical image (Section 2.3). 2.1 Discrete Glass Patterns Let v(r) be a vector field defined on IR2 and let us consider the following differential equation: dr = v(r) (1) dt We indicate the solution of (1), with the initial condition r(0) = r0 , by r(t) = Φv (r0 , t). For a fixed value of t, Φv is a map from R2 to R2 , which satisfies the condition Φv (r, 0) = r. Let S = {r1 , ..., rN } be a random point set and let be Φv (S, t)  {Φv (r, t)|r ∈ S}. Using this notation, we define the Glass pattern Gv,t (S) associated with S, v, and t as follows:  Gv,t (S)  S Φv (S, t) (2) Examples of GP generated by linear differential equations are shown in Fig. 1. More general vector fields give rise to more sophisticated geometries, which are related to the streamlines of v(r) (Fig. 2). 2.2 Continuous Glass Patterns (CGP) In order to extend the GP formalism to the continuous case, we first represent a point sets S with a binary fields bS (r) which, by definition, takes the value 1 Fig. 2. From left to right: vector field v(x, y) = [y 2 − 1 + 13 xy, 31 (y 2 − 1) − xy]⊺ , the trajectories which solve the corresponding differential equation ṙ = v(r), and a corresponding GP Glass Patterns and Artistic Imaging 1037 for r ∈ S and is equal to 0 for r ∈ / S. It is straightforward to see that the binary field associated with the superposition of two point sets S1 and S2 is equal to the maximum of bS1 (r) and bS2 (r). Therefore, from (2), we see that the binary field associated with a GP is equal to: bGv,t (S) (r) = max{bS (r), bS [Φv (r, t)]} (3) The generalization of (3) to the continuous case is straightforward: first, a continuous set of patterns bS [Φ(r, τ )], with τ ∈ [0, 1] is considered, instead of only two as in (3); second, any real valued random image z(r) can be used instead of a Poisson process bS (r). Specifically, a continuous Glass pattern (CGP) Gv,z (r) is defined as follows: (4) Gv,z (r)  max {z[Φv (r, τ )]} τ ∈[0,1] We indicate by Av (r) the arc of streamline r(t) = Φv (r, t) with t ∈ [0, 1], i.e. Av (r)  {Φv (r, t)|t ∈ [0, 1]}, and rewrite (4) more compactly as: Gv,z (r) = max {z(ρ)} (5) ρ∈Av (r) Examples of CGP are shown in Fig. 3, first row, where z(r) is random noise, in which the histograms of all images have been equalized for visualization purposes. These patterns are related to the vector fields v(r) = [y, −x]⊺ (rotation) and v(r) = [x, y]⊺ (isotropic scaling) and, as we see, CGP exhibit similar geometric structures to the corresponding discrete GP. 2.3 Cross-CGP Finally, in order to transfer the structure of a CGP to a color image I(r), we define the cross-CGP Cv {z(r), I(r)} as follows: Cv {z(r), I(r)}  I {ρ0 (r)} , ρ0 (r)  arg max {z(ρ)} ρ∈Av (r) (6) In other words, instead of directly considering the maximum of z(r) over Av (r), we first identify the point ρ0 (r) which maximize z(ρ), and we take the value of I(r) at that point. It is easy to see that if the input image I(r) coincides with z(r), cross-CGP coincide with CGP: Cv {z(r), z(r)} = Gv,z (r) (7) Examples of cross-CGP are shown in Fig. 3, second row, for a natural image I(r). As we see, both randomness and the geometric structure of a GP are present in a cross-CGP. Given v(r), z(r), and I(r), cross-CGP can be computed in a straightforward way by integrating (1) numerically, and by taking the maximum of z(r) over Av (r), as indicated in (5). We implemented (5) by means of an optimized version of the Euler algorithm whose pseudo-code is shown in Fig. 4. Our algorithm starts with a cross-CGP computed over a 1-pixel-long arc Av (r), and doubles the length of Av (r) at each iteration. Therefore, the computation time is logarithmic with the length of Av (r). 1038 G. Papari and N. Petkov Fig. 3. Examples of CGP and cross-CGP. First row: random image z(r) and CGP obtained from it, by means of the vector fields v(r) = [y, −x]⊺ and v(r) = [x, y]⊺ respectively. Their geometrical structure is analogous to the discrete case. Second row: an input image and cross-CGP related to the same random image z(r) and the same vector fields v(r). Cross-CGP succeeds in transferring randomness and geometric structure of a GP to an image. 3 Proposed PRA In this section, we describe a simple approach to add artistic effects to a photographic image I(r), as depicted in Fig. 5. The idea is to compute the cross-CGP Cv (z, I) of I(r), where z(r) is random noise and v(r) is a vector field extracted from the input image. In order to determine v(r), we first compute the orientation θσ (r) of the color gradient of I(r). Following [32], we define θσ (r) as the direction of the eigenvector associated with the maximum eigenvalue of the following matrix: Kσ (r) = 3   ⊺  ∇σ I (i) (r) ∇σ I (i) (r) (8) i=1 where I (i) (r) is the i − th color component of I(r) and ∇σ I (i) (r) is the result of the convolution of I (i) (r) with the gradient of a Gaussian function with standard deviation σ. Then, the vector field v(r) is chosen to have constant length a and to form a constant angle θ0 with the direction θσ (r) of ∇σ I, where a and θ0 are input parameters: v(r) = a[cos (θσ (r) + θ0 ) , sin(θσ (r) + θ0 )]⊺ (9) Glass Patterns and Artistic Imaging 1039 SubRoutine FastCrossCGP(I, z, Vx, Vy, N iter, h) %I = Input image I(r) %z = Random noise z(r) % Vx, Vy = x and y components of v(r) % N iter = Number of iterations %h = Step size of the Euler algorithm For i = 1 : N iter { I1 = evaluate(I, Vx, Vy, h); z1 = evaluate(z, Vx, Vy, h); For each (x,y) { if z(x,y) ≤ z1(x,y) I(x,y) = I1(x,y); } z = max(z, z1); Wx = evaluate(Vx, Vx, Vy, h); Wy = evaluate(Vy, Vx, Vy, h); Vx = Vx + Wx; Vy = Vy + Wy; } Return I; Fig. 4. Pseudo-code of the proposed implementation of cross-CGP. The subroutine ’evaluate’, which is not detailed here, takes in input three real valued functions I(x, y), Vx (x, y), and Vy (x, y), with (x, y) ∈ ZZ 2 , and a scalar h, and returns the function U (x, y)  I[x + h · Vx (x, y), y + h · Vy (x, y)], which is computed by means of bilinear interpolation. Fig. 5. Scheme of the proposed PRA The value of θσ (r) is undefined on points r for which the eigenvalues of Kσ (r) are equal. For such points we take by definition v(r) = 0. Finally, the output y(r) of the proposed operator is simply the cross-CGP associated with the input image I(r), the vector field v(r) defined in (9), and a random noise z(r): y(r) = Cv {z(r), I(r)} (10) An example of the output of the proposed operator is shown in Fig. 6 for the input image of Fig. 3, second row (921 × 660 pixel), for σ = 6, a = 18 and θ0 = π/2. As we see, the output presents curved brush strokes oriented along 1040 G. Papari and N. Petkov Fig. 6. Output of the proposed operator for the input image of Fig. 3, second row (a) (b) (c) (d) Fig. 7. (a) Input image, (b) output of the proposed operator, (c) noisy input image and (d) output of the proposed operator. Adding a small noise to the input image makes impressionist whirls visible on texture-less area too. Glass Patterns and Artistic Imaging 1041 Table 1. Values of the parameters used for the studied approaches, with each color component of the input image ranging between 0 and 1 Algorithm Proposed approach IR [1] AV [6] Parameter values a = 16, σ = 6, λ = 0.3, θ0 = π/2 l = 5, w = 1, ∆r = ∆g = ∆b = 0.06, ∆I = 0.15 Segm. levs. = 48, Enh. levs. = 96, Brush. Art. lev. = 40, α = 50% the object contours. It is easy to prove that for θ0 = π/2 the streamlines of v(r) are closed curves, thus, the brush strokes tend to form whirls which are typical of some impressionist paintings. The proposed operator does not introduce new colors to the input image. Consequently, brush strokes are not visible on non-textured areas (Fig 7b). However, this problem is solved by simply adding a small monochromatic random noise to the input image before computing the cross-CGP (Fig. 7c-d). Concerning the input parameters, a determines the length of the brush strokes, σ influences the smoothness of the lines traced out by the brush strokes and the (a) (b) (c) (d) Fig. 8. (a) An input image and outputs of (b) the proposed algorithm, (c) AV [6], and (d) IR [1] 1042 G. Papari and N. Petkov size of the impressionist whirls, and θ0 controls the angle between each brush stroke and the nearest edge. 4 Results and Comparison In this section, some experimental results are presented and commented. Our approach is compared with two of the most popular PRA, namely the impressionist rendering (IR) proposed in [1] and the NPR technique called artistic vision (AV) presented in [6]. IR consists in rendering overlapping rectangular brush strokes of a given size and orientation. In AV, curved brush strokes of different sizes (a) (b) (c) (d) Fig. 9. (a) An input image and outputs of (b) the proposed algorithm, (c) AV [6], and (d) IR [1] Glass Patterns and Artistic Imaging (a) (b) (c) (d) 1043 Fig. 10. Illustration of the influence of the parameter θ0 . (a) Input image and outputs of the proposed operator for (b) θ0 = π/2, (c) θ0 = π/4, and (d) θ0 = 0. are rendered by means of a more sophisticated segmentation approach. We show some results in Figs. 8-10; a larger set of examples is available online1 . Unless differently specified, we use the parameter values according to Table 1. The output of the proposed operator for the input images of Figs. (8-9)a is shown in Figs. (8-9)b. As we see, our operator effectively mimics curved brush strokes oriented along object contours, and the whirls that are present in contourless areas resemble some impressionist paintings. In Figs (8-9)c, the outputs of AV are shown for the same input images; though simulation of curved brush strokes is attempted, several artifacts are clearly visible, especially on flat areas. IR (Figs. (8-9)d) does not produce artifacts, but it tends to produce blurry contours and fails in rendering impressionist whirls. In Fig. 10, the output of the proposed operator is shown for different values of the parameter θ0 (θ0 = π2 , π4 , 0), which controls the angle between the brush strokes and the nearest object contour. As we see, different artistic effects can be achieved by varying the value of θ0 . Specifically, for θ = π/2, the strokes follow the object contours and form whirls in flat areas, while for θ = 0, the strokes are orthogonal to the contours and build star-like formations in flat regions. 1 http://www.cs.rug.nl/˜imaging/PSIVT2009 1044 5 G. Papari and N. Petkov Summary and Conclusions We use the theory of GP for unsupervised automatic painterly rendering. Specifically, we show that transferring the microstructure of a GP to a photographical image results in a nice artistic effect. The proposed algorithm is simple, can be implemented very efficiently, does not introduce ’magic numbers’ and, unlike many other methods, is expressed by a compact mathematical expression, which makes it suitable for further theoretical analysis. Due to its simplicity, input parameters are easily interpretable, and undesired artifacts that affect some existing algorithms are avoided. Moreover, the possibility to achieve considerably different artistic effects by simply changing the values of certain input parameters (such as θ0 , see Fig. 10) makes the method versatile and promising. This work improves a previous work of us presented in [33] in terms of a smaller amount of computational steps and input parameters. Experimental results on a broad range of input images validate the effectiveness of the proposed method. References [1] Litwinowicz, P.: Processing images and video for an impressionist effect. In: Siggraph, pp. 407–414 (1997) [2] Shiraishi, M., Yamaguchi, Y.: An algorithm for automatic painterly rendering based on local source image approximation. In: NPAR, pp. 53–58 (2000) [3] Li, N., Huang, Z.: Zhiyong Huang. Feature-guided painterly image rendering. In: ICIP, pp. 653–656 (2002) [4] Haeberli, P.: Paint by numbers: Abstract image representations. Computer Graphics 24(4), 207–214 (1990) [5] De Carlo, D., Santella, A.: Stylization and abstraction of photographs. ACM Transactions on Graphics (TOG) 21(3), 769–776 (2002) [6] Gooch, B., Coombe, G., Shirley, P.: Artistic vision: painterly rendering using computer vision techniques. In: NPAR, pp. 83–90 (2002) [7] Hertzmann, A.: Painterly rendering with curved brush strokes of multiple sizes. In: Siggraph, pp. 453–460 (1998) [8] Kasao, A., Miyata, K.: Algorithmic painter: a NPR method to generate various styles of painting. The Visual Computer 22(1), 14–27 (2006) [9] Collomosse, J.P., Hall, P.M.: Salience-adaptive painterly rendering using genetic search. International Journal on Artificial Intelligence Tools 15(4), 551–575 (2006) [10] Santella, A., DeCarlo, D.: Abstracted painterly renderings using eye-tracking data. In: Spencer, S.N. (ed.) NPAR, pp. 75–82. ACM Press, New York (2002) [11] Hertzmann, A.: A survey of stroke-based rendering. IEEE Computer Graphics and Applications 23(4), 70–81 (2003) [12] Olsen, S.C., Maxwell, B.A., Gooch, B.: Interactive vector fields for painterly rendering. In: Proceedings of Graphics Interface 2005, pp. 241–247 (2005) [13] Arnheim, R.: Art and Visual Perception: A Psychology of the Creative Eye. University of California Press (1974) [14] Arnheim, R.: Toward a psychology of art. University of California Press, Berkeley (1966) [15] Ogden, R.M.: Naive geometry in the psychology of art. American Journal of Psychology 49(2), 198–216 (1937) Glass Patterns and Artistic Imaging 1045 [16] Smith, M.A., Bair, W., Movshon, J.A.: Signals in macaque striate cortical neurons that support the perception of Glass patterns. Journal of Neuroscience 22(18), 8334–8345 (2002) [17] Smith, M.A., Kohn, A., Movshon, J.A.: Glass pattern responses in macaque V2 neurons. Journal of Vision 7(3), 5 (2007) [18] Yen, S.C., Finkel, L.H.: Extraction of perceptually salient contours by striate cortical networks. Vis. Res. 38(5), 719–741 (1998) [19] Li, Z.: A neural model of contour integration in the primary visual cortex. Neur. Comp. 10(4), 903–940 (1998) [20] Maloney, R.K., Mitchison, G.J., Barlow, H.B.: Limit to the detection of Glass patterns in the presence of noise. J. Opt. Soc. Am. A 4(12), 2336–2341 (1987) [21] Wilson, H.R., Wilkinson, F.: Detection of global structure in Glass patterns: implications for form vision. Vision Research 38(19), 2933–2947 (1998) [22] Seu, L., Ferrera, V.P.: Detection thresholds for spiral Glass patterns. Vision Research 41(28), 3785–3790 (2001) [23] Dakin, S.C.: The detection of structure in Glass patterns: Psychophysics and computational models. Vision Research 37(16), 2227–2246 (1997) [24] Prazdny, K.: Psychophysical and computational studies of random-dot moire patterns. Spatial Vision 1(3), 231–242 (1986) [25] Phillips, T.H., Rosenfeld, A.: A simplified method of detecting structure in Glass patterns. Pattern Recognition Letters 4(3), 213–217 (1986) [26] Wilson, J.A., Switkes, E., De Valois, R.L.: Glass pattern studies of local and global processing of contrast variations. Vision Research 44(22), 2629–2641 (2004) [27] Amidror, I.: Unified approach for the explanation of stochastic and periodic moirés. Journal of Electronic Imaging 12(4), 669–681 (2003) [28] Amidror, I.: Glass patterns as moiré effects: new surprising results. Optics Letters 28(1), 7–9 (2003) [29] Amidror, I.: The theory of the Moiré phenomenon, volume 1: periodic layers. Springer, Heidelberg (2000) [30] Glass, L.: Looking at dots. Math. Intell. 24(4), 37–43 (2002) [31] Amidror, I.: Dot trajectories in the superposition of random screens: analysis and synthesis. Journal of the Optical Society of America A 21(8), 1472–1487 (2004) [32] Cumani, A.: Edge detection in multispectral images. CVGIP 53(1), 40–51 (1991) [33] Papari, G., Petkov, N.: Continous glass patterns for painterly rendering. IEEE Transactions on Image Processing (to appear)