A New Measure of Probabilistic Entropy and its Properties

Applied Mathematical Sciences, Vol. 4, 2010, no. 28, 1387 - 1394 A New Measure of Probabilistic Entropy and its Properties Rajneesh Kumar Department of Mathematics Kurukshetra University Kurukshetra, India [email protected] Subash Kumar Department of Mathematics S.S.M.I.T, Dinanagar, India Anil Kumar Department of Mathematics S.S.C.E.T., Badhani Pathankot, India INTRODUCTION The concept of entropy in communication theory was first introduced by Shannon [13] and it was then realized that entropy is a property of any stochastic system and the concept is now used widely in different disciplines. The tendency of the systems to become more disordered over time is described by the second law of thermodynamics, which states that the entropy of the system cannot spontaneously decrease. Today, information theory is still principally concerned with communications systems, but there are widespread applications in statistics, information processing and computing. A great deal of insight is obtained by considering entropy equivalent to uncertainty, the generalized theory of which has well been explained by Zadeh [15]. R. Kumar, S. Kumar and A. Kumar 1388 The uncertainty associated with probability of outcomes, known as probabilistic uncertainty, is called entropy, since this is the terminology that is well entrenched in the literature. Shannon [13] introduced the concept of information theoretic entropy by associating uncertainty with every probability distribution P = ( p1 , p 2 , ...., p n ) and found that there is a unique function that can measure the uncertainty, is given by n H (P ) = − ∑ pi In pi (1.1) i =1 The probabilistic measure of entropy (1.1) possesses a number of interesting properties. Immediately, after Shannon gave his measure, research workers in many fields saw the potential of the application of this expression and a large number of other measures of information theoretic entropies were derived. Renyi [11] defined entropy of order α as: n ⎛ n ⎞ 1 (1.2) Hα ( P ) = In ⎜ ∑ piα ∑ pi ⎟ , α ≠ 1, α > 0 ⎟ 1 − α ⎜⎝ i = 1 i =1 ⎠ which includes Shannon’s [13] entropy as a limiting case as α → 1. Zyczkowski [16] explored the relationships between the Shannon’s [13] entropy and Renyi’s [11] entropies of integer order. Havrada and Charvat [5] introduced first non-additive entropy, given by: ⎡ n ⎤ α ⎢ ∑ pi ⎥ − 1 ⎢i = 1 ⎥ (1.3) H α ( P ) = ⎣ 1−α ⎦ , α ≠ 1, α > 0 2 −1 Kapur [7] generalized Renyi’s [11] measure further to give a measure of entropy of order ‘α’ and type ‘β’, viz., n ⎛ n ⎞ 1 (1.4) Hα , β ( P) = In ⎜ ∑ piα + β −1 ∑ pi β ⎟ , α ≠ 1, α > 0, β > 0,α + β –1 > 0 ⎟ 1 − α ⎜i =1 = i 1 ⎝ ⎠ The measure (1.4) reduces to Renyi’s [11] measure when β = 1, to Shannon’s [13] measure when β = 1, α → 1. When β = 1, α → ∞, it gives the measure H ∞ ( P) = − ln pmax Many other probabilistic measures of entropy have been discussed and derived by Brissaud [1], Chakrabarti [2], Chen [3], Garbaczewski [4], Herremoes [6], Lavenda [8], Nanda and Paul [9], Rao, Yunmei and Wang [10], Sergio [12], Sharma and Taneja [14] etc. The applications of the results obtained by various authors have been provided to various fields of Mathematical Sciences. In section 2, we have introduced a new generalized probabilistic information theoretic measure. A new measure of probabilistic entropy 1389 2. A New Generalized Information Theoretic Measure based upon Probability Distributions In this section, we propose a new generalized information measure for a n ⎧ ⎫ probability distribution P = ⎨( p1 , p2 , ..., pn ) , pi ≥ 0, ∑ pi = 1⎬ and study their i =1 ⎩ ⎭ essential and desirable properties. This generalized entropy depending upon n real parameters α1 , α 2 ,..., α n is given by the following mathematical expression: n α + α + ... + α 1 n −1 ∑ p i (P) = i = 1 Hn α , α , ..., α 1 − α − α − .... − α 1 n 1 n −1 2 (2.1) where n n and α + ∑ α ≠ 1, α + ∑ α > 1 i i i =1 α ≠ 1, α > 0, α ≥ 0 i =1 i (2.2) n If ∑ α = 0, then α ≠ 1, α > 0 .Thus, we see that the proposed measure (2.1) i i =1 becomes n α ∑ p −1 i H n (P) = i = 1 α 21 − α − 1 (2.3) which is Havrada and Charvat’s [5] measure of entropy of order α. The measure (2.3) again reduces to Shannon’s [13] measure of entropy as α → 1 . Thus, we see that the measure proposed in equation (2.1) is a generalized measure of entropy. Next, we study some important properties of this generalized measure. The measure (2.1) satisfies the following properties: (i) It is continuous function of p1 , p2 ,..., pn , so that it changes by a small amount when p1 , p2 ,..., pn change by small amounts. (ii) (iii) It is permutationally symmetric function of p1 , p2 ,..., pn , that is, it does not change when p1 , p2 ,..., pn are permuted among themselves. Hn α , α ,..., α 1 (iv) (P) ≥ 0 n n α + α + .... + α n −1 1 ∑ p i ( p , p ,..., p , 0) = i = 1 H n +1 α , α ,..., α 1 2 n 1 − α − α − .... − α 1 n n −1 1 2 R. Kumar, S. Kumar and A. Kumar 1390 = Hn α , α ,..., α 1 (P) n This property says that entropy does not change by the inclusion of an impossible event with probability zero. n (v) Since H α , α ,..., α 1 ( P ) is an entropy measure, its maximum value must n occur. To find the maximum value, we proceed as follows: Let n α + α + .... + α n −1 1 ∑ p i ⎛ n ⎞ f ( p) = i = 1 − λ ⎜ ∑ p − 1⎟ ⎜ ⎟ i 1 − α − α − .... − α ⎝i = 1 ⎠ n −1 1 2 Then, we have α + α + .... + α −1 n 1 p ( ... ) α α α + + + ∂f n 1 1 = −λ 1 − α − α − .... − α ∂p n 1 1 2 −1 α + α + .... + α −1 n 1 p α α α ( + + ... + ) ∂f n 1 2 = −λ 1−α −α − .... −α ∂p n −1 1 2 2 . . . α + α + .... + α − 1 n 1 p ( ... ) + + + α α α ∂f n n 1 = −λ 1 − α − α − .... − α ∂p 1 n −1 n 2 For maximum value, we take ∂f ∂f ∂f = = ... = = 0 ∂p ∂p ∂p 1 2 n which gives α +α + .... +α −1 α +α + .... + α −1 α + α + .... +α −1 n n n 1 1 1 (α +α + ... + α ) p (α +α + ... +α ) p (α + α + ... + α ) p n 1 n n n 1 1 2 1 = = ... = 1−α −α −.... −α 1−α −α − .... −α 1−α −α − .... −α n −1 n −1 n −1 1 1 1 2 2 2 which is possible only if p1 = p = ... = pn n Thus ∑p i =i i = 1 gives p1 = p = ... = pn = 1 n A new measure of probabilistic entropy 1391 Hence, we see that the generalized entropy measure (2.1) possesses maximum value n 1 and this value subject to natural constraint ∑ pi = 1 arises when p1 = p2 = ... = pn = . n i =i This result is most desirable. (vi) The maximum value is an increasing function of n. To prove this result, we have f ( p) = 1 − α − α n 1 − α − α 2 Thus f ' ( p) = 1 1 − .... − α − .... − α n − 1 n − 1 −α − α − .... − α 1 n (1 − α − α − ... − α ) n 1 n 1 − α − α − .... − α 1 n −1 2 α + α + .... + α n 1 (α + α + ... + α − 1).2 n 1 = >0 α + α + .... + α α + α + .... + α n − 2).n n 1 1 (2 since n n i =1 i =1 α + ∑ α ≠ 1, α + ∑ α > 1 i i Hence maximum value is an increasing function of n. (vii) Recursivity property : To prove that the measure (2.1) is recursive in nature, we consider n α + α + .... + α 1 n −1 n + ∑ p i i=3 1 − α − α − .... − α 1 n −1 2 α + α + .... + α ( p + p , p p ,..., p ) = H n −1 1 2 3, 4 α , α ,..., α n 1 n 1 1 α + α + .... + α α + α + .... + α n − p n 1 1 n − p 1 2 1 − α − α − .... − α 1 n −1 2 n α + α + .... + α 1 n −1 ∑ p i + i =1 1 − α − α − .... − α 1 n −1 2 α + α + .... + α ( p + p2 ) = 1 ( p1 + p2 ) R. Kumar, S. Kumar and A. Kumar 1392 α + α + .... + α α + α + .... + α ⎡ ⎤ ⎞ n ⎛ p n 1 1 ⎢ ⎛⎜ p1 ⎞⎟ ⎥ 2 ⎟ +⎜ − 1⎥ ⎢⎜ p + p ⎟ ⎜ ⎟ p p + α + α + .... + α ⎢ ⎝ 1 2 ⎠ ⎥ 2⎠ ⎝ 1 n 1 =− p + p ⎢ ⎥ 1 2 1 − α − α − .... − α 1 n −1 ⎢ ⎥ 2 ⎢ ⎥ ⎢⎣ ⎥⎦ n α + α + .... + α 1 n −1 ∑ p i + i =1 1 − α − α − .... − α 1 n −1 2 ( ) ( ) =− p + p 1 2 α + α + .... + α 1 ⎛ p ⎞ p 1 + 2 ⎟ + Hn nH ⎜ ( p , p ,..., p ) α ,α ,...,α 1 2 2⎜ p + p n p +p ⎟ 1 n 1 2 1 2 ⎝ ⎠ Thus, we have proved that Hn α , α , ..., α 1 ( p , p , ..., p ) = H n − 1 ( p + p , p p ..., p ) 1 2 1 2 3, 4 n n α , α , ..., α n n 1 ( + p +p 1 2 ⎛ p ⎞ p 1 + 2 ⎟ nH ⎜ 2⎜ p + p p +p ⎟ ⎝ 1 2 1 2⎠ α + α + .... + α ) 1 This shows that the measure (2.1) possesses recursivity property. (vii) Additive property: To show that the measure (2.1) is non-additive, we consider α + α + .... + α n m ⎛ ⎞ n −1 1 ∑ ∑ ⎜ p q ⎟ i j ⎠ ⎝ i =1 j =1 H n, m (P ∪ Q ) = α , α , ..., α 1 − α − α − .... − α n 1 n −1 1 2 ⎛ n α + α + .... + α ⎞ 1 n −1⎟ ⎜ ∑ p i ⎜ ⎟ ⎛⎜ m α + α1 + .... + αn ⎞⎟ = ⎜ i =1 −1 ⎟⎜ ∑ q j 1 − α − α − .... − α ⎟ 1 j = 1 n ⎜ 2 ⎠ − 1 ⎟⎝ ⎜ ⎟ ⎝ ⎠ ⎛ n α +α +.... +α ⎞ ⎛ m α +α1 +.... +αn ⎞ n 1 −1⎟ −1⎟ ⎜ ∑ qj ⎜ ∑p i ⎜ ⎟ ⎜ ⎟ j =1 +⎜ i =1 + ⎜ ⎟ ⎟ 1−α −α −.... −α 1−α −α −.... −α ⎜ n n 1 1 ⎜2 −1 ⎟ 2 −1 ⎟ ⎟ ⎜ ⎟ ⎜ ⎝ ⎠ ⎝ ⎠ A new measure of probabilistic entropy 1393 ⎛ n α+α +....+α ⎞ ⎛ m α+α1+....+αn ⎞ n −1⎟ ⎜ ∑ q −1⎟ ⎜ ∑p 1 j ⎛ 1−α−α −....−α ⎞ ⎜i =1 i ⎜ ⎟ ⎟ j =1 1 n −1⎟ =⎜2 ⎟ ⎜ ⎜ ⎟ ⎜ 1−α−α −....−α ⎟ 1−α−α −....−α ⎝ ⎠⎜ 2 1 1 n −1 ⎟ ⎜ 2 n −1 ⎟ ⎟ ⎜ ⎟⎜ ⎝ ⎠⎝ ⎠ ⎛ n α + α + .... + α ⎞ ⎛ m α + α1 + .... + αn ⎞ 1 n −1 ⎟ ⎜ ∑ q −1 ⎟ ⎜ ∑ p j i ⎜ ⎟ ⎜ ⎟ j =1 +⎜ i = 1 + ⎜ ⎟ ⎟ 1 − α − α − .... − α 1 − α − α − .... − α n −1 ⎟ ⎜ 2 n −1 ⎟ 1 1 ⎜ 2 ⎟ ⎜ ⎟ ⎜ ⎝ ⎠ ⎝ ⎠ ⎛ 1 − α − α − .... − α ⎞ n − 1 ⎟ .H n ( P ). H m (Q ) + H n ( P ) + H m (Q ) 1 = ⎜2 ⎜ ⎟ ⎝ ⎠ which shows that the generalized entropy (2.1) is non-additive. References [1] Brissaud, J.B. (2005): "The meaning of entropy", Entropy, 7 (1), 68-96. [2] Chakrabarti, C.G. (2005):"Shannon entropy: axiomatic characterization and application”, Int. Jr. Math. Math.Sci., 17, 2847-2854. [3] Chen, Y. (2006): “Properties of quasi-entropy and their applications”, J.Southeast Univ. Nat.Sci., 36 (2), 222-225. [4] Garbaczewski, P. (2006): “Differential entropy and dynamics of uncertainty", J.stat.Phys.,123, 315-355. [5] Havrada, J.H. and Charvat, F. (1967): "Quantification methods of classification process:Concept of structural α-entropy", Kybernetika, 3, 3035. [6] Herremoes, P.(2006):"Interpretations of Renyi entropies and divergences", Phys.A., 365 (1), 57-62 . R. Kumar, S. Kumar and A. Kumar 1394 [7] [8] Kapur, J.N. (1967): "Generalized entropy of order α and type β", Maths. Semi., 4, 79-84. Lavenda, B. H. (2005) : "Mean Entropies", Open System Infor. Dyn., 12, 289-302. [9] Nanda, A. K. and Paul, P. (2006): "Some results on generalized residual entropy", Information Sciences, 176 (1), 27-47. [10] Rao, M.C., Yunmei, V.B.C. and Wang, F. (2004): "Commulative residual entropy : a new measure of Information”, IEEE Trans. Inform. Theory, 50 (6), 1220-1228. [11] Renyi, A. (1961): "On measures of entropy and information", Proc. 4th Ber. Symp. Math. Stat. and Prob., 1, 547-561. [12] Sergio, Verdu (1998): “Fifty years of Shannon theory”, IEEE Trans. Inf. Theory, 44(6), 2057-2078. [13] Shannon, C. E.(1948):"A mathematical theory of communication”, Bell. Sys. Tech. Jr., 27, 379-423, 623-659. [14] Sharma, B.D. and Taneja, I.J. (1975): "Entropies of type (α, β) and other generalized measures of information theory", Met., 22, 202-215. [15] Zadeh, L.A. (2005) : "Towards a generalized theory of uncertainty (GTU)- an outline", Information Sciences, 172, 1-40. [16] Zyczkowski, K. (2003) : “Renyi extrapolation of Shannon entropy”, Open Syst. Inf. Dyn., 10 (3), 297-310. Received: October, 2009