Perceptual Assessment of the Degree of Russian Accent

Perceptual Assessment of the Degree of Russian Accent Lya Meister Laboratory of Phonetics and Speech Technology Institute of Cybernetics Tallinn University of Technology Akadeemia tee 21, Tallinn 12618, Estonia [email protected] Abstract This paper deals with the perceptual assessment of Russian-accented Estonian. Speech samples were recorded from 20 speakers with a Russian background; clips of about 20 seconds from each speaker were selected for this perceptual study. The accentedness was rated in two tests: first, 20 native Estonian speakers judged the samples and rated the degree of foreign accent on a six-point interval scale; secondly, two experienced phoneticians carried out a perceptual study of the same samples and compiled the list of pronunciations errors. The results of both listening tests were highly correlated – the higher the degree of accentedness given to a L2-speaker by naïve listeners, the more pronunciation errors were found by trained experts. The classification of most frequent pronunciation errors based on acoustic-phonetic features is given, as well. language (L2) speaker deviates from that of L1speakers (Southwood & Flege, 1999). On the contrary, a trained phonetician should be able to identify and classify different accent phenomena as well as describe them in terms of deviations of acoustic-phonetic features. Following the findings and methodology presented in a recent paper (Meister, 2006; for methods employed in different studies see Jesney, 2004) on the accentedness rating of foreign-accented Estonian, two further listening experiments have been designed. The aim of these experiments is to compare the accentedness ratings given by naïve listeners, and the results of perceptual analysis of pronunciation errors carried out by experienced phoneticians. It is expected that the results of these two groups of raters harmonize quite well, i.e., the higher the accentedness ratings by naïve listeners of L2 speakers are, the more pronunciation errors are listed by experts. The study serves also a longterm goal – the development of criteria for speaking proficiency assessment, including the degree of FA. 2 1 Introduction Native speakers/listeners can easily identify nonnative speech and are able to rate the degree of foreign accent (FA). Naïve listeners‘ judgments of FA degree are based on their general perceptual impression rather than on conscious use of acoustic-phonetic knowledge about their own first language (L1). Accentedness ratings result in the degree of global foreign accent which is an impressionistic measure to which the speech of a second 2.1 Method Speech samples and speakers The speech material used in the study was recorded from 20 L2-speakers (14 female, 6 male) during the high-level language test at the National Examination and Qualification Centre. One of the subtasks the examinees have to perform is the conversation in pairs on a given topic which should demonstrate different speaking skills: expression of opinion, argumentation, turn-taking and carry- Joakim Nivre, Heiki-Jaan Kaalep, Kadri Muischnek and Mare Koit (Eds.) NODALIDA 2007 Conference Proceedings, pp. 345–348 Lya Meister ing on the conversation, etc. (Pajupuu et al., 2002). It is expected that a person with high-level language skills is able to communicate in written and spoken Estonian with near-native proficiency. The recordings of the conversations were carried out using a digital recorder (sampling frequency 44.1 kHz, 16 bit, mono) and a high-quality microphone placed at a ca. 1 m distance from the speakers. With each pair of subjects, six to eight minutes of spontaneous conversation was recorded. A continuous clip of speech with the duration of ca. 20 seconds from each subject's speech was chosen for perceptual assessment. The clips were stored into an audio file in random order with an inter-stimuli interval of five seconds. In addition to the speech recordings, each subject filled out a questionnaire concerning their linguistic background, age of L2 acquisition, use of L1, L2, etc. The summary of the speakers' information is presented in Table 1. 2.2 2.3 Experimental setup Before the test a foreign accent scaling technique was introduced and several examples of L2 speech with different degrees of accentedness were played to the listeners. The participants were instructed to focus only on deviations in pronunciation, while grammatical and lexical errors should be ignored. In the first part of the experiment the stimuli were played to subjects from a notebook computer via headphones in a quiet environment. The task of the judges was to rate the degree of foreign accent of each stimulus on an interval scale from 1 – "no foreign accent" to 6 – "very heavy foreign accent". The group of naïve listeners heard each stimulus only once; during the inter-stimulus intervals they had to write down their ratings on an answer sheet. The duration of the listening session was about nine minutes. In the second part of the experiment, two experts carried out an exhaustive perceptual analysis of each stimulus and compiled the list of perceived pronunciation errors classified into five major groups typical to Russian-accented Estonian: (1) deviation of temporal structure, (2) location of word stress, (3) quality of vowels and diphthongs, (4) palatalization, and (5) voicing of voiceless consonants (Meister and Meister, 2005). In the first stage the experts carried out error analysis independently from each other using repeated listening: this resulted in two different lists of pronunciation errors. Later, the disagreements in errors were discussed and analyzed together until the experts reached a common agreement. Listeners Two groups of listeners were employed in the study. First, a group of naïve (non-linguist) listeners was composed of 20 native Estonians (10 female, 10 male) in the age range of 17 to 62. All of them had some knowledge of Russian and diverse exposure to foreign-accented Estonian spoken by Russians; none of them reported any hearing problems. A second group of judges consisted of two trained phoneticians (native Estonians, one female, one male, both 49 years of age) with good knowledge of Russian and experience in experimental studies of Estonian as L2. Table 1. Summary of the background information of L2-speakers (EST = Estonia(n), RUS = Russia(n), UKR = Ukraine (Ukrainian)). L2-speakers' data Speaker's ID Sp1 Sp2 Sp3 Sp4 Sp5 Sp6 Sp7 Sp8 Sp9 Sp10 Sp11 Sp12 Sp13 Sp14 Sp15 Sp16 Sp17 Sp18 Sp19 Sp20 Age 52 23 19 19 16 25 26 32 19 18 20 51 19 43 20 33 18 46 45 32 Gender Country of birth Age of L2 acquisition Language(s) used at home Language(s) used at work Friends include Estonians F M F F F M F F F F M F M F M F F M F F Est Ukr Est Est Est Est Est Est Est Est Rus Est Est Est Est Est Est Rus Rus Est 5 9 5 7 7 1 12 12 9 5 9 5 9 8 9 20 5 30 23 9 Rus Rus Rus Rus Rus Ukr Rus Rus Rus Rus Est Rus Rus Rus Rus Est Rus Rus Est Est Rus Rus Est Rus Est Rus Est Rus Est Rus Est Rus Rus Est No Yes No Yes Yes No No Yes No No No 346 Est Rus Est Rus Rus Rus Est Rus Est Rus Rus Rus Est Est Rus Est Rus Yes Yes Yes Yes Yes No Yes Yes Yes Perceptual Assessment of the Degree of Russian Accent Table 2. The results of the perceptual assessment in ascending order by the mean of the perceived degree of global foreign accent (L2 speakers marked as Sp1…Sp20, raters marked as R1…R20). Perceptual ratings given by 20 raters (R1 - R20) Sp14 Sp12 Sp4 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19 R20 2 1 1 1 1 1 1 1 2 1 1 1 1 1 1 2 1 1 1 1 Mean 1,2 Stdev 0,37 CI 95% 0,16 Sp6 Sp5 Sp13 Sp1 Sp9 Sp8 Sp19 Sp20 Sp15 Sp3 Sp11 Sp17 Sp18 1 2 1 2 1 1 1 1 2 1 1 2 1 1 1 2 1 1 1 1 1 1 1 1 1 2 1 2 1 1 2 2 2 2 2 3 1 1 1 1 1 1 3 1 1 3 1 1 1 2 1 3 1 1 1 3 2 2 1 1 2 2 2 1 2 2 1 2 2 2 3 3 1 2 2 3 2 2 1 2 3 3 2 2 2 3 2 2 2 3 2 3 2 2 3 3 2 2 2 2 3 2 2 2 2 4 2 3 3 3 2 2 4 3 4 4 3 2 2 2 Sp7 Sp10 Sp16 Sp2 2 3 3 2 3 4 2 2 3 4 3 4 2 3 4 4 3 2 2 3 3 4 3 2 3 4 2 [6] 2 5 3 3 3 4 3 4 5 3 4 5 4 4 4 3 4 5 2 5 4 5 4 4 3 3 2 3 4 2 2 5 4 4 4 3 5 4 5 5 3 3 3 4 3 4 4 4 4 3 3 3 3 3 3 [2] 5 4 3 5 5 5 4 3 4 4 3 4 5 3 4 4 3 4 4 4 5 [6] 5 3 5 5 4 5 4 5 5 4 5 3 3 3 5 4 4 3 4 5 3 4 5 5 5 3 4 5 4 4 5 3 5 4 6 3 5 4 5 6 4 3 5 5 5 4 6 5 4 4 5 3 5 3 5 5 5 3 5 6 4 6 6 4 4 5 4 5 5 4 5 3 4 6 5 4 5 4 5 5 6 6 5 5 4 5 5 5 [3] 4 5 4 5 5 5 5 5 [3] 5 6 6 6 6 6 5 6 5 6 5 5 6 4 6 6 5 5 5 4 6 6 5 6 6 6 5 6 5 6 5 5 6 [3] 6 6 6 6 6 4 6 6 6 6 6 6 6 6 6 6 5 6 6 4 6 6 1,3 0,44 0,19 1,5 0,60 0,27 1,6 0,83 0,36 2,0 0,60 0,27 2,4 0,49 0,21 2,7 0,80 0,35 2,9 0,79 0,35 3,4 0,96 0,42 3,6 1,05 0,46 3,8 0,72 0,31 3,9 0,81 0,35 4,2 0,83 0,37 4,2 0,77 0,34 4,5 1,00 0,44 4,7 0,92 0,40 4,8 0,60 0,26 5,5 0,61 0,27 5,5 0,61 0,27 5,8 0,64 0,28 M e an acce nt s core with CI 95% Interval scores 6 5 4 3 2 1 Sp14 Sp12 Sp4 Sp6 Sp5 Sp13 Sp1 Sp7 Sp10 Sp16 Sp2 Sp9 Sp8 Sp19 Sp20 Sp15 Sp3 Sp11 Sp17 Sp18 L2 spe ake rs Figure 1. The mean scores of global foreign accent with a confidence interval of 95 %. L2 speakers (Sp1 … Sp20) ordered by the mean accent score in ascending order. 3 Results The rating results of the first group of judges (Table 2 and Figure 1) show high inter-rater consistency. Correlation for all possible pairwise combinations of two raters was computed while a few outliers were excluded from the statistics (see Table 2 numbers in square brackets). The average correlation is r = 0.85 (min r = 0.7, max r = 0.96); a correlation of 0.75 is considered acceptable (Shrout and Fleiss, 1979). Good inter-rater correlation shows that the duration of stimuli of 20 sec- onds is sufficient for reliable results (cf. (Meister, 2006), where five- and 60-seconds clips were used). Also, the narrower six-point interval scale (compared to the nine-point scale used in (Meister, 2006)) may result in less dispersed ratings. Variability of judgments among different listeners and the occurrence of few deviating ratings suggest that listeners' internal standards of accentedness are different. Also, it can not be excluded that grammatical and lexical errors made by L2 speakers influenced the individual accent scores. 347 Lya Meister Table 3. Classification of pronunciation errors of L2 speakers in ascending order by error rate. Type and amount of errors Speaker ID Sp14 Sp12 Sp4 Sp6 Sp5 Sp1 Sp13 Sp10 Sp7 Sp2 Sp8 Sp9 Sp16 Sp3 Sp15 Sp11 Sp20 Sp17 Sp19 Number of words 34 40 40 45 33 40 36 23 41 34 56 23 33 23 27 31 29 27 23 Temporal structure 2 3 2 3 2 4 4 8 5 10 5 9 9 9 10 8 Word stress Vowel quality 1 1 1 2 1 2 2 1 7 1 1 4 1 3 6 2 2 Summary The results of the two groups of listeners are highly correlated − the correlation between the mean accent score (Table 2) and the error rate (Table 3) is 0.94. It has been shown that for L1 speakers of a non-quantity language it is difficult to acquire a contrastive temporal category of L2 as a quantity language (McAllister et al., 2000). The same seems to hold true for the case of Russian as L1 and Estonian as L2 – the errors in the temporal domain contribute most to the error rate and probably to the perceived degree of FA, as well. Further work will focus on the analysis of relationships between the degree of global FA and the types of pronunciation errors, as well as the role of deviations of acoustic features in the perception of accentedness. 348 Voicing of consonants 1 The findings of two experts (Table 3) show that the most frequent errors are related to temporal structure, voicing of voiceless consonants, and quality of some vowels and diphthongs; other errors are less frequent. These results confirm earlier findings (Meister and Meister, 2005). In order to compare different L2-speakers, a simple measure of error rate has been formed by dividing the total number of errors by the number of words produced by the speaker during a 20 second clip (see Table 3). 4 Palatalization 1 3 2 1 2 4 2 5 3 3 4 6 3 3 4 1 8 6 5 4 3 Total number of errors 0 0 0 3 3 4 6 7 13 11 19 11 17 15 18 21 20 19 18 Error rate 0 0 0 0,07 0,09 0,10 0,17 0,30 0,32 0,32 0,34 0,48 0,52 0,65 0,67 0,68 0,69 0,70 0,78 References Jesney, K. 2004. The Use of Global Foreign Accent Rating in Studies of L2 Acquisition. Calgary, AB: University of Calgary Language Research Centre Reports. McAllister, R., Flege, J. and Piske, T. 2000. Aspects of the Acquisition of Swedish Quantity by Native Speakers of English, Spanish and Estonian. In: Proceedings of FONETIK 2000. Skövde, Sweden. Meister, L. and Meister, E.. 2005. Acoustic correlates of Russian accent in Estonian. In: Proceedings of SPECOM 2005, University of Patras, 437 - 440. Meister, L. 2006. Assessment of the degree of foreign accent: a pilot study. In: Fonetiikan Päivät 2006 = The Phonetics Symposium 2006, University of Helsinki, 53:113 - 119. Pajupuu, H., Reins, P. and Kerge, K. 2002. Eesti keele kõrgtaseme test. Käsiraamat. Tallinn: Riiklik Eksami- ja Kvalifikatsioonikeskus. Shrout, P.E. and Fleiss, J.L. 1979. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86, 420-428. Southwood, H. and Flege, J. 1999. Scaling foreign accent: direct magnitude estimation versus interval scaling. Clinical Linguistics & Phonetics, Vol. 13, No. 5, 335-349.