Books by Rainer Kurz
Kogan Page Book , 2008
Book Chapter
A book chapter outlining the origin of the Great 8 Competencies and the 'World of Work' (WoW) model.
Conference Presentations by Rainer Kurz
International Congress of Psychology (ICP) in Prague, 2024
This symposium brings together psychometric testing experts to share pioneering insights into adv... more This symposium brings together psychometric testing experts to share pioneering insights into advances in Computer Based Assessment (CBA).
The first paper reviews milestones in the development and evaluation of CBA systems based on Bartram & Bayliss (1984). Validation principles are illustrated through the correlation of personality, ability, interest, and motivation scales with Great 8 competency factors.
The second paper outlines contemporary advances in personality, ability and competency assessment and reporting through Computer-Based Test Interpretation (CBTI) in the light of Bartram (1994) which shaped BPS and EFPA test review processes. Versatile use of assessment data for multi-level reporting across tools are illustrated and backed with validation data.
The third paper illustrates applications of Sociomapping (Bahbouh, 2012) to psychometric assessment reporting at group and construct set level. Topographical representation of relationships between Big 5 and Great 8 constructs align to Stability and Plasticity in Cybernetic Big 5 Theory (DeYoung, 2015). Case studies on a 'Talent' group and a vet team are presented.
The fourth paper outlines a leading-edge approach involving continuous assessment of wellbeing which presents a measurement model for Employee Experience (EX) based on continuous adaptive micro interactions. The system captures trends in EX data in teams and organisations over time, while accounting for the dynamic relationship between contextual factors in the work environment and internal mindset factors that together influence an individual's EX.
The fifth paper outlines how test review processes have been impacted by advances in CBA. Issues include the use of item banks and CBTI reports products where reviewers will depend on technical documentation featuring psychometric data, especially validation results, to assess the appropriateness of the interpretations.
As artificial intelligence becomes more prevalent, it is timely to discuss past, present and future of CBA. The discussant slot will review the contributions and draw out themes, challenges, and opportunities.
Discussant: Dragos Iliescu, University of Bucharest
Intelligent Testing Systems: Past, Present and Future
Rainer Hermann Kurz, PhD
HUCAMA Analytics, London
Bartram & Bayliss (1984) wrote about the future use of computers in assessment. This paper reviews issues surrounding seven components of a fully automated Intelligent Testing System (ITS) and demonstrates how empirical data can be used to build valid system. 1. Test choice requires a sophisticated understanding of the assessment purpose, tools available and the importance of the scales. 2. Administration requires robustness against Intellectual Property content theft and faking. 3. Scoring requires transparency, sensitivity, and validity. 4.Interpretation needs to be based on valid information presented in understandable form 5. Feedback must be framed appropriate for the recipient whether candidates or 4rd parties. 6. Decision-making must be embedded within legal frameworks and proportionate to the quality of the evidence gathered. 7. Monitoring of outcomes is crucial to establish and enhance validity.
With the advent of AI there is a risk that poor assessment practices will proliferate especially if poor off-line processes are perpetuated and magnified through AI. On the other hand, AI that builds on and integrates sound testing practices could add value.
A data set where 250 individuals completed ability, personality, motivation, and competency assessments is utilised to illustrate data-driven development approaches. Predictors for the Great 8 Competencies (Kurz & Bartram, 2002) are calculated based on the meta-analysis of Bartram (2005) to illustrate how the scales of these five assessment modalities inter-relate. The two highest correlations for the factors are identified for each scale to facility the development of expert systems based on Great 8 and Big 5 models – whether using conventional or AI approaches.
Computer-Based Assessment across Personality, Ability and Competency Factors
Michele Guarini
HUCAMA Group, Copenhagen
This paper builds on the pioneering work of Bartram (1994) on Computer-Based Test Interpretation (CBTI) with a lens on group reporting based on the Great 8 Success Factors inspired by Kurz & Bartram (2002).
Personality Factors:
The modular range features general, professional, and executive level versions with 80, 160 and 240 questions measuring 16, 32 and 48 facets respectively grouped into 8 factors.
A. The Role Wheel Report uses ipsatised data (Bartram, 1996) to remove the effect of individual response style for enhanced group reporting.
B. The Leadership Report maps 48 personality facets to 8 Primary Colours of Leadership constructs (Pendelton, Furnham & Cowell, 2021) with an observed validity with external reviewer ratings of .50 (N=113).
C. The Aspects Reports covers emotional and operational themes that underpin potential and performance fully integrating Emotional Intelligence and Learning Agility constructs. The median construct convergence for the nearest counterpart scale was .63 for the 15 EQi 2.0 facets (N=101) and .70 for its five higher-order compounds.
Ability Factors:
This assessment consists of diagrammatic, numerical, spatial, and verbal component tests with a time limit of 8 minutes each. Rule-based item generation builds on Kurz (1990) and reporting on Kurz (2000) featuring Supra-scores across areas and sub-scores for speed, accuracy and caution.
Competency Factors:
This inventory features an overarching GETTING IT RIGHT, GETTING ALONG, GETTING AHEAD and GETTING ALIGNED model that builds on the Schwartz (1993) values circumplex and the Hogan & Holland (2004) view on performance. Each quadrant pairs up two Great 8 factors. Extreme tie-breaker data is collected and item level results shown.
A Potential & Performance Solution gap analysis tool brings together reporting across personality, ability and competency assessments for individuals and groups using Sten scores on a dynamic dashboard that enables multi-level integration, interpretation, and interrogation of data.
Sociomapping and Team Profile Analyzer in Psychometric Assessment
Pauline Willis
Lauriate, Australia
Sociomapping (Bahbouh, 2012) is an innovative method for tracking quality and frequency of communication in organisation. This paper outlines two applications of the underlying methodology to psychometric assessment.
STORM software uses scale correlations to produce a topographical ‘heat map’ that indicates the centrality of scales and maps out the relationship between constructs. Correlations (N=308) of Big 5 scales (based on NEO) with Great 8 (Kurz & Bartram, 2002) constructs revealed the centrality of Emotional Stability together with motivational (Need for Achievement & Power) constructs to the variable set. Stability (Alpha) vs Plasticity (Beta) meta-factors delineated one axis whereas People vs. Task delineated the other. The graph illustrates how constructs interrelate and facilitate understanding of the nature of ‘derailment’ scales. A cross-validation on N=466 largely confirm the results using different questionnaires.
Team Profile Analyzer (TPA) software produces a map that indicates the centrality of individuals to groups and similarities between group members. In the ‘Sociomap of Profile Similarity’ each group member is represented by a point and mutual distances represent mutual similarity of individual profiles. A heat map colour scheme indicates centrality. Personality assessment results for 16 ‘Elite’ performers across business, arts and sports were analysed using TPA. 10 group members shared many characteristics whereas 6 were different at the highest level of analysis with lower-level scores illuminating the origin of higher-order trends. The analysis explored differences between business leaders and others as well as sex differences. TPA analysis on Personality Factors results for a veterinary team will also be presented.
The applications demonstrate the power of group level reporting across psychometric results. The question arises how advances in academic theory building, such as the Periodic Table of Personality (Woods & Anderson, 2016) and Cybernetic Big 5 Theory (DeYoung, 2015), can be built upon and integrated with the Sociomapping methodology.
Insights in Motion: A Comprehensive Model for Tracking Employee Experience Over Time
Richard T. Justenhoven, PhD
Welliba, Germany
The varying stability across time and situations different constructs exhibit is well known and subject of ongoing research (Steyer et al., 2015). Talent assessment and management tools continue to evolve, and technological advancements enable capturing data in ever increasing breadth and depth.
This enables increasingly nuanced approaches to measuring constructs over time. This paper presents a measurement model for Employee Experience (EX) based on continuous adaptive micro interactions (CadaMint) as one example of this (Preuss et al., 2023). Grounded in Self-Determination-Theory (Deci & Ryan, 2000; Yeager & Dweck, 2020) and the Job-Demands-Resources Model (Bakker & Demerouti, 2007) CadaMint captures trends in EX data in teams and organisations over time, while accounting for the dynamic relationship between contextual factors in the work environment and internal mindset factors that together influence an individual's EX.
As data on EX often covers quarterly or annual cycles, CadaMint accounts for decreasing reliability of data on individual level as measurements age and increases when new measurements for the same constructs are added. This is achieved through a set of characteristics assigned to each instrument and variable, determining how measurements age. Interactions of different factors and fluctuations over time not only pose challenges to measurement, but also to the way outputs are presented to HR profes...
International Congress of Psychology, 2024
This paper outlines a synthesis of Cybernetic Big 5 Theory (DeYoung, 2015) and Great 8 Competenci... more This paper outlines a synthesis of Cybernetic Big 5 Theory (DeYoung, 2015) and Great 8 Competencies (Kurz & Bartram, 2002) to address vexing problems in psychometric assessment practice.
Digman (1997) found Alpha (Agreeableness, Conscientiousness & Emotional Stability) and Beta (Extraversion & Openness) higher-order meta-factors of the Big 5 that DeYoung, Peterson & Higgins (2001) renamed ‘Stability’ and ‘Plasticity’. Judge, Piccolo, & Ilies (2004) found high validity in the Ohio leadership studies for ‘Consideration’ and ‘Initiating Structure’ which appear to cover people and task aspects respectively.
In line with Barrick & Prengler (2018). Kurz (2023) related both to the universal values circumplex of Schwarz (1992) to create a four-quadrant model and developed the Personality Factors range of questionnaires to operationalise Cybernetic Great 8 Theory that offers opportunities to enhance assessments in several ways.
‘RESPONSE STYLE’ is a score based on the sum of all positive qualities that approximates the General Factor of Personality (GFP) which Van der Linden et al (2017) considered a factor of Social Effectiveness that is very closely related to Emotional Intelligence.
In addition to normative reporting, Intra-Personal scores based on an item-level variation of Bartram (1996) generate reports that are ipsatised thus resolving the normative-ipsative debate.
‘STABILITY’ and ‘PLASTICITY’ scores are displayed, and their difference interpreted as a contrast between dependability and proactivity. Extreme difference scores are considered ‘out-of-balance’ and indicative of increased derailment risk'.
‘TASK’ and ‘PEOPLE’ scores are displayed, and their difference interpreted with the aid of paradoxes based on a quasi-circumplex arrangement of Success Factors.
Modular general (PF16), professional (PF32) and executive (PF48) versions cater for different job levels.
Psychometric data on the development sample (N=466) and case study vignettes will be presented. Parallels and differences between Big 5 and Great 8 approaches, and links to the Periodic Table of Personality will be discussed.
ABP Conference, 2024
Rainer has been at the forefront of Computer Based Assessment since 1986 when in his first year P... more Rainer has been at the forefront of Computer Based Assessment since 1986 when in his first year Psychology project he tortured friends with an audio-visual reaction time test he programmed on his Commodore C64.
Studying at the University of Hull under Prof Dave Bartram he secured his first job at SHL and proceeded to explore rule-based item generation, open answer format and speed-accuracy trade-odd in his MSc project.
At SHL he researched MS-DOS, Windows, and WWW test technology. He developed Competency Potential algorithms, ‘Which Way’ career apps and the Great 8 Competencies (Kurz & Bartram, 2002).
Further milestones involved the development of Saville Consulting Wave. Performance 360 and Swift aptitude tests.
At Cubiks be pushed forward the revolutionary PAPI Dynamic Report and the Horizon solution with dynamic top-tie breaker and ‘nipsatised’ scoring.
Most recently he has been presenting on the Periodic Table of Personality and how Cybernetic Big 5 Theory (DeYoung, 2005) could be improved.
He is a passionate advocate for good assessment practice and has shared very critical views on psychometric assessment issues in the TI newsletter of the International Test Commission and at conferences across Europe.0
Rainer will review developments in the field, explain why he hates multiple regression and share his concerns about artificial intelligence.
ICP Conference in Yokohama, 2016
Ability test construct validation study illustrating concerns that EFPA convergent construct vali... more Ability test construct validation study illustrating concerns that EFPA convergent construct validity minimum value of .55 is set too high at a counter-productive level and that the earlier standard of .45 applied in the 'BPS (1996). Review of Personality Instruments (Level B) For Use in Occupational Settings' is more suitable.
ITC COLLOQUIUM, 2021
Paper in the ITC Colloquium 11th July 2021 Symposium 382: Opportunities and Challenges for Person... more Paper in the ITC Colloquium 11th July 2021 Symposium 382: Opportunities and Challenges for Personality Assessment
ITC COLLOQUIUM, 2021
Paper in the virtual ITC Conference Symposium 'Exploring Career Fit using a Range of Constructs, ... more Paper in the virtual ITC Conference Symposium 'Exploring Career Fit using a Range of Constructs, Taxonomies and Techniques'
Association of Test Publishers (ATP), 2019
This poster outlines development of the 'Horizon' assessment solution initially for the China mar... more This poster outlines development of the 'Horizon' assessment solution initially for the China market with three key innovations in testing. For the PAPI 3 SL N version of the 'Personality and Preference Inventory' a novel questionnaire assessment with 162 questions was developed that dynamically combines natural (normative) responding with intrapersonal (ipsative) scoring and a Top Tie Breaker mechanism that identifies Inhibitors. Personality facets are aggregated through validated algorithms into Competency Predictions within a Capability Model. A Cultural Agility output was developed from Chinese characteristics.
BPS DOP Conference, 2020
Personality Assessment Science & Practice:
Mapping Measures to the Periodic Table of Personality
... more Personality Assessment Science & Practice:
Mapping Measures to the Periodic Table of Personality
Symposium at the BPS DOP 2020 Conference
Psychological Assessment at Work
This symposium maps personality questionnaire scales to the Periodic Table of Personality (Woods & Anderson, 2016) using the Trait Descriptive Adjectives (TDA) measure of Goldberg (1992). PCA with Varimax rotation on results for N=1213 professionals and managers were used to create orthogonal regression-based factor scores. The mean age of this group was 46.06 (SD 12.86). 63% of the group identified as Female.
The first two papers are based on a co-validation sub-group (N=128) that completed PAPI 3 SL, HPI, HDS and MVPI questionnaires in addition to TDA.
The first paper is concerned with the construct validation of a Big 5 + Achieving research model operationalised through 30 PAPI scales.
The second paper is concerned with the construct validation of the scales of the Hogan questionnaires.
The third paper maps 10 Lumina Spark Aspects representing both poles of each Big 5 factor and 40 sub-ordinated Qualities with adaptive and maladaptive sub-scores.
Paper 1 Summary:
Kurz (2020). Locating Scales of a Multi-level ‘Big 5 + Achieving’ Measure on the Periodic Table of Personality
The first paper is concerned with the construct validation of a Big 5 + Achieving research model operationalised through 30 PAPI scales with TDA scores (N=128) following the methodology of Woods & Anderson (2016). Scores for 20 PAPI dimensions and 10 Facets were aggregated into six factor scores using integer weights based on a prior large-scale validation study. Factors were paired up into Engaging, Enterprising and Excelling clusters which in turn were aggregated into a Performing Total score. TDA correlations largely were as expected determining construct locations on the Periodic Table of Personality. The model resolves the conundrum that Achieving constructs are variously located within Conscientiousness or Extraversion domains, or even at the opposite end of Agreeableness. The three-cluster model builds on the Three Effectiveness Factors (Kurz, Saville & MacIver, 2009) which in turn were inspired by the Alpha and Beta factors of Digman (1997). The results show strong overlap between the Performing Total based on PAPI and the sum of the TDA factors representing the General Factor of Personality (Musek, 2007). The results provide confirmation for the validity and value of multi-level measurement of personality variables.
Paper 2 Summary:
Groenewald & Kurz (2020). Mapping Hogan Bright Side, Dark Side and Inside Scales to the Periodic Table of Personality.
This paper is concerned with the construct validation of Hogan HPI, HDS and MVPI scale constructs with TDA scores (N=128) following the methodology of Woods & Anderson (2016). Scores for all scales as well as summary scores within and across instruments were mapped to the Periodic Table of Personality.
HPI results closely concur with the relevant TDA Big 5 scores although Learning Approach was largely independent.
HDS scores correlated negatively with Extraversion and Emotional Stability for Moving Away scales, positively with Extraversion and Openness for Moving Against scale and positively with Conscientiousness and Agreeableness for Moving Towards scales.
MVPI scales were less strongly correlated with TDA Big 5 scores but generally followed a pattern compatible with the research of Schwartz (1992) on universal values.
The results shed further light on the nature of bright side, dark side and inside measures by mapping them onto the Periodic Table of Personality.
Paper 3 Summary:
Desson (2020). Mapping Lumina Spark to the Periodic Table of Personality
This paper is concerned with the construct validation of Lumina Spark scale constructs with TDA scores (N=671) following the methodology of Woods & Anderson (2016). Scores for 10 Aspects covering the opposing poles of each Big 5 factor and 40 qualities were correlated with orthogonal TDA scores following Woods & Anderson (2016). Most correlations are in line with expectations although a few deviate somewhat e.g. there is more overlap than expected in that Conscientiousness (convergent thinking) and Openness (divergent thinking) attract opposing constructs rather than showing independence. Introverted and Discipline Driven are ‘factor-pure’ aspects whereas Observing, Intimate, Collaborative, Regard for Others, Focuses Feelings, Resilient and Responsive achieve this status at Qualities level.
Maladaptive sub-scales generally follow the correlation pattern of adaptive sub-scales but vary considerably in their correlation with Emotional Stability.
For the most part, Aspects and Qualities were found to correlate as expected, and thus shows the robust convergent validity between Lumina Spark and the TDA. Furthermore, by conceptualising opposite sectors of the circumplex model as discrete dimensions, we were able to show that these sectors are not necessarily direct opposites of each other, with one being the low form of its opposite, but instead discrete Aspects and Qualities can be mapped to these sectors.
BPS DOP Conference in Stratford-upon-Avon, 2004
Presentation on the development of the 'Great 8' that was originally submitted as part of a sympo... more Presentation on the development of the 'Great 8' that was originally submitted as part of a symposium for the 2003 conference.
EAWOP Congress in Lisbon, 2003
Poster Overview
Intelligence Theory
Multiple Intelligences (MI)
The Differential Reasoning Mode... more Poster Overview
Intelligence Theory
Multiple Intelligences (MI)
The Differential Reasoning Model (DREAM)
Competency Behaviour Survey:
Multiple Intelligences Inter-correlations
Interest Correlates
Personality Correlates
Competency Correlates
The World of Work (WoW) Model
Competency at Work
The Differential Reasoning Model (DREAM)
Review of the ability literature and ‘Occupational Testing’ practice lead to the development of the model (Kurz, 2000)
Model contrasts two categories of ability assessment :
Aptitudes (Reasoning/Learning) vs.
Achievements (Knowledge/Skills)
Model distinguishes general abilities (expected to have high traditional ‘g’ loadings) from distinct abilities (expected to have low traditional ‘g’ loadings) with 2 Sectors each:
General (Educational/Practical) vs.
Distinct (Physical/Social)
The four sectors cover 3 Ability Areas each
General Reasoning Ability ‘R’ defined as ‘unit weight’ average of six reasoning areas: V, N, C, D, S, M
Competency Behaviour Survey
Pilot research study (Briceno, 2002)
Joint project with Goldsmith MSc student
Target sample: >120 SHL staff worldwide
Operationalisation:
.xls Spreadsheet Player
Biodata
Ability self-ratings on 12 Multiple Intelligences
‘Great 8’ Competencies (Kurz & Bartram, 2002)
Collection of existing psychometric data for Construct Validation:
Occupational Personality Questionnaire: Big 5
Career Pathfinder In-depth: RIASEC
Multiple Intelligences in the World of Work
Miles (2000): ‘Emotional intelligence may be better conceptualised as an array of behavioural skills dependent on attributes of both personality and ability.’ ‘ …it is probable that emotional intelligence may possess both construct validity and practical utility as a competency (constellation of behaviours) rather than as a particular trait or ability’
Kurz and Bartram (2002) define competencies as ‘sets of behaviours that are instrumental in the delivery of desired results or outcomes. ‘
Kurz (1999) developed the ‘World of Work’ (WoW) model to map out the elements that underlie ‘Person * Environment Interaction’ at work
Basis for SHL Competency Framework (Kurz & Bartram, 2002)
EAWOP Congress in Lisbon, 2003
On-screen Aptitude Testing @ SHL: The Research Questions
SHL has 25 years of experience in uti... more On-screen Aptitude Testing @ SHL: The Research Questions
SHL has 25 years of experience in utilising computers in assessment. Since 1990 three generations of on-screen aptitude tests were developed, and a number of studies conducted to research their statistical properties as well as candidate responses. This poster pulls together data from seven studies to address the following questions:
Q1. Are the on-screen and p&p tests psychometrically equivalent?
Q2. Which administration format do candidates prefer?
Q3. On which version do candidates feel more anxious?
Q4. How do results change with technology advances?
Three Generations of SHL On-Screen Aptitude Tests
International Testing System (1990)
MS-Dos Operating System
Verbal (VMG1) and Numerical Reasoning (NMG1)
in 10 Languages
Expert (1993-)
Windows Operating System
VMG1, NMG1, Basic Checking (CP7.1), Diagrammatic Series (DC3.1), Spatial Reasoning (SIT7), Mechanical Comprehension (MT4.1)
On-line Solutions (1997-)
WWW Administration System using Java Applets
Practice Tests
Access Ability (Verbal & Numerical) for unsupervised ‘Distance Assessment’
Verbal (VMG3) & Numerical (NMG3) Reasoning
On-Screen Test Design
In the computerisation of ‘EXPERT’ lessons learned from the MS-DOS predecessor (see Kurz, Torrijos-Sanchez & Bartram (1994) where incorporated e.g. no on-screen calculator; use of mouse/touch screen instead of keyboard; ergonomic design of displays, controls, instructions and examples; and ‘Failsafe’ operation functionality geared towards (virtual) self-administration (see Kurz, 1998). The aim was to achieve transparency, consistency and acceptability across more than 20 questionnaires and 80 tests used with candidates across the entire working population, in over 20 languages. Evaluations by Kurz (1997) demonstrated high candidate satisfaction with instruction and example design, and favourable responses to the Graphical User Interface features. The development of WWW On-line aptitude tests – whether for Supervised or Unsupervised use, posed a number of additional challenges summarised by Evans (2002) relating to user interface, response mode and system feature design.
Study Designs
Test-Retest across Paper & Pencil and Computer mode:
Identical tests with time gap: Study 1, 3-6
Identical test in single session: Study 2
Parallel tests in single session: Study 7
All studies with 2 experimental groups:
Condition P&P-C: Paper & Pencil followed by Computer
Condition C-P&P: Computer followed by Paper & Pencil
Randomised allocation to conditions (except Study 7)
Study 4 also had P&P-P&P and C-C condition
Evaluation forms
Subjects: University Students Study 1, 2, 3 & 5; A-level Students Study 6 & 7; Employees Study 4
Locations: UK apart from Study 5 (Germany)
Psychometric Equivalence Issues
The results of the 7 studies fully support the findings of Mead & Drasgow (1993) that Power tests transfer well to computer format, and Speed tests are problematic. Average sample size weighted Test-Retest reliabilities reached very satisfactory levels of .75 for Verbal, and .80 for Numerical Reasoning, in line with ordinary p&p Test-Retest reliabilities reported in the MGIB manual. The results of Study 6 suggest that other ‘Power’ tests also transfer well to computer format. The research on the Basic Checking test demonstrated the need for different norms, and suggests that Speed tests may well have slightly different measurement characteristics in computerised format. Interestingly, Agapitou (1993) found slightly higher validities for the computerised version of CP7.1 in the prediction of competencies, and Wheeley (1994) found superior validities for computerised versions of VMG and NMG in the prediction of A-level results.
Conclusions
It is reassuring to know that the popular verbal and numerical reasoning tests can be considered equivalent across modes of administration. This means that current manuals and norms can be retained. For more specialised ‘Power’ tests in the Diagrammatic, Spatial and Mechanical area the limited results so far also suggest equivalence. The research on the Clerical test has highlighted the difficulty of achieving equivalence for Speed tests. New norms are required, and further studies to ascertain any differences in the construct measured, and validity. The candidate responses overall (sample size weighted average) show a clear preference for on-screen tests which is coupled with a slight reduction of test anxiety in that mode. Increased candidate familiarity with computers and technological advances make on-screen assessment the natural test administration medium in the new Millennium.
References
Agapitou, G. (1993). Evaluation of the equivalence of the automated mode of administration and the time limit effect on performance in selection tests. MSc Dissertation. University of Hull.
Aldridge, S. (1994). Psychometric testing and computer based assessment. MSc dissertation. University of East London.
Bartram, D. (1994), Computer-based Assessment. International Review of Industrial and Organizational Psychology, 9, 31-69.
Clausen, M. & Wolf, M. (1995). Vergleich zwischen paper-pencil und computerunterstuetzter Fassung von VMG1 and NMG1. University of Kiel.
Evans, T. (2002). Equivalence between Internet-based and paper-and-pencil cognitive ability tests and factors associated with the test mode effect. MSc dissertation. Birkbeck College.
Kurz, R., Torrijos-Sanchez, M. & Bartram, D. (1994). Computer based Assessment: Equivalence or Superiority? Paper at the BPS Occupational Psychology Conference, Birmingham, UK.
Kurz, R. (1997). Developing ‘Expert’ computerised tests. Paper presented at the Fifth European Congress of Psychology in Dublin.
Kurz, R. (1998). Perspectives on the Design of Computer Based Tests. Paper at the Computers in Psychology Conference, York, UK.
Kurz, R. (2000). The Facets of Occupational Testing: General Reasoning Ability, Residual Aptitudes & Speed-Accuracy Balance. PhD dissertation. Manchester: UMIST.
Mead, A. D. & Drasgow, F. (1993).Equivalence of computerized and paper-and-pencil cognitive ability tests: A Meta-analysis. Psychological Bulletin, 114, 449-458.
Torrijos-Sanchez, M. (1991). Evaluation of the automated from of the MGIB tests. MSc dissertation. University of Hull.
Wheeley, B. (1994). An investigation into the differences between computer administration and paper and pencil administration of occupational ability tests. BSc dissertation. Aston University, Birmingham.
EAWOP Congress in Lisbon, 2003
Performance at Work
Competencies have become a powerful tool for assessing the performance of pe... more Performance at Work
Competencies have become a powerful tool for assessing the performance of people at work because
they are firmly linked to observable behaviour.
they give managers and their staff a common language for discussing development and career potential issues,
they provide a way to express the goals, culture and values of the Organization in terms of the behaviour expected of employees.
Kurz and Bartram (2002) define competencies as:
sets of behaviours that are instrumental in the delivery of desired results or outcomes.
Integrating Competencies and Assessment Psychology
Conventional Competency Modelling approach:
Client-specific
Pragmatic
Varied
Local linking of assessments to competencies
SHL Competency Framework:
Universal framework (can be tailored to clients)
Theory-based
Standardised
Clear a-priori predictor-criterion links
Integrating psychometric theory and competencies practice to create scientifically sound assessment solutions
Research Basis of ‘Great 8’ Competencies
Statistical modelling & expert equation validation:
Kurz (1999) validated expert system equations predicting the 16 IMC competencies from 30 personality, verbal and numerical ability scales across 7 IVS studies; Average validities of 0.50 for self ratings and 0.30 for boss ratings
Goto (1999) cross-validated equations on 2 Asian samples with average validities of .25 for boss ratings
Inspection of r-matrices from these studies suggested most of predictor-criterion variance could be accounted for by 8 criterion factors
Content analysis: Mapping of IMC, CCCI and WSCI competencies (16 each) into a 20 dimension model
Logical concordance: expert judgements (Warr, 1999) of OPQ-IMC scale item content overlap
Theoretical: Big 5 Personality Model, General Mental Ability (GMA), Need for Achievement, Need for Control
‘Great 8’ Competency Factors
LEADING & DECIDING Taking Control & Exercising Power
SUPPORTING & CO-OPERATING Showing Agreeableness & Cooperation
INTERACTING & PRESENTING Showing Extraversion & People Orientation
ANALYSING & INTERPRETING Reasoning with Words, Numbers & Diagrams
CREATING & CONCEPTUALISING Showing Openness to New Ideas & Experiences
ORGANISING & EXECUTING Showing Conscientiousness & Compliance
ADAPTING & COPING Showing Emotional Maturity & Stress Tolerance
ENTERPRISING & PERFORMING Pursuing Challenges & Achievements
SIOP Conference, 2003
SIOP Paper 2003
The paper presents a criterion-centric approach to validation. It argues that th... more SIOP Paper 2003
The paper presents a criterion-centric approach to validation. It argues that the workplace behaviors or competencies we are interested in predicting can usefully be defined in terms of eight broad factors: the Great Eight. Evidence from 33 validation studies is reviewed that supports the utility of this structure. Personality-based
predictors of the Great Eight show strong correlations with their respective line manager or supervisor ratings of the eight competencies, while correlations with other competencies (i.e. those that do not match the predictor) average around zero. Ability tests correlate, as predicted, with three of the Great Eight competencies. In combination, ability and personality data yield corrected sample-weighted correlations
ranging from 0.23 to 0.44 for the eight predictor-competency pairs. Canonical correlation between the uncorrected predictor and criterion correlation matrices show that personality and ability measures together share over 40% of the criterion variance (R=0.65). Results are also provided showing how the Great Eight relate to overall
measures of job performance and to ratings of promotability. Finally, comparisons are drawn between using the Great Eight and the Big Five as models for the personality predictor domain.
See also Bartram (2005) JAP article.
BPS Occupational Psychology Conference, 2003
Baron, H., Bartram, D. & Kurz, R. (2003). The Great 8 as a framework for validation research. Pro... more Baron, H., Bartram, D. & Kurz, R. (2003). The Great 8 as a framework for validation research. Proceedings of the BPS Occupational Psychology Conference, 71-74.
BPS DOP Conference, 2003
What sort of validation?
Process by which the utility of some of assessment can be demonstrated.... more What sort of validation?
Process by which the utility of some of assessment can be demonstrated.
Utility is related to increasing or enhancing organisational effectiveness.
Conventionally:
Measure validity in terms of predictor-criterion correlations
Typically these are ‘many-to-few’ or ‘many-to-one’ relationships
Assess utility using ‘dollar criterion’
Are we asking the right question?
Conventional approach is predictor-centric.
We ask questions that categorise the world in terms of psychological measurement:
What can we predict with instrument X?
OPQ32, MQ, MGIB,
What is the value of ability tests as predictors of training outcome?
Utility equations based on composite predictor ‘r’ values and dollar criterion estimates
What question should we ask?
To get beyond bland generalisations, need a well-articulated model of the domain of work-place behaviours
We need to ask ‘How do we best predict Y?’
Where ‘Y’ is some meaningful, well-defined and important aspect of workplace behaviour.
Our model of the criterion domain not our model of the predictor domain should drive validation research.
Focus on the predictor domain has resulted in clear well articulated specification of that, but a fuzzy ill-defined specification of the criteria.
Criterion-focus re-defines predictors as measures of competency potential
Our validation methodology needs to distinguish:
Competency potential: Measures of dispositional attributes, personality and emotional intelligence, critical faculties, managerial judgement
Competencies: Behaviours, personal performance, and colleagues’ perceptions
Outcomes: Achievements, impacts on organisation’s business and financial performance, track record etc
Competencies
Competencies have become a powerful tool for assessing the performance of people because
they are firmly linked to observable behaviour.
they give managers and their people a common language for discussing development and career potential issues,
they provide a way to express the goals, culture and values of the Organization in terms of the behaviour expected of employees.
Kurz and Bartram (2002) define competencies as:
sets of behaviours that are instrumental in the delivery of desired results or outcomes.
What is a ‘competency framework’
Lots of people talk about competency frameworks, when what they mean are just collections of competencies.
A framework is an articulated set of relationships
It defines the components of a model.
How those components relate to each other
How they relate to other constructs (performance, personality etc).
It is evidence-based.
Deep and surface structures
The framework specifies the generic ‘deep structure’ of the competency domain
Specific competency models are ‘surface structures’
Measurements of competencies are ‘surface measures’.
‘Construct validity’ of competency measures can vary as a function of:
Measurement method – ratings, exercises etc
Assessor perspective – self, boss, peer, etc
A new validation model
The competency framework provides a common basis for describing both predictor measures and criterion measures
Great 8 predictor scores can be generated from combinations of ability, personality and motivation measures
Great 8 criterion scores can be generated by mapping competency ratings to the competency framework
Validity can then be examined using a one-to-one mapping of Great 8 predictor-criterion pairings.
ICP Congress in Montreal, 1996
Presentation Outline
Computer Based Assessment in Careers Guidance
Study Objectives & Hypothe... more Presentation Outline
Computer Based Assessment in Careers Guidance
Study Objectives & Hypothesis
Sample & Methodology
Results & Discussion
Conclusions
Study Objectives
Research psychometric properties of the AIMS
modules
Investigate Construct Validity by integrating AIMS
under the headings of the Holland Theory of
Vocational Type
Gather candidate’s views on instruments and process
Hypothesis: test results likely to be robust; AIMS
components are likely to correlate in a meaningful way
to the Holland types; candidates are likely to feel
positive about materials used, process and outcome
Sample
N = 152 UK students and adults in higher education
75% no test experience
Mean age 25.34 with SD of 10.39
37.5% male, 62.5% female
Methodology
Mouse Practice
6 Ability Tests (Verbal, Numerical, Diagramming,
Checking, Spatial and Mechanical)
Advanced Occupational Interest Inventory
Motivation Questionnaire
Occupational Personality Questionnaire
Evaluation Forms
Conclusions
Strong support for using AIMS in Careers Guidance
Successful AIMS integration under Holland Types
Blueprint for “The automated Assessment /
Development Centre”
Future Research:
- Factor analyse General and / or Management Interest
Inventory to generate Holland Types
- Computerised Versions of GAP
- More universal Profile of Occupational Preferences
- Integrated Ability, Interest, Motivation and Style
EXPERT Systems
Test User Conference in Scarborough, 1999
Paper at the Fourth Test User Conference, Scarborough, 10 June 1999
Presentation Outline
Backg... more Paper at the Fourth Test User Conference, Scarborough, 10 June 1999
Presentation Outline
Background
Project Objectives & Hypothesis
Methodology & Samples
Results & Discussion
Conclusions & Outlook
Background:
Competencies
Definition of Competencies: Multidimensional
Constructs that drive successful behavior patterns
Competency Constructs (Campbell 1993, Kurz 1999):
Multiply determined by antecedents (Ability, Interest,
Motivation, Style) and determinants (Knowledge, Skill,
Experience, Qualification)
Competency Requirements: Behavior Requirements
multiply determined by Job Objectives, Tasks, Context
and Culture
Competency Behaviors: Observed Behavior Patterns
that reflect successful Performance at Work
Background:
Managerial Competencies
Boyatzis (1982)
Schroder (1989)
Sparrow (1993)
Dulcewics (1992)
Inventory of Management Competencies (SHL,1992)
Perspectives in Management Competencies
(SHL,1994)
Background:
Validity of Ability & Personality Variables
Validity Reviews (e.g. Ghiselli, 1973)
Validity Generalisation (e.g. Hunter & Schmidt, 1986)
Meta Analysis of Validity Studies (e.g. Schmitt et
al.,1984; Barrick & Mount, 1989)
Meta Analysis of 20 SHL Validity Studies (Robertson &
Kinder, 1993)
International Validation Studies (Gibbons et al., 1995)
Background:
Computer Based Test Interpretation
(CBTI)
since 1960’ies especially for MMPI and other clinical
instruments
since 1987 Occupational Personality Questionnaire
(OPQ) Expert System Management Skills Potential
1990 Work Profiling System: Human Attribute Based
Person-Job Match
1995 OPQ Competency Based Narrative
1995 Competency Based Person-Job Match
1999 DecisionMaker
1999 OPQ32 vs. IMC Potential
Methodology
Standardised Validation Studies as pioneered in 4
AMT - IMC Studies (SHL, 1992)
Concurrent Design
Predictors:
- Occupational Personality Questionnaire (CM4.2)
- Verbal Critical Reasoning (VMG)
- Numerical Critical Reasoning (NMG)
Criteria:
- Inventory of Management Competencies (16 scales)
- Normative and Ipsative Sub-scales
- ‘Boss’ & ‘Self’ Ratings
Validation Studies -
Sample Sizes, Country, Industry Sector
A: N=131, UK, Distribution
B: N=34, UK, Banking
C: N=90, UK, Financial Services
D: N=114, UK, Paper Manufacturing
E: N=68, UK, Banking
F: N=103, USA, Rubber Manufacturing
G:N=503, Turkey, Glass Manufacturing
Designing Prediction Equations
Statistical Analysis: First order and multiple
correlations between predictors and criteria (within
and across studies)
Expert Judgement: A-priori hypothesised relations
Social Judgement: Logical Concordance (Warr, 1997):
Content Validity: Justifiable from Job Analysis
Measurement: Exclusion of Artefacts (Ipsativity, Halo)
Inclusion of Scales meeting all Criteria with Unit
Weights of 1, 2 ,3 or 4
Results:
Validity
Validation of Weighted Composite Prediction Scores
Predictions are overall stable & valid across jobs,
organisations and cultures for most competencies
Good validities for Ability (Median around .25; up to
.47)
High Validities for Personality variables vs. ‘Boss’
(Median around .3; up to .52)
High Validities for Personality variables vs. ‘Self’’
(Median around .5; up to .76)
Conclusions & Outlook
Highly successful Research & Development Project
Impressive Validity Evidence
Competency Based P-J Match now in use world-wide
Competency Based OPQ Narrative displaying &
verbalising Prediction Scores under Preparation
Future Research & Development:
- Integration of other predictors
- Integration under different categories
- User friendly report design
- Documentation
European Congress of Psychology, 2001
Poster at the European Congress of Psychology
Dr. Rainer H. Kurz, SHL Group plc
London, 2 July 2001
Uploads
Books by Rainer Kurz
Conference Presentations by Rainer Kurz
The first paper reviews milestones in the development and evaluation of CBA systems based on Bartram & Bayliss (1984). Validation principles are illustrated through the correlation of personality, ability, interest, and motivation scales with Great 8 competency factors.
The second paper outlines contemporary advances in personality, ability and competency assessment and reporting through Computer-Based Test Interpretation (CBTI) in the light of Bartram (1994) which shaped BPS and EFPA test review processes. Versatile use of assessment data for multi-level reporting across tools are illustrated and backed with validation data.
The third paper illustrates applications of Sociomapping (Bahbouh, 2012) to psychometric assessment reporting at group and construct set level. Topographical representation of relationships between Big 5 and Great 8 constructs align to Stability and Plasticity in Cybernetic Big 5 Theory (DeYoung, 2015). Case studies on a 'Talent' group and a vet team are presented.
The fourth paper outlines a leading-edge approach involving continuous assessment of wellbeing which presents a measurement model for Employee Experience (EX) based on continuous adaptive micro interactions. The system captures trends in EX data in teams and organisations over time, while accounting for the dynamic relationship between contextual factors in the work environment and internal mindset factors that together influence an individual's EX.
The fifth paper outlines how test review processes have been impacted by advances in CBA. Issues include the use of item banks and CBTI reports products where reviewers will depend on technical documentation featuring psychometric data, especially validation results, to assess the appropriateness of the interpretations.
As artificial intelligence becomes more prevalent, it is timely to discuss past, present and future of CBA. The discussant slot will review the contributions and draw out themes, challenges, and opportunities.
Discussant: Dragos Iliescu, University of Bucharest
Intelligent Testing Systems: Past, Present and Future
Rainer Hermann Kurz, PhD
HUCAMA Analytics, London
Bartram & Bayliss (1984) wrote about the future use of computers in assessment. This paper reviews issues surrounding seven components of a fully automated Intelligent Testing System (ITS) and demonstrates how empirical data can be used to build valid system. 1. Test choice requires a sophisticated understanding of the assessment purpose, tools available and the importance of the scales. 2. Administration requires robustness against Intellectual Property content theft and faking. 3. Scoring requires transparency, sensitivity, and validity. 4.Interpretation needs to be based on valid information presented in understandable form 5. Feedback must be framed appropriate for the recipient whether candidates or 4rd parties. 6. Decision-making must be embedded within legal frameworks and proportionate to the quality of the evidence gathered. 7. Monitoring of outcomes is crucial to establish and enhance validity.
With the advent of AI there is a risk that poor assessment practices will proliferate especially if poor off-line processes are perpetuated and magnified through AI. On the other hand, AI that builds on and integrates sound testing practices could add value.
A data set where 250 individuals completed ability, personality, motivation, and competency assessments is utilised to illustrate data-driven development approaches. Predictors for the Great 8 Competencies (Kurz & Bartram, 2002) are calculated based on the meta-analysis of Bartram (2005) to illustrate how the scales of these five assessment modalities inter-relate. The two highest correlations for the factors are identified for each scale to facility the development of expert systems based on Great 8 and Big 5 models – whether using conventional or AI approaches.
Computer-Based Assessment across Personality, Ability and Competency Factors
Michele Guarini
HUCAMA Group, Copenhagen
This paper builds on the pioneering work of Bartram (1994) on Computer-Based Test Interpretation (CBTI) with a lens on group reporting based on the Great 8 Success Factors inspired by Kurz & Bartram (2002).
Personality Factors:
The modular range features general, professional, and executive level versions with 80, 160 and 240 questions measuring 16, 32 and 48 facets respectively grouped into 8 factors.
A. The Role Wheel Report uses ipsatised data (Bartram, 1996) to remove the effect of individual response style for enhanced group reporting.
B. The Leadership Report maps 48 personality facets to 8 Primary Colours of Leadership constructs (Pendelton, Furnham & Cowell, 2021) with an observed validity with external reviewer ratings of .50 (N=113).
C. The Aspects Reports covers emotional and operational themes that underpin potential and performance fully integrating Emotional Intelligence and Learning Agility constructs. The median construct convergence for the nearest counterpart scale was .63 for the 15 EQi 2.0 facets (N=101) and .70 for its five higher-order compounds.
Ability Factors:
This assessment consists of diagrammatic, numerical, spatial, and verbal component tests with a time limit of 8 minutes each. Rule-based item generation builds on Kurz (1990) and reporting on Kurz (2000) featuring Supra-scores across areas and sub-scores for speed, accuracy and caution.
Competency Factors:
This inventory features an overarching GETTING IT RIGHT, GETTING ALONG, GETTING AHEAD and GETTING ALIGNED model that builds on the Schwartz (1993) values circumplex and the Hogan & Holland (2004) view on performance. Each quadrant pairs up two Great 8 factors. Extreme tie-breaker data is collected and item level results shown.
A Potential & Performance Solution gap analysis tool brings together reporting across personality, ability and competency assessments for individuals and groups using Sten scores on a dynamic dashboard that enables multi-level integration, interpretation, and interrogation of data.
Sociomapping and Team Profile Analyzer in Psychometric Assessment
Pauline Willis
Lauriate, Australia
Sociomapping (Bahbouh, 2012) is an innovative method for tracking quality and frequency of communication in organisation. This paper outlines two applications of the underlying methodology to psychometric assessment.
STORM software uses scale correlations to produce a topographical ‘heat map’ that indicates the centrality of scales and maps out the relationship between constructs. Correlations (N=308) of Big 5 scales (based on NEO) with Great 8 (Kurz & Bartram, 2002) constructs revealed the centrality of Emotional Stability together with motivational (Need for Achievement & Power) constructs to the variable set. Stability (Alpha) vs Plasticity (Beta) meta-factors delineated one axis whereas People vs. Task delineated the other. The graph illustrates how constructs interrelate and facilitate understanding of the nature of ‘derailment’ scales. A cross-validation on N=466 largely confirm the results using different questionnaires.
Team Profile Analyzer (TPA) software produces a map that indicates the centrality of individuals to groups and similarities between group members. In the ‘Sociomap of Profile Similarity’ each group member is represented by a point and mutual distances represent mutual similarity of individual profiles. A heat map colour scheme indicates centrality. Personality assessment results for 16 ‘Elite’ performers across business, arts and sports were analysed using TPA. 10 group members shared many characteristics whereas 6 were different at the highest level of analysis with lower-level scores illuminating the origin of higher-order trends. The analysis explored differences between business leaders and others as well as sex differences. TPA analysis on Personality Factors results for a veterinary team will also be presented.
The applications demonstrate the power of group level reporting across psychometric results. The question arises how advances in academic theory building, such as the Periodic Table of Personality (Woods & Anderson, 2016) and Cybernetic Big 5 Theory (DeYoung, 2015), can be built upon and integrated with the Sociomapping methodology.
Insights in Motion: A Comprehensive Model for Tracking Employee Experience Over Time
Richard T. Justenhoven, PhD
Welliba, Germany
The varying stability across time and situations different constructs exhibit is well known and subject of ongoing research (Steyer et al., 2015). Talent assessment and management tools continue to evolve, and technological advancements enable capturing data in ever increasing breadth and depth.
This enables increasingly nuanced approaches to measuring constructs over time. This paper presents a measurement model for Employee Experience (EX) based on continuous adaptive micro interactions (CadaMint) as one example of this (Preuss et al., 2023). Grounded in Self-Determination-Theory (Deci & Ryan, 2000; Yeager & Dweck, 2020) and the Job-Demands-Resources Model (Bakker & Demerouti, 2007) CadaMint captures trends in EX data in teams and organisations over time, while accounting for the dynamic relationship between contextual factors in the work environment and internal mindset factors that together influence an individual's EX.
As data on EX often covers quarterly or annual cycles, CadaMint accounts for decreasing reliability of data on individual level as measurements age and increases when new measurements for the same constructs are added. This is achieved through a set of characteristics assigned to each instrument and variable, determining how measurements age. Interactions of different factors and fluctuations over time not only pose challenges to measurement, but also to the way outputs are presented to HR profes...
Digman (1997) found Alpha (Agreeableness, Conscientiousness & Emotional Stability) and Beta (Extraversion & Openness) higher-order meta-factors of the Big 5 that DeYoung, Peterson & Higgins (2001) renamed ‘Stability’ and ‘Plasticity’. Judge, Piccolo, & Ilies (2004) found high validity in the Ohio leadership studies for ‘Consideration’ and ‘Initiating Structure’ which appear to cover people and task aspects respectively.
In line with Barrick & Prengler (2018). Kurz (2023) related both to the universal values circumplex of Schwarz (1992) to create a four-quadrant model and developed the Personality Factors range of questionnaires to operationalise Cybernetic Great 8 Theory that offers opportunities to enhance assessments in several ways.
‘RESPONSE STYLE’ is a score based on the sum of all positive qualities that approximates the General Factor of Personality (GFP) which Van der Linden et al (2017) considered a factor of Social Effectiveness that is very closely related to Emotional Intelligence.
In addition to normative reporting, Intra-Personal scores based on an item-level variation of Bartram (1996) generate reports that are ipsatised thus resolving the normative-ipsative debate.
‘STABILITY’ and ‘PLASTICITY’ scores are displayed, and their difference interpreted as a contrast between dependability and proactivity. Extreme difference scores are considered ‘out-of-balance’ and indicative of increased derailment risk'.
‘TASK’ and ‘PEOPLE’ scores are displayed, and their difference interpreted with the aid of paradoxes based on a quasi-circumplex arrangement of Success Factors.
Modular general (PF16), professional (PF32) and executive (PF48) versions cater for different job levels.
Psychometric data on the development sample (N=466) and case study vignettes will be presented. Parallels and differences between Big 5 and Great 8 approaches, and links to the Periodic Table of Personality will be discussed.
Studying at the University of Hull under Prof Dave Bartram he secured his first job at SHL and proceeded to explore rule-based item generation, open answer format and speed-accuracy trade-odd in his MSc project.
At SHL he researched MS-DOS, Windows, and WWW test technology. He developed Competency Potential algorithms, ‘Which Way’ career apps and the Great 8 Competencies (Kurz & Bartram, 2002).
Further milestones involved the development of Saville Consulting Wave. Performance 360 and Swift aptitude tests.
At Cubiks be pushed forward the revolutionary PAPI Dynamic Report and the Horizon solution with dynamic top-tie breaker and ‘nipsatised’ scoring.
Most recently he has been presenting on the Periodic Table of Personality and how Cybernetic Big 5 Theory (DeYoung, 2005) could be improved.
He is a passionate advocate for good assessment practice and has shared very critical views on psychometric assessment issues in the TI newsletter of the International Test Commission and at conferences across Europe.0
Rainer will review developments in the field, explain why he hates multiple regression and share his concerns about artificial intelligence.
Mapping Measures to the Periodic Table of Personality
Symposium at the BPS DOP 2020 Conference
Psychological Assessment at Work
This symposium maps personality questionnaire scales to the Periodic Table of Personality (Woods & Anderson, 2016) using the Trait Descriptive Adjectives (TDA) measure of Goldberg (1992). PCA with Varimax rotation on results for N=1213 professionals and managers were used to create orthogonal regression-based factor scores. The mean age of this group was 46.06 (SD 12.86). 63% of the group identified as Female.
The first two papers are based on a co-validation sub-group (N=128) that completed PAPI 3 SL, HPI, HDS and MVPI questionnaires in addition to TDA.
The first paper is concerned with the construct validation of a Big 5 + Achieving research model operationalised through 30 PAPI scales.
The second paper is concerned with the construct validation of the scales of the Hogan questionnaires.
The third paper maps 10 Lumina Spark Aspects representing both poles of each Big 5 factor and 40 sub-ordinated Qualities with adaptive and maladaptive sub-scores.
Paper 1 Summary:
Kurz (2020). Locating Scales of a Multi-level ‘Big 5 + Achieving’ Measure on the Periodic Table of Personality
The first paper is concerned with the construct validation of a Big 5 + Achieving research model operationalised through 30 PAPI scales with TDA scores (N=128) following the methodology of Woods & Anderson (2016). Scores for 20 PAPI dimensions and 10 Facets were aggregated into six factor scores using integer weights based on a prior large-scale validation study. Factors were paired up into Engaging, Enterprising and Excelling clusters which in turn were aggregated into a Performing Total score. TDA correlations largely were as expected determining construct locations on the Periodic Table of Personality. The model resolves the conundrum that Achieving constructs are variously located within Conscientiousness or Extraversion domains, or even at the opposite end of Agreeableness. The three-cluster model builds on the Three Effectiveness Factors (Kurz, Saville & MacIver, 2009) which in turn were inspired by the Alpha and Beta factors of Digman (1997). The results show strong overlap between the Performing Total based on PAPI and the sum of the TDA factors representing the General Factor of Personality (Musek, 2007). The results provide confirmation for the validity and value of multi-level measurement of personality variables.
Paper 2 Summary:
Groenewald & Kurz (2020). Mapping Hogan Bright Side, Dark Side and Inside Scales to the Periodic Table of Personality.
This paper is concerned with the construct validation of Hogan HPI, HDS and MVPI scale constructs with TDA scores (N=128) following the methodology of Woods & Anderson (2016). Scores for all scales as well as summary scores within and across instruments were mapped to the Periodic Table of Personality.
HPI results closely concur with the relevant TDA Big 5 scores although Learning Approach was largely independent.
HDS scores correlated negatively with Extraversion and Emotional Stability for Moving Away scales, positively with Extraversion and Openness for Moving Against scale and positively with Conscientiousness and Agreeableness for Moving Towards scales.
MVPI scales were less strongly correlated with TDA Big 5 scores but generally followed a pattern compatible with the research of Schwartz (1992) on universal values.
The results shed further light on the nature of bright side, dark side and inside measures by mapping them onto the Periodic Table of Personality.
Paper 3 Summary:
Desson (2020). Mapping Lumina Spark to the Periodic Table of Personality
This paper is concerned with the construct validation of Lumina Spark scale constructs with TDA scores (N=671) following the methodology of Woods & Anderson (2016). Scores for 10 Aspects covering the opposing poles of each Big 5 factor and 40 qualities were correlated with orthogonal TDA scores following Woods & Anderson (2016). Most correlations are in line with expectations although a few deviate somewhat e.g. there is more overlap than expected in that Conscientiousness (convergent thinking) and Openness (divergent thinking) attract opposing constructs rather than showing independence. Introverted and Discipline Driven are ‘factor-pure’ aspects whereas Observing, Intimate, Collaborative, Regard for Others, Focuses Feelings, Resilient and Responsive achieve this status at Qualities level.
Maladaptive sub-scales generally follow the correlation pattern of adaptive sub-scales but vary considerably in their correlation with Emotional Stability.
For the most part, Aspects and Qualities were found to correlate as expected, and thus shows the robust convergent validity between Lumina Spark and the TDA. Furthermore, by conceptualising opposite sectors of the circumplex model as discrete dimensions, we were able to show that these sectors are not necessarily direct opposites of each other, with one being the low form of its opposite, but instead discrete Aspects and Qualities can be mapped to these sectors.
Intelligence Theory
Multiple Intelligences (MI)
The Differential Reasoning Model (DREAM)
Competency Behaviour Survey:
Multiple Intelligences Inter-correlations
Interest Correlates
Personality Correlates
Competency Correlates
The World of Work (WoW) Model
Competency at Work
The Differential Reasoning Model (DREAM)
Review of the ability literature and ‘Occupational Testing’ practice lead to the development of the model (Kurz, 2000)
Model contrasts two categories of ability assessment :
Aptitudes (Reasoning/Learning) vs.
Achievements (Knowledge/Skills)
Model distinguishes general abilities (expected to have high traditional ‘g’ loadings) from distinct abilities (expected to have low traditional ‘g’ loadings) with 2 Sectors each:
General (Educational/Practical) vs.
Distinct (Physical/Social)
The four sectors cover 3 Ability Areas each
General Reasoning Ability ‘R’ defined as ‘unit weight’ average of six reasoning areas: V, N, C, D, S, M
Competency Behaviour Survey
Pilot research study (Briceno, 2002)
Joint project with Goldsmith MSc student
Target sample: >120 SHL staff worldwide
Operationalisation:
.xls Spreadsheet Player
Biodata
Ability self-ratings on 12 Multiple Intelligences
‘Great 8’ Competencies (Kurz & Bartram, 2002)
Collection of existing psychometric data for Construct Validation:
Occupational Personality Questionnaire: Big 5
Career Pathfinder In-depth: RIASEC
Multiple Intelligences in the World of Work
Miles (2000): ‘Emotional intelligence may be better conceptualised as an array of behavioural skills dependent on attributes of both personality and ability.’ ‘ …it is probable that emotional intelligence may possess both construct validity and practical utility as a competency (constellation of behaviours) rather than as a particular trait or ability’
Kurz and Bartram (2002) define competencies as ‘sets of behaviours that are instrumental in the delivery of desired results or outcomes. ‘
Kurz (1999) developed the ‘World of Work’ (WoW) model to map out the elements that underlie ‘Person * Environment Interaction’ at work
Basis for SHL Competency Framework (Kurz & Bartram, 2002)
SHL has 25 years of experience in utilising computers in assessment. Since 1990 three generations of on-screen aptitude tests were developed, and a number of studies conducted to research their statistical properties as well as candidate responses. This poster pulls together data from seven studies to address the following questions:
Q1. Are the on-screen and p&p tests psychometrically equivalent?
Q2. Which administration format do candidates prefer?
Q3. On which version do candidates feel more anxious?
Q4. How do results change with technology advances?
Three Generations of SHL On-Screen Aptitude Tests
International Testing System (1990)
MS-Dos Operating System
Verbal (VMG1) and Numerical Reasoning (NMG1)
in 10 Languages
Expert (1993-)
Windows Operating System
VMG1, NMG1, Basic Checking (CP7.1), Diagrammatic Series (DC3.1), Spatial Reasoning (SIT7), Mechanical Comprehension (MT4.1)
On-line Solutions (1997-)
WWW Administration System using Java Applets
Practice Tests
Access Ability (Verbal & Numerical) for unsupervised ‘Distance Assessment’
Verbal (VMG3) & Numerical (NMG3) Reasoning
On-Screen Test Design
In the computerisation of ‘EXPERT’ lessons learned from the MS-DOS predecessor (see Kurz, Torrijos-Sanchez & Bartram (1994) where incorporated e.g. no on-screen calculator; use of mouse/touch screen instead of keyboard; ergonomic design of displays, controls, instructions and examples; and ‘Failsafe’ operation functionality geared towards (virtual) self-administration (see Kurz, 1998). The aim was to achieve transparency, consistency and acceptability across more than 20 questionnaires and 80 tests used with candidates across the entire working population, in over 20 languages. Evaluations by Kurz (1997) demonstrated high candidate satisfaction with instruction and example design, and favourable responses to the Graphical User Interface features. The development of WWW On-line aptitude tests – whether for Supervised or Unsupervised use, posed a number of additional challenges summarised by Evans (2002) relating to user interface, response mode and system feature design.
Study Designs
Test-Retest across Paper & Pencil and Computer mode:
Identical tests with time gap: Study 1, 3-6
Identical test in single session: Study 2
Parallel tests in single session: Study 7
All studies with 2 experimental groups:
Condition P&P-C: Paper & Pencil followed by Computer
Condition C-P&P: Computer followed by Paper & Pencil
Randomised allocation to conditions (except Study 7)
Study 4 also had P&P-P&P and C-C condition
Evaluation forms
Subjects: University Students Study 1, 2, 3 & 5; A-level Students Study 6 & 7; Employees Study 4
Locations: UK apart from Study 5 (Germany)
Psychometric Equivalence Issues
The results of the 7 studies fully support the findings of Mead & Drasgow (1993) that Power tests transfer well to computer format, and Speed tests are problematic. Average sample size weighted Test-Retest reliabilities reached very satisfactory levels of .75 for Verbal, and .80 for Numerical Reasoning, in line with ordinary p&p Test-Retest reliabilities reported in the MGIB manual. The results of Study 6 suggest that other ‘Power’ tests also transfer well to computer format. The research on the Basic Checking test demonstrated the need for different norms, and suggests that Speed tests may well have slightly different measurement characteristics in computerised format. Interestingly, Agapitou (1993) found slightly higher validities for the computerised version of CP7.1 in the prediction of competencies, and Wheeley (1994) found superior validities for computerised versions of VMG and NMG in the prediction of A-level results.
Conclusions
It is reassuring to know that the popular verbal and numerical reasoning tests can be considered equivalent across modes of administration. This means that current manuals and norms can be retained. For more specialised ‘Power’ tests in the Diagrammatic, Spatial and Mechanical area the limited results so far also suggest equivalence. The research on the Clerical test has highlighted the difficulty of achieving equivalence for Speed tests. New norms are required, and further studies to ascertain any differences in the construct measured, and validity. The candidate responses overall (sample size weighted average) show a clear preference for on-screen tests which is coupled with a slight reduction of test anxiety in that mode. Increased candidate familiarity with computers and technological advances make on-screen assessment the natural test administration medium in the new Millennium.
References
Agapitou, G. (1993). Evaluation of the equivalence of the automated mode of administration and the time limit effect on performance in selection tests. MSc Dissertation. University of Hull.
Aldridge, S. (1994). Psychometric testing and computer based assessment. MSc dissertation. University of East London.
Bartram, D. (1994), Computer-based Assessment. International Review of Industrial and Organizational Psychology, 9, 31-69.
Clausen, M. & Wolf, M. (1995). Vergleich zwischen paper-pencil und computerunterstuetzter Fassung von VMG1 and NMG1. University of Kiel.
Evans, T. (2002). Equivalence between Internet-based and paper-and-pencil cognitive ability tests and factors associated with the test mode effect. MSc dissertation. Birkbeck College.
Kurz, R., Torrijos-Sanchez, M. & Bartram, D. (1994). Computer based Assessment: Equivalence or Superiority? Paper at the BPS Occupational Psychology Conference, Birmingham, UK.
Kurz, R. (1997). Developing ‘Expert’ computerised tests. Paper presented at the Fifth European Congress of Psychology in Dublin.
Kurz, R. (1998). Perspectives on the Design of Computer Based Tests. Paper at the Computers in Psychology Conference, York, UK.
Kurz, R. (2000). The Facets of Occupational Testing: General Reasoning Ability, Residual Aptitudes & Speed-Accuracy Balance. PhD dissertation. Manchester: UMIST.
Mead, A. D. & Drasgow, F. (1993).Equivalence of computerized and paper-and-pencil cognitive ability tests: A Meta-analysis. Psychological Bulletin, 114, 449-458.
Torrijos-Sanchez, M. (1991). Evaluation of the automated from of the MGIB tests. MSc dissertation. University of Hull.
Wheeley, B. (1994). An investigation into the differences between computer administration and paper and pencil administration of occupational ability tests. BSc dissertation. Aston University, Birmingham.
Competencies have become a powerful tool for assessing the performance of people at work because
they are firmly linked to observable behaviour.
they give managers and their staff a common language for discussing development and career potential issues,
they provide a way to express the goals, culture and values of the Organization in terms of the behaviour expected of employees.
Kurz and Bartram (2002) define competencies as:
sets of behaviours that are instrumental in the delivery of desired results or outcomes.
Integrating Competencies and Assessment Psychology
Conventional Competency Modelling approach:
Client-specific
Pragmatic
Varied
Local linking of assessments to competencies
SHL Competency Framework:
Universal framework (can be tailored to clients)
Theory-based
Standardised
Clear a-priori predictor-criterion links
Integrating psychometric theory and competencies practice to create scientifically sound assessment solutions
Research Basis of ‘Great 8’ Competencies
Statistical modelling & expert equation validation:
Kurz (1999) validated expert system equations predicting the 16 IMC competencies from 30 personality, verbal and numerical ability scales across 7 IVS studies; Average validities of 0.50 for self ratings and 0.30 for boss ratings
Goto (1999) cross-validated equations on 2 Asian samples with average validities of .25 for boss ratings
Inspection of r-matrices from these studies suggested most of predictor-criterion variance could be accounted for by 8 criterion factors
Content analysis: Mapping of IMC, CCCI and WSCI competencies (16 each) into a 20 dimension model
Logical concordance: expert judgements (Warr, 1999) of OPQ-IMC scale item content overlap
Theoretical: Big 5 Personality Model, General Mental Ability (GMA), Need for Achievement, Need for Control
‘Great 8’ Competency Factors
LEADING & DECIDING Taking Control & Exercising Power
SUPPORTING & CO-OPERATING Showing Agreeableness & Cooperation
INTERACTING & PRESENTING Showing Extraversion & People Orientation
ANALYSING & INTERPRETING Reasoning with Words, Numbers & Diagrams
CREATING & CONCEPTUALISING Showing Openness to New Ideas & Experiences
ORGANISING & EXECUTING Showing Conscientiousness & Compliance
ADAPTING & COPING Showing Emotional Maturity & Stress Tolerance
ENTERPRISING & PERFORMING Pursuing Challenges & Achievements
The paper presents a criterion-centric approach to validation. It argues that the workplace behaviors or competencies we are interested in predicting can usefully be defined in terms of eight broad factors: the Great Eight. Evidence from 33 validation studies is reviewed that supports the utility of this structure. Personality-based
predictors of the Great Eight show strong correlations with their respective line manager or supervisor ratings of the eight competencies, while correlations with other competencies (i.e. those that do not match the predictor) average around zero. Ability tests correlate, as predicted, with three of the Great Eight competencies. In combination, ability and personality data yield corrected sample-weighted correlations
ranging from 0.23 to 0.44 for the eight predictor-competency pairs. Canonical correlation between the uncorrected predictor and criterion correlation matrices show that personality and ability measures together share over 40% of the criterion variance (R=0.65). Results are also provided showing how the Great Eight relate to overall
measures of job performance and to ratings of promotability. Finally, comparisons are drawn between using the Great Eight and the Big Five as models for the personality predictor domain.
See also Bartram (2005) JAP article.
Process by which the utility of some of assessment can be demonstrated.
Utility is related to increasing or enhancing organisational effectiveness.
Conventionally:
Measure validity in terms of predictor-criterion correlations
Typically these are ‘many-to-few’ or ‘many-to-one’ relationships
Assess utility using ‘dollar criterion’
Are we asking the right question?
Conventional approach is predictor-centric.
We ask questions that categorise the world in terms of psychological measurement:
What can we predict with instrument X?
OPQ32, MQ, MGIB,
What is the value of ability tests as predictors of training outcome?
Utility equations based on composite predictor ‘r’ values and dollar criterion estimates
What question should we ask?
To get beyond bland generalisations, need a well-articulated model of the domain of work-place behaviours
We need to ask ‘How do we best predict Y?’
Where ‘Y’ is some meaningful, well-defined and important aspect of workplace behaviour.
Our model of the criterion domain not our model of the predictor domain should drive validation research.
Focus on the predictor domain has resulted in clear well articulated specification of that, but a fuzzy ill-defined specification of the criteria.
Criterion-focus re-defines predictors as measures of competency potential
Our validation methodology needs to distinguish:
Competency potential: Measures of dispositional attributes, personality and emotional intelligence, critical faculties, managerial judgement
Competencies: Behaviours, personal performance, and colleagues’ perceptions
Outcomes: Achievements, impacts on organisation’s business and financial performance, track record etc
Competencies
Competencies have become a powerful tool for assessing the performance of people because
they are firmly linked to observable behaviour.
they give managers and their people a common language for discussing development and career potential issues,
they provide a way to express the goals, culture and values of the Organization in terms of the behaviour expected of employees.
Kurz and Bartram (2002) define competencies as:
sets of behaviours that are instrumental in the delivery of desired results or outcomes.
What is a ‘competency framework’
Lots of people talk about competency frameworks, when what they mean are just collections of competencies.
A framework is an articulated set of relationships
It defines the components of a model.
How those components relate to each other
How they relate to other constructs (performance, personality etc).
It is evidence-based.
Deep and surface structures
The framework specifies the generic ‘deep structure’ of the competency domain
Specific competency models are ‘surface structures’
Measurements of competencies are ‘surface measures’.
‘Construct validity’ of competency measures can vary as a function of:
Measurement method – ratings, exercises etc
Assessor perspective – self, boss, peer, etc
A new validation model
The competency framework provides a common basis for describing both predictor measures and criterion measures
Great 8 predictor scores can be generated from combinations of ability, personality and motivation measures
Great 8 criterion scores can be generated by mapping competency ratings to the competency framework
Validity can then be examined using a one-to-one mapping of Great 8 predictor-criterion pairings.
Computer Based Assessment in Careers Guidance
Study Objectives & Hypothesis
Sample & Methodology
Results & Discussion
Conclusions
Study Objectives
Research psychometric properties of the AIMS
modules
Investigate Construct Validity by integrating AIMS
under the headings of the Holland Theory of
Vocational Type
Gather candidate’s views on instruments and process
Hypothesis: test results likely to be robust; AIMS
components are likely to correlate in a meaningful way
to the Holland types; candidates are likely to feel
positive about materials used, process and outcome
Sample
N = 152 UK students and adults in higher education
75% no test experience
Mean age 25.34 with SD of 10.39
37.5% male, 62.5% female
Methodology
Mouse Practice
6 Ability Tests (Verbal, Numerical, Diagramming,
Checking, Spatial and Mechanical)
Advanced Occupational Interest Inventory
Motivation Questionnaire
Occupational Personality Questionnaire
Evaluation Forms
Conclusions
Strong support for using AIMS in Careers Guidance
Successful AIMS integration under Holland Types
Blueprint for “The automated Assessment /
Development Centre”
Future Research:
- Factor analyse General and / or Management Interest
Inventory to generate Holland Types
- Computerised Versions of GAP
- More universal Profile of Occupational Preferences
- Integrated Ability, Interest, Motivation and Style
EXPERT Systems
Presentation Outline
Background
Project Objectives & Hypothesis
Methodology & Samples
Results & Discussion
Conclusions & Outlook
Background:
Competencies
Definition of Competencies: Multidimensional
Constructs that drive successful behavior patterns
Competency Constructs (Campbell 1993, Kurz 1999):
Multiply determined by antecedents (Ability, Interest,
Motivation, Style) and determinants (Knowledge, Skill,
Experience, Qualification)
Competency Requirements: Behavior Requirements
multiply determined by Job Objectives, Tasks, Context
and Culture
Competency Behaviors: Observed Behavior Patterns
that reflect successful Performance at Work
Background:
Managerial Competencies
Boyatzis (1982)
Schroder (1989)
Sparrow (1993)
Dulcewics (1992)
Inventory of Management Competencies (SHL,1992)
Perspectives in Management Competencies
(SHL,1994)
Background:
Validity of Ability & Personality Variables
Validity Reviews (e.g. Ghiselli, 1973)
Validity Generalisation (e.g. Hunter & Schmidt, 1986)
Meta Analysis of Validity Studies (e.g. Schmitt et
al.,1984; Barrick & Mount, 1989)
Meta Analysis of 20 SHL Validity Studies (Robertson &
Kinder, 1993)
International Validation Studies (Gibbons et al., 1995)
Background:
Computer Based Test Interpretation
(CBTI)
since 1960’ies especially for MMPI and other clinical
instruments
since 1987 Occupational Personality Questionnaire
(OPQ) Expert System Management Skills Potential
1990 Work Profiling System: Human Attribute Based
Person-Job Match
1995 OPQ Competency Based Narrative
1995 Competency Based Person-Job Match
1999 DecisionMaker
1999 OPQ32 vs. IMC Potential
Methodology
Standardised Validation Studies as pioneered in 4
AMT - IMC Studies (SHL, 1992)
Concurrent Design
Predictors:
- Occupational Personality Questionnaire (CM4.2)
- Verbal Critical Reasoning (VMG)
- Numerical Critical Reasoning (NMG)
Criteria:
- Inventory of Management Competencies (16 scales)
- Normative and Ipsative Sub-scales
- ‘Boss’ & ‘Self’ Ratings
Validation Studies -
Sample Sizes, Country, Industry Sector
A: N=131, UK, Distribution
B: N=34, UK, Banking
C: N=90, UK, Financial Services
D: N=114, UK, Paper Manufacturing
E: N=68, UK, Banking
F: N=103, USA, Rubber Manufacturing
G:N=503, Turkey, Glass Manufacturing
Designing Prediction Equations
Statistical Analysis: First order and multiple
correlations between predictors and criteria (within
and across studies)
Expert Judgement: A-priori hypothesised relations
Social Judgement: Logical Concordance (Warr, 1997):
Content Validity: Justifiable from Job Analysis
Measurement: Exclusion of Artefacts (Ipsativity, Halo)
Inclusion of Scales meeting all Criteria with Unit
Weights of 1, 2 ,3 or 4
Results:
Validity
Validation of Weighted Composite Prediction Scores
Predictions are overall stable & valid across jobs,
organisations and cultures for most competencies
Good validities for Ability (Median around .25; up to
.47)
High Validities for Personality variables vs. ‘Boss’
(Median around .3; up to .52)
High Validities for Personality variables vs. ‘Self’’
(Median around .5; up to .76)
Conclusions & Outlook
Highly successful Research & Development Project
Impressive Validity Evidence
Competency Based P-J Match now in use world-wide
Competency Based OPQ Narrative displaying &
verbalising Prediction Scores under Preparation
Future Research & Development:
- Integration of other predictors
- Integration under different categories
- User friendly report design
- Documentation
The first paper reviews milestones in the development and evaluation of CBA systems based on Bartram & Bayliss (1984). Validation principles are illustrated through the correlation of personality, ability, interest, and motivation scales with Great 8 competency factors.
The second paper outlines contemporary advances in personality, ability and competency assessment and reporting through Computer-Based Test Interpretation (CBTI) in the light of Bartram (1994) which shaped BPS and EFPA test review processes. Versatile use of assessment data for multi-level reporting across tools are illustrated and backed with validation data.
The third paper illustrates applications of Sociomapping (Bahbouh, 2012) to psychometric assessment reporting at group and construct set level. Topographical representation of relationships between Big 5 and Great 8 constructs align to Stability and Plasticity in Cybernetic Big 5 Theory (DeYoung, 2015). Case studies on a 'Talent' group and a vet team are presented.
The fourth paper outlines a leading-edge approach involving continuous assessment of wellbeing which presents a measurement model for Employee Experience (EX) based on continuous adaptive micro interactions. The system captures trends in EX data in teams and organisations over time, while accounting for the dynamic relationship between contextual factors in the work environment and internal mindset factors that together influence an individual's EX.
The fifth paper outlines how test review processes have been impacted by advances in CBA. Issues include the use of item banks and CBTI reports products where reviewers will depend on technical documentation featuring psychometric data, especially validation results, to assess the appropriateness of the interpretations.
As artificial intelligence becomes more prevalent, it is timely to discuss past, present and future of CBA. The discussant slot will review the contributions and draw out themes, challenges, and opportunities.
Discussant: Dragos Iliescu, University of Bucharest
Intelligent Testing Systems: Past, Present and Future
Rainer Hermann Kurz, PhD
HUCAMA Analytics, London
Bartram & Bayliss (1984) wrote about the future use of computers in assessment. This paper reviews issues surrounding seven components of a fully automated Intelligent Testing System (ITS) and demonstrates how empirical data can be used to build valid system. 1. Test choice requires a sophisticated understanding of the assessment purpose, tools available and the importance of the scales. 2. Administration requires robustness against Intellectual Property content theft and faking. 3. Scoring requires transparency, sensitivity, and validity. 4.Interpretation needs to be based on valid information presented in understandable form 5. Feedback must be framed appropriate for the recipient whether candidates or 4rd parties. 6. Decision-making must be embedded within legal frameworks and proportionate to the quality of the evidence gathered. 7. Monitoring of outcomes is crucial to establish and enhance validity.
With the advent of AI there is a risk that poor assessment practices will proliferate especially if poor off-line processes are perpetuated and magnified through AI. On the other hand, AI that builds on and integrates sound testing practices could add value.
A data set where 250 individuals completed ability, personality, motivation, and competency assessments is utilised to illustrate data-driven development approaches. Predictors for the Great 8 Competencies (Kurz & Bartram, 2002) are calculated based on the meta-analysis of Bartram (2005) to illustrate how the scales of these five assessment modalities inter-relate. The two highest correlations for the factors are identified for each scale to facility the development of expert systems based on Great 8 and Big 5 models – whether using conventional or AI approaches.
Computer-Based Assessment across Personality, Ability and Competency Factors
Michele Guarini
HUCAMA Group, Copenhagen
This paper builds on the pioneering work of Bartram (1994) on Computer-Based Test Interpretation (CBTI) with a lens on group reporting based on the Great 8 Success Factors inspired by Kurz & Bartram (2002).
Personality Factors:
The modular range features general, professional, and executive level versions with 80, 160 and 240 questions measuring 16, 32 and 48 facets respectively grouped into 8 factors.
A. The Role Wheel Report uses ipsatised data (Bartram, 1996) to remove the effect of individual response style for enhanced group reporting.
B. The Leadership Report maps 48 personality facets to 8 Primary Colours of Leadership constructs (Pendelton, Furnham & Cowell, 2021) with an observed validity with external reviewer ratings of .50 (N=113).
C. The Aspects Reports covers emotional and operational themes that underpin potential and performance fully integrating Emotional Intelligence and Learning Agility constructs. The median construct convergence for the nearest counterpart scale was .63 for the 15 EQi 2.0 facets (N=101) and .70 for its five higher-order compounds.
Ability Factors:
This assessment consists of diagrammatic, numerical, spatial, and verbal component tests with a time limit of 8 minutes each. Rule-based item generation builds on Kurz (1990) and reporting on Kurz (2000) featuring Supra-scores across areas and sub-scores for speed, accuracy and caution.
Competency Factors:
This inventory features an overarching GETTING IT RIGHT, GETTING ALONG, GETTING AHEAD and GETTING ALIGNED model that builds on the Schwartz (1993) values circumplex and the Hogan & Holland (2004) view on performance. Each quadrant pairs up two Great 8 factors. Extreme tie-breaker data is collected and item level results shown.
A Potential & Performance Solution gap analysis tool brings together reporting across personality, ability and competency assessments for individuals and groups using Sten scores on a dynamic dashboard that enables multi-level integration, interpretation, and interrogation of data.
Sociomapping and Team Profile Analyzer in Psychometric Assessment
Pauline Willis
Lauriate, Australia
Sociomapping (Bahbouh, 2012) is an innovative method for tracking quality and frequency of communication in organisation. This paper outlines two applications of the underlying methodology to psychometric assessment.
STORM software uses scale correlations to produce a topographical ‘heat map’ that indicates the centrality of scales and maps out the relationship between constructs. Correlations (N=308) of Big 5 scales (based on NEO) with Great 8 (Kurz & Bartram, 2002) constructs revealed the centrality of Emotional Stability together with motivational (Need for Achievement & Power) constructs to the variable set. Stability (Alpha) vs Plasticity (Beta) meta-factors delineated one axis whereas People vs. Task delineated the other. The graph illustrates how constructs interrelate and facilitate understanding of the nature of ‘derailment’ scales. A cross-validation on N=466 largely confirm the results using different questionnaires.
Team Profile Analyzer (TPA) software produces a map that indicates the centrality of individuals to groups and similarities between group members. In the ‘Sociomap of Profile Similarity’ each group member is represented by a point and mutual distances represent mutual similarity of individual profiles. A heat map colour scheme indicates centrality. Personality assessment results for 16 ‘Elite’ performers across business, arts and sports were analysed using TPA. 10 group members shared many characteristics whereas 6 were different at the highest level of analysis with lower-level scores illuminating the origin of higher-order trends. The analysis explored differences between business leaders and others as well as sex differences. TPA analysis on Personality Factors results for a veterinary team will also be presented.
The applications demonstrate the power of group level reporting across psychometric results. The question arises how advances in academic theory building, such as the Periodic Table of Personality (Woods & Anderson, 2016) and Cybernetic Big 5 Theory (DeYoung, 2015), can be built upon and integrated with the Sociomapping methodology.
Insights in Motion: A Comprehensive Model for Tracking Employee Experience Over Time
Richard T. Justenhoven, PhD
Welliba, Germany
The varying stability across time and situations different constructs exhibit is well known and subject of ongoing research (Steyer et al., 2015). Talent assessment and management tools continue to evolve, and technological advancements enable capturing data in ever increasing breadth and depth.
This enables increasingly nuanced approaches to measuring constructs over time. This paper presents a measurement model for Employee Experience (EX) based on continuous adaptive micro interactions (CadaMint) as one example of this (Preuss et al., 2023). Grounded in Self-Determination-Theory (Deci & Ryan, 2000; Yeager & Dweck, 2020) and the Job-Demands-Resources Model (Bakker & Demerouti, 2007) CadaMint captures trends in EX data in teams and organisations over time, while accounting for the dynamic relationship between contextual factors in the work environment and internal mindset factors that together influence an individual's EX.
As data on EX often covers quarterly or annual cycles, CadaMint accounts for decreasing reliability of data on individual level as measurements age and increases when new measurements for the same constructs are added. This is achieved through a set of characteristics assigned to each instrument and variable, determining how measurements age. Interactions of different factors and fluctuations over time not only pose challenges to measurement, but also to the way outputs are presented to HR profes...
Digman (1997) found Alpha (Agreeableness, Conscientiousness & Emotional Stability) and Beta (Extraversion & Openness) higher-order meta-factors of the Big 5 that DeYoung, Peterson & Higgins (2001) renamed ‘Stability’ and ‘Plasticity’. Judge, Piccolo, & Ilies (2004) found high validity in the Ohio leadership studies for ‘Consideration’ and ‘Initiating Structure’ which appear to cover people and task aspects respectively.
In line with Barrick & Prengler (2018). Kurz (2023) related both to the universal values circumplex of Schwarz (1992) to create a four-quadrant model and developed the Personality Factors range of questionnaires to operationalise Cybernetic Great 8 Theory that offers opportunities to enhance assessments in several ways.
‘RESPONSE STYLE’ is a score based on the sum of all positive qualities that approximates the General Factor of Personality (GFP) which Van der Linden et al (2017) considered a factor of Social Effectiveness that is very closely related to Emotional Intelligence.
In addition to normative reporting, Intra-Personal scores based on an item-level variation of Bartram (1996) generate reports that are ipsatised thus resolving the normative-ipsative debate.
‘STABILITY’ and ‘PLASTICITY’ scores are displayed, and their difference interpreted as a contrast between dependability and proactivity. Extreme difference scores are considered ‘out-of-balance’ and indicative of increased derailment risk'.
‘TASK’ and ‘PEOPLE’ scores are displayed, and their difference interpreted with the aid of paradoxes based on a quasi-circumplex arrangement of Success Factors.
Modular general (PF16), professional (PF32) and executive (PF48) versions cater for different job levels.
Psychometric data on the development sample (N=466) and case study vignettes will be presented. Parallels and differences between Big 5 and Great 8 approaches, and links to the Periodic Table of Personality will be discussed.
Studying at the University of Hull under Prof Dave Bartram he secured his first job at SHL and proceeded to explore rule-based item generation, open answer format and speed-accuracy trade-odd in his MSc project.
At SHL he researched MS-DOS, Windows, and WWW test technology. He developed Competency Potential algorithms, ‘Which Way’ career apps and the Great 8 Competencies (Kurz & Bartram, 2002).
Further milestones involved the development of Saville Consulting Wave. Performance 360 and Swift aptitude tests.
At Cubiks be pushed forward the revolutionary PAPI Dynamic Report and the Horizon solution with dynamic top-tie breaker and ‘nipsatised’ scoring.
Most recently he has been presenting on the Periodic Table of Personality and how Cybernetic Big 5 Theory (DeYoung, 2005) could be improved.
He is a passionate advocate for good assessment practice and has shared very critical views on psychometric assessment issues in the TI newsletter of the International Test Commission and at conferences across Europe.0
Rainer will review developments in the field, explain why he hates multiple regression and share his concerns about artificial intelligence.
Mapping Measures to the Periodic Table of Personality
Symposium at the BPS DOP 2020 Conference
Psychological Assessment at Work
This symposium maps personality questionnaire scales to the Periodic Table of Personality (Woods & Anderson, 2016) using the Trait Descriptive Adjectives (TDA) measure of Goldberg (1992). PCA with Varimax rotation on results for N=1213 professionals and managers were used to create orthogonal regression-based factor scores. The mean age of this group was 46.06 (SD 12.86). 63% of the group identified as Female.
The first two papers are based on a co-validation sub-group (N=128) that completed PAPI 3 SL, HPI, HDS and MVPI questionnaires in addition to TDA.
The first paper is concerned with the construct validation of a Big 5 + Achieving research model operationalised through 30 PAPI scales.
The second paper is concerned with the construct validation of the scales of the Hogan questionnaires.
The third paper maps 10 Lumina Spark Aspects representing both poles of each Big 5 factor and 40 sub-ordinated Qualities with adaptive and maladaptive sub-scores.
Paper 1 Summary:
Kurz (2020). Locating Scales of a Multi-level ‘Big 5 + Achieving’ Measure on the Periodic Table of Personality
The first paper is concerned with the construct validation of a Big 5 + Achieving research model operationalised through 30 PAPI scales with TDA scores (N=128) following the methodology of Woods & Anderson (2016). Scores for 20 PAPI dimensions and 10 Facets were aggregated into six factor scores using integer weights based on a prior large-scale validation study. Factors were paired up into Engaging, Enterprising and Excelling clusters which in turn were aggregated into a Performing Total score. TDA correlations largely were as expected determining construct locations on the Periodic Table of Personality. The model resolves the conundrum that Achieving constructs are variously located within Conscientiousness or Extraversion domains, or even at the opposite end of Agreeableness. The three-cluster model builds on the Three Effectiveness Factors (Kurz, Saville & MacIver, 2009) which in turn were inspired by the Alpha and Beta factors of Digman (1997). The results show strong overlap between the Performing Total based on PAPI and the sum of the TDA factors representing the General Factor of Personality (Musek, 2007). The results provide confirmation for the validity and value of multi-level measurement of personality variables.
Paper 2 Summary:
Groenewald & Kurz (2020). Mapping Hogan Bright Side, Dark Side and Inside Scales to the Periodic Table of Personality.
This paper is concerned with the construct validation of Hogan HPI, HDS and MVPI scale constructs with TDA scores (N=128) following the methodology of Woods & Anderson (2016). Scores for all scales as well as summary scores within and across instruments were mapped to the Periodic Table of Personality.
HPI results closely concur with the relevant TDA Big 5 scores although Learning Approach was largely independent.
HDS scores correlated negatively with Extraversion and Emotional Stability for Moving Away scales, positively with Extraversion and Openness for Moving Against scale and positively with Conscientiousness and Agreeableness for Moving Towards scales.
MVPI scales were less strongly correlated with TDA Big 5 scores but generally followed a pattern compatible with the research of Schwartz (1992) on universal values.
The results shed further light on the nature of bright side, dark side and inside measures by mapping them onto the Periodic Table of Personality.
Paper 3 Summary:
Desson (2020). Mapping Lumina Spark to the Periodic Table of Personality
This paper is concerned with the construct validation of Lumina Spark scale constructs with TDA scores (N=671) following the methodology of Woods & Anderson (2016). Scores for 10 Aspects covering the opposing poles of each Big 5 factor and 40 qualities were correlated with orthogonal TDA scores following Woods & Anderson (2016). Most correlations are in line with expectations although a few deviate somewhat e.g. there is more overlap than expected in that Conscientiousness (convergent thinking) and Openness (divergent thinking) attract opposing constructs rather than showing independence. Introverted and Discipline Driven are ‘factor-pure’ aspects whereas Observing, Intimate, Collaborative, Regard for Others, Focuses Feelings, Resilient and Responsive achieve this status at Qualities level.
Maladaptive sub-scales generally follow the correlation pattern of adaptive sub-scales but vary considerably in their correlation with Emotional Stability.
For the most part, Aspects and Qualities were found to correlate as expected, and thus shows the robust convergent validity between Lumina Spark and the TDA. Furthermore, by conceptualising opposite sectors of the circumplex model as discrete dimensions, we were able to show that these sectors are not necessarily direct opposites of each other, with one being the low form of its opposite, but instead discrete Aspects and Qualities can be mapped to these sectors.
Intelligence Theory
Multiple Intelligences (MI)
The Differential Reasoning Model (DREAM)
Competency Behaviour Survey:
Multiple Intelligences Inter-correlations
Interest Correlates
Personality Correlates
Competency Correlates
The World of Work (WoW) Model
Competency at Work
The Differential Reasoning Model (DREAM)
Review of the ability literature and ‘Occupational Testing’ practice lead to the development of the model (Kurz, 2000)
Model contrasts two categories of ability assessment :
Aptitudes (Reasoning/Learning) vs.
Achievements (Knowledge/Skills)
Model distinguishes general abilities (expected to have high traditional ‘g’ loadings) from distinct abilities (expected to have low traditional ‘g’ loadings) with 2 Sectors each:
General (Educational/Practical) vs.
Distinct (Physical/Social)
The four sectors cover 3 Ability Areas each
General Reasoning Ability ‘R’ defined as ‘unit weight’ average of six reasoning areas: V, N, C, D, S, M
Competency Behaviour Survey
Pilot research study (Briceno, 2002)
Joint project with Goldsmith MSc student
Target sample: >120 SHL staff worldwide
Operationalisation:
.xls Spreadsheet Player
Biodata
Ability self-ratings on 12 Multiple Intelligences
‘Great 8’ Competencies (Kurz & Bartram, 2002)
Collection of existing psychometric data for Construct Validation:
Occupational Personality Questionnaire: Big 5
Career Pathfinder In-depth: RIASEC
Multiple Intelligences in the World of Work
Miles (2000): ‘Emotional intelligence may be better conceptualised as an array of behavioural skills dependent on attributes of both personality and ability.’ ‘ …it is probable that emotional intelligence may possess both construct validity and practical utility as a competency (constellation of behaviours) rather than as a particular trait or ability’
Kurz and Bartram (2002) define competencies as ‘sets of behaviours that are instrumental in the delivery of desired results or outcomes. ‘
Kurz (1999) developed the ‘World of Work’ (WoW) model to map out the elements that underlie ‘Person * Environment Interaction’ at work
Basis for SHL Competency Framework (Kurz & Bartram, 2002)
SHL has 25 years of experience in utilising computers in assessment. Since 1990 three generations of on-screen aptitude tests were developed, and a number of studies conducted to research their statistical properties as well as candidate responses. This poster pulls together data from seven studies to address the following questions:
Q1. Are the on-screen and p&p tests psychometrically equivalent?
Q2. Which administration format do candidates prefer?
Q3. On which version do candidates feel more anxious?
Q4. How do results change with technology advances?
Three Generations of SHL On-Screen Aptitude Tests
International Testing System (1990)
MS-Dos Operating System
Verbal (VMG1) and Numerical Reasoning (NMG1)
in 10 Languages
Expert (1993-)
Windows Operating System
VMG1, NMG1, Basic Checking (CP7.1), Diagrammatic Series (DC3.1), Spatial Reasoning (SIT7), Mechanical Comprehension (MT4.1)
On-line Solutions (1997-)
WWW Administration System using Java Applets
Practice Tests
Access Ability (Verbal & Numerical) for unsupervised ‘Distance Assessment’
Verbal (VMG3) & Numerical (NMG3) Reasoning
On-Screen Test Design
In the computerisation of ‘EXPERT’ lessons learned from the MS-DOS predecessor (see Kurz, Torrijos-Sanchez & Bartram (1994) where incorporated e.g. no on-screen calculator; use of mouse/touch screen instead of keyboard; ergonomic design of displays, controls, instructions and examples; and ‘Failsafe’ operation functionality geared towards (virtual) self-administration (see Kurz, 1998). The aim was to achieve transparency, consistency and acceptability across more than 20 questionnaires and 80 tests used with candidates across the entire working population, in over 20 languages. Evaluations by Kurz (1997) demonstrated high candidate satisfaction with instruction and example design, and favourable responses to the Graphical User Interface features. The development of WWW On-line aptitude tests – whether for Supervised or Unsupervised use, posed a number of additional challenges summarised by Evans (2002) relating to user interface, response mode and system feature design.
Study Designs
Test-Retest across Paper & Pencil and Computer mode:
Identical tests with time gap: Study 1, 3-6
Identical test in single session: Study 2
Parallel tests in single session: Study 7
All studies with 2 experimental groups:
Condition P&P-C: Paper & Pencil followed by Computer
Condition C-P&P: Computer followed by Paper & Pencil
Randomised allocation to conditions (except Study 7)
Study 4 also had P&P-P&P and C-C condition
Evaluation forms
Subjects: University Students Study 1, 2, 3 & 5; A-level Students Study 6 & 7; Employees Study 4
Locations: UK apart from Study 5 (Germany)
Psychometric Equivalence Issues
The results of the 7 studies fully support the findings of Mead & Drasgow (1993) that Power tests transfer well to computer format, and Speed tests are problematic. Average sample size weighted Test-Retest reliabilities reached very satisfactory levels of .75 for Verbal, and .80 for Numerical Reasoning, in line with ordinary p&p Test-Retest reliabilities reported in the MGIB manual. The results of Study 6 suggest that other ‘Power’ tests also transfer well to computer format. The research on the Basic Checking test demonstrated the need for different norms, and suggests that Speed tests may well have slightly different measurement characteristics in computerised format. Interestingly, Agapitou (1993) found slightly higher validities for the computerised version of CP7.1 in the prediction of competencies, and Wheeley (1994) found superior validities for computerised versions of VMG and NMG in the prediction of A-level results.
Conclusions
It is reassuring to know that the popular verbal and numerical reasoning tests can be considered equivalent across modes of administration. This means that current manuals and norms can be retained. For more specialised ‘Power’ tests in the Diagrammatic, Spatial and Mechanical area the limited results so far also suggest equivalence. The research on the Clerical test has highlighted the difficulty of achieving equivalence for Speed tests. New norms are required, and further studies to ascertain any differences in the construct measured, and validity. The candidate responses overall (sample size weighted average) show a clear preference for on-screen tests which is coupled with a slight reduction of test anxiety in that mode. Increased candidate familiarity with computers and technological advances make on-screen assessment the natural test administration medium in the new Millennium.
References
Agapitou, G. (1993). Evaluation of the equivalence of the automated mode of administration and the time limit effect on performance in selection tests. MSc Dissertation. University of Hull.
Aldridge, S. (1994). Psychometric testing and computer based assessment. MSc dissertation. University of East London.
Bartram, D. (1994), Computer-based Assessment. International Review of Industrial and Organizational Psychology, 9, 31-69.
Clausen, M. & Wolf, M. (1995). Vergleich zwischen paper-pencil und computerunterstuetzter Fassung von VMG1 and NMG1. University of Kiel.
Evans, T. (2002). Equivalence between Internet-based and paper-and-pencil cognitive ability tests and factors associated with the test mode effect. MSc dissertation. Birkbeck College.
Kurz, R., Torrijos-Sanchez, M. & Bartram, D. (1994). Computer based Assessment: Equivalence or Superiority? Paper at the BPS Occupational Psychology Conference, Birmingham, UK.
Kurz, R. (1997). Developing ‘Expert’ computerised tests. Paper presented at the Fifth European Congress of Psychology in Dublin.
Kurz, R. (1998). Perspectives on the Design of Computer Based Tests. Paper at the Computers in Psychology Conference, York, UK.
Kurz, R. (2000). The Facets of Occupational Testing: General Reasoning Ability, Residual Aptitudes & Speed-Accuracy Balance. PhD dissertation. Manchester: UMIST.
Mead, A. D. & Drasgow, F. (1993).Equivalence of computerized and paper-and-pencil cognitive ability tests: A Meta-analysis. Psychological Bulletin, 114, 449-458.
Torrijos-Sanchez, M. (1991). Evaluation of the automated from of the MGIB tests. MSc dissertation. University of Hull.
Wheeley, B. (1994). An investigation into the differences between computer administration and paper and pencil administration of occupational ability tests. BSc dissertation. Aston University, Birmingham.
Competencies have become a powerful tool for assessing the performance of people at work because
they are firmly linked to observable behaviour.
they give managers and their staff a common language for discussing development and career potential issues,
they provide a way to express the goals, culture and values of the Organization in terms of the behaviour expected of employees.
Kurz and Bartram (2002) define competencies as:
sets of behaviours that are instrumental in the delivery of desired results or outcomes.
Integrating Competencies and Assessment Psychology
Conventional Competency Modelling approach:
Client-specific
Pragmatic
Varied
Local linking of assessments to competencies
SHL Competency Framework:
Universal framework (can be tailored to clients)
Theory-based
Standardised
Clear a-priori predictor-criterion links
Integrating psychometric theory and competencies practice to create scientifically sound assessment solutions
Research Basis of ‘Great 8’ Competencies
Statistical modelling & expert equation validation:
Kurz (1999) validated expert system equations predicting the 16 IMC competencies from 30 personality, verbal and numerical ability scales across 7 IVS studies; Average validities of 0.50 for self ratings and 0.30 for boss ratings
Goto (1999) cross-validated equations on 2 Asian samples with average validities of .25 for boss ratings
Inspection of r-matrices from these studies suggested most of predictor-criterion variance could be accounted for by 8 criterion factors
Content analysis: Mapping of IMC, CCCI and WSCI competencies (16 each) into a 20 dimension model
Logical concordance: expert judgements (Warr, 1999) of OPQ-IMC scale item content overlap
Theoretical: Big 5 Personality Model, General Mental Ability (GMA), Need for Achievement, Need for Control
‘Great 8’ Competency Factors
LEADING & DECIDING Taking Control & Exercising Power
SUPPORTING & CO-OPERATING Showing Agreeableness & Cooperation
INTERACTING & PRESENTING Showing Extraversion & People Orientation
ANALYSING & INTERPRETING Reasoning with Words, Numbers & Diagrams
CREATING & CONCEPTUALISING Showing Openness to New Ideas & Experiences
ORGANISING & EXECUTING Showing Conscientiousness & Compliance
ADAPTING & COPING Showing Emotional Maturity & Stress Tolerance
ENTERPRISING & PERFORMING Pursuing Challenges & Achievements
The paper presents a criterion-centric approach to validation. It argues that the workplace behaviors or competencies we are interested in predicting can usefully be defined in terms of eight broad factors: the Great Eight. Evidence from 33 validation studies is reviewed that supports the utility of this structure. Personality-based
predictors of the Great Eight show strong correlations with their respective line manager or supervisor ratings of the eight competencies, while correlations with other competencies (i.e. those that do not match the predictor) average around zero. Ability tests correlate, as predicted, with three of the Great Eight competencies. In combination, ability and personality data yield corrected sample-weighted correlations
ranging from 0.23 to 0.44 for the eight predictor-competency pairs. Canonical correlation between the uncorrected predictor and criterion correlation matrices show that personality and ability measures together share over 40% of the criterion variance (R=0.65). Results are also provided showing how the Great Eight relate to overall
measures of job performance and to ratings of promotability. Finally, comparisons are drawn between using the Great Eight and the Big Five as models for the personality predictor domain.
See also Bartram (2005) JAP article.
Process by which the utility of some of assessment can be demonstrated.
Utility is related to increasing or enhancing organisational effectiveness.
Conventionally:
Measure validity in terms of predictor-criterion correlations
Typically these are ‘many-to-few’ or ‘many-to-one’ relationships
Assess utility using ‘dollar criterion’
Are we asking the right question?
Conventional approach is predictor-centric.
We ask questions that categorise the world in terms of psychological measurement:
What can we predict with instrument X?
OPQ32, MQ, MGIB,
What is the value of ability tests as predictors of training outcome?
Utility equations based on composite predictor ‘r’ values and dollar criterion estimates
What question should we ask?
To get beyond bland generalisations, need a well-articulated model of the domain of work-place behaviours
We need to ask ‘How do we best predict Y?’
Where ‘Y’ is some meaningful, well-defined and important aspect of workplace behaviour.
Our model of the criterion domain not our model of the predictor domain should drive validation research.
Focus on the predictor domain has resulted in clear well articulated specification of that, but a fuzzy ill-defined specification of the criteria.
Criterion-focus re-defines predictors as measures of competency potential
Our validation methodology needs to distinguish:
Competency potential: Measures of dispositional attributes, personality and emotional intelligence, critical faculties, managerial judgement
Competencies: Behaviours, personal performance, and colleagues’ perceptions
Outcomes: Achievements, impacts on organisation’s business and financial performance, track record etc
Competencies
Competencies have become a powerful tool for assessing the performance of people because
they are firmly linked to observable behaviour.
they give managers and their people a common language for discussing development and career potential issues,
they provide a way to express the goals, culture and values of the Organization in terms of the behaviour expected of employees.
Kurz and Bartram (2002) define competencies as:
sets of behaviours that are instrumental in the delivery of desired results or outcomes.
What is a ‘competency framework’
Lots of people talk about competency frameworks, when what they mean are just collections of competencies.
A framework is an articulated set of relationships
It defines the components of a model.
How those components relate to each other
How they relate to other constructs (performance, personality etc).
It is evidence-based.
Deep and surface structures
The framework specifies the generic ‘deep structure’ of the competency domain
Specific competency models are ‘surface structures’
Measurements of competencies are ‘surface measures’.
‘Construct validity’ of competency measures can vary as a function of:
Measurement method – ratings, exercises etc
Assessor perspective – self, boss, peer, etc
A new validation model
The competency framework provides a common basis for describing both predictor measures and criterion measures
Great 8 predictor scores can be generated from combinations of ability, personality and motivation measures
Great 8 criterion scores can be generated by mapping competency ratings to the competency framework
Validity can then be examined using a one-to-one mapping of Great 8 predictor-criterion pairings.
Computer Based Assessment in Careers Guidance
Study Objectives & Hypothesis
Sample & Methodology
Results & Discussion
Conclusions
Study Objectives
Research psychometric properties of the AIMS
modules
Investigate Construct Validity by integrating AIMS
under the headings of the Holland Theory of
Vocational Type
Gather candidate’s views on instruments and process
Hypothesis: test results likely to be robust; AIMS
components are likely to correlate in a meaningful way
to the Holland types; candidates are likely to feel
positive about materials used, process and outcome
Sample
N = 152 UK students and adults in higher education
75% no test experience
Mean age 25.34 with SD of 10.39
37.5% male, 62.5% female
Methodology
Mouse Practice
6 Ability Tests (Verbal, Numerical, Diagramming,
Checking, Spatial and Mechanical)
Advanced Occupational Interest Inventory
Motivation Questionnaire
Occupational Personality Questionnaire
Evaluation Forms
Conclusions
Strong support for using AIMS in Careers Guidance
Successful AIMS integration under Holland Types
Blueprint for “The automated Assessment /
Development Centre”
Future Research:
- Factor analyse General and / or Management Interest
Inventory to generate Holland Types
- Computerised Versions of GAP
- More universal Profile of Occupational Preferences
- Integrated Ability, Interest, Motivation and Style
EXPERT Systems
Presentation Outline
Background
Project Objectives & Hypothesis
Methodology & Samples
Results & Discussion
Conclusions & Outlook
Background:
Competencies
Definition of Competencies: Multidimensional
Constructs that drive successful behavior patterns
Competency Constructs (Campbell 1993, Kurz 1999):
Multiply determined by antecedents (Ability, Interest,
Motivation, Style) and determinants (Knowledge, Skill,
Experience, Qualification)
Competency Requirements: Behavior Requirements
multiply determined by Job Objectives, Tasks, Context
and Culture
Competency Behaviors: Observed Behavior Patterns
that reflect successful Performance at Work
Background:
Managerial Competencies
Boyatzis (1982)
Schroder (1989)
Sparrow (1993)
Dulcewics (1992)
Inventory of Management Competencies (SHL,1992)
Perspectives in Management Competencies
(SHL,1994)
Background:
Validity of Ability & Personality Variables
Validity Reviews (e.g. Ghiselli, 1973)
Validity Generalisation (e.g. Hunter & Schmidt, 1986)
Meta Analysis of Validity Studies (e.g. Schmitt et
al.,1984; Barrick & Mount, 1989)
Meta Analysis of 20 SHL Validity Studies (Robertson &
Kinder, 1993)
International Validation Studies (Gibbons et al., 1995)
Background:
Computer Based Test Interpretation
(CBTI)
since 1960’ies especially for MMPI and other clinical
instruments
since 1987 Occupational Personality Questionnaire
(OPQ) Expert System Management Skills Potential
1990 Work Profiling System: Human Attribute Based
Person-Job Match
1995 OPQ Competency Based Narrative
1995 Competency Based Person-Job Match
1999 DecisionMaker
1999 OPQ32 vs. IMC Potential
Methodology
Standardised Validation Studies as pioneered in 4
AMT - IMC Studies (SHL, 1992)
Concurrent Design
Predictors:
- Occupational Personality Questionnaire (CM4.2)
- Verbal Critical Reasoning (VMG)
- Numerical Critical Reasoning (NMG)
Criteria:
- Inventory of Management Competencies (16 scales)
- Normative and Ipsative Sub-scales
- ‘Boss’ & ‘Self’ Ratings
Validation Studies -
Sample Sizes, Country, Industry Sector
A: N=131, UK, Distribution
B: N=34, UK, Banking
C: N=90, UK, Financial Services
D: N=114, UK, Paper Manufacturing
E: N=68, UK, Banking
F: N=103, USA, Rubber Manufacturing
G:N=503, Turkey, Glass Manufacturing
Designing Prediction Equations
Statistical Analysis: First order and multiple
correlations between predictors and criteria (within
and across studies)
Expert Judgement: A-priori hypothesised relations
Social Judgement: Logical Concordance (Warr, 1997):
Content Validity: Justifiable from Job Analysis
Measurement: Exclusion of Artefacts (Ipsativity, Halo)
Inclusion of Scales meeting all Criteria with Unit
Weights of 1, 2 ,3 or 4
Results:
Validity
Validation of Weighted Composite Prediction Scores
Predictions are overall stable & valid across jobs,
organisations and cultures for most competencies
Good validities for Ability (Median around .25; up to
.47)
High Validities for Personality variables vs. ‘Boss’
(Median around .3; up to .52)
High Validities for Personality variables vs. ‘Self’’
(Median around .5; up to .76)
Conclusions & Outlook
Highly successful Research & Development Project
Impressive Validity Evidence
Competency Based P-J Match now in use world-wide
Competency Based OPQ Narrative displaying &
verbalising Prediction Scores under Preparation
Future Research & Development:
- Integration of other predictors
- Integration under different categories
- User friendly report design
- Documentation
'Applications of the Periodic Table of Personality: Mapping Hogan Insight Series (HPI Bright Side, HDS Dark Side and MVPI Inside) vs. HUCAMA Personality and Competency Factors'