Conference Presentations by Maximilian Jansen
International Congress of Psychology (ICP) in Prague, 2024
This symposium brings together psychometric testing experts to share pioneering insights into adv... more This symposium brings together psychometric testing experts to share pioneering insights into advances in Computer Based Assessment (CBA).
The first paper reviews milestones in the development and evaluation of CBA systems based on Bartram & Bayliss (1984). Validation principles are illustrated through the correlation of personality, ability, interest, and motivation scales with Great 8 competency factors.
The second paper outlines contemporary advances in personality, ability and competency assessment and reporting through Computer-Based Test Interpretation (CBTI) in the light of Bartram (1994) which shaped BPS and EFPA test review processes. Versatile use of assessment data for multi-level reporting across tools are illustrated and backed with validation data.
The third paper illustrates applications of Sociomapping (Bahbouh, 2012) to psychometric assessment reporting at group and construct set level. Topographical representation of relationships between Big 5 and Great 8 constructs align to Stability and Plasticity in Cybernetic Big 5 Theory (DeYoung, 2015). Case studies on a 'Talent' group and a vet team are presented.
The fourth paper outlines a leading-edge approach involving continuous assessment of wellbeing which presents a measurement model for Employee Experience (EX) based on continuous adaptive micro interactions. The system captures trends in EX data in teams and organisations over time, while accounting for the dynamic relationship between contextual factors in the work environment and internal mindset factors that together influence an individual's EX.
The fifth paper outlines how test review processes have been impacted by advances in CBA. Issues include the use of item banks and CBTI reports products where reviewers will depend on technical documentation featuring psychometric data, especially validation results, to assess the appropriateness of the interpretations.
As artificial intelligence becomes more prevalent, it is timely to discuss past, present and future of CBA. The discussant slot will review the contributions and draw out themes, challenges, and opportunities.
Discussant: Dragos Iliescu, University of Bucharest
Intelligent Testing Systems: Past, Present and Future
Rainer Hermann Kurz, PhD
HUCAMA Analytics, London
Bartram & Bayliss (1984) wrote about the future use of computers in assessment. This paper reviews issues surrounding seven components of a fully automated Intelligent Testing System (ITS) and demonstrates how empirical data can be used to build valid system. 1. Test choice requires a sophisticated understanding of the assessment purpose, tools available and the importance of the scales. 2. Administration requires robustness against Intellectual Property content theft and faking. 3. Scoring requires transparency, sensitivity, and validity. 4.Interpretation needs to be based on valid information presented in understandable form 5. Feedback must be framed appropriate for the recipient whether candidates or 4rd parties. 6. Decision-making must be embedded within legal frameworks and proportionate to the quality of the evidence gathered. 7. Monitoring of outcomes is crucial to establish and enhance validity.
With the advent of AI there is a risk that poor assessment practices will proliferate especially if poor off-line processes are perpetuated and magnified through AI. On the other hand, AI that builds on and integrates sound testing practices could add value.
A data set where 250 individuals completed ability, personality, motivation, and competency assessments is utilised to illustrate data-driven development approaches. Predictors for the Great 8 Competencies (Kurz & Bartram, 2002) are calculated based on the meta-analysis of Bartram (2005) to illustrate how the scales of these five assessment modalities inter-relate. The two highest correlations for the factors are identified for each scale to facility the development of expert systems based on Great 8 and Big 5 models – whether using conventional or AI approaches.
Computer-Based Assessment across Personality, Ability and Competency Factors
Michele Guarini
HUCAMA Group, Copenhagen
This paper builds on the pioneering work of Bartram (1994) on Computer-Based Test Interpretation (CBTI) with a lens on group reporting based on the Great 8 Success Factors inspired by Kurz & Bartram (2002).
Personality Factors:
The modular range features general, professional, and executive level versions with 80, 160 and 240 questions measuring 16, 32 and 48 facets respectively grouped into 8 factors.
A. The Role Wheel Report uses ipsatised data (Bartram, 1996) to remove the effect of individual response style for enhanced group reporting.
B. The Leadership Report maps 48 personality facets to 8 Primary Colours of Leadership constructs (Pendelton, Furnham & Cowell, 2021) with an observed validity with external reviewer ratings of .50 (N=113).
C. The Aspects Reports covers emotional and operational themes that underpin potential and performance fully integrating Emotional Intelligence and Learning Agility constructs. The median construct convergence for the nearest counterpart scale was .63 for the 15 EQi 2.0 facets (N=101) and .70 for its five higher-order compounds.
Ability Factors:
This assessment consists of diagrammatic, numerical, spatial, and verbal component tests with a time limit of 8 minutes each. Rule-based item generation builds on Kurz (1990) and reporting on Kurz (2000) featuring Supra-scores across areas and sub-scores for speed, accuracy and caution.
Competency Factors:
This inventory features an overarching GETTING IT RIGHT, GETTING ALONG, GETTING AHEAD and GETTING ALIGNED model that builds on the Schwartz (1993) values circumplex and the Hogan & Holland (2004) view on performance. Each quadrant pairs up two Great 8 factors. Extreme tie-breaker data is collected and item level results shown.
A Potential & Performance Solution gap analysis tool brings together reporting across personality, ability and competency assessments for individuals and groups using Sten scores on a dynamic dashboard that enables multi-level integration, interpretation, and interrogation of data.
Sociomapping and Team Profile Analyzer in Psychometric Assessment
Pauline Willis
Lauriate, Australia
Sociomapping (Bahbouh, 2012) is an innovative method for tracking quality and frequency of communication in organisation. This paper outlines two applications of the underlying methodology to psychometric assessment.
STORM software uses scale correlations to produce a topographical ‘heat map’ that indicates the centrality of scales and maps out the relationship between constructs. Correlations (N=308) of Big 5 scales (based on NEO) with Great 8 (Kurz & Bartram, 2002) constructs revealed the centrality of Emotional Stability together with motivational (Need for Achievement & Power) constructs to the variable set. Stability (Alpha) vs Plasticity (Beta) meta-factors delineated one axis whereas People vs. Task delineated the other. The graph illustrates how constructs interrelate and facilitate understanding of the nature of ‘derailment’ scales. A cross-validation on N=466 largely confirm the results using different questionnaires.
Team Profile Analyzer (TPA) software produces a map that indicates the centrality of individuals to groups and similarities between group members. In the ‘Sociomap of Profile Similarity’ each group member is represented by a point and mutual distances represent mutual similarity of individual profiles. A heat map colour scheme indicates centrality. Personality assessment results for 16 ‘Elite’ performers across business, arts and sports were analysed using TPA. 10 group members shared many characteristics whereas 6 were different at the highest level of analysis with lower-level scores illuminating the origin of higher-order trends. The analysis explored differences between business leaders and others as well as sex differences. TPA analysis on Personality Factors results for a veterinary team will also be presented.
The applications demonstrate the power of group level reporting across psychometric results. The question arises how advances in academic theory building, such as the Periodic Table of Personality (Woods & Anderson, 2016) and Cybernetic Big 5 Theory (DeYoung, 2015), can be built upon and integrated with the Sociomapping methodology.
Insights in Motion: A Comprehensive Model for Tracking Employee Experience Over Time
Richard T. Justenhoven, PhD
Welliba, Germany
The varying stability across time and situations different constructs exhibit is well known and subject of ongoing research (Steyer et al., 2015). Talent assessment and management tools continue to evolve, and technological advancements enable capturing data in ever increasing breadth and depth.
This enables increasingly nuanced approaches to measuring constructs over time. This paper presents a measurement model for Employee Experience (EX) based on continuous adaptive micro interactions (CadaMint) as one example of this (Preuss et al., 2023). Grounded in Self-Determination-Theory (Deci & Ryan, 2000; Yeager & Dweck, 2020) and the Job-Demands-Resources Model (Bakker & Demerouti, 2007) CadaMint captures trends in EX data in teams and organisations over time, while accounting for the dynamic relationship between contextual factors in the work environment and internal mindset factors that together influence an individual's EX.
As data on EX often covers quarterly or annual cycles, CadaMint accounts for decreasing reliability of data on individual level as measurements age and increases when new measurements for the same constructs are added. This is achieved through a set of characteristics assigned to each instrument and variable, determining how measurements age. Interactions of different factors and fluctuations over time not only pose challenges to measurement, but also to the way outputs are presented to HR profes...
Uploads
Conference Presentations by Maximilian Jansen
The first paper reviews milestones in the development and evaluation of CBA systems based on Bartram & Bayliss (1984). Validation principles are illustrated through the correlation of personality, ability, interest, and motivation scales with Great 8 competency factors.
The second paper outlines contemporary advances in personality, ability and competency assessment and reporting through Computer-Based Test Interpretation (CBTI) in the light of Bartram (1994) which shaped BPS and EFPA test review processes. Versatile use of assessment data for multi-level reporting across tools are illustrated and backed with validation data.
The third paper illustrates applications of Sociomapping (Bahbouh, 2012) to psychometric assessment reporting at group and construct set level. Topographical representation of relationships between Big 5 and Great 8 constructs align to Stability and Plasticity in Cybernetic Big 5 Theory (DeYoung, 2015). Case studies on a 'Talent' group and a vet team are presented.
The fourth paper outlines a leading-edge approach involving continuous assessment of wellbeing which presents a measurement model for Employee Experience (EX) based on continuous adaptive micro interactions. The system captures trends in EX data in teams and organisations over time, while accounting for the dynamic relationship between contextual factors in the work environment and internal mindset factors that together influence an individual's EX.
The fifth paper outlines how test review processes have been impacted by advances in CBA. Issues include the use of item banks and CBTI reports products where reviewers will depend on technical documentation featuring psychometric data, especially validation results, to assess the appropriateness of the interpretations.
As artificial intelligence becomes more prevalent, it is timely to discuss past, present and future of CBA. The discussant slot will review the contributions and draw out themes, challenges, and opportunities.
Discussant: Dragos Iliescu, University of Bucharest
Intelligent Testing Systems: Past, Present and Future
Rainer Hermann Kurz, PhD
HUCAMA Analytics, London
Bartram & Bayliss (1984) wrote about the future use of computers in assessment. This paper reviews issues surrounding seven components of a fully automated Intelligent Testing System (ITS) and demonstrates how empirical data can be used to build valid system. 1. Test choice requires a sophisticated understanding of the assessment purpose, tools available and the importance of the scales. 2. Administration requires robustness against Intellectual Property content theft and faking. 3. Scoring requires transparency, sensitivity, and validity. 4.Interpretation needs to be based on valid information presented in understandable form 5. Feedback must be framed appropriate for the recipient whether candidates or 4rd parties. 6. Decision-making must be embedded within legal frameworks and proportionate to the quality of the evidence gathered. 7. Monitoring of outcomes is crucial to establish and enhance validity.
With the advent of AI there is a risk that poor assessment practices will proliferate especially if poor off-line processes are perpetuated and magnified through AI. On the other hand, AI that builds on and integrates sound testing practices could add value.
A data set where 250 individuals completed ability, personality, motivation, and competency assessments is utilised to illustrate data-driven development approaches. Predictors for the Great 8 Competencies (Kurz & Bartram, 2002) are calculated based on the meta-analysis of Bartram (2005) to illustrate how the scales of these five assessment modalities inter-relate. The two highest correlations for the factors are identified for each scale to facility the development of expert systems based on Great 8 and Big 5 models – whether using conventional or AI approaches.
Computer-Based Assessment across Personality, Ability and Competency Factors
Michele Guarini
HUCAMA Group, Copenhagen
This paper builds on the pioneering work of Bartram (1994) on Computer-Based Test Interpretation (CBTI) with a lens on group reporting based on the Great 8 Success Factors inspired by Kurz & Bartram (2002).
Personality Factors:
The modular range features general, professional, and executive level versions with 80, 160 and 240 questions measuring 16, 32 and 48 facets respectively grouped into 8 factors.
A. The Role Wheel Report uses ipsatised data (Bartram, 1996) to remove the effect of individual response style for enhanced group reporting.
B. The Leadership Report maps 48 personality facets to 8 Primary Colours of Leadership constructs (Pendelton, Furnham & Cowell, 2021) with an observed validity with external reviewer ratings of .50 (N=113).
C. The Aspects Reports covers emotional and operational themes that underpin potential and performance fully integrating Emotional Intelligence and Learning Agility constructs. The median construct convergence for the nearest counterpart scale was .63 for the 15 EQi 2.0 facets (N=101) and .70 for its five higher-order compounds.
Ability Factors:
This assessment consists of diagrammatic, numerical, spatial, and verbal component tests with a time limit of 8 minutes each. Rule-based item generation builds on Kurz (1990) and reporting on Kurz (2000) featuring Supra-scores across areas and sub-scores for speed, accuracy and caution.
Competency Factors:
This inventory features an overarching GETTING IT RIGHT, GETTING ALONG, GETTING AHEAD and GETTING ALIGNED model that builds on the Schwartz (1993) values circumplex and the Hogan & Holland (2004) view on performance. Each quadrant pairs up two Great 8 factors. Extreme tie-breaker data is collected and item level results shown.
A Potential & Performance Solution gap analysis tool brings together reporting across personality, ability and competency assessments for individuals and groups using Sten scores on a dynamic dashboard that enables multi-level integration, interpretation, and interrogation of data.
Sociomapping and Team Profile Analyzer in Psychometric Assessment
Pauline Willis
Lauriate, Australia
Sociomapping (Bahbouh, 2012) is an innovative method for tracking quality and frequency of communication in organisation. This paper outlines two applications of the underlying methodology to psychometric assessment.
STORM software uses scale correlations to produce a topographical ‘heat map’ that indicates the centrality of scales and maps out the relationship between constructs. Correlations (N=308) of Big 5 scales (based on NEO) with Great 8 (Kurz & Bartram, 2002) constructs revealed the centrality of Emotional Stability together with motivational (Need for Achievement & Power) constructs to the variable set. Stability (Alpha) vs Plasticity (Beta) meta-factors delineated one axis whereas People vs. Task delineated the other. The graph illustrates how constructs interrelate and facilitate understanding of the nature of ‘derailment’ scales. A cross-validation on N=466 largely confirm the results using different questionnaires.
Team Profile Analyzer (TPA) software produces a map that indicates the centrality of individuals to groups and similarities between group members. In the ‘Sociomap of Profile Similarity’ each group member is represented by a point and mutual distances represent mutual similarity of individual profiles. A heat map colour scheme indicates centrality. Personality assessment results for 16 ‘Elite’ performers across business, arts and sports were analysed using TPA. 10 group members shared many characteristics whereas 6 were different at the highest level of analysis with lower-level scores illuminating the origin of higher-order trends. The analysis explored differences between business leaders and others as well as sex differences. TPA analysis on Personality Factors results for a veterinary team will also be presented.
The applications demonstrate the power of group level reporting across psychometric results. The question arises how advances in academic theory building, such as the Periodic Table of Personality (Woods & Anderson, 2016) and Cybernetic Big 5 Theory (DeYoung, 2015), can be built upon and integrated with the Sociomapping methodology.
Insights in Motion: A Comprehensive Model for Tracking Employee Experience Over Time
Richard T. Justenhoven, PhD
Welliba, Germany
The varying stability across time and situations different constructs exhibit is well known and subject of ongoing research (Steyer et al., 2015). Talent assessment and management tools continue to evolve, and technological advancements enable capturing data in ever increasing breadth and depth.
This enables increasingly nuanced approaches to measuring constructs over time. This paper presents a measurement model for Employee Experience (EX) based on continuous adaptive micro interactions (CadaMint) as one example of this (Preuss et al., 2023). Grounded in Self-Determination-Theory (Deci & Ryan, 2000; Yeager & Dweck, 2020) and the Job-Demands-Resources Model (Bakker & Demerouti, 2007) CadaMint captures trends in EX data in teams and organisations over time, while accounting for the dynamic relationship between contextual factors in the work environment and internal mindset factors that together influence an individual's EX.
As data on EX often covers quarterly or annual cycles, CadaMint accounts for decreasing reliability of data on individual level as measurements age and increases when new measurements for the same constructs are added. This is achieved through a set of characteristics assigned to each instrument and variable, determining how measurements age. Interactions of different factors and fluctuations over time not only pose challenges to measurement, but also to the way outputs are presented to HR profes...
The first paper reviews milestones in the development and evaluation of CBA systems based on Bartram & Bayliss (1984). Validation principles are illustrated through the correlation of personality, ability, interest, and motivation scales with Great 8 competency factors.
The second paper outlines contemporary advances in personality, ability and competency assessment and reporting through Computer-Based Test Interpretation (CBTI) in the light of Bartram (1994) which shaped BPS and EFPA test review processes. Versatile use of assessment data for multi-level reporting across tools are illustrated and backed with validation data.
The third paper illustrates applications of Sociomapping (Bahbouh, 2012) to psychometric assessment reporting at group and construct set level. Topographical representation of relationships between Big 5 and Great 8 constructs align to Stability and Plasticity in Cybernetic Big 5 Theory (DeYoung, 2015). Case studies on a 'Talent' group and a vet team are presented.
The fourth paper outlines a leading-edge approach involving continuous assessment of wellbeing which presents a measurement model for Employee Experience (EX) based on continuous adaptive micro interactions. The system captures trends in EX data in teams and organisations over time, while accounting for the dynamic relationship between contextual factors in the work environment and internal mindset factors that together influence an individual's EX.
The fifth paper outlines how test review processes have been impacted by advances in CBA. Issues include the use of item banks and CBTI reports products where reviewers will depend on technical documentation featuring psychometric data, especially validation results, to assess the appropriateness of the interpretations.
As artificial intelligence becomes more prevalent, it is timely to discuss past, present and future of CBA. The discussant slot will review the contributions and draw out themes, challenges, and opportunities.
Discussant: Dragos Iliescu, University of Bucharest
Intelligent Testing Systems: Past, Present and Future
Rainer Hermann Kurz, PhD
HUCAMA Analytics, London
Bartram & Bayliss (1984) wrote about the future use of computers in assessment. This paper reviews issues surrounding seven components of a fully automated Intelligent Testing System (ITS) and demonstrates how empirical data can be used to build valid system. 1. Test choice requires a sophisticated understanding of the assessment purpose, tools available and the importance of the scales. 2. Administration requires robustness against Intellectual Property content theft and faking. 3. Scoring requires transparency, sensitivity, and validity. 4.Interpretation needs to be based on valid information presented in understandable form 5. Feedback must be framed appropriate for the recipient whether candidates or 4rd parties. 6. Decision-making must be embedded within legal frameworks and proportionate to the quality of the evidence gathered. 7. Monitoring of outcomes is crucial to establish and enhance validity.
With the advent of AI there is a risk that poor assessment practices will proliferate especially if poor off-line processes are perpetuated and magnified through AI. On the other hand, AI that builds on and integrates sound testing practices could add value.
A data set where 250 individuals completed ability, personality, motivation, and competency assessments is utilised to illustrate data-driven development approaches. Predictors for the Great 8 Competencies (Kurz & Bartram, 2002) are calculated based on the meta-analysis of Bartram (2005) to illustrate how the scales of these five assessment modalities inter-relate. The two highest correlations for the factors are identified for each scale to facility the development of expert systems based on Great 8 and Big 5 models – whether using conventional or AI approaches.
Computer-Based Assessment across Personality, Ability and Competency Factors
Michele Guarini
HUCAMA Group, Copenhagen
This paper builds on the pioneering work of Bartram (1994) on Computer-Based Test Interpretation (CBTI) with a lens on group reporting based on the Great 8 Success Factors inspired by Kurz & Bartram (2002).
Personality Factors:
The modular range features general, professional, and executive level versions with 80, 160 and 240 questions measuring 16, 32 and 48 facets respectively grouped into 8 factors.
A. The Role Wheel Report uses ipsatised data (Bartram, 1996) to remove the effect of individual response style for enhanced group reporting.
B. The Leadership Report maps 48 personality facets to 8 Primary Colours of Leadership constructs (Pendelton, Furnham & Cowell, 2021) with an observed validity with external reviewer ratings of .50 (N=113).
C. The Aspects Reports covers emotional and operational themes that underpin potential and performance fully integrating Emotional Intelligence and Learning Agility constructs. The median construct convergence for the nearest counterpart scale was .63 for the 15 EQi 2.0 facets (N=101) and .70 for its five higher-order compounds.
Ability Factors:
This assessment consists of diagrammatic, numerical, spatial, and verbal component tests with a time limit of 8 minutes each. Rule-based item generation builds on Kurz (1990) and reporting on Kurz (2000) featuring Supra-scores across areas and sub-scores for speed, accuracy and caution.
Competency Factors:
This inventory features an overarching GETTING IT RIGHT, GETTING ALONG, GETTING AHEAD and GETTING ALIGNED model that builds on the Schwartz (1993) values circumplex and the Hogan & Holland (2004) view on performance. Each quadrant pairs up two Great 8 factors. Extreme tie-breaker data is collected and item level results shown.
A Potential & Performance Solution gap analysis tool brings together reporting across personality, ability and competency assessments for individuals and groups using Sten scores on a dynamic dashboard that enables multi-level integration, interpretation, and interrogation of data.
Sociomapping and Team Profile Analyzer in Psychometric Assessment
Pauline Willis
Lauriate, Australia
Sociomapping (Bahbouh, 2012) is an innovative method for tracking quality and frequency of communication in organisation. This paper outlines two applications of the underlying methodology to psychometric assessment.
STORM software uses scale correlations to produce a topographical ‘heat map’ that indicates the centrality of scales and maps out the relationship between constructs. Correlations (N=308) of Big 5 scales (based on NEO) with Great 8 (Kurz & Bartram, 2002) constructs revealed the centrality of Emotional Stability together with motivational (Need for Achievement & Power) constructs to the variable set. Stability (Alpha) vs Plasticity (Beta) meta-factors delineated one axis whereas People vs. Task delineated the other. The graph illustrates how constructs interrelate and facilitate understanding of the nature of ‘derailment’ scales. A cross-validation on N=466 largely confirm the results using different questionnaires.
Team Profile Analyzer (TPA) software produces a map that indicates the centrality of individuals to groups and similarities between group members. In the ‘Sociomap of Profile Similarity’ each group member is represented by a point and mutual distances represent mutual similarity of individual profiles. A heat map colour scheme indicates centrality. Personality assessment results for 16 ‘Elite’ performers across business, arts and sports were analysed using TPA. 10 group members shared many characteristics whereas 6 were different at the highest level of analysis with lower-level scores illuminating the origin of higher-order trends. The analysis explored differences between business leaders and others as well as sex differences. TPA analysis on Personality Factors results for a veterinary team will also be presented.
The applications demonstrate the power of group level reporting across psychometric results. The question arises how advances in academic theory building, such as the Periodic Table of Personality (Woods & Anderson, 2016) and Cybernetic Big 5 Theory (DeYoung, 2015), can be built upon and integrated with the Sociomapping methodology.
Insights in Motion: A Comprehensive Model for Tracking Employee Experience Over Time
Richard T. Justenhoven, PhD
Welliba, Germany
The varying stability across time and situations different constructs exhibit is well known and subject of ongoing research (Steyer et al., 2015). Talent assessment and management tools continue to evolve, and technological advancements enable capturing data in ever increasing breadth and depth.
This enables increasingly nuanced approaches to measuring constructs over time. This paper presents a measurement model for Employee Experience (EX) based on continuous adaptive micro interactions (CadaMint) as one example of this (Preuss et al., 2023). Grounded in Self-Determination-Theory (Deci & Ryan, 2000; Yeager & Dweck, 2020) and the Job-Demands-Resources Model (Bakker & Demerouti, 2007) CadaMint captures trends in EX data in teams and organisations over time, while accounting for the dynamic relationship between contextual factors in the work environment and internal mindset factors that together influence an individual's EX.
As data on EX often covers quarterly or annual cycles, CadaMint accounts for decreasing reliability of data on individual level as measurements age and increases when new measurements for the same constructs are added. This is achieved through a set of characteristics assigned to each instrument and variable, determining how measurements age. Interactions of different factors and fluctuations over time not only pose challenges to measurement, but also to the way outputs are presented to HR profes...