Measurement and Evaluation (Research Methodology)

This empirical study explores the evolving landscape of assessment within the Nigerian university system, aiming at the understanding of current practices, challenges, and prospective innovations. Employing a mixed-methods approach, the research combines quantitative data from surveys of 500 students and 200 faculty members across five major universities with qualitative insights from indepth interviews with 30 educational stakeholders, including administrators and policy-makers. The findings reveal a predominant reliance on traditional examination methods, often criticized for fostering rote learning rather than critical thinking and problem-solving skills. Key challenges identified include large class sizes, inadequate infrastructure, and inadequate training for Lecturers to update their skills in alternative assessment techniques. Despite these obstacles, there is a growing recognition of the need for more holistic and formative assessment strategies, such as continuous assessment, project-based learning, and digital assessments, which better align with global educational standards and the demands of the 21st-century workforce. The study highlights successful pilot programs and best practices from select institutions that have integrated technology-driven assessments and competency-based evaluations. Recommendations for policy reforms and capacity-building initiatives are provided to facilitate a gradual transition towards more innovative and effective assessment methodologies. This research underscores the imperative for systemic change in the Nigerian universities' education assessment framework to enhance educational quality and student outcomes in an increasingly dynamic and competitive global context.

Customers select suppliers based on the relative importance of different attributes such as quality, price, flexibility, and delivery performance. This study examines the difference between managers' rating of the perceived importance of different supplier attributes and their actual choice of suppliers in an experimental setting. We use two methods: a Likert scale set of questions, to determine the importance of supplier attributes; and a discrete choice analysis (DCA) experiment, to examine the choice of suppliers. The results indicate that although managers say that quality is the most important attribute for a supplier, they actually choose suppliers based largely on cost and delivery performance. Research Question I: What are the perceived importance of quality, cost, delivery, and flexibility attributes in the supplier selection process? Research Question II: How do managers tradeoff among quality, cost, delivery and flexibility attributes when actually choosing a supplier? We used two different data collection and analyses procedures to explore the above research questions. A survey instrument containing Likert-type scale questions was used to explore Research Question I. We used an econometric procedure known as discrete choice analysis (DCA) for quantifying the relative weights of attributes when actual supplier choice is made (Research Question II). The following sections describe the research design, data collection and analyses procedures, and discuss the empirical results. Research Method An empirical study was designed to evaluate the supplier selection process. The sample group consisted of manufacturing companies, specifically metal processing and producers of small machine tools and tooling in the western United States. The population included 323 companies with SIC codes of 33, 34, and 35 registered with two western states (Utah and Arizona) in 1994. The actual data from the managers were collected during the first half of 1995. Operations managers or the managers with purchasing/supplier selecting responsibility were contacted by phone and were requested to participate in the survey. It was also ensured that the company manufactured appropriate product(s) (with SIC code 33, 34 or 35), and confirmed that the company employed at least 15 people. From the original population, 139 managers agreed to participate in the study. Survey instruments along with the cover letter, a postage-paid business reply envelope, and a forwarding letter from the management department of a large state university in the western United States were mailed to the managers.

Customers select suppliers based on the relative importance of different attributes such as quality, price, flexibility, and delivery performance. This study examines the difference between managers' rating of the perceived importance of different supplier attributes and their actual choice of suppliers in an experimental setting. We use two methods: a Likert scale set of questions, to determine the importance of supplier attributes; and a discrete choice analysis (DCA) experiment, to examine the choice of suppliers. The results indicate that although managers say that quality is the most important attribute for a supplier, they actually choose suppliers based largely on cost and delivery performance. Research Question I: What are the perceived importance of quality, cost, delivery, and flexibility attributes in the supplier selection process? Research Question II: How do managers tradeoff among quality, cost, delivery and flexibility attributes when actually choosing a supplier? We used two different data collection and analyses procedures to explore the above research questions. A survey instrument containing Likert-type scale questions was used to explore Research Question I. We used an econometric procedure known as discrete choice analysis (DCA) for quantifying the relative weights of attributes when actual supplier choice is made (Research Question II). The following sections describe the research design, data collection and analyses procedures, and discuss the empirical results. Research Method An empirical study was designed to evaluate the supplier selection process. The sample group consisted of manufacturing companies, specifically metal processing and producers of small machine tools and tooling in the western United States. The population included 323 companies with SIC codes of 33, 34, and 35 registered with two western states (Utah and Arizona) in 1994. The actual data from the managers were collected during the first half of 1995. Operations managers or the managers with purchasing/supplier selecting responsibility were contacted by phone and were requested to participate in the survey. It was also ensured that the company manufactured appropriate product(s) (with SIC code 33, 34 or 35), and confirmed that the company employed at least 15 people. From the original population, 139 managers agreed to participate in the study. Survey instruments along with the cover letter, a postage-paid business reply envelope, and a forwarding letter from the management department of a large state university in the western United States were mailed to the managers.

This research examines whether environmentally based intervention strategies increase elementary students' selection of white milk. At intervention school one, white milk was easily accessible, but students had to ask for chocolate milk. As an outcome, students increased their selection of white milk compared to control school students (p≤0.001). At intervention school two, the visual cue of a threefold greater quantity of white compared to chocolate milk did not significantly alter selection patterns. This research indicates that requiring students to ask for an item rather than self-serve can help modify food selections and serve as a tool for obesity prevention.

Aim: Digital intake estimation has been shown to be strongly correlated with real-time visual estimation of dietary intake in cafeterias and long term care settings. The objectives of the present study were to estimate intra-and interrater agreement for digitally captured plate waste in hospitalised adults. Methods: Lunch plates (n = 106) of hospitalised adults were photographed immediately prior to tray collection. Two raters used the modified Comstock method to assess food waste from each of the digital images. Additionally, a randomly selected subset of photographs was re-estimated 72 hours following the first estimation. Inter-and intrarater reliability were estimated by calculating the intraclass correlation coefficients (ICC). Results: In the total population, the ICC for interrater reliability was 0.95 (95% CI 0.93-0.97, P < 0.0001) for the entire meal, 0.94 (95% CI 0.9-0.96, P < 0.0001) for vegetables, 0.96 (95% CI 0.9-0.96, P < 0.0001) for starch and 0.96 (95% CI 0.94-0.97, P < 0.0001) for the main course. ICC for intrarater reliability was 0.99 (95% CI 0.97-1.0, P < 0.0001) for the entire meal, 0.95 (95% CI 0.86-0.98, P < 0.0001) for vegetables, 0.97 (95% CI 0.92-0.99, P < 0.0001) for starch and 0.99 (0.98-1.0, P < 0.0001) for the main course. Findings were similar for regular diets and somewhat weaker for soft diets. Conclusions: Dietary intake estimated from digital images has a high degree of both inter-and intrarater reliability in hospitalised patient populations.

Purpose. The purpose of this study is to establish the psychometric properties of 22 measures from a community-based participatory research (CBPR) conceptual model. Design. The design of this study was an online, cross-sectional survey of academic and community partners involved in a CPBR project. Setting. CPBR projects (294) in the United States with federal funding in 2009. Subjects. Of the 404 academic and community partners invited, 312 (77.2%) participated. Of the 200 principal investigators/project directors invited, 138 (69.0%) participated. Measures. Twenty-two measures of CBPR context, group dynamics, methods, and health-related outcomes were examined. Analysis. Confirmatory factor analysis to establish factorial validity and Pearson correlations to establish convergent and divergent validity were used. Results. Confirmatory factor analysis demonstrated strong factorial validity for the 22 constructs. Pearson correlations (p &lt; .001) supported the convergent and divergent...

The study investigated items that were prone to guessing in Senior School Certificate Examinations (SSCE) Economics multiple-choice tests among students in Kwara State, Nigeria. The 2016 West African Senior Secondary Certificate Examinations (WASSCE) and National Examinations Council (NECO) Economics multiple-choice test items were subjected to guessing parameters. This study adopted a descriptive survey research design, carried out in 25 randomly selected public Senior Secondary Schools in Kwara State. A multi-stage sampling technique was used to sample 768 Senior School III students. The instruments used for data collection were 2016 WASSCE and NECO multiple-choice Economics items. Guessing parameter was computed using WinGen IRT software package used to compute and answer research question 2 (What are those 2016 WASSCE and NECO Economics multiple-choice test items prone to guessing?). The hypothesis was tested using the chi-square statistic at 0.05 level of significance. One of the findings revealed that items prone to guessing in WASSCE and NECO were of the same rate. The finding also revealed that no significant difference existed in items guessed in WASSCE and NECO. One of the recommendations revealed that the classroom teachers and professional developers should avoid clauses such as "except", all of the above", "none of the above" and "not" to mention a few. This could reduce the guessing tendency in multiple-choice items.

Service organizations are increasingly utilizing advanced information and communication technologies, such as the Internet, in hopes of improving the efficiency, cost-effectiveness, and/or quality of their customer-facing operations. More of the contact a customer has with the firm is likely to be with the back-office and, therefore, mediated by technology. While previous operations management research has been important for its contributions to our understanding of customer contact in face-to-face settings, considerably less work has been done to improve our understanding of customer contact in what we refer to as technology-mediated settings (e.g., via telephone, instant messaging (IM), or email). This paper builds upon the service operations management (SOM) literature on customer contact by theoretically defining and empirically developing new multi-item measurement scales specifically designed for assessing technology-mediated customer contact. Seminal works on customer contact theory and its empirical measurement are employed to provide a foundation for extending these concepts to technology-mediated contexts. We also draw upon other important frameworks, including the Service Profit Chain, the Theory of Planned Behavior, and the concept of media/information richness, in order to identify and define our constructs. We follow a rigorous empirical scale development process to create parsimonious sets of survey items that exhibit satisfactory levels of reliability and validity to be useful in advancing SOM empirical research in the emerging Internet-enabled back-office.

The marketization of the university is reflected in the demand that universities constantly competeon the international market for publications, research funding, scientific labour and students. Today, university researchers have thus become ‘entrepreneurs’ whose most important task is to produce bibliometrically competitive knowledge, generate external funding and offer programmes centred on maximizing candidates ‘employability’. Researchers are increasingly measured and promoted based on their ‘performance’ and ‘innovation’ in these areas rather than their academic skills and knowledge. Research freedom includes not being forced to conduct research aimed at specific journals, selected by politicians or the university management based on bibliometric rankings; a selection that is often a far cry from individual researchers’ and research groups’ academic specialist expertise.

This paper draws on the work of Charles E.
Osgood · which introduces the visual semantic
differential scale and utilization as a suggested approach
in measuring meanings of concepts and concept
clusters. The introduction of the visual semantic
differential scale is not a method to replace the
traditional semantic differential scale. The Visual
Semantic Differential Scale incorporates a photograph
or image to assess the extent to which people come to
define, interpret and understand their world. The paper
makes recommendations· on the use of the visual
semantic differential scale and issues associated with
reality and validity the scale.

The impact of economic recession on the education system of Nigeria at the tertiary level was examined in this study. The study is a descriptive survey research. It was carried out in two universities, two polytechnics and one college of education. Three research questions guided the study. A sample of 304 male and female participants responds to the fifteen item questionnaire which yielded the necessary data. Quantitative data were analysed using descriptive statistics, percentages and charts while qualitative data were recorded and discussed. The findings of the study show that economic recession have affected the system of education in Nigeria in various forms. Based on the findings, recommendations were made on the need to improve education system in Nigeria.

The study investigated the force of using mean scores in assessing the difference in performance of examinees in WAEC and NECO mathematics objective test instruments and student ability parameters in Cross River State. The study used the mathematics objective test instruments for WAEC and NECO examinations on students preparing for their external examinations under a very serene examination atmosphere. The study was based on one research question which was eventually converted to the statement of hypothesis. The design of the study was the expost facto design. The population consist of all senior secondary school three students. Both the stratified and simple random sampling techniques were used for sampling out 731 students from 27 secondary schools. The factorial analysis of variance was carried out on; sex, age and location of schools/ students and their ability parameters in mathematics. It was revealed that there were no significant differences in students' performance in sex in both WAEC and NECO, but there was a significant difference in WAEC in terms of school location; urban students outperformed their rural counterparts. It was recommended by implication that PTA, Government should jointly encourage the posting of qualified mathematics teachers in rural areas, in order to bridge the between the urban and rural areas.

Objective: To assess whether a Smarter Lunchroom intervention based on beha-vioural economics and adapted for students with intellectual and developmental disabilities would increase the selection and consumption of fruits, vegetables and whole grains, and reduce the selection and consumption of refined grains. Design: The 3-month intervention took place at a residential school between March and June 2012. The evaluation employed a quasi-experimental, pre–post design comparing five matched days of dietary data. Selection and plate waste of foods at lunch were assessed using digital photography. Consumption was estimated from plate waste. Setting: Massachusetts, USA. Subjects: Students (n 43) aged 11–22 years with intellectual and developmental disabilities attending a residential school. Results: Daily selection of whole grains increased by a mean of 0?44 servings (baseline 1?62 servings, P 5 0?005) and refined grains decreased by a mean of 0?33 servings (baseline 0?82 servings, P 5 0?005). The daily consumption of fruits increased by a mean of 0?18 servings (baseline 0?39 servings, P 5 0?008), whole grains increased by 0?38 servings (baseline 1?44 servings, P 5 0?008) and refined grains decreased by a mean of 0?31 servings (baseline 0?68 servings, P 5 0?004). Total kilojoules and total gram weight of food selected and consumed were unchanged. Fruit (P 5 0?04) and vegetable (P 5 0?03) plate waste decreased. Conclusions: A Smarter Lunchroom intervention significantly increased whole grain selection and consumption, reduced refined grain selection and consumption , increased fruit consumption, and reduced fruit and vegetable plate waste. Nudge approaches may be effective for improving the food selection and consumption habits of adolescents and young adults with intellectual and developmental disabilities.

A general method for the statistical analysis of Guttman scales is presented. Scale gradient analysis is shown.tn to be an alternative to the coefficient of reproducibility approach. Scale items are interpreted as treatments, groups of respondents as randomized blocks, and the optimally designed perfect scale distribution as a covariate. The structure of a scale matrix is viewed as a two-way analysis of covariance with randomized blocks. The regression coefficient for the sums of observed response scores for each item and subgroup on the predicted perfect scores is  called the scale gradient. The Proportion of residual variance in observed scores explained by the scale assumption is measured, by a statistic called, the determinant of scalability. Multivariate techniques are presented, for tsting the equality of scalability for a variable in independent populations, and for linear combinations of scale gradients
from independent populations. The method is illustrated by a scale of work alienation.

Studies of contemporary research in industry and the academy often note the increasing coexistence of market and academic norms and practices. This article suggests that, at least in industry, these conflicting norms and practices are often preserved by loose coupling between market pressures and the research organization. Based on a two-year case study, this article examines the imposition of tight coupling at an industry lab that had previously been able to maintain some of norms and practices associated with the academy. Tight market coupling limited the role of judgment in the governance of research. First, a market-based quantitative project selection system delegitimized expert judgment. As a result, projects with obvious market and technical uncertainties were more difficult to justify. Second, objective, market-based oversight also limited the role of expert judgment. Tracking the return on investment of scientists’ hours with time cards had the effect of discouraging time spent on learning or exploratory work. In many cases, researchers argued that displacement of judgment for a quantitative governance system caused work to shift away from research and toward product development.

This Impact Report identifies and summarises the diverse impacts, resulting from the £500m of UK funding of Science and Technology in 2013, using numerous quantitative metrics and short case study extracts.  It shows how the varied impacts of a national science and technology organization and its many funded programmes can be annually identified, captured, tracked, and summarised in a public document.

The report was produced for, and by, the Science and Technology Facilities Council (STFC) -  the UK Research Council responsible for funding: (1) all UK astronomy, particle and nuclear physics research, (2) related UK science and technology facilities (e.g. as Harwell and Daresbury), (3) the UK contribution to international science (e.g. CERN, international telescopes and space missions), (4) innovation and business spin-off programmes, and (5) UK public engagement activities.

The Report was co-researched and co-authored by the STFC Impact Team, including myself while Impact Evaluation Manager in 2013, with Dr. Claire Dougan-McCallie (lead author/Head of Impact) and Jenny Beard, with additional input from senior managers, and many data and case study contributions from other key managers across STFC.

Varied key impacts are identified and reported including:

• Research amongst the best (by citation impact) in astronomy, nuclear and particle physics.
• Supported research of 226 Principal Investigators in 70 universities, led to 1,100 peer-reviewed papers in 2012.
• Facilities (the Diamond Light Source, ISIS and CLF) were used in 2012/13 by over 4,200 unique users , in over 2,400 experiments, producing over 1,000 papers in peer-reviewed journals.

• Technology from CERN benefited the UK economy by over £100 billion every year.
• Funding the development of the UK’s space, internet and computer animation industries, returned over £500 billion to the UK economy per annum.
• Funded Innovation Campuses hosted over 230 enterprises and supported over 5,000 jobs.
• UK industry won £43 million in contracts from the international subscriptions funded, and had accumulated a total of £150 million since 2005.

• Funded research inspired future generations to study STEM subjects, attracting 90% of UK undergraduates to study physics, funded 782 PhD students  and 16,800 student training days.
• Between 2009 to 2012 public engagement programmes engaged people on 58 million occasions, with 2 million people engaged in face-to-face activities in 2013 alone.

This research was funded by, and implemented within, a UK national research organisation, while the author was the KE and Impact Evaluation Manager responsible for assessing impact on a £15m UK government-funded research programme.

This work was also submitted as the authors MBA dissertation which was accepted and awarded in 2009.

ABSTRACT: A methodology and methods ‘toolbox’ is proposed for conducting impact research and assessment from within research programmes. The emphasis is upon how to clarify and assess the wider impacts from the early stages onwards.

The approach explicitly acknowledges, and attempts to address, the known issues and challenges of impact assessment while drawing upon the resources and opportunities available to research managers and researchers within such programmes.

The study has two triangulating foundations. Firstly, it examines the academic literature to help identify and design the pilot methodology and methods. Secondly it trials these in a real programme to both test and improve them.

The study was conducted over one year within one leading UK research organisation, where two research programmes (and around 60 research projects of £15m total value) were being assessed for their impacts. The study draws upon actual impact management experience and reflection, prospective (ex ante) and retrospective (ex post) assessments, participatory contributions from leading researchers undergoing assessment, and consultation with external users. The findings of this study outline the pilot and its first iteration. Recommendations are made for improvement to the methodology and methods.

A subsequent follow-on bid to UK government incorporating the impacts identified from implementing this methodology, successfully led to a further £15m of government funding.

It is expected the pilot methodology and methods will be applicable to other organisations, conducting and funding such research programmes, required to research and assess the wider research impacts.

Although primary focus of implemetation trials was upon research in physical sciences, technology, high-tech engineering, computing/IT, and environment, the wider literature study, the methodology design and justification, the methods proposed and used, and the resulting general recommendations and lessons are expected to apply more broadly to many other research areas, where the impact is to be found more widely in the society, economy, government or public sector or within organizations, policies, professions or the public.

Interest in this work is expected from research programme managers, impact managers and assessors, management consultants advising in this area, and government funding managers, all seeking to embed a developing approach to understand, capture, and enhance wider impact.

More specifically, according to the law, main ANVUR activities include: • The evaluation of the quality of processes, results, and products of all the activities concerning research, teaching and technological transfer performed by Universities and Research bodies • The definition of methods and criteria -based on objective parametersto be used for the periodical accreditation of Universities and research bodies (undergraduate and postgraduate courses and curricula) In its activities, ANVUR makes use of bibliometric information contained in the major international databases More specifically, since 2010 ANVUR has been involved in three main projects implying the use of bibliometric information and indicators: • The quality of research evaluation exercise for the period 2004-2010 (VQR 2004-2010, from the Italian Valutazione della Qualità della Ricerca), results of which will be presented on July 16, 2013 • The National Habilitation program, which is currently under way • The bi-annual report on the Italian Research system, which is scheduled for release in autumn this year 6

NARCH). To incorporate a community-based participatory research (CBPR) approach within this funded research, a national advisory committee was formed that included principal investigators from the NCMHD and NARCH partnerships, other CBPR national experts, community members involved in selected CBPR projects, and representatives of the Navajo Nation. c21.indd 371 c21.indd 371 7/31/08 7:31:12 PM 7/31/08 7:31:12 PM

How do we classify and evaluate published research articles? The discussion that follows will challenge aspects implicit in standard methods of classification, while illustrating the importance of definition as a tool for enhancing the value derived from reading research articles. The task will provide a brief resumé of the contents of two published academic articles, methods of classifying research articles will be explored and, finally, an alternative method of evaluation will be explored.