Measurement and Evaluation (Research Methodology)

The study investigated the force of using mean scores in assessing the difference in performance of examinees in WAEC and NECO mathematics objective test instruments and student ability parameters in Cross River State. The study used the mathematics objective test instruments for WAEC and NECO examinations on students preparing for their external examinations under a very serene examination atmosphere. The study was based on one research question which was eventually converted to the statement of hypothesis. The design of the study was the expost facto design. The population consist of all senior secondary school three students. Both the stratified and simple random sampling techniques were used for sampling out 731 students from 27 secondary schools. The factorial analysis of variance was carried out on; sex, age and location of schools/ students and their ability parameters in mathematics. It was revealed that there were no significant differences in students' performance in sex in both WAEC and NECO, but there was a significant difference in WAEC in terms of school location; urban students outperformed their rural counterparts. It was recommended by implication that PTA, Government should jointly encourage the posting of qualified mathematics teachers in rural areas, in order to bridge the between the urban and rural areas.

This Impact Report identifies and summarises the diverse impacts, resulting from the £500m of UK funding of Science and Technology in 2013, using numerous quantitative metrics and short case study extracts.  It shows how the varied impacts of a national science and technology organization and its many funded programmes can be annually identified, captured, tracked, and summarised in a public document.

The report was produced for, and by, the Science and Technology Facilities Council (STFC) -  the UK Research Council responsible for funding: (1) all UK astronomy, particle and nuclear physics research, (2) related UK science and technology facilities (e.g. as Harwell and Daresbury), (3) the UK contribution to international science (e.g. CERN, international telescopes and space missions), (4) innovation and business spin-off programmes, and (5) UK public engagement activities.

The Report was co-researched and co-authored by the STFC Impact Team, including myself while Impact Evaluation Manager in 2013, with Dr. Claire Dougan-McCallie (lead author/Head of Impact) and Jenny Beard, with additional input from senior managers, and many data and case study contributions from other key managers across STFC.

Varied key impacts are identified and reported including:

• Research amongst the best (by citation impact) in astronomy, nuclear and particle physics.
• Supported research of 226 Principal Investigators in 70 universities, led to 1,100 peer-reviewed papers in 2012.
• Facilities (the Diamond Light Source, ISIS and CLF) were used in 2012/13 by over 4,200 unique users , in over 2,400 experiments, producing over 1,000 papers in peer-reviewed journals.

• Technology from CERN benefited the UK economy by over £100 billion every year.
• Funding the development of the UK’s space, internet and computer animation industries, returned over £500 billion to the UK economy per annum.
• Funded Innovation Campuses hosted over 230 enterprises and supported over 5,000 jobs.
• UK industry won £43 million in contracts from the international subscriptions funded, and had accumulated a total of £150 million since 2005.

• Funded research inspired future generations to study STEM subjects, attracting 90% of UK undergraduates to study physics, funded 782 PhD students  and 16,800 student training days.
• Between 2009 to 2012 public engagement programmes engaged people on 58 million occasions, with 2 million people engaged in face-to-face activities in 2013 alone.

The impact of economic recession on the education system of Nigeria at the tertiary level was examined in this study. The study is a descriptive survey research. It was carried out in two universities, two polytechnics and one college of education. Three research questions guided the study. A sample of 304 male and female participants responds to the fifteen item questionnaire which yielded the necessary data. Quantitative data were analysed using descriptive statistics, percentages and charts while qualitative data were recorded and discussed. The findings of the study show that economic recession have affected the system of education in Nigeria in various forms. Based on the findings, recommendations were made on the need to improve education system in Nigeria.

This research was funded by, and implemented within, a UK national research organisation, while the author was the KE and Impact Evaluation Manager responsible for assessing impact on a £15m UK government-funded research programme.

This work was also submitted as the authors MBA dissertation which was accepted and awarded in 2009.

ABSTRACT: A methodology and methods ‘toolbox’ is proposed for conducting impact research and assessment from within research programmes. The emphasis is upon how to clarify and assess the wider impacts from the early stages onwards.

The approach explicitly acknowledges, and attempts to address, the known issues and challenges of impact assessment while drawing upon the resources and opportunities available to research managers and researchers within such programmes.

The study has two triangulating foundations. Firstly, it examines the academic literature to help identify and design the pilot methodology and methods. Secondly it trials these in a real programme to both test and improve them.

The study was conducted over one year within one leading UK research organisation, where two research programmes (and around 60 research projects of £15m total value) were being assessed for their impacts. The study draws upon actual impact management experience and reflection, prospective (ex ante) and retrospective (ex post) assessments, participatory contributions from leading researchers undergoing assessment, and consultation with external users. The findings of this study outline the pilot and its first iteration. Recommendations are made for improvement to the methodology and methods.

A subsequent follow-on bid to UK government incorporating the impacts identified from implementing this methodology, successfully led to a further £15m of government funding.

It is expected the pilot methodology and methods will be applicable to other organisations, conducting and funding such research programmes, required to research and assess the wider research impacts.

Although primary focus of implemetation trials was upon research in physical sciences, technology, high-tech engineering, computing/IT, and environment, the wider literature study, the methodology design and justification, the methods proposed and used, and the resulting general recommendations and lessons are expected to apply more broadly to many other research areas, where the impact is to be found more widely in the society, economy, government or public sector or within organizations, policies, professions or the public.

Interest in this work is expected from research programme managers, impact managers and assessors, management consultants advising in this area, and government funding managers, all seeking to embed a developing approach to understand, capture, and enhance wider impact.

A general method for the statistical analysis of Guttman scales is presented. Scale gradient analysis is shown.tn to be an alternative to the coefficient of reproducibility approach. Scale items are interpreted as treatments, groups of respondents as randomized blocks, and the optimally designed perfect scale distribution as a covariate. The structure of a scale matrix is viewed as a two-way analysis of covariance with randomized blocks. The regression coefficient for the sums of observed response scores for each item and subgroup on the predicted perfect scores is  called the scale gradient. The Proportion of residual variance in observed scores explained by the scale assumption is measured, by a statistic called, the determinant of scalability. Multivariate techniques are presented, for tsting the equality of scalability for a variable in independent populations, and for linear combinations of scale gradients
from independent populations. The method is illustrated by a scale of work alienation.

Virtual machines (VMs) are underlying technologies of IT solutions such as cloud computing. VMs provide ease of use through their on-demand characteristics and provide huge benefits in terms of lowering costs and improving scalability. VMs are also being used as malware detection systems, and with the rapidly expanding usage of mobile devices, besides of their usage as honeypots, VMs are coming to be used as emulators for detecting malware in apps. This is due to the limited resources, such as processing power, available in mobile devices. Currently, the security of applications for mobile devices is checked by running them in VM environments before they are released to the end user. We argue that such a process may cause or overlook serious security threats to the end user. In particular, if a piece of malware can detect its current running environment, it may change its behavior such that it doesn't perform malicious operations in environments it suspects to be emulators. In this way, when the malware detects that its running environment is on a VM, it may be able to hide from the security system on the VM. This is a potential security hazard for end users, especially users of mobile devices. In this paper, we present a VM detection method that we argue could be used for remotely detecting VM environments. The detection method works by analyzing the pattern of IP timestamps in replies sent from the target environment. The method does not require any installation of software on the target machine which further increase its potential harm if it were to be used by malware to detect VM environments. In this paper, we also present a technique to disguise a real PC machine such that it shows the similar IP timestamp patterns as the VM. By using this technique, malware may not be able to differentiate between a real machine and a VM, thus providing protection to PC end users.

The marketization of the university is reflected in the demand that universities constantly competeon the international market for publications, research funding, scientific labour and students. Today, university researchers have thus become ‘entrepreneurs’ whose most important task is to produce bibliometrically competitive knowledge, generate external funding and offer programmes centred on maximizing candidates ‘employability’. Researchers are increasingly measured and promoted based on their ‘performance’ and ‘innovation’ in these areas rather than their academic skills and knowledge. Research freedom includes not being forced to conduct research aimed at specific journals, selected by politicians or the university management based on bibliometric rankings; a selection that is often a far cry from individual researchers’ and research groups’ academic specialist expertise.

This study examines the types of situations, or triggers, which cause embarrassment and their relationship with attachment and communication apprehension. Two embarrassment measures were compared: Modigliani’s (1968) embarrassability scale and Sabini, Siepmann, Stein, and Meyerowitz (2000) embarrassment questionnaire. Participants (N = 350) completed embarrassment and personality measures. Correlational and regression analyses were conducted. All embarrassment measures were related to communication apprehension and the anxious attachment dimension, but not the avoidant attachment dimension. Not surprisingly, all embarrassment measures were significantly positively correlated with one another. The Center of Attention embarrassment trigger significantly predicted communication apprehension above and beyond the other embarrassment measures in the regression model. In contrast, Faux Pas and Sticky Situations embarrassment triggers were independently associated with anxious attachment. The implications for the conceptualization and measurement of embarrassment are discussed.

Service organizations are increasingly utilizing advanced information and communication technologies, such as the Internet, in hopes of improving the efficiency, cost-effectiveness, and/or quality of their customer-facing operations. More of the contact a customer has with the firm is likely to be with the back-office and, therefore, mediated by technology. While previous operations management research has been important for its contributions to our understanding of customer contact in face-to-face settings, considerably less work has been done to improve our understanding of customer contact in what we refer to as technology-mediated settings (e.g., via telephone, instant messaging (IM), or email). This paper builds upon the service operations management (SOM) literature on customer contact by theoretically defining and empirically developing new multi-item measurement scales specifically designed for assessing technology-mediated customer contact. Seminal works on customer contact theory and its empirical measurement are employed to provide a foundation for extending these concepts to technology-mediated contexts. We also draw upon other important frameworks, including the Service Profit Chain, the Theory of Planned Behavior, and the concept of media/information richness, in order to identify and define our constructs. We follow a rigorous empirical scale development process to create parsimonious sets of survey items that exhibit satisfactory levels of reliability and validity to be useful in advancing SOM empirical research in the emerging Internet-enabled back-office.

Studies of contemporary research in industry and the academy often note the increasing coexistence of market and academic norms and practices. This article suggests that, at least in industry, these conflicting norms and practices are often preserved by loose coupling between market pressures and the research organization. Based on a two-year case study, this article examines the imposition of tight coupling at an industry lab that had previously been able to maintain some of norms and practices associated with the academy. Tight market coupling limited the role of judgment in the governance of research. First, a market-based quantitative project selection system delegitimized expert judgment. As a result, projects with obvious market and technical uncertainties were more difficult to justify. Second, objective, market-based oversight also limited the role of expert judgment. Tracking the return on investment of scientists’ hours with time cards had the effect of discouraging time spent on learning or exploratory work. In many cases, researchers argued that displacement of judgment for a quantitative governance system caused work to shift away from research and toward product development.

How do we classify and evaluate published research articles? The discussion that follows will challenge aspects implicit in standard methods of classification, while illustrating the importance of definition as a tool for enhancing the value derived from reading research articles. The task will provide a brief resumé of the contents of two published academic articles, methods of classifying research articles will be explored and, finally, an alternative method of evaluation will be explored.