Test of professional competence: first pilot examination

Test of Professional Competence First Pilot Examination Report on First Pilot Feedback Paul Maharg 2 Contents Page Introduction 3 Preparation 4 Quantitative Feedback 5 Qualitative Feedback 7 Group Interviews 11 Conclusions 12 Recommendations 13 Appendix 1: Comparison of Statistical Charts 14 Appendix 2: Qualitative Feedback 24 Appendix 3: Group Interview Reports 37 Test of Professional Competence, Pilot I Report 21.5.02 3 Introduction Arising from the work of the Test of Professional Competence (TPC) Panel, and adhering to guidelines set out by Education and Training Committee, it was determined that there would be a Test of Professional Competence that, inter alia, included an open-book assessment of trainees’ skills and knowledge. To this end, the Panel drew up extensive learning outcomes, assessment outcomes and administrative and procedural documentation for the examination. In order to prepare for the introduction of the open-book assessment of trainees’ professional competence, the TPC panel decided to hold two pilots. The first pilot, which is the subject of this report, would be small-scale, and recommendations from it would inform the much larger-scale second pilot. Following on from the results of these pilots, it was determined that the TPC Panel would make recommendations to Education and Training Committee as to the form and nature of a test of professional competence open-book examination. The Law Society therefore assisted Dr Paul Maharg in the administration of the first pilot, which is the subject of this report. It was determined that there would be three areas of assessment in the first pilot, namely Public Administration, Company & Commercial and Criminal. Examination authors were appointed and given guidelines and basic training on the requirements of the task. The openbook papers were written, cross-checked for validity, fairness and reliability, and the papers were copied for the 24 trainees who had volunteered for the pilot assessment. These trainees were drawn from the year above that which would first be sitting the TPC in earnest. The examination took place in the Law Society building at Drumsheugh Gdns, and the papers were collected and marked. The purpose of this report is to determine the success of the first pilot, and to make recommendations as to the form and substance and procedures to be adopted in the second. Please note that it should be considered in the light of the authors’ and markers’ experiences, their assessment scheme, and the results of the pilot. A second pilot report will be drawn up, similarly, at the end of the second pilot, and the general success or otherwise of this mode of assessment will be the subject of discussion and analysis by the TPC panel in the first instance, and thereafter by Education and Training Committee, who will report to the Council of the Law Society of Scotland. Test of Professional Competence, Pilot I Report 21.5.02 4 Preparation of examiners Examiners for the areas of Criminal, Public Administration and Company and Commercial were appointed, and several meetings were held at which they were briefed using materials prepared by the TPC committee. The aims of the examination, context and format were explained. Authors wrote first drafts of examination papers which were commented on, redrafted and then copied for the exam. It was interesting to note that while authors in Company/Commercial (CC) and Public Administration (PA) had adhered to the guidelines for interaction with the examinees over supplementary materials, the Criminal (C) author had much more dialogue between the examinees and himself. This had consequences for the C. assessment, as we shall see, but for now it should be noted that while this is a possible route for assessment, the quality assurance risk is proportionately higher, particularly if authors deal in this way with larger numbers of trainees and across areas of law. Test of Professional Competence, Pilot I Report 21.5.02 5 Quantitative Feedback The results exhibit a fair degree of uncertainty about the standard of work expected in the assessment, especially among CC trainees. It is probably a factor that contributed to the uncertainty surrounding the underlying purpose of the assessment, as demonstrated in Q.14. This is something that the Law Society may want to improve for the second pilot. The trainees who will be sitting the second pilot will not, of course, be those who will sit it in earnest. Nevertheless, the second pilot is a useful opportunity to practise the communication of aims, criteria and standards (more of which, below). CC trainees appeared to have encountered the assessment topics previously in their traineeships, but PA trainees did not (Q.2). This goes to the heart of some comments made by trainees in the group session discussion, in which they pointed out that such was the variety of traineeships within a particular area that some trainees would have had experience of a topic, while others would not. It is difficult to propose action on this point. Arguably, and in a sense this goes back to earlier comments made by the author of this report, the Law Society ought to commission a study of competence in the trainee work place, particularly if the assessment is based upon work-place competence. However at this late stage it is not practicable to do this. It should be borne in mind that a possible ground of appeal is that a trainee is tested in an area of work of which he or she has had no prior experience. Most trainees were equivocal about whether the assessment would motivate them to do their best work (Q.3). Possibly if they were sitting the exam in reality this would change, but it is significant that the exam did not engage their interest very much. Q.4 attracted much more unanimity. Almost all agreed that the process of preparing for the exam was not heavy, and this was borne out by the responses to Q.10 and, as regards the element of time in the exam itself, Q.12. Q.17, querying burden of work and comprehension, also bears this out: most trainees (over half, and CC in particular) felt that the volume of work did not affect their ability to comprehend the conceptual material. Q.5 showed a mixed response across the three categories, with the majority of CC trainees commenting positively. The difficulty of the examination, memory work required, practical skills and understanding attracted moderate responses. Q.11, stating there was too much assessment of knowledge, attracted more agreement than disagreement, while most trainees thought that they had done well in the assessment (Q.19), bearing out their experience of the examination as relatively easy. Q.18 is a key question on whether the assessment tested traineeship learning. A third of the trainees thought it did not. The reasons for this are explored in the extended comments made by students on this question. Q.20 again demonstrates an equivocal response. A substantial proportion thought that this assessment was not a good way of assessing whether a trainee should be allowed into the profession (10), while nine thought it was, with four unsure. Test of Professional Competence, Pilot I Report 21.5.02 6 These results support the comments made by trainees on the experience of sitting the examination, both written comments and the results of the three group interviews. As they stand, the results do not point to the assessment being either markedly rigorous or a comprehensive test of competence. Test of Professional Competence, Pilot I Report 21.5.02 7 Qualitative Feedback Public Administration The general opinion was that the examination was too academic: This was not a test of ability or competence as a trainee solicitor. There was over-emphasis on knowledge and not enough emphasis on skills. I expected an assessment with various tasks set. The papers given in advance appeared to indicate this. I obviously read papers wrong – was expecting a brief memo, draft byelaws or management rules, letter to address complaint, etc In addition, the LSS need to give thought to widening the variety of papers for trainees. Even in an area as apparently compact as Public Administration it is clear that there are at least two constituences: As a Scottish Executive Trainee, I felt that the content of the paper focused too heavily on areas which related to Local Authorities (eg making byelaws etc). I have never dealt with such powers in my traineeship. It was also pointed out that open book examinations are not necessarily appropriate assessment instruments for all forms of leal tasks: Preparing a report is not normaly done in a single draft in such a short time frame. Topic too wide for time allowed. Company & Commercial CC trainees were the most negative of the three groups of trainees as regards the usefulness of the examination as a form of traineeship assessment: This was the lowest common denominator assessment. No scope whatsover for any creative legal or commercial thinking and almost no test of actual legal knowledge. […] As it stands this really doesn’t scratch the surface of competency and, if more public, would further diminish the opinion of lawyers in the eyes of the public if this is the benchmark of competency. The same trainee argued the case for an oral exam, but there are a number of problems associated with this. He/she also noted that Everyone was able to use his or her Diploma notes, which are two years old. This sits awkwardly with any kind of development during the traineeship. This is a crucial point, and one that goes to the heart of the way in which the professional regime has been constructed. As the TPC and PCC committes have pointed out, there needs to be a matrix of development throughout the professional training regime and across the various elements of the training programme. The development within Diploma, and across Diploma, PCC and traineeship needs to be addressed by the LSS: Test of Professional Competence, Pilot I Report 21.5.02 8 The commercial test I sat was very similar to exercises undertaken in management and formation of companies during the Diploma and did not reflect the fact that I am now carrying out the work. The restricted nature of the open book assessment attracted negative comment: It was an unrealistic test given a normal work environment and the rule that only one book was permitted – you cannot choose between a textbook and statutes! The sense of an academic test dressed up as an case study was further emphasised by the lack of contact with parties: There seemed little interaction with the parties unlike the other test subjects and no communication response… […] No research was required. […] Where were the questions on ethics? Do corporate lawyers not need professional ethical standards? These points illustrate the considerable difficulty of producing scenarios, instructions and contexts for as close a simulation to reality as is needed to achieve a reasonable simulation of practice within the traineeship. There seemed to be an uneasy mix (present in all papers) of the academic question-setting and problem-based approach. This point was dwelt on at some length by one trainee, who pointed out the fundamentally different form of problem-solving that is adopted in any form of examination, as opposed to office practice: … someone who is both competent and professional would go and look up a book or refer to a colleague if they were unsure of how to approach a problem. However, an exam format not only encourages but requires people to grasp at answers and write down what you think might be right so as you have a chance of any marks going – this is what we were always told to do in school, ‘never leave a question blank, you won’t get any marks that way!’ However this approach is the antithesis of competence and professionalism (despite being the way to score well in exams!) […] competent professionals will always ensure by research and double checking that their approach to a problem is the best one – it is this mindset that should be encouraged – not grasping at answers – that surely is negligent. Even allowing for exaggeration, it must be acknowledged that there are telling points made here about the trainee’s experience of sitting the exam vis-à-vis his/her experience of practice. The TPC sits uneasily between an academic test, and a true competence assessment. The latter involves visits to the workplace, the construction of a matrix of competences to be assessed, and the assessment of the matrix through observation of practice in the workplace. Close comments on the questions bore out the above remarks: To do well in this assessment all you really need is a good memory To do well you needed a good set of styles! No memory testing at all required. If memory was not needed, neither were skills thought to be a particularly important element, either: There was over-emphasis on skills at the expense of knowledge in my assessment I don’t feel I really used any skills during the assessment. Impossible to separate skills from knowledge The overemphasis was definitely towards knowledge. Test of Professional Competence, Pilot I Report 21.5.02 9 Criminal If trainees were puzzled as to where ethics appeared in the CC materials, it was not the case in Criminal: Expected exam to be related far more to specifics of case and how we would run/prepare defence ourselves. Instead it was more concentrated on ethics and scenarios which do no arise on a day-to-day basis and not particularly related to case itself. However, the same comments were made regarding the relationship of the exam to traineeships: If the assessment was intended to examine my traineeship and if I had only attempted to answer those questions, which I should be expected to answer as a result of my training, then I would definitely fail. I would only have been in a position to answer part of 3 of 10 questions. This is not in itself a comment on the paper, but on the difficult relationship between knowledge and skills, between academic assessment and trainee experience that is present throughout most of these comments. There is not necessarily a polarisation between knowledge and skills: rather, the polarisation occurs in the form of the assessment, which is uneasily sited half-way between academic examination and competence-based assessment. As a result, trainees are uncertain how to respond to the questions: as a trainee, or as a student? Once again, there is a convincing case made for the necessity to widen the types of exam papers made available to Criminal trainees dependent on the nature of their traineeship: … as a trainee for the Crown Office and Procurator Fiscal Service I could not answer questions on:• legal aid • client relations • conflict of interest • defence preparation of a case The trainee who pointed this out was also complimentary of the paper, and did say that in his/her opinion, there would not be much required to redraft this paper to suit a Crown Office trainee. Another trainee was less impressed: To do well in this assessment all you really need is a good memory Exam was too academic/theoretical. Cramming up on Criminal Procedure ‘theory’ is all that was necessary. This assessment was a good test of our practical knowledge and skills Exam could have tested more of our actual practical skills used on a dayto-day basis. The assessment was of course limited in the forms of such practical skills it could have tested. Nevertheless, this was a comment that was made by a number of trainees. This is quite a fundamental point in the assessment of competence. In most forms of competence assessment, the assessment is work-based. Assessors visit the workplace, and observe the trainee doing tasks in the work context. The TPC is quite different, in that trainees are taken out of the work context and assessed in the exam room. Preparation is carried out in the workplace, but the final product of assessment by which trainees will be assessed is performed in the exam room. Even given the relative freedom of a quasi-open book system, this Test of Professional Competence, Pilot I Report 21.5.02 10 end-point affected the ways that trainees prepared for the case-studies. As we shall see below, very probably what they produced in the exam room as a solution might not have been what they might have done in the office; or at least the process of arriving at the solution was different. In effect, the format of the examination is skewing the competence base of the assessment. Test of Professional Competence, Pilot I Report 21.5.02 11 Group interviews Three group interviews were carried out, each with a spread of the three categories of trainees. It was hoped that the interviewers could thus gain a sense of the contrasting experiences across the assessment. The sense of contrast was certainly borne out by the responses within the groups. Most of the responses reinforced points made in the questionnaires, and to this extent validated both the qualitative and quantitative data there. They generally support the comments made on the questionnaire. Many comments refer to the lack of challenge in the exam, and point to the reduced place of skills within them. Not many of them can be said to be positive about the content of the assessment, though some were prepared to acknowledge that the form of the assessment may be more successful if the content is improved. Test of Professional Competence, Pilot I Report 21.5.02 12 Conclusions This pilot reveals that there are a number of fundamental issues that remain unresolved, but perhaps the most fundamental is the relationship of the three important landmarks in the assessment: the assessment criteria underlying the assessment, the work experience of trainees who are sitting the assessment, and the format of the examination. All of these must be aligned with each other if there is to be an assessment that succeeds in its aims of testing professional competence. Thus the criteria must relate to what takes place in the office, and what trainees actually do there. The experience of trainees in the office must take account of the criteria, and they must know that what they do will be a preparation for assessment. Most important is the effect that the variety of trainees’ work experience has upon the Law Society’s ability to set an examination that is fair and valid for all. This is especially the case in professional education where, as we have seen, the potential variety of experience, even in this relatively small sample, is considerable. At present, neither current trainees nor their trainers have knowledge of the assessment criteria. It is unlikely that they are implementing the criteria in any meaningful way. The eight areas of legal practice under assessment need to be reconsidered. In each of the three areas in the pilot there were trainees who found difficulty in the assessment because they had not worked within the area in the ways intended by the examiner. In Criminal, there were both Crown and defence trainees; in Public Administration there were local authority and Executive trainees, while more than one trainee sitting the Company & Commercial assessment point out that corporate, commercial & company work can cover a huge range of tasks and knowledge components. The information flow to current trainees and supervisors may need to be reconsidered. Supervisors should be aware of the importance of giving trainees experience in tasks that prepare the trainees in the assessment criteria for the examination. The second pilot may be best to focus on fewer areas of assessment, and ensure that the training of authors is tied securely to assessment criteria and, where possible, description of, or knowledge about, trainee work practices. It must be said that the authors for the first pilot produced exams that were product of much thought and diligence on their part, and not a little originality. However the feedback on the examination from the trainees suggests that everyone in the process has underestimated the difficulty in designing successful competencebased assessment resources. Should the second pilot be subject to the same level of negative feedback from trainees, it may be necessary to reconsider either postponement or abandonment of the test element of the TPC. This obviously is a major decision, and not one to be undertaken lightly. Test of Professional Competence, Pilot I Report 21.5.02 13 Recommendations 1. Widen the categories of examination areas being assessed by the LSS. Bearing in mind the lengthy debates that took place in the TPC committee on this subject, the LSS may require to carry out field work on the types of legal work carried out by trainees in order to determine the categories of traineeship work. This field work will require to be updated on a regular basis by administrators, and carried out to a lesser extent in subsequent years. 2. Write a document, addressed for trainees, that explains in detail the purpose of the TPC, its assessment criteria, and the links between tasks carried out in the workplace and the TPC assessment 3. Exemplars for trainees that will set the standard of work to be achieved in the areas of the second pilot. 4. Train authors extensively in setting competence-based assessments, ie on specifically skills-based assessment, the writing of case studies and setting of questions on case studies. Set aside at least two days in which this training can take place. 5. Reduce the number of areas in the second pilot in order to focus on intensive training of TPC authors 6. Increase the practice complexity of the examination materials, while still keeping it within the boundary of trainee work experience 7. Inform training supervisors in firms on the rationale and working plan for the TPC 8. Develop the links between the TPC assessment outcomes and the rest of the postgraduate training programme, and publicise this 9. Assessments should be planned on a matrix, so that there is some continuity of assessment of the TPC criteria between areas of law being assessed 10. Clarify instructions to trainees regarding use of materials in the examination room 11. Widen the categories of texts that can be brought into the examination. 12. Assign weightings to questions 13. If the second pilot is not successful, either postpone the TPC for a subsequent year, or abandon the current TPC assessment format and remit to the TPC Committee for consideration. Test of Professional Competence, Pilot I Report 21.5.02 14 Appendix One: Comparison of Statistical Charts Question 1: It was easy to know the standard of work expected of me in this assessment. 10 8 6 4 4 2 4 3 2 1 3 2 2 1 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Definitely disagree No Response Pub Administration Question 2: I had not dealt in my traineeship with the topics being assessed. 10 8 7 6 4 4 2 4 2 1 1 1 1 2 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Test of Professional Competence, Pilot I Report Definitely disagree No Response Pub Administration 21.5.02 15 Question 3: This assessment will motivate trainees to do their best work. 10 8 6 4 4 2 4 3 3 3 2 1 1 1 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Definitely disagree No Response Pub Administration Q4: The preparation was too heavy. 10 8 6 6 5 4 5 3 2 2 2 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Test of Professional Competence, Pilot I Report Definitely disagree No Response Pub Administration 21.5.02 16 Q5: The assessment I was given tried to cover too many topics. 10 8 7 6 4 4 2 3 2 2 2 2 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Definitely disagree No Response Pub Administration Q6: The assessment was too easy. 10 8 6 5 4 3 3 2 2 2 3 2 2 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Test of Professional Competence, Pilot I Report Definitely disagree No Response Pub Administration 21.5.02 17 Q7: To do well in this assessment all you really need is a good memory. 10 8 6 5 4 4 2 4 3 2 1 2 1 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Definitely disagree No Response Pub Administration Q8: This assessment was a good test of our practical knowledge and skills. 10 8 7 6 4 4 3 2 2 1 2 2 1 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Test of Professional Competence, Pilot I Report Definitely disagree No Response Pub Administration 21.5.02 18 10 8 6 5 4 4 2 3 2 2 2 1 1 2 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Definitely disagree No Response Pub Administration Q10: It would be possible to get through this assessment just by working hard the day before. 10 8 6 6 4 4 2 2 2 2 1 2 1 2 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Test of Professional Competence, Pilot I Report Definitely disagree No Response Pub Administration 21.5.02 19 Q11: There is too much assessment of knowledge in this exam. 10 8 6 4 4 2 3 3 3 3 3 2 1 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Definitely disagree No Response Pub Administration Q12: We were given enough time to understand the things we had to learn for the assessment. 10 8 6 5 4 4 2 4 3 2 2 1 1 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Test of Professional Competence, Pilot I Report Definitely disagree No Response Pub Administration 21.5.02 20 Q13: The assessment I was given was a good test of my ability as a trainee solicitor in this area of law. 10 8 6 6 4 4 2 3 3 2 2 1 2 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Definitely disagree No Response Pub Administration Q14: The aims of this assessment were not clear to me. 10 8 6 4 4 4 3 2 2 2 2 2 1 2 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Test of Professional Competence, Pilot I Report Definitely disagree No Response Pub Administration 21.5.02 21 Q15: There was over-emphasis on skills at the expense of knowledge in my assessment. 10 8 6 4 4 4 3 2 2 2 2 1 1 2 1 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Definitely disagree No Response Pub Administration Q16: This assessment will put trainees under too much pressure. 10 8 6 5 4 4 2 3 2 2 2 1 1 1 1 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Test of Professional Competence, Pilot I Report Definitely disagree No Response Pub Administration 21.5.02 22 Q17: The sheer volume of work to be done for this assessment means you can’t comprehend it all thoroughly. 10 8 6 6 4 4 2 3 2 2 2 2 1 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Definitely disagree No Response Pub Administration Q18: This assessment tested what I learned on my traineeship. 10 8 6 6 5 4 2 3 3 3 2 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Test of Professional Competence, Pilot I Report Definitely disagree No Response Pub Administration 21.5.02 23 Q19: I feel confident that I’ve done well in this assessment. 10 8 6 6 4 4 3 3 2 2 2 1 1 1 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Definitely disagree No Response Pub Administration Q20: I think an open book assessment is generally a good way of assessing whether a trainee should be allowed into the profession. 10 8 6 4 2 3 3 3 2 2 1 1 3 2 1 2 1 0 Definitely agree Agree / reservations Criminal Not sure / doesn't apply Disagree Company & Commercial Test of Professional Competence, Pilot I Report Definitely disagree No Response Pub Administration 21.5.02 24 Appendix 2: Qualitative Feedback Company and Commercial Comments General § This was the lowest common denominator assessment. No scope whatsoever for any creative legal or commercial thinking and almost no test of actual legal knowledge. Of course trainees know how to fill in forms (share certs……) Of course trainees know how to copy and adapt style documents (Board Minutes/resolutions). It would be a harder and a better test if they were NOT allowed open book etc and were only allowed statutes: this would mean we had to understand how resolutions and minutes were structured and would demonstrate genuine thought in approaching the problems. TPC is not a difficult obstacle to get over as it is currently formulated and I think firms and trainees will become irritated by TPC very quickly if it does not actually TEST. I think an oral part of the exam would be an excellent addition. This could range from specifics (e.g. “what is the effect of S80 Companies Act 1985, when would you expect to come across it? Etc) to more general (e.g. “how would you go about researching this particular problem? Etc). Oral element would be unknown therefore would ensure real preparation and, more importantly, knowledge and understanding by the trainee. As it stands this really doesn’t scratch surface of competency and, if more public, would further diminish the opinion of lawyers in the eyes of the public if this is the benchmark of competency. With regard to preparation very little required. Everyone was able to use his or her diploma notes, which are 2 years old. This sits awkwardly with any kind of development during the traineeship. Where were the questions on ethics? Do corporate lawyers not need professional ethical standards? § The test covered issues such as employment law which is not covered in my traineeship in the corporate department. From the papers it was unclear what would be assessed in the exam. It was an unrealistic test given a normal work environment and the rule that only one book was permitted – you cannot choose between a textbook and statutes! There seemed little interaction with the parties unlike the other test subjects and no communication response for the ……. No research was required. There appeared to be no ethical issues in the test. There appeared to be great disparity between the papers as to the skills and general research and preparation required prior to the test being sat. The writing of minutes etc as required in the test was somewhat tedious. Most workplaces have styles on computer for this purpose. The time was tight because of the amount of writing required. I know what to do but it took ages to write it all out. § I felt that a set of good styles (board minutes etc) was what was needed to do well. Several of the questions simply mirrored copying styles and I felt this was a waste of time. As it took so long to copy out styles, this left little time for the few questions that required more thought. Test of Professional Competence, Pilot I Report 21.5.02 25 § § This assessment dealt too much with one very narrow area. If you have a good memory or can remember which of your style minutes to bring it was just an exercise in parrot copying. There was one throwaway at the end which related to ethics and only one small part: like a question. It might have made you think more if there was less emphasis on mechanical production of documents than if you had actual questions relating to practical work.……………………………………………………………………………………………………………… …………………………………………………… I was bored writing minutes in the office you never write like that you use a style because that sort of thing rarely changes…………………………………. …………………………………………………………………………………. …………………………………………………………………………………. In general the set up of the test seems like a good idea – hypothetical file and time to consider the issues followed by additional information and assessment thereon. However, the assessment was simply an exercise in being able to write quickly and copy out a style from the folder of notes. The subject matter of the file was entirely relevant to what I expected but the way of assessing did not seem relevant or helpful – the better questions were those which required you to think about what you would do e.g. how to protect business from claims? Breach of RC, list the documents needed to effect the allotment but then having to write out all those documents from scratch was a waste of time and not relevant in that you never have to do this in practice. Writing a letter would be a good question but I have been told in practice that you would never write a letter about directors’ § I was confused beforehand as to whether this was a commercial property exam or corporate – this should be clear from the outset. § I don’t agree with the introduction of the test. The commercial test I sat was very similar to exercises undertaken in management and formation of companies during the Diploma and did not reflect the fact that I am now carrying out the work. I don’t think the test was particularly knowledge based and was essentially an exercise in copying pre directed board minutes and resolutions and also the somewhat tedious task of filling in ……… If after 5 years at Uni you couldn’t copy documents then I think there would really be something wrong which would be highlighted long before you sat this exam. From the information I was supplied with (not the file) I was not sure what was expected of me. I don’t think it was made clear what the assessment hoped to achieve or even whether it was to be a practical and knowledge based exam. I am also concerned over the ……..of the exams. For example the “commercial development” can be very ….. and may not cover any corporate work. For example I worked in social housing in the commercial department and therefore did not do any corporate work at that point. It would seem ….. to those who do other aspects of commercial work to have to sit such a narrow exam. Test of Professional Competence, Pilot I Report 21.5.02 26 § The materials were incomplete in that no details of the allottee other than his surname were given which meant that candidates could not complete the tasks which they were asked to. Candidates should have been advised to bring or been supplied with a selection of style minutes as not many people will remember exactly what the text of resolutions should be and a lot of time is wasted filling out headings for board minutes. § I was fortunate in that I’d completed seats in corporate and employment teams prior to sitting the pilot TPC. I believe these gave me a good understanding of the relevant issues. In the preparatory information, there was an emphasis on ethics and conduct which I don’t feel was reflected in the test. The chronology of events, i.e. whether Mr McNish was resigning as a director prior to or following the allotment of shares to Mr Masden was unclear, which led to confusion. I think this exam should be completely open-book as trainees should have available all the materials that would be available to them in the office – I see no reason for this to be otherwise. I’m not sure the test was difficult enough, although this might be due to my level of experience in these particular areas. I do think the file gave a sufficient indication of the issues that were raised in the test. Not sure that there was any real level of legal analysis tested. The test did not require the trainee to give commercial advice. § It seemed to me the exam was attempting to assess skill, knowledge and professional ethics – each of these being the presumed constituents of professional competence. However, while I believe this underlying aim is possible to be examined in this way, I believe strongly that someone who is both competent and professional would go and look up a book or refer to a colleague if they were unsure of how to approach a problem. However, an exam format not only encourages but requires people to grasp at answers and write down what you think might be right so as you have a chance of any marks going – this is what we were always told to do in school “never leave a question blank, you won’t get any marks that way!” However, this approach is the antithesis of competence and professionalism (despite being the way to score well in exams!) Rather than just writing down anything, and competent professionals will always ensure by research and double checking that their approach to a problem is the best one – it is this mindset that should be encouraged = not grasping at answers – that surely is negligent. Thus, some, if not all of the answers given in an exam involve the complete opposite approach from which certainly I would take in real life. § Generally this was not a test of professional competence. It was a test of how good one’s preparation skills are i.e. did you run off enough styles the day before? There were no commercial aspects to this exam. Purely corporate. Either change exam or change title – too much time wasted on research into areas not examined e.g. lease/guarantee/accounts. Test of Professional Competence, Pilot I Report 21.5.02 27 Although I spent around 5 hours preparing (would have been more had I not been so busy at work), I am concerned that preparation will take up too much time. The trainees’ careers will be riding on passing. How many days off will they be allowed to take to research and prepare? Accuracy of instructions should be checked. Should management accounts not have read ‘to….2001’ not to….2000’. Also profit for year, should that not have been for period ending ‘in 2001’? Allow statutes to be taken in. What does question 12 mean in the questionnaire? PS Next time you pilot this, give the trainees a table at which they can complete this questionnaire – that way you will be able to read the answers! Test of Professional Competence, Pilot I Report 21.5.02 28 Company & Commercial Comments on Particular Questions 1. It was easy to know the standard of work expected of me in this assessment. § § § Standard was professions competence but what is that? Especially if whole of Scotland and wide range of firms. It was difficult to know exactly what was expected before hand. Correspondence gave no indication of standard, seems similar to ‘fit and proper’ as in still a woolly concept. 3 This assessment will motivate trainees to do their best work. § § § Only because they need to pass it to qualify. Any assessments on which your result would determine whether you qualify would motivate a trainee. Obvious, if they can’t enter the profession. 5. The assessment I was given tried to cover too many topics. § § When you have to write Minutes, Resolutions etc from scratch there was too much to do. Probably too few rather than too many. 6. The assessment was too easy. § Not so much the assessment itself but with the materials in advance. 7. To do well in this assessment all you really need is a good memory. § § § § All you need is a comprehensive folder of notes and an ability to write quickly. To do well you needed a good set of styles! No memory testing at all required. Good memory is essential for competent solicitors. *Felt very much like an exam in that working within the area probably conveyed very little benefit. A good textbook and some styles would suffice. Practical application is not simply a case of knowing how to write a Minute of Meeting. The last question had very little to do with practical issues. *Refers to questions 7, 8 and 11. 8. This assessment was a good test of our practical knowledge and skills. § § The subject matter was a good test but the way of testing rendered the exercise fairly pointless. True at basic level. 9. The assessment I was given tested my understanding of the law in this area. Test of Professional Competence, Pilot I Report 21.5.02 29 § § Was made to think about the area of law but test was fairly pointless. Exam was very practical which does reflect what is done in practice in traineeship. 10.It would be possible to get through this assessment just by working hard the day before. § § And photocopying styles from firm precedent book. As long as one had some prior experience too. 13. The assessment I was given was a good test of my ability as a trainee solicitor in this area of law. § § § All I did was copy out styles I had in my folder of notes. Very basic – administrative level. Ability as a trainee is not simply being able to follow a style. 14. The aims of this assessment were not clear to me. § I thought I understood before the test but then if was not what I had expected. 15. There was over-emphasis on skills at the expense of knowledge in my assessment. § § § I don’t feel I really used any skills during the assessment. Impossible to separate skills from knowledge The overemphasis was definitely towards knowledge. 17. The sheer volume of work to be done for this assessment means you can’t comprehend it all thoroughly. § § Plenty of time to prepare but during exam too much writing was required. Most ‘work’ is actually the experience from the department. 18.This assessment tested what I learned on my traineeship. § § § The subject matter was relevant to my experience but the test was not. Tested what I learned in relation to this area of the law only! Yes but to a limited extent. 19.I feel confident that I’ve done well in this assessment. § Quod erat demonstrandum. 20.I think an open book assessment is generally a good way of assessing whether a trainee should be allowed into the profession. § § § Yes in practice there is always a textbook to refer to so why not in the exam. More reflective of real life practice. In reality a trainee will have resources available (as well as people to ask) so open book seems the right way to go. Skills at knowing where to look, rather than knowledge of what is there. Test of Professional Competence, Pilot I Report 21.5.02 30 Public Administration General Comments § The public administration paper was too academic based. What was required was pure research in law of fireworks with only a small amount of application. I understood the aims of the TPC and as a result expected a more skills based approach. Asking candidates to perform a variety of tasks would have been more useful. Preparation of a briefing note was not a proper assessment of the vast area of public administration. As a local authority trainee I have only on one occasion dealt with byelaws and have not had exposure to management rules of fireworks etc before this assessment. This was not a test of ability or competence as a trainee solicitor. There was over emphasis on knowledge and not enough emphasis on skills. I expected an assessment with various tasks set. The papers given in advance appeared to indicate this. I obviously read papers wrong – was expecting a brief memo, draft byelaws or management rules, letter to address complaint etc. Useful to have papers a month before because trying to find time to study while working and with other commitments was difficult. I would not think that the content of the paper would be relevant for those not in a local authority, such as trainees in the Scottish Executive. Assessment did not test competence of me as a trainee solicitor or what I learned in my traineeship. § As a Scottish Executive Trainee, I felt that the content of the paper focused too heavily on areas which related to Local Authorities (e.g. making byelaws etc). I have never dealt with such powers in my traineeship. Although part of the question related to powers of the Executive, this was not a particularly large part of the question as a whole. Also, the test was largely knowledge based (asking for the legal background as well as powers of local authorities). Many of the skills employed in my traineeship relate to research and advising clients of powers etc. So, in that regard, I had to use these skills before coming to the test when preparing. However, it was not clear before the test that the question would be so heavily knowledge based. This made it difficult to prepare for. It was also unclear what one was permitted to take into the exam (e.g. would you bring statutes etc in your A4 folder?). Also this questionnaire is confusing as in the second part of the table the answer symbols were reversed! Public Administration Comments on Particular Questions 1. It was easy to know the standard of work expected of me in this assessment. Test of Professional Competence, Pilot I Report 21.5.02 31 § § § 2. I had not dealt in my traineeship with the topics being assessed. § § 3. Not really sure – you will want to do well but I think some trainees will treat it too much like an academic exam i.e. have to know everything about the subject. The assessment I was given tried to cover too many topics. § 7. Councils do varied work. While I had not dealt with this particular topic it was a realistic possibility that I could have been given something similar at work. Ethics point would get raise with manager and they would deal with it. Had a complaint about fireworks two week ago and had to investigate it. This assessment will motivate trainees to do their best work. § 5. Really had no idea how much depth of knowledge or standard would be expected. Not really apparent from info. Sent before exam. Preparing a report is not normally done in a single draft in such a short time frame. Topic too wide for time allowed. This was explained in the notes given to us before we sat the exam. It was very clear what was expected of us. To do well in this assessment all you really need is a good memory. § You don’t need a good memory, but it aims to stimulate an office environment is more practical. 8. This assessment was a good test of our practical knowledge and skills. § Too much emphasis on what law is.(refers to Qs 9 & 11 also) 10. It would be possible to get through this assessment just by working hard the day before. § If can get the resources and organise them well you could do it the day before. 13. The assessment I was given was a good test of my ability as a trainee solicitor in this area of law. § § 16. Was quite good in the way that you would be asked to give advice in the workplace. However, the ethics question was a bit random and not that likely to happen to a trainee. I agree, a lot of work we do is research based to find out the law. This assessment will put trainees under too much pressure. § I don’t feel it will at all, they would be used to doing research like that during the traineeship. 17. The sheer volume of work to be done for this assessment means you can’t comprehend it all thoroughly. § How much work was expected? Old file in office cut research time. Test of Professional Competence, Pilot I Report 21.5.02 32 19. I feel confident that I’ve done well in this assessment. § No idea – was not sure about how much legal knowledge was being assessed or if it was the way you phrased your correspondence. 20. I think an open book assessment is generally a good way of assessing whether a trainee should be allowed into the profession. § § § Absolutely no point in closed book then it would just be the same as an academic exam – not assessing professional competence. Only in combination with other things. You are allowed to look at books and other information in the profession and it is only right that trainees are allowed to do the same. Criminal General Comments § In general exam was fairly good test of expected level of knowledge of trainee in my position. However, preparation in four weeks prior to exam did not seem to tie in to exam in same way as I expected it to. Expected exam to be related far more to specifics of case and how we would run/prepare defence ourselves. Instead it was more concentrated on ethics and scenarios which do not arise on a day-to-day basis and nor particularly related to case itself. § I deliberately did not prepare for this assessment, as I believed the exam was to reflect the knowledge and skills I had acquired during my traineeship. Had I studied for it my answers would probably have been more knowledgeable and academic. Criminal Comments on Particular Questions 1. It was easy to know the standard of work expected of me in this assessment. § Had been advised to be prepared to run a Summary Criminal Trial – but no indication of actual format of exam. Perhaps more info would have helped. Assumed that questions would have concentrated in far more detail on actual circumstances of case i.e. actually preparing pleas etc. More emphasis on how you would have actually prepared defence. § There was not enough advice in advance of what would be expected and pre-empting the exam topics was not possible. This increases the amount of work and it is already an anxious and burdensome hurdle for those Test of Professional Competence, Pilot I Report 21.5.02 33 hoping to qualify. § We had no knowledge of the exam format/content. A ‘sample’ paper would have given a good indication of the type of exam we should prepare for. § I was unsure of the exam format and on which area of criminal law it was to be on i.e. practical issues/court procedure/pre trial preparation. 2. I had not dealt in my traineeship with the topics being assessed. § Had not dealt specifically in traineeship with some issues raised in exam but was still fairly relevant in brining these to my attention. Could have dealt with more issues however that would more commonly arise. § If the assessment was intended to examine my traineeship and if I had only attempted to answer those questions, which I should be expected to answer as a result of my training, then I would definitely fail. I would only have been in a position to answer part of 3 of 10 questions. I don’t feel that I can answer this anonymously. The examiner made an effort to present a balanced paper and aspects of it were fairer than I had anticipated. However, as a trainee for the Crown Office and Procurator Fiscal Service I could not answer questions on:-legal aid -client relations -conflict of interest -defence preparation of a case I attempted six questions because I had done a lot of academic revision prior to the exam – considerably more than the recommended one day. I found the topics examined interesting and challenging and feel that witness difficulties/child witnesses/duties to the court are proper topics for this exam. However, there should have been (and it would not require much redrafting) alternative questions. Sixteen Crown trainees will sit this exam in its first year and they should not be prejudiced. There was reference in one question to a plea of guilty. Whilst a defence trainee can write why they would advise a client to plead guilty or not, the Crown cannot reveal policy decisions on why a plea is/isn’t acceptable. The issue of disclosure and confidentially ought to be addressed. Overall I was challenged and impressed by the topics but feel that the questions must be phrased so that they can be addressed from either perspective. Professional ethics are clearly important but there are neutral topics (duty to court/CPD), which could be examined. § Although had dealt with topic – scenarios had never come up and knowledge did not always extend to scenario – was however able to look up explanation. § Exam questions focused on many points of law, which possibly arise once or twice in a blue moon! Not very real to true practice! 3. This assessment will motivate trainees to do their best work. § § A substantial pay-rise is the only motivation! I think trainees will obviously want to do well in the exam but it is difficult to know the assessments intentions – is it to test your ability to study for an exam or to test the quality of your training. Test of Professional Competence, Pilot I Report 21.5.02 34 11.To do well in this assessment all you really need is a good memory. Exam was too academic/theoretical. Cramming up on Criminal Procedure ‘theory’ is all that was necessary. § I think that if you did not know the answer to the question in many situations you could look up Green Statutes to see if there was anything in Criminal Procedure(s) Act 1995 which could help. § 8. This assessment was a good test of our practical knowledge and skills. § § Exam could have tested more at our actual practical skills used on a dayto-day basis As with question 7 if you don’t have the knowledge you can refer to Greens. Perhaps this is a good thing because in practice lawyers don’t tend to carry around all the relevant sections of legislation for everything. Usually they have knowledge of what to refer to, however, and an understanding of what the law is without the precise subsection of the Act. 9. The assessment I was given tested my understanding of the law in this area. § I felt that parts of the test were irrelevant as they dealt with situations that were rare/unusual and had little relation to day-to-day problems that a trainee would encounter. 10. It would be possible to get through this assessment just by working hard the day before. § A few days studying. 12.We were given enough time to understand the things we had to learn for the assessment. § Thought a 4-week period prior to the test could have provided better preparation than it did. It was unclear what was expected of us during this time in terms of returning work/learning new info? Not having the opportunity to have a day’s study leave from work also had an effect. 13. The assessment I was given was a good test of my ability as a trainee solicitor in this area of law. § Exam simply did not focus on ‘practice’ – too much theory! 14. The aims of this assessment were not clear to me. § 16. § Aims of test not overly clear prior to exam – should perhaps have been more emphasis on this. This assessment will put trainees under too much pressure. Without knowing the format of the test, the level of knowledge expected and the overall importance of the test, it is difficult to feel any pressure (this may be because we know this is a pilot test). Test of Professional Competence, Pilot I Report 21.5.02 35 17 The sheer volume of work to be done for this assessment means you can’t comprehend it all thoroughly. Without knowing the format of the test, the level of knowledge expected and the overall importance of the test, it is difficult to feel any pressure (this may be because we know this is a pilot test). § 18. This assessment tested what I learned on my traineeship. The practical work prior to assessment tested what I had learned more than the scenarios produced in the exam. § Exam questions focused on many points of law which possibly arise once or twice in a blue moon! Not very real to true practice! § 19.I feel confident that I have done well in this assessment. § I was not expecting the test to be this way and therefore had not prepare the way necessary for this test. 20. I think an open book assessment is generally a good way of assessing whether a trainee should be allowed into the profession. § Exam should definitely be open-book. Believe it or not, in practice, solicitors very often need to read books or notes to get the right answers. An open-book exam gives the impression of how it would be in practice dealing with the same issues. § An open book test is the only way to test knowledge/ability in a practical style. Test of Professional Competence, Pilot I Report 21.5.02 36 Appendix 3: Group Interview Reports Group 1: Group Facilitator, Linda Thomson This group was unanimous in its support for the general format of the TPC, ie case study to be issued in advance and preparation allowed. The group felt that this format replicates practice, in that trainees could discuss the case with their trainers, consult books etc. The group made the following constructive general suggestions, regarding aspects of the TPC which could be improved: • • • • • • • • • • A weighting could be assigned to each question, to provide guidance about how much time to spend Instructions about which/how many books trainees may take into the exam should be clarified Statutes should be allowed in addition to a textbook – this would make the TPC even more realistic, because statutes are always to hand in a solicitor’s office. Consideration should be given to allowing a copy of Greens Solicitors Professional Handbook to be taken in to the exam. In relation to the above point, it should be noted that firms may not have enough copies in their libraries for every trainee sitting the TPC In general, the content of the TPC was at an appropriate level for trainees. The inclusion of ethics issues was unrealistic – in practice, these would be dealt with by a partner. However the trainees in the group acknowledged that a trainee should be able to spot an ethics issue, without having to resolve it. Past papers or sample papers would be very useful, because this is an unfamiliar exam format. Most of the trainees had been allowed one day of study leave but had found this to be more time than was required. In some respects, more useful preparation could be done in the office because colleagues and resources were on hand The test was not particularly taxing and it had been quite easy to predict what some of the questions would be The test was more knowledge-based than the trainees had expected, and less skills based Comments relating to each of the three papers are as follows: Company/Commercial • • • The topic was relevant There was too much copying of minutes and filling in of forms, which took up a large part of the three hours allowed for the test. This in turn meant that trainees were rushing to get finished on time. Copying and form filling did not test trainees’ competence. It was suggested that the forms and the minutes should be supplied to the trainees with some parts already completed. The paper would have been more testing if there had been more questions of a “what would you do and how would you do it” nature. Test of Professional Competence, Pilot I Report 21.5.02 37 • • • Being asked to write to a director to set out his duties was unrealistic – this would not happen in practice. The mention of a lease in this paper was considered to be a red herring and somewhat misleading. There was a danger that trainees would go off on a tangent and not focus on the real issues. The trainees were surprised that there were no questions dealing with accounts rules. Criminal Litigation • • • • • • Corresponding with the examiner was considered to be very useful and had been successful The questions could have been answered without reference to the file of materials One trainee had no idea that ethics issues would arise. However those that did arise were pertinent Three hours was more time than was required Some unusual legal points came up in the questions which tested the trainees academically rather than in terms of their skills Two of the questions were very similar Administrative Law • • Three hours was too long – although the trainee who made this comment did acknowledge that she finished the test well before any of her fellow trainees sitting this paper The differences between the experience of a trainee in a Local Authority and a trainee in the Scottish Executive had not been accommodated in this paper Test of Professional Competence, Pilot I Report 21.5.02 38 Group 2: Group Facilitator, Liz Campbell Company/Commercial Law There was no opportunity for creativity – the paper simply said do this and do that. There were inconsistencies in the paper, especially in relation to the allotment of shares and the chronology was unclear. There was differing information about the company secretary in the papers circulated in advance and the papers handed out on the day. There was no research involved. It was easy to predict the type of issues which would come up as the paper was document and procedure based. A different tack could have been taken to test far more such as the ability to express oneself to clients. This only tested my memory. I only spent three hours preparing for the examination and would not say I was under-prepared. The examination needs to be targeted at a higher level. I simply sat and wrote thinking it can’t be that simple. It was so narrow, it needs to be more practical to test the diversity of training. There was no client contact. The last question was daft as it was so wide ranging. It was something one would probably never be asked and would certainly not answer it in the way expected in the examination. There was no clear link to what had been covered in the PCC. It was no different to what we are doing on the Diploma. Filling in the forms was not a test and in one case there was not address for one of the people. Public Administration There was nothing to pick out yourself. It was all based on research and knowledge – not real skills. There are such differences between Local Government and the Scottish Executive. How was this test assessing the traineeship? I was trying to anticipate what might come and had researched by-laws and management rules but the paper did not cover this. I learnt something about an area I knew nothing about before. It needs to be assessing more skills – a variety of tasks to be set even if on a restricted subject. Test of Professional Competence, Pilot I Report 21.5.02 39 It needs to be set on a subject that can be treated equally whether you are training with the Scottish Executive or a Local Authority. Alternatively, there should be two topics. It needs to assess a very practical application. Criminal Law It was easy to pick out what the examiner was trying to test us on. There were no past papers available and this is the only effective means of research. We need more guidance on what might come up. It was very much based on problem solving. It was all academic, there were only two parts I could have answered from my work experience in the Crown Office. It was difficult for me as it was based so much on the defence perspective. I learnt a lot. There were a lot of points that made me think. The only fair thing is to have two separate papers or questions that can be answered by either side. General Points There was confusion about the materials which were allowed into the examination room, in particular in relation to text books and statutes. Queried why statutes were not allowed in addition to textbooks and why Criminal Law candidates were allowed to bring only Green’s Statutes (the copies of which they had were two years out of date) and not the office annotated copy. In some areas, it was difficult to identify one relevant textbook. From the guidance they had obtained through accessing the website, many trainees expected a big emphasis on ethics but this did not happen in reality in the test in any area other than Criminal Law where candidates expressed a view that they had anticipated much more procedure and had been surprised by the level of ethics in the paper. There was a feeling expressed by some of the group that there should be more guidance given with the initial papers as to what might come up on the day of the test. However, others felt that the 4-week period allowed sufficient time in which to research possible angles. There was general concern about the different amounts of time which trainees had been permitted by their firms for preparation. It was considered that the Law Society recommendation should be greater than one day. On the other hand, the Company/Commercial candidates felt that a few hours had been more than sufficient. There was general feeling that the Criminal paper had been the best drafted. One trainee suggested that the Company/Commercial paper should take the form of an oral with an exit interview after about 23 months. There should be two set questions but most of it should be specific to the person and that would give a clearer picture of competence. Test of Professional Competence, Pilot I Report 21.5.02 40 It was also suggested that the Company/Commercial paper should have taken the angle of “Based on this file, as a new solicitor, do you consider there is anything wrong with the way the company was incorporated and how would you fix it?”. Several trainees requested feedback on performance and asked that they be given their marked papers back for them to provide us with further feedback. Test of Professional Competence, Pilot I Report 21.5.02 41 Group 3: Group Facilitator, Paul Maharg Abbreviations: C= Criminal trainees CC = Corporate & Commercial trainees PA = Public Administration trainees We focused on the questions raised in the presentation. PM invited feedback on the first questions in turn; but trainees ranged widely over the issues, and so PM has produced the following summary of the discussion. Experience of the examination PA pointed out that the instructions on what you cd bring in to the examination were too vague, and needed to be redrafted and sharpened. It would be helpful to provide reasons why particular texts (or their absence) were part of the examination. CC trainees agreed - too vague, esp corporate instructions. All trainees felt that the eight areas of practice need to be further defined. Eg the commercial exam was really corporate, and would have serious implications (given the current level of information) for a trainee who had done nothing but commercial work. It was queried whether the exam would work, because every trainee gets different training, and it is thus difficult to cater for all trainees' learning. PM raised the point of generic skills. Is it not the case that there is a generic skill set for the different areas of practice. Trainees pointed out that there is a different atmosphere in the examination room. It is not practice, and thus, if practical competences of the office are the focus of the test, then these are not being tested. Too artificial. PM asked if they had had the set of skills relating to the TPC. They did not. It is clear that there is a considerable amount of communication required to be done in order to alert both firms and trainees to the importance of these assessment outcomes. CC agreed with this in particular. PA case file was felt to be a realistic topic, with file of around 3 pages of A4, but trainees needed more time to produce more than a first draft. The C trainee felt that up to the exam itself, there was a realistic test of skills of the criminal trainee. But exam scenarios were not ones the trainees would be likely to encounter, eg death of complainant. It was felt that the aims of the assessment were unclear. Was it testing office practice, or something else? CC trainees were very definite that there was too low a level of knowledge and skills needed in the test. Basically they thought that much of the activity during the examination consisted of form filling, and little else. They could not see the relevance of this as a test of professional competence. The questions relating to this activity were similarly not at all a test of knowledge or skill. One trainee pointed out that there were a number of issues that arose from the transaction, or could possibly arise; but that there was no space in the Test of Professional Competence, Pilot I Report 21.5.02 42 examination paper to comment on these possible issues. He felt that these issues ought to be brought to the surface more in the examination. Possibly trainees might be asked for their reflections on the development of a case. CC trainees were particularly critical about the presence of the narrative. PM agreed that this tended to reduce the element of practice-based problem-solving, and ought not to have formed part of the examination. C trainee agreed, as did PA. PA and C files were presented much more as if they were real files, and as a result the assessment was more realistic. Having said this, though, the C trainee pointed out that she found she had left the file at home for the examination. But she did not need it in the examination for the type of questions that were asked. Professional ethics issues CC trainees could not see any. PA trainee considered that as far as the professional ethics element was concerned, it was brought in far too late -should have been flagged earlier, and then developed in the later materials. But nothing complicated. It would still need to be what a trainee might encounter, so that the trainee could present a 'history', as it were, of the professional ethical issues as these arose in the development of the case. PA trainee pointed out that the division of Executive and local authority could cause problems regarding subject matter and trainee experience. CC trainees thought that skills assessment was at a very low level in the corporate activity. Legal research PM asked about the quality of legal research done by trainees. All treated the case as if they were dealing with a real case. CC trainees thought that having one day off would probably lead to trainees treating it as an academic activity, rather than a real case that had real limitations to it in terms of the time that trainees could give to it in the office, alongside all the other cases on the go at any particular time. PA trainee consulted statutes, in-depth, plus discussions with colleagues. He thought it was a good scenario, realistic, with interesting issues arising from it. Felt it could have been better developed before the examination. Above all, it was practical. Criminal trainee basically considered charges, defences, etc, and did little research. Discussed with colleagues. Suggested adding to this a drafting activity, such as drafting joint minute of agreement Commercial trainees admitted theirs was a practical activity, but it was too much of a proforma activity. They did consult others, but wanted to see another stage where they wd get information on the original file. In other words, they wanted to see more office realism, and more developmental complexity. They also saw no reason for the ' red herrings'. Cut out narrative. Improvements: Information to trainees needs to be much much better. Test of Professional Competence, Pilot I Report 21.5.02 43 Some trainees especially CC were definitely of the opinion that there was no need for a TPC, or if there needed to be an assessment, this was definitely not it. Questions in the CC materials didn’t relate to the materials -- ethical points, but difficult to integrate these into the scenarios There was general discussion about the firms and their place in the traineeship structure. PM described the trainee logs, but this did nothing to assuage their concern about the different types and levels of training available to trainees, and whether this could be reasonably assessed by examination. Test of Professional Competence, Pilot I Report 21.5.02