Knowledge Management Systems Assessment: A Conceptual Framework And A Methodological Proposal1 Mario L Vasquez2 Carlos F. Gomez3 Ana M.

Lopez4 Jorge Robledo 5 Abstract Modern management has discovered in knowledge a never ending source of wealth an d competitive advantage for organisations. Nevertheless, for this to be as effecti ve as it promises, a sound conceptual framework and good methodologies for Knowledge Management (KM) are needed. This paper addresses these two critical aspects of KM, focusing on the a ssessment function of a KM System (KMS). The paper starts describing the conceptual approach adopted, and then continues presenting the methodology proposed, aiming at obtaining a comprehensive profile of a KMS to assess the maturity level of the system. A clear differentiation of system manag ement and operational functions is introduced, to produce a set of indicators that clearly identifies system s critical resources and functions. These indicators are regarded as key informati on to start an improvement process of the KMS. The methodological proposal is based on a conceptual model which articulates thr ee theoretical contributions: the feedbacked management cycle inspired in organisat ional 1 Paper submitted for IAMOT 2000, Conference on Technology Management, Track 15 Knowledge Management 2 Mario L Vasquez ([email protected]) is an associate consultant of Xeon Techno logy Corporation. Systems Engineer from EAFIT University and MSc in Technology Manage ment from UPB University (Medellin, Colombia). He has worked as an IT Business Develo per and Consultant for eight years and KM researcher for the last two. 3 Carlos F. Gomez ([email protected]. ) is an Organisational Planning Specialist from Empresas Publicas de Medellin. Mechanical Engineer and MSc in Technology Managem ent from UPB University. 4 Ana M. Lopez ([email protected]) is an Information Technology Consulta nt from Unisys de Colombia. Systems Engineer from EAFIT University and MSc in Technology Management from UPB University. 5 Dr. Jorge Robledo ([email protected]) is the Director of the Technolog

y Management and Policy Research Group, UPB University. He has worked before for Colciencias (the Colombian Institute for the Development of Science and Technology) and for sever al electrical utilities as consulting engineer. He holds a professional diploma in Mechanical Engineering (National University of Colombia), a MSc in Electrical Generation Systems (Valle University at Cali, Colombia) and Ph.D. in Science and Technology Policy Studies (University o f Sussex at Brighton, UK).

cybernetics; the classification of technological resources suggested by the UN E SCAP; and the Capability Maturity Model (CMM) developed by the Software Engineering Institute at CarnegieMellon University. The second part of the paper presents and discusses the results of a field test of the methodology, carried out in a planning unit of a local energy business. The resu lts, although preliminary, show that the proposed methodology is capable of producing a compre hensive profile and a helpful assessment of the KMS, identifying critical aspects of its resources and KM functions, and thus providing useful insights on where and how to proceed with a KM improvement plan. Keywords Knowledge Management, Knowledge Management Systems, Measurement, Organisational Measurement, Knowledge Management Measurement, Knowledge Management Systems Assessment Introduction ...knowledge is now as important if not more important as factor of production than labour and capital. The reduced cost and increasing efficiency in information pr ocessing and telecommunications are speeding the creation and dissemination of knowledge. And we may be in the midst of a major paradigm shift in the way we process and disseminate inf ormation a shift to an integrated global market for trade, finance, and knowledge. (World B ank, 1999, Part One) The authors, as many other people in the world, believe (as the World Bank also does) that knowledge is the resource that will drive the evolution of the 21st century world of business. Nevertheless, in order to realise the potential of this resource, every organisa tion must address a conscious and deliberated effort in managing their knowledge effectively. This paper proposes two key elements to make this happen: a conceptual framework and a methodology focused on the assessment function of a KMS. A proposal of a Systemic Model When approaching a brand new subject of study as KM one faces the potential prob lem of dealing with a subject not well defined. Thus, for a clearer understanding it is necessary to start defining some basic terms as used in this paper, seeking to create a commo n language to lay foundations for the proposed model. The first term, of course, is knowledge, which is defined here as the element th

at adds value to the information in a process which converts it into decisions. This def inition embeds the need for a clear differentiation between information and knowledge that has been addressed by researchers around the world, as Thomas Davenport (1997) who suggests the follow ing: 2

Data: Simple representations of the state of the world, easy to structure, trans fer and capture with machines and frequently quantifiable. Information: Data embedded with relevance and propose. Requires a unit for analy sis. It is necessary to achieve consensus about its meaning and requires human intervention . Knowledge: Information processed by human mind including reflections, synthesis and contexts. With characteristics such as: difficult to structure, difficult to capture using machines, frequently tacit, difficult to transfer. From a broader perspective, it has to be considered that knowledge is evolving a s a strategic resource for post-industrial organisations. The OECD considers that we are at the beginning of a new economy, the knowledge based economy , based over intangible res ources such as technology, information and learning. Knowledge is now recognised as the driver of productivity and economic growth, l eading to a new focus on the role of information, technology and learning in economic p erformance. The term knowledge-based economy stems from this fuller recognition of the place o f knowledge and technology in modern OECD economies. (OECD, 1996, p. 3) Searching for an effective response to the importance of knowledge within organi sations, the effective management of this resource appears as a key responsibility for succes s. As Wiig suggests: KM [Knowledge Management] in its broadest sense, is a conceptual framework that encompasses all activities and perspectives required to gain an overview of, dea l with, and benefit from the corporation s knowledge assets and their conditions. It pinpoints and prioritises those knowledge areas that require management attention. It identifies the salie nt alternatives and suggests methods for managing them, and conducts activities required to achi eve desired results. (Wiig, 1993, p. 18) Based on this arguments, the authors propose a KM model, which fully integrates these concepts and theoretical propositions to structure a coherent framework for the methodological development that follows. This model, as illustrated in Figure Error! Unknown sw itch argument., is based on the management cycle (Planing, Direction, Execution and C ontrol). Knowledge Strategies Knowledge Products Results policies & Plans resources Execution Reports& Indicators Data & information

Figure Error! Unknown switch argument.. KM systemic model. The Execution function is made up of Knowledge Activities, which include, from a systemic perspective, all the activities performed toward an effective incorpora tion of knowledge Planning Control Direction (Knowledge activities) Process 3

within a specific organisational Knowledge Sensitive Process, adding value to th e process from an external customer perspective. From a theoretical view, a taxonomy of these a ctivities may be defined in the following manner, as Wiig (1993) suggests: Gather, Organize, Prep are, Transfer and Use of Knowledge. The input for these activities are Knowledge Resources, and their outputs are a set of Products which are the added value of the Knowledge Activities. The Knowledge Re sources are provided by the Direction function of the KMS. The Knowledge Activities act in t he Process where knowledge produces end Results from an organisational perspective. The out put feedsback the Planning and Direction functions via the Control function, providing re levant information such as reports and indicators for analysis, decision taking and sup ervision. Methodological approach for assessing a KMS Current organisational management theories position measurement as a key element to achieve effectiveness in resource management; it settles the bases for planing a nd direction through control. Following these statements, the authors suggest a methodologica l proposal based on a gradual approach to the object under study using a lineal phase strat egy. The proposed phases are now explained. Process recognition The objective of this stage is to identify core processes within the organisatio n under study. Questions such as: which are and how are structured the core processes wi thin the organisation from a business, political ad social perspective?, have to be answe red during the first phase of the assessment procedure. This stage produces a document with the core processes and their main internal a nd external interactions. Knowledge Sensitive Processes identification The objective of this stage is to identify the processes where knowledge may add greater value. This proved to be a difficult task, whereas the literature does not repor t any research on this topic. However, based upon the conception of knowledge as an element that a dds value to information converting it into quality decisions, a link may be established betw een the knowledge sensitive processes and the factors guaranteeing decision making quali ty, an issue

which has been studied by several analysts. Therefore, as a methodological proposal for Knowledge Sensitive Processes identification, the authors worked out a procedure based on the six links of the decision quality chain discussed by Matheson and Matheson (1998, p. 24), building a sensitivity m atrix to grade each organisational process for every decision-quality link and, afterwards, gen erating a weighted indicator to reflect sensitiveness to knowledge. These links are: (1) A ppropriate frame and perspective for decisions, (2) Creative, doable alternatives, (3) Meaningful , reliable 4

information, (4) Clear values and trade-offs, (5) Logically correct reasoning an d (6) Commitment to action. Comprehensive Profile of a KMS At this point we reach the core of this paper, that is, the methodological tool that will assess the maturity level of a KMS within an organisation. To achieve this goal, the authors build a conceptual model articulating three theoretical contributions in a joint model which supports the profile assessment tool. But firstly we will introduce the basic models to d erive into the comprehensive profile assessment tool. Feedbacked management cycle This is a fundamental conceptual framework which defines the structure of the KM S, as mentioned above. Based on organisational cybernetics, a KM may be described as a system made up of four basic functions: Planing, Direction, Execution and Control. The first three are linked in a row delivering the knowledge that will rule organisation s operational functions. The Control function provides feedback to the whole system. The objective is to impr ove key variables (innovation, quality, diversity of alternatives, opportunity and cost of studies - in the case studied) of Knowledge Sensitive Processes to finally contribute to achieve selected end Results for the organisation (e.g. present net value of the recommended investme nts portfolio for the case of an energy business). Technological resources The UN ESCAP proposes a resource based approach to model an organisation, with a

classification of technological resources which we have found of particular rele vance to the methodology, as follows: Humanware: Related with organisation s human capital and the efficient use of work ers knowledge in core business areas, with a special focus on Knowledge Workers6. Infoware: Associated with explicit organisational knowledge, it includes concept ual frameworks, methods, tools and decision criteria used within processes. Technoware: Related with the information technology required to achieve KM management objectives. This is the technology that leverages the process. Orgware: Associated with organisational structures, practices and definitions fo r managing the other three resources.

The Capability Maturity Model This model, from the Software Engineering Institute at Carnegie-Mellon Universit y, was originally developed in 1991 as a proposal to effectively manage software produc tion processes, but later it was extended to different type of organisations and organisational units. 6 In this paper, the term Knowledge Worker refers to a person that develops a know ledge intense activity within the process being studied; it is different from Knowledge Manager , which applies to the person in charge of planning, directing and controlling the KMS. 5

According to this model, organisational processes evolve through five phases whi ch describe the maturity level of the organisation, as follows (Paulk et al, 1993): Initial. The process is characterized as ad hoc, few routines are defined, and s uccess depends on individual effort and heroics. Repeatable. Associated with the definition of politics for process management an d the presence of some proceedings to restore them. Defined. The process for both management and operational activities is documente d, standardized, and integrated. Managed. Detailed measures of the process and quality are collected. Both proces s and products are quantitatively understood and controlled. Optimizing. Continuous process improvement is enabled by quantitative feedback f rom the process. Tool for assessing the Comprehensive Profile of a KMS To articulate all these concepts in an operational tool and apply it in a field test, the authors designed a questionnaire based on four tables, one for each technologica l resource, where each row represents an element of the Management Cycle and each of these h as five choice answers which represent the adapted levels of the CMM. In that way each w orksheet assesses the maturity level of one specific technological resource from the orga nisational management perspective7. The first requirement of this methodology is the selection of a Knowledge Manage r to lead the whole process. This official has to be well trained both in the assessm ent methodology and in KM. He or she will be in charge of a team of Knowledge Workers from the p rocess selected to perform the assessment. After selecting the team, it is fundamental to address effective training sessio ns on the methodology and the theoretical model. The extent of this training will be defin ed by the Knowledge Manager based on his/her preliminary assessment of the KMS maturity le vel. This stage is crucial for the assessment success. If the Knowledge Manager is dealing with a low maturity process he/she will be advocated to select a larger team of Knowledge W orkers and reinforce the training sessions, but, on the other hand, if he/she is dealing wi th an organisation in an intermediate maturity level, the training will become a light workshop and co

ntinuous coaching. Only after a successful training, the Knowledge Manager may proceed to apply the questionnaires to the team. In the questionnaires, the Knowledge Worker will not only be asked to identify the current status but also the perceived development trend of the m aturity level of each resource referred to every management function. 7 This paper does not cover the assessment of the operational side of the KMS. F or further information refer to Gomez, Lopez y Vasquez (1999). 6

To summarise all the results, an automated tool may be applied to create two sum mary tables: the current KMS profile and the trend profile. From these tables, the to ol generates two radial graphs with 16 vertexes (each vertex represents a resource graded for one specific function of the management cycle). These diagrams provides a glance overview of the curre nt state of the KMS (expressed by Knowledge Workers) and the evolutionary trend. For further analysis, the authors extracted a couple of indicators from the summ ary tables that will be discussed in more detail in the section devoted to the field test. The n ext step after the assessment (not within the scope of this paper) is the definition of actions tow ard improvement of the KMS. Application of the model and methodological proposal to a planning unit within a local energy utility. After this theoretical presentation, it is necessary to find a mean of contrasti ng the methodological proposal. This process will contribute to validate the potential application of the methodology. The authors selected the perspective of a planning unit of a typical electricity business in a developing economy (further called CUX "Consulting Unit X"). This kind of depa rtment is responsible for making long and mid term decisions, where there is great uncerta inty and unreversibility, with maximum impact on the future development of the organisati on. Under these circumstances an effective KM is mandatory to reduce uncertainty and impro ve quality in the decision making process. The exercise was planned as a laboratory test of the methodology and it was dire cted by the authors playing the role of Knowledge Managers; the selected Knowledge Worke rs were energy generation planning specialists with several years of experience that wer e self interested in KM issues and voluntarily participated in the test. From a potential universe of 40 experts, an eight planning specialist team was chosen to answer the questionnaire with conti nuous coaching from the Knowledge Manager. The process of gathering answers lasted for 15 days. The following sections highlights major findings in this process. Process recognition The process recognition stage shows four core processes within the planning unit as follows:

Strategic Planning: It includes prospective planning and technological intellige nce; it defines strategic areas and mid term objectives (3 to 5 years) but also short te rm ones. New Business Planing: It is in charge of gathering information about and previou sly selecting new markets investment opportunities, as well as defining new business lines related with power generation and their marketing strategies. Studies Plan: Pre-selecting and studying new energy generation projects. It incl udes inspection, pre-feasibility and feasibility studies, and involves compiling the catalogue of studies. 7

Investment options analysis: It is responsible for standardizing projects with a pproved feasibility and other investment options, seeking to elaborate the optimal portf olio of investment and desinvestment. Support processes: It backs all the chain processes through a sort of shared org anisational resources such as: human development, finance, information technology and legal support. Knowledge Sensitive Processes identification With the input of the Process Recognition stage, the Knowledge Manager proposes to a couple of experts the exercise of grading the six decision-quality conditions fo r each process in an ascendant scale from 1 to 5. An extra factor was included to weight every con dition according to its intrinsic sensitivity to KM. Table summarises the findings for this particular case. Table. Knowledge Sensitive Processes identification. Condition Weight Process Strategic Planning New Business Planning Studies plan Investment Opt. Analysis Appropriate frame 20% 5 4 4 3 Creative, doable alternatives 20% 4 4 4 1 Meaningful, reliable info. 25% 3 5 5 4 Clear values and tradeoffs 15% 4 4 4 5 Logically correct reasoning 15% 3 4 4 5 Commitment to action 5% 2 3 3 1 TOTAL 100% 3.7 4.2 4.2 3.35 Based on this methodology, the processes most sensitive to knowledge are New Bus iness Planning and Studies Planning. Each of these has to be on direct focus of the KMS; also, each demands particula r consideration due to its diverse characteristics. As an example, we will explore the KMS focused on the process of Studies Plan , where the final product is a highly competitive, s

tandardized, qualified and diverse portfolio of feasibility studies. This process requires nu merous knowledge areas to obtain this portfolio joined with the following sub-processes: (1) Oppo rtunity studies, (2) Sector and market studies, (3) Pre-feasibility Studies and (4) Internal feasibil ity studies and also acquisition of external studies. These sub-processes are linked in a sequential manner building an event chain. The amount of information is overwhelming along this chain and it i s mandatory adding value to it toward a better final portfolio. This and other findings show how this process will certainly improve through an effective management of the KMS. 8

Comprehensive Profile of a KMS At this stage, the eight Knowledge Workers answered the assessment questionnaire . The Knowledge Manager collected and summarised the data for analysis using a compute rised spreadsheet specifically designed for this exercise. The final results are shown in Table, Table and Figure Error! Unknown switch argument. as follows. Table. Current Profile Resources Management Functions level Critical Resource LevelPlanning Direction Executio n Control H. Human Capital Management. Humanware 1.6 1.6 1.4 1.3 1.3 S. Explicit Knowledge Mgmnt. Infoware 1.9 1.3 1.3 1.8 1.3 T. Information Tech. Mngmnt. Technoware 2.0 1.9 2.5 1.6 1.6 O. Organisational Knowl. Mngmnt. Orgware 1.3 1.5 1.5 1.1 1.1 System s average level 1.3 Table. Natural Trend Profile Resource Management Functions level Critical Resource LevelPlanning Direction Executio n Control H. Human Capital Management. Humanware 3.1 2.9 3.1 2.9 2.9 S. Explicit Knowledge Mgmnt. Infoware 3.0 2.6 3.0 2.9 2.6 T. Information Tech. Mngmnt. Technoware 3.1 3.0 2.9 2.8 2.8 O. Organisational Knowl. Mngmnt. Orgware 2.9 2.6 2.8 2.3 2.3 9

Hp 0 1 2 3 4 5 Hd He Hc Sd Se ScTd Te Tc Op Od Oe Oc Current Sp Trend Tp System s average level 2.6 Figure Error! Unknown switch argument.. Comprehensive Profile of the KMS The tool also provides statistical dispersion indicators (using the same table f ormat) which, when reviewed, revealed a great dispersion among experts answers. It is a lso meaningful that the assessment of the current status showed major variation than the expect ed trend assessment. This fact itself represents an important indicator which may be cons idered from two points of view: The assessment method uses a complex tool which requires a depth understanding o f the basic model. It is reasonable to think that this is a major cause of dispersion due to the incipient maturity level of the organisation under study. The perception of the reality under study is not homogeneous. This issue has to be considered by the Knowledge Manager as a potential variation generator. It is de sirable to have a group of experts as uniform as possible, or, even better, to develop an i terative exercise where some accepted degree of consensus can be reached. Even these preliminary conclusions importantly restrain further analysis, underm ining the confidence in the information gathered; they highlight the importance of further

development of the training strategy for the assessment process and also a better definition of strategies for Knowledge Workers selection and interaction. Critical issues of the KMS Analysis from the Profile obtained

The proposal for analysis is based on a group of indicators derived from the tab les already shown; they are: KMS Critic Maturity Level (SCML), Resource Critic Matur ity Level (RCML), KMS Average Maturity Level (SAML), KMS Critical Aspects and Development Objectives. 10

As an overall definition for the analysis, it is important to reinforce that the Knowledge Manager in his/her role of assessment leader has the authority to define the rul es for indicators analysis. Critical Aspects of the KMS: As shown in Table and Table the Orgware control function (1.1; 2.3)8 is in a critical state but with a natural improvement trend. The reader may also find 13 other functions in the same criti c level that deserves further analysis. It is important to call reader s attention to the gap b etween current and expected maturity levels for some functions. This may also be caused by the two sources of uncertainty expressed before. RCML: Based on the chain rule (a chain is as strong as the weakest link) and considering that all the functions in the management cycle are linked in a feedb acked loop, the RCML is defined as the minimum level of each resource among the functions. For t he CUX, the results show that the lowest maturity resource is Orgware, followed by Humanware and Infoware. SCML: Again based on the chain rule, the SCML is defined as the minimum maturity level among functions. For the CUX this indicator sets a level 1 (Initial level) and shows a natural development trend to level 2 (Repeatable level). The values 1.1 and 2.3 are rounded to 1 and 2 by the Knowledge Manager. SAML: Defined as the arithmetic average among RCML. For the CUX as shown in Table it is graded 1.3, which compared with the average grade 1 (fraction elimin ated by the Knowledge Manager) in the SCML does not show any unbalance development sympt oms. In Table it can be seen that the trend shows an expected growth to level 2.6, which falls in the expected natural development to a maturity level 2, showing a balanced evolu tion of the system. Development Objectives: Based on the previous analysis, the Knowledge Manager ha s now enough resources (based on quantifiable and certain information) to set up d evelopment objectives to address identified organisational weaknesses and define future inv estment. Analysing the set of indicators extracted from CUX, the first priority is the Co

ntrol function applied to Orgware. Other weak functions are Humanware Control, Infoware Directi on, Infoware Execution and Orgware Planning. This overview presentation of the field test results gives clear signs of how th is tool may help a Knowledge Manager assess the maturity level of an organisation s KMS and ad d important value to its management and improvement planning. 8 The values in parenthesis represents: (current maturity level, trend maturity level) 11

Final remarks There are strong reasons to believe that a systematic, conscious and deliberated application of KM within organisations through sound and proved methodologies is a crucial factor to develop a sustainable competitive advantage in the new knowledge-based economy. However, better and coherent conceptual frameworks and useful methodologies are still needed to advance in this direction. This paper described a research effort carried out to make some contributions to this difficult task. From a conceptual perspective, the aim was to develop a working definition and structure of a KMS, based on which we could propose an as sessing method which help manage the whole system. The conceptual framework and the asse ssment methodology proposed in this paper was preliminary tested in a local electrical utility, proving its potential for .either planning the KMS installation or improving an existing one . However, there are still a long way to go and much research is needed in the development side o f the assessment methodology, not to mention the conceptual difficulties which have to be overcom e. On the other hand, an ulterior analysis of the results also suggests that the co nceptual framework and the assessment methodology might be applied to diverse organisations, regard less their economic sector, size and geographical situation, offering a tool to begin to cl ose the comparability gap between organisations. 12

References Davenport, T. (1997): Information Ecology, Oxford University Press, Oxford, 255 p. ESCAP (1988): A framework of technology for development, UN Economic and Social Commission for Asia and the Pacific, Bangalore, India. Gomez, C, Lopez, A y Vasquez, M (1999). Propuesta Metodolgica para el Control y l a Medicin en la Gestin del Conocimiento. Maestra en Gestin Tecnolgica, UPB, 164 p. Matheson, D. and Matheson, J. (1998): The Smart Organization, Creating Value thr ough Strategic R&D, Harvard Business School Press, Boston, 292 p. OECD (1996): The Knowledge Based Economy, Paris, STI, 57 p. Paulk, M. et al (1993): The Capability Maturity Model for Software, v. 1.1., Sof tware Engineering Institute, Carnegie Mellon University, 26 p. Wiig, K.M. (1993): Knowledge Management Foundations, Schema Press, Arlington (Te xas), 474 p. World Bank (1999): World Development Report 1998, [www document], URL: http://www.worldbank.org/html/fpd/technet/wdr98/partone.htm 13