ࡱ> PRMNO'` bjbjLULU :.?.?[%J "    $& Pd& ^`"        $0h q" ,@   X J Q }XDu(]l<hfhPQh i.dO[Nd& D,j8P/-& j8P& & &  MAKING EVALUATIONS OF CAPACITY DEVELOPMENT PROGRAMS MORE INFLUENTIAL. F Kotvojs, 1; R Hurworth2 1Kurrajong Hill, 701 Dignams Creek, Narooma, 2546, Australia (fiona.kotvojs@bigpond.com) 2Centre for Program Evaluation, The University of Melbourne, Australia (r.hurworth@unimelb.edu.au) Abstract Capacity development is core to the activities supported by most development assistance agencies. However, the development of capacity is rarely evaluated. Where it is evaluated, the quality of evaluations is usually poor and shows that capacity development has been largely unsuccessful. Also, when evaluations have been undertaken, it is difficult to identify the influence these have made on program implementation or capacity development policy. Furthermore, evaluations of capacity development programs have rarely met the utility standard. In particular, they seldom provide the types of information that intended users need. Patton (2002) suggests that for evaluations to be utilised, they need to answer questions that are of interest to intended users. Consequently, this paper attempts to identify the types of questions that should be posed by intended users of evaluations of donor-funded capacity development programs. Findings presented in this paper result from an analysis of 49 interviews held with a range of intended user groups. These groups include: donors (those responsible for the program, senior managers and internal evaluation specialists); those in the partner agency (counterparts and managers); those working directly on the program (managers and Advisers); and evaluators. It was found that there was one particular question each user group wanted evaluations of capacity development to answer. Beyond this, the broad questions user groups wanted answered and the focus of these varied. Finally, there appeared to be little relationship between questions users wanted answered and those generally used for evaluations which are based on the criteria established by the Development Assistance Committee (DAC). Introduction Over the last ten to fifteen years, the focus of development assistance has moved to the provision of support for what is known as capacity development. This move is due to both individual agencies recognising the dependency of sustainability on capacity development and the broader articulation of the importance of capacity development by the United Nations. As a result, capacity development is now core to the activities supported by most development assistance agencies. For instance, it is estimated that at least 25% of donor funds are committed to capacity development (Whyte, 2004; Watson, 2006; Loveridge 2007). In some cases, this is even higher; for example, 50% of World Bank funds for Africa are used to support capacity development (World Bank, 2005). However, capacity and the development of capacity are rarely evaluated. Indeed, the World Bank (World Bank, 2005, xiv) notes that most of its capacity development activities are not routinely tracked, monitored or evaluated. In many cases, what is measured is at an output level rather than at an outcome level, and primarily serves an accountability function (Whyte, 2004; Carman, 2007; Taut, 2007). Where capacity development had been evaluated, the quality was often found to be poor (Forss and Carlsson, 1997; Picciotto, 2003; Bollen et. al., 2005; Watson, 2006). From the limited number of evaluations undertaken it has been shown that capacity development has been largely unsuccessful. For example, achievements were below expectations across Africa (World Bank, 2005) and only 30 - 40% of capacity development effort by the World Bank was found to have been effective (World Bank, 1998). The reasons given for this are many and include issues with program design, environmental factors, and management (Watson, 2006). In addition, after a review of the literature, it has been impossible to identify examples of how results from capacity development evaluations have been used. It has also become evident that these results have rarely been used by other activities for which the findings could be relevant. The reasons for this situation have included: lack of ownership of the results by those who could use them; lack of time for the users to read relevant reports; evaluations undertaken too late for the results to be of use; and failure to produce useful information. Furthermore, current practice usually fails to meet the utility standards in the Standards for Program Evaluation set by the Joint Committee on Standards for Educational Evaluation. It was also noted that; one of the biggest problems we have is that we are not producing the sort of information that is useful [T10]. Thus, current practice does not reflect the well-documented need for evaluations to address such issues if the findings are to be used (see Cousins and Leithwood, 1986). Therefore, in order to encourage the increased use of evaluation findings associated with capacity development initiatives, this paper attempts to identify what information intended users would find useful. Definitions of Capacity and Capacity Development In international development, capacity development has evolved from a range of approaches to the provision of development assistance in developing countries. However, no universal definitions of capacity or capacity development have been established. As Morgan (2006) stated; Capacity can be everything or nothing, when it is claimed to be everything, it adds up to nothing". This sentiment has been echoed by others over a considerable period of time (Lusthaus, 1999). Nevertheless, for this paper, the widely accepted Organisation for Economic Cooperation and Developments definition (2006) of capacity as; the ability of people, organizations and society as a whole to manage their affairs successfully will be used. The definition of capacity development this paper uses is that of the Australian Agency for International Developments (AusAID) (2004): capacity development is . . . the process of developing competencies and capabilities in individuals, groups, organisations, sectors or countries which will lead to sustained and self generating performance improvement". Types of Evaluation Questions Required for Development Assistance Programs In 1991, the Development Assistance Committee (DAC) established criteria for evaluation of development assistance (DAC, 1991). It was agreed at that time that there were two key purposes for evaluation of development assistance: To improve future aid policy, programmes and projects through feedback of lessons learned; To provide a basis for accountability, including the provision of information for the public. Subsequently it was determined that four key aspects should be addressed in relation to development assistance programs. These pertain to: Relevance: The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor. Effectiveness: The extent to which an aid activity attains its objectives. Efficiency: Whether the aid used resources as perspicaciously as possible in order to achieve the desired results. It is a measure of the cost of the outputs in relation to the inputs. Sustainability: Whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Such questions are used frequently as the basis for developing a scope of service for evaluations of capacity development. In addition, from experience, accountability is often included explicitly in the scope of services to capture the second DAC evaluation purpose outlined above. Methodology This research concerns the first stage of a broader research project being undertaken as part of a Doctorate. It specifically aims to identify questions that intended users would want any evaluation of capacity development to answer. The information gained will be used to determine the usefulness of the Capacity Development Evaluation Framework (CDE Framework) in evaluation of capacity development initiatives (Kotvojs, 2009). Firstly, those expected to use the results from this research were identified. These groups included donors (those individuals responsible for the program, senior managers and internal evaluation specialists); those in the partner agency (counterparts and managers); those working directly in the program (management, Advisers/consultants); and other evaluation specialists. After discussion, AusAID identified the staff in Canberra and at two AusAID offices in developing countries (posts) who would be the most appropriate to interview i.e. people involved in capacity development activities and interested in evaluation. In the Solomon Islands, counterpart agency managers for the programs that are being used as case studies in the broader research were invited to participate. In Indonesia, this was not possible as the program is already experiencing difficulty in meeting senior managers for evaluations. As a result, these data do not include any information from Indonesian counterpart agencies. Then the Managing Contractor, Program Director, Team Leader and Advisers on each of the case study programs were also interviewed. In addition, people who were to be part of one of the intended user groups were contacted between November 2010 and June 2011. They were also invited to participate. This resulted in three people from NZAID and three from international Non-Government Organisations (INGO being included. As a result, a total of 51 people gave consent to be interviewed. The interviewer used a semi-structured interview. Initially material was digitally recorded; however this made a number of interviewees uncomfortable so subsequent interviews were documented by hand. Copies of transcripts of interviews were supplied to each interviewee to check for rigour. Subsequently, two participants working in Solomon Islands (one AusAID and one Adviser) withdrew approval to use their data and so this data was not included in the ensuing analysis. It should be noted that a number of interviewees fell into more than one user group category and thus their material has been analysed in each (Table 1). Table  SEQ Table \* ARABIC 1. Number of People Interviewed From Each Intended User Group. AusAID (Canberra)AusAID (Indonesia)AusAID (Solomon Islands)Managing ContractorProgram DirectorTeam LeaderAdviserM&E rolePartner AgencyInternational Non Government Organisation (INGO)NZAIDTotal79367671143349 Each person interviewed has been given an identifier from [T1] to [T49]. This is used to reference all quotes in this paper. The identifier is designed to maintain confidentiality of individual while providing a clear audit trail if needed. Data were coded and analysed specifically to determine what key questions each intended user group would like answered in evaluation of capacity development initiatives. In addition, data were analysed regarding each user groups: definition of capacity development; awareness of existing models used for evaluation of capacity development; experience of evaluation of capacity development; and criteria for a framework for evaluating capacity development. A summary of the results of the analysis was promised to all participants. Perception of Capacity Development Evaluation Practice All of those interviewed who were able to comment on the quality of evaluations of capacity development considered that the standard was poor. This was often attributed either to a poor understanding of capacity development by all involved in the program or to issues with the original project design. Furthermore, for many, it was not clear if capacity development was a process or a product. This was illustrated by comments such as If capacity development is a means, you monitor capacity development as a process; if it is an end, you need to start looking for signs that the capacity is different and that it is sustainable. So this is really still a big dilemma within AusAID [T28]. This confusion has been present for a long time (Brown, 2001). In addition, many were unclear as to how capacity development differed from the rest of development because (capacity development) is all we do in development. It is what we do, so it isnt evaluated separately [T18]. Others also noted a lack of understanding of capacity development strategies [T6, T7, T8, T13, T16, T18, T19, T43, T46, T47, and T48]. It was suggested that these factors meant that capacity development often failed to be considered explicitly in evaluations of AusAID-funded activities. Indeed, from interviews carried out, it appears that NZAID dont intend to measure capacity development as a discrete item [T47]. And, as indicated above, a number of those interviewed from all user groups noted that the quality of program designs was a major contributor to the poor quality of capacity development evaluations. They suggested that many designs did not address monitoring or evaluation of capacity development adequately [T18]. In other cases, those interviewed noted that program outcomes were not articulated clearly or were unrealistic [T12, T13, T19]. This made it extremely difficult to implement quality evaluations. There were many weaknesses identified with current practice in the evaluation of capacity development. One of the most common was thought to be the lack of use of evaluation findings3. This was mainly identified by AusAID staff. The reasons for this reflected a lack of application of basic good evaluation practice. One key issue was that the purpose of the evaluation was not clear from the outset. As one AusAID evaluation expert stated: I cant really say what the purpose of the evaluations of capacity development have been [T18]. This was probably a consequence of dealing with multiple clients [T10, T26] and a lack of effective evaluation planning [T18, T31]. For example, a review of evaluations contracted by AusAID in 2009 in Indonesia revealed that only about a third had developed an evaluation plan [T18]. Without a clear purpose (confirmed through the development of a program logic), it is unlikely that an evaluation can answer the questions in which the client is interested and hence findings from the evaluation be used. So, without planning an evaluation effectively an unsatisfactory evaluation in practice is likely to be the outcome (Owen and Rogers, 1999, p73). The Questions Intended Users Want Answered There was only one question that most of those interviewed (and most of each of the user groups) wanted answered. This was What changes have happened as a result of the support? Another two questions were of interest to almost half of those interviewed: How have these changes occurred? and What lessons have been learnt? Finally, whether the objective had been achieved, was identified by a smaller number as something needing investigation. Associated with each of these, there were significant variations of areas of interest for each user group (Table 2). Table  SEQ Table \* ARABIC 2. Questions of Primary Interest to Different User Groups. User groupDonorPartnerManagersAdvisersResponsible for M&E What were the changes?XXXXXHow did the change occur?XXWhat lessons were learnt?XXHas the objective been achieved?X A further three questions were identified by many in several user groups (Table 3). Table  SEQ Table \* ARABIC 3. Questions of Secondary Interest to Different User Groups. User groupDonorPartnerManagersAdvisersResponsible for M&E Has the objective been achievedXWhat progress has been made?XXAre the benefits sustainable?XXX What Were The Changes? All respondent groups interviewed were most interested in any changes that resulted from the capacity development. Their main interest was a demonstrable change in practice. What is striking is that all counterparts and most nationals wanted such information and only one national or counterpart did not identify this as a question they wanted answered. Counterparts tended to focus on changes at the individual, organisation and service delivery levels that resulted from training. This probably reflected that counterparts generally perceive capacity development and training to be almost synonymous. As one person remarked: I would want the evaluation to show that what he has done is benefiting his work and the Ministry and the country as a whole. What has been achieved from the training opportunity that he or she is going through [T5]. The other group of respondents who were most concerned with this question was AusAID, in particular those at the Post. They too wanted to see behavioural change - so whether the activity had changed; the ability of the people or the organisation whose capacity we are developing to do their job [T14]. As with partner agencies, there was a focus on application of training and behavioural change. However, it was recognised that at the moment AusAID tend to ask whether Ministry X is capacitated to Y. We often say they are able to do rather than they do [T19]. Thus, the need to move from asking whether program participants are able to do something, to whether learning is being applied was identified. Then, some at AusAID identified that the perspective taken in relation to change should be from the perspective of the beneficiary, rather than the donor [T15, T13, T22] and also that the degree of change need not be huge. In fact, . . . as an outsider looking at something that appears to be a very small change, it may actually be a very significant change for the people in that context. How we judge the value of the outcome needs to be from the perspective of the beneficiary [T13]. Those who had responsibility for monitoring or evaluation of the activity were also interested in identifying the changes that had occurred. However, their focus was more on organisational change than it was for partners or AusAID, as this quote indicates: The key question for me is what do you do differently now in comparison to what you did before because of the programme. . . This also refers to different systems and values being in place and applied that were not there before. For example, for a governments programme there may be new regulations in place which lead to new units being put in place to address the issues-- or the individual in the units producing things differently. If there has been an increase in planning and budget capacity, then you would look to see whether allocations have improved, they are more realistic, or whether there are other changes in line with what you expect. [T30] Most of those employed on the program (Program Director, Team Leader and Advisers) were also concerned with questions dealing with change. However, it was not as consistent a priority for them as for counterparts or AusAID. As with those responsible for evaluation, those employed on the program also emphasised change at an organisational level so that: Id want to know if there was demonstrable change, the impact it had made on the area, and whether this change was sustainable. Under that there are a lot of little elements, like whether staff performance and motivation has increased, whether the new procedures and systems are in place and being used. But overarching it all is whether there has been demonstrable change and therefore if the organisation is operating at the level we want [T44, T45] Regardless of the user group respondents represented or their experience in capacity development evaluation, those interviewed wanted the evaluation to identify the changes that had occurred. The changes respondents wanted to identify were related to several areas such as gains in knowledge and skills as well as in individual or organisational change. One person also identified that the evaluation should be able to show changes in competency of leadership [T19]. Among INGOs and those working at an activity level there was also a desire to see whether there had been changes in confidence and motivation. This was because they saw this as a key step in the change theory for capacity development [T48]. However, determining the degree of change in motivation and confidence was seen to be difficult to ascertain [T19, T27, T43]. Meanwhile, most of the NZAID interviewees considered that changes in the ability of the beneficiary to be able to address their own problems were key [T48, T49]. This is interesting because, despite AusAIDs definition of capacity development, which includes the phrase self generating performance improvement, the ability of an individual, organisation or nation to deal with their own problems was only identified by one of those interviewed from AusAID. As this person pointed out: What evaluation of capacity building should be asking is the extent to which we are contributing to (for example) Papua New Guineas ability to solve their own problems. If they cant, then we havent contributed to building capacity. [T10] Furthermore, very few respondents suggested that the degree of change needed to be quantified. Both respondents who did mention this were from AusAID and both recognised that they had limited experience in evaluation of capacity development [T18, T22] One other person from AusAID commented that; If it doesnt measure the softer side it isnt useful [T19]. Nevertheless, several others stated that it would be useful if the results could be quantified as this would allow comparison between programs and support the political use of evaluation [T13] However, they realised the potential difficulties associated with producing quantitative results to measure such changes. These included partner agency concerns with ratings [T22] and difficulty in establishing reliable indicators [T29]. It was also recognised that the achievement of change takes time and that it is often important to identify any progress towards change rather than just assessing the degree of change at the end. Therefore, one person said they: would want some demonstration of progressive improvements in capacity. I say progressive as capacity development is inherently over a long period of time, it is incremental, it is building on one stage before moving to the next. [T40] Perhaps the following encapsulates much of what all the user groups want evaluation of capacity development programs to achieve: I want (the evaluation) to be able to tell me the situation at the start, what the program hoped to achieve, the aspirations (and) what was achieved. [T9] Questions About How the Change Occurred The donors (including AusAID, NZAID, and INGOs) were all particularly interested in detail about how any change occurs. This included how effective the particular capacity development strategies being implemented were and the impact of environmental factors on the outcomes. Some of the AusAID respondents identified how they use this information. For example, I want to know what changed for the participants, why did it change, and what else was going on in the environment. If you have this information, you can form a judgement. [T13] With this information, AusAID can determine; whether we are using the right approach to capacity development i.e. on the job training, formal training [T13] and change the strategy if needed. [T26] Of those employed to support program implementation, identification of how the change occurred was of most importance to Advisers. In general, the latter were more concerned with the effectiveness of different capacity development strategies than in the impact of environmental factors on the outcomes. Where the Adviser wanted the evaluation to identify the impact of the enabling environment, it remained closely related to the strategies being considered. For example, one Adviser wanted the evaluation to answer; Could we have used a different process to meet the things that were constraints-- such as cultural mores?. [T43] However, questions about how the change occurred were not generally raised by activity managers. Only one manager (who was also an Adviser) stated that the evaluation should support; Learning about the capacity development process itself in a particular context in which you are working and what does or doesnt work. [T36] Similarly partners (and in general nationals) did not identify the reasons why the change occurred as a question the evaluation of capacity development needed to answer. Furthermore, those who had relatively little experience in evaluation of capacity development were less likely to want the evaluation to consider how the change occurred than were those who had extensive experience in this area. This was regardless of which user group they represented. Questions on Lessons Learned Questions about lessons learned were of most interest to donors and Advisers (especially Team Leaders). It was not a question any partners interviewed required evaluations to answer. Most Advisers were interested in finding out what could be applied to improve the implementation of the activity in which they were engaged. Comments such as; (I want to know) which aspects of the programme worked well, which didnt, how can they be strengthened: how can they be done better [T1] were typical. They did not identify a need to find out lessons for application to other projects. Those directly involved in the management of activities (Program Directors and Team Leaders) were also more interested in identifying lessons that could be applied to the activity being evaluated rather than what could be applied elsewhere. For the few Team Leaders who did want evaluations to provide lessons that could be used on other activities, it was not a key part of the evaluation. For example, one suggested that; An added bonus would be that it gives me takeaways that I can apply to other programs. [T37] This was in contrast to donors who were most interested in lessons that could be applied to other initiatives. Interestingly, in this regard AusAID Post staff in Indonesia were far more interested in lessons learnt that could be applied in other contexts than were any other groups of donors (including within AusAID). While none of the donor respondents stated explicitly that they wanted evaluations to identify lessons that could be applied to the current initiative, most indicated that they wanted to know if something had occurred and, if not, why not? [for example T18]. However, most expressed the desire for the evaluation to identify what could be applied in future phases, to other sectors or to other programs [for example T9, T13, T15, T16, T17, T49]. The statement that the evaluation should; help in the thinking about the lessons learnt, specifically so that you can say this is what we could do differently in future [T9] confirms this. Progress Toward and Achievement of Objective All partners interviewed wanted the evaluation of capacity development to address achievement of the agreed objectives. They each emphasised the need to focus and reflect on what the original objectives were. For example: An evaluation is always based on what the objectives of the program were. This is the only way to measure success, look at what the achievements were. Otherwise it is outside the program objective and is subjective. [T5] Managing Contractors, Program Directors and Team Leaders also wanted evaluations to identify whether the objectives were being achieved. Program Directors considered this to be one of the more important questions that needed to be answered. They also identified demonstration of progress (or otherwise) as a necessary key focus of capacity development evaluation. Additionally, very few people identified that they wanted the evaluation of capacity development to identify both progress and achievement of objectives [T9, T26, T33, T40]. Those who did were spread across user groups. While many of the AusAID staff interviewed also referred to achievement of objective (and, more broadly, accountability), it was not nearly as important a question as the evaluation questions previously discussed. From the responses in relation to interest in the broader accountability question, this interest was primarily to enable AusAID staff to provide information for briefings when required [T26] As one person interviewed from AusAID put it; I think accountability would be the rhetoric, but as we dont look at the quality of capacity development, Id argue the extent to which it was rhetoric or reality [T18]. Development Assistance Committee Criteria Of the four DAC criteria, none was consistently identified by the users as a question capacity development evaluations should address. Only effectiveness (the extent to which an aid activity achieves its objective) was identified to any extent, and even then, only by some of the user groups. Sustainability was not frequently identified as a question capacity development evaluations should consider. When raised, it was most frequently by those involved in managing implementation i.e. Managing Contractors, Program Directors and Team Leaders. It was also identified by a small number of donor staff. Few of those interviewed identified relevance or efficiency as an aspect that they wanted addressed. In fact, relevance was only identified by a few AusAID staff from the Indonesia Post and NZAID, and there was no consistent pattern across the three people who mentioned it. There was one who considered whether the technical content was relevant, another whether the capacity development approach was relevant, and the third whether the outcomes remained relevant [T49, T12, T18]. Efficiency was only raised as an issue that capacity development evaluation should address by a small number of those interviewed amongst donors and those responsible for monitoring and evaluation of the activity. It tended to be raised more frequently by those with more expertise in evaluation of capacity development initiatives than those with little experience. When mentioned the emphasis tended to be on whether the capacity development strategy being used was the most cost effective approach to achieve the results. One evaluator wondered how efficiency could be calculated for capacity development [T31] while another indicated that, in general, it would be impossible to produce a quantitative answer [T34]. Other Questions A number of people identified the need for the evaluation of capacity development to test the theory of change [T14, T15, T18, T23, T26, T34, T43, T48]. This was mainly requested by AusAID staff (primarily at the Indonesia Post) and those involved in evaluation of the initiative. For some [T18, T23], the focus was a concern about the robustness of the links between outcome levels. For others it was whether changes at an individual level were transferring to groups and being embraced at the organisational level [T34]. In these cases, the intent was to move the focus of the evaluation questions away from the lower levels (inputs and outputs, or individual) to higher levels (outcomes or organisational change). Another issue raised was the difficulty in attributing a change to the particular development intervention. At higher-level outcomes, this becomes a problem due to the range of initiatives supported by partner governments, donors, INGOs, or communities that may also be contributing to the outcome, as well as the time it takes to achieve change at this level. The need to be able to demonstrate that the support provided by a particular initiative did lead to the capacity development outcome being considered was identified by donors and Managing Contractors alike. However, those interviewed focused on contribution rather than attribution and many interviewed recognised that making a causal link at the high level of outcomes was extremely contentious [T10, T24]. Those interviewed also identified a range of other questions that they wanted answered by evaluations of capacity development initiatives. These included: What unintended outcomes occurred?, What opportunities had been missed?, Were the inputs of high quality? and What was the extent of partner ownership of the initiative?. However, each of these questions was only identified by a small number of people (at most three). Differences Between, and Within, User Groups The diversity of questions user groups wanted capacity development evaluation to answer, varied between user groups. For example, partners were really only interested in the changes that had occurred, whether objectives had been achieved, and to a lesser degree, the extent of cost efficiency. Other user groups had a more diverse range of interests. There were also differences in the questions that AusAID staff at Canberra and those at the Post wanted the evaluations to answer. Those at the Post were more interested in the changes that had occurred and why they occurred. By contrast, those based in Canberra were more interested in progress and the achievement of objectives. There is also some indication that questions of interest vary between Posts depending on the context. For example, capacity development initiatives in the Solomon Islands are being implemented in a Post-conflict situation. Here, AusAID staff could see little progress in capacity development and there was a sense that seeking to develop capacity was premature. This may explain why staff in this location were mainly interested in what changes were occurring and why these changes did or did not occur. This was in contrast to Indonesia where programs are larger, partner agencies better resourced and AusAID staff have been part of a program to develop their own capacity in monitoring and evaluation. It is likely that this program accounts for their greater understanding of what was and was not possible for evaluation. The AusAID staff interviewed in Indonesia were more interested in the evaluation identifying lessons that could be generalised and applied to other capacity development initiatives than was the case elsewhere. Furthermore, the range of questions AusAID staff at the Post wanted answered was generally less than that for those based in Canberra. Similarly, Advisers identified fewer questions to be answered than Program Directors or Managing Contractors. But what is clear is that, overall, the closer a person was to implementation of the Program, the narrower the focus of questions they wanted answered. This is shown schematically in Figure 1.  Figure  SEQ Figure \* ARABIC 1. Schematic diagram showing generalised relationship between the diversity of questions evaluation of capacity development is to answer and the intended users proximity to the initiative. Implications For the findings of evaluations of capacity development initiatives in international development to be more influential, it is clear that, among other things, they must meet the information needs of the intended users. At present, this is not the case and the different information needs of users on the whole do not appear to be recognised. Instead, the evaluation questions seem only to be driven by the DAC criteria. To ensure evaluations meet the information needs of intended users, the specific purpose of the evaluation and the questions that need to be answered to achieve this purpose must be considered when developing the Terms of Reference. These questions may not necessarily be derived from the DAC criteria. This will involve identification of the audience for the evaluation and recognition that each audience has different questions they want answered. Patton (2002, p173 - 174) emphasises the importance of this in maximising the use of the evaluation findings. So the research findings presented here, suggest that the questions different intended user groups wanted answered broadly reflected the different ways in which they will use the evaluation findings. In summary, there are those who: Want, primarily, to use the findings to improve the implementation of the existing initiative (mainly Advisers). Their focus is on identifying progress and learning which strategies are/are not leading to change so that they can apply lessons learned to improve outcomes. This information needs to be provided on an almost continual basis throughout the life of the initiative. Want to improve future capacity development initiatives (mainly AusAID). Their focus is on what influences the change (both the effectiveness of different capacity development strategies and factors in the environment) so that lessons learned can be applied to other initiatives Need to focus on accountability (Partners, Managing Contractors and Program Directors, AusAID at Canberra). Their focus is on whether the objective has been achieved and to a lesser extent on cost efficiency. Thus, end users of the information acquired have different requirements in terms of the evaluation. For example, the types of questions asked are likely to be different and the time at which the evaluation findings are required will vary. Therefore, to expect a single evaluation to meet all these conflicting requirements to the satisfaction of every intended user is unrealistic. Again, clarity in the purpose of the evaluation will help overcome such issues. Meanwhile, recognition of points raised in this paper may lead to more regular internal evaluations (i.e. monitoring) undertaken by the initiative which ask about the progress that has occurred and which strategies and factors in the environment have/have not contributed to this. The provision of such information should lead to improved implementation of existing initiatives. Moreover, AusAIDs Mid-Term Reviews and Independent Completion Reviews may focus on information of a summative nature that can support decisions about future initiatives (including future support to the same partner agency). These may also ask about which strategies and factors in the environment have/have not contributed to change and the achievement of objectives. In summary, we who are evaluators of international development assistance programs must take a more professional approach to evaluations. Rather than accept the Scope of Services without question, we may need to assist AusAID to clarify the purpose of any evaluation they wish to undertake; how it is intended to use the findings; and from this, determine the key questions that particular evaluations need to answer. To assist with the exercise, questions in Table 4 can be used as a guide4. We must also seek to ensure that the utility standards are met and apply good evaluation practice. Table  SEQ Table \* ARABIC 4. Questions of Primary Interest to Different User Groups. User groupDonor5PartnerManagersAdvisersResponsible for M&ECanberraPostWhat were the changes?XXXXXXHow did the change occur?Did the capacity development strategy work or not?XXXWhat factors in the enabling environment influenced the change? XXWhat lessons were learned?For application to this initiative.XXFor application to other initiatives.XXHas the objective been achievedXXWhat progress has been made?XXAre the benefits sustainable?XXXConclusion It is unlikely that clarity will be gained in the near future on what is meant by capacity development, and particularly whether it is a process (a means) or a product (an end). Perhaps from an evaluators perspective this is not essential as the two questions intended users most want answered reflect both aspects. The question: What has changed? captures the achievement of the objective, the product. While the question: How has the change occurred? (in terms of the effectiveness of different capacity development strategies and factors in the environment which impacted upon the change process) reflects the process. As evaluators, gaining clarity about the purpose of the evaluation and how the findings are to be used, we can help identify which of these questions we must answer and reflect whether capacity development is being considered as a means or an end. Recognising the different uses of the findings of evaluation of capacity development, evaluators should ensure that reporting supports these uses. The reports must be short, answer the questions of interest to the intended user and be timely. This will generally mean a change in current practice. For example, it may mean production of various summary reports for each intended user group in order to meet their needs before an integrated report is finalised. From the interviews, if capacity development evaluations provide pertinent information, there will be a significant improvement in how well these evaluations meet the needs of all intended users. But answering the questions of interest to intended users is only one step toward making evaluations of capacity development programs more influential. It is clear that we must all focus more on all of the utility standards. Key Messages for Evaluation of Capacity Development Identify what behaviour (individual, organisational, sectoral or environmental) has changed or is changing. Tell a simply story of change. Recognise that quantitative work is not essential. Show contribution, not attribution. Keep the report short. Comply with the utility standards. Acknowledgements This paper was presented at the Australasian Evaluation Society International Conference, Sydney, Australia, 29 August 2 September 2011. We would like to thank the Australian Government, Norton Rose, Cardno Emerging Markets and all those interviewed for their support of this research. The views in this paper are our own and do not necessarily represent those of AusAID, the Australian Government, Norton Rose or Cardno Emerging Markets. References AusAID. (2004). Capacity Development Principles and Practices. Internal Working Document, 2 November (2004). AusAID, (2008). Activity-level Monitoring & Evaluation. Brown L, LaFond A, Macintyre K, (2001). Measuring Capacity Building. MEASURE Evaluation. www.cpc.unc.edu/measure Cousins JB and Leithwood KA (1986). Current Empirical Research on Evaluation Utilisation. Review of Educational Research, 56 (3), 331 64. DAC (1991). Principles For Evaluation of Development Assistance. Development Assistance Committee. OECD. Paris. Forss, K., and Carlsson, J. (1997). The Quest for QualityOr Can Evaluation Findings be Trusted? Evaluation, 3, 481-501. Kotvojs. F. (2009). Development of Framework for Evaluating Capacity Development Initiatives in International Development. Australasian Evaluation Society Conference 2009, Canberra Australia. Loveridge D. (2007). Adaptive Management for International Development Programs. Australasian Evaluation Society Conference 2007, Melbourne Australia. Lusthaus, C., Adrien, M., and Pertinger, M. (1999). Capacity Development: Definitions, Issues and Implications for Planning, Monitoring and Evaluation. Universalia Occasional Paper. No. 35, September 1999 Morgan, P. (2006). The Concept of Capacity Study on Capacity, Change and Performance. European Centre for Development Policy Management. Stockholm. Organisation for Economic Cooperation and Development. (2006). Glossary of Statistical Terms Statistics Portal (Web) http://stats.oecd.org/glossary/index.htm Last accessed on: 10/06/2009 Owen J and Rogers PJ (1999). Program evaluation: forms and aproaches Allen and Unwin, Australia Patton, M.P (2008). Utilization Focused Evaluation. 4th edition. Sage Publications. London. Patton, M.P (2002). Qualitative Research and Evaluation Methods. 3rd Edition. Sage Publications. London. Taut ,S. (2007). Studying Self-Evaluation Capacity Building in a Large International Development Organization American Journal of Evaluation : 28; 45-57 UNDG. (2002). Report of the UN Inter-Agency Workshop on Capacity Development Geneva, 20 - 22 November 2002 http://www.undg.org/archive_docs/6351-Report_of_the_UN_Inter-Agency_Workshop_on_Capacity_Development__Geneva_20-22_November__2002.pdf Last accessed on: 9/06/2009 Watson, D. (2006). Monitoring and Evaluation of Capacity and Capacity Development. European Centre for Development Policy Management Discussion Paper No. 58B Whyte, A. (2004). Landscape Analysis of Donor Trends in International Development Human and Institutional Capacity Building. A Rockerfeller Foundation Series http://www.rockfound.org/library/04landscape.pdf Last accessed on: 3/3/2009 World Bank. (2005). Capacity Building in Africa: An Organisation for Economic Cooperation and Development Evaluation of World Bank Support Washington: World Bank Organisation for Economic Cooperation and Development  UNDPs (UNDG, 2002) definition is the other commonly used definition. Capacity refers to the ability of individuals, communities, institutions, organizations, social and political systems to use the natural, financial, political, social and human resources that are available to them for the definition and pursuit of sustainable development goals.  AusAID defines accountability as; The obligations of partners to act according to clearly defined responsibilities, roles and performance expectations, often with respect to the prudent use of resources, delivery of quality outputs and the achievement of meaningful results. (AusAID, 2008, p10) 3 Other major weaknesses identified which relate to utilisation included: lack of evaluator expertise, unclear evaluation purpose, weaknesses with reporting (including being too complex, too long, and presented after the information is needed for decision-making), and a lack of ownership of the evaluation by the intended user. These all reflect a lack of compliance with the JCSEE utility standards. 4 It should be noted, though, that it will not be appropriate for all evaluations of capacity development initiatives. 5 This mainly refers to AusAID.     PAGE  PAGE 9 Intended users distance from the initiative Diversity of questions EFQR^_ab ' ) 1 F l e ̿{o{o{o{o`oQoQohdjh B*mH phsH h{dh B*mH phsH h B*mH phsH hVh B*mH phsH h 7.h 6h 7.h 6H*\h 7.h mH sH h 7.h ^JmH sH h 7.h 6^Jh 7.h 6H*\^Jh 7.h 5CJ\aJh 7.h 5CJH*\aJh 7.h 5CJaJh  h 7.h Fa ' C#&e !!P""^##d$$" & F gd gd "gd $a$gd gd [6e f &.prwZq=H}-0(+3ABC_j !lwĽĵĬĬĥĬĬ˞˚˚h"Z h"Zh  h h3Fhh@ hh@6h3Fh 6 hh  hGlh h hN;B*mH phsH hq>B*mH phsH h B*mH phsH hB*mH phsH <%& px"%&ei13;ʼʵʵvh(h1UB*aJphh(h1U0JB*phh(h1UB*phh!|+h1U0JhGlh1U]aJ h1U]aJ hGlh1U h"Zh1U h\h1Uh1Uh_;h+eh 0J hW'h h h"Zh h"Zh h"Zh hh@%$@b e !!!P"##t$u$~%%(#)))++---..//0000T0U0f0ĽĽĶĶĬĠĶĶĕĎĆĆ}vfh'h0)5CJ\^JaJ h+yh0)hW+mHnHujh0)U hh0)hh0)B*phh0)B*mH phsH jh0)0JU h"Zh0) h Eh0)h0)h(h0)0JB*phh0)B*phh(h0)B*ph.jh(h0)0JB*CJOJQJUph&$%&'&)+@-//T0U0g0z0000000011!1"1$1 "$$Ifa$gdM Ff, "$Ifgd @&gd gd "gd f0g0y0z0000000000000001111 1"1#1$1%1&1'1(1)1*1+1,1-1.1/1012131415161718191;12"3i4j4t4u4}4~416)7*7+7,7/7o777ڼڱڪڡhdzh0)0J h6>)h0)h1 h0)0J6h6>)h0)0J h@Ph0) hNh0) h"Zh0)h0)h'h0)^Jh'h0)5CJ\^JaJh'h0)5?$1&1(1*1,1.10131517191<1=1>1123G4~4O89;@@BCCgd "gd@H"gd Ff "$$Ifa$gdM 7788E8F8G8L8M8889999999<<<<<<=>> > >N>"?#???9@}@@@AAAAAARBeBBCCȻ퉙 h@Ph0) hUh0)hh0)0J h0)h@Phr h_h0)h_h0)0J h"Zh0)h0)0J6CJOJQJh@Ph0)6CJOJQJh@Ph0)6CJ hdzh0) h0)0Jh0)hdzh0)0J h0)0J0C CCC C!C\CCCCCCCCCCDD$D%D(D)D*D+DDDDDDDDD EFEGEfEgEkElEEEEEHH HHHHHIIIIJ#Jºº³¬¤hQMh0)0J h h0)h_;h0)6 hsh0) h0)h@Pjh0)Uh0) hh0)h'h0)CJaJh'h0)5CJaJhW+mHnHu h@Ph0)jh@Ph0)U7C[C\CgCmCuC~CCCCCCCCCC "$$Ifa$gd  "$Ifgd Ff "$$$Ifa$gd $gd $$@&a$gd CCCC7." "$$Ifa$gd  "$Ifgd kd $$IfTlֈU :A SS| t0333333333333644 laTCCCCC "$$Ifa$gd  "$Ifgd  $$Ifa$gd CCCC7." "$$Ifa$gd  "$Ifgd kd $$IfTlֈU :A SS| t0333333333333644 laTCCCDD "$$Ifa$gd  "$Ifgd  $$Ifa$gd DD$D%D7.. "$Ifgd kd$$IfTlֈU :A SS| t0333333333333644 laT%D'D(D)D*D "$Ifgd  $$Ifa$gd  "$$Ifa$gd *D+D,DDD722-gd "gd kdq$$IfTlֈU :A SS| t0333333333333644 laTDDDDDDDE EE E@EBECEDEEEFE$$$$Ifa$gdRn "$$$IfgdRnFf"$$$$Ifa$gdRn$$gdRn $$$@&a$gdRn$$@&a$gd oMfFEGEdEfE7+"$$$$Ifa$gdRn "$$$IfgdRnkd$$IfTlֈU :A SS| t0333333333333644 laTfEgEiEjEkE$$$$Ifa$gdRn"$$$$Ifa$gdRn "$$$IfgdRnkElEEE7, "$$$Ifa$gdRn "$$IfgdRnkd$$IfTlֈU :A SS| t0333333333333644 laTEEEEE "$$$Ifa$gdRn "$$IfgdRnEEEEGH72-22gd "gd kd$$IfTlֈU :A SS| t0333333333333644 laTHHKLMNIQRrTWYZ]^_```a2bbc-fvg@i]igdX=gd gd<gd[gdc "gd gds#J(J)JJ/K0K5K`LpLLMMMMMM`NaNN-O2O?QAQGQHQIQRRdTeTfTgThTpTqT>V?VBVCVV.W/W1W3W4W5WWWWWWWKXMXTXh0)B*phhVghh0)B*ph h:gh0) hUh0) h#h0)hSh0)0J6hvHh0)6 hc h0) h0)h@Ph_;h0)6 hsh0) h0)6hQMh0)6h0) h h0) h_;h0)6TXUXXXX Y Y Y#YYYZZZZZZZ%[[[[[[[[[[[[[[ \\\\\\\i]k]m]n]]]]]^^v_~_``` ```````ݢ h}mh0) h<h0) h8 h0)h0)OJQJ h0)0Jh+vh0)0J h4{h0)hh0)6 h[h0) hUh0)h0! h0)6h@Ph0)6h0) h:gh0)<```1b2b3b4bbbbbbbccccccccccce#f$f&f'f,ffjgkglgmgrgsgtgugvg?i@ijк좜씍서}v h1h0) h Bh0)h Bh0)0J hU=lh0)hU=lh0)6 h0)0Jh32h0)0J h0)0JhNh0)0J hNh0) hbh0)h h0)6 h h0) hX=h0) hUh0) h0)h@Ph0) h<h0)h}mh0)6*]ijkmnmqqyrXsuewx=xcyz||K[*+܆ ič{"gdZPlgdQzgdHtgd "gd jjjkk:k;k?kLmmmmmmn:noooopppppppVqXq\qlqmqqzrPsQsRsSsXsdwewfw x xxxz޹޹޹ްޡޓޓބ|h=3Eh0)6 hQzh0)hh0)6 hHth0) htoh0) hnh0)hlh0)6hnh0)0J hh0) hlh0) h@Ph0) h/=?h0)h/=?h0)0Jh0) h""h0)h""h0)6 h0)0Jh;fh0)0J.zzzzB|C|&_ƅDž35:;uwxҎӎ{|23hKf0J5CJaJh'h0)CJaJh'h0)5CJaJ hph0) hKf0JhW+mHnHujh0)Ujh0)UmHnHtH u hUh0) hk}h0) h0)hlJ heh0)h0) h0)h@P6{}[h =&Z֛H 3 "$$Ifa$gd $a$gd  $@&a$gd " & Fgd "gd gd gdZPl"gdZPl 234GHabkl àĠŠƠȠɠʠ &+-123456[\]^_abc賨賨賨蝕賨賨hSCJaJh'hSCJaJh 9h CJaJ$h 9h CJOJQJaJnH tH $h'h CJOJQJaJnH tH h 9CJaJ hh h'h CJaJh'h 5CJaJ:345>CDEFGH_acegik "$Ifgd Ff "$$Ifa$gd Ffkl$ "$Ifgd kd$$IfTl      ֞+~  S S S  |  t0644 la`T "$Ifgd  $$Ifa$gd  "$$Ifa$gd 9 "$$Ifa$gd $" & F$Ifgd kd$$IfTl      ֞+~ SSS| t0644 la`T ĠŠƠȠɠ "$Ifgd 9$x$Ifa$gd 9 "$$Ifa$gd 9ɠʠ $" & F$Ifgd kd$$IfTl      ֞+~ SSS| t0644 la`T  "$Ifgd 9$x$Ifa$gd 9 "$$Ifa$gd 9/$ "$Ifgd kd $$IfTl      ֞+~  S S S  |  t0644 la`T/012345 "$Ifgd  $$Ifa$gd  "$$Ifa$gd 9 "$$Ifa$gd 56Z$" & F$Ifgd kd!$$IfTl      ֞+~ SSS| t0644 la`TZ[]^_ab "$Ifgd 9$x$Ifa$gd 9 "$$Ifa$gd 9bc$" & F$Ifgd kd"$$IfTl      ֞+~ SSS| t0644 la`T "$Ifgd 9$x$Ifa$gd 9 "$$Ifa$gd 9ڡܡޡ âƢCQRbqt£ܣޣ|ҧۧ;M^hd}h:hU hUhd}hI$hm}hShh $h'h CJOJQJaJnH tH $h 9h CJOJQJaJnH tH h'h CJaJh 9h CJaJ>$ "$Ifgd kd#$$IfTl      ֞+~ SSS| t0644 la`T "$Ifgd  $$Ifa$gd  "$$Ifa$gd ء$ "$Ifgd kd$$$IfTl      ֞+~ SSS| t0644 la`Tءڡۡܡޡߡ $$Ifa$gd  "$Ifgd  "$$Ifa$gd 9 "$$Ifa$gd $ "$Ifgd kd%$$IfTl      ֞+~ SSS| t0644 la`T "$Ifgd  "$$Ifa$gd  $gdmkd&$$IfTl      ֞+~ SSS| t0644 la`TRO)'1"$$ & Fn$d !%d $&d !'d $N!O$P!Q$]ngd$TH6"$$$n$d !%d $&d !'d $@&N!O$P!Q$]n^a$gd$TH"gd:"gd  ^lȨ()yөک%IJ=A{!"ƬǬ89ƭЭ񻴭h h 6 h_sh  h!|+h  hY*h  h!:h  h h@Ph@Ph h@PCJ h h  hd}6 h 6h)h 6 h$TH56h)h$TH56h  hgSh hd}7'JK\"Ǭ:ƭ7qհj% "^`gd  "^`gdgd "gd /"$ & Fn$d !%d $&d !'d $N!O$P!Q$]ngd$TH67Ů*+,Zeq{~¯ӯ)ѱ%')PT IYZ[\yz hKf0J h0)0Jh h0)mH sH h h0)OJQJnH tH h3Fh0)6h0)jh0)0JUh5h H*h_h H*hMh 6h!:h 6 h!:h  h_sh h  hMh 6OYZ[y"$a$gd gd gd "gd  "^`gd z$%&()/0123567OSbcd{ſŶֲֶ h_sh h h$ h0)CJ hm%CJ h0)CJhm%0JmHnHuh"Z h"Z0Jjh"Z0JUjhM[UhM[ hKf0JhKf#&'(34567cd{|}~gd gdZPlh]hgd  &`#$gd 21h:p . A!"#n$n% K 01h:p . A!"#n$n% P0 UU*$$If!v h555555555 5 65 T5 #v #v 6#v T#v :V l4n @@@@@@@@@@@@ txðððððððððððð0333333333333)v , 5 5 65 T5 f4pxððððððððððððytM Tkd$$IfTl4n  Y m 2  6T @@@@@@@@@@@@ txðððððððððððð0333333333333000044 laf4pxððððððððððððytM T $$If!v h555555555 5 65 T5 #v #v 6#v T#v :V l t0333333333333, 5 5 65 T5 ytM Tykd$$IfTl  Y m 2  6T t0333333333333000044 laytM Tk$$If!vh5A 5S5S55|5#vA #vS#v#v|#v:V l @@@@@@ t<ðððððð033333333333365A 5S55|5ap<ððððððTkdH $$IfTlֈU :A SS| @@@@@@ t<ðððððð0333333333333644 lap<ððððððT$$If!vh5A 5S5S55|5#vA #vS#v#v|#v:V l t033333333333365A 5S55|5aT$$If!vh5A 5S5S55|5#vA #vS#v#v|#v:V l t033333333333365A 5S55|5aT$$If!vh5A 5S5S55|5#vA #vS#v#v|#v:V l t033333333333365A 5S55|5aT$$If!vh5A 5S5S55|5#vA #vS#v#v|#v:V l t033333333333365A 5S55|5aTk$$If!vh5A 5S5S55|5#vA #vS#v#v|#v:V l @@@@@@ t<ðððððð033333333333365A 5S55|5ap<ððððððTkdQ$$IfTlֈU :A SS| @@@@@@ t<ðððððð0333333333333644 lap<ððððððT$$If!vh5A 5S5S55|5#vA #vS#v#v|#v:V l t033333333333365A 5S55|5aT$$If!vh5A 5S5S55|5#vA #vS#v#v|#v:V l t033333333333365A 5S55|5aT$$If!vh5A 5S5S55|5#vA #vS#v#v|#v:V l t033333333333365A 5S55|5aT$$If`!vh555S55|5#v#v#vS#v#v|#v:V l44 @@@@@@ t<ðððððð06+++++555S55|5a`p<ððððððT$kdz$$IfTl44      ֈ~ ``S``|` @@@@@@ t<ðððððð0644 la`p<ððððððT$$If`!vh555S5S55|5#v#v#vS#v#v|#v:V l44 @@ tðð06+++++55S55|5a`pFðððððððT0kd>$$IfTl44      ֞+~  SS S  |  @@ tðð0644 la`pFðððððððT$$If`!vh555S5S55|5#v#v#vS#v#v|#v:V l t0655S55|5/ a`T$$If`!vh555S5S55|5#v#v#vS#v#v|#v:V l t0655S55|5/ a`T$$If`!vh555S5S55|5#v#v#vS#v#v|#v:V l t0655S55|5/ a`T$$If`!vh555S5S55|5#v#v#vS#v#v|#v:V l t0655S55|5/ / a`T$$If`!vh555S5S55|5#v#v#vS#v#v|#v:V l t0655S55|5/ a`T$$If`!vh555S5S55|5#v#v#vS#v#v|#v:V l t0655S55|5/ a`T$$If`!vh555S5S55|5#v#v#vS#v#v|#v:V l t0655S55|5/ a`T$$If`!vh555S5S55|5#v#v#vS#v#v|#v:V l t0655S55|5a`T$$If`!vh555S5S55|5#v#v#vS#v#v|#v:V l t0655S55|5a`T$$If`!vh555S5S55|5#v#v#vS#v#v|#v:V l t0655S55|5a`T'666666666vvvvvvvvv66666686666666666666666666666666666666666666666666666666hH666666666666666666666666666666666666666666666666666666666666666666@@@ NormalCJ_HaJmH sH tH Z@Z i Heading 1$<@&5CJ KH OJQJ\^JaJ \@\ _ Heading 2$<@& 56CJOJQJ\]^JaJDA@D Default Paragraph FontRi@R  Table Normal4 l4a (k(No List jj d< Table Grid7:V0B'B @> O Footnote TextCJaJ@&@@ OFootnote ReferenceH*2O!2 sQuote 7^76HOH s Quote Char6OJQJ_HmH sH tH 4 @4 %d0Footer  9r .)@. %d0 Page NumberjQ@j k Body Text 3 d x-B*CJKHOJQJ_HaJmH phsH tH <+< }/ Endnote TextCJaJ>*> }/Endnote ReferenceH*DOD -K Char CharCJ_HaJmH sH tH LO"L #l> Body text "$xa$CJOJQJaJtH LO1L "l>Body text CharOJQJ_HmH sH tH *W@A* uStrong5\6U@Q6 _s Hyperlink >*B*phNYbN 3F Document Map&-DM OJQJ~42a.FGHIPGBPP.FGHILZ:4:Fa' C #&eP^d&!#@%'T(U(g(z(((((((())!)")$)&)()*),).)0)3)5)7)9)<)=)>)1*,,1388:;;;;;;;;;;;;;;;;;;;;;;;;;;;;;<<<$<%<'<(<)<*<+<,<<<<<<<<<= == =@=B=C=D=E=F=G=d=f=g=i=j=k=l==========?@@CDEFIIJrLOQRUVWXXXY2ZZ[-^v_@a]abcefmiiyjXkmeop=pcqr|tKw[w*z+}~ ią{}[h =&Z֓H 345>CDEFGH_acegikl˜ĘŘƘȘɘʘ /0123456Z[]^_abcؙڙۙܙޙߙ RO)'JK\"Ǥ:ƥ7qըj%OYZ[6d| 0 0"0F"0F 0"0"0"0 0"0"0"0@"0@"0@0@"0@"0@0@"0<@ "0<@ "0<@"0<@ "0<@ "0<@ "0<@ "0<@"0< @0@"0@"0@"0@"0@"0 @0@"0t)@"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @0t) @"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @"0t) @0t) @"0t)@"0@"0t) @0@"0!-@"0!-@"0!- @0@"08@"08@"08 @0@0:@"0: @"0: @"0: @"0: @"0: @"0: @0: @"0: @"0: @"0: @"0: @"0: @"0: @0: @"0: @"0: @0: @"0: @"0: @0: @0: @"0: @"0: @0: @"0: @"0: @0: @0: @"0: @"0: @"0: @0: @"0: @"0: @0: @"0:@"0:@0: @0 @0@0<@"0< @"0< @"0< @"0< @"0< @"0< @0< @"0< @"0< @0< @0< @0< @0< @0< @"0< @"0< @"0< @"0< @0< @0< @0< @"0< @"0< @"0< @"0< @"0< @"0< @0< @"0<@0<@"0=@"0=@"0=@"0=@"0=@0=@"0=@0=@0=@0=@"0=@"0=@0=@"0=@"0=@0=@"0=@0=@0<@"0Z@"0Z@0Z@"0Z@"0Z@"0Z@"0Z@0<@"0?c@"0?c@"0?c@"0?c@"0?c@0<@"08l@"08l@"08l@"08l@"08l@0<@"0r@"0r@"0r@"0r@0<@"0z@"0z@"0@0<@"0@"0@"0@"0@"0@"0@"0 @0@"0@"0@"0@ "0@ "0@ "0@"0@"0@"0@"0@"0 @0@0@"0 @"0 "0 "0 "0 "0 0  "0 "0 "0  "0  "0  "0  "0 0 "0 "0 "0 "0 "0 "0 "0 0 @"0 @"0 @0 @0 @"0 @"0 @0 @0  "0 "0 @0 @0 @"0 @"0 @0 @0 @ "0 @"0 @0 @0 @"0 @"0 @0 @0 @"0 @"0 @0 @0 @"0 @"0 @0 @0 @ "0 @"0 @0 @0 @"0 @"0 @0 @0 @ "0 @"0 0 0 "0 "0 0 0 "0 "0 "0 "0 0 "0 "0 0 "0 "0 "0 "0 "0 0 0 0 "0 "0 "0 "0 "0 "0 "0 0 "0"0"0"0"0 @"0@ "0@ "0@ "0@ "0@ "0@ "0"0 0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0@0@0 0000,)(00d)(Aa' C #&eP^d&!#@%'T(U(!)")<)=)>)1*,,1388:;;;;;;;;;;<<*<+<,<<<<<= =F=G=k=l=====?@@CDEFIIJrLOQRUVWXXXY2ZZ[-^v_@a]abcefmiiyjXkmp=pcqr|tKw[w*z+}~ ią{}[h =&Z֓Hbc RO)'JK\"Ǥ:ƥ7qըj%OYZ00D6 m(-0a< o6 m(-0< o> m(-0< o6 m(-0(< o> m(-0< o6 m(-0< o> q(-0< o6 q(-0 < o0000@000>q(-0< o6q(-0< o00 @d0,U00`;6q(-03< o6q(-0< o6q(-0< o6T(-0y< o6T(-0< o0000000 0 6T(-0< o6T(-0r < o6T(-0!< o6T(-0$< o6T(-0)< o>T(-0 )< o6m(-0})< o>m(-0)< o6m(-0J*< >m(-0*< o00< 0 0 ;6T(-0-&< o6m(-0w,< o>m(-0,< o6m(-0F/< o6m(-00< o6m(-03< o>-07< o6`V-07< o6`V-09< o6`V-0:< o>-0:< o6`V-0p:< o>-0:< o6`V-0:< o>-0:< o6`V-0:< o>-0:< o6`V-0:< o>-0:< o6`V-0;< o>ʱ-0;< o6Pq-0?;< o6Pq-0@;< o6Pq-0A;< o6Pq-0;< o>ʱ-0;< o6Pq-0;< o>ʱ-0;< o00< 0 0;00< 00|600< 00600< 00>ʱ-0<< o6Pq-0<< o6Pq-0l>< o6Pq-0 @< o6Pq-0@< o6Pq-0C< o6Pq-0D< o6Pq-0E< o6Pq-0F< o6Pq-01I< o6m(-0J< o6m(-06M< o6m(-0P< o6m(-0R< o6m(-0U< o6m(-0V< o6m(-0{W< o6m(-0cX< o6m(-0X< o>m(-0< o6m(-0Y< o6m(-0Z< o6m(-0-[< o6m(-0[< o6m(-0\< o6m(-0^_< o6m(-0`< o>m(-0b< o6m(-0b< o6<'-0ec< o6<'-0ve< o6<'-0g< o6<'-0i< o><'-0Kk< o6<'-0xk< o6<'-0Wl< o6<'-03m< o6<'-0io< o><'-0q< o6<'-07r< o6<'-0as< o6<'-0t< o6<'-0w< o><'-0py< o6<'-0y< o6<'-0|< o6<'-0< o><'-0l< o6<'-0< o6<'-0}< o6<'-0ʆ< o6<'-0Ɉ< o6<'-0< o6l-0< o6l-0< o>l-0y< o@06l-0n< o6l-0< o6l-0< o6l-0)< o>l-0@< o6l-0< o6l-0< o6l-0\< o6l-0ߝ< o6l-0)< o>l-0*< o6l-0< o>l-0< o6l-0< o> l-0< o6 l-0E< o> l-0< o6 l-0< o> l-0< o6 l-0< o6 l-0< o6 l-0'< o6 l-0< o>l-00< o6l-0d< o6l-0Ѫ< o6l-0< o6l-0'< o6<'-0L< o6<'-0c< o6 <'-0< o> <'-0< o6 <'-0< o6 <'-0#< o6 <'-0R< o6 <'-0^< o6 <'-0_< o6 <'-0̭< o6 <'-0< o6 <'-0w< o6 <'-0< o6 <'-0t< o6 <'-0< o6 <'-0< o6 <'-0E< o6 <'-0< o6 <'-0< o6 <'-0b< o6 <'-0ij< o6 <'-0"< o6 <'-0< o6 <'-0'< o6 <'-04< o6 <'-0Ӷ< o6 <'-0< o6 <'-0< oFa' C #&eP^d&!#@%''T(U(g(z(((((((())!)")$)&)()*),).)0)3)5)7)9)<)=)>)1*+,,01388:;;;;;;;;;;;;;;;;;;;;;;;;;;;;;<<<$<%<'<(<)<*<+<,<<<<<<<<<= == =@=B=C=D=E=F=G=d=f=g=i=j=k=l==========?@@CDEFIIJrLOQRUVWXXXY2ZZ[-^v_@a]abcefmiiyjXkmeop=pcqr|tKw[w*z+}~ ią{}[h =&Z֓H 345>CDEFGH_acegikl˜ĘŘƘȘɘʘ /0123456Z[]^_abcؙڙۙܙޙߙ RO)'JK\"Ǥ:ƥ7qըj%OYZ[y&'(34567cd{|}~00"0"00"0"0"00"0"0"0"0"00"0"00"0 "0 "0"0 "0 "0 "0 "0"00"0"0"0"0"0"00"0"0 "0 "0 "0 "0 "0 "0 "0 "0 "0 "0 "0 0 "0 "0 "0 "0 "0 "0 "0 "0 "0 "0 "0 "0 0 "0"0"0"00"0"0"0"00"0"0"000"0 "0 "0 "0 "0 "0 0 "0 "0 "0 "0 "0 "0 0 "0 "0 0 "0 "0 0 0 "0 "0 0 "0 "0 0 0 "0 "0 "0 0 "0 "0 0 "0"00000"0 "0 "0 "0 "0 "0 0 "0 "0 0 0 0 0 0 "0 "0 "0 "0 0 0 0 "0 "0 "0 "0 "0 "0 0 "00"0"00"0"00"00"00"0"00"0"00"000"0"00"0"0"0"00"0"0"0"0"00"00"0"000"0"0"0"00"0"0"00"0"0"0"0"0"000"0"0"0 "0 "0 "0"0"0"0"0"000"0 "0 "0 "0 "0 "0 0 "0 "0 "0 "0 "0 "0 "0 0 "0 "0 "0 "0 "0 "0 "0 0 "0 "0 "0 0 "0 "0 0 0  "0 "0 "0 0 "0 "0 0 0  "0 "0 "0 0 "0 "0 0 0 "0 "0 "0 0 "0 "0 0 0  "0 "0 "0 0 "0 "0 0 0  "0 "0 "0 0 "0 "0 0 0 "0 "0 "0 "0 0 "0 "0 0 "0 "0 "0 "0 "0 0 0 0 "0 "0 "0 "0 "0 "0 "0 0 0"0"0"0"0"0 "0 "0 "0 "0 "0 "0"00"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0b0@0@"0@0@0@000@000z@000z@000z@000z@0@0@0@0@0@000z00000000AFa' C #&eP^d&!#@%''T(U(g(z(((((((())!)")$)&)()*),).)0)3)5)7)9)<)=)>)1*+,,01388:;;;;;;;;;;;;;;;;;;;;;;;;;;;;;<<<$<%<'<(<)<*<+<,<<<<<<<<<= == =@=B=C=D=E=F=G=d=f=g=i=j=k=l==========?@@CDEFIIJrLOQRUVWXXXY2ZZ[-^v_@a]abcefmiiyjXkmeop=pcqr|tKw[w*z+}~ ią{}[h =&Z֓H 345>CDEFGH_acegikl˜ĘŘƘȘɘʘ /0123456Z[]^_abcؙڙۙܙޙߙ RO)'JK\"Ǥ:ƥ7qըj%OYZ[y67c 0 0"0"0 0"0"0"0 0"0"0"0@"0@"0@0@"0@"0@0@"0@ "0@ "0@"0@ "0@ "0@ "0@ "0@"0 @0@"0@"0@"0@"0@"0@"0 @0@"0@"0 @"0 @"0 @"0 @"0 @"0 @"0 @"0 @"0 @"0 @"0 @"0 @0 @"0 @"0 @"0 @"0 @"0 @"0 @"0 @"0 @"0 @"0 @"0 @"0 @0 @"0@"00?0-@"0 @00B0*@"0@"0@"0 @0@"0@"0@"0 @0@0@"0 @"0 @"0 @"0 @"0 @"0 @0 @"0 @"0 @"0 @"0 @"0 @"0 @0 @"0 @"0 @0 @"0 @"0 @0 @0 @"0 @"0 @0 @"0 @"0 @0 @0 @"0 @"0 @"0 @0 @"0 @"0 @0 @"0@"0@0 @0 @0@0@"0 @"0 @"0 @"0 @"0 @"0 @0 @"0 @"0 @0 @0 @0 @0 @0 @"0 @"0 @"0 @"0 @0 @0 @0 @"0 @"0 @"0 @"0 @"0 @"0 @0 @"0@0@"0@"0@0@"0@"0@0@"0@0@"0@0@"0@"0@0@"0@"0@0@"0@0@0@"0@"0@0@"0@"0@"0@"0@0@"0@"0@"0@"0@"0@0@"0@0@"000@"0@0@"0@"0@"0@"0@0@"0@"0@"0@0@"0@"0@"0@"0@"0@"0@0 @0@"0@"0@"0@ "0@ "0@ "0@"0@"0@"0@"0@"0 @0@0@"0 @"0 "0 "0 "0 "0 0  "0 "0 "0  "0  "0  "0  "0 0 "0 "0 "0 "0 "0 "0 "0 0 @"0 @"0 @0 @0 @"0 @"0 @0 @0  "0 "0 @0 @0 @"0 @"0 @0 @0 @ "0 @"0 @0 @0 @"0 @"0 @0 @0 @"0 @"0 @0 @0 @"0 @"0 @0 @0 @ "0@"0@0@0@"0@"0@0@0 @ "0@"000"0"000 "0"0"0"00"0"00 "0"0"0"0"0000 "0 "0 "0 "0 "0 "0 "00l  0"0"0"0"0 @"0@ "0@ "0@ "0@ "0@ "0@ "0"0 0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"0"000800t4444 @0 @0 0000*@0A $$$$$$$$$'e f07C#JTX`jz^z_bcdfhiyz{}~$$1CCCCCD%D*DDFEfEkEEEH]i{3kɠ /5Zbء'`egjklmnopqrstuvwx|a'((;; ;<<<       '!!8GH@ GD(    n!(* @C  S"?t A s *X99? "`n!(* B <vvB33 "`n9"U( fB C s *D33 #w((`B D c $D33 O!PF)`B E c $D33 OF)(G)"  )(+ F3  C"? })(* G 6vvG33 "` )(+ B S  ?{@d@#d T@ OLE_LINK1FL3&3&L923&323& 523& 923&823&L423&L323&723&L72 3& 72!3&72"3&62#3&62$3&L62%3& 62&3&52'3&L52(3&42)3&L82*3&=2+3&82,3& 82-3&=2.3&4 K/3& K03&13&ܸ23&33&\43&53&ܹ63&73&\83&93&ܺ:3&;3&\<3&=3&ܻ>3&?3&\@3&A3&ܼB3&C3&\D3&E3&ܽF3&G3&\H3&I3&ܾJ3&K3&\L3&M3&ܿN3&O3&\P3&Q3&R3&S3&\T3&U3&V3&W3&\X3&Y3&Z3&Í[3&\Í\3&Í]3&Í^3&č_3&\č`3&ča3&č W!W!&"&"""&&](](o(o(((66RROfOf0x0xggPPƃƃAA55//]ff^^tt{ݪݪFFjjz77       !"#$%&'()*+,-./0123456789:;=<>?@ABCEDFGHIJK _!_!5"5"""&&e(e(x(x(((66 R RXfXf9x9xoo__σσII""==ǢǢ 44eooggyLLy==  !"#$%&'()*+,-./0123456789:;=<>?@ABCEDFGHIJK>J*urn:schemas-microsoft-com:office:smarttags PersonName9K*urn:schemas-microsoft-com:office:smarttagsplaceBL*urn:schemas-microsoft-com:office:smarttagscountry-region8F*urn:schemas-microsoft-com:office:smarttagsCity=*urn:schemas-microsoft-com:office:smarttags PlaceName=*urn:schemas-microsoft-com:office:smarttags PlaceType LKJLKKFKKLLKKLKFKLKLKLLKLKLKFKFKLKLKLKFKFKKFFKLKKFFLKFLKKFKFLKFKFKFKKe n    # v|8!>!k!q!&&U([(g(m(z(("/(/U1[1445566HANABBCCFF!J'JKNPN|PP#Q%Q&Q*Q+Q/QxQ~QS%SSSXXXXYY[ [:f@fffmmnnUo[o.s4sxx{{#)&,ʣУ"(Ф֤IR7<FNqz&/`ity![]c%(256OSET]9:N]^d!!!!%%'(++(/)/E0F0001155 88;";#;Z;<<<<@@AABBDDFF3K7KUNZN4R=RVVXXNYY[[$^&^bbFf]fgg6o8ot&txx{{v~x~"}6YcU]")*`aƤ !:]^ťƥѥݥ().?AĦŦ+,pq§;<*+]j%Aɪʪܪ%&23EPWҫޫ۬߬ Iei[%(256OS33333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333lldd!!'(Q+Q++/,/"7#7;!;<<LL Q Q Q#Qddee e effAfAfffhhvjvjzjzjjjWkWknnnnoo5o5ododo3p3pJpO2fղ԰ծ!SZ[wh<'z<|44dh]h^`OJQJo(hHh^`OJQJ^Jo(hHohpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hHhhh^h`OJQJo(hHh88^8`OJQJ^Jo(hHoh^`OJQJo(hHh  ^ `OJQJo(hHh  ^ `OJQJ^Jo(hHohxx^x`OJQJo(hHhHH^H`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hH9 i^`io(hH.9 ^`hH.9 pLp^p`LhH.9 @ @ ^@ `hH.9 ^`hH.9 L^`LhH.9 ^`hH.9 ^`hH.9 PLP^P`LhH.^`CJOJQJo(^`CJOJQJo(opp^p`CJOJQJo(@ @ ^@ `CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo(h^`OJQJo(hHh^`OJQJ^Jo(hHohpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hH^`CJOJQJo(^`CJOJQJo(opp^p`CJOJQJo(@ @ ^@ `CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo(9 i^`io(hH.9 ^`hH.9 pLp^p`LhH.9 @ @ ^@ `hH.9 ^`hH.9 L^`LhH.9 ^`hH.9 ^`hH.9 PLP^P`LhH.h^`OJQJo(hHh^`OJQJ^Jo(hHohpp^p`OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hH^`OJPJQJ^Jo(^`OJQJ^Jo(hHopp^p`OJQJo(hH@ @ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoPP^P`OJQJo(hHh>>^>`OJQJo(hHh  ^ `OJQJ^Jo(hHoh  ^ `OJQJo(hHh^`OJQJo(hHh~~^~`OJQJ^Jo(hHohNN^N`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hH^`CJOJQJo(^`CJOJQJo(opp^p`CJOJQJo(@ @ ^@ `CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo(hhh^h`OJQJo(hHh88^8`OJQJ^Jo(hHoh^`OJQJo(hHh  ^ `OJQJo(hHh  ^ `OJQJ^Jo(hHohxx^x`OJQJo(hHhHH^H`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hH^`CJOJQJo(^`CJOJQJo(opp^p`CJOJQJo(@ @ ^@ `CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo(^`OJPJQJ^Jo(^`OJQJ^Jo(hHopp^p`OJQJo(hH@ @ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoPP^P`OJQJo(hH^`CJOJQJo(^`CJOJQJo(opp^p`CJOJQJo(@ @ ^@ `CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo(hhh^h`OJQJo(hHh88^8`OJQJ^Jo(hHoh^`OJQJo(hHh  ^ `OJQJo(hHh  ^ `OJQJ^Jo(hHohxx^x`OJQJo(hHhHH^H`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hH^`CJOJQJo(^`CJOJQJo(opp^p`CJOJQJo(@ @ ^@ `CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo([w!DfJT@/M|44c-i.GSZ*21pO I i Tz&'z1%7                  >r                 >r                                                               @ @$6 h@m4\H` 8 M >/ c Rn' 0!vHlrQz+ rgTD ""l"?#b$&( (6>)W+L,3m-Ox/Gb6 9_;(n?@lICo DD=3EI"FG$THkWJlJMrOU$U1Ui9V"ZM[g[\f/\c^Ay^ZctdKfVghbiYjlU=lZPl}mpon&%otqt/Guwv_|+d}m}u@iX=@1S8%,N`HtNe O[d}F ZD Q9IYdN;Vq7 0a$k%oR!':b"N`hkH<7q>nHu$T<M c=m%p0I~4lN:g*(:bxq1?_I8SkrRz0)+@HQsp T(U(g(z(((((((())!)")$)&)()*),).)0)3)5)7)9)<)=)p/4[;\;g;m;u;~;;;;;;;;;;;;;;;;;;;;;;<<<$<%<'<(<)<*<+<<<<<<= == =@=B=C=D=E=F=G=d=f=g=i=j=k=l======== QH 345>CDEFGH_acegikl˜ĘŘƘȘɘʘ /0123456Z[]^_abcؙڙۙܙޙߙ [\5-nD=n 0 0@I@{`@UnknownFaculty of Education G:Ax Times New Roman5Symbol3& :Cx ArialcMS Sans-SerifTimes New Roman5& >[`)Tahoma?& Lucida Sans{ Lucida GrandeFranklin Gothic Medium Cond;Wingdings?5 :Cx Courier New"hGaaMLF,/Y>0\F!n4d 2qHX ?_2IThere were three questions that most of those interviewed wanted answeredFionaFionaP            Oh+'0(8 HT t   LThere were three questions that most of those interviewed wanted answeredFionaNormalFiona6Microsoft Office Word@Ik@^|V@!rX@Vx}X,/՜.+,04 hp  Toshiba>Y' JThere were three questions that most of those interviewed wanted answered Title  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~      !"#$%&'()*+,-./0123456789:;<>?@ABCDFGHIJKLQRoot Entry F`Q}XSData '1TableWordDocument:SummaryInformation(=DocumentSummaryInformation8ECompObjq  FMicrosoft Office Word Document MSWordDocWord.Document.89q