重口50度灰

This report presents the planning, approach and findings of the evaluation of the 2024 annual international conference of the 重口50度灰. It reports on the merit and worth of the conference for the 重口50度灰and the range of conference delegates and identifies opportunities for improvement. 

A utilisation focussed, mixed-method convergent design was used to analyse data from interviews, post conference survey and other sources (SLIDO data, conference program, registration data and observations). The report conveys a high degree of satisfaction and value perceived by delegates and positive developments towards advancing the strategic priorities of the AES. 

Evaluation lead: Ruth Walker, Student-evaluator , Master of Evaluation , The University of Melbourne 
With support from Wen Chee in data analysis, presentation and interpretation

Evaluation led by Helen McInerney, a Capstone student undertaking the Melbourne University Master of Evaluation in a volunteer capacity, supported by a research assistant and fellow Masters鈥� student, Carla Pozo.

This report presents the design, planning, approach and findings for the evaluation of the 重口50度灰鈥檚 (AES) annual International Evaluation Conference which was delivered in Brisbane, Queensland from 27 to 29 September 2023. The evaluation was commissioned by the AES, and undertaken by University of Melbourne Master of Evaluation students, supported by the AES.

The evaluation was undertaken as a mixed methods approach, with the key performance indicator focusing on delegates鈥� satisfaction with different aspects of the conference. The evaluation answered two key evaluation questions (KEQ) as set out below.
Throughout this evaluation, particular attention was paid to the diverse perspectives of conference attendees, particularly through the lens of First Nations delegates, evaluators at the beginning, middle, and expert levels of their careers, and non-evaluators who interface with the evaluation profession in varying capacities.

Evaluation Lead: Dr Wan Yi Lee Independent Evaluator, with support from Dr Emily Saurman in scoping, and interview and survey design

This report presents the planning, approach and findings of the evaluation of the annual international evaluation conference 鈥楢ES22鈥� of the 重口50度灰. With support from the 重口50度灰as both commissioner and provider of the conference, an independent evaluator was engaged to undertake the project.

Two key evaluation questions (KEQs) were developed in collaboration with key 重口50度灰stakeholders: (1) How valuable was the conference for the range of delegates; and (2) To what extent did the conference contribute to advancing the strategic priorities of the 重口50度灰Board?

A three-phased fixed mixed method convergent design was conceptualised for the project. Data collection was undertaken both during and after the conference via interviews and an online post-conference survey with participants. The overarching purpose of the evaluation was to determine the overall value of the conference and to find areas for improvement within the conference, or 重口50度灰services. The report conveys a high degree of value expressed by the evaluation community regarding participants鈥� involvement in AES22 and positive developments towards addressing the strategic priorities of the AES.

The 2022 重口50度灰 (AES) mentoring program ran for six months. It consisted of five groups with six members each, supported by a project officer and a volunteer working group. Two groups focused on general professional development and three on specific evaluation topics of interest to members. The program was evaluated on the basis of three online surveys and two focus groups, as well as regular discussions with mentors and associate mentors.

The mentoring program was generally found to be successful and worth repeating in future years. Participants reported that they benefitted overall from participation, particularly if they were actively involved in their groups. No mentees expressed dissatisfaction with the program, although some constructive suggestions were given.

The 重口50度灰 (AES) is pleased to share this piece of research about the evaluation field. The study was originally conceived as a way of better understanding the promise and practice of evaluation across Australia.

This project was led by the 重口50度灰Relationships Committee, drawing on support and input from many 重口50度灰members, including a dedicated State of Evaluation working group. KPMG is acknowledged for its contribution to the project. The data collection was completed in mid-2022.

This being the first study of its kind in Australia, it is likely to be repeated periodically. The 重口50度灰Board recognise the report as a starting point for further exploration and discussion. Any interesting and emerging trends may be further or more deeply investigated. In the meantime, the report findings are helpful for the 重口50度灰in setting its strategic directions beyond 2023 and in tailoring support for members who work in the evolving field of evaluation.

Please email aes@aes.asn.au if you would like to give specific report feedback to Relationships committee.

During the Pilot, a group of talented CPE Evaluation Capstone students: Kate Glastonbury, Ion Ioannidis, Bronwyn Ledgard, Martina Donkers and Georgia Pinto evaluated the program. This is their report.

Evaluation by Tara Collins

This report presents the planning, methodology, key findings and recommendations of the evaluation of FestEVAL 2020. Designed and delivered by the 重口50度灰 (AES), FestEVAL was held from 21 to 25 September 2020 and offered a series of free, online evaluation-themed activities, centred around the theme of 鈥淐elebrating Evaluation鈥�.

The evaluation was led by a volunteer Master of Evaluation student, Ms Tara Collins. Ms Collins benefited from the support and guidance of the AES, as well as that of an experienced evaluation fellow, Dr Dorothy Lucks.

COVID has required the 重口50度灰and its members to work in fundamentally different ways. The 重口50度灰has implemented a range of measures to maintain services for its members given COVID restrictions which have prevented face to face activities and suspended the conference for 2020. We surveyed 重口50度灰members and others from the evaluation sector to identify other initiatives that the 重口50度灰could take to improve services during the COVID period and beyond.
The objectives of the survey were to identify:

1. Actions/initiatives that the 重口50度灰could undertake in the immediate term to enhance the services being offered to and accessed by its members.
2. How services should be delivered in the longer term post the COVID restrictions (should mode of operation return to pre COVID or should some of the changes continue on).

Evaluation lead Emily MacKay, with support from Penny Smallwood in scoping, survey design and quantitative analysis

This report presents the planning, approach and findings of the evaluation of the annual international evaluation conference of the 重口50度灰 鈥楢ES18鈥�. With support from the 重口50度灰as both commissioner and provider of the conference, two volunteer external evaluators were engaged to undertake the project; one (the report鈥檚 author) a Master of Evaluation student.

Two key evaluation questions (KEQ) and a series of sub-questions were developed in collaboration with key 重口50度灰stakeholders: (1) How valuable was the conference for the range of delegates; and (2) To what extent did the conference contribute to advancing the strategic objectives of the 重口50度灰Board?

A three-phased fixed mixed method convergent design was conceptualised for the project. Data collection was undertaken both during and after the conference via interviews and an online survey post-conference with participants. The overarching purpose of the evaluation was to determine the overall value of the conference and to find areas for improvement within the conference, or 重口50度灰services. The report conveys a high degree of value expressed by the evaluation community regarding participants鈥� involvement in AES18 and positive developments towards addressing the strategic objectives of the AES.

By Greet Peersman and Patricia Rogers

Commissioned by the 重口50度灰Board, this report outlines possible pathways to achieve one of the AES鈥� Strategic Priorities 鈥� 鈥榯o strengthen the capacity and professionalism of the evaluation sector鈥�. By professionalism, we do not mean working to make evaluation an exclusive profession, with barriers to entry including compulsory accredited qualifications and licensing. Instead, we mean continuously improving the quality of evaluation practice and products.

Evaluation by Jen Thompson

This report communicates the approach, methodologies and findings of the evaluation of the 2017 重口50度灰International Conference. The evaluation was conducted by a solo evaluator, Jen Thompson, a student of Melbourne University鈥檚 Master of Evaluation. The 重口50度灰was the commissioner of this evaluation and the author was hosted as an intern while completing the evaluation and post-Capstone obligations.

Two key evaluation questions (KEQ) and nine sub-questions were developed in consultation with the AES2017 conference organisation committee. The KEQs asked 鈥淒id the conference satisfy the professional interests of conference delegates?鈥� and 鈥淒id the conference satisfy the strategic interest of the 重口50度灰Board?鈥�.

A mixed method research approach was employed which included two data collection points; 20 at-conference interviews and a post-conference on-line survey of 215 participants. The evaluation primarily explored merit (quality) and worth (value) of the conference as it was perceived by participants and expressed in terms of satisfaction levels.

Evaluation by Benedict Shipp

This report is an evaluation of the 重口50度灰International Evaluation Conference that was held in Perth from September 19-21, 2016. Themed 鈥淟andscapes鈥�, the 重口50度灰conference attracted more than 300 participants. The 重口50度灰conference was preceded by two days of pre-conference workshops, which were out of the scope for this evaluation.

The 2016 conference had 110 presentations over 3 days, plus a special session of the 重口50度灰Annual General Meeting and the 重口50度灰Fellows. Each presentation had been allocated to one of 6 conference strands (excluding the 重口50度灰Annual General Meeting):

鈥� Plenary session

鈥� Evaluation Landscapes

鈥� Natural and Built Landscapes

鈥� Organisational, Political and Economic Landscapes

鈥� Social and Cultural Landscapes

鈥� Special Session.

This report gathered data from three sources 鈥� an Online Attendee Survey, an Online Non-Attendee Survey and semi-structured interviews conducted during the three days of the conference.

All the data collected was collated and used to answer the key evaluation question: 鈥極verall, did the conference attendees perceive there was value in attending the conference?鈥�

Evaluation by Emma Walters

The 重口50度灰2015 International Evaluation Conference was held in Melbourne from September 6-9, 2015. Themed 鈥淩eaching Across Boundaries鈥�, the 重口50度灰2015 conference attracted over 500 participants. Data collection from attendees and non-attendees was conducted during and after the conference to answer the following evaluation questions: Overall, did the conference attendees perceive there was value in attending the conference? If so, why? If not, why not? Specific questions this evaluation report addresses are:

1. What were the reasons participants registered for the 重口50度灰2015 conference?
2. What were the reasons that non-participants did not register for the 重口50度灰2015 conference?
3. Did conference participants find the conference valuable? If so, why? If not, why not?
4. What elements of the conference were most valuable to participants?
5. What elements of the conference were least valuable to participants?
6. Do participants perceive they are likely to change their evaluation practice in any way as a result of attending the conference? If so, how?
7. How could the conference be improved in future?

Attendees of the conference perceived there was value in attending: approximately four-fifths of respondents (79%) found it 鈥渇airly valuable鈥� or 鈥渆xtremely valuable鈥�, although this is a small drop from the 85% who found the 2014 conference 鈥渇airly鈥� or 鈥渆xtremely valuable鈥�. The 重口50度灰2015 attendees particularly valued the networking and learning opportunities offered.

Evaluation by Charlie Tulloch and Sigrid Patterson

The Darwin 2014 Australasian Evaluation Society (AES) Conference was held between 8 and 12 September 2014 at the Australian Centre for Indigenous Knowledges (ACIKE), located at the Casuarina Campus of Charles Darwin University (CDU). The conference鈥檚 theme was 鈥榰nleashing the power of evaluation鈥�. It featured two days of pre-conference workshops, two and a half days of conference activities, a conference opening event, a social program, international keynote speakers, a broad range of presentations by attendees and a range of other activities.

An evaluation was sought by the 重口50度灰to assess the value of the conference, along with ways to improve the conference over the next three to five years. 重口50度灰Board members and key Darwin event organisers helped to shape the focus of the evaluation against three key themes: the participant experience; organisation and logistics; and strategic alignment of the conference with AES鈥� goals.

The evaluation was conducted by two Master of Evaluation students at the University of Melbourne, with support from the Centre for Program Evaluation and the AES, in particular two evaluation 鈥榝ellows鈥� who provided expert guidance throughout the process.

The evaluation deliverables include this report on findings and appendices, associated data collection tools that could be used in future years and a separate report for the 重口50度灰detailing post-event participant and non-participant survey findings.

The findings focus largely on identifying critical challenges and areas for improvement in future.

Evaluation by David Turner

The 2013 Australasian Evaluation Society (AES) conference took place in Brisbane, Queensland, and has been evaluated on the basis of an online survey of conference participants. This represents the third year in which the approach of an online survey after the conference has been used.

Responses to the conference were generally positive. Overall, participants rated the conference experience positively. Learning opportunities, networking, and skill development were recognised as positive elements of the experience.

Comparison with the previous two years, however, shows cause for concern. Assessments of the learning and skill development opportunities were lower than for previous conferences, and negative comments were made about the quality of the program and some elements of the conference arrangements. New aspects of the conference, such as the use of social media, continue to attract little support or use.

Some suggestions are provided for consideration in planning future conferences.

Evaluation by David Turner

The 2012 Australasian Evaluation Society (AES) conference took place in Adelaide, South Australia. The conference was attended by over 400 people, with a wide range of participants that included delegates from Papua New Guinea, Thailand, the USA, the UK, and other countries. Pre-conference workshops were held to provide professional development, and are included in this report. The primary source of feedback from participants, for this evaluation, was provided by an online survey, with a 57% response rate.

Evaluation results show generally positive responses to the conference. Most respondents said the conference provided them with value in a range of areas, such as knowledge of new evaluation approaches or methods, awareness of research being done elsewhere, and opportunities to develop new evaluation skills. Some concerns remain about presentation quality, opportunities to interact effectively with peers, and the adequacy of the time allowed for presentations.

This conference was the first to embrace new social media technologies, but with a low take-up rate.

Suggestions are provided for building on the 2012 experience in future.

Evaluation by David Turner

This report summarises feedback on the 2011 Australasian Evaluation Society (AES) international conference in Sydney, Australia. It reflects a decision to go beyond the evaluations done on previous conferences, which focused heavily on participant satisfaction. Instead, this year鈥檚 evaluation looked at what participants learned from the conference and how likely they think they are to make use of new contacts, information or skills in their future work.

Responses to an online survey indicated that participants assessed their experience highly in terms of what they learned and how they expect to use it. Concerns and criticisms were raised about issues that reduced the opportunity for effective interaction with other participants, or otherwise reduced the opportunity to build on the conference for professional development. Recommendations are given for addressing these concerns and building on this year鈥檚 evaluation approach for future conferences.