Guidance Matrix: Standard 6

‘Continuous evaluat[ion] and review’ refers to a cyclical approach to evaluation and review that is ongoing rather than relying on ad-hoc activities and/or only occurring at a particular point in time. The AMC does not specify the frequency or method of particular evaluation and review activities, but the cycle of evaluation and review should include a variety of measures, be supported by evidence and demonstrate that the provider makes the necessary changes to the curriculum and program as a result of the evaluation to manage concerns about, or risks to, the quality of the program (6.1.1).

‘Communities’ include experts with lived experience of local health care, particularly those from communities who experience health inequities and Aboriginal and/or Torres Strait Islander and Māori communities. Feedback from all relevant stakeholders may be collected through different means that reflect the needs of different stakeholders, such as surveys, focus groups or consultations, as long as this feedback is authentically, regularly and systematically sought (6.1.2).

Providers should explain under Standard 6.1: Continuous review, evaluation and improvement how student, community and other stakeholder feedback is sought and analysed in the service of evaluation. The outcomes of this feedback process for communities, including how communities contribute to the program through evaluation, should be explained under Standard 1.2: Partnerships with communities and engagement with stakeholders. The details of student representation processes should be explained under Standard 1.3: Governance.

Providers should take care that feedback, particularly student feedback, is appropriately handled to maintain confidentiality or provide avenues for deidentified feedback when relevant, maintaining a focus on program evaluation and improvement (6.1.2).

‘Other education providers’ refers mainly to those who provide primary medical programs. Providers may also consider collaborating with prevocational training providers*, specialist training providers, and other health profession education providers on continuous evaluation and review, where relevant (6.1.3).

Documentary evidence could include:

• Policies, strategies and frameworks around the process of continuous evaluation, review and improvement, indicating the areas evaluated and reviewed.
• Descriptions of the continuous evaluation and review process and governance that demonstrate how these ensures that the provider can responsively and effectively manage concerns about or risks to program quality.
• Descriptions of the range of evaluation methods used, key findings and how the evaluation informs decisions about change.
• Reports from recent curriculum/educational reviews.
• Reports from other evaluation activities of different elements of the program.
• Agendas and minutes from key evaluation-related governance meetings.
• Descriptions of how feedback is regularly and systematically sought from students, and how the provider ensures the views of broad student cohorts are captured through this process.
• Descriptions of how feedback is regularly and systematically sought from staff, prevocational training providers, health services and communities.
• Documents related to evaluation and continuous improvement containing analysis of feedback.
• Descriptions of the collaborative links between the provider and the program, and other education providers, including the nature of the links, exchanges of students and staff, and research collaboration.
• Descriptions of the process for considering national and international developments in medicine and medical education, and how this influences and informs the program.

Interview and observational evidence could include:

• Discussions with staff responsible for continuous evaluation and review on how these activities are undertaken in practice.
• Discussions with student leadership, students, staff, prevocational training providers, health services and communities, on how their feedback is sought and contributes to the program.

There are no examples at this time.

There are no resources at this time.

‘Determin[ing] that all students meet the medical program outcomes*’ across ‘student cohorts and graduate cohorts’ refers to evidencing that phase/year cohorts of students and graduates are consistently achieving all the expected skills, knowledge and behaviours of medical students at different stages of the program (6.2.1). Providers define the graduate outcomes under standard 2.1.1, and outline learning outcomes for each stage of the program under standard 2.2.9.

‘Ensur[ing] that the outcomes of the medical program* are similar’ across ‘student cohorts and graduate cohorts’ refers to ensuring that phase/year cohorts of students and graduates are consistently performing in, progressing through and graduating from the program. The analysis should be sufficient to reveal any differences in performance and allow the provider to understand and address root causes (6.2.2).

‘Student characteristics’ refers to key student demographic, entrance qualifications, and entry pathway features. These characteristics should be relevant to the nature of the student cohort, as outlined under standard 4.1.2, and groups of students who may require additional health and learning support, as referred to under standard 4.2.3; along with other relevant demographic and admissions data regularly captured by the provider (6.2.3).

The evaluation of outcomes of the medical program for cohorts of students from equity groups and Aboriginal and/or Torres Strait Islander and Māori communities should be linked to the consideration of infrastructure and supports for these groups and communities under standard 4.1.3 (6.2.4).

‘Aboriginal and/or Torres Strait Islander and Māori education experts’ can be staff of or external to the provider. Experts should have experience with and/or qualifications in education and/or evaluation (6.2.4).

Documentary evidence could include:

• Descriptions of the student cohort and graduate cohort performance analysis process, including for consistency of medical program outcomes achievement and similarity of outcomes of the medical program.
• Descriptions of processes applied to identify and address any deficit in medical program outcomes achievement and similarity of outcomes of the medical program.
• Pass rates for each phase/year and major component of the program, and graduation rates, for at least the last three years, including for cohorts of students from equity groups and Aboriginal and/or Torres Strait Islander and Māori students.
• Assessment performance for key assessments and components of the program, including for cohorts of students from equity groups and Aboriginal and/or Torres Strait Islander and Māori students.
• Descriptions of the mechanisms and tools used to track graduates, both internal (e.g. provider graduate surveys) and external (e.g. the Medical Schools Outcomes Database).
• Evaluation reports resulting from analysis of consistency of medical program outcomes achievement and similarity of outcomes of the medical program, including for cohorts of students from equity groups and Aboriginal and/or Torres Strait Islander and Māori students.
• Descriptions of how Aboriginal and/or Torres Strait Islander and Māori experts inform and review evaluation of including for cohorts of students from equity groups and Aboriginal and/or Torres Strait Islander and Māori student cohorts.

Interview and observational evidence could include:

• Discussions with staff responsible for outcome evaluation on how these activities are undertaken in practice.
• Discussions with staff on student selection, curriculum and student support committees on the outcome evaluation data they receive and how it is used.

There are no examples at this time.

There are no resources at this time.

The ‘shar[ing]’ the ‘outcomes of evaluation, improvement and review processes’ refers to providing detailed outcomes and accessible summaries of these outcomes directly to students and the staff, clinical supervisors and community members contributing to program delivery (6.3.1).

Stakeholders with an interest in graduate outcomes may include:

• Prevocational training providers, particularly those who train many of the provider’s graduates.
• Training sites.
• Health jurisdictions.
• Community groups.
• Aboriginal and/or Torres Strait Islander and Māori people, communities and organisations.

‘Mak[ing] evaluation results available’ refers to providing the overall results to key individual representatives/ leaders of stakeholder groups. ‘Consider[ing] their views’ refers to being responsive to feedback on the processes of continuous evaluation and improvement of the program (6.3.2). The provider should explain under Standard 6.3: Feedback and reporting how stakeholders are provided with the results of evaluation and how their views on the continuous evaluation and improvement processes are considered. The provider should explain under Standard 1.2:  how stakeholders directly contribute to evaluation processes.

Documentary evidence could include:

• Descriptions of the process for reporting and sharing the outcomes of evaluation, improvement and review processes.
• Descriptions of the process for sharing evaluation results with stakeholders.

Interview and observational evidence could include:

• Discussions with members of admissions and course advisory committees on the governance reporting processes.
• Discussions with students, staff, clinical supervisors, and community members contributing to program delivery on their awareness of the outcomes of evaluation, improvement and review processes.
• Discussions with prevocational training providers on their awareness of evaluation results, and the provision of feedback on evaluation and improvement processes.

An example from Notre Dame School of Medicine

Notre Dame University School of Medicine is placing an increased focus on reporting evaluation findings to the student cohort to ensure students are aware of evaluation results and improvements made based off feedback given. Evaluation findings are provided to stakeholders for review and consideration with an expectation that improvements and changes to the program will be identified and reported. These findings are subsequently made available to the students through various avenues including:

• Orientation week quality presentation which outlines the evaluation plan for the year ahead; purpose of quality activities; and recent outcomes of evaluations, including changes and improvements to the student learning experience.
• Bi-annual summary of evaluation results and identified improvements presented in infographic format and published on the Learning Management System for student review.
• Quality presentation at the staff-student liaison meetings.
• Student representation on the School’s National Quality Committee which discusses quality initiatives and evaluation development.
• Summary of changes to the program in the course outline summary.

Contact – School of medicine email
som.quality@nd.edu.au

There are no resources at this time.