Assessment Innovation
Project – Contributing to a safe and competent workforce: Understanding International Medical Graduate (IMG) assessment experiences and performance
This project is designed to provide information on the experiences of international medical graduates (IMGs), recruiting health services and other key stakeholders in the common assessment and registration pathways for medical practitioners in Australia. The outcomes of this project will be used to inform policy development and to improve IMG assessment pathways.
This project will adopt a mixed method research approach. It will combine the views of IMGs regarding pathways and career experiences with qualitative insights of key stakeholders and quantitative datasets. Specifically, it will draw on this evidence base to develop IMG personas and journey maps, and to shed light on the role IMGs play in the medical workforce.
An Advisory Group will provide expert guidance, advice and support to the project and the AMC. The Advisory Group will be made up of a Chair appointed by the AMC and other members; Health consumer and community, Council of Presidents of Medical Colleges (CPMC), Postgraduate medical education councils who support IMGs, Executive staff of health services, Medical Board of Australia, Ahpra, Australian Medical Association, AMC committees, Aboriginal and/or Torres Strait Islander, Relevant Government Agencies via the Department of Health and Aged Care, International Medical Graduates, IMG Recruiter, IMG examiners and supervisors, Workplace Based Assessment (WBA) provider, and Bridging course providers.
Project Outcomes
This project aims to:
- Review the literature to explore the experiences of IMGs, to identify trends, gaps and good practice to inform quality improvements.
- Identify ‘quick wins’ for improvement through preliminary research and stakeholder engagement for action at the AMC, and a system level, including opportunities for interagency action.
- Provide a more differentiated, evidence-based picture of the heterogeneity of IMGs and their current and future potential contribution to the healthcare of Australian communities through detailed quantitative and qualitative analysis.
- Inform a clearer understanding of the thinking, feeling, and experiences of IMGs as well as perspectives of other stakeholders through evidence-based analysis.
- Inform policy development linked to the National Medical Workforce Strategy, improve IMG assessment pathways and communication channels, and identify further support for candidates to improve retention and progression.
Project Scope and Deliverables
The proposed project scope includes the common assessment pathways for IMGs. The scope will be confirmed by the project Advisory Group.
The project will take place in three stages, with corresponding deliverables:
Stage 1: A Data Diagnostic and Quick Wins report
Stage 2: IMG Personas and Interim Report
Stage 3: Journey Maps + Final Report
Opportunities to Contribute
This project is based on co-design methodologies. The Advisory Group will provide input through zoom meetings and follow up advice through surveys and focus groups. The project will also include a consultation phase which provides the opportunity for broader stakeholders of IMG assessment to review the proposed deliverables and shape the end products.
Online Clinical Examination
In June 2020, the AMC Board of Directors approved the development of a proposal to implement a new delivery format for the AMC clinical examination. The proposal presented long-term opportunities to improve quality and accessibility, and consider hybrid examination models.
Work commenced on the Examination in late 2020 under the guidance of the Clinical Examination Project Steering Group. The online clinical examination retained the structure of 14-scored stations and two unscored stations (pilots) with a required pass mark of 10 or higher out of 14 scored stations. The Zoom platform was chosen for the interactions between invigilator, a simulated patient and the candidate. The invigilator function was designated as key and was initially allocated to AMC staff with expansion to the NTC Marshal program.
Test examinations
Three test examinations were conducted between in February 2021 as proof of concepts to test online examination delivery theories, technology and processes on a small scale by using AMC staff (only) as examination participants.
Mock examinations
Three mock examinations were conducted later in February and early March 2021.
The purpose of the mock examinations were to:
- Conduct full end-to-end test of all technology and operations at scale with university students as candidates
- Re-enforce training with practice
- Test examination content for online delivery; and
- To ensure consistency between users guide for candidates, simulated patients, examiners and invigilators.
Pilot examinations
In early March 2021, pilot examinations were conducted with international medical graduates as part of the final testing. The test candidates were offered an opportunity to attempt the examination under ‘real’ conditions as part of the testing. The candidates could pass the examination or be offered another attempt free of charge if they failed.
Official launch of Clinical Online Examinations
The AMC launched the online clinical examination on Tuesday, 16th of March 2021 with the AMC website updated with relevant information, candidate resources including access to the candidate briefing video and the updated examination specifications (policies). Scheduling priority was provided to candidates listed on the waiting list from postponed 2020 examinations. Candidates residing in Australia were provided priority to support progression into the workforce and the successful delivery of the online examination.
In April 2021, a trial with overseas based candidates was successfully completed and the online examination was made available to overseas based candidates from July 2021.
Further Development
Further development of the online clinical examination will include the following –
- More automated processes
- Up-scaling the examination
- Digital marking. The AMC has recently trialled digital marking. This involves Examiners marking an online marksheet. The trial of online marksheets has been successful and will be fully implemented in online clinical examinations from 2022.
- Investigating hybrid examination models.
Examiner Decision Making (Edm) Research Project
The Examiner Decision Making (EDM) project was originally approved in 2018, and is aimed at quality improvement of the AMC clinical examination to better understand how examiners form judgements leading to candidate results. It was expected the research would be generalisable and contribute to publishable research and assessment literature from the AMC.
To continue the project, alternative approaches to clinical examination video review and examiner interviewing were needed to be established along with ethics approval.
However the initial timeline was not able to be delivered due to the impact of COVID 19 on the cancellations of clinical examination at the AMC National Test Centre. At a Clinical Examination Research Group meeting held in October 2021, the group discussed modifying the direction of the project and recognised a substantial potential of investigating examiner decision making in the online environment. While the online examinations will continue in 2022 they are also expected to form a part of the AMCs clinical examination directions in the future.
Approval has been granted for the Clinical Examination Research Group to resume the Examiner Decision Making project from early 2022, to study examiner decision making in the online environment.
Australian Medicine In Context Publication
The Anthology Editorial Group, led by Editor-in-chief and Chair, Professor Tim Wilkinson, are progressing with the re-write of the AMC Anthology publication. The editorial group was established to review the 2003 publication with a focus on ensuring the content remains current and reflects medicine in the context of Australian practice. The Anthology forms the basis of all examination content.
The Anthology Editorial Group adapted a new name for the previous Anthology, with the new title confirmed as “Australian Medicine in Context – key presentations and topics for entry to practice”.
An important consideration to note involves the review of Aboriginal, Torres Strait Islander, and Maori related content. The content relating to Aboriginal and Torres Strait Islander people has been written by individuals who are respected Aboriginal medical education professionals. In addition to the respected writers, the editorial group has placed a strong importance on engagement with Aboriginal and Torres Strait Islander people for guidance and peer review of the related content.
This publication will also focus on patient presentations, with less emphasis on medical conditions and a strong emphasis on professional practice.
The ongoing work will now include iterations of editing and indexing, graphic design, marketing, and financial modelling before publication. As the publication forms the basis of all examination content, a new classification system of examination content is being designed and implemented to align with the MCQ item bank and clinical examination scenarios.
An additional stream of work will align the AMC MCQ questions and Clinical scenario banks to the new publication which will now serve as the basis of the AMC blueprints.
This project has been handed over to the Strategy, Policy and Research team under the leadership of Ms Theanne Walters (DCEO), whilst the Assessment team will continue to collaborate with the Accreditation Section.
AMC APP
The AMC currently publishes a Handbook of Multiple Choice Questions to assist international medical graduates in preparing to take the AMC MCQ examination. The AMC Mobile Practice Test (App) is a project aimed at digitising the publication into an interactive mobile application. The App will utilise archived MCQ questions to create a study resource for international medical graduates and medical students preparing for examinations.