Sub menu

 

 

Evidence requirements for software using AI

Last updated

Certain requirements must be met to include a product in the Australian Register of Therapeutic Goods (ARTG). This includes demonstrating compliance with the Essential Principles. This page outlines requirements for medical devices that use Artificial Intelligence (AI) and machine learning.

Manufacturers (namely, the developer, organisation or individual responsible for development) of software medical devices that include AI must have evidence of how the device complies with the Essential Principles.

This evidence must be sufficiently transparent to enable the evaluation of the safety and performance of the product, and must include:

  • what the Artificial Intelligence/Machine Learning model is doing and how it contributes to the intended purpose of the device.
  • a description of the algorithm and model design including information on the training and testing phases of the AI model and how this relates to the intended purpose.
  • information about the data used for training and testing:
    • a statistically informed justification for dataset sizes.
    • a description of the populations represented in the data, for example, age, gender, and ethnicity.
    • a justification for how this data would be generalisable or appropriate for the Australian population and sub-populations for whom the AI is intended to be used, in the context of the relevant Australian clinical practice standard for which the device is used. Independent global draft consensus standards could provide a basis for structuring this information.
  • risk management evidence to show how risks have been addressed including AI specific risks such as (but not limited to) overfitting, bias and performance degradation such as data drift.
  • evidence to show how the device meets requirements for clinical evidence.

Evidence requirements continue through the product lifecycle and include robust post-market monitoring practices to ensure continued device performance and model accuracy.

Pre-market requirements for software using machine learning

We have adopted the International Medical Device Regulators Forum (IMDRF) guidance ‘Good machine learning practice for medical device development: Guiding principles’.  Sponsors can apply this guidance to compile evidence of compliance with the Essential Principles.

The guidance specifies 10 guiding principles that address the data-driven nature of AI use in medical devices over the lifecycle of the product.

Table 1 summarises the key elements of the guiding principles. Sponsors are encouraged to review the guidance in full prior to submitting an application.

Table 1: Key Good Machine Learning Principles
Software Lifecycle ComponentGood Machine Learning Practice to be addressed
Design
  • The intended purpose and device description should enable understanding of the device’s use within the clinical workflow, including the desired benefits and associated patient risks.
  • Model choice and design are evaluated and shown to be suited to the available data and support the active mitigation of known AI specific risks, like overfitting, performance degradation, and security risks.
  • Considerations include the impact on both the overall intended patient population and its subgroups as well as uncertainty and variability in the device inputs, outputs, and clinical use conditions.
Data management
  • Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including factors related to patients, sites, and data acquisition, are considered and addressed to assure independence.
  • Use accepted methods to develop fit-for-purpose reference standards – i.e. the expected result determined via a predefined, reproducible process (for example, consensus of qualified clinical experts using standardised criteria, adjudication with a third reader, or pathology/laboratory confirmation).
  • Document the rationale for the choice of reference standards based on the device’s intended use/ intended purpose and assessment of their suitability and limitations to address the intended use environment, to ensure clinically relevant, well-characterised data.
  • If available, accepted reference standards in model development and testing that promote and demonstrate model robustness and generalisability across the intended patient population are used.
  • Practices are adopted to support the rights, safety, and welfare of patients, including through the ethical use of patient data.
Development and training
  • Good software engineering, medical device design, and security practices are implemented throughout the total product life cycle.
  • Model design is implemented and maintained with attention to the fundamentals:
    • robust software engineering practices
    • usability
    • data quality assurance
    • data management
    • cybersecurity
    • quality management practices.
  • Design processes are adopted that can appropriately record and communicate decisions and rationale, as well as ensure traceability, reproducibility, data authenticity, confidentiality, integrity, and availability.
  • The infrastructure needed for model deployment, monitoring, and maintenance is documented.
Testing and evaluation
  • The performance of the device is assessed in the context of the intended use environment and clinical workflow, considering interactions with healthcare providers, consumers, and caregivers where applicable. The intended use and clinical workflow assessed includes the Australian context.
  • Human factors considerations are addressed, including for example, user skills, user expertise, user understanding of the model outputs and limitations, potential for overreliance, level of device autonomy, and user error, for normal use and reasonably foreseeable misuse.
Clinical validation
  • Clinical evaluation includes the use of datasets that are representative of the intended patient population:
    • data collection protocols aim to ensure that the relevant characteristics of the intended patient population (for example, in terms of age, gender, sex, race, ethnicity, geographical location, medical condition)
    • intended use environment
    • measurement inputs are sufficiently represented in a sample of adequate size in the datasets for training, testing, and monitoring so that results can be reasonably generalised to the intended Australian population.
  • The clinical evaluation report should include test plans and resulting data to demonstrate clinically relevant device performance addressing:
    • intended patient population
    • relevant subgroups
    • clinical environment
    • use by the human-AI team
    • measurement inputs
    • potential confounding factors.
  • The extent of external validation is proportionate to risk.
Transparency (including labelling)
  • Users are provided clear and contextually relevant information including:
    • the product’s intended purpose and indications for use, benefits and risks
    • performance of the model for appropriate subgroups, the study methodology, characteristics of the data used to train and test the model
    • acceptable inputs, known limitations, user interface interpretation
    • clinical workflow integration of the model
    • basis for model output (to the extent possible).
Risk management
  • In the context of the intended use within the clinical workflow, associated patient risks are well understood.
  • Methodical risk management practices are implemented throughout the total product life cycle.
  • Data is collected to manage any unintended bias or dataset drift, and to identify circumstances and subgroups where the model may underperform including over time.
Post-market monitoring
  • Monitoring plans should demonstrate risk-proportionate ongoing monitoring of deployed models in “real world use” to ensure maintained safety and performance.
Product types