Clinical evaluation is ongoing during the life cycle of a medical device.
Sponsors can use many different types of evidence to demonstrate performance and safety of medical devices. These types of evidence can include traditional clinical investigations as well as observations through real world practice
Clinical evidence for medical devices can include data regarding the usage, or the potential benefits or risks, of a therapeutic good derived from sources other than traditional clinical trials, including real world evidence and patient reported outcomes.
This data is part of the clinical evidence for the medical device and can be used to support a premarket application or as part of post-market monitoring as the TGA can accept real world observational data in conjunction with other data if it demonstrates sufficient evidence to meet the essential principles,
Sponsors are required to provide ongoing evidence to support their device, including identification of any new safety, clinical performance or effectiveness information identified during its use.
Manufacturers should periodically review performance, safety, and the benefit-risk assessment for the medical device through a clinical evaluation. The clinical evidence should be updated accordingly, using data generated from programs such as adverse event reports, published literature, further clinical investigations, and real world evidence.
Important aspects of real world evidence to consider include the type of evidence it can provide. It can reflect a scale of outcome, a time in point of clinical practice, or the impact of a product on the healthcare system. However, the context in which the data is viewed and analysed as well as potential confounding details can impact on the quality of the evidence.
A critical evaluation is then undertaken to provide supportive clinical evidence for a therapeutic good.
The quality of the data assessment is important to the applicability of outcomes, for example:
- percentage of data collected per subject
- range, median
- degree of missing data resulting in exclusion and number of subjects affected
- useability, health literacy factors
- calibration aspects if applicable
- RWD independent of training data sets.
In the post-market phase, manufacturers are expected to implement and maintain surveillance programs that routinely monitor the safety and review the performance and effectiveness of the medical device as part of their Quality Management System. Real world evidence can provide valuable data for ongoing device performance and safety.
Detailed advice on the use of clinical evidence, including preparing a clinical evaluation report, is provided in the Clinical evidence guidelines: Medical devices.
Types of clinical experience data and real world evidence
Device registries
Device registries are systematic data collections of medical outcomes following use of medical devices. They play a unique and important role in medical device surveillance providing impartial de-identified data about procedures and devices not regularly collected by other means. They can also give valuable information on device performance, including clinical or functional outcomes and quality of life.
The Australian Commission on Safety and Quality in Health Care provide a Framework for Australian clinical quality registries. They also developed the Australian Register of Clinical Registries, which lists the purpose and organisation of clinical registries at all stages of development.
Some are dedicated medical devices registries while other registries may include data on related medical devices.
Examples of Australian medical device registries include:
- Australian Breast Device Registry (ABDR)
- Australian Orthopaedic Association National Joint Replacement Registry (AOANJRR)
- Australasian Pelvic Floor Procedures Registry (APFPR)
- Victorian Cardiac Outcomes Registry (VCOR)
- Dental Implant Registry (DIR)
Powerful potential of registries
In late 2009 the DePuy ASR Metal on Metal hip implants were withdrawn from the Australian market after analysis of the AOANJRR showed the rate of early revision surgery for these implants was high compared to other hip replacements. The AOANJRR demonstrated the power of registries as it:
was the first to identify that the ASR was a prosthesis that was associated with a higher than anticipated revision rate and this led to the prostheses being withdrawn in Australia in 2009 almost a year earlier than the worldwide withdrawal
Other clinical experience data
Examples of other clinical experience data:
- Electronic Health Records (EHRs)
- Insurance company data, including claims and billing activities
- Product, disease, quality and procedure registries
- Patient-generated data including in home-use settings
- Data gathered from other sources that can inform on health status, such as mobile devices.
The FDA document Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices provides further guidance.
Overseas regulatory approvals
Systematically evaluated data collected from post-market surveillance in other jurisdictions may also be useful in the Australian context. Such data can be included in the clinical evidence.
Observational and epidemiological studies
Observational studies do not intervene (unlike clinical trials) but rather observe health outcomes, often in populations (i.e., epidemiology). Such studies may be relevant to medical devices as they provide data from often larger populations or patient groups.
Useful resource
The US FDA have published a commentary, Randomized, observational, interventional, and real-world—What's in a name?, on the potential use of observational studies for regulatory decision making.Typically, evidence from publications would not be considered real world evidence since it is in summary form. Where a clinical study underpins the publication, the expectation is that all the underlying clinical data from the study would be available.
Post-market data
Post-market data may be collected by manufacturers, sponsors, regulatory agencies, registries, or others. This data is a type of real world evidence. It is important that a sponsor provides all available post-market data in submissions to the TGA.
Post-market data may be for the specific device, or a claimed equivalent device. If the data is about an equivalent device, then evidence about differing features, functionality, components, biocompatibility, etc, of the device must be provided. As more data for the device itself is available over time, less post-market data for equivalent devices should be used. Eventually, only data for the device itself should be used.
Post-market data can, and should if available, be used to support the safety and performance claims of the device. It guides risk identification, assessment, and mitigation of both known and emerging hazards. Post-market data is important when the number of patients exposed through routine clinical usage exceeds those studied in clinical investigations, or newly identified risks may alter the benefit-risk profile for the device.
Emerging hazards
Sometimes new risks emerge or are identified after pre-market approval, when more patients have been exposed to the medical device after release into the market. The identification of long term harm associated with urogynaecological mesh and breast implant associated-anaplastic large cell lymphoma (BIA-ALCL) are examples of emerging hazards.
Other examples of real world data:
- Annual reports of post-market data for implantable Class IIb devices; Class III devices (including active implantable medical devices (AIMDs)) and Class 4 in vitro diagnostic (IVD) devices.
- Adverse events reportable to the governing bodies of the countries when a medical-device-related event leads to or potentially leads to death or serious injury.
- Information about recall actions and suspension or cancellation of marketing approval (in any jurisdiction).
Post-Market Clinical Follow-up (PMCF) studies can be used to collect more clinical data. They often rely on systemic collection and evaluation of real world data.
Potential of Unique Device Identification (UDI)
The TGA is implementing a Unique Device Identification system for Australian medical devices. When fully implemented, the label of most devices will include a UDI in both a human and machine-readable form (such as a barcode). Globally harmonised, core data about those devices will be publicly available through the Australian UDI Database (AusUDID).
This is expected to provide additional benefits in relation to collection of real world evidence. This includes:
- an improved ability to identify models of devices nationally and globally
- improved ability for data sharing across regulators and the healthcare industry
- enhanced research and analysis through the uniform documentation of devices in electronic health records, clinical information systems, clinical quality registries and other data stores.
More information about UDI’s is available on the dedicated UDI hub.
International guidelines
Post-Market Clinical Follow-Up Studies (IMDRF/MDCE WG/N65) provides information on examples of, and considerations, when using clinical experience data, and Clinical Evidence – Key Definitions and Concepts (IMDRF MDCE WG/N55), Clinical Evaluation (IMDRF MDCE WG/N56) and Clinical Investigation (IMDRF MDCE WG/N57) may also be informative.
The US FDA have also published useful guidance on Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices and Randomized, observational, interventional, and real-world—What's in a name?
Digital, software and artificial intelligence implications
Real world evidence from software has the potential to augment clinical trials and complement traditional clinical evidence. A key question is when to use synthetic data or real world evidence, or when a hybrid approach is optimal.
The use of for digital devices is not a suitable substitute for testing the functionality and other aspects of the product (such as resilience, error handling, fault tolerance, or performance). Technical and analytical validation of software is still required to demonstrate safety and accuracy.
Considerations of circumstances that best fit real world evidence include:
- type of data
- source of data
- structured vs unstructured data
- gaps or mismatches in data
- fit of the data to the scope of the software.
Although validation evidence sourced from real world evidence may be available in high volumes, it may sometimes be incomplete or partial, biased, contain gaps or represent only the most exercised use cases or functions. This may make it difficult to demonstrate reproducibility of the results generated using real world evidence.
Real world data and software
Data may be sourced from applications, sensors, IVDs, EHRs, general practice software, telehealth, or other sources such as consumer devices. Large data sets can be sourced in this way; however, they are often difficult to join accurately with data from other sources due to a lack of consistency in standardising data items.
Existing obligations under the medical device regulatory framework, including the essential principles, still apply for software-based medical devices for safety and performance, and in addition there are software specific requirements such as cyber security and version control – the software-based medical devices webpage has details about these obligations and links to guidance.
Specific real world data issues that need to be considered by sponsors and manufacturers in sourcing, storing, and using the data:
- Data governance framework
- Sourcing, collection and how the data is used, feeds into analytics
- Storage - on devices, cloud, which jurisdictions, copies
- Third party access - Application Programming Interfaces (APIs), on-selling
- Security
- End of life – disposal, retirement, porting
- Other obligations such as privacy and consent
Post-market use of real world evidence for software
Real world evidence may be used to:
- Give further external validation of devices to support generalisability of model performance, i.e. ability to perform in a new use environment or a new sample of patients.
- Assist in expansion of indications or claims for a product that is already in the market, provided that the study design is appropriate to the claims.
- Track end points beyond model accuracy, for example mortality, rate of Intensive Care Unit admission or other patient outcomes.
Key Issues
Confidence in the accuracy of the real world evidence is paramount and, in the case of Artificial Intelligence, this means transparency of data generation, sources, labelling and characteristics.
Variability of data output from different devices, across different uses on the same or similar patients or conditions, and by the same or different operators, may make the data difficult to interpret or to check the accuracy. This may impair the ability to use the data as evidence.
If you are aware of emerging opportunities around real work evidence and patient reported outcomes, we would like to hear from you. Please contact us through the Medical Device Information Unit