Favouring quality improvement initiatives: the experience of the Belgian College of Radiation Oncology

Introduction

Background

Radiation therapy (RT) has an integral role in multidisciplinary cancer treatment in a curative setting, e.g., in a radical approach for organ preservation or as a (neo-) adjuvant approach before or after other oncology treatments. It also has an important benefit in the palliative setting and to a lesser extend in the treatment of non-malignant diseases (1). The accurate delivery of a radiation treatment is of primordial importance in radiation oncology, particularly as patients benefit from ever shorter treatment approaches delivering high doses to the target volume, a strategy referred to as hypo-fractionation (2). This requires conformal and intensity modulated treatments to minimize the dose delivered to surrounding organs at risk. Hence, all possible measures should be put into place to ensure the safe and effective delivery of those treatments (3,4).

Rationale and knowledge gap

The European Council Basic Safety Standards Directive (BSSD) Directive of 1997 (Council Directive 97/43/Euratom) required that Member States take “all reasonable steps to reduce the probability and the magnitude of accidental or unintended [radiation] doses of patients” in radiotherapy. This is further detailed in the more recent 2013/59/Euratom directive (5). This directive requires that member states establish incident reporting and learning systems, carry out prospective risk analysis of all the processes involved as part of their quality assurance (QA) programme, as well as take all the steps necessary to ensure radioprotection of exposed citizens in terms of justification, optimisation and dose limitations. In addition, the directive states that departments need to carry out clinical audits “in accordance with national procedures” in order to assess good clinical practice. As such, European member states have the obligation to translate these requirements into their own legal framework. However, the extent to which this is actually converted into practice in the different countries varies greatly. A survey on carried out by the European Society of Radiology on BSSD compliance showed highly variable responses in between radiology departments and countries and this even more so following the COVID-19 pandemic (6). The European commission’s QuAdrant [Quality Improvement Through Clinical Audit in Diagnostic (including Interventional) Radiology, Radiotherapy and Nuclear Medicine including Therapies] project aims at promoting and enhancing BSSD 2013/59/Euratom compliance, and supporting clinical audit practice. Its work packages have also demonstrated variables degrees to which radiology and radiotherapy departments have implemented the requirements with a limited number of countries such as the UK, Finland, Switzerland, Norway and Belgium sharing their successful implementation of clinical audits (7).

In May 2000, the Belgian government established the College of Physicians for Radiation Oncology Centres (further referred to as ‘the College’) with the aim of evaluating and improving quality of radiotherapy. Eight radiation oncologists are appointed by the Minister of Health for a term of 6 years. A balance is sought between men and women, academic and non-academic, Flemish and French speaking members. The members are supported by a working group of experts, consisting of medical physicists, radiation therapists (RTT), radiation oncologists, quality managers, and, upon demand, delegates from the ministry and from the Belgian Cancer Registry. The missions of the College are financially supported by the Belgian Cancer Plan in addition to the College receiving operating money from the Belgian Ministry of Health.

Objective

Over the past decades, the College has been successful in developing and implementing a number of quality improvement initiatives, focusing on different aspects of radiotherapy practice (Figure 1). From a clinical point of view, contouring guidelines have been published, followed by national intervision projects, important to optimise and homogenise radiation treatment preparation. Covering the entire radiotherapy process, national peer review clinical audits are undertaken, based on the International Atomic Energy Agency (IAEA) Quality Assurance Team for Radiation Oncology (QUATRO) methodology (8). In parallel, the College also supports external dosimetry audits to ascertain the correctness of the delivered dose, per beam and for various complex techniques. In addition, a structure to analyse, report and benchmark radiotherapy events has been installed. Finally, the College has also defined a set of structural and patient-related process and outcome quality indicators (QIs), which have been collected and analysed on a yearly basis to serve as a reference for all participating centres.

Figure 1 Timing of the different cited projects of the College. QUATRO, Quality Assurance Team for Radiation Oncology; BELdART, Belgian Dosimetry Audits in RadioTherapy.

The objective of this paper is to describe these various quality-oriented projects. Each of these initiatives have the common aim of identifying areas of possible improvement in the delivered care, and defining collective improvement actions through sharing of experience and benchmarking.

Intervision projects for target volume delineation

Optimal target volume definition and delineation is a prerequisite to ensure optimal radiation treatment and benefit from technical improvements. To ensure consistent target volumes, persistent efforts are needed from the radiation oncology community to provide detailed contouring guidelines and hence reduce interobserver variability.

Because of the great diversity between the different hospitals in Belgium, both in the treatment of rectal tumors and their outcome, a national platform, PROCARE (PROject on CAncer of the Rectum) was set up by a multidisciplinary working group to advocate for standardization and quality control in rectal cancer care. In 2007, the PROCARE group outlined a number of national guidelines with recommendations for diagnosis, treatment and follow-up (9). In a next phase, about forty QIs were formulated in order to be able to test the quality of care (10). An accurate analysis of the registered data made it possible to provide feedback to the various hospitals in a constructive manner.

In addition to mapping the quality of care and offering “best clinical practice” guidelines, the PROCARE platform also offered concrete assistance in implementing these guidelines through training programs provided to surgeons as well as online review platforms such as PROCARE RX in which the radiologists could anonymously have their findings on computed tomography (CT) and/or magnetic resonance imaging (MRI) revised by an expert radiologist. Besides activities in the diagnostic and surgical setting, a central review platform for clinical target volume (CTV) delineation was launched, in an attempt to improve delineation uniformity in rectal cancer radiotherapy (11). Between March 2010 and September 2012, 20 centres submitted 1,255 rectal cancer cases, of which 1,224 were analysed. A high level of agreement on CTV delineation was observed between the participating centres and the central review facilities from the beginning of the project. Moreover, the instant feedback by e-mail gave the centres the opportunity to pick up the reasons for the suggested modifications. This resulted in a significant improvement in CTV delineation and reduction of interobserver variability across participating Belgian radiation oncology departments.

In 2013, a national QA project for breast radiotherapy was started, named PROCAB (PROject of CAncer of the Breast). Guidelines for the delineation of the regional lymph node areas (RLNAs) were published, approved on a national and European level (12) In contrast to previous experiences, the guidelines were based on the anatomy of the blood vessels, rather than on bony structures, which made them more patient-tailored, precise and applicable, independent of the treatment position. The delineation guidelines provided a meticulous description, supported by a clear delineation atlas using CT-images.

Subsequently, the project investigated the effect of central review on the interobserver variability and quality of the delineation of the RLNAs (13). In total, 1,009 CTVs from 23 RT departments were centrally reviewed and scored. A significant decrease in errors over time (learning curve) was observed, supporting the benefit of a central review on the conformity to the guidelines.

After these rectal and breast cancer projects, ProCaLung is now being introduced with the aim to improve the quality of mediastinal node positive non-small cell lung cancer radiotherapy through peer-review of mediastinal nodal target Volume Delineation (14). This project is discussed extensively in another article in this issue.

In 2017, the PROCAHN project (PROject on CAncer of Head and Neck) was supported with the goal to map the interobserver variability in tumor delineation in the head and neck region between the different radiotherapy centers in Belgium (15). Since availability of guidelines did not guarantee uniform delineations, the investigators recently set up a subsequent study, where the implementation of most recent delineation guidelines by Grégoire et al. [2018] on interobserver variability will be assessed (16).

The precision of radiotherapy treatment execution relies on the accurate delineation of target volumes. The College’s intervision projects have, as such, actively participated in diminishing the variability of target volume delineations all the while ensuring the dissemination of guideline-based good practice. Ideally, this type of initiative should be instigated for all radiotherapy indications but also for all major updates in clinical guidelines.

B-QUATRO peer-reviewed clinical audits

Clinical audits is a process in which actual clinical practice is compared to standard of care in order to identify potential areas of improvement leading to better quality of care. More particularly, peer-reviewed external clinical audits consist of the review of actual practice by expert individuals that are professionally active in the field. Clinical audits lead to the identification—by the auditors—of areas of practice needing to be improved (5). As described by Scalliet et al. [2015], since 2011, the College organizes peer-reviewed clinical audits in all Belgian radiation oncology departments based on the IAEA QUATRO tool, quality criteria and philosophy (17-19). The multidisciplinary auditing team is composed of a medical physicist, a radiation oncologist and an RTT, delegated by the Belgian radiation oncology departments. The auditors are all professionally active members who have been trained in the audit methodology and who perform this task on a voluntary basis, with a budget only foreseen to cover for travel and accommodation. The first audit cycle was initiated in 2011: 5 departments being audited every year, resulted in the successful auditing of all Belgian radiation oncology departments over 5 years. All departments actively collaborated in the process by accepting the audits.

In partnership with the IAEA, a modified version of the QUATRO tool, named B-QUATRO, was introduced in 2015 to further fine-tune the auditing tool for the Belgian context and to integrate concepts related to newer technologies. It also included a chapter focusing on quality criteria defined to evaluate quality management systems. As a result, a quality manager was also integrated in the auditing team (20,21). Since 2017, a second cycle of audits has thus been initiated, using the B-QUATRO document, this time with a team of 4 auditors. This second cycle of audits should be finalized by beginning of 2023, the coronavirus disease 2019 (COVID-19) pandemic having led to a one-year pause of the audits.

Following each audit, the audited department is provided with a written report covering a set of recommendation emitted by the auditors. This will assist departments in setting up quality improvement initiatives. Anonymized audit reports are presented to the College, at yearly auditors’ meetings, and via the College report to the health ministry.

Following the first cycle of audits, departments were surveyed in order to evaluate the relevancy and impact of the emitted recommendations (20). The survey revealed that clinical audits are used by departments in a constructive way to identify areas in need of improvement. Furthermore, the audited and the auditing team both benefit from this peer review process through in-depth exchange of experience and good clinical practice, and this, for the benefit of both the patients and the teams.

Peer reviewed clinical audits established in Belgium since 2011 and have thus shown to be a constructive tool for quality improvement though the identification of elements or processes needing to be improved but also through the communication and exchange of best practice between auditors and auditees. The continuity of these valuable audits relies on the existence of active and trained auditors and entities that formally support the organization of these audits.

National dosimetric audits—Belgian Dosimetry Audits in RadioTherapy (BELdART)

The success of radiotherapy is also impacted by the correctness of the delivered dose. Independent dosimetry audits can be conducted to identify eventual discrepancies in the dose delivered by the clinical equipment versus the dose that is expected to be delivered and they can provide support in the solution of identified discrepancies. In this way, continuous improvement in the accuracy of dosimetry can be brought into daily practice and severe failures can be prevented. To make access to dosimetry audits easily available to all Belgian RT centers, the Belgian Cancer Plan supports another action: BELdART based on alanine/Electron Paramagnetic Resonance (EPR) and radiochromic film dosimetry (22-28). A steering committee composed of five medical physicists functions as advisory body.

Since 2009, BELdART has been developing audit programs for basic dosimetry for photon and electron beams. This beam calibration audit is basic and fundamental, as it affects all patients for any specific treatment machine. Since 2016, BELdART also offers audits for various complex techniques. In the proposed so called end-to-end (E2E) tests, an anthropomorphic phantom is put through the entire chain of procedures that a patient would go through including simulation, planning and treatment delivery.

As a consequence, current services offered by BELdART to the Belgian RT departments are two-fold.

First, beam output checks, performed in reference conditions for photon and electron beams are performed, along with the possibility for more measurements in non-reference conditions for photon beams using alanine/EPR dosimetry (29). A second part consists of E2E tests executed in anthropomorphic phantoms for intensity-modulated radiation therapy (IMRT), prostate volumetric-modulated arc technique (VMAT, intracranial Stereotactic Radiation Surgery (SRS) and lung stereotactic body radiation therapy (SBRT) using a combination of alanine/EPR and film dosimetry (27,28). Between December 2016 and November 2021, the beam output has been measured for 116 beams and 86 of those beams have received a complete verification in water. E2E audits have been performed for prostate treatment for 55 beams, while 26 beams for cranial SRS and 14 beams for SBRT have been evaluated. Due to an increase in the request of audits for flattening filter free (FFF) beams, the configuration of the alanine detectors had to be adapted. As a general observation, the measurements done by BELdART did not deviate from the stated calculated dose by more than 5% for any center. For the 116 Beam Output audits, 90% was within 2% of the stated dose. For the 86 beams that were fully verified in water, 84 beams had an agreement within 5%. For 2 tests in small fields, the difference with the stated dose was just slightly above 5% and it is still unclear what the contribution of the positioning uncertainty is in those small fields. For the E2E tests, in the prostate, the measurements for 87% of the centers were within 3% of the calculated dose, and all were within 5%. In SRS E2E tests, 77% of the participating centers could deliver the dose within 3% and all centers within 5%. The E2E audit for SBRT is still new but of the 14 participating centers, 13 had measurements within 3% of the calculated dose and all were within 5%. The results for the film measurements are confirming these numbers.

The dosimetric validation of the dose measured by the alanine detectors has been performed by the IAEA using the BELdART material and showed results within experimental uncertainty for all clinical MV energies. The BELdART team also conducts R&D to develop future audit procedures for additional treatment techniques. This includes projects regarding the development of movable phantoms (in collaboration with Maastro, NL), the use of alanine in proton therapy [in collaboration with Studiecentrum voor Kernenergie (SCK) or in Ultra-High Dose Rate (FLASH)-RT (in collaboration with UAntwerpen] (30).

The accurate delivery of radiotherapy highly relies on the preciseness of the dose delivered by the equipment in place in radiotherapy departments. The external dosimetric audits organized though BELdART have allowed to provide an objective independent evaluation of the dose delivered by Belgian radiotherapy equipment all the while having to stay in line with the new treatment modalities that have become available in the recent years.

A national event reporting system—PRISMA-RT.be

As part of the requirements for the Belgian Cancer Plan, a national event reporting system had to be set up. The main focus of this system is quality improvement by inter-centre comparison of the causes leading to near-misses and adverse events. The system acts complementary to the already existing mandatory declaration of accidents to the Federal Agency of Nuclear Control (FANC), the national competent agency that, amongst others, governs all medical uses of ionizing radiation. For such a system of inter-comparison to succeed, a common methodology is needed across all radiotherapy centres.

The PRISMA (Prevention, Recovery and Information for Monitoring and Analysis) analysis methodology was developed in 1992 by T. W. van der Schaaf at the Technical University of Eindhoven initially for the chemical process industry, but shortly thereafter adapted for use in the medical field (31,32). At its core a fault tree analysis methodology, which distinguishes itself by also focusing on near miss reporting and the various factors leading to human behavioural error. Its function in an effective safety management system is the identification of root causes underlying near-misses and accidents, an essential feedback mechanism that supplements Healthcare Failure Mode and Effect Analysis (HFMEA) and other proactive process risk analysis techniques.

Identified root causes are categorized with the Eindhoven classification model (ECM) of system failure. It distinguishes between technical, organizational, human behavioural and external root causes that can affect a human operator. A simplified version of Rasmussen’s skill-rule-knowledge model of human errors is implemented in the human behaviour root cause subcategory (33). The ECM is a goal-directed classification, aimed at effective mitigation and prevention, not at attributing guilt or blame. Root causes with different classifications will typically have a different type of corrective action associated with them. Examples of such actions are technical changes, reviewing of procedures, information sessions and additional staff training.

The emphasis on human factor engineering principles led to PRISMA being adopted in 2008 as methodology of choice for the Dutch radiotherapy quality improvement organization PRISMA-RT, initiated by Petra Reijnders and Huub Backes of Maastro clinic (Maastricht, The Netherlands) and Anne Joustra, Catharina hospital (Eindhoven). This project deployed a web-based analysis application and a benchmark application, developed by TPSC (Alkmaar, The Netherlands), to all 17 participating radiotherapy centres. The primary goal was to improve radiotherapy processes and patient safety by an inter-centre comparison of root cause profiles. This is supported by regular training sessions to standardize the use of the analysis methodology. In addition, the PRISMA methodology was augmented by including radiotherapy-specific context variables to the root cause classifications. Context variables serve as tags to the otherwise unqualified ECM categories, for example highlighting a specific technique or the process involved in a root cause. The context variables allowed the organization to use the benchmark to steer specific projects (34).

In 2010, the College started PRISMA-RT Belgium. Following the Dutch example, a network was created of radiotherapy quality managers and the TPSC applications were made available to the participating centres. The context variables were extensively reviewed and adapted to the Belgian radiotherapy landscape. A similar programme of training sessions was started. During the lifetime of the project, an interface to the benchmark was developed by TPSC so users of third-party software would be able to contribute. At this time, one additional vendor, Infoland (Veldhoven, The Netherlands), has added the necessary functionality in their software.

Currently, the majority of radiotherapy centres are participating in this network. The experience shows that establishing a common methodology of analysis has served as a uniting factor in the Belgian radiotherapy quality management community. It acts as a common language to discuss projects and issues during biyearly meetings and training sessions. In addition, PRISMA-RT has been adopted as the analysis methodology for the mandatory declaration of accidents. These analyses are shared anonymously with all radiotherapy centres by the competent authority FANC.

The PRISMA-RT methodology of incident and near-incident analysis has thus been successfully implemented in all radiotherapy departments. Surveys among the quality managers indicate that the analysis of near-misses with PRISMA-RT together with regular meetings and training sessions are experienced as valuable driving forces in quality improvement projects. The intrinsic utility of the benchmark is currently being evaluated. An extensive analysis is in progress of the approximately 20,000 root cause classifications collected in the database.

National radiotherapy-specific QI project

In the quest to offer the best quality of care to RT patients, it is of uttermost importance that departments assess the actual performance or quality levels attained and compare these to expected or desired outcomes.

QIs are measurable and objective elements or data points that can be used as guides to evaluate and improve overall quality of patient care. More often than not, these QIs are developed using the Donabedian model in which the QI covers either the structural dimension or measures actual processes and outcomes (35,36). A clear definition and continuous collection of expert approved QIs allows for the monitoring of quality performance over time. Even more, centres or countries can compare their results with others through the process of benchmarking if QIs are collected on a multicentric or multi-national basis (37-41). Through this process, it is possible to potentially identify disparities between the QIs that are collected and what is considered standard of care or optimal quality of care. The identification of those gaps can then lead to implementation of improvement actions aiming at minimising those gaps and optimising overall quality of care. While a QI program may be developed at a local level (department or institutional level), it may also be driven by a regulatory demand and/or be initiated by a scientific or professional body, with the ultimate goal to favour quality improvement.

With this objective, a collective work—bringing together the College and the representatives of the Belgian Quality Managers in radiotherapy Association (QMRT.be)—established a list of radiotherapy-specific structural, process and outcome indicators, considered as fundamental. The process and outcome QIs focused on 3 pathology groups: head and neck, breast and prostate cancer patients. Once the QIs were defined and agreed upon, a test phase was launched in June 2015, collecting the full set of structural QIs for 2015, but limiting the patient-specific QIs to a restricted number of patients. Once the data collection test phase was validated and optimized, the capture of the defined QIs was initiated in 2016 and repeated on a yearly basis with an overall department participation of 95%.

This voluntary and yearly QI data collection has given rise to individualized benchmarking reports that are sent out to the participating centres. Although the overall Belgian landscape of radiation oncology departments is quite harmonious, comparison of QI data between departments has allowed the identification of areas for improvement and has favoured departmental and national quality improvement projects such as changes in treatment techniques used (i.e., use of breath hold for breast cancer patients) or decreased delays between simulation and start of treatment. The yearly collection of QIs has also allowed the monitoring of trends such as the number of equipment, estimated workload per professional group or the number of radiotherapy treatments over time, and has shown to be useful to monitor the impact of COVID-19 on the Belgian radiotherapy practice, as was also done in other countries across Europe (42-46).

As such, through this College initiative, it has been shown that it is feasible to collect defined QIs at a national level, and that such a project can boast of a very high participation rate. It is the hope of the College to be able to collect and analyze QIs at a larger scale by including more patient specific QIs and increasing the number of patients for which this data is collected. This can be favoured through tools that would allow for the automation of data extraction from existing radiotherapy information systems or hospital electronic health records (47-49). This path is currently investigated in the Belgian landscape along with the national collection of Patient Reported Outcome measures (PROMs) (50).

Conclusions

For Radiation Oncology, the Belgian government established the College of Physicians for Radiation Oncology Centres 20 years ago with the aim of evaluating and improving quality of radiotherapy. The 8 radiation oncologists, appointed by the government, are supported by a working group of experts, consisting of medical physicists, RTT, radiation oncologists, quality managers, and, upon demand, delegates from the ministry and from the Belgian Cancer Registry. In the recent decade, radiation oncology has become more and more complex requiring thorough QA programs and continuous quality improvement initiatives.

In the last 15 years, the College has been successful in implementing national quality improvement projects. These include the implementation of peer-reviewed target volume contouring programs, clinical audits, external beam dosimetry audits, national adverse event analysis systems and RT-specific QIs.

Contouring programs have shown that clear guidelines, continuous education of professionals and extensive quality control can reduce the interobserver and intercenter variability in target delineation. Clinical audits are used by departments in a constructive way to identify areas in need of improvement and are a way of exchange of experience and good clinical practice.

Dosimetric audits can support the implementation of new, more complex techniques and because of the national basis with feedback to all centers, it is possible to benchmark with other centers with similar equipment. The experience with our national event reporting system showed that establishing a common methodology of analysis can serve as a uniting factor, facilitating discussions. Finally, it has been shown that it is feasible to collect defined QIs at a national level, and that such a project can boast of a very high participation rate leading to change in organisational and clinical practice.

The establishment of a QA program and quality management system is a necessary step that all radiotherapy departments should take in order to ensure the optimal and safe delivery of radiotherapy treatments. This is particularly important in the context that radiotherapy is potentially a high-risk procedure relying on the accurate delivery of high doses within a complex environment. Patients undergoing a radiation treatment must be able to trust that they are treated optimally and in this way have the lowest chances of treatment morbidity and toxicity and the best chances of local tumour control, survival and quality of life. The creation of national quality-oriented initiatives can, as such, serve as guides that can help and accompany departments in ensuring that this is the case. The existence of the College—composed of professionally active RT members—has been a necessary structure that favoured the implementation of these in Belgium. The existence and sustainability of entities such as the College is thus of uttermost importance if these types of initiatives are to be maintained.

Acknowledgments

The authors would like to thank all Belgian radiotherapy departments for their involvement in the various quality improvement initiatives.

Funding: This work was in part supported by the Belgian Federal Government for Public Health through the reimbursement of operating costs.

Footnote

Provenance and Peer Review: This article was commissioned by the Guest Editors (Paul Van Houtte and Dirk Van Gestel) for the series “Quality Assurance in Radiotherapy” published in Precision Cancer Medicine. The article has undergone external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://pcm.amegroups.com/article/view/10.21037/pcm-22-15/coif). The series “Quality Assurance in Radiotherapy” was commissioned by the editorial office without any funding or sponsorship. AV declares being paid part time by the Belgian College of Radiation Oncology as well as being lecturer for the Internal Atomic Energy Agency. BY, NR, BR are members of the BELdArt initiative and benefits from a contract with the Institut Roi Albert II within the framework of the Belgian Cancer plan. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Leer JW, van Houtte P, Seegenschmiedt H. Radiotherapy of non-malignant disorders: where do we stand? Radiother Oncol 2007;83:175-7. [Crossref] [PubMed]
2. Rodin D, Tawk B, Mohamad O, et al. Hypofractionated radiotherapy in the real-world setting: An international ESTRO-GIRO survey. Radiother Oncol 2021;157:32-9. [Crossref] [PubMed]
3. Kron T, Fox C, Ebert MA, et al. Quality management in radiotherapy treatment delivery. J Med Imaging Radiat Oncol 2022;66:279-90. [Crossref] [PubMed]
4. Vaandering A, Jornet N, Scalliet P, et al. Doing the right thing: Quality in radiotherapy, a European perspective. Radiother Oncol 2018;127:161-3. [Crossref] [PubMed]
5. Council Directive 2013/59/Euratom of 5 December 2013 laying down basic safety standards for protection against the dangers arising from exposure to ionising radiation, and repealing Directives 89/618/Euratom, 90/641/Euratom, 96/29/Euratom, 97/43/Euratom and 2003/122/Euratom. Available online: https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2014:013:0001:0073:FR:PDF
6. The current status of uptake of European Basic Safety Standard (2013/59/Euratom) requirements: results of a follow-up survey in European radiology departments. Insights Imaging 2021;12:139. [Crossref] [PubMed]
7. Constant improvement in quality and safety of radiology, radiotherapy and nuclear medicine through clinical audit D4.4: WP4 Workshop Report. 2022. Available online: http://www.eurosafeimaging.org/wp/wp-content/uploads/2022/08/QuADRANT-WP4-Workshop-Report_for-website.pdf
8. International Atomic Energy Agency. Comprehensive Audits of Radiotherapy Practices: A Tool for Quality Improvement. OnkologiaGumedEduPl [Internet]. Vienna, 2007;152. Available online: https://www.iaea.org/publications
9. Penninckx F, Roels S, Leonard D, et al. Kwaliteit van rectale kankerzorg, fase 1: Een praktijkrichtlijn voor rectale kanker. Good Clinical Practice (GCP). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE), 2007:KCE Reports 69A.
10. Vlayen J, Verstreken M, Mertens C, et al. F. Kwaliteit van rectale kankerzorg – Fase 2: ontwikkeling en test van een set van kwaliteitsindicatoren. Good Clinical Practice (GCP). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE), 2008: KCE Reports 81A.
11. Joye I, Lambrecht M, Jegou D, et al. Does a central review platform improve the quality of radiotherapy for rectal cancer? Results of a national quality assurance project. Radiother Oncol 2014;111:400-5. [Crossref] [PubMed]
12. Verhoeven K, Weltens C, Remouchamps V, et al. Vessel based delineation guidelines for the elective lymph node regions in breast cancer radiation therapy - PROCAB guidelines. Radiother Oncol 2015;114:11-6. [Crossref] [PubMed]
13. Kindts I, Vandermeulen A, Verhoeven K, et al. A central review platform improves the quality of regional lymph node delineation for breast cancer radiation therapy. Int J Radiat Oncol Biol Phys 2016;96:E38-9. [Crossref]
14. Charlier F, Descamps T, Lievens Y, et al. ProCaLung – Peer review in stage III, mediastinal node-positive, non-small-cell lung cancer: How to benchmark clinical practice of nodal target volume definition and delineation in Belgium. Radiother Oncol 2022;167:57-64. [Crossref] [PubMed]
15. van der Veen J, Gulyban A, Nuyts S. Interobserver variability in delineation of target volumes in head and neck cancer. Radiother Oncol 2019;137:9-15. [Crossref] [PubMed]
16. Grégoire V, Evans M, Le QT, et al. Delineation of the primary tumour Clinical Target Volumes (CTV-P) in laryngeal, hypopharyngeal, oropharyngeal and oral cavity squamous cell carcinoma: AIRO, CACA, DAHANCA, EORTC, GEORCC, GORTEC, HKNPCSG, HNCIG, IAG-KHT, LPRHHT, NCIC CTG, NCRI, NRG Oncology, PHNS, SBRT, SOMERA, SRO, SSHNO, TROG consensus guidelines. Radiother Oncol 2018;126:3-24. [Crossref] [PubMed]
17. Scalliet PGFederal College of Radiotherapy, Brussels. Clinical radiotherapy audits in Belgium, 2011-2014. Cancer Radiother 2015;19:621-3. [Crossref] [PubMed]
18. Rosenblatt E, Zubizarreta E, Izewska J, et al. Quality audits of radiotherapy centres in Latin America: a pilot experience of the International Atomic Energy Agency. Radiat Oncol 2015;10:169. [Crossref] [PubMed]
19. Izewska J, Coffey M, Scalliet P, et al. Improving the quality of radiation oncology: 10years' experience of QUATRO audits in the IAEA Europe Region. Radiother Oncol 2018;126:183-90. [Crossref] [PubMed]
20. Vaandering A, Lievens Y, Scalliet P, et al. Feasibility and impact of national peer reviewed clinical audits in radiotherapy departments. Radiother Oncol 2020;144:218-23. [Crossref] [PubMed]
21. Batamuriza A, Blondiau E, Crohain J, et al. QMRT’s tool: A proposal for a complementary document to QUATRO [Internet]. Brussels; 2017 [cited 2019 Sep 18]. Available online: http://qmrt.be/downloads/QMRTtool2017.pdf
22. Schaeken B, Cuypers R, Lelie S, et al. Implementation of alanine/EPR as transfer dosimetry system in a radiotherapy audit programme in Belgium. Radiother Oncol 2011;99:94-6. [Crossref] [PubMed]
23. Schaeken B, Cuypers R, Goossens J, et al. Experimental determination of the energy response of alanine pellets in the high dose rate 192Ir spectrum. Phys Med Biol 2011;56:6625-34. [Crossref] [PubMed]
24. Anton M. Development of a secondary standard for the absorbed dose to water based on the alanine EPR dosimetry system. Appl Radiat Isot 2005;62:779-95. [Crossref] [PubMed]
25. Anton M. Uncertainties in alanine/ESR dosimetry at the Physikalisch-Technische Bundesanstalt. Phys Med Biol 2006;51:5419-40. [Crossref] [PubMed]
26. Niroomand-Rad A, Blackwell CR, Coursey BM, et al. Radiochromic film dosimetry: recommendations of AAPM Radiation Therapy Committee Task Group 55. American Association of Physicists in Medicine. Med Phys 1998;25:2093-115. [Crossref] [PubMed]
27. Low DA, Dempsey JF. Evaluation of the gamma dose distribution comparison method. Med Phys 2003;30:2455-64. [Crossref] [PubMed]
28. Yalvac B, Reniers B. PO-1783: A method for performing film dosimetry as part of a postal audit service by a recalibration process. Radiother Oncol 2020;152:S994. [Crossref]
29. Yalvac B, Reulens NR, Schroeyers W, et al. BELdART: A Belgian dosimetry audit programme in radiotherapy based on alanine/EPR and radiochromic film dosimetry. SSDL Newsletter 201;70:24-6.
30. De Saint-Hubert M, De Angelis C, Knežević Ž, et al. Characterization of passive dosimeters in proton pencil beam scanning - A EURADOS intercomparison for mailed dosimetry audits in proton therapy centres. Phys Med 2021;82:134-43. [Crossref] [PubMed]
31. van der Schaaf T. Near miss reporting in the chemical process industry. Eindhoven University of Technology; 1998. Available online: https://pure.tue.nl/ws/files/3420504/384344.pdf
32. van Vuuren W, Shea CEE, van der Schaaf TWW, et al. The development of an incident analysis tool for the medical field. 1997;25. EUT - BDK report. Dept. of Industrial Engineering and Management Science. Available online: http://alexandria.tue.nl/repository/books/493452.pdf
33. Rasmussen J. Human errors. A taxonomy for describing human malfunction in industrial installations. Journal of Occupational Accidents 1982;4:311-33. [Crossref]
34. Reijnders P, Joustra A. Opzetten van thematisch onderzoek met behulp van meldingsdata binnen de vereniging PRISMA-RT MAASTRO clinic (afdeling radiotherapie) Maastricht Nederland Monique Roozen. [cited 2022 Apr 1]; Available online: http://www.prisma-rt.nl/
35. Albert JM, Das P. Quality assessment in oncology. Int J Radiat Oncol Biol Phys 2012;83:773-81. [Crossref] [PubMed]
36. Albert JM, Das P. Quality indicators in radiation oncology. Int J Radiat Oncol Biol Phys 2013;85:904-11. [Crossref] [PubMed]
37. Kay JFL. Health Care Benchmarking. Medical Bulletin 2007;12:22-7.
38. Sampurno F, Cally J, Opie JL, et al. Establishing a global quality of care benchmark report. Health Informatics J 2021;27:14604582211015704. [Crossref] [PubMed]
39. Ellis J. Sharing the evidence: clinical practice benchmarking to improve continuously the quality of care. J Adv Nurs 2000;32:215-25. [Crossref] [PubMed]
40. Vuk T. Quality indicators: a tool for quality monitoring and improvement. ISBT Science Series. 2012;7:24-8. [Crossref]
41. World Health Organization. Regional Office for Europe, European Observatory on Health Systems and Policies, Busse R, Klazinga N, Panteli D, Quentin W. Improving healthcare quality in Europe: characteristics, effectiveness and implementation of different strategies. 2019 Available online: https://books.google.be/books?hl=en&lr=&id=e_q2DwAAQBAJ&oi=fnd&pg=PA31&dq=measuring+healthcare+quality&ots=zwLZpWVFNH&sig=uobmS-v9TZp_rl2i4E1B4xSEHr0#v=onepage&q=measuring%20healthcare%20quality&f=false
42. Vaandering A, Ben Mustapha S, Lambrecht M, et al. Impact of the COVID-19 Pandemic on Patients and Staff in Radiation Oncology Departments in Belgium: A National Survey. Front Oncol 2021;11:654086. [Crossref] [PubMed]
43. Spencer K, Jones CM, Girdler R, et al. The impact of the COVID-19 pandemic on radiotherapy services in England, UK: a population-based study. Lancet Oncol 2021;22:309-20. [Crossref] [PubMed]
44. Slotman BJ, Lievens Y, Poortmans P, et al. Effect of COVID-19 pandemic on practice in European radiation oncology centers. Radiother Oncol 2020;150:40-2. [Crossref] [PubMed]
45. Achard V, Aebersold DM, Allal AS, et al. A national survey on radiation oncology patterns of practice in Switzerland during the COVID-19 pandemic: Present changes and future perspectives. Radiother Oncol 2020;150:1-3. [Crossref] [PubMed]
46. Jereczek-Fossa BA, Pepa M, Marvaso G, et al. COVID-19 outbreak and cancer radiotherapy disruption in Italy: Survey endorsed by the Italian Association of Radiotherapy and Clinical Oncology (AIRO). Radiother Oncol 2020;149:89-93. [Crossref] [PubMed]
47. Bibault JE, Giraud P, Burgun A. Big Data and machine learning in radiation oncology: State of the art and future prospects. Cancer Lett 2016;382:110-7. [Crossref] [PubMed]
48. Nyholm T, Olsson C, Agrup M, et al. A national approach for automated collection of standardized and population-based radiation therapy data in Sweden. Radiother Oncol 2016;119:344-50. [Crossref] [PubMed]
49. Potters L, Ford E, Evans S, et al. A Systems Approach Using Big Data to Improve Safety and Quality in Radiation Oncology. Int J Radiat Oncol Biol Phys 2016;95:885-9. [Crossref] [PubMed]
50. Caissie A, Brown E, Olson R, et al. Improving patient outcomes and radiotherapy systems: A pan-Canadian approach to patient-reported outcome use. Med Phys 2018;45:e841-4. [Crossref] [PubMed]