Original article by Olivia S. Chung1*, Alisha M. Johnson1, Nathan L. Dowling2, Tracy Robinson3, Chee H. Ng2, Murat Yücel1†, and Rebecca A. Segrave1†.
1BrainPark, Turner Institute for Brain and Mental Health and Monash Biomedical Imaging Facility, Monash University, Melbourne, VIC, Australia
2Professorial Unit, Department of Psychiatry, The Melbourne Clinic, The University of Melbourne, Melbourne, VIC, Australia
3School of Nursing, Paramedicine and Healthcare Sciences, Charles Sturt University, Bathurst, NSW, Australia
Abstract
Therapeutic virtual reality (VR) has the potential to address the challenges of equitable delivery of evidence-based psychological treatment. However, little is known about therapeutic VR regarding the perspectives and needs of real-world service providers. This exploratory study aimed to assess the acceptability, appropriateness, and feasibility of therapeutic VR among clinicians, managers, and service staff working in mental healthcare and explore potential implementation barriers and enablers. Eighty-one staff from a network of private psychiatric hospitals in Victoria, Australia (aged M + SD: 41.88 + 12.01 years, 71.6% female; 64% clinical staff) completed an online survey, which included the Acceptability of Intervention Measure (AIM), Appropriateness of Intervention Measure (IAM), and Feasibility of Intervention Measure (FIM). While 91% of participants had heard about VR technology, only 40% of participants had heard of therapeutic VR being used in mental healthcare, and none had used therapeutic VR in a clinical setting. Most participants perceived VR to be acceptable (84%), appropriate (69%), and feasible (59%) to implement within their role or service and envisioned a range of possible applications. However, participants expressed concerns regarding safety, efficacy, and logistical challenges across clinical settings. Findings suggest a strong interest for therapeutic VR among Australian mental health providers working in the private system. However, dissemination efforts should focus on addressing identified barriers to ensure mental health providers are adequately informed and empowered to make implementation decisions.
Introduction
With the burden of mental illness rising across all countries at a projected cost of $16 trillion to the global economy by 2030 (1), there is an urgent need for innovation to ensure more effective, equitable and timely access to mental health services. Leveraging digital technologies has been highlighted as a key strategy to meet unmet mental health needs (2, 3). However, implementation of digital health technologies into clinical practice has been challenging (4). Notably, the COVID-19 pandemic has catalyzed rapid adoption of digital and online platforms across healthcare systems that have previously been underutilized despite wide availability and demonstrated efficacy (5). For instance, in the US, over two-thirds of psychologists had never used videoconferencing to deliver services prior to 2018 (5). Within this new era of digital healthcare is an opportunity for therapeutic virtual reality (VR) to be integrated as an innovative treatment tool.
VR is an immersive technology that uses computer-generated, 3D-environments to transport people into realistic scenarios, typically through a head-mounted display (HMD) (6). By effectively engaging participants’ natural sensorimotor contingencies, VR creates a sense of presence (i.e., illusion of “being there”) and embodiment (i.e., illusion of virtual body ownership) (7, 8), which can be exploited to enhance traditional psychological therapies. Research over two decades has investigated VR’s potential to provide systematically controllable, ecologically valid environments to safely modify maladaptive cognitive, emotional, and physiological responses (9). Presently, over 30 randomized control trials support VR exposure therapy (VRET) as an effective treatment for anxiety-related disorders (i.e., phobias, post-traumatic stress disorder (PTSD), panic disorder), with equivalent effect sizes (10) and attrition rates compared to in vivo exposure therapy (11, 12). Moreover, there is growing evidence supporting its utility in assessing and treating a broader range of conditions including autism-spectrum disorders, addiction, depression, eating disorders, pain, and psychosis (13, 14). The potential benefits of VR are numerous. For instance, VR enables patients greater opportunity for direct learning with the support of a clinician in simulations of real world situations where their psychological difficulties occur may accelerate treatment gains (15). VR can also address practical challenges associated with accessing relevant therapeutic stimuli (e.g., phobic stimuli), which can be impractical, costly, or even risky to recreate in conventional clinical settings (16). VR may also be a more acceptable treatment approach (e.g., compared to in vivo exposure) to patients (17), which may encourage earlier or sustained help-seeking in patients who find traditional treatment aversive.
While technology barriers (e.g., high cost, heavy apparatus, unreliable performance) have previously hindered meaningful translation from research to clinical settings, the release of consumer HMDs in 2016 has greatly altered implementation considerations. Since then, the technology has rapidly evolved, with entry-level headsets providing more immersive, ergonomic VR experiences (i.e., enabled by six degrees of freedom head-tracking, refresh rates of ≥90 Hz, wider field of view) than their predecessors at increasingly accessible price-points (Table 1 provides an overview of most popular VR systems commercially available and newly released). Concurrently, the therapeutic VR market has expanded, with several vendors offering various subscription plans (see Table 2). Yet, despite demonstrated efficacy and improved technology availability, therapeutic VR has seen limited uptake in mainstream clinical settings (28, 29). This reflects the broader pattern of slow evidence-based practice (EBP) adoption by healthcare systems (30), as it is estimated that only about half of EBPs are successfully incorporated into routine clinical practice, taking 17 years on average (31). This “time lag” highlights the inherent complexities of practice and policy in local contexts. As the application of VR in mental healthcare is emerging, it is timely to better understand the perspectives of frontline stakeholders.
According to Proctor et al. (32), the acceptability, appropriateness, and feasibility of a novel EBP are the outcomes most valuable to assess during early-stage implementation, as they are essential pre-conditions for successful and sustained uptake. Acceptability is the perception that an EBP is agreeable or satisfactory (e.g., in content, comfort, complexity, credibility); appropriateness is the perception that the EBP is useful or compatible with a given setting, provider or consumer (e.g., clinical suitability, organizational mission); and feasibility is the degree to which an EBP can be successfully carried out within a given setting. To date, studies investigating therapeutic VR implementation are limited, with most conducted prior to 2016 and focused specifically on its application in exposure therapy, stroke rehabilitation or pain (33–39). Only one study has examined contemporary attitudes toward VRET among clinicians practicing cognitive behavioral therapy (CBT), with results indicating that concerns may have shifted from predominantly technology-related reservations (e.g., operating difficulties, poor immersion) to therapeutic efficacy (28). However, the perspectives of cross-disciplinary clinicians, service directors, managers, and administrators working in mental healthcare remain poorly understood, despite their strong influence on therapeutic VR integration into practice and management of related service operations. Focusing on providers working in the privately funded system is a strategic starting point as they are frequently earlier adopters of new evidence-based practices compared with public system settings (40).
The current study sought to survey knowledge, attitudes, and perceived implementation barriers and enablers for therapeutic VR among clinicians and non-clinical service staff working across a network of private psychiatry hospitals in Victoria, Australia. Two explorative research questions were investigated: (i) do clinicians and non-clinical staff find therapeutic VR to be acceptable, appropriate, and feasible to implement within mental healthcare? and (ii) what are the potential barriers to, and enablers for, the implementation of therapeutic VR in mental healthcare?
Methods
Ethics Statement
The study was approved by the Melbourne Clinic Human Research Ethics Committee (#304) and the Monash University Human Research Ethics Committee (#13284). Informed consent was implied through survey completion. The ethics and consent statement and survey questions are included in the Supplementary Material.
Participants and Procedure
Eighty-one clinical and non-clinical health service staff (aged M = 41.88 ± 12.01 years; 71.6% female; 64% working in a clinical role) were recruited from the Healthscope network of private psychiatric hospitals in Melbourne, Victoria, as part of Australia’s largest private healthcare provider for mental health issues and for substance use disorders. Participants were recruited from hospitals via site-specific staff email lists between May to September 2019, with the inclusion criteria being a current Healthscope staff member or student, no exclusion criteria were applied. The email included a brief explanation about the study and link to the online survey. At the end of the survey, participants were invited to enter their details for a prize draw of a 1x $200 gift-card in appreciation for their time and effort. Respondents worked across three hospital sites in metropolitan and regional Victoria, including The Melbourne Clinic (n = 63, 78%), The Geelong Clinic (n = 10, 12%), and The Victoria Clinic (n = 8, 10%). Table 3 presents additional participant characteristics.
Survey
The survey was delivered online in English via Qualtrics, median completion time was 17.1 min, and was subdivided into six sections (i.e., five response sections and one information section). The sections are described below in order of completion.
I. Demographics. This section included questions regarding participant demographics (e.g., age, gender) and their professional background (e.g., primary role, years in current or similar role, clinical settings worked in, and patient age groups and primary clinical diagnoses worked with).
II. Prior knowledge and impression of VR. This section included questions about participants baseline knowledge of VR, including whether they had heard of VR (“Yes,” “No”), previously tried VR (“Yes,” “No,” “Not sure”), where prior VR experiences occurred (e.g., personal home, gaming outlet, museum), and what their overall impression of VR was generally (“positive,” “neutral,” “negative”), prior to completing the survey. An optional short answer question was included to further explore participants’ impressions.
III. Prior knowledge and impression of therapeutic VR. This section asked participants whether they had heard of therapeutic VR being used therapeutically in medicine, psychology, or psychiatry (“Yes,” “No”), used therapeutic VR with patients (“Yes,” “No”), and what their overall impression of VR as a therapeutic tool in mental healthcare was (“positive,” “neutral,” “negative”) prior to completing the survey. An optional short answer question was included to further explore participants’ impressions.
IV. Information section. As VR and its applications in mental healthcare are still emerging, it was anticipated that participants’ knowledge would be variable. Thus, to increase participants’ capacity to provide informed responses (i.e., section VI of the survey), information about the current state of VR technology, efficacy evidence, uptake, side-effects, cost, logistics, billing, and a 2-min video demonstration of a clinical VR application to treat persecutory delusions was provided (see Supplementary Material; Figures 1A,B depicts screen-captures of the video shown to participants). The information section was reviewed by the research team for consensus around the accuracy and neutrality of information (i.e., not presenting information in a positively or negatively biasing manner) at the time of data collection.
V. Impression of therapeutic VR after information provision. This section asked participants to re-rate their overall impression of VR as a therapeutic tool in mental healthcare (“positive,” “neutral,” “negative”) after being provided with information about therapeutic VR. An optional short answer question was included to further explore participants’ impressions.
VI. Acceptability, Appropriateness and Feasibility. Following provision of the information section, participants completed the implementation outcome measures, Acceptability of Intervention Measure (AIM), Appropriateness of Intervention Measure (IAM) and the Feasibility of Intervention Measure (FIM) (41). These were modified by substituting “evidence-based practice” with “therapeutic virtual reality.” Each scale consisted of four items rated on a 5-point Likert scale (1 = “strongly disagree” to 5 = “strongly agree”). A cut-off score of ≥ 15 indicated whether a participant perceived VR as acceptable, appropriate, or feasible to implement within their role or service setting. These scales have demonstrated good structural validity (Cronbach’s alphas of 0.85 for AIM, 0.91 for IAM and 0.89 for FIM) and test-retest reliability (Cronbach’s > of 0.83 for AIM, 0.87 for IAM and 0.88 for FIM). This section also included optional short answer questions to further explore participants’ perceptions about the usefulness of therapeutic VR and potential barriers to its use within one’s clinical role or service.
Data Analysis
Quantitative: The datasets for analysis in this manuscript are available upon request, without reservations, to all researchers. Of the 97 recorded survey attempts, only 81 contained full datasets which were included in analyses. Analyses were conducted using SPSS Statistics Version 22. Descriptive statistics were used to summarize data related to demographics, knowledge and attitudes, and AIM, IAM and FIM total scores. Due to small, unequal sample sizes and indications of non-normality and negative skew on AIM, IAM and FIM scores through Shapiro-Wilks tests and visual analysis of graphs, non-parametric tests were conducted (42). Mann-Whitney non-parametric tests were used to explore differences on AIM, IAM, and FIM scores between clinical and non-clinical staff, participants who had or had not tried VR, and males and females. Spearman’s rho correlations were performed between years worked in current or similar role and respective AIM, IAM, and FIM scores. All tests were two-tailed, with an alpha level of 0.05 (43).
Qualitative: Data was de-identified and initially divided into responses for each short answer question, which were subsequently considered as a whole during category development. Comments (n = 251) were analyzed with broad thematic grouping using a deductive approach informed by Proctor et al.’s (32) implementation outcomes. To provide further structure, responses were sorted into second-order codes based on stakeholder position (e.g., clinician, patient, service), then further classified as a barrier or facilitator. Barriers were factors that would prevent or hinder use, implementation, or uptake, while enablers were factors that would enhance or increase use, implementation, or uptake. Extraneous comments were excluded from further analysis. An individual response could contribute to more than one category if several issues were addressed. The number of comments reported in tables are the number of comments that contained information for each sub-category.
Data Integration: Data were collected using a sequential explanatory strategy (44), with quantitative data collection preceding free-text comments, which were used to contextualize and triangulate quantitative data on AIM, IAM, and FIM.
Results
Knowledge of VR
Prior to undertaking the survey, 91% (n = 74) of participants had heard of VR technology, and 42% (n = 34) of participants had previously tried VR at commercial gaming outlets (n = 15), residential homes (n = 10), museums, shopping centers, and theme parks (n = 8) or as part of a research study (n = 1). Of those who indicated they had not tried or were unsure if they had tried VR (n = 47), 92% indicated they would like to.
Knowledge of Therapeutic VR
Prior to undertaking the survey, 40% (n = 32) of participants had heard of VR being used clinically in mental healthcare. None of the participants reported having used therapeutic VR with a patient.
Attitudes Toward Therapeutic VR
Prior to reading the information section of the survey, 65% of participants had a positive impression and 36% had a neutral impression of VR being used in mental healthcare. After reading the information section, 84% of participants reported a positive impression, 14% reported a neutral impression, and 1% reported a negative impression (n = 1 missing). Seventeen participants (21%) changed their impression from neutral to positive after reading the information section, while two others (3%) changed their impression from positive to neutral or neutral to negative.
Acceptability, Appropriateness, and Feasibility
Descriptive statistics for group comparisons on AIM, AIM, and FIM scores are presented in Table 4. After reading the information section of the survey, 84% of participants felt that VR was acceptable, 69% of participants felt that VR was appropriate, and 59% of participants felt that VR was feasible to introduce into private mental health services or their clinical role. Clinicians perceived VR to be less appropriate, z(N = 81) = 2.15, p = 0.03, r = 0.24, and less feasible, z(N = 81) = 2.15, p = 0.03, r = 0.24, to implement than non-clinical staff, however, there was no difference in acceptability scores, z(N = 81) = 0.17, p = 0.86, r = 0.02. Hospital workers who had previously tried VR perceived it to be more acceptable, z(N = 77) = 1.96, p = 0.02, r = 0.22, and more appropriate, z(N = 77) = 2.46, p = 0.005, r = 0.28, than those who had not tried VR, however there was no statistically significant difference in feasibility scores, z(N = 77) =1.56, p = 0.12, r = 0.18. There were no statistically significant differences between males and females in scores for acceptability, z(N = 80) = 1.93, p = 0.05, r = 0.22, appropriateness, z(N = 80) = 0.07, p = 0.95, r = 0.01, or feasibility, z(N = 80) = 0.38, p = 0.70, r = 0.04. Spearman correlations conducted between years worked in current or similar role and scores for acceptability, rs = 0.01, p = 0.99 appropriateness, rs = −0.039, p = 0.73 and feasibility, rs = −0.0454, p = 0.69, were not statistically significant.
Barriers and Enablers to Implementation
Among free-text comments, 80% were related to acceptability, 44% were related to appropriateness, and 32% were related to feasibility. The most common barriers and enablers reported for each category are outlined in Table 5.
Acceptability
Barriers: The most common barrier to acceptability related to perceived technical difficulties and limitations of VR. Participants anticipated limitations in VR, including the “reliability of equipment” (P07, Psychiatrist) and “customisability of software to meet the patient’s individual needs/fears” (P70, Researcher). Participants expressed concerns about VR being an “artificial environment… similar to a game and not actually getting on the train or in the lift” (P39, Clinical Manager), thus preventing gains made in VR transferring into real-world improvements. Staff also queried whether clinical VR research could keep pace with the rapidly evolving technology [e.g., technology rapidly evolving but may mean is unable to be empirically tested in a considered way (P33, Researcher)]. Other barriers related to staff “not ever experiencing VR” (P10, Nurse) or a “lack of knowledge” (P3, Program Manager) about the efficacy of therapeutic VR and recommended practice (e.g., contraindications, treatment timing, dosage, after-care). Participants also expressed concerns that private health providers might look to “replace service provision with virtual reality” (P20, Intake Coordinator) and that VR might replace “human interactions and connections” (P53, Psychologist). A lack of “willingness to try” (P10, Nurse) VR due to factors including age and discomfort with technology was another perceived barrier for use, with participants anticipating that “older patients may not embrace the technology” (P16, Organizational Administrator) and that it “may only appeal to younger or tech savvy patients” (P68, Student).
Enablers: Perceived clinical and practical benefits of VR were the most common enablers to acceptability. Participants viewed VR as “another tool in the toolbox” (P78, Service Manager), with potential to “make intervention quicker [due to] opportunity to practice [skills] in a realistic setting before moving into real life” (P39, Clinical Manager). Participants perceived benefits in being able to access “environments… otherwise difficult to access” (P4, Program Manager), managing “difficult scenarios with low risk” (P45, Nurse), making treatment “less reliant on the quirks of humans” (P39, Clinical Manager), and enhancing control in exposure therapy (e.g., “if it becomes overwhelming, is easily removed and stopped” (P16, Organizational Administrator). Other enablers related to positive attitudes about technology-use in clinical practice (e.g., “it is the way the future is heading to be implementing more technology into therapy,” [P47, Psychologist]), and perceptions that it could “reach people who have difficulty engaging in treatment services in typical ways” (P33, Researcher). Participants felt VR would “appeal to a younger market [and] different cohort of patients” (P56, Program Manager), and benefit patients who had “difficulty articulating their experiences, [are] experientially avoidant” (P33, Researcher), or who “lack capacity to organize themselves” to complete exposure tasks when at home” (P64, Occupational Therapist). Prior experiences with VR enhanced participants’ understanding of its therapeutic potential [e.g., “During the game…dealing with heights, of which I’m afraid…this was affected” (P15, Nurse)], while knowledge of its clinical applications enhanced its credibility [e.g., “Barbara Rothbaum at Emory in USA is already using this to treat PTSD.” (P09, Psychiatrist)]. Broader dissemination of “impartial… findings” (P20, Intake Coordinator) particularly around efficacy, and allowing staff to” try and experience the feeling of VR” (P35, Nurse) were strategies suggested to further enhance acceptability.
Appropriateness
Barriers: Among barriers to appropriateness, most comments related to concerns around clinical risk and safety. Participants expressed concerns that VR would be used “without adequate training…[or] proper assessment of the client” (P26, Psychologist), which could “exacerbate symptoms” (P27, Psychologist), or lead to “avoidance of the real world” (P15, Nurse). Participants were also concerned about “potential unknown adverse reactions… [in patients with] a history of dissociation, derealization” (P33, Researcher), “severe PTSD and paranoia” (P18, Nurse), “psychosis” (P22, Program Manager) as well as “adverse reactions with medication” (P16, Organizational Administrator). Participants wondered “how patients with auditory hallucinations would cope” (P48, Nurse) and worried that patients may become “too overwhelmed from what they are seeing…and not speak up” (P60, Nurse) resulting in patients or staff injury or equipment damage. Participants also worried about immersion side effects including “disorientation and nausea [based on] personal experience” (P32, Psychiatric Nurse) and “electrical cords exposed [presenting a] hanging risk” (P69, Researcher). Other barriers related to perceived incompatibility with one’s treatment philosophy (e.g., “Art therapy seeks to stimulate creativity and ‘play’. It is strength based and virtual reality would not be of assistance” [P30, Art Therapist]).
Enablers: The most common enablers to appropriateness were perceptions that therapeutic VR would be “safe for clients” (P49, Psychologist) and was suitable for the scope of clinical presentations encountered and treatments offered by their service [e.g., “great opportunity…not unlike offering ACT, DBT and TMS.” (P65, Service Manager)]. Participants perceived VR as having “many applications across diagnostic groups” (P76, Program Manager), including anxiety disorders, PTSD, OCD, and addictions. Staff also felt that it was compatible with a range of interventions, including “graded exposure” (P40, Occupational Therapist), “role play” (P15, Nurse), “mindfulness” (P20, Intake Coordinator), “challeng[ing]…maladaptive beliefs” (P62, Nurse), “relaxation…to manage stress” (P23, Nurse), “biofeedback” (P64, Occupational Therapist), as well as relapse management [e.g., “safe way of simulating exposure to risk situations without the problems of relapse” (P39, Clinical Manager)] and harm minimization [e.g., “helping patients deal with suicidal or self-harm thoughts without actually doing these acts.” (P18, Nurse)]. Participants also felt that VR could help in the functional recovery of patients requiring “a slow return to community/social exposure” (P35, Nurse) and to “step up and/or step down for admissions to hospital” (P33, Researcher). Another facilitator was perceived reputational benefits, which could help position their service as a specialist clinic [e.g., “Would be a great adjunct to the PTSD program…and further [our] name as center of excellence in PTSD.” (P09, Psychiatrist)].
Feasibility
Barriers: A major barrier to feasibility related to logistical constraints of service settings. Participants noted that within the psychiatric hospital setting, clinicians “mainly work in group” (P39, Clinical Manager), and perceived VR to be an “isolating activity” (P31, Exercise Physiologist) that would be practically difficult to incorporate into group sessions [e.g., “couldn’t take time to (use VR) with each individual while others waited” (P57, Psychologist)]. Participants anticipated challenges to obtaining adequate funding “for 1:1 work alongside group work” [P64, Occupational Therapist], to purchase and maintain “multiple units” (P31, Exercise Physiologist), and ensuring that content would be “appropriate for all group members” (P31, Exercise Physiologist). Other common barriers related to a lack of expertise to provide “adequate training for clinicians” (P62, Nurse), “time required to train and use [VR and having] limited physical space” (P38, Psychologist) and the “portability/ transport of equipment and set up in outreach” (P52, Nurse). Participants also perceived that VR may not be viewed as financially lucrative enough to prioritize investing in by the service’s private owners given its lack of service reimbursement [e.g., “I am concerned about how it is billed and therefore financially lucrative for the private health sector.” (P56, Program Manager)].
Enablers: The most common enablers to feasibility related to the availability of VR “headsets [and] proper training” (P20, Intake Coordinator) for staff to deliver it. Participants felt that barriers related to “funding, space, resources [could] be overcome with planning” (P56, Program Manager), however, noted that clinical consultation was needed and that “psychologists or psychiatrists [should] be responsible for at least overseeing VR implementation.” (P10, Nurse). Participants suggested that “private health non-inclusion” could be overcome by offering VR as “part of therapy in a structured admission” (P63, Service Manager). Introducing therapeutic VR through a research trial was also seen as facilitatory, as had been previously done with other recently introduced treatments to services [e.g., “It is a great opportunity to do a research trial much like when TMS was commenced.” (P63, Service Manager)].
Discussion
In light of recent commercial availability of affordable VR hardware and the demonstrated efficacy of numerous therapeutic VR environments, mental health providers have an opportunity to capitalize on the potential benefits of this immersive technology. Understanding the perceived acceptability, appropriateness, and feasibility of implementing therapeutic VR among frontline stakeholders is a fundamental step to ensuring its timely, effective, and sustained uptake in clinical settings. Overall, findings demonstrated a strong interest in therapeutic VR among Australian private mental health service staff, despite low familiarity with the technology and its clinical applications. The majority (84%) of respondents had a positive impression of VR after provision of general information, most (84%) staff reported that therapeutic VR was acceptable, and over half indicated it was appropriate (69%) and feasible (59%) to implement within their role or service. These findings suggest that the potential benefits of VR are largely self-evident to staff, but that limited awareness of the clinical evidence-base and available systems to integrate into current treatment and service models remain key barriers to uptake.
Findings suggest value in broad stakeholder engagement, as clinicians perceived VR to be less appropriate and feasible to implement than non-clinical service staff. This may reflect possible tension between managers, who typically decide whether new treatments are practical to introduce and oversee their integration into service operations, and clinicians, who refer patients for and deliver treatment. Addressing clinicians’ reservations will be important as research suggests that negative attitudes have a stronger influence on future VR use (i.e., non-use) among CBT therapists than positive attitudes (28), and that failure to shift negative beliefs can adversely impact treatment availability and quality (45). Notably, psychologists and allied health professionals found VR least appropriate and feasible to implement. Thus, differentially targeting these disciplines during implementation efforts could be useful, for instance, by incorporating VR into graduate training programs, as has occurred in surgical and nursing education (46, 47).
Among the most common barriers staff reported were perceived risk and safety issues. This suggests that a major focus of education and training should involve addressing negative preconceptions about patient suitability (e.g., contraindications) and providing guidance around the likelihood and management of potential adverse events (e.g., patient distress, side-effects, medication interaction). Inaccurate beliefs about treatment consequences (e.g., iatrogenic effects, reduced therapeutic alliance) have been a well-documented barrier to the dissemination of in vivo exposure therapy, which remains underutilized even among trained clinicians (45). Current evidence suggests that concurrent use of common pharmacological treatments (e.g., olanzapine, antipsychotics, antidepressants) with VR have minimal adverse effects (20, 23, 48, 49) and that alliance when using VR is similar to face-to-face therapy (50, 51). However, both areas remain under-researched and warrant further attention. Interestingly, information about a VR application to treat persecutory delusions did not completely allay participants’ concerns about its appropriateness for psychosis treatment, suggesting that knowledge alone may be insufficient to change attitudes.
Another common barrier reflected perceived technical challenges and therapeutic limitations (e.g., low customisability, real-world generalisability), consistent with barriers identified in previous research (28, 34, 35). These unfavorable perceptions are likely attributable to low familiarity with VR technology generally, as participants with prior VR experiences perceived it to be more acceptable and appropriate to implement in mental healthcare than those without. Segal et al. (35) similarly found that therapists with greater knowledge or interest in VR perceived it as having greater clinical utility. A lack of experience with immersive VR technology or experiences exclusively in recreational contexts could negatively skew perceptions about its clinical utility by underselling the sophistication and quality of purpose-built therapeutic environments optimized for high-fidelity HMDs. Indeed, more advanced immersive VR technology has been associated with higher levels of presence and subjective anxiety (52), which has important implications for exposure therapy, whereby corrective emotional processing requires sufficient activation of one’s fear structure (53). These concerns will likely diminish as VR becomes more commonplace across society (e.g., entertainment, education, medicine), and as results of clinical research are more widely disseminated. For instance, meta-analytic findings suggest that gains made through VRET for specific phobias transfer to the real-world (52, 54), though further investigation in other psychiatric conditions is required to substantiate efficacy. Similarly, customizability of therapeutic VR programs (i.e., grading virtual scenes for increasing degrees of exposure difficulty) is becoming increasingly common (55, 56).
Encouragingly, staff perceived VR as having broad applicability across mental health conditions, interventions, and assessments. These findings are consistent with Lindner et al. (28), highlighting greater scope for collaboration between VR developers and clinical stakeholders to develop applications beyond exposure therapy (e.g., relapse prevention, harm minimization). Greater stakeholder consultation during application development may also help maximize its utility in applied service contexts and address practical challenges across service settings. For instance, staff questioned the feasibility of VR in group therapy, which was identified as a predominant treatment delivery mode. While current therapeutic applications offer limited solutions to this, advancements in multi-user VR systems may yet open possibilities for immersive group therapy in the future (57).
Staff also anticipated challenges to adequate resourcing (e.g., trained staff, rooms, purchasing, and maintaining multiple HMDs). Given that VR-based therapies have yet to demonstrate superiority over current treatment approaches (58, 59), and that few products are being reimbursed as a specific treatment by private health funds, highlighting VR’s potential to address existing challenges in clinical practice may be a useful strategy to promote uptake. For instance, research suggests that consumers perceive VRET as more acceptable than in vivo exposure and that it can enhance motivation and engagement with treatment (17, 60). Evidence of cost-effectiveness could also be an incentive, as has been shown with psychosis and combat PTSD populations (61, 62). Additionally, automated VR treatments have shown early promising results for acrophobia and psychosis (15, 28) and may offer a low-cost strategy to scale effective intervention (3). Nonetheless, participants’ concerns about VR replacing service provision suggest that automated applications could risk enhancing unfavorable perceptions of VR, thus further research on their efficacy and safety and careful attention during their dissemination is warranted.
Our findings have practical implications for the implementation of VR applications in mental healthcare. Participants limited knowledge of VR and its therapeutic applications speaks to a need for greater education and training, which could be delivered through graduate training programs, e-learning modules, and interactive workshops. The lack of resources to guide implementation has been identified as a barrier to the use of technology-enabled treatments more broadly (3). Thus, there is a need to develop evidence-based practice guidelines to ensure safe and ethical usage of evidence-based therapeutic VR applications.
It is important to note that while commercially available products (see Table 2) are developed with evidence-based principles in mind, most have not undergone, or are in the process of being tested for efficacy in randomized controlled trials (29). Similarly, freely available VR experiences are often marketed as having a mental health focus on open application marketplaces (e.g., SteamVR, Oculus), without verification of clinical utility or safety (63). Thus, there is a risk of unvalidated VR programs being marketed for both therapist-assisted or self-directed therapy, with potential to cause harm to users and diminish the credibility of validated programs (9). Nonetheless, evidence-based interventions (e.g., exposure therapy, arousal reduction, behavioral activation) in routine care often involve use of imaginal, visual, or auditory stimuli to elicit target emotional states (e.g., anxiety, disgust, craving, enjoyment, relaxation), for which the clinician and patient will often creatively source. Thus, VR scenarios will at least have value as an adjunct to these interventions. In future, we may see greater involvement of regulatory bodies in the dissemination of digital therapeutic, including VR therapies, as administrations in the US, EU, UK, and Australia, have or are in the process of reforming their regulatory framework for therapeutic software in recognition of their growing ubiquity and potential risk profile (64).
From a technical standpoint, the rapid pace of VR development also presents challenges with synchronization and compatibility of software across systems (59). Moreover, the literature to date has suffered from ambiguous terminology, leading to inadequate specification and misclassification of VR, which could diminish literature validity and provider confidence in therapeutic VR. Thus, as the field progresses, it stands to benefit from greater standardization of VR, for instance, with Takac et al.’s (65) proposed hardware-based VR qualification matrix, as well as development of a therapeutic VR resource directory. A similar initiative funded by the Australian government (i.e., e-Mental Health in Practice) successfully raised awareness of evidence-based digital mental health interventions among primary healthcare providers. Thus, these resources will likely be critical to enhancing providers’ “technological competence” in selecting and recommending appropriate VR hardware.
The current study adds to the literature by documenting the knowledge, attitudes, and perceived implementation barriers for therapeutic VR use among a broader stakeholder group (i.e., clinical, managerial, and operational staff perspectives), and including validated implementation outcome measures. However, the results should be considered with some caveats in mind. Limitations include the modest sample size and specificity to the Australian private psychiatry setting providing services to a predominantly adult patient population, which may limit generalisability. Another limitation related to participants’ low familiarity with VR, necessitating provision of general information to encourage informed responses. Nonetheless, given the limited uptake of VR in mainstream clinical settings across Australia and other comparable countries (e.g., US, UK), these findings are likely representative of current knowledge and diffusion of VR in private psychiatric services, and will therefore be relevant to early dissemination efforts in similar settings. An important next step that future research should address will be to document the perspectives of staff from public healthcare services, those working with pediatric and adolescent patient groups, and critically, those of consumers of mental health services.
We conclude that while there is a clear appetite for VR among Australian mental health providers working in the private system, concerns related to safety, efficacy, and logistical barriers warrant attention. Addressing unhelpful beliefs and knowledge and skills gaps will help ensure that prospective providers are adequately informed and empowered to adopt VR into clinical practice. Further research focused on formulating and evaluating implementation strategies is needed to promote effective and sustained uptake.
Data Availability Statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics Statement
The studies involving human participants were reviewed and approved by Melbourne Clinic Human Research Ethics Committee (#304) and the Monash University Human Research Ethics Committee (#13284). Written informed consent was not provided because informed consent was implied through survey completion.
Author Contributions
AJ and RS designed the study. AJ and ND collected the data. OC analyzed the data and drafted the manuscript. OC, ND, TR, CN, MY, and RS critically reviewed the data interpretations and draft and made significant contributions to the final version. All authors contributed to the article and approved the submitted version.
Funding
This work was supported by philanthropic investment from The David Winston Turner Endowment Fund, Wilson Foundation. The funding sources had no role in the design, management, data analysis, interpretation, and write-up of the data.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Acknowledgments
We would like to acknowledge and thank Professor Daniel Freeman and his team from the Department of Psychiatry, University of Oxford for the use of their video titled VR for paranoia in the survey and screenshots from the video in this publication. We are also extremely grateful to the Wilson Foundation and David Winston Turner Endowment Fund whose generous philanthropic investment in the BrainPark research team and facility made this research possible.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2022.792663/full#supplementary-material
References
1. Patel V, Saxena S, Lund C, Thornicroft G, Baingana F, Bolton P, et al. The Lancet Commission on global mental health and sustainable development. Lancet. (2018) 392:1553–98. doi: 10.1016/S0140-6736(18)31612-X
2. Arigo D, Jake-Schoffman DE, Wolin K, Beckjord E, Hekler EB, Pagoto SL. The history and future of digital health in the field of behavioral medicine. J Behav Med. (2019) 42:67–83. doi: 10.1007/s10865-018-9966-z
3. Fairburn CG, Patel V. The impact of digital technology on psychological treatments and their dissemination. Behav Res Ther. (2017) 88:19–25. doi: 10.1016/j.brat.2016.08.012
4. Torous J, Bucci S, Bell IH, Kessing LV, Faurholt-Jepsen M, Whelan P, et al. The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality. World Psychiatry. (2021) 20:318–35. doi: 10.1002/wps.20883
5. Pierce BS, Perrin PB, Tyler CM, McKee GB, Watson JD. The COVID-19 telepsychology revolution: A national study of pandemic-based changes in U.S. mental health care delivery. Am Psychol. (2020) 76:14–25. doi: 10.1037/amp0000722
6. Freeman D, Reeve S, Robinson A, Ehlers A, Clark D, Spanlang B, et al. Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychol Med. (2017) 47:2393–400. doi: 10.1017/S003329171700040X
7. Matamala-Gomez M, Maselli A, Malighetti C, Realdon O, Mantovani F, Riva G. Virtual body ownership illusions for mental health: a narrative review. J Clin Med Res. (2021) 10:139. doi: 10.3390/jcm10010139
8. Bohil CJ, Alicea B, Biocca FA. Virtual reality in neuroscience research and therapy. Nat Rev Neurosci. (2011) 12:752–62. doi: 10.1038/nrn3122
9. Rizzo AS, Koenig ST. Is clinical virtual reality ready for primetime? Neuropsychology. (2017) 31:877–99. doi: 10.1037/neu0000405
10. Carl E, Stein AT, Levihn-Coon A, Pogue JR, Rothbaum B, Emmelkamp P, et al. Virtual reality exposure therapy for anxiety and related disorders: a meta-analysis of randomized controlled trials. J Anxiety Disord. (2019) 61:27–36. doi: 10.1016/j.janxdis.2018.08.003
11. Benbow AA, Anderson PL. A meta-analytic examination of attrition in virtual reality exposure therapy for anxiety disorders. J Anxiety Disord. (2019) 61:18–26. doi: 10.1016/j.janxdis.2018.06.006
12. Fernández-Álvarez J, Rozental A, Carlbring P, Colombo D, Riva G, Anderson PL, et al. Deterioration rates in virtual reality therapy: an individual patient data level meta-analysis. J Anxiety Disord. (2019) 61:3–17. doi: 10.1016/j.janxdis.2018.06.005
13. Emmelkamp PMG, Meyerbröker K. Virtual reality therapy in mental health. Annu Rev Clin Psychol. (2021) 17:495–519. doi: 10.1146/annurev-clinpsy-081219-115923
14. Riva G, Malighetti C, Serino S. Virtual reality in the treatment of eating disorders. Clin Psychol Psychother. (2021) 28:477–88. doi: 10.1002/cpp.2622
15. Freeman D, Haselton P, Freeman J, Spanlang B, Kishore S, Albery E, et al. Automated psychological therapy using immersive virtual reality for treatment of fear of heights: a single-blind, parallel-group, randomised controlled trial. Lancet Psychiatry. (2018) 5:625–32. doi: 10.1016/S2215-0366(18)30226-8
16. Boeldt D, McMahon E, McFaul M, Greenleaf W. Using virtual reality exposure therapy to enhance treatment of anxiety disorders: identifying areas of clinical adoption and potential obstacles. Front Psychiatry. (2019) 10:773. doi: 10.3389/fpsyt.2019.00773
17. Garcia-Palacios A, Botella C, Hoffman H, Fabregat S. Comparing acceptance and refusal rates of virtual reality exposure vs. in vivo exposure by patients with specific phobias. Cyberpsychol Behav. (2007) 10:722–4. doi: 10.1089/cpb.2007.9962
18. Garcia LM, Birckhead BJ, Krishnamurthy P, Sackman J, Mackey IG, Louis RG, et al. An 8-week self-administered at-home behavioral skills-based virtual reality program for chronic low back pain: double-blind, randomized, placebo-controlled trial conducted during COVID-19. J Med Internet Res. (2021) 23:e26292. doi: 10.2196/26292
19. Spiegel B, Fuller G, Lopez M, Dupuy T, Noah B, Howard A, et al. Virtual reality for management of pain in hospitalized patients: A randomized comparative effectiveness trial. PLoS ONE. (2019) 14:e0219115. doi: 10.1371/journal.pone.0219115
20. Pot-Kolder R, Geraets CNW, Veling W, van Beilen M, Staring ABP, Gijsman HJ, et al. Virtual-reality-based cognitive behavioural therapy versus waiting list control for paranoid ideation and social avoidance in patients with psychotic disorders: a single-blind randomised controlled trial. Lancet Psychiatry. (2018) 5:217–26. doi: 10.1016/S2215-0366(18)30053-1
21. Nijman SA, Veling W, Greaves-Lord K, Vos M, Zandee CER, Aan Het Rot M, et al. Dynamic interactive social cognition training in virtual reality (DiSCoVR) for people with a psychotic disorder: single-group feasibility and acceptability study. JMIR Ment Health. (2020) 7:e17808. doi: 10.2196/17808
22. Goldenhersch E, Thrul J, Ungaretti J, Rosencovich N, Waitman C, Ceberio MR. Virtual reality smartphone-based intervention for smoking cessation: pilot randomized controlled trial on initial clinical efficacy and adherence. J Med Internet Res. (2020) 22:e17571. doi: 10.2196/17571
23. Rothbaum BO, Price M, Jovanovic T, Norrholm SD, Gerardi M, Dunlop B, et al. A randomized, double-blind evaluation of D-cycloserine or alprazolam combined with virtual reality exposure therapy for posttraumatic stress disorder in Iraq and Afghanistan War veterans. Am J Psychiatry. (2014) 171:640–8. doi: 10.1176/appi.ajp.2014.13121625
24. Difede J, Malta LS, Best S, Henn-Haase C, Metzler T, Bryant R, et al. randomized controlled clinical treatment trial for World Trade Center attack-related PTSD in disaster workers. J Nerv Ment Dis. (2007) 195:861–5. doi: 10.1097/NMD.0b013e3181568612
25. McLay RN, Wood DP, Webb-Murphy JA, Spira JL, Wiederhold MD, Pyne JM, et al. randomized, controlled trial of virtual reality-graded exposure therapy for post-traumatic stress disorder in active duty service members with combat-related post-traumatic stress disorder. Cyberpsychol Behav Soc Netw. (2011) 14:223–9. doi: 10.1089/cyber.2011.0003
26. Veling W, Lestestuiver B, Jongma M, Hoenders HJR, van Driel C. Virtual reality relaxation for patients with a psychiatric disorder: crossover randomized controlled trial. J Med Internet Res. (2021) 23:e17233. doi: 10.2196/17233
27. Donker T, Cornelisz I, van Klaveren C, van Straten A, Carlbring P, Cuijpers P, et al. Effectiveness of self-guided app-based virtual reality cognitive behavior therapy for acrophobia: a randomized clinical trial. JAMA Psychiatry. (2019) 76:682–90. doi: 10.1001/jamapsychiatry.2019.0219
28. Lindner P, Miloff A, Zetterlund E, Reuterskiöld L, Andersson G, Carlbring P. Attitudes toward and familiarity with virtual reality therapy among practicing cognitive behavior therapists: a cross-sectional survey study in the era of consumer VR platforms. Front Psychol. (2019) 10:176. doi: 10.3389/fpsyg.2019.00176
29. Bell I, Nicholas J, Alvarez-Jimenez M, Thompson A, Valmaggia L. Virtual reality as a clinical tool in mental health research and practice. Dialogues Clin Neurosci. (2020) 22:169–77. doi: 10.31887/DCNS.2020.22.2/lvalmaggia
30. Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An introduction to implementation science for the non-specialist. BMC Psychol. (2015) 3:32. doi: 10.1186/s40359-015-0089-9
31. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med. (2011) 104:510–20. doi: 10.1258/jrsm.2011.110180
32. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. (2011) 38:65–76. doi: 10.1007/s10488-010-0319-7
33. Kramer TL, Pyne JM, Kimbrell TA, Savary PE, Smith JL, Jegley SM. Clinician perceptions of virtual reality to assess and treat returning veterans. Psychiatr Serv. (2010) 61:1153–6. doi: 10.1176/ps.2010.61.11.1153
34. Schwartzman D, Segal R, Drapeau M. Perceptions of virtual reality among therapists who do not apply this technology in clinical practice. Psychol Serv. (2012) 9:310–5. doi: 10.1037/a0026801
35. Segal R, Bhatia M, Drapeau M. Therapists’ perception of benefits and costs of using virtual reality treatments. Cyberpsychol Behav Soc Netw. (2011) 14:29–34. doi: 10.1089/cyber.2009.0398
36. Glegg S, Holsti L, Velikonja D, Ansley B, Brum C, Sartor D. Factors influencing therapists’ adoption of virtual reality for brain injury rehabilitation. Cyberpsychol Behav Soc Netw. (2013) 16:385–401. doi: 10.1089/cyber.2013.1506
37. Markus LA, Willems KE, Maruna CC, Schmitz CL, Pellino TA, Wish JR, et al. Virtual reality: feasibility of implementation in a regional burn center. Burns. (2009) 35:967–9. doi: 10.1016/j.burns.2009.01.013
38. Levac D, Glegg SMN, Sveistrup H, Colquhoun H, Miller PA, Finestone H, et al. knowledge translation intervention to enhance clinical application of a virtual reality system in stroke rehabilitation. BMC Health Serv Res. (2016) 16:557. doi: 10.1186/s12913-016-1807-6
39. Schmid L, Glässel A, Schuster-Amft C. Therapists’ perspective on virtual reality training in patients after stroke: a qualitative study reporting focus group results from three hospitals. Stroke Res Treat. (2016) 2016:6210508. doi: 10.1155/2016/6210508
40. Aarons GA, Sommerfeld DH, Walrath-Greene CM. Evidence-based practice implementation: the impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implement Sci. (2009) 4:83. doi: 10.1186/1748-5908-4-83
41. Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. (2017) 12:108. doi: 10.1186/s13012-017-0635-3
42. Hills A. Foolproof Guide to Statistics Using IBM SPSS. Frenchs Forest, NSW: Pearson Custom Publishing (2011).
43. Armstrong RA. When to use the Bonferroni correction. Ophthalmic Physiol Opt. (2014) 34:502–8. doi: 10.1111/opo.12131
44. Curry LA, Krumholz HM, O’Cathain A, Plano Clark VL, Cherlin E, Bradley EH. Mixed methods in biomedical and health services research. Circ Cardiovasc Qual Outcomes. (2013) 6:119–23. doi: 10.1161/CIRCOUTCOMES.112.967885
45. Farrell NR, Deacon BJ, Kemp JJ, Dixon LJ, Sy JT. Do negative beliefs about exposure therapy cause its suboptimal delivery? An experimental investigation. J Anxiety Disord. (2013) 27:763–71. doi: 10.1016/j.janxdis.2013.03.007
46. Pottle J. Virtual reality and the transformation of medical education. Future Healthc J. (2019) 6:181–5. doi: 10.7861/fhj.2019-0036
47. McCarthy CJ, Uppot RN. Advances in virtual and augmented reality—exploring the role in health-care education. J Radiol Nurs. (2019) 38:104–5. doi: 10.1016/j.jradnu.2019.01.008
48. Knaust T, Felnhofer A, Kothgassner OD, Höllmer H, Gorzka R-J, Schulz H. Virtual trauma interventions for the treatment of post-traumatic stress disorders: a scoping review. Front Psychol. (2020) 11:562506. doi: 10.3389/fpsyg.2020.562506
49. Difede J, Cukor J, Wyka K, Olden M, Hoffman H, Lee FS, et al. D-cycloserine augmentation of exposure therapy for post-traumatic stress disorder: a pilot randomized clinical trial. Neuropsychopharmacology. (2014) 39:1052–8. doi: 10.1038/npp.2013.317
50. Miragall M, Baños RM, Cebolla A, Botella C. Working alliance inventory applied to virtual and augmented reality (WAI-VAR): psychometrics and therapeutic outcomes. Front Psychol. (2015) 6:1531. doi: 10.3389/fpsyg.2015.01531
51. Ngai I, Tully EC, Anderson PL. The course of the working alliance during virtual reality and exposure group therapy for social anxiety disorder. Behav Cogn Psychother. (2015) 43:167–81. doi: 10.1017/S135246581300088X
52. Ling Y, Nefs HT, Morina N, Heynderickx I, Brinkman W-P. A meta-analysis on the relationship between self-reported presence and anxiety in virtual reality exposure therapy for anxiety disorders. PLoS ONE. (2014) 9:e96144. doi: 10.1371/journal.pone.0096144
53. Foa EB, Kozak MJ. Emotional processing of fear: exposure to corrective information. Psychol Bull. (1986) 99:20–35. doi: 10.1037/0033-2909.99.1.20
54. Morina N, Ijntema H, Meyerbröker K, Emmelkamp PMG. Can virtual reality exposure therapy gains be generalized to real-life? A meta-analysis of studies applying behavioral assessments. Behav Res Ther. (2015) 74:18–24. doi: 10.1016/j.brat.2015.08.010
55. Rizzo AS, Difede J, Rothbaum BO, Reger G, Spitalnick J, Cukor J, et al. Development and early evaluation of the Virtual Iraq/Afghanistan exposure therapy system for combat-related PTSD: Virtual Iraq/Afghanistan PTSD exposure therapy. Ann N Y Acad Sci. (2010) 1208:114–25. doi: 10.1111/j.1749-6632.2010.05755.x
56. Cullen AJ, Dowling NL, Segrave R, Morrow J, Carter A, Yücel M. Considerations and practical protocols for using virtual reality in psychological research and practice, as evidenced through exposure-based therapy. Behav Res Methods. (2021) 53:2725–42. doi: 10.3758/s13428-021-01543-3
57. Slater M, Sanchez-Vives MV. Enhancing our lives with immersive virtual reality. Front Robotics AI. (2016) 3:74. doi: 10.3389/frobt.2016.00074
58. Dellazizzo L, Potvin S, Luigi M, Dumais A. Evidence on virtual reality-based therapies for psychiatric disorders: meta-review of meta-analyses. J Med Internet Res. (2020) 22:e20889. doi: 10.2196/20889
59. Meyerbröker K. Virtual reality in clinical practice. Clin Psychol Psychother. (2021) 28:463–5. doi: 10.1002/cpp.2616
60. Guillén V, Baños RM, Botella C. Users’ opinion about a virtual reality system as an adjunct to psychological treatment for stress-related disorders: a quantitative and qualitative mixed-methods study. Front Psychol. (2018) 9:1038. doi: 10.3389/fpsyg.2018.01038
61. Pot-Kolder R, Veling W, Geraets C, Lokkerbol J, Smit F, Jongeneel A, et al. Cost-effectiveness of virtual reality cognitive behavioral therapy for psychosis: health-economic evaluation within a randomized controlled trial. J Med Internet Res. (2020) 22:e17098. doi: 10.2196/17098
62. Wood DP, Murphy J, McLay R, Koffman R, Spira J, Obrecht RE, et al. Cost effectiveness of virtual reality graded exposure therapy with physiological monitoring for the treatment of combat related post traumatic stress disorder. Stud Health Technol Inform. (2009) 144:223–9. doi: 10.3233/978-1-60750-017-9-223
63. Best P, Meireles M, Schroeder F, Montgomery L, Maddock A, Davidson G, et al. Freely available virtual reality experiences as tools to support mental health therapy: A systematic scoping review and consensus based interdisciplinary analysis. J Technol Behav Sci. (2021) 7:1–15. doi: 10.1007/s41347-021-00214-6
64. Australian Government Department of Health. Therapeutic Goods Administration. Medical Devices Reforms: Medical Device Software Regulation. (2020). Available online at: https://www.tga.gov.au/regulation-software-medical-device (accessed January 5, 2021).
65. Takac M, Collett J, Conduit R, De Foe A. Addressing virtual reality misclassification: a hardware-based qualification matrix for virtual reality technology. Clin Psychol Psychother. (2021) 28:538–56. doi: 10.1002/cpp.2624
