Inside Oversight

Adam Roy:
Hello, this is Adam Roy with the Veteran Affairs Office of Inspector General. You're listening to Inside Oversight, our podcast that dives into published reports and features interviews and discussions with the OIG employees who did the work. Today, we're covering a recent management advisory memorandum published on the topic of wait times, specifically how the Veterans Health Administration calculates and discloses them. To help me do this, I've got Daniel Morris, a director with our Audits and Evaluations Office, joining me. Daniel, go ahead and introduce yourself and tell the listener a little bit about what you do here for the OIG.

Daniel Morris:
Yeah, thanks Adam. As you said, I'm a director in the Office of Audits and Evaluations. I've been conducting this oversight worked for VA OIG for about 15 years now. Many of the projects I've been involved in over that time of focused on the area of access to care, including patient wait times, appointment scheduling practices, consult management, Community Care programs in VA.

Adam Roy:
Great, glad you're glad you're here with us today. Now the VA OIG has published previous reports on wait times, and it's certainly a topic that generates a lot of interest on the Hill, among veterans service organizations, and among veteran patients and their families. So, let's just start with what's different about this report?

Daniel Morris:
Yeah, you're right in that it's a different than prior reports and assessments. This one focused on how VA publicizes, really how they calculated and disclose its appointment wait time data. So, unlike audits and reviews of the past, this work didn't assess the timeliness of appointments or staff compliance with scheduling policies or procedures, for instance, but really looking at how they disclose and calculate their VA appointment wait times data. It's complex. It's complicated, a lot of different terminology. And there's so many ways to do it, so many ways to calculate appointment wait times, and certainly no one standard way that everyone agrees on. So, since 2014, a law dictates that VA publish data about patient’s wait times for medical appointments in the VA facilities. So, they do this, VA does this on a dedicated website. Certainly, fair to say that calculating these times can be pretty complex as I mentioned. It's been challenging for VHA to consistently implement a suitable methodology that really reflects the amount of time patients wait and to be transparent and accurate in that reporting. And so that's really what this report focuses on, Adam.

Adam Roy:
Perfect, and let's talk about that and what led you and your team down this path. But first, let's share with the listener a little bit about the management advisory memorandum or as we call it the MAM, you know what it is, and you know why we do them.
Daniel Morris:
Yeah, so what we're doing with our results in this project is we're laying them out in the in the MAM like you mentioned so that goes from the OIG to the department, and the IG we issue these memos when it when important circumstances on areas of concern are identified in our work, in the course of our work, particularly when it's an immediate action by VA can help reduce any further risk or issues. So, this memo in particular, it addresses concerns that were raised about how VHA was reporting on wait times for appointments, and it's really meant to alert VA of what we identified in the course of that examination. So, memos such as this one, they're published just like any other report unless otherwise prohibited. And it's hoped that the information that we have in this memo will really help VA advance its efforts.

Adam Roy:
Excellent. Thanks, Daniel. So, let's go back and, you know, how did this all get started? How did your team get involved?

Daniel Morris:
So, this memorandum addresses concerns that were raised about how VHA was reporting wait times for appointments, and we received these concerns in June of 2021 when a complaint alleges to us that two years prior, in the fall of 2019, the then acting principal deputy under secretary for health had been informed that VHA's wait times reporting may be misleading, but then the VHA took no action in response to that.
So, after our initial examination, what we found was there was no basis to proceed with the misconduct investigation of that particular VHA official. We found no evidence of an intent or an effort to mislead. We did find that there were some efforts to improve the wait time disclosures. They had been under consideration, but based on interviews, they'd been deferred by other priorities such as the COVID-19 pandemic.
Each of the witnesses that we interviewed told us that that the VHA official in question had been supportive of the efforts to refine that approach to measuring wait times. Uh, in fact, that they had instructed other staff to update the disclosures on the websites to account for new methodologies, but we found that that action had not been taken yet.

Adam Roy:
OK, so it's in kind of, in summary, your team determined, you know, at first that no misconduct, uh misconduct wasn't present, and it was other urgent priorities, you know, obviously the pandemic that delayed VHA's efforts to update its website, specifically on how wait times are calculated, what methodologies are deployed. However, though, as the report reveals, you did observe that VHA failed to present wait times to the public clearly and consistently and as the report indicates VHA calculated wait times multiple ways.

Daniel Morris:
Right, Adam, and that's what we want to get across in this in this memo to VA. Since at least 2014, VA used different, several different methodologies really, particularly using different start dates for calculating the wait times that they report online as well as, you know, when they're determining community care eligibility criteria.
So, we found that the methodologies that VHA used sometimes deviated from their own policies and their own announcement that they made in 2014 about how they would calculate wait times. Uhm, and by doing that, by using inconsistent methods and start dates and not clearly describing them when you do post that information, it can be misleading and misinterpreted by the user. Uh, certainly, it affects the overall calculations, depending on when you start that calculation. And, like I said, whoever is using that information and assessing what the wait times is perhaps not getting the full picture depending on what calculation is used.

Adam Roy:
Big picture-wise, this report details several observations made by your team, in a way that the main problems we want to share, that you want to share with the VA leaders. Can you briefly describe these areas these observations that were made, and then we'll go into more detail on some of those?

Daniel Morris:
Yeah, and there are some overlapping issues here. Overall, the team made the following determinations. I'll put him in four categories. First, VA developed the standards in 2014 for how they wanted to calculate what they publicly reported wait times. So, at that time they decided to use the provider’s clinically indicated date or, if that wasn't present, a veteran’s preferred date.
The second point, this MAM shows that when and how VHA uses an inconsistent or different start date for its calculating wait times it's going to be potentially misleading and result in inaccurate reporting.
A third, VHA eventually published the wait time data based on start dates that were inconsistent with its policy and previously stated methodologies. And they were also inconsistent with the description that they had on their own websites.
And then the final main point that will make in this MAM is that VA implemented new access standards in 2019 under the Mission Act for determining community care eligibility in which they decided to use a different start date, and that's going to be the date of the appointment request.

Adam Roy:
OK, so let's start with those standards. You said, you know, as a result of the Choice Act in 2014, VA has been publishing wait times on a public facing website. The VA 's methodology for calculating wait times was reported to Congress, and I want you to share with the listener sort of the process on how VHA reported wait times and then break down the different defined dates that were and are being used to identify the start of a wait period. Or maybe just, simply put, how is wait time calculated?

Daniel Morris:
It gets technical. It is important really to first understand the general process and various start dates that VA used in determining their wait times. So, for VA facility appointments, requests for care can be made by the veteran themselves or a lot of times it's by a provider requesting care by way of a consult or an order. Then a scheduler identifies an available appointment date and time for that requested clinic and creates the appointment in the scheduling system. So, what VA does with this data is that it aggregates into and produces reports and that's the way that they showed the timeliness of its care. Uhm. Their appointment scheduling processes, they include several different time stamps, and this is where it gets a little bit complicated uh because they've used a lot of different timestamps when they measure these appointment wait times, especially in the with the starting dates. And so, in our report we talked about a number of those different timestamps, including the request date, which—like I mentioned—is usually when the veteran or provider asks for that appointment. The clinically indicated data is something that the provider would put in a consult indicating when the veteran should be seen.
A preferred date is uh appears when a veteran requests for specific date if there's not a clinically indicated date provided.
And then a create date really represents when the scheduler took the action to make that appointment in the scheduling system. And then you have the appointment date, which is when the care was delivered.

Adam Roy:
Okay, Daniel, so VHA 's used request date, clinically indicated date, preferred date, create date, appointment date. You know, these different start, these different dates to identify start times and ultimately wait times. What they used in 2014 may have changed over the last several years and in as there's not been a lot of consistency in their approach.

Daniel Morris:
Yeah, so over the past several years VA used several different methodologies in calculating the wait times for different purposes. Even measured internally prior to 2014, uh, they generally used a measurement from the create date. Since then, they've used a number of different methods. And they all really depends on what are their posting it and how they're describing it, that that's really important to the user that's reading what the wait times are, but generally speaking, you're right, Adam. They've used different methodologies for different purposes.

Adam Roy:
And what isn't acceptable wait time? Did the VA set a goal for that?

Daniel Morris:
So, there's no one standard methodology that everybody is agreed on. The Choice Act defined VA’s wait time goal—so this was in 2014. The Choice Act defined the goal is being no more than 30 days from the date which a veteran requests an appointment.
The Choice Act also allowed VA to develop its own methodology, which it did, so long as they reported that to Congress and made that notice in the Federal Register. So, VA did that in 2014 on the heels of the Choice Act and they decided at that time that they were going to post wait time data by facility on their public website and that they were going to use the preferred date for both new and established patient appointments. That's how they were going to display their wait times.

Adam Roy:
Okay, and then in 2017 VHA began posting wait times on a newer public website, The Access to Care website. In here, the start date to calculate wait times was not consistent with the original website’s methodology nor consistent with what was presented to Congress and described in VHA published policy. Can you talk about the differences there?

Daniel Morris:
Right. So, starting in 2017, we have VA posting wait time data on a newer different public website while the other website is still up and both available to the public. So, VA announced that this new website, which is called the Access to Care website, it was really aimed at giving the users, really giving veterans, a specific view of, a user-friendly view of, what wait times were at that at their own facility for this specific service that they were looking for, whether it's primary care or a specific type of type of specialty care.
So now in 2017, you have two public websites with the wait time data that are using different methodologies.

Adam Roy:
Okay, and so these different start date definitions that you’ve talking about we've kind of covered here. The report does a really good job outlining. It provides a good example. Let's figure one for reference if you're going to go find this report, listeners out there, that talks about different starting points result different starting points result in different way time calculations. Can you sort of walk through that example for the listener, kind of help provide sort of a map on how different, using different start dates lead to different data points?

Daniel Morris:
Right, it can be confusing, which is why it's important that VA clearly really indicates what starting points they're using when they're displaying or presenting this information.
And so, I'll provide you an example. But just generally, you know, the intervals between that that request state that we've talked about or, you know, that create data or even the clinically indicated date to the point where the patient is actually seen—those intervals can vary greatly and it really just depends on the facility and the service and the timing of it.
So, for example, we've got a scenario in the report, an example that you mentioned, illustrating, you know, how you could include different wait times for the same appointment. Uh and so this example, let's say it's requested on June 28th and it's a VA provider requesting a specialty care appointment, a cardiology appointment, for the veteran. So, they’re requesting it on June 28th. They put in their consult and that they enter a clinically indicated date of the same day, which is not unusual, typically means the provider wants this patient to be seen as soon as possible.
OK, so then that consult goes over to the cardiology department. In this example a few weeks later, on July 21st, so about three weeks later, the appointment is then created by the scheduler in that department. And then uh again several weeks later, on September 2nd, the patient is actually seen for an appointment.
So, in this example we have a uh request day, we have appointment create date, and we have an appointment completed date. Uh if the wait time is reported using the request date to the appointment date, so from point A to C, I'll call it, it would show a wait time of 66 days. However, if you're reporting wait times using the create date, when that appointment was created, so really kind of B to C, it's going to show a shorter wait time of 43 days and not include that first window, which I'll call is the time to schedule the appointment.

Adam Roy:
Right and then that in the report, you indicate, the report indicates that there's multiple instances where the create date was made two, five, even more days after the initial request date. So, what sort of impact did that have on calculating wait times overall?

Daniel Morris:
Correct, and this is just one example. Our look through VA appointment data shows there's varying gaps from the date of request to the appointment create date. You know, sometimes it could be the same day, but usually that's not the case. There's a there's one, a few, several days that pass until, between that request and the appointment create date So what VA’s missing when it rolls up their average wait times data in this way is really that time to schedule. And so, if you think about it from a couple different perspectives, from first from the veteran’s perspective, you know, they're looking at it as, you know, “I requested or my provider requested me to be seen in this service on this date. I'm waiting from that point in time, that request date.”
Uhm. However, if and when VA is calculating using starting on that create date, then they're then they're not including that time that it takes, however long, a few days, several days, a couple weeks, to get that appointment scheduled in the system. So, they're missing out on that on that portion of the report of the of the time that the veterans waiting.

Adam Roy:
So that's going add some confusion and then if we return back to the 2017 Access to Care website, uh we have a situation here where the report described how VHA calculated average wait times as vague. Could you expand upon that?

Daniel Morris:
Yeah, vague because it doesn't directly state what's used for new patient appointments. What it does, what the description suggests, is that wait times for new point patient appointments are calculated from the request state. Uh the site goes on, and I'll quote the site, VA’s website here, it says,
Request is the most accurate measure for new veterans because it represents the actual veteran, actual average days the veteran has waited for an appointment.
However, what we found in our analysis of this website data for new patient appointments is that VHA actually measured them from the create date and not the request date as the website purports and as we talked about in the example you know, we now know what the gap what the difference is between those two.
Further, supporting the analysis that that we had were our interviews with VHA officials and when they told us that they've been calculating the wait times for new patient appointments from the create date on this website since it was created in 2017.

Adam Roy:
So, for the last 4 years, give or take, VHA has operated two websites that calculated wait times differently.

Daniel Morris:
Right, and so they have the initial website, public website and now this Access to Care website, a few years later using two different methodologies. But the initial website was still up and running, and so for we have a good comparison chart in our report shows that for a four-year time period they had both up and both representing VA patient appointment wait times but using a different measurement on each. So, it can get confusing and I, I think I mentioned this earlier, when VA’s not clear on what it's using it can certainly be misleading to the user of the data, whether that's the veterans or media or other stakeholders. So, with this website, with this Access to Care website, what we have is a description that suggests one thing and it's measured from another. And, you know, given the sensitivity of wait times over the past years and they attention that it's brought, it's just really important for VA to be as transparent as possible on what they're displaying and what these wait times represent.

Adam Roy:
Absolutely, that makes sense. Let's change gears a little bit. So, the report also talked about a 2019 study that attempted to compare VA’s wait times with wait times in the private sector. Can you, kind of, briefly share some background on that?
Daniel Morris:
Yeah, so in addition to the publicizing the wait times on the access to care website, in January 2019 JAMA published a study and it was authored by then current and former VA leaders, and what it did was compare wait times for new patient appointments in the VA versus that of the private sector. The study methodology section stated that the VA wait times were calculated from the day that a veteran requested an appointment. So, again we have that request date described in its methodology. Now, we weren't able to precisely verify what method was used for certain based on the data. However, according to the VHA, according to VHA officials that we interviewed, the data VA used was measured from the create date, and again we have the difference there at we have talked about already.

Adam Roy:
OK, I kind of staying in 2019, let's kind of hone in on the Mission Act a little bit, which expanded veterans access to health care in the community or at non VA facilities. Uh, you know, specifically the Mission Act implemented new standards. And so how did those how did these new standards relate to wait times?

Daniel Morris:
Yeah, so the Mission Act, like you said, it continued community care, but there were some adjustments to the eligibility criteria, eligibility requirements. So, VA created their regulations for the Mission Act that went into effect in June of 2019, like you mentioned, in which one of those, one of those pieces of criteria, one of those standards is that a veteran is eligible for community care if the VA can't schedule them in house uh within 20 days for primary care or 28 days for specialty care, for instance. And within 20 days and 28 days of what? Well, this one is of request date. So, VA’s regulations that they created for the Mission Act standards is starting with the request state.
So what VA is using to determine, it's a different earlier start date than the point that they're measuring from for average wait times displayed online right, and again it's measuring from that request date whether that's a request from a VA provider when they enter the consult or requests from the veteran directly.

Adam Roy:
So, there's it's, it's two different approaches.

Daniel Morris:
Two different approaches, and there's an important distinction here, uh, because, you're right, the average wait times posted online are not the same as community care eligibility determinations. Two different purposes, two different approaches. Uh. VA’s reported wait time data online that we've talked about leading up to this, those are retrospective appointment data. Those are completed appointments that already happened, uh, generally based on the last months of appointments, and again, like we talked about, there's those who are measured from the create date.
Now, with Community Care eligibility determinations, they are looking at the schedule and real time, staff are looking at the schedule in real time taking that request, whether it's from the veteran or provider, and looking ahead in the schedule and determining “Can, can this VA clinic see this patient within the 20-or 28-days standards?” And if they can, they will schedule them. If they cannot, then it's a discussion with the with the patient, with the veteran to say, “Now, you know, we can't see you within the standards, you know the next available is say 35 days. Now, you have the option. Do you want to explore seeking community care or do you want to wait for a VA appointment?” So that's the difference there.

Adam Roy:
OK, now will VHA, you know, work to align wait time calculations with these community care standards?

Daniel Morris:
It appears so because what VA has done with having two different approaches and two different purposes is really caused a lot of confusion inside and outside VA.
And so, there's a lot of confusion, a lot of conflation between average wait times posted online versus “Does that equate to am I eligible for Community Care?” So, the short answer to that is no. They don't relate. It’s two different purposes two different measurements. But to avoid that VA has recognized that there's some confusion there, and what they've told us is that since 2019 they've been discussing how to align their average wait time data that they post online to how they're determining community care to make it more apples-to-apples because right now it's not, and they want to be consistent.
Uh, so that means they're really wanting to measure the average wait times online from the request state too as much as possible. So, they've acknowledged that when we interviewed them in July, June and July of 2021, they indicated that they were still discussing the matter. We mentioned earlier that, you know, that this had been deferred because of other priorities such as COVID, but now they're rediscussing the matter again and it appears that they're moving that way. However, no changes have been made as of the time that that we drafted our report.

Adam Roy:
So, Daniel, what needs to happen next to improve weight time consistency and transparency?

Daniel Morris:
Well, the bottom line is VHA needs to consider a consistent methodology that they're going to be able to accurately describe, and that's really also going to help them out, help them communicate those methods more transparently in the future.
And not get into uh conflating the two and, you know, public users being confused on one versus the other and how they relate or don't relate.

Adam Roy:
Thanks, Daniel. Is there anything else you want to add today?

Daniel Morris:
Yeah, so just to wrap up, I mean, given the significance of wait times in the past and the attention that it gets and you know, how useful this information can be to veterans and other users. You know, it's really important that, like we mentioned before, VA is clear and transparent in what they're posting that that really gives veterans a true understanding of what they're looking at.
You know, by putting this information in the MAM, you know, we're hopeful that it's it gives a good clear walkthrough of the processes and the challenges involved in, and for VA leaders to take it and potentially continue their improvement efforts on making this as clear as possible for themselves and for the veterans. Uh, you know, with the MAM, there's no formal recommendations like we would a typical report but we do ask VA that to let us know what their plans are to take this information into uh improve upon these issues. And we’ll monitor that. We’ll, we will continue to monitor, you know, any potential updates that they're inducting on their website as we think that they're doing. You know, potentially revisit this topic in the future, depending on those actions. Like I said, it's an important topic for veterans and other stakeholders. And for those listening, the report that you'll be able to find online, it's called Concerns with Consistency and Transparency in the Calculation and Disclosure of Patient Wait Time Data. Uh and, Adam, it’s a really good opportunity to, you know, discuss this topic with a lot of senior VHA officials and really get their perspectives and understandings and really understand the challenges behind this issue.

Adam Roy:
Daniel, appreciate it. That's really good insight. As Daniel mentioned, you can find Concerns with Consistency and Transparency and the Calculation and Disclosure of Patient Wait Time Data on our website, www.va.gov/oig. Appreciate your time, Daniel, and thanks for stopping by.

Daniel Morris:
Thanks, Adam