By Lindsey Hendrix

A team at the Texas A&M University Health Science Center (Texas A&M Health) led by Joshua Kissee worked day, night and weekends to quickly develop a new system for scheduling vaccine appointments at the Brazos County Community COVID-19 Vaccination Hub, which just surpassed a milestone of vaccinating 65,000 people.

In Brazos County, work started with the formation of the COVID-19 Vaccination Task Force on Jan. 6. Members of the task force knew they were going to have to vaccinate about 100,000 people and would need a system that could handle scheduling them all.

“We can set up all of the pieces we need to operationalize the plan, but if we can’t get people there, then we’re not going to be successful,” said Monica Martinez, task force member and emergency management coordinator at Texas A&M. “Part of an emergency manager’s job is to look at resources, who we know and what they can do. I knew the Office of Information Technology at Texas A&M Health had a system that might work for this, so we started those conversations.”

A Mad Dash

In early January, Kissee received a call from Greg Hartman, chief operating officer and senior vice president of Texas A&M Health, asking if his team could develop a scheduling system for the hub. It needed to collect specific information about individuals requesting appointments, confirm appointments across multiple parties and meet security standards to comply with the Health Insurance Portability and Accountability Act (HIPAA).

“I didn’t know if we could solve it, but I told him we were willing to work with people to see if we could actually make this thing happen,” Kissee said. “And so that really kicked off this kind of scramble in the very beginning. We wrote that software in about eight days which is the fastest I have ever seen software be developed.”

Kissee’s team was able to develop the program so quickly thanks to a head start provided by REDCap, a secure web application the university uses to build and manage online surveys and databases, mainly for research. The team had experience using the program for collecting and reporting COVID-19 testing data over the summer and had already migrated it to an environment that could scale up to meet a high demand. That migration turned out to be critical for the new vaccination scheduling software, which would eventually receive overwhelming traffic.

The team piloted the registration system to about 300 front-line workers on Jan. 28 — just three weeks after the task force formed. On that day, two developers sat at the clinic with laptops and made code changes on the fly based on immediate feedback from the operations team. They continue to fine-tune the software as new quirks emerge.

One of the variables the team did not account for at first was instances when multiple people in the same household share an email address. The system was designed under the premise that a person’s email address is their identity.

“There was some real custom coding that had to be put into place to accommodate for that,” Kissee said. “That was one of those curveballs that we had to react to, but we never would have known it without actually being there at the clinic having conversations with the people that are there, getting all of these stories and being responsive to design for that.”

Going Live

The system went live in early March for anyone in the county eligible for a vaccine under Phase 1A and 1B of the state’s rollout plan (people aged 65 and older, and people 16 and older with risk for severe COVID-19 illness). When eligibility opened to people 50 and older on March 15, traffic to the system was so immense that it crashed.

“That’s when we saw about 60,000 requests in a 60-second period, which is pretty heavy,” Kissee said. “We scratched our heads because we designed REDCap inside AWS (Amazon Web Services) to scale, and it did scale, but the database wasn’t working just the way that we wanted.”

At the same time Kissee and his team were scrambling to solve the problem, they learned that in just four days, Texas was going to open eligibility to all people over the age of 16.

“I don’t think I slept too much that weekend, and neither did anyone on our development team,” Kissee said.

Finally, they came up with a solution. Inspired by a system that has been used in health care for years, the team built a digital waiting room. To protect the database and prevent traffic from overwhelming that piece of the system, the waiting room feature only accepts a set number of people into the system per second. For the user, this displays as a countdown to their turn.

But it didn’t work well the first time they tried it.

“There were some timing issues with how people accessed the registration queue during extremely heavy registration traffic. And so, the team really went to work at a very hard solution,” Kissee said.

The Big Launch

At 9:45 a.m. on March 26 — 15 minutes before the scheduling website opened to the entire population of Brazos County — a group of 11 people gathered on a call to watch and monitor the system. They included Kissee, two developers, three system administrators, a technical architect, a communication professional and three people from AWS.

They watched as about 160,000 requests hit the system in a 60-second period. “Those were unprecedented numbers that no one on our team had seen before and that we don’t see at Texas A&M very often.”

A flurry of vaccine appointment registrations equivalent in size to most of the entire population of College Station swarmed the system for the 5,000 vaccines the Texas Department of Health and Human Services allocated to the hub that week.

“Fortunately, after all the problem-solving we did end to end, the system worked very well and all the appointments were filled in about four hours,” Kissee said.

The team now feels confident they can handle the load to the system and will continue improving it through the end of the program.

“I’ve been doing this for a very, very long time,” said Task Force Chief W. James Stewart. “I was managing things before there was computers. Over the last 40 years I have had a lot of experience with IT divisions both in the military, at the university and there at Brazos County, and this is by far one of the best. As far as we were concerned as Josh’s customers, we never got excuses. He just fixed the problems as they came up.”

“I think the most important thing throughout this process has not been technical. It has been the partnership with all the diverse stakeholders across the hub and us listening and being responsive to their needs to code and build this software,” Kissee said. “The magic to the sauce is bringing all the skill sets together and then having a strong focus on the people that we’re working with, listening more than we’re speaking and focusing on trying to solve problems.”

This article by Lindsey Hendrix originally appeared on Vital Record.