PORTLAND, Ore. (Portland Tribune) — After months of issuing reports modeling the future behavior of COVID-19, Gov. Kate Brown and Oregon health officials last week issued a tacit admission that the pandemic had blindsided them.
On Friday, Nov. 13, Gov. Kate Brown announced a two-week freeze on public activity throughout Oregon, citing an “alarming spike” in COVID-19 cases and a modeling report bearing dire projections.
That’s significant because just two weeks before, the state’s modeling showed Oregon as having the coronavirus at a standstill — neither gaining or losing ground. Brown went from confidently describing plans to reopen schools on Oct. 30 to stressing on Friday that “it’s a very dangerous situation.”
Throughout the pandemic, Brown’s announcements about the virus often have coincided with the projections’ twice-monthly release. At times she’s cited the report to portray her decisions as backed by scientific certitude.
Emails released under Oregon’s public records law, however, show the job of running models for the Oregon Health Authority has been like piloting a high-speed luge through an obstacle course — often colliding with technical difficulties, unexpected results, and numbers that don’t quite add up.
Oregon’s projections of expected viral trends have swung dramatically at times, and never more so than last week. The latest report showed transmission — essentially the reproduction rate of the coronavirus in Oregon — jumping nearly 50% in just two weeks’ time.
For the first time since the start of the pandemic, last week’s Health Authority’s projections used different modeling software. Publicly framed as a matter of efficiency, the change was driven by months of stress and previously undisclosed difficulties, emails obtained by the Portland Tribune show.
The newly adopted software painted a significantly more alarming picture than other models were estimating for the state in the same time frame. Officials say the newly deployed software is more sensitive, and its findings are backed by real-world events. In the last two weeks, hospital intensive care units have nearly filled up thanks to severe COVID-19 cases — a trend likely to be followed by an increase in deaths.
The Portland Tribune and Pamplin Media Group’s papers are a KOIN 6 News media partner
State health officer Dr. Dean Sidelinger told the Tribune that he and other officials are confident in the modeling, but know its limitations that are detailed in public records.
“Models are only as good as the assumptions and the data that goes into them,” he said. “The modeling is definitely science based, but then there’s an art to how do we interpret it. … We use it for planning purposes, and I think for that it’s definitely helping us.”
In the beginning: An information gap
On March 11, Brown wanted information, and Sidelinger was under pressure to provide it.
Days before, Washington Gov. Jay Inslee had estimated that 1,000 people were infected with the coronavirus in his state. For a week, Oregon reporters had asked for an equivalent number in this state.
Sidelinger “urgently” wanted an Oregon estimate before he met with the governor that afternoon, wrote modeling manager Julie Maher in an email to the source of Inslee’s numbers, the Bellevue-based Institute for Disease Modeling, or IDM. Billionaire philanthropists Bill and Melinda Gates fund the nonprofit.
IDM couldn’t meet that deadline, Director Mike Famulare responded, but it would take the job. He stressed the urgency for Oregon to lock down.
“The only problem that is doubling in size every week is COVID,” he wrote, adding that it will “exceed the burdens of any other social impacts if left unchecked.”
On March 12, Brown declared a number of restrictions, including a ban on large gatherings. Four days later, she closed K-12 schools statewide. One week after that, she announced a new batch of restrictions titled “Stay home, save lives.”
A partnership was born: IDM agreed to work unpaid for Oregon. The state would plug in numbers of confirmed infections, deaths and COVID-19 hospitalizations each day. IDM’s epidemiological modeling software would spit out scenarios of what the disease likely was doing.
The projections had a purpose: to persuade people to change their behavior for the common good.
“Epidemiology is a science of possibilities and persuasion, not of certainties or hard proof,” journalist Charles Duhigg wrote in April in The New Yorker magazine.
On March 17, IDM produced its first preliminary report for Oregon. But state officials wanted them weekly, and Maher and research analyst Erik Everson soon took over the job.
Modeling: Educated guesses, tweaking, assumptions
Maher and Everson used IDM software called Covasim, plugging in data, then essentially tweaking assumptions so that the trend lines fit the data — a process called calibration. They then extended those lines into the future and offered different possible trends — while stressing that they were offering projections, not predictions.
The job entailed long hours gathering data, reading the latest research, and battling data discrepancies and scenarios that changed dramatically from week to week. Records show that the two analysts sometimes struggled to fit the software model to real-life data about outbreaks, hospitalizations and COVID-19 test results.
Some of the issues:
• On March 31, Maher asked IDM why numbers in IDM’s modeling calibrations didn’t match the cumulative infections being reported by the model.
• On April 6, Sidelinger said a draft IDM report “severely undercounts” the number of hospitalizations in Oregon.
• On May 1, Everson and Maher noted that updated software was producing “volatility” including “very different calibration results” based on the same data.
• On May 6, Maher and Everson wrote to IDM, warning that projections showing that diagnoses of infections were catching up to the actual projected number — an impossibility given that many cases of the coronavirus go undiagnosed.
• That same day they noted a “big downward shift” in projected numbers “between last week and this week’s scenarios (which) seemed rather surprising.”
• On July 7, Maher wrote “It looks like we (were) overpredicting hospitalizations for a while (mid-April —mid-June).”
• On Aug. 5, Maher informed Sidelinger of some erroneous percentages in a report published the week before, asking if the report posted online could be replaced.
• On Sept. 14, Everson asked IDM for help, noting a “considerable discrepancy” in the software’s daily tally of cumulative infections.
Expecting precision was misguided, IDM senior research scientist Assaf Oron responded to one of Maher’s questions at one point. The modeling “forecasts account for many sources of uncertainty … but as a simplified model for a complex reality they cannot account for all such sources.”
This, he wrote, is why IDM claims a confidence level of only 80% — meaning a wide range of possible outcomes. It didn’t “seem right to pretend to know ‘the truth with 95 % confidence,'” he wrote.
As for the bugs in the software, IDM frequently updated it, forcing the state’s analysts to adapt. Emails show they nipped and tucked assumptions to make the curves fit the latest data — routinely changing them as new reports were published.
Staff struggles to keep up
The stress in dealing with coronavirus modeling took its toll.
In a May 13 email to Sidelinger, Maher asked for a week off from modeling work, noting the challenge of the modeling’s “bugs,” upgrades and making the calibration curves fit the data. The emails are full of similar comments.
“The ongoing updates to fix bugs & improve the code has made calibration time consuming & comparing from week to week difficult,” she wrote. “We could really use some time to regroup, better understand the increased complexity of the Covasim code.… I also could benefit from a couple of days off to rest my brain.”
The state soon switched to twice-monthly projection reports of weekly reports.
In a July 14 email, Everson called the work “exhausting.”
Sidelinger told the Tribune the work is emotionally draining because of the stakes involved and the impact on people’s lives.
One of the “major limitations” of the state’s modeling is its reliance on week-old data to make sure it’s complete, Sidelinger said. That means the reports don’t always capture recent changes and can be out of date immediately.
Of course, the reports haven’t always dictated policy, either. Brown’s flurry of moves in May to reopen the state came immediately following a projections report noting the major risks of doing so. Documents later showed Sidelinger and his staff would have preferred she waited until the virus was under better control.
The modeling has helped Maher and Everson flag interesting developments using real-life data.
“We’re seeing changes in how the virus is spreading among different age groups, communities and localities in Oregon,” Everson wrote in a June 30 email to IDM, asking to discuss potential modeling improvements.
They also learned about the limitations of the modeling, even as medical professionals have complained that coronavirus tests often produce false negatives.
In an email to IDM, the Oregon analysts noted research finding a sensitivity, or accuracy, rate for detecting the virus of only 70% to 80%.
IDM, however, responded its model assumes 100% sensitivity.
One IDM supervisor in August conceded that the “rush” to provide modeling for use in the pandemic had led to “temporarily abandoning” more scientific approaches used in the past. “We do our best to make reasonable assumptions … but it remains a work in progress.”
After repeatedly exploring whether the state could stop issuing its own reports using Covasim and instead rely on other sources, emails show, Maher in August began asking about software called Rainier. The much simpler program, also produced by IDM, doesn’t model hospitalizations or deaths, simply projecting the growth trend.
On Sept. 17, three days after Everson sent an email notifying IDM that the numbers again were not adding up in Covasim, the nonprofit sent an email giving the state access to Rainier.
In the future, reports will come out once every three weeks. And they won’t be nearly as specific, allowing “for a more timely turnaround,” said the state’s latest report.
Already used in Washington, how the different software performs in Oregon remains to be seen. Sidelinger said that considering the surge of cases in the last week, “that modeling report is probably on the optimistic side of things.”