Anthony Neumann – City of Oshkosh

One chilly morning in January 2020, Tony Neumann got the call every IT director dreads. Some strange splash screens—also known as boot or welcome screens—were popping up on devices given to employees of the City of Oshkosh, Wisconsin, where Neumann has worked since 1999. Before long, staffers couldn’t use their devices. Neumann tried to remote in and couldn’t break through at all.

So, he had the reporting staffer physically shut everything down. Then he drove to the office, and after looking at several servers and seeing what was on them, he knew. It was a ransomware attack.

“From the initial notification on a field device to the time I got there, there was a few hours’ delay, and by then the damage was done,” Neumann recalls. “Right away, I convened with the city leadership, and we activated the emergency operations center to declare an emergency.”

Anthony Neumann | IT Director | City of Oshkosh

Everything connected to the network at the time of the attack was down. That included law enforcement systems and all other workstations: 19 servers and 449 endpoints “were basically boat anchors,” Neumann says.

Once the emergency was declared, Neumann and his team could make emergency purchases without going through the city council. But the city had already made its most important purchase: cyber liability insurance, which Oshkosh had just bought the previous year.

The insurance company looped in a legal firm and a forensic analysis team, and Neumann and his colleagues had to wait while the forensic team gathered the evidence it needed. Meanwhile, two field agents from the Federal Bureau of Investigation and the cyber response team from the Multi-State Information Sharing and Analysis Center—a fusion center run by the Department of Homeland Security—joined Oshkosh law enforcement in responding to the attack.

Neumann never contacted the email address that appeared on the splash screens, so he never found out what the ransom was. But the forensic analysis revealed that the cyberattack likely came from Russia.

“Unfortunately, the nature of these things is that you don’t actually catch them,” Neumann says. “The best you can do is take all the servers that they’re using and blacklist them.”

Taking stock and learning lessons

As Neumann and his team were picking up the pieces, they identified some of the strengths the city had in weathering the attack. One was its disaster recovery backup plan, which included key phone numbers and a sequence of people to contact. Another was its financial planning—the city paid a $10,000 deductible on its cyber liability insurance, and insurance covered the rest of the $132,000 in expenses.

Then there was the IT team itself.

“They all came together, and everybody basically worked as PC hardware technicians, which, for some of them, was way outside their wheelhouse,” Neumann says. “They were able to adapt very quickly.”

Neumann presenting at a community workshop

Learning the lessons of the ransomware attack also required Neumann and his colleagues to identify areas for improvement. For example, they realized they needed to do more end-user training and retain more workstation images.

Today, the IT department retains copies of all images on every vintage machine, because the hardware platform changes every year. Unlike servers, on which one can use the backup and restore method, workstations are a case of “set it and forget it.” The fastest way to get them back up and operational after an event is reimaging, which delivers the preconfigured antivirus and Microsoft Office software as well as the relevant domain information.

Endpoint protection and user training

As soon as they’d recovered, Neumann and his team implemented end-user training, the first year of it paid for through the cyber liability insurance. They did a baseline test first, and Neumann says the results were “eye-opening.”

They now do a mandatory annual training program as well as random monthly simulations. For instance, a recent simulation involved phishing emails timed to coincide with the planning of Oshkosh’s capital improvement budget; some people fell for the fake capital improvement plan emails because the timing was right.

“Month by month, we have seen the click rate or the data entry rate drop significantly,” Neumann says. “But we still have people that fall for some, because as time goes on, we get more clever in crafting the emails.”

City Hall, City of Oshkosh

The IT department also upgraded its endpoint protection product. The previous product had been a signature-based antivirus protection system that failed to ward off the ransomware attack.

“It only takes me once to learn a lesson,” Neumann says. “[This new product] is advanced enough to where, for example, if it sees ransomware, it will actually shut the network card down on that machine, so that it can’t propagate across the network. And then they contact us and say, you’ve got a device that we just quarantined. And then we would go physically unplug the network cable and see what’s going on with it.”

The new product was highly recommended during a conference of the Governmental Information Processing Association of Wisconsin, a statewide organization Neumann joined decades ago comprised of IT leaders from all of Wisconsin’s cities, counties, villages and state agencies.

Building to last

Through nearly 25 years with the city, Neumann has come to love collaborating with all facets of local government. The top virtues of his job, as he tells prospective new hires, are its variety and its sense of higher purpose.

“I’ve grown to call this home, and I’m hoping to retire from this position,” he says. “And I’m very fortunate in that nearly all my staff has got at least a decade here.”

IT Department Receiving Employee of the Year after Ransomware Recovery

Neumann is not just looking for staff with good credentials. He’s looking for people with a passionate commitment to the trade they’ve selected: “Because if they don’t have the passion, they’re not going to last,” he says.

But day in and day out, he takes great pride in the work his department has done. They’ve helped the whole city, he says, and have led the government in adapting to today’s technological environment.

“While we’re not manufacturing anything, we’re still building something,” he says. “We’re building an infrastructure for the future of the city to ride and thrive on.”

View this feature in the Summer I 2023 Edition here.