Dallas police evidence loss could have been easily avoided, experts say

By Michael Williams and Maggie Prosser
6:00 AM on Sep 16, 2021 CDT

Original article: https://www.dallasnews.com/news/2021/09/16/dallas-police-evidence-loss-could-have-been-easily-avoided-experts-say/

While local and federal authorities investigate the loss of at least 22.5 terabytes worth of data, mostly Dallas police investigative files, the inquiries seek to answer two basic questions: How was it possible for one employee to cause the massive loss, and what could be done to prevent something similar from happening?

Experts in information technology and cybersecurity who spoke with The Dallas Morning News said a loss of that magnitude could have been easily mitigated — or avoided — had basic safeguards been in place to protect sensitive data.

“I’m disappointed in the lack of controls, disappointed that this happened and surprised that such a major error could have occurred,” said Dr. Costis Toregas, director of The George Washington University’s Cybersecurity and Privacy Research Institute. “If it was a small community with a part-time IT guy, I could understand, but we’re talking about the city of Dallas.”

The city said the employee who was responsible for the missing evidence had lost data on at least three occasions, prompting the FBI to open its own investigation into whether it was purposeful. The Dallas Police Department previously cleared the employee of intentional wrongdoing.

The employee, who has declined to comment to The News, has not been charged with a crime.

‘Failed to follow established procedure’

The first known deletion occurred after the employee, who has since been fired, “failed to follow proper, established procedures,” the city said in a written statement. The loss happened in late March, when the employee was supposed to move 35 terabytes of data from online storage to a physical city drive. The procedure was expected to take about five days. But the employee “failed to follow established procedure” and wound up deleting the files from the city’s network drive, Dallas Chief Information Officer Bill Zielenski told city officials during a meeting last month.

The employee canceled the deletion when colleagues told him files were disappearing. By then, 22 terabytes had been lost.

Officials said 14 terabytes were recovered from that initial batch of data. But late last month, officials uncovered an additional 15 terabytes of data that had been missing.

Officials say the current loss is about 22.5 terabytes of data, the equivalent of about 7,500 hours of HD video; about 6 million photos; or 150 million pages of Microsoft Word documents. But an ongoing audit, which is due to be finished later this month, could reveal more.

The city did not alert the Dallas County district attorney’s office about the loss until early August. District Attorney John Creuzot then wrote a memo to defense attorneys about the missing evidence, which made the public aware of what had happened.

The audit also revealed the employee had a “pattern of error” and had lost data on at least two other occasions.

In a memo last month, Dallas City Manager T.C. Broadnax outlined new policies, including informing city leaders of any data compromises within two hours of learning about them. Two IT employees will now oversee the movement of any data. Also, a 14-day waiting period will be instituted before data is permanently deleted, and a review will take place to analyze how the city stores and archives data.

Data should have been backed up more than once

Toregas said having two employees overseeing the movement of any data should have been the procedure to begin with. Other procedures that could have mitigated the loss, Toregas said, include aggressively managing the directory of people who have access to the data and segmenting data so large swaths can’t be affected at once.

Andrew Wildrix, chief information officer of INTRUSION, a Plano-based cybersecurity firm, said that if he had to guess, the employee moved the data instead of copying it. If that’s what happened, Wildrix said, it was a fundamental mistake.

“I would imagine that an organization of that history and size would have had safeguards put in place, but it’s obvious those were ignored,” he said.

Storing large amounts of evidence, files and police body camera footage on antiquated, physical hard drives has been outpaced by the volume of digital evidence, said Johnny Nhan, a criminal justice professor at Texas Christian University.

Nearly every police investigation now has a digital component, whether it’s a laptop or cellphone recovered at a crime scene or digital evidence, which needs to be copied onto police servers and preserved, Nhan said.

“Any type of crime scene requires some sort of data storage, so that’s becoming more and more important,” he said. “As there’s more computing requirements, storage is going to be an issue, just preserving digital data is going to be an issue moving forward.”

Nhan said modernized data storage practices include paid “cloud storage services,” which automatically back up data and upload large amounts of footage like police body camera video. These systems create copies and “multiple levels of redundancy” to data, which help to restore files if they are lost.

“If [police departments] have a diligent IT department, then that data would be backed up more than once,” Nhan said.

Ed Claughton, chief executive officer of PRI Management Group, a firm that provides criminal justice agencies with records management, IT and crime data consulting, said data loss usually happens when data is being uploaded or migrated to a new system or server, but he’d never seen such a large amount of data go missing.

Claughton said the best mitigation effort against losing data is having a “two-party validation process,” in which two people must approve or review each step of transferring or deleting data. He also suggested departments “map” where data is being stored and back up any critical data before deleting it from somewhere.

The city said it is conducting a “top-to-bottom assessment” to improve its systems and processes, and now requires that two people, not one, handle file transfers to ensure no steps are missed — in line with Claughton’s recommendations.