The AI Police

A new company makes artificial intelligence software that’s in use at a handful of police departments. Can it make law enforcement more transparent?

By Bloomberg LP

here’s a story Brett Goldstein ’96 likes to tell. It starts on a Friday night in 2010 with him sitting in a darkened Crown Victoria on a Chicago street, poring over maps. Goldstein was a commander at the Chicago Police Department, in charge of a small unit using data analysis to predict where certain types of crimes were likely to occur at any time. Earlier that day, his computer models forecast a heightened probability of violence on a particular South Side block. Now that he and his partner were there, Goldstein was doubting himself.

“It didn’t look like it should be a target for a shooting,” he recalled. “The houses looked great. Everything was well manicured. You expect, if you’re in this neighborhood, you’re looking for abandoned buildings, you’re looking for people selling dope. I saw none of that.”

Still, they staked it out. Goldstein’s wife had just given birth to their second child, and he was exhausted after a day in the office. He started to doze off. Goldstein’s partner argued that the data must be wrong. At 11 p.m., they left.

Several hours later, Goldstein woke up to the sound of his BlackBerry buzzing. There had been a shooting—on the block where he’d been camped out. “This sticks with me because we thought we shouldn’t be there, but the computer thought we should be there,” said Goldstein. He took the near-miss as vindication of his vision for the future of law enforcement. “I do believe in a policeman’s gut. But I also believe in augmenting his or her gut,” he said.

Seven years after his evening on the South Side, Goldstein threw on a gray suit and some aerodynamic sunglasses and headed out from his hotel in Midtown Manhattan into New Jersey. This spring, he founded CivicScape, a technology company that sells crime-predicting software to police departments. Nine cities are either using the software or in the process of implementing it, including four of the country’s 35 largest cities by population. Departments pay between $30,000 a year to use the software in cities with less than 100,000 people to $155,000 a year in cities with populations that exceed 1 million. Goldstein wanted to check in on the two clients who were furthest along—the police departments in the New Jersey towns of Camden and Linden.

Goldstein likes to harp on his own lack of charisma, but he’s well suited to be a pitchman for police departments. In Chicago, he rose from patrol officer to the city’s chief data officer over a seven-year government career and regularly drops a few war stories from the streets into his conversations with cops. He’s also peddling something that every department is after nowadays: technological sophistication. The criminal justice system produces reams of data, and new computing methods offer to turn any pool of numbers into something useful. Today, almost every major police department in the country is using or has used some form of commercial software that makes predictions about crime, either to determine what blocks warrant heightened police presence or even which people are most likely to be involved. Technology is transforming the craft of policing.

Not everyone is rubbing their hands in anticipation. Many police officers still see so-called predictive policing software as mumbo jumbo. Critics outside of law enforcement argue that it’s actively destructive. The historical information these programs use to predict patterns of crime aren’t a neutral recounting of objective fact; they’re a reflection of socioeconomic disparities and the aggressive policing of black neighborhoods. Computer scientists have held up predictive policing as a poster child of how automated decision making can be misused. Others mock it as pseudoscience. “Systems that manufacture unexplained ‘threat’ assessments have no valid place in constitutional policing,” wrote a coalition of civil rights and technology associations, including the ACLU, the Brennan Center for Justice, and the Center for Democracy & Technology, in a statement last summer.

A numbing progression of police shootings in the past several years serves as a reminder of what’s at stake when police officers see certain communities as disproportionately threatening. Over the course of eight days in late June, juries failed to convict officers who killed black men in Minnesota, Ohio and Wisconsin. In each case, the officer’s defense relied on his perception of danger. The worst-case scenario with predictive policing software is deploying officers to target areas with their ears raised, leading them to turn violent in what would otherwise be routine encounters.

Photo by Misha Friedman for Bloomberg

The police departments Goldstein visited in New Jersey didn’t raise any questions about fairness during his recent trip—but there was skepticism nonetheless. He had barely started speaking to a group of top officers in the Linden Police Department when the man who handled the city’s procurement process confessed how wary he was of software vendors’ magical-sounding claims. Goldstein nodded along. When he was a cop, he said, he hated sitting through “the vendor nonsense.” Goldstein launched into a sing-song voice: “‘Oh, you’re going to have a flying car, and it’s going to stop people, and you’re going to be Super Po-Po!’ They’ll promise you anything.”

Goldstein’s company does make one unusual promise, which it thinks can satisfy skeptics in law enforcement and civil rights circles simultaneously. Other companies that make predictive software for criminal justice settings keep their algorithms secret for competitive reasons. In March, CivicScape published its code on GitHub, a website where computer programmers post and critique one another’s work. It was an unprecedented move, and it caused an immediate stir among people who follow the cop tech industry. “They’re doing all the things I’ve been screaming about for years,” said Andrew Ferguson, a professor at the University of the District of Columbia’s law school and author of the forthcoming book The Rise of Big Data Policing.

Posting computer code online won’t erase the worries about predictive policing. There are still concerns about how CivicScape responds to perceived shortcomings, and there’s also the big question of what police departments do with the intelligence it produces. But more than any other company, CivicScape has turned itself into a test case for what it means for law enforcement to use artificial intelligence in a way that’s transparent and accountable—and whether that’s even possible.

Goldstein, 43, didn’t start off wanting to be a cop. He was the director of information technology at OpenTable, the online restaurant reservation company, but he began to question the significance of that work after 9/11. In 2004, Goldstein saw an advertisement for the Chicago Police Department’s entry exam. “I’m like, ‘what does it hurt to take the police exam? I like taking tests,’” he recalled. Goldstein took the test, did well, and in 2006 left OpenTable.

After 13 months as a beat cop, Goldstein began working to develop computer models to anticipate where crime would happen. He was later promoted to commander and put in charge of a new unit dedicated to this strategy. The unit was providing intelligence that far exceeded what it had been using before, according to Michael Masters, who first met Goldstein during his academy days when Masters was an adviser to Mayor Richard M. Daley, then moved to the police department and now works at CivicScape. “We were well ahead of our time,” said Masters. Goldstein was perfectly placed to build technology into the daily work of policing. “You don’t have people who were cops, and have ridden in squad cars, building these tools,” Masters said.

Like any fast riser at a slow-moving institution, Goldstein was a polarizing figure. There were running rumors that he had some family connections at City Hall, and he had trouble developing any tough guy credibility—even after he apprehended a shooter who killed a man in front of Goldstein’s family on his day off. Longtime officers simultaneously thought it was simple to predict broad patterns of crime, which consistently centered on the same areas of the city, and impossible to anticipate specific offenses. On Second City Cop, a popular, anonymous blog, Goldstein was dubbed Golden Boy, and his unit was called the Crystal Ball Unit. Neither was meant as a compliment. Goldstein’s critics would gloat when a shooting occurred a block from one of his target areas, and they’d occasionally berate him in person at headquarters.

Photo by Misha Friedman for Bloomberg

Goldstein admitted he failed to win over his critics, and his unit was disbanded when the head of the department stepped down in 2011. And he acknowledged that he never came up with a rigorous way to test the impact his techniques had on crime rates. Goldstein moved to City Hall, then left government in 2013. Since then, Goldstein has run a venture capital fund, held academic positions, and sat on the board of Code for America, a nonprofit dedicated to help governments use technology. With the Crystal Ball Unit gone, people on both sides of the debate in Chicago retreated to the comfort of their preconceived notions.

The Camden County Police Department’s Real-Time Tactical Operational Intelligence Center is a Rorschach test on how you feel about tech in law enforcement. The RT-TOIC, as it’s known, is a windowless room from which the department runs its technological initiatives. When Goldstein visited this month, the futuristic sheen had been undermined by the failure of the building’s air conditioning—it remained inhabitable only with a bunch of full-blast floor fans. Still, the department’s leadership thinks the RT-TOIC represents the future of policing, not just in Camden, but everywhere.

About a dozen people were at work inside, most of them sitting at stations displaying between four and six computer screens. Large screens showing maps and footage from surveillance cameras were displayed on the wall. Analysts monitored social media for accounts that have referred to known crimes.

Camden integrated CivicScape into the RT-TOIC three months ago. The company’s maps are always running, changing every hour to reflect updated data. When targets change, analysts switch their screens to the surveillance cameras pointed at those blocks. Officers translate what’s happening in the
RT-TOIC to the cops on the street. The guys in patrol cars don’t know whether an order is derived from some newfangled math, the judgment of a superior officer or a mixture. The ambiguity is deliberate, said Kerry Yerico, the department’s director of criminal intelligence and analysis.

On the day of Goldstein’s visit, Yerico and Lieutenant Jeremy Merck, the watch commander on duty, were discussing an area that CivicScape had flagged. Merck immediately recognized the area—his officers had said that drug dealers were ramping up operations there. They had deployed extra officers.

Neither Yerico nor Merck knew exactly how the department’s computers and humans had homed in on the same spot. The guts of CivicScape’s predictive system are a series of neural networks. Neural networks, named because their design mimics the structure of neurons in the human brain, examine large data sets in which the inputs and outcomes are labeled. They then determine patterns they can use to predict what will happen when presented with new data.

Neural networks are favored by computer scientists working with huge data sets, but one of their shortcomings is their opaqueness. Unlike an algorithm in which a human has consciously told the system what to think about each factor, neural networks find their own paths and can’t effectively explain to humans what they’ve done. This has the potential to make CivicScape even less transparent than other predictive-policing software, which use different types of algorithms.

Scott Thompson, Camden’s police chief, said he hasn’t heard any criticism about transparency. For its part, CivicScape said its openness comes from inviting discussion about the types of data its models use.

Kristian Lum and William Isaac, researchers who have written their own statistical models for the Human Rights Data Analysis Group demonstrating how bias works in predictive policing, have examined the code. They both described CivicScape’s move as positive but withheld praise until they see how the company followed through.

I do believe in a policeman’s gut. But I also believe in augmenting his or her gut.

A significant shortcoming with CivicScape’s code repository, said Isaac, is that it has posted generic code when in practice it adapts its system for each separate client. Police departments often resist releasing data they’re not mandated to make public by law, and Goldstein acknowledged that his clients will not allow him to share some of the data that he’s using to produce predictions. It’s hard for an observer to assess what an algorithm does without access to either the final version of the code or a full set of the data.

“I think it’s a straddle” between the desires of police departments and the public, said Goldstein. “I’d rather take this step and move forward than not take a step because we know there are imperfections.” Isaac said that CivicScape isn’t fully in control of how its system works but that Goldstein’s attitude illustrates the problematic approach companies and government agencies take toward predictive tech. Goldstein rightly pointed out that secrecy has been established as the baseline, then demanded credit for any steps he takes in the other direction. “Should it be up to him?” said Isaac. “I think that’s kind of a false choice.”

Linden, like many of CivicScape’s clients, is a small department going through a transition. Its chief, Jonathan Parham, started in late 2016. He is a lifelong Linden resident, a 25-year veteran of the force, and its first African-American chief. He also thinks police departments have overemphasized arresting people.

Parham said he sees predictive policing as a way to offset brain drain. A lack of experience in the department has left Linden’s officers without a basic understanding of its communities. “We’re looking at the absence of personal knowledge of your area and supplementing that with technical knowledge of the frequency of the crimes,” said Parham.

This is a common argument for adding computers. Camden’s department cited a similar need. All the cameras, social media analysis, and automated forecasting are supposed to help the department cover the most ground with the fewest officers.

To Isaac, this is a myopic way to approach criminal justice. Even the act of updating target areas hourly distorts the situation by ignoring any potential service that can’t be carried out by a cop over the next 60 minutes. The potential advantages of predictive tech are undermined by restricting it to something that cops use to catch supposed bad guys. “What you are left with is a perceived chess match between cops and robbers,” said Isaac. “That’s a very simplistic version of what crime really is.”

Parham offered a similar view. The worst thing Linden’s police department could do, he argued, was to believe that Goldstein really did have mystical math that would allow officers to drive to the specific locations of shootings just as they were about to occur. During the meeting with Goldstein, he told his department’s brass that it needed to avoid turning policing into a video game.

Parham recalled a training exercise he helped run. Officers were sent to a train station and told to interview people in the station, to treat people as more than potential perps. After 10 minutes, each officer would write down what he had learned. The winner was the person with the most useful information.

The officers who were logging the most arrests always performed the worst, Parham said. If law enforcement is a matter of receiving a target from a computer and then attacking that target, it doesn’t matter how precise the computer model is. Parham’s job is to produce cops who are better at the train station drill. “Our officers, the more technologically savvy they get, the less human they become,” said Parham. “I don’t want that.”