The crime analysis and accountability system known as Compstat, developed by the New York Police Department in 1994, is the most revolutionary public-sector achievement of the last quarter-century. Since its inception, Compstat has driven crime in New York down an astounding 77 percent; veterans of the Compstat-era NYPD who have gone on to run police departments elsewhere have replicated its successes. Other government agencies, both in New York and nationally, have applied the Compstat model to their own operations, using minutely analyzed data to hold managers accountable for everything from improvements in public health to decreases in welfare dependency to road repairs.
Now, however, a survey of retired NYPD commanders by two criminologists purports to cast doubt on the wisdom of Compstat. Compstat has undermined the reliability of the NYPD’s crime-reporting system, say former John Jay College of Criminal Justice professor Eli Silverman and Molloy College professor John Eterno. Weekly meetings in which top brass grill precinct commanders about crime on their watch place too much pressure on the commanders, resulting in data manipulation, the professors charge. “Those people in the Compstat era . . . felt less pressure to maintain the integrity of the crime statistics,” Eterno told the New York Times. “The system provides an incentive for pushing the envelope,” Silverman added. Silverman and Eterno anonymously asked several hundred retired NYPD captains if they were “aware” of instances of commanders changing crime data. About half of the survey respondents (157 of 309) said that they were aware of changes to crime reports. A follow-up question asked whether the changes were ethically appropriate. Of the 160 respondents who answered, 22.5 percent thought that the changes were ethical, while 53.8 percent believed that the changes were highly unethical.
Critics of the NYPD have seized gleefully on the study, which has not yet been publicly released. The New York Times obligingly wrote it up on its front page and followed up with several articles on the topic.
NYPD foes can put away their party hats. Nothing in the survey discredits Compstat or its crime-fighting accomplishments. Eterno’s claim about a decreased emphasis on crime-data integrity in the Compstat era is demonstrably false. It is ludicrous to suggest that a department where the top brass did not even get crime data until six months after the crime and then did nothing with them—as was the case in the pre-Compstat era—cared more about the accuracy of crime statistics than one in which every deployment decision is made based on the minute-by-minute reality of crime on the streets. Nor does the study, which has several design flaws, cast any doubt on the city’s record-breaking crime drop. Given the enormous efforts that the NYPD makes to ensure the validity of its statistics, the study ultimately comes down to a dangerous argument against accountability systems per se.
“The message from the powers-that-be is clear and unequivocal—DO NOT COOK THE BOOKS.” So responded a retired NYPD captain to my e-mailed questions about the Eterno-Silverman study. My correspondent is no friend of the department; he still bears the animosity of the rank and file against headquarters. He readily conceded, however, the all-consuming emphasis that the Compstat-era department places on data accuracy. Other commanders echo this view. “The biggest sin you could commit in the NYPD was fudging,” says Steve McAllister, a former borough chief in the transit bureau who now serves as crime-control strategy specialist in the Newark Police Department. “The consequences of fudging were worse than the consequences of a crime increase. You would lose your command and be vilified. It was a career-ender.”
Silverman and Eterno’s claim that Compstat has led to a weakening of “pressure to maintain the integrity of the crime statistics” ignores both the actual history of the Compstat era and the philosophy behind it. NYPD resources devoted to data integrity have steadily increased during the Compstat years. The department did not even begin routine audits until 1999. In standard auditing practice, the size of the audited sample mirrors changes in the volume of data; in the NYPD, however, even though the number of reported crimes has dropped sharply, the number of audited crime reports has grown. In 2009, 47,510 crime reports were audited, compared with 29,277 in 1999, despite budget cuts over the last nine years that have forced the attrition of 5,000 cops. The crime misclassification rate—often the product of innocent error—has dropped from 4.4 percent in 2000 to 1.5 percent in 2009.
The NYPD goes beyond accepted auditing conventions in other ways as well, as an independent assessment of the department’s data-quality controls in 2005 observed. When the New York State Comptroller reviewed the integrity of the NYPD’s crime reporting in 2000, it used a typical sampling practice: it selected 25 precincts for auditing and cut off its audit early after finding no significant evidence of misreporting in the first 17 precincts it analyzed. The NYPD, however, does not merely sample among its reporting units; it audits each command twice a year and samples every crime category within that command, observed New York University professor Dennis Smith and SUNY Albany professor Robert Purtell in their 2005 study.
If a precinct is suspected of manipulating data—if it shows a sharp crime drop when surrounding precincts show an increase, say—investigators will scrutinize up to 5,000 crime reports from that precinct. The chance of hiding misbehavior under such scrutiny is almost nil. “When the Quality Assurance Division [the auditors] comes in, they turn everything inside out and peruse every statement in the record,” says Steve McAllister. “If I’m trying to change burglaries to criminal trespass, I can’t create a system that would rewrite every victim statement in my file.” As the commanding officer of the 66th Precinct in Brooklyn, McAllister experienced one of the department’s most intense data audits. A malcontent from the community had falsely accused McAllister of lying about crime. “My borough commander called me to task,” McAllister recalls. “I said: ‘Audit me.’ We had the burden of discrediting everything [the malcontent] said.” The department’s Quality Assurance Division pulled thousands of crime reports from McAllister’s watch and subjected them to individual scrutiny, including reinterviewing victims and witnesses. None gave evidence of cheating.
Smith and Purtell compared the NYPD’s auditing processes with those of eight other large departments. “No other department has systems in place that come close to matching the set of high level, overlapping, systematic audits of crime reporting now employed by NYPD,” they concluded. The NYPD’s integrity apparatus encompasses both a Data Integrity Unit that focuses on coding accuracy within computer reporting systems and the Quality Assurance Division that audits the processes that lead up to computerized crime reports. Whereas in other departments, underreporting scandals have been discovered by outsiders, the NYPD itself discovers instances of questionable reporting, observed Smith and Purtell. Confirmed cases have resulted in serious sanctions. The 11 confirmed instances of intentional misreporting since 2002 have led to four commanding officers being stripped of their commands and a range of other discipline against lower-level officers.
The NYPD’s intense dedication to reliable data grows out of the Compstat philosophy itself. One of Compstat’s most powerful accomplishments was to yoke crime data and analysis intimately to deployment decisions. Compstat can accomplish its crime-lowering goal only if the crime data that commanders scrutinize so obsessively is accurate. “Jack Maple [Compstat’s flamboyant co-originator] stressed the necessity for accurate and timely intelligence,” recalls Edmund Hartnett, a former executive officer of the NYPD’s Narcotics Division and now commissioner of the Yonkers police department. “Everything that we do is based on crime data; if the data is not accurate or you are missing dots on the map, you’re not deploying properly.” A pattern of felony thefts ignored because of data manipulation will spread and worsen, a contagion that no police commissioner should be confident can be manipulated out of public awareness. In the Compstat era, commissioners live and die by the reality of crime. It is overwhelmingly in their self-interest to ensure faithful crime reporting since, without it, they will misallocate their scarce crime-fighting resources.
The Silverman-Eterno survey offers no evidence that New York’s reported crime drop is inaccurate. The magnitude of that drop is simply too great to be affected by a few instances of commanders reclassifying a theft from a felony ($1,000 of stolen goods) to a misdemeanor (theft under $1,000) by cross-checking on eBay a victim’s assessment of the value of the stolen goods. In order to design a system that would significantly hide crime, a retired officer says, you would need multi-level coordination from a precinct’s patrol officers through its desk officers and crime-analysis unit and finally to its executive and commanding officers. “People can’t agree on lunch orders. Do you really think that a conspiracy of this intricacy could operate without discovery?” he asks.
The most authoritative independent check on a department’s reported crime trends backs New York’s crime data. The Justice Department’s National Crime Victimization Survey showed an even stronger correlation with the NYPD’s crime data after Compstat began, two federal Bureau of Justice Statistics researchers concluded in 2003. No one has ever suggested that homicides can be reclassified as mere assaults or hidden altogether. The NYPD’s 77 percent homicide drop has led most other crime categories. If felonies were being systematically reclassified as related misdemeanors, the misdemeanor rate should rise. But misdemeanors have dropped along with felonies.
Moreover, the public has confirmed the NYPD’s crime reporting with its feet. Few of the new residents and business owners who have flocked to the city’s once violence-saturated neighborhoods since crime began dropping have heard of Compstat, much less checked Compstat data on the Internet. Rather, they have based their relocation decisions on word of mouth and their own lived experience of an area. Every marker in the city’s revitalized neighborhoods—from the absence of the once ubiquitous No radio signs in cars, to the opening of national chain stores, to the appearance of children riding their bikes on the sidewalk—buttresses the reality of the crime drop.
There are several reasons, however, to question the significance of Silverman and Eterno’s findings. The professors received responses from 323 former NYPD captains and above who retired sometime after 1995, when Compstat was in full force, out of 1,200 surveys that were mailed out. We don’t know when they retired; it could have been in 1996 or 2006. If the former, their knowledge of the post-Compstat department is limited. We do not know how many different incidents of report revision are being referred to by the 87 respondents who said they were “aware” of changes to crime reports and who found the changes “highly unethical.” It could be 87; it could be 17. The survey did not ask if the respondents had personally witnessed data manipulation or had just heard about a case. Nor did it ask if the manipulation was detected and what the consequences were. If retired editors at the New York Times were asked if they were aware of reporters making up quotes, the positive response rate would likely be 100 percent, since every editor would be aware of Jayson Blair, the Times reporter forced to resign in 2003 for fabricating sources and plagiarizing articles.
Nearly 34 percent of the respondents who mailed back their survey (166 individuals) had retired before 1995. Silverman and Eterno do not disclose what percentage of these pre-Compstat-era captains said that they were aware of changes to crime data, a figure that would provide some benchmark for their thesis. Even if the percentage is lower than in the post-Compstat era, however, that difference could simply demonstrate the greater publicity attendant on data-changing charges in the Compstat era. The only finding in the professors’ survey that backs up their claim that the department is less concerned about data integrity since Compstat is the slightly lower rating that post-1995 retirees give to the department’s demand for integrity in crime statistics compared with pre-1995 retirees: post-1995 retirees rate the demand for data integrity at a mean of 6.52 out of ten, compared with a mean of 7.18 for pre-1995 retirees. This is meager evidence, though. There is no way to know whether the pre- and post-1995 retirees interpret the survey’s one-to-ten rating scale identically. A more reliable indicator would have been to ask the post-1995 retirees to compare the department’s data accuracy pre- and post-Compstat, but this the authors apparently failed to do. In any case, the slight difference in the ratings given to the demand for data integrity does not come close to overriding the significance of the massive auditing apparatus that the department has put in place since Compstat began.
Silverman and Eterno provide selected anonymous comments from survey respondents at the end of their summary. It doesn’t paint a pretty picture, though not necessarily for the reason that Silverman and Eterno intend. Many of the respondents display the petty resentments toward higher-ups that are depressingly on view at the unofficial website for NYPD cops, NYPD Rant. They charge favoritism and unfair treatment by managers. To what extent the survey answers were driven by this resentment—and to what extent the favoritism charges are true and not just sour grapes—is unknowable. But a New York Times profile, meant to buttress the Silverman-Eterno thesis, unwittingly shows how that resentment and self-pity undercut the significance of the survey’s already paltry findings. An anonymous retired captain who took part in the survey tells the Times that he was himself subjected to investigation after crime in his precinct dropped far more than in neighboring precincts:
Eventually, [the captain] was called in before the department’s Internal Affairs Bureau, and accused of “improper classification” of some of those crime reports.
The commander insisted Sunday he had done nothing wrong. . . . But ultimately, he said, he decided to retire after 20 years on the job because he said the department threatened to punish him for the offenses it believed he had committed. Wrongdoing was never proved in the matter, but he was also never able to prove that he acted properly, he said.
“It eats its own,” the commander said of the department. “That’s what it does. But I was doing the right thing for the department. The moral of the story is, ‘Don’t try to be perfect; don’t try to do too much.’”
Amazingly, the Times thinks that this lachrymose tale supports the thesis that the department is lax about crime accuracy. The paradox of a department punishing people for the data improprieties that it is said to be indifferent to is an irony that no one pushing the anti-Compstat story seems troubled by.
Compstat indisputably created hard feelings in the department. It was a radical change to the status quo that upset longstanding hierarchies. It demanded measurable output from officials who had long been accustomed to coasting. Especially in its early years, Compstat could be a brutal experience; whether it still is is unknowable, since Commissioner Ray Kelly has closed it off to outside observers. It is no coincidence that the Captains Union funded the Silverman-Eterno study; the radical fringe in all the department’s unions wants Compstat thrown out. “‘The crap rolls downhill,’ as they say,” observes an NYPD veteran and police commissioner at another department. “When a CO is yelled at at Compstat [meetings], he comes back to the precinct and yells at his cops to start taking proactive action against the weed-smokers and guys drinking on the corner. The cops think that if we got rid of Compstat, they could ride around the sector like they used to 16 years ago waiting for victims to call in crimes.”
The effect of Silverman and Eterno’s poll, whether intentionally or not, is to cast doubt on all accountability systems. Without doubt, once an institution starts scrupulously measuring results and holding managers responsible for those results, some people will try to game the system. The school accountability movement has produced numerous efforts to manipulate test scores, some more deceitful than others. The question is: what is the alternative? Short of getting rid of accountability altogether, the best that a system can do is to work tirelessly to uncover fraud. That the NYPD clearly does. Would it be better to lighten up on the department’s relentless focus on outcomes? Compstat produced a sense of urgency about fighting crime that had never existed before. And unlike many police agencies that also experienced a crime drop in the 1990s, the NYPD has never lost that sense of urgency. The results speak for themselves: whereas the crime drop has leveled off or reversed itself in several cities that were considered competitors to New York in the 1990s, the NYPD has kept its crime rout going, allowing New York to hold on to its title as the safest large city in the country for a decade.
Perhaps there are additional measures that the department could take to beef up its already unparalleled integrity systems, such as making deputy inspectors in the borough commands responsible for quality control, or auditing the big precincts more frequently. But Compstat has accomplished what no other police reform ever has: driving down crime to record-breaking lows and focusing the department relentlessly on crime reduction. New Yorkers would be foolish to call for a watering down of the intense pressure it puts on commanders to perform.