By Cullen O’Keefe
This is Part III of a series examining the proposed ordinance Project Green Light (PGL) Ordinance, which would require every business in Detroit open after 10 pm to participate to pay for cameras allegedly monitored live by the Detroit Police Department. Part I: Project Green Light (the reality, not the hype) – Part II: Lies, Damn Lies, and Statistics – Part III: Who Is Really Getting Paid.
Today is primary election day and Detroiters are culling the political herd to pick the final Mayoral, City Council, and City Clerk candidates for the November ballot. Most Detroiters didn’t pay much attention to the primary campaigns; they will start to focus after the primary. What most missed was a preview of the general election campaigns to come, with crime reduction (alongside jobs) being THE issue.
Crime. Detroit’s original sin. It gets blamed for White flight, Black flight, blight, unemployment, food deserts, and high insurance rates. Crime is the ultimate barometer of the division between Downtown/Midtown (aka The 7.2) and The Neighborhoods (aka, 98% of the City). Leaving aside cause and effect, it’s natural that every politician and civic leader wants to lower crime. And get credit for it. Without addressing the messy underlying causes. Or spending money.
Enter, Project Green Light (PGL), a voluntary, “real-time” surveillance program that doesn’t cost tax payers a penny. After a year of rave review, apparently the Mayor and Chief Craig decided the only way to make PGL better is to remove “voluntary” from the description. But first, let’s take a look at those rave reviews, shall we?
Mayor Mike Duggan and the Detroit Police Department frequently assert that the program has reduced crime at participating sites. This of course is the rationale for the PGL Ordinance – in fact, the rationale for continuing PGL at all. Local news outlets frequently assume that these assertions are true. But research shows that similar widespread police surveillance systems have been ineffective at preventing and solving crime, so it makes sense to take a close look at the Mayor’s claims about PGL.
Unfortunately, the claims of the PGL’s efficacy don’t stand up to even modest scrutiny.
The following is typical of Mayor Duggan’s claims about PGL:
Since it was launched, Project Green Light has helped to bring about sustained reductions in violent crime at or near participating businesses. A recent analysis of reported crimes shows that the original 8 Green Light gas stations continue to see a nearly 40% reduction in violent crime. Newer Green Light businesses have experienced a roughly 20% average reduction in violent crime compared to the same time period the year before. Since Project Green Light has been up and running, the city has experienced a 40% reduction in carjackings, as well.
Newer analyses of PGL, however, show that the purported effect has “subsequently dropped to [a] 10 percent [reduction in violent crime] since the program started.” DPD spokesman Michael Woody attempted to downplay this much more modest, revised effect size, saying that the revised number was due to newer PGL participants: “‘[i]f we started at a 40 percent reduction from last year at all Green Light locations, and we take on a new facility that’s had a lot of crime since last year, it will affect the overall number.’”
Despite the availability of newer data – with shows a much smaller impact by PGL – PGL advocates keep citing the high reduction rate from the original eight sites only.
Why The Claims Fail
Meaningless Sample Size
DPD is of right: Crime reduction rate for PGL participants would be higher if we ignored PGL sites with higher crime rates. And the Tigers would have a better won-loss record if we ignored their losses. But an honest statistical analysis can’t just include the data that supports the researcher’s preferred outcome.
Woody does make one good point, though: statistics from small samples are vulnerable to significant changes due to random variation within the sample. This is why researchers prefer larger sample sizes: it helps them separate the signal (i.e., any real difference from the factor they are studying) from the noise (i.e., any differences due to other factors and random variation). What Woody fails to mention, though, is that this gives us a very good reason to be skeptical of DPD’s claims about PGL. As mentioned above, the most frequently cited statistic on the effect of PGL uses a sample size of only eight. This is so small as to render any statistics derived from it virtually useless: with such a small sample, it is extremely difficult to justify claiming that any noticeable “effect” on crime rates is due to PGL rather than expected random variation. Test this yourself. Survey 8 of your friends on anything, anything at all – shoe size, car color, number of keys in their pockets. Now, how representative is that for the entire city?
Since bigger samples tend to be more accurate, there is good reason to think that that lower figure is more accurate than DPD’s preferred statistic. Now that more than 180 business participate in PGL, DPD has the opportunity to perform more rigorous statistical analyses of PGL’s efficacy. Despite that, they continue to primarily cite data from the first eight businesses.
Regression to the Mean
The greater the deviation from its mean of an observation, the greater the probability that the next observation will be closer to the mean.”
–James Nicholson, The Concise Oxford Dictionary of Mathematics (Christopher Clapham & James Nicholson eds., online 5th ed. 2016).
A common statistical phenomenon, regression to the mean, explains why businesses with a higher-than-average crime rate one year (such as many PGL participants) should expect to see less extreme crime rates the next year, regardless of whether they participate in PGL.
The principle is counterintuitive, but easily stated: for certain types of observations, “the greater the deviation from its mean of an observation, the greater the probability that the next observation will be closer to the mean.” (James Nicholson, The Concise Oxford Dictionary of Mathematics (Christopher Clapham & James Nicholson eds., online 5th ed. 2016)).
Thus, extremely high observations will tend to be followed by lower observations. Many of the PGL-participating businesses decided to enroll after experiencing high crime rates. Thus, such stores should expect to experience a subsequent drop in crime rates, regardless of whether they participate in PGL. Without an adequate explanation of why any purported decrease in crime at PGL sites is not attributable to regression to the mean, DPD is incorrect to conclude that the decrease is causally attributable to PGL.
Lack of Control Group
We found no statistically significant difference in the crime rates between the PGL and non-PGL gas stations (paired t-test, p = 0.405)
DPD’s statistics are also flawed because they do not use a control group; that is, they do not compare crime trends at PGL sites with similarly situated non-PGL sites. Without a control group comparison, DPD cannot say whether the purported drop in crime at PGL sites is due to PGL or other factors. For example, Detroit’s violent crime rate has been declining for several years. DPD deserves credit for that. But without a control group, DPD can’t justify the claim that drops in crime at PGL sites are due to PGL rather than the other forces responsible for the general downward trend in Detroit crime rates.
We attempted to perform such a controlled analysis of PGL ourselves. We found 19 PGL-participating gas stations with another, non-PGL gas station nearby. For each pair, we then compared 2017 crime rates through July 19, 2017. We found no statistically significant difference in the crime rates between the PGL and non-PGL gas stations (paired t-test, p = 0.405). Thus, we see no reason to conclude that crime rates at PGL sites are lower than those at non-PGL sites.
Studies show that the non-video aspects of PGL, such as improved lighting and increased police patrols, reduce crime rates.
So, there is no evidence to support the claim that PGL has decreased crime at all, much less by the 40% the Mayor and DPD keep claiming. But let’s say for a moment those claims were true, there is still no justification for the claim that the “live” camera feed is is the reason.
PGL is multi-faceted. Current PGL-participating businesses not only stream live high-definition video to DPD, but also install improved lighting, post signage indicating PGL participation, install a green strobe light, receive regular patrols from DPD, and receive higher priority for calls to DPD. Thus, even if PGL lowers crime rates at participating sites, it is currently impossible to determine whether this is due to the live video feed to DPD, or due to the less invasive aspects of PGL like improved lighting, increased police patrols, quicker police response times, signage, or the presence of video cameras generally. DPD’s statistics, even if meaningful, don’t support the claim that the invasive live video feed to DPD is the cause (or even a partial cause) of the program’s supposed success. Indeed, studies show that the non-video aspects of PGL, such as improved lighting and increased police patrols, reduce crime rates. Thus, we have good reason to doubt that the live video feed to DPD explains any drop in crime at PGL sites.
Finally, Mayor Duggan, DPD, and other proponents of PGL have proposed to make PGL participation mandatory for Detroit businesses open past 10 PM. See Hunter, supra. These proponents often cite the purported success of PGL as justification: they “hope the [mandate] will result in similar drops in crime citywide.” See id. However, even if PGL has been successful so far, there is no reason to think that it would continue to be effective if expanded citywide. Since we do not know whether any drop in crime at PGL sites is due to the live video feed or due to the other, less invasive aspects of PGL, the only way to justify such a claim is to have the proposed citywide mandate mirror all aspects of PGL as currently constituted. However, it would be impossible for DPD to make regular patrols at all Detroit businesses open past 10 PM, or to prioritize calls to DPD from all such businesses. Thus, businesses forced to participate in PGL by the proposed ordinance would not receive some of the primary benefits that current PGL participants receive. DPD is therefore unjustified in expecting that mandatory citywide participation will be similarly effective to the current, voluntary system.
The Mayor and DPD are not, to be generous, accurately describing PGL or it’s success. To be blunt, their claims are statistically unsound: the small sample size, lack of a control group, and failure to account for regression to the mean all undermine any claim that PGL reduces crime. Even if such a claim were true, DPD would be unjustified in ascribing that reduction to the live video feed. One can only wonder why they are so determined to pass an ordinance requiring live video without further evidence that the live feeds themselves, and not other aspects of PGL, are responsible for any drop in crime.
Detroiters are being asked to accept a major imposition on their privacy without adequate reason to think it will reduce crime. The significant legal, social, and economic problems with mandatory, privately funded, citywide surveillance cannot be justified without high-quality statistical evidence of its efficacy. DPD currently lacks such evidence.