Terrapin Tables

I will say I feel bad for New Haven officials - they probably would've gotten sued whether or not they certified the results. I believe the city was right to pause for a moment and to assess whether or not the test was biased, and I probably wouldn't have argued against such an action if I was part of the team that did that, but I do feel that given the specifics of this case, the correct judgment was reached.

The legalese is hard for me to parse, but I feel that ultimately the city used the results of the test as prima facie evidence that the test had a racial bias, and it appears that that alone is not enough evidence to suggest the test was flawed. If 18 of those that passed were white, out of 118, that does seem to ring alarm bells. But how many civil agencies give tests like this every year? Maryland alone has 139 fire agencies and 95 law enforcement agencies. Add up all the civil agencies across all the jurisdictions we have in this country, and you're bound to have outliers like this pop up (I think... Roy?)

I'm somewhat familiar with the case, but since I don't know all that much about it (and am not taking the time to dig it up) I'll speak in some abstraction. (And sorry for the all-caps below--I don't know how to italicize in these windows.)

Seeing group-wide disparities in the rates of "passing" the test (or anything else--being hired, promoted, graduating, etc.) is NOT sufficient to establish bias. What is required, however, is if there are differential rates CONDITIONAL ON ABILITY. That is, if you took a white firefighter and a black firefighter OF EQUAL ABILITY and they had different probabilities of passing the test, then there would be a problem that may be termed "bias".

This notion of conditioning on ability (that is, thinking of two separate test-takers of equal ability) may be strange, and novices tend to resort to reporting the overall pass rates. But it is important to do that conditioning. For example, look at the racial profile of the NBA. On the surface, it appears to heavily discriminate in favor of blacks. But on deeper consideration that's not enough. What would be enough for bias is if black players and white players of equal ability were being offered contracts at different rates.

Now for some technical terms. The difference in the overall pass rate between groups is called "impact". That is, the test evidences adverse "impact" for blacks relative to others. But it's not necessarily bias. For bias, we'd have to argue that the blacks were of equal ability (which of course may be the case).

Technical asides:
1. In the educational measurement literature, what I've been referring to as bias is actually called "differential functioning". A bit more is required for "bias", but we're essentially there and I won't bother to get into the finer-grained distinction.
2. To get at "bias" (differential functioning) we don't necessarily need to identify blacks and whites of the same ability, match them up, and see if they pass at the same rate. There are model-based approaches that are used and subpopulation distributional assumptions may invoked to augment the data one has at hand.

If you want to learn more, check out differential item functioning (e.g., http://en.wikipedia.org/wiki/Differential_item_functioning for the big idea) or take my item response theory class.