Source: Fig 1 of Doleac and Stein (working paper)
In several previous posts, I’ve written about the value of moving beyond simple measures of treatment effects in experiments. That is, in addition to measuring an effect of a treatment, we would like to know 1) when the effect might be big or small and 2) why the effect occurs. In this post, I’ll write about the online field experiment of Doleac and Stein (2013) that makes use of digital technologies to study discrimination in market transactions and that attempts to understand the when and why of the effect they measured.
To provide some context for the Doleac and Stein study, there is intense debate about the amount of racial discrimination that takes place in contemporary America. Experimental approaches are a promising way to meet this measurement challenge because they can clearly detect any difference in outcomes for two people who are identical other than their race. There are two main experimental approaches to measuring discrimination—correspondence studies and audit studies—and the Doleac and Stein study combines some of the best features of both approaches.
Correspondence studies, which usually involve sending written application materials to potential employers, signal the race of the applicant by manipulating the applicant’s name. A great example of a correspondence study is Bertrand and Mullainathan’s (2004) paper with the memorable title “Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination.” Correspondence studies have relatively low cost per observation, which enables a single researcher to collect thousands of observations in a typical study. But, correspondence studies of racial discrimination have been questioned because names potentially signal many things in addition to the race of the applicant. That is, names such as Greg, Emily, Lakisha, and Jamal may signal social class in addition to race. Thus, any difference in treatment for resumes of Greg’s and Jamal’s might be due to more than presumed race differences of the applicants. Audit studies, on the other hand, involve hiring actors of different races to apply in person for jobs or housing. A great example of an audit study is Ayres and Siegelman (1995) in which actors negotiated the purchase of new cars and found that black customers were consistently quoted higher prices than white customers. Unfortunately, audit studies also have limitations, and they are extremely expensive per observation. This cost structure typically limits the size of audit studies to hundreds of observations. Doleac and Stein take advantage of a web-based marketplace perform something a hybrid. They are able to collect data at relatively low cost per observation, resulting in thousands of observations (as in a correspondence study), and they are able to signal race using photographs, which creates a signal of race that is unconfounded (as in an audit study).
To study the effect of race on market transactions the authors advertised thousands of iPods in an online marketplace (e.g., craigslist). The advertisements that they posted varied along three main dimensions. First, they varied the characteristics of the seller, which was signaled by the hand photographed holding the iPod as shown at the top of this post [white, black, white with tattoo]. Second, they varied the asking price [$90, $110, $130]. Third, they varied the quality of the ad text [high-quality and low-quality (e.g., cApitalization errors and spelin errors)]. Thus, the authors had a 3 X 3 X 2 design which was deployed across more than 300 local markets ranging from towns (e.g., Kokomoe, IN and North Platte, NE) to mega-cities (e.g., New York and Los Angeles). Averaged across all conditions, the outcomes were better for the white seller than the black seller, with the tattooed seller having intermediate results. For example, white sellers received more offers and had higher final sale prices.
Beyond these average effects, the experimental design allows for richer comparisons that can help us understand where and why discrimination might be happening. For example, one prediction from theory is that discrimination would be less in markets that are more competitive. Using the number of offers received as a proxy for market competition, the authors found that black sellers do indeed receive worse offers in markets with a low degree of competition between buyers. Further, by comparing outcomes for the ads with high-quality and low-quality text, the authors find that ad quality does not impact the disadvantage faced by black and tattooed sellers. Finally, taking advantage of the fact that advertisements were placed in more than 300 markets, the authors find that black sellers are more disadvantaged in cities with high crime rates and high residential segregation.
None of these results give us a precise understanding of exactly why black sellers had worse outcomes, but, when combined with the results of other studies, they can begin to inform theories about the causes of racial discrimination in different types of economic transactions. Thus, by making use of the web, Doleac and Stein were able to combine the scale of correspondence studies with an unconfounded signal of race. Their design could serve as a useful model for others interested in studying discrimination.
To read the entire paper, check out:
For a related study, check out:
Wheels on the bus 
