facebook app skip to content
Primary navigation

Survey Methodology

Sample Population

The sample for the Workforce Diversity Survey (WDS) was taken from the respondent population of the second quarter 2015 Minnesota Job Vacancy Survey (JVS), a twice annual survey of hiring among Minnesota employers conducted by the Labor Market Information Office at the Department of Employment and Economic Development.

From the roughly 7,000 JVS respondents, Labor Market Information analysts constructed a sample of 3,265 respondents for the WDS. The sample was specifically created to allow the results to be representative of businesses of the same size, in the same region, and in the same industry. The sample was drawn assuming a 60 percent response rate and allowing for a margin of error of 5 percent at 95 percent confidence.

Cleaning of the sample produced a final sample of 3,158. The 107 deleted records were all instances of separate locations of a single firm, where hiring occurs from a centralized location.

Survey Campaign

The sample of 3,158 was split into two groups: those for whom an email address could be obtained by a Google and Linkedin search and those for whom a public email address was unavailable. In only 80 cases did the Google and Linkedin search provide no appropriate contact name, whether or not it provided an email address; in those cases, we directed the survey to "Human Resources Manager".

The survey was administered via two methods: paper mailings to those without email addresses (approximately 1,293 contacts) and electronic mailings to those with email addresses (approximately 1,865 contacts). The email contacts were provided with an online survey to complete, and the paper mail contacts were mailed a self-addressed, stamped return envelope with a paper version of the survey, which also included a short URL where they could complete the survey online if they chose.

The first round for survey completion began on Thursday, February 4, 2016 and ended on Friday, February 26, 2016. A second round was opened beginning Monday, March 14, 2016 and ending Friday, March 25, 2016. The second window was opened for approximately 1,533 contacts who had either not responded during the first round or who we determined we had been unable to successfully contact during the first round due to bounced emails or returned to sender envelopes.

Ultimately, surveys were successfully sent to 2,904 contacts, of which 1,049 responded, a response rate of 36 percent.

Discussion of Nonresponse Bias

We did not meet our target of a 60 percent response rate, opening the possibility that our survey results are compromised by nonresponse bias. The presence of nonresponse bias means the results are not generalizable to the general population. We do not find evidence that our results are biased in this way; this section explains why.

Response Priors

The Job Vacancy Survey (JVS) achieves a 70 percent response rate, supplementing paper surveys with phone calls and automatic submissions by larger firms. Because the WDS is a new survey, we had few priors about what response rate we should expect from a combination of paper and online surveys emailed to contacts. The questions we ask in the WDS tend to be more subjective, more time intensive, and cover the sensitive topic of racial diversity in the workplace, suggesting we would get a lower response rate than the JVS. However, we sampled only those employers who had recently responded to the JVS, suggesting these employers are open to responding to government surveys and implying we could expect potentially even a higher response rate than the JVS. In the end, the sample was pulled assuming a 60 percent response rate.

Comparing Late Responders to Early Responders

Given our final response rate of 36 percent, we investigated the extent to which our results suggest significant nonresponse bias by comparing late responders (a proxy for nonresponders) to earlier responders1. If responses are generally consistent between these groups, we have a strong case that our results do represent Minnesota employers with a five percent margin of error at 95 percent confidence.

We are particularly interested in whether each of our stratifications appears robust to nonresponse bias. Because of our low response rate, we choose to remain conservative in our reporting by stratification. We are leaving off the table any two-way tabulations by stratification (e.g. firm size with industry, region with firm size, etc.). We are also grouping the 20 industries into 12 supersectors, guided by definitions from the Bureau of Labor Statistics.

We define a late responder as one who submitted their response among the last 25 percent of responders, an early responder as one who submitted a response among the first 25 percent, and a middle responder as one who submitted a response in the middle 50 percent of responders. Our strategy of dual response methods (mailing a paper copy of the survey versus submitting responses online) means that late responders are overwhelmingly mail responders. Because mail responders could differ from online responders, we control for this timing difference by calculating the response percentiles within both the mail and the online groups. Once we have early, middle, and late responders appropriately flagged, we combine the mail and online groups for the analysis presented here.

Our sample is largely free of significant differences between early, middle, and late responders, and in no case is a stratum significant in both questions. However, Natural Resources and Mining is significant or near significant (P=0.06) in both questions, so we report results for this industry with a word of caution.

1. This is a well-established practice in the field since the seminal Smith, Tom W. 1984. “Estimating Nonresponse Bias with Temporary Refusals,” Sociological Perspectives, 27(4).

P-value on Chi-squared (or Fisher exact) test results ON TWO SAMPLE QUESTIONS
  Does anyone in your current workforce at this location identify as a race or ethnicity other than white, non-Hispanic? Please indicate whether the following statement is true: My business/organization is actively trying to increase racial diversity in entry-level positions.
Total sample 0.49   0.08  
Northwest 0.10   0.21  
Northeast 0.78   0.13  
Central 0.14   0.15  
Twin Cities 0.46   0.98  
Southwest 0.81   0.90  
Southeast 0.37   0.13  
Firm size
Small 0.74   0.41  
Medium 0.01 ** 0.40  
Large 1.00   0.94  
Construction 0.65   0.85  
Education 0.64   0.51  
Financial Activities 0.71   0.47  
Government 0.21   0.64  
Health Services 0.91   0.81  
Leisure and Hospitality 0.48   0.03 *
Manufacturing 0.29   0.05  
Natural Resources and Mining 0.06   0.04 *
Other Services 0.04 * 0.52  
Professional and Business Services 0.70   0.04 *
Trade, Transportation, and Utilities 0.21   0.31  

** p < .01; * p < .05

back to top