LBNL Culture Survey: Methodology Overview
Culture Survey.mp4Survey Development Approach
Over the course of April through December 2023, the Lab's Culture Data Scientist engaged in the following activities to develop the LBNL Culture Survey:
Reviewed academic literature for validated survey instruments related to engagement and similar outcomes
Used validated questions (and will conduct further validation)
Checked for face validity via stakeholder meetings
Cross-referenced with existing surveys done by divisions or by UC Berkeley and past surveys
Chose comparable questions where possible
Used similar constructs to prior surveys, with some new types of questions related to learning processes
Held discussions about key priorities with Lab Directorate, ERGs, IDEA Chairs Council, divisional representatives from user facilities, HR Division Partners, and other stakeholder groups
Took best practices from division surveys and tested survey questions on diverse groups in the Lab
Sampling & Weighting Strategy
This culture survey went to all Lab employees, both represented and non-represented. The survey will not consider affiliates/contingent workers this round.
Over 50% of employees took the survey. It is worthwhile to listen to the voice of such a large proportion of employees. The high response rate also makes it likely that the survey captured a wide range of employee feedback.
Even given the high response rate, there is always a risk that those who take the survey are systematically different than those who do not. To help alleviate this risk, we used propensity scores to weigh the results by Area, Employee Class, gender and race/ethnicity.
Survey Validation
We rely on theoretical justification and past research for construct creation. However, we make minor modifications based on survey validation. In particular:
Confirmatory factor analysis (CFA) to test whether the factorial structure was consistent
Inter item correlation within a construct to consider adjusting questions based on reliability
We did not do quantitative pre-survey validity testing because:
Survey constructs and most items have been validated in prior surveys.
Constructs overlap so we do not expect a validation to show distinct constructs.
We do not want to take out a large subsample of the Lab for a validation
There is strong interest in each specific question
Data Sources
We will merge data collected by the survey with other data sources to understand the relationships between the different cultural constructs and areas of workplace well-being that the survey measures. Only the Culture Data Scientist will have the employee-level information linking survey responses to these data sources, and will immediately delete individual names -- please see the Data Privacy page. All reporting will be aggregated and confidential. These will include:
Retention
Years at the Lab
Seniority
Job type
Area and division
Race/ethnicity
Career or term
Employee class & work group (e.g., administator, scientist, engineer)
Manager status
Gender
Transparency is Essential.
Privacy & confidentiality of your survey data will be handled with the utmost care. Multiple protocols have been put in place to protect your data and ensure that the LBNL Culture Survey is a feedback channel you can trust.
LBNL Culture Survey: Outputs
Overview
Construct scores: Scores by construct are shown using the average responses of the questions in that construct.
Descriptive statistics: Average score of each construct will be shown by the different variables listed in data sources (such as by Area and years at the Lab). We also considered intersectional variables such as gender and job type.
We ran t-tests to check for constructs that were statistically significantly different from overall results apart from the group under consideration. Due to the large number of tests conducting, we applied a Benjamini-Hochberg correction for false discovery rates. We also applied a finite population correction given the large proportion of the Lab population that responded.
OLS regression: A regression helps to clarify which constructs most drive the employee engagement (i.e., an outcome measure in the survey), including the variables listed in the Data Sources section above as covariates.
Benchmarks: We considered available industry benchmarks for Net Promoter Scores and prior divisional surveys as points of reference. This survey will serve as a baseline to look at cultural change over time.
Detailed Results
These show:
Question-level averages
Net Promoter Scores
Top-5 Likes, Dislikes and Motivators
Results are also shown by a variety of groups:
Demographic groups
Employee Class and work group
Seniority, work mode, and tenure
Qualitative Data
Qualitative responses were stripped of all identifiers and read by at least one member of Berkeley Lab’s Learning & Culture Office. All responses were then qualitatively coded by theme in MAXQDA2024 simultaneously using an inductive (i.e., allowing themes to emerge from patterns in responses) and a deductive (i.e., applying the 11 culture constructs measured quantitatively in the LBNL culture survey) approach.
Background on Method: see Stefan Rädiker and Udo Kuckartz’s “Analyzing Open-Ended Survey Questions with MAXQDA: Step-by-Step” (2020, MAXQDA Press).
Use of Multiple Codes: Multiple codes were applied to the same comment or comment fragment whenever appropriate
Mixed-Methods: Culture survey results, analyses, and key findings were generated by looking across both quantitative (statistically significant & practically meaningful) and qualitative data sets. MAXQDA24 enables crosstabulations of certain statistics such as race, ethnicity, gender, Area, division, etc. as well as measures of code frequency.
Featured "Quotes": To protect the anonymity of employees, all listed qualitative quotes are AI-generated summaries, reconfigurations, and/or composites of several thematically-linked/similarly-coded qualitative comments submitted by survey respondents. These reconfigurations/composite quotes were derived by first collecting de-identified comments and/or relevant comment fragments related to a specific code and training ChatGPT to implement the same process of rewriting comments & combining them to produce these final results.