Sampling Techniques in Survey Construction and Data Analysis: Strengthening Validity and Generalizability in Statistical Procedures

Abstract

Librarians around the world are proudly user-focused in making decisions about collections, services, and programs for the patrons that visit our physical libraries and our web-based collections. Assessment activities are a critical aspect of the work we perform, and most of us have some experience with managing surveys and interpreting findings. Conducting surveys is one of the most common ways we collect data on patron needs, through directly mailed letters and through links we provide to surveys on our web sites and in emails to groups. But do we always know whether our approaches are strong enough to make effective decisions based on the results of our surveys? The goals of this presentation are to provide guidance on best practices for different types of sampling--ranging from convenience sampling to random sampling to stratified sampling--in order to strengthen the validity of findings and create a path for generalizability to the populations we serve. Issues that arise from different types of sampling techniques, such as confirmation bias and maturation effects, will be discussed with suggestions on how to limit their influence on respondents. In order to estimate the strength of the sample, statistical techniques such as Confidence Levels (CL) and Confidence Intervals (CI) will also be discussed with hands on examples of how to compute them. As a way to increase the reliability of the surveys we produce, models such as Inter-rater Reliability (IR) and Intraclass Correlation Coefficient (ICC) Analysis will also be introduced. Although most of our surveys focus on descriptive statistics, in cases of inferential statistics, Effect Size will also be presented as a way to interpret the strength of particular variables about which we are asking our patrons to respond. Although these terms might seem foreign to some of the participants in this session, the underlying theme of this presentation will to be to demystify these techniques. I will introduce them in ways that make sense to people who do not have a background in statistical procedures, and a major goal will be for people to walk away from this presentation feeling confident about the new knowledge they may have learned that will help them in their daily work lives. A presentation at the "Demystifying Statistics and Evaluation in Libraries" Satellite Meeting, organised by the Statistics and Evaluation Section and held at the University of Utrecht in Utrecht, The Netherlands from 17–18 August 2023.