Many coalitions are expected to implement a student survey as part of their evaluation efforts. However, with limited resources for evaluation, limited evaluation expertise, and the increasing work burden facing coalitions, selecting the appropriate student survey and/or creating one’s own is not a simple process. Below is some background information and tips to keep in mind when selecting a student survey and conducting an evaluation. We realize that coalitions may collect data on a number of different age groups; however, the information below focuses on data for youth 12-18 years of age.
Surveys
Whenever
possible coalitions should link-up with existing survey efforts in the community,
such as alcohol and other drug student surveys being conducted in schools or
community groups either through the school district, the city, or the state.
If specific questions need to be asked as part of funding requirements, then
it may be possible to add an item or two to an existing survey effort instead
of overburdening students with a separate survey effort. If a state-level or
national survey is being conducted, coalitions should determine if sufficient
sampling has or can occur so that data can be disaggregated to the local level
as needed. If your coalition targets efforts at the city level, then county,
state or national level data will not allow you to track changes in your community.
There are a number of different self-report student surveys. Before selecting a survey, be sure that it covers the type of information you want to know. For example, if you use a risk and protective factor framework, does the survey include questions on the type of risk and protective factors your coalition wants to target in its community? Some popular student surveys include the Youth Risk Behavior Survey, American Drug and Alcohol Survey, Search Institute’s Developmental Assets Survey, Communities That Care Youth Survey, and the National Parents’ Resource Information for Drug Education (PRIDE) Survey.
All coalitions that receive Drug-Free Communities Support Program (DFCSP) funding must collect information on the DFCSP’s 4 core measures. Please note that perception of disapproval by peers and adults does not mean parents must be surveyed. It is the respondent’s perception of disapproval by both peers and adults that should be measured. Additionally, the coalitions must collect, at a minimum, information on youth abuse/use of alcohol, tobacco and marijuana.
The 4 core measures
are:
1. Age of onset of any drug use
2. Frequency of use in the last 30 days
3. Perception of risk or harm
4. Perception of disapproval by adults
When
a survey can’t be implemented
If, for whatever reason, your coalition/community cannot conduct a student survey
(no money; too few students to properly survey with sufficient anonymity; etc.),
every effort must be made to collect other sources of substance abuse/use data.
Intermediate (e.g., risk and protective factors, social indicators) as well
as long-term outcomes (e.g., substance use prevalence levels) should be obtained
that are tied to your coalition’s logic model and strategic plan.
There are free/public-use measures available to coalitions. The measures on the CSAP Core Measures Initiative Web site are reliable and valid and can be down-loaded for use. These measures are comprised of common risk and protective factor predictors of substance abuse, as well as substance abuse/use measures.
Proxy measures are those measures that can substitute for student survey data when they can not be collected. These consist of social indicator data, such as substance-involved emergency rooms visits, traffic accidents and fatalities, etc. Examples of frequently used indicators that predict substance abuse are provided below. These indicators have also been validated through research.
In small/rural communities, classroom and grade levels have so few students that conducting a survey may result in a breach of confidentiality, as individual students could be singled out based on their responses. In this situation, a coalition or school may be able to conduct grade-level or school-wide focus groups for gathering information on perceptions of and behaviors involving substance use in the community.
Evaluation
Coalitions are being held to higher and higher standards these days. Funders
in both the public and private sectors are expecting coalitions to demonstrate
positive outcomes. Accountability to these funders and to your own community
is critical. In this day and age, coalitions can not afford to not evaluate
their efforts. So it is imperative that coalitions evaluate their interventions
and strategies to determine their effectiveness. Evaluation measures should
be tied to the goals of the programs, policies and practices that are part of
the coalition’s logic model/strategic plan. For example, an intervention
targeted at increasing successful compliance checks at alcohol retail outlets
should measure the number of alcohol retails outlets within the target community,
the number of compliance checks performed, and the # of successful compliance
checks.
If your coalition does not have staff members who are qualified to conduct evaluation activities, then the coalition should contact a local university or community college for help. There are always graduate students looking for class or thesis projects that could lend a hand for very little or no cost. Coalitions should make sure that students have some faculty supervision while on the evaluation project with the coalition.
Additional suggestions to guide your coalition evaluation can be found in the National Coalition Institute’s Evaluation Primer, which provides coalitions, especially Drug-Free Communities Support Program Grantees basic information to help them develop an evaluation plan. Click here to access the Evaluation Primer in PDF format.
Another helpful document is Strategizer 48 “Guiding Principles for Coalition Evaluation,” a technical assistance manual that provides a framework to help coalitions ensure their evaluation is meaningful and useful, not only for funders but for the coalition and community. Click here to access Strategizer 48 in PDF format.
For further information about selecting
surveys and evaluating your coalition’s efforts, please contact Evelyn
Yang, Evaluation and Research Manager at eyang@cadca.org.
Resources for Outcome/Evaluation Measures