Surveys and Evaluation: Where To Start?

The Institute's Evaluation and Research staff receive calls weekly from coalitions that need to monitor students' use of substances and related behavior problems. Increasingly, institutions that provide public and private funds demand accountability. Thus, coalitions must evaluate their interventions and strategies to determine their effectiveness. Below is some background information and tips to keep in mind when selecting a survey and conducting evaluation.

We realize that coalitions may collect data on a number of different age groups; however, this information focuses on data for youth 12-18 years of age.

Surveys
When possible, communities should piggyback on existing survey efforts, such as alcohol and other drug student surveys being conducted in schools or by community groups either through the school district, the city, or the state. If a state-level or national survey is being conducted, coalitions should determine if sufficient sampling has or can occur so that state data can be disaggregated to the local level as needed. State-level, county-level or national data will not do a coalition any good when it comes to tracking changes at the community level.

There are a number of different self-report student surveys. Before selecting a survey, be sure that it covers the type of information you want to know. For example, if you use a risk and protective factor framework, does the survey include questions on the type of risk and protective factors your coalition wants to target in its community?

Any survey conducted by a community that receives a Drug-free Communities Support Program grant MUST include questions on the DFCSP 4 core measures. Please note that perception of disapproval by peers and adults DOES NOT mean peers or parents must be surveyed; rather, it is the respondent's perception of disapproval by both peers and adults that should be measured.

The 4 core measures are:

1. Age of onset of drug use (incl. alcohol, tobacco and marijuana)
2. Frequency of use in the last 30 days
3. Perception of risk or harm
4. Perception of disapproval by peers or adults

Comparison of Leading Youth Surveys (MSWord Version) (PDF Version)

When a Survey Can't Be Implemented
If, for whatever reason, your coalition cannot conduct a student survey (no money; too few students to properly survey with sufficient anonymity; etc.), every effort must be made to collect other sources of substance abuse/use data. Intermediate (e.g., risk and protective factors, archival indicators) as well as long-term outcomes (e.g., substance use prevalence levels) should be obtained that are tied to your coalition's interventions, logic model and strategic plan.

The measures on the CSAP Core Measures Initiative website are reliable and valid and can be downloaded for use. These measures are comprised of common risk and protective factor predictors of substance abuse.

Proxy measures (i.e., measures that can substitute) for student survey data can consist of archival indicator data, such as substance-involved emergency rooms visits, traffic accidents and fatalities. A set of validated indicators that predict substance abuse is provided in the Intermediate Outcomes: Proxy Measures that Predict Substance Abuse chart .

In small/rural communities, classroom and grade levels have so few students that conducting a survey may constitute a breach of confidentiality, as individual students could be singled out based on their responses. In these cases, a coalition or school may be able to conduct grade-level or school-wide focus groups for gathering information on perceptions of and behaviors involving substance use in the community.

Evaluation
In this day and age, accountability is key. The way to demonstrate that your coalition is making a difference is by evaluating its efforts. If your coalition does not have staff members who are qualified to conduct evaluation activities, then the coalition should contact a local university or community college for help. There are always graduate students looking for class or thesis projects that could lend a hand for very little or no cost. Coalitions should make sure that students have some faculty supervision while on the evaluation project with the coalition.

Evaluation measures should be tied to the goals of the particular interventions and practices that are part of the coalition's strategic plan. For example, an intervention that targets parents' perceived risk of substance use and communication with their children should measure those facets of the parenting program.

For more information contact Evelyn Yang, Manager of Evaluation and Research for CADCA's National Coalition Institute. Please contact her for further information about selecting surveys and evaluating your coalition's efforts.  She can be reached at eyang@cadca.org.