In a recent Academic Impressions webcast, student learning assessment expert John Hoffman polled student affairs representatives from 200 institutions of higher education about their data collection methods for assessing student learning. Participants were asked to select their two most common methods of gathering data. The results were dismaying but perhaps unsurprising:
- 89% use student surveys, and
- 61% count student contacts.
…but, only:
- 28% rely on institutional data from IR.
- 22% make use of focus groups.
- 20% collect data within a course, workshop, or program.
- 18% rely on formal observations.
- 16% use interviews.
The reliance on surveys is telling. Surveys appear easy to design and deploy, but Hoffman cautions that there are a number of shortcomings with using surveys as your primary method of collecting data to assess student learning:
- With each passing year, students show greater evidence of survey fatigue, leading to diminishing returns in response rates.
- Surveys offer primarily indirect measures of student learning.
KEYS TO EFFECTIVE ASSESSMENT: “ROLLING UP YOUR SLEEVES”
“Surveys are valuable, but often what we get is the student’s self-report of their learning. Yes, you can test them within the survey, for instance by asking if the student can name three examples of services provided by your career development center — that invites more direct assessment of what they have learned. But to really get the data we need, we have to roll up our sleeves and collect data during interaction with the students.”
John Hoffman, California State University, Fullerton
KEYS TO EFFECTIVE ASSESSMENT: FOCUS ON COHORT LEARNING
“You might be asking, ‘How am I supposed to assess learning in a specific session? Do I need to write a student learning outcome for each session and then design an assessment within that one session?’ That would be cumbersome! But we can engage in some ongoing assessments.
“Assessment has more to do with cohort learning than individual student learning. I’ll use an academic example. When I grade students taking a course, I assign grades to specific students based on their individual work. That is not assessment, that’s grading. My assessment is: After I’ve graded all of their papers, I will take the rubric and tally-mark: for each student learning objective, how many students who completed this assignment were advanced? How many were competent? How many were at a ‘needs improvement’ level? You are assessing the learning of a student cohort.”
John Hoffman, California State University, Fullerton
Direct Measures of Student Learning
So how do you move beyond surveys? Here are two methods that require relatively low effort to implement:
1. Have students create learning products.
For example, if your career development center is giving a workshop on resume-building, have students create resumes as part of the workshop. Then collect copies of the resumes (or use a smartphone to capture them through photography) and assess them according to a rubric that maps to your career development center’s student learning objectives. This allows for a much more direct assessment of what the students have learned than an end-of-workshop would.
Or suppose you are holding mock interviews. With the students’ permission, record the interviews. You can then send copies of the videos to the students with their feedback. You can also view a number of the videos, rubric in hand, and assess where the students are in their progress toward learning outcomes you’ve identified.
2. Record direct observation.
Hoffman offers the example of “counselor notes.” Students come in to the career center or to academic advising with different needs at different times at the year, and at different points during their progress toward a degree. Identifying these patterns, you can write learning outcomes that describe what you hope first-year students will know or be able to do after their first counseling sessions in the fall. And you can write outcomes for sophomores, juniors, and seniors.
A best practice when working on-on-one with students in either developmental advising or counseling in the career center is to take notes at the end of the session (or at the first available break in the day). You might even use a quick form to populate a database with these notes.
What This Lets You Do
Here’s an example of how a simple addition to your process (such as adding counselor notes) can empower more direct assessment of student learning:
- Perhaps you write two learning outcomes for this student cohort. Perhaps you want these first-year students to (A) engage in career exploration, and (B) recognize that the tests and tools available, while useful, will be limited, and that they will need to engage in more analysis and problem-solving in order to apply what they learn from Myers-Briggs or from a strengths assessment to their career exploration.
- When a student comes in for a counseling appointment, you engage in formative assessment with a few questions. Maybe you find that the student hasn’t done any career exploration. Or maybe the student is asking what they should major in; maybe they are beginning to think about their long-term goals.
- When appropriate, the appointment becomes an opportunity for an intervention. Perhaps you recommend that the student take a strengths inventory.
- You collect data by taking notes. Was the student interested in taking a strengths inventory? And did they show evidence of dualistic thinking (did they want to just check a box and get an answer), or did they show awareness that the inventory would provide information that they would need to reflect on as they make decisions?
- Summative assessment: You look back over your counselor notes. How many students in this cohort came in with an interest in career exploration?
- Watch for opportunities for an educator intervention. If you have many students meeting outcome (A) but few meeting outcome (B) — that is, they’re engaging in career exploration but they want to check the box to get a quick and easy answer — then it’s time to ask whether there is something new that you can do during your counseling sessions to encourage and empower students to approach the tools less dualistically.
A CAVEAT
“Of course, students will naturally move away from dualistic thinking over the course of student development, so I can’t just say that students in their senior year are asking certain more sophisticated questions because they came to the career development center.
“But I can watch for patterns.
“What questions are asked by seniors who come into the center for the first time in their senior year? And what questions are asked by seniors who come into the center but have been involved in counseling sessions or our programming over the past few years?”
John Hoffman, California State University, Fullerton