lagged evaluation

Image

The standard time to evaluate a course is at the end of the semester.  That is when we typically solicit feedback from students in order to improve our course for the future.  But, that’s not the only time that we can get feedback.  If we got feedback during the class, we could use it to improve that particular class, and, in general, iterate and improve more quickly.  And, if we get feedback long after the class is over, I think that we can gain insights about deeper and more interesting kinds of things.  Did the students actually remember what we taught?  What turned out to be most useful to them?  What did they think was useful that turned out not to be?  Therefore, to evaluate my course from last spring, I did an evaluation this spring.  Here’s what I learned from my lagged evaluation.

In the spring of 2013, I taught a course for first year graduate students called Sociology 504: Social Statistics (basically, linear regression building toward the general linear model).  At the end of the semester, I evaluated the course and thought about how I could make it better next time.  One of my goals for the course is to empower students to do their own careful and creative quantitative social science research.  However, neither I nor the students could really evaluate how well the course achieved this goal because they had not really done their own research yet.  It was kind of like trying to evaluate sky driving classes before the students go sky diving.

After taking my class in the spring of 2013, all of the students took a class called the empirical seminar where they used everything they learned in their first year classes to write an empirical research paper of publishable quality.  Thus, the end of the empirical seminar  provided me with a great chance to do a second evaluation, after they had tried to really apply the ideas that I had been teaching them.

On the course evaluation I used immediately at the end of my class, I asked the two students two main questions:

  1. What do you think were most important things that you learned in Soc 504?
  2. What would have liked to learn more about in Soc 504?

The first question was designed to see if the students picked up what I thought were the main themes of the course.  The second question was designed to see what I could have done better.  Then I asked these exact same questions to students one year later, after they had tried to use what they learned in my class to complete a research paper.  Comparing the answers, two main themes emerged:

  • Skepticism fades and R endures: When writing about the most important things that they learned right at the end of the class, many students described what I would summarize as skepticism about the “general statistical recipe” (i.e., run a regression and look for statistically significant coefficients).  That skepticism was great because that was one of my main themes for the course.  A few students mentioned R but mostly as an afterthought.  However, one year later, many students wrote that R was the most important thing they learned, and almost nobody wrote about skepticism with the general statistical recipe.  I’m not sure if this is because students have completely internalized that skepticism, and, therefore, don’t remember learning it.  Or, perhaps they have lost some of their skepticism.  I’ll have to ask them so that I can find out.  If the skepticism has faded, I’ll have to develop new ways to make that stick.
  • Nobody cares about data cleaning until they care about data cleaning: At the end of the course, no student expressed a desire for more practice with recoding, data cleaning, and other kinds of research nuts-and-bolts.  However, after they spent a year struggling through this issues while writing an empirical paper, many students mentioned these as topics that they would have liked to learn more about.  Since good procedures have been developed for data cleaning and data management (here and here), I’m going to introduce those ideas to the students so that they don’t have to attempt to re-invent them.

At the end of the post, I’ve put wordles showing the differences in the actually words used at end-of-course and end-of-course + 1 year.

Based on this experience, I think I’m going to try lagged evaluations again with other courses.  Here are some issues that I’m going to consider for the future.

  • Design the first evaluation with later evaluations in mind: In this case, I had the idea for a lagged evaluation long after my initial evaluation.  So if I wanted to compare answers for a set of questions over time,  I was limited to the questions that I asked initially.  Just to be clear, I also included questions in the lagged evaluation that were not in the initial evaluation, and these were useful too, particularly about tweaks to the syllabus.
  • Longer lags: In this case, the evaluation was one year after the course.  Should I try to survey these students again in the future?  If so, when?
  • Survey evaluations vs. tests: In this case, all of the evaluation was based on what the students said they liked and did not like.  In the future, it might be interesting to use some kind of exam to measure what they actually know how to do.  However, I think it could be difficult logistically; it seem hard to get students who are not in your course to take something like a one-hour statistics exam.
  • Systems for dealing with non-response: In this case, we had 100% response rate because Princeton Sociology has amazing graduate students.  However, in other settings and as the lag increases, the rate of non-response is likely to increase and potentially be correlated course experience (e.g., students that loved the course might be more likely to participate).  Therefore, I need to develop a better system to track non-response even in an anonymous survey.
  • Learn more about other survey-based methods of course evaluation: I’m not the first person to do survey-based evaluations of a course, and I am ignorant of much of the earlier work in this area.  In the future, I would like to know more about how other people have tried to solve this problem, particularly when the goal is to improve instruction (rather than evaluate faculty).

Overall, I found this to be a very helpful experience, and I’m going to do it again.  It has given me concrete ways to improve my course (e.g., more practice with real data, probably through an assignment to replicate a published paper) and something to investigate further (e.g., whether students maintained some of their skepticism).

Here are the wordles showing the differences in the actually words used at end-of-course and end-of-course + 1 year.

  • What do you think were most important things that you learned in Soc 504?

End-of-course

Image

End-of-course + one year

wordle_most_important

  • What would have liked to learn more about in Soc 504? 

End-of-course

Image

End-of-course + one year

Image

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s