GenEd Sustainability Learning Outcomes (SLO) Committee Meeting
Friday, November 1, 2013
Jeffords 326, 9:30-10:30am

Agenda

  1. Draft outcomes and the vetting process (10 min)
  2. Assessment models with John Ryan (20 min)
  3. Realistic assessment models for UVM with John Ryan (30 min)

Guest: John Ryan, Director of Institutional Research at UVM

MINUTES

In attendance: Laura Hill Bermingham (co-chair), Deane Wang (co-chair), Lisa Watts Natkin, Tarah Rowse, Christine Vatovec, Rycki Maltby, Sam Ghazey, Allan Strong, Brian Reed, Marilyn Lucas, John Ryan, Laura Webb, Wendy Verrei-Berenback, Stephanie Kaza

Not in attendance: Brian Lee, Sandy Wurthmann, Leon Walls

Laura HB: Deane and I made some edits. Including and introduction and preambles to the outcomes. Brought the intro to the "21st century." Preambles are new.

Deane: The preamble is the context for the learning outcome.

Rycki/Christine: Can we format so that it is clear which are preambles and maybe number the learning outcomes?

Some formatting adjustments were made.

Laura HB: Maybe the next meeting will be dedicated to getting and putting feedback together.

Assessment

Brian Reed: As we work on each of the gened outcomes, we are trying to integrate assessment as we go. That will look different in different cases. There are multiple ways to do it. Assessment is critical.

Deane: I don't see that there will be an office of University assessment, right?

Brian Reed: Correct. We don't have a plan for that infrastructure. The Provost is engaged around assessment though. Done with Faculty Senate right now. There will be some sort of central investment

John Ryan: I think we are moving towards some middle ground, at least functionally. We work to support in terms of design, thinking, information, research, data. What are the best ways to leverage existing sources of information? What are the things that are already or could be happening at the curricular course level? This meeting today is an example of the kind of participation that is needed.
  • How can we work with things that faculty are already doing in order to provide meaningful in depth information?
  • If we are interested in growth or change over time how can we look at that?

Laura HB: Do you have collaboration with individual faculty members? It is a faculty driven effort?

John Ryan: The primary vehicle for GenEd has been in these kind of settings. I've done a number of ad-hoc things. CTL does some support around learning outcomes. Yes, it is faculty driven and there is some support.
  • This is an evolutionary process. Modeling the process over time.
  • I do quite a bit of literature review on assessment. Especially in terms of challenges or hurdles in terms of assessment.
  • It does need to be a faculty and student centered process.

Brian Reed: A larger issue in terms of GenEd is how we manage it in the long-run? For example is it possible that this becomes a standing committee? How does it work with the Faculty Senate? Right now it is piecemeal, but we need to have a conversation about how we do this?
  • What are the mechanisms for oversight?
  • How do we do faculty development in terms of assessment?

Deane: So the place where everything might come together could be a curriculum committee?

Brian Reed: If we are in the business of creating university wide curriculum then how will we manage it? Maybe a new committee.

John Ryan: We need to communicate that we are doing this process. We think we maybe need to go in this direction. We will try and report back.

Laura HB: We don't have any reporting. We don't have any data. We would need some of this.

John Ryan: Already in my mind from a quick review of the SLOs I have some idea for assessing progress, change, or achievement. It may not require a lot of additional or new work. We need some coordination and alignment to pull it together.

  • If we don't have an assignment, exam, paper that we can look at, then how might we do it?

Laura Webb: What about looking at Sustainability Faculty Fellows as a practice run?

Wendy VB: Yes, we might be able to embed an exercise and work with the current cohort.

John Ryan: We will want to try and tackle the outcomes in mutliple ways.
  • Direct - paper, exam, assignment. Look at these to evaluate.
  • Indirect - outside of a graded assignment. How do we think the students are doing?
  • Applying some of the very basic principles we might apply in our own research or scholarship.

Deane: In the context of formative assessment, if an academic unit chooses to own those learning outcomes, then the nature of the assessment would be different than if departments are just accepting that students can do it outside their unit?

John Ryan: It would be good to map who is doing what where. We need to understand what is set up in different areas.

Laura HB: There is an enormous amount of collaboration and communication needed to do this.

John Ryan: Yes.

Brian Reed: I think we should have a discussion within the Faculty Senate exec committee and then the Faculty Senate. What is the structure? How do we want to move forward? We need to have this conversation. I'd like to approach Julie Roberts on this.

John Ryan: UVM was an early participant in NSSE (National Survey of Student Engagement). We've done it every 3rd year since 2002. This is a broad institutional level of data. If there is an opportunity to participate in a consortium such as this we may be able to do it more often. Doing an off cycle could be done. The goal would be to support one or two or three of the learning outcome areas.
  • It's over 6,000 for the whole institution.
  • Census approach with 1st years and seniors.
  • Informed by the literature on high impact practices.
  • Retention, academic achievement, integration, and "kind of" satisfaction

Deane: Are they coded so that you can get the same student again?

John Ryan: Yes, we had 200 from 2008 to 2011.

Brian Reed: This is one way that we can get some data. We will still want to do some things that are more granular.

John Ryan: Responses are confidential but they are not de-identified. We can do some modeling work to look at relationships in variables. We can connect it to other databases in some ways.

  • What are the potential effects or relationships between gains in sustainability outcomes and student engagement outside of sustainability?
  • We survey all first year and all seniors and generally get around 32-35% response rate. It's robust enough for institutional analysis.
  • Non-response bias? There are a variety of techniques used to address that. By and large you are getting close to the population (1 in 3) in terms of characteristics.

Laura HB: How is the data made available to the faculty?

John Ryan: All the data is online on the IR website. Individual datasets or requests can be done individually with us.

Deane: So, institutional level student self reported survey like NSSE, then class level data, then something else... a picture starts to emerge from different levels of assessment.

Brian/John: Yes, that's it.

Brian Reed: Diversity is grappling with this too. They did one indirect assessment (surveymonkey, focus groups).

John Ryan: How do D1/D2 courses fit in with the diversity learning outcomes?

Laura HB: What about incorporating co-curricular activities in terms of learning outcomes? Would NSSE capture this?

John Ryan: There are some co-curricular questions.

Laura HB: What about NSSE getting too big?

John Ryan: That's my job to look at that. The modules tend to be pretty short. 3 to 5 items. They can be rotated. We get the whole dataset.

Deane: There's a lot of material on the wiki in terms of how other institutions are doing assessment.

Deane: Is what we as a committee are doing is recommending general approaches and then have "experts" refine that? Or should our process say we will do this, we want this asked, this is the process (in other words more specific recommendations all through the committee)?

Brian Reed: There are some experts around campus who can assist. We can make those connections.

John Ryan: There is opportunity for having research done on campus around assessment.
  • Think of this as a first step and I can reconnect with you all in the future.