OSP

Assessing Student Learning Using AAC&U VALUE Rubrics

Submitted by jasmith1113 on 19 January 2010 - 5:00pm.

in

Practice Type:

Portfolio: Assessment

Related Links:

AAC&U VALUE Rubrics

Three Canoes

AAC&U Essential Learning Outcomes:

Civic knowledge & engagement (local & global)

Creative thinking

Critical thinking

Ethical reasoning & action

Foundations & skills for lifelong learning

Information literacy

Inquiry & analysis

Integrative learning

Intercultural knowledge & competence

Oral communication

Problem solving

Quantitative literacy

Teamwork

Written communication

Purpose of Practice:

This practice is a proof of concept demonstration of one way to implement the AAC&U VALUE Rubrics in Sakai/OSP. We have created a matrix for a possible undergraduate learning experience, a set of evaluation forms using the AAC&U VALUE Rubrics, and a glossary explicating AAC&U Essential Learning Outcomes. These Sakai/OSP data structures can be used to create a portfolio implementation to assess student learning across and beyond the curriculum.

Our example is not meant to be used as is, but was created to suggest that the AAC&U Value Rubrics can be implemented in Sakai/OSP to support a variety of educational processes in specific institutional contexts. We invite users to set up these OSP data structures in Sakai and consider ways they could be further customized to capture the needs of a specific educational context.

More Details

Goals:

Our primary goal is to demonstrate that Sakai/OSP can be used to implement the AAC&U VALUE Rubrics within a customizable portfolio workflow.

Our secondary goal is to show how OSP Evaluation Forms can be used to create rubrics that allow instructors to click the descriptor for each criterion that respresents the work of a specific student and, assuming that each criterion is equally weighted, view the average score for that student on that rubric.

Tools Used:

We have uploaded the following Sakai/OSP data structures as a resource in Open Ed Practices:

An AAC&U Learning Outcomes Matrix with VALUE Rubric Evaluation Forms. The matrix has a row for each VALUE Rubric and columns for General Education, Coursework, Co-Curricular Activities, and Internships and Employment.
The VALUE Rubric Evalution Forms and their customized renderers as separate OSP data structures (for use in other matrices or wizards).
A Glossary defining the 15 AAC&U Essential Learning Outcomes for mouse-over and pop-up definitions in the matrix.
A Read Me file with instructions for installation.

One way to implement these data structures in a Sakai/OSP version 2.6 portfolio workflow is as follows:

The system admin sets osp.experimental.assignments = true in the institutional instance of Sakai/OSP.
The system admin or instructor sets up a portfolio site with the Resources, Matrices, Forms, Glossary, Evaluations, and Assignment tools and imports the data structures we have provided.
The instructor adds and posts assignments in the Assignments tool indicating to students in the assignment instructions which VALUE Rubrics will be used to evaluate each assignment once submitted.
The instructor adds the appropriate assignments to each matrix cell according to the VALUE Rubrics that will be used to evaluate the assignment.
The student completes one or more assignments and submits them for evaluation.
The matrix automatically posts the assignment to appropriate matrix cells for evaluation using the rubrics specified by the instructor.
The student reviews the assignments to be evaluated by each rubric (within the context of general education, coursework, co-curricular activities, and/or internships and
employment).
The student completes a reflection indicating his/her progress on the relevant skill
or understanding and submits the cell for evaluation.
The instructor may access the matrix of any student at any time by using the drop-down menu at the top of the matrix to view the matrix of a particular student.
The instructor may use the drop-down menu in the matrix to fill out a (formative) feedback form for any matrix cell of any student at any time.
When the student submits a matrix cell for evaluation, the instructor responsible for evaluating the cell will be able to access that cell through the Evaluations tool or via the drop-down menu in the matrix.
The instructor assigned to each particular cell uses the Evaluations or Matrix tools to access the student submission and review the evidence and reflection in each submitted matrix cell.
The instructor evaluates the evidence using the clickable evaluation rubric form in each submitted matrix cell. The evaluation form records the scores and calculates the average, assuming that all criteria receive equal weight.
The instructor saves the evaluation form and decides whether to complete the cell (lock it from any further change), leave it as pending evaluation, or return it to ready status for more work by the student.
The student reviews the instructor's evaluation via his/her copy of the matrix.

A second way to use the data structures would be to use the guidance features in each matrix cell to suggest to students what evidence to supply for evaluation with each VALUE Rubric. A drawback (or possible advantage) of this method is that students would have to think more analytically about how their available evidence fits each educational context and each essential learning outcome. Students would also have to take the initiative in associating evidence with multiple matrix cells in order to address all the learning outcomes. The matrix could be modified to have only one or two columns as a means of simplifying the process.

Lessons Learned:

There were two challenges in thinking through this practice. The first challenge was to create a viable Evaluation form for each of the AAC&U VALUE Rubrics. This challenge was met with rubric forms that match the display of each rubric on the AAC&U VALUE Rubric website, allow instructors to rate each criterion by clicking its descriptor, and providing for a total calculation of the score for each student on each instructor's rating of the rubric.

Login or register to post comments

AAC&U Essential Learning Outcomes matrix and VALUE Rubrics

Submitted by smkeesle on 18 January 2010 - 8:05pm.

in

Resource Type:

OSP Matrix

AAC&U Essential Learning Outcomes:

Civic knowledge & engagement (local & global)

Creative thinking

Critical thinking

Ethical reasoning & action

Foundations & skills for lifelong learning

Information literacy

Inquiry & analysis

Integrative learning

Intercultural knowledge & competence

Oral communication

Problem solving

Quantitative literacy

Teamwork

Written communication

Introduction

This is a package of demonstration material that illustrates one way that Sakai could be used to implement portions of an assessment strategy using the AAC&U Essential Learning Outcomes and VALUE rubrics. It is not a comprehensive solution and should be regarded as an early release. Feedback of any sort is welcome. Below we attempt to qualify this implementation with some assumptions and related limitations:

Assumptions and Limitations

Each AAC&U VALUE rubric includes a statement that says:

AAC&U Programs VALUE: Valid Assessment of Learning in Undergraduate Education

Login or register to post comments
Read more
1 attachment

Custom Renderer to Change Checkbox/Multi-Select Choice

Submitted by ericaack on 5 March 2009 - 5:28pm.

in

Resource Type:

OSP Form

This is a simple OSP form with a custom renderer that changes the way that OSP chooses whether to display checkboxes or a multi-select box. It would also change the same choice between radio button and a single select box. By default, checkboxes and radio buttons are used when there are fewer than 5 items in an enumeration. (The choice between checkboxes and radio buttons depends on whether there can be more than one selection. For example, maxOccurs=1 gives radio buttons for fewer than 5 items, but a single select list for more than 5 items.)

Login or register to post comments
Read more
1 attachment

Education 578 International Teaching Assistant Program

Submitted by clarevdb on 27 February 2009 - 2:59pm.

in

Training in this program is open to all graduate students whose native first language is not English and those who are currently serving as TAs or will hold teaching assistant duties during Fall 2009 semester. Selection to to the program is based on the immediacy of the TA assignment and evidence of need, as determined by a 30-minute ITAP Language Assessment (ILA).

Course Information

Course Number/ID:

EDUC 5780

Course Length (number of weeks):

15

Course Delivery Mode:

Hybrid/Blended (some face-to-face and some online interactions)

Average Number of Enrolled Students:

Between 30 and 60 students

Course Level:

College/University

Course Development & Delivery

Course Contributors:

Theresa Pettit, Faculty/Project Director
Clare van den Blink, Project Director
Kimberly Kenyon, Faculty/Project Director

Stew Marklel, Faculty
Stephanie Hanson, Faculty
Jody Gabler, Faculty
Nina Loney, Faculty
Pamela Pollock, Faculty

Eric Machan Howd, Admin/Inst.Design

Course Development:

The EDUC5780 course is designed to include a range of student activities and assignments to meet the educational objectives. Students record audio assignments each week, and video recording of teaching exercises. A key challenge for the course is how to easily submit this large collection of assignments, and have a way for faculty to provide feedback for each assignment. The course is designed with a range of assessment strategies from face-to-face feedback with instructors, online assessment of submitted audio & video assignments, and peer review. Students receive weekly feedback on their assignments and progress in meeting the course goals.
Another requirement of the course is to have students share their video with other students for peer review. Since the program consists of a series of courses, student assignments are retained and reviewed across several courses and several semesters. To meet these challenges, the Sakai OSP tool was selected. The portfolio features were critical to collecting and assessing student work. It keeps a record of student progress, and whether students are meeting objectives. Student artifacts include a variety of digital media files: audio files, MP3, WMA, video, Adobe pdfs and MS Office files. Using Sakai’s portfolio CLE allows students to use an electronic portfolio to collect artifacts and assists instructors in the creation of student-centered tasks. It provides participants with a portfolio of work that allows both instructor and students to monitor learner progress.
Since OSP is part of the Sakai learning system, the courses were able to seamlessly post assignments that were then submitted through the portfolio tool. A portfolio tool alone was not sufficient in meeting the needs of the courses.
A preliminary evaluation of International Teaching Assistant (ITA) students who participated in a pilot study affirmed that ITA students were using technology to enhance their language development, and that they believed that technology is an effective tool for improving language. Learner engagement and motivation are two important factors in an ITAs success. Offering ITAs creative ways to engage with digital media was at the forefront of the creation of the ITAP learning system. A small pilot study about the use of Sakai, found that Sakai CLE was a more robust system than other technologies. As a result it was implemented for courses in the program. A post pilot study evaluation tool was then used to assess the current pilot program. The focus of the evaluation was to examine the impact of this instructional design module on student's perception of their language gains. A method for comparing the oral proficiency levels of ITAs was built into the final analysis. Qualitative data was compared with quantitative findings to better inform CTE-ITAP's future initiatives.
The first phase in developing this course (and those across the program) in Sakai OSP was to define the learning outcomes and assessments within the curriculum in order to determine the design of the portfolio matrix. Much of this work had already been accomplished in the design of the courses themselves, so the focus of building the matrices was to settle on a consistent design of all four matrices, across all four courses in the program. Once that was determined, the last phase of course development was to adjust the courses and matrices within Sakai, according to the pedagogical principles and requirements of the program.

Course Delivery:

EDUC5780 is the first course that students in the ITAP must complete in the four-course program. EDUC5780, as well as the other three courses in the program, are delivered as ‘hybrid’, or ‘blended’, courses through Sakai’s portfolio CLE.
Incoming TAs whose first language is not English, and who are tapped to hold TA duties in an upcoming semester, are scheduled for a screening interview (ILA). Subsequently, they are informed of the program's purpose and assessment process. The screening of English language skills provides a critical juncture for the program. The level of spoken English, as well as listening comprehension skills, need to meet established criteria to assume TA responsibilities. The program assumes that if an ITA passes the initial language screening, then the ITA’s communication skills will be “successful” in the classroom. If an ITA passes the screening, then she/he can proceed as a TA.
If a student is required to take classes wirth the ITAP, he or she enrolls in EDUC 5780. The student is required to maintain of portfolio of all work: recordings of audio journals, oral and written reflective assignments, and video recordings of teaching practices/sessions. All of these artifacts are submitted to the Sakai portfolio matrix and commented upon by the instructor; the student also reflects upon their artifacts/learning throughout the course via the CLE and in small-group, face-face consultations.
At the end of the EDUC 5780 course, the ITA's level of language proficiency is re-assessed by an ITAP instructor. When ITAs have been determined to have “passed”, meaning they have successfully been able to communicate an Advanced Low level on the ILA, they are no longer required to participate in the program. If the ILA result is below the Advanced Low level, the ITA will be required to enroll in EDUC 5790. If ITAs reach the Intermediate High on the ILA (one level below Advanced Low), they can hold TA duties concurrent with EDUC 5790.
The method of course delivery is not only strongly ‘learning-centered’, but also heavily ‘assessment-centered’. The process of submitting ‘artifacts’ to a portfolio matrix, receiving feedback/assessment on those artifacts, supports research that shows that assessment-centered learning environments offer many opportunities for feedback and revision, and tie directly into the student’s learning goals. The strength of ITADP’s innovative design and use of the Sakai OSP supports that claim.
The final outcome of the program assumes that successful communication skills in a classroom will lead to an improvement in undergraduate instruction.

Course Self-Assessment

Communication & Collaboration Self-Assessment:

Excellent

Communication & Collaboration Evidence:

Communication and collaboration in this course are one of the most important aspects of the innovation. Students have a lot of communication/collaboration with their instructors regarding the evidence that they submit to their portfolio matrix. Students regularly meet with instructors, both in-class and through small-group conferences, to reflect upon their learning and their evidence. The frequency of adding learning artifacts to the portfolio and receiving feedback on those artifacts instills a large amount of assessment, regular assessment, in this course, so learner input and reflection are highly valued in the pedagogy behind the course design. Community building takes place outside and around the course, through the ITADP program itself. The power of this use of the Sakai portfolio CLE lies in the constant reflection, self-assessment and instructor assessment that students engage in on a regular basis.

Learning Material Self-Assessment:

Excellent

Learning Material Evidence:

Our self-assessment of the learning material in this course (andprogram) has found that students clearly understand the components,structure and roles of the Sakai portfolio CLE.
Students in the program reported that course interactions were easy,and that the assignments for completing their portfolio matrix wereclear and concise. The ‘hybrid’ model of delivering the learning(in-class and Sakai portfolio CLE) also gave the student moreopportunities to reflect upon their evidence and learning throughoutthe semester.

Learning Outcomes & Assessment Self-Assessment:

Excellent

Learning Outcomes & Assessment Evidence:

The overall goal of this project and, in part , the goal for this course, was to develop and implement a working prototype of a “Portfolio Learning System” for the International Teaching Assistant Development Program (ITADP).
Our self-assessment reiterated that the mission of this program is to enable ITAs to enter the Cornell classroom as confident instructors who can communicate their domain expertise in an interactive, student-led environment. Curriculum in four courses addresses instructional strategies as well as general language proficiency. In addition to coursework, students attend weekly small group tutorial sessions during which instructors work with participants to create an individualized plan for language development. The prototype was successfully created using Sakai OSP.
We found that the new portfolios enable ITADP students to create, store, and manage audio and video recordings for their assignments in all four courses ITA courses. If students are enrolled for four semesters, even with different instructors along the way, they can maintain the same portfolio throughout the entire period. As portfolios grow in size and quality, students gain evidence to document their progress in spoken English. The same goes for documentation of their teaching abilities. The system also enables instructors and program administrators to review submitted materials. Students have the ability to hone their critical thinking/judgment and problem solving skills through multiple means. Also, students are able to receive individual feedback on their learning progress through multiple assessment opportunities in the course. These assessments provide feedback to help students reach target levels with consideration of individual student's personal styles and needs.
At present, all sections and all courses with ITADP are using the online space. It has been the experience of program administrators and instructors that, compared to the “old way” of handling audio files through conventional e-mail, Sakai has “distinct advantages” and is more streamlined. No longer does the program have to concern itself with managing a large number of files, which now reside on an external server. In addition, shared artifacts eliminate the complications of losing files from individuals computers. The impact on the program’s workflow is great in that this course, and others in the program, reduce a lot of the administrative overhead so that the focus of the teaching and learning can be more about the ‘learning’ (teaching, student interaction, reflection, etc.) and less about the logistics of storing files, evidence, etc.

Course Look & Feel, Web Usability Self-Assessment:

Effective

Course Look & Feel, Web Usability Evidence:

The Sakai OSP course is designed to be clear and concise. We’ve strived to make the look/feel as simple and straightforward as possible. Students are only required to interact with the assignment tool and matrix (the main tools) to do their own work. The design enhances the presentation and communication of key information/content throughout the course. The design of EDUC 5780 mirrors the design of the other three courses in the program; this consistency enhances student learning throughout the ITADP program. Students also have the opportunity to build their own on-line portfolio at different points in the program which supports self-reflection and self-assessment.
When teaching staff were asked to elaborate on the strengths and weaknesses of Sakai, similar comments about its strength arose. Strengths included Sakai’s ability to track progress, its ability to store work over an entire semester (or longer) and its capability to “keep materials all in one place for easy viewing.” Additional comments included Sakai’s progressive feel and noted improvement over conventional e-mail. The instructor self-evaluation portion of this project/course found that instructors were generally satisfied with its innovative nature and especially liked the way Sakai was more accommodating to larger file sizes.
Additionally, when ITADP students were asked if they were satisfied with Sakai, a combined 62% of respondents replied "Agree" or "Strongly Agree". In the strength's section (open text field) of the online survey, 11 out of 15 students mentioned Sakai's effective matrix design and clear organization of information. Other strengths noted were ease of use, accessibility of files and ability to upload a variety of file types. In the weaknesses section (open text field), less than 1/3 felt the interface was complicated and the menu somewhat confusing.

Learner Support Self-Assessment:

Excellent

Learner Support Evidence:

Support for any innovative program is crucial. The ITAP has addressed this fact through a variety of support resources: a student-orientation session, a ticket-based software help system, a faculty training session and a support wiki. The Sakai OSP courses in this program contain a variety of resources and contact information and support documentation is created ‘on-the-fly’ when needed on a support wiki. EDUC 5780 users report great satisfaction with learner and instructional support.
The student-orientation session introduces students to the CLE as well as to the particular requirements for using the CLE throughout the semester. Courses also contain a web-link to a support wiki; this wiki contains step-step instructions on how to submit evidence to the matrix, trouble-shooting tips/tricks as well as extensive information about the OSP. The wiki is regularly updated with new information regarding the CLE. The ‘ticket-based’ help system allows us to track document fixes for future reference. In most cases, solutions to issues are also cross-posted to the support wiki for student/faculty reference. Students can e-mail support requests to this system at any time, and requests are usually responded to within a day.

Teaching Innovation

Teaching Innovation:

Students in the ITA program are charged with making their teaching better. Reflective processes and features of this course enable students to attain that goal.
As illustrated in the ‘Learning Outcomes Self-Assessment’, we found that the new portfolios enable ITADP students to create, store, and manage audio and video recordings for their assignments. As the portfolios grow in size and quality, students document their progress in spoken English, as well as in teaching skills, over multiple semesters. The system also enables instructors and program administrators to review submitted materials.
The design of the courses in this program requires regular interactions between student and faculty. Administrative time savers are critical. At present, all sections and all courses within ITAP are using the online space. It has been the experience of program administrators and instructors that,that, compared to the “old way” of handling audio files through conventional e-mail, Sakai has “distinct advantages” and is more streamlined. No longer does the program have to concern itself with managing a large number of files. They now reside on an external server. Such self-assessment is clear evidence of ‘innovation’ making the teaching process more efficient and productive.

Login or register to post comments
Read more

How do I use OSP Forms, Wizards, Matrices and Templates from this library in my instance of OSP?

Submitted by smkeesle on 8 October 2008 - 1:52pm.

in

The OSP tools allow you to import and export data structures into a zip file format. To import a zip file that you find in this library you need to download it from this web site, upload it into resources in an instance of Sakai and import the zip file from the appropriate tool.

To import an OSP form:

Login or register to post comments
Read more

Example Form

Submitted by smkeesle on 6 October 2008 - 7:16pm.

in

Resource Type:

OSP Form

This form includes several types of fields and may be useful to developers learning OSP.

Login or register to post comments
1 attachment

LaGuardia Community College

Submitted by beynon on 18 February 2008 - 2:54am.

in

So many students come from so many different places to study together at LaGuardia (over 160 different countries at last count) that we call ourselves "The World’s Community College." Being The World’s Community College is not just a statement about who we are, it's a statement about who we aspire to be. Our goal is to educate leaders for a global workforce and a global citizenry.

Related Links:

LaGuardia Community College website

Login or register to post comments

University of Delaware

Submitted by mathieu on 11 February 2008 - 5:45pm.

in

Located in Newark, Delaware, USA, University of Delaware enrolls over 16,000 undergraduates and nearly 3,000 graduate students. As a state-assisted, privately controlled institution, the University seeks to enroll students from diverse backgrounds and a wide variety of geographic regions.

Related Links:

UDLMS Project Site

Related Links:

Sakai@UD Gateway

Login or register to post comments
Read more

What's the difference between a Matrix and a Wizard in OSPortfolio?

Submitted by trillich on 10 January 2008 - 5:46am.

in

A matrix is a two-dimensional wizard. Really.

A hierarchical wizard is a tree-type structure of categories with nodes that lead to "wizard pages".
A sequential wizard is just a flat step-by-step sequence of "wizard pages".
A matrix is a row-and-column grid of cells, each of which takes you to a "wizard page".

That's it!

Login or register to post comments
Read more

How to develop an OSPortfolio template

Submitted by trillich on 10 January 2008 - 5:30am.

in

How do I develop a workflow for creating an XSL ePortfolio template inside Sakai and OSP?

This is a two-step process. First you'll need to get your hands on the XML that you'll be creating your XSL for (you do this by creating a pass-through template so you'll have the resulting XML to work with) Second, you can develop the XSL and your XPATH expressions on your local computer with minimal fuss.

Login or register to post comments
Read more

How do I join the Sakai or OSPortfolio community?

Submitted by trillich on 10 January 2008 - 5:10am.

in

Short version:

http://collab.sakaiproject.org/

Long version:

If you're tinkering with Sakai or its Electronic Portfolio "Open Source Portfolio (OSP)" you might have questions -- or you might find yourself wanting to contribute back to the community (documentation, quality-assurance testing, project management, java or XML coding...)

Login or register to post comments
Read more

What is the relationship between OSP and the Sakai Project and its Sakai Foundation?

Submitted by angell on 3 December 2007 - 10:33pm.

in

Open Source Portfolio (OSP) is a set of integrated tools that live inside the full Sakai Collaboration and Learning Environment (CLE).

OSP and Sakai are both community source projects of the non-profit Sakai Foundation.

See the FAQ: Can I use OSP without using Sakai?

Login or register to post comments

Can I use OSP without using Sakai?

Submitted by angell on 3 December 2007 - 10:07pm.

in

OSP
Sakai

Yes.

Although Open Source Portfolio runs in and depends on the base Sakai Collaboration and Learning Environment, you can enable and use OSP tools without turning on or using any Sakai collaboration or course tools.

Alternatively, you can enable and use only those Sakai tools that may be useful in your specific portfolio environment, for example, Sakai's announcements, mailling list/archive or calendaring tools.

Login or register to post comments
Read more

rSmart

Submitted by angell on 12 September 2007 - 1:37pm.

in

rSmart is a Sakai Commercial Affiliate, actively involved in ongoing development and support of Sakai and OSP within the Sakai community. Over the years we've helped over 40 institutions implement Sakai and OSP pilots or production instances on their campuses. rSmart also offers the rSmart Sakai CLE, an open-source, commercially-supported release of Sakai that provides easy configuration, a polished user interface, a quality-assured expanded toolset, and comprehensive documentation and training.

Related Links:

rSmart Website