Prior to January 2004, Assessment at The College of New Jersey was driven by external accreditation

A Case Study of Assessment of the Academic Major and the Quantitative Reasoning Component of the Liberal Learning Curriculum at The College of New Jersey and Its Possible Application Elsewhere

Edward Conjura and Cathy Liebars

Department of Mathematics and Statistics

The College of New Jersey

Table of Content

Introduction Page 2

PREP/SAUM Workshops: Their Nature and the Role Played

In Assessment of the Major at TCNJ Page 3

Assessment in the Department of Mathematics and Statistics

Prior to January 2004 Page 5

Redefining Academic Majors Page 6

Implementing the Assessment Cycle Page 7

Feedback from Students Page 9

Assessment of the Quantitative Reasoning Component

of Liberal Learning Page 10

Summary and Conclusions Page 11

Appendix A: Major Program Goals Matrix By Course for the Mathematics Major

Appendix B: Major Program Goals Matrix By Course for the Mathematics Education Major

Appendix C: Assessment Cycle Matrix for the Mathematics Major

Appendix D: Rubric Used For Evaluation of Student Presentations in Geometry

Appendix E: Rubric Used For Evaluation of Student Teaching Presentations

Appendix F: On-Line Survey Questions

........................................................Page 1.....................................................

Introduction

Prior to January 2004, Assessment at The College of New Jersey (TCNJ) was primarily driven by external accreditation bodies such as The Association to Advance Collegiate Schools of Business (AACSB), the Accreditation Board for Engineering and Technology (ABET) and the Nation Council for Accreditation for Teacher Education (NCATE). The Schools of Business, Engineering and Education and departments such as Computer Science were veterans at gathering information and using it as evidence of curricular effectiveness. On the other hand, with few exceptions, departments such as Mathematics and Statistics tended to rely on piecemeal data and anecdotal evidence about job placement and graduate school acceptances to judge the quality of the job being done. Assessment was something that administrators wanted departments to do but something that they had little interest in doing. It was low on the priority list of things that needed to be done by individual faculty, if on the list at all. However, the winds of change were becoming stronger and louder. The Middle States Commission on Higher Education had already made it clear that assessment had become the backbone of their revised standards. To add momentum in the direction of the need for assessment, in late 2003 the College began going through a complete academic transformation. Degree requirements were changed from 120 credits to 32 course units. General Education was given the new title of Liberal Learning and completely overhauled, and academic majors were being redefined. The number of courses in the major was to be reduced but the courses were to be “transformed” to become “more intensive.” The college also adopted a writing intensive curriculum that was intended to help students develop the skills necessary for today’s world, and it implemented a comprehensive service learning initiative that began as part of the First Year Experience within Liberal Learning and is currently being expanded across the curriculum to integrate additional service learning into the second year and beyond. New features, such as an optional fourth meeting hour and a required capstone experience were also introduced. In summary, the entire college went through transformative change.

This paper chronicles the work done by members of the Department of Mathematics and Statistics at the College of New Jersey in developing and implementing an assessment plan for evaluation of academic majors offered by the department. In particular, it will focus on the work done by Edward Conjura (Ed) and Cathy Liebars (Cathy) and the role they played in educating, recruiting and training colleagues who helped make any and all successes concerning assessment possible. To a lesser extent this report will also discuss the work done in assessing the effectiveness of the Quantitative Reasoning component of the Liberal Learning Curriculum at TCNJ. It is the hope of the authors that others might learn from the success and failures experienced at TCNJ and perhaps help others in their assessment endeavors.

........................................................Page 2.....................................................

PREP/SAUM Workshops: Their Nature and the Role Played in Assessment of the Major at TCNJ

Much of the credit for the accomplishments achieved in the Mathematics and Statistics Department at TCNJ related to assessment can be given to the training and experiences that Cathy and Ed had through their participation in PREP/SAUM Workshops on Assessing the Undergraduate Program in Mathematics. First, the workshops provided valuable training. Second, the task based nature of the workshops provided motivation and valuable experience. Third, participation in the workshops directly and indirectly established them as leaders and experts on assessment, and added credibility to the team’s efforts back at TCNJ, both with colleagues and administrators.

Between March 5^th and 8^th of 2004, Cathy and Ed attended a workshop held in High Point, North Carolina. The training received there was of great value and included becoming familiar and comfortable with all of the following:

The Assessment Cycle

1. Set Learning Goals and Objectives

2. Design Strategies to Accomplish Objective

3. Determine Areas and Methods of Assessment

4. Gather Assessment Data

5. Use Assessment Data to Improve the Program.

Assessment of the Major

Some Approaches that have been Effective

Recommendations for the Major

CUPM Curriculum Guide: 2004

Listening to and Learning from Students

Tools for Assessment

Portfolios

Research on Learning

Knowing what Students Know

........................................................Page 3.....................................................

Teams from twenty eight colleges and universities participated in this workshop. Each team gave a report describing the current status of its assessment plan and activities involved in implementing their plan. Each team also developed an assessment project that would be carried out during the upcoming year. A progress report on this project was to be given in 2005 at the second PREP/SAUM workshop to be held in Atlanta immediately following the Joint MAA/AMS meeting. The proposed project for TCNJ was to have Cathy and Ed work with other members of the department to implement methodologies to assess successes and failures in achieving program goals for the major programs in Mathematics, Mathematics Education and Statistics that are offered by the department. In addition to completing work already begun on phase 1 of the Assessment Cycle, they planned to put particular emphasis on phases 2, 3 and 4. It can’t be over emphasized how much value there was in hearing what was going on at the twenty seven other schools and gaining insight as to where TCNJ was in the spectrum of involvement in assessment. Both Cathy and Ed came away with some good feelings but a realization that there was much more to be done.

The 2005 PREP/SAUM workshop was held between January 8^th and 10^th at Clayton College in Atlanta. The main two parts of the agenda included team progress reports on work done between March 2004 and January 2005 and a report on future plans for assessment of the major. Proposed plans were to be worked on and reported on the following year at the 2006 PREP/SAUM workshop. At the 2005 workshop Sharon Navard (Sharon) joined Cathy and Ed as a team member. Sharon was acting as a representative of the Statistics Program, Cathy represented Mathematics Education and Ed represented the major program in Mathematics. Part of the TCNJ proposal was to have Cathy, Ed and Sharon act as liaisons and ambassadors within the department to help develop and implement an assessment program for the major. Details of the content of the progress report and the report on future plans can be found in the following sections of this report.

The 2006 PREP/SAUM workshop was held on January 10^th and 11^th in San Antonio just prior to the Joint MAA/AMS Meeting. The workshop began with progress reports being given by each team. Presentations were followed by Consulting Sessions with experts on assessment, including Bernie Madison, Bill Marion, Bill Martin, Bonnie Gold, Kathy Safford, Bill Haver, Dick Jardine, Peter Ewell and Lynn Steen. The last part of the workshop involved plenary panels dealing with two topics:

Achieving Faculty Buy-In to Assessment

Learning Research and Assessment.

........................................................Page 4.....................................................

Assessment in the Department of Mathematics and Statistics Prior to January 2004

In 2003 TCNJ began going through what has affectionately become known as “transformative change.” This change involved a complete overhaul of General Education, which was assigned the new title of Liberal Learning. Faculty workload was being changed from a standard teaching load of 24 semester hours per year to 18 semester hours per year. Graduation requirements were changed from between 120 to 128 credits to 32 course units. And, academic majors were being completely redesigned so that the number of required course was lowered while at the same time the new and transformed courses were to become more intensive. By any known standards the job being undertaken was massive, intensive and all consuming.

The Department of Mathematics and Statistics was one of the first on campus to develop transformed major programs and transform specific courses within the major. It was also one of the first to offer such courses and begin teaching under the reduced faculty load. It was also a leader in developing an assessment plan. However, prior to 2004 assessment efforts were modest at best. The department had strived to use evidence-based thinking in its evaluation of teaching, learning and program quality. Formative assessment was used to monitor student learning and to adjust instructional strategies and activities. However, such assessment normally would occur at the classroom level for the purpose of improving teaching and learning. The Mathematics Education Major was without question the only program involved in formal assessment activities, but this was primarily the result of requirements imposed by NCATE Accreditation Requirements. The best word to describe assessment of the Mathematics and Statistics majors was ad hoc. Job placement, student participation in Research Experiences for Undergraduates and acceptances in graduate school served as the primary measures of program quality and the effectiveness of the academic program. This is not to say that such data does not measures quality, but as used, the known data focuses on the successes of a few rather than measure the overall quality of the program or what impact the program had on accomplishing success for the individual. Throughout academic year 2003-2004 and into academic year 2004-2005 the department for the first time began to work on a comprehensive plan for assessment.

........................................................Page 5.....................................................

Redefining Academic Majors

During Academic Year 2004-2005 “transformative change” was a major institutional effort at TCNJ. The number of courses required for graduation was reduced in number with the expectation that the new and fewer transformed courses would be more intense and demanding of students. All degree requirements and requirements for the major were completely redefined. As part of this process, goals were set for each academic major program and each course within the major, and a plan was required for evaluating whether goals were being met. For example, once goals for the major were developed, the next step was to develop a matrix that indicated how, and to what extent each course addressed those goals. Appendix A (http://www.tcnj.edu/~conjura/Assessment/AppendixA.htm)

is sample matrix for the Mathematics Major (MATA) and Appendix B (http://www.tcnj.edu/~conjura/Assessment/AppendixB.htm) is an example for Mathematics Education (MATT). This step was followed by the development a second matrix that indicated an assessment strategy for how achievement of major learning outcomes would be measured, the cycle or timeframe for making measurements, findings that resulted from the measurements, and actions taken as a result of the findings. Appendix C (http://www.tcnj.edu/~conjura/Assessment/AppendixC.htm) contains the Assessment Cycle Matrix for the MATA Major

During the fall semester of 2004 Ed Conjura worked with the Mathematics Committee, Cathy Liebars worked with the Professional Committee, and Sharon Navard worked with the Statistics Committee on the refining of their respective Course Outcomes Matrix. It should be noted this was an iterative process. For example, it was not until the end of the fall 2004 semester that the mathematics and statistics major program committees had finalized the details of the capstone experience for their programs. This added another requirement in both cases that was not reflected in existing matrices. During the spring 2005 semester these same committees began and completed work on their respective Assessment Cycle Matrix.

........................................................Page 6.....................................................

Implementing the Assessment Cycle

During academic year 2005-2006 each of the major programs chose one program goal and gathered data that would be used to measure success in achieving the goal. Statistics chose the goal that “students will be able to apply statistical procedures to real-life problems regarding data analysis.” Their method was to use an assignment in a required course on Regression Analysis. Student work on this assignment was reviewed by members of the Statistics Committee to evaluate performance. This course was also taken by students who are not Statistics majors and so all student results were reviewed. As expected, performance by statistics majors was exceptionally higher than the average for all students and they exhibited a high standard of knowledge and application. The Statistics Committee was very please with the data since it confirms that the stated goal was being achieved. As a follow up to this assignment and the results, the Statistics Committee will continue this evaluation on a periodic basis. The Mathematics Education group chose the goal that “students will be able to clearly communicate mathematical ideas and concepts.” They used a required Geometry course as the test ground. Each student was required to give an oral presentation on a topic in Geometry. A rubric used to evaluate the presentations can be found in Appendix D (http://www.tcnj.edu/~conjura/Assessment/AppendixD.htm). Two different instructors participated in the experiment. The results produced limited value if one looked solely at the evaluation scores assigned since one instructor had students give presentations on topics in Euclidean Geometry and the other chose topics in Non-Euclidean Geometry. As expected the evaluations were better for students who were allowed to present on topics from Euclidean Geometry, which they have greater knowledge of and are more comfortable with. A lesson learned is that when multiple instructors are involved, they must coordinate to guarantee that data collected is consistent and measuring the same thing. However, by simply making this assessment evaluation, the Mathematics Education Committee is guaranteeing that student will have the opportunity to communicate mathematics before they have to do so as part of their field experience, and this is good. The Mathematics Major group chose the goal that “students should be able to read, write and explain a proof.” A portfolio of student work on proofs was collected in required mathematics courses that have a significant focus on proofs. Instructors were asked to submit examples of good work on proofs and work that was not so good. This experiment proved to be of limited value since it involved samples from courses spread out over several years and the experiment was just too free in form to allow meaningful data to be collected and meaningful interpretations to me made that would accurately measure success or failure in achieving the goal. However, the portfolios of proof do serve two useful purposes. First, we now have portfolio data on proofs, and preliminary examination of the data appears to support preconceived notions of the abilities of our students to do proofs.

........................................................Page 7.....................................................

Much was learned from experiments conducted in spring 2006. Perhaps the most valued lesson was confirmation of the obvious, which is that the quality of the planning part of the cycle is directly correlated to the quality and usefulness of the data that results. Undaunted by the limited success in spring 2006, each group is currently involved in a new assessment experiment in academic year 2006-2007. The Mathematics and Statistics program committees have chosen to assess the stated program goal that “students should be able to read mathematics, write about mathematics, and explain mathematics to others.” These groups have chosen the major requirement of a Capstone Experience for the data gathering platform. During the academic year 2004-2005, each major program committee designed a required capstone experience for their respective major that was offered for the first time in the fall 2006 The Capstone Experience for mathematics and statistics is implemented by having each student chose an upper level course they take and this course acts as the basis for their capstone experience. The student is then required to write and additional research paper in this course, and give a presentation on their work. Statistics has chosen the student run seminar series as the forum for the presentations and the mathematics group has made the presentations an added course requirement with each student in the course required to attend a fixed percentage of all of the talks. Also, instructors for these Capstone Experiences are required to maintain a file of the student papers and grades assigned. They must write an end-of-semester report on their experiences and meet with the appropriate program committee and future Capstone Experience Instructors to report their findings. The Mathematics Education program chose the goal that “students will use a variety of instructional strategies in their teaching” and they are using student teaching as the basis for the capstone experience, and the platform for their data gathering. They will use a rubric that was jointly developed by members of the School of Education to evaluate student teaching performance. Appendix E (http://www.tcnj.edu/~conjura/Assessment/AppendixE.htm) contains the program specific part of this rubric.

........................................................Page 8.....................................................

Feedback from Students

A goal set by Ed in March of 2004 was to improve feedback from students concerning their perceptions of the quality of their educational experience at TCNJ and in particular, within the Department of Mathematics and Statistics. It was decided to conduct a formal survey of graduates. In May 2004 each graduate received a paper version of a Graduate Survey Form and was asked to use a stamped self-addressed envelope to return it in. There was a disappointing 20% return rate on the first survey. A lesson quickly learned was to get information from students before they leave and so a more proactive approach was taken. First, an on-line version of the survey was developed. For June 2005 graduates, an exit interview was conducted by Ed with each graduate in order to make the survey process more personal and to assure them that the purpose for the survey was to make our programs better. During that interview, students were asked to complete the online form of the survey, but on their own time. This approach resulted in a 72% return rate. For the 2006 graduates an even more proactive approach was taken. A group meeting was scheduled where pizza was served. The same explanations that Ed gave individually the year before was given to the group in attendance. The students were than asked to complete the survey before they left. 85% of the graduates attended the meeting and all in attendance completed the survey. You can see the survey and some results of the survey at http://www.tcnj.edu/~conjura/SeniorSurvey/SeniorInfo.html.

A sample survey can be found in Appendix F (http://www.tcnj.edu/~conjura/Assessment/AppendixF.htm). Currently under development is an online version of a survey intended for alumni. Each program committee has been asked to come up with questions they would like to ask graduates who have been away from the college for three or more years. The questions should be ones that can be used to gain information about the quality of the program and provide data that can be used to evaluate achievement of program goals. During spring 2007 this survey will become available online.

........................................................Page 9.....................................................

Assessment of the Quantitative Reasoning Component of Liberal Learning

Assessment of the Quantitative Reasoning component of the Liberal Learning Curriculum has evolved slowly. During academic year 2004-2005 three calculus courses were selected (Mat125: Calculus for Business and Social Sciences, Mat127: Calculus A and Mat128: Calculus B) and a group of faculty used a rubric to evaluate whether the final exams covered all major topics in the course syllabus and whether they addressed stated course objectives. The evaluation was done on an anonymous basis and in order to put a positive spin on the assessment being done, examples of those exams that were judged to be the best were circulated to all instructors who were assigned to teach those courses during subsequent semesters. This project has been so successful that it is now being expanded to include more courses. During academic year 2005-2006 the Mathematics Education group used a rubric to evaluate the final exam for Mat105: Mathematical Structures and Algorithms for Educators, which is one of two required courses for all elementary, special education, and deaf and hard of hearing education majors who do not have mathematics as a specialty.

In fall 2006 a new college-wide committee was formed with the title of Quantitative Reasoning (QR) Advisory Committee. This committee has the following charges and contains membership from five different schools within the college. Ed is chair of this committee and assessment activities will begin late spring of 2007.

· Establishes guidelines for the submission of courses, programs, or equivalent sustained experiences meeting the QR requirement of the Liberal Learning Program.

· Reviews QR proposals and recommends them to the Liberal Learning Program Council (LLPC) or returns them to submitters with reasons for rejections and/or recommendations for improvement.

· Works with departments and programs to refine and perfect proposals.

· Reviews the goals and outcomes for QR, recommending to LLPC changes it sees fit.

· Creates guidelines for assessing existing and future QR courses, programs, or equivalent sustained experiences.

........................................................Page 10.....................................................

Summary and Conclusions

This report began by giving much credit for any success in assessment to the training and experiences that Cathy, Ed and Sharon received through participation in the PREP/SAUM Workshops held in 2004, 2005 and 2006. It is fitting that the report should end by repeating that testimonial. In 2004 at High Point, North Carolina the team learned several valuable lessons that have served as the keystone of their work with assessment. First, you should think small, especially in the beginning. Second, you must remember that you should be doing assessment to learn something about yourself that will help you to improve what you do. Third, you can’t do it all on your own and if you want people to buy into doing assessment, you need to somehow show them that it is something that is useful. The team has tried to live by these rules, and sometimes they actually have done so. They have had successes and failures. They have also learned from their mistakes and try not to repeat them too often. In all cases they have enjoyed what they are doing, which is as it should be. Along the way they have even gotten others to get involved, which is perhaps their greatest success. Finally, Cathy and Ed hope that perhaps others can benefit and learn from their personal experiences and the work done by them and others in the Mathematics and Statistics Department at TCNJ. Additional information, reports and case studies on assessment can also be found at these references .