Applicable Conference Topics: Evaluating teaching methods, Computer science education research, Impact of technology on the curriculum, Innovative instructional methods, Electronic forms of class support, Instructional technology, Emerging technologies.
Deborah Knox
Department of Computer Science
The College
of New Jersey
Box 7718
Ewing, NJ 08628-0718
Telephone:
609-771-2193
Fax: 609-637-5183
knox@tcnj.edu
Nell Dale
Department of Computer Sciences
The University
of Texas at Austin
Taylor Hall 2.124
Austin, TX 78712-1188
Sally Fincher
Computing Laboratory
University of Kent at
Canterbury
Canterbury, Kent CT2 7NF, UK
Dialing code: 01227
(UK) or +44 1227 (International)
1. Abstract
When an instructor adopts teaching materials, he/she wants some measure of confidence that the resource is effective, correct, and robust. The measurement of the quality of a resource is an open problem. It is our thesis that the traditional approach to peer review is not sufficient to insure the quality of teaching tools. This working group will focus on the evaluation process by applying a new review model to teaching resources. The working group will refine the model, and will work towards the development of reviewer training materials.
The new model of review will be applied (at least) to the resources of the Computer Science Teaching Center, which is a digital library for computer science educators. The development of the CSTC is supported by the National Science Foundation and by the ACM Education Board. Our work will also be made available to other initiatives of this kind - the JLearn repository (http://jlearn.mit.edu), the Java Co-odination Teaching Group at Monash Univeristy, Australia (http://www.sd.monash.edu.au/jtcg/)and those working within the Educational Object Economy model (http://www.eoe.org).
2. Problem Statement
One focus of the Computer Science Teaching Center, CSTC, is on increasing the availability of materials to enhance the teaching and learning of computer science. This digital library (see Section 3 below) is being designed to support the access of quality teaching materials, including peer reviewed materials. The question of what makes a laboratory project "good" or what makes a visualization demonstration "worthy" of class or lab time is an underlying research question. The investigation of the single aspect of peer review of teaching materials is the aspect of the work to be conducted by this working group.
A fundamental question is how any digital resources not developed "in house" enhances the learning experience of our students. However, before computer science education researchers can propose studies to answer this question, we must insure that the materials they are testing are of a consistent quality. The first phase of this open question is to validate the quality of the material, i.e., to provide a level of confidence that the material is sound and well-founded for the topic.
It is important to insure the quality of the materials available for a number of reasons:
As professionals, we accept a variety of measures of quality. These measures include the peer review of written material to be published in journals or presented at conferences, and established criteria for accreditation of programs of study or institutions.
The evaluation of teaching materials is an open research question. In the area of computer science education, we are accustomed to reviewing papers describing teaching methods or projects, e.g., the SIGCSE Technical Symposium, but in general do not have a resource for refereeing teaching materials. A panel discussion at the 1998 SIGCSE Technical Symposium highlighted a number of printed journals that exist for publishing papers pertaining to computer science education, but these journals have not traditionally included actual materials (labs, tools, techniques, visualizations). The CSTC will provide the resources, however, we need to establish the methodology of review of teaching materials.
3. The CSTC Project
The Computer Science Teaching Center is an internet-based repository of peer reviewed reusable teaching resources for computer science educators. Its development is being supported by the NSF and the ACM Education Board. It builds upon existing efforts, including the SIGCSE Computing Laboratory Repository and the Visualization Resource Center [Knox97, Grissom97]. The project answers the need for a digital library to support undergraduate education in the sciences, as presented in recent workshop reports [NRC98, NSF98]. The CSTC is similar to a vision proposed by the ACM Strategic Directions in Computing Research Workshop [Tucker96].
Outcomes of the CSTC project include a framework for developing and distributing peer reviewed reusable teaching resources. The first phase of development, scheduled to be completed in 1998, supports a repository of tools and techniques to use visualizations in the classroom, computing laboratory materials, and NSF Education Innovation (EI) project results of courseware and tools. In the future, this framework can be used for a variety of teaching resources such as curriculum development, test banks, lecture notes, and syllabi.
4. Working Group Goals
We want to work with a group of people to insure that the standards set by the reviewing model are appropriate and consistent. While a few people are able to develop and implement the digital library foundation of the CSTC, it requires the input of a larger group of people, with differing professional and institutional perspectives, to establish an overall methodology for the review of teaching materials.
The working group will consider:
This working group builds upon the work of the 1998 Dublin working group, who started collecting materials on assessment of teaching materials (http://www.tcnj.edu/~cstc) and made recommendations to utilize an Editorial Board and a formal review process [Grissom98, ACM].
We have led initial efforts in the evaluation of laboratory materials. At the 1997 ITiCSE Conference, a working group convened to discuss the peer review of laboratory materials [Joyce97]. This group categorized submissions to the predecessor of the CSTC and identified qualities of a good lab, e.g., portability, completeness, outstanding content, successfully class-tested and subsequent revision prior to review, stimulates learning, stimulates student interest in the topic, and flexibility. These were features recommended for identification during a peer review process. This initial attempt to identify qualities of good lab materials is only a beginning to the process of insuring quality resources. A more formal approach needs to be established.
Prior to the working group interactions, we will be completing a review of the current models of review. These include traditional approaches as well as alternative models, such as those proposed for portfolio assessment. In our work thus far, we have identified a number of models to examine, including:
These approaches will be examined in light of the materials to be reviewed (lab exercises or projects), and one or more suitable models will be outlined. These will be prepared during Winter/Spring 1999 before the working group starts its efforts together. The working group will help evaluate the new review process by application to actual materials.
It is anticipated that the materials prepared by the working group will be utilized by the editors of the CSTC, and modified as needed when applied to other types of resources. Future groups of reviewers will be monitored to insure consistency.
We foresee establishing a long-term working relationship with the working group members as the CSTC provides services to all CS educators. Working group members may choose to continue their involvement in the CSTC project as reviewers and submitters of materials. We anticipate that some working group members may become more closely involved in expanding the Resource Centers beyond the initial three areas.
5. Working Group Chairs and Qualifications
Dr. Deborah Knox, Dr. Nell Dale, and Dr. Sally Fincher are most willing to serve as co-chairs of a working group on the validation of the quality of teaching materials. Based on our experience, it is beneficial to have more than one person responsible for guiding the group and insuring that the report is generated in a timely fashion. Deborah Knox will serve as the primary contact.
Deborah Knox has been a computer science faculty member since 1987. She has developed and used laboratories in support of the computer science curriculum for over ten years. She has co-led three previous working groups at the SIGCSE/SIGCUE ITiCSE Conference. She developed the prototype SIGCSE Computing Laboratory Repository. She has maintained active involvement at the SIGCSE Symposium and at ITiCSE with a variety of presentations. She is an invited feature columnist in the quarterly SIGCSE Bulletin. She was an invited participant at the 1997 National Research Council workshop on "Developing a Digital National Library for Undergraduate Science, Mathematics, Engineering,and Technology Education" (www.nap.edu/readingroom/enter2.cgi?ED.html) and also at the 1998 NSF SMETE Library Workshop. (http://www.dlib.org/smete/public/report.html)
Nell Dale has been on modified service with the Computer Sciences Department at the University of Texas at Austin since her retirement in 1995. She teaches on alternate semesters, including a semester at Uppsala University in 1997. She has been actively involved in generating laboratory materials for ten years, having published her first laboratory course in 1990. Determining an objective way to judge the quality of such materials is one of her top priorities.
Sally Fincher works within the Computing Laboratory at the University of Kent at Canterbury, UK. She has run the Computer Science Discipline Network (http://www.cs.ukc.ac.uk/national/CSDN) for the last four years and is also manager for Project EPCoS (Effective Projectwork in Computer Science - accessible as a link from the CSDN site, above) which is involved in preparation of materials for transfer from one context (or institution) to another.
6. Other Potential Participants
Previous working group members who have worked on any of the three previous working groups co-led by Knox and people who have volunteered to serve as reviewers of the CSTC materials are likely to be interested in participating in this proposed working group activity. The co-chairs will send targeted invitations to these groups of people.
7. References
[ACM]
ACM Newsletter Editor Manual: Summarizing the Differences
Between Refereeing, Formal Reviewing, and Reviewing.
[ACM98]
ACM Journal of Experimental Algorithmics, http://www.jea.acm.org/.
[Chapman98]
Chapman, Gail, private correspondence, April 14,
1998.
[Grissom97]
S. Grissom, "The Visualization Resource Center
for Computer Science Educators," http://www.uis.edu/~grissom/VRC.
[Grissom98]
S. Grissom, D. Knox (joint chairs), E. Copperman,
W. Dann, M. Goldweber, J. Hartman, M. Kuittinen, D. Mutchler,
N. Parlante, "Developing a Digital Library of Computer Science
Teaching Resources, Report of the ITiCSE'98/ACTC'98 Working Group
on
The Online Computer Science Teaching Center," ACM ITiCSE 98, Dublin, Ireland, August 1998, SIGCSE Bulletin, December 1998.
[ETS93]
The 1992 Advanced Placement Examinations in Computer Science
and Their Grading" Copyright 1993 The College Entrance Examination Board.
[JEA98]
"Reviewing JEA Submissions," http://www.jea.acm.org/guidelines_eval.html.
[Joyce97]
D. Joyce, D. Knox (joint chairs), J. Gerhardt-Powals, E. Koffman, W. Kreuzer, C. Laxer, K. Loose, E. Sutinen, A. Whitehurst "Developing Laboratories for the SIGCSE Computing Laboratory Repository: Guidelines, Recommendations, and Sample Labs Report of the Working Group on Designing Laboratory Materials for Computing Courses," ACM ITiCSE 97, Uppsala, Sweden, June 1997, SIGCSE Special Issue 1997, pp. 1-12.
[Knox97a]
D. Knox, "SIGCSE Computing Laboratory Repository,"
[NRC98]
National Research Council, "Developing a Digital National Library for Undergraduate Science, Mathematics, Engineering, and Technology Education," National Academy Press, 1998, http://www.nap.edu/readingroom/enter2.cgi?ED.html.
[NSF98]
National Science Foundation, "Report of the SMETE Library Workshop," William Y. Arms, Steering Committee Chair, http://www.dlib.org/smete/public/report.html.
[Tucker96]
A. Tucker (ed.), "Strategic Directions in Computer Science Education," ACM Computing Surveys, 28(4), December, 1996.