R W Brown
Senior Lecturer
Royal Melbourne Institute of Technology University
In virtually all educational institutions, economic factors related to physical resources and limited staff numbers, preclude individual student instruction or sole use of equipment in typical laboratory and project exercises.
Small groups of students simultaneously and jointly conduct laboratory experiments or work on an assignment. There are also other reasons for making students operate in teams such as the optimistic belief that through ``learning teams'' students will jointly help each other acquire knowledge and skill.
There are however, common problems associated with teams which often result in significant numbers of students gaining very little from the team exercise. Aggressive individuals can deprive less assertive members of vital experience by dominating the operation of experiments or exercises [1]. A more common problem is that some team members can shirk their responsibilities, knowing that they will normally get the same mark as the more industrious or skilled members who do the bulk of the work. Thus team discipline can be a serious problem, with dedicated and industrious members powerless to enforce attendance at meetings or to command fair contribution of effort. The team ``passengers'' not only get a free ride, they damage the morale of the student team and diminish the outcome of the project.
The project supervisor is also faced with daunting problems with the use of teams. How can the supervisor determine individual marks? Marking of teams is easy and efficient, but the supervisor can usually only guess at the contributions of team members. In addition, the enforcing of team discipline devolves totally to the supervisor since the individual team members have no real power apart from peer group pressure that is often ineffective.
However, individual team members usually know with considerable insight, who are the real contributors on their team and who are the passengers. If team members had influence over individual marks, then not only could individual assessment be more accurate, but the team would have considerable coercive power to induce team members to participate and contribute fairly.
The idea of giving students the task of marking themselves and others is generally viewed with disfavour by experienced academics and is often prohibited by educational institutes, with good reason. Students by definition, usually do not have a mature grasp of the knowledge aimed at in an experiment or project. Therefore, they cannot correctly judge the marks that should be allocated. Thus self-marking schemes are usually unacceptable to course accrediting bodies.
However, the students can judge their relative contributions to the team effort quite accurately, even though they cannot be expected to judge the absolute value of the work. Since the absolute value of the team's work can be assessed by the supervisor or another examining body, the individual mark can be estimated by combining the supervisors mark with a relative rating supplied by the students.
Since 1991, a system that uses relative peer and self assessment by each team member, has been used in the Communication &Electronic Department and Computer Systems Department at the RMIT.
The system uses votes supplied by the team members and a spreadsheet, to calculate a weighing factor for each student. This weighing factor is multiplied by the team mark obtained from the supervisor, to obtain the mark for an individual. The weighing factor is a ratio of the average of the assessments of the individual, to the average of all assessments of all individuals.
Some students will get a higher than team mark by this system and some will get a lower than team mark. The average of all marks of all team members will still be the mark given by the supervisor. Student votes or assessments do not normally change the average team mark.
Let us consider an example. Suppose there are four team members, John, Betty, Chan and Sifnos. The supervisor has assessed their team project as worth 80%. The Autorating system is used to obtain a voting form from each team member with ratings for all team members including the voter. These votes could be in percentage units. John for example, rates himself as 87.5%, Betty as 100%, Chan as 75%and Sifnos as 62.5%. Similarly, votes from the other team members are also obtained. They are entered in a spreadsheet as shown in Table 1.
The number for each person in the Adjustment Factor column is the average of the four votes for that person, divided by the average of all votes. This is then multiplied by the Group Mark to obtain the Final Mark for each student. The average of all the final marks should be the same as the group mark. Thus some students obtain marks greater than the group mark, and some less. The only situation that will cause the team average to alter, is where the adjusted student mark exceeds 100%. In this case, the supervisor caps the mark at 100%, thus altering the overall average. This occurs rarely in practice.
The Autorating system can be exploited to automatically penalise team members who fail to vote and to compensate undermanned teams. When a team member(s) fails to vote, that person's column(s) is filled in by the supervisor with zeroes and with 25%for all voting team members. This results in a penalty of about 10%for the non-voter(s). This is not absolute but varies according to mark value and the number in the team. However, the automatic nature of the process relieves the supervisor of the difficulty of assigning penalties.
In the case of undermanned teams, columns are created for ``phantom'' team members to bring the team up to full strength. These columns are then filled in by the supervisor with zero for all members, and the rows corresponding to the phantom members are also filled with zeroes. The votes of the actual team members are entered in the normal way. The result is a compensated calculation of the final mark for the real team members. The outcome should be carefully checked and weighed by the supervisor.
Several caveats should be observed. The cooperation of the students is essential to the success of the system. To engender trust, the votes must be kept absolutely confidential. This precludes use of the system in two-person teams, as it is possible for one to derive the other person's vote. However, if the supervisor makes known his/her intention or prerogative of also entering a vote, the problem disappears.
The use of descriptive rather than numerical assessment has also been found to be essential to the effectiveness of the system. Although these descriptive assessment levels are later converted to numerical equivalents by the supervisor, it has been found that students tend to give very high percentage marks to each other if the voting system is numerical. However, they seem to be much more comfortable with descriptive ratings, such as ``Excellent, Very Good, etc.''. The numerical correspondence of subjective rating levels should not be advertised.
Finally, the students must be enjoined to give realistic ratings. Some students find the act of ``judging'' their peers distasteful and tend to give ``Excellent'' to everybody. If a student vote is quite different from the team performance, then the vote should be rejected and handed back to the voter for re-submission. Students are happier with the voting requirement if it is pointed out to them, that assessing people may be distasteful, but it has to be done by managers and supervisors in the real world every day in order to determine pay raise and promotions for example.
Numerical ratings were found to be unsatisfactory because of an apparent psychological compulsion of students to give high numerical marks. A descriptive system of nine discrete levels spaced between 100%and 0%in 12.5%intervals, has been found to work well. The descriptors are:
Although it might be thought that such quantised levels would give too rough a grading, in practice, variation between team member votes, effectively gives a continuum of marks between discrete levels.
The system has been in routine use since 1991 in the undergraduate courses in the Departments of Electronic &Communications, and Computer Systems. It has been found to have a salutary effect on team work. Because students know that they will be judged by their peers they more conscientiously attend meetings and participate. Complaints from students about non-performing individual team members are now rare. Surprisingly, when used on groups of two, students often simultaneously rated their partner higher than themselves. This attests to the honour practiced by most students.
The system has relieved the supervisor of the almost impossible task of accurately assessing individual student performance in teams.
In practice, the adjustments are not extreme. In an analysis of the correction factors obtained from the votes of 100 second year students, the mean was 1.02, the standard deviation was 0.14 and the range was 1.31. If the factors of two students are eliminated, the three figures become respectively, 1.03, 0.12 and 0.73.
The two students had correction factors of 0.22 and 0.68. It has been our experience that students very rarely give zero or very low marks. Thus the low ratings for these two students indicated a serious problem, later borne out by the fact that both dropped out of their course.
The non-unity mean indicates an error in the data supplied by the tutors in this example.
Since 1991, when the system first went into routine use in the first and second year of the Design and Innovation subject strand and in some single semester subject units, only one complaint has been received from students regarding marking. In one student survey, strong support of the system was recorded, with the attendant comment of a student expressing delight that hard workers in a team could be rewarded through the system whilst non-workers could be penalised. Supervisors have found students to be surprisingly honest when rating themselves and their peers. However, in a survey [2] of second year students, conducted by student representatives, around 30%of students indicated that they did not like the Autorating system.
An automatic, simply implemented and operated system for determining individual team member marks from a team-mark, has proven itself reliable and effective. In addition to simplifying the assessment tasks of supervisors, the system also facilitates effective team work and devolves some power to student team members to reward deserving individuals in teams. It also makes it possible for good students to achieve satisfactory results even though the team assessment may be less than satisfactory. The Autorating system is now being used in other subjects at the RMIT and is being considered for adoption by academics at some other universities.
