Student Peer Evaluations Using the Analytic Hierarchy Process Method

Les Frair
Foundation Coalition
Department of Industrial Engineering
P.O. Box 870288
University of Alabama Tuscaloosa, AL 35487

Abstract:

Employment of the Analytic Hierarchy Process (AHP) to assess the contributions of engineering student team members is described. The students perform this assessment as well as the instructor. The assessments are correlated with the individual team role-assignments to making a final determine of the contribution of the individual to the team effort. Using this evaluation technique appears to provide candid student peer input for evaluations.
AHP can be characterized as a multi-criteria decision technique in which qualitative factors are of prime of importance. A model of the problem (teaming contribution) is developed using a hierarchical representation. At the top of the hierarchy is the overall goal or prime objective one is seeking to fulfill. The succeeding lower levels then represent the progressive decomposition of the problem. The knowledgeable parties (e.g. individual team members) complete a pair-wise comparison of all entries in each level relative to each of the entries in the next higher level of the hierarchy. The composition of these judgments fixes the relative priority of the entities at the lowest level (e.g. individual team members) relative to achieving the top-most objective.
A description of AHP for teams within a production engineering class is described. First the lack of success with traditional student questionnaires to asses team performance is described followed by a description of the what appears to be more meaningful results when AHP is used. Finally, several complicating factors associated with this experiment, some tentative conclusions and a recommendation for continued investigation of the use of AHP for student evaluations are described.

The Analytic Hierarchy Process (AHP)

The technique is especially suited for application to project evaluation in which qualitative factors dominate. It can be characterized as a multi-criteria decision technique that can combine qualitative and quantitative factors in the overall evaluation of alternatives. This section provides an introduction to AHP with an emphasis on the presentation of the general methodology. No attempt is made to prove the mathematical foundations for AHP, rather the interested reader is referred to [1] and [2].

Step 1.: Develop the hierarchical representation of the problem. At the top of the hierarchy is the overall objective and the decision alternatives are at the bottom. Between the top and bottom levels are the relevant attributes of the decision problem, such as selection criteria and the various ``actors'' (individuals, agencies and organizations), if appropriate, that provides significant input on the decision process. The number of levels in the hierarchy depends on the complexity of the problem and the analyst/decision maker model of the problem hierarchy.
Step 2.: Generate relational data for comparing the alternatives. This requires the analyst (decision maker) to make pairwise comparisons of elements at each level relative to each activity at the next higher level in the hierarchy. In the system example, the importance of each criterion relative to system acceptance needs to be established.

In AHP a relational scale of real numbers from 1 to 9 is used to systematically assign preferences. When comparing two attributes (or alternatives) A and B, with respect to U in a higher level, the following numerical relational scale is used: