Ade Mabogunje, Larry J. Leifer,
Center for Design Research
Stanford University, Stanford, CA 94305-2232
Raymond E. Levitt,
Center for Integrated Facility Engineering
Stanford University, Stanford, CA 94305-4020
Catherine Baudin,
Artificial Intelligence Research Branch
NASA Ames Research Center, Moffett Field, CA 94035
This paper describes an ongoing project to develop the managerial framework for such a training. We begin by adapting the simulation model of an engineering organization to a project-based design class. We hypothesize that it is possible to simulate these classes with at least the same degree of realism as current computer simulations of engineering organizations.
To illustrate the potential impact of this approach on design performance, we present preliminary result from the computer simulation study of ME210. ME210, Mechatronic Systems Design, is a graduate-level course based on Industry-sponsored projects. Students, in three-person teams, work on one project for nine months. The inputs to the simulation program are such variables as the class organizational structure, physical layout, team composition, and communication technologies. The principal output is the schedule-performance achieved by each team and the class as a whole. While we were able to prove the hypothesis, the results demonstrated the need for theories of learning processes that are specific to project-based classes.
Within the last decade, in the field of organizational modeling, there has been a shift from the procedural representation of decision making through numerical algorithms to the representation of decision making through symbol-driven search processes [1]. This shift has made it possible to build more realistic models of organizations [2]. Also within the last decade, there has been a shift in engineering schools from organizationally simple lecture-based classes to organizationally more complex hands-on project-based classes [3] [4] [5][6]. This shift and added complexity come at a time when theories of learning specific to project-based classes are relatively ad hoc and not yet well established (see [7] for example). Since project-based engineering classes are organizationally similar to manufacturing organizations or construction firms, we believe we may be able to develop theories of learning that are specific to project-based classes by relying on the relatively well established theories of decision-making that have been widely used to model and simulate complex engineering organizations. In particular, we hypothesize that it is possible to simulate project-based engineering classes with at least the same degree of realism as current computer simulations of engineering organizations.
In order to closely follow the line of development of this study, one must have a good appreciation for the meaning of three main terms introduced in the preceding section. These terms are, ``degree of realism'', ``computer simulation of engineering organizations'', and ``project-based classes''. The first of these terms will be illustrated pictorially, and the latter two will be described by using concrete examples.
The degree of realism is the most critical benchmark to judge simulations [8]. Figure 1 illustrates the triad of entities that constitutes a simulation, namely an aspect of reality, a conceptual model and a simulation model. The figure also shows the two primary sets of actions relating to these entities. The first set comprises actions used to produce simulations: observe, program, compare. The second set comprises actions used to evaluate simulations: verify, validate. We can therefore observe that the degree of realism is the result of an evaluative action and that it tells us how close a simulation model is to reality.
VDT, Virtual Design Team, is a software that performs computer simulations of engineering organizations. In general, the conceptual model in VDT seeks to explain how the duration and quality of an engineering project are affected by actor variables, task variables and organizational variables. Actor variables include elements such as the skill of the actor with respect to a particular task, her preferences for using certain communication devices during task execution and her position within the organizational hierarchy. Task variables include a description of the level of complexity of a task and the degree of uncertainty associated with the task activity. Organizational variables include such variables as the structure and communication policy.
The relationship among these variables is based on a combination of Galbraith's theory of information processing in firms [9] and heuristics for estimating the duration of tasks and quality of decision making during the execution of a project. Paraphrased, Galbraith describes knowledge work as routine work, punctuated by exceptions-situations where the information needed to complete a task exceeds the information available to the worker. He goes further to describe how the organization-nominally a hierarchy, but evolving to a matrix as needed-serves as an exception handling system with exceptions flowing along communication channels to other workers who have the information (and authorization) needed to complete the task. He explains how hierarchies can get overloaded with exceptions when uncertainty is high relative to the skill levels of the first line workers and describes alternative strategies for dealing with this information overload. The heuristics for estimating the duration of tasks and quality of decision making focuses on the process by which the first line worker handles exceptions. The worker can do this by doing nothing (default delegation), by reporting to a colleague (lateral communication) or by reporting to a supervisor (vertical communication). In communicating with the colleague or supervisor, a further choice is made with respect to the communication medium to use, a memo, a fax, a telephone, or a face-to-face meeting. These choices are constrained by the organizational structure and the communication policy within the organization. In addition to the constraints on these choices, the choices have consequences. These consequences affect other members of the organization and ultimately the total time to execute a given task.
In 1992, VDT was used to simulate a 120-person team designing a petroleum refinery [10] and in 1993, it was used to simulate a 20-person team designing parts of an offshore oil-production platform [11]. These two examples were shown to exhibit a high degree closeness to reality. In the former case for example, the simulation was used to predict the effects of: changing decision making from decentralized to centralized, and adding or removing voice mail. The initial result demonstrated that the effect of these types of changes on the organizational output were qualitatively consistent between practice, theory and the VDT simulation model [12].
ME210 is a Mechatronics Systems Design class that uses industry-sponsored engineering projects as a framework for teaching graduate students the product development process. Every year, during the fall quarter, the students are supported in forming three person teams which then bid on 12 to 18 projects submitted by a variety of companies. Over the next seven months the students propose alternative designs, investigate these alternatives, build, and test a functional prototype of the preferred design. This prototype along with a detailed report of its design specification and rationale are handed off to the client at the end of the academic year. ME210 includes several other experienced participants besides the instructor. This includes advanced graduate student teaching assistants, team coaches and Industrial liaisons (See Figure 2). The course, under various professors and with constant refinement, has been offered for over twenty years and several observers including class alumni working in industry have described the class setting and design activities as a microcosm of engineering practice in industry.
The premises of this study are (1) the activities in a project-based class are similar to the activities in a construction project and (2) VDT was successfully used to simulate two construction projects. The hypothesis is that the degree of realism between ME210-VDT and the ME210 class will be comparable to that between VDT and the Construction projects.
A standard procedure has been developed for modeling projects and organizations in VDT [13]. It requires one to specify attributes of actors such as their task qualification and their team experience. This information was obtained from the ME210 instructor and a former teaching assistant. The procedure also requires one to describe the project tasks in terms of the level of requirement complexity, the level of solution complexity, and the level of task uncertainty. The techniques of Quality Function Deployment, Functional Requirements Decomposition, and Design Structure Matrix were used to compute the values for these task parameters. The required information was obtained from previous class projects and ongoing development work on Dedal, an information retrieval tool [14]. In addition to the description of actors and tasks, the VDT model has two sets of independent variables: the communication tools and the organizational structure. Figure 3 is a black-box illustration of the variables just described.
For a detailed explanation of all the variables modeled in VDT, the reader should see reference [13]. For the software modifications that were made in order to model a multi-project, multi-team organization like the ME210 class and the values assigned to the variables modeled in ME210-VDT the reader should see reference [15]. For illustrative purposes we will describe and give examples of four of these variables: (1) level of formalization; (2) project noise; (3) task experience; and (4) level of participation.
The level of formalization characterizes an organizational culture with respect to formal meetings: how regularly they are scheduled and how strictly the attendance is mandated. For example when meetings with the instructor are confined only to office hours, the level of formalization is high. The project noise is a measure of the amount of distractions (non-project related activities) that have to be accommodated at the same time as the project. The task experience is an estimate of the students previous experience with respect to a particular task. For example, a student majoring in mechanical engineering may have a low task experience on an electrical engineering task. Finally, level of participation describes the part-time ratio of coaches, teaching assistants, industrial liaisons and instructors relative to the students who are assumed to be working full time on the project (100%participation)
After scanning the output of initial runs and reflecting on the goals of our experiment, we decided to focus on the following summary output measures: Schedule Quality, Verification Quality, Default Delegation.
Schedule quality was chosen because it reflects actual duration of the project relative to a computed duration. Because we are modeling a class which has a fixed deadline as opposed to a more standard design project which may run over-time, we had to develop an interpretation of schedule quality that fit our real problem. We assume that a lower measure of schedule quality may indicate a reduction in project quality because in reality the students must submit the final project documentation by a specified date. If simulation results indicate a low schedule quality, the project may need to be turned in before it is actually ready (the project is not yet fully completed or bugs remain to be worked out).We also recognize that in the case of low schedule quality, the students may opt to work all night, in which case the project and final documentation may be fully completed on time. In such a case, project quality may or may not suffer. Verification quality was chosen because it gives an indication of how exceptions were handled. Uncorrected exceptions are a key component determining the quality of the final design prototype. Verification failures might also indicate the quality of the design instruction process in ME210. Our final measure of interest is default delegation. We chose this measure because we believe in our case that it can be interpreted as an indicator of the quality or effectiveness of the ME210 teaching staff. It might also indicate the degree to which the teaching staff is overloaded. We believe this could be the case because most of our activities are assigned to students in the design teams and most of the exceptions are generated at this level. Therefore, most default delegations represent the number of problems students encountered which they decided to pass upward in the organization structure for decisions and which were not attended to before a certain time (approximately 1 week in our model).
Over many years of teaching ME210 the instructor has made a number of changes to the class in attempts to improve the class, or make it more in-line with his philosophy of the design experience. For example, email and network access and use have recently been encouraged, the physical layout of the lab has been changed, and the role of team coaches has been varied over the years. For our experiment, we are interested in modeling
changes to the class along two dimensions: the formal organization structure, and the physical layout in the lab. We test a 2x2 matrix represented by the four configurations shown in Figure 4.
The base-case organization structure (Figure 5a) is a dual reporting structure. In this setting and according to Galbraith's model which was
paraphrased earlier, those exceptions encountered by students in class-related activities may be directed to the teaching assistants and the instructor and those exceptions encountered in project-related activities may be directed to the coaches and the client. The experimental structure (Figure 5b) is a pure hierarchy in which the coaches are a more formal part of the class and able to address both class and project related issues. In addition, we assume this structure creates an additional load on the coaches because they are required to attend class, so that they are less available to help students outside of class (project participation reduced from 20%to 10%).
The base-case physical layout is a large lab with open work spaces for the student design teams. In this configuration student projects and progress are readily visible to all design teams. The experimental layout is one in which team work spaces are partitioned so that each team's work is not as apparent to the other teams. We expect a number of effects due to the addition of partitions, and model these as an increase in formalization, decrease in project noise, and decrease in task experience, since students no longer have the benefit of easily observing, interacting with, and learning from the experience of other groups.
Our preliminary simulations show that the combination of an open work space and a dual reporting structure resulted in the best overall schedule quality. (See Figure 7). The results also showed that a partitioned work space and the dual reporting structure resulted in the worst schedule quality and default delegation In addition, the results show that given a purely hierarchical structure, a partitioned work space resulted in a better performance. Therefore if for some reason, the instructor needed to make changes in certain class policies, for example to mandate coaches to attend class lectures, it would be advisable to simultaneously make changes in the physical layout of the class in order to keep the class performance as high as possible.
The foregoing results are evocative because the recommendations that can be deduced from them raise a number of questions. These questions range from ``Are you serious ?'' to ``You must be joking'' and finally to ``Could it possibly be true ?'' Our answer is ``no'' to all three questions, but this is a no with several ``yeses'' underneath.
To address this paradox and hence draw some conclusions for our hypothesis we must return to the issue of realism and see how we stand from three main viewpoints of realism namely validity, theory, and representation. There is not enough space to expand on any of these viewpoints, we therefore will note only the essential positions.
From a validity standpoint, the simulation is not valid since learning is not represented in anyway and this is one phenomenon of the class environment that affects an actor's skill over time and changes the rate of task execution. Furthermore, the results have not been statistically validated.
But, on closer inspection, the relationship between the results and the experiment design, i.e. best schedule performance corresponds to open work space and dual reporting structure, seems to be corroborated by empirical
data showing that communication among technical professionals in R&Dlaboratories is a significant determinant of the technical performance and productivity of R&Dproject teams [16]. This thus provides some face validity for the simulation.
From a theoretical viewpoint, the absence of an appropriate theory of learning in project based courses, severely limits our ability to interpret the results of the simulation. In other words we cannot say what will really happen and not happen without having actually tried it. However, this same uncertainty is very characteristic of project based courses and Galbraith's theory seems very applicable.
With respect to representation, we fail on realism especially since the actor's memory is not adequately represented and we do not have a way of representing the social interaction patterns that occur during project execution. Minor differences in the quality of these interactions have been linked to relatively large differences in project duration [17]. Current extensions to VDT are addressing these problems by developing new representations (agents, access structures, and structuration theories) to explain how organizations adapt to new technology introductions [18]. In contrast to the representation of actor's memory which is not realistic, the representation of tasks which is derived from widely used design tools like the quality function deployment (QFD) and PERT charts is realistic. Furthermore, it appears that as tasks become more dependent on technological devices, the proportion of task duration that can be attributed purely to human cognition is reduced. For further discussion of this, see Nass and Mason [19]. Thus for simulations such as VDT, which model man-machine organizations, we can expect an increase in the reliability of prediction for project duration.
This last point is perhaps the most powerful as regards the issue of realism: It provides a clear direction and strong motive for further development of this genre of simulation models. Technically, this further development will necessitate a closer coupling between the simulation environment and the class environment as shown in Figure 8. With regards to motivation, it should be clear by now that the complexity of project-based classes will limit the ability of instructors in the areas of curriculum planning, prediction and assessment. Therefore the externalization of the instructor's class concept in simulation models like VDT will help at the beginning of the year for planning purposes, and at the end of the year for diagnostic and assessment purposes.
Based on our hypothesis, it is obvious that we can simulate project-based classes with qualitative results similar to current computer simulations of engineering organizations. It is also obvious that the learning process in project-based classes cannot yet be realistically simulated. Students learn a multitude of skills and concepts in project based classes and we can begin to systematically study the different effects. We therefore need more precise benchmarks of learning in these types of classes and a good theoretical base. Overall, further research will be needed and the results of this study are indicative of the potential benefits we can expect in future.
We would like to thank Michelle Baron and Bart Balocki who worked on ME210-VDT during the CE251 course and to acknowledge many helpful discussions with Yan Jin, and John Chachere.
