Vol. 3, No. 1, Spring 2008
| Looking for Beta Testers for Online Assurance of Learning Assessment As a part of the ongoing efforts to improve the Marketplace simulation, we have been working to develop an Assurance of Learning Assessment tool that will help instructors gauge students’ progress and learning during the simulation. We have reached the beta-testing phase of development and are looking for several schools that would be willing to administer the assessment to their students during the spring term. The beta-version for the assessment will be available starting March 17th and it is built to be administered during Q6 (preferably just before students wrap-up Q6 decisions). The beta-version has been developed to work with the following games:
Instructors who use any of the above games and are willing to serve as beta-sites for the Assurance of Learning Assessment, please contact Leff Bonney at the University of Tennessee as soon as possible. Leff can be reached by telephone at +1 865.974.4179 or by email. What is the Online Assurance of Learning Assessment? Three of the primary goals of the Marketplace simulation are to help students 1) develop an integrative perspective on business decision making; 2) learn how to use analytical tools and thought processes to make well informed business decisions; and 3) determine how to align the firm’s strategy and tactics to market conditions. However, Marketplace has had no formal procedure for evaluating the students’ ability to achieve these goals. Nor has there been a way to measure the students’ ability to 1) perceive important elements in the environment and 2) connect the dots between changes in the market and the necessary changes in strategy and tactics. Of course, teams that achieve exceptional performance on the balanced score card most likely exhibit this ability, but what about the other teams in the exercise? Since it is unclear if students are developing an integrative mindset and effectively using the tools of management with regard to their strategy and tactics, the developers of Marketplace are attempting to develop a new assessment tool to determine what is being learned. This new tool, the Assurance of Learning Assessment (AOLA), is currently in the developmental stage. Specific measurement goals of the tool are to 1) pinpoint what students know about their business in terms of market, marketing, manufacturing, financial, and accounting information; 2) determine the degree to which the students understand the implications of this knowledge; and 3) determine if they are pursuing the appropriate strategy given their understanding of the business and market. The AOLA is a battery of questions which are divided into five main sections. Each section is comprised of questions associated with each functional area within the firm as well as questions about the market in general. The first section assesses the students’ understanding of their team’s position relative to its competitors on various business metrics that are drawn from each of the business functions. The second section assesses the students’ ability to predict how well their firm will perform on several of these metrics in the upcoming quarter. Section three evaluates the students’ knowledge of the market in general. The fourth section assesses the students’ ability to predict what the competitive landscape will look like in the upcoming quarter. Finally, the fifth section assesses the students’ ability to link their knowledge about the market to appropriate strategies and tactics. After the quarter is wrapped up and processed, students will receive a scoring sheet which shows each question, the student’s answers, the correct answer, and a brief comment as to why the information is important to know. In addition, students will receive a diagnostic sheet indicating how well they did relative to other students in the course and how well their team scored compared to other teams. Instructors will receive a report showing how every individual in the course scored on the assessment as well as comparative statistics at the team level (see below for specific reports available). The diagnostic report can be used to coach students as they compete in the final quarters of operation. It can also be used as input on adjustments to the course design. What is required of Instructors wishing to use the online AOLA? Using the online AOLA is simple. Instructors administer the AOLA just before the wrap-up of quarter 6. The goal is to give the students ample time to study the market and operational data and make their decisions for the current quarter. We want to take our measurements when almost all of the work has been done. By administering the AOLA at the end of the quarter, the students should have spent dozens of hours studying information and making decisions. They should have a good sense of their position in the market, maybe not the specific details but a good understanding. We have already seen that the students who know how well their firm is operating and doing in the market, perform better on the balanced scorecard. It is also preferable that the AOLA be administered in a classroom situation where the students would not be allowed to review their software and obtain answers by checking specific data. Our goal is to measure what the students know, have concluded, and have acted upon regarding the market and their business. We are not trying to measure their ability to find information - that should already be an active part of their decision making. Also, we are not trying to measure their knowledge of specific data points. We are trying to determine the conclusions that they should have drawn from having studied the available information and the students’ use of the tools of management. At the designated time, instructors send students a link to the AOLA tool. This action can be executed from within the Web Instructor for Marketplace. The assessment should take students approximately 30 minutes to complete. After all of the students have finished the assessment and the decisions for that period are wrapped up and processed, instructors can release the results and the scoring sheet mentioned above. The software will automatically grade each student’s assessment and provide the diagnostic reports to the instructor. What specific diagnostics are provided by the AOLA? The AOLA is not simply a grading system. Rather, it is a feedback system. AACSB is interested in more than grades; it wants each school to help its students discover their weaknesses so that they can adjust their own study to improve their performance. Therefore, a major objective of the assessment is to give students feedback 1) on what they know and do not know about their business and 2) how their knowledge compares to others in the industry. If a student is underperforming in a functional area or in general and he/she can see how this lack of knowledge or understanding relates to the firm’s performance in the industry, then perhaps the student will be motivated to concentrate on their weaker areas. AACSB is also interested in providing feedback to the instructors and curriculum designers in order to facilitate continuous improvement. The AOLA is designed to highlight which aspects of the students’ understanding of business are well understood and which are not well understood. Where there are consistent weaknesses, the instructor may wish to add lecture or written material to deal with these them. In some cases, the discovered weaknesses may relate to what the students should have known prior to enrolling in the simulation course - in other words, prerequisites. In these cases, the reports can be used to persuade other faculty or curriculum committees to make adjustments in their curricula. In light of these objectives, we have prepared a number of metrics to be reported back to students and instructors. Students only receive information that is relevant to them, while instructors get the full range of diagnostics. We are working to have full diagnostic functionality by March 10th, however some diagnostics may not be available in time for beta testing. Below is a list the diagnostics that have been or are being developed.
Marketplace Community Newsletter, issued quarterly. www.marketplace-simulation.com |
back
to top
Previous
Article