Achiever Spring 1999 - UMUC Alumni Magazine

When our parents asked what we learned in kindergarten each day, the answers were usually easy. "I learned to make paper snowflakes," or, "I learned to write my name," we would answer. But as we got older, that question, like so many others, became more nuanced and the answers more abstract. Does performance in the classroom correlate with performance in the workplace? What kind of proficiency can a college course offer that on-the-job training cannot? What, ultimately, do we mean when we refer to learning? These and other related questions are the focus of an institution-wide initiative on assessing learning outcomes (ALO) spearheaded by UMUC’s Institute for Research and Assessment in Higher Education (IRAHE).

"ALO is not just restricted to content knowledge," says Joan Krejci, director of IRAHE. "It’s not just a matter of looking at a final exam, because sometimes exams measure the whole compass of a course and sometimes they don’t. And it’s certainly not a matter of looking at final grades, because grades often involve a number of factors besides what one has learned." Instead, ALO implements new strategies to measure not just skills and knowledge, but general attitudes, dispositions, and goals; to assess an individual’s abilities when entering a course or program of study, and upon completing it; and to determine whether a course or program of study teaches what it is intended to teach—and whether those goals and outcomes are in harmony with the demands of the workplace.

"That," Krejci says, "involves gathering opinions from the main stakeholders in the various programs—current students, alumni, and employers—so that we can be sure that we’re aiming for the right outcomes. Just a few days ago, for example, we invited in three focus groups made up of people from the business community, and a facilitator asked them, ‘As our students complete a graduate degree program in management, what should they be able to do for your company?’ The kind of feedback that produces is just tremendously valuable."

The ALO initiative sprang from a benchmarking study conducted by UMUC and the American Productivity and Quality Center (APQC). The study focused on strategies for evaluating the learning process. After in-depth analysis of a number of organizations, both in and outside of higher education, UMUC and APQC produced a detailed report on the effectiveness of those strategies.

"There were a number of people in leadership positions [at UMUC] who were involved in that study. We felt that it was important, after the study was completed, to bring it to the faculty members and program directors to continue the momentum of the study and try to apply some of the principles that we had learned about assessing learning outcomes," Krejci said.

With that in mind, teams of faculty members and program directors from both the Graduate School of Management & Technology and Undergraduate Programs have been working to develop what Krejci calls "assessment pilots." In Undergraduate Programs, one pilot focuses on developing assessment strategies for BMGT 364 Management and Organization Theory, while the other seeks to develop similar strategies for a series of accounting courses. In the Graduate School, one pilot reviews the intended learning outcomes of management programs in light of the needs of stakeholders—students, alumni, and employers. The other focuses on evaluating a student’s preparedness to enter a graduate-level course or program. It is, by all accounts, a daunting task.

"We go through syllabi for each class," says Joyce Shirazi, associate director of technology and engineering management programs, "and talk with professors who teach the classes, trying to identify the essential skills and knowledge for students entering the class. It’s not something that’s very easy to do. Even in a technical field, a professor might say, ‘Students should have a background in art history.’ When I ask why, he or she might say, ‘Well, they need to know how to think out of the box.’"

"So that goes in," Shirazi says. "It all gets considered. And that’s just one question, for one class, from one professor. It’s a lot of detail and a lot of work." To a layperson, all that work might seem like an exercise in reinventing the wheel. After all, don’t tests like the GED, SAT, and GRE serve to assess learning outcomes? The differences, says Kevin Michel, academic director of accounting in Undergraduate Programs, are subtle but important, and exist on several levels. "Typical achievement tests evaluate knowledge," says Michel. "The assessment instruments that we are working to develop will evaluate not only knowledge, but skills, abilities, and whether individuals can apply the acquired knowledge to specific situations."

ALO, says Michel, is a tool. It is not an obstacle students must overcome, or a standard they must meet, before they can be admitted to—or graduate from—a program.

Even in a technical field, a professor might say, ‘Students should have a background in art history.’ When asked why, he or she might say, ‘Well, they need to know how to think out of the box.’

"The focus of ALO is not just a judgment at one point in time. We are trying to measure improvement over a period of time, so that an individual or institution or employer can assess someone’s capabilities and, from that assessment, develop a plan to improve those capabilities."

Rich Neidig, acting associate dean for academic affairs in the Graduate School, points out that effective learning-assessment strategies can be of immediate benefit to students, saving them both time and money.

"We all know that education, especially for our students, is expensive," Neidig says, "not necessarily in terms of dollars, although that is a bite too, but in terms of the commitment they have to make as working adults. What we’re trying to do is see that they get the most possible content from every hour they spend in the classroom or online, and one way to do that is to offer them the chance to make certain they are optimally prepared for the content they will encounter."