Looking Forward by Looking Back : Determining the Value of External Program Reviews

As Canadian universities are pressured to exhibit a range of organizational accountabilities, their various academic, administrative, and support units are required to adopt more integrated and inclusive approaches to assessment and planning. Within the purview of university continuing education units, individual program areas are expected to not only demonstrate their effectiveness and efficiency in response to a multiplicity of pressures, but also to display some form of quantifiable improvement. At the same time, limited resources compel continuing education as a whole to find ways to show evidence of and display their organizational achievements and outcomes. In 2006, Continuing Studies at Simon Fraser University (SFU) began modifying their unit-wide and external review processes for individual programs to bring them more in-line with academic faculties and departments, which are subject to external reviews on a regular basis. This article explores SFU’s approach to external reviews through a case study of one particular program review, Résumé Alors que les universités canadiennes reçoivent de la pression pour démontrer une gamme de responsabilités organisationnelles, leurs diverses unités académiques, administratives et d’appui doivent adopter de nouvelles approches plus intégrées et inclusives à l’évaluation et à la planification. Du ressort des unités d’éducation permanente universitaire, les secteurs individuels des programmes doivent non seulement démontrer leur efficacité et efficience en réaction à de multiples pressions, mais aussi faire preuve d’une forme quelconque d’amélioration quantifiable. En même temps, les ressources limitées poussent les secteurs d’éducation permanente dans leur ensemble à trouver des façons de démontrer et exposer leurs accomplissements et leurs résultats. En 2006, le département d’éducation permanente de l’université Simon Fraser (USF) a commencé à modifier son processus d’examen externe et à l’intérieur de l’unité pour les programmes individuels pour être davantage en ligne avec les facultés et départements académiques, qui sont assujettis à

exhibit a range of organizational accountabilities, their various academic, administrative, and support units are required to adopt more integrated and inclusive approaches to assessment and planning.Within the purview of university continuing education units, individual program areas are expected to not only demonstrate their effectiveness and efficiency in response to a multiplicity of pressures, but also to display some form of quantifiable improvement.At the same time, limited resources compel continuing education as a whole to find ways to show evidence of and display their organizational achievements and outcomes.
In 2006, Continuing Studies at Simon Fraser University (SFU) began modifying their unit-wide and external review processes for individual programs to bring them more in-line with academic faculties and departments, which are subject to external reviews on a regular basis.This article explores SFU's approach to external reviews through a case study of one particular program review,

Introduction
In the beginning, God created the heaven and the earth . . ."Behold," God said, "It is good." . . .And on the seventh day God rested from all His work.His archangel came then unto Him asking, "God, how do you know what you have created is 'very good'? What are your criteria?On what data do you base your judgment?Just exactly what results were you expecting to attain?And aren't you a little close to the situation to make a fair and unbiased evaluation?" God thought about these questions all that day and His rest was greatly disturbed.On the eighth day God said, "Lucifer, go to hell."Thus was evaluation born in a blaze of glory. . . .From Halcolm's "The Real Story of Paradise Lost" (Patton, 1997, p. 2) As Canadian universities feel the strain to demonstrate greater fiscal and organizational accountability, a more integrated approach to strategic planning and assessment is required.Specifically, university continuing education units and individual program areas are expected to demonstrate (and usually increase) their effectiveness and efficiency in response to a variety of internal and external pressures.
Yet, additional resources rarely accompany these heightened expectations, so self-funded ways to exhibit organizational achievements and outcomes are needed.Only a limited number of options are available: ignore or downplay the expectations, potentially sacrifice programs or narrow the scope of activities, or consider new or alternative approaches often drawn from corporate or other sectors outside higher education.Since each option may threaten to compromise the breadth and quality of the unit's programs and approach, there is a need to periodically review the unit's various activities in a systematic manner.
For university continuing education units, the challenges are often compounded by their somewhat peripheral role within university organizational architecture, their need to be responsive to students, external clients and groups, internal university structures, and the expectation that they will, in general, fund their own activities.An added complication in large units is that they are often an amalgamation of various academic, entrepreneurial, and administrative subunits with their own distinctive approaches.The various faculty, programming, administrative staff, and advisory groups in these large university continuing education units often have differing roles and responsibilities with markedly different outlooks on aspects of their work.Further, extension students often come from diverse sectors of society and hold different perspectives on the role of universities within society and the place of education in their own lives.
As a consequence, programs catering to these various groups often evolve their own distinctive cultures and approaches that emphasize the value and achievements of their own members while downplaying contributions of other groups.The potential lack of understanding and mutual respect can undermine effective collaboration, waste scarce resources, diminish noting the key findings and decisions made as a result of the reviewer's recommendations.des examens externes sur une base régulière.Cet article explore l'approche de l'USF aux examens externes via une étude de cas d'un examen de programme particulier, en notant les principaux résultats et les décisions prises suite aux recommandations de l'examinateur.program effectiveness, and undermine reputations within the institution and among its various constituencies.Developing shared understandings of common challenges that confront university continuing education in general, and each program or unit in particular, is an important step to strengthening unit and program identity and promoting more effective collaborations with various constituencies.Addressing such issues in a deliberate and inclusive way can be regarded as a form of community engagement that seeks to encompass a wide variety of stakeholders in reviewing and assessing program worth.Although community outreach and engagement is often considered an essential aspect of university practice, and university continuing education practitioners in particular have developed a wide range of experience and expertise, it is surprising how rarely universities use such perspectives in their own reviews and analyses.
The choice of whether to use an internal or external evaluator for such a review can be a weighty one.While both types may appear to have biases (Love, 1991), external evaluators are typically seen as impartial and objective.Conley-Tyler (2005, p. 8) proposes that this is especially the case where internal evaluators might be seen as possessing political motivations or biases, and that "there remains a risk that internal evaluators will not be seen as objective and reliable as external evaluators."Alternatively, Patton (1997, p. 138) suggests that this fundamental issue of internal versus external could also be called the "in-house" versus "outhouse" issue, where evaluation knowledge and commitment may stay "in-house" with an internal evaluator while an external evaluator may cause programmatic difficulties due to lack of organizational knowledge and norms-ending up with "outhouse" work.
However, Patton and others (e.g.Mathison, 1994;Shapiro & Blackwell, 1987) also point out that these two categories are not mutually exclusive, and that there can be a productive partnership where external evaluators provide the "validity and fairness of the evaluation process while the people internal to the program collect and/or analyze the evaluation data" (1997, p. 142).
In this article we discuss Simon Fraser University's (SFU) approach to conducting reviews of its various university continuing education program areas in a way that enhances the involvement of others-both internal and external-while addressing some of the cultural and organizational barriers.A general introduction to SFU's overall approach is provided and then explored in some detail through a case study of one particular program review.The discussion of process, results, recommendations, and actions taken may benefit other continuing education programmers considering the value of program reviews for their own program areas.

The Simon Fraser University Approach
Like most academic units in Canadian universities, Continuing Studies at SFU is subject to periodic external review.These reviews take several months and consist of three elements.First, the preparation of a detailed self-study document: a description and overview of the unit's mission, structure, personnel, programs and services, and key constituencies; the contexts in which it works; its major competitors; and, a self-analysis of its strengths, weaknesses, opportunities and threats, and the key issues and challenges that it faces.Second, a small team of professional experts-usually drawn from corresponding units in other universities-visits for a few days and interviews key personnel both within and external to the unit.This team prepares a final report with some recommendations for further consideration that is then-the third elementsent first to the unit for comment and then opened up to broader discussion throughout the university and subsequent decisions about the report's recommendations.
This approach is designed to review the unit as a whole and does not focus in depth on its constituent subunits or program areas.It provides comparatively little help for the university to Looking Forward by Looking Back Canadian Journal of University Continuing Education / Vol.36, No. 2 fall 2010 Revue Canadienne de L'Éducation Permanente Universitaire / Vol.36, N o 2 automne 2010 http://ejournals.library.ualberta.ca/index.php/cjuce-rcepuassess the particular strengths and approaches of the smaller units.For SFU, this meant that the 20-plus distinct program areas that comprise Continuing Studies were never reviewed on any systematic basis.This naturally led to some questions regarding the independence, relevance, and usefulness of such reviews.In 2006, Continuing Studies began modifying the unit-wide and external review process for individual program use to bring Continuing Studies more in-line with academic faculties and departments where external reviews are conducted on a regular basis.
Of course, program reviews and evaluations have been an essential element in UCE for many years (Davie, 1994;Queeney, 1995;Wiesenberg, 2000), though not specifically at a program-area level.However, rather than assume a universal model and apply it to a particular circumstance, SFU's approach was designed to provide an overarching framework while also accommodating each program's distinct approach, focus, and clientele.Following are SFU's general guidelines for establishing program reviews: A good Continuing Education program is planned to be contextual in nature and influenced by the social, economic, cultural, and political climate of its environment.The following broad questions are meant to discover how well the ________ Program at Simon Fraser University responds to its overall environment.Specifically: • What are the strengths and weaknesses of the Program from the participants', instructors', staff members', and any relevant others' perspectives?• What value do participants place on the programs and courses they've taken?• Are the planning, implementation and evaluation processes effective and efficient?In consultation with the Dean's office, the program director and staff modify these general questions to meet specific program requirements and supplement them with other questions to meet additional issues and concerns.The program staff also prepare a self-study document that describes the program's overall aims, approach, and organization, details its recent activities and achievements, and suggests people from various groups (such as students, staff, faculty, advisory board members, and community representatives) who might be contacted for their opinions.At that point, the review is handed over to an associate dean who chooses an appropriate external reviewer (usually a continuing education colleague from a neighbouring but non-competing institution), decides on a feasible timeline, and organizes site visits and stakeholders to be interviewed.Generally, a draft of the report is reviewed within about three months.
In contrast with faculty reviews, which normally involve a panel of three or four members, only one reviewer is selected for SFU's program reviews.This decision is taken on financial grounds as much as any other; no central funds are allocated for the reviews so each program must bear the costs of their own.Of course, the identity of the external reviewer is a crucial decision, one that can significantly affect the outcome of the review and, potentially, the future direction of the program.The associate dean has the responsibility here.He or she solicits the names of several potential reviewers, determines their suitability, and makes the final choice.Such an approach removes the program staff from any 'conflict of interest,' underscores the seriousness and rigour with which SFU regards the undertaking, and provides for some consistency across the several reviews carried out each year.Also, as associate deans have unit-wide responsibilities and are often concerned with relationships with other departments at their own and neighbouring institutions, their involvement in the administration of the review allows them to promote (and sometimes defend) the process at intra-and inter-institutional levels.

Case Study: Management and Professional Programs
In 1984, SFU opened a campus in downtown Vancouver in close proximity to the business community.Beginning in the late 1990s and on into the early 2000s, enrolment in the core Certificate in Management program delivered at this campus began to drop.In 2004, the program director restructured the management program's portfolio, and began a self-study of the certificate in management in 2006.
After the completion of the self-study, Management and Professional Programs (MPP) began the external review process.The available offerings at this time were the certificate in management (with various specializations); a diploma in applied project management; and some limited customized training, with two new certificates in career development and business analysis soon to be launched.There were also two partnered programs-one with a professional association and one with a European business school-and the programming for two endowments related to the Salvation Army and the Bank of Montreal.
The self-study of the Certificate in Management Program was conducted by Cathie Dunlop, program director of the research and evaluation unit within Continuing Studies at SFU, and its objectives ranged from building a comprehensive portrait of the program with feedback from key stakeholders to informing future decisions regarding program positioning and growth.The methodologies utilized were interviews (phone and e-mail), in-class and on-line surveys, and document/website reviews.
Before the self-study, the status quo of this fundamental certificate could be categorized as "the Good, the Bad, and the Ugly.""The Good" were established courses, dedicated instructors and staff, and a new campus in Surrey, BC, one of Canada's fastest growing cities.Even with limited marketing and fluctuating demand this certificate had the largest student base in Continuing Studies and was able to steadily generate net revenue.
"The Bad" included limited program governance, poor program awareness (even among Continuing Studies staff), declining demand for the program and courses, too much reliance on associations and professional designations to drive enrolment, and increasing competition."The Ugly" included aging print promotional materials and limited web marketing, a downtown campus with a high fixed-cost structure that required high course enrolments to be financially viable, and a vicious cycle of softening enrolment.
Students' reasons for enrolling in the program, by order of importance, included enhancing work-related skills and career advancement, accreditation in professional associations, interest in course topic, and the reputation of the program.Interestingly, these results closely mirrored findings from a survey of certificate program students undertaken a decade earlier at the University of Calgary (Hutton, 1997), with the exception that students in the Calgary study stated their top reason for enrolling in a program was to broaden their knowledge while professional accreditation was much lower in their priorities.While the students indicated they were generally satisfied with the courses and instructors, their greatest complaint was the limited availability and infrequent offerings of courses, and that the courses were not transferable to degree programs.Results of other interviews with instructors and associations indicated that the strengths of the program were the links to professional associations/designations, the quality of instructors, and the course content, while a perceived weakness was the ineligibility of courses for degree credit transfer.

Key Findings, Recommendations, and Implementation
Four fundamental and explicit questions taken from SFU's general guidelines for program reviews guided the review process and the external reviewer during this review.These questions were: Where does the MPP portfolio fit within a crowded education market and what are the portfolio's strengths and weaknesses?How does the MPP portfolio compare to similar units at other universities?What are the opportunities for future growth?Is the administrative structure adequate to support current needs and future growth?The recommendations of the review fell into six main areas: (1) program portfolio, (2) marketing, (3) students, (4) instructors, (5) program administration, structure, and governance, and ( 6) alternate delivery methods.
The external reviewer's report acted as a roadmap for the program area, indicating what was working and what needed improvement, where to focus growth and development, and where to proceed with caution.For MPP, the external reviewer's report, along with the program director's formal response, became the foundation for an active and ongoing planning process.After a meeting to review the findings and recommendations, the dean, associate dean, and program director prioritized the issues and immediate actions, determining where the program area needed to "take the long view."Over the two years since the program review was completed, MPP has undertaken several initiatives, using the reviewer's recommendations as a guide.

Program Portfolio
The external reviewer's report recommended the adoption of a "blue ocean strategy" to capitalize on new (or changing) market opportunities.In blue ocean strategy, the key is to create "uncontested market space that [makes] the competition irrelevant" (Kim & Mauborgne, 2005, p. 1).A red ocean represents all industries in existence, where boundaries are known and rules understood, where rivals compete for a piece of the existing demand: as the space gets more crowded, increasing competition turns the water bloody.A blue ocean represents the unknown market space where demand is formed by creating totally new industries or, more commonly, from within a red ocean by changing the boundaries of an existing industry.Cirque de Soleil is a perfect example of blue ocean strategy as it reinvented the circus and pulled in a new kind of customer who typically patronized the theatre, ballet, or opera, and who was prepared to pay higher prices for an unrivalled entertainment experience.For MPP, their new Certificate in Restorative Justice (developed in response to a request from the Salvation Army) was an example of blue ocean thinking that capitalized on reconfiguring current market spaces.The program area has also intentionally taken a long-term view with regard to moving into new market areas, such as the area of customized or contract training.
More program specializations have allowed the MPP area to leverage existing courses by bundling them differently and thus increase their demand.

Marketing
One of the review's recommendations concerning program-specific marketing was to capitalize on efforts of Continuing Studies to build brand awareness in the local market.Therefore, the marketing priority for MPP is now to "sell" the university name first and the link to a professional designation second.The relationship with professional associations remains important, but not as important as the SFU Continuing Studies brand.

Students
The review recommended capitalizing on MPP's alumni network by improving communication with past and present students.MPP has therefore begun offering alumni events for key programs, where past and present students come together to hear from an industry-specific guest speaker.Where appropriate, these events involve the local chapter of the relevant professional association.

Instructors
The review recommended that MPP improve instructor support, communications, and development, perhaps through such methods as an instructor newsletter, or instructor professional development and appreciation events.This recommendation is being addressed as a Continuing Studies-wide initiative, beginning with the introduction of instructor awards.

Program Administration, Structure & Governance
The review highlighted that, without the involvement and support of academic faculty, it would be a challenge for MPP to grow and evolve its program portfolio.One of the key recommendations was to hire an assistant director.The program area was at a critical juncture, and additional human resources were a key factor for moving forward with many of the recommendations and growing the program portfolio.[moved from pg. 6] The reviewer also strongly supported the area's efforts to establish a steering committee that included five academic faculty from the three faculties aligned with MPP programs, particularly the Faculty of Business Administration.Planning is underway to improve program governance and link the academic units more closely to MPP programming, with initiatives such as formally establishing an academic lead/coordinator for each certificate and diploma.
The review further emphasized that, in terms of building strategic internal relations with other faculties, it has been beneficial to expand and broaden the MPP portfolio.Although the program area began with a focus on business programs-recognizing that business and management will remain the core for the foreseeable future-recently MPP has expanded to other programming areas.A new cluster of programs related to Career and Life Planning is closely Looking Forward by Looking Back Canadian Journal of University Continuing Education / Vol.36, No. 2 fall 2010 Revue Canadienne de L'Éducation Permanente Universitaire / Vol.36, N o 2 automne 2010 http://ejournals.library.ualberta.ca/index.php/cjuce-rcepualigned with the Faculty of Education (counselling psychology and career development), while another new cluster related to Justice and Conflict Resolution is aligned with the School of Criminology (restorative justice), each with its own steering committee.This broadens MPP's base of support within the university and lessens the reliance on management programming, which can be a contentious programming area for many continuing education units.Even within the core Business and Management cluster, the steering committee includes faculty from the Department of Economics and the Centre for Tourism Policy and Research.Expanding these linkages across the university has helped to strengthen support and interest in both the business management programs and Continuing Studies.

Alternate Delivery Methods
Although the reviewer suggested that MPP consider increasing its online programming, given that Continuing Studies has been somewhat slow in deploying online delivery more broadly for non-credit programs, a program-specific approach to the use of technology in program delivery was chosen.The first fully online certificate, the Certificate in Restorative Justice, was recently launched.As well, several of the courses in the Certificate in Management now use a blendedprogram delivery model.

Growth in Management and Professional Programs
As can be seen in Table 1, the MPP programs have grown significantly in the two years since the program review was completed.In 2007/08, the Certificate in Management was the core program within the program area, representing 55% of all courses offered.By 2009/10, several new certificates had been added, doubling the number of courses offered while maintaining the average enrolment per course.The Certificate in Management now represents only 23% of all courses offered.Over this same time period, fiscal-year revenue for MPP increased from approximately $780,000 to over $1.5 million.

Lessons Learned
There were several lessons learned from both the process and the conclusions of this external program review.First, on a macro level, the choice of an external versus internal reviewer was considered justified, bringing impartiality, outside expertise, and the ability to be objective when evaluating the program.Often external reviewers bring relevant views and perspectives from their own experience that may suggest different ways of dealing with similar issues seen at other institutions.Finally, the reviewer is able to examine the program through an objective lens that allows the program area to be seen as it really is, and also as how it could be.This forces us as "insiders" to reflect on both the reality and the potential vision of what our programs are and could be when presented with the results of these reviews.
The external review process is able to make use of an outside perspective to take a longterm view and make long-term recommendations.While the direct financial cost of an external reviewer may be higher than using an internal reviewer, in this instance the end result was felt to more than justify the cost.With the valuable data contributed by the self-study, this review might be seen as an internal-external combination, which Patton notes "may be more desirable and more cost-effective than either a purely internal or purely external evaluation" (1997, p. 142).The risk in this approach lies in the vigilance required to shepherd recommendations and the accompanying changes through the university bureaucracy without losing sight of them or the desired end result.
Second, clarity regarding the vision or goal of the review is essential in order to guide the evaluation process and ensure it stays on track, producing useful data.Is the review meant to recommend large-scale changes that might question the very existence of the program area?Should the review compare the program area to similar programs at other institutions?What geographic range should be considered?Should opportunities for future growth be recommended, or is it more a matter of program retrenching?
Third, buy-in from the stakeholders is essential to both the process, and for guaranteeing that the end results of the review possess value and credibility.The review process must make a commitment to hearing the views of the stakeholders, no matter the nature of those views.
Finally, there is value in the review recommendations suggesting levels of changes, so programmers may choose to implement recommendations in stages, perhaps in line with desired results or available resources.Analogous to breakthrough strategy, this approach may also have the benefit of easing in some changes that might otherwise ruffle the feathers of program staff.It is useful to consider the important versus the urgent issues (Covey, Merrill, & Merrill, 1996) arising from the review, and take the long view in terms of implementing review recommendations.Often day-to-day issues force us to concentrate our energy on upfront urgent problems while eclipsing our ability to see how dealing with important issues may eventually prevent the appearance and reappearance of urgent problems.Susan Burgess est directrice des programmes de formation continue en gestion et de formation professionnelle continue au sein du département d'éducation permanente de l'Université Simon Fraser.Avant de se joindre à l'USF, elle était consultante pour les industries du tourisme, de l'hospitalité et du développement immobilier où elle a complété de nombreuses études de faisabilité, des plans d'affaires et des évaluations pour des clients de partout en Amérique du Nord, en Europe et au Moyen-Orient.
Diane Dutton has been extensively involved in continuing education at the Universities of Calgary and Alberta for over twenty years, and in continuing management education throughout North America.Her recently completed EdD dissertation examines sessional faculty in Canada and their motivation to teach in a post-secondary setting.Currently Diane is working with Volunteer Alberta to identify and develop a competency framework and related tools for senior leaders and managers in the non-profit/voluntary sector in rural Alberta.
Tom Nesbit is associate dean of Continuing Studies at Simon Fraser University.A former tradeunion official, he has worked as an adult and continuing educator in Great Britain, Sweden, the United States, and Canada.His research interests include social class, workers' and workplace education, adult numeracy, and the institutional provision of lifelong learning.He is editor-inchief of the Canadian Journal for the Study of Adult Education.
• Are available resources being used effectively?• Does the Program do what it claims?• Does the administrative structure of the Program appropriately support its purposes?• Are the staffing resources adequate and effectively used?• Is the Program being appropriately governed?• Are there appropriate opportunities for those involved in the local and disciplinary communities to comment upon and influence the Program and suggest potential areas of growth?(Simon Fraser University, 2007) In the spring of 2010, this approach gave rise to a five-course Certificate in Management Programs (in collaboration with Sing Tao Newspapers), which is taught wholly in Mandarin to support the learning needs and career aspirations of new Chinese-speaking immigrants to Greater Vancouver.The program review encouraged the exploration of other professional designations and program alliances, which is now a top priority for MPP, hence the development of a new Certificate in Human Resources Management linked to the Certified Human Resources Professional designation.Program formats have been expanded to offer certain management courses in a blendedlearning format, the addition of a part-time version of the Career Development Practitioner Certificate (previously only offered full-time), and the development of the online Certificate in Restorative Justice (the program area's first fully online program). http://ejournals.library.ualberta.ca/index.php/cjuce-rcepu

Table 1 :
Management and Professional Programs, Enrolment Statistics Summary http://ejournals.library.ualberta.ca/index.php/cjuce-rcepu Looking Forward by Looking Back Canadian Journal of University Continuing Education / Vol.36, No. 2 fall 2010 Revue Canadienne de L'Éducation Permanente Universitaire / Vol.36, N o 2 automne 2010 http://ejournals.library.ualberta.ca/index.php/cjuce-rcepuSimon Fraser University.(2007).Guidelines for Continuing Studies program reviews: Continuing Studies.Vancouver, BC: Simon Fraser University.Susan Burgess is the director of Management and Professional Programs within Continuing Studies at Simon Fraser University.Prior to joining SFU, she consulted in the tourism, hotel, and real estate development industries, completing numerous feasibility studies, business plans, and valuations for clients throughout North America, Europe, and the Middle East. Biographies