As part of contemporary Triple Helix operations, universities are more and more frequently asked to collaborate with other social actors to generate knowledge that supports their innovation and development activities and to foster economic growth. In addition to research and education tasks, universities are developing formal and informal collaborations with industrial organizations to fulfill the third mission of societal effectiveness and to receive funding from research grants. These university-industry research and development projects, at the societal level and from the viewpoints of policymakers and financiers, aim to create economic growth and to support the innovation and learning capabilities of the organizations, among other benefits. One key challenge to managing these collaborative activities between universities and other societal organizations is related to measuring and evaluating them. Therefore, this study focuses on supporting the future development of evaluation and performance measurement for the operational-level activities of university–industry–government relationships. The study explores the performance measurement practices currently used by universities and financiers and intends to increase the understanding of both current performance measurement practices and the challenges regarding university–industry (UI) collaboration projects. The empirical data for this study were gathered from UI research projects in Finland. As part of the management of university–industry–government relations and Triple Helix operations, the results of this study reveal some challenges in the evaluation and measurement of the collaborative projects in the three phases explored: before the project, during the project, and after the project. The study’s results indicate that UI collaboration projects’ performance evaluation practices are based mainly on the tasks promised in the projects’ applications. Furthermore, measurements are mainly used for external-reporting purposes. To support the future development of the evaluation and performance measurement of university–industry–government relations, some propositions for overcoming these challenges are presented, and the potential implications of the findings for scholars, university members, financier representatives, and policy makers are discussed.
دور قياس الأداء في مشاريع التعاون بين الجامعة والصناعة في إطار التصرف في العمليات المنجزة ضمن ”المراوح الثلاثة“
Tero Rantala, Juhani Ukko, Minna Saunila
Tero Rantala, Juhani Ukko, Minna Saunila
Le rôle de la mesure de performance dans les projets de collaboration université-industrie dans le cadre de la gestion des opérations de la Triple Hélice
Tero Rantala, Juhani Ukko, Minna Saunila
Dans le cadre des opérations contemporaines de la Triple Hélice, les universités sont invitées à collaborer de plus en plus avec d’autres acteurs sociaux pour générer des savoirs qui soutiennent leurs activités d’innovation et de développement et favoriser la croissance économique. En plus des tâches de recherche et d’enseignement, les universités développent des collaborations formelles et informelles avec des organisations industrielles pour remplir la troisième mission d’efficacité sociétale et obtenir des subventions de recherche. Ces projets de recherche et développement université-industrie, au niveau de la société et du point de vue des décideurs politiques et des financiers, visent entre autres à créer la croissance économique et à soutenir les capacités d’innovation et d’apprentissage des organisations. Un défi majeur dans la gestion des activités de collaboration entre les universités et d’autres organisations de la société est lié à leur mesure et à leur évaluation. Par conséquent, cette étude vise à soutenir le développement futur de l’évaluation et de la mesure de la performance des activités de niveau opérationnel dans les relations université-industrie-gouvernement.
Triple Hélice – mesure de la performance – gestion de la performance – collaboration université-industrie – gestion de projets
O papel da medição da performance em projetos de colaboração universidade-indústria como parte da gestão de operações « Triple Helix »
Tero Rantala, Juhani Ukko, Minna Saunila
Como parte das operações contemporâneas da Tríple Helix, as universidades são cada vez mais frequentemente chamadas para colaborar com outros atores sociais na gestão do conhecimento que apoia suas atividades de inovação e desenvolvimento e para fomentar o crescimento econômico. Em além das tarefas de pesquisa e educação, as universidades estão desenvolvendo formais e informais colaborações com organizações industriais para cumprir a terceira missão da sociedade de efetividade social e receber financiamento de bolsas de pesquisa. Essas universidades-indústria projetos de pesquisa e desenvolvimento, a nível social e dos pontos de vista dos decisores politicos e dos financiadores, visam criar crescimento econômico e apoiar o capacidades de inovação e aprendizagem das organizações, entre outros benefícios. Um desafio chave para gerar essas atividades colaborativas entre universidades e organizações sociais esta relacionado à medição e avaliação delas. Portanto, isso estudo foca em apoiar o desenvolvimento futuro de avaliação e medição da performance ao nível operacional das atividades universidade-indústria-governo.
Triple-Helix – Medição da performance – Gestão da performance – Universidade- Industria colaboração – Gestão de projetos
El papel de la medición del rendimiento en los proyectos de colaboración entre la universidad y la industria como parte de la gestión de las operaciones de Triple Helix
Tero Rantala, Juhani Ukko, Minna Saunila
Como parte de las operaciones contemporáneas de Triple Helix, se les pide a las universidades con cada vez más frecuencia que colaboren con otros actores sociales para generar sus actividades de innovación y desarrollo y para fomentar el crecimiento económico. Además de las tareas de investigación y educación, las universidades están desarrollando colaboraciones formales y informales con organizaciones industriales para cumplir la tercera misión de eficacia social y recibir subvenciones a la investigación. Estos Universidad-Industria proyectos de investigación y desarrollo, a nivel social y desde el punto de vista de políticos y financistas, tienen como objetivo crear crecimiento económico y apoyar las capacidades de innovación y aprendizaje de las organizaciones, entre otros beneficios. Uno desafío clave para la gestión de estas actividades colaborativas entre universidades y otros organizaciones sociales esta relacionado con medirlas y evaluarlas. Por lo tanto, este estudio se centra en apoyar el desarrollo futuro de la evaluación y medida del desempeño de las actividades de nivel operativo das relaciones de la universidad,industria-gobierno.
Triple Helix – Medición del rendimiento – Gestión del rendimiento – Universidad-Industria Colaboración – Gestión de proyectos
Collaboration activities between governmental organizations, industry, and academia have been acknowledged as important mechanisms for successful technology and knowledge transfer, which forms a foundation for global-, national-, and regional-level innovation and economic growth (Miller et al. 2016; Urbano and Guerrero 2013; Etzkowitz and Leydesdorff 1995). Currently, the Triple Helix of university–industry–government relations provides surroundings and frameworks for different organizations to overcome blockages to innovation in regional innovation systems (Champenois and Etzkowitz 2018; Etzkowitz 2008). Collaborative partners in innovation and development activities are no longer sought only by other public and private organizations; today, universities are commonly seen as important partners in these activities. Organizations of all sizes are increasingly forming collaboration activities and research and development projects with universities to maintain and develop their competitive advantages (Peças and Henriques 2006; Perkmann et al. 2011). As such, within recent years, the expansion of universities’ entrepreneurial activities has resulted in various helix models, such as the Triple Helix model of government, academia, and industry (Miller et al. 2016). By participating in activities formed under Triple Helix models, (instead of isolated research and development activities), different organizations are for example looking for external governmental funding support by forming collaboration activities and projects with universities. While research into university–industry (UI) collaborations have traditionally focused on the transfer of intellectual capital (e.g. licensing and commercialization), some studies have contended that these alliances have more multiform natures (e.g. Agrawal 2001; Mäkimattila et al. 2015). As part of the Triple Helix-related operations, traditional collaborations typically enable organizations to participate with universities in alliances or projects focused on specific technical or scientific subject areas. Today’s partnerships and collaboration projects, however, are increasingly being formed on the basis of more open and innovation-based strategies (e.g. Bahemia et al. 2018; Bogers et al. 2017). In these collaborations, organizations are interested, for example, in gaining social capital or innovation capabilities rather than solving specific technical problems (Perkmann and Walsh 2007; Pinheiro et al. 2016). As part of the Triple Helix-related operations, these more open strategies aiming to increase social capital and innovation capabilities create possibilities but also cause challenges for university-industry collaborations, and, thus, more research and understanding of the execution and management of these collaboration activities is needed. A main challenge in terms of managing UI collaboration activities as part of Triple Helix operations is related to the collaborations’ evaluation and performance measurement activities. Therefore, the aim of this study is to explore the current operational-level performance measurement practices used and challenges faced by university project managers and financier representatives when managing collaborative UI research and development projects under the Triple Helix umbrella. In this study, exploring the phenomenon at the operational level involves those who are executing the projects, including, for example, the project managers who are part of daily project operations and the financier delegates who actively follow the operations performed in those projects.
While a growing part of the Triple Helix operations are pursued in diverse types of research and development projects between universities and other societal organizations and even though the literature on project management (PM) has recognized and suggested performance measurement practices (Cao and Hoffman 2011; Ling et al. 2009; Thomas and Fernandez 2008), Perkmann et al. (2011) argued that a key challenge for the management of university-industry collaborations is how to assess and evaluate them. Further, Immonen and Cooksy (2014) found in their study that there were difficulties in developing indicators for the kinds of performance that research institutions are expected to demonstrate on outputs, outcomes, and impacts of these collaborations. In addition to performance measurement of the projects pursued by industrial organizations, the literature on the performance measurement of UI collaborations has suggested practices and indicators to support their management (cf. Albats et al. 2018; Perkmann et al. 2011). However, as in traditional industrial PM contexts, commonly implemented performance measurement frameworks and practices are rare in UI collaborations. Thus, in order to be able to evaluate and manage Triple Helix operation, it is important to more understand both the meanings and aims of the performance measurement in these UI projects as well as the elements that support or complicate these activities.
Even though there seems to exist an interest towards the new and alternative methodologies of analyzing and evaluating university–industry–government relations (e.g. Park 2014), the previously developed and suggested performance measurement frameworks and tools seem to remain insufficiently utilized in practice. Some studies (e.g. Rantala and Ukko 2018) have explored the phenomenon of measuring the performance of UI collaborative activities from the perspective of industrial organizations. Their findings show that although the industrial organizations in these collaborations are interested in their social-level outcomes, they lack understanding and capabilities regarding measurement. Rather, they seem to focus on the collaborations’ business perspectives and the use of traditional business-related measurement practices. Furthermore, based on the results of Rantala and Ukko (2018), the industrial organizations participating in collaborative UI projects agreed that in the “big picture,” performance measurement should be conducted by other stakeholders. Thus, the present study focuses on increasing understanding of contemporary performance measurement practices and the challenges that university project managers and financier representatives face when managing UI collaboration activities as part of Triple Helix operations. The empirical data for this study were gathered from Finnish UI research projects that were all receiving governmental funding support. The data for the study were collected from interviews with collaborative research project managers and financier delegates. As the majority of the performance measurement practices, reporting, and bureaucratic aspects of UI collaborations are usually handled by the university project managers and financier representatives, this study focuses on exploring the phenomena from these perspectives. The study is executed through the following research questions:
RQ1 How is performance measurement used to steer university–industry collaborative projects at the operational level?
RQ2 How are the actions, success, and effects of the university–industry collaborative projects evaluated at the operational level?
RQ3 What challenges do the performance measurement of the university–industry collaborative projects raise at the operational level?
As part of the future development of the university–industry–government relations, the growing interest in collaborative UI projects necessitates research on performance measurement to obtain a better empirical understanding of their current practices and challenges. Although many governments and universities around the world have allocated funding mechanisms and resources to support the forming of Triple Helix activities (Champenois and Etzkowitz 2018; Croce et al. 2014) and although collaboration activities between universities and other societal organizations have been championed as likely to play significant roles in generating economic growth in the future, both types of activities must more and more often deal with limited resources and funding support. Economic recession has decreased the funding available to university-industry collaborations and research and development projects, and current grant competitions are ever more competitive. While making decisions concerning Triple Helix operations, politicians and financiers are in the challenging situation of finding the best possible partnerships and projects to support. For that reason, diverging stakeholders are asking better methods and practices for performance measurement to support the management of these collaborations (e.g. Rantala and Ukko 2019). Therefore, we believe that all of the stakeholders in these collaborations can find the results of this study beneficial for developing better solutions for managing and evaluating university–industry–government relations and Triple Helix operations.
2 University–Industry Partnerships and Collaboration Activities
Universities as well as private and public organizations have been reported to play important roles in generating new economic growth, innovation, knowledge, and product development (e.g. Lundberg and Andersen 2012; Perkmann et al. 2013; Clark 2001). In addition to the execution of the third mission, universities are also seeking increasing amounts of external funding from different sources, such as Horizon 2020, European Regional Development Funds, and national programs, to bolster their other operations and educational programs. Economic recessions are also forcing universities to modify their operations and compete for external funding. Universities, therefore, are under increasing pressure to interact with their surrounding society to support their research and development activities, as society’s expectations of the economic returns from basic research and development activities have increased (Schartinger et al. 2001; Lundberg and Andersen 2012). As such, policymakers are more frequently encouraging universities to develop their third mission of fostering partnerships and collaboration between academic researchers and knowledge users (Gulbrandsen and Slipersaeter 2007). Among the various ways of establishing these activities, commercializing knowledge and licensing innovations have been the focus of attention, both in the academic literature and in the policy community (O’Shea et al. 2008). While these are clearly important ways for academic research to contribute to both the economy and society, research is in fact transferred into practice in numerous ways (Rantala and Ukko 2018). Currently, UI collaboration activities are of growing interest to various societal stakeholders, and the methods of executing collaborations include many different types of options and possibilities. Generally, collaborative activities between universities and other societal organizations can be either formal or informal, including, for example, graduate education, providing collaborative research, launching technologies or start-ups, and exchanging resources and staff (Hsu et al. 2015). Ankrah and Al-Tabbaa (2015) found that the most typical forms of UI collaboration are alliances, consortia, networks, and joint ventures. The present study defines UI collaborative research and development projects as formal contract-based ones involving stakeholders from universities, industrial organizations, and financial institutions. What further differentiates these projects from other types of collaborative activities among universities is that these projects have targeted agreements and durations. Not only by generating short- and long-term commercialized outcomes, but also by supporting intra- and inter-organizational development, UI projects play important roles in developing the innovation capabilities of organizations. It is not merely the traditional stakeholder groups in organizations (e.g. customers and competitors) that can produce knowledge that supports competitive advantages and organizational development; university research also plays an important role in the innovation activities of industrial organizations (Cohen et al. 2002; Salter and Martin 2001). The role of academic research and partnerships with universities, thus, is more than simply creating patents or licensed products; it also involves supporting organizations’ learning processes and innovation capabilities. The academic and industry involvement in knowledge transfer include formal activities, such as collaborative or contract research, as well as informal activities, like advising and networking with practitioners (Perkmann et al. 2013).
Because the funding programs in many countries require academics to make a societal impact outside of their universities, understanding how collaboration activities and partnerships result in such benefits, while still maintaining scientific quality, is of great relevance (Perkmann et al. 2013). Further research on the impacts and effectiveness of collaborations could enable policymakers to make considered judgements as to which behaviors and forms of collaboration to promote and under which conditions they are most likely to achieve their scientific and economic objectives (Perkmann et al. 2013). Although UI projects offer potential advantages, they are not without their problems. For one, the structural differences between universities and industrial organizations are a major challenge. While academic research tends to have a long-term orientation, organizations are more commonly interested in short-term outcomes and rapidly commercializable innovations (Perkmann et al. 2011). Another problem is rooted in confidentiality. In order to allow the commercial exploitation of their research results, researchers might be required to delay or even forgo publication (Geuna 2001). Researchers’ publishing activities might also be hindered by academic journals’ lack of interest in industry-connected work that is not perceived as novel or academically relevant. In addition, researchers might spend time and resources on activities that are not conducive to academic output (Calderini et al. 2007; Perkmann and Walsh 2009). Perkmann et al. (2011) presented aspects of how UI projects differ from traditional research and development projects pursued by industrial organizations. First, in collaborative projects that focus on generating scientific knowledge that is far from commercialization, the value of any project outputs are difficult to evaluate. Second, in UI projects, the outputs are likely to be more complex than in traditional settings. Third, project benefits may be realized only over the long term (Voytek et al. 2004). Given the long-term nature of some UI projects, many of the benefits derived from these collaborations might be realized only after a considerable time (Perkmann et al. 2011).
In addition to their challenges and differences, the impact of industry partnerships and collaboration activities on universities has also drawn the attention of researchers; some have emphasized the benefits of partnerships, while others have feared that growing involvement in these activities might have negative effects on academic priorities (Perkmann and Walsh 2009). Perkmann et al. (2013) suggested there is a partial lack of understanding about the consequences of academic engagement and showed that evidence on these collaborations’ impacts on other university activities, like research and teaching, is scarce; therefore, it cannot be assumed that collaboration activities are always beneficial and should be promoted. Salimi et al. (2015) found that doctoral students participating in collaborative UI projects achieved higher performance than those participating in non-collaborative activities in terms of several performance dimensions, including the number of publications and the number of citations. The knowledge transfers from universities to industries have been also the interest of several academic studies (cf. Albats et al. 2018; Anderson et al. 2007), and the reverse direction of information and knowledge flow (from industries to universities) has also been examined (cf. D´Este and Patel 2007). Even though the importance of the knowledge gathered from industry organizations in collaboration projects has been recognized as playing an important role in academics’ careers, the evaluation of the gathered and transferred knowledge seems to be rare.
3 Need for Performance Measurement in University–industry Collaboration Projects
Organizations collaborating in research and development projects with universities have recognized the need for more systematic evaluation and measurement, both to assess their outcomes and to monitor ongoing activities in order to implement timely improvements (Perkmann et al. 2011). Consequently, developing performance measurement systems for the organizations’ collaborations and results is paramount, not only for the organizations but also for the universities (Piva and Rossi-Lamastra 2013). In addition, for the performance measurement interests of industrial organizations and universities, less attention has been paid to the measurement practices and challenges of the financier representatives of these collaborative projects (c.f. Bark et al. 2016). However, some studies have shed some light on their evaluation perspectives (cf. D´Este et al. 2012; Plewa et al. 2013). Even though the evaluations of the UI collaborations have been studied from different perspectives, academics still emphasize the shortcomings in the existing performance measurement practices applied by external evaluators (Albats et al. 2018; Kauppila et al. 2015; Perkmann et al. 2011).
The literature on performance measurement recognizes the trend toward inter-organizational collaboration and regularly calls for more research to be conducted on performance measurement in collaborative organizations and projects (Bititci et al. 2012). As with other types of collaborations, UI research and development projects involve organizations that work together in relationships of equality, commitment, and trust; that exchange information, share activities, and combine resources; and that complement and enhance each other’s capacities in order to achieve mutual benefits and a common purpose by sharing risks, responsibilities, and rewards (Bititci et al. 2004). In UI projects, organizations can utilize the external knowledge produced by researchers, who in turn can examine how their newest findings operate in real-world contexts. Several researchers have claimed, however, that in order to generate such advantages, the network structures of these collaborations need to be fully understood, carefully measured, and appropriately managed (cf. Bititci et al. 2012). For example, a study by Saunila and Ukko (2013) showed that the exploitation of external knowledge can be improved significantly through performance measurement. UI collaborations can be arranged in different ways, and the participating organizations may have different expectations for them. To support the performance measurement activities of collaborative projects and to increase the understanding of the context in which measurement and evaluation takes place, it is important at the beginning of the collaboration to ensure that all the participating organizations have a common understanding of the partnership’s and project’s vision and targets (Busi and Bititci 2006; Ukko et al. 2015). The purpose and methods of the performance measurement should be defined jointly by the participating organizations, and common visions for and targets of the collaboration should be clarified.
As a result of the increased interest in these collaborative research projects, some frameworks and indicators have recently been proposed in an effort to improve their management and evaluation, for example:
A Balanced Scorecard for measuring the impact of UI collaboration (Al-Ashaab et al. 2011)
Critical key performance indicators for UI collaboration (Albats et al. 2018)
Criteria-based success evaluation model for assessing the UI research collaboration (Iqbal et al. 2011)
Factors in the success of R&D cooperative agreements between firms and research organizations (Mora-Valentin et al. 2004)
A performance measurement system to evaluate the success in UI alliances (Perkmann et al. 2011).
These frameworks and tools, however, are mainly theoretical and show the need for more operational-level empirical evidence and understanding to support their implementation and further development (Perkmann et al. 2011). In other words, it appears that no empirical studies have implemented and further developed these suggested frameworks based on feedback and experiences gathered while using them. Given this, theoretical-based frameworks need to be better understood to enable them to be implemented and used more widely in practice.
3.1 Project Management and Performance Measurement
Today, individual development activities among industrial organizations are often pursued and managed in the form of different types of research and development projects. PM has become popular, and visions, methods, and good practices that promote PM can be found (Korhonen 1992; Lauras et al. 2010). One of the primary challenges in PM is to achieve all of a project’s objectives and goals while adhering to project constraints (Harrison and Lock 2004). As a part of PM operations, Project Management Information Systems (PMSIs) should provide project managers with decision-making support for organizing, planning, and controlling projects (Caniëls and Bakens 2012). However, according to the literature on PM, most project managers seem to be dissatisfied with the information provided by the current PMSIs. Among the PMSIs, performance measurement plays an important role in ensuring a project’s success and its usefulness to the participating organization(s) (Pillai et al. 2002). According to the authors, in a controlled environment, the organizational and project performances are known to be sensitive to the metrics of measurement.
Previous studies on PM have identified a wide range of practices that can be used to evaluate the outcomes of projects and the input characteristics that impact outcomes, for example, the two-step performance measurement approach (Cao and Hoffman 2011), project scoping practices, time management practices, cost control practices, quality control practices, risk management practice, human resource management (Ling et al. 2009), and methods for evaluating IT-process success (Thomas and Fernandez 2008). However, many of these practices are not widely implemented; for example, project schedules are still used as the sole project performance measurement in many organizations. Studies on project performance measurement have also used different methods based on different viewpoints and factors that affect project performance (Xu and Yeh 2014). For example, while measuring overall project performance, multicriteria decision-making methods have been used to aggregate multiple performance measures under various application contexts (cf. Marques et al. 2011; Barfod 2012; Xu and Yeh 2014). One of the proposed performance measurement approaches to support PM is cross-project learning, which has been identified as vital for organizations seeking to continuously improve PM practices (Cao and Hoffman 2011; Lewis 2000). Cao and Hoffman (2011) posited that the first step in cross-project learning is to identify outstanding projects that can serve as role models. The authors further demonstrated that performance measurement allows the creation of incentives that will likely yield higher performance. Xu and Yeh (2014) argued that performance measurement of ongoing and completed projects would help establish benchmarks for high-performance projects (for cross-learning) and identify the inefficiencies in low-performance projects (for potential improvement) but stated that project assignment and project performance measurement are treated as two separate research problems in PM.
In summary, both academics and practitioners seem to be aware that project performance measurement plays an important role in the management of operational level activities of university-industry-government relations, but successfully implemented, comprehensive performance measurement systems seem to be rare.
4 Research Design
The purpose of this study is to explore the current operational-level performance measurement practices and challenges of universities and financier representatives as a part of managing university-industry-government relations. These phenomena are explored from the viewpoints of the university project managers and financier representatives of the collaborative research and development projects conducted between universities and other societal organizations. As a majority of the performance measurement practices, reporting, and bureaucratic aspects of UI research projects are usually handled by the university project managers and financier representatives, the phenomena was explored from these perspectives. The empirical data for this study were gathered from 12 Finnish UI research projects that were established to find answers to certain problems identified by funding programs, to enhance and support organizations’ innovation activities and capabilities, and to produce new academic knowledge. The selection of the explored projects was based more on their natures than their specifics (Perkmann and Walsh 2009). As part of the Triple Helix operations and university–industry–government relations, the explored research and development projects focused on management, engineering, and economic issues as well as involved industrial partners who did not have long backgrounds or histories in collaborating with universities. The projects lasted between two and three years, which is considered a normal period for these kinds of applied UI projects.
The data for the study were collected from interviews with twelve university project managers representing three different universities in Finland. The motivation for selecting three different universities was to gather empirical evidence of the phenomena from different regions in Finland, and, more importantly, from projects funded by different external funding sources. Generally, the universities selected do not differ markedly from one another in any way except size. All follow the Finnish national funding mechanism for universities and Finnish national university law.
The interviews lasted from 60 to 90 minutes. The interview participants were chosen by how they had actively been part of managing and measurement of these projects, which were funded by different external sources. In other words, they were academic staff (post-doctoral researchers, junior researchers, and project researchers) who had received the rights and responsibilities of project managers. All had been actively participating in the activities and operations of their projects, and all had been responsible of evaluating and measuring them. It is important to note these sources in order to ensure a common understanding and holistic picture of the different evaluation methods used in different granting decisions: two of the chosen projects were funded by the European Commission’s Seventh Framework program (FP7), three by the Finnish Funding Agency for Technology and Innovation, and seven by European regional development funds. The interviews were semi-structured and used the following themes from the current literature on performance measurement and UI collaborations as the basis for the data collection:
The performance processes and methods used to evaluate the projects before they were undertaken (Al-Ashaab et al. 2011; Iqbal et al. 2011; Mora-Valentin et al. 2004; Perkmann et al. 2011)
The performance measurement challenges before the projects were undertaken (Al-Ashaab et al. 2011; Iqbal et al. 2011; Mora-Valentin et al. 2004; Perkmann et al. 2011)
The performance measurement processes and methods used to evaluate the projects while they were underway (Al-Ashaab et al. 2011; Iqbal et al. 2011; Mora-Valentin et al. 2004; Perkmann et al. 2011)
The performance measurement challenges while the projects were underway (Al-Ashaab et al. 2011; Iqbal et al. 2011; Mora-Valentin et al., 2004; Perkmann et al. 2011)
The performance measurement processes and methods used to evaluate the outcomes and effects of completed projects (Al-Ashaab et al. 2011; Iqbal et al. 2011; Mora-Valentin et al. 2004; Perkmann et al. 2011)
The performance measurement challenges of the outcomes and effects of completed projects (Al-Ashaab et al. 2011; Iqbal et al. 2011; Mora-Valentin et al. 2004; Perkmann et al. 2011)
To explore these issues from the viewpoints of the project financiers, two financier representatives from the European Regional Development Funds of Finland and two members of the Finnish Funding Agency for Technology and Innovation were also interviewed. Semi-structured interviews were conducted with the same themes and factors as were used with the university project managers to ensure comparability with the evaluation processes employed and the challenges that were included. Table 1 summarizes the process of data collection.
4.1 Data Analysis
The data collected from the interviews were primarily analyzed through a multi-coding process to generate patterns, which, during the analyzing process, were structured into more generic patterns. In the first round of the coding process, the data were analyzed and arranged through patterning the role of performance measurement in UI research projects from the viewpoints of both university and financier representatives. These findings regarding the role of performance measurement in UI research projects were used in the subsequent coding process. In other words, after both the role of and the purposes for using the performance measurement had been understood and patterned, the next aim of the coding was to understand what types of measurement practices were used and what challenges existed. Thus, a second round of coding was conducted with the more generic patterns related to the current performance measurement challenges and practices in UI research projects. Both university and financier representatives’ viewpoints were arranged during the second round of the process. Again, the findings of the second coding round were used as a basis for the third coding round. In the final round of coding, the patterns structured during the second round were further arranged into more generic patterns identifying the individual measurements that both university project managers and financier representatives used to measure the performance of these collaborative research projects. In each stage of the coding process, research triangulation was used to validate the structured patterns. Three researchers who had expertise in performance measurement conducted the triangulation process. After each coding round, the findings and patterns discovered were analyzed both by each researcher individually and by all three researchers together to avoid single-observer bias. The overall coding process of the data is summarized in Table 2.
In this section, the findings of the research are presented according to three different phases of the UI projects. The first phase includes the performance measurement activities during the preparation of the projects, when they were not yet funded or had gotten the promise of funding but had not begun. The second phase includes the performance measurement activities while the projects were ongoing. The third phase includes the projects’ evaluations after they were finished, from immediate to long-term evaluation activities.
5.1 Performance Measurement Before the Projects
The results from the interviews revealed that most of the explored projects did not determine any performance measurement methods or individual measurements for the projects before starting them. Although measurement methods were not determined at all or in only a cursory fashion in many of the collaborations, the university project managers mainly agreed that there should be better performance measurement methods for collaborative projects before they are launched or are in the preparation phase. Measurement activities concerned with evaluation should be conducted in the project’s preparation and planning phase to enable understanding of the motivations of various stakeholders to participate in the project. For example, for industrial organizations, there are various motivations to participate in collaborative activities with universities. In many cases, these motivations are not obvious to other project stakeholders. If some development work regarding performance measurement activities were conducted during the project’s preparation and planning phase, it might identify motivations and aims, thereby making it possible to set targets for them. The same applies to other stakeholders, including individual researchers participating in collaborations. Although they usually receive salaries from these collaborative projects, salaries are not the only motivation for researchers to participate. Their other motivations may include joint publications with industrial organizations. Again, if the motivations of other stakeholders could be identified, targets could be set for them, which could support the preparations for the project and further the performance measurement activities.
A leading reason that project managers cited for the lack of performance measurement activities at the preparation phase was the current market situation. Many funding calls by Horizon 2020 or the Finnish Funding Agency for Technology and Innovation, for example, asked about the involvement of industry or other private-sector organizations. Private-sector organizations in Finland and many other European countries, however, are continuing to deal with a serious economic slowdown, so they are keeping a careful eye on any new development and collaboration activities, at least those executed with universities. This explains why the project managers commonly agreed that, in many cases, it is not possible to evaluate and design projects too carefully beforehand; universities are forced to collaborate with those partners that are willing to participate in projects. (Note: the quotations from the study participants have been slightly edited from the Finnish–English translations for clarity.)
There have been some cases where we did some preparations and tried to do some performance measurement related to evaluating the best possible partners beforehand. After searching for and exploring four or five suitable candidates, none were interested in collaborating with the project. We did not have the chance to change the project plan because of the theme of the funding call, and we had to pick the partners that were interested in collaborating with us, even though they were not the best ones. Since then, we haven’t systematically evaluated partners beforehand. (University project manager)
In addition to the challenge of finding suitable partners, there were serious challenges related to actual project preparation. Many of the project managers mentioned that it is quite common that the majority of the project proposals and applications are rejected because of the current competitive situation. They reported that university project managers use numerous working hours writing project applications and proposals that are rejected and are not funded. Evaluations of how much resources are sacrificed to that work do not seem to exist.
It is actually surprising that no one seems to be interested in the resources that are sacrificed to rejected project applications. In the current situation, we have to write, for example, let´s say four or five different applications in order to get one funded. Sometimes some part of that work can be re-used in later applications, but usually, that work will be forgotten. We should at least evaluate how much resources are used for “unnecessary” work that could have been used, for example, for journal writing. Moreover, we should somehow to be able to evaluate what we have learned from that work. (University project manager)
The current market situation has also caused some problems for the financier representatives’ measurement activities. In many cases, the funding calls asked that SMEs participate in the collaborative projects because a large part of today’s economic and new job growth comes from SMEs and startups. These firms, however, have suffered even more from market challenges than large and multinational companies. Therefore, financier representatives have to evaluate whether a potential organizational partner actually has enough resources to participate in a collaborative project:
Over the last two years, we have had several cases where the SMEs faced bankruptcy while they were participating in a collaboration project with the university. After these cases, we have had to be much more careful in evaluating whether the SMEs have the resources to handle their daily operations throughout the course of the whole project. (Financier representative, Finnish Funding Agency for Technology and Innovation)
After the preparation phase, but before the projects are launched, there are several challenges regarding project performance measurement. One challenge at this phase of a UI project is the lack of communication between academic project managers and financier representatives concerning the actual measurement of the project’s performance. Although universities, industrial organizations, and financier representatives are all interested in project performance measurement, few resources are deployed to establish common measurement methods and goals. There are no standard procedures by which to form collaborative research projects so that the aims and wishes of the participating organizations are defined and determined jointly with all parties. Even in those cases where there were a few agreed-upon measurements or evaluation methods, most university project managers wished there were better methods, both to steer the projects and to evaluate their outcomes:
Once the project application is accepted, it is common to just start the project by following the project plan. If there are several industrial partners involved, there is not usually a common view about the goals and wishes of the other project participants. The goals and aims promised in the project plans are used as the goals of the project. It quite often happens that industrial partners are unfamiliar with these goals. It might be a good practice to design some kind of measurement system or method both to explore and make explicit the expectations of all of the partners and to generate measures to follow them. (University project manager)
Table 3 summarizes the current performance measurement practices and challenges of the university project managers and financier representatives before the projects are launched (and in the beginning of the projects).
5.2 Performance Measurement During the Projects
Although the university project managers felt that performance measurement practices should be more clearly designed with the participating organizations before a project begins, they thought they had thorough understandings of what was going on in their own projects. This indicates that they need better performance measurement methods or individual measurements, not to increase their own understandings of the projects, but, rather, to make the outcomes and results of the projects transparent to other stakeholders:
It’s not that I don’t know what is happening in the project; it’s more like the other partners of the project are unfamiliar with the project’s actions. We did not have any performance measurement or evaluation systems before the project. I think that by following up to see if the aims and wishes of the organizations had been reached, this might have helped steer the project. It also would have helped other partners to follow the actions in other cases and work packages. (University project manager)
The interviews also revealed that it is fairly common to use the original project application as the steering and measurement framework for a project. Usually, the project’s application presents various work packages and tasks that execute and fulfill the themes of the funding programs. Furthermore, a project’s application usually includes some targets that the project aims to achieve. These targets and aims usually reflect the actions and operations that will be conducted in the work packages and tasks. Thus, the performance measurement activities of these UI collaboration projects seem to focus on following work packages and tasks and achieving the promised targets. Both the financier representatives and university project managers seemed to be quite satisfied if the project followed the original plan. The university project managers generally agreed that the measurements and goals determined beforehand did not overly restrict their activities during the projects, though some challenges remained:
We had a situation where we had a chance to attain a very radical innovation. If we had chosen to do that, we would have had to deviate from the original project plan. Once we weren’t sure if we could reach that innovation, the project manager decided not to do it. It wasn’t mentioned in the original plan, and it was much easier to report the use of the resources as they were meant in the beginning of the project. If we had had some kind of measure to try to evaluate the effectiveness of that possible radical innovation, and we had had a better understanding of the initial situation, trying to reach the innovation probably would have been easier. (University project manager)
For the financier representatives, challenges arose when the projects did not follow the original project plans:
Of course, it’s possible to make changes in the project while it’s already underway. However, in many cases, it is quite difficult to evaluate these changes. Usually the situations where the changes are required are unexpected, for both the universities and the industrial organizations. (Financier representative, European Regional Development Funds)
Other challenges that financier representatives faced during project performance measurement were lack of time and other resources. During different funding programs and individual funding calls, there are many different projects funded, and the financier representatives feel that in many cases, they lack the time and other resources to focus on the projects’ performance measurement. That is why they mainly focus on the measurement of the tangible aspects of the projects, for example, money and salaries used, workshops arranged, and reports published. When using this kind of measurement practice, they are mainly using the project application/project plan as a measurement framework.
From the viewpoint of the university project managers, systematic performance measurement of various elements during the project occur at a relatively low level, even though they are important from the perspective of their own operations and innovativeness:
Amount and quality of accumulated data. The respondents noted that data quality is not usually evaluated. Rather, it is simply gathered from the project (e.g., from workshops, interviews, and surveys) and used afterward for research purposes.
Increased understanding and learning. None of the interviewees mentioned that they had any measurement methods of their own to determine their increased understanding or changes in welfare or the work climate. The project managers also indicated that they were unaware of which university innovation capabilities could or should be evaluated. Despite the current lack of measurement, all the interviewees agreed that it would be very useful to evaluate these issues if there were suitable methods or frameworks to do so.
Satisfaction of industry partners. Despite the importance of industry involvement, there did not seem to be a systematic evaluation of industry partner satisfaction. While no one suggested any one glaring cause, they did mention a few minor issues: “It is not conventional to evaluate the satisfaction of the industry partners;” “No one asks for that information” and “The expectations of the industry organizations are not known, so it is hard to evaluate their satisfaction.”
Rather than measuring their own operations, the university representatives expressed more interest in using performance measurement to evaluate issues that need to be reported to project financiers. It is likely that a few measurements borrowed from the project application are used to steer projects. These tools, however, are usually quantifiable; for example, they could calculate the number of new companies or jobs directly resulting from a project without paying attention to the increased innovation capabilities that could be used in future projects.
The interviews also explored who should be responsible for the measurement of these UI partnerships and projects. It was commonly agreed that performance measurement activities should include members of both academia and industry. The university project managers noted that it was difficult to try to evaluate what was happening in the industry organizations during the partnerships if they were not actively involved in the measurement. The respondents also stated that it might be challenging for industry organizations to evaluate these activities by themselves. They usually lacked an understanding of the projects’ academic elements, like data quality and publications, and were typically not interested in learning about those aspects:
It is basically impossible for a researcher to evaluate or measure the activities happening inside the industrial organizations. That is why someone should be named responsible for the evaluation of the organizations, as well. (University project manager)
5.3 Performance Measurement After the Projects
The performance measurement activities conducted after collaborative research projects are completed—not only to make a project’s actual results and effects visible immediately after its completion, but also to track results in the ensuing years—posed a major challenge to both university project managers and financier representatives. In many of the cases, the project managers had the strongest understandings of the projects and their results. If they were unavailable during the follow-up period, evaluation was extremely difficult:
There have been a few collaborative research projects where we followed up with them three or five years after they were finished. However, in many cases, the people who were part of the original project were no longer available for this follow-up evaluation work: they either no longer worked there, or they were at a new position. There do not seem to be any measurements or evaluation methods that we could use to follow up with projects after they are completed. Normally, the results of a project are analyzed and reported straight after a project is finished, and then people move on to the next project. That makes following up with these projects extremely difficult. (Financier representative, Finnish Funding Agency for Technology and Innovation)
For the university project managers, the challenge was usually connected with the partnership or project funding. They received salaries from projects while they were running them, and after a project was finished, they usually moved on to the next one. Projects with salaries allocated for future performance measurement and evaluation work were extremely rare:
If there were some resources allocated to conducting follow-up performance measurement after the projects (say, one month of the year for five years), and if we had some proper evaluation methods, it would be possible to evaluate more meaningfully the actual effectiveness of the activities that were carried out during a project. (University project manager)
An important societal-level outcome of different funding programs is bringing innovations, new products, and novel services to the market. From this viewpoint, which is shared by innovation theory, UI research projects and partnerships should also create ideas and frameworks for future collaborative projects. The financier representatives also reported that it was difficult to compare the results and effectiveness of projects:
At the moment, we do not have the proper tools to evaluate and compare the effectiveness of projects that are carried out in different geographical areas. We get the results and the project reports from the organizations that executed the project, and after that, we are supposed to compare these results with the results from other areas. At the moment, however, this is very difficult to do. (Financier representative, European Regional Development Funds)
In order to provide a more empirical understanding of the role of evaluation and performance measurement in university–industry–government relations, this study explored the current operational-level performance measurement practices and challenges of university project managers and financier representatives as part of managing UI research and development projects. Due to the increased interest in the evaluation of Triple Helix operations, performance measurement of the UI projects, performance measurement frameworks, individual measures, and indicators have been suggested in previous studies (Al-Ashaab et al. 2011; Albats et al. 2018; Iqbal et al. 2011; Mora-Valentin et al. 2004; Perkmann et al. 2011). This study, however, revealed that the university project managers and the financier representatives demonstrated a lack of familiarity with the presented frameworks and indicators and lack the proper process models to implement them. Even though there were some individual cases where performance measurement practices were slightly jointly designed by the university and industrial organizations, financier representatives have been excluded from these design activities. Regarding the findings from the theoretical perspective on UI collaboration and performance management, there seems to be no culture in which performance measurement activities are designed collaboratively by university project managers, industry organizations, and financier representatives. Though Lauras et al. (2010) argued that each project manager should develop a range of KPIs for the projects, we believe that in order to develop KPIs to support the evaluation of the collaborative research and development projects involved in Triple Helix operations, university project managers need more academic and practical-level support to better recognize the current performance measurement frameworks suggested by academics. This would enable them to develop and implement performance measurement practices and use existing models and processes to support planning, controlling, and evaluating the projects.
Some of the previous research on PM (e.g. Cao and Hoffman 2011; Ling et al. 2009; Thomas and Fernandez 2008) has revealed that, even though a wide range of practices and measures that can be used to evaluate and steer projects have been proposed, in many cases, project schedules are still used as the sole performance measurement approach. From a theoretical perspective and in contrast with the performance measurement practices of industrial projects, this study shows that in the management of university–industry–government relationships, original project applications are commonly used as the sole performance measurement framework for steering and evaluating UI research and development projects. These results are in line with Xu and Yeh’s (2014) findings, which showed that project assignment and project performance measurement are treated as two separate research problems in PM. Though university project managers and financier delegates seem to be satisfied if the projects proceed according to the project applications, it causes challenges for evaluation. The projects’ applications, and the tasks and goals promised in them, are written before the projects are started, and they are basically only educated guesses of how projects will be executed. As UI research projects evolve and change their forms and goals over time, it is also important to review and update the performance measurement frameworks and individual measurements during the projects. This would increase the dynamism of the measurements, better support the evaluation and management of the projects, and further support the management of university–industry–government relations.
Currently, as part of the Triple Helix operations and the collaborative research and development activities, university project managers’ performance measurement activities seem to be focused on qualitative success measures executed during the project and immediately after the projects are finished. Moreover, they seem to be focusing mainly on the issues that need to be reported to financier representatives. Though the knowledge flow from industry to university is an important part of academics’ careers (e.g. D’Este and Patel 2007), university project managers are not currently actively evaluating the quality of knowledge they can gather from the projects. Further, this study found that, though university project managers might have some ways to evaluate the most suitable partners for the projects and data gathering, they are commonly forced to collaborate with industry organizations that are willing and able to participate in projects, rather than with suitable partners. It was reported that once they knew they were not collaborating with the most optimal partners, they knew they probably would not get the best possible data and knowledge, and the project managers would have to deal with the data they got. The situation seems to be frustrating for the university project managers. The results support the findings of Perkmann and Walsh (2009), which showed that applied collaboration with industry organizations might distract academics from engaging in long-term academic research because, among other reasons, the data and results gathered from collaborative projects might not be publishable in academic journals because they are not sufficiently systematic.
Although universities participate in Triple Helix operations and collaborative research and development projects mainly to fulfill their third mission of societal effectiveness and meet funding program requirements for industry involvement, they should also pay attention to the measurement of their own improvements in innovation activities and capabilities. While universities are interested in the short-term benefits from these projects, for example, reaping academic publications and funding, their long-term aspects also warrant attention. The funding programs to support university-industry collaborations in many countries and at the European Union level will become increasingly competitive, thus requiring universities to pursue more innovative partnerships and collaborative research projects in order to succeed. Focusing solely on measuring resources and outputs does not fully capture all the components of innovation capability. The measurement of innovation capabilities is an issue that has received significant attention in academic circles in the last few years. Saunila and Ukko (2012) presented a conceptual framework for measuring innovation capability and its effects. They argued that simply knowing how many new innovative processes, actions, or products have been initiated is insufficient if there is no understanding about their connections to performance. This approach, however, remains novel and rare, even in academic circles. Regarding the theoretical implications of the study’s findings, in addition to the challenges caused by a lack of familiarity with evaluating innovation capabilities, a lack of continuity among university project managers causes further evaluation challenges. The majority of the interviewed project managers had fixed-term employment contracts. Commonly, there is uncertainty about future working possibilities in the universities and what will happen after the projects are finished. Several project managers argued that it is hard to focus on evaluating the long-term aspects of the projects (such as increase in innovation capabilities or changes in work climate) when they do not know if they are long-term parts of the organizations.
For the financier representatives, the most significant challenges in the performance measurement of UI projects as part of the evaluation of Triple Helix operations seem to be related to cross-project learning (e.g. Cao and Hoffman 2011), meaning it is difficult for them to recognize outstanding projects and actions that might serve as role models for others. While some individual projects turn out to be successful and thus provide a valuable example of Triple Helix operations, there do not seem to be evaluation criteria that explain their success. Therefore, it is difficult to transfer the successful elements to future (or other ongoing) projects and collaborative operations or use them as evaluation indicators. Thus, the financier representatives also face challenges when trying to compare different ongoing projects in different universities and different geographical areas. The lack of continuity among the university project managers also causes challenges for the financier representatives. The performance measurement activities and reporting are mainly handled by one person (usually the project manager), and if that person is working in another organization in the future, the future evaluation of the projects is extremely difficult. It was commonly suggested by the university project managers and financiers that more than one person should be responsible for the performance measurement in UI projects to support the operational-level evaluation of university–industry–government relations. It would not only support future evaluation activities but also help to distribute the lessons learned from the operational level projects more effectively. These findings are in line with the conclusions of Perkmann et al. (2011), which pointed out that the dissemination of the performance measurement should concisely communicate the objectives of the project to all the participants in the collaborative projects and, therefore, increase the motivation for evaluation.
As part of the university–industry–government relations and Triple Helix operations, UI activities and projects serve the interests not only of university researchers and industrial organizations but also of financiers and policymakers. Due to increased competition in the universities’ and industry organizations’ operating environments, policymakers are continually looking for better ways to evaluate which collaborative research projects and UI partnerships should be funded. The results of this study show that there are challenges from the viewpoints of university project managers and financier representatives in the performance measurement of these activities at all three stages explored: before, during, and after the projects. The methods or individual measurements used to evaluate these collaborations are based primarily on the project applications, which are obviously established before the project. The current methods and measurements used to steer projects and to make the results visible do not assess either increased learning or innovation capabilities of the participating organizations.
The results of the study further indicate that, at the moment, no commonly implemented performance measurement systems exist among universities, financier representatives, and industrial organizations to help steer collaborative projects as part of the Triple Helix operations. Some individual measurements are used to follow specific aspects, such as the resources used or workshops arranged, but they are mainly applied from the project applications and evaluation instructions and implications given by the funding agencies, and comprehensive performance measurement systems are lacking. Even though some tools and frameworks to evaluate the collaborative projects in the context of university–industry–government relations have been suggested, both university project managers and financier representatives of the collaborative research and development projects are unfamiliar with them and do not know how to implement them. There does not exist a culture where performance measurement activities would have been commonly deigned and adopted with university project managers, industry organizations and financier representatives. The results of this study also indicate that as part of their Triple Helix-related operations, universities are presently more interested in measuring the issues that they report to financiers and the academic world than in evaluating their intra-organizational learning and increasing their own innovation capabilities. While in the short term this approach may support the evaluation of the societal effectiveness of UI collaborations, in the long term, these partnerships should generate new knowledge and increase the innovation capabilities of the participating organizations to support long-term economic growth at a broad societal level.
The financier representatives in the collaborative research and development projects conducted between universities and other societal organizations are currently using evaluation criteria provided by the funding calls, as well as the aims and goals promised in the project applications, while evaluating the performance of UI research projects. The biggest challenges they seem to face in these actions are related to cross-project evaluation. They struggle to find successful projects that could be used as benchmarks for other projects, and the comparison of different projects poses challenges. The lack of continuity of the university project managers’ work contracts poses performance measurement challenges for both project managers and financier representatives. When project managers cannot be sure of the continuity of their work, they do not see it meaningful to evaluate the long-term aspects of these collaborations. As many of the advantages gained from these projects are only realized long after the projects, financier delegates find it difficult to evaluate the projects if the project managers have moved to other positions and organizations and are unavailable. The challenge is that the performance measurement of the UI research projects relies too heavily on the project managers; therefore, these activities should be disseminated more widely.
To support the operational-level management and evaluation of university–industry–government relations and to develop the future performance evaluation practices of the UI research projects, the creation of a collaborative design and implementation culture for performance measurement processes is needed to implement the theoretical tools and frameworks. This culture should involve all the participating stakeholders, including the financier representatives, in designing, building, and implementing the measurement frameworks for UI collaborations, and support evaluation and management throughout entire projects. This would enhance the measurement effectiveness of such collaborations at both the operational and societal levels.
One theoretical implication of this study is the increase in the understanding of the current performance measurement activities that university project managers and financier representatives pursue in UI projects as part of managing university–industry–government relations. In addition, the study increased the empirical understanding of the current challenges related to the performance measurement of these projects. Because operational-level activities of Triple Helix operations UI projects seem to be of growing interest to both academics and industry representatives, the results of this study can be utilized to develop better performance measurement tools and frameworks that could be more easily adopted in such contexts. While developing these frameworks, the academics should also pay attention to the viewpoints, practices, and challenges of the financier representatives, not focus only on universities’ and industry organizations’ perspectives.
As managerial implications, this study increases the university project managers’ and financier representatives’ understandings of each other’s performance measurement practices and challenges. As it seems to be quite common that university project managers and financier representatives do not design performance measurement tools for these collaborative projects, the results of this study can support their understanding of each other’s viewpoints and increase the interplay between them. Only following the instructions provided by the funding calls or only following the research plans causes challenges in performance measurement. To be aware of these challenges and of the performance measurement activities of the others, more comprehensive and suitable evaluation methods can be developed to support the management of UI projects.
The limitation of this study is that the empirical evidence was gathered from Finland, and some cultural aspects that are country specific might exist. For that reason, it may be important and valuable to execute comparative studies in other countries. Further, as the UI research and development projects seem to be growing, it would be reasonable to investigate how a culture of PE design and implementation could be fostered and supported in these collaborations. It is also important to find performance measurement practices for financier representatives to support their cross-project evaluations. Finally, though the lack of continuity among the university project managers’ working conditions cannot be directly solved by performance measurement activities, it is important to explore how these challenges could be met for long-term performance evaluation.
This article is part of the Special Issue Science Parks and Innovation. The guest editor is Christiane Gebhardt. This collection of articles has been initiated in the framework of the THA (Triple Helix Association) and IASP (International Association of Science Parks).
Agrawal, A. (2001) University-to-industry knowledge transfer: literature review and unanswered questions. International Journal of Management Reviews 3: 285–302.
Al-Ashaab, A., Flores, M., Doultsinou, A., Magyar, A. (2011) A balanced scorecard for measuring the impact of industry–university collaboration. Production Planning & Control: The Management of Operations 22(5/6): 554–570.
Albats, E., Fiegenbaum, I., Cunninghma, J.A. (2018) A micro level study of university industry collaborative lifecycle key performance indicators. Journal of Technology Transfer 43(2): 389–431.
Anderson, T.R., Daim, T.U., Lavoie, F.F. (2007) Measuring the efficiency of university technology transfer. Technovation 27(5): 306–318.
Ankrah, S., Al-Tabbaa, O. (2015) Universities–industry collaboration: a systematic review. Scandinavian Journal of Management 31: 387–408.
Bahemia, H., Sillince, J., Vanhaverbeke, W. (2018) The timing of openness in a radical innovation project: a temporal loose coupling perspective. Research Policy 47: 2066–2076.
Barfod, M.B. (2012) An MCDA approach for the selection of bike projects based on structuring and appraising activities. European Journal of Operational Research 218: 810–818.
Bark, R.H., Kragt, M.E. and Robson, B.J. (2016) Evaluating and interdisciplinary research project: lessons learned for organisations, researchers and funders. International Journal of Project Management 34: 1449–1459.
Bititci, U., Carengo, P., Dörfler, V., Nudurupati, S. (2012) Performance measurement: challenges for tomorrow. International Journal of Management Reviews 14: 305–327.
Bititci, U.S., Martinez, M., Albores, P., Parung, J. (2004) Creating and managing value in collaborative network. International Journal of Physical Distribution & Logistics Management 34(3): 251–268.
Bogers, M. et al. (2017) The open innovation research landscape: established perspectives and emerging themes across different levels of analysis. Industry and Innovation 24(1): 8–40. https://doi.org/10.1080/13662716.2016.1240068.
Busi, M., Bititci, U. (2006) Collaborative performance measurement: a state of the art and future research. International Journal of Performance and Productivity Management 55: 7–25.
Calderini, M., Franzoni, C., Vezzulli, A. (2007) If star scientists do not patent: the effect of productivity, basicness and impact on the decision to patent in the academic world. Research Policy 36(3): 303–319.
Caniëls, M.C.J., Bakens, R.J.J.M. (2012) The effects of Project Management Information Systems on decision making in a multi project environment. International Journal of Project Management 30: 162–175.
Cao, Q., Hoffman, J.J. (2011) A case study approach for developing a project performance evaluation system. International Journal of Project Management 29: 155–164.
Champenois, C., Etzkowitz, H. (2018) From boundary line to boundary space: the creation of hybrid organizations as a Triple Helix micro-foundation. Technovation 76/77: 28–39.
Clark B. (2001) The entrepreneurial university: new foundations for collegiality, autonomy, and achievement. Higher Education Management 13(2): 9–24.
Cohen, W., Nelson, R., Walsh, J. (2002) Links and impacts: the influence of public research on industrial R&D. Management Science 48(1): 1–23.
Croce, A., Grilli, L., Murtinu, S. (2014) Venture capital enters academia: an analysis of university-managed funds. Journal of Technology Transfer 39(5): 688–715.
D’Este, P., Patel, P. (2007) University–industry linkages in the UK: what are the factors underlying the variety of interactions with industry? Research Policy 36(9): 1295–1313.
D’Este, P., Guy, F., Iammarino, S. (2012) Shaping the information of university–industry research collaborations: what type of proximity does really matter? Journal of Economic Geography 13(4): 537–558.
Etzkowitz, H., Leydesdorff, L. (1995) The Triple Helix university–industry–government relations: a laboratory for knowledge based economic development. EASST Review 14(1): 14–19.
Geuna, A. (2001) The changing rationale for European university research funding: are there negative unintended consequences? Journal of Economic Issues 35: 607–632.
Gulbrandsen, M., Slipersaeter, S. (2007) The third mission and the entrepreneurial university model. In: Daraio A and Slipersaeter, C (eds) Universities and Strategic Knowledge Creation: Specialization and Performance in Europe. Edward Elgar: Cheltenham, pp. 112–143.
- Search Google Scholar
- Export Citation
( , Gulbrandsen, M. Slipersaeter, S. ) 2007 The third mission and the entrepreneurial university model. In: (eds) Universities and Strategic Knowledge Creation: Specialization and Performance in Europe. and Daraio A Slipersaeter, C Edward Elgar: Cheltenham, pp. 112– 143.
Hsu, D.W.L., Shen, Y.C., Yuan, B.J.C., Chou, C.Y. (2015) Toward successful commercialization of university technology: performance drivers of university technology transfer in Taiwan. Technological Forecasting and Social Change 92: 25–39.
Immonen, S., Cooksy, L. (2014) Using performance measurement to assess research: lessons learned from the international agricultural research centres. Evaluation 20(1): 96–114.
Iqbal, A.M., Khan, A.S., Iqbal, S., Senin, A.A. (2011) Designing of success criteria-based evaluation model for assessing the research collaboration between university and industry. International Journal of Business Research and Management 2(2): 59–73.
Kauppila, O., Mursula, A., Harkonen, J., Kujala, J. (2015) Evaluating university–industry collaboration: the European Foundation of Quality Management excellence model-based evaluation of university-industry collaboration. Tertiary Education and Management 21(3): 229–244.
- Search Google Scholar
- Export Citation
( , Kauppila, O. , Mursula, A. , Harkonen, J. Kujala, J. ) 2015 Evaluating university–industry collaboration: the European Foundation of Quality Management excellence model-based evaluation of university-industry collaboration. Tertiary Education and Management 21( 3): 229– 244.
Lauras, M., Marques, G., Gourc, D. (2010) Towards a multi-dimensional project Performance Measurement System. Decision Support Systems 48: 342–353.
Ling, F., Low, S., Wang, S., Lim, H. (2009) Key project management practices affecting Singaporean firms’ project performance in China. International Journal of Project Management 27(1): 59–71.
Lundberg, H., Andersen, E. (2012) Cooperation among companies, universities and local government in a Swedish context. Industrial Marketing Management 41: 429–437.
Marques, G., Gourc, D., Lauras, M. (2011) Multi-criteria performance analysis for decision making in project management. International Journal of Project Management (29): 1057–1069.
Mäkimattila, M., Junell, T., Rantala, T. (2015) Developing collaboration structures for university–industry interaction and innovations. European Journal of Innovation Management 18(4): 451–470.
Miller, K., McAdam, R., McAdam, M. (2016) A systematic literature review of university technology transfer from a quadruple helix perspective: toward a research agenda. R&D Management 48(1): 7–24.
Mora-Valentin, E.M., Montoro-Sanchez, A., Guerras-Martin, L.A. (2004) Determining factors in the success of R&D cooperative agreements between firms and research organizations. Research Policy 33: 17–40.
O’Shea, R., Chugh, H., Allen, T. (2008) Determinants and consequences of university spinoff activity: a conceptual framework. Journal of Technology Transfer 33: 653–666.
Park, H.W. (2014) An interview with Loet Leydesdorff: the past, present, and future of the triple helix in the age of big data. Scientometrics 99(1): 199–202.
Peças, P., Henriques, E. (2006) Best practices of collaboration between university and industrial SMEs. Benchmarking: An International Journal 13(1): 54–67.
Perkmann, M. et al. (2013) Academic engagement and commercialization: a review of the literature on university–industry relations. Research Policy 42: 423–442.
Perkmann, M., Neely, A., Walsh, K. (2011) How should firms evaluate success in university–industry alliances? A performance measurement system. R&D Management 41(2): 202–216.
Perkmann, M., Walsh, K. (2009) The two faces of collaboration: impacts of university–industry relations on public research. Industrial and Corporate Change 18(6): 1033–1065.
Perkmann, M., Walsh, K. (2007) University–industry relationships and open innovation: towards a research agenda. International Journal of Management Reviews 9(4): 259–280.
Pillai, A.S., Joshi, A., Rao, K.S. (2002) Performance measurement of R&D projects in a multi-project, concurrent engineering environment. International Journal of Project Management 20: 165–177.
Pinheiro, M.L., Serôdio, P., Pinho, J.C., Lucas, C. (2016) The role of social capital towards resource sharing in collaborative R&D projects: evidences from 7th framework programme. International Journal of Project Management 34(8): 1519–1536.
Piva, E., Rossi-Lamastra, C. (2013) Systems of indicators to evaluate the performance of university–industry alliances: a review of the literature and directions for future research. Measuring Business Excellence 17(3): 40–54.
Plewa, C., Korff, N., Johnson, C., Macpherson, G., Baaken, T., Rampersad, G.C. (2013) The evolution of university–industry linkages—a framework. Journal of Engineering and Technology Management 30(1): 21–44.
Rantala, T., Ukko, J. (2019) Performance evaluation to support European regional development—university–industry perspective. European Planning Studies 27(5): 974–994.
Rantala, T., Ukko, J. (2018) Performance measurement in university–industry innovation networks: implementation practices and challenges of industrial organisations. Journal of Education and Work 31(3): 247–261.
Salimi, N., Bekkers, R., Frenken, K. (2015) Does working with industry come at a price? A study of doctoral candidates performance in collaborative vs. non-collaborative PhD projects. Technovation 41/42: 51–61.
Salter, A., Martin, B. (2001) The economic benefits of publicly funded basic research: a critical review. Research Policy 30(3): 509–532.
Saunila, M., Ukko, J. (2013) Facilitating innovation capability through performance measurement: a study of Finnish SMEs. Management Research Review 36(10): 991–1010.
Saunila, M., Ukko, J. (2012) A conceptual framework for the measurement of innovation capability and its effects. Baltic Journal of Management 7(4): 355–375.
Schartinger, D., Schibany, A., Gassler, H. (2001) Interactive relations between universities and firms: empirical evidence for Austria. Journal of Technology Transfer 2(3): 255–268.
Thomas, G., Fernandez, W. (2008) Success in IT projects: a matter of definition? International Journal of Project Management 26(7): 733–742.
Ukko, J., Pekkola, S., Saunila, M., Rantala, T. (2015) Performance measurement approach to show the value for the customer in an industrial service network. International Journal of Business Performance Management 16(2/3): 214–229.
Urbano, D., Guerrero, M. (2013) Entrepreneurial universities: socioeconomic impacts of academic entrepreneurship in a European region. Economic Development Quarterly 27(1): 40–55.
Voytek, K.P., Lellock, K.L., Schmit, M.A. (2004) Developing performance metrics for science and technology programs: the case of the manufacturing extensions partnership program. Economic Development Quarterly 18: 174–185.
Xu, Y., Yeh, C.-H. (2014) A performance-based approach to project assignment and performance evaluation. International Journal of Project Management 32: 218–228.
Tero Rantala is a researcher and doctoral student at Lappeenranta University of Technology. His current research focuses on evaluation of university-industry partnerships and performance measurement and management, including the measurement of the effectiveness of innovation activities in private and public sector organizations.
Dr. Juhani Ukko is a Senior Researcher at Lappeenranta University of Technology, School of Engineering Science. His current research interests involve different areas of performance management and measurement, covering the assessment and measurement of service operations, networks, and innovation capability.
Minna Saunila is a Senior Researcher at Lappeenranta University of Technology, School of Engineering Science. Her current research focuses on performance measurement and management, including the measurement of innovation capability and the relationship between innovation capability and organizational performance in small and medium sized enterprises.