Chapter 22 Trying to Say ‘No’ to Rankings and Metrics

Case Studies from Francophone West Africa, South Africa, Latin America and the Netherlands

In: Socially Responsible Higher Education
Authors:
Florence Piron
Search for other papers by Florence Piron in
Current site
Google Scholar
PubMed
Close
,
Tom Olyhoek
Search for other papers by Tom Olyhoek in
Current site
Google Scholar
PubMed
Close
,
Ivonne Lujano Vilchis
Search for other papers by Ivonne Lujano Vilchis in
Current site
Google Scholar
PubMed
Close
,
Ina Smith
Search for other papers by Ina Smith in
Current site
Google Scholar
PubMed
Close
, and
Zakari Liré
Search for other papers by Zakari Liré in
Current site
Google Scholar
PubMed
Close
Open Access

Abstract

In this chapter, continuing from , the authors examine examples from Africa, Latin America and Europe to answer the question of how to encourage academics to conduct research that meets society’s needs and enhance people’s rights, while preserving academic freedom. In particular, they ask how we can use the tools and devices devised by knowledge democracy (science shops, participatory research, community based research) to emancipate open access from the enclosures that for-profit publishers are still trying to impose on academia.

1 Introduction

In Chapter 6, we demonstrated that the metrics and rankings system not only hinders the development of universities’ and researchers’ conscience of their social responsibility, but also produces exclusionary effects in the global South or in non-Anglophone European scholarly communities. We also hinted that more knowledge democracy is possible within a polycentric system, taking into account local values and priorities.

In this chapter, we explore four case studies illustrating this polycentricity and its difficulties. We begin this world tour by giving an overview of the situation in two very different African contexts: Sub-Saharan French-speaking Africa, where the open science movement is still very recent and timid, and where research infrastructures are very poorly supported, and South Africa, which is predominantly English-speaking and has strong research infrastructures. In contrast, open science is a well-established tradition in Latin America, presented in the following section. We end the chapter with a European case, that of the Netherlands. This country was chosen because it is both very committed to research and open science, while having a strong tradition of knowledge democracy.

2 Francophone Sub-Saharan Africa

For many Francophone Sub-Saharan African universities, CAMES (African and Malagasy Council for Higher Education), a not-for-profit pan-African organisation based in Ouagadougou, is responsible for the promotion and tenure of academics through well-organised pan-African international committee (CCI) meetings once a year. A global policy orientation document in French lists the elements that academics must present to pass to the next level in their career (associate professor, full professor) (African and Malagasy Council for Higher Education, 2017). This document clearly outlines the importance of publishing in “international” journals or “outside the applicant’s university and country of practice”, but does not explicitly mention the impact factor of these journals. However, “international” here clearly means “from the North”. In fact, other internal policy papers show that for four of these CCI committees (Mathematics, Physics and Chemistry Committee, Engineering Sciences Committee, Natural Sciences Committee. and Science and Technology of Physical and Sports Activities Committee), it is necessary for scientists to have at least two papers in a journal indexed in the Web of Science or Scopus to be promoted. We found a few testimonies of scientists having failed to get their promotion because of their inability to publish in such a journal. Since many get promoted in these disciplines, we deduce that they manage to publish there, perhaps with co-authors from the North. A more detailed survey needs to be done.

Another interesting promotion criterion can be found in the CAMES global policy paper about promotion and tenure: “service to the community”, a form of social responsibility that can be linked to knowledge democracy. This includes all the services that an academic has to render to the university (participating in committees, etc.), but also mentions their responsibilities towards society, including:

  1. Participating in civic activities related to their specialty
  2. Getting involved in a university research centre, company, non-government organisation, community partnerships, etc. (our translation).

0The idea that service to the community should be included as a criterion for promotion of academics could be a starting point for the creation of an effective knowledge democracy in this part of the world. The creation of 10 science shops (in Benin, Burkina Faso, Cameroon, Ivory Coast, Guinea, Niger and Senegal) within the SOHA1 network is also an exciting new social and intellectual movement that will contribute to these goals (Piron et al., 2021).

Research shows that scholarly communities in this region of the world have been very slow to appreciate the opportunities of open access practices and policies, even if many interesting initiatives have been launched, mainly by librarians and the CAMES2 (Piron et al., 2017). This may explain the very small number of open access journals from Francophone Sub-Saharan Africa. On a positive note, the more numerous local (printed) journals do not interest much the captains of the English-language, for-profit scientific publishing industry, therefore remaining free to treat “local” issues.

In this environment, a Francophone African platform of open access journals has just been created. The Grenier des savoirs3 is a platform that brings together 15 multidisciplinary journals, focussing on themes of importance to Sub-Saharan Francophone Africa. It aspires to be indexed in the DOAJ. Funded by Éditions science et bien commun, located in Canada, while registered in Benin, the platform uses innovative Wordpress-based software. The editorial boards of the platform have together drafted a common, bold and forward-looking editorial policy, using open access under the CC-BY-SA license. In order to reach a larger African audience having problems of connecting to the Web, the Grenier des savoirs’ software includes the possibility of printing each issue of a journal. Other characteristics of the editorial policy are translation of abstracts into an African language, inclusive writing, a constructive process of peer review, the possibility of practicing open peer review, and a diversity of text types that promote the diversity of knowledges.

A condition for authors to publish in the Grenier des savoirs is that they accept a commitment that includes the following values of knowledge democracy:

I understand that I have to aim for the accessibility of my text, so that it can be read and used in the world of research and teaching, but also in civil society, administrations, companies, etc. Therefore, I am committed to minimizing jargon or explaining all the key concepts I use, especially if they are rare or little known.

I understand that the Grenier des savoirs, in its quest for cognitive justice, encourages me to use and cite the work of women researchers and work from other regions of the Global South, such as Latin America or Asia.

But its most striking innovation regarding knowledge democracy is the creation of a Social Relevance Committee, alongside the Scientific Committee. This is how it is defined on its website:

The mission of the Social Relevance Committee is to prevent the Grenier des savoirs from becoming an ivory tower, cut off from the concerns of the contemporary world and to ensure, through dialogue with the journal teams, that they remain focused on socially relevant issues for Africa and Haiti. This committee also supports the journals in their efforts to highlight both their African identity and their universality, i.e., their relevance to all humanity.

The Committee is made up of people who are not employed by a university or research centre as researchers: they may or may not have a PhD, be in graduate studies, work in the public service or in an NGO, as an entrepreneur or self-employed. What these citizens have in common is that they are aware of the need to promote socially responsible research in Africa and Haiti and wish to contribute to it. They believe in the capacity of the Grenier des savoirs to embody this project while aiming for a high level of scientific quality. They have no vested interest in any for-profit industry or business that would wish to influence their participation on the committee.

In February 2020, three journals of the Grenier des savoirs have already published their first issue, and six journals have launched their first call for papers. Nearly 80 abstracts were received within a few months (Grenier des savoirs, 2020). The enthusiasm of the editorial boards for the project and for open access shows the potential for editorial innovation in Sub-Saharan French-speaking Africa, without the weight of the impact factor.

3 South Africa

According to UNESCO, South Africa is a leading African country in terms of Open Access (OA) policies at the governmental and grass-roots levels. The Academy of Science of South Africa (ASSAf) plays a leadership role in promoting high quality OA publishing across all disciplines through its Scholarly Publishing Programme, which includes a National Scholarly Editors’ Forum (established in 2007) and the SciELO South Africa platform, which aggregates 79 high quality journals adhering to both ASSAf and Scientific Library Online (SciELO) Brazil standards. The SciELO South Africa collection was certified in April 2013 as a regular, operational national collection, indexed in the SciELO Network Global Portal. All SciELO journals appear on the Web of Science (WoS) search portal within the SciELO Citation Index. SciELO South Africa is an automatically accredited index of the South Africa Department of Higher Education and Training (DHET) (see Chapter 6), together with five other accredited lists. Since 2011, South African university libraries have started offering journal hosting services to South African scholarly journals, in addition to hosting solutions offered by African Journals Online (AJOL) and commercial publishers such as AOSIS. In all three cases (university libraries, AJOL and AOSIS), the Public Knowledge Project (PKP) Open Journal Systems (OJS) open source software is used.

In South Africa, the DHET Research Outputs Policy (2015) outlines the criteria for the measurement and evaluation of research outputs of public higher education institutions (HEIs), and serves as a tool for the distribution of research subsidy to public HEIs in South Africa. The Department subsidises institutions and not individual authors or academics. Without being prescriptive, it cautions institutions against directly incentivising individual authors, as this practice can promote perverse behaviour, in some cases. Publishing in high impact factor journals is not mentioned as a requirement for funding, and funding is solely allocated based on the number of articles published in scholarly journal titles in the DHET accredited lists (WoS, Scopus, IBSS, Norwegian List Level 2, SciELO SA and DHET lists).4 Since universities compete with one another in terms of global rankings, and since university rankings are, amongst others, based on bibliometric data obtained from prominent commercial publishers, it stands to reason that the pressure to publish in high impact factor journals stems from the institutions themselves.

The DHET constantly evaluates the South African scholarly landscape, which is evident through the following studies commissioned by them in the past few years.

  1. The quality of South Africa’s research publications (2020) (unpublished)
  2. Twelve Years Later: Second ASSAf Report on Research Publishing In and From South Africa (2019)
  3. Scholarly Books: Their Production, Use and Evaluation in South Africa Today (2009)
  4. Report on a Strategic Approach to Research Publishing in South Africa (2006)
  5. Multiple reports on the peer-review of South African scholarly journals

The studies are used to inform policy and decision-making.

Recent developments and initiatives are an indication that South Africa is increasingly favouring Open Access and Open Science, including the following:

  1. The pilot pan-African Open Science Platform project (2016 to 2019) conducted a landscape study of what is happening on the continent, in terms of Open Science, and frameworks for policy, capacity building, incentives, and e-infrastructure were developed to guide African countries at the national level.5
  2. The SA-EU Open Science Dialogue report (2018) was released as an outcome of the SA-EU Strategic Partnership Dialogue Facility. The intention of the framework was to support the South African Department of Science and Technology in the formulation of an Open Science policy and, more broadly, to assist all actors involved in the R&D process with the adoption of Open Science practices and principles. In South Africa, Open Science also has tremendous potential for creating a more inclusive society, towards citizen science.
  3. The White Paper on Science, Technology and Innovation was adopted as policy by the South African government in 2019.
  4. The South Africa Science Engagement Strategy aims at communicating and engaging the wider community more fully in science and in an understanding of the knowledge economy to which we aspire.

4 Latin America

The situation in Latin America presents many different challenges from Africa. The continent has the oldest tradition of open access publishing, due to researchers’ efforts to expand academic knowledge, starting in the late 18th century. According to Cetto and Alonso-Gamboa (1998), academic journals emerged during the last decades of the Spanish colonial regime, when the scientific community was very small. The first scientific periodical publication in the New Spain was the medical journal Mercurio Volante, published in 1772 by José Ignacio Bartolache y Díaz Posada, a physician and mathematician described as “a man with no teacher who conversed only with the dead [via books]” (Glick, 1991).

Since then, scientific publications in Latin America have been managed by the community of scholars, most of them affiliated to universities (whether private or public), research centres, and research societies. All these higher education institutions work as publishers. For instance, universities publish journals and books through their editorial presses or research departments. Most of the journals’ editors-in-chief and editorial board members are faculty members of universities who do the editorial work ad honorem. With regard to journals, their content has been available in electronic format with no cost to readers or authors since the early 1990s. This is thanks to the efforts of different groups of scholars concerned about the under-representation of Latin American production in the commercial journal circuits (Cetto & Alonso-Gamboa, 1998).

Two major projects can be attributed to this goal. First, in 1995, the project Latindex was launched, thanks to the support of the Universidad Nacional Autónoma de México (Mexico), where the bibliographic databases CLASE (1975) and PERIODICA (1978) were created, to list the products of research (articles, essays, reviews, biographies, etc.) in Latin America and the Caribbean. This pioneering project established a set of criteria to evaluate journals, which, at the same time, set the basis for following the developments on open access indices and databases. Second, in 1997, the Scientific Library Online (SciELO) was launched in Brazil, with the support of the Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP). Currently, SciELO works as a network of peer reviewed journals that has a presence in 14 Latin American countries, Spain, South Africa and Portugal.

Besides these two major initiatives, other systems of information in Latin America have developed, in the 21st century, to counterbalance the predominance of mainstream journals, where Latin American authors are underrespresented. Redalyc, CLACSO Library, and the network of repositories La Referencia are amongst the more relevant. Those systems index millions of documents produced by Latin American authors, or elsewhere, while maintaining a focus on this region as a unit of analysis. For instance, in La Referencia, there are 190,150 doctoral theses produced in 10 countries (Argentina, Brazil, Chile, Colombia, Costa Rica, Ecuador, El Salvador, Mexico, Peru and Uruguay). In addition, Latin America is the region where more journals use Open Journal Systems (OJS) than anywhere else in the world (2,840 installations in 2018), which shows the benefits of the Public Knowledge Project (PKP). As a result, “several universities of the region have developed complete collections of their journals on OJS platform” (UNESCO, 2017).

Moreover, some representatives of the organisations mentioned above have pushed national policies of research dissemination, based on the principle of science as a common good. Leading scholars working for those projects have successfully influenced the legislation of governments in Argentina, Mexico and Peru between 2013 and 2014 to include open access.

Despite all these endeavours, most national systems of evaluation of research in Latin America remain built on the simplistic assumption that researchers must publish in ‘international’, ‘top-ranked’, ‘high Index Factor’ journals to demonstrate the quality of their work (Alperin & Rozemblum, 2017; Vasen & Lujano Vilchis, 2017). This is reflective of researchers in Anglophone Africa, Europe and North America. This has created a paradox that Alperin and Fischman (2015) identified. On the one hand, most of the governments in Latin American countries invest resources, directly or indirectly, in creating and improving peer reviewed journals, according to internationally recognised quality standards. But, on the other, these publications fail to receive recognition of being as valuable as mainstream journals in the research evaluation system. This is illustrated in the case of disciplines such as natural sciences or medicine, where national policies of evaluation of research explicitly require authors to publish in journals indexed in Science Citation Index (SCI). Latin American journals continue to be underrepresented in these domains.

The ‘Latin American circuit of scientific journals’ has successfully boosted the dissemination of documents. However, “it has not yet been able to offer regional indicators of the published ST production for its valorization in evaluation processes” (Beigel, 2019, p. 1). In fact, national systems of evaluation in Argentina (Conicet), Brazil (Qualis/Capes), Colombia (Publindex) and Mexico (Conacyt) rely on the data from the Journal Citations Report, Scimago Journal Rank, and, in some cases, Google Scholar metrics such as h5-index.

Despite these hurdles, some initiatives are being developed to tackle the inequalities within evaluation systems based on ‘international’ citation indices. For instance, Beigel (2018) has proposed the creation of an ‘anti-ranking’ system to measure scientific production ‘in the periphery’. She has worked on a set of research evaluation indicators “to break the vicious circle that commercializes evaluative cultures” (Beigel, 2019, p. 2). The purpose of her work is to provide ‘circulation indicators’ for open access, peer reviewed articles published in the Latin American circuit of journals, to reach policy making processes at national and institutional levels.

Recently, some countries in Latin America are moving towards reforms of their scientific evaluation systems, especially in the social sciences and humanities. The rationale underpinning the reform is the rights perspective (right to science, education, and information) and the reaffirmation of the discourse on knowledge and science as common goods (CLACSO-CONACTY, 2019). The reflections, expressed in a meeting on scientific evaluation held in 2019 by the Latin American Council of Social Sciences (CLACSO) and Mexico’s National Council of Science Technology (CONACYT) of Mexico, suggest that there is political willingness to generate an evaluation scheme at the regional revel. The proposal seeks to include qualitative evaluation criteria, as well as diversity of outputs of academic work, such as “artistic production, social impact, outreach, and contributions to public debate” (CLACSO-CONACTY, 2019). This could be a good opportunity to counteract some trends in science policy that have reduced the relevance of the Latin American circuits of knowledge circulation.

5 The Netherlands

The dominant methods of evaluating research in the Netherlands has long been based on the number of articles published in high ranked (Scopus, Web of Science) journals. In other words, evaluation was based on citation analysis performed by for-profit companies, including major publishers such as Elsevier.

Now that OA is rapidly becoming the norm, with official government policy demanding 100% open access by 2020, standards regarding assessment of scientists are also changing. The major Dutch research funder, NWO,6 is going to implement a policy of evaluating research based on the quality of published articles, not on where these are published. The number of articles published is also less relevant than the importance of the individual articles.

According to VSNU,7 NWO (Netherlands Organisation for Scientific Research) and other academic organisations in the Netherlands, the quest for a new approach that recognises and rewards academics has begun.8 NWO and ZonMw9 signed the San Francisco Declaration on Research Assessment (DORA) in 2019. Furthermore, NWO will look for ways to increase the weight of research quality and anticipated impact in its evaluation of researchers and proposals. This process will be done in consultation with ZonMw, the VSNU and academics. The latest conference took place on November 15, 2019 in Rotterdam, the Netherlands under the title ‘‘Room for everyone’s talent: towards a new balance in the recognition and rewards of academics’’.10

The conference envisaged a renewal in three areas:

  1. Differentiation of career pathways: Universities and University Medical Centres want to provide academic staff with a choice for specific focus areas – teaching, research, knowledge transfer and/or leadership.
  2. Renewal of research assessment methodologies: New approaches to evaluating research quality and impact are emerging. The promotion of open science is integral to this development.
  3. Team science: Alongside recognition and reward for individual accomplishments, there is a push to award the collaborative efforts and accomplishments of teams with the consideration that they deserve.

The main obstacles to be overcome in the Netherlands are human attitudes and habits, alongside publisher monopolies and lobbying activities. A great deal of hope exists concerning the attitudes and habits of researchers, given the long tradition of connecting science and society through science shops in the Netherlands. Socially relevant research has helped develop this goal through a major network called the Living Knowledge Network. Since 2000, conferences have been organised by bringing together all those involved in doing or supporting research with and for communities.11

This is combined with promoting open science and publishing in open access avenues.12 However, many individual scientists, especially ones with tenured jobs, still value publishing in Scopus or Web of Science ranked journals rather than in open access journals. These same scientists are pressuring the younger generation to publish in the sphere of the knowledge economy, even if there are promising signs of a changing attitude among policy makers. Dutch universities are increasingly negotiating journal article subscriptions deals on their terms with major publishers, and publishers are being forced to change tactics because of large scale subscription cancellations.

The main danger lies in the possibility that big publishers will maintain their grip on scholarly publishing by monopolising Open Access in the same way that they managed to monopolise subscription publishing. The emergence of so-called mirror journals (fully open access journals with the same editorial board and infrastructure as a subscription counterpart) is a scheme to ensure high income from publishing, in the event that subscription journals disappear. Despite their intention, the Coalition S instigated Publishing Agreements (Read and Publish) could, instead of making open access a cheaper publishing reality, make publishing even more expensive for the community.

6 Conclusion

These four examples show that all regions still suffer from an under-valuation of locally produced knowledge, mainly because the value of research is seen as being dependent on it being published in Northern journals, ranked through a system that is controlled by commercial companies owned by scientific publishers and linked industries.

The way out is demonstrated to be to:

  1. Publish in locally owned journals and platforms

  2. Publish with not-for-profit organisations and publishing companies

  3. Evaluate all knowledge outputs, with far more emphasis on criteria other than mere citations, such as social relevance, impact on the UN SDGs, and relevance to solving local, regional and global problems in health, agriculture and other fields.

Acknowledgement

The authors gratefully acknowledge Paul Gregory Murphy for the revision of their English.

Notes

1

SOHA means Science ouverte en Haïti et en Afrique – Open Science in Haiti and Africa. It is a long-term research action project led by Florence Piron.

2

For example, the CAMES launched its institutional repository in 2019 at https://savoirs.c3mes.online

4

DHET has included the complete set of DOAJ listed journals in Janunary 2021. See https://bit.ly/3sUsOfn

5

A new continental platform for the open access publishing of journals, monographs and textbooks in Africa has been developed by South Africa’s University of Cape Town (UCT) through its library service. See https://bit.ly/3kKM5wO

6

Netherlands Organisation for Scientific Research.

7

Association of Universities in the Netherlands.

9

Netherlands Organisation for Health Research and Development.

12

In February 2021 an Open Access platform for Dutch academic journals was launched. See https://bit.ly/3kI77w5

References

  • African and Malagasy Council for Higher Education (Conseil africain et malgache pour l’enseignement supérieur). (2017). Guide for the evaluation of teacher-researchers and researchers (GEE-CC) within the framework of the Inter-African Consultative Committees (IACs) 2017–2021 (p. 60). CAMES.

    • Search Google Scholar
    • Export Citation
  • Alperin, J. P., & Fischman, G. E. (2015). Scientific journals made in Latin America. In Made in Latin America: Open access, scholarly journals, and regional innovations. Consejo Latinoamericano de Ciencias Sociales.

    • Search Google Scholar
    • Export Citation
  • Alperin, J. P., & Rozemblum, C. (2017). La reinterpretación de visibilidad y calidad en las nuevas políticas de evaluación de revistas científicas. Revista Interamericana de Bibliotecología, 40(3). http://www.memoria.fahce.unlp.edu.ar/library?a=d&c=arti&d=Jpr7928

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Beigel, M. F. (2018). Las relaciones de poder en la ciencia mundial: Un anti-ranking para conocer la ciencia producida en la periferia. http://ri.conicet.gov.ar/handle/11336/94914

    • Search Google Scholar
    • Export Citation
  • Beigel, M. F. (2019). Indicadores de circulación: Ciencia, tecnología y política, 2(3), 028028. https://doi.org/10.24215/26183188e028

    • Search Google Scholar
    • Export Citation
  • Cetto, A. M., & Alonso-Gamboa, O. (1998). Scientific periodicals in Latin America and the Caribbean: A global perspective. Interciencia, 23(2).

    • Search Google Scholar
    • Export Citation
  • CLACSO (Latin American Council of Social Sciences) & CONACYT (National Council of Science and Technology). (2019). Latin American Forum on Scientific Evaluation Rapporteurship. https://www.clacso.org/actividad/relatoria-foro-latinoamericano-sobre-evaluacion-cientifica/

    • Search Google Scholar
    • Export Citation
  • Glick, T. F. (1991). Science and Independence in Latin America (with special reference to New Granada). The Hispanic American Historical Review, 71(2), 307334. https://doi.org/10.2307/2515643

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Grenier des saviors. (2020). Les revues du Grenier des savoirs prennent leur envol [Blog post]. Science Afrique. https://www.scienceafrique.org/blog/2020/03/02/les-revues-du-grenier-des-savoirs-prennent-leur-envol/

    • Search Google Scholar
    • Export Citation
  • Piron, F., Diouf, A. B., Dibounje Madiba, M. S., Mboa Nkoudou, T. H., Ouangré, Z. A., Tessy, D., Achaffert, H. R., Pierre, A., & Liré, Z. (2017). Open access as seen from Sub-Saharan French speaking Africa. Revue française des sciences de l’information et de la communication. Online. https://doi.org/10.4000/rfsic.3292

    • Search Google Scholar
    • Export Citation
  • Piron, F., et al. (2021). Building science shops in Africa to fight cognitive injustices. Manuscript.

  • UNESCO. (2017). PKP – Public Knowledge Project. http://www.unesco.org/new/en/communication-and-information/portals-and-platforms/goap/key-organizations/latin-america-and-the-caribbean/pkp/

    • Search Google Scholar
    • Export Citation
  • Vasen, F., & Lujano Vilchis, I. (2017). Sistemas nacionales de clasificación de revistas científicas en América Latina: Tendencias recientes e implicaciones para la evaluación académica en ciencias sociales. Revista Mexicana de Ciencias Políticas y Sociales, 62(231). http://dx.doi.org/10.1016/S0185-1918(17)30043-0

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Collapse
  • Expand

Socially Responsible Higher Education

International Perspectives on Knowledge Democracy

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 962 83 10
PDF Views & Downloads 745 72 6