Czech Academy of Sciences Evaluation – Beyond Metrics, Accent on Peer Review

A total of 408 teams from 52 CAS institutes and 3,841 research outputs – those were the parameters of a large-scale evaluation of the CAS research institutes recently completed by the Czech Academy of Sciences in collaboration with nearly two thousand international experts. How did the evaluation unfold, and what did it reveal? Patrik Španěl from the Academy Council of the CAS explains below.

Patrik Španěl has taken part in previous rounds of the internal evaluation of the Czech Academy of Sciences (CAS) in a range of roles – as a team leader, Deputy Director of the J. Heyrovský Institute of Physical Chemistry of the CAS, and Chair of the institute’s board. This has given him an in-depth understanding of both the principles and procedures and how they have evolved over the past twenty years.

*

Do you have any experience with evaluations at other institutions?

Yes – from the UK’s Research Excellence Framework, which I took part in as a professor at Keele University. I have also completed hundreds of peer-review assessments of studies, dissertations, and grant proposals – both Czech and international, including for the European Research Council. Over the past year, I have also experienced the role of evaluator at the University of Basel.

Why does the Academy conduct its internal evaluation regularly, once every five years?

The purpose of the evaluation is to protect our investments in science, support the quality and integrity of research, and ensure research is conducted in an ethically responsible manner. That is why we apply an evaluation model based on international peer review, one that takes multiple criteria into account and respects disciplinary context. It is also aligned with the principles of CoARA and DORA, to which the Czech Academy of Sciences subscribes.

Patrik Španěl from the Academy Council of the CAS.

How does the evaluation reflect these principles?

Primarily in the fact that results were assessed through independent expert judgment rather than bibliometric indicators. Looking ahead, we should further consider how to better separate the evaluation of research contribution from the evaluation of publications – publications are a means of communicating science, not the ultimate purpose of research.

In the Czech Republic, however, we are used to bibliometrics.

I consider bibliometrics an outdated way of evaluating science – it often relies on proxy indicators: the journal, citation counts, rankings, quartiles. But the quality of a paper is not determined by a journal’s Article Influence Score, and the quality of a research team cannot be judged by the number of publications. Goodhart’s law applies here: when a measure becomes a target, it ceases to be a good measure, and the system begins to adapt to it.

What are the strengths of peer review?

Peer review assesses the actual substance and context of scientific work. It does not simply look at how many outputs were produced or where they were published – it asks about their quality, originality, significance, and methodological rigor. That is why peer review is irreplaceable.

When did preparations begin for the evaluation covering 2020–2024?

Preparations were already underway under the previous Academy Council, which established the foundations and methodology which we then followed. The then Coordination Board for the Evaluation also played a key role.

What was its role?

It addressed procedural matters. Initially, it helped select suitable experts for the Academy Council, assessed potential conflicts of interest, and issued recommendations on how to submit research teams for evaluation. Once the new Academy Council was elected in March 2025, the Coordination Board was re-established. In the subsequent phase, its work again focused on conflicts of interest and on assessing the legitimacy of objections raised by CAS institutes regarding first-phase reports and the final reports of the evaluation panels.

How did this evaluation differ from the previous one?

There was a shift toward the actual purpose of research itself – its outputs, context, and long-term contribution. Greater respect was given to the focus of the CAS institutes and their specificity in regard to individual scientific fields and specializations. This is why 52 international panels were established – one for each CAS research institute. We also moved away from proxy metrics, separated the criteria for evaluating research teams and institutes, and shifted applied research results to the second phase, where they were assessed in context via case studies. In short, the evaluation put emphasis on expert judgment, more in-depth feedback, the abandonment of proxy metrics, and greater transparency.

One of the guiding principles was also ‘formative character.’ What does that mean exactly?

It means that the evaluation was not intended simply to rank CAS institutes or place them on a scale. The main goal was to provide qualified, actionable feedback – identifying strengths, weaknesses, risks, and opportunities for further development.

You mentioned that each CAS institute had its own evaluation panel. Did you also tailor the evaluation by scientific field?

Yes, but not in the sense of applying different standards. Our aim was to ensure that quality would be assessed by people who truly understand the field and the specializations. Panels therefore included experts familiarized with the CAS institute’s areas of research, capable of assessing them in both field-specific and interdisciplinary contexts.

How many researchers were involved?

A total of 341 experts served on the international panels, most frequently from Germany, France, and the UK. The same international panel handled both phases of the evaluation, ensuring consistency. The difference lay in the role of additional evaluators: 1,552 remote external evaluators participated only in the first phase, assessing selected outputs. In the second phase, only the evaluation panels were involved.

Let’s look more closely at the evaluation phases. What was the goal of the first phase?

Research-oriented institutes submitted 3,941 selected outputs from 408 teams. These were assessed by remote evaluators with the aim of obtaining expert judgments on the quality of the results.

Was it difficult for the panels to reach agreement on the final assessment?

It was challenging, not least because we did not rely on shorthand metrics such as a journal’s Article Influence Score. Each output was assessed as a concrete piece of work – in terms of what it contributes, how original it is, how significant it is, and how methodologically rigorous it is. Typically, each output was reviewed by two remote evaluators, and the panels then worked with their reports. In roughly half of all cases, the two evaluators’ assessments were in agreement.

And when they disagreed?

Differences had to be interpreted in context. In peer-review assessments, this is entirely natural – disagreement does not necessarily indicate an error; it often reflects differing emphases on originality, significance, or rigor. In more complex or borderline cases, informed discussion and expert consensus are more valuable than any numerical indicator.

What were the results of the first-phase evaluation?

They showed a very high level of quality: 17% of outputs – more than 600 – were rated Outstanding, 53% Excellent, 27% Standard, 2% Acceptable, and 0.1% Unsatisfactory.

What do these grades mean exactly?

‘Outstanding’ denotes an exceptional result that advances existing knowledge and is characterized by originality, significance, and methodological rigor. ‘Excellent’ refers to a highly valuable result with a clear scientific contribution. ‘Standard’ indicates a solid, well-executed piece of research. ‘Acceptable’ denotes an output with more limited contribution, and ‘Unsatisfactory’ refers to work that does not meet the expected scientific standard.

Which results performed best?

I would prefer not to turn the evaluation into a laundry list of top ten publications – that is what we sought to avoid. The aim was not to identify “the best paper from the Czech Academy of Sciences,” but to assess the quality of a representative set of outputs within their field-specific context.

Distribution of CAS research outputs rated on originality, significance, methodological rigor, and scientific contribution.

Let’s move on to the second phase of the evaluation.

During the second phase, the panels assessed not only research outputs but entire teams and institutes as a whole. They worked with written materials and reports from the first phase, and also took part in site visits and discussions with leadership, teams, students, and early-career researchers. The second phase therefore examined not only scientific quality, but also societal relevance, quality of management, and the strategic vision of each CAS institute.

What can you learn during a site visit that you cannot get from documents?

On site, you can feel out the atmosphere – you see how leadership and teams communicate, how they work with early-career researchers, how effectively infrastructure is used, and whether a development strategy exists only on paper or is actually reflected in leadership practice. Dialogue is also crucial. The panels can verify how CAS institutes communicate their set priorities, how they respond to questions, how they perceive their own strengths and weaknesses, and how they think about the future. In this sense, the visit is not merely a formality – it is an integral part of peer review.

When did the site visits take place?

In September and October 2025 – depending on the size of the CAS institute, they lasted between one and three days.

The panel had thirty minutes allotted for each research team in person. Isn’t that too little?

This was not an isolated half-hour in which the panel was expected to learn everything all at once. The meetings were meant to verify and refine the picture the panel had already formed. In a well-prepared evaluation, a focused discussion can be highly effective.

What was the outcome of the second phase of the evaluation?

The main output – and thus result – consists of the final reports, which include evaluations of both the research teams and institutes, along with recommendations for their further development. At the same time, the reports place the performance of teams and institutes within an international context in their respective fields.

How is the Czech Academy of Sciences planning to use the evaluation results overall?

They will serve as a basis for strategic management and for setting institutional funding for the development of CAS institutes for 2027–2031. Discussions with individual CAS institutes will take place in September and October 2026. However, the results are not translated into funding mechanically.

How should CAS institutes work with the results?

They should use them to shape or refine their strategies. The evaluation is intended to help CAS institutes reinforce strong directions, identify weaknesses, and develop plans for further growth. The results will be published at the end of the year, making them accessible to the wider scientific community.

What will happen to CAS institutes or teams that receive unfavorable evaluations?

If the evaluation identifies any weaknesses, this is not a sanction, but a basis for change. CAS institutes should reflect on which observations are most important for them and how to move forward after taking them into account. In development discussions, we want to focus on how CAS institutes respond to the recommendations, what their strategy and aim is, and how they plan to achieve their goals.

Did the evaluation also reveal broader shortcomings within the Academy?

Indirectly, yes. For instance, it highlighted how strongly funding and evaluation systems shape institutional behavior. If the system is overly tied to metrics, short-term outputs, and project pressure, it narrows the space for long-term development, new teams, and higher-risk research directions. This is not just an issue for the Czech Academy of Sciences alone, but more broadly for Czech science policy as a whole.

What do you mean by that?

We need greater trust in high-quality evaluation, more room for long-term research programs, stable and predictable institutional funding, and less mechanical reliance on metrics. Research cannot be effectively managed via spreadsheets alone. It is also clear that, for example, the situation in which PhD students from universities conduct their research at institutes of the Czech Academy of Sciences is not standard internationally and is seen by foreign experts as a certain issue.

How did the international experts perceive the Academy and its internal evaluation?

We are still analyzing their feedback. From what we know so far, however, it was a positive experience for them. Some have already expressed interest in working with us on future evaluations, which we greatly appreciate. Recruiting high-quality international experts is no easy feat – they have to understand the specifics of the Czech Academy of Sciences, the Czech research environment, and the evaluation methodology, while also remaining independent. High-quality experts, however, are essential for the credibility of peer review.

When will preparations begin for the next evaluation round?

Immediately following reflection on the completed one. We are currently gathering input from CAS institutes, observers, experts from the evaluation panels, and the organizational team. A summary of this feedback will be presented to the Academy Council in early 2027.

Are you already planning any changes?

The next round should shift the focus even further away from research outputs toward contributions to knowledge, research programs, team culture, and long-term impact. Publications will remain important, but they must not become an end in themselves. In the age of AI – when it will become increasingly easy to convincingly imitate useful results with fabricated texts – it is even more important to ask where genuine human scientific contribution lies.

We are nearly halfway through the next five-year evaluation period, yet the results of the current one are only now available. Will CAS institutes have enough time to respond to the recommendations?

Some changes – for example in management, strategy, or work with early-career researchers – can begin immediately. Other changes will only become visible over a longer time frame. But evaluation is not a snapshot; it does not assess a state at a single moment in time. It takes into account the developmental trajectory of CAS institutes and their ability to respond to recommendations.

What would you highlight in conclusion?

I would like to thank everyone involved in the evaluation. This expert, formative assessment based on peer review has provided us with high-quality feedback. I believe we are setting an example for others. The evaluation of the Czech Academy of Sciences truly meets international standards.

Prepared by: Zuzana Dupalová and Luděk Svoboda, External Relations Division, CAO of the CAS
Translated by: Tereza Novická, External Relations Division, CAO of the CAS
Photo: Jana Plavec, External Relations Division, CAO of the CAS

 The text is released for use under the Creative Commons license.

Read also

• Ordovician Fauna Beneath Prague – New Arthropod Found During Metro Line Dig
• Wildfires on the Rise in Europe and Set to Double by 2100. What Can We Do?
• The CAS Photogenic Science Contest Invites Entries
• The CAS Launches New Research Programs as Part of Its Strategy AV21
• Academy Assembly – Thirteen Priorities to Boost Basic and Applied Research
• We Must Prioritize Science, Czech President Petr Pavel Says on His Academy Visit
• Radomír Pánek Visits CAS Institutes, Aiming to Strengthen the Academy’s Unity
• Small but Mighty – The Enceladus Moon Is Flexing with Saturn’s Magnetic Field
• Light vs. Antibiotics – Can We Get Harmful Substances Out of Our Water?
• Better ECGs and Industrial Superlasers – Real-World Results of CAS Research