KRUNOSLAV CAPAK, director of the Croatian Institute of Public Health, announced today that serological tests which could provide an answer to the question how many people in Croatia came into contact with the virus are underway.

"The Teaching Institute of Public Health 'Dr. Andrija Stampar' managed to procure the tests, and they are the ones to be contacted for any further information. As far as the CIPH is concerned, we received the first tranche of tests, and the validation ended yesterday. I was informed that the tests are sensitive and fairly reliable. The testing can begin, "Capak explained.

What could testing be good for?

Serological tests, which reveal who has developed antibodies to the SARS-CoV-2 virus, should show who has been infected with COVID-19, regardless of whether they had symptoms or not. They have been recognized in the world as very important and multi-purpose.

First of all, they should reveal how many people in a country have recovered from COVID-19 and how many infected people went under the radar. This would allow experts to have a clearer idea of ​​how deadly the disease is, that is, what its IFR is (the ratio of deaths divided by the number of actual infections). Since it is estimated that somewhere between 30 and 80% of those infected do not develop any symptoms, it is expected that the IFR could be significantly lower than the CFR, the ratio of deaths divided by the number of confirmed infections.

Another important purpose of testing is to assess the ratio of people in the population who might have developed immunity – since they have antibodies. That way, one could get some idea of how far the population of a country or a part of it is from collective immunity. Since COVID-19 is a highly contagious disease, more contagious than influenza (R0 = 1.4 – 5.7), about 70% of the population should get infected in order for the collective immunity to be acquired.  

The third useful purpose of the test which was advertised was to help normalize life. Namely, people who test positive would know that they are immune to COVID-19, and they could get back to normal life. Positive test results would serve as a sort of a passport that would allow for unhindered interaction with others, both healthy people, and those who may be ill. This would especially be useful to medical professionals.  

Finally, the tests could be used in the future to see if developed vaccines create immunity in clinical trial subjects.

The race to produce tests

Considering the great potential of serological testing in this state of the pandemic, dozens of companies around the world are urgently developing various tests and launching them on the market.

The British Prime Minister Boris Johnson even called them a turning point in the pandemic. However, it turned out that their potential was overestimated, with the problems pertaining to them being underestimated in several different ways.

One of the problems is that they have not yet shown sufficient reliability for all of the above purposes.

Furthermore, there is no unequivocal confirmation that the existence of antibodies entails protection from the disease. It is also not clear for how long such a protection lasts, if it even exists.

Finally, due to all of the above, it has become very questionable whether it is ethical to tell someone that they are positive, which could be interpreted as getting the green light to get back to normal.

The UK has learned this lesson the hard way after the 3.5 million tests it recently commissioned from several companies proved insufficiently reliable.

Numerous experts have therefore expressed serious doubts concerning the very purpose of such testing.

"No test is better than a bad test," says Michael Busch, the director of the Vitalant Research Institute in San Francisco.

Estimates of the possible number of infected individuals in Croatia

Despite these doubts, the Czech Republic recently conducted massive serological testing which, quite expectedly, showed that very few people have had contact with the virus in that country. On a sample of 26,549 people, 107 positives were identified. Their estimates are that no more than 4 to 5% of the population is infected. In the Czech Republic, 7,899 people have been confirmed as infected, and 258 have died so far. According to the same study, the number of people infected with the virus who show no symptoms could range from 27% to 38%.

Rough estimate of infected individuals in Croatia

Unsurprisingly, people living in Croatia would like to know the answer to the question from the title.

One way of estimating the said number is based on the number of confirmed cases in Croatia, which currently amounts to 2112. The number of symptomatic cases that epidemiologists missed should also be added to this number – according to the London School of Hygiene and Tropical Medicine, that number amounts to approximately two thirds. Finally, asymptomatic cases should be added to all symptomatic cases – estimates of their share range from 31% up to 88%, with the mean value being around 56%.

According to these very rough estimates (supposing that only 30% of symptomatic cases were recorded, as well as that about 60% of the asymptomatic ones should be added), one can conclude that around 11,000 people in Croatia have been infected so far – about 0.26% of the population.

As we can see, we are still very far from a collective immunity, whose threshold amounts to at least 70%.

The problem of test unreliability

The idea is that serological testing could give much more reliable estimates than the above rough estimates. But there are some problems here – namely, even very high-quality serological tests can provide unreliable results.

Martin Pozar, an expert in statistics and data processing, provided a detailed explanation of why that is the case on his blog.

He points out that the reliability or unreliability of estimates based on serological tests stems from the reliability level of the results of serological tests, as well as from the size and representativeness of the sample of subjects. Simply put, if the tests are unreliable or if the sample is poorly selected, the results will be unreliable as well.

"The reliability of serological test results is expressed by the sensitivity and specificity of the test. The test is 100% sensitive if it can successfully detect the presence of antibodies in all cases when they are indeed present, and 100% specific if a negative test result always points to the actual absence of antibodies. The results of serological testing, however, are generally not as reliable. On April 23, the British Center for Epidemic Analysis and Modeling released this report, providing an overview of the reliability of different currently available serological tests for the SARS-CoV-2 virus, which shows that their sensitivity ranges widely, from approximately 50% to 90%, while their specificity ranges from 60% to almost 100%," writes Pozar.

The fewer infected, the greater the error of tests

It is important to emphasize that the uncertainty of the estimate arising from the unreliability of tests is significantly more pronounced in cases when the actual number of infected people in the total population is low, as is the case in Croatia.

"For example, using a test that has a relatively high specificity (98%), it is to be expected that the result of false-positives will amount to 2% even when there are no infected people in the population at all. When the actual situation is such that there are less than 2% of infected individuals in the population, false-positive results of this serological test can lead to a significant overestimation of their number. Based on the previous simple estimate, it is expected that the number of infected individuals in Croatia is very small, only about 0.26%. Therefore, there is a high probability of overestimating the number of people who are actually infected, even when using a serological test that has a relatively high reliability," says our statistician.

What results could testing in Croatia provide?

Our experts announced that serological testing on the general population in Croatia was to be conducted on a sample of 1100 subjects.

Pozar points out that, assuming that 0.2% of people in Croatia are actually infected, about two positive individuals should appear in a sample of 1,100 subjects. However, even if the test was about 98% specific, an additional 22 false-positives may appear – 2% of false positives on a sample of 1100 amounts to 22. This shows that even very reliable serological tests can have unreliable results.

There is also a problem of sample representativeness, and the method of subject selection is very important. For example, all regions should be evenly represented in the sample based on the number of inhabitants, regardless of how many people were infected in each individual region. If the sample is selected unevenly, there can be an overrepresentation of subjects from a region where the infection rate was higher or lower than average. Furthermore, if the volunteers for the testing are found online, one could expect that those who have recently had some respiratory issues would be more likely to respond, namely out of curiosity, which could increase the ratio of those who have actually recovered from COVID-19.

In his blog, Pozar illustrates how high the uncertainty degree pertaining to the research conducted in Croatia could be.

"For example, with 5% of positive results and specificity of the test amounting to 98%, we could conclude that between 0.6% and 4.8% of people were infected, i.e., between 20,000 and 200,000 individuals," says Pozar.

Simply put, the range would be so large that, if the testing was conducted in Croatia, we would only be able to conclude that we have relatively few infected, but little more could be done.

The higher the number of infected, the higher the accuracy

Test result accuracy increases significantly with the ratio of infected individuals in the population.

"There are handy online calculators that can confirm this. For example, MEDCALC shows that, with a test which is 100% sensitive and 98% specific, and a disease prevalence of only 0.2%, which might be expected in Croatia, the PPV of the test amounts to approximately 9%. This means that the probability that someone who has a positive test is actually infected amounts to only 9%. On the other hand, all those who are negative on the test will actually be negative, i.e., not infected. With a disease prevalence of about 20%, the PPV increases to approximately 90%. The problem is that we don't know what the actual prevalence is. That is exactly what we are trying to determine," says Pozar.

Since Capak said that the tests cost around HRK 40,000-50,000, conducting a nationally representative field research could end up costing more than HRK 100,000. That may not be a large sum, but if the result were definitely unusable, it would be a waste of money.

Testing will ultimately serve a purpose

However, our epidemiologist Branko Kolaric from the Faculty of Medicine in Rijeka says that research, in order to be useful, should not necessarily be done on a sample from the entire population, i.e., on a representative sample, but instead, it can be done on groups at increased risk and on those at lower risk, which is called Sentinel sampling.

"This type of testing will show us the prevalence of the virus in specific groups, among people in nursing homes, the elderly, voluntary blood donors, pregnant women, etc. We expect to find a higher prevalence of the virus among some of these groups and a lower prevalence in others relative to the general population. As a result, we will also have a higher predictive value of the tests. Finally, in addition to these tests, we will conduct additional confirmatory tests in order to make sure if those who tested positive on serological tests were actually infected. This, along with appropriate modeling, will give us a clearer picture of the situation in the entire population," says Kolaric.

Slovenes are more transparent

Pozar says that Slovenes are conducting their research in two waves.

"In the first (current) wave, they are using serological tests and the RT-PCR method (picture above), while in six months, during the second wave, they will conduct only a serological test on the same subjects. They planned a sample of 3,000 people, sent calls, and received an answer from about 1,400 subjects," says our statistician.

It is interesting and important to mention that Slovenes published basic data regarding their research quite transparently. According to the results they published today, there are 41 serologically positive subjects, which amounts to about 3% of the sample. However, it is not evident from their research if they made the necessary corrections, nor is it clear how reliable their tests were.

"I hope that our test results will be published transparently as well. All information about the methodology and tests should be published in order for one to be able to assess the quality of the results. Ideally, a clear description of the methodology and sample should be published, and so should a complete research data framework, the computer code used for processing and, of course, data regarding the manufacturer and the reliability of the tests. The research cannot be deemed credible without this information, not only because of the possibly controversial methodological choices, but also because it would be unknown what choices were made," says Pozar.

When will we have reliable tests?

The following question arises: when will we have tests that would provide us with significantly more reliable data, able to create a clear picture of the number of infected?

Vladimir Trkulja, a full professor of pharmacology at the Faculty of Medicine's Department of Pharmacology, says that the answer to this question depends on several factors, most of which are yet unknown.

"For example, if in the near future, in about four to six months, there is a new, highly successful treatment strategy – one or a combination of several treatment options or prevention strategies, a successful vaccine, or if it is shown that the pandemic is, so to speak, naturally withdrawing, the efforts aimed at developing these types of tests could be somewhat decreased, since the matter will no longer be as important," says Trkulja.

"If the situation does not change, the efforts will certainly continue in that direction, and significant progress could be expected within the same period, in about four to six months. This progress, however, will not fix the above-mentioned problems. The test may become more reliable, but if the antibodies are short-lived and do not guarantee an actual longer-term immunity, it will have little influence on everyday life. If there was a vaccine, there would be no problems even in case of just short-term immunity – we could be vaccinated again, for example every year or so, as is the case with the flu. Therefore, it is very likely that significantly improved tests could appear by the end of the year," says our pharmacologist, noting that the role of such tests will depend on other factors.

He warns that only the tests with a specificity of the order of 99.99%, as is the case with laboratory tests for HIV, can be considered reliable in such a way that, no matter what the ratio of infected people in the population is, even if it amounts to only 1%, can have the so-called positive predictive value close to 100% with proper testing.

"This means that, when a person of unknown status has a positive result on the test, they actually have antibodies to COVID-19 of a certain class, in a certain amount," says Trkulja.

What are these tests good for?

Nevertheless, he believes that despite all limitations, existing serological tests can be useful for a variety of purposes.

"Not really in the way that is most often mentioned, but they can be applied in other ways.

For example, let's take the people who have been suspected of being infected with COVID-19 over the past two months as an example. They had a mild fever and some other mild symptoms, and at an early stage, they once underwent a diagnostic test that was negative. This is, of course, possible due to the properties of the test, due to possible problems regarding sampling, etc. The diagnostic test was not repeated, and those people now feel well and have no symptoms. It is realistic to assume that the actual ratio of those who were actually infected with COVID-19 among such people is high, let's say about 50%. A serological test with a specificity of 98%, cited here as an example of a somewhat problematic test, would result in a positive predictive value of 98% in such a population. This means that we could be 98% certain that a person actually recovered from COVID-19 if the test result was positive, and 100% sure that they did not if the test was negative. Thus, existing serological tests have a purpose. The key is how they are used. Also, combined with the so-called RNA diagnostic tests, they could help diagnose unclear cases. But in the case of mass testing aimed at assessing the rate of infection in the population, they would not be beneficial, that is, they would not provide reliable and important information," concludes Trkulja.