Testing Vaccine Efficacy Against COVID

By Kashif Peshimam

Introduction

Dealing with COVID has been a big part of daily life for about one and a half years now. After extensive quarantines, social distancing, mask mandates, and more, we finally have vaccines going around. But this makes us wonder, how are the vaccines making an impact on the spread of the coronavirus? With many people tired of remote working and other changes to our lives, some optimism can be gained if things look like they are getting better. Now that vaccines have been going around for a few months, we have a sizable amount of data to try and make a conclusion on the efficacy of the vaccines.

In this tutorial we are going to walk through the data science process when it comes to Data Collection, Data Management, Exploratory Data Analysis, Hypothesis Testing, Machine Learning, and Conclusions we can draw.

We are going to be analyzing data from two sources. The first is the data on vaccinations by country, and can be found at https://www.kaggle.com/gpreda/covid-world-vaccination-progress?select=country_vaccinations.csv.

The second dataset is daily covid data by country which includes variables like number of cases, deaths, and so on. The second dataset can be found at https://www.kaggle.com/josephassaker/covid19-global-dataset?select=worldometer_coronavirus_daily_data.csv.

The first step to our analysis is data collection. We have our datasets to take from, but first we need to import the necessary libraries.

1. Data Collection

Now we need to read in the data.

When we read in the data, we are also going to create a deep copy of the dataframes we create in order to have an original copy should we need to go back to refer to something after altering the dataframes for analysis.

2. Data Management

As can be seen, the first table on vaccinations starts from February 2021 while the second table on daily data starts out in 2020. Since we want to be comparing the vaccine efficacy to variables like daily new cases, we are going to filter out rows for dates we don't need.

Another important thing to realize is that upon inspection of the country names, they differ in their naming. For example, one table might use 'and' the other table may use 'And'. In order to avoid issues with this, we are going to capitalize the whole country name for each row in each table.

Now that we have filtered out rows containing dates that cannot correspond to the vaccinations data table, we need to filter out the dates that do not correspond to the daily data table from the vaccinations table. In this case, it is just one day, 2021-5-12. So all we need to do is go through and drop rows for that particular day.

Note that we also need to capitalize the country names here too.

As can be observed from checking the number of rows for each of the newly filtered tables, we see that they do not match. This could be because of an inconsistency in the number of countries. We can count the number of countries to make sure they match. If there is not a match, we can drop the countries that do not appear in both tables.

As we suspected, there is an inconsistency in the number of countries. Therefore we can now try to find out which countries we need to drop, and drop them.

It seems as though we have encountered a large naming convention disconnect between the two tables. Although we know that the outputted length of countries to drop is 75, since most countries appear twice in the printed country names but with slightly different names, about 37 countries are not synchronized out of about 210.

Based on this, one option we have is to go ahead with about 180 countries for the vaccine efficacy analysis and still get decent results. Another option is to select major countries from this list, and fix the naming coventions so that we can ahead with the analysis with a few more countries. A final option would be to manually fix each and every one of the names which would take a long time and be inefficient. How about you take a moment and consider which of these options is the best.

If you have all the time in the world, you may think that the final option is the best. However, the second option most likely the most efficient route and the one we will go. If you examine the country names you would see extremely relevant countries such as the United Kingdom and the United States. Adding these could be beneficial to our vaccine efficacy analysis. Something to also consider is that many of these countries to drop in question are islands or very small. So they may not reflect the general global trends and could end up being outliers. Another thing to note is that some of these countries may also be lagging behind in terms of giving adequate data either by censorship or poor data collection ability.

Based on this, we are going to take 'UNITED KINGDOM':'UK', 'UNITED STATES':'USA', 'VIETNAM':'VIET NAM', 'STATE OF PALESTINE':'PALESTINE', and 'NORTH MACEDONIA':'MACEDONIA'. So at this point, we need to go through both tables again and fix the names we have chosen and then drop the rest of the names we chose not to save.

Let's check if the rows are now in order.

It appears that there still is some disconnect with the number of rows in each table. This is likely due to a deeper issue with the number of dates for each country not being uniform throughout. However after this much data cleaning, there is enough of a organized format among the tables to explore the data because many values in the tables are NaN. NaN means that the the value is 'Not a Number,' and they are generally put or found in tables when the value is missing. By taking the averages for each country, we should still be able to explore the data and try to come to a conclusion about vaccine efficacy. All we need now is to make sure the countries are synchronized for both tables.

Now let's repeat the same code above to double check the country counts are the same for each table, and let's also double check that the tables now have the same countries.

As can be seen, the countries are the same in both tables, just out of order. We can now continue to the next phase of the data science pipeline, which is Exploratory Data Analysis or EDA.

3. Exploratory Data Analysis (EDA)

Now comes the exploratory data analysis. This is where we try to visualize the data and notice patterns that can help us in our question. It may cause us to go back to different steps of the data science process to tweak things as necessary. We may also gain a better understanding of what we should be testing.

At this point, it is important to realize that the data science process is not necessarily a linear process from data collection to conclusion. Most of the time, a big part of the process is going back and forth between different parts of the process. For example, if visualizing a variable is problematic due to outliers, then it may help to go back to the Data Management portion of the data science process to clean the data further.

With that out of the way, we know we want to find some relationship between the increase in vaccinations and the coronavirus weakening. To see if the data implies that, we can start off by visualizing the rates of daily cases against the rates of daily vaccininations per month. In order to do this, we will need to calcualte the rates for each of those variables.

Just a reminder, now is a good time to make sure that the presence of an NaN did not cause any country to be completely skipped from being included into the rate lists. Remember when we were gathering the daily vaccination data into a list, we skipped including a country for a particular row if there was an NaN value for the number of daily vaccinations.

We can examine the lengths of the two 2-tuple lists we created to see if there are any disconnects.

So it turns out that there were two countries that were left out. All we have to do now is find out which ones are included in the larger list (the list of case rates), and drop those two countries. It turns out that we marked the countries that were skipped, so we can use that to help us pinpoint which 2 countries to remove.

So now we know that the two countries are Tajikistan and Yemen, so we can drop them from the country_case_rate list.

Now we have equal rates lists for vaccinations and cases, so we can go ahead and plot. What we want to plot is the Rate of Daily Cases against the Rate of Daily Vaccinations.

That looks awfully hard to understand. As you probably have understood by now, data science involves a lot of trial and error. You generally are not going to get everything perfect on the first try. By the looks of this graph, there are about 5 outliers, 3 of which are very significant. It is a good idea to understand which countries are so significantly standing out. This way we can remove them possibly research the circumstances they may be in that may cause such a significant deviation from the main cluster.

In order to accomplish this, we can go through the list of rates for vaccinations, and label the outliers with extreme values.

We know the high vaccination rates for China, India, and the United States are very high because of their role in making the vaccines. We also know from the news that India has been going through an extremely hard time with COVID. With such a high population count of over 1 billion people, it is understandable how India can have such a high vaccination rate yet still seem to have the worst increase in cases. More can be read at https://www.wsj.com/articles/why-indias-second-covid-19-surge-is-much-worse-than-its-first-11621071001 to learn more about India's struggle.

Now that we know the significant outliers, we can remove them.

We can now attempt to plot the scatter plot of the Rate of Daily Cases against the Rate of Daily Vaccinations again as before.

This still does not look good when it comes to trying to establish find vaccine efficacy. Based on the cluster, it looks as if there is a almost somewhat of an increasing trend in rate of cases when it comes to an increase in the rate of vaccinations. One factor that could be changing the view of the data is that the rates need to be segmented for each country. Since this is data on a highly contagious virus, different countries have been jumping in and out of surges. Therefore perhaps it is wrong to look at countries' rates as one rate per country but rather it should be done with multiple rates per month. This may account for any jumps in surges so that the data may be spread apart.

Let's go ahead an calculate the rates per month.

Now let's make sure that unshared values between the two lists are removed, NaN values are removed, and outliers are removed.

This does not look that promising. Based on the cluster, it looks as though that the linear relationship wouldn't be too strong. It also may be wrong to look at rate of vaccinations, because at the beginning of the month there could be a large amount that have an impact against the spread of COVID, yet towards the end of a month there could be view, and thereby showing a negative rate for the month. It may be better to instead have the cumulative vaccinations per hundred people per month instead. We can compare against the rate of cases still, because this could tell us how the spread of COVID is progressing as more people are vaccinated.

We can now follow the same cleaning process to filter out unshared values between the lists, NaN values, and outliers.

This looks more likely to yield a linear relationship than comparing Rate of Vaccinations. Ideally, we would hope for a negative linear regression line, which indicates that the cases are decreasing as the rate of vaccinations increase. Now that we have established something of a possible relationship that can show vaccine efficacy, we can move on to test it.

4. Hypothesis Testing

This is where we start to learn about hypothesis testing. But first, what is a hypothesis? Well, a hypothesis is a claim about a single parameter, several parameters, or a probability distribution. So Hypothesis testing is checking the validity of such a claim. In it, we have two scenerios. The first is that the null Hypothesis denoted $H_{0}$ which is what is assumed to be true by default before the test. The second is the alternate hypothesis denoted $H_{a}$ which is a claim generally opposite to the null hypothesis.

Generally there are about 5 steps to hypothesis testing. Let's briefly summarize them.

  1. Stating the null and alternate hypotheses.
  2. Calculating the test statistic
  3. Calculating the P-value
  4. Making a statistical conclusion with regards to the P-value compared to the level of significance.

You may be wondering, what is a test statistic, a P-value, and the level of significance? First, the level of significance also known as Alpha, is related to the confidence level at which we want to perform the test. The Confidence level is equal to 100(1 - Alpha)%, so when the Confidence level is high like 95%, Alpha would be 5% or 0.05. The test statistic is the standardized score of the sample statistic when the null hypothesis is assumed to be true. The P-value is the probability that the test statistic will take on values as extreme as the test statistic, when the null hypothesis is true. However, we let the machine learning do the work to produce the P-value which we are looking for to make the statistical conclusion.

With that out of the way, let's start hypothesis testing. We want to test for a significant linear relationship between x and y values for a linear regression test. You should recall that for linear regression, lines take up the form y = mx + b or in other words y = $B_{1}$x + $B_{0}$ where $B_{1}$ is the slope.

We want to know whether the Rate of Daily Cases vs Cumulative Number of Those Fully Vaccinated per Hundred People by Month has a significant linear relationship. Let's perform this hypothesis test at a 0.05 level of signficance.

$H_{0}$: $B_{1}$ = 0

$H_{a}$: $B_{1}$ $\neq$ 0

where $B_{1}$ is the slope of the true regression line.

Let's take a moment to understand the hypotheses we just declared. If the slope is equal to 0 like in the null hypothesis, then it means that there is no significant linear relationship between the x and y variables. If the slope is not equal to 0 like in the case of the alternate hypothesis, then it means that there is a significant linear relationship between the x and y variables. Now we need to figure out the P-value. This is where we can bring in machine learning.

Machine Learning

Now let's get into the machine learning. What is machine learning? To keep it simple, Machine learning is training models by fitting those models with data. This can enable us to predict more values based on our model. In this case, we want to create a linear regression model based on the Rate of Daily Cases vs Cumulative Number of Those Fully Vaccinated per Hundred People by Month.

There are many libraries that offer ways to train linear regression models. Among the famous ones are Sklearn and StatsModel. How about we use a bit of both in order to shed more light on how training models work. We can learn about Sklearn at https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html. We already have the values laid out from our latest scatter plot. Let's modify those values so that we can use them to train the linear regression model. But before, let's clear any NaN values that may be present. Once we train the model, we can print out the value of the regression line equation.

Take a look at the linear regression line we calculated using the Sklearn library. It looks very promising, as the slope is negative implying that the rate of daily cases is decreasing as more people are vaccinated. However, it is important to understand that linear regression models can produce equations even when there is no significant linear relationship between x and y. This is because linear regression models are mostly calcualted using a least squares approximation, where the calculated regression line minimizes the sum of the squared deviations from the line.

Let's take a look at the regression line from the model on the graph.

To become more confident with the equation our model produced, let's determine the P-value. In order to accomplish this, we can try out another Machine Learning library, StatsModel. StatsModel makes it significantly easier to calculate the P-value. However there are important considerations to make when training the model. Machine Learning libraries are not always going to adhere to the same exact format, so it is very important to pay heed to the documentation. We can learn about StatsModel at https://www.statsmodels.org/stable/regression.html#.

The P-value in this case would be about 0.014. Using this, let's continue on with our hypothesis test to determine the statistical significance.

To determine the statistical significance, we need to compare the P-value we calculated with the level of significance. If the P-value is less than or equal to Alpha (the level of significance), then we reject $H_{0}$ in favor of $H_{a}$. However if the P-value is greater than Alpha, then we fail to reject $H_{0}$.

With StatsModel, the default level of significance is 0.05 which is what we chose earlier. Our P-value as we already stated is 0.014. Comparing them, we see that Alpha is greater which leads us to reject $H_{0}$ in favor of $H_{a}$.

5. Conclusion

Now that we have rejected the null hypothesis in favor of the alternate hypothesis, let's comprehend what this means in terms of vaccine efficacy. This means that based on the sample data, we have sufficient evidence to conclude that there is a significant linear relationship between the Rate of COVID Cases and the Cumulative Number of Those Fully Vaccinated per Hundred People at a 0.05 level of significance.

Overall, our linear regression model gave us the equation y = -1.3722702484693745x + 9.802660202474815 which is promising given that we also found the linear relationship to be signficant. Reiterating the description of the equation, it tells us that for an increase in number of people vaccinated per hundred people, the rate of cases decreases. This is something that would potentially convince us that the vaccine is working and the number of cases are slowing down. However, it is important to understand that there are numerous factors that go into testing vaccine efficacy around the world such as population density, government regulation on peoples' activities, COVID variants, and a lot more. It is important to understand that in this tutorial we made analyzed vaccine efficacy based on two different datasets. There were many missing values, and many disconnects that had to be accounted for. In the future, a more complicated analysis would likely be more accurate with better collected data, and more datasets that give insight to things like the population density, and social norms of a country.

Overall, I hope you learned a lot from this tutorial, and if you missed any resources linked throughout the tutorial you can check them out here:

Vaccine dataset: https://www.kaggle.com/gpreda/covid-world-vaccination-progress?select=country_vaccinations.csv

Daily Data dataset: https://www.kaggle.com/josephassaker/covid19-global-dataset?select=worldometer_coronavirus_daily_data.csv

India's Struggle with COVID: https://www.wsj.com/articles/why-indias-second-covid-19-surge-is-much-worse-than-its-first-11621071001

Machine Learning Libraries:

Thanks for reading!