Lab 2: Randomization & Non-Compliance

Lecture Slides & Data

Recap

Selection bias

In the lecture we discussed selection bias as the difference in the potential untreated outcomes between those who were treated and who were not treated. Stated formally:

\[E[Y_{1i} | D_i = 1] - E[Y_{0i} | D_i = 0] = \underbrace{E[Y_{1i} - Y_{0i} | D = 1]}_{ATT} + \underbrace{E[Y_{0i} | D_i = 1] - E[Y_{0i} | D_i = 0]}_\text{Selection bias}\] Using experiments we can get rid of selection bias as in expectation units across treatment arms are similar on all (un)observed traits and only differ in terms of treatment assignment $D_I$.

Given that, in expectation potential outcomes are orthogonal (independent) to treatment assignment: $E[Y_{1i} | D_i = 1] = E[Y_{1i}]$ and also $E[Y_{0i} | D_i = 0] = E[Y_{0i}]$

And also because pre-treatment differences in potential outcomes are, on expectation, equal between units assigned to the treatment and the control group. Stated formally: \[E[Y_{0i} | D_i = 1] = E[Y_{0i} | D_i = 0]\] Using both of these assumption the ATT is equal to the ATE. Stated formally:

\[ ATE = E[Y_{1i} | D_i = 1] - E[Y_{0i} | D_i = 0] = \underbrace{E[Y_{1i} - Y_{0i} | D = 1]}_{ATT} + \underbrace{E[Y_{0i} | D_i = 1] - E[Y_{0i} | D_i = 0]}_\text{0}\]

Randomisation

We also discussed ways that we randomly allocate units to either treatment or control:
- Simple random assignment: a procedure such a die roll or coin toss that gives each subject identical probability of being assigned to treatment.
- Complete random assignment: where m of N units are assigned to the treatment.
- Stratified or block randomisation: We conduct complete randomisation within selected groups or within blocks, respectively. Blocks should be determined by a pre-treatment covariate that predicts outcomes.

One-sided compliance

Often times, we assign units to the treatment group, but they are not eventually treated. Usually not taking the treatment is correlated with the outcome. Thus, we need to figure out how to estimate the treatment effect taking into account that we may have either one-sided or two-sided non compliance.

Naive solution: Exclude those not treated, and calculate the different-in-means using only treated units. However, if we do this randomisation is broken.
Intention to treat (ITT): Difference in outcomes between units assigned to the treatment group versus those in the control group without taking into account compliance rates.

\[ ITT = E[Y_{1i}] - E[Y_{0i}] \]

Estimate the Complier Average Causal Effect (CACE), which is the treatment effect amongst compliers.

\[ CACE = \frac{ITT}{E[D_{1i}]} \] where $E[D_{1i}]$ is the difference of treatment compliance rates between the treated and the control group.

Before starting this seminar

Create a folder called “lab2”
Download the data (you can use this button or the one at the top, or read csv files directly from github):
Open an R script (or Markdown file) and save it in our “lab2” folder.
Set your working directory using the setwd() function or by clicking on “More“. For example setwd(“~/Desktop/Causal Inference/2022/Lab2”)
Let’s install an load packages that we will be using in this lab:

library(jtools) # generate plots with regression coefficients
library(stargazer) # generate formated regression tables 
library(modelsummary)  # generate formatted regression tables
library(texreg) # generate formatted regression tables
library(nnet)  # If you want to use glm() to check for balance
library(haven) # upload .dta files into a R environment 
library(tidyverse) # to conduct some tidy operations
library(ggplot2) # to generate plots 
library(AER) # to run IV regressions

Seminar Overview

In this seminar, we will cover the following topics:

Uselm() function to retrieve the ATE.
Report regression estimates stargazer package and coefficients using the coefplot package
Evaluate differential effects conducting an interaction.
Conduct a simple and stratified random assignment.
Calculate the proportion of never takers, always takers, and defiers.
Retrieve the Intention-to-treat using the mean(), lm(), ivreg() functions.
Estimate the Compliers Average Causal Effect using the mean() function and Two-Stage-Least Squares using the ivreg() function.

Experiments

You already know various types of experiments, such as survey experiments, lab experiments, field experiments.

How do natural experiments fit in?

IR: Rotating Presidency of the Council of the EU (Carnegie/Marinov 2017)
CP: Population size & electoral system (Eggers 2015)
many others…
IR/CP: Election Observation (Hyde 2007)

Election Monitoring

Today We will be working with data from Susan Hyde’s work on the Observer Effect in International Politics, where she analyses a natural experiment in Armenia.

Hyde sought to identify a relatively broad but surprisingly understudied research question: Do international monitoring missions have an effect on electoral outcomes?

Satisfactorily answering this question is harder than it seems. As Hyde reasons, cross-sectional comparisons (i.e. comparing mutliple countries) as well as cross-temporal comparisons cannot provide robust evidence of an observer effcect. Endogeneity will inevitably play a role in such analyses. She find a solution to this problem by observing the microlevel. Instead of looking at outcomes at national level, she wants to find out what effect observers have on election-day fraud. To do so, she analyses individual polling stations.

Hyde selects the 2003 presidential elections in Armenia for her analysis, for three particular reasons:

There clearly was some fraud on election day (remember: we need variation in the outcome). The election has been described as ‘one of the dirtiest even Armenians can remember’ (p.47).
Disaggregated data were available (note that this was by no means a given in 2003). [Do you think reporting election results online becoming a standard in virtually all countries has impacted fraud? Why (not)?]
Assignment was ‘as-if’ random by the OSCE. Hyde mentions some possible probems associated with the randomness of observer assignment but argues that these were mitigated by the OSCE’s approach.

Given the assumption of random assignment, Hyde conceives of the assignment of observers as natural experiment. The outcome of interest is the incumbent’s vote share as the incumbent candidate is likely to use the administrative apparatus to commit fraud. The assumption of randomness entails that the expected outcome of both observed and unobserved polling stations would be the same if no observer mission took place.

Hyde concludes that the observer effect is signifcant. That is, the presence of international observers on election day prevents fraud in the observed polling stations. She finds that between observed and unobserved polling stations there was a 5.9% difference in incumbent vote share in the first round and a 2% difference in the second round. Note that these are simple differences in means.

A description of the variables is listed below:

Variable	Description
`mon_voting`	Binary variable equal to 1 if observers were present in polling station in Round 1
`kocharian`	Vote share of the incumbent (Robert Kocharian) in Round 1
`mon_voting_R2`	Binary variable equal to 1 if observers were present in polling station in Round 2
`KocharianR2`	Vote share of the incumbent (Robert Kocharian) in Round 2
`osce_assignment`	Binary variable equal to 1 if OSCE was assigned to observe the polling station (Note: these are fictional data)
`pollingstation`	Name and number of polling station
`subregion`	Armenian region the polling station is located
`urban`	Binary variable equal to 1 if the polling station is located in an urban area
`voterspersqkm`	Continuous variable indicating population density

Now let’s load the data. There are two ways to do this:

You can load the brands dataset from your laptop using the read_dta() function. Remember to load the haven package. If you stored your data in the same directory that set your working directory, you only need to write the name of the dataset, plus the extension (.dta).

# setwd("C:/Desktop/Causal Inference/2022/Lab2")
# 
# library(haven)
# hyde2007 <- read_dta("hyde2007.dta") # loading your data from your laptop

Or you can load the data from the course website using the read.dta() function. Write the following url: https://raw.githubusercontent.com/dpir-ci/CI22/gh-pages/docs/data/hyde2007.dta inside the read.dta() function

hyde2007=read_dta("https://raw.githubusercontent.com/dpir-ci/CI22/gh-pages/docs/data/hyde2007.dta")
# updating the file from the course website.

Let’s start this lab by inspecting the data. You can use either head() or glimpse() function. You may want to select the variables that you want to inspect and then combine these variables using the cbind() function. The syntax would be the following head(cbind(dataframe$variable1, dataframe$variable2...dataframe$variable3)).

Exercise 1: Inspect the following variable: pollingstation, urban,voterspersqkm,kocharian,mon_voting,mon_voting_R2, KocharianR2, osce_assignment.

Reveal Answer

head(cbind(hyde2007$pollingstation, hyde2007$urban, hyde2007$voterspersqkm, hyde2007$kocharian, hyde2007$mon_voting, hyde2007$mon_voting_R2, hyde2007$KocharianR2, hyde2007$osce_assignment))

     [,1]                                [,2] [,3]              
[1,] "<U+00A0> 1671. Agarak, Club"       "0"  "27.9790992736816"
[2,] "<U+00A0> 1620. Arevis, School"     "0"  "77.8955993652344"
[3,] "<U+00A0> 0691. Urtsalanj, School"  "0"  "88.6297988891602"
[4,] "<U+00A0> 0959. Norabak, secondary School" "0"  "41.4747009277344"
[5,] "<U+00A0> 1188. Shamlugh, Cultural House" "0"  "63.1679000854492"
[6,] "<U+00A0> 0678. Erasghavan, School" "0"  "88.6297988891602"
     [,4]                [,5] [,6] [,7]                [,8]
[1,] "1"                 "1"  "0"  "0.977800011634827" "1" 
[2,] "0.942900002002716" "0"  "0"  "1"                 "1" 
[3,] "0.529999971389771" "1"  "0"  "0.676199972629547" "1" 
[4,] "0.693899989128113" "1"  "0"  "0.868399977684021" "1" 
[5,] "0.506200015544891" "0"  "0"  "0.874300003051758" "1" 
[6,] "0.685899972915649" "1"  "0"  "0.776600062847137" "1"

Now let’s check how many observations were observed or treated. The treatment variable is mon_voting variable. We can use the table() function to get the number of polling stations were monitored. The syntax is the following: table(df$variable):

Exercise 2: How many polling stations were observed in the first round?

Reveal Answer

table(hyde2007$mon_voting)


   0    1 
1016  748

We can see that out of the 1764 polling stations,748 were monitored.

Now that we have inspected the data, let’s calculate the difference in means of Kocharian’s vote share (kocharian) between monitored and not monitored polling stations. Let’s check if we get the same numbers reported by Hyde.

Exercise 3: Calculate the difference in means using the mean() function. Remember to subset the data by setting a logical condition within the squared brackets [df$variable == TRUE, ]. The data frame will subset the data selecting the rows that meet the logical condition. Here is the syntax to calculate the difference in means mean(df$outcome[df$treatment == 1])

Reveal Answer

diffmeans  <- mean(hyde2007$kocharian[hyde2007$mon_voting==1]) - mean(hyde2007$kocharian[hyde2007$mon_voting==0], na.rm=T)
diffmeans

[1] -0.05867601

We got the same estimates as Hyde. This difference in means that can be interpreted as an average decrease in vote share of 5.9 percentage points in support of Kocharian in polling station that were monitored versus those that were not monitored.

We can also get the difference in means using the t.test() We can do this by retrieving the means of each groups from a two sample t-test output.

Exercise 4: Perform a two sample t-test using the t.test() function. The syntax is as follows: t.test(df$outcome ~ df$treatment). Use the assignment operator <- to store the output of the t-test into an object, which allows us to retrieve the means of each group from this test.

Reveal Answer

t.test1 <-t.test(hyde2007$kocharian ~ hyde2007$mon_voting)
t.test1


    Welch Two Sample t-test

data:  hyde2007$kocharian by hyde2007$mon_voting
t = 5.9911, df = 1675.8, p-value = 2.546e-09
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 0.03946639 0.07788563
sample estimates:
mean in group 0 mean in group 1 
      0.5419031       0.4832271

To calculate the difference in means we need to retrieve the means of both monitored and not monitored groups from the t-test output. To do this, we need to know the names that the t.test() function gives to each parameter that it generates. To do that, we can use the names() function. The syntax is as follows: names(object) .

names(t.test1) # we can see the names of each of the parameters generated by the t.test function.

 [1] "statistic"   "parameter"   "p.value"     "conf.int"    "estimate"   
 [6] "null.value"  "stderr"      "alternative" "method"      "data.name"

We see that the t.test() function stores the means and calls them “estimate” .

We can retrieve the means from the object where we stored the t-test output by including a dollar sign $, plus estimate. We can also use the round() function to round off values to a specific number of decimal values. The syntax of the round() function is the following round(x, digits = 2) where x is the object (variable). The second argument is the number of decimals that you want to report.

Exercise 5: Obtain the mean vote share for Kocharian in observed and unobserved polling stations. Use the output of the t.test using the following syntax: t.test.object$estimate.Then, retrieve the means of the monitored and not monitored polling stations using the following syntax: t.test.object$estimate[2] (for the monitored polling stations) and t.test.object$estimate[1] (for not monitored polling stations). Finally, subtract these two objects and round their difference, reporting 3 decimals. Use the round() function to do this.

Reveal Answer

t.test1$estimate

mean in group 0 mean in group 1 
      0.5419031       0.4832271

diff_means_ttest <- round(t.test1$estimate[2] - t.test1$estimate[1], 3) # w
diff_means_ttest

mean in group 1 
         -0.059

We see that we obtained the same result as before: -0.059. As you can see, this is not the most straightforward way to retrieve the difference in means.

We know that we use regressions to analyse data from randomised experiments. It turns out that regressing an outcome variable on a treatment variable yields a slope coefficient identical to the difference in means between two groups. In addition, the resulting intercept corresponds to average outcome among the control (untreated) units. Bear in mind that the difference in mean estimator is equal to the coefficient of the treatment variable, only when the treatment variable is binary.

Exercise 6: Regress the outcome variable kocharian on the treatment variable mon_voting variable. Interpret the coefficient of the mon_voting variable. Is the coefficient identical to the difference in means obtained before? Load the stargazer package and report this model in a regression table. The syntax of the stargazer model is the following stargazer(model, title="Results", align=TRUE, type = "text") where model is the object where you stored the output of your regression. Include a title to your regression table and called it “Results”. Also set the align argument equal to TRUE, so the numeric values in the same column will be aligned at the decimal mark. Finally, set the type argument equal to "text" to specifies what type of output the command should produce. You can also use modelsummary() or texreg() to report regression tables, feel free to use these packages as well.

Reveal Answer

library(stargazer)
ols1=lm(kocharian ~ mon_voting, data=hyde2007)
stargazer(ols1, title = "Results", align = TRUE, type = "text")  # using stargazer


Results
===============================================
                        Dependent variable:    
                    ---------------------------
                             kocharian         
-----------------------------------------------
mon_voting                   -0.059***         
                              (0.010)          
                                               
Constant                     0.542***          
                              (0.006)          
                                               
-----------------------------------------------
Observations                   1,764           
R2                             0.019           
Adjusted R2                    0.019           
Residual Std. Error      0.206 (df = 1762)     
F Statistic          35.022*** (df = 1; 1762)  
===============================================
Note:               *p<0.1; **p<0.05; ***p<0.01

From the regression we obtained the same value as when we “manually” calculate the difference in means before. The beta coefficient of the mon_voting variable means that the vote share for Kocharian decreased by 5.9 percentage points once international observers monitor the election process. We also see that the intercept is equal to 0.542. This parameter represents the average vote share for Kocharian in those polling stations that were not not monitored.

As we did last week, we would like to look check that the randomisation worked. In particular, we would like to look at whether there is balance between urban and rural areas.

Exercise 7: Create a contingency table using the tab() function. Get the number of polling stations that were monitored versus those that were not monitored (variable: mon_voting), both for urban and rural areas (variable: urban). The syntax is the following: table(df$variable_1, df$variable_2). Are there differences in the number of polling stations monitored between these two areas?

Reveal Answer

table(hyde2007$urban, hyde2007$mon_voting)

We can see that there are more polling stations were monitored in urban areas (463) versus in rural areas (285).

Now, let’s use prop.table() to check whether there is unbalance between urban and rural areas. You can do this by storing the previous table in a object, and then insert this object inside the prop.table() function.

Exercise 8: Run the prop.table() function. Do you find unbalance in terms of the number of polling stations between urban and rural?

Reveal Answer

prop.test(table(hyde2007$urban, hyde2007$mon_voting))


    2-sample test for equality of proportions with continuity correction

data:  table(hyde2007$urban, hyde2007$mon_voting)
X-squared = 21.088, df = 1, p-value = 4.388e-06
alternative hypothesis: two.sided
95 percent confidence interval:
 0.06279495 0.15695353
sample estimates:
   prop 1    prop 2 
0.6369427 0.5270684

From the prop.table() function, we find that around 63.6% of the polling stations in the rural area were not monitored, whereas in the urban area roughly 52%. We find statistically significant difference in the proportions of polling stations that were not observed.

We can also check for balance by regressing the treatment assignment variable mon_voting on the covariate(s) of interest urban. Let’s do that in the following exercise.

Exercise 9: Regress mon_voting on urban. Again, report the results in a regression table. Change the title to “Randomisation check”.

Reveal Answer

balance_urban=lm(mon_voting ~ urban, data=hyde2007)

stargazer(balance_urban, title = "Randomisation check", align = TRUE, type = "text") # using stargazer


Randomisation check
===============================================
                        Dependent variable:    
                    ---------------------------
                            mon_voting         
-----------------------------------------------
urban                        0.110***          
                              (0.024)          
                                               
Constant                     0.363***          
                              (0.018)          
                                               
-----------------------------------------------
Observations                   1,764           
R2                             0.012           
Adjusted R2                    0.012           
Residual Std. Error      0.491 (df = 1762)     
F Statistic          21.777*** (df = 1; 1762)  
===============================================
Note:               *p<0.1; **p<0.05; ***p<0.01

What do we conclude? Do we have reason to think this might affect the validity of the results? One key feature of randomised experiments is that they can generate unbiased estimates of the average treatment effect regardless if there is imbalance. We also know that we can use the covariate urban to eliminate observed differences between treatment and control groups and reduce variability in outcomes. However, we should be suspicions that we find such a large difference between the two areas areas.

Let’s ignore the differences in the number of polling stations monitored between urban and rural areas. Let’s look at heterogeneous treatment effects and identify whether there is a differential response to international monitoring between urban and rural areas. For that, we could interact the treatment variable mon_voting with the urban variable. The syntax is the following lm(Outcome variable = Treatment variable + Covariate + Covariate * Treatment variable)

Exercise 10: Run a regression where the dependent variable is voter share kocharian is regressed on mon_voting, the urban variable and an interaction between mon_voting and urban variable. Interpret the coefficient of the mon_voting and the interaction. Do you find evidence of a differential response to being monitored between polling stations in urban versus rural areas? Also, interpret the coefficient of the urban variable. Present the results in a regression, but also include the previous model conducted in Exercise 6 (bivariate model), so you can compare both models. The syntax using stargazer is the following stargazer(model1, model2, title = “Models, align =TRUE, type =”text").

Reveal Answer

ols1_interaction =lm(kocharian ~ mon_voting + urban + mon_voting*urban, data=hyde2007)

stargazer(ols1, ols1_interaction, title = "Models", align = TRUE, type = "text")


Models
=====================================================================
                                   Dependent variable:               
                    -------------------------------------------------
                                        kocharian                    
                              (1)                      (2)           
---------------------------------------------------------------------
mon_voting                 -0.059***                 -0.028*         
                            (0.010)                  (0.015)         
                                                                     
urban                                                 0.003          
                                                     (0.013)         
                                                                     
mon_voting:urban                                     -0.050**        
                                                     (0.020)         
                                                                     
Constant                    0.542***                 0.540***        
                            (0.006)                  (0.009)         
                                                                     
---------------------------------------------------------------------
Observations                 1,764                    1,764          
R2                           0.019                    0.025          
Adjusted R2                  0.019                    0.023          
Residual Std. Error    0.206 (df = 1762)        0.205 (df = 1760)    
F Statistic         35.022*** (df = 1; 1762) 14.863*** (df = 3; 1760)
=====================================================================
Note:                                     *p<0.1; **p<0.05; ***p<0.01

First, we find that controlling for urban the point estimate of the mon_voting yields a much smaller treatment effect (roughly 2 times smaller). Now the ATE is -0.028 but yet statistically significant. The non-zero coefficient on the interaction between urban and mon_voting means that support to being monitored has a differential effect on vote share for Kocharian between urban vs rural areas. This interaction coefficient (0.050) is the average difference in the effect of mon_voting between urban and rural areas. This substantively means that the average effect of monitoring is associated with a 5%point decrease in vote share for Kocharian, when me move from a rural to urban area. Finally, the coefficient from the urban variable indicates the average difference in vote share for Kocharian between urban and rural areas, when there is no monitoring mon_voting = 0. Here, we don’t find a significant effect. This is unexpected as urban areas usually show different voting patterns than rural regions do. Given the imbalanced patterns of observation, the monitoring effect is likely to absorb the rural-urban difference.

Round 2 of monitoring

In this study, the OSCE (Organization for Security and Co-operation in Europe) conducted a second wave of inspections for the second round of the elections. Let’s quickly replicate the same analyses that we did before.

Exercise 11: Run a regression using the outcome variable KocharianR2 regressed on the treatment assignment variable mon_voting_R2. You can use the summary() or tidy() function to report your results. Is the treatment effect weaker or stronger in this round?

Reveal Answer

ols2 <- lm(KocharianR2 ~ mon_voting_R2, data=hyde2007)
summary(ols2)


Call:
lm(formula = KocharianR2 ~ mon_voting_R2, data = hyde2007)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.69251 -0.10281 -0.00287  0.11904  0.32733 

Coefficients:
               Estimate Std. Error t value Pr(>|t|)    
(Intercept)    0.692511   0.004781 144.841   <2e-16 ***
mon_voting_R2 -0.019841   0.008043  -2.467   0.0137 *  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.1614 on 1761 degrees of freedom
  (1 observation deleted due to missingness)
Multiple R-squared:  0.003444,  Adjusted R-squared:  0.002878 
F-statistic: 6.085 on 1 and 1761 DF,  p-value: 0.01373

# tidy format
ITT <- tidy(ols2) %>%
  filter(term == "mon_voting_R2") %>%
  pull(estimate)
ITT

[1] -0.01984091

This looks like a somewhat weaker effect of the observers. We find that the average treatment effect is now -0.01, which means that there is a drop of 1 percentage point on the vote share for Kocharian in the second round of elections when polling stations are being monitored.

Exercise 12: Again, as we did before, let’s check if there is balance for the urban variable in the second round. Regress the treatment assignment variable mon_voting_R2 on urban. Report the results using either summary() or the tidy() function.

Reveal Answer

ols2_urban=lm(mon_voting_R2 ~ urban, data=hyde2007)
summary(ols2_urban)


Call:
lm(formula = mon_voting_R2 ~ urban, data = hyde2007)

Residuals:
    Min      1Q  Median      3Q     Max 
-0.4147 -0.4147 -0.2764  0.5853  0.7236 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  0.27643    0.01689  16.365  < 2e-16 ***
urban        0.13828    0.02267   6.099 1.31e-09 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.4733 on 1762 degrees of freedom
Multiple R-squared:  0.02067,   Adjusted R-squared:  0.02012 
F-statistic: 37.19 on 1 and 1762 DF,  p-value: 1.313e-09

We find again unbalance between urban and rural areas. This linear probability model suggest that being in a urban area is associated with an increase on the probability of being observed by 13% more than an rural area.

Given that we are working with a binary treatment variable, we could also use glm(). Try on your own and interpret the results from this estimation.

Let’s perform the same analysis that we did before for the first round to obtain the ATE for the second round. Include the variable urban and also include interaction between urban and mon_voting_R2.

Exercise 13: Find the treatment effect using the lm() function. Include the urban variable as a covariate. Also include the interaction between mon_voting_R2 with urban. Interpret the results for both mon_voting_R2 an the interaction term. Report your results in a regression table. Use either stargazer(), screenreg() or modelsummary(). Give the following names to your model: “Second round”. Report the results with 3 decimals.

Reveal Answer

ols2_interaction=lm(KocharianR2 ~ mon_voting_R2 + urban + urban*mon_voting_R2, data=hyde2007)

library(texreg)
screenreg(ols2_interaction, digits = 2, custom.model.names = c("Second round"),
          custom.coef.names = c("Intercept", "Monitor", "Urban", "Monitor:Urban"))


===========================
               Second round
---------------------------
Intercept         0.67 *** 
                 (0.01)    
Monitor           0.01     
                 (0.01)    
Urban             0.04 *** 
                 (0.01)    
Monitor:Urban    -0.06 *** 
                 (0.02)    
---------------------------
R^2               0.01     
Adj. R^2          0.01     
Num. obs.      1763        
===========================
*** p < 0.001; ** p < 0.01; * p < 0.05

We find that the treatment effect for the second round is 0,01. Slightly smaller and no longer statistical significant. However, we do find differential response to the treatment as the interaction is statistically significant. This coefficient of the interaction can be interpreted as follows: The average effect of monitoring on vote share for Kocharian is about 6 percentage points less, when we move from polling stations in rural to urban areas.

Let’s include a in a single regression table some of the regressions that we have conducted so far. You can use the stargazer() function, but feel free to use texreg() or modelsummary().

Exercise 14: Include the regression models into single regression table. Change the title to: “Regression results - All rounds”. Call bivariate models “Baseline” and models with interaction “Interaction”. To do this, include the following argument column.labels = c("Name 2", "Name3"). Also, set labels for the dependent variables that you analysed the first round and the second round. You can do this by adding the following argument dep.var.labels = c("First Round", "Second Round"). Finally, let’s change the labels of the variables by including the covariate.labels = c("label variable 1", "label variable 2")

Reveal Answer

stargazer(ols1, ols1_interaction, ols2, ols2_interaction, title = "Regression results - All rounds", column.labels = c("Baseline", "Interaction", "Baseline", "Interaction"), dep.var.labels = c("First round", "Second round"), covariate.labels = c("Monitored", "Urban", "Monitor:Urban", "Monitor round 2:Urban", "Monitor round 2" ), align = TRUE, type = "text")


Regression results - All rounds
======================================================================================================================
                                                            Dependent variable:                                       
                      ------------------------------------------------------------------------------------------------
                                         First round                                     Second round                 
                              Baseline               Interaction               Baseline              Interaction      
                                (1)                      (2)                     (3)                     (4)          
----------------------------------------------------------------------------------------------------------------------
Monitored                    -0.059***                 -0.028*                                                        
                              (0.010)                  (0.015)                                                        
                                                                                                                      
Urban                                                   0.003                                         0.038***        
                                                       (0.013)                                         (0.010)        
                                                                                                                      
Monitor:Urban                                          -0.050**                                                       
                                                       (0.020)                                                        
                                                                                                                      
Monitor round 2:Urban                                                                                 -0.060***       
                                                                                                       (0.017)        
                                                                                                                      
Monitor round 2                                                                -0.020**                 0.013         
                                                                               (0.008)                 (0.013)        
                                                                                                                      
Constant                      0.542***                 0.540***                0.693***               0.673***        
                              (0.006)                  (0.009)                 (0.005)                 (0.007)        
                                                                                                                      
----------------------------------------------------------------------------------------------------------------------
Observations                   1,764                    1,764                   1,763                   1,763         
R2                             0.019                    0.025                   0.003                   0.014         
Adjusted R2                    0.019                    0.023                   0.003                   0.012         
Residual Std. Error      0.206 (df = 1762)        0.205 (df = 1760)       0.161 (df = 1761)       0.161 (df = 1759)   
F Statistic           35.022*** (df = 1; 1762) 14.863*** (df = 3; 1760) 6.085** (df = 1; 1761) 8.210*** (df = 3; 1759)
======================================================================================================================
Note:                                                                                      *p<0.1; **p<0.05; ***p<0.01

From the previous analysis we might wonder to what extend the rounds are independent of each other. Hyde states the following in this regard:

Hyde concludes that her “results suggest that if first-round monitoring took place then second-round monitoring had only a marginal additional deterrent effect.” (p.56). She claims that first-round observation has a persistent effect.

Let’s see whether monitoring in the two rounds was independent. First, we check the number of polling stations observed per round.

sum(with(hyde2007, mon_voting==0 & mon_voting_R2==0))

[1] 756

sum(with(hyde2007, mon_voting==1 & mon_voting_R2==0))

[1] 385

sum(with(hyde2007, mon_voting==0 & mon_voting_R2==1))

[1] 260

sum(with(hyde2007, mon_voting==1 & mon_voting_R2==1))

[1] 363

Well, in Round 2 more stations were observed that had been monitored in Round 1 than those that had not been monitored. Let’s confirm significance. For that we can conduct a two sample test of proportions.

prop.test(table(hyde2007$mon_voting_R2, hyde2007$mon_voting))


    2-sample test for equality of proportions with continuity correction

data:  table(hyde2007$mon_voting_R2, hyde2007$mon_voting)
X-squared = 98.233, df = 1, p-value < 2.2e-16
alternative hypothesis: two.sided
95 percent confidence interval:
 0.1965443 0.2939381
sample estimates:
   prop 1    prop 2 
0.6625767 0.4173355

What do we conclude?

The evidence suggest that monitoring in round 2 was not indeed random and it was conditional on the results of the first round.
If we apply experimental standards (which is what we should do), we conclude that Round 2 monitoring was not independent from Round 1 monitoring.

Finally, let’s plot all coefficients from the regressions conducted in Exercises 6, 10, 11 and 13 together. There are multiple packages to plot regression estimates. One of them is jtools. We can use the plot_summs function to to create regression coefficients plots with ggplot2.

The first step is to store all regressions into separate objects. You should have 4 models in 4 separate objects: two bivariate models and two models with interactions. We can store all of the objects in a list. A list in R is object that can contain multiple objects. This facilitates the syntax that we need to use whenever a plot with multiple models.

The second step is to include this list the plot_summs() function. Add the following argument inside of the plot_summs() function, scale = TRUE. This generates a version of the plot. Also, include the following argument: robust = TRUE. Setting robust = TRUE the plot will generate confidence intervals using robust standard errors.

Exercise 15: Generate a plot with the coefficients of all four models using `plot_summs() function

Reveal Answer

coefficients <- c("(Intercept)" = "Intercept","mon_voting" = "Monitor 1", "urban" = "Urban", "mon_voting:urban" = "Monitor 1 x Urban", "mon_voting_R2" = "Monitor 2", "mon_voting_R2:urban" = "Monitor 2 x Urban")

plot_summs(models, scale = TRUE, robust = TRUE)

Randomisation

We already found a meaningful imbalance. Polling stations in urban areas were much more likely to be observed than rural ones in the first round of the elections. We know that we can’t control for all possible confounders, but is this imbalance a problem for the validity of Hyde’s research design? Why?

Let’s check if we get a similar result for population density to make sure the imbalance is not simply caused by the way the urban variable was constructed.

Exercise 16: Find out whether randomization worked regarding population density (voterspersqkm).

Reveal Answer

#Population Density
ks.test(hyde2007$voterspersqkm[hyde2007$mon_voting==0],
        hyde2007$voterspersqkm[hyde2007$mon_voting==1])


    Two-sample Kolmogorov-Smirnov test

data:  hyde2007$voterspersqkm[hyde2007$mon_voting == 0] and hyde2007$voterspersqkm[hyde2007$mon_voting == 1]
D = 0.26816, p-value < 2.2e-16
alternative hypothesis: two-sided

# Lookin at means only
t.test(hyde2007$voterspersqkm~hyde2007$mon_voting, data=hyde2007)


    Welch Two Sample t-test

data:  hyde2007$voterspersqkm by hyde2007$mon_voting
t = -12.156, df = 1221.9, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -914.4994 -660.3231
sample estimates:
mean in group 0 mean in group 1 
       443.6667       1231.0779

Given that population density is a continuous variable, we can use a t-test to look at differences in means and a ks.test for meaningful differences in the distribution between observed and unobserved polling stations.

All these stats show that there is an imbalance between observed and unobserved polling stations. That is, they treatment (monitoring) was not randomly received with regard to population density and urban areas. This is a very concerning problem as we know from various studies that urban and rural demographics are different, particularly when it comes to political predisposition. The problem here is that what we might think to be the treatment effect is actually just this difference we pick up in our results due to selection bias.

Exercise 17: Randomise the assignment to treatment. Use (i) complete random assignment and (ii) stratified randomisation.

Since randomisation was not, in fact, successful in the natural setting, let’s ourselves create a variable for random treatment assignment. We assume that 50% of all polling stations can be observed.

Note: There are two ways to do this:
- You can use either the rbinom function to randomly assign a dummy variable, which follows the following syntax: `rbinom(n, size, prob), where n is the number of observations, size is the number of trials per observation and prob the porbability of success (or a positice result)
- Or you use the simple_ra function from the randomizrpackage. You only need to specify arguments for n and probability.

Reveal Answer

hyde2007$random_assignment <- rbinom(n=1764, size=1, prob=0.5)

table(hyde2007$random_assignment)


  0   1 
877 887

# or
library(randomizr)
hyde2007$random_assignment <- simple_ra(N = 1764, prob = 0.5)

table(hyde2007$random_assignment)


  0   1 
897 867

[See more ways to randomise using the randomizr package]

The randomizr package includes various options for randomisation. You can easily achieve block randomisation or even randomise the assignment of various treatments. These are just a few examples:

hyde2007$multiarm_assignment <- complete_ra(N = 1764, prob_each = c(0.2, 0.3, 0.5), conditions = c("Treatment 1", "Treatment 2", "Control")) ## multiple treatments 

table(hyde2007$multiarm_assignment)


Treatment 1 Treatment 2     Control 
        353         529         882

hyde2007$block_assignment <- block_ra(blocks = hyde2007$urban,
prob_each = c(.1, .1, .8))

table(hyde2007$block_assignment)


  T1   T2   T3 
 177  176 1411

Assume now that the OSCE has been made aware of this problem regarding rural areas. For some reason, urban polling stations are more likely to be monitored than rural ones. For their next assignment, they want to make sure rural polling stations are just as likely to be observed as urban ones. To have enough resources to ensure that every single polling station that is assigned to be observers will in fact be observed, they reduce the overall number of polling stations to 30% but use stratified randomisation.
Note: The easiest way to do this is using the strata_rs command from the randomizr package. You just need to specify the strata and prob arguments.

Reveal Answer

#install.packages("randomizr")
library(randomizr)
hyde2007$stratified <- strata_rs(strata = hyde2007$urban, prob = .3)

This should be much better now. Nonetheless, we should make sure randomisation was successful - using random assignment as well as stratified randomisation - to avoid working with flawed data.

# Complete Random Assignment Balance Checks
t.test(hyde2007$voterspersqkm~hyde2007$random_assignment, data=hyde2007)


    Welch Two Sample t-test

data:  hyde2007$voterspersqkm by hyde2007$random_assignment
t = 0.1181, df = 1760.2, p-value = 0.906
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -116.4803  131.4066
sample estimates:
mean in group 0 mean in group 1 
       781.2257        773.7626

prop.test(table(hyde2007$urban, hyde2007$random_assignment))


    2-sample test for equality of proportions with continuity correction

data:  table(hyde2007$urban, hyde2007$random_assignment)
X-squared = 0.54101, df = 1, p-value = 0.462
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.06685097  0.02932145
sample estimates:
   prop 1    prop 2 
0.4980892 0.5168539

# Stratified Randomisation Balance Checks
t.test(hyde2007$voterspersqkm~hyde2007$stratified, data=hyde2007)


    Welch Two Sample t-test

data:  hyde2007$voterspersqkm by hyde2007$stratified
t = -0.21412, df = 991.99, p-value = 0.8305
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -150.5202  120.9042
sample estimates:
mean in group 0 mean in group 1 
       773.1169        787.9249

prop.test(table(hyde2007$urban, hyde2007$stratified))


    2-sample test for equality of proportions with continuity correction

data:  table(hyde2007$urban, hyde2007$stratified)
X-squared = 1.0575e-29, df = 1, p-value = 1
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.04302202  0.04490877
sample estimates:
   prop 1    prop 2 
0.7006369 0.6996936

This looks good - actually, it couldn’t be better: look at the p-value of 1! Assignment to treatment is now balanced. Let’s have a look at the regional allocation of election observers nonetheless.

aggregate(hyde2007[, c("mon_voting", "random_assignment", "stratified")], list(hyde2007$subregion), mean)

            Group.1 mon_voting random_assignment stratified
1           Abovyan  0.4814815         0.5555556  0.2592593
2          Ajapnyak  0.5897436         0.3846154  0.2051282
3          Alaverdi  0.3200000         0.5200000  0.3000000
4          Anrashat  0.7142857         0.2380952  0.2380952
5           Arabkir  0.6818182         0.4090909  0.2500000
6            Ararat  0.6666667         0.6000000  0.2666667
7           Armavir  0.3600000         0.3600000  0.3200000
8         Arshaluys  0.2812500         0.4687500  0.4062500
9             Artik  0.4000000         0.3750000  0.2500000
10         Ashtarak  0.2812500         0.5000000  0.3125000
11             Avan  0.5238095         0.6190476  0.1428571
12           Ayntap  0.5333333         0.4666667  0.3333333
13             Berd  0.4186047         0.6046512  0.2325581
14      Byureghaven  0.3611111         0.3888889  0.2222222
15     Charentsavan  0.2222222         0.5555556  0.2592593
16          Eghvard  0.6153846         0.5384615  0.5384615
17          Erebuni  0.7692308         0.5000000  0.3461538
18            Gavar  0.2500000         0.5357143  0.2500000
19            Goris  0.3893805         0.5398230  0.3185841
20           Gyumri  0.1676301         0.4797688  0.3121387
21          Hrazdan  0.5862069         0.5172414  0.2758621
22           Ijevan  0.1351351         0.4864865  0.2702703
23   Kanaker-Zeytun  0.3571429         0.6428571  0.2857143
24            Kapan  1.0000000         0.0000000  0.0000000
25          Kentron  0.8600000         0.5000000  0.4600000
26 Malatia-Sebastia  0.7777778         0.5555556  0.2500000
27          Martuni  0.4074074         0.4074074  0.3333333
28            Masis  0.3793103         0.5517241  0.3448276
29       Myasnikyan  0.3000000         0.4750000  0.2750000
30         Nor-Nork  0.8333333         0.4166667  0.2708333
31          Parakar  0.4062500         0.4375000  0.4062500
32            Sevan  0.3023256         0.6046512  0.1860465
33        Shengavit  0.7142857         0.4642857  0.3750000
34            Talin  0.2753623         0.4927536  0.3188406
35           Tashir  0.2222222         0.4444444  0.3333333
36         Vanadzor  0.3934426         0.4590164  0.2622951
37         Vardenis  0.3400000         0.4600000  0.2200000
38    Vargharshapat  0.6666667         0.5000000  0.2500000
39             Vedi  0.4074074         0.5925926  0.3333333
40     Yeghegnadzor  0.3750000         0.5750000  0.4750000

It looks like the problem with first round monitoring is not merely a concentration in urban areas, but an overproportionate presence of monitors in the capital, Yerevan! Arabkir, Erebuni, Kentron, Malatia-Sebastia, Nor-Nork, Shengavitare all districts of Yerevan and vastly overrepresented. The table also shows that this problem would not have appeared if observers had been randomly assigned. This is a good example that context really matters for natural experiments!

Make sure you will be using the osce_assignment variable for the next parts of the lab. It provides a fictional example of potential assignment by the OSCE, allowing us to work with the same data.

Non-Compliance

One of the reasons for the imbalance between rural and urban polling station could be that observers did not adhere to their assigned polling stations. In that case, one-sided non-compliance would be a problem. Polling stations that were supposed to be monitored were, in fact, not monitored on election day. Let’s quickly have a look at the compliance status and the broad picture. Let’s look again at the contingency table that you did using the osce_assignment and the mon_voting_R2 variables.

In experiments with one-sided compliance units or subjects can be divided into two groups: Compliers and Never takers . However, in the case of two-sided non-compliance there are subjects that be divided into further groups adding the so-called are called Always takers . These are subjects/units that regardless of their treatment assignment, they always end up taking the treatment.**

Tricky Exercise 18: Generate again a contingency table using the osce_assignment and the mon_voting_R2 variables. Calculate the proportion of Always takers.

Reveal Answer

#Let's remember the compliance status [1] 883 [2]258 [3] 0 [4]623
tab.1 <- table(hyde2007$osce_assignment, hyde2007$mon_voting_R2)
tab.1

#Always-Takers  
# Under one-sided non-compliance there are no Always takers, so this should be zero
# 0 / (883 + 0) = 0 
at <- tab.1[3]/(tab.1[1] + tab.1[3])  
at

[1] 0

As pointed before, there are no Always takers . in experiments with one-sided compliance. Also, if there were Always takers ., we would expect some of polling stations assigned to control (row = 0) being monitored (column = 1).

We also have the so-called Never takers . These are units that never take the treatment regardless of whether they are assigned to treatment group or control group.

Exercise 19: Calculate the proportion of Never takers in the study. Use the same contingency table as before. Hint: Look at the number of units within those assigned to treatment, and in particular the number of polling stations that were not monitored.

Reveal Answer

tab.1

#Never Takers
# 258/(258 + 623) = 0.29
# This means that from those polling stations that were assigned to treatment, 29% will not be monitored
nt <- tab.1[2]/(tab.1[2] + tab.1[4])
nt

[1] 0.292849

We can calculate the proportion of Never takers by looking the units within the treatment group. The number of Never takers are those assigned to treatment, but did not end up taking the treament. That is equal to 258. Then, the proportion is equal to 258 divide by the total number of polling stations assigned to the treatment 258 + 623 = 881. We cannot identify the Never takers in the control group, as we cannot to disentangle whether units are not taking treatment because they are Compliers or because in fact they are Never takers .

Finally, we have the so-called Compliers . These are units that follow whatever their assignment is (if assigned to treatment, they do take the treatment; if assigned to control, they don’t). For example, if a polling station was assigned to be monitored, it ends up being monitored. Conversely, if a polling station was assigned to not be monitored, it ends up not being monitored.

Exercise 20: Calculate the proportion of Compliers in the study. Use the same contingency table as before. Hint: Look again at the units in the treatment group. In particular the number of polling stations that were monitored.

Reveal Answer

#Compliers (No Defiers By Assumption)
1 - at - nt

[1] 0.707151

So we have 623 units assigned to the treatment and ended up being monitored. Thus, the number of compliers is equal to 623/(258+623) = 0.70. But we can also obtain the proportion of Compliers by subtracting the proportion of Never takers and Always takers from 1.

An important underlying assumption for identification is that we rule out the existence of the alleged Defiers. These units do the opposite of whatever their assignment is (if assigned to treatment, they don’t do the treatment; if assigned to control, they do the treatment).

Intent-To-Treatment (ITT) & Complier Average Causal Effect (CACE)

Using the ITT is one possible solution to dealing with non-compliance. Remember that the ITT describes intended assignments, not actual treatment. In doing so,it does not simply discards non-compliance.

Let’s start by calculating the ITT manually, using differences in means. Remember that the ITT describes the difference in the outcome between the units assigned to treatment and those which were not. In other words, it is the change in expected outcomes when a unit moves from an assigned control group to an assigned treatment group:

\[ITT = E[Y_1~_i] - E[Y_0~_i]\]

Note that you find this parameter also written as ITT.y, which highlights its referring to the effect in the outcome, y.

Exercise 21: Using the information on random assignment (osce_assignment), calculate the Intent-to-Treat effect (ITT) in Round 2 of the election.

Reveal Answer

There are two ways to calculate the ITT.y. First, the difference in means of the outcome variable between the group of units assigned to treatment, on the one hand, and the units assigned to the control group, on the other hand, is equivalent to the ITT.

#ITT or ITT.y
itt.y <- (mean(hyde2007$KocharianR2[hyde2007$osce_assignment==1], na.rm = TRUE) - mean(hyde2007$KocharianR2[hyde2007$osce_assignment==0], na.rm = TRUE))
itt.y

[1] -0.01544005

Second, you can run a bivariate OLS regression. As you learned last week, a the coefficient of a bivariate OLS is equivalent to the difference in means if the independent variable is binary.

itt_fit <- lm(KocharianR2 ~ osce_assignment, data= hyde2007)
summary(itt_fit)


Call:
lm(formula = KocharianR2 ~ osce_assignment, data = hyde2007)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.69322 -0.10448 -0.00248  0.11818  0.32222 

Coefficients:
                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)      0.693215   0.005439 127.456   <2e-16 ***
osce_assignment -0.015440   0.007694  -2.007   0.0449 *  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.1615 on 1761 degrees of freedom
  (1 observation deleted due to missingness)
Multiple R-squared:  0.002282,  Adjusted R-squared:  0.001715 
F-statistic: 4.027 on 1 and 1761 DF,  p-value: 0.04492

The results are the same, the ITT is about-0.015. The difference in the outcome between polling stations assigned to treatment and those that weren’t amounts to 1.5%points in the incumbent’s vote share. Does this go together with the reported results?

If there is no non-compliance and assignment to treatment perfectly predicts receiving the treatment, the ITT is equivalent to the ATE. In our natural experiment, however, many polling stations that should have been observed were not. Accordingly, the ITT.y underestimates any true treatment effect. If we don’t care about treatment effects but are only interested in the change in outcome a certain programme or policy brings about, the ITT can be a quantity of interest.

The ITT thus describes the intent-to-treat effect of assignment to treatment (z) on the outcome (Y). Be aware that this means the outcomes could be influenced either by the assignment to the treatment or the actual treatment.

Let’s therefore now look at the intent-to-treat effect of assignment to treatment (z) on treatment (d). This second ITT, sometimes called `ITT.d, describes the the proportion of subjects who are treated in the event that they are assigned to the treatment group, minus the proportion who would have been treated even if they had instead been assigned to the control group:

\[ITT_D = E[d_i(1)] - E[d_i(0)]\]

Note that in the case of one-sided no compliance, e.g. the case without always-takers as we are facing here, the ITT.D reduces to $E[d_i(1)]$. In our analyses, we assume there were no always-takers, i.e. polling stations that were not supposed to be monitored but were, in fact, monitored.

Exercise 22: Calculate the Intent-to-Treat effect on treatment (ITT.d) in Round 2 of the election.

Here, again, we have the same two options we used to estimate the ITT. Both differences in means and a bivariate OLS will provide the ITT.d. In the former case, it is the difference in means of the treatment variable, which is equivalent to the share of treated polling stations per group, between polling stations selected for observation missions, on the one hand, on those that haven’t been selected, on the other hand. In other words, we calculate the share of monitored polling states over all stations assigned for monitoring and subtract the share of monitored stations in the control group from the former.

Reveal Answer

#ITT_D
itt.d <- (mean(hyde2007$mon_voting_R2[hyde2007$osce_assignment==1], na.rm = TRUE) - mean(hyde2007$mon_voting_R2[hyde2007$osce_assignment==0], na.rm = TRUE))
itt.d

[1] 0.707151

Equivalently, on OLS regression treatment on assignment to treatment will provide the same result:

## ITT_D using OLS 
itt_d_fit <- lm(mon_voting_R2 ~ osce_assignment, data=hyde2007)
summary(itt_d_fit)


Call:
lm(formula = mon_voting_R2 ~ osce_assignment, data = hyde2007)

Residuals:
    Min      1Q  Median      3Q     Max 
-0.7071  0.0000  0.0000  0.2928  0.2928 

Coefficients:
                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)     1.565e-15  1.083e-02    0.00        1    
osce_assignment 7.072e-01  1.532e-02   46.15   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.3218 on 1762 degrees of freedom
Multiple R-squared:  0.5473,    Adjusted R-squared:  0.547 
F-statistic:  2130 on 1 and 1762 DF,  p-value: < 2.2e-16

The results are the same, the ITT.d is about0.71. In other terms, only 71% of all polling stations that should have been observed were observed during the first round of the elections.

We (and Hyde), however, do not care very much about the differences in outcomes or the ITT itself. The aim of this natural experiment is finding out if electoral monitoring significantly reduces fraud - we want to identify the treatment effect. As stated above, the ITT is not equivalent to the ATE due to non-compliance.

In this case, the complier average causal effect (CACE) can be estimated, which corresponds to the ATE among compliers:

\[CACE = \frac{ITT_Y}{ITT_D} = \underbrace{E[(Y_i(d=1) - Y_i(d=0))}_\text{ATE}|\underbrace{d_i(1)=1}_\text{Among Compliers}]\]

Note that the CACE is sometimes referred to as local average treatment effect (LATE).

Exercise 23: Calculate the CACE using the ITT.y and ITT.d

You can simply divide the ITT by the ITT.D or use the calculations for both quantities and merge them into a single formula.

Reveal Answer

#CACE in one line: ITT_Y divided by ITT_D 
(mean(hyde2007$KocharianR2[hyde2007$osce_assignment==1], na.rm = TRUE) - mean(hyde2007$KocharianR2[hyde2007$osce_assignment==0], na.rm = TRUE))/(mean(hyde2007$mon_voting_R2[hyde2007$osce_assignment==1], na.rm = TRUE) - mean(hyde2007$mon_voting_R2[hyde2007$osce_assignment==0], na.rm = TRUE))

[1] -0.02183417

cace <- itt.y/itt.d # This should be the same! 
print(cace)

[1] -0.02183417

We can also calculate the significance of the CACE. To do so, we calculate the standard errors of the CACE (this is in fact, an approximation): $SE(CACE) \approx SE(ITT.Y) / ITT.D$

se_cace <- summary(itt_fit)$coefficients[4]/itt_d_fit$coefficients 
print(se_cace)

    (Intercept) osce_assignment 
   4.916583e+12    1.088013e-02

Finally, you can then calculate the p-value as we did last week:

t_cace <- cace / se_cace
print(t_cace)

    (Intercept) osce_assignment 
  -4.440922e-15   -2.006793e+00

p <- (1 - pnorm(abs(t_cace), 0, 1)) * 2
p

    (Intercept) osce_assignment 
     1.00000000      0.04477166

Exercise 23b: Calculate the CACE using a two-stage least-squares (2SLS) estimator

As we will be seeing in a few weeks time, we can also use two-stage least-square (2SLS) regressions to calculate the CACE. This regression is frequently used in instrumental variable designs, but don’t worry about that for now. All we need to know is that it can be applied to identify the CACE, too.

The logic of the 2SLS estimator is that it involves two regressions. in a first step, the (random!) assignment to treatment is used to predict actual treatment. Assignment to treatment itself is assumed to only influence the outcome through the reception of treatment and is, thus, not part of the second equation. The effect of the treatment in the second equation is estimated using the predicted values for receiving the treatment.

While this seems complicated, what 2SLS does is addressing endogeneity: It accounts for the possibility that receiving treatment itself can be related to some confounder that also affects the treatment.

The 2SLS estimator can be implemented relatively easily using the ivreg command. The syntax is as follows: ivreg(Outcome ~ Treatment|Assignment, data=).

Reveal Answer

#CACE via 2SLS
iv_reg <- ivreg(KocharianR2 ~ mon_voting_R2|osce_assignment, data=hyde2007)
summary(iv_reg)


Call:
ivreg(formula = KocharianR2 ~ mon_voting_R2 | osce_assignment, 
    data = hyde2007)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.693215 -0.103031 -0.002715  0.118335  0.328619 

Coefficients:
               Estimate Std. Error t value Pr(>|t|)    
(Intercept)    0.693215   0.005436 127.528   <2e-16 ***
mon_voting_R2 -0.021834   0.010874  -2.008   0.0448 *  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.1614 on 1761 degrees of freedom
Multiple R-Squared: 0.003409,   Adjusted R-squared: 0.002843 
Wald test: 4.032 on 1 and 1761 DF,  p-value: 0.0448

The estimated CACE is equivalent to the one from our manual estimation, -0.0218. That is, the average treatment effect among polling stations that were supposed to be observed and, in fact, were observed corresponds to a reduction in the incumbent’s vote share of 2.2%-points.

Note that as compared to a naive OLS, this takes into account that (non-random) non-compliance takes place instead of disregarding non-compliance. Nor does it bulk together different effects as the ITT does in case of non-compliance.

Exercise 24: Plot the estimated coefficients from (i) the 2SLS regression and (ii) the naive OLS.

There are many ways to do this. The most common package to plot coefficients is coefplot, but depending the estimator used, some adjustment might. Sometimes it can even be necessary to manually retrieve coefficients and confidence intervals.

Note that the 2SLS regression increases the uncertainty around the point estimate compared to the OLS regression.

Show Example

The modelplot command is also a powerful tool to plot your regression output:

#install.packages("modelsummary")
library("modelsummary")

# OLS
m_ols <- lm(KocharianR2 ~ mon_voting_R2, data=hyde2007)

# Combining OLS and IV to a list of models
m_list <- list(OLS = m_ols, IV = iv_reg)
msummary(m_list)

	OLS	IV
(Intercept)	0.693	0.693
	(0.005)	(0.005)
mon_voting_R2	-0.020	-0.022
	(0.008)	(0.011)
Num.Obs.	1763	1763
R2	0.003	0.003
R2 Adj.	0.003	0.003
AIC	-1423.1
BIC	-1406.7
Log.Lik.	714.547
F	6.085

# Plotting the Coefficients
modelplot(m_list, conf_level = 0.95, coef_omit = "Intercept")

# You can also customise the plot using the ggplot syntax:
modelplot(m_list, conf_level = 0.95, coef_omit = "Intercept", coef_rename = c("mon_voting_R2"="Monitoring (R2)")) +
  scale_color_brewer(type = 'qual') +
  ggtitle("Coefficient Plot") +
  xlab("Coefficients and 95% Confidence Intervals") +
  theme_classic()

Lab 2: Randomization & Non-Compliance

Week 2 HT 2022

Lecture Slides & Data

Recap

Selection bias

Randomisation

One-sided compliance

Seminar Overview

Experiments

Election Monitoring

Round 2 of monitoring

Randomisation

Non-Compliance

Intent-To-Treatment (ITT) & Complier Average Causal Effect (CACE)