Preliminary Analysis

<<<<<<< HEAD ======= >>>>>>> 6056c29c711690a9ae14a8e4ec179ee3356985e0

Importing Packages

library(tidyverse)
library(readxl)
library(corrplot)
library(ggthemes)
theme_set(theme_clean())

Importing and Cleaning Data

district_data <- read_excel("../raw/NHGIS_District_data.xlsx")

Fortunately, there isn’t much initial restructuring required of this data.

names(district_data)

##  [1] "School ID"                                       
##  [2] "State"                                           
##  [3] "Geographic School District"                      
##  [4] "Children 5-17 (SAIPE Estimate)"                  
##  [5] "% Poverty (SAIPE Estimate)"                      
##  [6] "% Single Parent Estimate"                        
##  [7] "Single Parent Margin of Error"                   
##  [8] "% HHs With Vulnerable Job Estimate"              
##  [9] "Vulnerable Job Margin of Error"                  
## [10] "% Crowded Conditions Estimate"                   
## [11] "HH With Crowded Conditions Margin of Error"      
## [12] "% No Computer or Internet Estimate"              
## [13] "No Computer or Internet Margin of Error"         
## [14] "% Children with Disability"                      
## [15] "Children with Disability Margin of Error"        
## [16] "% Linguistically Isolated Children"              
## [17] "Linguistically Isolated Children Margin of Error"

Let’s work on renaming our variables to more R-friendly names.

names_list <- c(
    "school_ID", 
    "state", 
    "dist",  # Geographic School District 
    "children", # Children 5-17 (SAIPE Estimate)
    "pct_pov", # % Poverty (SAIPE Estimate)
    "pct_SP", # % Single Parent Estimate
    "SP_MOE", # Single Parent Margin of Error
    "pct_HHVJ", # % HHs With Vulnerable Job Estimate
    "HHVJ_MOE", # Vulnerable Job Margin of Error
    "pct_CC", # % Crowded Conditions Estimate
    "CC_MOE", # HH With Crowded Conditions Margin of Error
    "pct_NCI", # % No Computer or Internet Estimate
    "nci_MOE", # No Computer or Internet Margin of Error
    "pct_CD", # % Children with Disability
    "CD_MOE", # Children with Disability Margin of Error
    "pct_CLI", # % Linguistically Isolated Children
    "CLI_MOE" # Linguistically Isolated Children Margin of Error
)
names(district_data) <- names_list
district_pcts <- district_data %>%
    select(-ends_with("MOE")) 

We make a new dataset, district_pcts, which contains all the values without any of the margins of error.

We can now do some preliminary analysis on this data. Let’s first see if there’s any NA/missing data:

district_data %>% 
    is.na() %>% 
    colSums()

## school_ID     state      dist  children   pct_pov    pct_SP    SP_MOE  pct_HHVJ 
##         0         0         0         0         0         0         0         0 
##  HHVJ_MOE    pct_CC    CC_MOE   pct_NCI   nci_MOE    pct_CD    CD_MOE   pct_CLI 
##         0         0         0         0         0         0         0         0 
##   CLI_MOE 
##         0

Fortunately, there don’t appear to be any NA values in any of the percentage columns.

Analysis

Variable Correlations

Let’s check out some variable correlations:

district_pcts[,4:ncol(district_pcts)] %>% 
    cor() %>%
    corrplot(method = "number")

Based on the correlations, it might make sense to plot some variables against each other to see some initial trends. We will choose a cutoff correlation value of 0.3 to distinguish variables which may be of interest. This is completely arbitrary, but limits the amount of plots we need to make such that the analysis is not entirely cumbersome.

Additionally, we will only consider school districts which have more than 100 students. This eliminates outlying values which appear for low numbers of students, such as measures equal to 100%. As an example, we can look at the variable pct_pov; there are a few data points at the 100% mark, so let’s look at these values:

district_data %>% 
    filter(pct_pov == 1) %>% 
    arrange(pct_pov) %>%
    select(state, dist, children, pct_pov)

## # A tibble: 2 × 4
##   state      dist                       children pct_pov
##   <chr>      <chr>                         <dbl>   <dbl>
## 1 Maine      Seboeis Plantation               17       1
## 2 New Mexico Wagon Mound Public Schools       59       1

Because of this low number of children affecting the values of the variables so much, we’ll set 100 to be that cutoff value and continue with analysis. Additionally, since we have over 10,000 rows in our dataset, we randomly sample 1000 of these values for each of the following plots.

Correlated Variable Plots

district_filtered <- district_data %>%
    filter(children > 100)
nrow(district_data) - nrow(district_filtered)

## [1] 790

ggplot(district_filtered[sample(1000),],
       aes(x = pct_pov, 
           y = pct_SP
       )) + 
    geom_point(alpha = 0.1) + 
    geom_smooth()

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

<<<<<<< HEAD

=======

>>>>>>> 6056c29c711690a9ae14a8e4ec179ee3356985e0

ggplot(district_filtered[sample(1000),],
       aes(x = pct_pov, 
           y = pct_CC
       )) + 
    geom_point(alpha = 0.1) + 
    geom_smooth()

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

<<<<<<< HEAD

=======

>>>>>>> 6056c29c711690a9ae14a8e4ec179ee3356985e0

ggplot(district_filtered[sample(1000),],
       aes(x = pct_pov, 
           y = pct_NCI
       )) + 
    geom_point(alpha = 0.1) + 
    geom_smooth()

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

<<<<<<< HEAD

=======

>>>>>>> 6056c29c711690a9ae14a8e4ec179ee3356985e0

ggplot(district_filtered[sample(1000),],
       aes(x = pct_SP, 
           y = pct_NCI
       )) + 
    geom_point(alpha = 0.1) + 
    geom_smooth()

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

<<<<<<< HEAD

=======

>>>>>>> 6056c29c711690a9ae14a8e4ec179ee3356985e0

ggplot(district_filtered[sample(1000),],
       aes(x = pct_CC, 
           y = pct_NCI
       )) + 
    geom_point(alpha = 0.1) + 
    geom_smooth()

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

<<<<<<< HEAD

=======

>>>>>>> 6056c29c711690a9ae14a8e4ec179ee3356985e0

ggplot(district_filtered[sample(1000),],
       aes(x = pct_CC, 
           y = pct_CLI
       )) + 
    geom_point(alpha = 0.1) + 
    geom_smooth()

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

<<<<<<< HEAD

=======

>>>>>>> 6056c29c711690a9ae14a8e4ec179ee3356985e0

Immediately from all of these plots, we can notice that the vast majority of values for any given predictor are below 50%. We can notice that, similar to what our correlation matrix would have us believe, there appears to be a positive correlation between all of these variables.

Distribution of Number of Students

To get a better idea at what sorts of numbers we’re looking at in terms of the number of children in each shcool district, let’s look at a summary of the distribution. We’ll look at all school districts rather than just those with more than 100 students:

summary(district_data$children)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##       0     445    1264    4353    3458 1213005

The mean of this distribution is well above the third quartile, implying that this data is heavily right-skewed. This makes sense, given the fact that there are so many small school districts and likely a few very large ones.

Because of the right-skewed nature of this data, let’s look at a log-transformed boxplot

district_data %>% 
    filter(children > 0) %>%
    ggplot(aes(x = children)) + 
    geom_boxplot() + 
    geom_jitter(aes(y = 0), alpha = 0.01, height = 0.05) + 
    scale_x_log10() + 
    geom_density()

<<<<<<< HEAD

=======

>>>>>>> 6056c29c711690a9ae14a8e4ec179ee3356985e0

Interestingly, when the number of children in each district is log-transformed, the density plot seems to be relatively symmetric about the median. Perhaps there’s some sort of log-normal distribution underlying the distribution of children in school districts in the U.S.?

Additionally, let’s see which school district has over a million estimated children aged 5 to 17.

district_data %>% 
    select(state, dist, children) %>%
    arrange(desc(children)) %>%
    head(5)

## # A tibble: 5 × 3
##   state       dist                                  children
##   <chr>       <chr>                                    <dbl>
## 1 New York    New York City Department Of Education  1213005
## 2 California  Los Angeles Unified School District     724446
## 3 Puerto Rico Puerto Rico                             529844
## 4 Illinois    Chicago Public School District 299      399883
## 5 Florida     Dade County School District             396516

Makes sense. Is this actually a school district, or is it a conglomerate of school districts? We’ll assume that the data is good, and that the New York City Department of Education is, in fact, classified as a school district.

<<<<<<< HEAD ======= >>>>>>> 6056c29c711690a9ae14a8e4ec179ee3356985e0