Make Decisions about the Civis Dataset

[DForbush]

Importing the Civis data is cumbersome for two reasons:

1. In the Global File, the following code (lines 49-53) takes a long time to run:
   civistable <- "cic.pdb2019trv3_us" civisdata <- read_civis(civistable, database="City of Chicago") #this will take a minute or two civisdata <- as.data.table(civisdata) civisdata <- civisdata[match(shp_tracts$GEOID, civisdata$gidtr)]

This is because the entire dataset is imported and then matched with the Chicago-specific census tracts. Is there a way, in the read_civis import line, to only import the Chicago-specific census tracts?

2. There are 522 columns in the Civis data set. We probably don't need most of these, and it makes the data unwieldy and slow. How should we filter which columns we want to use?

[DForbush]

Another thought about this: the column names in the Civis dataset, based off of the Census Planning Database, are impossible to understand on their own. We need to rely on the Census documentation (available here ) to interpret the variables. There's no way that anybody else reading these reports is going to be able to understand the column names on their own. This isn't a problem for the data table, because it is easy to rename the columns. But it is a problem for the map- if you click on specific Census tracts in the map, the column name and value appears. We probably will need to rename the columns to make them understandable. Should we do that? And if so, where is the best place in the code to do it?

[geneorama]

I agree, I would consider keeping a lookup table

[sherryshenker]

I've shared you both on a table called cic.data_dict. This has a mapping of raw columns to more human-readable column names. Let me know if its helpful!