Skip to content

Commit f080d5e

Browse files
Megha18jainMegha18jain
authored
Merge branch 'master' into stormnoaa
2 parents 7b25643 + 74311f7 commit f080d5e

48 files changed

Lines changed: 4629 additions & 554 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎scripts/us_census/pep/monthly_population_estimate/preprocess.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -318,7 +318,7 @@ def _transform_data(self, df: pd.DataFrame, file: str) -> None:
318318
ascending=False,
319319
inplace=True)
320320
# Data for 2020 exists in two sources, causing overlap. We'll eliminate duplicates
321-
#self._df.drop_duplicates("Date", keep="last", inplace=True)
321+
self._df.drop_duplicates("Date", keep="last", inplace=True)
322322
self._df.drop(['date_range'], axis=1, inplace=True)
323323
float_col = self._df.select_dtypes(include=['float64'])
324324
for col in float_col.columns.values:

‎statvar_imports/child_birth/metadata.csv‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
1-
parameter,value
2-
#places_within,country/USA
3-
output_columns,"observationAbout,observationDate,value,variableMeasured,observationPeriod"
4-
header_rows,1
5-
mapped_columns,5
6-
dc_api_root,https://api.datacommons.org
1+
parameter,value
2+
#places_within,country/USA
3+
output_columns,"observationAbout,observationDate,value,variableMeasured,observationPeriod"
4+
header_rows,1
5+
mapped_columns,5

‎statvar_imports/crdc/instructional_wifi_devices/common_pvmap.csv‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
1-
key,,,,,,,,
1+
key,p1,v1,p2,v2,,,,
22
COMBOKEY,#Format,observationAbout=nces/{Data:0>12},statType,measuredValue,,,,
33
YEAR,observationDate,{Data},,,,,,
4-
SCH_INTERNET_FIBER,#ignore,ignore
5-
SCH_INTERNET_WIFI,#ignore,ignore
6-
SCH_INTERNET_SCHDEV,#ignore,ignore
7-
SCH_INTERNET_STUDDEV,#ignore,ignore
4+
SCH_INTERNET_FIBER,#ignore,ignore,value,{Number},,,,
5+
SCH_INTERNET_WIFI,#ignore,ignore,value,{Number},,,,
6+
SCH_INTERNET_SCHDEV,#ignore,ignore,value,{Number},,,,
7+
SCH_INTERNET_STUDDEV,#ignore,ignore,value,{Number},,,,
88
SCH_INTERNET_WIFIENDEV,populationType,School,measuredProperty,count,internetSubscriptionType,WifiEnabledDevice,value,{Number}
99
,,,,,,,,
1010
JJ:Yes,educationalInstitutionType,JuvenileJusticeFacility,,,,,,

‎statvar_imports/us_urban_school/covid_directional_indicators/README.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
#### Copyright 2025 Google LLC
2+
####
3+
#### Licensed under the Apache License, Version 2.0 (the "License");
4+
#### you may not use this file except in compliance with the License.
5+
#### You may obtain a copy of the License at
6+
####
7+
#### https://www.apache.org/licenses/LICENSE-2.0
8+
####
9+
#### Unless required by applicable law or agreed to in writing, software
10+
#### distributed under the License is distributed on an "AS IS" BASIS,
11+
#### WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
#### See the License for the specific language governing permissions and
13+
#### limitations under the License.
14+
15+
-----
16+
17+
# US_UrbanSchool_Covid_Directional_Indicators Import
18+
19+
The primary purpose of this import is to capture information about the method and extent of instruction offered by schools during the Coronavirus Pandemic, which significantly affected traditional learning environments.
20+
21+
-----
22+
23+
## ⚙️ Workflow
24+
25+
The workflow for this data import involves two main steps: downloading the necessary files and then processing them.
26+
27+
### Step 1: Download the Source Data
28+
29+
To acquire the necessary data files, execute the download script `download.py`.
30+
31+
```bash
32+
python3 download.py --data_type=covid
33+
```
34+
35+
All downloaded files will be stored in the directory `input_files`.
36+
37+
### Autorefresh type
38+
39+
This import uses a fully automated refresh process.
40+
41+
-----
42+
43+
### Step 2: Process the Files
44+
45+
After successfully downloading the files, run the processing script to generate the final output artifacts. There are two ways to do this:
46+
47+
**Option A: Use the `run.sh` script**
48+
49+
The `run.sh` script automates the processing of all the downloaded files.
50+
51+
**Run the following command:**
52+
53+
```bash
54+
sh run.sh
55+
```
56+
57+
**Option B: Manually Execute the Processing Script**
58+
59+
#### **Execution Commands**
60+
61+
The final output is generated by processing the downloaded data using the stat_var_processor.py script, which is located in the data/tools/statvar_importer/ directory.
62+
63+
```bash
64+
python3 stat_var_processor.py \
65+
--input_data=../../../statvar_imports/us_urban_school/covid_directional_indicators/input_files/*.csv \
66+
--pv_map=../../../statvar_imports/us_urban_school/covid_directional_indicators/covid_directional_indicators_pv_map1.csv \
67+
--config_file=../../../statvar_imports/covid_directional_indicators/covid_directional_indicators/covid _directional_indicators_metadata.csv \
68+
--places_resolved_csv=../../../statvar_imports/covid_directional_indicators/covid_directional_indicators/place_resolved.csv \
69+
--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \
70+
--output_path=../../../statvar_imports/us_urban_school/covid_directional_indicators/output_files/covid_directional_indicators_output1
71+
```
72+
73+
```bash
74+
python3 stat_var_processor.py \
75+
--input_data=../../../statvar_imports/us_urban_school/covid_directional_indicators/input_files/*.csv \
76+
--pv_map=../../../statvar_imports/us_urban_school/covid_directional_indicators/covid_directional_indicators_pv_map2.csv \
77+
--config_file=../../../statvar_imports/covid_directional_indicators/covid_directional_indicators/covid _directional_indicators_metadata.csv \
78+
--places_resolved_csv=../../../statvar_imports/covid_directional_indicators/covid_directional_indicators/place_resolved.csv \
79+
--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \
80+
--output_path=../../../statvar_imports/us_urban_school/covid_directional_indicators/output_files/covid_directional_indicators_output2
81+
```
82+
83+
```bash
84+
python3 stat_var_processor.py \
85+
--input_data=../../../statvar_imports/us_urban_school/covid_directional_indicators/input_files/*.csv \
86+
--pv_map=../../../statvar_imports/us_urban_school/covid_directional_indicators/covid_directional_indicators_pv_map3.csv \
87+
--config_file=../../../statvar_imports/covid_directional_indicators/covid_directional_indicators/covid _directional_indicators_metadata.csv \
88+
--places_resolved_csv=../../../statvar_imports/covid_directional_indicators/covid_directional_indicators/place_resolved.csv \
89+
--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \
90+
--output_path=../../../statvar_imports/us_urban_school/covid_directional_indicators/output_files/covid_directional_indicators_output3
91+
```
92+
```bash
93+
python3 stat_var_processor.py \
94+
--input_data=../../../statvar_imports/us_urban_school/covid_directional_indicators/input_files/*.csv \
95+
--pv_map=../../../statvar_imports/us_urban_school/covid_directional_indicators/covid_directional_indicators_pv_map4.csv \
96+
--config_file=../../../statvar_imports/covid_directional_indicators/covid_directional_indicators/covid _directional_indicators_metadata.csv \
97+
--places_resolved_csv=../../../statvar_imports/covid_directional_indicators/covid_directional_indicators/place_resolved.csv \
98+
--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \
99+
--output_path=../../../statvar_imports/us_urban_school/covid_directional_indicators/output_files/covid_directional_indicators_output4
100+
```
101+

‎statvar_imports/us_urban_school/covid_directional_indicators/covid_directional_indicators_metadata.csv‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
parameter,value
2+
header_rows,1
3+
output_columns,"observationDate,observationAbout,variableMeasured,value"
4+
#comments,manually added YEAR column for resolving observationDate
5+
url,https://civilrightsdata.ed.gov/data
6+
#multi_value_properties,"observationDate, observationAbout"
7+
#input_rows,150

‎statvar_imports/us_urban_school/covid_directional_indicators/covid_directional_indicators_pv_map1.csv‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
key,,,,,,,,,,
2+
YEAR,observationDate,{Data},populationType,PublicSchool,measuredProperty,count,statType,measuredValue,medicalCondition,COVID19
3+
LEA_STATE,observationAbout,{Data},,,,,,,,
4+
SCH_DIND_INSTRUCTIONTYPE:A,instructionMode,InstructionInPerson ,"#Aggregate
5+
",sum,value,1,,,,
6+
SCH_DIND_INSTRUCTIONTYPE:B,instructionMode,InstructionInRemote ,"#Aggregate
7+
",sum,value,1,,,,
8+
SCH_DIND_INSTRUCTIONTYPE:C,instructionMode,InstructionInHybrid,"#Aggregate
9+
",sum,value,1,,,,
10+
SCH_DIND_INSTRUCTIONTYPE:D,instructionMode,NoEffect,"#Aggregate
11+
",sum,value,1,,,,

‎statvar_imports/us_urban_school/covid_directional_indicators/covid_directional_indicators_pv_map2.csv‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
key,,,,,,,,,,
2+
YEAR,observationDate,{Data},populationType,PublicSchool,measuredProperty,count,statType,measuredValue,medicalCondition,COVID19
3+
LEA_STATE,observationAbout,{Data},,,,,,,,
4+
SCH_DIND_INSTRUCTIONTYPE:A,instructionMode,InstructionInPerson ,,,,,,,,
5+
SCH_DIND_INSTRUCTIONTYPE:B,instructionMode,InstructionInRemote ,,,,,,,,
6+
SCH_DIND_INSTRUCTIONTYPE:C,instructionMode,InstructionInHybrid,,,,,,,,
7+
SCH_DIND_INSTRUCTIONTYPE:D,instructionMode,NoEffect,,,,,,,,
8+
SCH_DIND_REMOTETYPE:A,studentAvailabilityMode,StudentInPerson,"teacherInstructionMode
9+
",TeachersInRemote,#Aggregate,sum,value,1,,
10+
SCH_DIND_REMOTETYPE:B,studentAvailabilityMode,StudentInRemote,"teacherInstructionMode
11+
",TeachersInRemote,#Aggregate,sum,value,1,,
12+
SCH_DIND_REMOTETYPE:C,studentAvailabilityMode,StudentInPersonAndRemote,"teacherInstructionMode
13+
",TeachersInRemote,#Aggregate,sum,value,1,,

‎statvar_imports/us_urban_school/covid_directional_indicators/covid_directional_indicators_pv_map3.csv‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
key,,,,,,,,,,
2+
YEAR,observationDate,{Data},populationType,PublicSchool,measuredProperty,count,statType,measuredValue,medicalCondition,COVID19
3+
LEA_STATE,observationAbout,{Data},,,,,,,,
4+
SCH_DIND_INSTRUCTIONTYPE:A,instructionMode,InstructionInPerson ,,,,,,,,
5+
SCH_DIND_INSTRUCTIONTYPE:B,instructionMode,InstructionInRemote ,,,,,,,,
6+
SCH_DIND_INSTRUCTIONTYPE:C,instructionMode,InstructionInHybrid,,,,,,,,
7+
SCH_DIND_INSTRUCTIONTYPE:D,instructionMode,NoEffect,,,,,,,,
8+
SCH_DIND_REMOTETYPE:A,studentAvailabilityMode,StudentInPerson,"teacherInstructionMode
9+
",TeachersInRemote,,,,,,
10+
SCH_DIND_REMOTETYPE:B,studentAvailabilityMode,StudentInRemote,"teacherInstructionMode
11+
",TeachersInRemote,,,,,,
12+
SCH_DIND_REMOTETYPE:C,studentAvailabilityMode,StudentInPersonAndRemote,"teacherInstructionMode
13+
",TeachersInRemote,,,,,,
14+
SCH_DIND_REMOTEAMOUNT:A,remoteInstructionTime,5OrMoreHours,#Aggregate,sum,value,1,,,,
15+
SCH_DIND_REMOTEAMOUNT:B,remoteInstructionTime,3To4Hours,#Aggregate,sum,value,1,,,,
16+
SCH_DIND_REMOTEAMOUNT:C,remoteInstructionTime,1To2Hours,#Aggregate,sum,value,1,,,,
17+
SCH_DIND_REMOTEAMOUNT:D,remoteInstructionTime,LessThan1Hour,#Aggregate,sum,value,1,,,,

‎statvar_imports/us_urban_school/covid_directional_indicators/covid_directional_indicators_pv_map4.csv‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
key,,,,,,,,,,
2+
YEAR,observationDate,{Data},populationType,PublicSchool,measuredProperty,count,statType,measuredValue,medicalCondition,COVID19
3+
LEA_STATE,observationAbout,{Data},,,,,,,,
4+
SCH_DIND_INSTRUCTIONTYPE:A,instructionMode,InstructionInPerson ,,,,,,,,
5+
SCH_DIND_INSTRUCTIONTYPE:B,instructionMode,InstructionInRemote ,,,,,,,,
6+
SCH_DIND_INSTRUCTIONTYPE:C,instructionMode,InstructionInHybrid,,,,,,,,
7+
SCH_DIND_INSTRUCTIONTYPE:D,instructionMode,NoEffect,,,,,,,,
8+
SCH_DIND_REMOTETYPE:A,studentAvailabilityMode,StudentInPerson,"teacherInstructionMode
9+
",TeachersInRemote,,,,,,
10+
SCH_DIND_REMOTETYPE:B,studentAvailabilityMode,StudentInRemote,"teacherInstructionMode
11+
",TeachersInRemote,,,,,,
12+
SCH_DIND_REMOTETYPE:C,studentAvailabilityMode,StudentInPersonAndRemote,"teacherInstructionMode
13+
",TeachersInRemote,,,,,,
14+
SCH_DIND_REMOTEAMOUNT:A,remoteInstructionTime,5OrMoreHours,,,,,,,,
15+
SCH_DIND_REMOTEAMOUNT:B,remoteInstructionTime,3To4Hours,,,,,,,,
16+
SCH_DIND_REMOTEAMOUNT:C,remoteInstructionTime,1To2Hours,,,,,,,,
17+
SCH_DIND_REMOTEAMOUNT:D,remoteInstructionTime,LessThan1Hour,,,,,,,,
18+
SCH_DIND_REMOTEPERCT:A,remoteInstructionPercent,[1 25 PercentStudents],#Aggregate,sum,value,1,,,,
19+
SCH_DIND_REMOTEPERCT:B,remoteInstructionPercent,[26 50 PercentStudents],#Aggregate,sum,value,1,,,,
20+
SCH_DIND_REMOTEPERCT:C,remoteInstructionPercent,[51 75 PercentStudents],#Aggregate,sum,value,1,,,,
21+
SCH_DIND_REMOTEPERCT:D,remoteInstructionPercent,Above75PercentStudents,#Aggregate,sum,value,1,,,,

0 commit comments

Comments
 (0)