CDCWonder_NNDSS_InfectiousAnnual Measles data refresh PR to bring configs

statvar_imports/us_cdc/nndss_annual_tables/README.md

@@ -0,0 +1,37 @@
+# CDCWonder NNDSS InfectiousAnnual Import
+
+## About the Dataset
+Notifiable Infectious Diseases Data: Annual tables from CDC WONDER which has the incident counts of different infectious diseases per year and are aggregated based on demographic factors. The statistics are reported by the 50 states, New York City, the District of Columbia, and the U.S. territories
+
+### Download URL
+
+Source URL {https://wonder.cdc.gov/nndss/nndss_annual_tables_menu.asp}.
+
+
+To download the latest versions of ALL datasets available, run the following command. Files will be downloaded and extracted to a nndss_Annual_data folder.
+
+### License
+
+This dataset is available for public use, license is available at https://www.cdc.gov/other/agencymaterials.html
+
+
+### Downloading and Processing Data
+
+
+   To download the data follow the below steps
+
+```
+	1.  Select the radio button for the category.
+   2.  Select all locations, all years, and all diseases.
+   3.  Save your selections.
+   4.  The corresponding table will become visible.
+   5.  To download the data, click on "Export."
+```
+
+   To execute the data run the below command, keep chaning the config
+
+```bash
+   python3 ../../../tools/statvar_importer/stat_var_processor.py --input_data='input_files/region_state/NNDSS_Annual_Summary_Data_2023.csv' --pv_map=regional_state_pvmap.csv --config_file=regional_state_metadata.csv --output_path=output/nndss_regional_state --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf |& tee check.log
+``` 
+
+

statvar_imports/us_cdc/nndss_annual_tables/measles_pvmap.csv

@@ -0,0 +1,43 @@
+prop,key,val,,,,
+Year,observationDate,{Number},populationType,MedicalConditionIncident,observationAbout,country/USA
+Sex Code:1,gender,Female,,,,
+Sex Code:2,gender,Male,,,,
+Sex Code:99,gender,CDC_GenderUnknownOrNotStated,,,,
+Age Code:1,age,YearsUpto1,
+Age Code:2,age,Years1To4,
+Age Code:3,age,Years5To14,
+Age Code:4,age,Years15To24,
+Age Code:5,age,Years25To39,
+Age Code:6,age,Years40To64,
+Age Code:7,age,Years65Onwards,
+Age Code:99,age,USC_AgeNotStated,
+Race Code:1,race,AmericanIndianAndAlaskaNativeAlone,
+Race Code:2,race,AsianOrPacificIslander,
+Race Code:3,race,BlackOrAfricanAmericanAlone,
+Race Code:4,race,WhiteAlone,
+Race Code:5,race,CDC_OtherRace,
+Race Code:99,race,CDC_RaceUnknownOrNotStated,
+Race Code:98,#ignore,Race,,
+
+Ethnicity Code:1,ethnicity,HispanicOrLatino
+Ethnicity Code:non-,ethnicity,NotHispanicOrLatino
+Ethnicity Code:99,ethnicity,CDC_EthnicityUnknownOrNotStated
+Ethnicity Code:98,#ignore,Ignore
+Regions Code:U.S.,residentStatus,USResident
+Regions Code:REGION1,#ignore,Ignore,
+Regions Code:REGION2,#ignore,Ignore,
+Regions Code:REGION3,#ignore,Ignore,
+Regions Code:REGION4,#ignore,Ignore,
+Regions Code:REGION5,#ignore,Ignore,
+Regions Code:REGION6,#ignore,Ignore,
+Regions Code:REGION7,#ignore,Ignore,
+Regions Code:REGION8,#ignore,Ignore,
+Regions Code:REGION9,#ignore,Ignore,
+Regions Code:REGION10,#ignore,Ignore,
+Regions Code:REGION11,residentStatus,NonUSResident
+
+Case Count,value,{Number},measurementMethod,CDC_NNDSS_Diseases_AnnualTables,observationPeriod,P1Y
+
+10140,medicalCondition,Measles,,,,
+10500,medicalCondition,Measles,medicalStatus,IndigenousCase,,
+10510,medicalCondition,Measles,medicalStatus,ImportedCase,,

statvar_imports/us_cdc/nndss_annual_tables/measles_pvmap_region.csv

@@ -0,0 +1,39 @@
+prop,key,val,,,,
+Year,observationDate,{Number},populationType,MedicalConditionIncident,observationAbout,country/USA
+Sex Code:1,gender,Female,,,,
+Sex Code:2,gender,Male,,,,
+Sex Code:99,gender,CDC_GenderUnknownOrNotStated,,,,
+Age Code:1,age,YearsUpto1,
+Age Code:2,age,Years1To4,
+Age Code:3,age,Years5To14,
+Age Code:4,age,Years15To24,
+Age Code:5,age,Years25To39,
+Age Code:6,age,Years40To64,
+Age Code:7,age,Years65Onwards,
+Age Code:99,age,USC_AgeNotStated,
+Race Code:1,race,AmericanIndianAndAlaskaNativeAlone,
+Race Code:2,race,AsianOrPacificIslander,
+Race Code:3,race,BlackOrAfricanAmericanAlone,
+Race Code:4,race,WhiteAlone,
+Race Code:5,race,CDC_OtherRace,
+Race Code:99,race,CDC_RaceUnknownOrNotStated,
+Race Code:98,#ignore,Race,,
+
+Ethnicity Code:1,ethnicity,HispanicOrLatino
+Ethnicity Code:non-,ethnicity,NotHispanicOrLatino
+Ethnicity Code:99,ethnicity,CDC_EthnicityUnknownOrNotStated
+Ethnicity Code:98,#ignore,Ignore
+Regions Code:U.S.,residentStatus,USResident
+Regions Code:REGION1,#ignore,Ignore,
+Regions Code:REGION2,#ignore,Ignore,
+Regions Code:REGION3,#ignore,Ignore,
+Regions Code:REGION4,#ignore,Ignore,
+Regions Code:REGION5,#ignore,Ignore,
+Regions Code:REGION6,#ignore,Ignore,
+Regions Code:REGION7,#ignore,Ignore,
+Regions Code:REGION8,#ignore,Ignore,
+Regions Code:REGION9,#ignore,Ignore,
+Regions Code:REGION10,#ignore,Ignore,
+Regions Code:REGION11,residentStatus,NonUSResident
+
+Case Count,value,{Number},measurementMethod,CDC_NNDSS_Diseases_AnnualTables,observationPeriod,P1Y

statvar_imports/us_cdc/nndss_annual_tables/regional_state_metadata.csv

@@ -0,0 +1,4 @@
+parameter,value
+header_rows,1
+mapped_columns,6
+output_columns,"observationDate,observationAbout,variableMeasured,value,measurementMethod,observationPeriod"

statvar_imports/us_cdc/nndss_annual_tables/regional_state_pvmap.csv

@@ -0,0 +1,141 @@
+prop,key,val,,,,
+Year,observationDate,{Number},populationType,MedicalConditionIncident
+Case Count,value,{Number},measurementMethod,CDC_NNDSS_Diseases_AnnualTables,observationPeriod,P1Y
+10140,medicalCondition,Measles,,,,
+10500,medicalCondition,Measles,medicalStatus,IndigenousCase,,
+10510,medicalCondition,Measles,medicalStatus,ImportedCase,,
+
+
+"Ala.",observationAbout,"geoId/01",
+"Alabama",observationAbout,"geoId/01",
+"Alaska",observationAbout,"geoId/02",
+"Amer. Samoa",observationAbout,"geoId/60",
+"American Samoa",observationAbout,"geoId/60",
+"Ariz.",observationAbout,"geoId/04",
+"Arizona",observationAbout,"geoId/04",
+"Ark.",observationAbout,"geoId/05",
+"Arkansas",observationAbout,"geoId/05",
+"C.N.M.I.",observationAbout,"geoId/69",
+"Calif.",observationAbout,"geoId/06",
+"California",observationAbout,"geoId/06",
+"Colo.",observationAbout,"geoId/08",
+"Colorado",observationAbout,"geoId/08",
+"Commonwealth of Northern Mariana Islands",observationAbout,"geoId/69",
+"Commonwealth of the Northern Mariana Islands",observationAbout,"geoId/69",
+"Conn.",observationAbout,"geoId/09",
+"Connecticut",observationAbout,"geoId/09",
+"D.C.",observationAbout,"geoId/11",
+"Del.",observationAbout,"geoId/10",
+"Delaware",observationAbout,"geoId/10",
+"District of Columbia",observationAbout,"geoId/11",
+"E.N. CENTRAL",observationAbout,"usc/EastNorthCentralDivision",
+"E.S. CENTRAL",observationAbout,"usc/EastSouthCentralDivision",
+"East North Central",observationAbout,"usc/EastNorthCentralDivision",
+"East South Central",observationAbout,"usc/EastSouthCentralDivision",
+"Fla.",observationAbout,"geoId/12",
+"Florida",observationAbout,"geoId/12",
+"Ga.",observationAbout,"geoId/13",
+"Georgia",observationAbout,"geoId/13",
+"Guam",observationAbout,"geoId/66",
+"Hawaii",observationAbout,"geoId/15",
+"Idaho",observationAbout,"geoId/16",
+"Ill.",observationAbout,"geoId/17",
+"Illinois",observationAbout,"geoId/17",
+"Ind.",observationAbout,"geoId/18",
+"Indiana",observationAbout,"geoId/18",
+"Iowa",observationAbout,"geoId/19",
+"Kans.",observationAbout,"geoId/20",
+"Kansas",observationAbout,"geoId/20",
+"Kentucky",observationAbout,"geoId/21",
+"Ky.",observationAbout,"geoId/21",
+"La.",observationAbout,"geoId/22",
+"Louisiana",observationAbout,"geoId/22",
+"MID. ATLANTIC",observationAbout,"usc/MiddleAtlanticDivision",
+"MOUNTAIN",observationAbout,"usc/MountainDivision",
+"Maine",observationAbout,"geoId/23",
+"Maryland",observationAbout,"geoId/24",
+"Mass.",observationAbout,"geoId/25",
+"Massachusetts",observationAbout,"geoId/25",
+"Md.",observationAbout,"geoId/24",
+"Mich.",observationAbout,"geoId/26",
+"Michigan",observationAbout,"geoId/26",
+"Middle Atlantic",observationAbout,"usc/MiddleAtlanticDivision",
+"Minn.",observationAbout,"geoId/27",
+"Minnesota",observationAbout,"geoId/27",
+"Miss.",observationAbout,"geoId/28",
+"Mississippi",observationAbout,"geoId/28",
+"Missouri",observationAbout,"geoId/29",
+"Mo.",observationAbout,"geoId/29",
+"Mont.",observationAbout,"geoId/30",
+"Montana",observationAbout,"geoId/30",
+"Mountain",observationAbout,"usc/MountainDivision",
+"N. Dak.",observationAbout,"geoId/38",
+"N. Mex.",observationAbout,"geoId/35",
+"N.C.",observationAbout,"geoId/37",
+"N.H.",observationAbout,"geoId/33",
+"N.J.",observationAbout,"geoId/34",
+"NEW ENGLAND",observationAbout,"usc/NewEnglandDivision",
+"Nebr.",observationAbout,"geoId/31",
+"Nebraska",observationAbout,"geoId/31",
+"Nev.",observationAbout,"geoId/32",
+"Nevada",observationAbout,"geoId/32",
+"New England",observationAbout,"usc/NewEnglandDivision",
+"New Hampshire",observationAbout,"geoId/33",
+"New Jersey",observationAbout,"geoId/34",
+"New Mexico",observationAbout,"geoId/35",
+"New York",observationAbout,"geoId/36",
+"Non-U.S. Residents",#ignore,"Ignore",
+"North Carolina",observationAbout,"geoId/37",
+"North Dakota",observationAbout,"geoId/38",
+"Ohio",observationAbout,"geoId/39",
+"Okla.",observationAbout,"geoId/40",
+"Oklahoma",observationAbout,"geoId/40",
+"Oreg.",observationAbout,"geoId/41",
+"Oregon",observationAbout,"geoId/41",
+"P.R.",observationAbout,"geoId/72",
+"PACIFIC",observationAbout,"usc/PacificDivision",
+"Pa.",observationAbout,"geoId/42",
+"Pacific",observationAbout,"usc/PacificDivision",
+"Pennsylvania",observationAbout,"geoId/42",
+"Puerto Rico",observationAbout,"geoId/72",
+"R.I.",observationAbout,"geoId/44",
+"Republic of Palau",observationAbout,"country/PLW",
+"Republic of the Marshall Islands",observationAbout,"country/MHL",
+"Rhode Island",observationAbout,"geoId/44",
+"S. ATLANTIC",observationAbout,"usc/SouthAtlanticDivision",
+"S. Dak.",observationAbout,"geoId/46",
+"S.C.",observationAbout,"geoId/45",
+"South Atlantic",observationAbout,"usc/SouthAtlanticDivision",
+"South Carolina",observationAbout,"geoId/45",
+"South Dakota",observationAbout,"geoId/46",
+"Tenn.",observationAbout,"geoId/47",
+"Tennessee",observationAbout,"geoId/47",
+"Tex.",observationAbout,"geoId/48",
+"Texas",observationAbout,"geoId/48",
+"U.S. Residents",#ignore,Ignore,
+"U.S. Residents, excluding U.S. Territories",#ignore,Ignore,,
+"U.S. Territories",#ignore,Ignore,
+"U.S. Virgin Islands",observationAbout,"geoId/78",,
+"US Virgin Islands",observationAbout,"geoId/78",
+"United States",observationAbout,"country/USA",
+"Utah",observationAbout,"geoId/49",
+"V.I.",observationAbout,"geoId/78",
+"Va.",observationAbout,"geoId/51",
+"Vermont",observationAbout,"geoId/50",
+"Virginia",observationAbout,"geoId/51",
+"Vt.",observationAbout,"geoId/50",
+"W. Va.",observationAbout,"geoId/54",
+"W.N. CENTRAL",observationAbout,"usc/WestNorthCentralDivision",
+"W.S. CENTRAL",observationAbout,"usc/WestSouthCentralDivision",
+"Wash.",observationAbout,"geoId/53",
+"Washington",observationAbout,"geoId/53",
+"Washington DC",observationAbout,"geoId/11001",
+"West North Central",observationAbout,"usc/WestNorthCentralDivision",
+"West South Central",observationAbout,"usc/WestSouthCentralDivision",
+"West Virginia",observationAbout,"geoId/54",
+"Wis.",observationAbout,"geoId/55",
+"Wisconsin",observationAbout,"geoId/55",
+"Wyo.",observationAbout,"geoId/56",
+"Wyoming",observationAbout,"geoId/56"
+"New York (excluding New York City)",#ignore,Ignore
+"New York City",#ignore,Ignore