Where does the U.S. Government Keep its Data?

Non-exhaustive list of R Package Sources for getting government data:

Principal Statistical Agency Programs

Inspiration and primary source for all of this information: Statistical Programs of the United States Government Fiscal Year 2017

“The Federal Statistical System of the United States is the decentralized network of federal agencies which produce data about the people, economy, natural resources, and infrastructure of the United States.” (Wikipedia)

I recently got the opportunity to visit the offices of the Committee on National Statistics and I realized I had no idea just how much statistical work the Federal government does. Before this, the only statistical agency I really knew about was the Census Bureau. Based on my complete and total ignorance about all of the work done and data collected by the U.S. federal government, I decided to compile together as many sources of data as I could. Wherever possible, I tried to find APIs. This list is also probably not exhaustive. There is also way more data from state and local governments on data.gov, but I have not found this site to be particularly helpful when just looking for possible interesting data sources. It’s better for very specific searches in my opinion.

These 13 principal agencies are crucial to collecting data on the population and resources of the United States. They “provide critical support for policymaking, program management, and evaluation.” Two of the 13 are independent: the National Center for Science and Engineering Statistics (NCSES), a part of the National Science Foundation (NSF), and the Office of Research, Evaluation, and Statistics (ORES), a part of the Social Security Administration (SSA).

(If you’re exhausted by the alphabet soup already, I suggest you stop reading….)

The report linked at the top contains descriptions of every agency and every statistical program. It also links to every program’s website and provides employment data tables for the 13 principal agencies. Below, the numbers (x/y) are x = the number of statisticians employed and y = the number of full-time employees in the agency as of 2017.

  1. Bureau of Economic Analysis (BEA): (12/509 \(\approx\) 2%)
    • Description: “BEA is responsible for the preparation, development, and interpretation of the Nation’s economic accounts.”
    • Home Agency: Department of Commerce (DOC)
    • Website
    • API
  2. Bureau of Justice Statistics (BJS): (42/69 \(\approx\) 61%)
    • Description: “BJS collects, analyzes, publishes, and disseminates statistical information on all aspects of the criminal justice system; assists State, Tribal, and local governments in gathering and analyzing justice statistics; and disseminates high value information and statistics to inform policymakers, researchers, criminal justice practitioners and the general public.”
    • Home Agency: Department of Justice (DOJ)
    • Website
    • Data
  3. Bureau of Labor Statistics (BLS): (168/2167 \(\approx\) 8%)
    • Description: “BLS is responsible for measuring labor market activity, working conditions, and price changes in the economy.”
    • Home Agency: Department of Labor (DOL)
    • Website
    • API (An R script for accessing their API!!)
  4. Bureau of Transportation Statistics (BTS): (15/84 \(\approx\) 18%)
    • Description: “BTS compiles, analyzes, and disseminates information about the Nation’s transportation systems, including the extent, use, condition, performance, and consequences of those systems.”
    • Home Agency: Department of Transportation (DOT)
    • Website
    • Data (an app)
  5. Census Bureau (Census): (2311/6551 \(\approx\) 35%)
    • Description: “Census measures and disseminates information about the Nation’s ever-changing economy, society, and institutions.”
    • Home Agency: DOC
    • Website
    • APIs (plural!)
  6. Economic Research Service (ERS): (1/359 \(\approx\) .3%)
    • Description: “ERS provides economic and other social science research and analysis to inform public and private decision making on agriculture, food, natural resources, and rural America.”
    • Home Agency: United States Department of Agriculture (USDA)
    • Website: https://www.ers.usda.gov/
    • API Prayer directed at Excel files here: https://www.ers.usda.gov/data-products/
    • Bonus: Cool sounding postdoc gig. Application due November 19, 2018!
  7. Energy Information Administration (EIA): (75/368 \(=\) 20%)
    • Description: “EIA collects, analyzes, and disseminates independent and impartial energy information to promote sound policymaking, efficient markets, and public understanding of energy and its interaction with the economy and the environment.”
    • Home Agency: Department of Energy (DOE)
    • Website: https://www.eia.gov
    • API: https://www.eia.gov/opendata/qb.php
  8. National Agricultural Statistics Service (NASS): (634/1037 \(\approx\) 61%)
    • Description: " NASS collects, summarizes, analyzes, and publishes data on the number of farms and land in farms; acreage, yield, production, and stocks of crops; inventories and production of livestock, including eggs and dairy products; prices received by farmers for products, prices paid for commodities and services, and related indexes; agriculture production and marketing data; cold storage supplies; agricultural chemical use; and other related areas of the agricultural economy."
    • Home Agency: USDA
    • Website: https://www.nass.usda.gov/
    • API: https://quickstats.nass.usda.gov/api (This was hard to find.)
  9. National Center for Education Statistics (NCES): (71/93 \(\approx\) 76%)
  10. National Center for Health Statistics (NCHS): (164/503 \(\approx\) 33%)
  11. National Center for Science and Engineering Statistics (NCSES): (25/52 \(\approx\) 48%)
    • Description: “NCSES is called on to support the collection of statistical data on research and development trends, the science and engineering workforce, U.S. competitiveness, and the condition and progress of the Nation’s science, technology, engineering, and mathematics (STEM) education; to support research using the data it collects and on methodologies in areas related to the work of the Center; and to support the education and training of researchers in the use of its own and other large-scale, nationally representative data sets.”
    • Home Agency: NSF
    • Website: https://www.nsf.gov/statistics/
    • API Prayer and patience: https://www.nsf.gov/statistics/data-tools.cfm
  12. Office of Research, Evaluation, and Statistics (ORES): (3/66 \(\approx\) 5%)
  13. Statistics of Income (SOI): (25/116 \(\approx\) 22%)

Other Federal Statistical Programs, by Department

Of the fifteen US federal government departments, fourteen have additional statistical programs (outside of the 13 principal statistical agencies), and there are even more programs in independent agencies. There are 115 of these statistical programs in total (yes, I counted them). For each one, I just link to the closest thing to a data website I could find.

  1. Department of Agriculture : https://www.usda.gov/media/digital/developer-resources
    1. Agricultural Research Service (ARS) : https://www.ars.usda.gov/research/datasets/
    2. Food and Nutrition Service (FNS) : https://www.fns.usda.gov/data-and-statistics
    3. Foreign Agricultural Service (FAS) : https://www.fas.usda.gov/data
    4. Forest Service (FS) : closest I could find to a data warehouse https://www.fs.fed.us/publications
    5. Natural Resources Conservation Service (NRCS) : https://www.nrcs.usda.gov/wps/portal/nrcs/detail/national/home/?cid=stelprdb1049255
    6. Risk Management Agency (RMA) : https://www.rma.usda.gov/ under the “Tools” menu
    7. World Agricultural Outlook Board (WAOB) : (just some reports) https://www.usda.gov/oce/commodity/index.htm
  2. Department of Commerce : APIs linked + code on Github!!!! https://github.com/CommerceGov/commerce.gov-api
    1. Economics and Statistics Administration (ESA) : https://www.commerce.gov/bureaus-and-offices/ousea
    2. International Trade Administration (ITA) : API here: https://developer.trade.gov/
    3. National Oceanic and Atmospheric Administration (NOAA) : https://www.ncdc.noaa.gov/cdo-web/webservices/v2 (R Package https://cran.r-project.org/web/packages/rnoaa/index.html)
    4. National Environmental Satellite, Data, and Information Service (NESDIS) : Cool maps and other viz https://www.nesdis.noaa.gov/content/imagery-and-data
    5. Marine Fisheries Service (NMFS) : https://www.fisheries.noaa.gov/resources/data-and-maps
    6. Patent and Trademark Office (PTO) : https://www.uspto.gov/learning-and-resources/open-data-and-mobility
  3. Department of Defense : https://data.defense.gov/
    1. Army Corps of Engineers (ACE) : https://www.iwr.usace.army.mil/about/technical-centers/wcsc-waterborne-commerce-statistics-center/
    2. Defense Health Agency (DHA) : https://health.mil/Military-Health-Topics/Health-Readiness/Armed-Forces-Health-Surveillance-Branch/Data-Management-and-Technical-Support
    3. Defense Manpower Data Center (DMDC) : https://www.dmdc.osd.mil/appj/dwp/data_reqs.jsp
  4. Department of Education: https://www2.ed.gov/rschstat/catalog/index.html, https://eddataexpress.ed.gov/?src=ft , https://usedgov.github.io/, https://usedgov.github.io/api/
    1. Institute of Education Sciences (IES) : https://ies.ed.gov/data.asp
    2. Office of Career, Technical, and Adult Education (OCTAE) : https://www2.ed.gov/about/offices/list/ovae/resource/index.html
    3. Office for Civil Rights (OCR) : https://ocrdata.ed.gov/
    4. Office of Elementary and Secondary Education (OESE) : https://www2.ed.gov/about/offices/list/oese/resources.html
    5. Office of Innovation and Improvement (OII) : https://innovation.ed.gov/resources/
    6. Office of Planning, Evaluation, and Policy Development (OPEPD) : https://www2.ed.gov/about/offices/list/opepd/reports.html
    7. Office of Postsecondary Education (OPE) : https://nces.ed.gov/ipeds/use-the-data
    8. Office of Special Education and Rehabilitative Services (OSERS) : https://rsa.ed.gov/
  5. Department of Energy : https://www.energy.gov/data/open-energy-data
    1. Office of Environment, Health, Safety, and Security (OEHSS) : https://www.energy.gov/ehss/policy-guidance-reports/databases
  6. Department of Health and Human Services : https://www.hhs.gov/developer.html, https://healthdata.gov/
    1. Administration for Children and Families (ACF) : https://www.acf.hhs.gov/reports#block-acf-theme-opre-reports
    2. Administration for Community Living (ACL) : https://agid.acl.gov/
    3. Agency for Healthcare Research and Quality (AHRQ) : https://www.ahrq.gov/data/resources/index.html
    4. Centers for Disease Control and Prevention (CDC) : https://data.cdc.gov/ ; https://github.com/CDCgov .
      1. Agency for Toxic Substance and Disease Registry (ATSDR) : https://www.atsdr.cdc.gov/publications_data_resources.html
      2. Center for Global Health : https://www.cdc.gov/globalhealth/index.html
      3. National Center on Birth Defects and Developmental Disabilities (NCBDDD) : https://www.cdc.gov/ncbddd/index.html
      4. National Center for Chronic Disease Prevention and Health Promotion (NCCDPHP) : https://www.cdc.gov/chronicdisease/data/indicators.htm
      5. National Center for Emerging Zoonotic and Infectious Diseases (NCEZID) : https://www.cdc.gov/ncezid/multimedia/infographics.html
      6. National Center for Environmental Health (NCEH) : https://www.cdc.gov/nceh/data.htm
      7. National Center for HIV/AIDS, Viral Hepatitis, Sexually Transmitted Disease, and Tuberculosis Prevention (NCHHSTP) : https://www.cdc.gov/nchhstp/data-tools-resources.htm
      8. National Center for Immunization and Respiratory Diseases (NCIRD) : https://www.cdc.gov/ncird/
      9. National Center for Injury Prevention and Control (NCIPC) : https://www.cdc.gov/injury/wisqars/index.html
      10. National Institute for Occupational Safety and Health (NIOSH) : https://www.cdc.gov/niosh/data/default.html
      11. Office of Public Health Scientific Services (PHSS) : https://www.cdc.gov/cpr/
    5. Centers for Medicare and Medicaid Services (CMS) : https://www.cms.gov/Research-Statistics-Data-and-Systems/Research-Statistics-Data-and-Systems.html
    6. Health Resources and Services Administration (HRSA) : https://data.hrsa.gov/
    7. Indian Health Service (IHS) : https://www.ihs.gov/dps/
    8. National Institutes of Health (NIH) : https://www.nimhd.nih.gov/resources/reports-data.html, https://ghr.nlm.nih.gov/about/data-files-api, https://report.nih.gov/databases.aspx
    9. National Cancer Institute (NCI) : https://api.seer.cancer.gov/, https://clinicaltrialsapi.cancer.gov/ , https://www.cancer.gov/search/results?swKeyword=api
    10. National Center for Complementary and Integrative Health (NCCIH) : https://nccih.nih.gov/research/statistics
    11. National Eye Institute (NEI) : https://nei.nih.gov/eyedata
    12. National Heart, Lung, and Blood Institute (NHLBI) : https://www.nhlbi.nih.gov/science
    13. National Human Genome Research Institute (NHGRI) : https://www.genome.gov/policyethics/legdatabase/pubsearch.cfm, https://www.genome.gov/10000018/computational-and-statistical-genomics-branch-csgb/
    14. National Institute on Aging (NIA) : https://www.nia.nih.gov/research/resources
    15. National Institute on Alcohol Abuse and Alcoholism (NIAAA) : https://www.niaaa.nih.gov/research/guidelines-and-resources/epidemiologic-data
    16. National Institute of Allergy and Infectious Diseases (NIAID) : https://www.niaid.nih.gov/research/resources
    17. National Institute of Biomedical Imaging and Bioengineering (NIBIB) : https://www.nibib.nih.gov/
    18. National Institute of Child Health and Human Development (NICHD) : https://www.nichd.nih.gov/research/resources/index
    19. National Institute on Deafness and Other Communication Disorders (NIDCD) : https://www.nidcd.nih.gov/
    20. National Institute of Dental and Craniofacial Research (NIDCR) : https://www.nidcr.nih.gov/research/data-statistics
    21. National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) : https://www.niddk.nih.gov/research-funding/research-resources
    22. National Institute on Drug Abuse (NIDA) : https://github.com/nih-nida
    23. National Institute of Environmental Health Sciences (NIEHS) : https://www.niehs.nih.gov/research/resources/databases/index.cfm
    24. National Institute of General Medical Sciences’ (NIGMS) : https://www.nigms.nih.gov/education/other-resources
    25. National Institute of Mental Health (NIMH) : https://www.nimh.nih.gov/health/statistics/index.shtml
    26. Office of the Director (NIH/OD) : https://acd.od.nih.gov/working-groups.html
    27. Office of the Assistant Secretary for Planning and Evaluation (ASPE) : https://aspe.hhs.gov/basic-search/data
    28. Office of Population Affairs (OPA) : https://www.hhs.gov/opa/
    29. Substance Abuse and Mental Health Services Administration (SAMHSA) : https://www.samhsa.gov/data/all-reports
  7. Department of Homeland Security : https://www.dhs.gov/dhs-developer-community
    1. Citizenship and Immigration Services (CIS) : https://www.uscis.gov/tools/reports-studies/immigration-forms-data
    2. Coast Guard : https://www.uscg.mil/
    3. Customs and Border Protection (CBP) : https://www.cbp.gov/trade/itrac-requests, https://www.cbp.gov/newsroom/stats
    4. Federal Emergency Management Agency’s (FEMA) : https://www.fema.gov/data-visualization, https://www.fema.gov/data-feeds
    5. Office of Immigration Statistics (OIS) : https://www.dhs.gov/immigration-statistics (Link was broken last I checked…)
  8. Department of Housing and Urban Development : https://data.hud.gov/
    1. Office of Housing (Housing) : https://www.hud.gov/program_offices/housing
    2. Office of Policy Development and Research (PD&R) : https://www.huduser.gov/portal/pdrdatas_landing.html
    3. Office of Public and Indian Housing (PIH) : https://www.hud.gov/program_offices/public_indian_housing/ih/pubs
  9. Department of the Interior : https://www.doi.gov/data, https://data.doi.gov/dataset
    1. Bureau of Land Management (BLM) : https://www.blm.gov/about/data
    2. Bureau of Ocean Energy Management (BOEM) : https://www.boem.gov/Maps-and-GIS-Data/, https://www.boem.gov/Renewable-Energy-Program-Mapping-and-Data/, https://www.data.boem.gov/ (woof)
    3. Bureau of Reclamation (BoR) : https://water.usbr.gov/ , https://www.usbr.gov/rsvrWater/HistoricalApp.html
    4. Fish and Wildlife Service (FWS) : https://www.fws.gov/southeast/data-management/, https://www.fws.gov/gis/data/national/
    5. Geological Survey (GS) : https://www.usgs.gov/products/data-and-tools/overview (NICE!)
    6. National Park Service (NPS) : https://www.nps.gov/gis/data_info/, https://irma.nps.gov/DataStore/, https://www.nps.gov/subjects/digital/nps-data-api.htm
    7. Office of Natural Resources Revenue (ONRR) : https://revenuedata.doi.gov/explore/#federal-disbursements, https://www.onrr.gov/About/production-data.htm
  10. Department of Justice : https://www.justice.gov/developer , https://data.ojp.gov/developer/index.html
    1. Bureau of Prisons’ (BOP) : http://prisondb.github.io/bopapidocs.html (not affiliated with bop.gov), https://www.bop.gov/about/statistics/
    2. Drug Enforcement Administration (DEA) : https://www.dea.gov/data-and-statistics, https://dea.ntis.gov/
    3. Federal Bureau of Investigation’s (FBI) : https://www.fbi.gov/wanted/api, https://crime-data-explorer.fr.cloud.gov/api
    4. National Institute of Justice (NIJ) : https://www.nij.gov/journals/258/Pages/forensic-databases.aspx, https://www.nij.gov/funding/data-resources-program/Pages/accessing.aspx
    5. Office of Juvenile Justice and Delinquency Prevention (OJJDP) : https://www.icpsr.umich.edu/icpsrweb/content/NACJD/archiving/deposit-ojjdp-data.html, https://www.ojjdp.gov/ojstatbb/dat.htmlm https://www.ojjdp.gov/ojstatbb/ezapop/
  11. Department of Labor: https://enforcedata.dol.gov/homePage.php, https://developer.dol.gov/
    1. Chief Evaluation Office (CEO) : https://www.dol.gov/asp/evaluation/PublicUseData.htm
    2. Employment and Training Administration (ETA) : https://www.doleta.gov/research/
    3. Mine Safety and Health Administration (MSHA) : https://www.msha.gov/data-reports/data-sources-calculators
    4. Occupational Safety and Health Administration (OSHA) : https://www.osha.gov/oshstats/index.html
    5. Office of Federal Contract Compliance Programs (OFCCP) : https://www.dol.gov/ofccp/BTN/index.html
    6. Wage and Hour Division (WHD) : https://catalog.data.gov/dataset/whd-compliance-full
  12. Department of State : https://www.state.gov/api/v1/docs/, https://www.state.gov/developer/, https://www.state.gov/data/
    1. Office of the U.S. Global AIDS Coordinator (OGAC) :https://www.pepfar.gov/priorities/data/index.htm, https://www.hiv.gov/locator#api
  13. Department of Transportation : https://www.transportation.gov/developer
    1. Federal Aviation Administration (FAA) : https://www.faa.gov/data_research/
    2. Federal Highway Administration (FHWA) : https://infopave.fhwa.dot.gov/Data/DataSelection
    3. Federal Motor Carrier Safety Administration (FMCSA) : https://dataqs.fmcsa.dot.gov
    4. Federal Railroad Administration (FRA) : https://safetydata.fra.dot.gov/officeofsafety/publicsite/downloaddbf.aspx
    5. Federal Transit Administration (FTA) : https://www.transit.dot.gov/ntd
    6. National Highway Traffic Safety Administration (NHTSA) : https://www.nhtsa.gov/research-data
    7. Pipeline and Hazardous Materials Safety Administration (PHMSA) : https://www.phmsa.dot.gov/data-and-statistics/pipeline/data-and-statistics-overview
  14. Department of Veterans Affairs : https://www.data.va.gov/, https://developer.va.gov/
    1. National Cemetery Administration (NCA) : https://gravelocator.cem.va.gov/
    2. VA Office of Inspector General (VAOIG) : https://www.va.gov/oig/publications/default.asp
    3. Office of Policy and Planning (OPP) : https://www.va.gov/HEALTHPOLICYPLANNING/OSPA_prod.asp
    4. Veterans Benefits Administration (VBA) : https://benefits.va.gov/benefits/
    5. Veterans Health Administration (VHA) : https://www.publichealth.va.gov/studies-data.asp?_ga=2.43552869.1979850101.1542340457-1023302291.1542340457, https://www.va.gov/health/
  15. Statistical Programs of Other Federal Agencies 57
    1. Broadcasting Board of Governors (BBG) United States Agency for Global Media (USAGM): https://www.usagm.gov/our-work/strategy-and-results/strategic-priorities/research-reports/
    2. Consumer Product Safety Commission (CPSC) : https://www.cpsc.gov/Recalls/CPSC-Recalls-Application-Program-Interface-API-Information, https://www.cpsc.gov/Newsroom/Downloadable-Data
    3. Corporation for National and Community Service’s (CNCS) : https://www.nationalservice.gov/impact-our-nation/evidence-exchange/basic-search
    4. Environmental Protection Agency (EPA) : https://www.epa.gov/enviro/web-services (loads of APIs)
    5. Equal Employment Opportunity Commission (EEOC) : https://www.eeoc.gov/eeoc/statistics/index.cfm,
    6. Institute of Museum and Library Services (IMLS) : https://www.imls.gov/research-tools/data-collection
    7. National Aeronautics and Space Administration (NASA) : https://api.nasa.gov/, https://www.nasa.gov/data/
    8. National Science Foundation (NSF) : https://www.nsf.gov/developer/
    9. Office of National Drug Control Policy (ONDCP) : https://www.whitehouse.gov/ondcp/ (I couldn’t find any data.)
    10. Small Business Administration (SBA) : https://www.sba.gov/about-sba/sba-performance/open-government/digital-sba/open-data/open-data-sources
    11. Social Security Administration (SSA) : https://www.ssa.gov/developer/, https://www.ssa.gov/data/
    12. U.S. Agency for International Development (USAID) : https://www.usaid.gov/results-and-data/data-resources, https://www.usaid.gov/developer

What did I learn?

Sam Tyner
Sam Tyner
Data Scientist

I am an applied statistician and data scientist, with a wide range of skills and experiences. I’m passionate about using data to make a difference.