There are now over 5.8 million smart meters operating across homes and businesses in the UK, and this number is expected to rise to over 50 million by 2020, driven by the Smart Meter Implementation Programme. This programme will provide high-resolution electricity and gas consumption data across millions of household across the UK, offering enormous potential for energy research and data-driven policymaking. However, the sheer volume of the data (e.g. > 20GB) means that even basic exploratory analysis is difficult using traditional desktop-driven methods of analysis. The aim of this page is to provide a curated list of useful software libraries and resources for analyzing smart meter data at scale. This list may also be useful for analyzing any regularly-spaced time series data, e.g. half-hourly weather data, monthly unemployment data, etc.
Inspired by awesome-python and awesome-R.
The table below lists some existing UK smart meter data sets available for download on the web.
| Name | Period | No. of Households | No. of Obs. | Sample Freq. | Data Recorded | Obs. Type | Data Format | Data License | Meter Type |
|---|---|---|---|---|---|---|---|---|---|
| Energy Demand Research Project | 2007-2010 | 14,000 | 400 million | 30 mins | Electricity, Gas | Aggregate | csv | EULA | SMETS 1 |
| Low Carbon London Project | 2013-2014 | 5,536 | 100 million | 30 mins | Electricity | Aggregate | csv | Open | SMETS 1 |
| REFIT Electrical Load Measurements | 2015-2017 | 20 | 1 billion | 8 seconds | Electricity | Aggregate, Appliance | csv | Open | Own device |
| UK Domestic Appliance-Level Electricity | 2015-2017 | 5 | 1 billion | 6-8 seconds | Electricity | Aggregate, Appliance | csv, FLAC | Open | Own device |
- EULA (End User License Agreement): Sign up and register with the UK Data Service to download.
- SMETS 1 (Smart Meter Equipment Technical Standards): UK government standard for smart metering equpiment.
Some other good resources include:
- Household Power/Appliance Open Access Datasets
- Smart Meter Data and Public Interest Issues - Annex A: Existing Data
A number of small samples of smart meter datasets can be found in the data directory.
These have been converted into feather format for speed and seamless interoperability
between R and other programming languages (e.g. Python).
A list of popular R libraries for analyzing smart meter data is provided below. A hands-on tutorial can be found for each library in the examples directory as an R notebook. This is to ensure that the code provided is both readable and reproducible. We follow Google's R Style Guide for R code for consistency and maintainability, with the exception of using = instead of <- for variable assignment. Whilst this is largely a matter of preference, = does allow for greater compatibility with other mainstream programming languages such as C, C++, Java, etc, and generally results in less obscure bugs in code as it disallows assignment within control structures (e.g. if()).