+
+
+
+
+
+
+
+
+
+
+
+
+{% block results %} {% endblock %}
+
+
+
+
+
+Skip to main content
+
+
+
+
+
+
+
+
+
+/search?q=
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# `StatsChat`
+[](https://github.com/mkenney/software-guides/blob/master/STABILITY-BADGES.md#experimental)
+[](https://codecov.io/gh/datasciencecampus/Statschat)
+[](https://twitter.com/DataSciCampus)
+[](https://github.com/datasciencecampus/Statschat/blob/main/LICENSE)
+[]()
+
+## Code state
+
+**Please be aware that for development purposes, these experiments use experimental Large Language Models (LLM's) not intended for production. They can present inaccurate information, hallucinated statements and offensive text by random chance or through malevolent prompts.**
+
+**Tested on OSX only**
+
+**Peer-reviewed**
+
+**Depends on external API's**
+
+**Under development**
+
+**Experimental**
+
+## Introduction
+
+This is an experimental application for semantic search of ONS statistical publications.
+It uses LangChain to implement a fairly simple embedding search and QA information retrieval
+process. Upon receiving a query, documents are returned as search results
+using embedding similarity to score relevance. Additionally, the relevant text is
+passed to a locally-hosted Large language Model (LLM), which is prompted to write an
+answer to the original question, if it can, using only the information contained within
+the documents.
+
+For this prototype, the program is run entirely locally; relevant web pages are scraped and the data
+stored in `data/bulletins`, the docstore / embedding store that is created is likewise
+in local folders and files, and the LLM and all other code is run in memory on your
+desktop or laptop.
+
+The search program should be able to run on a system with 16GB of ram. The LLM is
+set up to run on CPU at this research stage. Different models from the Hugging Face
+repository can be specified for the search and QA functions.
+
+## Installation
+
+The project requires specific versions of some packages so it is recommended to
+set up a virtual environment. Using venv and pip:
+
+```shell
+python3.10 -m venv env
+source env/bin/activate
+
+python -m pip install --upgrade pip
+python -m pip install -r requirements.txt
+```
+
+### Pre-commit actions
+This repository contains a configuration of pre-commit hooks. These are language agnostic and focussed on repository security (such as detection of passwords and API keys). If approaching this project as a developer, you are encouraged to install and enable `pre-commits` by running the following in your shell:
+ 1. Install `pre-commit`:
+
+ ```shell
+ pip install pre-commit
+ ```
+ 2. Enable `pre-commit`:
+
+ ```shell
+ pre-commit install
+ ```
+
+Once pre-commits are activated, whenever you commit to this repository a series of checks will be executed. The pre-commits include checking for security keys, large files and unresolved merge conflict headers. The use of active pre-commits are highly encouraged and the given hooks can be expanded with Python or R specific hooks that can automate the code style and linting. For example, the `flake8` and `black` hooks are useful for maintaining consistent Python code formatting.
+
+**NOTE:** Pre-commit hooks execute Python, so it expects a working Python build.
+
+## Usage
+
+By default, flask will look for a file called `app.py`, you can also name a specific python program to run.
+With `--debug` in play, flask will restart every time it detects a saved change in the underlying
+python files.
+The first time you run the app, any ML models specified in the code will be downloaded
+to your machine. This will use a few GB of data and take a few minutes.
+App and search pipeline parameter are stored and can be updated by editing `app_config.toml`.
+
+We have included three EXAMPLE scraped data files in `data/bulletins` so that
+the preprocessing and app can be run as a small example system without waiting
+on webscraping.
+
+### To webscrape the source documents from ONS
+#### By default we have limited the script to retrieving 10 actual articles (`statschat/webscraping/main.py`, line 61), this limit is easily edited out to allow the program to run to completion.
+```shell
+python statschat/webscraping/main.py
+```
+
+### To create a local document store
+```shell
+python statschat/preprocess.py
+```
+
+### To run the interactive app
+
+
+
+```shell
+flask --debug run
+```
+or
+```shell
+python app.py
+```
+
+The flask app is set respond to https requests on port 5000. To use the user UI navigate in your browser to http://localhost:5000.
+
+The API default url would be http://localhost:5000/api. See [API endpoint documentation](docs/api/README.md) for more details (note, this is a work in progress).
+
+
+### Search engine parameters
+
+There are some key parameters in `app_config.toml` that we're experimenting with to improve the search results,
+and the generated text answer. The current values are initial guesses:
+
+| Parameter | Current Value | Function |
+| --- | --- | --- |
+| k_docs | 10 | Maximum number of search results to return |
+| similarity_threshold | 1.0 | Cosine distance, a searched document is only returned if it is at least this similar (EQUAL or LOWER) |
+| k_contexts | 3 | Number of top documents to pass to generative QA LLM |
+
+### Alternatively, to run the search evaluation pipeline
+
+The StatsChat pipeline is currently evaluated based on small number of test question. The main 'app_config.toml' determines pipeline setting used in evaluation and results are written to `data/model_evaluation` folder.
+
+```shell
+python statschat/model_evaluation/evaluation.py
+```
+
+
+## Testing
+
+Preferred unittesting framework is PyTest:
+
+```shell
+pytest
+```
+
+# Data Science Campus
+At the [Data Science Campus](https://datasciencecampus.ons.gov.uk/about-us/) we apply data science, and build skills, for public good across the UK and internationally. Get in touch with the Campus at [datasciencecampus@ons.gov.uk](datasciencecampus@ons.gov.uk).
+
+# License
+
+
+
+The code, unless otherwise stated, is released under [the MIT License][mit].
+
+The documentation for this work is subject to [© Crown copyright][copyright] and is available under the terms of the [Open Government 3.0][ogl] licence.
+
+[mit]: LICENSE
+[copyright]: http://www.nationalarchives.gov.uk/information-management/re-using-public-sector-information/uk-government-licensing-framework/crown-copyright/
+[ogl]: http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/
diff --git a/app.py b/app.py
new file mode 100644
index 0000000..4062e7b
--- /dev/null
+++ b/app.py
@@ -0,0 +1,160 @@
+import toml
+import logging
+
+import pandas as pd
+
+from datetime import datetime
+from flask import Flask, render_template, request, jsonify
+from flask.logging import default_handler
+from markupsafe import escape
+from werkzeug.datastructures import MultiDict
+
+from statschat.llm import Inquirer
+from statschat.utils import deduplicator
+from statschat.latest_flag_helpers import get_latest_flag, time_decay
+
+
+# Config file to load
+CONFIG = toml.load("app_config.toml")
+
+# define session_id that will be used for log file and feedback
+SESSION_NAME = f"statschat_app_{format(datetime.now(), '%Y_%m_%d_%H:%M')}"
+
+logger = logging.getLogger(__name__)
+log_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+logging.basicConfig(
+ level=logging.INFO,
+ format=log_fmt,
+ filename=f"log/{SESSION_NAME}.log",
+ filemode="a",
+)
+logger.addHandler(default_handler)
+
+
+# define global variable to link last answer to ratings for feedback capture
+last_answer = {}
+feedback_file = f"data/feedback/{SESSION_NAME}.csv"
+pd.DataFrame(
+ {"question": [], "answer": [], "confidence": [], "timing": [], "feedback": []}
+).to_csv(feedback_file, index=False)
+
+# initiate Statschat AI and start the app
+searcher = Inquirer(**CONFIG["db"], **CONFIG["search"], logger=logger)
+
+
+def make_query(question: str, latest_max: bool = True) -> dict:
+ """
+ Utility, wraps code for querying the search engine, and then the summarizer.
+ Also handles storing the last answer made for feedback purposes.
+
+ Args:
+ question (str): The user query.
+ latest (bool, optional): Whether to weight in favour of recent releases.
+ Defaults to True.
+
+ Returns:
+ dict: answer and supporting documents returned.
+ """
+ now = datetime.now()
+ # TODO: pass the advanced filters to the searcher
+ # TODO: move deduplication keys to config['app']
+ docs = deduplicator(
+ searcher.similarity_search(question),
+ keys=["section", "title", "date"],
+ )
+ if len(docs) > 0:
+ if latest_max:
+ for doc in docs:
+ # Divided by decay term because similarity scores are inverted
+ # Original score is L2 distance; lower is better
+ # https://python.langchain.com/docs/integrations/vectorstores/faiss
+ doc["weighted_score"] = doc["score"] / time_decay(
+ doc["date"], latest=latest_max
+ )
+ docs.sort(key=lambda doc: doc["weighted_score"])
+ logger.info(
+ f"Weighted and reordered docs to latest with decay = {latest_max}"
+ )
+
+ answer = searcher.query_texts(question, docs)
+ else:
+ answer = "NA"
+
+ results = {
+ "answer": answer,
+ "question": question,
+ "references": docs,
+ "timing": (datetime.now() - now).total_seconds(),
+ }
+ logger.info(f"Received answer: {results['answer']}")
+
+ # Handles storing last answer for feedback purposes
+ global last_answer
+ last_answer = results.copy()
+
+ return results
+
+
+app = Flask(__name__)
+
+
+@app.route("/")
+def home():
+ advanced = MultiDict()
+ return render_template("statschat.html", advanced=advanced, question="?")
+
+
+@app.route("/advanced")
+def advanced():
+ advanced = MultiDict(
+ [("latest-publication", "Off"), ("bulletins", "on"), ("articles", "on")]
+ )
+ return render_template("statschat.html", advanced=advanced, question="?")
+
+
+@app.route("/search", methods=["GET", "POST"])
+def search():
+ question = escape(request.args.get("q"))
+ advanced, latest_max = get_latest_flag(request.args, CONFIG["app"]["latest_max"])
+ logger.info(f"Search query: {question}")
+ if question:
+ results = make_query(question, latest_max)
+ return render_template(
+ "statschat.html", advanced=advanced, question=question, results=results
+ )
+ else:
+ return render_template("statschat.html", advanced=advanced, question="?")
+
+
+@app.route("/record_rating", methods=["POST"])
+def record_rating():
+ rating = request.form["rating"]
+ logger.info(f"Recorded answer rating: {rating}")
+ last_answer["rating"] = rating
+ pd.DataFrame([last_answer]).to_csv(
+ feedback_file, mode="a", index=False, header=False
+ )
+ return "", 204 # Return empty response with status code 204
+
+
+@app.route("/api/search", methods=["GET", "POST"])
+def api_search():
+ question = escape(request.args.get("q"))
+ _, latest_max = get_latest_flag(request.args, CONFIG["app"]["latest_max"])
+ logger.info(f"Search query: {question}")
+ if question:
+ results = make_query(question, latest_max)
+ logger.info(f"Received {len(results['references'])} documents.")
+ return jsonify(results), 200
+ else:
+ return jsonify({"error": "Empty question"}), 400
+
+
+@app.route("/api/about", methods=["GET", "POST"])
+def about():
+ info = {"version": "ONS StatsChat API v0.1", "contact": "dsc.projects@ons.gov.uk"}
+ return jsonify(info)
+
+
+if __name__ == "__main__":
+ app.run(debug=False, host="0.0.0.0")
diff --git a/app_config.toml b/app_config.toml
new file mode 100644
index 0000000..86e0a35
--- /dev/null
+++ b/app_config.toml
@@ -0,0 +1,30 @@
+[db]
+faiss_db_root = "db_lc"
+embedding_model = "sentence-transformers/all-mpnet-base-v2" # "sentence-transformers/paraphrase-MiniLM-L3-v2"
+
+[setup]
+directory = "data/bulletins"
+split_directory = "data/full_bulletins_split"
+split_length = 1000
+split_overlap = 50
+
+[search]
+model_name_or_path = "google/flan-t5-large" # "lmsys/fastchat-t5-3b-v1.0" "google/flan-t5-large" "google/flan-ul2"
+k_docs = 10
+k_contexts = 3
+similarity_threshold = 1.0 # Threshold score below which a document is returned in a search
+return_source_documents = false
+llm_summarize_temperature = 0.0
+llm_generate_temperature = 0.0
+
+[app]
+latest_max = 2 # Takes value int >= 0, commonly 0, 1 or 2
+
+[NYI]
+prompt_text = """Synthesize a comprehensive answer from the following text
+ for the given question. Provide a clear and concise response, that summarizes
+ the key points and information presented in the text. Your answer should be
+ in your own words and be no longer than 50 words. If the question cannot be
+ confidently answered from the information in the text, or if the question is
+ not related to the text, reply 'NA'. \n\n
+ Related text: {summaries} \n\n Question: {question} \n\n Answer:"""
diff --git a/codecov.yml b/codecov.yml
new file mode 100644
index 0000000..1af4451
--- /dev/null
+++ b/codecov.yml
@@ -0,0 +1,16 @@
+comment: true
+
+coverage:
+ status:
+ project:
+ default:
+ target: auto
+ threshold: 80%
+ informational: true
+ patch:
+ default:
+ target: auto
+ threshold: 20%
+ informational: true
+
+ignore: ["tests"]
diff --git a/data/.gitkeep b/data/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/data/bulletins/.gitkeep b/data/bulletins/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/data/bulletins/EXAMPLE-2015-03-10_child-and-infant-mortality-in-england-and-wales-2013.json b/data/bulletins/EXAMPLE-2015-03-10_child-and-infant-mortality-in-england-and-wales-2013.json
new file mode 100644
index 0000000..f7177bc
--- /dev/null
+++ b/data/bulletins/EXAMPLE-2015-03-10_child-and-infant-mortality-in-england-and-wales-2013.json
@@ -0,0 +1,152 @@
+{
+ "id": "2015-03-10_child-and-infant-mortality-in-england-and-wales-2013",
+ "title": "Child and infant mortality in England and Wales: 2013",
+ "url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10",
+ "release_date": "2015-03-10",
+ "release_type": "bulletins",
+ "latest": false,
+ "url_keywords": [
+ "peoplepopulationandcommunity",
+ "birthsdeathsandmarriages",
+ "deaths"
+ ],
+ "contact_name": "Elizabeth McLaren",
+ "contact_link": "mailto:vsob@ons.gov.uk",
+ "content": [
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#main-points",
+ "section_header": "1. Main points",
+ "section_text": "There were 2,686 infant deaths (deaths under 1 year) in England and Wales in 2013, compared with 2,912 infant deaths in 2012 and 6,381 in 1983 In 2013, the infant mortality rate was 3.8 deaths per 1,000 live births, the lowest ever recorded in England and Wales. This compares with an infant mortality rate of 4.0 deaths per 1,000 live births in 2012 and 10.1 deaths per 1,000 live births in 1983 Infant mortality rates were lowest for babies of mothers aged 25 to 29 years (3.4 deaths per 1,000 live births) and highest for babies of mothers aged under 20 years (6.1 deaths per 1,000 live births) In 2013, the infant mortality rates for very low birthweight babies (under 1,500 grams) and low birthweight babies (under 2,500 grams) were 164.0 and 32.4 deaths per 1,000 live births respectively",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#summary",
+ "section_header": "2. Summary",
+ "section_text": "This bulletin presents final statistics on infant deaths and childhood deaths that occurred in England and Wales in 2013. It also contains additional analyses by some of the key risk factors affecting infant deaths, including age of mother and birthweight. These characteristics are derived from linking the death registration to the corresponding birth registration record. Data are also available for babies born in 2012 who died before their first birthday: the 2012 birth cohort for infant deaths.This is the first time that we have published 2013 figures on infant and childhood mortality, based on occurrences. It is also the first time that the 2012 birth cohort data for infant deaths have been published.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#background",
+ "section_header": "3. Background",
+ "section_text": "Although infant mortality rates have continued to fall in England and Wales over the past 30 years, the rates of change varied over the period. The change in the first half of the period was more than twice that in the second half. General improvements in healthcare and more specific improvements in midwifery and neonatal intensive care can partly explain the overall fall in the rate of change (National Children\u2019s Bureau, 2014).Despite the downward trend, evidence in the Marmot Review: Fair Society, Healthy Lives noted that factors, including births outside marriage, maternal age under the age of 20 and deprivation, were independently associated with an increased risk of infant mortality. The review went on to say that \u2018low birthweight in particular is associated with poorer long-term health outcomes and the evidence also suggests that maternal health is related to socio-economic status\u2019.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#infant-and-perinatal-mortality-rates",
+ "section_header": "4. Infant and perinatal mortality rates",
+ "section_text": "There were 2,686 infant deaths in England and Wales in 2013, resulting in an infant mortality rate of 3.8 deaths per 1,000 live births (the lowest rate ever recorded in England and Wales). Since 1983, when the rate was 10.1 deaths per 1,000 live births, there has been a 62% fall in infant mortality rates in England and Wales. This continues the overall decline in infant mortality rates in England and Wales over the past 30 years (Figure 1). The infant mortality rate in 2012 was 4.0 deaths per 1,000 live births. Over the same period, there has been a similar fall in neonatal mortality rates (deaths under 28 days) and postneonatal mortality rates (deaths between 28 days and 1 year). The neonatal mortality rate fell by 54%, from 5.9 deaths per 1,000 live births in 1983, to 2.7 deaths per 1,000 live births in 2013. The postneonatal mortality rate fell by 72% over the same period, from 4.3 deaths per 1,000 live births in 1983, to 1.2 deaths per 1,000 live births in 2013.In 2013, there were 3,284 stillbirths and 1,423 deaths at age under 7 days, resulting in a perinatal mortality rate of 6.7 deaths per 1,000 total births. Since 1983, when the perinatal mortality rate was 10.4 deaths per 1,000 total births, the rate has fallen by more than a third.",
+ "figures": [
+ {
+ "figure_title": "Figure 1: Infant, neonatal and postneonatal mortality rates, 1983 to 2013",
+ "figure_subtitle": "England and Wales",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10/86444f44&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ }
+ ]
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#linking-birth-and-death-records",
+ "section_header": "5. Linking birth and death records",
+ "section_text": "Linking birth and infant death records improves our understanding of the key characteristics of the baby\u2019s parents that were registered on the birth registration record (background note 4). In 2013, 98% of infant deaths in England and Wales were successfully linked to their corresponding birth registration record. The linkage rate for infant deaths has remained consistent since the linking exercise began.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#cause-of-infant-deaths",
+ "section_header": "6. Cause of infant deaths",
+ "section_text": "The broad ONS cause groups showed that immaturity-related conditions, for example, respiratory and cardiovascular disorders, were the most common cause of infant deaths in 2013, with 44% due to these causes. Immaturity-related conditions accounted for 14% of all postneonatal deaths and 57% of all neonatal deaths. Congenital anomalies were another major cause group, accounting for 28% of all infant deaths. Congenital anomalies accounted for 32% of all postneonatal deaths and 27% of all neonatal deaths.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#age-of-mother-at-birth",
+ "section_header": "7. Age of mother at birth",
+ "section_text": "The infant mortality rate for all infant deaths linked to their corresponding birth registration record was 3.8 deaths per 1,000 live births in 2013. For these linked deaths, infant mortality rates were lowest for babies of mothers aged 25 to 29 years (3.4 deaths per 1,000 live births) and highest for mothers aged under 20 years (6.1 deaths per 1,000 live births).",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#birthweight",
+ "section_header": "8. Birthweight",
+ "section_text": "Low birthweight, one of the known risk factors for infant deaths, can be caused by a number of factors. For example, smoking has been identified as a major risk factor contributing to low birthweight. Babies born to women who smoke weigh, on average, 200g less than babies born to non-smokers (NHS, Why should I stop smoking if I\u2019m pregnant?). In 2013, the infant mortality rates for very low birthweight babies (under 1,500 grams) and low birthweight babies (under 2,500 grams) were 164.0 and 32.4 deaths per 1,000 live births respectively. This is much higher than the rate of 1.3 deaths per 1,000 live births among babies of normal birthweight (over 2,500 grams).For babies of low birthweight, the infant mortality rate was highest among mothers aged under 20 years (44.8 deaths per 1,000 live births) and lowest among mothers aged 40 and over (28.9 deaths per 1,000 live births) (Figure 2).",
+ "figures": [
+ {
+ "figure_title": "Figure 2: Infant and neonatal mortality rates for low birthweight babies: by age of mother, 2013",
+ "figure_subtitle": "England and Wales",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10/864f732a&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ }
+ ]
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#socio-economic-status",
+ "section_header": "9. Socio-economic status",
+ "section_text": "Significant differences in infant mortality rates by socio-economic group persist in England and Wales (Oakley et. al. 2009 (720.5 Kb Pdf)). One measure of social circumstances is that of occupational status. This information is collected at birth. Infant mortality rates were highest for the National Statistics Socio-economic Classification (NS-SEC) groups describing routine and manual occupations (Groups 5 to 7) with 4.5 deaths per 1,000 live births (the three-class version of NS-SEC has been used, there is more information in background note 9). In contrast there were 2.8 deaths per 1,000 live births for higher managerial, administrative and professional occupations (Groups 1.1, 1.2 and 2) and 3.6 deaths per 1,000 live births for intermediate occupations (Groups 3 and 4). For the 2012 data year onwards, the way that socio-economic status is reported has changed; details can be found in background note 8.Similar patterns in perinatal mortality by socio-economic group were recorded, with mortality rates highest for the NS-SEC groups describing routine and manual occupations (7.8 deaths per 1,000 total births). Higher managerial, administrative and professional occupations had a perinatal mortality rate of 5.5 deaths per 1,000 total births while for intermediate occupations the rate was 6.6. These variations may be the result of the link between lower socio-economic status and poorer maternal health, which can ultimately affect infant mortality rates (Oakley et. al. 2009 (720.5 Kb Pdf)).",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#1-mothers-country-of-birth",
+ "section_header": "10. Mother\u2019s country of birth",
+ "section_text": "The infant mortality rate for babies of mothers born outside the UK was 4.2 deaths per 1,000 live births compared with 3.6 deaths per 1,000 live births for mothers born inside the UK. The highest infant mortality rates were for babies of mothers born in the Caribbean (9.0 deaths per 1,000 live births) and mothers born in Central Africa (8.3 deaths per 1,000 live births) (please note this rate has been calculated from fewer than 20 deaths and its reliability as a measure maybe affected by the small number of events). Babies of mothers born in the Caribbean also had the highest stillbirth rate (8.3 deaths per 1,000 total births). Differences in infant mortality rates by mother\u2019s country of birth are likely to reflect underlying factors including mother\u2019s age, together with a range of other socio-demographic characteristics mentioned earlier in this bulletin.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#1-child-mortality-rates",
+ "section_header": "11. Child mortality rates",
+ "section_text": "Between 1983 and 2013, the age-specific mortality rate for children aged 1 to 14 years fell by 66%, from 29 deaths per 100,000 population in 1983 to 10 deaths per 100,000 in 2013. The age-specific mortality rate for children aged 1 to 4 years also fell by 66% over the same period, from 44 deaths per 100,000 population in 1983 to 15 deaths per 100,000 in 2013. Over the past 30 years child death rates from respiratory and circulatory diseases in England and Wales have been falling, as they have for the whole population, reflecting advances in medical care and preventative measures generally. In 2013, congenital related conditions and cancers were the most common form of death for children aged under 16 years.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#1-birth-cohort-for-infant-deaths-2012",
+ "section_header": "12. Birth cohort for infant deaths 2012",
+ "section_text": "Singleton and multiple births using the 2012 birth cohort tablesOf the 729,674 live births in 2012, a total of 2,939 infants had died before their first birthday, resulting in an infant mortality rate of 4.0 deaths per 1,000 live births. There were 3,558 stillbirths and 1,577 deaths at age under seven days over the same time period, resulting in a perinatal mortality rate of 7.0 deaths per 1,000 total births (live births and stillbirths).The infant mortality rate for multiple births in the 2012 birth cohort was nearly five times higher than for singletons (17.7 deaths per 1,000 live births compared with 3.6 deaths per 1,000 live births). This was most marked in the first 28 days of life (neonatal deaths) when the mortality rate for multiple births was nearly six times higher than for singletons (13.8 deaths per 1,000 live births compared with 2.4 deaths per 1,000 live births). For those who survived beyond their first month but died before their first birthday (postneonatal deaths), the mortality rate was over three times higher for multiple births (3.9 per 1,000 live births) compared with singleton births (1.1 deaths per 1,000 live births) (Figure 3). On average multiple births tend to have lower birthweights than singletons, which is one reason why the infant mortality rate is around five times higher for multiple births than for singleton births. Just over half of multiple birth babies (56% of those with a known birthweight) were low birthweight (less than 2,500 grams) and 8.9% of those with a known birthweight were very low birthweight (less than 1,500 grams) (Figure 4). Multiple pregnancies are also associated with a higher risk of stillbirth, death under 28 days and child disability (Human Fertilisation and Embryology Authority (HFEA)). In contrast, 5.4% of singletons were born with a low birthweight and 0.9% were very low birthweight. The main reason why multiple birth babies tend to have lower birthweight than singleton babies is because multiple births rarely go to term.The highest infant mortality rates were for the extremely low birthweight babies (less than 1,000 grams). For singletons the rate was 316.3 deaths per 1,000 live births and for multiples the rate was 331.5 deaths per 1,000 live births.Although most multiple births occur naturally, many occur as a result of fertility treatment. On average, 1 in 5 of In Vitro Fertilisation (IVF) pregnancies result in multiple births compared with 1 in 80 for women who conceive naturally (HFEA). With approximately 17,800 IVF babies born in the UK in 2012, this contributes considerably to the multiple birth rate (HFEA, 2014). In 2009, the Human Fertilisation and Embryology Authority launched the elective single embryo transfer (eSET) policy, which allowed centres to develop their own eSET strategy, with the aim to reduce the UK IVF multiple pregnancy rate to 10% over a period of years (HFEA).Age of mother at birth using the 2012 birth cohort tablesNearly two-thirds (64%) of all multiple births in 2012 were to women aged 30 years or over compared with 49% of all singleton births. The 2012 birth cohort tables for infant deaths show that mothers aged under 20 years have the highest infant mortality rate for singletons (5.9 deaths per 1,000 live births). Younger mothers (under 20 years) also have the highest infant mortality rate for multiple births (36.1 deaths per 1,000 live births).Marital status and registration type using the 2012 birth cohort tablesThe infant mortality rate for babies born inside marriage was lower than for those born outside marriage. The 2012 birth cohort for infant deaths shows that for singletons, there were 3.1 deaths per 1,000 live births inside marriage and 4.1 deaths per 1,000 live births outside marriage. For multiple births there were 15.2 deaths per 1,000 live births inside marriage and 21.2 deaths per 1,000 live births outside marriage.The infant mortality rate for singletons was highest for those registered solely by their mother, or registered jointly by parents living at different addresses (5.9 and 4.9 deaths per 1,000 live births respectively). For multiple births the infant mortality rate was highest for those jointly registered by both parents living at different addresses (25.3 deaths per 1,000 live births) compared with 21.7 deaths per 1,000 live births for those jointly registered by parents living at the same address. The very small numbers in these groups affects the robustness of estimated mortality rates. Differences in mortality rates by marital status and birth registration type will also reflect complex underlying factors including mother\u2019s age and social circumstances (Messer, 2011). For married women the infant mortality rate for singleton births was higher for women who have previously had three or more children (4.7 deaths per 1,000 live births) compared with women who have had no previous children (3.5 deaths per 1,000 live births). Other factors may be relevant here, especially the mother\u2019s age. For multiple births, the infant mortality rate was higher for women who have previously had two children (16.4 deaths per 1,000 live births).Socio-economic status using the 2012 birth cohort tablesThe 2012 birth cohort tables for infant deaths show that for singleton births, the highest infant mortality rate was for the NS-SEC groups describing routine and manual occupations (Groups 5 to 7) with 4.2 deaths per 1,000 live births (the three-class version of NS-SEC has been used, background note 9 has more detail). In contrast there were 2.6 deaths per 1,000 live births for higher managerial, administrative and professional occupations (Groups 1.1, 1.2 and 2) and 3.5 deaths per 1,000 live births for intermediate occupations (Groups 3 and 4). The same pattern was found for multiple births where there were smaller numbers, although the infant mortality rate for each group was between five and six times higher than for singletons. For the 2011 data year onwards, the way in which socio-economic status is reported has changed; details can be found in background note 8.Studies have shown that infant mortality rates are comparatively higher for low income families (Duncan and Brooks-Gunn, 2000). Mothers from routine and manual occupation are less likely to breastfeed (NICE, 2008) which can result in poorer immunity and poorer digestive health for the baby.",
+ "figures": [
+ {
+ "figure_title": "Figure 3: Infant, neonatal and postneonatal mortality rates for singleton and multiple births, babies born in 2012",
+ "figure_subtitle": "England and Wales",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10/39550c02&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ },
+ {
+ "figure_title": "Figure 4: Live births: birthweight (grams) for singleton and multiple births, babies born in 2012",
+ "figure_subtitle": "England and Wales",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10/046cb1c7&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ }
+ ]
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#1-users-and-uses-of-infant-mortality-statistics",
+ "section_header": "13. Users and uses of infant mortality statistics",
+ "section_text": "Infant mortality is seen as a key measure among health outcomes and there is a long established link between social and health inequalities, and infant mortality. The Department of Health (DH) is a key user of child mortality statistics. Infant mortality continues to take a central role in DH\u2019s work on tackling health inequalities within the NHS Outcomes Framework 2014/15 and the Public Health Outcomes Framework.There are also two specific users of the birth cohort data; the Department of Midwifery and Child Health, City University London; and Public Health England, who provide information and intelligence to improve decision making for policy makers, commissioners, managers, regulators, and other health stakeholders working on children's, and maternal health, for example by providing service snapshots for infant mortality and stillbirths for local areas. Other users of infant mortality data include academics, independent researchers, charities and media.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#1-changes-to-child-mortality-outputs",
+ "section_header": "14. Changes to child mortality outputs",
+ "section_text": "Changes to child mortality publications were outlined in an information note, available on our website. These changes are being implemented.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#15further-information",
+ "section_header": "\n15\n.Further information\n",
+ "section_text": "",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#16references",
+ "section_header": "\n16\n.References\n",
+ "section_text": "",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#17background-notes",
+ "section_header": "\n17\n.Background notes\n",
+ "section_text": "",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/childhoodinfantandperinatalmortalityinenglandandwales/2015-03-10#18methodology",
+ "section_header": "\n18\n.\nMethodology\n",
+ "section_text": "",
+ "figures": []
+ }
+ ]
+}
\ No newline at end of file
diff --git a/data/bulletins/EXAMPLE-2016-02-16_consumer-price-inflation-uk-january-2016.json b/data/bulletins/EXAMPLE-2016-02-16_consumer-price-inflation-uk-january-2016.json
new file mode 100644
index 0000000..34d9def
--- /dev/null
+++ b/data/bulletins/EXAMPLE-2016-02-16_consumer-price-inflation-uk-january-2016.json
@@ -0,0 +1,97 @@
+{
+ "id": "2016-02-16_consumer-price-inflation-uk-january-2016",
+ "title": "Consumer price inflation, UK: January 2016",
+ "url": "https://www.ons.gov.uk/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016",
+ "release_date": "2016-02-16",
+ "release_type": "bulletins",
+ "latest": false,
+ "url_keywords": [
+ "economy",
+ "inflationandpriceindices"
+ ],
+ "contact_name": "James Tucker",
+ "contact_link": "mailto:cpi@ons.gov.uk",
+ "content": [
+ {
+ "section_url": "https://www.ons.gov.uk/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016#main-points",
+ "section_header": "1. Main points",
+ "section_text": "The Consumer Prices Index (CPI) rose by 0.3% in the year to January 2016, compared with a 0.2% rise in the year to December 2015.This is the third consecutive month of small increases, with the rate in January 2016 being the same as it was in January 2015.The main contributors to the rise in the rate were motor fuels, and to a lesser extent food, alcoholic beverages and clothing.Air fare prices partially offset the rise in the rate, falling by more than they did a year ago. This followed a large increase in prices in December 2015.CPIH (not a National Statistic) grew by 0.6% in the year to January 2016, up from 0.5% in December 2015.From this month, CPI and CPIH indices have been re-referenced and published with 2015=100. This does not impact on published inflation rates. Further information is available in the background notes.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016#a-brief-description-of-consumer-price-inflation",
+ "section_header": "2. A brief description of Consumer Price Inflation",
+ "section_text": "Consumer price inflation is the speed at which the prices of goods and services bought by households rise or fall. Consumer price inflation is estimated by using price indices. A way to understand a price index is to think of a very large shopping basket containing all the goods and services bought by households. The price index estimates changes to the total cost of this basket. An infographic explains how consumer price inflation is calculated. Consumer price indices are published monthly.A price index can be used to measure inflation in a number of ways. The most common is to look at how the index has changed over a year. This is calculated by comparing the price index for the latest month with the same month a year ago. This is known as the 12-month inflation rate. This bulletin measures inflation to January 2016, so the 12-month rate measures changes in prices between January 2015 and January 2016.A range of measures of consumer price and other price inflation are published. A tale of many price indices summarises information on the different measures.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016#consumer-prices-index-cpi",
+ "section_header": "3. Consumer Prices Index (CPI)",
+ "section_text": "What is the CPI?The CPI is a measure of consumer price inflation produced to international standards and in line with European regulations. First published in 1997 as the Harmonised Index of Consumer Prices (HICP), the CPI is the inflation measure used in the government\u2019s target for inflation.The CPI is also used for purposes such as uprating pensions, wages and benefits and can aid in the understanding of inflation on family budgets. For more information see Users and uses of consumer price inflation statistics (100.5 Kb Pdf).Latest figure and long-term trendThe CPI 12-month rate (the amount prices change over a year) between January 2015 and January 2016 stood at 0.3%, a small increase compared with December 2015, when the rate was 0.2%. A rate of 0.3% means that a basket of goods and services that cost \u00a3100.00 in January 2015 would have cost \u00a3100.30 in January 2016.The last year, 2015, was a year of very low inflation, with the rate being at or close to zero for most of the year. Looking across 2015 overall, prices for transport costs, food and non-alcoholic beverages and (to a lesser extent) recreational and cultural goods and services had a downward pull on the rate of inflation. These were counterbalanced by an upward pull from price movements for other goods and services, most notably restaurant and hotel bills, and education costs such as university tuition fees. The rate has seen small increases for 3 consecutive months beginning in November 2015, although it remains low in the historic context.Figure A shows the contributions to the CPI 12-month rate in January 2016 compared with the contributions to the 12-month rate a year earlier. A larger version of the chart can be viewed by clicking on it (HTML version only). Figure B shows the CPI 12-month rate over the last 10 years. Table A shows the CPI 1-month rate (the amount prices change between 2 consecutive months), 12-month rate and index values for the last year. A larger version of the chart can be viewed by clicking on it (HTML version only). Consumer Prices Index (CPI): What are the main movements?This section explains which goods and services had the biggest impact on the change to the 12-month rate between December 2015 and January 2016 and, where relevant, considers the longer-term inflationary trends for these goods and services.The change in the CPI 12-month rate can be calculated by comparing the 12-month rates for 2 consecutive months. An alternative, and equally valid, approach is to calculate it by comparing the price change between the latest 2 months and the price change between the same 2 months a year ago. Explaining the contribution to change in the 12-month rate (37.1 Kb Pdf) is a diagram explaining the calculation.The CPI decreased by 0.8% between December 2015 and January 2016, compared with a decrease of 0.9% between the same 2 months a year earlier. The 1-month movement was therefore 0.1 percentage points higher this year compared with a year ago, leading to a rise in the CPI 12-month rate.Between December 2015 and January 2016, the main upward contributions to the change in the CPI 12-month rate came from: motor fuels and lubricants: overall, prices decreased by 2.6%, compared with a larger fall of 6.8% a year ago. The largest upward contribution to the change in the 12-month rate came from prices for petrol, which dropped by 1.9%, compared with a larger fall of 7.3% between the same 2 months a year ago. A similar, though less pronounced, effect was seen for diesel, with prices falling by 4.0%, compared with a fall of 6.0% a year ago food and non-alcoholic beverages: the upward contribution came from food prices which, overall, fell by 0.6% between December 2015 and January 2016, compared with a larger fall of 1.0% between the same 2 months a year ago. The overall upward contribution was due to smaller upward pressures for a variety of products. Notably, prices for shop-bought milk saw a small decrease in price, whereas last year prices fell at a number of major supermarket chains, resulting in a larger decrease alcoholic beverages and tobacco: the upward contribution came from prices for alcoholic beverages which, overall, increased by 5.2% between December 2015 and January 2016, compared with an increase of 3.2% between the same 2 months a year ago. Beer and spirits were the main contributors to the rise. Prices for beer increased between December 2015 and January 2016, which is a return to the more usual pattern of price rises into the new year, following a fall in January last year clothing and footwear: the upward contribution came mainly from prices for clothing which, overall, decreased by 3.4%, compared with a larger fall of 3.9% between the same 2 months a year ago. The overall upward contribution resulted from smaller upward pressures for a variety of items of clothing The only substantial downward contribution to the change in the CPI 12-month rate between December 2015 and January 2016 came from: air fares: prices, overall, fell by 35.8% compared with a smaller fall of 17.1% a year ago. The downward contribution to the change in the 12-month rate was greater than the upward contribution from motor fuels, which has resulted in a downward contribution from transport overall. It is important to note that air fare prices are highly variable. They typically drop in January following increases in December, with the increase in December 2015 being the largest December rise since 2002 Figure C shows the contributions to change from each part of the CPI basket of goods and services. A larger version of the chart can be viewed by clicking on it (HTML version only).",
+ "figures": [
+ {
+ "figure_title": "Figure A: Contributions to the CPI 12-month rate: January 2015 and January 2016",
+ "figure_subtitle": "UK",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016/ea3e8095&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ },
+ {
+ "figure_title": "Figure B: CPI 12-month inflation rate for the last 10 years: January 2006 to January 2016",
+ "figure_subtitle": "UK",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016/3e77f61d&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ },
+ {
+ "figure_title": "Figure C: Contributions to the change in the CPI 12-month rate: January 2016",
+ "figure_subtitle": "UK",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016/39e639b0&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ }
+ ]
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016#cpih",
+ "section_header": "4. CPIH",
+ "section_text": "CPIH is currently undergoing re-assessment to evaluate the extent to which it meets the professional standards set out in the Code of Practice for Official Statistics. CPIH was first assessed in 2013. In August 2014 its National Statistics status was removed pending work to improve the methods for measuring owner occupiers\u2019 housing costs in the index. Full details can be found on the UK Statistics Authority website and in an explanatory note (313.9 Kb Pdf) on our website. The improvements from the resulting development work were introduced as part of the February 2015 dataset with the historical series revised back to 2005. Further information on the changes is available in 2 articles: Improvements to the measurement of Owner Occupiers\u2019 Housing Cost and Private Housing Rental Prices (2.48 Mb Pdf) and Revising the weight of Owner Occupiers\u2019 Housing in CPIH (197.4 Kb Pdf).CPIH is a measure of UK consumer price inflation that includes owner occupiers\u2019 housing costs (OOH). These are the costs of housing services associated with owning, maintaining and living in one\u2019s own home. OOH does not include costs such as utility bills, minor repairs and maintenance, which are already included in the index.CPIH uses an approach called rental equivalence to measure OOH. Rental equivalence uses the rent paid for an equivalent house as a proxy for the costs faced by an owner occupier. In other words this answers the question \u201chow much would I have to pay in rent to live in a home like mine?\u201d for an owner occupier. OOH does not seek to capture increases in house prices. Although this may be inconsistent with some users\u2019 expectations of measures of OOH, the inclusion of an asset price and therefore capital gains would make the index less suitable for a measure of consumption. OOH currently accounts for 16.5% of the expenditure weight of CPIH. This compares with a weight of 19.5% in 2005.Currently, the method of calculation, the population coverage and the basket of goods and services are the same as the Consumer Prices Index (CPI), with the exception of OOH. The method of deriving the weights for CPIH and the data used for these are also the same as for CPI, with the exception of OOH. This can result in some differences from the CPI.In January 2016, the 12-month rate (the rate at which prices increased between January 2015 and January 2016) for CPIH stood at 0.6%, up from 0.5% in December 2015. The difference between the CPI and CPIH annual rates in January 2016 was 0.3 percentage points, the same as the difference in December 2015.Owners occupiers\u2019 housing costs increased by 0.3% between December 2015 and January 2016, compared with 0.2% between these months a year earlier. This means that they had a small impact on the change on the CPIH 12-month rate between the 2 months.Figure D shows the CPIH and OOH component 12-month rates since January 2006 (the earliest date for which the official CPIH 12-month rate can be calculated). The CPI 12-month rate has been included for comparative purposes. Table B shows the CPIH and OOH component 1-month and 12-month rates and index values for the last year. A larger version of the chart can be viewed by clicking on it (HTML version only).",
+ "figures": [
+ {
+ "figure_title": "Figure D: CPIH, OOH component and CPI 12-month rates since January 2006",
+ "figure_subtitle": "UK",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016/7cc4501b&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ }
+ ]
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016#retail-prices-index-rpi-and-rpij",
+ "section_header": "5. Retail Prices Index (RPI) and RPIJ",
+ "section_text": "In accordance with the Statistics and Registration Service Act 2007, the Retail Prices Index and its derivatives have been assessed against the Code of Practice for Official Statistics and found not to meet the required standard for designation as National Statistics. The full assessment report can be found on the UK Statistics Authority website.The RPI is a long-standing measure of UK inflation that has historically been used for a wide range of purposes such as the indexation of pensions, rents and index-linked gilts. For further information see Users and uses of consumer price inflation statistics (100.5 Kb Pdf).RPIJ is an improved variant of the Retail Prices Index, which is calculated using formulae that meet international standards. The rationale for creating RPIJ was to give users a better alternative to the RPI if their needs were for a measure of inflation based on the same population, classifications, weights, etc as the RPI. Currently, RPIJ also acts as an analytical series in that it allows users to see the impact of using the Jevons (which meets international standards) in place of the Carli formula (which does not meet international standards) in the RPI. The use of the different formulae at the elementary aggregate level is currently the only difference between the 2 indices. Detailed goods and services indices are not produced for RPIJ.In January 2016, the 12-month rate for RPIJ stood at 0.7%, up from 0.5% in the year to December 2015.The RPI 12-month rate for January 2016 stood at 1.3%, meaning that it was 0.6 percentage points higher than it would have been had it used formulae that meet international standards.Figure E shows the RPI and RPIJ 12-month rates for the last 10 years. Over this period the RPIJ 12-month rate has been, on average, 0.5 percentage points lower than the RPI but the difference has increased to an average of 0.6 percentage points over the last 3 years. A larger version of the chart can be viewed by clicking on it (HTML version only).Table C shows the RPI and RPIJ 1-month and 12-month rates and index values for the last year. For users who want to understand the causes of the difference between the CPI and RPI, please see Table 5 in the Consumer Price Inflation Reference Tables.",
+ "figures": [
+ {
+ "figure_title": "Figure E: RPI and RPIJ 12-month rates for the last 10 years: January 2006 to January 2016",
+ "figure_subtitle": "UK",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016/90b5f947&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ }
+ ]
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016#guide-to-data",
+ "section_header": "6. Guide to data",
+ "section_text": "Table D outlines where data for all consumer price inflation statistics can be found.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016#7background-notes",
+ "section_header": "\n7\n.Background notes\n",
+ "section_text": "",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/economy/inflationandpriceindices/bulletins/consumerpriceinflation/january2016#8methodology",
+ "section_header": "\n8\n.\nMethodology\n",
+ "section_text": "",
+ "figures": []
+ }
+ ]
+}
\ No newline at end of file
diff --git a/data/bulletins/EXAMPLE-2016-02-18_adult-smoking-habits-in-great-britain-2014.json b/data/bulletins/EXAMPLE-2016-02-18_adult-smoking-habits-in-great-britain-2014.json
new file mode 100644
index 0000000..bfa4b25
--- /dev/null
+++ b/data/bulletins/EXAMPLE-2016-02-18_adult-smoking-habits-in-great-britain-2014.json
@@ -0,0 +1,79 @@
+{
+ "id": "2016-02-18_adult-smoking-habits-in-great-britain-2014",
+ "title": "Adult smoking habits in Great Britain: 2014",
+ "url": "https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/healthandlifeexpectancies/bulletins/adultsmokinghabitsingreatbritain/2014",
+ "release_date": "2016-02-18",
+ "release_type": "bulletins",
+ "latest": false,
+ "url_keywords": [
+ "peoplepopulationandcommunity",
+ "healthandsocialcare",
+ "healthandlifeexpectancies"
+ ],
+ "contact_name": "Dr Craig Orchard",
+ "contact_link": "mailto:socialsurveys@ons.gov.uk",
+ "content": [
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/healthandlifeexpectancies/bulletins/adultsmokinghabitsingreatbritain/2014#main-points",
+ "section_header": "1. Main points",
+ "section_text": "19% of adults in Great Britain currently smoke, down from a peak of 46% in 1974 when the series began. Among smokers, the average consumption was 11.4 cigarettes a day \u2013 the lowest daily cigarette consumption since the series began, the peak being in 1976 at 16.8 cigarettes a day. 20% of men currently smoke \u2013 the lowest value since the series began in the 1970s. In 1974, half (51%) of men smoked. 17% of women smoke \u2013 a slight increase on the previous year. The peak for women was in 1974 at 41%. Those aged 25 to 34 are the most likely to smoke but smoke the lowest amount per day at 9.6 cigarettes while smokers aged 50 to 59 smoke the most at 13.4 cigarettes a day. In 2014, 55% of people who had previously smoked had quit. 23% of those with personal annual income up to \u00a39,999 are current smokers, compared with 11% of those whose annual income is \u00a340,000 or more. Just over 1 in 10 babies were born to mothers who smoke. There are an estimated 2.2 million current e-cigarette users in Great Britain, 4% of the population. 74% of former e-cigarette users smoke cigarettes. 53% of e-cigarettes users vape as an aid to quit smoking. 76% of e-cigarettes users think exposure to non-users has no health impact.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/healthandlifeexpectancies/bulletins/adultsmokinghabitsingreatbritain/2014#summary-of-findings",
+ "section_header": "2. Summary of findings",
+ "section_text": "This report highlights some of the characteristics of adult (16 years old and over) smokers in 2014 and the findings on e-cigarette users in Great Britain in 2015 using our Opinions and Lifestyle Survey.The percentage of current smokers in Great Britain has remained at 19% compared with 2013, but the percentage of male smokers has reached its lowest ever value of the series. Average daily cigarette consumption has also reached its lowest value of the series.The percentage of people using e-cigarettes is 4%. Women are slightly more likely to be e-cigarette users than men. The main reason given for using e-cigarettes was to aid in quitting smoking.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/healthandlifeexpectancies/bulletins/adultsmokinghabitsingreatbritain/2014#cigarette-smoking",
+ "section_header": "3. Cigarette smoking",
+ "section_text": "Smoking is the leading cause of preventable death in Great Britain. In 2013, almost 80,000 deaths were attributable to smoking in England. Estimates from the Scottish and Welsh governments suggest that smoking is responsible for around 13,500 deaths per year in Scotland and 5,500 in Wales. Exposure to second-hand smoke (passive smoking) can lead to a range of diseases, many of which are fatal, with children especially vulnerable to the effects of passive smoking.Smoking also has economic costs, adding significantly to the burden on the NHS. Research from Oxford University suggests that smoking cost the NHS in the UK \u00a35.2 billion in 2005/06. It is estimated that in 2013/14, approximately 4% of all hospital admissions in England for those aged 35 and over were attributable to smoking.Reducing the prevalence of cigarette smoking is therefore a key objective for the government and devolved administrations. The government has set a smoking prevalence target for England of 18.5% by 2015. The Welsh government has a target of 16% by 2020. The Scottish government has a target of 5% by 2034.The UK government and Welsh and Scottish governments have published the papers \u201cHealthy Lives, Healthy People \u2013 A Tobacco Control Plan for England\u201d, \u201cTobacco Control Action Plan for Wales\u201d and \u201cCreating a Tobacco-Free Generation \u2013 A Tobacco Control Plan for Scotland\u201d. These set out their respective strategies for reducing the proportion of the population that smokes and the harm caused by tobacco use.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/healthandlifeexpectancies/bulletins/adultsmokinghabitsingreatbritain/2014#smoking-data",
+ "section_header": "4. Smoking data",
+ "section_text": "Proportion of male smokers reaches its lowest valueMen are more likely to smoke than women and in 2014 across Great Britain, 20% of men aged 16 and over smoked cigarettes. This was a fall on the previous year and the lowest smoking prevalence among men at any time over the past 40 years \u2013 the highest prevalence was in 1974 at 51%.For women aged 16 and over, 17% smoked in 2014, a slight increase on the previous year; the peak for women was in 1974 at 41%. For men and women combined, smoking prevalence was 19%.Across all age groups, men are more likely to smoke than women and the most common age group for people to smoke is 25 to 34, where 24% of people within this age group smoke cigarettes. The next most common age group is 16 to 24 (23% smoke) followed by 35 to 49 (21%). From the age of 50 smoking prevalence starts to fall, in particular for those aged 60 and over (11%). As well as a fall in the prevalence of smoking, there has also been a fall in the number of cigarettes consumed daily. In 2014, around 11.4 cigarettes were smoked per day, the lowest at any point over the past 40 years and down from 12.1 a year earlier. 1976 saw the highest number of cigarettes smoked per day at 16.8.Men smoke on average 1.7 more cigarettes a day. While those aged 25 to 34 are the most likely to smoke, the smokers within this group smoke the lowest amount per day at 9.6 cigarettes, while smokers aged 50 to 59 smoke the most at 13.4 cigarettes a day.Comparing 2013 and 2014, each group\u2019s average daily consumption fell, apart from men aged 60 and over, whose cigarette consumption rose from 13.7 to 14.6, and women aged 50 to 59 whose consumption rose from 12.0 to 12.3. Despite the rise, 12.3 was the second lowest average daily consumption for this age group since 1971. Most of the sex and age breakdowns had their lowest value in 2014; this can be seen in table 2 of the smoking reference tables.Generally from the age of 25, smoking prevalence among the population falls and this is reflected in the data on people who have quit. As people get older they are more likely to have quit \u2013 partly reflecting that they have had more time to do so. In 2014, 55% of people who had previously smoked had quit. The proportion of men who have quit smoking (55%) is slightly higher than the percentage of women who have quit (54%). Women have seen a decrease in the percentage of cigarette smokers who have quit in each age group. Most of the age groups for men have seen increases in the percentage of smokers who have quit except for the oldest 2 age groups which had stayed the same or seen a fall. The proportion of cigarette smokers who have quit is the proportion of all those who said that they have smoked cigarettes regularly, who do not currently smoke. Individuals with an annual income of less than \u00a310,000 are twice as likely to smoke as those with an income of \u00a340,000 and overGenerally, as personal incomes increase people are less likely to smoke. In 2014, 23% of those with an annual personal income of less than \u00a310,000 were current smokers while just 11% of those with an income of \u00a340,000 or more smoked.Those with the lowest incomes are also less likely to have quit smoking. Of those who had ever smoked and had an income of less than \u00a310,000, 48% had quit, while 69% of those who had ever smoked and had an income of \u00a340,000 or more had quit.Individuals with a personal income of less than \u00a320,000 smoked on average between 11.6 and 11.8 cigarettes per day and as well as being less likely to smoke overall, those with an income of \u00a340,000 or more also smoked less per day at 10.0 cigarettes. Just over 1 in 10 babies born to mothers who smokeThe Health and Social Care Information Centre (HSCIC) publishes statistics on women\u2019s smoking status at the time of delivery in England. Over the period 2014/15, around 11.4% of pregnant women were recorded as smokers at the time of delivery. This is down from 12.0% a year earlier.Estimates on the smoking status during pregnancy from the Opinions and Lifestyle Survey suggest that 17% of pregnant women aged 16 to 49 were smokers in 2014, up from 11% a year earlier. However, these estimates are based on a small sample of women and just 70 pregnant women were in the survey in 2014. Therefore there is a margin of uncertainty with the estimates from such a small sample of people. Upper and lower confidence intervals represent the range in which we are 95% confident the exact figure falls and this range in 2014 is between 8% and 26%. This overlaps with the previous year\u2019s confidence intervals and so there are no statistically significant differences in the prevalence of smoking while pregnant. For robustness it is advised to use the statistics collected at the time of delivery when reporting on the smoking status of pregnant women.Other characteristics of smokersThe data discussed in this publication can be found in the reference tables section of the bulletin. In the reference tables, there is also data that has not been included within the commentary including: proportion who have never smoked cigarettes, by sex and age, Great Britain, 1974 to 2014 cigarette smoking status and the proportion of cigarette smokers who have quit, by highest qualification level, Great Britain, 2014 cigarette smoking habits, by economic activity, Great Britain, 1990 to 2014 cigarette smoking status and the proportion of cigarette smokers who have quit, by socio-economic classification, Great Britain, 2014 cigarette smoking status and the proportion of cigarette smokers who have quit, by relationship status, Great Britain, 2014 adult cigarette smoking habits, by sex and whether dependent children living in household, Great Britain, 2000 to 2014 cigarette smoking status and the proportion of cigarette smokers who have quit, by age and whether lone person household, Great Britain, 2014 type of cigarette smoked, by sex, Great Britain, 2014 There is also information on cigarette smoking prevalence since 2010 in the Integrated Household Survey (IHS). These are broadly comparable with the Opinions and Lifestyle Survey (OPN) estimates of cigarette smoking prevalence. The IHS sample is far larger than the OPN sample. This leads to more precise estimates, especially at lower level geographies. The IHS asks questions around smoking of those aged 18 and over, whereas the OPN also asks the questions of 16 and 17 year olds. The construction of the proportions of the population who have never smoked cigarettes, and those who are ex-smokers, also differ, as the OPN asks an additional question around this.",
+ "figures": [
+ {
+ "figure_title": "Figure 1: Percentage of smokers by age and sex, 2014",
+ "figure_subtitle": "Great Britain",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/peoplepopulationandcommunity/healthandsocialcare/healthandlifeexpectancies/bulletins/adultsmokinghabitsingreatbritain/2014/061c62dc&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ },
+ {
+ "figure_title": "Figure 2: Proportion who smoke and proportion who have quit smoking, 2014",
+ "figure_subtitle": "Great Britain",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/peoplepopulationandcommunity/healthandsocialcare/healthandlifeexpectancies/bulletins/adultsmokinghabitsingreatbritain/2014/a4366e66&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ },
+ {
+ "figure_title": "Figure 3: Average daily consumption by income band, 2014",
+ "figure_subtitle": "Great Britain",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/peoplepopulationandcommunity/healthandsocialcare/healthandlifeexpectancies/bulletins/adultsmokinghabitsingreatbritain/2014/8101d2cd&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ }
+ ]
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/healthandlifeexpectancies/bulletins/adultsmokinghabitsingreatbritain/2014#e-cigarette-data-2015-provisional-data",
+ "section_header": "5. E-cigarette data (2015 provisional data)",
+ "section_text": "Half of e-cigarettes users vaping as an aid to quit smokingIn Great Britain, there were an estimated 2.2 million current e-cigarette users in 2015, 4% of the population. There were 3.9 million former users of e-cigarettes and a further 2.6 million people said they had tried an e-cigarette but never went on to use it.Half (53%) of the 2.2 million current e-cigarettes users said their main reason for vaping was to aid themselves in quitting smoking. Just over 1 in 5 (22%) gave their main reason for vaping was because they felt e-cigarettes were less harmful than cigarettes. Despite the cost difference between vaping and smoking, just 9% gave this as the main reason. A further 9% said their main reason was because they could use e-cigarettes indoors.For both current and ex-smokers, the main reason for using e-cigarettes was to help them quit smoking. The second most popular reason for both groups was that they were perceived to be less harmful, although this was the reason for 31% of ex-smokers compared with 15% of current cigarette smokers. Among current cigarette smokers, 15% said that their main reason for using e-cigarettes was that they could be used indoors, compared with 1% of ex cigarette smokers.There are some current users of e-cigarettes who had never previously smoked, although this number is relatively small, at 56,000 (0.1% of the population or 3% of current users). Three-quarters of former e-cigarette users smoke cigarettesOf the former e-cigarette users, around three-quarters said they were currently smoking cigarettes and 59% of the current users said they also used cigarettes. Current e-cigarette users smoke a slightly lower number of cigarettes a day on average (11.2 per day) than the average for all smokers (11.4 per day). Former e-cigarette users\u2019 average daily cigarette consumption (12.4) is higher than those who have never used an e-cigarette (10.9).Focusing on those who currently smoke, around 36% had never used an e-cigarette. In 2015, 1.3 million people were current e-cigarette and cigarette smokers. Three-quarters of e-cigarettes users think exposure to non-users has no health impactOf those surveyed who were current or former users of e-cigarettes, 76% felt that exposure of the vapour of e-cigarettes to non-users had no health impact. On the other hand, 41% of people who have never smoked or vaped felt that exposure had a damaging impact on their health.In terms of personal health impact, three-quarters (73%) of all those surveyed regardless of smoking or vaping status felt that e-cigarettes were less harmful than cigarettes. However, current or former e-cigarettes users gave a stronger opinion that e-cigarettes were less harmful than people who only smoked or had never smoked.The percentages of e-cigarette users are similar for men and women but there is a difference in age profile of those who use them. The most common age group for women to vape were those aged 35 to 44, whereas for men the most common groups are 45 to 54 and 55 to 64.Most people (67%) use an e-cigarette on a daily basis and a further 19% use one at least once a week, with the most common choice of e-cigarette being one that doesn\u2019t resemble a cigarette.More data can be found in the reference tables provided.E-cigarettes have been sold since 2004 and in Europe since 2006. Their popularity and availability has increased, which has led to debate around their use. Some feel that e-cigarettes could renormalise smoking, or could be a gateway to smoking by introducing non-smokers to nicotine. Others feel that they could be a useful tool in the effort to reduce tobacco consumption. To date, e-cigarettes have mainly been marketed as a cheaper and healthier alternative to smoking. However, the long-term health effects of using e-cigarettes have yet to be established. This has led to a World Health Organisation call for tighter controls on e-cigarettes.",
+ "figures": [
+ {
+ "figure_title": "Figure 4: Percentage of smokers by e-cigarette use, 2014",
+ "figure_subtitle": "Great Britain",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/peoplepopulationandcommunity/healthandsocialcare/healthandlifeexpectancies/bulletins/adultsmokinghabitsingreatbritain/2014/f6dc3f96&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ }
+ ]
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/healthandlifeexpectancies/bulletins/adultsmokinghabitsingreatbritain/2014#6background-notes",
+ "section_header": "\n6\n.Background notes\n",
+ "section_text": "",
+ "figures": []
+ }
+ ]
+}
\ No newline at end of file
diff --git a/data/bulletins/EXAMPLE-2023-05-19_public-opinions-and-social-trends-great-britain-4-to-14-may-2023.json b/data/bulletins/EXAMPLE-2023-05-19_public-opinions-and-social-trends-great-britain-4-to-14-may-2023.json
new file mode 100644
index 0000000..221951e
--- /dev/null
+++ b/data/bulletins/EXAMPLE-2023-05-19_public-opinions-and-social-trends-great-britain-4-to-14-may-2023.json
@@ -0,0 +1,111 @@
+{
+ "id": "2023-05-19_public-opinions-and-social-trends-great-britain-4-to-14-may-2023",
+ "title": "Public opinions and social trends, Great Britain: 4 to 14 May 2023",
+ "url": "https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023",
+ "release_date": "2023-05-19",
+ "release_type": "bulletins",
+ "latest": false,
+ "url_keywords": [
+ "peoplepopulationandcommunity",
+ "wellbeing"
+ ],
+ "contact_name": "Ben Harris, Hannah White, David Ainslie, Tim Vizard",
+ "contact_link": "mailto:policy.evidence.analysis@ons.gov.uk",
+ "content": [
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023#main-points",
+ "section_header": "1. Main Points",
+ "section_text": "The following information is for the latest survey period 4 to 14 May 2023, based on adults in Great Britain. When asked about the important issues facing the UK today, the most commonly reported issues continue to be the cost of living (91%), the NHS (82%), the economy (73%), and climate change and the environment (59%). Around two-thirds of (67%) adults reported that their cost of living had increased, compared with a month ago. The most common reasons reported by adults who said their cost of living had increased compared with a month ago were an increase in the price of food shopping (96%), an increase in gas or electricity bills (74%), and an increase in the price of fuel (38%). Around 6 in 10 (59%) reported watching (or planning to watch) the coronation of King Charles III and Camilla, Her Majesty the Queen on TV; this proportion increased with age, being reported by 39% of people aged 16 to 29 years, 56% of those aged 30 to 49 years, 62% of those aged 50 to 69 years, and 82% of those aged 70 years or over. Around half (49%) of adults reported trying to contact a GP practice for themselves or someone else in the past month. Around 4 in 10 (40%) adults who had tried to contact a GP practice for themselves or someone else in the past month reported it was easy or very easy to contact the GP practice, and 36% reported that it was difficult or very difficult. Around 6 in 10 (60%) of those who tried to contact a GP practice for themselves or someone else in the past month described their overall experience of the GP practice as very or fairly good, while 22% reported that it was very or fairly poor.",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023#important-issues-facing-the-uk",
+ "section_header": "2. Important issues facing the UK",
+ "section_text": "Estimates in this release are based on data collected between 4 and 14 May 2023 (the \"latest period\") and 19 April and 1 May 2023 (the \"previous period\"). Analysis is based on adults in Great Britain.Important issuesIn the latest period, we asked adults what they feel are important issues facing the UK today. The most commonly reported issues were: the cost of living (91%) the NHS (82%) the economy (73%) climate change and the environment (59%)",
+ "figures": [
+ {
+ "figure_title": "Figure 1: The cost of living and the NHS continue to be the most commonly reported important issues",
+ "figure_subtitle": "Proportion of all adults in Great Britain, October 2022 to May 2023",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023/ebf7d799&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ }
+ ]
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023#cost-of-living",
+ "section_header": "3. Cost of living",
+ "section_text": "Over 9 in 10 (93%) people reported that their cost of living had increased, compared with a year ago. Compared with one month ago, 67% reported an increase in their cost of living (70% in the previous period).Reasons for cost of living increasesThe most commonly reported reasons given by adults for the rise in their cost of living over the past month were increases in: the price of food shopping (96%) their gas or electricity bills (74%) the price of fuel (38%) Data released in our Consumer price inflation (CPI), UK: March 2023 bulletin show that food costs and housing costings (including energy, electricity and other fuel) continue to be causes of CPI.Actions taken because of the rising cost of livingThe most common actions reported by all adults because of the rising cost of living in the latest period were: spending less on non-essentials (66%) shopping around more (49%) using less fuel such as gas or electricity in their home (49%) spending less on food shopping and essentials (44%)",
+ "figures": [
+ {
+ "figure_title": "Figure 2: Around two-thirds (66%) of adults reported spending less on non-essentials because of the rising cost of living",
+ "figure_subtitle": "Proportion of adults taking actions because of increases in the cost of living, Great Britain, December 2021 to May 2023",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023/55b5a167&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ }
+ ]
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023#gp-practice-access",
+ "section_header": "4. GP practice access",
+ "section_text": "In the latest period, we asked adults about their experiences of accessing a GP practice.Around half (49%) of adults reported trying to contact a GP practice for themselves or someone else in the past month.Of these adults: around 7 in 10 (68%) reported they had managed to make contact on the same day they tried (had received a response from a receptionist or clinician or been able to book an appointment) 8% reported they had made contact on the day after 15% reported they had made contact two or more days later 10% reported they didn't manage to make contact with the GP practice Around 4 in 10 (40%) of these adults reported it was easy or very easy to contact the GP practice, whereas 36% reported that it was difficult or very difficult. Around 6 in 10 (60%) of these adults described their overall experience of the GP practice as very or fairly good, and 22% reported that it was very or fairly poor. More information about the methods people contacted their GP practice by and any next steps they took after making contact is available in the GP practice access dataset.A more detailed looked at people's experience of GP practice access is included in our Impact of winter pressures on different population groups in Great Britain article. You can also explore estimates of how people feel about their GP practice experience in England using estimates from the GP patient survey.",
+ "figures": [
+ {
+ "figure_title": "Figure 3: Around 6 in 10 (60%) adults described their overall experience of the GP practice as very or fairly good",
+ "figure_subtitle": "Proportion among adults who tried to contact a GP practice in the past month for themselves or someone else, Great Britain, 4 to 14 May 2023",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023/974fa2b9&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ }
+ ]
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023#personal-well-being",
+ "section_header": "5. Personal well-being",
+ "section_text": "During the latest period, we continued to ask respondents about their personal well-being. Average levels of personal well-being were: life satisfaction: 6.9 in the latest period (6.9 in the previous period) feeling that the things done in life are worthwhile: 7.2 in the latest period (7.2 in the previous period) happiness: 7.0 in the latest period (7.1 in the previous period) anxiety: 3.9 in the latest period (3.9 in the previous period) Adults in Great Britain, March 2020 to May 2023 Notes: Questions: \"Overall, how satisfied are you with your life nowadays?\", \"Overall, to what extent do you feel that the things you do in your life are worthwhile?\", \"Overall, how happy did you feel yesterday?\", \"Overall, how anxious did you feel yesterday?\". These questions are answered on a scale of 0 to 10, where 0 is \"not at all\" and 10 is \"completely\". Base: all adults. We also asked adults how often they feel lonely. Around a quarter (26%) of adults reported that they felt lonely often, always, or some of the time (25% in the previous period).Section 3 of our Personal well-being quarterly estimates technical report provides more information on the seasonal variation associated with measures of personal well-being. You can learn more about the Measures of National Well-being from our Quality of life in the UK: May 2023 bulletin and our Measures of National Well-being dashboard, which provides longer-term trends in well-being since 2012.Download the data",
+ "figures": [
+ {
+ "figure_title": "4",
+ "figure_subtitle": "Figure 4: Levels of personal well-being",
+ "figure_url": "https://www.ons.gov.uk/visualisations/dvc2595/wellbeing/wrapper/index.html",
+ "figure_type": "interactive"
+ }
+ ]
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023#kings-coronation",
+ "section_header": "6. King\u2019s coronation",
+ "section_text": "The coronation of King Charles III and Camilla, Her Majesty the Queen, took place on 6 May 2023, during the latest period of our survey.We asked adults what activities they did or planned to do over the coronation weekend, 6 to 8 May 2023. Around 6 in 10 (59%) people reported they watched, or planned to watch, the Coronation Service on TV. This proportion increased with age, being reported by: 39% of those aged 16 to 29 years 56% of those aged 30 to 49 years 62% of those aged 50 to 69 years 82% of those aged 70 years or over Figure 5 shows the proportion of adults undertaking a range of activities we asked about.",
+ "figures": [
+ {
+ "figure_title": "Figure 5: Around 6 in 10 (59%) adults reported they planned to or did watch the Coronation Service on TV",
+ "figure_subtitle": "Proportion of all adults in Great Britain, 4 to 14 May 2023",
+ "figure_url": "https://www.ons.gov.uk/chartimage?uri=/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023/59504068&width=&hideSource=true",
+ "figure_type": "chartbuilder_image"
+ }
+ ]
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023#public-opinions-and-social-trends-data",
+ "section_header": "7. Public opinions and social trends data",
+ "section_text": "",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023#measuring-the-data",
+ "section_header": "8. Measuring the data",
+ "section_text": "This release contains data and indicators from a module being undertaken through the Office for National Statistics's (ONS's) Opinions and Lifestyle Survey (OPN).Breakdowns by age and sex, including confidence intervals for the estimates, are contained in our Public opinions and social trends, Great Britain datasets.Where changes in results from previous weeks are presented in this bulletin, or comparisons between estimates are made, associated confidence intervals should be used to assess the statistical significance of the differences.Sampling and weightingIn the latest period (4 May to 14 May 2023), we sampled 4,961 households. This sample was randomly selected from people who had previously completed the Labour Market Survey (LMS) or OPN. The responding sample for the latest period contained 2,045 individuals, representing a 41% response rate.Survey weights were applied to make estimates representative of the population (based on ONS population estimates). Further information on the survey design and quality can be found in our Opinions and Lifestyle Survey Quality and Methodology Information (QMI).",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023#related-links",
+ "section_header": "9. Related links",
+ "section_text": "",
+ "figures": []
+ },
+ {
+ "section_url": "https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023#1-cite-this-statistical-bulletin",
+ "section_header": "10. Cite this statistical bulletin",
+ "section_text": "",
+ "figures": []
+ }
+ ]
+}
diff --git a/data/feedback/.gitkeep b/data/feedback/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/data/test_outcomes/.gitkeep b/data/test_outcomes/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/docs/api/README.md b/docs/api/README.md
new file mode 100644
index 0000000..ce09432
--- /dev/null
+++ b/docs/api/README.md
@@ -0,0 +1,97 @@
+The ONS StatsChat API lets you retrieve ONS publications based on search question.
+ +| Method | +Endpoint | +Description | + + +
|---|---|---|
| GET | +/search | ++ Search for a question. + | +
| GET | +/bulletins | ++ TODO: Search bulletins. + | +
| GET | +/articles | ++ TODO: Search articles. + | +
| GET | +/datasets | ++ TODO: Search datsets. + | +
| POST | +/bulk | ++ TODO: Runs a batch of search terms. + | +
| Method | +Endpoint | +Description | + + +
|---|---|---|
| GET | ++ | + No custom endpoint implemented + | +
| Method | +Endpoint | +Description | + + +
|---|---|---|
| GET | +/about | ++ Get version information. + | +
| GET | +/options | ++ TODO: Return something useful + | +
Search for an answer to free text question.
+ +GET /search
| Parameter name | +Value | +Description | +Additional | + + +
|---|---|---|---|
| q | +string | +Free text question. | ++ Required + | +
| latest | +string | +TODO: Weight to prioritise latest publication. (0 = No prioritisation) | +
+ Optional
+ Default: NA + |
+
| limit | +string | +TODO: Specifies the number of references to return. | +
+ Optional
+ Default: 10 + Maximum: 100 + |
+
Success. A json return of answer and references.
+ +Bad request. Indicates an issue with the request. Further details are provided in the response.
+ +TODO: Server too busy. The Address Index API is experiencing exceptional load.
+ +TODO: Internal server error. Failed to process the request due to an internal error.
+ + + + +curl #API_URL#/search?q=how+many+people+watched+coronation
+{
+ "answer":"Around 6 in 10 people reported they watched, or planned to watch, the Coronation Service on TV.",
+ "confidence":"55%",
+ "references":[
+ {
+ "context":"... reported by adults who said their cost of living had increased compared with a month ago were an increase in the price of food shopping (96%), an increase in gas or electricity bills (74%), and an increase in the price of fuel (38%). Around 6 in 10 (59%) reported watching (or planning to watch) the coronation of King Charles III and Camilla, Her Majesty the Queen on TV; this proportion increased with age, being reported by 39% of people aged 16 to 29 years, 56% of those aged 30 to 49 ...",
+ "date":"19 May 2023",
+ "figures":[],
+ "score":"55%",
+ "section":"1. Main Points",
+ "section_url":"https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023#main-points",
+ "source":"2023-05-19_public-opinions-and-social-trends-great-britain-4-to-14-may-2023",
+ "title":"Public opinions and social trends, Great Britain: 4 to 14 May 2023",
+ "url":"https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023"
+ },
+ {
+ "context":"... Her Majesty the Queen, took place on 6 May 2023, during the latest period of our survey.We asked adults what activities they did or planned to do over the coronation weekend, 6 to 8 May 2023. Around 6 in 10 (59%) people reported they watched, or planned to watch, the Coronation Service on TV. This proportion increased with age, being reported by: 39% of those aged 16 to 29 years 56% of those aged 30 to 49 years 62% of those aged 50 to 69 years 82% of those aged 70 years or over Figure ...",
+ "date":"19 May 2023",
+ "figures":[
+ {
+ "figure_subtitle":"Proportion of all adults in Great Britain, 4 to 14 May 2023",
+ "figure_title":"Figure 5: Around 6 in 10 (59%) adults reported they planned to or did watch the Coronation Service on TV","figure_type":"chartbuilder_image",
+ "figure_url":"https://www.ons.gov.uk/chartimage?uri=/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023/59504068&width=&hideSource=true"
+ }
+ ],
+ "score":"55%",
+ "section":"6. King\u2019s coronation",
+ "section_url":"https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023#kings-coronation",
+ "source":"2023-05-19_public-opinions-and-social-trends-great-britain-4-to-14-may-2023",
+ "title":"Public opinions and social trends, Great Britain: 4 to 14 May 2023",
+ "url":"https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/publicopinionsandsocialtrendsgreatbritain/4to14may2023"
+ }
+ ]
+}
+ + Answer based on following publication(s): +
+ {% else %} ++ Most relevant publication(s): +
+ {% endif %} ++ Context: {{row['page_content']}} +
+ {% for image in row['figures'] %} + {% if image['figure_type']=='interactive' %} + + {% else %} + }}){kind=small}
+ {% endfor %}
+  }}){kind=spacer}
+
+
+
+