Connecting to the StatXplore API in PowerBI

Office for National Statistics
Office for National StatisticsOffice for National Statistics

This is a guide prepared by Suffolk County Council on connecting to the StatXPlore API using PowerBI, when constructing dashboards.

Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Connecting to the StatXplore API using Power BI
Introduction
The StatXplore API allows users to retrieve data as it is updated. The response sent by the API
can appear challenging to process, but data can be successfully extracted given the consistent
layout of the response.
The response layout requires two queries; one to retrieve labels for each of the fields requested
(such as date or geography), plus an additional query for the data itself. These queries must be
linked to use the labels alongside the data.
This guide is split into two main sections: how to query the API and how to process the
response. The Appendix contains examples of query bodies, a query template, and a full query
example (excluding API Key).
The method described in this guide was initially created with assistance from colleagues at IJYI.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Querying the StatXplore API
Prerequisites
To query the StatXplore API, you will need:
a. A personal API Key. This is obtained by registering on the StatXplore website. Always
use your personal API Key. If the user assigned to your API Key deregisters from
StatXplore, your queries will not function. Once registered, your API Key can be found
in the Account section of the website. Use the link in the top right once registered and
logged in:
b. The body of the query to send to the API. The easiest way to construct this is to use the
StatXplore website to build the table of data you need, then export it as an Open Data
API query (.json). Save this file so that you can refer to it later. This export will need
further editing before use in a query, see Preparing the Query Body.
c. A program to prepare the query body. Notepad++ is recommended, although
alternative software can be used.
d. A template to put your query body into. An example template can be found in the
Appendix of this document.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Preparing the Query Body
The .json file that is exported from StatXplore (see Prerequisites) must be edited before use.
This needs to be done for each query that you use. See the finished Example query body (1) and
Example query body (2) in the Appendix.
1. Open the .json file exported from StatXplore in step B of the Prerequisites in Notepad++.
2. Remove the text relating to date fields from the recodes section, but not from the
dimensions section. This is done so that values from all dates are returned, not only
those dates specified on the StatXplore website. Without this step, queries will not
return new data. Remove the text from and including the comma preceding the date
field name, up to and including the right parenthesis after the ‘total’ line of the date
field (see examples below where the highlighted text should be deleted).
3. Remove the text relating to any other fields for which you want to return all available
data rather than only the values initially specified. Use the same technique as above:
remove the text from and including the comma preceding the field name, up to and
including the right parenthesis after the ‘total’ line of the field. This could be used, for
example, to see values for all the employment indicators for Alternative Claimant Count.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Remove the text relating to these fields from the recodes section, but not the
dimensions section. Removing text from the dimensions section would remove this field
from the query entirely.
4. Replace every instance of a speech mark (") with a double speech mark (""). Note that
this must be the two vertical lines; using the quotation marks in Word (“ and ”) will not
be recognised and will cause the query to fail.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Creating a Query
1. In Power BI desktop, add a new data source and use the Blank Query option.
2. With this new blank query created, open the Advanced Editor. Select everything in the
box, delete it, then paste in the Query Template.
3. The query template creates the variables url, body and header:
• url is the web address used for any query that aims to bring back data from the
StatXplore API. This does not need to change.
• body is the part of the query that tells the API which data to return. Paste your
prepared query body between the speech marks in the body line.
• header is where your personal API Key is added to authenticate access to the API.
Paste your API key between the speech marks in the header line.
The variable response uses the url, body and header variables as inputs to the Web.Contents
function. response is returned as json using the Json.Document function.
Click Done and wait for the response from the API. If your query has worked, you should see
this:
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
If so, you can now access and transform the data.
If asked for credentials for this data source, select Anonymous credentials and Public privacy
level.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Accessing and Transforming Data
Overview
The response from the API should look like this:
The data returned by the query comes from the fields and cubes rows of this table. fields
contains the labels for your dataset such as geography, date and sex. cubes contains the values
of your dataset. The yellow words List and Record act as links and can be used to navigate
through the json document.
By default, there are no labels assigned to the values in the cubes section. The data is
presented in a hierarchy, but there are no labels assigned to each level in the hierarchy.
Fortunately, the response sent by the API will always follow a set of rules. The rules of the API
response are:
a. The hierarchy of data within the cubes is always in the same order as the List of fields.
b. The labels for each field are always in the same order as the List of records within a
field.
These rules mean that, if expanded correctly, a flat table can be created with each combination
of labels in the order that they occur in the data. The general process for this method is:
1. Transform the list of fields into a single row.
2. Expand each column, working from left to right, to obtain the labels. Each expansion will
create new rows for each combination of existing labels and the newly expanded labels.
3. In a duplicate query, expand the list of values to new rows. Continue expanding to new
rows until the data appears.
4. The order of rows in the labels and values tables is identical. Assign each row in both
tables an index value to allow the tables to be merged or linked.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
5. Merge the values table into the label table using the index columns and expand out the
values column into the labels table. Alternatively, create a relationship between the
index columns in the modelling view and use both tables.
Worked Example: Households on Universal Credit
To show more effectively how to transform the data, I have included a worked example of a
connection to the database for Households on Universal Credit. This uses the example query in
the Appendix. The step numbers correspond to the numbers from the Overview.
Step 1: Transform the list of fields into a single row
Begin from the response provided by the API (see Creating a Query). Click on the List in the
fields row.
From the Transform menu, select Convert To Table. This allows the Records to be manipulated.
Expand the column of Records to get a List for the labels for each field (there are two in this
example, area and date). Click the Expand button at the top right of the column. Select only the
items checkbox, then click OK.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
The Lists need to be switched from rows to columns. Use the Transpose option from the
Transform menu.
There is now a column for each field in the query.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Step 2: Expand the labels table to obtain labels for each row
Each column should now be expanded to obtain the labels for each field. The rows will be
created in the order that they will appear in the data. As each column is expanded, new rows
are added for each existing row.
Click the Expand button at the top-right of the first column. Select Expand to New Rows. This
will create a new row for each of the labels in this field.
Then click the Expand button again and select only the Labels checkbox.
Then click the Expand button for the third time and select Expand to New Rows.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
This has added to labels for the first field. Now, return to the start of this step and repeat the
process for all the field columns, working from left to right, until the rightmost column has
labels rather than Lists. When expanding each column to new rows for the first time, a new row
will be created for each label within the new row, for each existing row. In this example, the
expansion of the second column results in 76 rows for each of the existing rows (Babergh, East
Suffolk et cetera) because the second field (month) has 76 labels.
Once all the columns have been expanded and the labels are shown, the labels table expansion
is complete. Each row will be a unique combination of labels which will appear in the same
order as the values table. Rename the columns and change their types accordingly.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Step 3: Expand the values table
Duplicate the previous query and delete all steps up to and including the ‘fields’ step.
Navigate to the Record in the Cubes row. Then select the Record within that, then the List
within that. This should display a List of Lists.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Next, convert this List to a table from the List Tools menu; click OK in the dialog box that
appears.
The table should now display one column with a List in each row. Each List must be expanded to
get the data values. Click the expand button at the top-right of the column and select Expand to
new rows. Repeat this until the data is displayed.
The data is now displayed with a row for each value. These rows correspond exactly to the
order of the rows in the labels table.
Rename the values column and change its type accordingly.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Step 4: Create index columns
The labels and values tables are now complete and ready to be linked. For both queries, add an
index column using the button in the Add Column tab. Use the same settings for the index
columns for both queries.
The tables can now be linked using modelling (relationships) or transformations (merging
queries).
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Step 5a: Linking by merging tables
To use the merging queries approach, open Transform data and select the labels table. From
the Combine section of the Home tab, select Merge Queries.
Select the index column from the labels table. Then select the values table from the first drop-
down box and select the index column. A Left Outer join will be required as the aim is to add
the value column from the values table to the labels table. All rows should match, if not there
has been an error in the expansion of the tables or the indexes have been created using
different settings. Click OK.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Click the expand button in the top-right of the new column. Expand out only the values column.
This will add the values to the labels table, resulting in one table that can be used to filter and
display data as required.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Step 5b: Linking by relationships
To use the modelling approach, load the data and switch to the Model view. Click and drag the
tables you will be using so that they are close together.
Click and drag the index column from one table to the other. This will automatically create a
relationship between these columns.
The relationship should always be of the type 1:1. This will be automatically calculated. If the
relationship is not of this type, there has been an error in the expansion of the tables or the
indexes have been created using different settings. To make the report easier to use, hide the
index columns from the report view using the eye icon.
The fields from the labels table can now be used to filter and display the data in the values
table.
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Appendix
Example query body (1)
{
""database"" : ""str:database:UC_Households"",
""measures"" : [ ""str:count:UC_Households:V_F_UC_HOUSEHOLDS"" ],
""recodes"" : {
""str:field:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE"" : {
""map"" : [ [
""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R
EGION:E07000200"" ], [
""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R
EGION:E07000244"" ], [
""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R
EGION:E07000202"" ], [
""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R
EGION:E07000203"" ], [
""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R
EGION:E07000245"" ] ],
""total"" : true
}
},
""dimensions"" : [ [ ""str:field:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE"" ], [
""str:field:UC_Households:F_UC_DATE:DATE_NAME"" ] ]
}
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Example query body (2)
{
""database"" : ""str:database:ACC"",
""measures"" : [ ""str:count:ACC:V_F_ACC"" ],
""recodes"" : {
""str:field:ACC:V_F_ACC:UK_COA"" : {
""map"" : [ [
""str:value:ACC:V_F_ACC:UK_COA:V_C_MASTERGEOG11_LA_TO_REGION_NI:E07000200"" ], [
""str:value:ACC:V_F_ACC:UK_COA:V_C_MASTERGEOG11_LA_TO_REGION_NI:E07000244"" ], [
""str:value:ACC:V_F_ACC:UK_COA:V_C_MASTERGEOG11_LA_TO_REGION_NI:E07000202"" ], [
""str:value:ACC:V_F_ACC:UK_COA:V_C_MASTERGEOG11_LA_TO_REGION_NI:E07000203"" ], [
""str:value:ACC:V_F_ACC:UK_COA:V_C_MASTERGEOG11_LA_TO_REGION_NI:E07000245"" ] ],
""total"" : true
}
},
""dimensions"" : [ [ ""str:field:ACC:V_F_ACC:UK_COA"" ], [
""str:field:ACC:F_ACC_DATE_new:DATE_NAME"" ], [ ""str:field:ACC:V_F_ACC:EMP"" ] ]
}
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Full example query
For privacy reasons the API key has been omitted. Copy and paste your personal API key
between the speech marks after APIKey= to get this query to work.
let
url = "https://stat-xplore.dwp.gov.uk/webapi/rest/v1/table",
body = "{
""database"" : ""str:database:UC_Households"",
""measures"" : [ ""str:count:UC_Households:V_F_UC_HOUSEHOLDS"" ],
""recodes"" : {
""str:field:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE"" : {
""map"" : [ [
""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R
EGION:E07000200"" ], [
""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R
EGION:E07000244"" ], [
""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R
EGION:E07000202"" ], [
""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R
EGION:E07000203"" ], [
""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R
EGION:E07000245"" ] ],
""total"" : true
}
},
""dimensions"" : [ [ ""str:field:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE"" ], [
""str:field:UC_Households:F_UC_DATE:DATE_NAME"" ] ]
}",
header = [ #"APIKey"="" ],
response = Web.Contents(url,[Content=Text.ToBinary(body),Headers=header]),
Source = Json.Document(response,1252)
in
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Source
Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023)
phc_analytics@suffolk.gov.uk
Query template
let
url = "https://stat-xplore.dwp.gov.uk/webapi/rest/v1/table",
body = "",
header = [ #"APIKey"="" ],
response = Web.Contents(url,[Content=Text.ToBinary(body),Headers=header]),
Source = Json.Document(response,1252)
in
Source

Recomendados

Connecting to the NOMIS API in PowerBI por
Connecting to the NOMIS API in PowerBIConnecting to the NOMIS API in PowerBI
Connecting to the NOMIS API in PowerBIOffice for National Statistics
77 vistas4 diapositivas
ONS Local presents Suffolk County Council's Cost of Living Dashboard por
ONS Local presents Suffolk County Council's Cost of Living DashboardONS Local presents Suffolk County Council's Cost of Living Dashboard
ONS Local presents Suffolk County Council's Cost of Living DashboardOffice for National Statistics
173 vistas17 diapositivas
What is copa master data por
What is copa master dataWhat is copa master data
What is copa master dataRajeev Kumar
2K vistas7 diapositivas
F.16 balance carry forward of gl accounts por
F.16 balance carry forward of gl accountsF.16 balance carry forward of gl accounts
F.16 balance carry forward of gl accountsFarooq Wangde
3.4K vistas7 diapositivas
Copa realignment por
Copa realignmentCopa realignment
Copa realignmentsuryanarayana tata
2.3K vistas11 diapositivas
06-manual-activos-fijos.pdf por
06-manual-activos-fijos.pdf06-manual-activos-fijos.pdf
06-manual-activos-fijos.pdf7158AS
169 vistas67 diapositivas

Más contenido relacionado

Similar a Connecting to the StatXplore API in PowerBI

1 Week 6 - What Well Be Working On This Week In th.docx por
1 Week 6 - What Well Be Working On This Week  In th.docx1 Week 6 - What Well Be Working On This Week  In th.docx
1 Week 6 - What Well Be Working On This Week In th.docxdorishigh
2 vistas71 diapositivas
Office excel tips and tricks 201101 por
Office excel tips and tricks 201101Office excel tips and tricks 201101
Office excel tips and tricks 201101Vishwanath Ramdas
1.4K vistas39 diapositivas
Whatsapp survery report por
Whatsapp survery  reportWhatsapp survery  report
Whatsapp survery reportKaran Kukreja
4.6K vistas23 diapositivas
Microsoft Excel- basics por
Microsoft Excel-  basicsMicrosoft Excel-  basics
Microsoft Excel- basicsjeshin jose
1.7K vistas58 diapositivas
ROLL NO 1 TO 9(G1) USE OF EXCEL IN CA PROFESSION (Final Draft).pptx por
ROLL NO 1 TO 9(G1) USE OF EXCEL IN CA PROFESSION (Final Draft).pptxROLL NO 1 TO 9(G1) USE OF EXCEL IN CA PROFESSION (Final Draft).pptx
ROLL NO 1 TO 9(G1) USE OF EXCEL IN CA PROFESSION (Final Draft).pptxDishantGola
405 vistas42 diapositivas
X rec extened reconciliation using excel vba por
X rec   extened reconciliation using excel vbaX rec   extened reconciliation using excel vba
X rec extened reconciliation using excel vbasenthilsundaresan
690 vistas8 diapositivas

Similar a Connecting to the StatXplore API in PowerBI(20)

1 Week 6 - What Well Be Working On This Week In th.docx por dorishigh
1 Week 6 - What Well Be Working On This Week  In th.docx1 Week 6 - What Well Be Working On This Week  In th.docx
1 Week 6 - What Well Be Working On This Week In th.docx
dorishigh2 vistas
Whatsapp survery report por Karan Kukreja
Whatsapp survery  reportWhatsapp survery  report
Whatsapp survery report
Karan Kukreja4.6K vistas
Microsoft Excel- basics por jeshin jose
Microsoft Excel-  basicsMicrosoft Excel-  basics
Microsoft Excel- basics
jeshin jose1.7K vistas
ROLL NO 1 TO 9(G1) USE OF EXCEL IN CA PROFESSION (Final Draft).pptx por DishantGola
ROLL NO 1 TO 9(G1) USE OF EXCEL IN CA PROFESSION (Final Draft).pptxROLL NO 1 TO 9(G1) USE OF EXCEL IN CA PROFESSION (Final Draft).pptx
ROLL NO 1 TO 9(G1) USE OF EXCEL IN CA PROFESSION (Final Draft).pptx
DishantGola405 vistas
X rec extened reconciliation using excel vba por senthilsundaresan
X rec   extened reconciliation using excel vbaX rec   extened reconciliation using excel vba
X rec extened reconciliation using excel vba
senthilsundaresan690 vistas
How to use vlookup in MS Excel por Jaspal Singh
How to use vlookup in MS ExcelHow to use vlookup in MS Excel
How to use vlookup in MS Excel
Jaspal Singh342 vistas
Handouts how to use microsoft access to combine cost and usage data for ejour... por Charleston Conference
Handouts how to use microsoft access to combine cost and usage data for ejour...Handouts how to use microsoft access to combine cost and usage data for ejour...
Handouts how to use microsoft access to combine cost and usage data for ejour...
Charleston Conference1.6K vistas
Since the instructions for the final project are standardized and .docx por edgar6wallace88877
Since the instructions for the final project are standardized and .docxSince the instructions for the final project are standardized and .docx
Since the instructions for the final project are standardized and .docx
Share point ssis adapters 2011 por Krishna Na
Share point ssis adapters 2011Share point ssis adapters 2011
Share point ssis adapters 2011
Krishna Na1.1K vistas
MS-EXCEL Assignment Help por Rahul Kataria
MS-EXCEL Assignment HelpMS-EXCEL Assignment Help
MS-EXCEL Assignment Help
Rahul Kataria13.2K vistas
Introduction to SiteCatalyst ReportBuilder por Peter O'Neill
Introduction to SiteCatalyst ReportBuilderIntroduction to SiteCatalyst ReportBuilder
Introduction to SiteCatalyst ReportBuilder
Peter O'Neill8.5K vistas
Create an Evaluation Matrix Chart in Excel that will let you com por co4spmeley
Create an Evaluation Matrix Chart in Excel that will let you comCreate an Evaluation Matrix Chart in Excel that will let you com
Create an Evaluation Matrix Chart in Excel that will let you com
co4spmeley33 vistas
Stage 3 System Design SpecificationBefore you begin this as.docx por whitneyleman54422
Stage 3   System Design SpecificationBefore you begin this as.docxStage 3   System Design SpecificationBefore you begin this as.docx
Stage 3 System Design SpecificationBefore you begin this as.docx
4b6c1c5c-e913-4bbf-b3a4-41e23cb961ba-161004200047.pdf por Nitish Nagar
4b6c1c5c-e913-4bbf-b3a4-41e23cb961ba-161004200047.pdf4b6c1c5c-e913-4bbf-b3a4-41e23cb961ba-161004200047.pdf
4b6c1c5c-e913-4bbf-b3a4-41e23cb961ba-161004200047.pdf
Nitish Nagar10 vistas
Developing a ssrs report using a ssas data source por relekarsushant
Developing a ssrs report using a ssas data sourceDeveloping a ssrs report using a ssas data source
Developing a ssrs report using a ssas data source
relekarsushant3.6K vistas
Excel Datamining Addin Beginner por excel content
Excel Datamining Addin BeginnerExcel Datamining Addin Beginner
Excel Datamining Addin Beginner
excel content6.3K vistas

Más de Office for National Statistics

SlideShare ONS Economic Forum Slidepack - 13 November 2023 por
SlideShare ONS Economic Forum Slidepack - 13 November 2023SlideShare ONS Economic Forum Slidepack - 13 November 2023
SlideShare ONS Economic Forum Slidepack - 13 November 2023Office for National Statistics
66 vistas40 diapositivas
SlideShare ONS Economic Forum Slidepack - 16 October 2023 por
SlideShare ONS Economic Forum Slidepack - 16 October 2023SlideShare ONS Economic Forum Slidepack - 16 October 2023
SlideShare ONS Economic Forum Slidepack - 16 October 2023Office for National Statistics
7 vistas36 diapositivas
So what does ‘Beyond GDP’ mean for the UK – 12 October 2023 por
So what does ‘Beyond GDP’ mean for the UK – 12 October 2023So what does ‘Beyond GDP’ mean for the UK – 12 October 2023
So what does ‘Beyond GDP’ mean for the UK – 12 October 2023Office for National Statistics
158 vistas37 diapositivas
GDP after 2025: updating national accounts and balance of payments – 11 Octob... por
GDP after 2025: updating national accounts and balance of payments – 11 Octob...GDP after 2025: updating national accounts and balance of payments – 11 Octob...
GDP after 2025: updating national accounts and balance of payments – 11 Octob...Office for National Statistics
145 vistas66 diapositivas
SlideShare Measuring the Economy Slidepack - 29 September 2023 por
SlideShare Measuring the Economy Slidepack - 29 September 2023SlideShare Measuring the Economy Slidepack - 29 September 2023
SlideShare Measuring the Economy Slidepack - 29 September 2023Office for National Statistics
92 vistas23 diapositivas
Why dashboards? por
Why dashboards?Why dashboards?
Why dashboards?Office for National Statistics
7 vistas3 diapositivas

Más de Office for National Statistics(20)

Último

Strategic Planning & Managment por
Strategic Planning & ManagmentStrategic Planning & Managment
Strategic Planning & ManagmentJo Balucanag - Bitonio
5 vistas31 diapositivas
ecb.sp231121_1~8df317dc17.en.pdf por
ecb.sp231121_1~8df317dc17.en.pdfecb.sp231121_1~8df317dc17.en.pdf
ecb.sp231121_1~8df317dc17.en.pdfSociété Tripalio
411 vistas17 diapositivas
Dr Jean Paul Latyr FAYE - 2023 ReSAKSS Conference.pptx por
Dr Jean Paul Latyr FAYE - 2023 ReSAKSS Conference.pptxDr Jean Paul Latyr FAYE - 2023 ReSAKSS Conference.pptx
Dr Jean Paul Latyr FAYE - 2023 ReSAKSS Conference.pptxAKADEMIYA2063
8 vistas24 diapositivas
Taking care of the elders por
Taking care of the eldersTaking care of the elders
Taking care of the eldersSERUDS INDIA
6 vistas6 diapositivas
A terrorist threat originating from Qatar. por
A terrorist threat originating from Qatar.A terrorist threat originating from Qatar.
A terrorist threat originating from Qatar.fasurijobaf
7 vistas1 diapositiva
Support a Child Bright Future kurnool por
Support a Child Bright Future kurnoolSupport a Child Bright Future kurnool
Support a Child Bright Future kurnoolSERUDS INDIA
8 vistas3 diapositivas

Último(20)

Dr Jean Paul Latyr FAYE - 2023 ReSAKSS Conference.pptx por AKADEMIYA2063
Dr Jean Paul Latyr FAYE - 2023 ReSAKSS Conference.pptxDr Jean Paul Latyr FAYE - 2023 ReSAKSS Conference.pptx
Dr Jean Paul Latyr FAYE - 2023 ReSAKSS Conference.pptx
AKADEMIYA20638 vistas
Taking care of the elders por SERUDS INDIA
Taking care of the eldersTaking care of the elders
Taking care of the elders
SERUDS INDIA6 vistas
A terrorist threat originating from Qatar. por fasurijobaf
A terrorist threat originating from Qatar.A terrorist threat originating from Qatar.
A terrorist threat originating from Qatar.
fasurijobaf7 vistas
Support a Child Bright Future kurnool por SERUDS INDIA
Support a Child Bright Future kurnoolSupport a Child Bright Future kurnool
Support a Child Bright Future kurnool
SERUDS INDIA8 vistas
Dr Getaw Tadesse - 2023 ReSAKSS Conference .pptx por AKADEMIYA2063
Dr Getaw Tadesse - 2023 ReSAKSS Conference .pptxDr Getaw Tadesse - 2023 ReSAKSS Conference .pptx
Dr Getaw Tadesse - 2023 ReSAKSS Conference .pptx
AKADEMIYA20639 vistas
IEA Report: The Oil and Gas Industry in NetZero Transitions por Energy for One World
IEA Report: The Oil and Gas Industry in NetZero TransitionsIEA Report: The Oil and Gas Industry in NetZero Transitions
IEA Report: The Oil and Gas Industry in NetZero Transitions
Arrow Adoption Training for Kinship Families por ArrowMarketing
Arrow Adoption Training for Kinship FamiliesArrow Adoption Training for Kinship Families
Arrow Adoption Training for Kinship Families
ArrowMarketing40 vistas
ΕΚΘΕΣΗ ΚΟΜΙΣΙΟΝ ΓΙΑ ΤΟΥΡΚΙΑ por ssuser9e6212
ΕΚΘΕΣΗ ΚΟΜΙΣΙΟΝ ΓΙΑ ΤΟΥΡΚΙΑΕΚΘΕΣΗ ΚΟΜΙΣΙΟΝ ΓΙΑ ΤΟΥΡΚΙΑ
ΕΚΘΕΣΗ ΚΟΜΙΣΙΟΝ ΓΙΑ ΤΟΥΡΚΙΑ
ssuser9e6212167 vistas
Social behavioural change to drive community ownership_ Divyang Waghela_Tata ... por India Water Portal
Social behavioural change to drive community ownership_ Divyang Waghela_Tata ...Social behavioural change to drive community ownership_ Divyang Waghela_Tata ...
Social behavioural change to drive community ownership_ Divyang Waghela_Tata ...
Moving up into upper secondary by Hannah Kitchen - OECD Education Webinar 23N... por EduSkills OECD
Moving up into upper secondary by Hannah Kitchen - OECD Education Webinar 23N...Moving up into upper secondary by Hannah Kitchen - OECD Education Webinar 23N...
Moving up into upper secondary by Hannah Kitchen - OECD Education Webinar 23N...
EduSkills OECD82 vistas
Mrs. Tsitsi Makombe - 2023 ReSAKSS Conference por AKADEMIYA2063
Mrs. Tsitsi Makombe - 2023 ReSAKSS Conference Mrs. Tsitsi Makombe - 2023 ReSAKSS Conference
Mrs. Tsitsi Makombe - 2023 ReSAKSS Conference
AKADEMIYA20635 vistas
UNiTE- Invest to Prevent Violence against Women & Girls! por Christina Parmionova
UNiTE- Invest to Prevent Violence against Women & Girls!UNiTE- Invest to Prevent Violence against Women & Girls!
UNiTE- Invest to Prevent Violence against Women & Girls!
Dr. Ousmane Badiane - 2023 ReSAKSS Conference.pptx por AKADEMIYA2063
Dr. Ousmane Badiane - 2023 ReSAKSS Conference.pptxDr. Ousmane Badiane - 2023 ReSAKSS Conference.pptx
Dr. Ousmane Badiane - 2023 ReSAKSS Conference.pptx
AKADEMIYA20637 vistas
Taking care for elders por SERUDS INDIA
Taking care for eldersTaking care for elders
Taking care for elders
SERUDS INDIA11 vistas

Connecting to the StatXplore API in PowerBI

  • 1. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Connecting to the StatXplore API using Power BI Introduction The StatXplore API allows users to retrieve data as it is updated. The response sent by the API can appear challenging to process, but data can be successfully extracted given the consistent layout of the response. The response layout requires two queries; one to retrieve labels for each of the fields requested (such as date or geography), plus an additional query for the data itself. These queries must be linked to use the labels alongside the data. This guide is split into two main sections: how to query the API and how to process the response. The Appendix contains examples of query bodies, a query template, and a full query example (excluding API Key). The method described in this guide was initially created with assistance from colleagues at IJYI.
  • 2. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Querying the StatXplore API Prerequisites To query the StatXplore API, you will need: a. A personal API Key. This is obtained by registering on the StatXplore website. Always use your personal API Key. If the user assigned to your API Key deregisters from StatXplore, your queries will not function. Once registered, your API Key can be found in the Account section of the website. Use the link in the top right once registered and logged in: b. The body of the query to send to the API. The easiest way to construct this is to use the StatXplore website to build the table of data you need, then export it as an Open Data API query (.json). Save this file so that you can refer to it later. This export will need further editing before use in a query, see Preparing the Query Body. c. A program to prepare the query body. Notepad++ is recommended, although alternative software can be used. d. A template to put your query body into. An example template can be found in the Appendix of this document.
  • 3. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Preparing the Query Body The .json file that is exported from StatXplore (see Prerequisites) must be edited before use. This needs to be done for each query that you use. See the finished Example query body (1) and Example query body (2) in the Appendix. 1. Open the .json file exported from StatXplore in step B of the Prerequisites in Notepad++. 2. Remove the text relating to date fields from the recodes section, but not from the dimensions section. This is done so that values from all dates are returned, not only those dates specified on the StatXplore website. Without this step, queries will not return new data. Remove the text from and including the comma preceding the date field name, up to and including the right parenthesis after the ‘total’ line of the date field (see examples below where the highlighted text should be deleted). 3. Remove the text relating to any other fields for which you want to return all available data rather than only the values initially specified. Use the same technique as above: remove the text from and including the comma preceding the field name, up to and including the right parenthesis after the ‘total’ line of the field. This could be used, for example, to see values for all the employment indicators for Alternative Claimant Count.
  • 4. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Remove the text relating to these fields from the recodes section, but not the dimensions section. Removing text from the dimensions section would remove this field from the query entirely. 4. Replace every instance of a speech mark (") with a double speech mark (""). Note that this must be the two vertical lines; using the quotation marks in Word (“ and ”) will not be recognised and will cause the query to fail.
  • 5. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Creating a Query 1. In Power BI desktop, add a new data source and use the Blank Query option. 2. With this new blank query created, open the Advanced Editor. Select everything in the box, delete it, then paste in the Query Template. 3. The query template creates the variables url, body and header: • url is the web address used for any query that aims to bring back data from the StatXplore API. This does not need to change. • body is the part of the query that tells the API which data to return. Paste your prepared query body between the speech marks in the body line. • header is where your personal API Key is added to authenticate access to the API. Paste your API key between the speech marks in the header line. The variable response uses the url, body and header variables as inputs to the Web.Contents function. response is returned as json using the Json.Document function. Click Done and wait for the response from the API. If your query has worked, you should see this:
  • 6. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk If so, you can now access and transform the data. If asked for credentials for this data source, select Anonymous credentials and Public privacy level.
  • 7. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Accessing and Transforming Data Overview The response from the API should look like this: The data returned by the query comes from the fields and cubes rows of this table. fields contains the labels for your dataset such as geography, date and sex. cubes contains the values of your dataset. The yellow words List and Record act as links and can be used to navigate through the json document. By default, there are no labels assigned to the values in the cubes section. The data is presented in a hierarchy, but there are no labels assigned to each level in the hierarchy. Fortunately, the response sent by the API will always follow a set of rules. The rules of the API response are: a. The hierarchy of data within the cubes is always in the same order as the List of fields. b. The labels for each field are always in the same order as the List of records within a field. These rules mean that, if expanded correctly, a flat table can be created with each combination of labels in the order that they occur in the data. The general process for this method is: 1. Transform the list of fields into a single row. 2. Expand each column, working from left to right, to obtain the labels. Each expansion will create new rows for each combination of existing labels and the newly expanded labels. 3. In a duplicate query, expand the list of values to new rows. Continue expanding to new rows until the data appears. 4. The order of rows in the labels and values tables is identical. Assign each row in both tables an index value to allow the tables to be merged or linked.
  • 8. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk 5. Merge the values table into the label table using the index columns and expand out the values column into the labels table. Alternatively, create a relationship between the index columns in the modelling view and use both tables. Worked Example: Households on Universal Credit To show more effectively how to transform the data, I have included a worked example of a connection to the database for Households on Universal Credit. This uses the example query in the Appendix. The step numbers correspond to the numbers from the Overview. Step 1: Transform the list of fields into a single row Begin from the response provided by the API (see Creating a Query). Click on the List in the fields row. From the Transform menu, select Convert To Table. This allows the Records to be manipulated. Expand the column of Records to get a List for the labels for each field (there are two in this example, area and date). Click the Expand button at the top right of the column. Select only the items checkbox, then click OK.
  • 9. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk The Lists need to be switched from rows to columns. Use the Transpose option from the Transform menu. There is now a column for each field in the query.
  • 10. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Step 2: Expand the labels table to obtain labels for each row Each column should now be expanded to obtain the labels for each field. The rows will be created in the order that they will appear in the data. As each column is expanded, new rows are added for each existing row. Click the Expand button at the top-right of the first column. Select Expand to New Rows. This will create a new row for each of the labels in this field. Then click the Expand button again and select only the Labels checkbox. Then click the Expand button for the third time and select Expand to New Rows.
  • 11. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk This has added to labels for the first field. Now, return to the start of this step and repeat the process for all the field columns, working from left to right, until the rightmost column has labels rather than Lists. When expanding each column to new rows for the first time, a new row will be created for each label within the new row, for each existing row. In this example, the expansion of the second column results in 76 rows for each of the existing rows (Babergh, East Suffolk et cetera) because the second field (month) has 76 labels. Once all the columns have been expanded and the labels are shown, the labels table expansion is complete. Each row will be a unique combination of labels which will appear in the same order as the values table. Rename the columns and change their types accordingly.
  • 12. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Step 3: Expand the values table Duplicate the previous query and delete all steps up to and including the ‘fields’ step. Navigate to the Record in the Cubes row. Then select the Record within that, then the List within that. This should display a List of Lists.
  • 13. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Next, convert this List to a table from the List Tools menu; click OK in the dialog box that appears. The table should now display one column with a List in each row. Each List must be expanded to get the data values. Click the expand button at the top-right of the column and select Expand to new rows. Repeat this until the data is displayed. The data is now displayed with a row for each value. These rows correspond exactly to the order of the rows in the labels table. Rename the values column and change its type accordingly.
  • 14. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Step 4: Create index columns The labels and values tables are now complete and ready to be linked. For both queries, add an index column using the button in the Add Column tab. Use the same settings for the index columns for both queries. The tables can now be linked using modelling (relationships) or transformations (merging queries).
  • 15. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Step 5a: Linking by merging tables To use the merging queries approach, open Transform data and select the labels table. From the Combine section of the Home tab, select Merge Queries. Select the index column from the labels table. Then select the values table from the first drop- down box and select the index column. A Left Outer join will be required as the aim is to add the value column from the values table to the labels table. All rows should match, if not there has been an error in the expansion of the tables or the indexes have been created using different settings. Click OK.
  • 16. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Click the expand button in the top-right of the new column. Expand out only the values column. This will add the values to the labels table, resulting in one table that can be used to filter and display data as required.
  • 17. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Step 5b: Linking by relationships To use the modelling approach, load the data and switch to the Model view. Click and drag the tables you will be using so that they are close together. Click and drag the index column from one table to the other. This will automatically create a relationship between these columns. The relationship should always be of the type 1:1. This will be automatically calculated. If the relationship is not of this type, there has been an error in the expansion of the tables or the indexes have been created using different settings. To make the report easier to use, hide the index columns from the report view using the eye icon. The fields from the labels table can now be used to filter and display the data in the values table.
  • 18. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Appendix Example query body (1) { ""database"" : ""str:database:UC_Households"", ""measures"" : [ ""str:count:UC_Households:V_F_UC_HOUSEHOLDS"" ], ""recodes"" : { ""str:field:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE"" : { ""map"" : [ [ ""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R EGION:E07000200"" ], [ ""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R EGION:E07000244"" ], [ ""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R EGION:E07000202"" ], [ ""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R EGION:E07000203"" ], [ ""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R EGION:E07000245"" ] ], ""total"" : true } }, ""dimensions"" : [ [ ""str:field:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE"" ], [ ""str:field:UC_Households:F_UC_DATE:DATE_NAME"" ] ] }
  • 19. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Example query body (2) { ""database"" : ""str:database:ACC"", ""measures"" : [ ""str:count:ACC:V_F_ACC"" ], ""recodes"" : { ""str:field:ACC:V_F_ACC:UK_COA"" : { ""map"" : [ [ ""str:value:ACC:V_F_ACC:UK_COA:V_C_MASTERGEOG11_LA_TO_REGION_NI:E07000200"" ], [ ""str:value:ACC:V_F_ACC:UK_COA:V_C_MASTERGEOG11_LA_TO_REGION_NI:E07000244"" ], [ ""str:value:ACC:V_F_ACC:UK_COA:V_C_MASTERGEOG11_LA_TO_REGION_NI:E07000202"" ], [ ""str:value:ACC:V_F_ACC:UK_COA:V_C_MASTERGEOG11_LA_TO_REGION_NI:E07000203"" ], [ ""str:value:ACC:V_F_ACC:UK_COA:V_C_MASTERGEOG11_LA_TO_REGION_NI:E07000245"" ] ], ""total"" : true } }, ""dimensions"" : [ [ ""str:field:ACC:V_F_ACC:UK_COA"" ], [ ""str:field:ACC:F_ACC_DATE_new:DATE_NAME"" ], [ ""str:field:ACC:V_F_ACC:EMP"" ] ] }
  • 20. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Full example query For privacy reasons the API key has been omitted. Copy and paste your personal API key between the speech marks after APIKey= to get this query to work. let url = "https://stat-xplore.dwp.gov.uk/webapi/rest/v1/table", body = "{ ""database"" : ""str:database:UC_Households"", ""measures"" : [ ""str:count:UC_Households:V_F_UC_HOUSEHOLDS"" ], ""recodes"" : { ""str:field:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE"" : { ""map"" : [ [ ""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R EGION:E07000200"" ], [ ""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R EGION:E07000244"" ], [ ""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R EGION:E07000202"" ], [ ""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R EGION:E07000203"" ], [ ""str:value:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE:V_C_MASTERGEOG11_LA_TO_R EGION:E07000245"" ] ], ""total"" : true } }, ""dimensions"" : [ [ ""str:field:UC_Households:V_F_UC_HOUSEHOLDS:COA_CODE"" ], [ ""str:field:UC_Households:F_UC_DATE:DATE_NAME"" ] ] }", header = [ #"APIKey"="" ], response = Web.Contents(url,[Content=Text.ToBinary(body),Headers=header]), Source = Json.Document(response,1252) in
  • 21. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Source
  • 22. Public Health & Communities Analytics | Suffolk County Council | Version 3 (September 2023) phc_analytics@suffolk.gov.uk Query template let url = "https://stat-xplore.dwp.gov.uk/webapi/rest/v1/table", body = "", header = [ #"APIKey"="" ], response = Web.Contents(url,[Content=Text.ToBinary(body),Headers=header]), Source = Json.Document(response,1252) in Source