Audit and Evaluation Branches
Natural Resources Canada and Statistics Canada
March 24, 2025
On this page
- List of acronyms and abbreviations
- Executive summary
- Introduction
- Relevance
- Effectiveness – Outcomes
- Effectiveness – Design and Delivery
- Lessons learned
- Conclusion
- Appendix A: Interviewee rating of achievement of CCEI outcomes
- Appendix B: Evaluation Team
List of acronyms and abbreviations
- ACS
- Assistant Chief Statistician
- ADM
- Assistant Deputy Minister
- CCEI
- Canadian Centre for Energy Information
- CER
- Canada Energy Regulator
- CSAC
- Canadian Statistics Advisory Council
- DSA
- Data Sharing Agreement
- EAC
- External Advisory Committee
- ECCC
- Environment and Climate Change Canada
- ECCP
- Energy and Climate Change Policy
- EDI
- Equity Diversity, and Inclusion
- EIA
- Energy Information Administration
- ESP
- Energy Statistics Program
- FPT
- Federal, Provincial, and Territorial
- GBA Plus
- Gender Based Analysis Plus
- GHG
- Greenhouse Gas
- HFE
- High Frequency Electricity Data
- IEA
- International Energy Agency
- IT
- Information Technology
- NRCan
- Natural Resources Canada
- Ofgem
- Office of Gas and Electricity Markets
- OGD
- Other Government Department
- StatCan
- Statistics Canada
- UK
- United Kingdom
- UN
- United Nations
- USA
- United States of America
Executive summary
This report presents the findings, conclusions, and recommendations from the Interdepartmental Evaluation of the Canadian Centre for Energy Information (CCEI) led by the Audit and Evaluation Branch of Natural Resources Canada (NRCan) with the support of the Audit and Evaluation Branch of Statistics Canada (StatCan).
This interdepartmental evaluation was approved in both NRCan’s 2022-27 Integrated Audit and Evaluation Plan and StatCan’s Integrated Risk-based Audit and Evaluation Plan 2022-23 to 2026-27. The evaluation methodology followed the Treasury Board Policy on Results (2016) and consisted of document and literature review, key informant interviews, and an international comparison. The evaluation period was 2018-19 to 2023-24. The CCEI is constantly evolving and as such, progress on outcomes may have been made following the evaluation period. Examples include the release of the .Stat Data Explorer (November 2023) and the High Frequency Electricity Data (HFED) Initiative (February 2024), whose impacts may not have been captured in the evaluation findings.
The CCEI is a partnership between NRCan and StatCan, in collaboration with Environment and Climate Change Canada (ECCC), and the Canada Energy Regulator to create a modern, independent energy information system. The CCEI aims to serve as a ‘one-stop shop’ for detailed regional and national energy information that is accurate, timely, transparent, comprehensive, user-friendly, internally consistent, free of charge, responsive to the needs of different users, and independent of political influence.
Housing the operational centre of CCEI within StatCan was intended to address recommendations from the 2018 report, Rethinking Canada’s Energy Information System: Collaborative Models in a Data-Driven Economy, from the House of Commons Standing Committee on Natural Resources. It included government working with industry, civil society, research institutions, Indigenous governments and communities, and provincial/territorial governments to create a provider of energy information and statistics as part of a collaborative model in a data-driven economy.
What the evaluation found
Relevance
The CCEI is relevant. The evaluation found that the need for CCEI is increasing given the importance of Canada’s energy transition. As an initiative, the CCEI is well-positioned to support Canada’s Net-zero Emissions Strategy, low carbon fuels initiatives, and open data strategies among other initiatives, in a way that is transparent and accountable. However, professional data users, referred to by the CCEI as ‘power-users,’ shared the perception that relevant energy data, including those related to Canada’s greenhouse gas (GHG) emissions, were not sufficiently granular or timely. While acknowledging this perception, program managers indicated that, regarding GHG emissions, the most comprehensive set of timely and granular data published by ECCC and StatCan was already available via the CCEI at the time of interviews. The evaluation noted that satisfying the need for more granular and timely data would involve broader systemic changes to Canada’s overall system of energy information and significant resourcing. Ongoing improvements in Canada’s overall system of statistical energy data, including the CCEI, are necessary to facilitate better decision making by policy makers, industry, researchers, and Canadians.
Effectiveness
The CCEI met its immediate outcomes. Canada’s new energy information system is relevant and collaborative and has made it easier for information users to find data and analyses to make informed decisions. However, there is still room to better leverage the value of federal family collaboration resulting from the CCEI. While oversight and reporting have continued largely unchanged since 2019, there was a change to a direct funding model and the operational portion of the program was mainstreamed into StatCan in 2021-22. Due to other operational changes over time such as CCEI’s engagement approach with stakeholders, and evolved information technology infrastructure, the evaluation identified a need for the CCEI to review and update its governance and performance measurement frameworks.
CCEI user visits have increased by at least 50% year on year since its establishment. Overall, web access numbers are variable, with approximately a few thousand page views per month, and approximately 50,000 page views per year. Further advancement of data visualizations or integration of narrative website elements (for example, use of text, graphs, story maps, etc.) may help to make CCEI information easier to absorb for both senior leaders and Canadians, which could aid in increasing awareness of the CCEI, and enhance the CCEI’s ability to communicate useful information about Canada’s energy systems. NRCan’s Energy Fact Book was found to be a useful model containing map data, graphs, and graphics. Exclusive posting of the Fact Book to the CCEI portal (formerly it was on the NRCan website) was associated with a significant spike in user traffic.
Beyond stakeholder perspectives, there is limited data available to assess the CCEI’s achievement of its intermediate and ultimate outcomes. These intermediate outcomes relate to energy information users basing their decisions on more relevant, consistent, and complete information, and increased trust in the Government of Canada’s energy data. Ultimate outcomes relate to use of information, advice, and stakeholder perspectives by decision makers in support of energy initiatives and lead to improved sustainability and competitiveness of Canada’s energy sector.
Stakeholder perspectives of progress against the intermediate and ultimate outcomes were mixed. While most felt that progress was being made on these outcomes and on data harmonization, several stated that the issues around the intermediate to ultimate outcomes are complex, and without public opinion research directly demonstrating outcome related data, they could not comment on progress attributable to the CCEI. Interviewees stated their positive view of StatCan as a trusted and politically neutral organization. There was a view by some that the CCEI was performing a ‘library’ function and not fully demonstrating federal leadership on improving the quality and timeliness of energy data in Canada.
There was an overall sentiment from almost all interviewees across stakeholder groups that the goal of the CCEI was too ambitious for the level of resources the program received, and that the CCEI had “accomplished a lot” with its available resources. CCEI program team members have stated that they often struggle to find a balance between addressing priorities based on what should be done to address stakeholder needs, and what can be done in practice given constraints on existing resources.
Lessons learned
A range of federal, provincial and territorial (FPT) and private sector partners mentioned the valuable collaboration that was occurring. The CCEI as a venue for collaboration and the CCEI itself for bringing energy information into a ‘one-stop shop’ were seen as best practices. However, some stakeholders noted that collaboration and communication with the CCEI was sometimes slow and intermittent. Securing meaningful provincial and territorial engagement was noted as a particularly challenging area. CCEI program team members noted it was often difficult to get feedback or explicit endorsement from senior officials. Interviewees also identified a need for senior leaders to communicate priorities that could be operationally actioned by the CCEI.
Feedback from stakeholders also indicated that the need to consult various governance bodies such as the External Advisory Committee and the Energy Mines and Ministers Conference were initially very useful but may now be ‘top-heavy’ processes. Stakeholders expressed interest in more opportunities to participate at the working level, with agendas and the timing of meetings geared to the needs of the involved parties at the level of energy data professionals.
CCEI program team members indicated that the CCEI was successful as a ‘pathfinder’ project towards a more inclusive and dynamic delivery model. The CCEI’s influence helped to create a change in the corporate culture of StatCan, which was previously heavily invested in producing data tables but largely left additional analysis or graphic representations of the data to other parties. Energy data professionals within academia and research institutes expect that the CCEI and the broader federal government should be able to match their level of presenting useful visualizations, which would make it easier to communicate quickly with senior leaders and a broader audience.
Recommendations and management response and action plan
In light of these findings, the following recommendations are proposed, along with the management response and action plan:
| Recommendations | Management Response and Action Plan |
|---|---|
|
Management Response: Management agrees. In response to Recommendation 1, working collaboratively, NRCan and StatCan will undertake medium- to long-term planning for the CCEI initiative, in consultation with FPT partners, while identifying opportunities to improve the collaborative work planning and prioritization process. While comparisons to the United States Energy Information Administration (EIA) are understandable, significant changes to CCEI’s mandate, budget, and legal authority would be required to match the standard of the EIA. Considering this, management will develop a five-year work plan with consistent year-to-year tracking of deliverables to prioritize improvements that can realistically be addressed by the CCEI within current funding levels. The expertise of FPT partners will be leveraged to identify best practices and support future work. Management agrees to explore the following in developing the work plan:
The five-year plan will build on recently launched foundational tools, including the .Stat Data Explorer and HFED visualization tool. The Data Explorer, launched in November 2024, will facilitate the publication of FPT data not currently on the CCEI website. CCEI will also continue to explore opportunities to expand the HFED visualization tool beyond publicly available data to address the needs of power-users producing complex visualizations and regionally based outputs. Position responsible: Director General (DG), Energy Policy Branch (EPB) and DG, Agriculture, Energy, Environment Statistics Branch (AEESB) Deliverables and timeline: A five-year work plan that consolidates consistent year to year tracking of progress developed in consultation with FPT partners and key stakeholders, and approved by the ADM, Energy Systems Sector, NRCan, and the ACS, Economic Statistics, StatCan (by the end of September 2026). |
|
Management Response: Management agrees. In response to Recommendation 2, working collaboratively, NRCan and StatCan, in consultation with other federal partner departments and provincial/territorial (PT) counterparts, will re-examine the CCEI’s governance in light of the strategic direction of the CCEI. Existing governance structures will be reviewed, and best practices will be appraised. Consideration will be given to maximizing the use of existing governance mechanisms or other meeting fora, balancing senior-level strategic engagement and working level discussions, with a view to leveraging subject matter expertise and optimizing participation/collaboration amongst federal departments and agencies. Management agrees to:
Position responsible: DG-EPB and DG-AEESB Deliverables and timeline: An updated governance structure with terms of reference for relevant committees, approved by the ADM Energy Systems Sector, NRCan and the ACS, Economic Statistics, StatCan (by the end of September 2026). |
|
Management Response: Management agrees. In response to Recommendation 3, working collaboratively, StatCan and NRCan will update the performance measurement framework for the CCEI initiative, aligned with Government of Canada policies and directives, including the Policy on Results and Directive on Results, and in consultation with StatCan and NRCan performance measurement groups. This will include selecting key performance indicators to effectively measure progress against outcomes, identifying data requirements and sources of data, and setting targets and timelines for achievement of performance measures. The framework will be reviewed on a regular basis to maintain robust performance measurement. Management agrees to:
Position responsible: DG-EPB and DG-AEESB Deliverables and timeline: A revised performance measurement framework with updated key performance indicators, approved by the ADM, Energy Systems Sector, NRCan, and the ACS Economic Statistics, StatCan (by end of March 2026). |
Introduction
This report presents the findings, conclusions, and recommendations from the Interdepartmental Evaluation of the Canadian Centre for Energy Information (CCEI) led by the Audit and Evaluation Branch of Natural Resource Canada (NRCan) with the support of the Audit and Evaluation Branch of Statistics Canada (StatCan).
In line with the commitment made by NRCan to the Treasury Board to evaluate the CCEI in collaboration with StatCan, conduct of this interdepartmental evaluation was approved in both the 2022-27 NRCan Integrated Audit and Evaluation Plan, and the StatCan Integrated Risk-based Audit and Evaluation Plan 2022-23 to 2026-27.
Program profile
Program context and rationale
The CCEI is a partnership between NRCan and StatCan, in collaboration with Environment and Climate Change Canada (ECCC), and the Canada Energy Regulator (CER) to create a modern, independent energy information system. Housing the operational centre of the CCEI within StatCan was intended to address recommendations from the 2018 report, Rethinking Canada’s Energy Information System: Collaborative Models in a Data-Driven Economy, from the House of Commons Standing Committee on Natural Resources that noted gaps in Canadian energy information.
The CCEI does not produce or originate raw data; it disseminates energy and climate change relevant data and information from NRCan, CER, and ECCC. While NRCan and ECCC are ministerial departments, StatCan, as the national statistics agency, and CER, as the federal energy regulator, are structured differently as departmental agencies, and by their enacting legislation, operate with a greater degree of independence.
The CCEI responds to gaps in Canada’s energy information system. Specifically, as noted in the Committee report, Canada’s system of energy information has been described as: decentralized and difficult to navigate, interpret, and verify; based on inconsistent data measurement and reporting standards; frequently out of date in a fast-paced economy with ever growing demand for real-time information; and often inaccessible, incomplete, or lacking.
To address the Natural Resources Committee report, in Budget 2019, the Government of Canada announced that it would provide NRCan with $15.2 million over five years, and $3.4 million per year ongoing, to establish the virtual CCEI delivered by StatCan. The Minister of Natural Resources announced StatCan’s launch of the CCEI portal in October of 2020.
As described in NRCan’s Evaluation of the Energy and Climate Change Policy (ECCP) Program (December 2020), “In the past several years, NRCan and its federal partners have been unable to answer important energy questions due to insufficient access to quality data and analysis (including raw and interpreted statistics).” This has negatively affected the Government’s ability to track emerging trends and evaluate economic, social and environmental impacts of moving to a clean energy future, to identify gaps and market failures that need to be addressed by governments, and to assess Canada’s position in an international context.
While the responsibility for utilization of the CCEI’s funding was later transferred to StatCan, the existing oversight and reporting has continued under the governance already in place, coordinated by NRCan.
The CCEI aims to serve as a ‘one-stop shop’ for detailed regional and national energy information that is accurate, timely, transparent, comprehensive, user-friendly, internally consistent, free of charge, responsive to the needs of different users, and independent of political influence.
The CCEI aims to address gaps in energy data and data quality and increase the consistency and coherence across existing data by standardizing energy definitions, measurements, and reporting across federal partners (i.e. NRCan, StatCan, ECCC, and CER) and provinces and territories (PT). The four federal departments and agencies have roles and responsibilities aligned with their mandate:
- NRCan collects and disseminates information to support its mandate to enhance the responsible development, use and competitiveness of natural resources;
- StatCan is the primary collector, compiler, and distributor of data and statistics in Canada, including energy data;
- ECCC produces information regarding the impact of environmental regulations on the energy sector as well as the energy sector’s impact on the environment; and
- The CER, as the federal regulator for pipelines, energy development and trade, collects and disseminates energy trade data, as well as production and pipeline throughput data.
Operational context and mandate
NRCan acts as the policy lead for the CCEI, and its activities contribute to the ECCP Program, managed by the Energy Systems Sector. The ECCP Program provides policy advice to the Minister of Natural Resources, senior government officials, and other stakeholders to inform decision-making on matters related to energy and climate, including the future pathway for Canada’s energy sector.
Within StatCan, the CCEI is operationalized by the Energy Statistics Program (ESP) which is part of the Centre for Energy and Transportation Statistics, a division under the Agriculture, Energy and Environment Statistics Branch of the Economic Statistics Field. The ESP, which has been in place at StatCan for many years, collects and disseminates primarily volumetric data on the production, distribution, transformation, and consumption of the various forms of energy (e.g., oil, natural gas, coal and coke, electricity, and refined petroleum products) for users across the country as well as meeting Canada’s international reporting obligations.
In delivering the CCEI, StatCan, as an independent agency, is an arms-length arbitrator of energy statistics. There is no direct ministerial involvement in methodological or technical issues.
CCEI’s mandate is to:
- compile energy data from several sources into a single easy-to-use website (i.e., centralized, accessible); and
- improve the overall quality of energy information available to Canadians, decision-makers, stakeholders, and industry (i.e., data quality, data quantity, availability).
The CCEI has three main lines of work:
- Access: Simplifying access to Canada’s energy information through development, launch, and maintenance of a virtual data centre that integrates data from a range of providers and makes it publicly available through a user-friendly website with advanced functionalities.
- Data: Working with partners to identify and mitigate data gaps (including through new data collection), standardize energy definitions, and reconcile data from various providers, and reduce overlap in data collection (e.g., through more specialized surveys).
- Engagement: Strategic direction in improving the state of Canada’s energy information is guided by structured engagement with stakeholders. This engagement draws on existing structures without requiring incremental resources.
The 2018 Natural Resources Committee report made recommendations that included government working with industry, civil society, research institutions, Indigenous governments and communities, and PT governments to:
- ensure that the proposed energy information provider is politically independent and has sufficient legislative power to collect, validate, analyse and distribute energy data under competitive timelines;
- establish sufficient safeguards to protect the sensitivity and/or confidentiality of energy data reported by the public, private companies and other organizations; and
- incorporate best practices from international counterparts, where appropriate.
While the Committee report notes that the above goals could be achieved by an entirely new energy information agency that would function independently of government, evidence collected by the Committee also references that setting up a new agency would likely be costly. The report also notes that the goals it expressed could also be accomplished through existing national data providers, such as StatCan. As announced in Budget 2019, the CCEI as an NRCan/StatCan partnership, operationalized by StatCan, was ultimately selected by the Government of Canada as the model to best address the report’s recommendations.
Governance
The CCEI governance structure is designed to enhance federal, provincial and territorial (FPT) collaboration and leverage existing structures. Figure 1 represents the original governance structure envisioned. However, the structure has changed over time with the inclusion of the Energy Steering Group and Consultative Council on Statistics and its subcommittees.
- The FPT Steering Committee is co-chaired by the Deputy Minister of NRCan and the Chief Statistician of StatCan and is responsible for identifying energy priorities and providing guidance on policy priorities for the annual plan of the CCEI.
- The FPT Steering Committee is supported by a federal Director General-level Implementation Committee that provides technical and operational expertise as well as updates on implementation of the CCEI.
- The External Advisory Body, now known as the External Advisory Committee (EAC), provides input and advice to the Steering Committee. It includes six to 10 external experts from academia, think-tanks, industry organizations and other non-governmental organizations, and municipalities. The EAC contributes to the continuous review of the CCEI’s statistical outputs, fosters program relevance, recommends priorities, and meets biannually.
- The Canadian Statistics Advisory Council (CSAC) is an independent body that advises the Chief Statistician in a transparent manner on the overall quality of the national statistical system. The CSAC is also responsible to produce and release an annual report on the state of this system.
- Through its existing Indigenous capacity building initiative, StatCan is developing focused Indigenous engagement to involve Indigenous Peoples as contributors and users of the CCEI.
- The FPT Consultative Council on Statistical Policy and its subcommittees comprise a network of 13 provincial/territorial official representatives of the FPT Technical Working Group, who collaborate with StatCan to determine data requirements, consult on current statistical activities, and coordinate the dissemination of StatCan's products to provincial and territorial governments.
Figure 1. Initial CCEI governance structure
Source: CCEI Overview Deck, June 2019.
Text version
CCEI Governance Diagram
Figure one. Initial CCEI Governance Structure. The diagram shows the initial Canadian Centre for Energy Information governance entities in boxes, and their relationship to the other entities. The Canadian Centre for Energy Information (housed at Statistics Canada) sits in the operations area, near the middle of the diagram. Below, the Federal Provincial and Territorial Technical Working Group (Cochairs Director-level representatives from NRCan and StatCan) also sits in the operations area. A line of five boxes sits above in the priority-setting area. The Federal Provincial and Territorial Steering Committee (co-chairs Deputy Minister NRCan and Chief Statistician) is supported by the Energy Steering Group. An arrow leads to this box from the adjacent External Advisory body. Another box contains Indigenous engagement, and the final box, the Canadian Statistics Advisory Council. The upper priority-setting entities are connected by arrows down to the operational level of the Canadian Centre for Energy Information. Source: Canadian Centre for Energy Information Overview Deck, June 2019.
Expected results
The original logic model for the CCEI is illustrated below as Figure 2. Stream two of the original logic model was discontinued due to resource constraints.
Figure 2. CCEI expected results stated in foundational documents
| Ultimate results | |||
|---|---|---|---|
| This program connects to NRCAN’s Energy and Climate Change Policy Program (ECCP) logic model’s intermediate outcome of “use of information, advice, and stakeholder perspectives by decision makers in support of energy initiatives that promote public confidence and clean growth,” which in turn leads to the ECCP Logic Model ultimate outcome of “improved sustainability and competitiveness of Canada’s energy sector.” | |||
| Intermediate results | |||
|
Energy information users base their decisions on more relevant, consistent, and complete information. Energy information users have increased trust in the Government’s energy data and analysis. |
Energy information users have increased trust in the Government’s energy data and analysis. Establishment of new industries and supply chains. |
||
| Immediate results | |||
| Stream 1 | Stream 2 | Stream 3 | |
| Energy information users can more easily find the data and analysis they need to make informed decisions | New analyses inform government decision-making and public debate on energy policy issues | Canada’s energy information system remains relevant and collaborative. | |
|
Output: User-friendly energy information hub with more comprehensive, timely data. Activity: Creating an energy information hub; investing in energy data. |
Output: Production of new analyses/reports. Activity: Expanding analytical capacity through hiring, training, and investing in new modeling capabilities. |
Output: Venue for joint decision-making; energy information hub and models integrate comprehensive FPT information. Activity: Launching FPT working table to bolster collaboration, including the establishment of improved data sharing procedures |
|
Logic model
Figure 3 shows an updated Logic Model that was used for this evaluation. The evaluation team worked with program staff to update the original Logic Model for the purposes of this evaluation as activities related to Stream 2 were removed from its mandate. More recent changes made to the intermediate and ultimate outcomes of the ECCP Program are not reflected here and are not expected to significantly impact the findings or recommendations of this evaluation.
Figure 3. Logic model
Access to data and energy info hub
- Input
Human and financial resources - Activity
Creating an energy information hub; investing in energy data - Outputs
User-friendly energy information hub with more comprehensive, timely data - Immediate results
Energy information users can more easily find the data they need to make informed decisions
- Intermediate results (i.e., cognitive and behavioural change)
Energy information users (the public, government, academics, think-tanks, industry, Indigenous groups):- based their decisions on more relevant, consistent, and complete information
- Have increased trust in the Government's energy data
- Ultimate Results (i.e., societal change)
Connects to ECCP Logic Model intermediate outcome of “use of information, advice, and stakeholder perspectives by decision makers in support of energy initiatives that promote public confidence and clean growth”, which in turn leads to the ECCP Logic Model ultimate outcome of “improved sustainability and competitiveness of Canada’s energy sector”
Joint governance
- Input
Human and financial resources - Activity
Launching FPT working table to bolster collaboration, including the establishment of improved data sharing procedures - Outputs
Venue for joint decision-making; energy information hub and models integrate comprehensive FPT information - Immediate results
Canada’s energy information system remains relevant and collaborative
About the evaluation
Objective, scope and approach
The evaluation objective was to examine the relevance and effectiveness of the CCEI. The scope covered the period from fiscal years 2018-19 to 2023-24 and focused on the following evaluation questions:
Figure 4. Evaluation questions
| Relevance |
|
| Effectiveness |
Immediate Outcomes
|
Intermediate Outcomes
Ultimate Outcomes
|
|
The approach and methodology used for the evaluation followed the Treasury Board Policy on Results (2016) and related Standards on Evaluation. It was also presented, reviewed, and endorsed by the Interdepartmental Evaluation Working Group.
The evaluation used the lines of evidence outlined in Figure 5 for data collection and analysis to address the issues of relevance and effectiveness. Data was mostly collected during the evaluation’s conduct phase FY 2023-2024. The .Stat Data Explorer launched in November 2023 and High Frequency Electricity Data was released in February 2024. Both were launched within the period of the evaluation; however, these new updates were too recent for stakeholders to consider in their feedback to the evaluation.
Figure 5. Lines of Evidence
Document and Literature Review
Foundational and strategic documents, project files (e.g., performance data), and reports on program results have been reviewed to answer evaluation questions related to relevance and effectiveness.
Key Informants Interviews (n=32)
Key informant interviews were used to answer evaluation questions pertaining to relevance (including evolving needs and priorities of stakeholders, as well as NRCan’s and StatCan’s continued relevance in this sector), and effectiveness (including perspectives on program design, delivery, implementation, and responsiveness to EDI factors, and the extent to which the CCEI is on-track towards the longer-term outcomes).
International Comparison
Comparison of Canada’s approach to the production and presentation of energy information with similar initiatives in the United States of America, European Union, United Kingdom, Australia, and New Zealand.
Limitations
The evaluation was designed to collect data using multiple lines of evidence to enhance the reliability of results and validity of findings. The following limitations should be considered when reviewing the evaluation findings:
- While all were invited, some PTs declined to participate in interviews. Those that chose to participate in interviews included British Columbia, Prince Edward Island, Newfoundland and Labrador, Nova Scotia, Quebec, and Yukon. This may have limited the evaluation’s effectiveness in collecting representative views on the main issues and improvements required related to the non-participating stakeholders. However, this limitation was mitigated by the document review which included the review of meeting notes and opinion-based evidence of CCEI program team members who had engaged with representatives from those parties, to give a generalized or indirect view of their perspectives.
- Although Indigenous stakeholders were invited to participate in the interview process, individuals representing Indigenous communities or organizations chose not to be interviewed. The evaluation was thus limited to secondary sources of information on Indigenous involvement rather than first-hand accounts that could reflect Indigenous voices.
- The documentation provided did not cover all performance indicators and measures in line with evaluation expectations; therefore, some of the evaluation’s analysis had to rely on proxy indicators (e.g. web access as a proxy for stakeholder awareness and use of the CCEI).
- While StatCan has conducted surveys of CCEI users, the evaluation did not conduct any public opinion research to obtain the views of Canada’s general public. All non-CCEI program interviewees regularly worked with energy data and information as part of their occupation and/or research, and their opinion-based evidence cannot be generalized to the Canadian population as a whole.
- Energy information data initiatives reviewed in the international comparison were limited to nations and government bodies recommended by interviewees.
- More than half of CCEI program team members interviewed were unaware of how diversity, inclusion, and accessibility are considered by the CCEI. Rather than a comment on the CCEI, this perhaps points to a relatively low awareness of how federal initiatives must show alignment to federal Gender Based Analysis Plus (GBA Plus) and Equity Diversity and Inclusion (EDI) requirements. This limited the evaluation’s ability to highlight CCEI’s positive GBA Plus and EDI contributions while also limiting the evaluation’s ability to identify potential gaps.
- The time available to assess the long-term outcomes of the CCEI was limited given the portal launched in 2020. This is supported by evaluation evidence, with the number of interviewees reluctant to rate progress against outcomes increasing from 0% to 42% as the outcomes tested became more long-term and aspirational as they moved further from the CCEI’s direct sphere of influence. Therefore, the evaluation team ensured that the evaluation was properly focused on the programs’ achievement of the intended outputs and immediate outcomes. Furthermore, beyond stakeholder perspectives, there is limited data available to assess the CCEI’s achievement of its intermediate and ultimate outcomes. The CCEI has not developed indicators or collected data to track how its information was used after its publication on the CCEI portal.
Relevance
Is the CCEI focusing on relevant needs and priorities within the changing context of the program’s operating environment?
Canada’s energy information has been characterised as: being scattered and hard to find; having gaps in the completeness, timeliness, granularity of energy data; having varying measurements and definitions across jurisdictions, having data that may be inaccessible due to confidentiality and difficult to navigate for non-experts; and having outdated data given it was not able to update information to reflect the rapid evolution of the energy sector.
There is evidence that the CCEI is focused on relevant needs and priorities, and that the need for the CCEI, especially in informing Canada’s energy transition, has grown since its inception.
The CCEI provides access to neutral, evidence-based information about energy that is critical to meaningful dialogue on Canada’s energy future. Priorities are determined by regularly engaging stakeholders through established avenues. The CCEI program team has led engagements of individuals and organizations from a broad range of backgrounds, perspectives, and expertise that include each PT, nonprofit organizations, Indigenous organizations, academia, and other federal government departments and agencies. Most interviewees acknowledged that the CCEI program team has done significant work to include input from stakeholders through its consultations, workshops and regular meetings and advice from the CCEI’s EAC.
The CCEI program team engaged and consulted with key stakeholders during its development and implementation. Engagements with FPT partners, the EAC, and other users indicate that the CCEI aligns with and responds to the needs of stakeholders, by performing actions such as the consolidation of energy data into one platform, the integration of StatCan employment data, and making progress towards the standardization of various energy data sets and definitions. While there are limitations imposed by the Statistics Act, interviewees have stated that the data could be more granular and timelier to better address their needs.
Alignment and response to departmental and governmental priorities, frameworks and strategies
There is evidence that the CCEI aligns with and responds to departmental and governmental priorities, frameworks and strategies. The CCEI directly supports or falls under existing federal government strategies and frameworks such as:
- The Statistics Act is a central piece of legislation that governs the CCEI. The Act gives Statistics Canada the authority to collect, compile, analyse, abstract, and publish statistical information on the economic, social and general conditions of the country and its people. It also mandates StatCan to collaborate with departments of government in the compilation and publication of statistical information, including statistics derived from the activities of those departments.
- The CCEI’s outcomes laid out in its logic model feed directly into NRCan’s ECCP program outcomes. The ECCP provides policy advice to the Minister of Natural Resources, senior government officials, and other stakeholders to inform decision-making on matters related to energy and climate including the future pathway for Canada’s energy sector. It serves as the federal government’s central energy policy hub in the transition to a low-carbon, sustainable, inclusive, and competitive resource economy. The ECCP also provides policy and engagement services to NRCan and partners (FPT partners and other government departments) to inform decision-making on matters related to energy and climate, including the future pathway for Canada’s energy sector.
- The CCEI is part of a broader pan-Canadian information system initiative that aims to provide credible, impartial, and comprehensive information that supports Canada’s transition to a low-carbon energy system. The CCEI directly contributes to Canada’s energy sector as it aims to increase the use of a consistent fact-base among government, stakeholders, and industry. Through these activities, the CCEI also supports clean growth strategies and aims to enhance the energy sector’s competitiveness, which are also described in the ECCP Performance Information Profile and the Pan-Canadian Framework on Clean Growth and Climate Change.
- The CCEI data sharing implementation system was built on existing data sharing agreements (DSAs) among FPT partners. By leveraging existing DSAs in its collaborative efforts with PTs, the CCEI aligns with broad federal government efforts to work with PTs to accelerate clean growth strategies and enhance the energy sector’s competitiveness.
There is strong alignment between CCEI and Canada’s Clean Technology Data Strategy (CTDS) established in 2017. This data strategy provides the latest information to measure the economic, environmental, and social contributions of the cleantech sector in Canada. Both initiatives strive to increase accessibility to a broad range of high-quality information on energy transition trends and renewable energy technologies. CCEI also hosts CTDS products such as the Natural Resources Satellite Account and the Environmental and Clean Technology Products Economic Account.
Other government department interviewees stated that the program aligns with and responds to departmental and governmental priorities, frameworks, and strategies, and they acknowledged being included in priority-setting initiatives with the CCEI. They stated that the information that the CCEI provides can support existing federal government initiatives such as the net-zero emission strategy, energy efficiency initiatives and low carbon fuels initiatives. This group also noted that the CCEI’s mandate on data overlaps with other existing government data initiatives, such as OpenGov (the Directive on Open Government and the National Action Plan on Open Government that detail Canada’s commitment to being more transparent and accountable).
The CCEI also deepens existing government commitments with many international data and reporting initiatives, such as:
- reporting and monitoring through the International Energy Agency;
- reporting and monitoring progress towards the United Nations’ (UN) 2030 Agenda for Sustainable Development;
- the UN Conference of European Statisticians; and
- the Oslo Group on Energy Statistics.
International comparison
A comparison of five energy data initiatives from the European Commission, United States of America (USA), United Kingdom (UK), Australia and New Zealand was conducted by the evaluation team. The CCEI’s foundational documents referenced the EIA of the United States as a model for the CCEI. The EIA was the most referenced international initiative that was similar to the CCEI. The EIA was cited by six interviewees as the standard the CCEI is trying to achieve. A comparison of the mandate and the resources of the EIA to the CCEI provides insight on the significant level of resources needed to achieve a platform that has relevant, complete, coherent, and timely data that are accessible and useable to the public. For example, while the EIA is able to focus and specialize in producing high quality energy data products in a timely manner, the CCEI’s five-year budget is roughly one tenth the size of the USA EIA’s budget for a single year (2023/2024). In addition, unlike Canada’s decentralized energy information system, the USA has a central agency responsible for energy. Furthermore, the international comparison revealed that comparable agencies in other jurisdictions have legal authority to collect certain energy and climate data to ensure the completeness and coherence of data that may exceed the authority of Canada’s legislation.
Other findings from the international comparison include:
- The United Kingdom’s Office of Gas and Electricity Markets (Ofgem) had the largest budget, but as the national energy regulator, it is also able to recuperate its costs.
- EIA, Ofgem and the European Commission provide information on consumer/public sentiment related to their services or the energy transition and climate policy.
- CCEI is the only website or web portal of those analysed that provides detailed information on gender, status, and visible minorities related to employment in the energy sector. No other initiative listed has detailed GBA Plus data available on the website.
- The EIA and Ofgem provide different pages for different audiences. This includes, for example, EIA teaching resources for children to encourage early energy literacy.
- The USA, UK, Australia, and the European Commission all have legislation that requires mandatory disclosure of different types of energy data and information.
Access to quality energy data in the absence of the CCEI
The CCEI consolidates energy information by linking energy data and information sources within its online portal so the information can be accessed more easily. This centralization of energy data makes access to existing information easier and more convenient. The CCEI facilitates finding the information by providing more coherent search terms and narrowing the scope of search within the single portal. Evidence from internal documents and interviews suggests that the CCEI has useful energy information. However, gaps in functionalities and data exist such as municipal and hydrogen data. Users often compensate for these gaps by accessing other websites and databases within and outside Government of Canada resources. Professional ‘power-users’ are still able to access information as they are familiar with the source data and statistics. None of the interviewees identified the CCEI as their primary source of energy information.
There is no other source on Canadian energy information that is attempting to centralize Canadian energy information into one portal.
Effectiveness – Outcomes
To what extent has the CCEI achieved its intended immediate outcomes as a result of its activities and outputs?
Immediate outcome: CCEI Energy information users can more easily find the data and analysis they need to make informed decisions.
When discussing the CCEI’s users, it is crucial to make the distinction between the general public and professional ‘power-users’ (i.e., energy data professionals and those most likely to make frequent, advanced use of energy information). According to interview evidence, the CCEI has information that is relevant, more complete, and coherent for the general public than that which was available in the period before the CCEI’s implementation. There is evidence, supported by the evaluation’s review of program documents, that the CCEI portal has improved access to energy data by consolidating around 1000 government and non-government sources into one portal.Footnote 1 However, it is difficult to determine if this improved access has led to more support for informed decisions. Stakeholders interviewed for the evaluation have stated that they have observed little awareness of the CCEI and have generally not seen it referenced in the broader energy discourse.
Portal users engaged by the evaluation indicated that the CCEI does not provide them with increased/better quality information and data in terms of completeness, timeliness, relevance, and coherence compared to what they accessed previously. While the CCEI portal centralizes data, it does not address existing issues with Canada’s energy information system related to data suppression (limitations including privacy related to the Statistics Act), coherence, and timeliness of data publications that professional ‘power-users’ raised.
Interviewees were asked to rate progress on CCEI outcomes on a zero-to-seven-point scale, from very low to very high (see detailed responses in Appendix A). On the immediate outcomes related to CCEI users being able to more easily find energy information, 81% of interviewees rated progress as acceptable to very high, indicating progress towards the outcome.
Immediate outcome: Canada’s energy information system is relevant and collaborative.
Interviews and stakeholder feedback from internal reviews revealed that the CCEI has improved access to energy information because it has consolidated energy data from various sources into its platform; however, there is still more room for improvement for professional ‘power-users.’ Given the current importance of climate change in the national dialogue, the importance of understanding the role of energy use in driving GHG emissions and climate change, as well as informing broader decision-making in the energy sector, the role of the CCEI is considered to be very important by all the stakeholder groups interviewed.
The CCEI succeeded in bringing data from different sources into one portal. The data and information provided by the CCEI is generally considered good by the interviewees and the feedback from external stakeholders that was extracted through the evaluation’s analysis of documents. However, as confirmed through interviews, the documentation indicates that data gaps that existed before the CCEI still exist, and professional ‘power-users’ already know where to find the energy data they need.
Mentioned previously, the CCEI has provided many opportunities for FPT engagement such as its FPT committee, as well as various other bilateral engagements, workshops, working groups, and user surveys. There is evidence that these engagements have contributed to Canada’s energy system being relevant and collaborative, as it has led to the coordination of energy publications and products, integrated FPT data, and inter-government agreements. However, other government departments (OGDs) such as the CER believe they could play a more strategic role in CCEI’s priority-setting and develop a more proactive relationship with the CCEI. Additionally, PT and OGD interviewees stated that the collaboration with the CCEI can be slow, but the process is positive overall.
The CCEI also leverages existing DSAs and legislation to maximise information sharing between governments at all levels. Continuous engagement with these bodies is also part of their plan for iterative improvement over time.
On the immediate outcome related to CCEI being relevant and collaborative, 73% of interviewees rated progress as acceptable to very high, indicating progress towards the outcome.
To what extent has the achievement of the immediate outcomes facilitated the achievement of the intermediate and ultimate outcomes?
Intermediate outcome: Energy information users base their decisions on more relevant, consistent, and complete information.
It is difficult to determine the extent that the CCEI has contributed to this outcome as there is a lack of evidence on how users use information from the CCEI. Other than the web metrics of user access to the published energy data and information, the CCEI does not have indicators or measures for continually tracking how the information is used or cited after publication.
Interviewees generally agreed that the consolidated data has made finding energy data in one platform easier. However, it is unclear if the general public is using the information provided by the CCEI to inform decisions. The evaluation did not conduct any public opinion research to close this information gap. Furthermore, none of the ‘power-user’ interviewees identified the CCEI as their primary source of information. They often preferred going directly to sources such as StatCan’s data tables or NRCan websites to collect information rather than through the corresponding link in the CCEI portal. In this way, ‘power-users’ generally continued to rely on their previously established methods for data gathering.
Internal documents revealed that the CCEI program team conducted a workshop that gathered feedback from the general public and ‘power-users.’ For ‘power-users,’ the internal documents indicate that one of the user’s main purposes for accessing the CCEI is to gather energy information to inform research, policy making, and modelling. While the general public had mostly no specific reason for using the CCEI, some indicated research and general interests as purposes for using the CCEI.
Of the interviewees that were asked to give a rating on the CCEI’s progress on “Energy information users base their decisions on more relevant, consistent, and complete information” 69% of the interviewees rated progress as acceptable to very high. These results suggest that most interviewees believe that the CCEI has made some progress towards this outcome.
Intermediate outcome: Energy information users have increased trust in the Government’s energy data.
Interviewees trusted the CCEI due to a common perception that the CCEI is a StatCan entity. The CCEI benefits from StatCan’s reputation as being a trusted source of data. The CCEI also increases trust by providing transparency in data discrepancies and methodologies.
Interviewees stated that as professional ‘power-users’ they generally trust CCEI data. However, for the general public, there is a broader trend of mistrust of major institutions such as government and holders of government data. The CCEI is not immune to these trends as the public often relies on other sources of information outside of government and the CCEI as a basis for decisions. Many interviewees stated that inconsistent data across different sources and suppressed data could potentially negatively impact trust among the general public in the CCEI’s information.
Of the interviewees that were asked to give a rating on the CCEI’s progress on “Energy information users have increased trust in the government’s data” 65% of interviewees rated progress as acceptable to very high. However, in the absence of available data, a relatively high percentage of interviewees (31%) were reluctant to quantify a rating. Excluding these results suggests that most interviewees believe that the CCEI has made some progress towards this outcome.
Ultimate outcome: “Use of information, advice and stakeholder perspectives by decision makers in support of energy initiatives that promote public confidence and clean growth.”
It is too early to assess the CCEI’s performance against its ultimate outcome. However, we noted significant limitations in the CCEI’s performance metrics related to this outcome. While it is difficult to measure the extent to which information is used to inform decisions, there was no information provided through the CCEI’s performance measurement system that could be used to measure this result. Interviewees agreed that the CCEI has information that can support energy initiatives, but whether the information is used to inform actual decisions that can be documented is unclear. While some indicated that in theory the program logic (including its ultimate outcomes) was generally sound, most interviewees provided no evidence that the information linked to the CCEI supported energy initiatives that promote public confidence and clean growth.
Of the interviewees that were asked to give a rating on the CCEI’s progress on “Use of information, advice, and stakeholder perspectives by decision makers” 50% of the interviewees rated progress as acceptable to very high. However, in the absence of available data, a relatively high percentage of interviewees (42%) were reluctant to quantify a rating. Excluding these interviewees increases the total positive response to 87%. These results suggest that most interviewees believe that the CCEI has made some progress towards this outcome.
Ultimate outcome: “Improved sustainability and competitiveness of Canada’s energy sector.”
It is too early to assess the CCEI’s performance against its ultimate outcome, the improved sustainability and competitiveness of Canada’s energy sector. It is important to acknowledge that the CCEI was only launched in 2020, thus the CCEI may have not had enough time to mature as a program to address its longer-term objectives. Some interviewees indicated that the ultimate outcome is intended as an aspirational target, and creating positive changes in Canada’s overall system of statistical data on energy is a long-term endeavour.
Some EAC interviewees shared their perception that the information on the CCEI is not at the level required for ‘power-users’ to use in their work as much of the information is not sufficiently timely and disaggregated to meet their needs, and not in the format needed by users like regulators. Conversely, several of the CCEI’s extended program team members shared the opinion that the CCEI’s information could contribute to this ultimate outcome. However, evaluation evidence on whether information from the CCEI contributed to decisions on sustainability and competitiveness in Canada was not available.
Several PT interviewees shared their perception that the CCEI has made progress towards this outcome; however, they suggested that there is still room for improvement towards increasing the granularity of data (for example, in providing sub-regional or municipal data), and improving the timeliness of data publications (for example, in providing reporting on daily or hourly electricity demand data by provinces or their utilities in Canada). The evaluation found that the CCEI has made some progress towards providing near-real time energy data - for example, the launch of its High-Frequency Electricity Data (HFED): Visualization Tool (beta) in February 2024, which provides electricity data for nine provinces and one territory. However, at the time of data collection, this tool had not yet been widely used or reviewed by the evaluation’s interviewees.
Of the interviewees that were asked to give a rating on the CCEI’s progress on “improved sustainability and competitiveness of Canada’s energy sector” only 42% of the interviewees gave a rating between acceptable and very high. However, a relatively high percentage of interviewees (42%) were reluctant to quantify a rating. Excluding these interviewees increases the total positive response to 73%. These results suggest that most interviewees believe that the CCEI has made progress towards this outcome.
Effectiveness – Design and delivery
To what extent has the CCEI been implemented and delivered as planned?
There were four areas in the design and delivery of the CCEI that evidence suggests were implemented as planned:
- Engagement – Despite the significant amount of engagement required and the small complement of staff, external stakeholders viewed the engagements they participated in as positive, as their needs were discussed and the workplans approved by all parties.
- EAC – The EAC was effectively set up as planned and is perceived to have been instrumental in the development of the CCEI. They are relied upon to provide their expertise in energy data and information that directly informs the activities and data products of the CCEI.
- CCEI portal and data consolidation – CCEI program team members stated that the CCEI has linked to around 1000 government and non-government sources and has made significant progress towards being a one-stop-shop for energy information. CCEI also became one of the first government energy data sources to move to the cloud. In addition to providing increased access to energy data, the CCEI provides access to interactive tools and data graphics. Additionally, CCEI’s program team members have worked to ensure that the portal adheres to accessibility and official language standards set by the Government of Canada. Program staff and OGD interviewees believe that the CCEI effectively leveraged existing resources to develop a portal in a cost-effective manner.
- Roles of NRCan and StatCan – All interviewees stated that NRCan and StatCan play suitable roles in fulfilling the delivery of the CCEI. StatCan is trusted and effectively plays the role as the central operational hub of the CCEI.
There were three challenging areas in the implementation and delivery of the CCEI:
- Governance structure impacting timeliness – The governance structure was labelled as “resource intensive” and “heavy” at the senior level. A program team member interviewee commented that the check-ins at the senior level were helpful at the beginning, but these high-level meetings may not need to be as frequent as the program matures. The move to virtual consultations during the COVID-19 pandemic, and the interruptions to national level meetings meant that the evaluation was limited in assessing the frequency and adequacy of meetings related to the governance structure, particularly those around the EAC and the FPT committee. Program team members also commented that high-level feedback from PTs that could be usefully applied at an operational level was generally not shared at FPT meetings.
- Information Technology (IT) infrastructure limitations – The initial StatCan IT infrastructure in place before the CCEI had to be further developed to host the IT functionalities required for the CCEI’s mandate. Developing and implementing tools for the portal, such as the dashboard and other tools that require specific IT supports required more time and money than originally planned, and the developments were not always “smooth.” CCEI team members were able to address the foundational IT gaps, but this resulted in unexpected progress delays that were generally outside of the CCEI’s control. The evaluation was not able to fully assess progress on resolving StatCan’s broader systemic issues around the adequacy of IT support. However, interviewees were generally positive regarding the influence that CCEI was having on StatCan’s corporate culture, indicating that perhaps some, but not all of the IT issues had been resolved.
- Integrating external data, data visualization and data standardization – Integrating data from across jurisdictions and organizations into the portal has proven more difficult and time-consuming than originally planned. This is not only due to the complexity of the task but also due to the complexity of the energy data situation across Canada and the difficulty of balancing the volume and broad range of priorities within the program.
Internal factors that significantly influenced the CCEI were as follows:
- Beta-Version of the CCEI Portal – The initial pivot to launching a beta-version of the CCEI (rather than a fully formed portal) positively impacted the progress of the CCEI.
- Governance structure – As previously mentioned, the governance structure was seen as top-heavy and resource intensive, primarily during the later stages of the implementation of the CCEI. Interviewees stated that while the top-heavy governance was beneficial in the early stages of the CCEI’s priority-setting and engagements; organizing and briefing these various high-level stakeholders required a significant number of resources that could be used to support more operational work that grew as the CCEI matured. Feedback from the PT partners tended to be minimal and was not informed by the operational knowledge needed around data methodologies and definitions.
- Leveraging existing resources and reliance on the goodwill of stakeholders (e.g., to provide useful feedback through the FPT Committee process, and without any participation funding provided by the CCEI). Overall, this resulted in a positive impact for the CCEI in the short-term as it maintained relatively low program costs.
- Resource constraints – There was an overall sentiment from almost all the interviewees across stakeholder groups that the goal of the CCEI was too ambitious for the level of resources the program received. CCEI program team members have stated that they often struggle to find a balance between addressing priorities based on what should be done to address stakeholder needs, and what can practically be done given constraints on existing resources.
- Change Management and Team Turnover – There were key staffing changes, mostly within StatCan, and turnover that occurred during various points of the CCEI’s implementation. The turnover in team members led to some confusion for points-of-contact within the CCEI program at StatCan, negatively impacting the coordination of efforts. The CCEI program was later integrated into StatCan’s ESP (as an operational decision, unrelated to staffing issues). This transition was seen as a positive change.
External factors that significantly influenced the CCEI were as follows:
- Global pandemic – The COVID pandemic and its response occurred in the midst of the implementation of the CCEI. Its operations shut down overnight, and the response to the pandemic was prioritized. This delayed CCEI’s launch by approximately four to six months. The launch was originally planned for the spring or summer of 2020 but occurred in October of 2020.
- Data suppression – As a result of the Statistics Act, StatCan typically uses suppression techniques to protect sensitive statistical information. These techniques involve suppressing (not displaying) data points that can directly or indirectly reveal potentially sensitive information about a respondent. This data suppression has impacted the completeness and timeliness of the data.
- Other government sources of energy data and information – OGDs, EAC, and PT interviewees have stated that they are aware of the CCEI and have used the portal to search for information. However, OGDs, PT and EAC interviewees generally rely on other sources of energy data to complete and publish their own research and analysis.
- Rapidly evolving nature of the energy transition and its data needs – In addition to the need to provide information to various stakeholder groups (e.g., general public and ‘power-users’), the complexity of the energy data landscape and energy sector in Canada has impacted the CCEI by creating an environment in which it is challenging for the CCEI to keep pace with the rapidly evolving landscape of the energy sector in Canada. Interviewees have referenced, for example, the rapid growth and evolution of the hydrogen sector. Current data does not reflect all of the sources and production methods and end uses of hydrogen. Interviewees also referenced the development of artificial intelligence that presents opportunities to “scrape” data from other sources but also creates potential issues around privacy and accuracy of the data.
To what extent has the CCEI Program considered GBA Plus/EDI factors?
Documents reviewed by the evaluation demonstrated evidence of the CCEI’s consideration of EDI factors. This is particularly true as it relates to an Indigenous lens on data, with efforts that included: having an Indigenous member on the EAC, CCEI collaboration with the CER on Indigenous projects, work with StatCan’s Centre for Indigenous Statistics and Partnerships and its Indigenous Statistical Capacity Development Initiative. Almost half of the program team interviewed, and a majority of the other interviewee groups mentioned the importance of work with Indigenous partners and communities, potentially pointing to greater success at the federal level in raising awareness on the issue. Program team awareness on other GBA Plus and EDI issues was generally low.
The CCEI also followed a pan-Canadian approach to engage all PTs through the Energy and Mines Ministers’ Conference. Independent non-governmental organizations such as environmental groups, private sector consultants, ‘think-tanks’ and academic institutes were also consulted with a view to improving access to data to support the energy transition.
The international comparison revealed differences in the nature of energy information reported to meet stakeholder needs. For example, unlike the CCEI, the EIA, Ofgem and the European Commission provided information on consumer/public sentiment related to their services and the energy transition and climate policy. The EIA and Ofgem also provide different pages for different audiences. However, CCEI is the only web portal or website of those analysed that provides detailed information on gender, status, and visible minorities related to employment in the energy sector. No other organization reviewed has detailed GBA Plus data available on the website.
All documents reviewed by the evaluation that were on the CCEI portal or StatCan’s various websites were available in both of Canada’s official languages, French and English. Interviewees stated that the CCEI was compliant with Government of Canada accessibility guidelines, and that processes were in place to ensure that data and information were accessible. There is evidence in the documents that the CCEI engaged with UX designers (their role is to make a product or service usable, enjoyable, and accessible) to ensure that the portal continually meets the needs of its users.
There was a fairly strong view among some stakeholders that because all humans are consumers of energy, the need for a program like CCEI to consider various EDI factors is of low relevance. The evaluation noted that given the importance of affordability dimensions in the broader energy discourse, and the diversity in energy systems across Canada, including the unique needs and challenges of access to energy in Canada’s north and remote regions, there is opportunity for more collection and dissemination of EDI relevant data. This additional data may have significant potential to provide answers to questions on whether access to energy employment and other benefits are conferred equitably.
The NRCan/StatCan satellite account human resources model was mentioned as a source of data on EDI and accessibility. Other than one interviewee who mentioned their work on the Equal by 30 initiatives to accelerate gender equality and diversity in clean energy, none of the interviewees elaborated on how or why having EDI or GBA Plus data was important. Rather they referred the issue primarily to the production of labour statistics (either by gender or Indigenous/non-Indigenous status).
Members of the EAC stated that they were part of an effective and diverse panel, and the presence of a member that is Indigenous was cited as proof that diverse voices are represented at the Committee.
Does the CCEI’s performance measurement strategy support program effectiveness?
Interviewees were positive on the value of CCEI’s progress on its monitoring to fulfil management information needs. This occurred mostly in the context of StatCan surveys of CCEI users to obtain feedback which was used to enhance the portal’s user experience and functionality. Data were also collected in the form of web analytics reports. Web metrics from the portal help to show that more users are accessing the CCEI, with approximately 50% growth in page views year-on-year since inception. This measure is a proxy measure for stakeholder awareness but cannot show how the CCEI contributes to its outcomes, such as demonstrating that information is being used to inform decisions.
An analysis of the program’s logic model, drafted by NRCan as part of its initial program design, which was later updated for this evaluation, shows that the outcome measures of the wider scope of activities for the NRCan ECCP Program (NRCan ‘parent’ program), from the Performance Information Profile, do not appear to effectively represent what may be required for the CCEI to improve its effectiveness. As the ECCP Program performs a mostly internal supporting policy function within NRCan, there is little performance information collected for the Performance Information Profile that can be related or directly attributed to the CCEI.
The outcomes and indicators outlined in the CCEI’s foundational documents align to outputs (not actual outcomes) and are consistent with the StatCan role as a data and information provider. Therefore, most of the outcomes examined as part of the evaluation had no supporting documentation: for example, there were no documents outlining cases where the CCEI was used in decision-making, and the program had no way to measure or collect information on the degree to which CCEI was ‘trusted’ by Canadians.
The CCEI program team, in consultation with FPT partners have taken steps to identify priorities that are related to data quality, and data gaps. Annual workplans define specific deliverables and project goals that are to be monitored at a program level. Most of the indicator information the evaluation reviewed references items that are on the annual plans; however, the evaluation was not able to consistently verify or analyse achieved outputs in relation to the annual plans due to the absence of a formal or centralized tracking system. Some of the CCEI outcomes, objectives and results could be better and more effectively measured and reported. It is likely that StatCan and NRCan would be required to consult collaboratively on updates to CCEI’s performance indicators, given NRCan’s role in requesting the original funding allocation, and StatCan’s role as the primary data provider.
When the development of the CCEI was initiated by NRCan and StatCan, it started with a special project status in StatCan. StatCan documented an extensive series of lessons learned in its internal-only document special project close-out report when the CCEI was mainstreamed into the operation of the ESP in 2022. This close-out report was drafted by StatCan to comply with the mandatory lessons learned reporting step (close-out) as part of the StatCan Departmental Project Management Framework process. The 2022 special project close-out report pointed to challenges in creating workplans for the IT work needed to launch the CCEI. There was no tracking of IT deliverables against the workplan, due to the lack of an agency-wide standardized tracking system for IT projects. According to the report, this points to the assumption that there are gaps in performance measurement and performance indicators in projects where significant IT resources are required. The evaluation was not able to gauge StatCan progress on the management response and action plan stemming from the CCEI special project close-out report.
With respect to performance information, the CCEI may not be optimizing opportunities to demonstrate why its efforts at continuous improvement are important in areas such as federal leadership on energy data and data harmonization. A strengthened performance measurement regime may also provide opportunities for the CCEI to draw lessons learned from examples of its successful stakeholder relationships. As energy statistical programming is relatively new to StatCan compared to its legislative census activities, it is critical for the CCEI to continue to illustrate its value as Canada approaches the period 2030 to 2050 to demonstrate progress on Canada’s climate goals.
Lessons learned
Are there any best practices data and data harmonization, lessons learned, and improvements required related to the design (planning process, governance, system of performance measurement) of the program?
Program documentation and interviews identified lessons learned. The evaluation evidence did not point to a need for an alternative delivery model or alternative designs for the CCEI. Rather, it suggests general ways to improve existing CCEI products and processes. Many of the stakeholders agreed that there was no need for the CCEI to “reinvent the wheel” or start over again. The concept of a ‘one-stop shop’ at a national level was seen to be a sound way to work towards greater completeness and coherence in Canada’s broader system of energy data.
The special project close-out report made approximately 15 recommendations for improvement, in the following areas: “engagement, working groups, consultations, and outreach;” “communication and organizational change management;” “IT;” “team management, communication and organizational management;” “project outcomes management;” “scope management;” and “cost management.” The report shows that StatCan has been proactive in its planning for its future. Individual items included considerations such as modifying the role of the EAC; offering smaller bilateral or regional meetings with more space for partners to share their perspectives and feedback; creating a pathway for other data centres; development of IT roadmaps for key deliverables; development of more horizontal approaches; improvement of data integration from the FPT partners (i.e., NRCan, CER, ECCC and the provincial and territorial partners), utilities, and others; and development of interactive data tools for advanced functionality. The evaluation was not able to track CCEI progress towards the recommendations of the close-out report. The evaluation’s recommendations are supported by some of the learnings mentioned in the report, and it is expected that NRCan/StatCan may be able to leverage the lessons learned and best practices that have been previously documented.
Possible CCEI improvements suggested varied by interviewee group. No strong convergence was noted on themes, apart from general statements that with more financial resources, the CCEI could perhaps improve or speed up its efforts. Interviewees were generally positive about what the CCEI had managed to achieve with its modest resources. Two CCEI program team member interviewees mentioned continuing work on greater integration of data visualization and portal improvements and increasing the degree to which other federal departments publish their data on the CCEI. There was evidence from two of the stakeholder groups that NRCan/StatCan could perhaps increase the federal leadership role in data harmonization.
The need to update the CCEI's governance structure including its priority setting process was highlighted by several interviewees. The evaluation noted that this view supported findings from the special project close-out report which states the CCEI should “continue regular engagement in developing priorities” and “leverage with FPT partners and stakeholders.”
The evaluation’s analysis showed that for interview evidence, the primary theme for lessons learned and best practices was CCEI’s value as a venue for collaboration which aligned to statements describing the CCEI itself as a best practice. Document review evidence also shows that StatCan consulted its partners on its efforts to improve the user experience, and advance other improvement items mentioned in the close-out report. The workplan development process was mapped out to incorporate feedback from priorities defined by FPT governance guidance, and EAC input, in addition to an internal prioritization process.
CCEI program team member interviewees shared that the CCEI was aware that it could not “do everything all at once” and had an effective process for prioritizing its work. However, some of the federal government interviewees noted difficulties in obtaining feedback via the high-level governance meetings for the CCEI. They also mentioned that it was often difficult to schedule Deputy Minister level participation in meetings, and that when workplans were presented there was often very little provincial or territorial feedback on the directions proposed for the CCEI. A majority of FPT interviewees were positive to very positive on the collaboration that was occurring through the CCEI. It may be assumed that opportunities for continued or expanded collaboration may be met with favourable reactions from at least some of the provincial/territorial departments and organizations that have been involved in the CCEI. One of the limitations of the evaluation was that not all provinces and territories opted to participate.
Two other themes were identified by the evaluation, stressing the importance of good communication between partners, and the importance of standards for coherence. Internal interviewees were positive about the value of listening to feedback and giving a voice to stakeholders. Two of the PT interviewees expressed a desire to have more regular updates from the CCEI and working group structures that prioritized incorporation of feedback from energy data professionals. They suggested that there be a recurring or perhaps more formal venue that was geared to the needs of partners in how often it would meet (for example, at least one interviewee suggested that monthly meetings would be too frequent, but that perhaps CCEI could convene working group meetings a few times per year). The evaluation noted that there could be more opportunities for working group activities to further align federal and PT efforts at furthering work on standards, including on international standards such as SDMX, a global initiative to improve statistical data and metadata exchange.
Having NRCan’s Energy Fact Book available exclusively on the CCEI was viewed positively by StatCan interviewees. It was also suggested that the CCEI try to acquire permission to post other ‘flagship’ products from federal or provincial partners. One of the territorial stakeholders interviewed stated that micro-data from their jurisdiction was being published on the CCEI through a DSA, which was seen as a positive model that supported the collaborative nature of the CCEI’s activities. The view was also shared on the value of having federal and provincial or territorial focal points through the CCEI that could function as resources for information exchange, and act as a question-and-answer resource.
Conclusion
Key findings and recommendations
The evaluation found that an increased need for the CCEI exists, given the importance of Canada’s energy transition. As an initiative, the CCEI is well positioned to support Canada’s 2030 Emissions Reduction Plan, the 2050 Net-Zero Emissions Accountability Act, low carbon fuels initiatives, and open data strategies among other initiatives, in a way that is transparent and accountable. However, advanced users of the CCEI shared the perception that relevant energy data, including related to Canada’s GHG emissions, were not sufficiently granular or timely. Ongoing improvements in Canada’s overall system of statistical energy data (not just for the CCEI) are necessary to facilitate better decision making by policymakers, industry, researchers, and Canadians.
Outcomes were initially defined by NRCan that aligned to the funding instruments that NRCan sought in collaboration with StatCan. The CCEI met its immediate outcomes related to Canada’s energy information system being relevant and collaborative, and information users more easily finding data and analyses to make informed decisions. A range of FPT and private sector partners mentioned the valuable collaboration that was occurring. The CCEI as a venue for collaboration and the CCEI itself for bringing energy information into a ‘one-stop shop’ were seen as best practices. There is room for improvement, as some stakeholders noted that collaboration and communication with the CCEI was sometimes slow and intermittent. Securing meaningful PT engagement was noted as a particularly challenging area. Feedback from stakeholders indicated that the governance structures such as the EAC and PT input from through the Energy Mines and Ministers’ Conference were initially very useful but may now be ‘top-heavy.’ Stakeholders expressed interest in more opportunities to participate at the working level, with agendas and the timing of meetings geared to the needs of the parties at the level of energy data professionals.
Stakeholder perspectives on progress against intermediate and ultimate outcomes were mixed. The intermediate outcomes related to energy information users basing their decisions on more relevant, consistent, and complete information, and increased trust in the Government of Canada’s energy data. Ultimate outcomes related to use of stakeholder perspectives by decision makers in support of energy initiatives, and the improved sustainability and competitiveness of Canada’s energy sector. While many felt that progress was being made on these outcomes and on data harmonization, many stated that the issues around the intermediate to ultimate outcomes are complex and difficult to measure. Some interviewees responded that they could not rate CCEI against its outcomes.
From the perspective of data professionals, some interviewees stated that without personally analysing some targeted data related to the outcomes, they could not comment on progress attributable to the CCEI. Furthermore, while they view StatCan as a trusted and politically neutral organization, some shared the perception that the CCEI was performing a ‘library’ function, and not fully showing federal leadership on improving the quality and timeliness of energy data in Canada. Updates to CCEI’s strategic direction, governance, and performance measurement frameworks and performance monitoring program were suggested by stakeholders and are supported by the evaluation’s review of the available evidence.
To further leverage the value of federal family collaboration that resulted from the CCEI, there are opportunities for the CCEI to continue to advance data visualizations available to users. NRCan’s Energy Fact Book was found to be a useful model containing map data, graphs, and graphics. Exclusive posting of the Fact Book to the CCEI (formerly it was on the NRCan website) was associated with a significant spike in user traffic. While CCEI user visits have increased by at least 50% year on year since its establishment, overall web access numbers are approximately a few thousand page views per month, amounting to approximately 50,000 page views per year. There is still room for improvement and visualizations may help to make CCEI information easier to absorb for both senior leaders and Canadians, which could aid in increasing awareness of the CCEI, and could enhance the CCEI’s ability to communicate useful information about Canada’s energy systems.
Appendix A: Interviewee rating of achievement of CCEI outcomes
Please rate, on a scale of 0-7, the level of progress made on the achievement of the following outcomes.
| CCEI Outcome | # of interviewee responses per rating (n = 26) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
|
0 Not applicable / Don’t know |
1 Very Low |
2 Low |
3 Slightly Low |
4 Acceptable (neither low nor high) |
5 Slightly high |
6 High |
7 Very high |
||
| 1 | CCEI energy information users can more easily find the data and analysis they need to make informed decisions. | 3 | 0 | 1 | 1 | 6 | 8 | 5 | 2 |
| 2 | Canada’s energy information system is relevant and collaborative. | 3 | 1 | 2 | 1 | 2 | 6 | 8 | 3 |
| 3 | Energy information users base their decisions on more relevant, consistent, and complete information. | 4 | 1 | 0 | 3 | 6 | 8 | 4 | 0 |
| 4 | Energy information users have increased trust in the government’s energy data. | 8 | 1 | 0 | 0 | 3 | 7 | 7 | 0 |
| 5 | Use of information, advice, and stakeholder perspectives by decision makers in support of energy initiatives that promote public confidence and clean growth. | 11 | 0 | 2 | 0 | 4 | 6 | 2 | 1 |
| 6 | Improved sustainability and competitiveness of Canada’s energy sector. | 11 | 1 | 1 | 2 | 4 | 4 | 3 | 0 |
Appendix B: Evaluation Team
From Natural Resources Canada:
- Michel Gould, Chief Audit and Evaluation Executive
- Stephanie Kalt, Director of Evaluation
- Jamie Riddell, Senior Evaluation Manager
- Edmund Wolfe, Senior Evaluator
- Carlo Magnaye, Evaluator
From Statistics Canada:
- Steven McRoberts, Chief Audit and Evaluation Executive
- Yen Linh Ly, Director of Evaluation
- Anita Chung, Senior Evaluation Analyst
The evaluation team would like to acknowledge those individuals who contributed to this project, particularly program representatives and members of the Evaluation Working Group from StatCan’s Energy Statistics Program, and NRCan’s Energy Systems Sector.