Introduction

The first seven components of DCAM define the capabilities required for best practice Data Management (DM). These DCAM best practices guide us regardless of the purpose to which the data is subsequently applied. The issues inherent in Artificial Intelligence/Machine Learning (AI/ML) and an organization’s Code of Data Ethics in these components are particularly relevant where Analytics consumes the data. Consumption of data for analytical purposes is increasingly important for many organizations. Analytics is dependent on high-quality, well-understood data. Analytics functions are, in general, dependent on data produced by areas outside Analytics or that is sourced from outside the organization.

The purpose of Analytics Management (AM) is to formalize how the Analytics activities of an organization are structured, executed, and managed and to ensure they are aligned with the DM activities. The degree to which Analytics teams are either centralized or distributed in an organization will depend on the structure and culture of the organization. However, synergies, efficiencies, and effectiveness will be maximized if the teams operate within a well-understood framework as part of a coherent Analytics strategy.

Definition

The AM component is a set of capabilities used to structure and manage the Analytics activities of an organization. The capabilities align AM with DM in support of business priorities. They address the culture, skills, platform, and governance required to enable the organization to obtain business value from analytics.

Scope

• Define the Analytics strategy
• Establish the AM function.
• Ensure that Analytics activities are driven by the Business Strategy and supported by the DM strategy.
• Ensure clear accountability for the analytics created and for their uses throughout the organization.
• Work with DM to align Analytics with Data Architecture (DA) and Data Quality Management (DQM).
• Establish an analytics platform that provides flexibility and controls to meet the needs of the different stakeholder roles in the Analytics operating model.
• Apply effective governance over the data analysis lifecycle. Governance includes tollgates for model reviews, testing, approvals, documentation, release plans, and regular review processes.
• Ensure that Analytics follows established guidelines for privacy, data ethics, model bias, and model explainability requirements and constraints.
• Manage the cultural change and education activities required to support the Analytics strategy.

Value Proposition

Organizations that best manage their Analytics functions and resources effectively maximize the benefits of combining advanced algorithms and high-quality data.

Overview

In simplest terms, analytics support decision making by analyzing data. They are not new. Statistical analysis of manually collected data pre-dates the age of computing. Advances in technology have increased the speed, variety, and sophistication of analysis, as well as the quantity and variety of data that can be analyzed. Analytics-based, real-time, sophisticated decision making is accelerating and has become a key business differentiator.

Diagram 8.1: Analytics Enablers and Stakeholders

Advances in technology and data include:

• Lower data storage costs, increasing the quantity of data that can be made available to analytics.
• New sources of data such as sensors, telematics, and satellite imagery, enabling new data sets and new combinations of data sets to be analyzed.
• NoSQL and graph database technologies are enabling greater varieties and quantities of both structured and unstructured data to be accessed and processed efficiently.
• Falling costs of processing and the ease with which cloud computing capacity can be scaled up and down, increasing the affordability of more sophisticated analytics techniques.
• Advanced data visualization simplifying how people explore and interpret very large volumes of analytics results.
• The evolution of ML, enabling models to make decisions with minimal or no human involvement.

Data scientists have the skills to understand a business need, bring together data sets, and apply advanced analytics techniques to address that need. In many organizations, they may be aligned more directly with the business units they support than with the technology teams that provide more traditional business intelligence analytics. In both circumstances, the organization must have a clear operating model that provides consistency and efficiency in how the activities are performed. Creators of this model must take care to retain the business alignment and agility that the organization demands.

The ever-increasing volume and variety of data available to an organization means the universe of issues that advanced analytics can be tasked with is boundless. The organization must have a clear strategy for how analytics will be used to support the businesses’ objectives. The operating model should ensure this Analytics strategy can be delivered, and a funding model must be established to sustain this effort.

In the absence of effective data management, a significant amount of an Analytics practitioner’s time is spent manipulating, cleansing, and transforming data in preparation for the analytics. Analytics must be aligned with the DM initiative. In particular, DA should ensure that data is understood and that authoritative sources are used where these are available. DQM will provide measures of data quality that Analytics should reference, and DQM should be used to manage data quality issues uncovered by Analytics practitioners as they prepare data for their models.

Not all Analytics activities will involve the creation of models that, once successfully tested, will run as production processes or services. Some analytics will be one-time exercises to investigate a historical issue or answer a specific question at the moment. There may also be experiments with new data sources or new analytical techniques. The platform that supports Analytics must be designed to accommodate these different types of activity and the specific needs of different stakeholders.

Most organizations will have restrictions on access to production data, whether driven by commercial sensitivity or personal privacy regulations. Analytics practitioners will often need to work with data that has been obfuscated in some way. Requirements for this must be understood and must be facilitated by the analytics platform.

Model governance and model transparency are critical to the controlled, auditable development and deployment of advanced analytics. Compliance with privacy regulations is vital when models are released as production assets or services. However, with advanced analytics, ML and AI, other governance and control issues also come into play. The need to explain decisions made by models may be a regulatory requirement. As with most considerations that need to be confirmed at the point of release into production, requirements for explainability need to be understood as early as possible in the model development. This need for early understanding also applies to model bias, particularly if this bias could result in prejudice against groups of individuals. In this case, the bias in the data sets that are used to train models needs to be controlled. These controls may need to be extended to data that the model encounters over time in production. Finally, the models, and the business issues they are addressing, need to be governed against the Code of Data Ethics of the organization. It is essential to determine if the way the model is using the data is ethical and appropriate. This determination then must guide the organization as it decides whether and how it should act on the decisions and recommendations the model makes.

Overcoming the challenges an organization will face in delivering its Analytics strategy will require cultural and behavioral changes. Analytics practitioners will require specific analytic skills. However, an awareness of both the value that analytics can bring and the potential for undesirable impacts created through analytics will be needed throughout the organization. Awareness of model bias and data ethics considerations will be required by those creating models as well as those approving them.

Core Questions

• Does the Analytics strategy clearly articulate the role that Analytics will play in delivering the business strategy?
• Is there buy-in to the Analytics strategy, operating model, and funding model across the organization?
• Are Analytics activities prioritized according to business priorities, and is the value delivered understood?
• Are AM and DM fully aligned?
• Does the analytics platform support the needs of Analytics initiatives and the Analytics operating model?
• Are processes in place to control the release of analytics models into production?
• Is approval and release of models aligned with privacy and data ethics governance?
• Are there initiatives in place to address the cultural change and Analytics practitioner education to enable Analytics success?

Core Artifacts

The following are the core artifacts required to execute an effective Analytics Management capability. Items with an ‘*’ link to published best practice guidelines.

• Analytics Strategy
• Analytics Classification System
• Analytics Operating Model
• Analytics Methodology
• Model Documentation Standard
• Data Obfuscation Strategies
• Model Testing, Approval, Release, and Regular Review Procedures
• Cultural Behaviors Gap Analysis and Initiatives Backlog
• Analytics Skills Gap Analysis

For an Analytics function to be sustainable, it must be formally established and recognized. It must be approved by management and supported by an approved funding model and an effective governance structure. Roles and responsibilities must be established, and an analytics methodology must be adopted.

Description

The Analytics strategy must be defined and formally empowered by senior management, and the role of the Analytics function must be communicated to stakeholders.

Objectives

• Formally establish the organization’s Analytics strategy.
• Obtain approval for the Analytics strategy from stakeholders.
• Obtain executive management support for the Analytics strategy.
• Communicate the role of Analytics across the organization through formal organizational channels.
• Secure the authority to enforce alignment with the Analytics strategy through policy and documented procedure.

Advice

The Analytics strategy must be aligned to the business and operational objectives of the organization and be aligned to the overarching Data Management Strategy (DMS).

The strategy presents the vision for Analytics, covering organization, leadership, techniques, platform, processes, and culture. It shows how to address the gaps and realize the vision. The goal of developing the Analtyics strategy is to capture the high-level objectives and translate them into achievable Analytics solutions.

Achieving stakeholder and executive buy-in is critical to the success of the strategy. A well-documented Analytics strategy is both a statement of approach and a marketing document for presentation to stakeholders and executive management. Effective communication of the strategy is key to empowering a coordinated approach across the organization and avoiding inconsistencies and inefficiencies of different business areas, taking different approaches.

Questions

• Is there a formal, documented strategy and approach for Analytics?
• Have stakeholders been identified and involved in the creation and approval of the strategy?
• Are the Analytics strategy and the vision of the DMS aligned?
• Does the Analytics strategy support the high-level objectives of the organization?
• Has executive management support for the strategy been obtained?
• Has the role of Analytics been formally communicated throughout the organization?
• Do policies and procedures exist with the authority to enforce alignment with the strategy?

Artifacts

• A Vision statement of the target state for Analytics and how it supports the organization’s objectives
• A prioritized approach to the initiatives required to realize the Analytics vision
• Documented alignment of the Analytics strategy and vision of the DMS
• List of stakeholders and evidence of bi-directional communication
• Evidence of formal communication of the strategy and approach
• Evidence of formal approval of the strategy
• Policies and procedures enforcing alignment with the strategy

Scoring

Description

The spectrum of analytics (e.g., MI (Management Information), BI (Business Intelligence), ML, or Descriptive, Diagnostic, Predictive, Prescriptive) employed by the organization is defined. Names and descriptions of the different types of analytics relevant to the organization are formalized in a categorization system. This system is used in statements of the scope of responsibility in the Analytics operating model. It provides a common language for the organization to refer to analytics and helps avoid confusion and misunderstanding.

Objectives

• Define the different types of analytics relevant to the organization.
• Validate the categorization system with stakeholders.
• Communicate the categorization system to Analytics practitioners.
• Adopt the categorization system in the Analytics operating model.

Advice

In defining the spectrum of analytics, examples will bring these to life. The boundaries of the analytics spectrum determine the activities to which the Analytics governance and best-practice frameworks apply. For example, an organization may define its lowest boundary of analytics such that static MI reports are out-of-scope, whereas self-service BI is considered in-scope.

In setting the categorization system, it is important to consider how to identify the stakeholders and to solicit their input. It is also important to identify who should receive communications regarding the outcomes of the process of setting the categorization system.

The categorization system for levels of analytics will not be static. It will work best by allowing for emerging levels of sophistication as Analytics matures and evolves.

Questions

• What types of analytics has the organization developed historically?
• What types of analytics does the organization foresee itself using in the future?
• Which stakeholders should be approached for input into the categorization system for levels of analytics?
• Who should receive the published categorization system, both within and outside the organization?

Artifacts

• Documentation of the categorization system
• Lists of stakeholders involved with the categorization system
• Distribution lists for publishable versions of the categorization system

Scoring

Description

An operating model is defined to describe how Analytics will be structured within the organization. It establishes the scope and coverage of the different parts of Analytics and designs how they should work together to serve the needs of the business.

Objectives

• Define the Analytics team's structure within the organization, delineating the scope and coverage of each.
• Define the roles and responsibilities within each type of Analytics team.
• Create the organizational structure described in the operating model.
• Obtain approval from stakeholders for the Analytics operating model.
• Define the interrelationships between the Analytics teams, between Analytics teams and the Data Management initiative and data teams, and between Analytics teams and the business functions.

Advice

The Analytics operating model encompasses disciplines from different areas of IT (Information Technology), business, and Analytics. It provides clarity on how these areas work together to provision the data required by analytical models and to deliver the models into production.

The Analytics operating model clarifies the ownership of line-of-business Analytics, cross-business functions, and any overarching Analytics program. It is aligned with other operating models within the organization. If there is a strong culture of federated operating models, this should be embraced. Conversely, if there is a strong centralized operating model culture, carefully consider the benefits before deviating from this approach.

Best practices suggest that the ideal make-up of an Analytics team includes people with mathematical, analytical, and technical skills and strong business acumen. The creation of centers of excellence should be considered. Establishing centers of excellence can focus discipline around Analytics and instill a level of confidence in the user community. Centers of excellence will bring together the IT, business, and analytical skills that better leverage the organization’s investment in technology. These centers transition Analytics from technical data gathering to providing solutions through organization-wide awareness, sharing, and best practice.

Having buy-in from stakeholders is key if the Analytics organization is to be effective. Stakeholders should be involved throughout the creation of the operating model to avoid misaligned expectations.

Questions

• Has the structure of the Analytics teams within the organization been defined?
• Have the roles and responsibilities of each type of Analytics team been defined?
• Does the team structure delineate the scope and coverage of each?
• Are the interactions of Analytics teams, the Data Management initiative, and the business defined?
• Have stakeholders approved the operating model?
• Has the functional structure described in the operating model been implemented?

Artifacts

• Operating Model Terms of Reference
• Organization structure of Analytics teams
• Role definitions and RACIs (both between Analytics teams and between Analytics and other functions)
• Process models for Analytics, DM and business interactions
• List of stakeholders and evidence of bi-directional communication
• Documented formal approval of the operating model

Scoring

Description

The Analytics funding model addresses the resources required for stakeholders to implement and sustain the Analytics function. The funding model aligns with the organization-wide funding process. It is enforced through the Analytics governance process.

Objectives

• Define and socialize the Analytics funding model with stakeholders.
• Collect and incorporate feedback into the model.
• Have stakeholders review and approve funding commitments for a sustainable function.
• Align the funding model with the overall funding approach of the organization.
• Measure benefits that help support the funding case.

Advice

The funding model must address all aspects of funding Analytics. Types of work beyond business-driven Analytics projects should be addressed. These include experimentation, foundational work, creation of re-usable data assets and ad-hoc, high-priority analysis, and more. Address the funding of software licenses and platform and infrastructure costs. When addressing people costs, take account of differentials in costs of Analytics specialists. Alignment with the overall funding approach of the organization may require the model to distinguish between Analytics initiatives aligned with operating units and aspects of the operating model that are not operating unit-specific.

Ensure that Analytics is funded as a sustainable function. Use the funding model to distinguish aspects of funding that are discretionary from those that are non-discretionary. Stakeholder involvement in the creation of the funding model is critical to obtaining a financial commitment that can be sustained.

Questions

• Has a funding model for Analytics been established?
• Is the model consistent with the established funding approaches and budget processes of the organization?
• Does the model address how Analytics will be funded as an ongoing, sustainable function?
• Are measurable benefits from Analytics initiatives used as support for the funding requirements?
• Has the model been socialized with stakeholders, and has their feedback been addressed?
• Has the model been formally approved?

Artifacts

• Documented funding model
• A written process to review, modify and validate resource plans
• Evidence of alignment with budget processes and organizational cycles
• Documentation of measured benefits
• List of stakeholders and evidence of bi-directional communication
• Funding plans and budget allocation
• Evidence of formal approval

Scoring

Description

Explicit governance of Analytics is in place across the organization. The governance structure oversees implementation and sustains the operating model for Analytics, implementation of the analytics platform, and ongoing initiatives to shape the Analytics culture of the organization. It ensures alignment of Analytics with business strategy, data ethics, and the DMP (Data Management Program).

Objectives

• Define the governance structure for Analytics.
• Document and share the governance structure with stakeholders.
• Create policies to enforce Analytics governance implementation.
• Establish governance forums with written and approved charters.
• Implement operating governance structures.
• Appoint stakeholders.
• Communicate stakeholder roles and responsibilities.
• Hold stakeholders accountable for their participation in Analytics via performance reviews and compensation considerations.

Advice

The governance structure should complement and align with the Analytics operating model. Care should be taken to ensure the governance structure does not constrain timely decision making. Members of the Analytics governance structure need to be clear about their roles and empowered to effect change. Ensure there is representation from all the disciplines required to resolve issues arising from identification of opportunities, data collection and provisioning, and model deployment.

Analytics will have dependencies on other areas of the organization for deliverables such as data collection and model deployment. It is particularly important to agree in advance to clear hand-offs and escalation paths. Consider having members of the governance organization reporting to senior management, such as the COO, to gain this empowerment.

Tool selection should be aligned with the business strategy and not driven just by what tooling or expertise exists in the organization.

Ethics and privacy are central themes in the governance of Analytics. Start by building on the ethics and privacy data governance processes and structures already in place. Governance processes need to demonstrate the fairness, transparency, and security of data usage.

Questions

• Do terms of reference for Analytics governance exist?
• Do policies exist that enforce Analytics governance implementation?
• Are there metrics for what is governed and measures of successful compliance?
• Is there a register of Analytics projects with up-to-date status available?
• Is there a model register with review dates and update history?
• Is there a tooling inventory with usage recommendations available?
• Is there RACI (Responsible, Accountable, Consulted, Informed) documentation for all stakeholders and participants in the Analytics governance process?
• Is the governance process demonstrating that ethics and privacy are governed?

Artifacts

• Analytics governance terms of reference
• Evidence of policies written, implemented and enforced to show that Analytics is properly governed
• Analytics governance metrics and measurements
• Analytics project tracking
• Model inventory, including model reviews and updates
• Tooling inventory and recommendations
• Analytics governance RACI

Scoring

Description

An analytics methodology provides a framework for the activities performed through the analytics lifecycle. The lifecycle begins with understanding the business problem and runs through to deployment, operation, and review of the solution. The framework provides a common language and structure for stakeholders to refer to the different stages and aspects of Analytics activities. A standard for model documentation ensures consistency across the organization in the way that model provenance, assumptions, inputs, outputs, parameters, and limitations are captured and communicated.

Objectives

• Define and adopt a standard for model documentation.
• Select or develop an analytics methodology that defines the analytics lifecycle for the organization.
• Obtain approval and support for the methodology and model documentation standard from stakeholders.
• Ensure the methodology and model documentation standards are understood and adopted by Analytics practitioners.
• Formally document the analytics methodology and model documentation standard.
• Provide feedback mechanisms for ongoing refinement and improvement of the methodology and model documentation standard.

Advice

Whether the organization should buy or outsource analytics solutions as opposed to internally developing them will depend on the maturity and culture of the organization. There will be more flexibility to define and specify the analytics methodology for internally developed analytics than for externally developed analytics. However, in both cases, it may be worthwhile to consider external best practice methodologies and adapting them to the organization.

To ensure that the analytics methodology is appropriate to the organization and to get stakeholder buy-in, key influencers and stakeholders should be involved in the selection or development of the analytics methodology. Form a community of “champions” to act as owners of the methods and drivers of improvement.

The analytics methodology should focus on significant steps in the end-to-end process rather than specific analytical or modeling techniques. Analytics tools come and go, while analytics methodology should be sustainable and stable. However, to reach a long-term stable analytics methodology, the organization must be open to methodology improvements based on experience. The analytics methodology should accommodate innovation as well as business use-case-driven analytics.

Questions

• Has an analytics methodology that defines the analytics lifecycle of the organization been selected or developed?
• Has a model documentation standard been created?
• Has approval and support for the methodology and model documentation standard from stakeholders been obtained?
• Has it been confirmed that the analytics methodology and model documentation standard have been understood and adopted by Analytics practitioners?
• Have the analytics methodology and model documentation standard been formally documented?
• Have feedback mechanisms for ongoing refinement and improvement of the analytics methodology and model documentation standard been developed and made available to stakeholders?

Artifacts

• Records of analytics methodologies considered by the organization
• Analytics methodology and terminology document
• Model documentation standard
• Records of approval by stakeholders of the analytics methodology and model documentation standard
• Evidence of adoption of the analytics methodology and model documentation standard by Analytics practitioners
• Feedback on refinements to the analytics methodology and model documentation standard
• Feedback from Analytics practitioners and other stakeholders
• Documentation of mechanisms for ongoing refinement of the analytics methodology and model documentation standard

Scoring

The Analytics and DM functions must be aligned and together must support the business goals. Analytics must be prioritized to meet the needs of business strategy and drive business value.

Description

Each organization adopts a unique business architecture. This architecture defines its structure, governance, processes, and information for decision making. The business architecture of the organization should inform the organizations’ analytical requirements. Those accountable for delivering analytics solutions are also accountable for engaging the business stakeholders, understanding the impact of business requirements on analytics, aligning analytical requirements, and ensuring that these requirements are delivered.

Objectives

• Review the business architecture (i.e., strategy, structure, governance, processes) – regardless of the degree of formalization.
• Understand and document key analytical requirements appropriate for the business.
• Align the Analytics capabilities and processes with business requirements.
• Communicate and anchor analytical requirements to stakeholders.

Advice

Not all businesses operate the same way. Priorities for decision making may vary across different parts of the same organization. Analytics leaders must engage with business leaders to understand their requirements and consider how Analytics should best align with the business architecture. Note that the business architecture is composed of a broad array of elements and not merely the IT architecture.

The operating model designers for Analytics must ensure it is informed by the business requirements and business architecture and is kept current with developing input from business leaders. The proximity of analytics to senior decision making may require parts of the Analytics organization to be more decentralized and embedded within multiple business units or functional areas. However, some analytics may best be enabled by a center of excellence.

The most effective Analytics organizations work on projects aligned to the business objectives and business architecture. Full engagement with business objectives ensures the Analytics portfolio is balanced: strategic vs. tactical; compliance vs. performance; BAU support vs. innovation; balanced scorecard vs. single metric.

Documenting the analytical requirements will achieve a clear and consistent understanding. Ongoing interaction with the evolving business needs keeps Analytics focused on immediate priorities.

Questions

• Is the Analytics operating model and plan documented?
• Have key dependencies been taken into account in the design of the Analytics organization and its connectivity to decision making?
• Does the Analytics plan include specific reference to business requirements?
• Have the senior executive and line-of-business executive teams endorsed the Analytics plan?
• Is there a defined process for ongoing updates of the Analytics plan?

Artifacts

• Evidence of alignment of Analytics with business strategy and plan
• Analytics operating model
• Documented alignment of Analytics and Business Architecture
• Details of business analytics requirements
• Evidence of senior executive input and approval

Scoring

Description

The business strategy drives the Analytics roadmap and activities. This fact keeps the focus on Analytics investments and how they can maximize business outcomes.

Objectives

• Understand leadership expectations and explore how Analytics can support strategic value creation.
• Work collaboratively with leadership to build a comprehensive list of current business opportunities and analytics use cases.
• Co-create an Analytics vision and roadmap that helps the alignment with business priorities, recognize short- and long-term focus areas, and creates senior management buy-in.
• Create a benefits case for each analytics use case and prioritize them according to business value through the Analytics prioritization process.
• Communicate the Analytics vision and the prioritization of analytics use cases to stakeholders.

Advice

Align the Analytics roadmap with the business strategy establishing a forum across the business to supervise prioritization of analytics use cases.

A framework to assess the benefits of analytics use cases should have clear dimensions for scoring and grading them. Scoring and grading prioritization should include, but not be limited to, the complexity, feasibility, regulatory requirements and business value of the analytics solution, the number of data sources involved, and the size of the user community impacted. The framework should be transparent, allowing the fair assessment of analytics use cases and the communication of use case prioritization with the business stakeholders.

The organization should ensure that Analytics activities include both those that drive business outcomes as well as those that develop the foundational and future state Analytics capabilities. Create a roadmap that contains a detailed list of initiatives that are sequenced to help the organization deliver on its strategic objectives, and establish processes to revisit sequencing as business priorities and market conditions change.

Questions

• Is there a formal, documented, and prioritized roadmap for Analytics?
• Does a process exist to ensure prioritizations are revisited and modified by accountable stakeholders?
• Does the Analytics roadmap articulate its support of the organization in its strategic business objectives?
• Has the Analytics vision and roadmap received appropriate high-level sponsorship?
• Do the analytics use cases document the assessments of their business value as well as their feasibility?
• Are there clear grading criteria for assessing or scoring the potential business value and feasibility of analytics use cases?
• Has communication support been established so business stakeholders can prioritize the use cases?

Artifacts

• Analytics vision statement or plan
• Prioritized roadmap of Analytics initiatives
• Details of use case benefits
• Evidence of high-level sponsorship and support

Scoring

Description

Analytics properly aligned to business objectives can deliver value in multiple ways. Examples include informing decisions, validating understanding, identifying process effectiveness, or suggesting actions. Analytics teams drive innovation, taking new ideas to the business as well as responding to business-led use cases. Appropriate analytics insights should be embedded in business information and processes. In many cases, this will include automation of analytics with technology for greater speed and efficiency.

Objectives

• Clarify which business questions, decisions, actions or processes would be impacted by the analytical solution in the formulation of a new analytics use case.
• Early in the design of any analytics solution, establish which user community will benefit from the analytical solution.
• Co-develop analytics solutions with relevant business teams.
• Ensure access to analytics outputs and that visualizations of them are readily available.
• Communicate and deploy outputs in ways the business users can easily consume them, including the development of good visualizations and user interfaces to ensure the explainability of the analytics outputs.

Advice

Analytics should both respond to business needs and proactively generate new insights. In each case, there is a responsibility to ensure the business can use the analytics outputs to support its decisions and act on the new insight.

Regular communication between the business and Analytics teams is critical for success. Interaction allows the analytic solutions to be co-developed so that they are focused on the right needs and delivered in a way that is most effective for the users to act. During this process, Analytics can add value by considering a broader array of data inputs and insights that may be derived from a mix of internal and external sources.

When designing new analytics use cases or initiatives, it is important to consider the actionability of insights. In some instances, limits on actionability may arise from ethical, legal, or commercial considerations. For example, while it may be technically feasible to predict which customers are most likely to churn, the business must be extremely careful when deciding whether to act on this insight, to avoid unintended consequences of contacting them. The use of personally identifiable information often requires special care for ethical and regulatory reasons.

All Analytics activities must have a purpose. Some analysis is likely to be investigative and so not directly actionable in a business process. However, these analyses should fit into a decision process where the exploratory analysis helps the business choose the next course of action.

Analytics teams have multiple options for the deployment of insights depending on the context and time-sensitivity of the use case. For example, the ability to automate or centralize critical algorithms while minimizing manual manipulation is likely to be more appropriate for high volume close to real-time decision making. In contrast, ongoing business processes with humans in the loop are more likely to need analytic outputs using visualizations and integrated business intelligence tools. In both scenarios, it is important to ensure the explainability of the analytics outputs so that stakeholders can gain trust in using the analytics.

Questions

• Do formal procedures exist for documenting the purpose, scope, and delivery of analytics use case insights?
• Does the analytics specification document identify the specific user community impacted?
• Does the analytics specification document identify how the outputs will be delivered, acted on, or embedded in the business process?
• Have BI visualization tools been deployed for the delivery of analytics?
• Is the analytics tooling readily available to all potential beneficiaries in the organization?
• Has the organization demonstrated the benefits of automating analytics that supports business processes?

Artifacts

• Analytics governance policies
• Analytics specification documents
• Evidence of alignment with Privacy and ethics policies
• Evidence of analytics automation
• Analytics output visualizations/li>

Scoring

Description

The objective of Analytics is to provide insights that are used to deliver outcomes of business value. The organization must routinely measure the impact and effectiveness of their analytics solutions.

Objectives

• Include ways to measure new and additional value created by analytics solutions early in the course of their development.
• Include ways to measure new and additional value created by analytics solutions post-implementation, as continued ingestion of new data can increase or decrease their value.
• Engage the stakeholders to create shared accountability and buy-in to quantified benefits driven by analytics.
• Measure the impact of analytics use cases, projects, and experiments.
• Communicate the observed and measured business value of analytics.

Advice

Organizations should consider in advance how the outcome of any analytical solution will be measured.

A useful starting point is to establish reliable comparison benchmarks from which to measure incremental impacts. Control groups and randomized experimental methods are good ways to establish these benchmarks. These tools become especially important to diagnose cause and effect in situations where actions based on analytics insight do not result in the anticipated benefits. For example, the failure of an analytics-based marketing campaign that does not deliver expected benefits may not simply result from failed analytics models. Still, it may arise from other factors such as poor marketing or data.

When the incremental value of analytics can be measured, the successful measurement can be applied to a whole portfolio of analytical solutions. The Analytics teams quantify the business value of their actions to the organization at large, justifying the initial and additional investments in analytical capabilities.

The systematic quantification and communication of business value support the development of a data-driven culture of inquiry.

There can be analytics where the benefits cannot be quantified. In this case, the qualitative benefits of these should be documented and recognized. The value of analytics that informs decisions not to change or take action should also be recognized.

Questions

• Does the analytics specification document mention use of KPIs (key performance indicators) to evaluate the analytics?
• Are KPIs, control groups, or other experimental methods routinely used to prove the incremental value of analytics against a benchmark?
• Do the control groups or other experimental methods in use successfully isolate the impact of analytic insights versus the impacts of other related business actions?
• Are the benefits of analytics communicated and understood by senior management?

Artifacts

• Analytics specification documents
• Evidence of the return on investment of analytics solutions
• Evidence of experimental design to evaluate the incremental value of analytics
• Post-implementation benefits evaluations
• Evidence of communication of analytics benefits to senior management

Scoring

Analytics must be an integrated part of the data ecosystem of the organization. It is dependent on data that often originates upstream in the organization and produces analytical products and services to support decision making within an organization. Alignment with Data Architecture (DA) capabilities is critical to ensure the reliability of data and assure confidence and trust in decisions taken based on them. Data lineage must be understood, approved business definitions must be referenced, and identification and classification standards must be followed. Consistent standards must be applied to the preparation of data for analytics.

Description

Knowledge about the lineage of data used in analytics provides a basis of trust for the analytical products and services they are used to create as well as the business decisions based on them. Data used in analytics must be clearly defined, referencing the organization’s business glossary and following the organization’s metadata standards. Authoritative data sources must be used to ensure completeness and fit-for-purpose quality. However, there will be cases where analytics will require data for which the authoritative source has yet to be agreed. Recording the use of such new or alternative sources is important to the integrity of Analytics. Data must be traceable from source to final product. Applied transformations or aggregations must be understood. If understanding is not possible, the lack of understanding should be highlighted, for example, in the case of ML. And the use of the transformation or aggregation in a model, calculation or visualization must be agreed by stakeholders to be reasonable and appropriate.

Objectives

• Document data sources and data analysis outputs by using the business glossary and metadata descriptions.
• Source data from authoritative data sources where relevant to ensure completeness and fit-for-purpose Data Quality (DQ).
• Understand data lineage from input or authoritative data source to final analytical product, including any transformations, calculations, and aggregations (or highlight where these are not known).
• Confirm stakeholder agreement that data is reasonable and appropriate for use in analytical products.

Advice

Throughout data analytics tools, models and processes, data lineage must be maintained. Where not possible, that fact must be highlighted. An acceptable data lineage includes metadata that provides a definition, description, or context for the data, including business, operational and technical characteristics. Data lineage must be established across platforms to ensure continuity of proper data usage throughout the data ecosystem (landing, staging, sandboxes, reports, and operational and analytical databases). Authoritative data sources should be leveraged where possible in data preparation.

Data lineage provides an audit trail that is key to post-decision defensibility of analytics and post-outcome accountability for business decisions. Data lineage should be part of any model, analytical method, or tool documentation and use the organization’s standards and processes. Analytics staff should be well-trained and versed in data lineage methods, processes, and documentation.

Data lineage solutions for analytics should be integrated with the overall lineage approach and tools of the organization that is established by the DA function.

Questions

• Are data sources and data analysis outputs well documented, and do they consistently reference business glossary and metadata descriptions?
• Is data sourced from authoritative data sources (where relevant)?
• Is data lineage well understood from input or authoritative data source to final analytical product?
• Does data lineage documentation include any transformations, calculations, and aggregations (or highlight where these are not known)?
• Are stakeholders in agreement that data is reasonable and appropriate for use in analytical products?

Artifacts

• Data lineage policy and procedures and systems that apply to data analytics tools and processes
• Analytical data architecture and data flow construct documentation
• Analytics documentation that includes data lineage
• Evidence of stakeholder agreement including formal sign-offs
• Data lineage training materials and evidence of competency

Scoring

Description

Clear, unambiguous business definitions are fundamental to trusted decision-making based on analytical products and services. Data consumed and produced by analytics must be mapped consistently to approved and understood business definitions.

In a mature data environment, business definitions are captured and maintained in authoritative sources (business and semantic glossaries), enriched by metadata and data lineage, and supported by data management policy and standards to ensure content quality, controlled access, and appropriate.

Objectives

• Use approved business definitions across all analytical products and services.
• Enable all Analytics processes and people to retrieve, understand, and trust the data they use based on clear business definitions.
• Drive accuracy, effectiveness, and efficiency of reporting by consistent usage of business definitions.
• Ensure Analytics stakeholder understanding of business definitions.

Advice

Confirm that the DMP has established authoritative, approved business glossaries. Reference to and use of the definitions in these glossaries by Analytics practitioners helps ensure that model inputs and results are understood consistently across Analytics and between Analytics and other areas of the organization.

Business glossaries should be accessible to Analytics practitioners and stakeholders through established data distribution services (see DCAM 4.1.4, Data Management is engaged in the definition and development of the organization-wide data distribution infrastructure). Controlled access to business glossaries enables approved definitions to be incorporated in model documentation and the visualization of results.

Employ feedback and contribution mechanisms for use by Analytics stakeholders. By providing feedback capabilities, stakeholders can clarify and improve definitions and propose new definitions for data created by analytics applications. Record Analytics’ interest in definitions as metadata to ensure they can be notified of definitional changes and address any impact on analytics products.

Discovery and experimentation may commence without a full and formal understanding of all business definitions, particularly where new data sets are involved. The organization should be clear where formalization of these is required before adoption into an analytical production environment, process, or product.

Questions

• Is an authoritative glossary of distinct and specific business definitions referenced and applied across all analytical products and services?
• Are all Analytics processes and people able to retrieve, understand, and trust the data they use based on clear business definitions?
• Do Analytics stakeholders have a common understanding of business definitions?

Artifacts

• Reference to authoritatively sourced business definitions in design documentation and analytical products and outputs
• List of Analytics stakeholders and evidence of bi-directional communication on business definitions
• Agreement on business definitions for data produced by analytics with verification by Analytics stakeholders
• Business definitions readily accessible to Analytics stakeholders
• Documentation evidencing linkage of business definitions in analytics to the organization’s metadata repository

Scoring

Description

Identification and classification standards organize data. They increase the ease with which data can be located and retrieved and secure integrity throughout the data analysis life cycle. Data classification should follow the organization’s data management, information security, and applicable regulatory and compliance standards. Analytics must respect these standards both as a consumer and producer of data.

Objectives

• Define clear roles and responsibilities, as both consumers and producers of data, for the identification and classification of analytical data and products in line with the organization’s data management, compliance, and information security policies.
• Safeguard confidentiality, integrity, and availability of data and analytical products throughout all analytics data lineage and analytics life cycle processes in line with the organization’s policies.
• Implement and carry data classifications throughout data processes, systems, and data analysis life cycles.
• Apply the identification and classification standards to data used or generated in analytics to make it easily searchable and trackable based on categories and criteria set by the organization.

Advice

Data identification and classification facilitates data discovery, querying, and retrieval. Data classifications and indexation can significantly improve data extraction process performance, storage, and encryption processes and facilitate the communication on data value and management within an organization.

The quantity and variety of data sets used and generated in analytics can create challenges for access and maintenance of data classifications. AI discovery techniques and automated systems that identify and classify data based on content, context, metadata, and patterns should be considered. Policies and procedures should be well defined yet straightforward enough for implementation and compliance by analysts.

The classification of data must include ethical, privacy, and regulatory practices.

Questions

• Are clear roles and responsibilities defined for both consumers and producers of data and the identification and classification of analytical data and products in line with the organization’s data management, metadata, compliance, and information security policies?
• Is confidentiality, integrity, and availability of data and analytical products safeguarded throughout all analytics data life cycle processes in line with the organization’s policies?
• Are data classifications implemented and carried throughout data processes (retrieval, transmission, copies), systems, and data analysis life cycles (i.e., sandbox, develop, test, pre-production, production)?
• Are identification and classification standards applied to data used or generated in analytics?

Artifacts

• RACI matrix that captures roles and responsibilities for data identification and classification in Analytics
• Evidence of the application of data classifications and indexes in analytical design documentation, testing documentation, and analytical output and products
• Audit trails and audit reports

Scoring

Description

Data preparation is the process of collecting, structuring, cleansing, and transforming data so it can be readily and accurately analyzed for business purposes. The data preparation process must be defined, and all process steps applied consistently to achieve fit-for-purpose data. Data preparation must be subject to the organization’s Data Governance (DG) policy and standards, including the use of the business glossary and metadata. The data must be from authoritative data sources, where relevant. Data preparation must be performed in a way that preserves data lineage and integrity.

Objectives

• Define standards for data preparation that include the identification of data required for the analysis, available authoritative data sources, and data accessibility. Also define standards for specification of data elements, DQ, need for data cleansing (including defect tracking and root cause fix) and conformance, transformation, and aggregation.
• Ensure data preparation follows the organization’s DM policy and DM standards and preserves data lineage and integrity.
• Create and maintain adequate documentation on data definitions, data sources and lineage, data usage, and data owners.
• Maximize the re-usability of any prepared data and data preparation processes to create efficiencies and time to market improvements for future data preparation needs.

Advice

Data preparation processes leverage authoritative data sources, business glossary, and metadata to provide accurate, well-defined data for consumption. When new data sets or data elements are sourced, documentation should be completed to ease reuse. A well-documented data catalog supports and significantly accelerates future data sourcing and wrangling processes.

Understanding DQ and determining whether the data is fit-for-purpose must be a key activity of the Analytics practitioners in the data preparation process. Data of higher quality can be re-used in other processes more readily. Profiling data can support data sourcing decisions. The profiling provides a statistics-based understanding of data content and DQ across multiple data sources.

Data preparation should be a repeatable process and a formalized best practice. Preference should be given to the use of self-service data preparation solutions versus spreadsheets. Spreadsheets are affordable but error-prone, difficult to control, and maintenance heavy. Data preparation tools can provide an effective and controlled data preparation process and the ability to integrate structured and unstructured data more efficiently. Strategies for agile creation of re-usable data sets should be considered to foster efficiency by reducing the need for internal approvals via obfuscation techniques.

Questions

• Do data preparation standards cover: the identification of data required for the analysis; available (authoritative) data sources; data accessibility; specification of data elements; DQ; data cleansing and conformance; transformation and aggregation?
• Does data preparation follow the organization’s DM policy and DM standards and preserve data lineage and integrity?
• Is adequate documentation maintained on data definitions, data sources, data lineage, data usage, and data owners?
• Are any prepared data or data preparation processes leveraged across multiple analytical processes, tools, or teams?

Artifacts

• Data preparation and analytics guidance and design documentation
• Evidence of adoption of data preparation standards by Analytics teams
• Data architecture and data lineage documentation
• Data definition, sourcing, lineage, usage and ownership documentation
• Evidence of re-use of data sets and data preparation processes

Scoring

Analytics must leverage the resources and activities of the Data Quality Management (DQM) function to understand the quality of the data input to analytics and ensure it is appropriate for the business purpose. DQM must be applied to data created by analytics and to any DQ issues uncovered.

Description

The business must define the quality of data required to support each analytics business case. The DQM function must be leveraged to understand the quality of the data sets used and created. Analytics must set and achieve the DQ requirements for consumed data.

Objectives

• Understand the quality of data required to meet the needs of the business case.
• Use DQM profiling and grading to understand the quality of the data used and created by analytics.
• Ensure the quality of data used and created is aligned to the requirements of the business case.
• Ensure prioritized data created by analytics is subject to DQM.

Advice

Analytics professionals may identify and accept differing DQ levels. For example, a business analyst that generates a report may not identify nor be concerned with some nulls in a data set. In contrast, a data scientist may not only discover them but need to address the nulls before building a model.

The acceptable level of DQ will differ by business case. When the decision based on the analytic output is higher risk, higher DQ thresholds may need to be met before proceeding. Analysis conducted on poor quality data is likely to be wrong. Low DQ also increases costs, risks, and the likelihood of ethics issues in the analysis. Analytics professionals must coordinate with the data owners to discuss requirements and DQ criteria. Data owners remain accountable for monitoring DQ and implementing changes, but Analytics practitioners should also contribute to improving DQ. There must be consistent communication among data owners and Analytics practitioners to preserve a consistent DQ framework and a consistent toolset.

Analytics practitioners should document findings, adjustments, assumptions, and decisions. Consideration should also be given to how the data was collected and stored before data analysis took place. Analytics practitioners must share their findings with the data owners as the findings often warrant a change to business processes or business decisions and business rules. Suitable processes must be in place to ensure that data generated as part of the Analytics process comes within the scope of DQM.

Questions

• Are the DQ requirements of business cases defined?
• Do Analytics practitioners reference DQ metrics?
• Is the quality of data assessed against the requirements of the business use case?
• Is business justification provided to support the prioritization of DQ improvement?
• Are data quality rules defined and communicated for data created by analytics?
• How do Analytics practitioners assess, document, and communicate DQ findings?

Artifacts

• DQ requirements for analytics business cases
• Evidence of assessment of DQ against DQ requirements and criteria
• Documented DQ processes for Analytics practitioners
• Quality rules for data produced by analytics
• Evidence of data produced by analytics being in scope of DQM

Scoring

Description

The DQM framework must support DQ issue management on analytics data sets so that the data user can understand the quality of the data being prepared and report DQ issues identified during preparation. Adoption of and integration with the existing DQM framework is required to ensure that all data is managed at its source, regardless of whether it originated inside or outside the data analytics environment.

Objectives

• Ensure the DQM framework is understood within Analytics.
• Provide DQ metrics for data used in the analytics environment.
• Identify the techniques and tooling required for analytics DQ issue reporting and management.
• Integrate the reporting and management of analytics DQ issues into the DQM framework.

Advice

DQM and Analytics must work together to ensure the analytics requirements are addressed in the DQ framework. Analytics stakeholders play a contributory role in DQM. As consumers and users of data, they are accountable for ensuring that the data being used in analytics are of a quality fit for the intended purpose. If data issues are discovered, analytic stakeholders must provide feedback to the source systems and contribute to the root-cause analysis. A clearly understood process for logging and prioritizing DQ issues must be developed and socialized.

Analytics practitioners leverage a variety of data types with an increased number of variables and a higher volume of data than ever before. This environment creates challenges in determining what data must be monitored and assessed as well as how the DQM framework should be applied. DQM must be able to evaluate the DQ needs of Analytics and propose appropriate, cost-effective solutions. DQM professionals should be open to new tools and techniques while also remaining cognizant of industry trends and best practices.

As an organization expands its use of analytics, DQM teams need to evolve in support of this. The team should assess its people, processes, and technology and share these findings with stakeholders to gain support for training, transformation, and tools. This assessment, evaluation, and implementation may need to occur faster than ever before, given the constant advancement within the analytics discipline.

Questions

• Is there a funded education and communication plan to ensure Analytics understands the DQM framework?
• What are the data quality dimensions for data used by Analytics practitioners? Do these need to be adjusted?
• What tools are available to measure DQ?
• How are DQ issues communicated between Analytics and the data producer?
• How has DQ improved over time?
• Is the data producer able to successfully implement changes within the timeframe required by Analytics?
• Is there an assessment of the cost of potential DQ remediation to weigh against the potential benefits to be derived from the analytics?

Artifacts

• A learning and development plan and supporting materials to educate Analytics practitioners on the DQM framework
• A communication plan and supporting material to convey the DQM framework to Analytics practitioners
• Evidence of DQ issues being reported by Analytics practitioners
• DQM team skills assessment and training
• DQM tools assessment
• DQM process reviews
• Evidence of collaboration between DQM team and Analytics

Scoring

For Analytics to be effective and efficient, it must be supported by a platform that is designed and implemented to meet its needs. The operating model drives many of these needs. The different requirements of production and non-production environments must be addressed, and there must be a version control regime for models that is appropriate for each of these environments. Strategies for obfuscation of sensitive data are required to maximize the reusability of data sets. The platform should provide appropriate flexibility to scale up and down.

Description

The analytics platform is a combination of tools, applications, and infrastructure that enables analytics to be created and executed in an organization. It must have the necessary capabilities to support the way that Analytics teams are structured and operate, and the way they engage with the business and stakeholders. It must enable Analytics to develop, test, govern, socialize, maintain, and mature analytical models. The platform must be flexible enough to allow a variety of individuals to interact appropriately with the system. These interactions must implement any required segregation of duties and the ability to set access rights to different users of the system. Beyond the Analytics teams, business users will need to validate and utilize the results of the analytics models.

Objectives

• Understand the platform requirements of the different roles defined in the operating model.
• Obtain stakeholder agreement to the requirements to be supported.
• Ensure the platform design takes account of the agreed requirements.
• Ensure the platform supports the necessary and appropriate enforcement of access rights.

Advice

The functional and non-function requirements to support the Analytics operating model must be understood before the procurement or development of the platform commences. Any pre-existing infrastructure and solutions should not constrain the requirements.

Facilitation of segregation of duties and support and control of different levels of access must be part of the design. The platform design should ensure ease of access and use. It should support the need for platform users to understand both the data being used and the results being produced.

Questions

• Have the platform requirements of different roles described in the operating model been defined?
• Have platform requirements been discussed and agreed with business and Analytics stakeholders?
• Are the requirements understood by those responsible for the procurement and implementation of the platform?
• Has the platform design been reviewed to confirm it addresses the agreed requirements?
• Has platform support of enforcement of access rights been validated?
• Can stakeholders easily review the outputs of the platform?
• Can the platform support users beyond the data analysts who work to produce the outputs?

Artifacts

• Documented Analytics operating model
• Documented assessment of the platform design support for the operating model
• Policies relating to segregation of duties and evidence of their review and approval
• Evidence of bi-directional communication with the business on the use of the platform

Scoring

Description

A separate environment is needed for data discovery and testing before delivering models into production. Testing in this low risk, sandbox environment ensures appropriate testing takes place before the model is used and relied upon by the organization. The sandbox must be provided appropriate capacity for its computational requirements determined by the business needs.

Objectives

• Define and support the requirements for flexibility and agility of innovation environments.
• Define and support the requirements to segregate the development and testing sandbox from the production environment.
• Establish distinct design processes for the sandbox, development, and production aspects of the platform.
• Establish distinct change control processes for the sandbox, development, and production environments.

Advice

The nature of analytics is trial and error, so development environments need a degree of agility that supports the ability to fail fast and fail-safe. Non-production environments, especially a sandbox/innovation environment, need a greater level of flexibility made possible by a sufficiently agile change management process that can address multiple scenarios.

There must be clear segregation of environments to ensure the change management process is appropriate for each level. It is change management that enables the Analytics teams to be productive. Change management processes for sandbox/innovation environments must be flexible enough to support adequate turnaround times for experimentation. A rigidly separate environment must be provided for non-production model testing, with clear demarcation separating it from production. The change management process for production must be appropriately robust in comparison to the more flexible non-production environments.

Questions

• Are there documented requirements distinguishing the sandbox/innovation environments and production environments?
• Are the memory, storage, and processing capabilities of the different environments documented?
• Has the business approved that the environments meet their requirements?
• Has the design process for each environment been assessed and put into operation?
• Are different change control processes in place for each environment to support the business requirements and risk appetite of the organization?

Artifacts

• Non-production environments strategy
• Documented designs of sandbox, development and production environments
• Documented change control processes
• Evidence of engagement with stakeholders of policy review/implementation
• Evidence of communication of strategy and policies

Scoring

Description

There must be controlled management of change to models developed within the analytics platform and a documented process for recording the changes. Effective governance and change control of the models must be in place and must be auditable.

Objectives

• Ensure changes to the analytics models are made with appropriate authorization.
• Ensure changes are tracked, and the nature of each change is documented and auditable.
• Ensure models are released into the production environment only through the controlled process of the appropriate test environments release mechanisms.
• Ensure appropriate versioning and archiving are in place for data sets used to train and validate models.
• Address requirements to re-run analytics on historical data with the version of a model used at that point in time.

Advice

Analytical models should follow the change control process and the organization’s software development life cycle process. Seek concurrence through discussion with the Technology function, the change management function, and connected business functions. Maintain documentation that explains changes made to models. This documentation must trace version history to know which versions of which models are used in which implementations. Document the process for the creation and storage of retrievable backups of previous versions of code and data sets. These backup files are needed in the event of errors in the upgrade or in case the analysis provided by previous versions needs to be validated.

Questions

• Is there a change control process for amendments to analytics models on the platform?
• Is the defined change control process in place and understood by the responsible individuals within the company?
• Does documentation exist to show and explain the changes made to models?
• Are previous code versions and data sets available to ensure understanding of previous model analyses?
• Does the change control process ensure appropriate authorization for changes?

Artifacts

• Model specifications
• Policies and procedures associated with version control for models and data sets
• Model testing, approval and review procedures
• Model release protocols and policies
• Evidence of model review and approval

Scoring

Description

Data obfuscation strategies can help to ensure that data used during the development phases of models comply with the appropriate regulatory regimes governing the organization and best practices regarding data privacy. Data obfuscation also applies to commercially sensitive confidential data, such as company results before their release. The obfuscation capability is important for non-production testing environments and, in some cases, can also be relevant to production environments. Where external professional testers have access to test environments, non-production environments must store the minimum amount of personally identifiable or commercially sensitive information.

Objectives

• Create a strategy or policy to understand what data is classed as commercially sensitive or personally identifiable information (PII), using the organization’s data classification system to help determine the level and sophistication of obfuscation that is required.
• Establish techniques to obfuscate data, to mask or blur it, as required by the business.

Advice

Identify the data classifications that determine what data is commercially sensitive or classed as PII (according to the relevant regulatory framework). Those classifications will drive the level of obfuscation or deletion that must be applied to the data. Confirm whether data obfuscation is needed for production and non-production models and whether it applies to model outputs as well as inputs. Governance of the appropriate application of obfuscation is covered in DCAM 8.6.3 – “Model approval and release is aligned with privacy governance.”

Questions

• Are data obfuscation requirements understood?
• Is there a process documented and followed to ensure regulatory compliance in the use of personal data and commercially sensitive data?
• If personal data needs to be used to ensure accurate reporting and prediction optimization, is there a process to ensure that the personal data utilized is obscured or deleted after use?
• Have obfuscation techniques been established and adopted?

Artifacts

• Evidence of data anonymization policies, referencing data types and detailing specific requirements for production and non-production environments
• Non-production environments strategy including the management of data replication across environments
• List of stakeholders and evidence of bi-directional communication and recognition of relevant policies
• Documentation of obfuscation techniques with guidance on how and when to apply them
• Evidence of data obfuscation/anonymization of relevant models/model outputs

Scoring

Description

Business requirements change rapidly. The analytics model environment must have the flexibility to cope with new data without requiring significant redesign. The model environment needs to be able to provide an increase in data computing power requirements to address forecasted business growth and the growing sophistication of models. The costs of computational power need to be tracked. There must be an understanding of how additional requirements could be accommodated, and potential costs must be estimated in advance.

Objectives

• Understand the processing power capacity and flexibility potentially needed for the analytics model environment.
• Support the ability to scale up and down to accommodate planned requirements.
• Establish a mechanism for estimating and tracking model processing costs.

Advice

A process should be established for stakeholders to provide direction on what future requirements the analytics models may be required to support. These requirements provide input for capacity planning. They should include data volumes and an estimate of the analytical workloads. The scalability requirements should be reflected in the technology roadmap for Analytics. They should establish a position for the use of both on-premise and cloud infrastructure as appropriate for the organization.

The costs of the analytics platform should be tracked and reviewed against expectations by the stakeholders to ensure the environments are sized appropriately. Assess and quantify the business benefits obtained from the model outputs to determine if the costs of the models and processing and the costs of idle capacity that supports scalability are justified.

Questions

• Is there business direction on future requirements and strategy that the analytics models are required to support?
• Are future data volumes and workloads understood and quantified?
• Are volume and workload requirements documented and reviewed by Analytics?
• Can the models accommodate increased data volumes or variables without requiring a complete redesign and the related costs?
• Are the costs of the analytics platform tracked and reviewed against expectations to ensure that environments are sized appropriately?
• Does the assessment include the costs of idle capacity?

Artifacts

• Documentation evidencing production and non-production low-level non-functional requirements
• Non-production environments strategy
• Cost tracking of both non-production and production environments, including any utility computing
• Evidence of spending reviews and ROI for production and non-production environments

Scoring

While some analytics activities will be exploratory or one-off, Analytics must be able to deploy models into production in a controlled and governed manner. Testing, approval, and release processes are central to this, along with processes for regular review of deployed models. These must be aligned with the organization’s governance approaches for data ethics and privacy. Requirements to understand and control model bias must be addressed, as must the need to be able to explain how model decisions have been reached.

Description

Before release into an operational environment, models must be tested to ensure that they are performing as expected and in consonance with all model specifications. The satisfactory outcome of testing and approval for release should be formalized by the official designated for this purpose. The model should be released following established release protocols. The performance of the model may change as ingested data changes over time, potentially affecting the efficacy of the model’s original intent. Its performance and its continued adherence to model specifications must be reviewed periodically, and any time there is a change to model specifications.

Objectives

• Define formal procedures for model testing, approval, release, and periodic review.
• Ensure the authority for model approval is clear and appropriate.
• Obtain stakeholder approval and support to enforce testing, approval, release, and review procedures.
• Ensure that models released or pending release into production environments are performing as expected and in consonance with each model’s specifications.
• Establish safeguards to ensure model release into production minimizes disruption to the organization’s operations and protects against data breaches and corruption of data.

Advice

As part of the model testing, a wide range of inputs must be run through the model to ensure that it is performing as expected. Such inputs sets may include back-test data, current data, and artificially created representative samples of input data that could theoretically, even if rarely, arise in operations. Model outputs should be checked for any unintended consequence arising from using the model, such as unfair bias or undesirable business outcomes. Model testing and controls should be automated where possible.

The individuals authorized to approve a model should have input to the specifications of the portfolio of evidence to be provided for review in the approval. They should have an understanding of how the model works and the business objectives it is designed to meet. They should also have a broader awareness of the overall business function context in which the model will operate and of governance and ethical parameters.

Questions

• Do formal procedures exist for model testing, approval, release, and periodic review?
• Is the authority for model approval designated at the appropriate level of seniority?
• Have stakeholders approved the enforcement of the procedures for testing, approval, release, and review?
• Do models perform as expected and in consonance with their specifications?
• Have safeguards been established to ensure model release minimizes disruption to operations and protects against data corruption and breaches?

Artifacts

• Model specifications
• Model testing, approval and review procedures
• Evidence of automation of model tests
• Model release protocols and policies
• Details of model input datasets and outputs
• Schedule of periodic model reviews
• Evidence of model review and approval
• List of stakeholders and evidence of approval of enforcement of procedures

Scoring

Description

The data ethics oversight function should outline guidance and the requirements to operationalize models. These requirements and the Code of Data Ethics must be followed and monitored during the model development, approval, and release process.

Objectives

• Align model development, approval, and release processes with the data ethics governance structure and guidance.
• Assign data ethics accountability for models during approval and release processes.
• Review and mitigate any weaknesses in the model approval and release process with the data ethics oversight function
• Involve analytics experts in the Code of Data Ethics development and continuous review.
• Execute ethical management.

Advice

Consideration of data ethics needs to be embedded in the end-to-end process of model development and management. Organizations should have policies, processes, and controls in place to document the ethical consideration as part of the approval and release process. Organizations should retrospectively embed similar policies retrospectively in existing models and any post operationalization monitoring process. This sub-capability is dependent on capability 6.6 – “Govern the Data Ethics.”

Model approval and release must take into account the data ethics considerations. These include the credible, responsible, and traceable sourcing and safe sharing of data. The customer’s best interests are paramount when the organization uses, models, and transforms data in ways that make the handling of data processed by the model explainable and transparent.

As data ethics is an emerging practice, there should be continuous dialogue between analytics experts, associated risk teams, and the data ethics oversight function to review and update requirements and guidance to ensure they are up to date.

Questions

• Has the data ethics oversight function provided guidance and requirements outlining how models adhere to data ethics?
• Is there a clear process in place for the ongoing monitoring of models to ensure they continue to meet the guidelines and requirements of the data ethics oversight function?
• Is the accountability for data ethics during the model approval and release process at an appropriate level of seniority?
• Is there a process in place to ensure a formal Code of Data Ethics and associated guidance are reviewed and remain up to date?

Artifacts

• Evidence of compliance with data ethics oversight function’s guidance
• Evidence of data ethics assessment included in the model review and approval policy
• Schedule of periodic reviews of the alignment of model development, approval and release processes with data ethics governance
• List of stakeholders and evidence of communication

Scoring

Description

The model approval and model release processes should include explicit checks and confirmation that they will be aligned with the organization’s privacy governance requirements.

Objectives

• Ensure privacy governance requirements are understood by those involved in model approval and release.
• Align model development, approval, and release processes with the privacy governance structure and guidance.
• Assign accountability for privacy compliance of models during approval and release.
• Engage the privacy oversight function in regular reviews of the effectiveness of the model approval and release process.
• Ensure that, where appropriate, models allow authorized third-party access to the data used to reverse engineer a decision to original, preprocessing data.

Advice

Compliance of a model with privacy requirements must be understood and confirmed before a model is released, but should already be taken into account at the design stage. Models and the governance principles are tightly interrelated, and compliance must be reconfirmed every time a model changes or the compliance rules are amended.

Questions

• Has the data privacy oversight function provided guidance and requirements outlining how models adhere to privacy requirements?
• Is there a model compliance review process that includes the privacy requirements before a model is released, or before changes to a model are released?
• Is there a process in place to review all models in production when the privacy requirements change?

Artifacts

• Model specifications with a description of personally identifiable information used and details of how it is protected
• Privacy governance controls and guidance
• Evidence of privacy review and history for all model releases and updates
• Evidence of privacy review and history for all updates of privacy requirements

Scoring

Description

Analytics stakeholders need to be aware of any prejudices and unfairness of models and any unintended consequences they may cause. This type of model bias is a governance issue and must be addressed with appropriate checks and balances that include awareness, mitigation, and controls. Models must be subject to active management of bias that provides for ongoing assessment and validation of algorithms, data sets, and model outcomes.

Objectives

• Establish procedures and controls to ensure that input data sets do not introduce model bias.
• Ensure that model bias and its effects are identified.
• Establish procedures for addressing model bias once identified.
• Ensure that the shortcomings of algorithms are understood, and they are not applied to questions where answers will be invalidated by algorithmic bias.

Advice

Analytics model results are based on the data inputs to the model, either when created and trained or when running in production. Bias arises when that data is not representative of the real world, whether missing key variables that would lead to different decisions or including human-produced content that incorporates biases of those persons. To minimize the harmful effects of bias, stakeholders need to be aware of the possible bias in any model. They must understand the various types of bias and how each can impact data, analysis, and decisions. Identification and management of bias must be incorporated in the formal procedures for model approval and regular review. Regular reviews of models must include monitoring for sudden or creeping bias being introduced as the models are exposed to new data in production. Many organizations are establishing Data Ethics committees, charged with reviewing models and their outcomes, for signs of intended or unintended bias.

Questions

• Are stakeholders aware of the potential sources of model bias?
• Is the analysis of model bias included in the analytics methodology and model documentation standard?
• Are there procedures and controls to address bias in model input data?
• Are model shortcomings documented and considered in decisions on the application of the model?
• Is model bias addressed in procedures of model approval and regular review?
• Are judgment-based decisions on model bias risk made at an appropriate level of seniority?

Artifacts

• Stakeholder education material on model bias
• Procedures for the analysis of model bias
• Procedures and controls for review of bias in input data sets
• Documented analysis of model bias
• Documentation of model shortcomings and limitations
• Evidence of model bias and shortcomings being considered in model deployment decisions

Scoring

Description

Requirements for explaining how a model works and reaches its outcomes should be precise and must be incorporated into the modeling process and the operational processes that support the models. Transparency of how a model works is critical to recognizing bias and aligning business processes and activities to achieve non-biased outcomes.

Objectives

• Define requirements for model explainability in business terms.
• Assign accountability for model explainability during the requirements phase of the modeling process.
• Perform additional steps to mitigate ambiguity in requirements for model explainability.
• Incorporate processes to ensure requirements for model explainability result in a clear understanding of how a model works and model transparency.

Advice

Model explainability requirements must specify how easily models can be understood and how easy it must be to understand the cause of decisions in a model. The goal is for stakeholders to be able to explain what models do and how they do it, both to internal and to external audiences. Model explainability must be able to stand up to internal audit or external regulatory reviews. Explanations should be consistent.

Identify and consult the stakeholders who can understand and discuss the requirements for model explainability. Include requirements for model explainability in the organization’s analytics methodology and model documentation standard. Embedding standards into the fabric of an organization is a critical step in achieving model explainability.

Update the organization’s processes, standards, and policies as the adoption of requirements for model explainability increases. Keep documentation and processes current. The alternative results in tribal knowledge, operational siloes, and increased risk that intellectual property and knowledge are lost, making model explainability more difficult.

Establish methods to monitor and continuously improve model explainability. The goal of continuous improvement is to provide opportunities for stakeholders to enhance model explainability to meet business objectives. Sources of improvement ideas include surveys, interviews and peer reviews, success stories, and analysis of gaps highlighted by monitoring and measurement. Bring new ideas and thinking into the organization by keeping pace with emerging and waning industry standards, technology improvements, and by comparing processes and outcomes to those of competitors, allies, and other industries.

Questions

• Are models explainable in business terms?
• Can the requirements for model explainability be traced to business processes, activities, and outcomes?
• Is there evidence of roles and responsibilities?
• Was accountability assigned during the requirements phase of the modeling process?
• Have steps or processes been identified to mitigate ambiguity?
• Are models easily interpreted?
• Is it easy to understand the basis of decisions in the models?

Artifacts

• Policies for requirements for model explainability including the types of model acceptable for business use cases
• Steps to mitigate ambiguity, both demonstrated and documented or observed
• Requirements for model explainability documented in analytics methodology and model documentation standard
• Minutes from peer-review sessions of model explainability
• Documentation of rationale for model choice
• Analytics practitioner role definitions that include responsibilities relating to model explainability
• Evidence of monitoring and continuous improvement efforts for model explainability

Scoring

Analytics practitioners require specialized skills. An education program must be in place to address any skills gaps identified. Effective Analytics will also require behavioral and cultural change across the organization.

Description

Unique cultural and behavioral changes must be identified and adopted for each organization to produce value from Analytics capabilities. Once defined and agreed, culture and behaviors should be measured at specific points throughout the journey.

Objectives

• Identify the behaviors that exist in the organization today that impact the effectiveness of Analytics.
• Define the behaviors needed to realize value from data and analytics.
• Create a list of behavioral recommendations.
• Develop a backlog of ideas, solutions, and concepts for adoption based on behavioral recommendations.
• Measure the behaviors at key milestones.

Advice

A combination of data, behavioral insights, and human-centered design needs to be applied to determine the behaviors required to enable an Analytics culture. Define the future behaviors necessary to enable the business and data strategy of the organization to realize the desired value. Use engaging discovery methods to obtain an understanding of the current behaviors prevalent in the organization.

Validate the observed and desired behaviors needed for the Analytics culture with other stakeholder groups to ensure they align with the organizational culture. These could include behaviors showing collaboration, positive attitudes towards experimentation and understanding of failure as part of learning, innovation, a culture of inquiry, etc. All of these can potentially be harnessed to deliver business value.

Questions

• Is there existing data for how people in the organization think, feel, and act in relation to Analytics?
• Does the organization understand how people apply analytics in the flow of their daily work?
• Have the desired behaviors to create value from analytics been defined, documented, and communicated?
• Have stakeholders validated the desired behaviors for the Analytics culture?

Artifacts

• Evidence of stakeholder engagement in the definition of desired behaviors
• Documentation of existing and desired behaviors
• Backlog of potential ideas, solutions, and concepts
• Results of assessments and measurement of behaviors

Scoring

Description

The required behaviors have been identified. The organization must conceive and create prototype initiatives that encourage and drive the desired behaviors. Once the value of those initiatives is proven, they must be embedded and sustained as ongoing activities.

Objectives

• Create prototypes of potential concepts, solutions, and initiatives that address culture gaps.
• Demonstrate business value by documenting that the initiatives lead to desired changes in behavior.
• Enable the adoption of cultural change initiatives as a business-as-usual activity.
• Measure the behaviors at key milestones.
• Ensure the Analytics culture takes account of the broader cultural drivers of the organization, such as confidentiality, risk appetite, innovation, and ethics.

Advice

A backlog of potential solutions (as identified in DCAM 8.7.1) is a prerequisite to embed initiatives that will address culture gaps. The solutions must be prototyped and tested to determine if they lead to the desired behavior changes. Those with a positive impact should be prioritized and refined for further rollout.

Work with the Analytics community to define the KPIs associated with the behaviors and build the business case for implementation of the solution as a business-as-usual activity. Obtain buy-in and sponsorship to drive the data and Analytics culture initiatives. Following implementation, measure the change in behavior and its impact at regular milestones.

Engage HR to ensure the Analytics culture is aligned with the broader organization and work with the organization’s Learning and Development team, if available, to co-develop education initiatives. Consider how communication of Analytics success stories can support cultural objectives.

Questions

• Have the solutions in the backlog been assigned relative priorities?
• Are the initiatives embedded in daily activities?
• Does a process exist to determine if new behaviors are sustainable and adding business value?
• Has a process to measure behaviors over time been defined?

Artifacts

• Education and behavioral initiatives roadmaps
• Results and insights from prototype solutions
• A documented process to determine sustainability and business value of new behaviors
• Evidence of behavioral measurement

Scoring

Description

The skills required to perform the Analytics practitioner roles and responsibilities are understood, and a learning map has been created to develop these skills.

Objectives

• Identify and align the skills required to fulfill roles and responsibilities at different levels.
• Create a learning map to provide Analytics practitioners with the right skills and tools to carry out their activities.
• Review and confirm that the learning map fulfills role definitions.
• Embed a process to refresh the learning map based on industry standards and developments.

Advice

Analytics practitioners need to be provided with the right education and training to build their skill set to maximize the value an organization gets from its Analytics capability. When developing the learning map, Analytics practitioners across all levels should be involved to ensure all the required skills are represented across Analytics. Roles and responsibilities should be used as a starting point to identify the skills and tools that will enable practitioners to perform these effectively. A current state assessment of existing skills, tools, and training should be performed to identify any gaps which should then be addressed within the learning map.

The learning map should be designed to provide Analytics practitioners with both practical and theoretical experiences. Consider experiences from structured learning, social learning, and experiential learning activities. The various experiences should be built into the learning map with enough time committed to each and delivery roadmap clear with milestones to monitor learning progress. The learning map must be refreshed at intervals aligned with the operating model and the learning experiences updated to reflect industry developments and best practices.

Questions

• Have skills requirements across Analytics practitioner levels been defined and agreed?
• Have stakeholders reviewed and approved the learning map?
• Does the learning map fulfill role definitions across all roles and levels?
• Do the learning maps cover all types of learning?
• Has a process to refresh the learning map been defined?

Artifacts

• Documented roles and responsibilities of Analytics practitioners
• Documented skills requirements and needs of Analytics practitioners
• Learning maps for each role type and level
• List of stakeholders and evidence of bi-directional communication and approval of the learning map
• Assessment of existing tools and training
• Documented process for regular review of learning maps

Scoring

Description

The education initiatives for Analytics practitioners are established with a process for continuous improvement. The success of the education initiatives must be measurable and monitored to ensure skills gaps are addressed on an ongoing basis.

Objectives

• Perform gap analysis to identify skills gaps for Analytics practitioners across levels.
• Design learning experiences based on the defined learning experience needs.
• Define continuously updated roadmaps to deliver ongoing education initiatives across practitioner levels.
• Embed a process to monitor learning progress and for continuous improvement.

Advice

The learning maps prescribed in DCAM 8.7.3 should be used as requirements for developing the education initiatives, centering the learning experience around the needs of Analytics practitioners. A gap analysis across Analytics should be performed to identify existing skills gaps and common challenges, which should drive the focus of the learning experiences. At this point, it should be considered whether hiring may be appropriate to address skills gaps, depending on the size of the gap and the degree of specialization of the skill. For example, it may be optimal to hire practitioners with particular skill sets to meet a new opportunity or act as a catalyst for raising other team members' skill levels.

The learning experiences should be designed to equip Analytics practitioners with both practical and theoretical backgrounds, combining structured learning, social learning, and experiential learning. Engaging experiences based on real-life scenarios should be developed to put new skills and tools to use in a simulated, safe environment. An example is to use gamification to motivate and engage practitioners to make the learning experience as useful as possible. The development of non-technical skills should also be considered, along with the rotation of people between Analytics and business functions to create a well-rounded skill set.

A roadmap to deliver the learning experience should be created within an iterative approach and a feedback mechanism to ensure education initiatives are effective. Embedded measures of success should be tracked with learning experiences that are continuously improved and updated with industry developments.

Questions

• Has a gap analysis to identify skills gaps been performed?
• Is a plan to resolve skills gaps in place?
• Has the education initiative been designed to provide practical and theoretical experiences?
• Has a learning experience roadmap been defined?
• Have success metrics been approved?
• Is a feedback mechanism in place?

Artifacts

• Skills gap analysis
• Backlog of learning experiences
• Learning experience delivery roadmap
• Education initiative success metrics
• Documented process for monitoring and improving learning progress

Scoring