Program and Project Management Assessment Report

MNsure Phase II Project
Deliverable #2 – Program and Project

Management Assessment
July 11, 2014
MNsure’s Accessibility & Equal Opportunity (AEO) office can provide this information in accessible formats for individuals with disabilities. Additionally, the AEO office
can provide information on disability rights and protections to access MNsure programs. The AEO office can be reached via 1-855-3MNSURE (1-855-366-7873) or
[email protected].
Document Control Information
Document Information
Document Name Deliverable #2- Program and Project Management Assessment

Project Name MNsure Phase II Project
Client State of Minnesota
Document Author Deloitte Consulting LLP
Document Version 2.0 FINAL
Document Status FINAL
Date Release 07/11/2014
Document Edit History
Version Date Additions/Modifications Prepared/Revised by
1.0 6/03/2014 Draft Deloitte Consulting LLP

Incorporated Feedback from MNsure, DHS, and MN.IT
2.0 7/11/2014 Deloitte Consulting LLP
stakeholders
-2-
Table of Contents
Sections Page Numbers

Lead Vendor Project Background 4–5
Executive Summary 6–8
Approach and Scope 9 – 11
Governance, Decision Making, and Accountability 12 – 28
Immediate Key Resource Needs 29 – 30
Communication and Information Flow 31 – 36
Status Reporting 37 – 42
Risk and Issue Management 43 – 47
Change Control 48 – 53
Defect Management 54 – 68
Test Management 69 – 83
Release Management 84 – 93
Appendices
Appendix A: Project Management Processes and Tools 94 – 115
Appendix B: Roles and Responsibilities
Appendix C: Interviews Conducted
-3-
Lead Vendor Project
Background
Lead Vendor Project Background
Project Background and Objective

Deloitte Consulting LLP (Deloitte) was engaged by the State of Minnesota to assess, identify potential impacts and provide
recommendations for the State’s consideration on the go-forward strategy for ongoing operations, 2015 open enrollment and beyond
Project Scope
1. Conduct an assessment of governance structure, decision-making processes, program and project management practices
and provide recommendations for consideration to implement governance structure, program and project management
controls and oversight
2. Conduct an assessment of the current state of the MNsure system from functional and technical perspective and provide
recommendations for consideration for the short-, mid-, and long-term
3. Perform the following project activities:
 Program and Project Management
 Project Planning
 Functional and Technical Systems Assessment
 Release Management Scope of this
 Defect and Issue Tracking deliverable
 Leadership and Planning of User Acceptance Testing (UAT)
Project Deliverables
Deloitte is contracted to produce five deliverables:
1. Report and reconciliation matrix of current status of Deliverables across existing vendor agreements
2. Project Management Analysis and Considerations Report
3. Phase 1 Functional and Technical Assessment Report with a categorization of key functional and system gaps and
considerations for a near-term system roadmap
4. Application Project Work Plan
5. Phase 2 Functional and Technical Assessment Report with a categorization of key functional and system gaps and
considerations for a mid-term and long-term
The focus of this deliverable is the Project Management Analysis and Considerations Report
-5-
Executive Summary
Executive Summary
Deloitte Consulting LLP (Deloitte) was engaged to conduct an assessment of the governance structure, accountability and
decision making, project management controls and software development lifecycle (SDLC) phases of testing, defect, and
release management. This assessment focused on identifying considerations for the State for the immediate term (calendar
year 2014) and for a sustainable project management structure and lifecycle.
During the Fall of 2013, much of the State’s efforts were focused on addressing issues that arose at the time of initial open
enrollment (October 1, 2013). During this period, we understand the governance and project management processes for the
project became less effective and resulted in a lack of coordination, integration and decision-making across the project teams
and stakeholders. Recognizing these challenges in early 2014, the State began to refresh efforts to reinstate its governing and
project management processes it had instituted at the outset of the project.
Deloitte identified observations, impacts and considerations in the following areas: (1) Governance; (2) Communication and
information flow; (3) Status reporting; (4) Risk management; (5) Issue management; (6) Change control; (7) Defect
management; (8) Testing management and (9) Release management. For each of the areas, the overall maturity of the
process/area was assessed against Deloitte’s proprietary project management methodology.
Governance: While positive efforts were noted in the reestablishment of a model of governance and related processes earlier
this year, their effectiveness remain diluted for a variety of structural, procedural, role definition, decision-making and
accountability challenges. The cumulative effect has been to create confusion among most leads and stakeholders, inconsistent
adherence to processes, untimely decision making and issue resolution. In addition to streamlining project execution
responsibility under a new Project Director role (within the Minnesota IT organization and has day to day responsibility for the
MNsure IT system project), the full establishment of a MN.IT MNsure Project Management Office (PMO), empowerment and
staffing of all governance bodies (including Change Control Board) was identified.
Prioritization of key tasks, activities and decisions made, need to be documented, communicated, and not revisited or changed.
MNsure IT system project work needs to be documented in an integrated project work plan to include testing and release
management activities built into the approach. Clarity of roles and establishing measurable accountability are key takeaways of
the observations.
-7-
Executive Summary (cont.)
Project management processes: Reconstitution of many critical project management processes were evident, however many
of these processes lacked consistency in operation and varied in maturity. The primary impacts to project effectiveness are
concentrated in a number of areas that are prioritized below:
 It was observed that testing of complex functionality (such as batches, interfaces and notices) often occurs directly in the live
production environment - where actual user processing occurs. This specific functionality was not tested with State
involvement, and broader testing was limited or entirely missing prior to promotion to production. The State needs to address
the barriers preventing thorough testing in the lower (earlier) environments. In addition to significant disruption risk to t he
production environment, the cost of remediating an issue found in production is generally significantly more costly than when
found much earlier in the testing cycle.
 Controls for risk, issue, and decision management (including logs and spreadsheets) are available for the project but there is
not active or consolidated management of these logs. Specifically, prioritization of risks and issues at an appropriate level does
not occur, nor does timely decision making occur. This can lead to issues, risks, and decisions not being fully understood,
communicated, or acted upon with the appropriate degree of prioritization.
 Tracking and timely reporting of current and cumulative project status is critical to understanding where the project stands at
any point in time and thereby allowing leadership to respond to issues, unplanned events, and resource impacts in particular.
Comprehensive status reporting for the project was not timely, consistent or fully representative of all IT vendor partners a nd
agency groups.
 System defects do not appear to be comprehensively captured, resulting in a far lower number of total open defects. Initial
reports showed only 60-162 total open defects. Upon follow-up and detailed analysis, 399 total open defects were identified.
The defect types are split roughly in half between product and functional issues, have been identified in the production
environment and fixes pending to be delivered by vendors were identified in lower environments. The State should validate and
confirm that this is the exhaustive list of defects and one system should be used to track and manage all defects. The non -
capture and active management of system defects will challenge system improvement efforts and may pose additional financial
burden on the State.
While our observations are pervasive across the governance model and project management processes – addressing these
needs with a positive impact to project momentum can usually be achieved in a short timeframe. The remainder of this
document provides the detail and considerations to affect this effort.
-8-
Approach and Scope
Approach
Deloitte’s approach to assessing the current project governance, project management and software development lifecycle
processes and tools was to interview stakeholders, review documents and processes, and identify gaps. Gaps were compared with
Deloitte’s Project Management Body of Knowledge (PMBOK) based project management methodology to develop considerations
for each of the assessment areas.
Inputs and Activities Outputs

Observations, Impacts and Considerations for:
 Governance, decision-making and
accountability
 Communication and information flow
 Status reporting
Identify Review Existing
Stakeholders Analysis  Risk management
 Issue management
 Change control
 Defect management
 Testing management
Deloitte’s Project Process Walkthroughs  Release management
Management
Methodology Proposed processes and tools (to-be implemented
by the MN.IT MNsure PMO) for:
 Status reporting
 Risk management
Interviews Document  Issue management
Reviews  Change control
 Defect management
 Testing management
 Release management
- 10 -
Scope
The scope of this assessment is to provide observations and considerations focused around governance, prioritization,
communication and information flow, status reporting, risk and issue management, defect ,test and release management
Governance Defect Management

Governance scope:
 Governance structure Defect management scope:
 Defect triage
 Accountability and decision-making
 Defect prioritization, ownership, resolution, closure
 Defect management tool
Communication and Information Flow  Defect dashboards and metrics
Communication and information flow scope:
 Internal and external stakeholders (agencies, vendors, health
plans, counties, navigators, brokers Test Management
 Information flow
Test management scope:
Status Reporting  Testing plan
 Testing lifecycle spanning unit test, integration, system test,
Status reporting scope: user acceptance test, production smoke test, and regression
 Reporting of status content test
 Status preparation and distribution  Performance test, security test, and American Disabilities Act
(ADA) testing types
Risk and Issue Management
Risk and issue scope: Release Management
 Risk/issue plan
 Risk/issue tools and maintenance Release management scope:
 Risk/issue prioritization and categorization  Release management plan
 Release schedule and calendar
 Release estimates standards
Change Control  Release checklists
Change control scope:  Release notes
 Change control board  Deployment standards
 Change control request process
 Change control log and request form
- 11 -
Governance, Decision-making, and Accountability
Project Governance – Overview
 Deloitte was engaged by the State of Minnesota (“the State”) to assess the project governance, organizational structure,
and project management approach and to recommend critical changes needed to improve overall management of the
project. The “project” is defined as the MNsure Phase II Project - which in short is the project to effect remediation and
enhancements to the system to fully enable the enrollment process for 2015 (which starts on November 15, 2014).
 Three primary state entities have a stake in the project – the Minnesota Insurance Marketplace (MNsure), the
Department of Human Services (DHS) and the Minnesota Information Technology agency (MN.IT). Our review included
understanding the business interests, relationships and impacts of these organizations on the project.
 Today a Board of Directors governs the relatively newly formed MNsure organization (“MNsure”). At a summary level, the
Board by its charter predominantly determines strategy and delegates “day to day” operational management to its
appointed Executive Director, while also maintaining particular focus on the financial underpinnings of the organization.
The business of the organization (including policy setting) is exclusively that of the Board, who nonetheless can delegate
responsibility to it’s executive director or a committee(s).
 The Department of Human Services (DHS), a more mature organization, is headed by a Commissioner with an
underlying executive team. The Commissioner has a permanent appointment on the MNsure Board of Directors.
 The Minnesota Information Technology agency (MN.IT) is led by a Commissioner and the organization has broad
ownership of the State’s technology assets and resources, and operates as a “shared services” organization for their
respective business customers, including DHS and MNsure.
 Although the MNsure and DHS organizations have unique business goals and interests, they share common interests as
they relate to providing health coverage to Minnesotans, and the underlying processes and system (“MNsure system”)
that enables that processing. MN.IT’s stake in the relationship relates to the enablement and ongoing management of the
technology system as a shared services entity.
- 13 -
Project Governance – Overview
 We understand from interviews, that events following the initial open enrollment period (October 2013) led to a significant
breakdown in project governance and management processes. This was due to the State being primarily focused on
addressing and remediating the issues that arose at the time of open enrollment.
 Following the departure of the Executive Director of MNsure, the Board of Directors essentially stepped into the operational
leadership role left void by her departure. Since that time the Board has continued to play a significant role in the operations
of the Exchange, was successful in appointing a new Executive Director, and working collectively with DHS and MN.IT has
started to reestablish critical governance and project processes.
 Unclear roles and responsibility, authority, lines of communication, reporting relationships, team and governing bodies
composition have added significant inefficiency and have fostered no or poorly informed decision-making across the project.
Priorities or decisions are not timely or at the appropriate governing level and often are revisited and or changed.
 Of particular concern, was the inconsistent and unclear engagement of MN.IT, the state’s information technology agency,
with broad responsibility for all state IT activities and assets, and their role in managing and delivering the system
(particularly within the context of the systems plan for the state enterprise).
 Management of the MNsure IT system development vendors has been inconsistent and appears to have impeded outputs
and progress.
 The dominant “engagement model” has tended towards a “siloed” approach among key stakeholder agency staff - further
exasperated by loose vendor management who themselves have operated in a silo from one another.
 Absence of a baseline and updated/maintained consolidated work plan for the project - that is comprehensive of all task
level details for all contributing resources and vendors through system delivery and post system go live stabilization – has
made project direction, execution, progress tracking and management challenging.
 A number of essential roles/positions were not defined/ vacant for the vast majority of the project (including Project Director,
Testing Lead) that further challenged the governance and project management processes and project effectiveness.
- 14 -
Project Governance – Summary Observations
Governance, Decision-making, and Accountability
Component Expectation Summary Observation
Relevant business interests, strategic intent and priorities Partially present
of all agency stakeholders are defined, duly aligned and While most near-term interests are known and a long term MN.IT@ DHS strategic plan exists;
represented in the form of a project long-term plan. alignment, prioritization and longer-term plans need to be finalized and communicated.
Partially present
A governance and enabling organizational model for the
Model components exist and are operating – clarification of select roles and project ownership,
project exists, is well defined, been duly constituted and
role realignment and addition of new roles should be considered. Upon finalization, clear and
understood by all impacted parties.
broad communication is needed.
Partially present
Roles and responsibilities of the project governing
For a select few governance bodies Roles and Responsibilities were reasonably well defined, in
body(ies) and participants are well defined and align with
general they were are not uniformly defined, clear or well understood across all impacted
the governance and organization model
parties.
Partially present
Clear delegation of responsibility and role definition of all
A number of committees exist (from direction setting, control to quasi-operational). Certain
committees (including appropriate peering of all
committees roles, operation and effectiveness may be diluting the overall governance process.
participants).
We observed some peering inconsistencies that should be addressed.
Partially present
Responsibility for priority setting is clear and priorities are
Many governing groups at MNsure set priorities but they are not established with a clear
adhered to once establish.
methodology. Once established, priorities are often modified or fully changed.
Partially present
Decision-making authority and escalation pathway is well Although partially evident and improvements over time were noted - to avoid decision-making
defined and understood by all affected parties. delays and to engage the most relevant experience/skills at the right time, there is a need to
(re)align and empower the decision-making process.
Key project governance and organizational roles are Partially present

staffed with appropriately experienced and skilled Although many roles exist, critical ones are absent (including Project Director and a number of
resources. subordinate but important roles such as a Testing Manager).
The enabling governance processes (including tools, Partially present
reports and meetings) to support an effective project While most processes exist, consolidation and further role and expectations clarifications are
exist. needed to improve effectiveness.
- 15 -
Project Governance – Detailed Observations
ID Observation Impact Considerations
1 Although the interests and goals of the Governance is reactive to latest Complete long-term business planning within the
Business Owners of the project developments and decision-making is not respective project Business Owners and
(MNsure and DHS) are generally well fully guided by a immediate and longer- consolidate those as they relate the project
defined and understood for the near- term strategy. Staff and vendors are (system). Together with near term interests set
term, they and their enabling tactics lack unclear on priorities, significant milestones the business prioritization for the project (with the
prioritization. Further, the long term or targets, and objectives for longer-term advice and guidance of the MN.IT organization).
strategic intent has been less clear. Finalization of this process should include
DHS has developed a 5 year The absence of a full and clear long-term relevant staff, vendors and other stakeholders
project/system modernization plan, plan adds the risk of inefficient investment
however MNsure’ s longer-term needs being made in how future system
are still in progress. The absence of components are developed
alignment and harmonization of these
longer-term interests is a barrier to
completing the project long-term plan
2 Fragmented and unclear decision- Project delivery resources progress is Clarify decision-making authority for each
making authority and role confusion impeded as issue resolution, prioritization governance body and representative role(e.g.
across the project at multiple levels is or other decision-making is bottle-necked PMT,CCB, EST); define the issue and decision-
leading to decision-making delays, or protracted making escalation pathway; set expectations and
bottle-necking of issues in need of accountability measures for each governance
resolution, protracted activity planning, body and role. Monitor and report periodically on
and unmet objectives expectations and accountability measures
3 Vendors are given conflicting direction Absent a common control point, vendors MN.IT should provide project execution
on priorities of work and system are unsure how to proceed, or proceed in leadership and management of all “SDLC” duties
requirements from business, conflict with other activities of the project including tasks such as managing technical
technology and management teams and lose production time clarifying tasks, teams to the project plan activities,
operating in “silos”, without coordination priorities, and requirements communicating priorities, managing system
or integration across State and vendor requirements, daily supervision of IT vendors,
teams. This appears to have coordinating and integrating across State and
exasperated the lack of coordination vendor teams, and providing to progress
and teaming across the legacy vendors reporting through the project governance
leadership and business owners
- 16 -
4 Participant roles and Participants are often unsure of what Establish/clarify governing body participant
responsibilities are not clearly contribution they should provide to the expectations, roles and responsibilities,
defined and adhered to within governing group leading in some cases to accountability and decision-making authority
governing bodies both under and over-involvement of
participants, lack of focus on critical aspects
and inefficient use of senior resources
5 Participants in certain governing By mixing participants of different Wherever possible, participants on the project
bodies are not peered at the organizational levels on a governance body, governance bodies should be at the same peer
same level, which may inhibit there is a risk of representation bias, uneven level. A review of current participants peering level
engagement and equitable engagement and value creation and across governing bodies and realignment as
decision-making decision-making independence needed is recommended
- 17 -
Project Governance – Detailed Observations (cont.)

6 The Minnesota Insurance Marketplace (MNsure) Over the past few months as the As the new Executive team solidifies, a
governance has been undergoing a transition new Executive Director and new project governance model (with
over the past year (separate and distinct from the Leadership team established itself clear implementation accountability),
project governance) and began operating the business full and appropriate engagement by
(including managing it’s stake in key agency stakeholders (DHS and
After the departure of its initial Executive Director the project to remediate and MN.IT) is completed and
(ED) and in the aftermath following the open enhance the underlying system), accompanying project processes
enrollment period, we understand the MNsure one of the unintentional (including transparent performance
Board (“Board”) essentially stepped into the ED consequences of the Board and reporting) is established, we would
role, responsible primarily for daily operations. Committee/Workgroups prevailing encourage a realignment of roles,
We further understand, that as a result of a operating mode may have responsibilities and decision-making
historical lack of operational transparency, timely contributed to project inefficiencies between the Board, the relevant Board
and accurate reporting and appropriate level workgroups/subcommittees and the
decision-making, the Board decided to further Until operational management and Executive team in the governance of
fortify the organizational leadership by decision-making rebalances the project
establishing several Board workgroups including between the Board and Executive
one focused on Technology and the project which Leadership there exists another
is the focus of Deloitte’s review layer of decision-making, coupled
with the risk of conflicting project
In more recent months, MNsure has successfully intervention and direction which
appointed a new Executive Director, added new may impact overall project
senior leadership capability and begun to realign effectiveness
the legacy organization. Coincident with these
changes, cooperation and coordination with the
other key agency stakeholders (DHS and MN.IT)
accelerated rapidly
- 18 -
7 Vendor management and direction Vendors and staff are confused about MN.IT be responsible for day-to-day
setting for the project system vendors who is responsible and has authority to management of SDLC duties including tasks
has vacillated and was often unclear direct System Development Life Cycle such as managing teams to the project plan
(SDLC) activities, and how to prioritize activities and supervision of IT vendors
their related activities, adversely
impacting project progress
8 State and vendor leadership, State and vendor SMEs are not utilized
Participation roles in governing groups
managers, and subject matter experts appropriately to inform the governing
(SMEs) are not involved at an groups, and leadership and managers should be defined for State and vendor
leadership. Management and staff should be
appropriate level in the governance are not providing appropriate input. It is
utilized at meetings as appropriate to provide
and decision-making unclear as to the how or why decisions
insights necessary to inform the governing
are made, and have limited visibility into
groups and provide clear communication
the process
and visibility to the decision-making process
9 The role of an overall Project Director Inconsistent application of Project Develop scope, roles and responsibilities,
for the project is not defined and leadership and management tasks and and reporting structures for a MNsure IT
accordingly unfilled. Furthermore, many are conducted to varying degrees Project Director and a MN.IT MNsure PMO.
there is no central PMO coordinating by multiple stakeholders resulting in lack Communicate to stakeholders the roles,
project functions of coordination responsibilities, and authorities of the Project
Director and the MN.IT MNsure PMO.
Coordinate and integrate the activities of the
MN.IT MNsure PMO with other agency
PMO’s within DHS, MNsure and MN.IT
10 Turnover of staff associated with the Institutional knowledge of governance Develop a workforce transition plan that
project has occurred in participant goals, activities, and outcomes is lost identifies project governance participant
roles in governance when turnover occurs. New staff in roles and documentation so that knowledge
governance roles need to be on-boarded transfer from one State staff person to a new
to the governance participant role State staff person to fulfill the governance
participation role
- 19 -
11 Agenda topics and discussions Topics and discussions outside of scope of Provide agenda items that are within scope of the
at governing groups are not at the group reduces the ability of the group governing group. Maintain facilitation for each
the appropriate level needed to to fulfill its role and responsibilities and can group that manages adherence to the scope for
meet the purpose of the result in conflicts. Project staff also the governing groups
governing group consume time and effort providing
materials that are outside of scope of the
governing groups
12 Meeting cadence including Improper meeting cadence, for instance Meeting cadence should be defined that allows
sequence and frequency of too frequent, encourages discussions that the goals, activities, and outcomes of the
meetings for governing groups are not within scope, or cadence that is not governing groups to be met while reducing
is not appropriate for the frequent enough prevents discussions that unnecessary meetings. A master schedule of all
objectives, activities, and are in scope. Currently activities such as meetings should be developed to manage
outcomes required of the meeting preparation, meeting time, and duplication, inefficiency and resource conflicts
governing groups post meeting activities are consuming
participant and project staff time
- 20 -
13 An integrated project work plan is not The State leadership and management Develop an integrated work plan for all IT-
established with project activities, have limited insight into the activities of related project activities. The plan serves as
dates, milestones, releases, specific vendors and dependencies which the primary document governing the activities
interdependencies, and resource reduces their ability to make decisions on the project including dates, milestones,
ownership of project activities based on planned activities deliverables, responsible parties, and
dependencies
14 Processes for deliverable submittal, State and vendors use time and effort Establish and document the standard
review, acceptance or rejection, determining what has been submitted, approval process for deliverables and
remedy, and invoicing are unclear to what should be approved or needs communicate to appropriate vendors and
the State and vendor partners additional activities to remedy, and what State staff
decisions should be made regarding
payment of invoices
15 The naming convention of the IT The system is intended to support multiple For clarity purposes - the system should be
system (MNsure system) being agency/organizational interests. assigned a unique name/moniker – that
synonymous with the governing Unwarranted and unintended confusion allows clear differentiation of the enabling
organization (MNsure) for the health can be caused when stakeholders system (project) from any vested
insurance exchange creates address organizational needs and issues agency/organization
confusion in communication and rather than the enabling IT system
direction-setting demands
- 21 -
16 MNsure work is divided into many The State and vendor project leaders Develop an integrated work plan for all IT-
projects with out a full documentation lack visibility into project dependencies related project activities including: design,
of dependencies or an overall project and activities on the project are not development, testing, and release
work plan or consolidated schedule to managed based on a plan or schedule activities. Empower the Project
drive project work Management Team, Project Director, and
the MN.IT MNsure PMO to manage and
drive the activities of the project based off
the project work plan
17 No one person is in charge of the day Gaps in accountability develop as Create a MNsure IT Project Director
to day operations for the MNsure IT governing groups spend energy position to manage the day to day work of
project determining who is responsible for a the project for both vendors and State
particular issue rather than effort to staff. This position should report to MN.IT
resolve the issue staff and coordinate frequently with
MNsure, DHS, MN.IT, and vendor
stakeholders
18 MNsure board working groups lack Frequent meetings drain time and Clearly articulate the role and objective of
clear cadence, definition, duration, or resources away from the MNsure MNsure Board working groups. Consider
role for MNsure leadership and staff as they prepare a sun setting or postpone work groups
and conduct briefings for Board during times of reduced activity on the
members MNsure project
- 22 -
Project Governance – Proposed Model
 The structure supports the two Business
Owners/Sponsors setting overall direction, policy and
Business Owners/Sponsors reviewing progress of the project based on their
strategic business needs. While some of those
interests are shared, many are independent of one
MN.IT MNsure DHS another. A process for setting, reconciling and
reviewing the related demands on the project will need
to be established
Executive  The MN.IT agency, as the State’s Information

Technology shared services agency is positioned to
Steering
provide input to this process and serve as technology
Committee advisor to the Business Owners
Project  The Executive Steering Committee has operational

Management responsibility for the success of the project and
Team primarily with the setting and monitoring of the tactics
Change Control to achieve the goals set by the Business Owners,
Board including major operational issue resolution. The
process for reviewing and reconciling competing
Project Director business demands on the project will be handled by
the Executive Steering Committee. The EST will
MN.IT MNsure provide progress updates to MNsure and DHS and
Project inform them of any serious risk s or issues related to
Management Office the goals the business owners have set
 The Project Management Team serves as the Project

Director’s immediate cross-agency operating group to
Change Training Communications Test Release Defect aide in resource, project and other operational issue
resolution
 The Project Director who is staffed by MN.IT as part of

Business Technology their delivery responsibility for the project and system,
IT vendors
Analysts/SME’s staff manages and oversees the day-to-day activities of the
project and the system delivery teams
- 23 -
Project Governance – Proposed Model
Governance Model Detail – Senior Level

 Today the MNsure Board executes its
responsibilities largely through several workgroups
Business Owners/Sponsors comprised of Board Members who in turn report
back to the collective Board on their respective
focus area (including Compliance, Financial,
MN.IT MNsure DHS Technology). Similar responsibilities for the DHS
agency are incorporated within the role of the DHS
Commissioner
 As it relates to the project, the Business Owners

MNsure Board
role is to set the strategic direction, based on their
Compliance Other …. business interests and mission/charter, for the
Workgroup Workgroup project and underlying system. The Board should
Financial Technology expect regular timely and relevant updates on
Workgroup Workgroup progress from the Executive Steering Committee,
as well as being made aware of significant critical
risks and issues as they arise, that may impact the
project and attainment of their goals. (In the case
Executive Steering Committee of MNsure, the Board may exercise these
responsibilities through it’s current workgroup
CORE MEMBERS: structure or another model)
MNsure CEO
DHS Deputy Director
MN.IT CEO  The Executive Steering Committee has
operational responsibility for the success of the
Supported by NON-CORE MEMBERS: project to achieve the business goals set by the
Senior Management Representation Business Owners. It will determine and execute
(incl. Legal, Finance, Technology, Admin, Operating) operational level tactics to achieve those goals,
resolve all issues escalated to it by the
downstream governance team(s), monitor project
progress and keep the Business Owners
accordingly apprised
- 24 -
Governance – Proposed Business Owners Framework
Role for the MNsure Project

The Business Owners provide overall guidance, policy setting and direction for the project based on their respective organizational strategy and
business needs. Where the Business Owners needs and priorities differ, due to mutually exclusive charter or strategies; they must reconcile those
differences and priorities such that the project direction is clear and unimpeded. The Business Owners retain broad oversight for the project and
should be kept apprised of progress through pre-determined executive updates. The Board should also expect to be made aware of any
significant critical events and risks that may impede progress and/or success of the project
Members Roles and Representation Key Responsibilities

 MNsure Board: Represented in the form of either it’s Board Chair  Strategic Planning
and/or another Board delegate (this could take the form of a  Policy determination/setting
representative Board workgoup)  Governance
 Organizational/project direction setting
 DHS Commissioner: The Commissioner for the Department of  Project review and monitoring (in particular where
Human Services significant events or risks may impede progress or
success)
 Note: While MN.IT is not considered a Business Owner, their value
as State IT “shared services” agency is recognized as important
and as such as should provide input and advisory support to the
Business Owners
Key Relationships
Key Decisions
 Strategic
 Policy
 Governor: Direct responsibility for the Department of Human  Communications themes and approach
Services;
 Legislature: Authorized the establishment of the Board and
authorized the Commissioner of DHS to serve as a Board member. Meeting Cadence
Exercises oversight of MNsure:
 One meeting per month (up to quarterly meetings when
project is stabilized and operating)
- 25 -
Governance – Proposed Executive Steering Committee Framework

The core Executive Steering Committee is comprised of the senior leadership peers from MNsure, MN.IT, and DHS, and provides the project
direction, monitoring and management of operational tactics needed to achieve the goals set out by the Business Owners. The Committee
provides overall coordination of efforts among leadership at MNsure, DHS and MN.IT. The process for reviewing and reconciling competing
business demands on the project will be handled by the Executive Steering Committee. In addition, the Committee serves to resolve issues and
assist with major project risk mitigation that is escalated up to them from downstream governance team(s)

 Executive level coordination between MNsure, DHS, MN.IT
 Direction to the Project Management Team on guidance and
policy set by the Business Owners
 CORE Members: MNsure Executive Director, DHS Deputy
 Overall management responsibility for operations, policy,
Commissioner, MN.IT CIO
technology, and communications on the MNsure IT system
project
 SUPPORT Members: Key Operating, Technical, Administrative,
 Review and approve resource plans and commitments
Legal, Financial representation. These members are not considered
 Establish priority criteria for project activities
voting members but advisory and support to the Core Members.
 Resolve issues escalated by the Project Management Team
Key Relationships Key Decisions

 High level project operational setting and that provides more
 Business Owners: Overall Strategy and Policy setting detailed direction to the project
 MNsure, DHS and MN.IT: Communication of plans and operational  Issues resolution (escalated by the PMT)
impacts and coordination with respective organizations  Strategic recommendations to the Board
 PMT: The PMT provides project updates to the Executive Steering  Statusing and communications recommendations to the
Committee and escalates issues and risks for final resolution Board
 Public and Media: These stakeholder groups look to the
leadership for information about the status of project Meeting Cadence
 Committees (incl. Health Industry Advisory Committee and the
Consumer and Small Employer Advisory): These groups consult  Weekly or bi-weekly until project key milestones are achieved
with and provide recommendations to operational leadership (can then revert to monthly)
- 26 -
Governance – Proposed Project Management Team (PMT) Framework
The Project Management Team (PMT) is comprised of business and technology managers that are peers from the three key stakeholder
agencies of MNsure, MN.IT, and DHS. The PMT reviews and approves more detailed administrative and operational project level activities and
decisions including forecasting, resourcing, planning, and prioritizing project activities, major enhancements, continuous improvements, and
maintenance of service delivery. Their direction to the Project Director is based on effective demand and capacity management of business and
technology agency resources, and management of cross agency interdependences and impacts. The PMT addresses risks, issues, and action
items escalated from the Project Director. The PMT operates within it’s authority and escalates issues to the EST as needed/required
Participant Roles and Representation Key Responsibilities

 Provide direction to the Project Director for managing all
 MN.IT: MN.IT is a member representing the overall technology
areas of the project including: scope, schedule, budget,
goals of the project and MN.IT provides technology knowledge and
quality, resources, communications, risk, procurement, and
expertise to the PMT. The MN.IT representative acts with MNsure
integration
and DHS input, as the decision authority for MNsure IT system
 Monitor the progress of project activities through the planning,
related decisions brought to the PMT
execution, monitoring, controlling, and closing of project
 MNsure: The MNsure representative is responsible for representing
phases
the business and technical goals of MNsure
 Finalize recommendations from the Change Control Board
 DHS: DHS is a member representing the interests of DHS
regarding change requests
programs that are affected/dependent by/on MNsure
 Vendors: Representatives from each of the IT vendors attend the
meeting and provide input as requested by the PMT members
Key Decisions
Key Relationships
 Remediation steps for issues that are impacting scope,
schedule, budget, quality, human resources,
communications, risk, procurement, and integration
 Executive Committee: Issues outside their authority or that cannot  Prioritization of risks and issues
be resolved by the Project Management Team should be escalated  Recommendations for change orders
to the Executive Committee for final decision/resolution  Release schedule
 Project Director: The PMT receives status from the Project  Identification of issues for escalation to the Executive
Director and the PMT provides guidance, decisions and issue Steering Committee
resolution support to the Project Director Meeting Cadence
 One meeting per week
- 27 -
Governance – Proposed Project Management Office (PMO) Framework
The MN.IT MNsure Project Management Office (PMO) provides support to the Project Director and to the project by providing tools and
processes, templates, standards, methodology, policies and procedures for activities including the project work plan, status reporting, risk and
issue tracking, change control, defect management, release management, testing management, and communication. The MN.IT MNsure PMO
coordinates with MN.IT PMO, MNsure PMO and DHS PMO. The MN.IT MNsure PMO has responsibility for “rolling-up” (consolidating) the
respective stakeholder PMO and vendor work plans and status reporting into a master plan and status report

 MNsure IT system Project Director: Directs the MNsure IT system  Manage the project work plan, status reporting, risk and issue
PMO and the activities of the MNsure IT system work plan tracking, change control, defect management, release
 MN.IT MNsure PMO: The MN.IT PMO maintains the integrated work management, testing management, and communication
plan for the MNsure IT system, and manages status reporting, risk  Provide reports to the Project Director on areas managed by
and issue tracking, change control, defect management, release the MN.IT MNsure PMO
management, testing management, and communication. The State  Communicate with key stakeholders
provides necessary staff to assist the PMO including staff such as:  Develop project status reports and distribute to stakeholders
change manager, release manager1, test manager1, defect manager1
and communications manager.
 Vendors: Project managers from each of the IT vendors are to
required to provide input to the PMO for each of the areas managed
by the PMO
1 These roles are described in their respective section on following slides

 Determination of tools and processes, templates, standards,
 Project Director: The Project Director guides the activities of the methodology, policies and procedures for project activities
MN.IT MNsure PMO. The PMO supports the activities of the Project  Assignment of risk and issue owners
Director in managing the MNsure IT system
 MN.IT and DHS: Other PMO staff coordinate with the MN.IT MNsure
PMO to maintain MN.IT MNsure PMO alignment with tools,
processes, templates, standards, methodology, policies and
procedures used by other State PMOs. Meeting Cadence
 Vendors: Vendor work plans are integrated into the master project
work plan, vendors provide input to the areas managed by the MN.IT  Coordination occurs daily as a matter of on-going operations
MNsure PMO
- 28 -
Immediate Key Resource Needs
Immediate Key Resource Needs for MNsure IT System Project
Top 5 Positions and Descriptions

Priority Position General Description
1 Project Director The Project Director for the MNsure IT system manages and oversees the day-to-day
activities of the project management lifecycle. This full time resource is empowered to make
key decisions for the MNsure IT system and will escalate issues to the Project Management
Team as needed. The MN.IT MNsure PMO will support the Project Director in driving
MNsure IT system activities
2 Test Manager Test Manager is accountable for testing, test plan development and execution in the
MNsure IT system including: test cases and scenarios, results and defect management,
testing status communications and defining entry and exit criteria across all test phases
(integration, system, performance/regression, and UAT)
3 Release The Release Manager will lead release management end-to-end activities and ensure
Manager compliance to quality in release management execution. The Release Manager defines and
enforces standards and processes for release management across all environments
4 Communications The Communications Manager is accountable for MNsure IT system communications
Manager affecting MNsure, MN.IT, DHS and their key stakeholder groups (vendors, carriers,
counties, navigators, and brokers). The Communications Manager is responsible for the
development of an integrated communication plan for project stakeholders and for
monitoring communications triggers such as updated release functionality, technical events,
and operational changes
5 Defect/Triage The Defect/Triage Lead will be a member of the testing team and will track and manage all
Lead defects, will lead defect and triage meetings, and will report on identified defects and their
status to the State and vendor partners
Other personnel gaps or needed positions were identified during the course of the governance assessment of the
MNsure organization, those considerations are identified in the remainder of Deloitte Deliverable #2: Program and
Project Management Assessment report.
- 30 -
Communication and Information Flow
Communications and Information Flow – Overview
 Communication is an integral part of the success of the MNsure IT system. Communications leverage familiar methods to
reinforce messaging and use multiple methods for each stakeholder group. Various communication methods are used
depending on the purpose of the message and its intended audience. Communications are used to either inform or engage
specific stakeholders. Selecting the appropriate method to target the right stakeholders is key to the successful execution o f the
communication at hand
 It was observed that communication silos exist within MNsure, MN.IT, DHS and their key stakeholder groups - vendors,
counties, navigators, and brokers. Meetings are being conducted and communications are being distributed within the
individual silos. An integrated communication plan for project stakeholders has not been developed. In addition,
communication ownership and triggers such as technical events, operational changes, and policy modifications have not been
defined. Vendor communications have not been formalized and vendors currently do not interact with end users of the
application. As part of information flow bidirectional communication occurs with feedback being actively solicited
 Summary Observations from the assessment, and additional detailed observations, impacts, and considerations for the
Communication and Information Flow area are presented on following slides
- 32 -
Communications and Information Flow – Summary Observations
Communication and Information Flow
Component Summary Observation
Integrated project communication plan with key communications events as well as the Not present
target audience, timing, delivery mechanism, key messages, and responsible parties An integrated communications plan does not
exist, communication occurs in silos
Stakeholder matrix implemented for project communications that identifies and Partially present
categorizes stakeholders and key areas for communication or focus Matrixes exist, however communication
categorization and focus is not included
Project templates, triggers, timing, and channels for communications Not present
Templates, triggers, and timing, are not
standardized and integrated with the project
Project communication creation, approval, distribution, and processes that formalize Not present
communications processes A formalized process is not documented for
communication creation, approval, and
distribution processes
Project communication feedback mechanisms that obtain bi-directional feedback Partially Present
Bidirectional feedback mechanisms have not
been fully and consistently implemented to
measure stakeholder engagement
Multiple forums and channels for project communications Partially present
Communication forums take place within
individual stakeholder groups. Additional
communication forums have not been
implemented for project communications such as:
• Newsletters
• Collaboration Groups
• Town halls
- 33 -
Communications and Information Flow – Detailed Observations
19 A consolidated plan and strategy for Communications with internal and Develop and manage to an integrated
stakeholder communications including external stakeholders are fragmented communication plan for stakeholders that
vendors, health plans, counties, and not formalized resulting in details: types of communications, target
navigators, brokers and internal stakeholders being updated on an ad- audience, timing, delivery mechanism,
stakeholders does not exist hoc basis that could result in messages, triggers, and responsible
inconsistent messages parties to standardize and formalize
communications
20 Communication channels are not Due to the lack of defined ownership Identify a Communications Manager that
managed across DHS, MNsure, and between business and technology is part of the MN.IT MNsure PMO and is
MN.IT; there are individual owners groups non-standard communications responsible for coordinating
responsible for communications that are sent which may lead to inconsistent communications related to the IT System
relate to the MNsure IT system, including stakeholder communication by both the across the project and is responsible for
distributing related communications business and technology groups about making sure that communications are
project-related decisions creating aligned and planned for with key system
confusion about operational milestones
procedures, schedule, policies and
technology
21 There is a lack of standardization in Details such as release status and MN.IT can define a set of triggers,
communications triggers, templates, and scope, release schedule, release templates, and processes for
processes for both business and MN.IT; functionality, and downtime may not communications as well as their audience,
communications, are not defined or reach the right stakeholders at the right focus can occur on the following technical
standardized for audience, templates, and time in the right format, leading to communications:
triggers misunderstandings and confusion and  Release plan
may limit ability to serve the customer  Release calendar
 Release notes
 System outages
 Testing status
- 34 -
Communications and Information Flow – Detailed Observations (cont.)
22 Internal stakeholders receive inconsistent As a result of inconsistent Solicit feedback and develop forums for
communications from various communications, confusion and internal stakeholder communications such
stakeholders such as business and miscommunication may occur and as town halls and newsletters to promote
technology groups and communications ultimately stakeholders could become open and transparent communications,
feedback is not solicited disengaged town halls occur quarterly and newsletters
are distributed monthly and engage
stakeholders to provide feedback
channels
23 MNsure and MN.IT communications with Vendors can receive informal MN.IT assumes the leadership role over
vendors are not structured and formalized contradictory guidance from MNsure communications with IT vendors, MN.IT
and vendors have limited involvement and MN.IT which could lead to works with MNsure and DHS operations
with user groups of the application inaccurate priorities, rework, and staff to help set priorities and the overall
confusion. Vendors do not receive plan and create focus groups that provide
feedback from end users of the user feedback to the vendors
application leading to missed
opportunities to improve the system
24 Meetings with stakeholders including Due to the lack of coordination around Incorporate health plans, navigators,
health plans, navigators, brokers, and stakeholder communications for health brokers, and counties into the overall
counties are scheduled but not plans, navigators, brokers, and MNsure communication strategy and
coordinated in terms of communication counties each group may have receive develop an integrated communications
content and messaging different messaging with different calendar and detail communication
content at different times triggers to synchronize communications
for these stakeholders to maintain a
defined communication schedule, so that
all stakeholders receive timely
coordinated messages
- 35 -
Communications and Information Flow – Detailed Observations (cont.)
25 Health plan meetings occur weekly to Health plan meetings are not Incorporate health plans into the overall
discuss business and technology integrated into an overall MNsure communication strategy and
processes but are not aligned with communication plan or strategy which develop a communications calendar and
MNsure communications and are tactical could lead to missed opportunities to detail communication triggers to for timely
in nature improve communications with Health and specific communications to relevant
Plans stakeholders
26 Forums for county communications exist Counties are one of the largest group Incorporate county information needs into
to share business policy and system of the MNsure system users and often the overall communication strategy and
information, however communication deal with some of the most complex detail triggers for policy, operational, and
gaps exist in terms of sharing policy, family situations It is critical that technology updates. Also consider
operational, and technology information communications for policy, operational, implementing additional county
and technology are targeted, concise, communications strategies such as:
and timely to prevent inaccurate  Testimonials
information  Fact Sheets
 Job Aids
 Frequently Asked Questions (FAQs)
 Newsletters
 Blogs or Collaboration Groups
27 The primary means of navigator Navigator communications are not Incorporate navigators plans into the
communications occur through weekly e- timely and this is causing frustration overall MNsure communication strategy
mail communications amongst this stakeholder group and develop a communications calendar
and detail communication triggers for
timely and specific communications
- 36 -
Status Report
Status Report – Overview
 The project status report presents information on the activities of the MNsure IT system project including an overall project status,
an executive summary, and updates from vendors on scope, resources, schedule, and quality. The status report utilizes
dashboards to provide succinct, clear information for executives and managers
 The project status report relies on close coordination between vendors and State resources to represent the project status. The
status report serves as an opportunity to communicate clearly across the project about activities and possible issues or risks that
may be present, and reduces the need for clarification or re-explanation of project status during the course of project activities. It
provides governance groups with appropriate information to allow the groups to make decisions that fulfill their responsibilities
 Summary Observations from the assessment, and additional detailed observations, impacts, and considerations for the Status
Reporting area are presented on following slides
- 38 -
Status Report – Summary Observations
Change Control
Present
Overall project status report including weekly project progress and
A status report is produced weekly, however not all
performance
key project health metrics are included
Not present
Executive Summary Section in status report An overall executive summary that provides a high
level overview of the status is not present
Not present
Summarized items requiring leadership attention A summary of executive items is not included in status
report
Partially present
Upcoming milestones detailed in report include future releases, policy or
Report describes some upcoming activities but does
business operations updates
not fully detail project interdependencies
Partially present
Updates from vendors called out in specific sections
Vendors provide only brief updates in the status report
Red, yellow, green status for scope Present
Not present
Red, yellow, green status for resources Not included in status report, a view of resources is not
present
Red, yellow, green status for schedule Present
Not present
Red, yellow, green status for quality Not included in status report, a view of quality is not
present
- 39 -
Status Report – Summary Observations (Cont.)
Change Control
Partially present
Project metrics included in status report Key metrics for managing the project are missing such
as variances and completion percentages
Partially present
Insights are provided at a summary level, but detailed
Project assessed using dashboards
dashboards do not exist for trends, change requests,
risks, issues
Partially present
Distributed appropriately to stakeholders Currently distributed to project leadership but not all
stakeholders
- 40 -
Status Report – Detailed Observations

28 The project is currently producing project Overall status may not be reported MN.IT MNsure PMO is responsible for
status reports, however gaps exist in accurately due to the lack of integration creating an integrated project status report
terms of consolidating information from between MN.IT, MNsure, DHS, and that includes status from stakeholders
many stakeholders including MNsure, vendors including MN.IT, MNsure, DHS, and
DHS, and vendors vendors
29 Executive level project dashboards do not Executives do not receive consolidated Develop and implement a project wide
currently exist for managing the MNsure dashboard views for the project making dashboard that will display overall status
project at an executive level or displaying it difficult to understand the full project and provide metrics for change requests,
impactful information to an executive status including budget, scope, and risks, and issues
audience schedule
30 Limited metrics reporting is included in the Limited metrics do not provide The MN.IT MNsure PMO is responsible
project status report sufficient information to decision for including additional metrics that
makers for the purposes of managing indicate the overall health of the project
the project and alert stakeholders to variances in
metrics as appropriate
 Key metrics include:
 Financial health variance
 Requirements volatility
 UAT test case first pass rate
 Execution issues
- 41 -
Status Report – Proposed Structure
The MN.IT MNsure PMO is responsible for consolidating information for the weekly status report. The Project Director reviews the
status report prior to distribution of the status report. The MNsure status report will be sent to a varied audience of stake holders that
includes agency executives, project leadership and management, and vendors
MNsure1 DHS1
MN.IT1 (Board of Directors)
Report Distributed
MN.IT MNsure Project

PMO Director
MN.IT
MNsure/DHS
Vendors Technical
Business
Leads
- 42 -
Risk and Issue Management – Overview
 Risk and issue management are similar processes that enable the Project Director and MN.IT MNsure PMO to
monitor identified risks and issues during the course of the project. Risks and Issues may be proposed at any time
during the project and once confirmed, they are added in JIRA, are managed or resolved as appropriate, and are
included in the weekly status report
 The assessment of risk and issue management included evaluating existing risk and issue management processes
and tools to provide assessment results, go-forward considerations, and an approach of how risks and issues can
be communicated across the project. The assessment was conducted across the elements of governance, process,
tools, and metrics for the entire issue and risk life cycle ranging from issue and risk reporting, tracking, assignment,
ownership, prioritization, resolution, and closure
 A risk is defined as an event that has not occurred that will, if it does occur, impact the project schedule, scope,
budget, or quality. Risks need to be managed in terms of impact and probability. Mitigation strategies need to be
defined for all risks. These will be tracked and published in the weekly status report and escalated if not resolved
timely to reduce the likelihood that they become issues
 An issue is defined as an event that has occurred that will impact the project schedule, scope, budget, or quality.
Unresolved Critical and High priority issues will be reported in the Weekly Status Report; medium issues greater
than 1 week past due will also be reported
 The MN.IT MNsure PMO will conduct a weekly risks and issues meeting to proactively manage MNsure IT system
issues and risks
 Summary Observations from the assessment, and additional detailed observations, impacts, and considerations for
the Risk and Issue Management area are presented on following slides
- 44 -
Risk and Issue Management – Summary Observations

Partially present
Risk/issue plan in project management plan Issues/risk management is present but plan is
over a year out of date
Partially present
Risk log present and currently maintained
Risk log is present but out of date
Issue log present and currently maintained Present
Partially present
Risk status present in risk log Risk status is present but risk log is over a year
out of date
Partially present
Closed risk documented in risk log Risk status categorized but risk log is over a
out of date
Risk types categorized (i.e., Cost, Functional, Quality, Organization,
Not present
Performance, Project Management, Resource, Schedule, Scope, Technical,
Risks in the log are not categorized by type
General)
Prioritization of risks and issues Partially present
Risk prioritization framework in place but risk
log is out of date
Issue types categorized (i.e., cost, functional, quality, resource) Partially present
Some issues are categorized but others are not
categorized
- 45 -
Risk and Issue Management – Detailed Observations
31 There is a lack of a formal risk/issue Leadership is challenged to identify
Implement a formal risk/issue escalation
escalation process risk/issues across the project and
process, this would limit a reactionary
holistically identify threats to the project
and inconsistent approach to mitigating
risks and issues
32 Risk and issue logs are not standardized Project status cannot be clearly Develop and manage risk and issue in
or used across the MNsure governance monitored without a central location to JIRA that will give each item a reference
structure track the progress or resolutions of number, owner, due date, and priority
tasks, this presents risks to project
schedule, costs, and scope
33 Risks and issues lack owners and priority, Due to limited detail for risk and Implement risk and issue through the
documentation of a process for escalating issues, decision-making can be MN.IT MNsure PMO to allow for scoring
issues and risks is limited prolonged leading to additional cycles (Probability * Impact) of risks and
to refine information document a process for risk and issue
management
34 Risk and Issue logs do not contain Risks and issue logs are incomplete Develop risk and issue management in
needed information to fully track risks and and project leaders cannot fully use JIRA that track item owners, priority,
issues as the arise on the project them in making project decisions owner, date creation, and criteria for
closure
- 46 -
Risk and Issue Management – Proposed Structure
The MN.IT MNsure PMO is responsible for managing the risk and issue processes. The PMT is responsible for risk and issue
resolution and escalation
Project
Management
Team

PMO Director
MN.IT MNsure/DHS
IT vendors
Technical Staff Business
- 47 -
Change Control
Change Control Process – Overview
 The change control process manages all changes requested during the MNsure IT system project. This includes technical
changes to application functionality, requested changes to schedule, and changes to scope. Change control is an integral part of
the project governance as it allows for changes to be proposed, approved and implemented through the appropriate governing
groups with responsibilities to manage change. Effective change control advises stakeholders and project team members of the
schedule for implementation of proposed changes
 Summary Observations from the assessment, and additional detailed observations, impacts, and considerations for the Change
Control area are presented on following slides
- 49 -
Change Control Process – Summary Observations
Change Control
Partially present
Change control log present and being used by project team members
Log is present but not used consistently at the CCB
Partially present
Impact analysis performed on change requests Impacts discussed at the CCB but level of analysis
is inconsistent
Partially present
Change control request template used for all change requests Request template in place, however it is
inconsistently used at the CCB
CCB operating on the project Present
Not present
Change requests/orders in project status report
Change requests/orders not in status report
- 50 -
Change Control Process – Detailed Observations
35 “Projects” are being proposed to Projects are being used to manage Project activities should be driven by an
governing bodies by various State project activities that should be driven by, and integrated project work plan that is used to
members and vendors to determine a prioritized, by an integrated project work determine and prioritize activities. A
group of activities that should be plan and these projects present change control process should be used to
conducted as a project and prioritized conflicting priorities and consume manage requests to deviate from the
resources to develop, discuss, and project plan (which is based on a baseline
determine validity set of requirements and approved design)
36 Change requests are made to vendors The State is being presented with Project activities should be driven by an
by various project team members without invoices for change orders from integrated project work plan that is used to
going through a formal change control vendors and the State is unable to determine and prioritize activities. A
process prior to the work being determine why a change was requested change control process should be used to
conducted or how the work was authorized to be manage requests to deviate from the
completed. Vendors are receiving project plan (which is based on a baseline
conflicting direction on activities and are set of requirements and approved design)
unclear on scope of activities
37 Decision prioritization of project change Priorities for change requests are Provide governing groups with
requests cannot be determined because undetermined or conflicting and the appropriate information to make decisions
governing groups are not provided with organization cannot provide effective regarding change requests to allow them
sufficient information such as impact direction to State staff and vendors to determine priorities including: a
analysis; including resource detailed work plan and an impact analysis
requirements and dependencies to other for requested changes to the overall
activities project plan
38 Change request logs are not Project status cannot be clearly The Project Director should oversee the
standardized or used across the project monitored without a central location to MN.IT MNsure PMO to develop and
track the progress or resolutions of manage change request in JIRA that will
tasks, this presents risks to project give each item a reference number,
schedule, costs, and scope status, justification, and impact summary
- 51 -
Change Control Process – Proposed Structure
The MN.IT MNsure PMO is responsible for coordinating change requests for submission to the CCB. The MNsure Project Director
then reports the proposed changes in the form of change requests to the CCB for their decision to approve or deny the change.
Following the CCB action, the approved change orders are reported to the PMT for consolidation into the overall release plan as
needed
Project
Change
Management
Control Board
Team

PMO Director Change Requests
Release Team
Business SME’s IT vendors Technology staff
- 52 -
Change Control Process – Proposed Change Control Board Framework

The Change Control Board evaluates proposed changes to the MNsure IT system. Changes can be proposed by MNsure, IT vendors, other state
agencies, and stakeholders using a MNsure IT system change request form. The Change Control Board evaluates, prioritizes, and approves or
denies requested changes for the MNsure IT system project. If approved, change requests become change orders and are passed to the PMT for
implementation into future releases
Members Roles and Responsibilities Key Responsibilities

 MN.IT: Chairs the Change Control Board and is responsible for  Evaluate change requests
managing the activities during the CCB meetings  Prioritize change requests and change orders
 Project Director: Responsible for presenting change requests to  Approve or deny change requests
the CCB
 DHS: DHS is a member and represents the interests of other DHS
programs that are affected by changes to the MNsure project
 MNsure: MNsure is a member and represents the business and
operations impact of changes as they relate to the MNsure IT
system
 Other Stakeholders: IT vendor project managers may be asked to
attend at the request of the chair to provide input to the change
request being discussed; other project members may be asked to
attend to provide input to the Board
Key Decisions
Key Relationships  Approve or deny change requests
 Project Management Team: Once change requests are approved

by the CCB, the project management team works with the release
manager, and technology and business stakeholders to evaluate the
implementation of the change Meeting Cadence
 Release Management Team: Provides input to inform the board's
discussion regarding the feasibility of the change requests
 Vendors: IT vendors provide input to the CCB and the PMT  One meeting per week
regarding the impacts of the change request
- 53 -
Defect Management
Defect Management – Overview
 Defect Management addresses all aspects of the defect life cycle from effective defect reporting and logging, ongoing review,
triage and prioritization, assignment to the appropriate owners for resolution, testing and validation, and certification to promote
the defect fixes to the production environment
 Defect Management is closely integrated with testing and release management, and effective defect management contributes to
the quality of the system
 Our defect management assessment spanned the review of existing defect management processes and tools to provide
assessment results and go-forward considerations, and the review of the current set of defects to support a re -prioritization to
align with State objectives. The assessment was conducted across the dimensions of governance, process, tools, and metrics fo r
the entire defect life cycle ranging from defect reporting, tracking, triage, assignment and ownership, prioritization, resol ution,
retest, and closure
 Summary Observations from the assessment, and additional detailed observations, impacts, and considerations for the Defect
Management area are presented on following slides
- 55 -
Defect Management – Summary Observations
Defect Management
Centralized owner or lead for defect management Not present
Overall Defect Manager and defect management team does
not exist
Centralized ownership of defect management tools Not present
The State does not maintain or have ownership of the
central defect repository (JIRA)
Consistent capture and recording of all reported defects accurately in the Not present
defect management tool (JIRA) Reported defects are not captured accurately in JIRA,
resulting in a far lower number of total open defects in JIRA
One system of record for production defects Not present
Initial JIRA reports showed only 60-162 total open defects.
Upon further follow-up and detailed analysis of JIRA, 399
total open defects were identified, including duplicates.
Right complement of a defect management team Partially present
Structured, coordinated Involvement of MN.IT, MNsure
business entities, and vendor teams in defect triage,
prioritization, and resolution is missing
Centralized access to defect management tools Partially present
Key State personnel do not have the right access setup to
close resolved defects in JIRA
Coordinated defect handling from multiple defect channels and sources Partially present
Defects reported from the field help desk, and various
contact centers are lost in transition and do not always make
it to JIRA
Established guidelines for the defect management life cycle Partially present
Documented and established guidelines for the defect life
cycle are not fully in place
- 56 -
Defect Management – Summary Observations (cont.)
Defect Management
Centralized defect triage Not present
Regular and structured defect triage meetings and process not in
place
Coordinated defect prioritization, ownership, resolution, closure Not present
Coordinated defect prioritization meetings, and tracking to resolution
and closure not in place
Pre-defined, timely defect resolution timeframes Not present
Defects from the go-live timeframe are still open/unresolved in JIRA
without clarity as to the expected resolution timeframe
Coordinated, cross-vendor defect resolution with limited churn and Not present
iterations Established meetings and processes for reviewing and confirming
defect fix cross-vendor impacts prior to resolution not in place. Can
lead to cross-module issues getting uncovered for the first time in
integrated test/UAT leading to multiple iterations and rework
Adequate defect resolution details Not present
Lack of clarity in JIRA when a defect is closed as to what was fixed,
and how the problem was addressed. Root cause details are missing.
Often, defects are closed prematurely in the early part of the SLDC
without UAT validation and approval
Robust defect management tool that supports defined defect Partially present
processes JIRA is not configured and setup to support defect management
processes needed on a project of this scale and complexity
Summary and detailed defect dashboard and metrics Not present
Detailed defect dashboards and metrics are currently not in place and
not being distributed to management or executive teams. A clear,
concise current state of defects not depicted in JIRA
- 57 -
Defect Management – Detailed Observations
39 Capture of reported defects is not Results in far fewer defects being Establish, communicate, and enforce
occurring consistently in the defect tracked and reported for the project, clear guidelines and setup resources to
management tool (JIRA) thereby not providing an accurate enable all reported defects to be captured
picture of system quality in JIRA
40 Total number of open defects not reported In the absence of an accurate system Establish a centralized owner for JIRA
consistently, and reported numbers very of record for defects, a clear picture of and enforce process for defect reporting,
low (range from 60-399 total open defects system quality cannot be procured, and capture, and management across the life
for the entire project) focused, prioritized plans for resolution cycle to create a clear defect picture for
cannot be put in place the project
41 Lack of a centralized owner for defect May lead to a lack of clarity and limited Designate a Defect Manager from MN.IT
management across the entire defect life understanding as to the current state of who is responsible and accountable for
cycle defects and can result in outcomes of defect management for UAT and
the defect management process not production. Define clear roles and
meeting expectations responsibilities and owners for each step
of the defect life cycle. Define clear
ownership for defect resolution
42 Access to JIRA is limited to a few This impacts the reporting of defects With the proposed MN.IT ownership of
individuals, access is not aligned with the and closure of reported defects, and JIRA, configure JIRA to meet project
duties of the individual, and licensing may impact the overall quality of the needs and align access groups and
issues have been observed, preventing system and detracts focus from the controls with project roles and
JIRA from scaling up for MNsure “real” set of defects responsibilities. Once the JIRA access
structure is established, review license
needs and upgrade as needed
- 58 -
Defect Management – Detailed Observations (cont.)
43 Ownership and maintenance of the defect A single, consolidated view of the Consider that MN.IT take responsibility
management tool (JIRA) is not with the universe of defects is lacking, thereby and accountability for the defect
State potentially limiting the ability to use management tool (JIRA), including setup,
defects as a reflection of system quality access, usage, and maintenance, to
effectively leverage the tool for defect
management
44 Access to JIRA is limited to a few This may impact the reporting of With the proposed MN.IT ownership of
individuals, access is not aligned with the defects and closure of reported JIRA, configure JIRA to meet project
duties of the individual, and licensing defects, and impacts the overall quality needs and align access groups and
issues have been observed, preventing of the system and detracts focus from controls with project roles and
JIRA from scaling up for MNsure the “real” set of defects responsibilities. Once the JIRA access
structure is established, review license
needs and upgrade as needed
45 Centralized, coordinated defect resolution Can result in multiple iterations of Established a centralized prioritization and
process does not exist across vendors testing and churn to successfully resolution plan following defect triage and
resolve a defect to closure, and can expand testing in the lower environments
cause rework and additional usage of to reduce churn, achieve more successful
resources such as people and time defect resolution with fewer iterations, and
prevent the scenarios of having to rush
partially tested code to the production
system
- 59 -
46 Limited resources (knowledge base and Lack of defect triage can lead to issues Setup a dedicated triage team structure
dedicated time) to triage and route lingering in production longer than they (MN.IT with support from business SMEs
defects for resolution should and as a result, can be an and vendor developers) for timely triage
impact to system quality and end user support
access to functionality
47 A single, consistent system of record for Lack of consistent usage of JIRA Establish, document, communicate, and
all defects is missing. JIRA has been results in defects being reported and implement clear defect reporting, tracking,
setup and is being used, but not tracked via email or not being reported and resolution guidelines and roles and
consistently and effectively at all, which can lead to issues responsibilities so that defect processes
lingering in production longer than they are being consistently followed across all
should and impacting the quality of the users and entities (MNsure, MN.IT, and
system DHS). Clarify specifically how JIRA will be
used for recording the right content and
for driving defects to closure
48 Established guidelines for defect May lead to fewer than actual defects Implement a full end-to-end defect
reporting, tracking, and resolution do not being reported. Insufficient information lifecycle, including guidelines for reporting
exist. Reported defects are missing may lead to issues with replicating the and detailed defect logging (including the
necessary pieces of data such as defect and causing it to be deemed severity level, detailed descriptions,
severity, priority, and associated business invalid (when it is not). Defects are not screenshots, etc.), tracking, and
function which makes the triage and logged at the correct severity level, resolution, with detailed processes, roles
prioritization a challenge. Definitions of impacting the ability to prioritize and fix and responsibilities for each stage of the
defect attributes and values, for example, critical issues. Defects are closed lifecycle. Analyze and revise definitions
defect severity, environment defect was prematurely outside the testing/SME and data values for defect fields in JIRA
identified in, are unspecified or setup as group with limited clarity on the
optional in JIRA resolution and root cause
- 60 -
49 Defect triage and prioritization are missing A clear representation of system Designate a MN.IT Defect Manager
and there is lack of clarity during triage on quality is lacking as defects are not accountable for production defects
items that function as designed vs. “real” being accurately reported and defect Conduct a defect clean-up effort in
defects queues are not being monitored, concert with vendor teams, business, and
triaged, and prioritized. This may MN.IT to bring the current set of defects
ultimately impact the quality of the up-to-date. Establish a team of defect
system personnel from MN.IT with business
SMEs in an advisory role to monitor defect
queues on an ongoing basis. Establish
recurring defect triage meetings with key
stakeholders (MN.IT, business, vendors)
to review defect status reports, key
findings, and triage outcomes. Have triage
outcomes include impact analyses to drive
prioritization of defect work load to the
vendor teams and developers and drive
defects to resolution
50 Defect triage and defined resolution In the absence of pre-defined Amend the contract to include official, pre-
timeframes are missing timeframes, there is limited defined timeframes for defect triaging and
accountability from the responsible resolution based on defect severity and
parties to turnaround defect triaging impact. Clarify scope and expectations
and resolution within expected around ownership and accountability for
timeframes. This causes defects to each step during the timeframe
linger in production and impact system
quality
- 61 -
51 Inadequate resources (masked Valid defects may get canceled if Dedicate a team (MN.IT, business, vendor
production data, environment setup, etc.) unable to be replicated and the developers) to have bi-directional
to triage and route defects for resolution corresponding issues may continue to communication and escalation in the
linger in production and impact system event of insufficient defect data. Take
quality and end user access to corrective measures to provide a
functionality production-like environment with masked
production data that is refreshed at
regular intervals, to facilitate successful
defect replication and to augment reported
defects with additional information for
developers to resolve. Provide access to
canceling and closing defects to a select
group of individuals
52 Issues reported via the field help desks, Can cause a breakdown of critical Identify and define possible sources of
contact centers, escalation centers and information flow and may result in defects. Identify clear roles and
other sources do not migrate effectively to delaying the resolution of critical responsibilities for each defect source.
the central source of defects (JIRA) defects and in causing ambiguity and Provide the required tools, skills, training,
uncertainty to the reporting party documented process, and dedicated
around the status and resolution of resources to the defect source centers.
reported issues Establish a mechanism to allow for bi-
directional communication and escalation
between the source centers and central
defect team on the project
53 When defects are resolved, there is lack Limits insight into the perceived quality Leverage the defect management tool
of clarity on what the root cause was or of the system and the volume of (JIRA) effectively to enforce that key data
how the defect was resolved potentially duplicate issues be entered as part of the defect lifecycle
process
- 62 -
54 Although JIRA is being used as the defect Defect reporting, tracking, and Setup and configure JIRA for the needs of
management tool, JIRA has not been prioritization are impacted as a result of the MNsure project. Identify critical defect
adequately setup, configured, and the limitations imposed by the current fields to be mandatory and create training
leveraged to an extent that could make setup and usage of JIRA guides for defect reporting so that defects
defect management more effective are reported consistently. Dedicate
focused resources and time to keep JIRA
up-to-date and conduct ongoing review
and triage to further defect prioritization
and resolution. Establish and
communicate clear guidelines on
managing the defect life cycle in JIRA
55 Status dashboard and metrics for defect Can lead to limited transparency and Conduct a clean-up of the current state of
management are not being created, visibility around the status of defects defects in JIRA. Define a team for daily
maintained, and distributed and can result in lack of clarity or an review, triage, clean-up, assignment, and
impact to perceived system quality. prioritization of defects. Define and
Can also hinder focus on the “real” publish a detailed defect status report that
issues and the ability to prioritize and includes data such as defect status,
resolve them to closure severity, priority, and impacted business
function. Define the frequency, content,
and audience for an executive summary
dashboard of defect results. Identify the
stakeholders who will receive dashboard
and drive outcomes and resolution
- 63 -
Defect Management – Proposed Structure
The MN.IT Defect Manager is responsible for defect clean-up and prioritization, defect assignment, tracking resolution, and closure.
The Defect Manager manages multiple teams that support the various defect management activities. The MN.IT Defect Manager
reports up to the overall Test Manager who is ultimately accountable for defect management, and who in turn, reports up to th e
MNsure Project Director

PMO Director
MN.IT Overall
Test Manager
MNsure/DHS MN.IT Defect

Business Manager
MN.IT Defect MN.IT Defect

MN.IT Defect
IT vendors Management Tools and
Triage Team Metrics Team
Team
- 64 -
Defect Management – Proposed Defect Management Team Framework

Addresses all aspects and activities of defect management from establishing and implementing processes throughout the defect lifecycle,
enforcing SLAs for tracking, and resolution of defects, and manage defect triage and prioritization. The team is led by a Defect Manager who is
responsible for this and who reports up to the overall Test Manager who is ultimately accountable for Defect Management
.
 MN.IT: Provides the Defect Manager who is responsible for owning  Monitor defect reporting and defect processes and their
and managing the process and a triage team experienced and adherence to established processes
knowledgeable in MNsure. Is responsible for maintaining the defect  Monitor defect queues
tool (JIRA)  Drive defect triage
 MNsure and DHS: Provides business analysts and SMEs for  Drive prioritization
subject matter clarifications and representation, sign-off, and  Monitor defect SLAs
approval at the triage and prioritization meetings  Track defects to resolution
 Vendors: Provide development leads as vendor product SMEs,  Defect status reporting
support for defect triage, estimation for prioritization of defects  Stakeholder communication
 Defects triage outcomes are confirmed

 Project Management Team: Understand the status and progress  Defect prioritization is established
of the current state of defects and the resolution plan relative to the  Escalation checkpoints are established, triggered, and
overall release schedule escalated issues are tracked to resolution
 MNsure and DHS: Active representation across business areas for  Resolution plans, including estimates, are confirmed and
review and approval of work prioritization communicated officially to vendor groups
 Vendors: Provide triage and produce SME support, provide
estimations to factor into the prioritization and defect resolution plan
for information about the status and future of MNsure
- 65 -
Defect Management – Defect Prioritization
The below aspects have been taken into consideration for defect prioritization for the MNsure IT system project
Scope:
 It was not possible to get a consistent result of the total universe of non -closed defects in JIRA
 Multiple data requests sent for a report of non-closed JIRA defects yielded inconsistent results, ranging from a total of 60
to 162 defects
 An independent assessment of the universe of total non-closed* defects in JIRA on 5/8/2014, was 399 defects, which is
the basis of this prioritization
Below is an existing breakdown of the 399 non-closed defects by Priority and Severity
Existing Breakdown of 399 non- Existing Breakdown of 399 non-
closed defects by Priority closed defects by Severity
Defects by Priority Defects by Severity

Blocker 1 - Total Failure
14 14 5
Critical 7 1.5 - Major blocking
11
42
78 Major 2 - Major no workaround
44 3 - Major w workaround
192 Minor
4 - Minor w workaround
Trivial 196
63 5 - Cosmetic
Deferred
80 Major
32 Pending PM
15 5 assignment Pending Assignment
A summary of the existing breakdown by Priority is: A summary of the existing breakdown by Severity is:
 48% have no value assigned for Priority (of which  11% have no value assigned for Severity
70% are major severity defects)  5% are categorized as SEV 1, 1.5
 39% are categorized as blocker/ critical/ major  71% are categorized as SEV 2, 3, Major
 13% are categorized as minor/trivial/deferred  13%% are categorized as SEV 4, 5
* Non-closed includes all statuses except closed, from all projects in JIRA (MNHIX*, SCM Team, Short Term Projects, Security Dom ain)
- 66 -
Defect Management – Defect Prioritization (cont.)
Re-Prioritized Resolution Criteria:

 The existing universe of 399 defects have been analyzed and grouped into the below three priorities to address for
resolution:
− Priority 1 – based on critical functionality (as defined in the Key Function Matrix (KFM) in the functional assessment
in deliverable 3)
− Priority 2 – based on functionality (outside of critical functionality addressed in priority 1). Defects still pending
triage and open more than 90 days are also included in this priority
− Priority 3 – based on functionality deferred or not in near-term scope, or internal, isolated, technical errors for
vendor-specific modules that may not be reproducible or are open for more than 90 days
Of the 399, non-closed defects, 52% are currently assigned Breakdown of 399 non-closed defects by proposed
a priority Resolution Priority
Defects breakdown by Priority for 207 Defects by Resolution Priority
defects (where Priority is populated)
Blocker, Critical, 53
15 Major Priority 1
37 Minor, Trivial
Priority 2
123 223
155
Deferred
Priority 3
A summary of the breakdown before re-prioritization is: A summary of the breakdown after re-prioritization is:
 75% are categorized as top priority  Priority 1:
 18% are categorized as low priority − Only 43% (of the previously categorized 207 defects) are at Priority 1
 7% are categorized as deferred − 55% of the total 399 are at Priority 1
 Priority 2:
− 39% (of the previously categorized 207 defects) are at Priority 2
− 30% of the total 399 are at Priority 2
 Priority 3:
− 9% (of the previously categorized 207 defects) are at Priority 2
− 15% of the total 399 are categorized as Priority 3
- 67 -
Defect Management – Defect Prioritization (cont.)
Key takeaways:
 30% of the non-closed defects are outstanding from 2013
 A large percentage of defects (75%) was tagged as top priority in the existing non-closed defect universe. This leads to
dilution of the concept of “top priority” and makes it challenging to arrive at a realistic, achievable, defect resolution pl an
 Going forward, existing definitions of Severity and Priority should be re-evaluated to refine the definitions and usage of
these fields during defect reporting, logging, triage, and prioritization
Attached are the prioritization results:

 Below is an embedded excel file that should be reviewed in its entirety for the detailed results of the defect prioritization
effort
 Given the volume of defect content, this document is being included in electronic format only
- 68 -
Test Management
Test Management – Overview
 Testing is an integral part of the Software Development Life Cycle (SDLC) because it validates the ability of components and the
system to meet business requirements. Testing verifies that the system works as designed and drives the identification and
management of defects in software quality towards resolution. Testing advises stakeholders, clients, and project team members
as to the software quality
 For a system implementation to be effective, quality must be built in from the beginning and across the entire SDLC ranging f rom
unit test during development to User Acceptance Test (UAT) and post-deployment validation in production. An organized, well
documented, and structured testing process creates transparency and accountability for quality at each step of the SDLC
 The testing assessment spanned current testing processes and tools across the testing phases to provide assessment results and
go-forward considerations. The assessment was conducted across the dimensions of governance, process, tools, and metrics for
the testing phases of unit test, integration test, system test, user acceptance test and production smoke test, and across the
testing types of performance test, automation test, security testing, and ADA testing
 Deloitte’s observations, impacts, and considerations for the Test Management area are presented on the following slides
- 70 -
Test Management – Summary Observations
Test Management
Test team by phase, where the team is well-defined with roles and Not present
responsibilities, including a Test Manager, Testers, Business SMEs, Overall Test Manager, Test Leads for each phase and the right
and Development/Product Support complement of test resources not in place
Thorough testing in each phase prior to UAT and production smoke Partially present
test Testing occurs directly in production and for the first time in UAT or
production for complex functionality and components such as
batch jobs and notices
Adequate testing training that ramps up the testing staff on critical Partially present
business functions Training occurs on an as-needed basis or not at all, and often,
business SMEs pick up testing due to limited functional knowledge
outside that group
Well-defined test strategy and approach Partially present
Does not exist for some phases (such as unit test or integration
test). May exist for phases such as UAT but is not documented.
Often created ad hoc and as-needed and not maintained or
tracked against
Detailed test plan outlining key components of a test phase Partially present
Not documented and may exist informally; often created just in
time but not maintained or tracked against
Clear, achievable test schedule maintained and updated to factor in Not present
dependencies and delays Pre-defined schedule does not exist. Delays in earlier test phases
or deployment delays to UAT not factored in to adjust the UAT
schedule, causing impacts to the available time for testing in UAT
Documented test scenarios and test cases Partially present
Not updated for ongoing functionality. High-level and usable by a
small group only; cannot be leveraged effectively by IT Testers and
other stakeholders
- 71 -
Test Management – Summary Observations (cont.)
Test Management
Documented test case traceability matrix Partially present
Only one point-in-time, outdated version. Not maintained
for ongoing changes in requirements and test cases
Clear, specific, well-documented, pre-approved entrance and exit criteria Not present
Acceptance and certification of functionality is done on a
qualitative basis by business SMEs and documented
entrance and exit criteria not present
Well-defined test data Partially present
Insufficient, often invalidated by multiple testers using the
same data set, and created manually for the most part
Re-usable and repeatable test data creation and automation testing Partially present
Limited means to effectively create large volumes of data
Robust test environment to support end-to-end testing Not present
Does not exist for UAT; interfaces, batches, notices cannot
be tested in the UAT environment
Formalized and documented smoke testing Partially present
Occurs to a limited extent
State-owned and managed performance testing Not present
Only 3 runs of performance test to-date, conducted by a
third party
Robust, repeatable regression testing Partially present
All regression testing owned by vendor and done primarily
in system test. Limited to no regression test in UAT
Testing of components such as interfaces, batches, notices, and reports Partially present
Limited, can test only in system test and not in UAT
- 72 -
Test Management – Summary Observations (cont.)
Test Management
Testing tools usage for test case creation and maintenance, test Partially present
execution, Performance testing, Security Testing, ADA Testing Limited, de-centralized, not coordinated and fully leveraged
Testing Dashboard and Metrics Not present
Executive and detailed dashboards for test metrics not present
- 73 -
Test Management – Detailed Observations
56 State (and specifically MN.IT) supervision Limited visibility and transparency to Designate a MN.IT Test Manager who is
of unit, integration, and system test testing in the lower environments may accountable for testing (including test
phases is limited. Each phase in the lower lead to unclear entry and exit criteria cases, results, defect management,
environments is owned and managed by and may result in more defects testing communications, stakeholder
a different stakeholder with a lack of identified in later stages of the project, involvement, entry and exit criteria) across
consistent processes across the board which may result in more time, cost, all test phases (unit, integration, system,
and resources expended for resolution UAT, production). Develop plan to
at that stage coordinate testing in lower regions and do
not wait to UAT. Create a team of MN.IT
test leads, wherein the leads report up to
the Test Manager and each lead is
aligned with and accountable for one test
phase each (unit, system, integration,
UAT, production, regression). For
instance, for UAT, there will be a test team
and test lead that report up to the Test
Manager (more details to follow in the
chart). Make provisions in the contract to
allow vendor teams to share unit and
integration test details with the State
57 The User Acceptance Test (UAT) team is Can impact the quality and Designate a MN.IT Test Lead, who
lacking the full complement of the right effectiveness of testing and overall reports to the overall MN.IT Test Manager,
mix of resources, knowledge base, and confidence in approving the release to and is accountable for UAT. Involve
stakeholders for testing production. Can also limit the ability to stakeholders from MN.IT, DHS, MNsure,
confirm if the release functionality and the vendor teams in UAT to augment
meets business requirements or not, the knowledge base and provide
which has the likelihood of impacting clarifications as to the build content.
end user access to functionality Establish a team and stakeholder
structure with clear expectations around
roles and responsibilities
- 74 -
Test Management – Detailed Observations (cont.)
58 UAT is conducted during a limited test Lack of documentations limits the Establish a consistent schedule and plan
window and on an unpredictable schedule testing team’s ability to write thorough for releases to UAT, outlining the timeline,
with insufficient knowledge as to the test cases targeted to test critical expectations, and criteria for UAT kick-off.
contents of the release, thereby resulting functionality, which thereby limits Plan for adequate buffer in the schedule
in incomplete testing. The contents of the testing effectiveness. Limited testing to factor in unknowns. Outline clearly and
release are not clearly documented via may lead to the release being specifically the contents of releases to
release notes, and documentation around prematurely promoted to production, UAT via release notes or other such
MNsure functionality is limited or entirely causing delayed identification of documentation. Proactively communicate
lacking in instances. Documentation is defects and regression items, and schedule changes to the UAT stakeholder
also not kept updated to reflect updates to increased time, cost, and resources to group. Create and maintain
functionality resolve issues found in production documentation around MNsure
functionality via up-to-date requirements
and design documents
59 There are instances where testing of Lack of testing of specific functionality Setup adequate resources (environment
complex functionality occurs directly and prior to production poses a risk of setup, data, people, time in the schedule)
for the first time in production regression, where existing functionality to initiate business user testing early on
is impaired, or the intended new and in advance of UAT and more
functionality deployed to production comprehensive testing during UAT to
does not work. This can result in avoid the situation of testing in production
impacting access to production and in
severe circumstances, even impact
production availability or uptime
- 75 -
60 The UAT phase has incomplete and Diminishes the effectiveness and intent Establish and implement process for the
inconsistent testing processes, of the UAT phase and can lead to areas outlined below:
specifically around test case scope and delayed identification of defects at a  Test cases: scope, creation, review,
creation, test execution, reporting and later stage or directly in production, traceability, sign-off, and maintenance
review of results, defect identification, resulting in increased time, cost, and to reflect new functionality
tracking, and resolution, established entry resources for defect resolution and  Test execution: data creation,
and exit criteria, and stakeholder increasing the risk of impacting end execution, tracking and reporting of
representation and communications user experience and access to critical results
functionality  Defect management: Identification,
reporting, tracking, communication, and
resolution of defects
 Established entry and exit criteria
 Stakeholder involvement in UAT and
timely communication of decisions and
outcomes
61 UAT is limited in its effectiveness as a This results in testing some Prioritize the setup of a UAT environment
result of environment constraints such as functionality for the first time in to allow for the testing of critical
the inability to test end-to-end scenarios, production and identifying and components such as interfaces, notices,
components such as interfaces, notices, resolving issues at that point, which reports, and batches. Build focused test
reports, and batches, and time based may result in expending more time, teams knowledgeable in testing each
scenarios that need time advancement cost, and resources, and delaying the component including stakeholders from
access of planned functionality to the MN.IT, MNsure, DHS, and the vendors.
end user Prioritize the addition of system
functionality to advance the time clock
- 76 -
62 UAT is limited in its effectiveness as a Limited testing leads to more issues
result of data constraints such as limited identified in production, resulting in Identify and allocate test resources to the
test data, lack of a means to automate expending more time, cost, and UAT team for supporting data
data creation, and lack of masked resources for resolution. Lack of management (creation and automation).
production data to replicate and retest masked production data in a secondary Setup a secondary environment with
production issues environment limits the ability to masked production data, or alternatively,
replicate and resolve critical production refresh this data periodically into UAT,
defects that may continue to linger in and provide access to this data to
production longer than they should and vendors, testers, and the business users,
impact the end user’s access to to allow for production issues to be
functionality replicated and resolved
63 Detailed security testing has been If identified code issues and gaps are Prioritize the remediation of security gaps
conducted, however, code corrective resolved timely, then the effectiveness with the vendors. Identify the list of all
actions suggested to some of the vendor of security testing can improve. pending gaps by vendor and create a
groups have not been prioritized and Depending on the type of gaps resolution plan in concert with the Security
implemented to date outstanding, those may result in team and MN.IT
security non-compliance and render
the product vulnerable to security
threats
64 ADA testing is still ongoing; the State is Unless the current plan with the third Suggest active monitoring, tracking, and
working with a third party vendor to party vendor is followed closely, any reporting of status against the plan, timely
assess any gaps in accessibility and potential gaps in accessibility and review of the assessment, and
disability compliance of the product and disability compliance may not be prioritization of resolution for any identified
ADA testing remediated in a timely manner gaps
- 77 -
65 The performance test efforts undertaken System performance may directly Identify a MN.IT owner for performance
by the State with SOASTA have translate to end user experience and testing, through its life cycle from testing
uncovered performance issues and gaps, access, and the user’s ability to to issue resolution and fix migration to
many of which are yet to be resolved. effectively use the MNsure website. production. Identify and designate a
These issues range from site capacity Resolution of lingering performance performance team within MN.IT to track
limitations, HTTP errors once the capacity issues can result in improving end user and monitor progress with each vendor
is reached, lower than expected response access and the number of successful via the issue resolution plan. Identify
times, throughput, bandwidth, and server enrollments critical performance attributes and
stability, and connection reset and other establish clear requirements for each
errors attribute. Work with SOASTA to
understand the current state against these
attributes. Prioritize and create a
resolution plan with the vendors for the
performance issues and gaps identified to
date and new gaps against the
established baseline. Rerun performance
tests with SOASTA at periodic intervals
monitor progress against the baseline
- 78 -
66 Testing tools currently used for test Effective usage of tools may enable In the near-term, analyze and leverage
execution (MS Excel) and defect better tracking of test case traceability existing tools effectively. This entails
management (JIRA) can be setup and to requirements, test case results, and activities such as setup, providing user
leveraged more effectively. Testing tools defect management. This may result in access to stakeholders, creating and
are currently not integrated or available to more transparency, accountability, and communication training guides for correct
all stakeholders accurate reporting of the outcomes of usage, ongoing tracking and monitoring of
testing data, ongoing review of the results and
tracking to expected outcomes. In the
long-term, assess integrated test and
defect management tools that provide
strong out-of-the-box capabilities that can
be leveraged on a project of this scale and
size
67 Status dashboard and metrics for test May limit the transparency and visibility Define and publish on a weekly basis a
management are not being created, around the status of testing which detailed test status report that outlines the
maintained, and distributed could limit the ability to drive to scope of testing, traceability to
successful outcomes and hinder the requirements, test execution results, test
full effectiveness of the testing process case first pass rate, defect density,
resolution plan, and plan as to additional
test cycles, if any. Define the frequency,
content, and audience for an executive
summary dashboard of test results.
Identify the stakeholders who will receive
dashboard and drive outcomes and
resolution
- 79 -
User Acceptance Test (UAT) Management – Proposed Structure
The MN.IT Test Lead is responsible for User Acceptance Test (UAT) and manages the Test Team and testing involvement with the
business entities and vendor groups. As referenced in ID 56 on slide 74, the structure below can be used for testing beyond UAT.
The MN.IT UAT Test Lead reports up to the MN.IT Overall Test Manager whose responsibility extends beyond UAT. The Overall Test
Manager is ultimately accountable for UAT. The Test Manager reports to the overall MNsure Project Director

PMO Director
MN.IT Overall
Test Manager
MNsure/DHS MN.IT UAT

Business Test Lead
MN.IT Test
IT vendors
Team
- 80 -
Test Management Team – Proposed User Acceptance Testing (UAT) Framework
Addresses all aspects and activities of UAT from test case and test data management, test execution, test status reporting and tracking, defect
reporting and tracking, to regression testing and certifying code readiness for production. The team is led by a UAT Lead who is responsible for
this phase and who reports up to the overall Test Manager who is ultimately accountable for UAT

 UAT Planning and Management
 MN.IT: Provides the Test Manager who is accountable for UAT, a  Test case management
UAT Lead who is responsible for day to day activities in UAT, and  Test data management
testers experienced in testing processes and tools. Provides UAT  Test execution
environment support and maintenance  UAT Environment management
 MNsure and DHS: Provide SMEs for subject matter clarifications  UAT Status Reporting and Tracking
and Q&A; and review/sign-off of test cases. Provide business  Defect Reporting and Tracking
representation for UAT sign-off/approval.  Discussing/approving entry and exit criteria
 Vendors: Provide development leads for Q&A and triage of defects  Stakeholder communication
or issues identified during UAT  Gate/Approval of code migration to production
 Confirm that entry criteria are met to start UAT

 Project Management Team: Understand the status and progress  Approve the scope and content of test cases
of UAT relative to the release schedule  Review and approve test case results
 MNsure and DHS: Understand the UAT plan in advance to  Agree on defect reporting guidelines, severity, and priority of
anticipate and plan resource needs and representation across all defects identified in UAT
key business areas  Approve functionality conformance to requirements
 Vendors: Need to provide triage support and Q&A for issues
 Approve and certify code readiness for production
identified during UAT
- 81 -
Test Management – Test Plan Outline
 The following slide illustrates a representative test plan outline, with key components to be included in the plan, for User
Acceptance Test (UAT)
 The creation and effective implementation of plans similar to this for other test phases – unit test, integration test, system test – and
other test types such as performance test, ADA test, and regression test are likely to result in structure and coordination for those
test phases and types
- 82 -
Test Management – UAT Plan Outline
ID Activity Description
1 Purpose and Scope The purpose provides an introduction to the test plan and outlines the intent and components of the plan. The scope highlights the high -
level functional requirements or functional areas that the test plan applies to.
2 Test Objectives This section will outline the motivating factors and expected outcomes of testing, the aspects that are in scope, and an overview of
planned tests with what’s included and what’s not
3 Test Strategy The Test Strategy establishes the foundation for all testing activities. It covers testing policies and processes to support the vario us test
levels and cycles. The Test Strategy will provide flexible, consistent delivery of testing services to drive improved quality , lower cost and
increase speed of delivery across the system.
4 Test Approach The Test Approach is created after the Test Strategy has been approved. It outlines the scope of the overall testing effort, the test levels
required for the project, the test team organization, the estimated effort needed to plan and execute, the issue resolution p rocess and the
roles/responsibilities of the team involved. The Test Approach is the predecessor to the detailed Test Plan.
5 Detailed Test Plan The Test Plan includes: the scope of the testing effort, roles and responsibilities of all team members providing support, th e schedule and
time frame for scenario development and testing, and a detailed overview of all activities involved in the system testing pro cess. The Test
Plan will identify the standards and metrics against which test activities are planned and measured.
6 Test Scenario, Test Cases, and Includes detailed definition of the test scenarios, review, and approval, and traceability of the test cases to requirements
Test Case Traceability Matrix
7 Entry and Exit Criteria The Test Entry Criteria help determine if the execution of a particular Test Plan can begin. All criteria within the Testing Approach must
be met or documented. Exceptions must be mutually agreed upon before testing can begin. The Test Exit Criteria will be used t o
determine if the execution of the Test Plan is complete and intended objectives are met. The criteria must be clearly documented upfront.
8 Test Data Requirements Outlines all aspects of test data management, including the types of data, how frequently data should be refreshed, mechanisms to
create and use data, any automated tools for creating data, and the resources and ownership of data management
9 Test Environmental Needs Includes the environment name and technical details, for the source and target systems as well as any tools used for testing.
Environment sizing and the intended number of testing iterations (assuming the target environment will be refreshed/cleared i n between
iterations) will be critical expectations to document. Access to the environment(s) should also be defined.
10 Staffing, Roles and This section outlines the required resources to address the test effort, main roles and responsibilities of these resources, along with
Responsibilities, Training Needs expected knowledge base and skill sets. The section also discusses how to approach training for the testing roles on the proj ect.
11 Test Schedule This section will include the key schedule milestones, the test schedule for detailed planning and iterations (execution cycles), number of
iterations, characteristics of each iteration (for example: size of load, timeframe, data variations), and the expected timeframe for each.
Depending on the solution, it may be advisable to begin with a subset of production data that represents the ‘basic’ or most common
business scenarios, and then perform iterations on more focused scenarios individually.
12 Testing Dashboard and Metrics Reports should be defined to be created regularly to track, manage, and communicate the progress and status of testing. These reports
include summary and detailed information of test scripts executed and defects discovered during testing. The reports are gene rated
based on the data elements in the Test Management Tool, which provides for customization of the attributes as needed.
13 Testing Risks, Dependencies, This section will identify potential risks, mitigation and contingency for each risk, and it’s likelihood. Any assumptions or depe ndencies
Assumptions, and Constraints likely to impact the test plan, test executions, or outcomes of testing should also be outlined here, with an escalation and mitigation plan.
- 83 -
Release Management
Release Management – Overview
 Release management activities include planning releases, scheduling releases, monitoring releases status, overseeing vendor
resources, aligning releases to business expectations, and ensuring release quality
 Currently release management for the MNsure project is present however opportunities exist to improve release management
 Release Management is closely integrated with testing and development and determines that code is deployed in the right
environment at the right time. The release manager coordinates release activities with change management, testing
management, and defect management to align activities across the project
 The components that were assessed for release management include release plans, calendars, roles and responsibilities,
prioritizations, release estimates, deployment standards and tools including release management checklists
 Summary Observations from the assessment, and additional detailed observations, impacts, and considerations for the
Release Management area are presented on following slides
- 85 -
Release Management – Summary Observations
Release Management
Release Plan that details the software release to all environments, identifies release Partially present
strategy, logistics, tasks, recovery and disaster plans, rollback plans, and pre and Some details are contained in individual documents
post-implementation activities
Integrated Release Calendar that provides a view of all activities such as Partially present
development and testing and details release dependencies such as vendor product A schedule has been developed, but it is missing
dependencies the integrated view
Compliance/standards champion present in the release management team with the Not present
ability to understand the requirements associated with the standards and able to Individuals are driving requirements to completion,
verify that the standards are appropriately implemented and adhered to in the however the no cross-organization role has been
application defined with expectations
Roles and Responsibilities of Release Management Organization that defines the Not present
organization structure and the role, responsibility, and activities Individuals are performing roles, however the roles
have not been defined with expectations
Release Notes that list all items delivered within a particular release for both Partially present
business and technical audiences Technical details are included based upon vendor
input, however the business focus of release notes
is limited
Prioritization Matrix that identifies importance of defects, enhancements, and is used Not present
to develop budgets for releases Prioritization occurs through informal processes
and no priorities are documented associated to
defects and requirements
Release Checklist a deployment tool that encourages the deployment process is Partially present
followed and may have environment specific features Checklists are being used for migrations, however
information sharing between environments is
limited, processes are not documented, and
checklists are not used to drive continuous
improvement
- 86 -
Release Management – Detailed Observations
68 A release management plan along with Makes it difficult to manage the overall MN.IT should develop a release
associated roadmap does not exist release management process including management plan along with a roadmap,
release planning and estimation, release the release management plan includes
governance including prioritization of the release planning, release governance,
requirements can lead to difficulty in process documentation, and
planning and executing releases and documentation standards
have appropriate requirements met within
the allocated budget
69 There is a lack of an overall release Results in quality and deployment issues Define the role of release manager and
manager across all environments, there such as missed deployment windows, provide the release manager the authority
is no one on point for maintaining an rework, incomplete regression testing, to lead release management end-to-end
overall release calendar and no single and missed requirements. This also to promote quality and improvement in
point of contact to drive deployments to results in different approaches being release management execution. Develop
each environment and encourage followed in each environment which can consistent standards and processes for
expected processes are being followed lead to confusion and inconsistent release management across all
consistently for each environment processes environments
70 Due to multiple entities involved in Can lead to confusion as to who is Define specific and clear roles and
release management, there is a lack of responsible and results in quality issues. responsibilities to improve the structure for
clarity around roles and responsibilities Can also lead to missed deployment release management
and a consolidated view thereof windows and rework along with budget
being spent on unsuccessful
deployments
71 An overall approach or strategy Prevents a holistic view of how guidelines MN.IT should define and incorporate a
associated to driving requirements and standards are met and can lead to role for a cross-organization
relative to standards is lacking. missed requirements compliance/standards champion into the
Individual business owners are release management team. That person
identified to drive requirements and should have the ability to understand the
implementation of functionality specific requirements associated with the
to their products standards and be able to verify that the
standards are appropriately implemented
and adhered to in the application
- 87 -
Release Management – Detailed Observations (cont.)
72 Deployment processes have not Results in inefficient deployment Develop a list of deployment processes
been documented and are only processes being executed and including deployment checklists for each
partially implemented environment discrepancies delaying the vendor and environment, implement
deployment of releases; this can lead to environment standards and documentation
deployment dates being missed and standards such as standardized release notes
lead to resources working on the same and standardized change controls, and find
deployment multiple times, thereby opportunities to streamline deployment through
wasting deployment resources and automated tools such as ClearQuest or other
possibly impacting the schedule of future MN.IT tools thereby reducing the resources
releases needed for deployments
73 Defect fixes, new code, and product Lower priority items may be fixed prior to Implement an estimation and prioritization
upgrades are not actively managed higher priority items and budget may be process associated to defects that uses
and prioritized by the State spent on fixing items that may not be a standardized tools such as JIRA and
priority. More complex, higher priority ClearQuest so that high priority defects can be
items remain unresolved, impacting estimated and scheduled for release. Ensure
availability of functionality and overall collaborative process is established between
product quality Release, Defect, and Test Managers
74 Mapping the dependencies Leads to unsupported combinations of The State should map the dependencies
between the various vendors in vendor packages, thereby increasing between the various vendors in terms of
terms of software versions needed risk and possibly requiring additional software versions needed in order to meet the
in order to meet the release testing or not meeting functional release schedule and determine if there are
schedule has not occurred requirements for the end user unsupported combinations of vendor
packages and determine associated
mitigations
75 Release testing by the State is Leads to identifying defects reactively The State should test prior to the User
primarily done in the User delaying releases and requiring Acceptance Testing Environment and be
Acceptance Testing Environment additional budget to resolve defects responsible for regression testing. This helps
confirm timeliness and quality around
deployment and testing, and prevent defects
from being identified in the User Acceptance
Environment Testing for the first time
- 88 -
76 Full requirements traceability does not Due to this lack of rigor, requirements Conduct a fit gap analysis of the current
exist due to the original requirements from for implementation have been missed application factoring in any assumptions
Maximus not being utilized; 2700 leading to additional spending to and gaps around underlying technologies
requirements were documented at varying remediate gaps downstream and pre-existing functionality, determine
levels of detail the associated gaps and develop a plan to
address the gaps
77 The current approach to documenting Leads to designing processes that do The State should document the
requirements is not standardized, at times not coincide with functionality of the requirements gathering process taking
a blank whiteboard is used versus a fit vendor applications and may result in into consideration the underlying
gap analysis for the vendor applications wasted coding effort or having to technologies and pre-existing functionality
rework the requirements. Assumptions
are also made around what exists out-
of-the-box and what functionality needs
to be built
78 Vendors have expressed concern about Results in conflicting priorities and The State should develop a matrix and
the lack of business ownership of rework due to confusion about the implement a process that indicates who is
requirements and the overall release requirements and their priority, this can responsible for owning business
management process including lead to missed requirements or work requirements and setting priorities
deployment management and support of being done on lower priority
business processes such as prioritization requirements requiring additional
budget to address the higher priority
requirements
- 89 -
79 Release notes are focused on the Stakeholders do not have exposure to Release notes should be improved to
software (vendor specific) and do not all the change requests that have been serve both the needs of the business and
effectively highlight the business changes implemented thereby making it difficult to provide a consolidated view of the
that are being deployed for the MNsure to understand what is delivered in each release. This allows the business to better
application release and communicate changes structure their UAT and execute
outward and making it more difficult to communication plans for users of the
test and validate the deployment application, and enable the technical
effectively teams to better understand all the
components that are being deployed and
any inter-dependencies between
components
80 Tactical planning has occurred for release Makes it challenging to understand The State should develop an integrated
management but an integrated calendar delivery schedules and dependencies calendar view of future releases along
view is lacking associated to releases, makes it more with all dependencies
difficult to plan for deployments and
testing, and may be more challenging
to identify code and version conflicts of
the various vendor packages
81 No prioritization process associated to Results in confusion about delivery and The State should develop a prioritization
requirements exists, and there is a lack of can lead to missed requirements, matrix associated to requirements and
formalized process associated to requiring additional budget to address formalize requirements definition
business requirements. Release missed requirements processes and release schedules so that
schedules are developed, but priorities high priority items are addressed
change and then schedules are adjusted
- 90 -
Release Management – Proposed Structure
The MN.IT Release Manager reports to the Project Director and is responsible for managing vendor deployments as well as the
MN.IT Deployment Team, the MN.IT Release Manager provides status to MNsure and DHS business stakeholders and to the PMO,
the MN.IT Release Manager is responsible for obtaining deployment approval in each environment and coordinating environment
changes with MN.IT Infrastructure Team

PMO Director
MNsure/DHS MN.IT Release

Business Manager
MN.IT
IT vendors Deployment
Team
MN.IT
Infrastructure
- 91 -
Release Management – Proposed Release Management Team Framework

Release Management addresses all aspects of deployment and release management including the development of the release management
strategy and plan, the governance of business requirements, prioritization, and definition of the roles and responsibilities for the deployment and
release teams

 MN.IT: MN.IT provides the Release Manager who executes the  Develop Release Management Plan
Release Management Plan and develops the Integrated Release  Complete Prioritization Matrix
Calendar  Create the deployment and release checklist
 MNsure: Provides business SMEs by functional areas and who are  Establish consistent deployment and release processes in
responsible for approving the Prioritization Matrix and the Release each environment
Roadmap  Manage Integrated Release Calendar
 DHS: DHS to represent the interests of other State programs that  Develop Release Roadmap
are affected by Release Management  Implement Documentation Standards
 Vendors: Project Managers from each of the IT Vendors are  Execute production and UAT deployments
recommended to attend the meeting and present their requested
changes as appropriate and discuss any dependencies

 Approve product upgrade deployments
 Approve the content for defect fixes and code upgrades
 Counties/ Providers/Brokers/Navigators: Several groups work
 Certify environment readiness before start of a deployment
directly with MNsure customers and have a need to know the plan
 Certify and approve completed deployments and release for
and business impacts of deployments
testing
 MNsure and DHS Stakeholders: Need to understand the
deployment and overall release schedule to allocate resources for
testing and support Meeting Cadence
 Vendors: Need to understand the implications of deployments
 As needed per workplan
- 92 -
Release Management – Plan Outline
ID Activity Description
1 Purpose and Scope The purpose provides an introduction to the release plan and outlines the intent and components of the plan. The scope highlights the
high-level functional requirements that are required to be implemented as part of release management
2 Current State Inventory As part of creating a release management plan, an initial step is to conduct a current state inventory of release management processes,
tools, and stakeholders
3 Release Strategy This section provides an overview of the strategy for future MNsure releases. The strategy includes understanding of any concurrent
project deployments to be included within the release, identification of any components or sub-systems that are impacted, and the
nature of impact. For MNsure, the release strategy incorporates the overall orchestration of resources, tasks, and environments
required to perform a successful release
4 Release Logistics This section documents the logistics for this release in terms of technology infrastructure, network, application, third party software and
resource needs. The logistics included here provide an overview of the process and components needed to orchestrate a successful
release into production or non-production environments
5 Release Estimation This section defines the estimating actions that need to be completed for a successful production or non-production release
6 Release Classification Releases will be managed by the Release Manager and grouped into Release Types such as Major, Minor, and Emergency. Individual
process steps may vary by release type
7 Roll Back Planning It may become necessary to revert to the pre-release state, if possible. Detailed steps should be developed for Rollback. These are
taken in the event that a contingency occurs during or after release that cannot be appropriately mitigated
8 Release Go-No Go Criteria The purpose of the Release Go-No Go Criteria Checklist is to evaluate the readiness of going live with the new system. The criteria
should be used to help aid the decision of whether or not to move a release into the production environment
9 Release Notes The Software Release Notes is the quality record that lists the items delivered within a particular release. The document includes
general information about the release, compatible products in the release, upgrades from previous releases, new features introduced in
the release and known limitations, bugs, and workarounds
10 Prioritization Process As part of the prioritization process, stakeholders need to prioritize defects, enhancements, and develop overall budgets for releases.
Prioritization correlates with release classification. The overall process for prioritization integrates with the release management
process and the overall governance structure
11 Roles and Responsibilities This section identifies the roles for performing the release and deployment activities and describes the responsibilities of identified roles
12 Release Processes All release management processes will be documented in each environment as part of developing the release management plan
- 93 -
Appendix A:
Project Management
Processes
Proposed Project Status Reporting Process
The MN.IT MNsure PMO will be in charge of triggering the status report data request process on a weekly basis. Under this structure,
they are responsible for compiling, synthesizing, and developing the weekly project status reports which, following the Project
Director’s approval, are distributed. Status report distribution list includes State executives and leaders, governance groups, vendor
partners, and external stakeholders
Project Action Change Decisions Issues

MN.IT Status Items Requests
4. Distribute
1. Submit Data
MNsure Request Project Status
Test Release Risks Work Plan
PMO Management
Management
2. Compile Data 3. Update JIRA with

Vendors
Request necessary data
2. Compile Data 3. Update JIRA with

MN.IT Request necessary data
DHS 2. Compile Data 3. Update JIRA with

MNsure Request necessary data
Legend
Inputs / Supporting
Task Step Procedure Decision
Outputs Outputs
- 95 -
Project Status Reporting: Executive Summary <dd-mmm-yyyy – dd-mmm-yyyy>
Project Status Summary Items Needing Leadership Attention
Y Scope Resources Schedule Quality Budget Request Priority/ Target

Overall ID Description Sev erity Resolution Date
Project Y R G Y G
Status - = = + - +
Project
Status
Sum m ary
Upcoming Deliverable and Key Milestone Status

Baseline Planned/Actual
Deliverable / Milestone Nam e Progress Finish Date Finish Date Status Com m ents
C
NS
Deliv erable Status and Milestone Summary Legend NS Not started C Completed G On track Y <1 week behind schedule R >1 week behind schedule
Proj ect Trends + Trending Up (Improving) = Flat Trend (Steady) - Trending Down (Declining)
- 96 -
Proposed Risk Management Process
Project Management Team assesses the severity of the risk and manages the risk once a mitigation strategy is determined
Project 1. Identify Risk

Team
Members
MN.IT 2. Validate JIRA Update JIRA

MNsure Entry
PMO
5. Develop Risk
Response
Project
Yes
Management
Team No 8.
3. Analyze Risk 4. Is Risk Severity 6. Monitor Risk Manage 9. Close Risk
High? Risk
No Yes
Risk 7. Is the
Risk
Owner Real?
Legend
Inputs / Supporting
Outputs Outputs
- 97 -
Proposed Issue Management Process
The objective of this process is to manage the issues identified in the project, which includes identifying, prioritizing, assigning,
monitoring, and closing issues through all project phases. Issues are managed on an ongoing basis and reviewed on a monthly basis.
JIRA is the tool used for issue management
Project 1. Record and Update JIRA

MN.IT
Management validate details 4. Track and follow
MNsure Plan about issue in up on issue
PMO JIRA
2. Assign 5. Change
proposed issue No 6. Closes issue
Request?
Project
Management
Team
3. Performs issue Yes
management
Change 5. Manages
Control Change
Board Requests
Legend
Inputs / Supporting
Outputs Outputs
- 98 -
Proposed Change Requests Process
The objective of this process is to identify, manage, and facilitate change control decisions on change requests to client contract terms
and/or project deliverables that have been signed off and placed under change control. Change requests are managed on an ongoing
basis and the change control board will meet on a weekly basis. JIRA is the tool used for change control
Project
MN.IT 2. Review
3. Review Impact Management
MNsure Change Request
Analysis Plan
Form Update JIRA
PMO
Change
Control 4. CCB Review
Yes
and Approval?
Board
No
1. Complete
Change Change Request
Requestor Form
PMO 5. Communicate
Communications Changes
Manager
Legend
Inputs / Supporting
Outputs Outputs
- 99 -
Proposed Change Request Template
The Change Request template will be used by stakeholders to initiate a Change Request. The template will be submitted to the MN.IT
MNsure PMO and will be used to record, track, and manage change requests throughout the life of the project. The PMO will keep
JIRA up to date based on any Change Request forms received. Once the Change Control Board makes a decision on a specific
Change Request the PMO will update JIRA to reflect it
- 100 -
Proposed Defect Management Process
The objective of the defect management process is to enable timely communication of defects through appropriate channels in an
effort to quickly triage and resolve issues as they are detected throughout the project lifecycle. Defect management occurs on an
ongoing basis. JIRA is the tool used for defect management
8. Create 12. Publish

5. Drive
MN.IT Defect 2. Drive defect defect executive and
defect
Manager triage
prioritization
resolution and detailed defect
retest plans dashboards
9. Retest defects
3. Conduct
MN.IT Defect 1. Intake and track defects
defect
Team Defects triage
to resolution and
closure
6.
Business Participate in 11. Validate and
defect approve defect
SMEs closure
4. prioritization
Participate
in defect
triage 7. Provide input
10. Support defect
Vendor to the defect
management and
Teams plans and
Q&A
resolution details
Legend
Inputs / Supporting Decision

Task Outputs Step Outputs Procedure
- 101 -
Proposed User Acceptance Test (UAT) Process
The objective of the UAT process is to develop the User Acceptance Test (UAT) Plan to validate that the system meets business
requirements. UAT is managed on an ongoing basis and MS Excel is used as the primary tool for test case creation, execution, and
maintenance
3. Approve the
Provide 10. Publish test
MN.IT Test Test Strategy,
notification as results and build
Approach, and
Leadership to a new build acceptance
Plan,
1. Create Test
4. Create the test
MN.IT Test Strategy, 6. Create
Approach, and scenarios and
test data
Team Plan,
cases 9. Review
7. Execute and
test cases Approve
2. Review and provide 5. Review and test results
Business input on the test provide feedback on
strategy, approach, the test scenarios
SMEs
and inputs and cases
Vendor 8. Support test

execution
Teams
Legend
Inputs / Supporting
Outputs Outputs
- 102 -
Proposed Release Management Process Flow
The objective of this process is to manage the migration of software changes developed and deployed in the form of packages
released to the production system. The process is managed on an ongoing basis. ClearQuest and JIRA are the tools used for release
management.
Release 2.
3. Develop 6. Deploy
Plan 1. Plan Develop 8. Close
MN.IT Release Release
Deployment Release Release
Plan
Schedule
4. Approve
PMT No
Release?
Yes
Release
Vendors Schedule
PMO 7. Communicate
Communications Release
Manager
Legend
Inputs / Supporting
Outputs Outputs
- 103 -
Appendix B:
Roles and Responsibilities
Roles and Responsibilities – Status reporting, vendor management, work plan, and
requirements RACI Matrix
The RACI matrix identifies the governing groups, agencies and stakeholders involved in governance that are responsible,
accountable, consulted and informed
MN.IT MNsure PMO

IT Vendors
MNsure
MN.IT
DHS
MNsure Project Activities
Compile and Verify Raw Metrics Data C C C C A, R

Populate Project Status Reports C C C C A, R
Analyze Metrics and Complete Report C C C C A, R
Distribute project status report to appropriate internal and external stakeholders C C C C A, R
Status reporting C C C C A, R
Vendor management A C C I R
Work plan management A C C C R
Work plan - updates and maintenance I I C A R
Develop Requirements R A I C I
Legend
R Who is Responsible? The person or entity assigned to do the work
A Who is Accountable? The person or entity that makes the final decision
C Who is Consulted? The person or entity assigned that must be consulted before a decision or action is taken
I Who is Informed? The person or entity that must be informed that a decision or action has been taken
- 105 -
Roles and Responsibilities – Risk Management RACI Matrix
MN.IT MNsure PMO

Project Management
Project Team
Team
Identify and analyze risk C A R

Determine risk severity A C R
Develop risk response for high level risk A C R
Monitor risk A C R
Manage risk A C R
Close risk A C R
Incorporate risk information into weekly status report A C R
Legend
- 106 -
Roles and Responsibilities – Issue Management RACI Matrix
MN.IT MNsure PMO

Project Management
Team
CCB
Record and validate issue A R C

Assign and prioritize issue A R C
Perform issue management A R C
Track and follow-up on issue R A C
Determine if issue requires a change request A R C
Close issue A R C
Document closed issue A R C
Incorporate issue information into weekly status report A R C
Legend
- 107 -
Roles and Responsibilities – Change control RACI Matrix
PMO Communications
MN.IT MNsure PMO
Change Requestor
Manager
CCB
Complete change request form R A C I

Review change request and perform impact analysis C R A I
Accept or reject change request C R A I
Draft change request communication C R C A
Approve and send change request communication C I A R
Maintain change requests in JIRA C R A I
Incorporate change request information into weekly status report A R C I
Legend
- 108 -
Roles and Responsibilities – Defect management RACI Matrix
accountable, consulted and informed.
MN.IT Defect Manager

MN.IT Defect Team
Business SMEs
IT Vendors
Drive defect triage calls A R I C

Conduct defect triage R A I C
Participate in defect triage R A C C
Prioritize defect A R I C
Provide input to the defect plans and resolution details C R I A
Create defect resolution and retest plans A R I C
Retest defects and track defects to resolution and closure R A I C
Support defect management and Q&A C R I A
Validate and approve defect closure C I A R
Publish executive and detailed defect dashboards A R I C
Incorporate defect information into weekly status report R A I C
Legend
- 109 -
Roles and Responsibilities – Testing management RACI Matrix
accountable, consulted and informed.
MN.IT Test Leadership

MN.IT Test Team
Business SMEs
IT Vendors
Write and execute UAT Scenario A C I R

Oversee UAT process R A I C
Test reports for prioritization and decision making A R I I
Perform load and performance testing A C I R
Produce UAT reporting and dashboards R A I C
Create Test Strategy, Approach, and Plan R A I C
Review and provide input on the test strategy, approach, and inputs C R A C
Approve the Test Strategy, Approach, and Plan A C R C
Create the test scenarios and cases R A I C
Approve the test scenarios and cases C R A C
Create test data R A I C
Execute test cases R A I C
Support Q&A for test execution C R I A
Review and approve test results C R A C
Publish test results and build acceptance A R I C
Legend
- 110 -
Roles and Responsibilities – Release management RACI Matrix
PMO Communication
Project Management
IT Vendors
MN.IT
Team
Team
Plan release R A C I
Develop release schedule and deployment plan R A C I
Review and approve release R A C I
Deploy release A R C I
Develop release notes R A C I
Draft release communications C R C A
Approve and send release communications C A C R
Close release I R C A
Document closed release in release management plan R A C I
Incorporate release information into weekly status report R A C I
Legend
- 111 -
Appendix C:
Interviews Conducted
Interviews Conducted
No. Organization Interview Subject Interview Date

1 MN.IT Testing May 1, 2014
2 MN.IT Governance May 2, 2014
3 MNsure Deployment Process May 2, 2014
4 MNsure Governance May 5, 2014
7 MN.IT Federal Hub Discussion May 7, 2014
9 DHS Governance May 7, 2014
12 MN.IT, IV&V Testing May 7, 2014
13 DHS Conversion Strategy May 7, 2014
14 EngagePoint Release Management May 7, 2014
- 113 -
Interviews Conducted (cont.)

18 MNsure Carrier Communication May 8, 2014
23 Connecture Release Management May 8, 2014
24 Connecture Testing May 9, 2014
26 Counties Governance, communication May 9, 2014
27 EngagePoint Governance May 12, 2014
28 IBM/Cúram Release Management May 12, 2014
30 Board of Directors Governance May 12, 2014
31 DHS, MN.IT Testing May 12, 2014
32 Carriers Governance May 13, 2014
33 MNsure Testing May 13, 2014
- 114 -
Interviews Conducted (cont.)

34 EngagePoint Testing May 13, 2014
35 IBM/Cúram Governance, communication May 13, 2014
36 DHS Testing May 13, 2014
37 PwC Governance May 13, 2014
38 DHS, MN.IT, MNsure Testing May 13, 2014
39 EngagePoint Testing May 13, 2014
40 Navigators Governance May 13, 2014
41 IV&V Governance May 14, 2014
42 DHS Testing May 14, 2014
44 DHS, MN.IT Governance May 15, 2014
45 Connecture Governance May 15, 2014
48 MNsure Brokers June 3, 2014
- 115 -
Disclaimer
This document may contain Confidential Information and is intended strictly for MNsure’ s internal use and
not for any third party. As such, Deloitte is not, by means of any resulting publication of this document,
rendering professional advice or services to any third party. Any resulting publication should not be used
by any third party as a basis for any decision or action that may affect its business. Third parties should
consult a qualified professional advisor before making any decision or taking any action that may affect its
business.
Deloitte shall not be responsible for any loss sustained by any third party who relies on any resulting
publication of this document.
About Deloitte
Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by
guarantee, and its network of member firms, each of which is a legally separate and independent entity.
Please see www.deloitte.com/about for a detailed description of the legal structure of Deloitte Touche
Tohmatsu Limited and its member firms. Please see www.deloitte.com/us/about for a detailed description of
the legal structure of Deloitte LLP and its subsidiaries. Certain services may not be available to attest clients
under the rules and regulations of public accounting.
- 116 -

Program and Project Management Assessment Report

Uploaded by

Copyright:

Available Formats

Program and Project Management Assessment Report

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Program and Project Management Assessment Report

Uploaded by

Copyright:

Available Formats

MNsure Phase II Project

Deliverable #2 – Program and Project

July 11, 2014

Document Name Deliverable #2- Program and Project Management Assessment

Document Edit History

Version Date Additions/Modifications Prepared/Revised by

1.0 6/03/2014 Draft Deloitte Consulting LLP

Sections Page Numbers

Project Background and Objective

Inputs and Activities Outputs

Governance Defect Management

Key project governance and organizational roles are Partially present

ID Observation Impact Considerations

ID Observation Impact Considerations

Executive  The MN.IT agency, as the State’s Information

Project  The Executive Steering Committee has operational

 The Project Management Team serves as the Project

 The Project Director who is staffed by MN.IT as part of

Governance Model Detail – Senior Level

 As it relates to the project, the Business Owners

Role for the MNsure Project

Members Roles and Representation Key Responsibilities

Role for the MNsure Project

Members Roles and Representation Key Responsibilities

Key Relationships Key Decisions

Participant Roles and Representation Key Responsibilities

 One meeting per week

Members Roles and Representation Key Responsibilities

Key Relationships Key Decisions

Top 5 Positions and Descriptions

ID Observation Impact Considerations

MN.IT MNsure Project

Risk and Issue Management

MN.IT MNsure Project

MN.IT MNsure Project

Business SME’s IT vendors Technology staff

Role for the MNsure Project

Members Roles and Responsibilities Key Responsibilities

 Project Management Team: Once change requests are approved

MN.IT MNsure Project

MNsure/DHS MN.IT Defect

MN.IT Defect MN.IT Defect

Role for the MNsure Project

Members Roles and Representation Key Responsibilities

Key Relationships Key Decisions

 Defects triage outcomes are confirmed

Defects by Priority Defects by Severity

Re-Prioritized Resolution Criteria:

Attached are the prioritization results:

MN.IT MNsure Project

MNsure/DHS MN.IT UAT

Role for the MNsure Project

Members Roles and Representation Key Responsibilities

Key Relationships Key Decisions

 Confirm that entry criteria are met to start UAT

MN.IT MNsure Project

MNsure/DHS MN.IT Release

Role for the MNsure Project

Members Roles and Representation Key Responsibilities

Key Relationships Key Decisions