FIVE YEAR STATE PLAN FFYs 2017 – 2021
SECTION V. EVALUATION PLAN
1. Outline how the Council will examine the progress made in achieving the goals of the State Plan. Since 1997 the Council has applied the National Baldrige Criteria for Performance Excellence, which is aligned to the Government Performance and Results Modernization Act of 2010 (GPRMA). The Council's State Plan, Annual Work Plan, Monthly Activity Reports, and Logic Model for each goal are all aligned to Baldrige and GPRMA.
The Council spent a year developing the current State Plan and began with a thorough understanding of past performance of both qualitative and quantitative results at a full-day meeting of past sub-grantees. The Council then received and reviewed the results of both an individual and family survey (n=531) as well as a provider survey (n=286) to provide a state of the state review. The goals were crafted by the Council members and presented for a public review and comment period. The objectives, rationale, allocations and logic models for every goal were reviewed and approved by the Council.
Once the State Plan is approved then the Council will move into preparation of RFPs and selection of sub-grantees. Performance contracts grounded in the AIDD performance measures and Baldrige principles are developed. Continuous quality improvement is stressed at every step. The Grant Review Committee and the full Council are involved in reviewing progress toward goals.
2. Describe how the Council will assess the effectiveness of the strategies used that contributed to achieving the goals of the State Plan. According to the Baldrige Criteria, effectiveness is defined as how well a process or a measure addresses its intended purpose. Determining effectiveness requires evaluating how well the process is aligned with the organization's needs and evaluating the outcome of the measure.
Assessing the effectiveness of strategies begins with the performance results reported by sub-grantees. If results are missing the Individual and Family Advocacy and the Systems Change performance targets and measures, then strategy or work processes should be reviewed.
Monitoring effectiveness of strategy or work processes begins with the sub-grantee. If the monthly or quarterly reports show any trend of lagging, then Council staff must review and discuss with the sub-grantee to determine if it is a systemic issue or another type of barrier. Any deficiencies in performance come before the Grant Review Committee first. Every effort must be made to assist sub-grantees in meeting performance measures. The Council can offer assistance of experts on the topic for training and technical assistance. Additional information is contained below in answer to questions 5-8.
3. Describe how the Council will examine the progress made in achieving the outcomes of the self-advocacy goal. The Council has been funding various self-advocacy efforts since the mid-1970s with a public education and media campaign that people with developmental disabilities are people first. Self-advocacy has grown over the years culminating in the reauthorization of the DD Act in 2000 mandating that Councils meet new federal requirements. The self-advocacy goal monitors progress of self-advocacy following the same processes outlined in questions 5-8 below.
The Council will follow its RFP process and supplier management system. Performance contracts will be executed that contain the federal performance measures. Tracking will be required of the number of people participating, customer satisfaction, training evaluation, and IPSII measurement. Reporting will occur on an ongoing basis and a mid-year face to face performance report will be required with the Grant Review Committee. At a systems level, self-advocacy is contained in the Olmstead measurable goal area of community engagement.
4. How will the annual review identify emerging trends and needs to update the CRA? In following the Baldrige Criteria, the Council undertakes ongoing and systematic environmental scanning which includes daily reviews of national listservs for news and updates (i.e. every Council member receives Inclusion Daily Express). As noted earlier, Council staff reviewed hundreds of Legislative reports, websites, and needs assessments to prepare the State Plan Comprehensive Review and Analysis. This process includes regular reviews of key state agency websites, the Legislative Reference Library acquisitions, and national PNS data collection websites.
The Council also sponsors customer and market surveys annually that enable in-depth study of a specific trends or needs such as employment, health care, IT, special education or public attitudes. The survey results are always presented to the full Council meeting, summarized in news releases and for the Council website. Copies are disseminated broadly through public media.
The Council receives regular updates at every meeting about the progress of the Jensen Agreement and the Olmstead Plan including: (1) progress toward the measurable goals; (2) the longitudinal study of Quality of Life outcomes; (3) the semi-annual Jensen reports to the Court and the quarterly Olmstead reports to the Court; (4) the semi-annual Status Conferences held with the US District Court; and (5) the in-depth reports submitted to the Olmstead Subcabinet. Reports to the Court provide information in three categories that are relevant for the Comprehensive Review and Analysis: (1) movement of people with developmental disabilities from segregated to integrated settings; (2) movement from the DD waiting list; and (3) any and all Quality of Life results. The Olmstead Plan has been another avenue to receive "real time" data from eight state agencies reporting about 39 measurable goals. The Court retains jurisdiction over the case until December 2019.
5 and 6. Explain the methodology (qualitative and quantitative) that will be used to determine if needs are being met and if Council results are being achieved. Include evaluation of consumer satisfaction. Since 1998, the Council has utilized quantitative and qualitative data to measure federal DD Act outcomes of independence, productivity, self-determination, integration and inclusion (IPSII).
In 2015, a qualitative study was conducted with 531 people with developmental disabilities and family members and 286 providers. The summarized results support the need for the federal lawsuit (Jensen Settlement Agreement) and court oversight of the Olmstead Plan, as a means to prompt statewide improvements in IPSII.
Every sub-grantee must report on applicable performance measures, use the federal customer satisfaction survey and the Council's IPSII pre- and post-evaluation forms where applicable. These results are submitted in program reports to the Council staff and summarized in reports to the Council (additional details in next section). Data is collected on an ongoing basis. Results are summarized in monthly activity reports, then mid-year supplier results, and then into annual Business Results, the federal Program Performance Report and the Council's Annual Report. The Baldrige Business Results present data in graphic format, showing trend lines for key business measures including IPSII results.
Customer satisfaction forms are reviewed for compliments or complaints that provide ideas for improvements or actionable items. Stakeholder Survey results are also reviewed to find opportunities for improvement. Continuous quality improvement experts often review all results to gain additional insights and objective verification. Return on Investment measures for Council results are also calculated in consultation with continuous quality improvement experts.
The Council can also monitor issues at a systems-level, not just Council results:
(1) The annual customer or market survey foster in-depth review of systems-level topics, providing the Council with both quantitative and qualitative feedback. For example, the Council experimented with a Narrative Research method to study education issues. Both quantitative and qualitative insights emerged from the stories and experiences of students and families. This initiative was a first for the Minnesota Department of Education.
(2) Finally, the Minnesota Department of Human Services adopted the National Core Indicators and the Olmstead Subcabinet has adopted a longitudinal survey approach to measure quality of life. These additional surveys will be important sources of information for identifying needs at a systems level.
7. and 8.Describe the Council's processes, procedures and role in reviewing and commenting on progress toward reaching the goals in the Plan. In following the Baldrige Criteria, the Council has aligned its approach to a "supplier management system" for all sub-grantees which begins with performance contracts that are aligned to the State Plan.
The Grant Review Committee is involved in the oversight of the supplier management system and the full Council can review results on an ongoing basis.
The Council expects that sub-grantees are in full compliance with all applicable state and federal laws, and other contract requirements. This is true regardless of the dollar amount involved, or the type or size of the business. These requirements include a record keeping system best suited to the business in order to monitor and track progress toward goals, monitor income and expenses, and other programmatic, financial and business transactions.
The Council maintains an online resource library of OMB Circulars, Federal Regulations, Questions and Answers about Sarbanes-Oxley, IRS publications, Minnesota statutes, small business development resources, and Minnesota nonprofit resources. This packet of information was converted to an online resource for quicker access.
Another part of the supplier management system is reporting based upon the contract terms including use of the AIDD customer satisfaction survey and federal outcomes of IPSII, and applicable performance measures. These reports accompany the financial reports. Performance reports are summarized and presented to the Grant Review Committee.
Each sub-grantee is expected to present results in-person to the Committee. This report includes updates, results achieved, process improvements and ideas or suggestions that could generate greater IPSII results. These in-performance reviews are summarized and presented to the full Council. In preparing for this State Plan, all suppliers presented at an all-day meeting in October 2015 so that all Council members were briefed on both the quantitative and qualitative results.
The Grant Review Committee reviews results and allocations annually at two separate meetings, with a preliminary allocation process in June and final funding recommendations and Council approval at the August Council meeting. All performance results are summarized for the full Council to review prior to approval of final funding recommendations.
The above process is summarized in an Annual Work Plan approved by the Council at the Annual meeting held in October. The Annual Work Plan is aligned to the Baldrige Criteria category headings so that all aspects of the organization are addressed. The State Plan goals are embedded in the Annual Work Plan.
Monthly Activity Reports are distributed at each Council meeting; these Reports are aligned to the Baldrige Criteria and the Annual Work Plan. These Activity Reports summarize progress data on goals, objectives, evaluation data, customer satisfaction data and IPSII results. The Executive Director's report at every Council meeting allows time for discussion of all results presented in the Monthly Activity Reports.
The annual results of monthly reports and all sub-grantees are also summarized into Baldrige Business Results for the full Council in December. These Business Results are posted online and the Annual Report is also posted. The Business Results are fully discussed, especially any exceptional performance (either above or below goals).