:: 1 Electronic Government Services - Customer Evaluation prepared for: Minnesota Governor's Council on Developmental Disabilities prepared by: project :: 1504 9 September :: 2003 :: 2 table of contents page 1 :: project overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 :: summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3 :: important criteria for evaluating EGS . . . . . . . . . . . . . . . . . . . . . . . . . 9 4 :: customer feedback on current sites . . . . . . . . . . . . . . . . . . . . . . . . . . 19 disability services - california & minnesota special education - oregon & minnesota rehabilitation services - texas & minnesota councils on developmental disabilities - north carolina & minnesota 5 :: customer hopes and visions for future EGS . . . . . . . . . . . . . . . . . . . 28 :: 3 1 :: project overview :: 4 Project Overview OBJECTIVES: The Governor's Council on Developmental Disabilities wants to support the advancement of Electronic Government Services (EGS), and increase the number of their primary customers who use EGS. From the government's viewpoint, EGS improves efficiency, accountability and responsiveness of government agencies while reducing costs. From the customer's viewpoint,EGS gives the citizen choice, convenience and control over relationships with government. The objective of this study is to evaluate the current state of EGS, and develop an understanding of the needs, requirements, expectations and desires of customers related to EGS. There have been top-down agendas from both the federal and state government leadership, especially the Ventura and Pawlenty administrations,encouraging Minnesota government agencies to develop Electronic Government Services (EGS). From the government's view point, EGS: Improves the delivery of services and products Reduces transaction costs (less expensive administratively) Is a strategic resource rather than an expense. (Parallel web and paper systems should be eliminated) Improves efficiency, accountability and responsiveness of government agencies Streamlines processes by causing reengineering to occur first Reduces bad-check processing time and expense, and reduces time in recognizing money From the customer's viewpoint, EGS: Gives the citizen choice, convenience and control over relationships with government Improves customer access and good will by reducing travel and waiting time Allow customers to get answers at Internet speed Is available 24/7 :: 5 Project Overview OBJECTIVES: Continued … The Governor's Council on Developmental Disabilities (GCDD) supports the EGS directive from the government leadership, and since 1997 Council funding has been shifting to EGS; but they are not aware of any studies that validate EGS for their primary customer group (individuals with developmental disabilities and their families). Over time, individuals with developmental disabilities and their families have used the web at increasing rates. In 2002, the GCDD had 120,000 customer contacts with 20% coming from EGS. In February 2000, the GCDD's own survey of 157 Partners in Policymaking(r) graduates, showed computer usage rates comparable to the national norms: 80% had access to computers 76% had access to the Internet 70% used e-mail In 2003, the GCDD had a total of 87, 261 unique visitors to its Web site and a total of 82,781 downloads. 37 :: 37 Print Publications Compared to Electronic Version The following is an example comparing print and electronic versions of one of the GCDD's publications across several factors. MARKET GROWTH 68,883 Unique Visitors 2907 Requestors AVAILABILITY 24/7/365 8/5/250 CYCLE TIME 3 clicks 3 days It's My Choice 75,000/3 Staff 20,000/3 Staff PRODUCTIVITY $2.27 $9.42 COST It's My Choice EGS PRINT FACTOR SYSTEM IMPROVEMENTS (ROI Impacts) (11 months FFY 03) :: 6 Project Overview PROJECT IMPLEMENTATION: # of completes This report contains the results of qualitative interviews conducted in the summer of 2003. Some consideration should be given to interpreting the findings of this research as it is qualitative in nature and based on subjective interpretations. Customers evaluated current Minnesota sites along with "best practices" counterpart state sites for four agencies: Disability Services, Special Education, Rehabilitation Services and the Council on Developmental Disabilities. They first reviewed the sites on their own using a set of attributes to help ensure thorough evaluations. This review process was then followed by in-depth personal interviews or mini groups. Respondent interviews were completed among the following customers: Parents of children with developmental disabilities 5 Parents of individuals in transition age with developmental disabilities 10 Adults with disabilities 3 Schools 3 Residential providers 5 Day programs 4 The qualitative interviews included both in-depth personal interviews which averaged one hour in length and mini focus groups which averaged one and a half hours in length. All interviews were conducted by professional moderators. Prior to articipating in the qualitative discussions, respondents reviewed the web sites of four Minnesota government agencies. The sites included: Disability Services Special Education Rehabilitation Services Council on Developmental Disabilities. For each agency, the respondents also reviewed the counterpart web site in another state. The counterpart web nsites were selected as "best practices" sites. This designation was determined from a previous evaluation conducted by an external consultant commissioned by the GCDD. For that evaluation, searches were conducted in all four categories up until the point where three sites had been identified in each category that scored well on a pre-determined set of criteria. These criteria included: Quality of content Ease of intuitive use Navigation speed Frequency of updates, Accessibility, Links to other information, sites Number/Organization/quality of links Positive portrayal of people with disabilities Overall :: 7 Project Overview PROJECT IMPLEMENTATION: Continued … A total of 45 states were visited before three were identified in each category. For the purpose of this research study, the top state in each area was selected for review. In order to give the respondents some framework for reviewing these sites prior to the interviews, they were given a list of attributes to help guide their review of each site. The list of attributes was a detailed and somewhat expanded list of the criteria used in the earlier review process. The additional attributes included: ability to locate site, interactive tools, ability to apply for services directly on-line, opportunity for citizen feedback, layout is visually pleasing and language is appropriate. Respondents were asked to rate each site on each attribute using a scale from -3 to 3, where -3 was the most negative, 0 was neutral and +3 was the most positive. The results of this preliminary review process were used as an interview discussion tool to assure respondents were adequately able to discuss each site, and not a quantitative survey instrument. Since the review of all eight sites was a very lengthy process, respondents were not required to complete all ratings of all sites. They were however encouraged to write notes so they could discuss their opinions of the eight websites during the research interviews. See an example survey instrument attached as an appendix to this report. :: 8 2 :: summary :: 9 Summary Respondents viewed EGS broadly as all services the government could provide electronically. The main components of this included: • information (documents, Frequently Asked Question's, program descriptions and eligibility criteria, etc.) • transactions (application for services, on-line courses/training, etc.) • connections Based on this consumer research the sites now provide information, transactions and links. There are also opportunities for improvement and expansion of web sites to encourage greater use of EGS. The good news is: • Customers provided clear, concrete direction • There is considerable consensus as to improvement opportunities Overview There is a huge opportunity for agencies to provide further Electronic Government Services to adults, family members and professionals who work with individuals with developmental disabilities. The Internet was recognized as having a tremendous potential to provide information and services to this group - This is an important project. Broad definition of Electronic Government Services Respondents viewed EGS broadly as all services the government could provide electronically. The main components of this included: • information (documents, FAQ's, program descriptions and qualification criteria, etc.) • transactions (application for services, on-line courses/training, etc.) • connections. These would "from the government's viewpoint, improve efficiency,accountability and responsiveness of government agencies while reducing costs. From the customer's view point, give the citizen choice, convenience and control over relationships with government." Providing information electronically as compared to either verbally in-person or by phone or through publication distribution can and has significantly reduced government costs while increasing service to customers. 10 :: 10 Summary 2. Make it nice • Visually pleasing • Positive portrayal of people with disabilities • Feedback opportunity for visitors Summary of Important Criteria for Government web sites and EGS. 3. Building Opportunities • Connections- facilitate communications between customers • Transactions • Organization of information (Information management) • And more Information! 1. The Basics - Help Find More Information • Contact information • Able to locate website • Navigation • Information • Links Help Help Find Find Information Information Note on Accessibility- This is a critical attribute to the individuals who require it to visit websites. Sites need to cover the basics first, then they can proceed up the ladder, making the sites friendly and developing additional opportunities. Some customer comments on Information: Information is power…making information unavailable or hard to find reflects a State that doesn't want to help. When they make information so buried, it seems they're trying to discourage people from applying for services. Note on Accessibility: While "Accessibility" or Bobby approval did not emerge as important criteria to the group overall, to the sub-group that requires this in order to navigate sites, it was of course the most critical attribute. 11 :: 11 3 :: 3 :: important criteria for evaluating EGS 12 :: 12 What criteria are important in evaluating a government web site? Easily navigated More, current information Links Contact info Visually pleasing Positive portrayal Feedback opportunity Easily located Eight broad criteria were identified as important to these customers. The first five are "the basics", and the last three are "making it nice". Based on the research, eight overall criteria emerged as being important. These were attributes the respondents described in response to the following questions: •How do you evaluate a government web site overall, that is, what's important to you? •What specific attributes are part of your evaluation criteria? •When it comes to electronic government services what's important to you? •What are your needs, requirements and expectations? 13 :: 13 What would be excellent execution? Easily navigated More, current information Links Contact info Visually pleasing Positive portrayal Feedback opportunity Important Criteria Easily located Most customers searched for government sites through search engines, such as Google, Yahoo,MSN. The web site should come up in the first page of options, preferably in the top three to five options, ideally in the first spot. The site needs to be easily accessible from the State's homepage, one click is ideal, two is good. Respondents were instructed to locate the web sites in two ways: "We are going to ask you to locate information in four (4) different areas on state government web sites. First, please locate the information as you normally might locate government information on the Internet. Secondly, if you do not normally locate information through the state's main executive branch web page, please then locate the information through this method." Access from State's homepage customer comments: "Disability pages shouldn't be buried deep within the related websites." 14 :: 14 Easily navigated More, current information Links Contact info Visually pleasing Positive portrayal Feedback opportunity Easily located What would be excellent execution? Important Criteria Alphabetical topic lists (i.e. MN Disability Services) Search capability within website (i.e. Texas Rehab) Pop-up windows when scrolling over buttons (i.e. Texas) Sidebars (i.e. MN Disability Services) Clearly labeled buttons/links Clear organizational categories (i.e. Oregon Special Ed) Multiple ways to navigate (i.e. North Carolina Council on Developmental Disabilities) Clearly labeled buttons/links: Oregon's home page is clearly set-up, you don't need to guess. Search capability: Great search window (Texas) - can type in what you are looking for. It is upfront and visible, right off the bat. I don't have a ton of time to sit and dig around. Pop-up windows when scrolling over buttons: If the categories are too generic, the cursor should drop down an index of what you can do (like "How to apply for services") Buttons are okay, but list subjects underneath or do a sidebar pop-up, like in Windows. Then you know what's there and you don't have to go back and forth. Multiple ways to navigate: There should be more than one way to navigate. People aren't all the same,they think differently. It doesn't hurt to have more than one way to get somewhere. 15 :: 15 Easily navigated More, current information Links Contact info Visually pleasing Positive portrayal Feedback opportunity Easily located What would be excellent execution? Important Criteria Language/Information appropriate for consumers as well as professionals Current •put the date it was updated clearly on the front page. •current information only, remove outdated information. More Information •Disabilities (from research to sociological) •Programs •Legislation, budgets •etc. Information is very important to these customers, and they are constantly looking for more information. Current: They especially need to keep up to date with all the budget cuts -- what programs are still active, and so on. You don't want to read a bunch of information and then realize it's not current. More Information: Information is power. Does the state want an empowered citizenship? When information is hard to find, it can cut down on the number of people applying for services. You can't have too much information. 16 :: 16 Easily navigated More, current information Links Contact info Visually pleasing Positive portrayal Feedback opportunity Easily located What would be excellent execution? Important Criteria Links to independent agencies, advocacy groups The Arclink.org is consistently cited as the "best practice" standard for links in terms of quality, quantity and organization. Customer comments: There should be links to independent agencies, not just government agencies. The Arc is a good example, they have great links. 17 :: 17 Easily navigated More, current information Links Contact info Visually pleasing Positive portrayal Feedback opportunity Easily located What would be excellent execution? Important Criteria Contact information specific to different functions/needs. Easy to find/on home page. Multiple contact options - phone and online. Detailed information and photos of contacts - more friendly. Ensure ability to respond to contacts in a timely manner. Customer comments: There was a feedback button, but I want a name and a phone number. This is not an unreasonable request. Disabilities are very personal. They should have more than one way to contact them. It's important that you can contact them online. It's easier, quick, less of a hassle. Also, you can be more assertive in your opinions. Convenience, I'm busy until the kids are in bed, before 9:00pm my time is not my time. You need to be able to respond. Don't ask for comments, questions or whatever unless you can respond and in a reasonable amount of time. An easy to find phone number right at the top. The consumer has the right to be lazy. 18 :: 18 Easily navigated More, current information Links Contact info Visually pleasing Positive portrayal Feedback opportunity Easily located What would be excellent execution? Important Criteria Color Uplifting, inviting, friendly, not cold or "bureaucratic" Graphics, pictures •To help organize the site (i.e. Oregon Special Ed) •To let people identify with site, "know it's for people like me" •To break up the text Text size (i.e. North Carolina Council on Developmental Disabilities) Customer comments: It's important that it looks good. Has to invite you to go inside. If it's boring then you don't want to look, especially for people with disabilities. Helps it be less intimidating. 19 :: 19 Easily navigated More, current information Links Contact info Visually pleasing Positive portrayal Feedback opportunity Easily located What would be excellent execution? Important Criteria Pictures (i.e. MN GCDD) Success Stories (i.e. MN GCDD) Customer comments: If there are no pictures of people with disabilities and no other portrayal of them, the only impression you get of people with developmental disabilities is that these are needy people who need a lot of services. 20 :: 20 Easily located Easily navigated More, current information Links Contact info Visually pleasing Positive portrayal Feedback opportunity Survey A close-ended survey makes it quick and easy to provide feedback (i.e. Texas Rehabilitation Services) What would be excellent execution? Important Criteria Open-ended text blocks to: •Offer specific compliments •Address specific concerns/lodge complaints Offer feedback opportunities for: •The agency's performance •The services •The website Customer comments: Make it easier to comment. The braver they seem, you know they're trying to do the right thing. Encouraging feedback helps things get better. In the feedback page folks can participate in a survey about the website. Makes it very easy to give feedback. A feedback opportunity is not important until you're having a problem. A feedback link tells me they're measuring performance, tends to drive system to improve over time. In the feedback page folks can participate in a survey about the website. Makes it very easy to give feedback. 21 :: 21 4 :: customer feedback on current sites 22 :: 22 California: Developmental Disability Services What are they doing exceptionally well, that would be a good learning example? Contact Information Respondents found this site to be average as opposed to "exceptional". http://www.dds.cahwnet.gov/ California was selected as a "best practices" site based on Information content. "Extensive information on available services; laws and regs; basic information on DD; section for vendors; Consumer guide to the Lanterman Act; appeal rights (with PDF forms on-line); Andrews Closure Plan; direct support training with PDF courses on-line." Customer comments There's good 'contact us', there's a number of different options for contacting in different areas. 23 :: 23 Minnesota: Disability Services Division Strengths Opportunities for Improvement More information Information Navigation Links Visually Pleasing- Graphics, Color http://www.dhs.state.mn.us/contcare/disability Customer comments I liked this site - seemed fast - many links to glance at quickly. Content rich. Layout simple and easy to follow. The clickable bullets with simple titles, not long descriptions with too much to read. They have all the manuals online. I love that. I have printed them all out - all the laws. 24 :: 24 Oregon: Special Education What are they doing exceptionally well, that would be a good learning example? Navigation Links Contact Information Visually Pleasing http://www.ode.state.or.us/sped/ Customer comments The Oregon site was beautiful. It was easy to read. I knew what they were talking about. It was logically organized. It was written for someone new to special ed. It was very thorough. Liked this site. It's very important to have rights/ complaints / appeals right up-front and visible. Contact - it actually gives you a person's name! It has pictures of the contact people. These are real people - and it lists the staff. The site had great links. I'm always impressed when they have links to advocacy organizations. They had listservs - where you can get on and talk about issues with other people. Oregon seems very open. The front page is set-up well, the labels are clear. I like the visuals, very helpful. The state has a whole department (for special education), that's something a web site can't overcome. They even have a "Need Help" icon that is really useful. Once one clicks on this it has links alphabetically organized by subject. 25 :: 25 Minnesota: Special Education Strengths Opportunities for Improvement More information Improving Easier to find site Faster Contact information Visual- increase text size, use the whole page http://education.state.mn.us/html/intro_support_special_ed Customer comments It's a big improvement. Most information is clear. It's becoming more parentfriendly. 26 :: 26 Texas: Rehabilitation Agency What are they doing exceptionally well,that would be a good learning example? Navigation Visually pleasing Feedback Opportunity Accessibility http://www.rehab.state.tx.us/ Customer comments Great cover page. Makes it easy to navigate. The upper left corner has clear topics to click. It was simple, they didn't use jargon. Most anyone could understand it. It was visually pleasing. It's a big deal, the pages shouldn't be boring. In the feedback page folks can participate in a survey about the website. Makes it very easy to give feedback. Had a map, broke down information by location. It would be easy to find the services that are nearest to me. When the mouse scrolls over something, it gives you a little narrative with information that helps you. (In terms of accessibility) this was the closest I have ever come across an ideal site. It was very clean- there was no superfluous language. 27 :: 27 Minnesota: Rehabilitation Services Strengths Opportunities for Improvement Navigation- less information on homepage, have a short list of links in a sidebar to the left or right (not "embedded" in main body), a home button for Disability Services. Information More information- information on independent living, clearer listing and explanation of programs/resources. http://www.mnwfc.org/rehab/ Customer comments The quality of information for people who are blind is very useful, all people with disabilities should have this kind of access and attention. The print is good, not busy. They often have transition information. Good place to look for job coaches, volunteer or part-time opportunities. The rest of the site is much better. It's very clear. Just follow your own example. 28 :: 28 North Carolina: Council on Developmental Disabilities What are they doing exceptionally well, that would be a good learning example? Navigation Information Contact Information Visually Pleasing Positive Portrayal http://www.nc-ddc.org/ Customer comments This is a good site for consumers. "Participant" encourages people to get involved. Very easy to navigate. Liked having the sidebar links and the alternative of clicking. There are two ways to get to the information. The first thing you see is the contact number. Nice big layout, visually pleasing. It has a better opening page than Minnesota. The links, the site map. The sidebar makes it easy to link, Minnesota you have to try and guess. 29 :: 29 Minnesota: Council on Developmental Disabilities Strengths Opportunities for Improvement Navigation - more, clearer navigational links on homepage, navigational links in a clear sidebar Information Links - more links, a list of disabilities with a number of sites to link to Information/Links - on where to go if one needs help to obtain services Visually Pleasing Positive Portrayal Interactive tools Contact Information http://www.mn.gov/mnddc Customer comments I liked just about everything. I love the clips. I love the section on the history of disabilities. The timeline. Has a lot of wonderful, interactive tools. I like the "success" stories. They are very powerful. The pictures of the council members and the information about them, made it seem like regular people. Not intimidating, all PhD's. You can identify with them. 30 :: 30 5 :: customer hopes and visions for future EGS 31 :: 31 Customer hopes and visions for future EGS. These are the building opportunities. Customers described their hopes and visions for future EGS in four main areas: Information Content Organization of Information (Information Management) Connections Transactions 32 :: 32 Customer hopes and visions for future EGS. Information Content Basic information on disabilities and disability services/programs for those first diagnosed, basic "How To's Database of grievances and findings Eligibility criteria available on line Range of all residential opportunities in a compiled list Basic information on disabilities / First Diagnosis An ideal web site from the perspective of someone new, would have basic information, easy to find contact information, clean language and no acronyms. Basic "How To's": How to report abuse and neglect, how to make an appeal, how to write an advocacy letter, how to write a letter of medical necessity. Database of Grievances Minnesota used to have a special ed database of complaints that were filed and the findings. It's important to learn from problems. Eligibility criteria They should make the qualification criteria available right online. Especially for adults with developmental disabilities and families that aren't sure, new diagnosis, wondering if they qualify, what they would need to qualify. Range of all residential opportunities in a compiled list It would be nice if they had pictures of places because it's hard for some people to get around. 33 :: 33 Customer hopes and visions for future EGS. Organization of Information (Information Management) Organized to help people get the information they need •something that let's you dictate the level of information you want •something that sorts through all available programs from multiple agencies (cross silo) and helps identify which programs are better for each individual. Keep track of information centrally/Standardized information format Organized to help people get the information they need I'd like to see some sort of ratings scale. For in-depth information you type in a '4', for general information a '3', for newly diagnosed a '1', for technical information a '6', etc. So you can find the type of information you need given your situation. There needs to be something that helps kids with multiple disabilities. Something to help them sort it out and find what they need. There is lots of information on all types of disabilities and services. People need help matching things up. The computers could do this for you. It could create a matrix. You could use a work-flow engine to match data to corresponding services. A self-service optimized government web site. Keep track of information centrally/Standardized information format Keep the school, county and all the agencies information all together. They're asking for the same information over and over again every six months. Everything is a separate packet of forms this thick. Something that streamlines the process when you have multiple children with developmental disabilities. 34 :: 34 Customer hopes and visions for future EGS. Forums / Listservs Links to advocacy organizations Find your governor / legislator Connection Introduction service between users On-line mentoring Information on groups you can join Site for kids Services for other cultures/ Information on what services are available for different cultures/ Languages other than English Classifieds/ Donations or postings for used equipment Forums / Listservs There should be something that connects consumers with others that have been there and figured it out. Mercedesshop.com is an example of a forum with a search engine. You can type in a problem and read through 20 examples of how other parents have solved that. Most sites have information from an educational, classroom standpoint; rather than a real world, school of hard-knocks. Introduction service It would be nice if they could link you to people. Like, "Click here if you would like to be linked to other people who are looking for alternative housing." Links to people with common issues. They could collect the names. Site for kids There should be a site for kids. Something to help them learn about their disability. It should link to other web sites for kids, so they can talk about it. There are so many places that segregate them and don't let them be kids. Something that will help them with their self-esteem, because God doesn't make junk. Something for kids would pull all the families together. Services for other cultures There should be a link for other cultures. They should be able to contact an advocate or translator to understand what's available in their language. 35 :: 35 Customer hopes and visions for future EGS. Transactions Submit applications for services on-line Online status of application Downloads (Applications, legislation, medical information, etc.) See the example return on investment chart on the next two pages for a comparison of costs for print versus downloaded publications. Ability to apply for services on-line You should be able to apply for services right online. In the very least you should be able to download the forms and have everything completed before you go into an agency. Otherwise you're wasting everyone's time, including a government worker. Online status of application You should be able to find out the status of your application. They could give you a number and you could look it up online. It's impossible to get a person on the phone to find these things out. Thank you for agreeing to participate in this important Electronic Government Services Assessment survey. The purpose of this survey is to evaluate current EGS for individuals with developmental disabilities and to help Minnesota state agencies improve provided services. You will be asked to evaluate state government web sites in four different areas: • Developmental Disabilities Services • Special Education • Rehabilitation Agencies • Councils on Developmental Disabilities For each area you will be assessing both the Minnesota site along with the site of one other state. You will be locating and evaluating a total of eight (8) web sites. This should take you a total of approximately 45 minutes depending on the amount of time you spend at each web site. You do not have to evaluate all the web sites at the same sitting, but please complete the entire evaluation of all eight sites prior to your scheduled research interview. Your opinions are very important and we appreciate your effort. Please respond keeping in mind there are no right or wrong answers, it is your opinions that count. All of the information you share while participating in this survey will be kept confidential. If you have any problems or questions please feel free to contact Tom Senn at MarketResponse. He can be reached at (612) 379-1645 or by e-mail at t.senn@marketresponse.com. Project #1504 July, 2003 Example Evaluation Sheet - Page 1 INTRODUCTION Appendix We are going to ask you to locate information in four (4) different areas on state government web sites. First, please locate the information as you normally might locate government information on the Internet. Secondly, if you do not normally locate information through the state's main executive branch web page, please then locate the information through this method. Project #1504 July, 2003 Evaluation Sheet - Page 2 EASE OF LOCATING WEB SITE/INFORMATION This site/information was easy to find. I was able to easily locate this site/information starting from the state's main executive branch page. Minnesota California Rating Developmental Disabilities Services Sites Please locate the developmental disabilities services site for the states of California and Minnesota, then rate how well you agree with each statement using the scale shown above. Neutral Disagree Neither Agree Agree Completely or Disagree Completely -3 -2 -1 0 1 2 3 The following is one set of survey instrument questions. This same set of questions was used to evaluate and compare Minnesota government agencies for Disability Services, Special Education, Rehabilitation Services, and Council on Developmental Disabilities with the best practices Web site. I am now going to ask you to evaluate each site on a number of attributes. Please take some time and visit the two sites listed below, these should be the same sites you located earlier. DEVELOPMENTAL DISABILITIES SERVICES SITES • California: http://www.dds.cahwnet.gov/ • Minnesota: http://www.dhs.state.mn.us/contcare/disability Please rate how well you agree or disagree with each of the following statements using the scale. Neutral Disagree Neither Agree Agree Completely or Disagree Completely -3 -2 -1 0 1 2 3 Content Minnesota California Rating Project #1504 July, 2003 Evaluation Sheet - Page 4 This site has the right quantity of information; neither too little or too much. It contains the right information, the information I would want to find. I am able to easily apply for services/memberships/etc. directly online. This site has good extended services such as online courses and other interactive tools. This site contains good contact information. I am able to contact someone from this agency online. This site provides good opportunities for citizen feedback on the quality of the site. This site provides good opportunity for citizen feedback on the quality of the agency's performance. The site displays an appropriate understanding of and positive portrayal of people with disabilities. Minnesota California Rating Project #1504 July, 2003 Evaluation Sheet - Page 3 Navigation Speed The site has good accessibility. The site has notice of Bobby approval. Accessibility I am able to easily search for and find the information I need. The site contains good buttons and links to make navigation easy. The layout is clear, not distracting. The text size and type is easy to read. The layout is visually pleasing. The language is appropriate and easy to understand. Organization/Ease of Use The information appears very current. Frequency of Updates There are a good number of helpful links to other sites and agencies that I would want to search for. The links to other agencies and information are current and work well. Links to Other Information/Sites Overall I am very satisfied with this site overall. The site opens at a good/appropriate speed. The speed of the transitions and links is good/appropriate for my Internet connection. The speed of downloads of forms/publications is good/appropriate for my Internet connection.