Home : Staff : John Butterworth :

Employment Data Systems : Florida's Agency for Persons with Disabilities

By:

Originally published: 3/2011

Series Introduction

The increasing emphasis on government accountability at the state and federal levels has increased interest in and use of outcome data. Moreover, research has found that high performing states in integrated employment generally have a clear and visible data collection system that provides individual outcome data (Hall et al, 2007). But what are the most important elements in designing and using a system? Stakeholders have raised questions regarding creating effective data collection systems, identifying variables with the most utility for influencing policy, and using data as a strategic planning tool. This series is intended to shed light on the successes and challenges of day and employment systems across several states and to provide strategies for other states as they examine their own data collection systems and their impact on their employment priorities for individuals with intellectual and developmental disabilities (ID/DD).

Background

The state of Florida's Agency for Persons with Disabilities (APD) has implemented a five-year employment initiative for people with ID/DD. One goal is to enable at least 50 percent of adults (ages 18 to 55) receiving APD-funded day services (including adult day training, supported employment, and non-residential supports and services), as of July 1, 2004, to achieve community employment

by July 1, 2009. Florida is specifically targeting a total of 25 percent of the adult day training population to be employed by July 1, 2009. The employment initiative requires that each area office has a plan with target goals that is updated semi-annually. Florida's Supported Employment Tracking System (SETS) was developed to provide a comprehensive and accurate picture of the state's progress in reaching its employment goals.

Prior to 2004, the state had a data collection system in place but it was less automated and standardized. The new database that is now in place reflects a more defined commitment to accuracy and buy-in from all those involved with employment services.

Data system basics

The design of the data system

Florida is divided into 14 regions within the APD system. Each region has a supported employment liaison who is responsible for collecting employment data. Across the state, supported employment liaisons may have additional responsibilities beyond employment, such as transition- age youth or supported living, and thus may have limited resources to focus on employment data. Accordingly, the data collection processes within the regions can differ as well.

On a monthly basis, each supported employment liaison documents the outcomes of individuals within his or her region. This is done through outreach to supported employment service providers and support coordinators in the region who are supporting individuals who are employed and receiving services from APD. Information is also solicited from Home- and Community-Based Services (HCBS) Waiver support coordinators, who have knowledge of the employment status of the individuals that they support (every individual on the HCBS Waiver has a support coordinator). Support coordinators are an important source of information that can supplement information from the SE service provider and create a more complete picture of an individual's situation.

What data elements are collected

A chart detailing the data elements collected is included as an appendix to this brief. One attempt at standardization and shared definitions has been made through the development of a field definitions form that offers guidance on each field within SETS.

Standardization across the state

Initially, establishing the system and collecting the data was a more cumbersome process than it is today. Overall respondents felt that SETS was an efficient way to collect the data and one that has established some consistency and familiarity among the providers. At this point providers are aware that their SE liaisons will be in contact with them for a monthly check-in. While the process is more streamlined, it is not standardized across the state. Variation exists in how SE liaisons collect the data and how they view the data collection process. In one area, for example, a liaison felt that this monthly contact is an entree to talk about questions or concerns. She uses these consistent check-ins with providers as a built-in opportunity for training and technical assistance in specific areas.

Depending on the region and the provider, some providers mail their data, while others use email. In one region, each provider develops their own form based on the data that the SE liaison requests for each individual working. Providers submit the data to the liaison, and he enters it into a SETS hardcopy book, and then into the electronic SETS data systems. In another region, the SE liaison works with her IT staff to get a point-in-time picture of the data at the end of each month, and she works with providers, coordinators, and families to get an accurate update of this data each month. Once she gets the data, she enters it into the SETS database. When working with a new individual, a census form is filled out with additional information, such as demographic information, that does not change from month to month. She also maintains a list of providers that she updates monthly in the event that a provider is no longer operating or no longer supporting people in her region in employment, or in the event that a new provider enters the system.

Who the data is collected on

Data is collected on all individuals who receive services from APD and who are working in the community and on individuals who are eligible to receive services (on the waiting list) and who are working. "Working in the community" is defined as individual or group employment (not more than eight people as part of a work crew or enclave) or employment with or among people without disabilities that pays at least minimum wage. For individuals in job development, data is not entered on their activities until the day they actually start their jobs and begin receiving a paycheck.

Frequency of data collection

Employment data is collected by APD on a monthly basis. An SE liaison noted that the frequency of the data collection lets providers know that it is a priority - the intensity of monthly data collection conveys the importance of data to APD. In addition, regular interaction with providers helps to give her a month to month compass point on where providers are in terms of working towards their goals.

While monthly data collection is more frequent than in other states (e.g., NH collects data bi-annually, and MA annually), one provider said that submitting data on a monthly basis is not cumbersome because she simply submits what is required for an additional certification of her agency (e.g. Commission on Accreditation of Rehabilitation Facilities), and her regional SE liaison extracts the data that he needs to complete SETS. One support coordinator felt that monthly data collection misses some of the job turnover that happens throughout the course of the month and that a weekly data collection could provide more accuracy. The majority of interviewees felt that a monthly data collection enabled a sufficiently frequent picture of the changes that occur within individual employment situations.

Data sources

There are several potential sources for employment data: waiver support coordinators (for those on the HCBS Waiver), supported employment service providers, general revenue coordinators (who work with individuals on the waitlist), and families. One SE liaison felt that waiver support coordinators drive the system, as they have the central files on all waiver participants. APD uses several sources because in some cases, one type of respondent may be more effective in reporting certain data. This process also provides a useful accuracy check on the data when it is corroborated by more than one source or when one source may be difficult to reach.

An SE liaison who participated in this research described the sources she accessed in the data collection process. Her first contact is with the service provider. She noted that if there was no job coach or provider involved, she would call the support coordinator. If the individual was on a waiting list for services, she would call the general revenue coordinator. SE liaisons have different relationships with the coordinators. One SE liaison felt that she had good relationships with the waiver support coordinators but was unfamiliar with the general revenue coordinators. When necessary, she will approach the individual or family members for data. In only one case, she approached the employer (Goodwill) for data. Another liaison felt that he had very good relationships with the general revenue coordinators. These relationships are critical in facilitating effective communication and outcomes data collection.

A prominent issue that arose during the conversations with SE liaisons was the level of effort involved in obtaining data monthly from providers. The consensus seemed to be that after the initial implementation of the system, providers became familiar with the type of data that was necessary to submit and the process became more streamlined. SE liaisons described their style with regard to interacting with providers. One liaison often goes directly to the director of the employment provider to obtain data and emphasizes the importance of data collection for many reasons, including communication with state legislators regarding funding needs. Another liaison has closer relationships with employment specialists who often know more specific information about the individuals they support. Regardless of what level SE liaisons works with providers, communication was an important theme.

Linked systems

In some states, the employment data collection process is linked to other systems such as quality assurance or billing. Respondents note that employment is indirectly linked to the quality assurance process. Quality assurance staff work at both the state and regional level. A state-level staff person said that supported employment is frequently on the agenda at quality assurance meetings and is promoted and has become "an integral part of everything the agency preaches."

How the data is used, analyzed, and shared

The data system is primarily used to measure progress towards the five-year employment initiative. Supported employment liaisons are very aware of their regional goals around employment. One liaison mentioned that he currently had more than 300 people working but that this number was approximately 100 shy of his region's target goal. Several noted that these goals are shared with providers. However, when one provider was asked about Florida's data system, although she provides data to her liaison, she did not regularly receive employment data back from APD. She has been told that their organization and their region as a whole are behind on its goals, but has not received any more specific information. This provider noted that better communication would be an area for improvement around using data to measure performance.

While APD communicates about data with its regional liaisons, and through them, with their service providers, one group that seems less informed about the data is support coordinators. Upon asking about the state's data collection system, a support coordinator said that he had no idea how the state uses the data or how it benefits individuals. His only experience with data is the limited amount that he is able to provide on a monthly basis when contacted by the liaison. This coordinator had little awareness about the employment initiative in general, perhaps suggesting a need for APD to expand the involvement of support coordinators in the prioritization of employment services. An APD official noted that each region's plan documents regular meetings with support coordinators, which are an opportunity to discuss employment and employment data. While some regions are reported to have good communication with support coordinators about employment as a priority, this is not necessarily standard across the state.

A critical message: Data is important

All sources of data, be it support coordinators or providers, need to value and buy into the concept that data collection is a key element in moving employment forward. Toward that end, in the past few years support coordinator training has started to address supported employment and data collection. An APD central office staff noted, "Every chance we get, we hit on data collection." APD trains their staff around the state, who in turn work with support coordinators and providers on how to report outcomes.

Data became an especially pressing issue for APD as the agency was rolling out its five-year initiative, because without data APD would be unable to track its progress. One respondent noted that the importance of providing data is conveyed through the level of intensity that SE liaisons spend in obtaining accurate data. While many providers have adjusted to the process and expect to be submitting data on a monthly basis, others have to be consistently "chased down." It is likely that this is an ongoing issue across regions in Florida and across providers in general.

How the data is analyzed and shared

Employment data is gathered locally and analyzed by region. Each region can view its own report, and can also generate reports that are statewide. The analysis is done at the local level, both by the SE liaisons and the program administrator within the area office. At the state level, analysis is done within the Employment Unit of APD. While regional office staff have provider-level data, available data reports do not compare one provider to another across the region or across the state.

Regarding dissemination of the data, reports are typically produced on a monthly basis and go out to each region and staff within APD central office. Within APD, there is regular communication about data. Every other month a two-hour conference call, which includes all of the area agency offices, serves as a forum for data discussions. APD also makes use of its internet and intranet to highlight data. Line graphs documenting the state's progress in meeting its employment goals are posted on its public website. On its intranet, area offices have access to a table that provides employment data for each area.

External outreach is usually done on the basis of requests or through state-level or national presentations. Data is shared quarterly with counterparts from Florida's Vocational Rehabilitation agency. Regarding families and individuals, respondents felt that there is little interest in data as families are more concerned with individual services. Moreover, since data is not available at the provider level, data does not help individuals or families make decisions about from which agencies to obtain services. Family Care Councils within each region are a forum to make data more accessible and meaningful to families and individuals.

An analysis of the system: Successes and challenges

What's working well

Respondents could point to many positive attributes about their data system. The use of a few different sources was cited as a check on the data for greater accuracy and access. The frequency of data collection was also mentioned as a boon, and this helps to cement relationships with providers and keep them familiar with the data collection process. Also, this monthly "compass point" keeps everyone aware of progress towards goals and keeps the five-year initiative on the minds of APD staff and providers.

A central APD staff person said that the current system provides "basic numbers that they can feel confident about, in terms of number of people who are employed and number of people who are moving out of adult day training." These numbers speak to precisely what the initiative sought to expand; in that sense, the system is measuring what it is meant to measure.

Challenges within the system

Respondents spoke of several challenges within the system, existing at multiple levels. At the most on-the- ground level, SE liaisons can often feel unsure that they are counting everyone there is to count. People on the waiting list for services who may already have jobs may not be part of the system unless their general revenue support coordinators are communicating with the SE liaison in the region. Also, individuals who obtain jobs without the help or knowledge of a provider or a support coordinator may not be counted initially, although eventually the information would come to light during communication with the support coordinator.

A predominant concern was the difficulty regarding tracking down and obtaining data from numerous support coordinators and employment providers. An APD staff person referred to this as a "weakness within the system." Currently, providing data is not a requirement in the Medicaid Handbook that guides service delivery for providers. Respondents were mixed in terms of the credence they gave this issue. Some felt that if it was in the handbook, APD may have more leverage in enforcing the data requirement. Others felt that providers treated it as a requirement because they know it is a necessary condition for staying in the good graces of APD. An additional issue is turnover among support coordinators. Helping new providers and support coordinators get up to speed on how to provide data may produce some delays in getting an accurate count of the number of individuals working.

Another level of challenge is in the type and quality of data that APD seeks to collect. At one point in time APD was interested in collecting data beyond wages and hours, to illuminate some quality of life issues on the job (e.g., use of natural supports). While domains such as opportunity for career advancement are included in the system, often this data is not accurate or missing. This level of detail, while cumbersome for providers with large workloads, is important in understanding quality of the job and impact it may have on one's life. While a focus on key outcomes is important in measuring progress towards a major goal such as the five-year initiative, many felt that a data system could do more to shed light not only on how many people are working but on how individuals' lives are improved through such experiences.

At a system level, respondents spoke of great difficulty in sharing data with Vocational Rehabilitation (VR). According to the SE liaisons, there was an emphasis on communicating with VR providers, as opposed to local VR offices. Collaboration with VR is especially important because when an individual obtains a job, it is more than likely that APD is tracking a VR outcome as VR is the funder of Phase 1, or initial job development, and job support services. Once the individual stabilizes on the job and transitions to Phase 2, the individual is then in "follow along" and transitions from VR to APD funding. At the Central Office level, APD has been trying to work more closely with VR through joint trainings and other initiatives, but progress is slower regarding data sharing.

Moving forward: Areas for future development

Working with VR

In addressing the challenges of working with VR for accurate data, APD staff are working toward better communication towards that end. They are interested in collaborating with VR for access to information such as how long it takes to get through the VR system, where individuals may get delayed, and where improvements are made. These conversations have taken part at the state level but have not fully reached the depths of the systems. One recommendation that came out of this research is the development of close working relationships among local VR and APD offices in order to better share information.

At the time of this writing, Florida's VR agency is currently dealing with service capacity issues and is in order of selection, whereby VR is able to accept only those individuals with the most significant disabilities who could benefit from VR services.

Data that addresses quality of life

Another challenge that is being considered is access to data that addresses quality issues. Data that speaks to quality of the job is important and yet difficult to collect. A respondent noted that while these topics were present during initial discussion and implementation of the data system, the system has not yielded as useful data as had been hoped. Respondents said that measurement of quality issues has fallen somewhat short and requires rethinking in order to yield data that will provide insights beyond hours and wages.

Accountability

Several suggestions were made regarding accountability within the system. Because so much of the data collection process rests within the purview of each SE liaison, a breakdown could occur if the person responsible for data in the region has competing priorities. Accountability is not only an issue at that level, but also at the level of the support coordinators and the providers. As mentioned, data collection is not a true requirement for service coordinators or providers. While it is not officially a requirement, many believe that there is the expectation, and even the obligation, to provide accurate data on a consistent basis. Respondents discussed the possibility of making a revision in the Medicaid Handbook and instituting data collection requirements.

Lessons learned and implications for other states

Respondents from Florida shared the lessons they have learned from their experiences. These include:

Conclusion

While Florida's system has its challenges, stakeholders agree that the frequency of the data collection helps to keep a keen eye towards progress in meeting their five-year initiative goals. Other states that are in the earlier stages of developing employment data collection systems can learn from APD's experiences and use their data systems to increase accountability, and enhance communication around expectations and priorities for the system.

Appendix Data elements included in APD's SETS

Name SS# Employed? (Y/N) Employment Consultant
Employer Job Title Minimum Wage or Higher? Pay per Hour
Average Hours per Week Date of Last Raise Small Group of 8 or Less? Integrated Work Setting?
General Revenue (GR)
Employment Project?
Obtained Job Funding from Where?
(GR, Medicaid, None)
Maintains Job with Funding from? Career Advancement Opportunities?
Paid Vacation? (Y/N) Paid Sick Leave? (Y/N) Retirement? (Y/N) Health? (Y/N)
Other? (List) Lost Job? (Y/N) Reason Lost Job Client Plans
DVR Referral? Date Referred to DVR Original Appointment Date Eligibility Date
Date Employed Date Plan Developed Modification Date

Reference

Hall, A.C., Butterworth, J., Winsor, J., Gilmore, D.S., & Metzel, D. (2007). Pushing the employment agenda: Case study research of high performing states in integrated employment. Intellectual and Developmental Disabilities, 45(3),182-198.

Acknowledgements:

The authors would like to thank the stakeholders in Florida for their contributions to and review of this brief. This project was supported, in part, by cooperative agreement #90DN0216. from the Administration on Developmental Disabilities, Administration for Children and Families, US Department of Health and Human Services. The opinions contained herein are those of the grantee and project participants and do not necessarily reflect those of the Administration on Developmental Disabilities.

For more information, please contact:

Allison Hall
Research Associate
Institute for Community Inclusion
University of Massachusetts Boston Allison.hall@umb.edu