Share

Software and Qualitative Analysis

7. Drawing and Verifying Conclusions

Tables 1-3 summarize the findings for a research site in a large cross-case research project (see tables for a brief explanation). Each of these three tables approaches description from a different direction. Compare the differences in the kinds of things you can learn depending upon how you approach organizing the data.

Table 4 organizes participant responses to one of the programs, component by component. Table 5 is a different animal altogether: a time-oriented matrix summarizing the major phases of the project over time. (See Miles and Huberman for an extensive set of types and examples of matrices.)

Tables 1-7 Credit: National Implementing Evidence-Based Practices Project

Table 1

Characteristics of the Agency Overall—Site 2: This descriptive matrix summarizes the essential characteristics of one of the agencies. Each row summarizes the data available for each topic.

Click on the table preview below to see the full table:

Table summarizing characteristics of the agency as discussed in text

Table 2

Characteristics of the SMI Program—Site 2: In contrast to Table 1, this matrix summarizes the characteristics of the program rather than of the agency. What are the differences in what you can learn?

Click on the table preview below to see the full table:

Table summarizing the characteristics of the SMI Program as discussed in text

Table 3

Quality Improvement Systems—Site 1: This matrix summarizes the data on hand for various topics having to do with quality improvement.

Click on the table preview below to see the full table:

Table summarizing quality improvement systems as discussed in text

Table 4

Summary of Internet Survey: This site summary gives a composite view of participants' assessments of the program, component by component.

Click on the table preview below to see the full table:

Table summerizes internet surveys as discussed in text

Table 5

Project Timeline—Site 2: This time-ordered matrix summarizes milestones and critical events for the key time periods of the project.

Click on the table preview below to see the full table:

Table summarizing project timeline as described in the text

Table 1

Characteristics of the Agency Overall—Site 2

Source Topic Data
A.1 Legal structure Private nonprofit
A.2 Governance Local board of directors (The Board includs professors, professionals, attorneys, family members, consumers, city administrator, state department workers, scientists, etc.)
A.4 Number of clinical sites Two main sites. Services include: Services for Homeless and for Persons with Severe and Persistent Mental Illness (including an Afro-centric and Multi-cultural, Deal/HOH approach), Case Management, Community Treatment Teams, Mobile Psychiatric Outreach and Project Work (Employment Assistance); “Apothecare” Pharmacy; Residential Programs; General Information. Services at the 4th St. Location include: Chemical Dependency Outpatient Services; Criminal Justice Programs; HIV/AIDS Programs/ Older Adult Mental Health Services; GLBTQ Adult and Adolescent Programs/Services
A.5 Main setting type Urban
A.6 Type of setting of implementing site #2  
A.7 Type of setting of implementing site #3  
A.8 Type of setting of implementing site #4  
A.9 Total annual operating budget of agency $16,747,936 for FY 2002
A.10 Number of consumers served by agency/year 6,000
A.11 Total hours of service by agency/year 141,046.30 units for FY 2002; about 1900-2000 units for SMD services. Outreach activities would account for an additional 1000 units, but may duplicate units already documented.
A.12 Number of full time equivalents (FTE) in agency Agency has a total staff of 236 employees for FY 2002. Of those, 18 are contingent. FTEs for the agency are as follows: 1.0 - 187, .9 - 1, .8 - 3, .6 - 2, .5 - 26, .4 - 4, .3 - and below - 7

Table 4

Summary of Internet Survey

Post Training 12 Month Followup
IRK Component Useful for Project Comments Strengths Weaknesses Improvements/ Additions
IRK Guide Moderate   Guide used primarily as a training tool. Cat used the Guide help integrate EBP principles into practice. The resources in the Guide tied in well with the overall practice. Two respondents used the IRK 0 times, five reported using it <3 times. Fewer, more concise handouts would be helpful.
Kickoff Introductory Power Point Presentation Moderate to Very Useful and easy to understand. Especially helpful in recognizing that outcomes are measured over longer periods of time.      
The Introductory Video Moderate to Very The personal testimony in the video was very effective. Rated as Moderately useful. All respondents used the video at least once. Video was used to engage staff, consumers, local officials. Video was an effective way to convey a sense of hope.    
Practitioner’s Written Practice Information Sheet Moderate        
Skills training Power Point Very Respondents agreed that the PP was easy to understand, gave them more skills, and provided motivation to do practice.      
EBP Workbook Moderate to Very Respondents agreed that the Workbook was easy to understand, gave them more skills, and provided motivation to do practice. Five staff used the Workbook beyond 2-3 times and 2 staff used it more than 6 times.    
EBP-Specific Training Video Very Respondents agreed that the Video was easy to understand, gave them more skills, and provided motivation to do practice. One respondent stated it helped to see what they had learned demonstrated in a clinical setting. Three respondents used the video and felt it was helpful after training. Used to look at motivational interviewing styles.   Could have had more interviews
Fidelity Measures for EBP Moderate to Very At least half of the respondents felt the measures were easy to understand, motivated them, were effective as guidelines and that the measures would be used in the workplace. One respondent felt that the measures were only a modified treatment plan and wasn’t sure how they would be used. One respondent used the scale at regular intervals. Scale also used to inform leadership team. One respondent stated it was used to generate change in practice.    
Outcome Measures Very Six of the ten respondents felt the information on outcome measures was helpful in all areas. Chapter was encouraging and having a resource for outcomes is helpful. Outcome measures were collected at least 2-3 times. Four respondents either did not read the chapter or did not use any of the information. Agency could not ID outcome measures for just DD clients.  
Cultural Competency Moderate to Very Majority of the ten respondents felt the information on cultural competency was helpful in all areas. The information was a good reminder to keep cultural aspects in mind whenever relevant. Five out of seven respondents did not read the chapter and most found the chapter only slightly effective.  

Table 5

Project Timeline—Site 2

Milestones   Critical Events
  • In early November, 2002, when another project site drops out, the agency agrees to participate.
  • CCEO Clinical Director and CAT meet with administration to plan Steering Committee and intensive training to start immediately.
 Pre- project  
  • Baseline Fidelity Site Visit, 11-20- 02
  • Plan is to target Criminal Justice and Homeless Dual teams, i.e., the most acute population.
11- 02
  • Designated IDDT team and TL are selected vs. recruited.
  • Local MH/AODA represented as not interested or knowledgeable about IDDT/EBPs.
  • IDDT Kickoff Event, 1-7-03
  • First meeting of the steering committee planned for after kickoff.
  • IRK intensive training underway but only two modules to be completed during January; senior management attends training with two teams and others.
  • Agency sends 8 staff to off-site CCOE training on Motivational Interventions and 9 staff to 2-day family interventions workshop.
  • CAT provides consultation on structure and functions of Steering Committee, emphasizes work on outcomes; first meeting scheduled for 2/28
1- 03
  • IRK training requiring more than 12 hours indicated in Toolkit.
  • Challenges encountered with set-up of Steering Committee and CAT suggests smaller work group in addition.
  • Agency stalling on convening SC; work on outcomes is delayed.
  • CAT observes that PL and TL are overburdened and that the planning process is “fraying around the edges.”
  • Two Steering Committee sub-group meetings; plans for large group to meet quarterly and smaller group to meet monthly; focus on outcomes.
2- 03
  • Administration wants to use Clusters outcomes data and CAT consults against it.
  • Administration presence in training is intimidating to staff.

Table 2

Characteristics of the SMI Program—Site 2

Source Topic Data
B1 Definition of SMI As per Clinical Director, using state MHA guidelines; Inclusion criteria were recently changed.
B.2.b Number of consumers (SMI) served /year 2080 SMD clients served in FY 2002 about 30% were classified as high intensity; 30-35% classified as moderate intensity; about 40% low intensity/recovery
B.2.c Total hours to (SMI) /year 81,443.9 - AOD: 3745.6; Crisis: 143.2; Assessment: 379.5; Med/Som: 11,868.7 - Ind. Couns: 1,465.7; Grp. Couns: 2,193.4; CSP Ind.: 45,916.9; CSP Grp: 675.1; Vocational: 9,971.9; Mental health/other: 5,083.9
B.2.d Total FTE for SMI program/year In SMD, there are 79 employees. FTEs for SMD area are as follows: 1.0 - 75, 2.0+ .75 - 1, 3.0+ .5-.3
B.3.c.1 Focus on professional guilds/Comments Yes
B.3.c.2 Segregation by profession/Comments Yes/ They do tend to group, but more likely by jobs than by profession, although those often coincide.
B.3.c.3 Teams function in multidisciplinary fashion/Comments Yes/ We rotate call, so teams must collaborate
B.3.c.4 Consumers/family members on paid staff/Comments Consumers
B.3.c.5 Paid peer support specialists/Comments Yes/ We developed a consumer clinical package and got it funded by a foundation. We try to move client to role of collaborators on their own treatment.
B.3.d Allegiance to professional organizations (unions)/Comments No

Table 3

Quality Improvement Systems—Site 1

Source Topic Data
B.6 Nature of staff training program Basic CM training in the first 6 months of hire through KU: there is an orientation and a mentor assigned. Each CM also gets $250 annually for training. Many go to advanced trainings through KU.
B.7 Nature of agency’s management information system Memos, administrative meetings, 2 hours per week. Attending are the Exec and Clincal Dirs and clinical leaders. Other staff is in charge of AIMS.
B.8 Nature of agency’s quality improvement program We have Utilization Review Committee and Quality Assurance Committee as weekly meetings. Risk Management Committee review incident reports.
B.11.a Do practitioners collect consumer outcome data? Yes
B11.b Timely reports to program leaders? Yes
B11.c Timely reports to practitioners? Yes

Tables 1-7 Credit:

Reprinted with permission from the National Implementing Evidence-Based Practices Project. I would like to thank my colleague, Greg McHugo, at the Dartmouth Psychiatric Research Center for helping to identify and providing the site reports from which these displays are drawn. Further information about the project can be found in:

Torrey, W.C., Lynde, D.W., & Gorman, P. (2005). Promoting the implementation of practices that are supported by research: The national implementing evidence-based practice project. Child and Adolescent Psychiatric Clinics of North America, 14, 297-306.

McHugo, G.M., Drake, R.E., Whitley, R., Bond, G.R., Campbell, K., Rapp, C.A., Goldman, H.H., Lutz, W., & Finnerty, M. (2007). Fidelity outcomes in the national implementing evidence-based practices project. Psychiatric Services, 58, 1279-1284.