Skip to site content

National Evaluation

a magnifying glass pointed at a stack of reports

Grantees are required to participate in a national evaluation of the 4-in-1 grant program to address the needs of the Urban Indian service population. The evaluation collects the following quantitative data from the Government Performance and Results Act (GPRA), National Immunization Reporting System (NIRS), and Uniform Data System (UDS). Through their quarterly progress reports, grantees provide narrative outcomes in the following areas:

  • The four health program areas, including their unmet needs and recommendations [PDF - 2.7 MB]
  • Integration of culture and tradition into interventions and practice
  • Practice- and evidence-based approaches
  • Findings from an evaluation of the grantees work across their immunization program [PDF - 2.7 MB]. (Click below under Immunization to watch webinar)
  • A comprehensive evaluation report from Grant Program years 2019-2022 (Click below under 4-in-1 Overview of Comprehensive Report Findings Webinar to watch webinar)
Date Topic Presentation Handouts Other Materials and Resources

05/04/2023

4-in-1 Overview of Comprehensive Report Findings Webinar

Presenters: Crystal Tetrick and Soultana Macridis

Duration/Topics: 40 minutes

Slides [PDF - 3.8 MB]

REVISED Final 4-in-1 Comprehensive Report [PDF - 1.7 MB]

In the initial cohort year, the national evaluation included a review of 29 grantees. The second cohort included 33 grantees. In the current cohort, there are 32 grantees. Once complete, the national evaluation will cover data from 2016 to 2027 as outlined below:

  • Cohort 1 - 2016 – 2019 (3 years; 29 grantees)
  • Cohort 2 - 2019 – 2022 (3 years; 33 grantees)
  • Cohort 3 - 2022 – 2027 (5 years; 32 grantees)

Evaluation Objectives

The evaluation helped the IHS accomplish three objectives:

  1. To reduce the reporting burden by designing a common progress reporting template and combine the unmet needs and recommendations within the quarterly reporting form,
  2. To aid in the initial web-based portal of an electronic reporting application — to support ease of access to qualitative and quantitative data elements.
  3. To document processes and design decisions, which were grounded in UIOs’ feedback.

National Evaluation Findings

The Indian Health Service’s (IHS) Office of Urban Indian Health Programs (OUIHP) conducted an evaluation of the 4-in-1 Grant Program for the 2019 Grant Program Year (April 1, 2019 – March 31, 2020) and the 2020 Grant Program Year (April 1, 2020 – March 31, 2021). Below is a highlight of the key summary findings organized by each of the four program areas. The key findings below are summarized from grantee-reported quantitative and qualitative data sources.

  • Analysis of 2018 to 2019 UDS data found increases for the AI/AN population total visits (+0.3%), medical visits (+1.8%), and other professional visits (+23.5%). Decreases were found for vision visits (-4.3%) and enabling visits (-6.4%).
  • Analysis of 2018 to 2019 GPRA data found the average HP/DP-related screening measures (Controlled BP <140/90 (-12.7%), DM Statin Therapy (-15.4%), Nephropathy Assessed (-31.1%), Pap Screening (-6.3%), Mammography (-14.2%), and Colorectal Cancer (-6.3%)) had a decrease, except for the increase of HIV screening (+5.8%).
  • Analysis of 2018 and 2019 GPRA data found the average HP/DP-related screening measures exceeding the national average targets in both years were Controlled BP <140/90, DM Statin Therapy, Nephropathy Assessed, and HIV Screening Ever. Childhood Weight Control exceeded the national average target in 2018, only.
  • Analysis of the grantee quarterly reports identified cooking classes that supported nutrition education and cultural activities, as well as physical activity programs that incorporated cultural activities to be program strengths.
  • Analysis of 2018 to 2019 GPRA data found the average immunization rates increased for influenza vaccinations for children ages 6 months to 17 years (+3.2%) and for adults 18 years and older (+1.2%), with decreases observed for children and adult composite immunizations (-10.7% and -4.9%, respectively).
  • Analysis of 2018 and 2019 GPRA data found that none of the average immunization rates met or exceed the national average targets.
  • Analysis of 2019 and 2020 NIRS reporting found that a total of 100 NIRS reports were submitted for both years. On average, 51% of grantees used the NIRS system in 2019 compared to an average of 50% of grantees who used the system in 2020.
  • Analysis of the Grant Program Year 2019 and 2020 grantee quarterly reports found that for immunization services, grantees continued to promote routine immunizations and COVID-19 vaccines, especially as the pandemic disrupted vaccination of the population served across a variety of settings (e.g., clinics and schools) and among all population groups (e.g., children, adults, and elders).
  • Watch previously recorded webinar: IHS Urban 4-in-1 Grant Overview of Immunization Findings: April 1, 2019 to March 31, 2022.
  • Analysis of 2018 to 2019 UDS data found that alcohol and substance abuse visits among the AI/AN population increased (+7%).
  • Analysis of 2018 to 2019 GPRA data found the average alcohol/substance abuse-related measures, Screening, Brief Intervention, and Referral to Treatment (SBIRT; -11.2%) decreased, whereas Tobacco Cessations (+0.3%) and Universal Alcohol Screening (+2.4%) increased.
  • Analysis of 2018 and 2019 GPRA data found the average alcohol/substance abuse-related measures that exceeded the national target included SBIRT in 2018 (+14.9%) and 2019 (+3.7), and Universal Alcohol Screening in 2018 (+1.4%).
  • Analysis of the grantee quarterly report found grantees’ alcohol/substance abuse programs were successful as a hybrid delivery service model that incorporates both in-person and virtual opportunities to engage with patients, reducing barriers to treatment.
  • Analysis of the 2018 to 2019 UDS data found mental health visits decreased slightly by 1.1% for the AI/AN population.
  • Analysis of 2018 to 2019 GPRA data found the average mental health measures decreased for Intimate Partner Violence/Domestic Violence (IPV/DV) Screening (-1%), Depression Screening for Adolescents 12-17 years (-12.2%), and Depression Screening for Adults Ages 18 and older (-10.5%).
  • Analysis of 2018 and 2019 GPRA data found the average mental health measure that exceeded the national average target was Depression Screening in 2018 by 6.4%.
  • Analysis of the grantee quarterly report found mental health service delivery by the grantees were successful because well-trained staff provided culturally competent services and trauma-informed care.

Integration of cultural practices and evidence-and-practice-based approaches into service delivery was a priority for most grantees. This can help improve and strengthen health care, primary care, and social support services and delivery.

  • Thirty of the 33 grantees who submitted the grantee quarterly reports for the 2019 and 2020 Grant Program Years mentioned the integration of cultural and traditional practices, as well as practice- and evidence-based approaches into services and delivery.
  • Grantees reported using 43 different types of cultural and traditional practices for the HP/DP, alcohol/substance abuse prevention, and mental health.
  • Examples of the cultural offerings integrated into programs included:
    • Traditional art forms (e.g., beading, singing, and drumming).
    • Traditional physical activities (e.g., dancing, peeling bark from teepee poles, a powwow for cardiovascular health, and sweat lodge ceremonies).
    • Discussion activities (e.g., talking circles, which promote respectful sharing, and elders’ storytelling, which helps participants learn about healing through their language).

Please contact the IHS for additional details: 4in1@ihs.gov.

Measuring Impact and Outcomes

Grantees expressed a uniform concern that measures do not fully express the impact and outcomes of all of their programs.

The previous reporting system restricted the grantee’s ability to report robust qualitative data. For example, there are no measures that document how practicing culture correlates with improved health. A few Urban Indian Organizations (UIO) self-reported the integration of culture primarily increased attendance in men’s and women’s groups and at the UIO in general. When UIOs self-report on information without standardization, the measures for data collection will vary by UIO.

To address these concerns, the Office of Urban Indian Health Programs (OUIHP) designed a new reporting template with an open-ended format so grantees have the ability to:

  • Describe how they weave culture and traditional practices into program services without having to quantify the effect of these efforts
  • Upload photos, short videos, and digital stories to support their responses

Sources of qualitative data included event sign-in sheets, activity leader attendance sheets, staff notes, anecdotal information from grantees, community advisory council feedback (when present), community governance board feedback, and local community feedback.

The process evaluation focused on the programs’ activities and whether they were implemented as planned and intended. As examples, process evaluation can entail tracking the number of direct contact hours, the number of clients who complete a program, the patients’ satisfaction, or even the delivery of activities and/or services.

Grantees were expected to perform a process evaluation at least bi-monthly, but most grantees monitor their progress weekly. Grantees felt strongly that the ability to report and reflect on program implementation is key to their continuous improvement and accountability to their community.

During site visits, grantees expressed a common understanding of the importance of achieving and reporting community-level, short-term outcomes, such as:

  • Establishing a community garden
  • Practicing cultural activities (like harvesting saguaro fruit)
  • Improving access to medical devices for elders (such as hearing aids for Veterans)

Grantees also want to share their unique program strengths, such as their exemplary staff, partnerships, and their ability to go beyond expected outcomes.

The proposed electronic reporting application includes fields that can better capture qualitative measures and enable the Indian Health Service (IHS) to track long-term grantee progress on current and new data fields. For example, the electronic reporting application will help the IHS discern a UIO’s capacity in various areas including:

  • Program description
  • Program updates
  • Evaluative capacity
  • Program impact
  • Program sustainability