To: Board of Supervisors
From: Supervisor Rex Bohn
Agenda Section: Initiated by Board Member
Vote Requirement: Majority
SUBJECT:
title
Recognition of Humboldt County’s Information Technology Professionals for Exemplary Work and Dedication During the CrowdStrike Emergency
end
RECOMMENDATION(S):
Recommendation
That the Board of Supervisors:
1. Recognize Humboldt County’s Information Technology professionals for their exemplary work and dedication during the CrowdStrike emergency; and
2. Present Certificates of Appreciation.
Body
STRATEGIC PLAN:
This action supports the following areas of your Board’s Strategic Plan.
Area of Focus: Core Services/Other
Strategic Plan Category: 3001 - Support a well-trained workforce
DISCUSSION:
On Thursday, July 18, 2024, at 9:30 p.m. a widespread issue with CrowdStrike's endpoint protection software caused Windows machines to experience Blue Screen of Death (BSOD) errors. This incident affected all CrowdStrike customers around the globe, including the County of Humboldt. Multiple county departments experienced significant disruptions as Windows machines became inoperable due to the BSOD errors.
At 9:30 p.m. that day, County IT Technician II Carson Browne discovered an issue during routine maintenance. By 10 p.m., Information Technology Director Scott Irving was notified, and IT Security Analyst II Scott Kimball was brought in, suspecting a potential security attack. Browne and Kimball identified CrowdStrike as the cause and found a workaround. By 11 p.m., the fix was in place.
Director Irving assembled a Critical Initial Response team consisting of himself, Carson Browne, Scott Kimball and County IT Technician II Shur Yang and they went to work getting critical services back online. By 2:30 a.m. Friday, July 19, 2024, all critical services for 24-hour facilities were back online.
At 6:30 a.m. on Friday, July 19, 2024, Director Irving brief DHHS Deputy Director Zach Smith over the phone on the outage. At 7 a.m. Director Irving returned to the courthouse to bring the Board of Supervisors, County Administrative Office, Treasurer/Tax Collector and Purchasing machines back online. By 8 a.m., Director Irving assembled county IT Division staff to devise a divide and conquer strategy to address as many machines as possible during normal working hours for Friday.
Staff working the issue on Friday included IT Project Manager, Sandy Allsop; Carson Browne; IT Systems Administrator I, Gabel Ammon; IT Systems Administrator I, Jonathan Danna; IT Systems Supervisor, Ulf Engert; Scott Irving; IT Security Analyst II, Omar Padilla; IT Technician II, Jesse Reid; IT Systems Supervisor, Tyson Sanchez; IT Applications Analyst II, Michael Tjoelker; IT Technician II, Anthony Williams; Shur Yang; and IT Applications Analyst I, Hussein Yazbeck.
The Office of Emergency Services was prioritized, Carson Browne took care of the southern sites in the county, Michel Tjoelker took care of the north, courthouse and surrounding buildings were handled by the desktop support team.
Tom Pinto Department (Dept.) Information Systems Analyst and Adrian Ratcliff Administrative Analyst I from the District Attorney’s Office were briefed on the fix for the District Attorney Offices’ computers, and they took care of their department.
Department of Health & Human Services (DHHS) Information Services (IS) assisted with Public Safety East, with Josh Burke, Information Systems Analyst II, getting the Willow Creek Sheriff’s substation and DHHS Willow Creek computers back online.
DHHS response was handed by DHHS IS staff: Dept. Information Systems Supervisor, Candace Moore; Senior Information Systems, Cory Cook; Dept. Information Systems Analyst II, Aaron Trussler; Dept. Information Systems Analyst II, Lisa McNiff; Dept. Information Systems Technician, Joe Lohberger; Dept. Information Systems Technician, Vanja Venezia; Dept. Information Systems Technician, Ryan Yakely; Dept. Information Systems Technician, Arthur Ecker; Dept. Information Systems Supervisor, Edgar Diaz; Dept. Information Systems Analyst II, Nathan Vegh; Dept. Information Systems Analyst II, Patricia Husted; Dept. Information Systems Analyst II, Joshua Burke; Dept. Information Systems Analyst II, Bob Quance; Dept. Information Systems Analyst II, Cassidy Bailey; Dept. Information Systems Supervisor, Russ Catalan; Dept. Programmer Analyst, Roger Kreutz; Dept. Programmer Analyst, Samantha Mena; Dept. Programmer Analyst, Scott Josephson; Dept. Information Systems Analyst II, Des Morphis; Dept. Information Systems Analyst II, William Classon; Dept. Information Systems Analyst II, Robert Beier; Dept. Information Systems Analyst II, Patrick Ory and Dept. Information Systems Analyst II, Matt Reinke.
DHHS IS staff worked diligently to restore mandated community services making them available to the public by the afternoon of July 19, 2024. By the end of the day staff had remediated 60 servers and nearly 1,000 staff computers. All remediations were done manually as every DHHS workstation hard drive is encrypted and security keys needed to be manually inputted.
Department of Child Support Services’ (DCSS) response was handled by DCSS IT staff Darrell Cline Information Systems Supervisor, Ryan Watts Information Systems Analyst and Joanna Hawley Jones Information Systems Analyst.
Saturday, July 20, 2024, Jesse Reid visited the Blue Lake Library and Ulf Engert visited the Trinidad Library to bring them back online.
By Monday July 22, 2024, County IT Desktop Support Specialists were able to perform the clean-up of any remaining system issues not addressed previously.
By Monday July 22, 2024, DHHS IS staff in the Help Desk unit were tasked with remediating the remaining 250 or so staff computers to bring them back online.
Carson Browne played a crucial role during this incident. His ability to work remotely allowed for the routine maintenance of the county’s infrastructure, which proved invaluable in this situation. Carson's immediate awareness of the issue enabled him to promptly contact Director Irving and mobilize the Critical Response Team. This swift action significantly reduced downtime for county departments. By Friday morning, county IT’s focus had shifted to restoring departmental computers, as all critical infrastructure, including the county’s servers, dispatch, and the correctional facility, were brought back online within four hours of the outage's onset. Carson’s quick thinking and decisive action were instrumental in county IT’s effective response.
Additionally, Scott Kimball and Shur Yang should be commended for their willingness to report to work in the middle of the night to assist in getting core services back online. Their dedication and commitment were vital in ensuring the rapid restoration of our critical systems.
Furthermore, Jesse Reid and Ulf Engert deserve recognition for their efforts on Saturday. Despite the weekend, they worked diligently to get the computers back up and running at the Trinidad and Blue Lake libraries while they were open, ensuring that these community resources remained available to the public.
SOURCE OF FUNDING:
All county departments
FINANCIAL IMPACT:
The financial impacts to the county were minimal due to immediate awareness of the issue and the deployment of county IT’s critical response team and communications with departments housing their own IT support. Additionally, the incident occurred afterhours on Thursday night and Fridays are generally a telework day for most employees whose departments are closed to the public on Fridays. Computers not connected to the network were not affected.
STAFFING IMPACT:
Other than the impacts to county IT staff, staffing impacts were minimal to county employees.
OTHER AGENCY INVOLVEMENT:
County Administrative Information Technology Division
Department of Health & Human Services Information Services
Department of Child Support Services
District Attorney Office
ALTERNATIVES TO STAFF RECOMMENDATIONS:
Board discretion.
ATTACHMENTS:
Certificates will be presented at the meeting.
PREVIOUS ACTION/REFERRAL:
Meeting of: N/A
File No.: N/A