Job Details
Senior Splunk Engineer - Infrastructure Operations
United States
Suitability/Public Trust
Fully remote
Information Technology
GovCIO is currently hiring for Systems Architect (Senior) /Senior Splunk Engineer - Infrastructure Operations of Infrastructure Operations to support our Administrative Office of the US Courts NLS project. The NLS currently ingest an average of 18-20TB of logging data daily across 60 indexers distributed in 2 data centers. This position is located within the United States and is fully remote.
**Responsibilities**
+ Design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment.
+ Monitor overall Splunk health through the Monitoring Console (DMC) including indexer, search head, and cluster master status.
+ Track indexing rates, license usage, queue health, and search concurrency to identify performance or ingestion issues early.
+ Monitor CPU, memory, and disk utilization across all Splunk components to ensure optimal resource usage.
+ Respond promptly to health alerts, DMC warnings, or anomalies observed on monitoring dashboards.
+ Investigate and resolve common user-reported issues such as access problems, failed searches, or non-triggering alerts.
+ Troubleshoot data ingestion, parsing, and indexing issues across Universal Forwarders, Heavy Forwarders, and HEC endpoints.
+ Investigate missing or duplicate logs, timestamp errors, or sourcetype misassignments and escalate complex parsing issues to Engineering.
+ Validate new data source onboardings by confirming sourcetype assignment, timestamp accuracy, and field extraction integrity.
+ Support data source owners with forwarder deployment, syslog setup, and connectivity troubleshooting during initial onboarding.
+ Maintain data flow visibility from source ? forwarder ? indexer to confirm data completeness and performance.
+ Rotate and update credentials, API keys, or tokens used in data inputs, integrations, alerts, and scheduled searches.
+ Manage RBAC user and role mappings, handling access requests, entitlement reviews, and permission troubleshooting.
+ Provide end-user assistance with SPL searches, reports, alerts, and dashboards, including query optimization tips.
+ Maintain and update knowledge base articles, SOPs, and FAQs for repeatable issues and troubleshooting steps.
+ Log and escalate platform or parsing issues to the Engineering team with evidence such as logs, screenshots, and correlation IDs.
+ Open and manage Splunk Support cases for platform-level bugs, license problems, or critical system faults.
+ Monitor and manage ITSI service health, including KPIs, correlation searches, NEAP policies, and summary index latency.
+ Troubleshoot ITSI-related issues such as broken KPIs, delayed episodes, or missing notable events.
+ Perform capacity management by monitoring index growth, bucket rotation, and frozen data retention policies.
+ Conduct periodic system maintenance tasks, including orphaned object cleanup and knowledge object review.
+ Verify and maintain compliance with data governance and retention policies, ensuring secure and auditable configurations.
+ Participate in DR testing and validation to ensure Splunk data recovery and HA configurations are functioning as expected.
+ Document incidents, RCA findings, and preventive actions for future reference.
+ Collaborate closely with the Engineering team for escalations, root-cause investigations, and deployment verifications.
**Qualifications**
Bachelor's with 10 years (or commensurate experience) OR Masters Degree or higher (in a related discipline) with 7 years experience
Required Skills and Experience
+ Expert skills in Enterprise Security, ITSI, SOAR, and the Slunk product line.
+ Able to design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment.
**Clearance Required:** Must be able to obtain and maintain AOUSC Public Trust
**Posted Salary Range**
USD $105,000.00 - USD $145,000.00 /Yr.
Suitability/Public Trust
Fully remote
Information Technology
GovCIO is currently hiring for Systems Architect (Senior) /Senior Splunk Engineer - Infrastructure Operations of Infrastructure Operations to support our Administrative Office of the US Courts NLS project. The NLS currently ingest an average of 18-20TB of logging data daily across 60 indexers distributed in 2 data centers. This position is located within the United States and is fully remote.
**Responsibilities**
+ Design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment.
+ Monitor overall Splunk health through the Monitoring Console (DMC) including indexer, search head, and cluster master status.
+ Track indexing rates, license usage, queue health, and search concurrency to identify performance or ingestion issues early.
+ Monitor CPU, memory, and disk utilization across all Splunk components to ensure optimal resource usage.
+ Respond promptly to health alerts, DMC warnings, or anomalies observed on monitoring dashboards.
+ Investigate and resolve common user-reported issues such as access problems, failed searches, or non-triggering alerts.
+ Troubleshoot data ingestion, parsing, and indexing issues across Universal Forwarders, Heavy Forwarders, and HEC endpoints.
+ Investigate missing or duplicate logs, timestamp errors, or sourcetype misassignments and escalate complex parsing issues to Engineering.
+ Validate new data source onboardings by confirming sourcetype assignment, timestamp accuracy, and field extraction integrity.
+ Support data source owners with forwarder deployment, syslog setup, and connectivity troubleshooting during initial onboarding.
+ Maintain data flow visibility from source ? forwarder ? indexer to confirm data completeness and performance.
+ Rotate and update credentials, API keys, or tokens used in data inputs, integrations, alerts, and scheduled searches.
+ Manage RBAC user and role mappings, handling access requests, entitlement reviews, and permission troubleshooting.
+ Provide end-user assistance with SPL searches, reports, alerts, and dashboards, including query optimization tips.
+ Maintain and update knowledge base articles, SOPs, and FAQs for repeatable issues and troubleshooting steps.
+ Log and escalate platform or parsing issues to the Engineering team with evidence such as logs, screenshots, and correlation IDs.
+ Open and manage Splunk Support cases for platform-level bugs, license problems, or critical system faults.
+ Monitor and manage ITSI service health, including KPIs, correlation searches, NEAP policies, and summary index latency.
+ Troubleshoot ITSI-related issues such as broken KPIs, delayed episodes, or missing notable events.
+ Perform capacity management by monitoring index growth, bucket rotation, and frozen data retention policies.
+ Conduct periodic system maintenance tasks, including orphaned object cleanup and knowledge object review.
+ Verify and maintain compliance with data governance and retention policies, ensuring secure and auditable configurations.
+ Participate in DR testing and validation to ensure Splunk data recovery and HA configurations are functioning as expected.
+ Document incidents, RCA findings, and preventive actions for future reference.
+ Collaborate closely with the Engineering team for escalations, root-cause investigations, and deployment verifications.
**Qualifications**
Bachelor's with 10 years (or commensurate experience) OR Masters Degree or higher (in a related discipline) with 7 years experience
Required Skills and Experience
+ Expert skills in Enterprise Security, ITSI, SOAR, and the Slunk product line.
+ Able to design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment.
**Clearance Required:** Must be able to obtain and maintain AOUSC Public Trust
**Posted Salary Range**
USD $105,000.00 - USD $145,000.00 /Yr.
Job #NLX290299241