Senior Data Engineer

Are you ready to turn complex data into powerful mission insight? NS2 Mission is seeking an experienced Senior Data Engineer to design, build, and optimize tools, code, and services that support complex data engineering activities across structured and unstructured data sources. This role is positioned in Chantilly, VA and will focus on enabling the ingestion, transformation, and delivery of high-quality data to support mission-critical analytics and decision-making.

Key Responsibilities

• Design and implement solutions for moving both structured and unstructured data using approved methods.
• Execute data ingestion processes to store data in local or enterprise-level repositories.
• Develop code and tools to format data for exploration and analysis.
• Analyze source data formats and collaborate with Data Scientists and partners to identify optimal transformations and standards aligned with mission objectives.
• Build and maintain ETL pipelines, including one-time and recurring data extraction, formatting, and transformation into enterprise or standalone data models.
• Perform O&M and enhancements to existing ETL processes using best practices and coding standards.
• Develop and deliver documentation for each project, including ETL mappings, user guides, and access instructions.
• Design and optimize scalable data pipelines using technologies such as Spark, Apache Iceberg, Trino, OpenSearch, EMR cloud services, NiFi, and Kubernetes containers.
• Ensure data lineage, pedigree, and security are maintained throughout the lifecycle.
• Clean and preprocess datasets to enable advanced analytics.
• Collaborate with enterprise working groups to advance data standards.
• Partner with engineers, data stewards, and mission stakeholders to derive actionable insights from data assets.
• Work closely with software engineers to update, configure, and maintain data services based on evolving requirements.
• Ensure data quality by collaborating with testing and quality teams to standardize conditioning pipelines.
• Adapt to diverse data types and formats, integrating new platforms and tools as needed.

Required Skills & Experience

• 10+ years of experience in:

  • Data lifecycle engineering

  • Development and maintenance of ETL tools and services

  • Cloud and on-premises data storage and processing solutions

  • Programming with Python, SQL, Spark, and related languages

  • Leveraging COTS and open-source tools such as Elasticsearch and NiFi

  • Supporting data workflows within Agile development lifecycles

Preferred Skills

• Experience applying AI and large language models (LLMs) to accelerate data processing and transformation.

Position Clearance Requirement:

TS/SCI with an FSP