Data Engineer
Apply NowCompany: Compunnel Software Group
Location: Cincinnati, OH 45238
Description:
Job Overview:
We are seeking a Data Engineer with a strong background in building scalable data pipelines and a passion for solving complex unstructured data ingestion challenges. This is an exciting opportunity to work in a forward-thinking environment focused on building a robust unstructured data fabric, integrating diverse data sources into modern data platforms like Redshift, S3, and Delta Lake.
Key Responsibilities:
Required Skills & Experience:
Nice to Have:
Education: Bachelors Degree
We are seeking a Data Engineer with a strong background in building scalable data pipelines and a passion for solving complex unstructured data ingestion challenges. This is an exciting opportunity to work in a forward-thinking environment focused on building a robust unstructured data fabric, integrating diverse data sources into modern data platforms like Redshift, S3, and Delta Lake.
Key Responsibilities:
- Design, build, and optimize data pipelines using AWS Glue, Python, and related technologies.
- Ingest structured and unstructured data from a variety of sources including
- Relational DBMS
- NoSQL databases
- APIs
- Document storage systems
- Manage data ingestion into Amazon Redshift, S3, and strategic Delta Lake (Iceberg format) for advanced analytics.
- Collaborate with cross-functional teams to evolve the unstructured data fabric, aligning with organizational data architecture strategies.
- Leverage Delta Sharing and ontology-driven design principles to enable seamless data sharing and discovery within Palantir and other platforms.
- Troubleshoot and optimize data transformation workflows, ensuring data quality and pipeline efficiency.
Required Skills & Experience:
- Strong proficiency in Python for data engineering tasks.
- Hands-on experience with AWS Glue for ETL/ELT processes.
- Solid understanding of working with unstructured data and designing ingestion strategies.
- Experience integrating with various data sources - RDBMS, NoSQL, APIs, documents (PDF, JSON, XML, etc.)
- Proficiency in working with Amazon Redshift, S3, and data lakes.
- Familiarity with Apache Iceberg or Delta Lake, and Delta Sharing protocols.
- Experience working in cloud-based data environments, preferably AWS.
Nice to Have:
- Experience with Palantir Foundry or similar platforms.
- Knowledge of ontology modeling or semantic data design.
- Familiarity with data governance, metadata management, and data cataloging.
- Exposure to DevOps practices and CI/CD pipelines for data workflows.
Education: Bachelors Degree