Data Engineer
Apply NowCompany: Compunnel Software Group
Location: Appleton, WI 54915
Description:
Job Summary
We are building an Enterprise Data Platform (EDP) entirely on the cloud, utilizing Databricks on AWS. This role will focus on leading data ingestion efforts, mentoring junior engineers, and optimizing data transformation capabilities while maintaining high standards for testing and data quality. The ideal candidate will have expertise in Python, PySpark, AWS, and DBT, with a strong understanding of cloud migration, data modeling, and data profiling.
Key Responsibilities
" Oversee and guide the data ingestion process, ensuring efficient and accurate data integration.
" Mentor junior engineers, promoting engineering excellence and improving delivery processes.
" Utilize extensive knowledge of various data structures (Oracle, DB2, Mainframe, MongoDB, DynamoDB) to optimize data ingestion and transformation.
" Analyze structured, unstructured, and semi-structured data to determine the best profiling and ingestion methods.
" Take full ownership of data sources and datasets, ensuring optimal ingestion strategies.
" Collaborate with staff engineers and other team members to drive a proactive and efficient work environment.
Required Qualifications
" Strong experience with Databricks, AWS, Python, Spark, and DBT.
" Proven track record in cloud migration projects, particularly moving on-premises systems to the cloud.
" Expertise in data modeling and transformation, with a focus on the Medallion architecture.
" Demonstrated ability to lead teams, mentor junior engineers, and drive engineering excellence.
" In-depth knowledge of various data structures, including Oracle, DB2, Mainframe, MongoDB, and DynamoDB.
" Experience with profiling structured, unstructured, and semi-structured data to optimize ingestion processes.
" Strong communication and interpersonal skills, with an accountable, collaborative, and proactive approach.
Location: Appleton, Wisconsin
Education: Bachelors Degree
We are building an Enterprise Data Platform (EDP) entirely on the cloud, utilizing Databricks on AWS. This role will focus on leading data ingestion efforts, mentoring junior engineers, and optimizing data transformation capabilities while maintaining high standards for testing and data quality. The ideal candidate will have expertise in Python, PySpark, AWS, and DBT, with a strong understanding of cloud migration, data modeling, and data profiling.
Key Responsibilities
" Oversee and guide the data ingestion process, ensuring efficient and accurate data integration.
" Mentor junior engineers, promoting engineering excellence and improving delivery processes.
" Utilize extensive knowledge of various data structures (Oracle, DB2, Mainframe, MongoDB, DynamoDB) to optimize data ingestion and transformation.
" Analyze structured, unstructured, and semi-structured data to determine the best profiling and ingestion methods.
" Take full ownership of data sources and datasets, ensuring optimal ingestion strategies.
" Collaborate with staff engineers and other team members to drive a proactive and efficient work environment.
Required Qualifications
" Strong experience with Databricks, AWS, Python, Spark, and DBT.
" Proven track record in cloud migration projects, particularly moving on-premises systems to the cloud.
" Expertise in data modeling and transformation, with a focus on the Medallion architecture.
" Demonstrated ability to lead teams, mentor junior engineers, and drive engineering excellence.
" In-depth knowledge of various data structures, including Oracle, DB2, Mainframe, MongoDB, and DynamoDB.
" Experience with profiling structured, unstructured, and semi-structured data to optimize ingestion processes.
" Strong communication and interpersonal skills, with an accountable, collaborative, and proactive approach.
Location: Appleton, Wisconsin
Education: Bachelors Degree