Data Engineer
Apply NowCompany: HonorVet Technologies
Location: Appleton, WI 54915
Description:
Job Title: Data Engineer
Location: Appleton, WI
Job Description
Job Description:
We are building out an Enterprise Data Platform (EDP) entirely on the cloud, utilizing Databricks on AWS. The project involves two teams focused on data ingestion, with a framework already established using Python and Spark for processing. The platform leverages DBT for data modeling. Our goal is to enhance data transformation capabilities while maintaining high standards for testing and data quality.
Key Responsibilities:
Required Skills and Experience:
Interview Process:
Location: Appleton, WI
Job Description
- Must have: PySpark, Python Databricks
Job Description:
We are building out an Enterprise Data Platform (EDP) entirely on the cloud, utilizing Databricks on AWS. The project involves two teams focused on data ingestion, with a framework already established using Python and Spark for processing. The platform leverages DBT for data modeling. Our goal is to enhance data transformation capabilities while maintaining high standards for testing and data quality.
Key Responsibilities:
- Lead Data Ingestion: Oversee and guide the data ingestion process, ensuring efficient and accurate data integration.
- Mentorship: Mentor junior engineers, promoting engineering excellence and improving delivery processes.
- Data Expertise: Utilize extensive knowledge of various data structures (Oracle, DB2, Mainframe, MongoDB, DynamoDB) to optimize data ingestion and transformation.
- Data Profiling: Analyze structured, unstructured, and semi-structured data to determine the best methods for profiling and ingestion.
- Ownership: Take full ownership of data sources and datasets, determining the best methods for ingestion and understanding the data thoroughly.
- Collaboration: Work closely with staff engineers and other team members to schedule and conduct interviews, ensuring a collaborative and proactive work environment.
Required Skills and Experience:
- Technical Proficiency: Strong experience with Databricks, AWS, Python, Spark, and DBT.
- Cloud Migration: Proven track record in cloud migration projects, particularly moving on-premises systems to the cloud.
- Data Modeling: Expertise in data modeling and transformation, with a focus on the Medallion architecture.
- Leadership: Demonstrated ability to lead teams, mentor junior engineers, and drive engineering excellence.
- Data Structures: In-depth knowledge of various data structures, including Oracle, DB2, Mainframe, MongoDB, and DynamoDB.
- Data Profiling: Experience with profiling structured, unstructured, and semi-structured data to optimize ingestion processes.
- Soft Skills: Accountable, collaborative, and proactive, with strong communication and interpersonal skills.
Interview Process:
- Round 1: Technical interview with staff engineers.