PYSPARK Tech Lead
Apply NowCompany: Diverse Lynx LLC
Location: Chicago, IL 60629
Description:
Required Skills : Cloudera HUE, PySpark, Spark Pyspark, Cloudera Data Platform, Apache Iceberg
Responsibilities :
Lead the design and implementation of data solutions using Cloudera HUE and Cloudera Data Platform
Supervise the integration of Apache Iceberg and Spark into existing data workflows
Provide technical guidance and mentorship to junior team members on PySpark best practices
Collaborate with cross-functional teams to understand data requirements and deliver scalable solutions
Ensure data quality and integrity through meticulous testing and validation processes
Develop and maintain documentation for data architectures and workflows
Optimize data processing pipelines for performance and efficiency
Conduct code reviews to ensure alignment to coding standards and best practices
Solve and resolve complex technical issues related to data processing
Stay updated with the latest advancements in data technologies and incorporate them into projects
Drive innovation by exploring new tools and techniques in the research and development domain
Communicate effectively with stakeholders to gather requirements and provide project updates
Contribute to the overall success of the company by delivering high-quality data solutions that drive business value
Qualifications
Possess strong expertise in Cloudera HUE and Cloudera Data Platform
Demonstrate proficiency in Apache Iceberg and Spark
Have extensive experience with PySpark for data processing
Show a solid understanding of data architecture and data engineering principles
Exhibit excellent problem-solving skills and attention to detail
Have a background in research and development is a plus
Display strong communication and collaboration skills
Be able to work effectively in a hybrid work model
Have a proactive approach to learning and staying current with industry trends
Be capable of mentoring and guiding junior team members
Demonstrate the ability to deliver high-quality solutions within deadlines
Show a commitment to continuous improvement and innovation
Possess a strong sense of ownership and accountability for project outcomes.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
Responsibilities :
Lead the design and implementation of data solutions using Cloudera HUE and Cloudera Data Platform
Supervise the integration of Apache Iceberg and Spark into existing data workflows
Provide technical guidance and mentorship to junior team members on PySpark best practices
Collaborate with cross-functional teams to understand data requirements and deliver scalable solutions
Ensure data quality and integrity through meticulous testing and validation processes
Develop and maintain documentation for data architectures and workflows
Optimize data processing pipelines for performance and efficiency
Conduct code reviews to ensure alignment to coding standards and best practices
Solve and resolve complex technical issues related to data processing
Stay updated with the latest advancements in data technologies and incorporate them into projects
Drive innovation by exploring new tools and techniques in the research and development domain
Communicate effectively with stakeholders to gather requirements and provide project updates
Contribute to the overall success of the company by delivering high-quality data solutions that drive business value
Qualifications
Possess strong expertise in Cloudera HUE and Cloudera Data Platform
Demonstrate proficiency in Apache Iceberg and Spark
Have extensive experience with PySpark for data processing
Show a solid understanding of data architecture and data engineering principles
Exhibit excellent problem-solving skills and attention to detail
Have a background in research and development is a plus
Display strong communication and collaboration skills
Be able to work effectively in a hybrid work model
Have a proactive approach to learning and staying current with industry trends
Be capable of mentoring and guiding junior team members
Demonstrate the ability to deliver high-quality solutions within deadlines
Show a commitment to continuous improvement and innovation
Possess a strong sense of ownership and accountability for project outcomes.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.