Data Engineer

Data Engineer

Position Summary:

The Data Engineer will support Transcend's Chief Data and Artificial Intelligence Officer, as well as be responsible for developing, optimizing, and managing data pipelines and architectures that support analytics, reporting, and business intelligence initiatives. The Data Engineer will collaborate closely with data analysts, software developers, and IT teams to ensure our data is reliable, scalable, and accessible for decision-making.

Duties/Responsibilities:

• Data Pipeline Development
  • Design, implement, and optimize end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.
  • Develop robust ETL (Extract, Transform, Load) processes to integrate data from diverse sources into our data ecosystem.
  • Implement data validation and quality checks to ensure accuracy and consistency.
  • Develop and maintain scalable data pipelines and build new API integrations to support increasing data volume and complexity.
• Data Modeling and Architecture
  • Design and maintain data models, schemas, and database structures to support analytical and operational use cases.
  • Optimize data storage and retrieval mechanisms for performance and scalability.
  • Evaluate and implement data storage solutions, including relational databases, NoSQL databases, data lakes, and cloud storage services.
  • Define company data assets, including Spark, SparkSQL, and HiveSQL jobs to populate data models.
• Data Integration and API Development
  • Build and maintain integrations with internal and external data sources and APIs.
  • Implement RESTful APIs and web services for data access and consumption.
  • Ensure compatibility and interoperability between different systems and platforms.
  • Design data integrations and a data quality framework.
• Data Infrastructure Management
  • Configure and manage data infrastructure components, including databases, data warehouses, data lakes, and distributed computing frameworks.
  • Monitor system performance, troubleshoot issues, and implement optimizations to enhance reliability and efficiency.
  • Implement data security controls and access management policies to protect sensitive information.
  • Design and evaluate open-source and vendor tools for data lineage.
  • Work closely with all business units and engineering teams to develop a strategy for long-term data platform architecture.
• Collaboration and Documentation
  • Collaborate with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision-making.
  • Document technical designs, workflows, and best practices to facilitate knowledge sharing and maintain system documentation.
  • Provide technical guidance and support to team members and stakeholders as needed.
  • Write unit/integration tests, contribute to engineering wiki, and document work.
  • Perform data analysis required to troubleshoot data-related issues and assist in their resolution.
• Additional duties and responsibilities may be assigned to support Agile project and team success.

Required Skills/Abilities:

• Organized with attention to detail.
• Excellent analytical, logical thinking, and problem-solving skills.
• Excellent verbal and written communication skills.
• Ability to build and optimize data sets, "big data" data pipelines, and architectures.
• Ability to perform root cause analysis on external and internal processes and data to identify opportunities for improvement and answer questions.
• Excellent analytical skills with working on unstructured datasets.
• Ability to build processes that support data transformation, workload management, data structures, dependency, and metadata.
• Experience with data pipeline and workflow management tools (e.g., Apache Airflow, AWS Glue).
• Proficiency in programming languages such as Python, Java, or Scala.
• Hands-on experience with cloud-based data solutions (AWS, Azure, or Google Cloud).
• Strong understanding of data warehousing concepts and tools (e.g., Snowflake, Redshift, BigQuery).
• Experience working with structured and unstructured data.
• Knowledge of data governance, security, and compliance best practices.

Education and Experience:

• Bachelor's degree in Computer Science, Data Science, or Information Science-related field required; Master's degree preferred.
• Experience with real-time data processing frameworks (e.g., Apache Kafka, Spark Streaming).
• Experience with data visualization tools (e.g., Tableau, Power BI, Looker).
• At least three years of related experience required.