Snowflake & DBT Engineering
Apply NowCompany: Tata Consultancy Services
Location: Plano, TX 75025
Description:
Job Title - Snowflake & DBT Engineering
Experience Range - 4+ years
Technical/Functional Skills
Snowflake, DBT
Roles & Responsibility :
. Snowflake & DBT Infrastructure Development:
* Design and implement scalable, modular, and reusable DBT models.
* Develop incremental and snapshot models to optimize data processing and reduce compute costs
* Implement Common Table Expressions (CTEs), materialized views, and table strategies to enhance performance.
* Define and implement DBT tests.
* Develop custom DBT macros.
* Optimize SQL queries for performance and cost efficiency.
* Develop and manage ETL/ELT pipelines using Snowflakes capabilities like Snowpipe, Streams, and Tasks.
2. Model Deployment:
* Ensure the smooth operation of deployed models by monitoring performance and identifying potential issues.
* Manage DBT projects using Git, ensuring proper version control and collaboration.
* Set up CI/CD pipelines to automate testing, validation, and deployment of DBT models
* Set up data sharing, masking, and encryption as needed.
3. Data Management:
* Maintain comprehensive DBT documentation for models, sources, and transformations.
* Ensure the quality and integrity of data used in dbt models by implementing data validation and cleaning processes.
* Implement data security measures to protect sensitive data.
* Ensure transparency in data workflows by maintaining well-documented YAML files.
4. Monitoring and Optimization:
* Set up alerting mechanisms for failures in DBT transformations.
* Create and monitor jobs using Control M for dbt models.
* Monitor execution times, identify performance bottlenecks, and apply optimizations.
* Ensure end-to-end data pipeline reliability, from ingestion to transformation and reporting.
5. Collaboration and Communication:
* Work closely with cross-functional teams, including data scientists, software engineers, and DevOps engineers.
* Develop and maintain documentation for Snowflake infrastructure and dbt workflows.
* Present findings and recommendations to stakeholders through clear and effective communication.
Salary Range - $80000 to $120000 a year
Experience Range - 4+ years
Technical/Functional Skills
Snowflake, DBT
Roles & Responsibility :
. Snowflake & DBT Infrastructure Development:
* Design and implement scalable, modular, and reusable DBT models.
* Develop incremental and snapshot models to optimize data processing and reduce compute costs
* Implement Common Table Expressions (CTEs), materialized views, and table strategies to enhance performance.
* Define and implement DBT tests.
* Develop custom DBT macros.
* Optimize SQL queries for performance and cost efficiency.
* Develop and manage ETL/ELT pipelines using Snowflakes capabilities like Snowpipe, Streams, and Tasks.
2. Model Deployment:
* Ensure the smooth operation of deployed models by monitoring performance and identifying potential issues.
* Manage DBT projects using Git, ensuring proper version control and collaboration.
* Set up CI/CD pipelines to automate testing, validation, and deployment of DBT models
* Set up data sharing, masking, and encryption as needed.
3. Data Management:
* Maintain comprehensive DBT documentation for models, sources, and transformations.
* Ensure the quality and integrity of data used in dbt models by implementing data validation and cleaning processes.
* Implement data security measures to protect sensitive data.
* Ensure transparency in data workflows by maintaining well-documented YAML files.
4. Monitoring and Optimization:
* Set up alerting mechanisms for failures in DBT transformations.
* Create and monitor jobs using Control M for dbt models.
* Monitor execution times, identify performance bottlenecks, and apply optimizations.
* Ensure end-to-end data pipeline reliability, from ingestion to transformation and reporting.
5. Collaboration and Communication:
* Work closely with cross-functional teams, including data scientists, software engineers, and DevOps engineers.
* Develop and maintain documentation for Snowflake infrastructure and dbt workflows.
* Present findings and recommendations to stakeholders through clear and effective communication.
Salary Range - $80000 to $120000 a year