Data Engineer
Apply NowCompany: VDart
Location: San Francisco, CA 94112
Description:
Job Title: Data Engineer
Location: San Francisco CA/Onsite
Duration: 3 Months extendable project
Technical Skills
Must Have
Advanced SQL
Apache Spark
Databricks
GitHub
Python
Terraform
Job Description
Keywords: PySpark, SQL, Databricks, GitHub, Python
Location: San Francisco CA/Onsite
Duration: 3 Months extendable project
Technical Skills
Must Have
Advanced SQL
Apache Spark
Databricks
GitHub
Python
Terraform
Job Description
- In this role, you will assist the threat detection team at Databricks in building threat detection pipelines (log analytics for security), using PySpark, SQL, and Databricks.
- You will help get these pipelines deployed over the terraform.
- For this specific contract, we are looking for someone to help us with platform migration involving migration of detection rules and content from a legacy platform and into databrick's native internal detection framework.
- This will involve creating equivalent PySpark log pipelines, rule configuration files, writing unit tests, performing data validation checks, and then deploying into production.
- Minimum of 3-5 years of experience is preferred.
Keywords: PySpark, SQL, Databricks, GitHub, Python