Big Data Engineer
Apply NowCompany: First Soft Solutions
Location: Monmouth Junction, NJ 08852
Description:
We are actively hiring for Big data engineer
Location : Remote
W2
Visa: USC, GC, GC EAD, H4 EAD
Looking for candidates with 10+ years experience
Must have hands-on experience with Databricks
Hands-on experience with high-velocity high-volume stream processing: Apache Kafka and Spark Streaming (A. Experience with real-time data processing and streaming techniques using Spark structured streaming and Kafka B. Deep knowledge of troubleshooting and tuning Spark applications)
Hands-on experience with Python and/or Scala i.e. PySpark/Scala-Spark
Hands-on experience building, testing, and optimizing 'Big Data' data ingestion pipelines, architectures and data sets
Experience in successfully building and deploying a new data platform on Azure/ AWS
Strong knowledge of Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series
Databricks UI, Managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity Catalog
Experience with data ingestion of different file formats across like JSON, XML, CSV
Experience with NoSQL databases, including HBASE and/or Cassandra
Knowledge of Unix/Linux platform and shell scripting is a must
Experience with Cloud platforms e.g. AWS, GCP, etc. Experience with database solutions like Kudu/Impala, or Delta Lake or Snowflake or BigQuery
Please share updated profile for *************
Location : Remote
W2
Visa: USC, GC, GC EAD, H4 EAD
Looking for candidates with 10+ years experience
Must have hands-on experience with Databricks
Hands-on experience with high-velocity high-volume stream processing: Apache Kafka and Spark Streaming (A. Experience with real-time data processing and streaming techniques using Spark structured streaming and Kafka B. Deep knowledge of troubleshooting and tuning Spark applications)
Hands-on experience with Python and/or Scala i.e. PySpark/Scala-Spark
Hands-on experience building, testing, and optimizing 'Big Data' data ingestion pipelines, architectures and data sets
Experience in successfully building and deploying a new data platform on Azure/ AWS
Strong knowledge of Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series
Databricks UI, Managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity Catalog
Experience with data ingestion of different file formats across like JSON, XML, CSV
Experience with NoSQL databases, including HBASE and/or Cassandra
Knowledge of Unix/Linux platform and shell scripting is a must
Experience with Cloud platforms e.g. AWS, GCP, etc. Experience with database solutions like Kudu/Impala, or Delta Lake or Snowflake or BigQuery
Please share updated profile for *************