Data Engineer

Anywhere
Posted 3 years ago
This position has been filled

Website Technical Options

We are seeking a Data Engineers to Design and Docu

We are seeking a Data Engineers to Design and Document the new architecture and development process to convert Database and data warehouse models into Hadoop based systems.

Develop the Large-scale data processing pipelines to handle Peta bytes of transaction data and egress to analytical sources.

Use the Scala 2.12 and Java 7 Programming to build the UDFs and supporting utilities for the standardizing the data pipelines

Work with Apache Spark 2.0 Utilizing the Spark SQL and Streaming components to support the intraday and real-time data processing

Develop Batch and streaming workflows with in-built Stone branch scheduler and bash scripts to automate the Data Lake system

Implement the Spark Best practices to efficiently process data to meet ETAs by utilizing features like partitioning, resource tuning, memory management and Check pointing features

Resolve any open Spark and Yarn resource management issues, OOM errors, Shuffle exceptions, heap space errors, Null Pointer Exceptions and schema compatibility in Spark

Work on multiple data formats like ORC, Parquet, Avro, JSON and XML etc

Work on converting the multiple SQL Server and Oracle stored procedures into Hadoop using Spark SQL, Hive, Scala and Java

Extract, Parse, Clean and ingest the incoming data feeds, data and transaction logs