Data Engineer

Anywhere
Posted 3 years ago
This position has been filled

Website Technical Options

Data Engineer will Design and Document the new

Data Engineer will Design and Document the new architecture and development process to convert Database and data warehouse models into Hadoop based systems.

Develop the Large-scale data processing pipelines to handle Peta bytes of transaction data and egress to analytical sources.

Work with Apache Spark 2.0 Utilizing the Spark SQL and Streaming components to support the intraday and real-time data processing

Use the Scala 2.12 and Java 7 Programming to build the UDFs and supporting utilities for the standardizing the data pipelines

Implement the Spark Best practices to efficiently process data to meet ETAs by utilizing features like partitioning, resource tuning, memory management and Check pointing features

Resolve any open Spark and Yarn resource management issues, OOM errors, Shuffle exceptions, heap space errors, Null Pointer Exceptions and schema compatibility in Spark

Work on multiple data formats like ORC, Parquet, Avro, JSON and XML etc

Develop Batch and streaming workflows with in-built Stone branch scheduler and bash scripts to automate the Data Lake systems

Work on converting the multiple SQL Server and Oracle stored procedures into Hadoop using Spark SQL, Hive, Scala and Java

Extract, Parse, Clean and ingest the incoming data feeds, data and transaction logs