Technical Options

Job Recruiter

  • About
  • Testimonials
  • Resources
  • Employers Services
  • Job Seekers
  • Jobs
  • Follow us on
  • Follow us on
  • Find us on
  • Send Resume
  • Contact
  • Home

1009: Data Engineer

February 18, 2019 By sysadmin

  • Anywhere
  • Posted 3 years ago
  • This position has been filled
Technical Options

Website Technical Options

Data Engineer will Design and Document the new

Data Engineer will Design and Document the new architecture and development process to convert Database and data warehouse models into Hadoop based systems.

  • Develop the Large-scale data processing pipelines to handle Peta bytes of transaction data and egress to analytical sources.
  • Work with Apache Spark 2.0 Utilizing the Spark SQL and Streaming components to support the intraday and real-time data processing
  • Use the Scala 2.12 and Java 7 Programming to build the UDFs and supporting utilities for the standardizing the data pipelines
  • Implement the Spark Best practices to efficiently process data to meet ETAs by utilizing features like partitioning, resource tuning, memory management and Check pointing features
  • Resolve any open Spark and Yarn resource management issues, OOM errors, Shuffle exceptions, heap space errors, Null Pointer Exceptions and schema compatibility in Spark
  • Work on multiple data formats like ORC, Parquet, Avro, JSON and XML etc
  • Develop Batch and streaming workflows with in-built Stone branch scheduler and bash scripts to automate the Data Lake systems
  • Work on converting the multiple SQL Server and Oracle stored procedures into Hadoop using Spark SQL, Hive, Scala and Java
  • Extract, Parse, Clean and ingest the incoming data feeds, data and transaction logs
  • Jobs | Job Seekers | Employers | Job Resources | Testimonials | About Us | Sitemap | RSS

    Copyright © 2022 Technical Options, Inc.