Technical Options

Job Recruiter

  • About
  • Testimonials
  • Resources
  • Employers Services
  • Job Seekers
  • Jobs
  • Follow us on
  • Follow us on
  • Find us on
  • Send Resume
  • Contact
  • Home

1037: Data Engineer

March 19, 2019 By sysadmin

  • Anywhere
  • Posted 4 years ago
  • This position has been filled
Technical Options

Website Technical Options

We are seeking a Data Engineers to Design and Docu

We are seeking a Data Engineers to Design and Document the new architecture and development process to convert Database and data warehouse models into Hadoop based systems.

  • Develop the Large-scale data processing pipelines to handle Peta bytes of transaction data and egress to analytical sources.
  • Use the Scala 2.12 and Java 7 Programming to build the UDFs and supporting utilities for the standardizing the data pipelines
  • Work with Apache Spark 2.0 Utilizing the Spark SQL and Streaming components to support the intraday and real-time data processing
  • Develop Batch and streaming workflows with in-built Stone branch scheduler and bash scripts to automate the Data Lake system
  • Implement the Spark Best practices to efficiently process data to meet ETAs by utilizing features like partitioning, resource tuning, memory management and Check pointing features
  • Resolve any open Spark and Yarn resource management issues, OOM errors, Shuffle exceptions, heap space errors, Null Pointer Exceptions and schema compatibility in Spark
  • Work on multiple data formats like ORC, Parquet, Avro, JSON and XML etc
  • Work on converting the multiple SQL Server and Oracle stored procedures into Hadoop using Spark SQL, Hive, Scala and Java
  • Extract, Parse, Clean and ingest the incoming data feeds, data and transaction logs
  • Jobs | Job Seekers | Employers | Job Resources | Testimonials | About Us | Sitemap | RSS

    Copyright © 2023 Technical Options, Inc.