Brief Description of position:

The Lead Data Engineer is responsible for the development of high performance, distributed computing tasks using Big Data technologies such as Hadoop (platform level), NoSQL, text mining and other distributed environment technologies based on the needs of the organization. The Lead Data Engineer is also responsible for analyzing, designing, programing, debugging and modifying software enhancements and/or new products used in distributed, large scale analytics solutions. 

PRINCIPAL DUTIES AND RESPONSIBILITIES Designing and Implementation:

Technical Requirement:

• Hands on coding experience in Big Data Technologies like Hadoop/Google Big Query/Redshift/Snowflake, KAFKA, Hive, Spark , Flink, Storm etc
• Hands on in building real time & batch ETL/ELT solution using open source technology like Spark/Flink/Storm/Kafka Streaming.
• Hands on in creating data model (ER and Dimension Model) to help data consumers to create high performance consumption layer.
• Strong experience in following OOPs programming(using Python/Java/Scala).
• Experience ins database concepts like OLTP , OLAP, Start & Snowflake Schema, Normalization and Denormalization etc.
• Experience working in Linux/Unix environment and exposure to command line utilities.
• Experience creating/supporting production software/systems and a proven track record of identifying and resolving performance bottlenecks for production systems.
• Familiar with open source workflow management software like Airflow/Oozie.
• Any additional knowledge of maintaining data quality by introducing data governance/validation framework would be good.
• Any experience in Google Cloud Platform and SAP will have added advantage.
• Any experience in Talend & Apache Nifi will add advantage.
• Experience in the creating scalable distributed system ti meet future data volume & processing need.
• Maintain knowledge base and processes across different phases of development life cycle. 

Scope of Work:

• Understand strategic requirements to ensure effective solutions meet business requirements.
• Building End to End Data engineering solutions to enable consumers seamless data access meeting availability and Quality SLA.
• Review and approve specifications to ensure consistency in approach and use
• Assist partners as a technical and business-savvy resource to software engineering personnel on a range of software design issues
• Perform systems and applications performance characterization and trade-off studies through analysis and simulation
• Continuous improvements in coding standards
• The role will be responsible for providing innovative operational solutions and best practices

MINIMUM JOB REQUIREMENTS:

• 5 plus years of experience in developing software applications including: analysis, design, coding, testing, deploying and supporting of applications
• 4+ years professional experience in building robust data pipelines and writing ETL/ELT code using Python/Java/Scala, Spark, Shell Scripts, SQL etc)
• Minimum 3 implementations under Big Data Ecosystem.
• Strong communication Skill.
• BS in Computer Science, Applied Mathematics, Physics, Statistics or area of study related to data sciences and data mining or relevant experience

Location

Hyderabad

Minimum Qualification:

Graduate