Key Responsibilities

Functional 

Responsibilities

Gather and process raw data at scale (including writing scripts, web scraping, calling  APIs write SQL queries, etc.)

Design, implement and maintain high performance big data infrastructure /systems,  & big data processing pipelines scaling to structured and unstructured business  requirements

Create web services / API based interfaces to modularize capabilities across lending  products

Developing prototypes and proof of concepts for the selected solutions, and  implementing complex big data projects

Drive data investigations across organizations and deliver resolution of technical,  procedural, and/or operational issues to completion

Conduct timely and effective research in response to specific requests (e.g. data  collection, summarization, analysis, and synthesis of relevant data and information)

Enhance/optimize data migration procedures to include data that is relevant for  building analytic systems

Work with business domain experts, data scientists and application developers to  identify data relevant for analysis and develop the Big Data solution

Adapting and learning new technologies surrounding Big Data ecosystems and  ensure learning and growth of team members

Job Requirements

Qualifications 

Masters or PhD in Computer science, Engineering (CS, ECE, IT etc.), Mathematics,  Computational physics

Experience 

4-8 years in data engineering or warehousing with 3+ years in high performance data  engineering roles

Functional Competencies

Big data tech: Spark, Data Lake, Airflow

Data pipelines with Kafka streams, RESTful APIs, API gateways etc

Heterogeneous data environments: Cloud (AWS) Relational, NoSQL, GraphDBs in  production

Programming languages like python. Shell script

S/w development practices: Agile, devOps CI/CD, automated testing / deployments,  WebServices, MLOps

Behavioral Competencies

Hands-on contributor, passionate about applying tech in businesses

Eye for detail

Strong Analytical & Interpersonal skill