Data Platform Architect
What you’ll do?
• You will drive the design & architecture of data platform,
infrastructure, and tools including but not limited to streaming
infrastructure, streaming & batch processing, metadata management, data
warehouse, data quality, data security etc.
• You will provide clear technical guidance to the team based on your own
working experience and industry best practices to deliver a modern data
processing platform that includes stream & batch processing powered by
Spark, data warehousing powered by Snowflake, data quality and reliability,
metadata management capabilities.
• You will build POCs to evaluate and make right technology choices.
• You will build prototypes to provide guidance for the rest of the data
engineering team.
• You will work closely with product management team with respect to
feature prioritization, delivery, high level estimates of effort and high and
mid-level designs while not losing sight of technical debt.
• You will collaborate and communicate effectively within the team and
across teams to deliver impactful data platform and services.
• You will understand how LendingClub’s data is used and what it all means.
Who You Are:
• You are passionate designing and leading the implementation of resilient,
distributed software platforms and large-scale data infrastructures.
• You have excellent listening skills and empathetic to others.
• You believe in simple and elegant solutions and give paramount importance
to quality.
• You’ve a track record of shipping working software fast and reliably.
What you’ll need to succeed:
Must have skills:
• B.S., M.S. or Ph.D. degree in computer science or a related field or
equivalent work experience.
• Expert level proficiency in writing production quality code, preferably Scala
& Python.
• Excellent knowledge and proven experience of common design patterns
and architectures used in distributed data processing, data quality, data
warehouses, metadata management etc.
• In-depth knowledge of and experience working on distributed data
processing systems using the open-source technologies like MapReduce,
Hive, Tez, Spark and Kafka.
• Experience building data platforms on public cloud infrastructure,
preferably AWS.
• Bonus points if experience working with EMR, Databricks and Snowflake.
Nice to have skills:
• Working knowledge of open-source ML frameworks and end-to-end model
development life cycle.
• Previous working experience with running containers (Docker/LXC) in a
production environment using one of the container orchestration services
(Kubernetes, Docker Swarm, AWS ECS, AWS EKS).
Job Features
Job Category | Architect |