GCP DATA ENGINEER

Role Description & Requirements

Role and Responsibilities

Key cloud engineer for customer engagementsCollaborate with lead Data Consultant on customer engagementsDefine solution architecture and detailed design for data platforms on GCPBuild solutions, i.e., perform hands on coding for the designed solutionInteract with customer data engineering team to provide sound advice and guidance on solution architecture as well as hands on coding solutionsPerform pair programming with customer’s data engineers and handover the developed solution to customer engineers.

Minimum Qualifications

Solution Architecture Knowledge and Expertise Required

○ Designing data platform architecture and design for

migrating open source or other public cloud based data platforms to Cloud (pref. GCP) Designing green field data platforms on cloud

○ Ability to perform research on tools and technology, identify important considerations for technology selection in the architecture, provide recommendations on tool selection

○ Define solution architecture and detailed design for

batch ingestion and processing of data from sources such as files, cloud and on-prem databases, streaming solutions for real time data ingestion and processing (such as log analytics, click-stream analytics, real time data ingestion from other types of sources)

○ Build data pipelines and CI/CD pipelines

○ In-depth knowledge of at least one public cloud, preferably GCP

Tools experience Required

○ GCP (GCP preferred, or equivalent in other public clouds acceptable)

○ GCP Dataflow, Pub/Sub, Cloud Composer, BigQuery, Cloud Run, Cloud

○ Functions,Spanner, CloudBuild, Vertex AI, Terraform

Coding expertise

○ Programming knowledge and willingness to be hands-on as part of day to day work

○ Knowledge of at least one (preferably both): Python, Java Preferred Qualifications

Solution Architecture Knowledge and Expertise Required

○ Experience with open source ecosystem and distributions such as Hadoop, Spark, Cloudera/Hortonworks

○ Understanding and experience with NoSQL databases such as HBASE, MongoDB

Tools experience Required

○ Other public clouds

○ Open source ecosystem and frameworks (Hadoop, SPARK, Oozie, Kafka, HBASE) Certification

At least one of below:

○ GCP Data engineering (preferred)

○ AWS Customer Experience

Experience in large enterprise environments, navigating architectural review and approval frameworks, lobbying design submissions, engaging cross-functional stakeholders (Preferably one of the Big Four AU Banks)