Principal Data Engineer

Job Description

Lead AI & Data Platform Engineer

• Location: Melbourne (Hybrid Work Model)
• Salary: $180,000 – $210,000 + Super
• Industry: Financial Services
• Contract Type: Permanent / Full-Time

About the Role

We are seeking a high-caliber Lead AI & Data Platform Engineer to spearhead the evolution of our next-generation data architecture. In this role, you will bridge the gap between advanced Data Engineering and production-grade MLOps, designing resilient, petabyte-scale systems within a highly regulated Financial Services environment.

You will be our internal Databricks Expert, driving the strategy and deployment of AI Agentic workloads, advanced analytics pipelines, and secure cloud infrastructure.

Key Responsibilities

1. Databricks Platform Excellence

• Architect and maintain our enterprise Lakehouse infrastructure utilising Delta Lake and Medallion topology (Bronze/Silver/Gold layers).
• Enforce robust data governance and secure discovery across the enterprise using Unity Catalog.
• Build, monitor, and scale production pipelines via Delta Live Tables.

2. Production AI & MLOps Infrastructure

• Deploy, monitor, and scale production-grade ML and AI Agentic workloads.
• Operationalise LLM and Generative AI pipelines utilizing Databricks Mosaic AI, MLflow, Langchain, and Vector Databases.
• Manage corporate feature stores and optimize model serving endpoints.

3. AWS Cloud Architecture & Infrastructure as Code (IaC)

• Design secure, high-performance cloud environments using Terraform.
• Manage intricate AWS configurations including S3 optimization, VPC peering, AWS PrivateLink, and granular IAM roles.
• Integrate cloud-native AI services like AWS Bedrock seamlessly into our data ecosystem.

4. Performance Tuning & Streaming

• Conduct advanced Spark optimization to handle high-concurrency, petabyte-scale data pipelines.
• Tune GPU clusters specifically optimized for heavy AI/ML training and inference workloads.
• Architect real-time streaming pipelines leveraging Kinesis or Kafka.

Your Profile

To be successful in this role, you will bring a blend of elite engineering skills and a deep understanding of modern data-ops.

Technical Requirements:

• Core Languages: Mastery of PySpark, Python, SQL, and Scala.
• AI Frameworks: Strong familiarity with PyTorch, TensorFlow, or Hugging Face.
• DataOps & CI/CD: Proven experience embedding automated testing and CI/CD pipelines into data deployments.
• Industry Experience: Prior experience within Financial Services or a similarly regulated environment is highly desirable (understanding of data privacy, compliance, and risk frameworks).

What’s on Offer

• Competitive Compensation: $180K – $210K base salary, depending on experience.
• Hybrid Flexibility: A healthy balance of remote work and collaborative time in our modern Melbourne CBD office.
• Cutting-Edge Tech Stack: Total backing to utilize the absolute latest in Generative AI, Mosaic AI, and cloud-native tools.