Search jobs
Back to search Apply now
Ref: #74340

Lead Data Engineer

  • Practice Data

  • Technologies Business Intelligence Jobs and Data Recruitment

  • Location London, United Kingdom

  • Type Contract

Data Engineering Consultant (Snr/Lead)
Role Overview
We are hiring a hands-on Databricks Engineer that has experience delivering modern data platforms on Databricks. This role requires a minimum of 2 years of Databricks experience gained recently (i.e., current/recent projects using modern platform capabilities).
Key Responsibilities

  • Engineering Delivery of Databricks lakehouse solutions from ingestion to curated serving layers.
  • Define and implement Medallion Architecture (Bronze/Silver/Gold) and reusable engineering patterns.
  • Build scalable ingestion pipelines using AutoLoader, Lakeflow Connect, batch/streaming, and incremental patterns.
  • Develop Declarative Pipelines with Expectations (DLT) to enforce and monitor data quality.
    Implement and operate Unity Catalog for governance, access control, lineage, and secure data sharing patterns.

  • Drive code quality and operational excellence (CI/CD approach, testing strategy, monitoring, incident triage).
  • Partner with architects, platform teams, and stakeholders to align delivery with enterprise standards.
  • Mentor engineers and act as the technical escalation point during delivery.

Minimum Experience (Filter Criteria)

  • Handson Databricks experience, in recent years (e.g., within the last 2–3 years), demonstrating usage of modern Databricks capabilities and patterns.
  • Evidence of production delivery (not trainingonly or lab-only exposure).

Must Have (Non-Negotiable)

  • Databricks Certification, at least one of: Databricks Certified Data Engineer Associate/Professional OR Databricks Certified Machine Learning Associate/Professional OR Databricks Certified Generative AI Engineer (Associate)
  • Unity Catalog handson experience:

Metastore/catalog design, grants, lineage, and secure access patterns.

  • Declarative Pipelines with Expectations (DLT):

Building pipelines, defining expectations, handling failures/quarantines, observability.

  • Ingestion engineering using Databricksnative approaches:
  • AutoLoader and/or Lakeflow Connect, streaming and incremental ingestion patterns.
  • Medallion Architecture implementation and best practices:
    Designing and implementing Bronze/Silver/Gold with practical decisions (schema evolution, CDC/upserts, SCD patterns, performance strategy). 

Should Have

  • Demonstrable use of latest Databricks capabilities (candidate can explain what they used recently and why).
  • Strong Databricks engineering fundamentals:
  • Delta Lake (MERGE, schema enforcement/evolution, OPTIMIZE/ZORDER, VACUUM)
  • Databricks Workflows / job orchestration
  • Productiongrade PySpark/SQL
  • Clear understanding of pipeline reliability:
    Observability, alerting, replay/backfill strategies, and operational runbooks.

Nice to Have

  • Lakeflow ingestion connectors (specific connector experience is a plus).
  • RBAC / masking implementations (rowlevel security, column masking, sensitive data handling) using Unity Catalog.
  • GenAI on Databricks: Mosaic AI, Vector Search, model serving, RAG pipelines, AI Functions.
    Lakebase awareness or handson experience.

  • Workload/query optimisation: Photon usage, cluster sizing, shuffle/skew mitigation, caching strategy, partitioning, file sizing.
  • Cost awareness and controls: Understanding DBU drivers, job vs allpurpose compute, cluster policies, monitoring and chargeback/showback patterns.
Fügen Sie eine Lebenslaufdatei an. Akzeptierte Dateitypen werden DOC, DOCX, PDF, HTML und TXT.

Wir laden Ihre Bewerbung hoch. Es kann einige Augenblicke dauern, bis Sie Ihren Lebenslauf lesen können. Bitte warten!