Philadelphia, PA
Fort Lauderdale, FL
Databricks Architect
building databricks lakehouse architecture from end to end
- REMOTE
- Philadelphia, PA +1
- $95 - $105
- Hourly
- Managed by Jobot Pro: Katrina McFillin
A bit about us:
We are a company focused on the exploration and advancement of critical metals projects globally.
Why join us?
As a Jobot consultant, you will receive the following benefits:
75% of your low deductible healthcare premium will be covered by Jobot with 25% covered for your dependents
access to a 401(k)
40 hours of PTO
75% of your low deductible healthcare premium will be covered by Jobot with 25% covered for your dependents
access to a 401(k)
40 hours of PTO
Job Details
You will be building a databricks lakehouse from end to end.
You will be leading a team setting up and configuring Databricks. You will establish a dev workspace, a storage account, and a key vault necessary for basic functionality to align with the aspects of the current and future state. Data Ingestion of the desired data and Unity catalog will be set up and configured.
Following the set-up and configuration of the Databricks medallion architecture, and, based on level of complexity and customization required, you will create one pipeline sourcing data in an ongoing or incremental manner. Data will be sourced and processed with limited transformations in the silver layer, and datasets will be prepared for BI and data exploration from gold.
You will evaluate the Platform Architecture to include its current state and future state to determine which of the Databricks features and tools should be included in the future state, taking into consideration relevant dependencies for MLops/Unity Catalog/etc.
To ensure the Databricks Lakehouse architecture is successful for their go-to-market product, additional deliverables will need to be completed. These include, but are not limited to 1) the establishing of a Production environment in Databricks, 2) the orchestration of Unity Catalog for future state and scalability of data, 3) contextualized MLops/MLflow expertise and implementation, 4) additional silver and gold layer transformations relevant to business needs for data science, BI tools, Web App integration, etc. These services will be agreed upon and documented (in backlog) in conjunction with defined success criteria.
Tech stack: React / C# / Azure / Apache Spark / Delta Lake / Unity Catalog / Databricks
You will be leading a team setting up and configuring Databricks. You will establish a dev workspace, a storage account, and a key vault necessary for basic functionality to align with the aspects of the current and future state. Data Ingestion of the desired data and Unity catalog will be set up and configured.
Following the set-up and configuration of the Databricks medallion architecture, and, based on level of complexity and customization required, you will create one pipeline sourcing data in an ongoing or incremental manner. Data will be sourced and processed with limited transformations in the silver layer, and datasets will be prepared for BI and data exploration from gold.
You will evaluate the Platform Architecture to include its current state and future state to determine which of the Databricks features and tools should be included in the future state, taking into consideration relevant dependencies for MLops/Unity Catalog/etc.
To ensure the Databricks Lakehouse architecture is successful for their go-to-market product, additional deliverables will need to be completed. These include, but are not limited to 1) the establishing of a Production environment in Databricks, 2) the orchestration of Unity Catalog for future state and scalability of data, 3) contextualized MLops/MLflow expertise and implementation, 4) additional silver and gold layer transformations relevant to business needs for data science, BI tools, Web App integration, etc. These services will be agreed upon and documented (in backlog) in conjunction with defined success criteria.
Tech stack: React / C# / Azure / Apache Spark / Delta Lake / Unity Catalog / Databricks
Easy Apply Now
Job Details