Earlier than making architectural choices, it’s price revisiting the broader migration technique. In our earlier submit, we launched Databricks Skilled Companies’ strategy to complicated knowledge warehouse migrations, highlighting the significance of early choices round technique and design. These foundational decisions immediately affect the goal platform’s implementation and structure.
We additionally launched two sequencing methods: ETL-first and BI-first. The BI-first strategy delivers fast worth by modernizing the consumption layer, whereas the ETL-first strategy focuses on upstream pipelines. Every has its place, relying on priorities.
On this submit, we discover some of the important design decisions: selecting between a Raise-and-Shift or Modernization strategy. We clarify what every strategy entails, when to make use of it, and the way to merge them right into a hybrid strategy for long-term success on Databricks.
From technique to migration strategy: selecting the correct path
After you’ve aligned on the broader migration technique—ETL-first or BI-first—the subsequent main choice is the way to construction the migration. Do you replicate what exists, or reimagine it for the longer term?
This architectural choice sometimes comes down to 2 core approaches:
- Raise-and-Shift: Transfer workloads as-is to speed up the migration
- Modernization: Redesign the platform to unlock long-term effectivity and scale
The best strategy relies on your objectives, constraints, and timeline. Under, we break down the tradeoffs of every and embody a hybrid mannequin that many organizations use to mix the very best of each.
Raise and shift migration
Raise-and-Shift includes transferring your present knowledge fashions and codebase to the brand new platform with minimal modifications. You don’t introduce new use instances, and the structure stays unchanged.
This strategy is interesting as a result of it’s simpler to scope, plan, and automate. Instruments like profilers and code analyzers assist measure workload patterns, complexity, and value, making it simpler to guage and execute.
Key advantages embody:
- Predictable timelines
- Automated tooling (e.g., code converters, reconciliation validators)
- Quicker migration when going through deadlines or expiring licenses
For instance, code converters can robotically deal with as much as 80% of scripts. Since performance stays the identical, validation and working queries on each methods and evaluating outputs are simpler.
On Databricks, Raise-and-Shift will get you off legacy platforms shortly whereas unlocking fast efficiency good points utilizing options like z-ordering and liquid clustering. After you’ve got migrated, your group can start incrementally modernizing the platform.
Modernize the migration sample
Modernizing, in distinction to Raise-and-Shift, means constructing a brand new knowledge platform in your goal system with out being constrained by your legacy structure. The main target shifts from merely migrating present property to reimagining use instances and designing for future wants. As an alternative of mapping outdated optimizations, you implement greatest practices and the well-architected pillars of the lakehouse.
On an open lakehouse, this includes refactoring code and re-architecting knowledge constructions to satisfy your group’s present and future scalability, efficiency, value, and functionality necessities, free from legacy limitations.
Tooling stays helpful, however extra for discovery and planning:
- Profilers and code analyzers assist stock what it is advisable to migrate
- Code converters and reconciliation instruments play a minimal position, since this isn’t a direct code translation
This strategy is right when you’ve got versatile timelines and an outdated or overly complicated legacy system, usually with hundreds of tables and scripts. Whereas beginning recent can really feel gradual and overwhelming, the long-term advantages are substantial: simplified structure, higher efficiency, and decreased upkeep overhead.
That stated, migrating hundreds of scripts usually means sustaining their upkeep complexity. If that appears daunting, take into account partnering with Databricks Skilled Companies or licensed migration consultants to assist information the planning and design section and guarantee a smoother path.
A hybrid strategy: elevate and shift, after which modernize
One other strategy is a hybrid migration technique that balances pace with long-term worth. You’ll start with the Raise-and-Shift strategy to eliminating their legacy platform as shortly as potential, particularly when going through pressing constraints like expiring licenses. Automation and repeatable tooling assist speed up this preliminary section and cut back threat throughout execution.
You may transfer into the modernization section after you migrate your workloads to Databricks.
Within the hybrid strategy, you:
- Combine new and fashionable knowledge sources
- Implement a knowledge product technique
- Allow superior analytics, AI, and new use instances that drive enterprise choices
This section usually requires architectural updates however permits you to evolve regularly. With a hybrid technique, you don’t should modernize all the pieces on day one—you construct on a secure basis whereas aligning with future necessities.
For those who’re pursuing this strategy, Databricks Skilled Companies and licensed companions might help information your roadmap, making certain a clean transition and a future-ready structure.
Our standpoint
Deciding on a migration strategy shouldn’t be a one-size-fits-all. The most typical strategy is a hybrid migration:
- Create a migration manufacturing unit that leverages automation instruments.
- Raise-and-shift the vast majority of the codebase.
- Allow out-of-the-box optimizations, similar to z-ordering and liquid clustering, to start out your modernization effort.
Databricks can act as your major knowledge warehouse. For instance, you’ll be able to migrate saved procedures to notebooks and use SQL Scripting for scalability and AI integration with out leaving the consolation of SQL. Migrating Transact-SQL to every other cloud knowledge warehouse requires an analogous effort to migrating that Transact-SQL to a pocket book with Python code wrapped round your SQL performance. The advantage of utilizing a pocket book is that you simply additionally get flexibility and a fantastic growth expertise.
What to do subsequent
Able to modernize your knowledge warehouse? Obtain our eBook, “Reworking Legacy Information Warehouses: A Strategic Migration Blueprint,” for detailed methods and greatest practices that guarantee a low-risk transition to the Databricks Information Intelligence Platform.
