We’re excited to introduce development branches, a safe and fast way for engineers to iterate new ideas on production projects.
The days when someone uploaded a file somewhere to a production environment and fixed it are prehistoric for developers. But in BI and data manipulation in general, it's often a slightly different song. DEV and TEST, if they exist at all, are seldom an image of production. Not every team can really afford to have a separate copy of the warehouse for every analyst, so they can see if their changes will break something or not.
Fortunately, thanks to the Development Branches in Keboola, literally everyone can start using it in a few clicks. You simply don't do it on production, but first try it sideways, collaborate with your teammates and then deploy it to production—or let someone responsible deploy it.
Real-life example, now it gets interesting
Imagine that your production system checks your inventory every 3 minutes and increases the supply of your best-selling items, so that they don't run out. And you need to intervene and, for example, adjust the algorithm to ignore items that do not make more than 5% of the whole order in the average shopping cart (in Amazon, for example, the display cleaner). But, in the 3-minute window, you don't have enough time to make a change and check it. What to do with this?
It is simple:
- You copy all input data sideways.
- You copy scripts and chunks of code that make application logic into the new environment.
- You simulate a change in input data between two runs.
- You now have a TEST environment where you can try the change.
- When done, you still have to test it to see if it breaks the follow-up steps.
All steps are easy if:
- You don't have much data.
- The data is not linked.
- Your requested change does not need to check the changed data output.
If your answer is "Yes” to a, b, or c, you can burn many times more hours preparing such an environment in comparison with the hours needed to modify the conditions that select items for stock transfer analysis.
And, precisely because of the disproportion between preparing the right test environment for the work you need to do, it's no surprise that people who work with data often don't test their work. One gets used to everything, but working directly on production is a dangerous, bad habit.
Keboola now solves this in a few clicks and makes your life much easier! So, no more epic fails in production, no more scary horror stories for newbies, no BDM.SALES_V4_JANUARY_TEST tables, just development branches available directly in the UI to test changes before you put them into production. It's that simple.
Development mode gives you the ability to:
- Create an unlimited number of development branches with custom names.
- Shadow copy the storage data to a separated branch.
- Describe your development branch and its purpose in markdown.
- Do rebase to propagate the latest changes from production to the selected development branch to stay up-to-date.
- Show all changes against production one-by-one in detail.
- Merge selected or all changes to production.
Why did we build it?
Before development branches, you'd have to manually copy your transformation in your production project, change it and try to run it while affecting your buckets in storage, relying on manual checks that you or someone from your team took necessary actions to prevent breaking your production project and pipeline. Now, you simply create a development branch and we take care of everything for you.
With two clicks, you can now create a "branch" where you test all your code on production data. You tune what you need, wait for the result and then merge it to production. Until you merge it, you can be sure that your changes will not affect the production environment. Without additional work, you have access to data that is generated in real time in the production environment. We even solve how to test the results without overwriting the production data—when it tries to write to the production data, we automatically virtualize DWH and produce a shadow copy of your data on-the-fly, where you can check your results.
What makes development branches great for development?
- It’s easy to use. Development branches can be created within a single click and mimic the known behavior of branches in GIT.
- It’s relevant. You have full control over the changes and merging them back to production projects.
- It’s safe. The created development branch doesn't interfere with your production project in any way offering you a safe environment for any experiment.
- It’s time-saving. No more config or data copying to local or between projects.