Notes taken by Horeb S.
In this section, we’ll learn the foundations of workflow orchestration, its importance, and how Kestra fits into the orchestration landscape.
Table of contents
We can say that an orchestrator is like an orchestra where several different instruments need to come together in unison. In the case of an orchestrator, instead of instruments, we have different tasks, services, applications, and data pipelines that need to work together harmoniously. An orchestrator helps coordinate and manage these various components, ensuring they execute in the right order, handle dependencies, and maintain proper data flow between them. It provides a centralized way to control and monitor complex workflows that span multiple systems and processes.
Orchestration and automation are very similar. But, orchestration is effectively automation and then some, whereas automation is about executing individual tasks or processes automatically. Orchestration involves managing multiple automated tasks, handling their dependencies, scheduling, error handling, and ensuring they work together as a cohesive system. Think of automation as a single musician playing their part, while orchestration is the conductor ensuring all musicians play together harmoniously.
Let’s discuss now a few common use cases for orchestrators, so we cen understand when we need to use them.
Here, orchestrators are key for being able to allow extracts, transform and load tasks, and
manage data pipelines across different systems and environments.
They help ensure data consistency, handle dependencies between data processing steps, and provide monitoring capabilities for the entire data workflow. This is particularly important in modern data architectures where data needs to be processed from multiple sources and transformed for various downstream applications.
Orchestrators can be useful for being able to build, test and publish codes to different places at the same time and manage all of the different steps to make sure they happen in unison. They help automate the software delivery process by coordinating various stages such as code compilation, running automated tests, security scans, and deploying to different environments. This ensures consistency and reliability in the software development lifecycle while reducing manual intervention and potential errors.
