Why Pollination partnered with Pipekit to scale their Argo Workflows implementation.
Challenged with hundreds of concurrent workflows and escalating costs, Pollination sought an expert Argo Workflows partner to navigate their business and technical challenges. They chose Pipekit to navigate them on their operational optimization journey. The result? 10x improvements to workflow execution speed, critical upstream open source bug fixes, and much-improved system reliability. This case study showcases the power of partnership, technical expertise, and a shared commitment to operational excellence, highlighting how Pollination turned its challenges into a 56% ROI.
Developed by Ladybug Tools, Pollination is a platform dedicated to advancing environmental and ecological research through cutting-edge technologies. A key component powering Pollination is Argo Workflows, the open source container-native workflow engine for orchestrating complex workflows and tasks. As their operations expanded, so did the associated complexity and scale of their workflows, which led to longer workflow run times and higher maintenance costs.
Results Snapshot
How do you refactor workflows to more efficiently accommodate the execution of small models on the Pollination Cloud Services platform? That was the first challenge. Ensuring a swift and seamless experience for Pollination’s users was imperative for Pollination’s business. Simultaneously, the second challenge revolved around providing reliability and greater capability for scaling.
At the core of these challenges was the optimization of short-lived task execution within Argo Workflows. The key question was how to engineer these workflows to accelerate task completion while minimizing resource utilization. This required a deep understanding of Argo Workflows’ intricacies and a creative approach that delivered greater workflow efficiency.
In the face of these challenges, Pollination recognized the importance of collaborating with a partner who possessed both technical expertise and a collaborative problem-solving approach. They found Pipekit could deliver what they needed.
To help the Pollination team reach its identified goals, we (Pipekit) planned a comprehensive scope of services designed to address the presenting challenges. Our engagement included:
Our work began with an in-depth analysis of Pollination’s Kubernetes stack and Argo Workflows setup, aiming to identify the most significant inefficiencies. This analysis revealed key bottlenecks and inefficiencies slowing workflow execution. Using Argo Workflows’ containerSet feature, we managed to improve execution speed by 10x and then refactored Pollination’s workflows to achieve these gains in resource efficiency.
Next, Pollination provided several issues they faced within the upstream Argo Workflows codebase. Pollination identified a user experience (UX) challenge with archived workflows. One week after assessing the issue, the Pipekit team scoped and developed a solution and got a PR merged upstream. Similarly, a Google Cloud Storage integration issue that hampered workflow execution was identified and resolved. The solution was merged upstream and a patch release was scheduled within two weeks.
Enhancing workflow execution meant stability and reliability were paramount for Pollination. Their team discovered a bug that caused workflows to hang after their engineers executed stop and termination commands. We went to work to pinpoint the root cause and developed a solution that was merged upstream. To ensure the fix was adopted as soon as possible, we created a production-grade fork of Argo Workflows for Pollination and kept it meticulously maintained. This approach allowed Pollination to work with precise control over the orchestrator’s functionality without compromising on speed and agility.
There was a great deal of collaboration between our two teams from the beginning through to the end of this process. After conducting a thorough analysis and developing quick and effective solutions, our optimization of Argo Workflows led to significant cost reductions, and increased operational efficiency.
Through our efforts, we were able to optimize Pollination’s use of Argo Workflows, resulting in impressive outcomes:
Ladybug Tools CEO & Co-founder, Mostapha Sadeghipour Roudsari, had this to say about working with Pipekit,
"It feels great working with Pipekit. The team is capable and honest. When we run into issues, they never pretend things are great when they aren't. They identify the problem, make a plan, and then get it done. I can rely on the Pipekit team's expertise just like ours."
Pollination was dedicated to finding ways to improve their performance. They sought deep Argo experts who could extend their technical capabilities through highly collaborative work and help guide them to quick performance and reliability gains. By refactoring workflow files and resolving key issues with Argo Workflows, the Pollination team realized marked platform efficiency and reliability gains. These improvements drove enhanced customer satisfaction and feedback and generated meaningful cost savings resulting from their significant ROI and reduced need to hire specialized engineering talent.
If you’re interested in learning more about how we can support your team in getting the most out of Argo Workflows, get in touch!
Sign up here for a free consultation call, or visit our professional services page to learn more.