Running Workflows 10x Faster: How Pollination Achieved 56% ROI Working with Pipekit

‍Brief Summary

Challenged with hundreds of concurrent workflows and escalating costs, Pollination sought an expert Argo Workflows partner to navigate their business and technical challenges. They chose Pipekit to navigate them on their operational optimization journey. The result? 10x improvements to workflow execution speed, critical upstream open source bug fixes, and much-improved system reliability. This case study showcases the power of partnership, technical expertise, and a shared commitment to operational excellence, highlighting how Pollination turned its challenges into a 56% ROI.

‍

Customer Background

Developed by Ladybug Tools, Pollination is a platform dedicated to advancing environmental and ecological research through cutting-edge technologies. A key component powering Pollination is Argo Workflows, the open source container-native workflow engine for orchestrating complex workflows and tasks. As their operations expanded, so did the associated complexity and scale of their workflows, which led to longer workflow run times and higher maintenance costs.
‍

‍Results Snapshot

• Achieved an impressive 56% ROI, equivalent to half a full-time Argo platform engineer's salary, highlighting substantial financial gains.
• Realized up to a 10x improvement in workflow execution speed and reduced pod requirements by one-third, enhancing task execution and resource efficiency at Pollination.
• Enhanced system reliability with a 99.8% success rate for the previously unstable server, demonstrating increased resilience and capacity during high-usage periods.
• Accelerated deployment of upstream fixes from months to days by creating a fork of Argo Workflows, reinforcing Pollination's agility in responding to evolving challenges.
• Avoided the need to hire a dedicated Argo platform engineer, resulting in significant cost savings thanks to Pipekit's optimization efforts.
  

‍

Business and Technical Challenges

Enhancing the Efficiency and Reliability of Pollination Cloud Services

How do you refactor workflows to more efficiently accommodate the execution of small models on the Pollination Cloud Services platform? That was the first challenge. Ensuring a swift and seamless experience for Pollination’s users was imperative for Pollination’s business. Simultaneously, the second challenge revolved around providing reliability and greater capability for scaling.

At the core of these challenges was the optimization of short-lived task execution within Argo Workflows. The key question was how to engineer these workflows to accelerate task completion while minimizing resource utilization. This required a deep understanding of Argo Workflows’ intricacies and a creative approach that delivered greater workflow efficiency.

In the face of these challenges, Pollination recognized the importance of collaborating with a partner who possessed both technical expertise and a collaborative problem-solving approach. They found Pipekit could deliver what they needed. 

‍

Customer Objectives

To help the Pollination team reach its identified goals, we (Pipekit) planned a comprehensive scope of services designed to address the presenting challenges. Our engagement included:

• Argo Workflows performance benchmarking — Pollination’s workflow definitions and configs were meticulously assessed with a focus on performance benchmarking and reliability. We conducted a detailed analysis of execution times, resource consumption, and bottlenecks to make it possible to target optimizations.
• Argo Workflows refactoring and optimization — Our two teams worked on a comprehensive refactoring, conducting a granular evaluation of Pollination’s workflow structures. To optimize workflows, we fine-tuned the orchestration process by grouping tasks efficiently to enhance execution speed and resource utilization.
• Open source bug resolution and feature development — Our Argo Workflows expertise allowed for quick diagnosis and resolution of open source bugs directly impacting Pollination’s platform reliability. After resolving these issues, the same fixes were introduced to the open source project as new features to enhance the experience for the community.
• Maintenance of a production-grade fork of Argo Workflows — Recognizing the importance of operational stability and agility, a robust, production-grade fork of Argo Workflows was created to meet Pollination's specific needs. This enhanced version enabled Pollination to expedite bug fixes and performance improvements.
  ‍

Approach

Our work began with an in-depth analysis of Pollination’s Kubernetes stack and Argo Workflows setup, aiming to identify the most significant inefficiencies. This analysis revealed key bottlenecks and inefficiencies slowing workflow execution. Using Argo Workflows’ containerSet feature, we managed to improve execution speed by 10x and then refactored Pollination’s workflows to achieve these gains in resource efficiency.

Next, Pollination provided several issues they faced within the upstream Argo Workflows codebase. Pollination identified a user experience (UX) challenge with archived workflows. One week after assessing the issue, the Pipekit team scoped and developed a solution and got a PR merged upstream. Similarly, a Google Cloud Storage integration issue that hampered workflow execution was identified and resolved. The solution was merged upstream and a patch release was scheduled within two weeks.

Enhancing workflow execution meant stability and reliability were paramount for Pollination. Their team discovered a bug that caused workflows to hang after their engineers executed stop and termination commands. We went to work to pinpoint the root cause and developed a solution that was merged upstream. To ensure the fix was adopted as soon as possible, we created a production-grade fork of Argo Workflows for Pollination and kept it meticulously maintained. This approach allowed Pollination to work with precise control over the orchestrator’s functionality without compromising on speed and agility.
‍

Outcomes

There was a great deal of collaboration between our two teams from the beginning through to the end of this process. After conducting a thorough analysis and developing quick and effective solutions, our optimization of Argo Workflows led to significant cost reductions, and increased operational efficiency.

Through our efforts, we were able to optimize Pollination’s use of Argo Workflows, resulting in impressive outcomes:

• Generated impressive ROI of 56% — The partnership with Pipekit resulted in an impressive ROI of 56%. These savings equated to half a full-time Argo platform engineer, highlighting the tangible financial benefits of optimization.
• Improved workflow execution by up to 10x — Pollination achieved an astounding up to 10x improvement in workflow execution speed. Simultaneously, the number of pods required for each workflow was dramatically reduced by one-third. These marked enhancements not only boosted task execution but also streamlined resource utilization, reinforcing Pollination's capability to handle diverse workloads with ease.
• Improved system reliability by 99.8% — The introduction of ContainerSets had a significant impact. The server that previously experienced stability challenges on high-usage days now experienced a surging success rate of up to 99.8%. This improvement in stability underscored the resilience of the system and its capacity to handle increased loads with greater confidence.
• Accelerated deployment of upstream fixes from months to days — Leveraging Pipekit's fork of Argo Workflows, Pollination slashed the deployment timeline for upstream fixes from months to mere days. This agile approach fortified Pollination's ability to swiftly adapt to evolving requirements and challenges, reinforcing the team’s overall agility.
• Avoided additional costs of dedicated Argo platform engineer — Due in large part to Pipekit’s optimization endeavors, Pollination ultimately avoided hiring a dedicated Argo platform engineer. This translated into substantial cost savings.

‍

Ladybug Tools CEO & Co-founder, Mostapha Sadeghipour Roudsari, had this to say about working with Pipekit,
"It feels great working with Pipekit. The team is capable and honest. When we run into issues, they never pretend things are great when they aren't. They identify the problem, make a plan, and then get it done. I can rely on the Pipekit team's expertise just like ours."

‍

Conclusion

Pollination was dedicated to finding ways to improve their performance. They sought deep Argo experts who could extend their technical capabilities through highly collaborative work and help guide them to quick performance and reliability gains. By refactoring workflow files and resolving key issues with Argo Workflows, the Pollination team realized marked platform efficiency and reliability gains. These improvements drove enhanced customer satisfaction and feedback and generated meaningful cost savings resulting from their significant ROI and reduced need to hire specialized engineering talent.