The story behind Stegflow¶

Photo by Dave Ruske, CC BY 4.0.

I have been building workflow engines for over twenty-five years: first for unified messaging systems (fax, email, telex), then for email archiving, and for the last decade around data warehouse automation. Orchestrating long-running, fault-tolerant processes is a problem I keep coming back to.

Looking for a workflow engine¶

In 2018, the team I worked with was building a data warehouse automation product based on the Data Vault approach. Automating a datavault based data warehouse is, at heart, an ETL problem¹: you need to generate and orchestrate data-loading workflows. So we went looking for a workflow engine.

We weighed several options at the time (Apache Airflow, Node-RED, Microsoft BizTalk, AWS Step Functions, to name a few). One thing guided the search from the start: I wanted to keep the pull-based worker architecture I had relied on in earlier systems, where workers pull work rather than having it pushed to them. Scaling then becomes simple: you start or stop workers and the load spreads itself, instead of the engine having to balance it. Step Functions won me over on that count. Besides the usual push model, where the engine invokes a service (a Lambda, for example), it also offers exactly that poll model, through its Activities and Workers. Long-running tasks are handled cleanly through heartbeats. Just as important, its language, the Amazon States Language, is specified in the open under an Apache license. I first used it in earnest on a data warehouse project for a client running Amazon Redshift and Step Functions.

One catch: it only runs in the cloud¶

There was one catch: Step Functions runs only in the cloud, at a cost that varies with usage. I could not build a product that depended on a paid online service, especially since most of our clients keep their data warehousing infrastructure disconnected from the internet. That constraint set the direction for everything that followed, from that first engine to Stegflow today: it had to run anywhere, on your own servers, in a private cloud...

Staying compatible with Step Functions matters for clients already on AWS: several of ours were using Redshift, so they could stay in an environment they already knew and supported, and the product would integrate smoothly.

A first implementation¶

The specification was open, so I wrote a first C# implementation of the States Language, based on a Java implementation that has since disappeared. It only generated workflow JSON. We then built a first Step Functions engine on top of it. That component is still part of the product today; it has been running in production for years and has kept evolving since, with each change to the specification ported as it came, so the product is compatible with AWS Step Functions for clients who want it, and self-contained for the others.

Starting over, in the open¶

Since 2023 I no longer work there full time and late in 2024 the specification changed significantly: JSONata, variables, and new API operations for versioning and aliases. Retrofitting the existing code would have been too invasive, and I wanted a free implementation of the States Language, one anybody could run and improve. So I started fresh, on a Domain-Driven Design architecture (an approach I picked up during the COVID period and wanted to try on a project like this). This new engine, Stegflow, goes further than that first version: it tracks the latest specification and adds multi-tenancy and fine-grained access control.

By late 2025 the engine was nearly complete, and I began the console in ASP.NET and htmx. I wanted to come back to ASP.NET with a simple, lightweight way to build dynamic pages, and htmx fits it well; React and Monaco are used only for the graph view and the JSON editor. In July 2026, I published the first Stegflow beta.

Where it is going¶

My aim is to take Stegflow wherever it proves useful, self-hosted and open. I will be using it myself for years to come, and it will keep evolving: many ideas are already in the pipeline (see the open issues on Codeberg). Contributions are welcome. It stays free software under the AGPL-3.0: broadly usable, while ensuring that improvements flow back to everyone.

Vincent Daron, creator of Stegflow. Reach me at @vdaron.