ElastiFlow Pipelines
A lightweight Go framework for building stateful, real-time data pipelines and reactive streaming apps with composable stages, windowing, and fan-out/in—pure Go.
- go
- stream-processing
- data-pipeline
- concurrency
- open-source
Highlights
- Engineered a production-ready MVP for building composable streaming/batch pipelines in pure Go.
- Provided a roadmap and led subsequent development to support stateful processing, windowing, and fan-out/in/broadcast patterns.
- Authored comprehensive GoDocs and examples to standardize adoption and usage across teams.
Screenshots
What it is
ElastiFlow Pipelines is a lightweight Go framework for building stateful, real-time data pipelines and reactive streaming applications.
It’s designed around Go’s native concurrency primitives (channels and goroutines) and a composable DataStream API. You can build pipelines as a series of stages (e.g., Map, Filter, Window) and scale throughput using familiar patterns like fan-out/fan-in, broadcast, and tee.
What I contributed
- Led development of the production-ready MVP, including core pipeline lifecycle, stage orchestration, and API ergonomics.
- Defined the roadmap and led subsequent development, expanding the framework to support real-world streaming needs like windowing and concurrency-friendly composition patterns.
- Authored the GoDocs and examples, making the API discoverable and adoption-friendly for engineers integrating the framework into production systems.
Outcome / impact
- Standardized a repeatable pattern for concurrent streaming processors in Go, reducing bespoke channel plumbing and sharp-edge concurrency code.
- Adopted internally at ElastiFlow as a reusable developer tool, then open sourced as a standalone module.
Tech (high-level)
Go · Generics · Channels/Goroutines · Context · Standard Library