Solutions · Open Source

Argus Flow

An open-source data pipeline distribution that gathers everything you need to run Apache NiFi 2.10.0 in production. Custom extension bundles (NARs), deployment packages (tar.gz, RPM, container images), a Kubernetes operator, and configuration/certificate/user-management tooling — consolidated from scattered repositories into a single monorepo so every artifact ships from one build pipeline at matching versions.

Apache License 2.0 · Open SourceApache NiFi 2.10.0 · Java 21GitHub Repository

Highlights

01

Extensions, distribution, and operator in one pipeline

NiFi extensions, the operator, and the certificate scripts used to live in separate repositories, which made versions drift apart. They are now a single monorepo: one make target produces NARs, tar.gz, RPM, and container images at matching versions.

02

12 bundles, 18 NARs, isolated dependencies

Extensions are split into domain bundles and each bundle is packaged as its own NAR, so third-party dependencies never collide across bundle boundaries. Deploy only what you need from CDC, lakehouse, Hive, Parquet, database, reporting, and flow analysis.

03

A NiFi 2.x Kubernetes operator without ZooKeeper

It uses NiFi 2.x native Raft coordination, so clusters run without ZooKeeper. NiFiCluster and NiFiFlow custom resources give declarative management, scale-down performs a safe Offload → Disconnect → Remove sequence, and cert-manager integration issues TLS certificates automatically.

04

Configuration, certificates, and accounts handled by tooling

Instead of hand-editing a 200-line nifi.properties and its XML files, an interactive wizard collects the values and shows a diff before saving. The --check diagnostic catches certificate SAN mismatches, upcoming expiry, and commented-out login providers — preventing failures such as Invalid SNI.

What's in the distribution

Extension bundles, deployment packages, the Kubernetes operator, and the operations tooling are built together from one repository, while each directory can still be built on its own.

NiFi Extensions
Maven · Java 21
12 bundles → 18 NARs with per-bundle dependency isolation
22 processors · 7 controller services
5 reporting tasks · 6 flow analysis rules
Database-backed authentication/authorization providers
Service contracts split into services-api bundles
Unified io.datadynamics.nifi.* packages
Distribution
tar.gz · RPM · container
Repackaging of the official NiFi binaries
nfpm-based RPM with systemd units
Container image with NARs, based on apache/nifi
Operations scripts bundled under bin/
Upstream pinned at build time, never vendored
make dist · make rpm · make docker-image
Kubernetes Operator
Python · kopf
NiFiCluster and NiFiFlow custom resources
Reconciles StatefulSet, Headless Service, ConfigMap
Safe offboarding on scale up/down
Automatic TLS issuance and renewal via cert-manager
Health-check-driven node reconnection
Prometheus metrics · Helm chart
Ops Tooling
argus-config · argus-ssl · argus-user
Interactive TUI for conf/, shipped as a zipapp
First-run wizard — address → TLS → login
--check diagnostics with CI-friendly exit codes
Non-interactive --set and --recipe automation
TLS certificate generation scripts
User-management CLI for database authentication
Technology Stack
Apache NiFi 2.10.0Java 21Maven 3.9.16+ (wrapper)Python 3.12+kopfHelmnfpmDocker / Podmancert-managerDebezium 3.2.2Apache IcebergDelta KernelApache KuduHive3 · ORCParquet · AvroPostgreSQL

Core Capabilities

From CDC, lakehouse, Hive, and file-format extensions to operational governance rules, database authentication, distribution packaging, the Kubernetes operator, and configuration/TLS automation — twelve pillars of NiFi operations in a single distribution.

Four CDC sources

Built on Debezium Embedded Engine 3.2.2, emitting change events from major relational databases as JSON FlowFiles.

CaptureChangeMySQL — binlog CDC
CaptureChangePostgreSQL — logical replication (pgoutput/decoderbufs)
CaptureChangeOracle — LogMiner (CDB/PDB)
CaptureChangeSQLServer — SQL Server CDC
PrimaryNodeOnly · at-least-once delivery

Lakehouse sinks

Load records directly into open table formats and columnar stores.

PutIceberg — writes to Apache Iceberg tables
PutDeltaLake — path-based sink on Delta Kernel
PutKudu — writes to Apache Kudu tables
HiveCatalogService · HadoopCatalogService

Database processors

Broader relational integration, from querying to bulk loading.

ExecuteSQL · ExecuteSQLRecord
ExecuteFastSQL — high-throughput streaming of large result sets
PutDatabaseRecord — INSERT/UPSERT
BulkOracleInsertProcessor — Oracle array-binding bulk insert

Hive / ORC

Hive3 integration and ORC output in a dedicated bundle.

SelectHive3QL · PutHive3QL · PutHive3Streaming
UpdateHive3Table — column and partition schema sync
TriggerHiveMetaStoreEvent
PutORC · Hive3ConnectionPool · VendorHive3ConnectionPool

File formats & record serialization

Readers and writers that let you control timestamp formatting.

PutParquet · MergeParquet
CSVReader with configurable timestamp formats
TimestampFormatAvroRecordSetWriter
TimestampFormatParquetRecordSetWriter

General-purpose processors

Recurring field requirements folded into the standard bundle.

PrePostExecuteStreamCommand — run pre/post commands alongside
MultilineCsvParser — CSV parsing with multiline fields
Bundle-level deployment installs only what you need

Five monitoring reporting tasks

Watch node resource state against thresholds and notify externally.

MonitorDiskUsageReportingTask — repository disk
MonitorMemoryUsageReportingTask — JVM heap
MonitorMemoryPoolReportingTask — individual memory pools
MonitorThreadReportingTask — thread state such as deadlocks
HttpNotificationReportingTask — webhook HTTP POST alerts

Six flow analysis rules

Operational governance rules that flag flow anti-patterns through static checks.

TimerThreadPoolCeilingRule — global thread count ceiling
ProcessorThreadShareLimitRule — single-processor thread share
ProcessorConcurrencyCapRule — concurrent task cap per type
ListingScheduleGuardRule — zero-second schedules on List processors
DeadEndFunnelRule — dead-end funnels causing backpressure
IcebergSinkMergeRule — small-file risk from a missing Merge

Database authentication & authorization

Providers built for environments where neither LDAP nor an external IdP is an option, so nobody has to hand-edit users.xml.

DbLoginIdentityProvider — user/password authentication from an RDB
DbUserGroupProvider — RDB-backed authorization
argus-user.sh user-management CLI
Disabled (commented out) by default, enabled only when needed
PostgreSQL driver bundled; MariaDB supplied separately

Distribution packaging

Repackages the official binaries into artifacts you can put straight into production.

tar.gz including extension NARs and operations scripts
RPM via nfpm with systemd units and sysconfig
Container image based on the official apache/nifi
Upstream pinned at build time rather than vendored

Kubernetes operator

Define NiFi 2.x clusters declaratively and automate scaling and updates.

NiFiCluster CR — replicas, resources, storage, JVM heap
NiFiFlow CR — GitOps deployment of Registry flows
Automatic node discovery via Headless Service DNS
Rolling restarts apply configuration without downtime
TLS through a cert-manager Issuer
Helm chart — CRDs, RBAC, Deployment

Configuration & TLS automation

Standardizes the configuration areas where things most often go wrong right after installation.

First-run setup carries the access address straight into the certificate SAN
--check diagnoses SAN mismatches, expiry, commented-out providers
Recipes: tls:generate · login:db · login:ldap · state:zookeeper
Preserves original comments and ordering, shows a diff before saving
Non-interactive mode prints certificate commands instead of running them

Editions

Use the entire open-source distribution freely as Community, or move to Enterprise when you need an operational SLA and dedicated technical support.

Community

Apache License 2.0 · Free

Use and operate the extension bundles, deployment packages, operator, and tooling without restriction.

Recommended

Enterprise

Enterprise customer support

Everything in Community plus SLA-backed dedicated support, deployment and migration services, and custom extension development.

Feature Comparison
Community
Enterprise
Core capabilities
12 NiFi extension bundles · 18 NARs
CDCLakehouseHive / ORCParquetDatabaseReportingFlow Analysis
Deployment packages (tar.gz · RPM · container image)
Kubernetes operator · Helm chart
Configuration TUI · diagnostics · TLS certificate tooling
Database-backed authentication and authorization
Support & services (Enterprise)
SLA-backed dedicated technical support
Priority hotfixes and security patches
Installation, deployment, and NiFi 1.x → 2.x migration
Training, onboarding, and pipeline architecture consulting
Custom processor development and roadmap priority
Air-gapped deployment and vendor JDBC driver integration
Cloudera Hive JDBCMariaDB Connector/J
Support channel
Support channel
GitHub Issues
Dedicated channel
Apache License 2.0 · Open Source

A NiFi production distribution released as open source

Argus Flow is published on GitHub under the Apache License 2.0. Extension bundle sources, distribution packaging, the Kubernetes operator, and the operations tooling are all open, so enterprises can audit the code, extend it for their own environment, and run it entirely on their own — even in air-gapped networks.

  • Apache 2.0 with no commercial-use restrictions
  • Upstream attribution and license headers verified in the build
  • Self-operated on-premises or air-gapped