Build a lakehouse
Collect database, file and API data into Iceberg tables for historical analytics and reuse.
Enterprise data platform
Open Lakehouse
From ingestion and storage to analytics across distributed data. A deployment and operations package combining Argus Flow, Apache Airflow 3, Starburst Enterprise and MinIO AIStor.
Modern Lakehouse — manage data with an open table format
Combine scalable object storage with analytical table management. Separate storage from compute and use Apache Iceberg to manage schemas, snapshots and data changes.
- 1.Prepare data
- Collect DB, file and API data through batch or CDC pipelines, validate and transform it, and publish through a supported table write path.
- 2.Manage open tables
- Iceberg tracks schemas, partition layouts, snapshots and data-file references. Parquet and table metadata files reside in the storage layer.
- 3.Separate compute and storage
- SQL and processing engines read the required files and produce results for BI, analytics and AI applications with authorized access.
- 4.Catalog and operations
- Discover tables and commit metadata changes through a REST catalog. Configure access, quality checks, monitoring and table maintenance together.
This is a conceptual architecture. SQL, batch and streaming are workload types; validate each engine’s read/write support and catalog compatibility. Size storage and compute for their respective requirements.
Read the official documentationFile format, table format, catalog and storage
Each layer has a distinct role in storing data and exposing a consistent table state.
| Concept | Responsibility |
|---|---|
| Parquet · file format | Stores records in columnar files. A file alone does not manage table-wide snapshots or commit state. |
| Apache Iceberg · table format | Manages a collection of data files as a table, with schema, partition layout, snapshots and file statistics. |
| REST catalog | Resolves table names to current metadata references and processes commits. Engines read data files from storage directly. |
| Object storage | Stores data and metadata files. MinIO AIStor provides the S3-compatible storage layer in this platform. |
In Open Lakehouse, Argus Flow and NiFi 2 collect and transform data, Airflow 3 coordinates tasks, Starburst performs SQL analysis and table processing, and MinIO AIStor stores files. The platform architecture below shows their connections.
How does Iceberg manage files as one table?
Follow an append to the orders table to distinguish file storage from a snapshot commit.
- 1.Resolve current state
- The catalog points to metadata M100, which resolves snapshot 100 and files A and B. Example amounts 120 and 80 total 200.
- 2.Prepare the append
- The writer prepares file C and metadata M101. Before commit, the current table still references M100 and queries do not include C.
- 3.Validate and commit atomically
- Validate the base metadata and update requirements, then switch the current reference to M101. On concurrent conflicts, refresh, revalidate and retry.
- 4.Read the chosen snapshot
- After commit, new queries can read snapshot 101 with A, B and C for a total of 250. Each query uses its chosen snapshot; retained earlier states remain queryable.
Uploading a file is not a table commit. Time travel requires retained snapshots and their referenced files. A table-level commit does not establish a distributed transaction across external databases.
Read the official documentationIceberg capabilities for analytics and operations
Connect data changes, historical queries and maintenance to business requirements.
| Capability | Practical value |
|---|---|
| Table-level transactions | Keep uncommitted changes out of current table queries and read data using the selected snapshot. |
| Time travel | Reproduce analysis using retained historical snapshots and compare states before and after changes. |
| Schema and partition evolution | Adapt supported column and partition definitions as data and query patterns change. This does not automatically reorganize every existing file. |
| File pruning | Use partition and column statistics to reduce files read. Results depend on layout, statistics and queries. |
| Data changes | Apply INSERT, UPDATE, DELETE and MERGE through supported engines. Design CDC capture, deduplication and change ordering separately. |
| Table maintenance | Compact small files, expire snapshots and remove unreferenced files while accounting for active jobs and historical retention. |
Check engine, connector, Iceberg table-version and catalog support. Validate concurrent engine use and write compatibility in the target deployment.
Starburst Iceberg reads, writes and maintenance documentationStore, connect and analyze together
Build history in your lakehouse and connect directly to operational data when needed.
Analyze across existing systems
Connect multiple sources through Starburst SQL and JOINs without requiring a complete copy first.
Deliver and operate
Data Dynamics combines architecture, installation, integration, validation, performance measurement and operating procedures.
One platform, two paths to your data
Combine ingestion into the lakehouse with direct, federated access to existing sources.
- 1.Ingest
- Argus Flow uses NiFi 2 processors and services to collect, validate and transform data.
- 2.Store
- Write files to MinIO AIStor, then commit Iceberg metadata to publish them in a table.
- 3.Connect & analyze
- Starburst connects Iceberg and external databases for SQL and cross-source JOINs.
- 4.Orchestrate
- Airflow 3 coordinates schedules, dependencies, retries and downstream tasks.
Iceberg is the table format and MinIO AIStor is the storage layer. The REST catalog is a reference configuration; implementation, versions, authentication and write paths are validated together.
Read the official documentationPlatform components and responsibilities
Connect the products through data flows and operational practices designed for your business.
| Component | Responsibility |
|---|---|
| Argus Flow · Apache NiFi 2 | Ingestion, CDC, transformation, delivery, queues and replay through processors and controller services. |
| Apache Airflow 3 | Schedules, dependencies and orchestration of downstream SQL, quality checks and table maintenance. |
| Starburst Enterprise | Data federation across databases, cross-source JOINs, Iceberg reads and writes, and distributed SQL. |
| MinIO AIStor | S3-compatible storage for raw objects, Parquet and Iceberg metadata, with commercial subscription and support. |
| Apache Iceberg · REST catalog | Snapshots and table metadata. Catalog implementation, authentication and write compatibility are validated during delivery. |
Data Federation: distributed data, one SQL query
Design connections across Oracle, Hive and Impala environments, plus many other data sources.
- 1.Connect sources
- Configure connector catalogs and authentication to access tables in multiple systems.
- 2.Write one query
- Reference tables from different catalogs within a single SQL statement.
- 3.Execute across sources
- Starburst queries sources, applies supported pushdown and joins the returned data.
- 4.Deliver results
- Provide integrated results to BI tools, SQL clients and applications.
Federation reduces the need for full central preloading. Query execution still transfers data, and source load and access permissions must be managed.
Read the official documentationConnect well beyond Oracle
Bring operational databases, existing analytics platforms, cloud warehouses and your lakehouse into a common SQL access path.
| Source category | Example sources |
|---|---|
| Relational databases | Oracle · PostgreSQL · MySQL · SQL Server · IBM Db2 · SAP HANA |
| Data warehouses | Teradata · Snowflake · Amazon Redshift · Google BigQuery |
| NoSQL & search | MongoDB · Cassandra · Elasticsearch |
| Existing big data platforms | Hive · Impala environments — integration depends on existing tables, storage and access methods |
| Platform lakehouse | Apache Iceberg tables stored on MinIO AIStor |
Features, authentication and licensing vary by connector. A dedicated, supported Impala connector is not assumed: data access and compatibility are validated for the existing environment. Generic JDBC is evaluated for PoC use with production support assessed separately.
Starburst connector supportArgus Flow ingestion and CDC, orchestrated by Airflow
See how processor-level data handling connects to the order of downstream tasks.
- 1.Capture changes
- CaptureChange processors collect database events; record readers and writers normalize the format.
- 2.Validate & upload
- Preserve keys, operations and source positions in Parquet on MinIO AIStor; route failures to a separate queue.
- 3.Commit to tables
- ExecuteTrinoStatement and its connection service submit Starburst SQL; changes become queryable after the Iceberg commit.
- 4.Apply & coordinate
- Design MERGE to handle at-least-once duplicates and deletes; Airflow coordinates quality checks, publication and maintenance.
The reference path combines Argus Flow S3 uploads with Starburst file import. add_files does not validate schemas: configure schema/partition checks, duplicate-import prevention and retention for registered files. Direct REST-catalog writes through NiFi PutIceberg are not assumed.
Read the official documentationMinIO AIStor storage and data protection
Enterprise storage delivered with a commercial subscription and technical support.
- 1.S3 storage
- Define paths and permissions for raw objects, Iceberg data files and metadata.
- 2.Access & encryption
- Configure TLS, service credentials, IAM policies and KMS integration.
- 3.Data protection
- Set distributed storage, versioning and replication policies, then verify backup and restore procedures.
- 4.Commercial support
- Include subscription, vendor support coordination and upgrade planning in the delivery scope.
Entitlements follow the contracted product; object retention and locking must align with Iceberg cleanup and snapshot expiration.
Read the official documentationRead what matters, process in parallel
Distinguish database pushdown from Iceberg scan optimization.
- 1.Source pushdown
- Run supported filters and aggregations in the remote database to reduce returned data.
- 2.Iceberg pruning
- Use partition and file metadata to narrow scans and select required columns.
- 3.Distributed processing
- Starburst workers execute scans, JOINs and aggregations in parallel.
- 4.Measure & improve
- Use statistics, EXPLAIN and query metrics to identify bottlenecks and tune real workloads.
Pushdown depends on the connector and query. MinIO AIStor does not execute SQL. Validate plans and measured performance against data distribution, statistics, network and concurrency.
Read the official documentationConnect pipeline operations with recovery
Observe pipelines, queries and storage; make recovery and table maintenance repeatable.
- 1.Pipeline operations
- Track queue pressure, ingestion lag, failed batches and checkpoints for safe replay.
- 2.Query & table health
- Review load and statistics; schedule small-file compaction and snapshot maintenance.
- 3.Recovery validation
- Verify restoration of catalog state, configuration and credentials as well as storage.
- 4.Delivery & support
- Agree acceptance criteria for correctness, ingestion lag, concurrent queries and recovery.
Apply retries to idempotent tasks. Snapshot expiration and orphan cleanup must respect active jobs and retention windows. Validate recovery objectives with restore exercises.
Read the official documentationAdopt and operate one platform package
Data flows, acceptance criteria and operating procedures accompany the deployment. Delivery scope and support terms are defined in the agreement.
PoC & architecture
Validate ingestion, federation and Iceberg integration with representative data and queries; define capacity, concurrency, security and recovery requirements.
Production implementation
Configure version combinations, templates, NiFi flows, Airflow DAGs, catalogs, authentication, dashboards, alerts and runbooks.
Operations & technical support
Coordinate incident response and vendor support, upgrade compatibility, performance reviews and backup/restore exercises.
This is an integrated deployment and operations package using product-specific consoles. MinIO AIStor and Starburst products, licenses and support scope are confirmed in the proposal and agreement.
Related solutions and partners
Explore component capabilities and extension options.