📦
A single Kubernetes CRD abstraction provisions and lifecycle-manages storage, databases, search, streaming, and analytics engines.
- One `DataResource` custom resource covers 17 documented types — `pvc/block`, `pvc/file`, `object`, `nfs`, `iscsi`, `postgres`, `keyvalue`, `search`, `kafka`, `clickhouse`, `warehouse/trino`, `lakehouse/iceberg`, `timeseries`, `vector`, `mariadb`, `mysql`, and `ferretdb`.
- The k8s-controller watches every DataResource and reconciles it against the matching upstream operator (CloudNativePG, Valkey Operator, OpenSearch Operator, Strimzi, or a cloud SDK for external mode).
- Status is reflected back onto `status.phase`, `status.conditions`, and `status.endpoints` so consumers can watch readiness the same way for every type.
- Three origination modes — managed, external, and imported — are selected per-resource via `spec.origination`.
💾
Ceph RBD block volumes, CephFS shared filesystems, and S3-compatible object buckets via Rook-Ceph, plus NFS and iSCSI bridging.
- `pvc/block` (Ceph RBD, ReadWriteOnce, up to 20K IOPS per volume) for databases, VMs, and single-pod workloads.
- `pvc/file` and `filesystem` (CephFS) for ReadWriteOnce and ReadWriteMany shared-filesystem access respectively.
- `object` (Ceph RGW) for S3-compatible buckets, up to 10K req/sec per bucket.
- `nfs` and `iscsi` bridge non-Kubernetes systems to Nest storage; both require an explicit client/initiator allow-list annotation or the resource fails to provision rather than opening access broadly.
- iSCSI targets get CHAP authentication auto-provisioned on first reconcile — a per-resource secret stored in the tenant namespace and owned by the DataResource, so it is removed with it.
🐘
PostgreSQL via CloudNativePG, and Valkey with primary + replica replication, both with automated failover.
- `postgres` provisions a CloudNativePG cluster with streaming replication; `replicas.write.count` defaults to 3 for managed HA.
- `keyvalue` (Valkey) with `replicas.write.count > 1` provisions a primary (pod-0) plus `replicaof` replicas; an init container on each replica configures replication and reads `masterauth` from the mounted auth secret, so the password never appears in the pod spec.
- This is primary/replica replication, not hash-slot sharding — every replica holds a full copy of the primary keyspace. `replicas.write.count <= 0` fails closed with a validation error.
- Applications without their own read/write splitting can instead connect through db-proxy for transparent routing.
🔎
OpenSearch (dedicated or shared multi-tenant), Kafka, ClickHouse, Trino, Iceberg, and vector databases.
- `search` (OpenSearch) supports both a dedicated single-tenant cluster and a shared multi-tenant `SearchPool`, with index-prefix isolation via OpenSearch security roles.
- `kafka` for event streaming, with tenant isolation via topic ACLs and consumer-group prefixes.
- `clickhouse` for columnar OLAP analytics, `warehouse/trino` for distributed SQL across sources, and `lakehouse/iceberg` (with its own REST catalog service) for open lakehouse tables with ACID transactions and time travel.
- `vector` (Milvus / OpenSearch Vector) for embedding similarity search, and `timeseries` (VictoriaMetrics) for metrics and observability data.
☁️
GA AWS EBS/S3, Azure Managed Disk/Blob, and GCP Persistent Disk/GCS block and object provisioning, alongside on-cluster managed mode.
- AWS, Azure, and GCP block and object provisioners are GA: full create, delete, and status-polling support, each with encryption at rest (KMS, Key Vault, or CMEK depending on provider).
- Resize/expand of an already-provisioned external volume or bucket is not implemented — a `size` change on an existing external resource is not applied. Scale the underlying cloud resource directly if needed.
- DigitalOcean, Linode, and generic S3-compatible providers are also supported; Vultr and Cloudflare R2 are accepted by the API but the controller provisioner is not yet implemented, so create requests error until it ships.
- DataResource lifecycle, tenant isolation, quota, and audit apply consistently across every external provider.
🏢
Every DataResource is scoped to a tenant, enforced at the API layer via JWT claims, with automatic per-tenant network isolation.
- `nest-api` validates the JWT `tenant` claim before any scope check and rejects requests without a valid tenant claim or where the claim does not match the requested `tenant_id`.
- On the first DataResource provisioned into a tenant namespace, the controller creates a `nest-default-deny` NetworkPolicy scoping that namespace to same-namespace ingress/egress, DNS to `kube-system`, and the gateway services on their provisioning ports. Creation is idempotent and best-effort.
- Shared resources add data-layer isolation on top: index-prefix isolation for a shared `SearchPool`, and per-tenant database accounts (with optional row-level security) for shared databases.
- Quota enforcement runs alongside isolation: the free tier caps a tenant at 5 DataResources, and the API returns a clear upgrade error once the limit is reached.
🔀
Managed (Nest provisions on-cluster), external (cloud-native AWS/Azure/GCP resources), or imported (register and observe existing resources).
- Managed: Nest provisions and fully lifecycle-manages the resource on Rook-Ceph, CloudNativePG, Valkey, OpenSearch, and Strimzi. Only mode with Data Protection Policies, DarkDrive-aware scheduling, CSI integration, and Eggs.
- External: Nest calls cloud-provider APIs directly for create/delete/status; there is no on-cluster provisioning path, so CSI, the injector webhook, and DarkDrive scheduling are not involved.
- Imported: Nest registers and health-probes a pre-existing resource without provisioning it, with optional managed credential rotation and failover.
- Tenant isolation, quota, audit logging, RBAC, and health probing are supported consistently across all three modes.
- Cost tracking is available in managed mode and in external mode (via each cloud provider’s cost APIs); it is unavailable for imported resources, since Nest has no cloud credentials scoped to a resource it did not provision.
🛡️
Automated snapshots, Velero backups, point-in-time recovery, and restore verification for managed resources.
- Local snapshots via Ceph VolumeSnapshot, on a cron schedule with configurable hourly/daily/weekly/monthly/yearly retention counts.
- Remote Velero backups to an object DataResource, with optional cross-region replication (configurable lag budget) and KMS encryption at rest.
- Point-in-time recovery with a configurable recovery window, for database engines with transaction-log support.
- Scheduled restore-verification test-restores to a scratch namespace catch backup corruption before it becomes an incident.
- Data protection is available in managed mode only — external and imported resources use the cloud provider’s or the source system’s own backup tooling.
🔁
db-proxy transparently splits reads and writes, load-balances replicas, and caches query results — built in-repo, replacing an external proxy previously used for this.
- Transparent read/write splitting: writes route to the primary, reads to replicas, even for clients with no split logic of their own. Session state (`SET`/`PREPARE`/`DECLARE`/temp tables/open transactions) pins subsequent queries to the primary per connection.
- Reads are load-balanced round-robin across all healthy replicas via an atomic counter that skips unhealthy ones.
- Security/query filtering: SQL-injection heuristics plus per-tenant blocked/allowed resource enforcement on the live path, hot-reloadable.
- High availability via blue/green primary switching and optional multi-write; a tenant-scoped, TTL’d, table-tag-invalidated Redis cache sits in front of query results.
- Proxies MySQL, PostgreSQL, and Redis wire protocols; configuration (routes, security policy, cache policy) is dynamic, read from a Redis config bus and written by the manager service.
🗄️
A four-tier storage architecture with automated age-based lifecycle transitions, backup integration, and cross-region replication.
- Tiers, lowest to highest cost per GB: `nvme-hot` (active data, <5ms latency, highest cost) → `ssd-warm` (recent backups, 30–365 days, medium cost) → `sata-bulk` (compliance archives, 1–7 years, low cost) → `sata-cold` (long-term retention, lowest cost).
- Objects are created in `nvme-hot` by default; lifecycle policies transition them to colder tiers automatically as they age, based on per-DataResource lifecycle annotations.
- Velero uses a Nest object DataResource as a BackupStorageLocation target, and cross-region replication provides multi-region copies for disaster recovery.
🥚
A named, versioned bundle of one or more DataResources and processors — the unit of composition in Nest.
- Group related resources (for example, a database, a Kafka stream, and a search index) into one logical unit, versioned and released together.
- Deploy an entire bundle as one manifest, share the same egg definition across multiple tenants, and discover its resources by label (`egg=<name>`).
- Eggs are available in managed mode only.
📋
Structured audit logging, policy evaluation, and lineage tracking across every DataResource.
- audit-service logs structured `AuditEvent`s for compliance-relevant event trails across Nest’s services, including a `nest.auth.scope_denied` event on every authorization denial.
- policy-engine evaluates `PolicyRule`s covering access, retention, data-residency, and DLP (data-loss-prevention) concerns against DataResources — distinct from the network-layer default-deny NetworkPolicy.
- schema-service and data-indexer introspect, cache, and classify DataResource schema and contents; lineage-collector records an OpenLineage-style trail of how data flows between DataResources.
