Hybrid‑Cloud Data Lake Migration for a National Grid Operator

# Challenge

The client operated a 12‑year‑old Hadoop cluster on a private data centre in the Midwest. Daily, 3.4 billion sensor readings from substations, SCADA feeds and market data were landed as Avro files in HDFS. The ingestion pipeline, built on MapReduce jobs scheduled via Oozie, introduced an average end‑to‑end latency of 12 seconds per record batch. During a high‑frequency market event in March 2025, the delay caused the operator to miss a price‑spike arbitrage window, resulting in a $4.3 M revenue loss.

Regulatory bodies required all raw telemetry to remain within national borders, but the operator also needed elastic compute for seasonal load spikes. Their on‑prem infrastructure was at 85 % capacity, with storage utilisation at 78 % and a projected OPEX increase of 14 % per year. The existing stack offered limited observability; failure modes such as HDFS block corruption were only discovered after downstream analytics failed.

The mandate was clear: migrate to a hybrid environment that kept raw data on‑prem, leveraged cloud burst capacity for heavy analytics, cut latency, and provided transparent audit trails.

# Approach

PPIL assembled a cross‑functional team of data engineers, cloud architects and compliance officers. The solution blueprint comprised four pillars:

1. Data Sovereignty Layer , Deploy a MinIO object store behind the perimeter firewall, exposing S3‑compatible APIs to on‑prem workloads. MinIO’s erasure‑coding kept data durable while satisfying residency rules.
2. Lake Format Modernisation , Replace raw HDFS directories with Apache Iceberg tables, enabling atomic metadata operations and time‑travel queries without full rewrites.
3. Kubernetes‑Native Orchestration , Run ingestion services as stateless containers on a Rancher‑managed Kubernetes 1.31 cluster, auto‑scaling with the cluster‑autoscaler based on CPU and memory thresholds.
4. Observability & Security Stack , Instrument services with OpenTelemetry, forward traces to Grafana Cloud, and enforce zero‑trust access via HashiCorp Vault‑integrated service accounts.

A phased migration plan reduced risk:

• Phase 1 (Proof‑of‑Concept) , Move a single substation feed (≈ 5 million records/day) to the new stack.
• Phase 2 (Scale‑Up) , Incrementally onboard additional feeds, targeting 50 % of total volume within three months.
• Phase 3 (Full Cut‑Over) , Decommission legacy Hadoop jobs, redirect all ingestion to the Kubernetes pipeline, and retire the old HDFS cluster.

# Implementation

1. Infrastructure Provisioning
Terraform modules defined both on‑prem VM pools (vSphere‑based) and the public‑cloud VPC (AWS us‑east‑2). The modules provisioned:

• 12 vCPU, 48 GB RAM nodes for MinIO (3‑node erasure set)
• 8 vCPU, 32 GB RAM worker nodes for Kubernetes (initially 10, autoscaled to 30)
• IAM roles linked to Vault for dynamic credential generation

2. Data Ingestion Service
A Go‑based microservice, StreamGate, subscribed to Apache Pulsar topics that mirrored the SCADA feed. The service performed:

// streamgate.go
package main

import (
    "context"
    "log"
    "github.com/apache/pulsar-client-go/pulsar"
    "github.com/minio/minio-go/v7"
)

func main() {
    client, err := pulsar.NewClient(pulsar.ClientOptions{
        URL: "pulsar://pulsar-gw.internal:6650",
    })
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()

    consumer, err := client.Subscribe(pulsar.ConsumerOptions{
        Topic:            "persistent://grid/telemetry",
        SubscriptionName: "streamgate-sub",
        Type:             pulsar.Shared,
    })
    if err != nil {
        log.Fatal(err)
    }
    defer consumer.Close()

    minioClient, _ := minio.New("minio.internal:9000", &minio.Options{
        Creds:  credentials.NewStaticV4("accessKey", "secretKey", ""),
        Secure: false,
    })

    for {
        msg, err := consumer.Receive(context.Background())
        if err != nil {
            log.Println(err)
            continue
        }
        // Write raw payload to MinIO as Iceberg data file
        _, err = minioClient.PutObject(context.Background(), "grid-lake", fmt.Sprintf("%s.avro", msg.ID()), bytes.NewReader(msg.Payload()), int64(len(msg.Payload())), minio.PutObjectOptions{ContentType: "application/octet-stream"})
        if err != nil {
            log.Println(err)
        }
        consumer.Ack(msg)
    }
}

The container image (golang:1.22‑alpine) was built via CI/CD pipelines in GitHub Actions, scanned with Trivy, and pushed to the private registry. Helm charts defined the deployment, health probes, and resource limits.

3. Iceberg Table Management
Using Spark 3.5 on the cloud burst cluster, nightly jobs compacted incoming Avro files into columnar Parquet, registering them with Iceberg via the Spark catalog. The catalog was stored in PostgreSQL (managed by RDS) to keep metadata centrally accessible.

4. Observability Integration
OpenTelemetry SDKs in Go and Java emitted spans to a Grafana Loki endpoint. Alerts triggered on ingestion latency > 2 seconds or MinIO node health degradation. All logs were encrypted at rest with AWS KMS.

5. Security Controls
Vault’s PKI engine issued short‑lived TLS certificates for each service instance. Policies limited MinIO bucket access to the specific substation prefix, preventing cross‑feed data leakage.

# Results

The pilot phase demonstrated a 1.3‑second latency for the test feed, confirming that the new stack could meet sub‑second requirements for market‑driven decisions. Scaling to 50 % of feeds reduced the on‑prem storage footprint by 30 % thanks to MinIO’s erasure coding and tiering to S3‑compatible cold storage. The operator reported a $1.2 M reduction in OPEX during the first year after decommissioning half of the Hadoop nodes.

Compliance audits showed 100 % adherence to the national data‑residency rule, as all raw files never left the domestic MinIO cluster. The zero‑trust model eliminated the need for static service credentials, cutting credential‑rotation effort by 70 %.

# Lessons Learned

1. Start Small, Validate Early , The single‑feed proof‑of‑concept revealed a hidden bottleneck in Pulsar’s consumer configuration (default receive queue size). Adjusting the queue avoided back‑pressure without extra hardware.
2. Metadata Management Pays Off , Switching to Iceberg eliminated the need for costly HDFS block repairs. The atomic commit model prevented half‑written files from corrupting downstream analytics.
3. Observability Must Be End‑to‑End , Early alerts on MinIO node health prevented a cascade failure that would have taken hours to diagnose under the old stack.
4. Policy‑Driven Security Reduces Human Error , Vault’s dynamic secrets removed the practice of embedding static keys in container images, a common source of leaks in legacy environments.
5. Hybrid Architecture Requires Clear Boundaries , Defining “raw” versus “processed” data zones helped the compliance team certify that only the raw zone stayed on‑prem, while processed aggregates could be safely placed in the public cloud for large‑scale analytics.

The project also highlighted the importance of capacity planning for the Kubernetes autoscaler. Initial limits were set too conservatively, causing a brief spike in queue length during a market surge. Re‑tuning the target CPU utilization from 70 % to 55 % smoothed the scaling curve.

# PPIL Takeaway

By treating data sovereignty as an architectural boundary rather than a compliance checkbox, PPIL helped the grid operator turn a legacy bottleneck into a flexible, observable platform that delivers real‑time insight while respecting national regulations.