IOMETE With Wasabi

IOMETE is a self-hosted data lakehouse platform, built on Apache Iceberg and Apache Spark, that runs SQL analytics, data engineering and AI/ML workloads on your own infrastructure. IOMETE is validated for use with Wasabi. This article outlines how IOMETE connects to a Wasabi bucket by creating a Spark catalog whose tables are stored in Wasabi, then shows how to create, query and modify an Iceberg table in it.

Reference Architecture

Users, BI tools, and dbt reach IOMETE through the IOMETE Gateway: the browser over HTTPS for the IOMETE Console (Admin Portal and Domain Portal), and SQL clients over JDBC/ODBC or Arrow Flight to the Spark compute clusters. Spark writes Iceberg data files to Wasabi. The Iceberg REST catalog writes table metadata files to Wasabi and keeps each table’s current metadata location in PostgreSQL. All storage traffic uses the S3 API over HTTPS to the bucket’s regional service URL.

Requirements

• IOMETE 3.19.1 or later (with access to the IOMETE Admin Portal)

• A running compute cluster in the IOMETE domain where you will use the catalog

• An active Wasabi Hot Cloud Storage account

  • Wasabi bucket (see Creating a Bucket)

  • Access & Secret Keys (see Creating a New Access Key)

Configuring Wasabi

1. Log in to the Wasabi Console.

2. Create a Wasabi bucket in the desired region. Do not enable versioning and object lock.

3. Create a policy with the below permissions. For more information, review Creating a Policy. Change YOUR_WASABI_BUCKET_NAME to your own Wasabi bucket name.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "s3:ListBucket",
           "s3:GetBucketLocation",
           "s3:ListBucketMultipartUploads"
         ],
         "Resource": "arn:aws:s3:::YOUR_WASABI_BUCKET_NAME"
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:GetObject",
           "s3:PutObject",
           "s3:DeleteObject",
           "s3:AbortMultipartUpload",
           "s3:ListMultipartUploadParts"
         ],
         "Resource": "arn:aws:s3:::YOUR_WASABI_BUCKET_NAME/*"
       }
     ]
   }

4. Create a user with programmatic access (create an API key) and apply the policy created above to the user. For more details, refer to Creating a User Account and Access Key.

Configuring IOMETE

Creating a Spark catalog

1. In the IOMETE Admin Portal, go to Data Governance.

2. Select Spark Catalogs.

3. Click New Spark catalog.

   

4. On the Managed Catalogs page, select the Iceberg Rest Catalog card at the top.

   Do not pick the card with the same name under External Catalogs, which connects to an existing external catalog.

   

5. Provide the following information.

   • Name—For example, wasabi_catalog.

   • Warehouse—Provide the Wasabi bucket URI, optionally with a prefix, for example, s3://iomete-lakehouse or s3://iomete-lakehouse/warehouse. With a prefix, the tables in Verifying the data in Wasabi will be under warehouse/wasabi_test/ instead of wasabi_test/.

   • Enable custom credentials—Click the checkbox to enable.

   • Endpoint—The Wasabi service URL for the bucket's region, for example https://s3.eu-west-1.wasabisys.com.

   • Access key—Wasabi access key created in the earlier section.

   • Secret key—Wasabi secret key created in the earlier section.

6. Click Test connection. Verify that the List, Write, Read and Delete checks all pass.

7. Click Create.

   

   This configuration example uses Wasabi’s eu-west-1 storage region. To use another Wasabi storage region, use the appropriate URL as described in Service URLs for Wasabi's Storage Regions.

8. The spark catalog created will be displayed below.

   

   For more information, refer to IOMETE documentation Iceberg REST Catalog.

Making the Catalog Available to a Domain

1. In the Admin Portal, go to Domains and open the domain that will use the catalog (for example, analytics-team).

2. Open the Catalogs tab, click Add Catalog and select Wasabi_catalog.

   

Granting Data Access to the Catalog

IOMETE controls access to databases, tables, and columns with data access policies. (For more information, refer to Data Access Policy.) Create a policy so the right users and groups can work with tables in wasabi_catalog.

1. In the Admin Portal, under Data Governance, select Data Security.

2. Select Access.

3. Click New access policy.

   

4. Provide a policy name (for example wasabi_catalog_access) and provide the configuration details below.

   • Resources—Set the following parameters.

     • Catalog—Provide the name of the catalog created earlier (in our example, it is wasabi_catalog)

     • Databases—*

     • Tables—*

     • Columns—*

   • In the Conditions sections, under Allow, select the groups or users that need access (for example, data-engineers) and choose their Permissions: All for full read and write access, or Select for read-only access.

   

   To give different teams different levels of access, narrow the databases or tables in the resources, or add more conditions to the policy. For Data Masking Policy and Data Row Level Filter, use the Masking and Row Level Filter policy types under the same Data Security page.

Creating and Querying an Iceberg Table

1. As a user assigned with the policy created earlier, switch to the domain in the IOMETE console and open the SQL Editor.

2. Select a running compute cluster. wasabi_catalog now appears in the Database Explorer.

3. Run the following statements.

   CREATE DATABASE wasabi_catalog.wasabi_test;
    
   CREATE TABLE wasabi_catalog.wasabi_test.test_table (
       id BIGINT,
       name STRING,
       created_at TIMESTAMP
   ) USING iceberg;
    
   INSERT INTO wasabi_catalog.wasabi_test.test_table VALUES
       (1, 'test-1', current_timestamp()),
       (2, 'test-2', current_timestamp()),
       (3, 'test-3', current_timestamp());
    
   UPDATE wasabi_catalog.wasabi_test.test_table SET name = 'updated' WHERE id = 1;
    
   DELETE FROM wasabi_catalog.wasabi_test.test_table WHERE id = 3;
    
   SELECT * FROM wasabi_catalog.wasabi_test.test_table ORDER BY id;
    
   SELECT committed_at, snapshot_id, operation
   FROM wasabi_catalog.wasabi_test.test_table.snapshots
   ORDER BY committed_at;

   The final SELECT returns the current rows, and the snapshots query lists one Iceberg snapshot per write.

   

   

   For more information on creating and querying a table, refer to IOMETE’s documentation about SQL Editor Overview.

Verifying the Data in Wasabi

In the Wasabi Console, open the bucket and browse to “wasabi_test/”. Each table has its own folder (for example, “test_table-<id>”) containing a “data/” folder with the Parquet files and a “metadata/” folder with the Iceberg metadata.

Configuring Wasabi as IOMETE’s Default Storage

To use Wasabi as the platform's primary storage instead (see Storage Configuration), set it in the IOMETE data plane Helm values (shown below) at install or upgrade time. In this mode, the bucket also holds IOMETE platform data, such as Spark event logs, SQL query results and SQL Editor worksheets, alongside table data.

Create the Secret in the IOMETE namespace before running the Helm install or upgrade.

kubectl create secret generic wasabi-storage-credentials \
  -n <iomete-namespace> --from-literal=secretKey='<wasabi-secret-key>'

storage:
  bucketName: "iomete-lakehouse"
  type: "s3_compatible"
  s3CompatibleSettings:
    endpoint: "https://s3.eu-west-1.wasabisys.com"
    accessKey: "<access-key>"
    secretKeySecret:
      name: wasabi-storage-credentials
      key: secretKey

Replace the bucketName and endpoint parameters with your bucket name and the region URL of the bucket.

Considerations

• The endpoint must match the region of the bucket. Using the wrong service URL allows reads but rejects writes and deletes.

• Leave versioning and Object Lock disabled on buckets that hold IOMETE tables. IOMETE stores tables in the Apache Iceberg format, and Iceberg table maintenance (snapshot expiration, compaction and orphan file removal) relies on permanently deleting files it no longer needs. With versioning or Object Lock enabled, those deletes only add delete markers, so old file versions stay in the bucket and storage keeps growing.

• Wasabi bills objects deleted before the minimum storage duration (90 days by default) as Timed Deleted Storage. Iceberg compaction and snapshot expiration regularly replace and delete files, so take this into account when scheduling table maintenance (see Table Maintenance Overview).