--- title: "SkyPilot With Wasabi" slug: "skypilot-with-wasabi" updated: 2026-10-06T15:53:12Z published: 2026-10-06T15:53:12Z canonical: "docs.wasabi.com/skypilot-with-wasabi" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.wasabi.com/llms.txt > Use this file to discover all available pages before exploring further. # SkyPilot With Wasabi SkyPilot is an open-source framework for running AI, machine learning, and general batch compute workloads across a wide range of infrastructure — including major public clouds, on-prem or self-hosted Kubernetes clusters, Slurm clusters, and existing machines. Rather than writing infrastructure-specific scripts for each provider, users describe a task once in a simple YAML file, and SkyPilot handles provisioning compute, syncing code and data, running setup and job commands, and tearing resources down afterward. This article documents the configuration required to run SkyPilot tasks that read data from and write results to Wasabi Hot Cloud Storage. ## Requirements The following software was used during testing of this configuration. - Operating System—This solution was tested with Ubuntu 26.04 Server edition. - Python—3.10. Installed automatically by uv if not already present. - uv—Tested with version 0.11.32. Astral's Python package/venv manager. Installed via the official install script. - SkyPilot—Tested with version 0.13.0. - Kubernetes distribution—MicroK8s. Tested with version v1.35.6. See “Known Issues — k3s” below before choosing a distribution. This solution was tested with Kubernetes, but other compute infrastructure such as AWS is also supported. - kubectl—Tested with v1.36.3. Installed via snap install kubectl --classic. A standalone binary is required; the microk8s kubectl alias is not sufficient for SkyPilot's internal use. - AWS CLI—Installed automatically inside the task pod during setup. Needed inside the remote pod to talk to Wasabi. - Active Wasabi account—See [Signing Up for Wasabi](https://docs.wasabi.com/docs/signing-up-for-wasabi) for instructions on how to sign up. - Wasabi bucket—See [Creating a Bucket](https://docs.wasabi.com/docs/working-with-buckets-and-objects#creating-a-bucket) for details. - [Wasabi Console](https://console.wasabisys.com/) access. ## Configuring Wasabi > Reference commands in this article use the placeholder YOUR_WASABI_BUCKET_NAME. Replace it with your own bucket name. Commands and output labeled “Example” are copied directly from the terminal during testing for this article, using a real bucket named mt-skypilot. 1. Log in to the [Wasabi Console](https://console.wasabisys.com). 2. Configure a policy for a SkyPilot user (that will be created in the next step) using the following policy. Replace YOUR_WASABI_BUCKET with the actual name of the bucket you want SkyPilot to use. See [Creating a Policy](https://docs.wasabi.com/docs/creating-a-policy) for details. ```json {  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": "s3:ListAllMyBuckets",      "Resource": "arn:aws:s3:::*"    },    {      "Effect": "Allow",      "Action": "s3:*",      "Resource": [        "arn:aws:s3:::YOUR_WASABI_BUCKET",        "arn:aws:s3:::YOUR_WASABI_BUCKET/*"      ]    }  ] } ``` 3. Create a skypilot user and attach the previously created policy to it. Use the following settings for the user. See [Creating a User](https://docs.wasabi.com/docs/creating-a-user-account-and-access-key#creating-a-user) for details. - Allow programmatic-only access, not console access. - Do not require Multi-Factor Authentication (MFA). - It is not necessary to assign the user to a group. - Save the access and secret keys in a secure location. ## Configuring SkyPilot 1. Install UV and create a Virtual Environment. SkyPilot requires Python 3.9–3.13 as of the writing of this article. UV is used to manage the Python environment. ```bash curl -LsSf https://astral.sh/uv/install.sh | sh source $HOME/.local/bin/env uv venv --seed --python 3.10 source .venv/bin/activate ``` Install SkyPilot with Kubernetes (or the appropriate compute infrastructure for your environment) support into the activated virtual environment. ```bash uv pip install "skypilot[kubernetes]" ``` 2. Install and configure a Kubernetes Cluster (if required). MicroK8s is recommended for a local, single-node Kubernetes cluster on Ubuntu. See the k3s section of [Known Issues](/v1/docs/skypilot-with-wasabi#known-issues) below for why k3s is specifically not recommended for this use case. ```bash sudo snap install microk8s --classic sudo usermod -aG microk8s $USER newgrp microk8s microk8s status --wait-ready microk8s enable dns microk8s enable hostpath-storage sudo snap install kubectl --classic mkdir -p ~/.kube microk8s config > ~/.kube/config chmod 600 ~/.kube/config kubectl get nodes ``` The output of `kubectl get nodes` should indicate a status of Ready for the node. Verify SkyPilot recognizes the cluster. ```plaintext sudo apt update && sudo apt install -y socat source .venv/bin/activate sky check ``` Expected output includes: ```plaintext 🎉 Enabled infra 🎉  Kubernetes [compute]    Allowed contexts:    └── microk8s ``` 3. Store Wasabi credentials locally. Create a dedicated AWS CLI profile for Wasabi, stored in its own credentials/config files separately from any existing AWS credentials, using the Access Key and Secret Key generated earlier. > This configuration example uses Wasabi’s us-east-2 storage region. To use another Wasabi storage region, use the appropriate URL for your bucket region as described in [Service URLs for Wasabi's Storage Regions](https://docs.wasabi.com/docs/service-urls-for-wasabis-storage-regions). ```bash sudo apt install awscli mkdir -p ~/.wasabi AWS_SHARED_CREDENTIALS_FILE=~/.wasabi/wasabi.credentials \  aws configure --profile wasabi # Enter the Wasabi Access Key and Secret Key when prompted. # Region can be left blank or set to us-east-1. # Set the output format to be json. AWS_CONFIG_FILE=~/.wasabi/wasabi.config \  aws configure set endpoint_url https://s3.us-east-2.wasabisys.com --profile wasabi AWS_CONFIG_FILE=~/.wasabi/wasabi.config \  aws configure set s3.addressing_style path --profile wasabi ``` Verify the profile works before proceeding. You should see a list of your buckets as the output. ```bash AWS_SHARED_CREDENTIALS_FILE=~/.wasabi/wasabi.credentials \ AWS_CONFIG_FILE=~/.wasabi/wasabi.config \  aws s3 ls --profile wasabi --endpoint-url https://s3.us-east-2.wasabisys.com ``` 4. Minimum required task YAML sections. 5. SkyPilot has no native “Wasabi” storage backend or cloud provider. Because Wasabi is S3-compatible, the working approach is to provision compute normally with SkyPilot, then use the AWS CLI directly inside the task's setup and run steps, pointed at Wasabi's endpoint instead of AWS. At a minimum, a task YAML needs the following four sections. - resources—Specifies the compute target. For Kubernetes: ```yaml resources:  infra: k8s ``` - file_mounts—Uploads the local Wasabi credential files to the remote pod so the AWS CLI running there can authenticate. ```yaml file_mounts:  ~/.wasabi/wasabi.credentials: ~/.wasabi/wasabi.credentials  ~/.wasabi/wasabi.config: ~/.wasabi/wasabi.config ``` - envs—Points the AWS CLI at the uploaded Wasabi credential files instead of the default ~/.aws location. ```yaml envs:  AWS_SHARED_CREDENTIALS_FILE: $HOME/.wasabi/wasabi.credentials  AWS_CONFIG_FILE: $HOME/.wasabi/wasabi.config ``` - setup—Installs the AWS CLI (not present by default in the SkyPilot task image) and downloads data from Wasabi before the job runs. Replace YOUR_WASABI_BUCKET with the name of your bucket. Use the URL associated with the region your bucket is in (the last line). ```yaml setup: |  export DEBIAN_FRONTEND=noninteractive  export TZ=Etc/UTC  apt_retry() {    local max_attempts=30    local attempt=1    until sudo -E "$@"; do      if [ $attempt -ge $max_attempts ]; then        echo "Giving up after $max_attempts attempts."        return 1      fi      echo "apt-get busy (lock held), retrying in 5s... (attempt $attempt/$max_attempts)"      sleep 5      attempt=$((attempt + 1))    done  }  apt_retry apt-get update  apt_retry apt-get install -y awscli  aws s3 sync s3://YOUR_WASABI_BUCKET ~/data \    --profile wasabi \    --endpoint-url https://s3.us-east-2.wasabisys.com ``` **Why the apt_retry wrapper is necessary:** SkyPilot installs its own runtime dependencies (rsync, curl, openssh-server, and so on) in the background when a pod first starts, using apt-get. If a task’s setup step also calls apt-get at the same time, both processes compete for dpkg’s lock file, resulting in errors such as "Could not get lock /var/lib/dpkg/lock-frontend." Wrapping apt-get calls in a retry loop (rather than trying to detect the lock ahead of time with tools like fuser, which may not be installed on the task image) reliably resolves this. DEBIAN_FRONTEND=noninteractive and TZ=Etc/UTC are also required — without them, installing awscli pulls in tzdata as a dependency, which prompts interactively for a timezone and hangs indefinitely, since no terminal is attached during setup. 6. Full example task YAML. The following complete example downloads data from a Wasabi bucket, generates a simple summary report and status file, and uploads the results back to the same bucket under an outputs/ prefix. Replace YOUR_WASABI_BUCKET (listed twice) and the endpoint URL with your own values. Name the file `wasabi_task.yaml`. ```yaml resources:  infra: k8s  cpus: 4+ file_mounts:  ~/.wasabi/wasabi.credentials: ~/.wasabi/wasabi.credentials  ~/.wasabi/wasabi.config: ~/.wasabi/wasabi.config envs:  AWS_SHARED_CREDENTIALS_FILE: $HOME/.wasabi/wasabi.credentials  AWS_CONFIG_FILE: $HOME/.wasabi/wasabi.config setup: |  export DEBIAN_FRONTEND=noninteractive  export TZ=Etc/UTC  apt_retry() {    local max_attempts=30    local attempt=1    until sudo -E "$@"; do      if [ $attempt -ge $max_attempts ]; then        echo "Giving up after $max_attempts attempts."        return 1      fi      echo "apt-get busy (lock held), retrying in 5s... (attempt $attempt/$max_attempts)"      sleep 5      attempt=$((attempt + 1))    done  }  apt_retry apt-get update  apt_retry apt-get install -y awscli  aws s3 sync s3://YOUR_WASABI_BUCKET ~/data \    --profile wasabi \    --endpoint-url https://s3.us-east-2.wasabisys.com run: |  echo "=== Data synced from Wasabi ==="  ls -la ~/data  mkdir -p ~/outputs  echo "=== Processing files ==="  {    echo "Wasabi Sync Test Report"    echo "Generated: $(date -u)"    echo "Hostname: $(hostname)"    echo ""    echo "Files found in ~/data:"    ls -la ~/data    echo ""    for f in ~/data/*; do      if [ -f "$f" ]; then        echo "--- $f ---"        echo "  Lines: $(wc -l < "$f")"        echo "  Words: $(wc -w < "$f")"        echo "  Bytes: $(wc -c < "$f")"      fi    done  } > ~/outputs/report.txt  echo "Task completed successfully at $(date -u)" > ~/outputs/status.txt  echo "=== Generated outputs ==="  cat ~/outputs/report.txt  echo "=== Uploading results back to Wasabi ==="  aws s3 sync ~/outputs s3://YOUR_WASABI_BUCKET/outputs \    --profile wasabi \    --endpoint-url https://s3.us-east-2.wasabisys.com  echo "=== Done ===" ``` 7. Create a test file in your Wasabi bucket. Before running the task, create a small test file locally using your favorite text editor (for example, vi, vim, or nano on Linux). ```plaintext nano test.txt ``` Enter the following line of text, then save and close the file. ```plaintext This is a test. ``` Log in to the Wasabi Console, navigate to **Buckets**, select the name of the bucket, and upload test.txt using the Console’s upload feature. This gives the task’s setup step something to download in the next section. ![](https://cdn.document360.io/bef0a1ea-7768-4d5a-b520-c4fe2f7fafad/Images/Documentation/Screenshot 2026-07-28 at 07-23-41.png) ![](https://cdn.document360.io/bef0a1ea-7768-4d5a-b520-c4fe2f7fafad/Images/Documentation/Screenshot 2026-07-28 at 07-25-04.png) 8. Running and verifying the task. ```plaintext sky launch -y -c wasabi-test wasabi_task.yaml ``` To confirm the results were written back to Wasabi, run the following commands. Replace YOUR_WASABI_BUCKET with the name of your bucket, and the endpoint URL with that of the region of your bucket. ```plaintext AWS_SHARED_CREDENTIALS_FILE=~/.wasabi/wasabi.credentials \ AWS_CONFIG_FILE=~/.wasabi/wasabi.config \  aws s3 ls s3://YOUR_WASABI_BUCKET/outputs/ \  --profile wasabi \  --endpoint-url https://s3.us-east-2.wasabisys.com ``` You should see report.txt and status.txt listed. Useful cleanup and follow-up commands: - sky status — list active clusters - sky logs wasabi-test — view job logs - sky down wasabi-test — terminate the cluster and stop billing/resource usage ## Known Issues ### k3s: Sky Launch Hangs Indefinitely During Provisioning During testing, a single-node k3s cluster (the Kubernetes distribution bundled by some Ubuntu server setups) caused every sky launch — including a trivial task with no file_mounts at all — to hang indefinitely at the Preparing SkyPilot runtime (1/3 - initializing) stage. Investigation ruled out the following as causes: - DNS resolution and general internet/package-mirror connectivity (all confirmed working) - The host firewall (ufw was inactive) - kubectl client/server version skew (both were identical) - Basic and multi-line streaming kubectl exec sessions (both completed successfully) - CPU, memory, and disk resource contention (all had ample headroom) - The kubectl exec transport protocol (forcing KUBECTL_REMOTE_COMMAND_WEBSOCKETS=true made no difference) The hang was isolated to SkyPilot’s internal file-sync step, which uses rsync tunneled through kubectl exec to transfer its runtime files into the pod. The actual rsync worker process inside the pod was found sleeping indefinitely, but the container’s security context blocked all process introspection tools (strace, ptrace-based inspection via /proc) even for the root user — consistent with a restrictive seccomp profile in this k3s configuration blocking ptrace() outright. This prevented the determination of the exact underlying cause. **Resolution:** Switching from k3s to MicroK8s on the same machine resolved the issue completely; the same trivial task was completed successfully on the first attempt after the switch. Until the root cause is identified, MicroK8s (or another distribution such as kind or minikube) is recommended over k3s for running SkyPilot locally on Ubuntu. ### aws: Command Not Found The default SkyPilot task image does not include the AWS CLI. It must be installed explicitly in the task’s setup step, as shown in the example YAML above. Installing it pulls in tzdata as a dependency, which requires DEBIAN_FRONTEND=noninteractive and a pre-set TZ environment variable to avoid an interactive prompt that would otherwise hang the setup step indefinitely. ### E: Could not get Lock /var/lib/dpkg/lock-frontend This occurs when the task’s apt-get call runs concurrently with SkyPilot’s internal background provisioning (which also uses apt-get to install its own runtime dependencies). Wrapping apt-get calls in a retry loop, as shown in the example setup step above, resolves this. If the error persists across many retries, check for a stale/orphaned apt-get process left over from a previous interrupted run (for example, one that hung on the tzdata prompt before this fix was applied) that may still be holding the lock; such a process can be found and terminated from inside the pod with ps aux | grep apt-get and sudo kill -9 .