Decoupling your data from WSL using VHD disk images
UPDATE: I wrote a script to automate all this: https://github.com/kmmiles/wsl-vhd-bash
WSL stores each distribution, and all files contained within it, in a single VHD disk image. There has been no decent way of decoupling that data. But with Windows 11 Build 22000, there are enough pieces of the puzzle to do this viably, and all from Linux.
Windows Requirements
- Windows 11 22000 or or higher.
- WSL (the feature itself) installed from the Microsoft Store: https://devblogs.microsoft.com/commandline/a-preview-of-wsl-in-the-microsoft-store-is-now-available/
- A working WSL distribution (Tested with Ubuntu, Debian 11, and Alma Linux 9)
The new --mount
option in WSL
I first learned that Microsoft included a new --mount
option in the wsl
utility that opens the door for some pretty neat things: https://learn.microsoft.com/en-us/windows/wsl/wsl2-mount-disk.
My first thought was to add a seperate partition to my drive, and use wsl --mount
to mount it within WSL, but…
“it’s not possible to use wsl –mount to read a partition on the boot device, because that device can’t be detached from Windows.”
Well, that sucks.
Using a virtual hard drive (VHD)
Fortunately WSL has added yet another option: wsl --mount --vhd
. That means a VHD disk image can be used instead.
I found guides and blogs detailing how to do this, but I wasn’t happy with them and wanted to do it entirely within WSL.
Install Linux dependencies
From a shell in your choice of WSL distribution…
Debian/Ubuntu: sudo apt install qemu-utils ntfs-3g util-linux
Redhat: sudo dnf install qemu-img ntfsprogs util-linux
Include support for btrfs
, ntfs
, exfat
, vfat
, fat
, and msdos
:
Debian/Ubuntu: sudo apt install util-linux qemu-utils btrfs-progs ntfs-3g exfat-utils exfat-fuse dosfstools
Redhat:
dnf install -y \
https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm
sudo dnf install util-linux qemu-img btrfs-progs ntfsprogs exfatprogs fuse-exfat dosfstools
Create the VHD
Use qemu-img
to create a 10GB VHD called mydisk.vhdx
.
qemu-img create -f vhdx /mnt/c/mydisk.vhdx 10G
NOTE: The VHD files must be stored on Windows.
Attach the VHD to WSL
First we need to “attach” aka “bare mount” the VHD. This exposes the device to Linux without actually mounting it:
wsl.exe --mount --vhd --bare 'C:\mydisk.vhdx'
Pro-tip: When calling Windows programs from WSL, you need only append the command with
.exe
.
Now use dmesg
to see which device it mapped to:
[227979.476940] scsi 0:0:0:4: Direct-Access Msft Virtual Disk 1.0 PQ: 0 ANSI: 5
[227979.478324] sd 0:0:0:4: Attached scsi generic sg4 type 0
[227979.478866] sd 0:0:0:4: [sdd] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
[227979.479499] sd 0:0:0:4: [sdd] Write Protect is off
[227979.479898] sd 0:0:0:4: [sdd] Mode Sense: 0f 00 00 00
[227979.480433] sd 0:0:0:4: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[227979.483982] sd 0:0:0:4: [sdd] Attached SCSI disk
As we can see, the device was attached to /dev/sdd
.
Create partition (optional)
For NTFS/vfat/exFAT/msdos VHD’s, you should probably use a partition. Windows requires one to mount natively (i.e. without WSL).
printf 'n\np\n1\n\n\nt 1\n7\nw\n' | sudo fdisk /dev/sdd
We created an HPFS/NTFS/exFAT partition on /dev/sdd1
.
Format
Now let’s format it with our filesystem of choice.
sudo mkfs.ext4 /dev/sdd # ext4
sudo mkfs.btrfs /dev/sdd # btrfs
sudo mkfs.ntfs /dev/sdd1 # ntfs
sudo mkfs.exfat /dev/sdd1 # exfat
sudo mkfs.vfat /dev/sdd1 # vfat
Mount non-FUSE filesystems e.g. ext2, ext3, ext4, btrfs
For native linux filesystems, we’ll use wsl.exe
for all mounting/unmounting.
First detach it:
wsl.exe --unmount '\\?\C:\mydisk.vhdx'
You could also just do
wsl.exe --unmount
, which will unmount anything you’ve mounted withwsl.exe
. Note the single quotes and\\?\
prefix. This is a Windows extended path prefix and required forwsl.exe --umount
.
Now mount it:
wsl.exe --mount --vhd --name "mydisk" 'C:\mydisk.vhdx'
If you omit
--name
the mountpoint will be the VHD path, stripped of delimiters e.g./mnt/wsl/Cmydiskvhdx
.
Optionally give read/write access to the current user:
sudo chown "$(id -un):$(id -gn)" /mnt/wsl/mydisk
Mount FUSE filesystems e.g. ntfs, exfat, vfat
For FUSE filesystems, the VHD must be attached with wsl.exe
and mounted with sudo mount
.
We already attached the VHD in the previous steps, so the first step can be skipped.
wsl.exe --mount --vhd --bare 'C:\mydisk.vhdx'
sudo mount -o "uid=$(id -u)" /dev/sdd1 /mnt/wsl/mydisk
chown
doesn’t work as expected, so the uid
option is passed to give read/write permission the current user.
This should be supported for ntfs, exfat, and vfat but if not just omit it.
Automatically mounting on startup
With a little effort, we can automatically mount the VHD when launching a distribution.
Create a shell script at /onboot
e.g. sudo editor /onboot
#!/bin/bash
/mnt/c/WINDOWS/system32/wsl.exe --mount --vhd --name 'mydisk' 'C:\mydisk.vhdx'
Give it execute permissions:
sudo chmod +x /onboot
Now sudo editor /etc/wsl.conf
and add:
[boot]
command = /onboot > /onboot.log 2>&1
Now shutdown WSL with wsl.exe --shutdown
. Upon relaunching the VHD should automatically be mounted.
If something goes wrong, check the output stored in
/onboot.log
Compact the VHD aka sparsify aka thin provision
After significant use and/or filling the VHD to capacity and free’ing up space, the actual size on disk will not be automatically freed. You can reclaim a percentage of this space with the following:
dd if=/dev/zero of=/mnt/wsl/mydisk/zeros
rm -f /mnt/wsl/mydisk/zeros
wsl.exe --unmount '\\?\C:\mydisk.vhdx'
qemu-img convert -f vhdx -O vhdx /mnt/c/mydisk.vhdx /mnt/c/mydisk-thin.vhdx
mv /mnt/c/mydisk-thin.vhdx /mnt/c/mydisk.vhdx
This is the equivalent of the
compact
feature of the Windowsdiskpart
tool.