# madbfs `madbfs` (Modern `adb` Filesystem, formerly `adbfsm`) is a userspace filesystem for Android via `adb` built using `libfuse`. ## Preview https://github.com/user-attachments/assets/dad81c5a-993e-480d-a329-d8fc560a69de > What's happening above: > > - viewing a video on Waydroid from host and copying it to host, and > - copying a picture from host to Waydroid. ## Motivation I want to manage my Android phone storage from my computer without using MTP. This project is inspired by the [`adbfs-rootless`](https://github.com/spion/adbfs-rootless) project. While `adbfs-rootless` works most of the time under light load it has reliability issues. Under high load (such as when a thumbnailer service is running) `adbfs-rootless` tends to crash, making it unusable until it is forcefully unmounted and its cache cleaned. The file I/O approach of `adbfs-rootless` is also similar to that of MTP, in which each file is first pulled from device (using `adb pull`) before any operations like reading and writing performed. If a write operation is performed, the file is then pushed back into the device (using `adb push`). This approach limits the size of the file that can be handled before the cache is filled (it uses `/tmp`). These limitations lead me to decide to rebuild the project from ground up in order to create a more stable, modern, and possibly faster solution. ## Features > TLDR: full file and directory traversal with concurrent file streaming approach, partial read/write, and active caching. - No root access required Non-rooted device will have standard file and directory access as regular user. Rooted device will have less constraints on file and directory access. - Full file and directory traversal Browse the entire file tree of your Android device (subject to permission constraints). - Read and write support Open, read, and write files (subject to permission constraints). - Create and delete files and directories Support creating new files and directories as well as deleting them (subject to permission constraints). - Rename and move Rename or move files and directory seamlessly (subject to permission constraints). - Modify file timestamps Update file access and modification times. - Automatic in-memory caching Recently accessed files are cached in memory using an LRU paging mechanism, allowing faster repeated access. - Streamed file access (partial read/write) Read files on-demand without pulling the entire file first. Unlike MTP’s "pull-whole-file" approach, this filesystem streams data over `adb`, enabling efficient access to large files or specific file segments. - Efficient resource use Loads only what you access, conserving memory usage and bandwidth. You can also control how much the cache stores file data. - Concurrent file access Allows for concurrent access to files and directories without blocking. This ability comes out-of-the-box by virtue of using FUSE and `adb`. - Flexible connection method `madbfs` offers two kinds of transport: - `adb` transport The simplest transport. It executes all FUSE operations by running `adb shell` commands like `dd`, `stat`, and `touch`. No additional component is required. - Proxy transport Communicates with a lightweight TCP server running natively on the Android device via a custom RPC protocol. It requires a server binary compiled for the phone architecture but has better I/O throughput than the `adb` transport. Both of the transport can work wired via USB or wireless in your local network. `madbfs` will automatically fall back to `adb` transport if the proxy transport is not available. - Resilient to disconnections Stays mounted even when the device disconnects. Cached files and directories remain accessible, and full functionality resumes automatically when the connection is restored. - Built with modern C++ Developed in C++23 using coroutine-style asynchronous programming for clean, lightweight, high-performance I/O. ## Dependencies - `madbfs` and `madbfs-msg` Runtime dependencies - adb Build dependencies - GCC or Clang (support C++23 or above, tested on GCC 15 and GCC 16) - CMake (version 3.22+) - Ninja - Conan Library dependencies - Boost (Asio, Process, and JSON component) - libfuse - rapidhash - spdlog - `madbfs-server` Build dependencies - Android NDK (support C++23 or above, tested on NDK version 29 (beta 1)) - CMake (version 3.22+) - Ninja - Conan Library dependencies - Asio (non-Boost variant) - spdlog ## Building > if you just want a prebuilt binaries, jump to [installation](#installation) Since this project dependencies are managed by Conan, you need to make sure it's installed and configured correctly on your system (consult the [documentation](https://docs.conan.io/2/installation.html) on how to do it). There are two CMake project that produces executables: - `madbfs`: produces `madbfs` and `madbfs-msg` - `madbfs-server`: produces multiple `madbfs-server` for each architecture Android supports `madbfs` project is compiled once for the host machine, while `madbfs-server` is compiled multiple times for different architectures using Android NDK. `madbfs` embeds the produced `madbfs-server` in order to simplify the software usage. This requirement makes it so that the compilation step of the project is quite strict. Because of that, I have written a handy script to automate this process. ### Using script ```sh ./build.sh # try adding --help to this script ``` In order to use this script, you are required have `ANDROID_NDK_HOME` variable defined and point to valid Android NDK. The script above will produce build directories at this specified stage: - `madbfs`: `build//` - `madbfs-server`: `madbfs-server/build/android--/` and `madbfs-server/build/android-all-/` > - `madbfs` executable can be obtained at `build//madbfs/madbfs` > - `madbfs-msg` executable can be obtained at `build//madbfs-msg/madbfs-msg` When the script is invoked with `--package` option, it will produce a `tar.gz` file containing `madbfs` and `madbfs-msg` and its hash at: - `build/package--` ### Manually - **`madbfs-server`** While Conan is a package manager, it can also be used as toolchain for cross-compiling via profile. A profile for build environment and host environment is required. I used profiles at `./.github/workflows/` both for development and CI. If you read `profile-server.ini` you might notice that `settings.arch` and `conf.tools.android:ndk_path` values are not set concretely, instead it uses environment variable. This is because the profile needs to be reused for multiple architecture and not hardcode Android NDK path on the profile respectively. > Conan uses build-host naming convention instead of host-target for cross-compilation For this instruction, I will use the profiles above. The first step is moving to `madbfs-server` directory from the root. Then you need to get build dependencies using Conan. At this point `ANDROID_NDK_HOME` and `MADBFS_SERVER_ARCH` (for `conf.tools.android:ndk_path` and `settings.arch` respectively) need to be define already. Assume the `${variables}` point to valid arch, profiles, and build type (capitalized) ```sh export ANDROID_NDK_HOME=path/to/android/ndk export MADBFS_SERVER_ARCH=${arch} conan install . --build missing -pr:b "${profile_client}" -pr:h "${profile_server}" -s build_type=${build_type} ``` > you can also use `-c:h tools.android:ndk_path=path/to/android/ndk` and `-s:h arch=${arch}` Conan options to specify the NDK path and arch respectively instead of environment variable Then you can generate and compile them: ```sh cmake --preset conan-android-${arch}-${build_type,,} cmake --build --preset conan-android-${arch}-${build_type,,} ``` > `,,` is bash parameter expansion for lowering string case This will produce `madbfs-server` executable at `./build/android--/madbfs-server` Do this repeatedly for all supported arch. - **`madbfs` (and `madbfs-msg`)** > This instruction will only focus on `madbfs` since `madbfs-msg` compilation is simple and doesn't need additional configuration. First, move to the root of the project, then install the dependencies. Assume the `${variables}` point to valid arch, profiles, and build type (capitalized) ```sh # notice that both build and host profiles point to ${profile_client} because it will be used on the build machine conan install . --build missing -pr:b "${profile_client}" -pr:h "${profile_client}" "&:build_type=${build_type}" -s build_type=Release ``` Then you need to prepare a directory that contains the `madbfs-server` for all architecture. The executables should be renamed to have each of their ABI (not arch): `madbfs-server-abi`. This step is required because `madbfs` expects `madbfs-server` files to have specified type but on which directory. ```sh mkdkir -p ${servers_path} # ... cp ./build/android-${arch}-${build_type,,} ${servers_path}/madbfs-server-${abi} # for each arch ``` To tell where `madbfs` can find the server files, you need to pass `MADBF_SERVER_BINARY_DIR` to CMake, which is done at generation time ```sh cmake --preset conan-${build_type,,} -D MADBFS_SERVER_BINARY_DIR="${servers_path}" ``` Building `madbfs` is then just this: ```sh cmake --build --preset conan-${build_type,,} ``` The executable can be found at `./build//madbfs/madbfs` (and `./build//madbfs-msg/madbfs-msg`) ## Installation You can download `madbfs` from the [GitHub releases page](https://github.com/mrizaln/madbfs/releases). There is no installation step required, as the application is built statically. You can place the binaries wherever you prefer. #### Installing from the AUR `madbfs` can be installed using your prefered AUR helper, as with every other AUR package. Using `yay`, you'd run the following command as superuser: ``` yay -S madbfs-bin ``` ## Usage The help message can help you start using this program ``` usage: madbfs [options] Options for madbfs: --serial= serial number of the device to mount (you can omit this [detection is similar to adb]) (will prompt if more than one device exists) --root= directory to mount on device as root of the filesystem (by default madbfs mounts the root path of the device) (the path must be absolute and points to an existing path) (if the path is a symlink, it will be resolved first) --log-level= log level to use (default: "warning") (enum: "trace", "debug", "info", "warning", "error", "critical", "off") --log-file= log file to write to (default: "-" for stdout) --cache-size= maximum size of the cache in MiB (default: 256) (minimum: 128) (value will be rounded up to the next power of 2) (ignored if 'no-cache' is provided) --page-size= page size for cache & transfer in KiB (default: 128) (minimum: 64) (maximum: 4096) (value will be rounded up to the next power of 2) (ignored if 'no-cache' is provided) --ttl= set the TTL of the stat cache of the filesystem in seconds (default: 60) (set to 0 to disable it) --timeout= set the timeout of every remote operation (default: 2) (set to 0 to disable it) --port= set the port number the server will listen on (default: 23237) --no-server don't launch server (will still attempt to connect to specified port) (fall back to adb shell calls if connection failed) (useful for debugging the server) --adb-only don't launch server and don't try to connect --no-cache don't use data caching Options for libfuse: -h --help print help -V --version print version -d -o debug enable debug output (implies -f) -f foreground operation -s disable multi-threaded operation -o clone_fd use separate fuse device fd for each thread (may improve performance) -o max_idle_threads the maximum number of idle worker threads allowed (default: -1) -o max_threads the maximum number of worker threads allowed (default: 10) -o allow_other allow access by all users -o allow_root allow access by root -o auto_unmount auto unmount on process termination ``` ### Unmounting To unmount the filesystem, you run this command like for any other FUSE filesystem: ```sh fusermount -u ``` You can also use the [IPC](./IPC.md) operation `unmount` for unmounting. This will unmount the filesystem on the next operation immediately. ### Selecting device You don't need to specify the device if there is only one device connected via `adb` on your computer. If there are more than one device then you can specify the serial using `--serial` option. If you omit the `--serial` option when there are multiple device connected to the computer, you will be prompted to specify the device you want to mount. ```sh $ madbfs [madbfs] checking adb availability... [madbfs] multiple devices detected, - 1: 068832516O101622 - 2: 192.168.240.112:5555 [madbfs] please specify which one you would like to use: _ ``` `madbfs` respects the env variable `ANDROID_SERIAL` (mimicking `adb` behavior) so you can alternately use it to specify the device. ```sh $ ANDROID_SERIAL=068832516O101622 ./madbfs [madbfs] checking adb availability... [madbfs] using serial '068832516O101622' from env variable 'ANDROID_SERIAL' ``` ### Mounting subdirectory `madbfs` supports mounting subdirectory. Use `--root` option to specify it. ```sh $ madbfs --root= ``` The `` is the subdirectory on your device you want to treat as the root of the filesystem (effectively mounting a subdirectory). The option only accepts absolute path and if the directory is a symlink, it will resolve it first (`/sdcard` -> `/storage/emulated/0`). Do note that mounting subdirectory might break symbolic links since it is possible that a link may contain components that are unreachable from the specified root. For example if you mount `/sdcard/` and have a link in `/sdcard/link` that points to `/storage/` the filesystem can't reach it, because `/storage/` is outside of `/sdcard/`. ### Specifying server and port number If you want the filesystem to use only use `adb` transport, use `--adb-only` flag. This flag prevents `madbfs` from pushing the server into your phone and running it. If you rather want to manually run the server yourself (for debugging purpose for example), use `--no-server` flag instead. The proxy communicates with `madbfs` over TCP enabled by port forwarding and by default it will listen on port `23237` (`adbfs` on dial pad). If you find this port to be not suitable for your use you can always specify it with `--port` option. ```sh $ madbfs --server= --port=23237 ``` ### To cache or not to cache By default `madbfs` caches file stat and file content (data) of files operated by the filesystem. While file stat caching is always active, file content caching can be disabled using `--no-cache` program option. This option will ignore other cache related options. ```sh $ madbfs --no-cache ``` The no-cache mode is basically a direct I/O, file content will always be read/written to device immediately via the connection method (proxy or adb transport). ### Cache size `madbfs` caches all the read/write operations on the files on the device. This cache is stored in memory. You can control the size of this cache using `--cache-size` option (in MiB). The default value is `256` (256 MiB). ```sh $ madbfs --cache-size=256 # 256 MiB of memory will be used as file cache ``` ### Page size In the cache, each file is divided into pages. The `--page-size` option dictates the size of this page (in KiB). Page size also dictates the size of the buffer used to read/write into the file on the device. You can adjust this value according to your use. ```sh $ madbfs --page-size=128 # read/write operations are communicated in 128 KiB chunks ``` ### Logging The default log file is stdout (specified by "-"; which goes to nowhere when not run in foreground mode). You can manually set the log file using `--log-file` option and set the log level using `--log-level`. ```sh $ madbfs --log-file=madbfs.log --log-level=debug ``` You can always watch the logs of the filesystem at runtime even if you don't specify a log-file beforehand by using IPC `logcat` operation (see [below](<#ipc-(and-madbfs-msg)>)). ### IPC (and `madbfs-msg`) Filesystem parameters can be reconfigured and queried during runtime though IPC using unix socket. The supported operations are: - help, - version, - info, - invalidate cache, - set page size, - set cache size, - set ttl, - set timeout, - set log level, and - unmount (on next FUSE operation) - logcat (read `madbfs` log in real-time, similar to `adb logcat`). The address of the socket in which you can connect to as client is composed of the name of the filesystem and the serial of the device. The socket itself is created in directory defined by `XDG_RUNTIME_DIR` environment variable (it's usually set to `/run/user/`). If the `XDG_RUNTIME_DIR` is not defined, as fallback, the directory is set to `/tmp`. The socket will be created when the filesystem initializes. For example, the socket path for a device with serial `192.168.240.112:5555`: ``` /run/user/1000/madbfs@192.168.240.112:5555.sock ``` If at initialization this socket file exists, the IPC won't start. This may happen if the filesystem is terminated unexpectedly (crash or kill signal). You need to remove this file manually if that happens. For the specification of the message protocol used on the IPC and how to use it read [IPC.md](./IPC.md) file. To make it easier for user to use the IPC without having to write their own socket code, I have created another executable: `madbfs-msg`. The possible operations are explained in [IPC.md](./IPC.md) file as well. ### Debug mode As part of debugging functionality `libfuse` has provided debug mode through `-d` flag. You can use this to monitor `madbfs` operations (if you don't want to use log file or want to see the log in real-time). If the debugging information is too verbose, you can use `-f` instead to make madbfs run in foreground mode without printing `fuse` debug information. ```sh $ madbfs --log-file=- --log-level=debug -f # runs in foreground (not daemonized) $ madbfs --log-file=- --log-level=debug -d # this will print the libfuse debug messages and madbfs log messages $ madbfs --log-file=- --log-level=debug -d 2> /dev/null # this will print only madbfs log messages since libfuse debug messages are printed to stderr ``` ## Benchmark Benchmark is done by writing a 64 MiB file using `dd` and then reading it back. The statistics printed by `dd` is used for the speed value so is for `adb push` and `adb pull`. The test is done on an Android 11 phone (armv8) using USB cable with proxy transport. As baseline, the speed on which an `adb push` (write) and an `adb pull` (read) operation is done on a file with the same size is measured. `madbfs` is launched using its default parameters (cache size = 256 MiB, page size = 128 KiB). The write command is as follows: ```sh dd if=/dev/random of=random bs=128K count=1024 ``` The read command is as follows: ```sh dd if=random of=/dev/null bs=128K count=1024 ``` ### Proxy transport - Write | operation | speed (MB/s) | relative speed | | :-------- | -----------: | -------------: | | adb push | 15.3 | 1.00 | | write | 13.1 | 0.86 | - Read | operation | speed (MB/s) | relative speed | | :-------------- | -----------: | -------------: | | adb pull | 11.7 | 1.00 | | read (no cache) | 10.8 | 0.93 | | read (cache) | 2100.0 | 179.49 | ### `adb` transport - Write | operation | speed (MB/s) | relative speed | | :-------- | -----------: | -------------: | | adb push | 15.3 | 1.00 | | write | 12.4 | 0.81 | - Read | operation | speed (MB/s) | relative speed | | :-------------- | -----------: | -------------: | | adb pull | 11.7 | 1.00 | | read (no cache) | 3.6 | 0.24 | | read (cache) | 2100.0 | 179.49 |