# storagekit (@mohamedhabibwork/storagekit)

> Unified TypeScript file management across Local filesystem, AWS S3, MinIO, Azure Blob Storage, Oracle OCI Object Storage, RustFS, Cloudflare R2 and Google Cloud Storage. One common API for common operations, with strongly-typed **provider-native options** preserved under a `native` key instead of being flattened to a lowest common denominator. Runs on Node.js >= 20, Bun >= 1.1 and Deno >= 2 (all smoke-tested in CI). Dual ESM/CJS with full TypeScript declarations. Custom drivers can be registered against the same API.

## Install

```bash
npm install @mohamedhabibwork/storagekit
# Provider SDKs are optional peer dependencies — install only what you use:
# S3 / RustFS / R2: npm i @aws-sdk/client-s3 @aws-sdk/lib-storage @aws-sdk/s3-request-presigner
# MinIO:       npm i minio
# Azure:       npm i @azure/storage-blob
# Oracle:      npm i oci-common oci-objectstorage
# GCS:         npm i @google-cloud/storage
# Local:       no extra dependency (Node built-ins only)
```

**Entrypoints** (import only what you use; the root entry bundles no cloud SDK):
`@mohamedhabibwork/storagekit` (core + factory + manager + copyBetween + upload intake),
`.../local`, `.../s3`, `.../minio`, `.../azure`, `.../oracle`, `.../rustfs`,
`.../r2`, `.../gcs` (each also exports `createXxxStorage`), `.../testing` (driver contract suite),
`.../testing/fake` (in-memory fake driver + createFakeStorage() for tests, no vitest dep),
`.../uploads` (upload intake helpers), `.../adapters/express`,
`.../adapters/fastify`, `.../adapters/formidable` (framework upload adapters).

Loading a driver without its SDK installed throws `StorageInvalidConfigError`
with the exact `npm install` command — never a module-not-found crash.

## Core rule

Common options are normalized and identical on every provider. Anything
provider-specific goes in `native`, strongly typed by the configured storage
type: `Storage<'s3'>` accepts real AWS fields (`StorageClass`,
`ServerSideEncryption`, `ACL`, …) while `Storage<'azure'>` accepts Azure
fields (`tier`, `conditions`, `tags`, …) and REJECTS AWS fields at compile
time. `native` is merged last and overrides common equivalents.

## Creating a storage (async)

```ts
import { createStorage } from '@mohamedhabibwork/storagekit';

// Returns Promise<Storage<'s3'>> — the type parameter is inferred from `type`
const s3 = await createStorage({
  type: 's3',                    // 'local' | 's3' | 'minio' | 'azure' | 'oracle' | 'rustfs' | 'r2' | 'gcs'
  bucket: 'uploads',
  region: 'eu-central-1',
});
```

Required config per type:
- `local`: `root` (required); optional `baseUrl` (powers `getUrl()`), `prefix`, `permissions`, `createDirectories`, `followSymlinks`
- `s3`: `bucket`; optional `region`, `endpoint`, `credentials`, `forcePathStyle`, `prefix`, `publicUrlBase`, `client` (inject S3Client), `clientOptions`
- `minio`: `bucket`, `endPoint`; optional `port`, `useSSL`, `accessKey`, `secretKey`, `region`, `prefix`, `publicUrlBase`, `client`, `clientOptions`
- `azure`: `container` + ONE auth route: `connectionString` | `accountUrl` (+ `credential`: shared key or TokenCredential) | `serviceClient` | `containerClient`; optional `prefix`, `publicUrlBase`
- `oracle`: `namespaceName`, `bucketName`; optional `region`, `prefix`, `publicUrlBase`, `auth` (`{type:'config-file'|'instance-principals'|'resource-principals'|'provider'}`), `authProvider`, `client`, `clientOptions`
- `rustfs`: `bucket` + `endpoint` (required); optional `region` (default `us-east-1`), `credentials`, `forcePathStyle` (default `true`), `prefix`, `publicUrlBase`, `client`, `clientOptions`. RustFS is S3-compatible but ships no first-party JS SDK — `native()` returns `S3Client` and every `native` option accepts the AWS SDK input fields.
- `gcs`: `bucket` (required); optional `projectId` (auto from Application Default Credentials), `keyFilename`, `credentials`, `apiEndpoint` (for emulators), `retryOptions`, `prefix`, `publicUrlBase`, `client`, `clientOptions`. Backed by `@google-cloud/storage` — uses GCS resumable uploads, V4 signed URLs, full `FileMetadata` on every operation. `native()` returns the SDK `Storage` instance.

All providers accept an optional `prefix` (e.g. `'production/'`): every
operation transparently stores/lists under that prefix and strips it from
results.

Second factory argument (all optional):
`{ hooks, onOperation, detectContentType }`. `detectContentType` defaults to
true and is a lightweight extension→MIME map; pass `contentType` explicitly
when it matters.

## Upload

```ts
// body: string | Buffer | Uint8Array | ArrayBuffer | Blob | Node Readable
await storage.upload('users/1/avatar.jpg', buffer, {
  contentType: 'image/jpeg',          // normalized
  metadata: { userId: '100' },        // normalized
  cacheControl: 'public,max-age=31536000',
  contentDisposition: '…', contentEncoding: '…',
  overwrite: false,                   // default true; false = HEAD check + StorageConflictError
  signal: controller.signal,          // cancellation everywhere
  multipart: { partSize: 10 * 1024 * 1024, concurrency: 4 },
  native: { StorageClass: 'INTELLIGENT_TIERING' },   // provider-native, typed
});
// → { path, size?, etag?, versionId?, url?, provider, native? }
```

Streams are never buffered into memory. Multipart mapping: S3 → `@aws-sdk/lib-storage`
(`partSize`, `concurrency` → `queueSize`); Azure → `uploadStream(bufferSize, maxConcurrency)`;
Oracle → native multipart API (partSize floored at 1 MiB, default 64 MiB);
MinIO → its internal multipart (size unknown); local → plain stream write.

Provider-native field examples:
- S3: `StorageClass`, `ServerSideEncryption: 'aws:kms'`, `SSEKMSKeyId`, `ACL`, `Tagging`
- Azure: `tier: 'Cool'`, `conditions`, `tags`, `blobHTTPHeaders`
- MinIO: `metaData: { … }` (extra x-amz-meta bag)
- Oracle: `storageTier: 'Archive'`, `ifMatch`, `opcMeta`, `opcChecksumAlgorithm`
- Local: `mode: 0o600`

## Framework uploads (multer, @fastify/multipart, formidable, web File)

```ts
// Core intake — works with ANY upload middleware. Import from 'storagekit/uploads'
// (also re-exported from the root entry).
import { saveUpload, sanitizeFilename, randomKey } from '@mohamedhabibwork/storagekit/uploads';
const saved = await saveUpload(storage, {
  body: file.stream,              // stream | Buffer | Blob
  fieldname: file.fieldname,
  originalName: file.originalname, // untrusted: used only for key EXTENSION + metadata
  mimeType: file.mimetype,
}, { directory: 'uploads' });      // → uploads/<uuid>.<ext from original name>
// saved: { key, originalName?, mimeType?, result }  (result = UploadResult)
// Auto metadata: originalname/fieldname/mimetype (merge via `metadata: {...}`,
// disable with `metadata: false`). `key` accepts a string or async resolver —
// both traversal-checked. `native`, `signal`, `overwrite`, `contentType` pass through.

// Upload intents — a declarative object you can build anywhere and run later
// (HTTP handler, queue consumer, CLI); re-runnable value, driver-agnostic.
import { defineUploadIntent, saveUploadIntent } from '@mohamedhabibwork/storagekit/uploads';
const intent = defineUploadIntent({ source: file, directory: 'uploads', metadata: { tenantId: 't1' } });
const saved2 = await saveUploadIntent(storage, intent);
// source: web File/Blob (name/type/size auto) | { body, fieldname?, originalName?, mimeType?, size? } | bare bytes.
// Fields: directory?, key? (string | resolver), contentType?, metadata? (| false), overwrite?, native?, signal?.

// Express / NestJS / koa-multer — multer storage engine ('storagekit/adapters/express'):
import multer from 'multer';
import { createMulterStorage } from '@mohamedhabibwork/storagekit/adapters/express';
const upload = multer({ storage: createMulterStorage(storage, { directory: 'uploads' }) });
app.post('/u', upload.single('avatar'), (req, res) =>
  res.json({ key: req.file!.key, etag: req.file!.etag, storagekit: req.file!.storagekit }));
// NestJS: FileInterceptor('avatar', { storage: createMulterStorage(storage) })
// Request aborted / handler error → stored object is deleted (_removeFile).

// Fastify 4/5 ('storagekit/adapters/fastify'):
import { saveFastifyFile } from '@mohamedhabibwork/storagekit/adapters/fastify';
app.post('/u', async (req) => {
  const part = await req.file();
  return saveFastifyFile(storage, part!, { directory: 'uploads' }); // consumes the stream
});

// Hono / Next.js / Remix / Nuxt / Elysia / Bun.serve / Deno.serve — web File:
import { saveWebFile } from '@mohamedhabibwork/storagekit/uploads';
const saved = await saveWebFile(storage, fileFromFormData, { directory: 'uploads' });

// formidable v2/v3 ('storagekit/adapters/formidable'):
import { saveFormidableFile } from '@mohamedhabibwork/storagekit/adapters/formidable';
const saved = await saveFormidableFile(storage, files.file, { directory: 'uploads' });
// Koa (@koa/multer) uses the same multer engine; busboy / GraphQL Upload →
// wrap the part into UploadFileInput and call saveUpload(). Serve files back
// via getUrl() (local needs baseUrl), getSignedUrl() (Oracle: PARs, not
// presigned), or download() + stream. Validate client MIME/size yourself
// (multer fileFilter + limits run before the engine).
```

Adapters import NOTHING from the frameworks (structural types only) — every
framework package stays an optional peer. Client filenames never control the
key stem: `randomKey()` uses a UUID; `sanitizeFilename()` keeps only a safe
basename/extension.

## Download

```ts
const dl = await storage.download('docs/report.pdf', {
  versionId: '…',                     // where the provider supports it
  range: { offset: 100, length: 50 }, // byte range where supported
  signal,
});
dl.stream            // Node Readable — pipe it somewhere
dl.contentType; dl.contentLength; dl.etag; dl.lastModified; dl.metadata; dl.versionId
await dl.buffer(); await dl.text(); await dl.json<MyType>();
```

Always consume the stream (or call `.buffer()/.text()/.json()`). If you abort,
`stream.destroy()` it.

## Exists / stat / delete

```ts
await storage.exists('a.txt');                    // HEAD-based, never downloads
const st = await storage.stat('a.txt');           // { path, size, contentType?, etag?, lastModified?, metadata?, versionId?, provider }
await storage.delete('a.txt');                    // IDEMPOTENT: deleting a missing object is a no-op
await storage.deleteMany(['a', 'b', 'c']);        // → { deleted: string[], failed: [{ path, error }] }
```

Missing file: `download`/`stat` throw `StorageNotFoundError`; `delete`/`deleteMany`
succeed silently. `exists` returns false.

## List / iterate

```ts
// default (recursive: false) = ONE level deep, like S3 with delimiter
const page = await storage.list({ prefix: 'users/100/', limit: 100, cursor });
page.files;        // StorageFile[]: { path, size?, etag?, lastModified?, metadata? }
page.directories;  // ['users/100/sub/'] — trailing slash, one level
page.cursor;       // opaque token — pass back as { cursor } verbatim
page.hasMore;

// recursive flat scan (all files under prefix, directories empty)
await storage.list({ prefix: 'uploads/', recursive: true, limit: 1000 });

// async iteration — pagination handled for you, yields files only
for await (const file of storage.iterate('uploads/')) { file.path; }
```

`limit` defaults to 1000 and bounds entries per page (files + directories).
Cursors are opaque driver-specific tokens: only round-trip them, never parse
or persist across config changes.

## Copy / move / cross-provider copy

```ts
await storage.copy('temp/a.jpg', 'images/a.jpg');   // server-side copy
await storage.move('temp/a.pdf', 'docs/a.pdf');     // copy + delete (rename on local)

import { copyBetween } from '@mohamedhabibwork/storagekit';
await copyBetween(srcStorage, 'docs/f.pdf', dstStorage, 'archive/f.pdf', {
  concurrency: 4, onProgress: (bytes, total) => {},
});
```

`copy` streams server-to-server within a provider. Cross-provider always
streams through the process (never fully buffered) — use `copyBetween`.

## URLs

```ts
await storage.getUrl('images/logo.png');      // unsigned; no network call
await storage.getSignedUrl('private.pdf', { expiresIn: 3600, action: 'read' }); // 'read' | 'write' | 'delete'
```

`expiresIn` is seconds, default 3600, hard-capped at 604800 (7 days) — out of
range throws. S3/MinIO/RustFS: presigned; Azure: SAS via shared key / connection
string (TokenCredential needs user-delegation via `nativeRequest`); local:
throws `StorageUnsupportedOperationError` (serve through your app); Oracle:
throws — it has no presigned URLs. `createOracleStorage()` (subpath import)
adds `createPreauthenticatedRequest()` for Oracle PARs, whose lifecycle is
persistent server-side and fundamentally different from presigned URLs.

## Escaping the abstraction

```ts
const client = storage.native();            // S3Client | ContainerClient | Minio.Client | ObjectStorageClient | {root, resolve}
await storage.nativeRequest((c) => c.anySdkCall());   // typed native client
storage.capabilities();                     // { signedUrls, multipartUpload, serverSideCopy, versioning, metadata, directories, bulkDelete }
```

Check `capabilities()` instead of assuming a feature exists; unsupported
features throw `StorageUnsupportedOperationError` rather than being silently
emulated.

## Errors

```ts
import { StorageNotFoundError, isStorageError } from '@mohamedhabibwork/storagekit';
try { await storage.download(p); } catch (e) {
  if (e instanceof StorageNotFoundError) { /* missing */ }
  e.provider;  e.operation;  e.path;  e.code;  e.cause;  // original SDK error
}
```

Classes: `StorageError` (base), `StorageNotFoundError`,
`StoragePermissionError`, `StorageConflictError`, `StorageInvalidConfigError`,
`StorageNetworkError`, `StorageQuotaError`,
`StorageUnsupportedOperationError`, `StorageInvalidPathError`. Use
`isStorageError(e)` when errors may cross entrypoint copies. Provider errors
are preserved on `cause` and never leak credentials into messages.

## Multiple disks / manager

```ts
const disks = createStorageManager({ default: 'uploads', disks: {
  uploads: { type: 's3', bucket: 'uploads' },
  temp: { type: 'local', root: './tmp' },
}});
await disks.disk('uploads').upload('a.txt', 'x');   // typed per disk config
await disks.defaultDisk();
await disks.warmup();               // create every configured disk up front
```

## Hooks & events

```ts
const storage = await createStorage(config, {
  hooks: { beforeUpload, afterUpload, uploadError, beforeDownload,
           afterDownload, downloadError, beforeDelete, afterDelete, deleteError },
  onOperation: (e) => e.provider && e.duration && e.success,
});
const off = storage.on('operation', (e) => {}); off();
// event: { provider, operation, path?, duration, success, error? }
```

Hook contexts are sanitized: credentials and signed URLs never appear.
Hook errors in `beforeX` abort nothing silently — `beforeX` errors propagate;
`xxxError` hooks never mask the original failure.

## Key semantics (do not get these wrong)

- Keys always use `/`. Input is normalized: backslashes, duplicate slashes,
  leading/trailing slashes, `.` segments. `..` traversal throws
  `StorageInvalidPathError`; local paths can never escape the configured root.
- `overwrite` defaults to true everywhere; `overwrite: false` costs one extra
  metadata round-trip.
- Deleting missing objects is a no-op on every provider (idempotent).
- `native` overrides common options when both set the same field.
- Everything network-bound accepts `signal`.
- Export maps are dual ESM/CJS with full TypeScript declarations; types work
  with `moduleResolution: 'bundler'` / `'node16'`.

## Best practices

1. Import provider subpaths (`.../s3`) in bundled apps so only one SDK ships.
2. `await createStorage(...)` — the factory loads the SDK and builds clients.
3. Stream large files; never `readFile` then upload the buffer.
4. Set `contentType` explicitly for user-facing correctness; extension
   detection is a convenience, not a guarantee.
5. Pass `signal` on long transfers and wire it to request timeouts.
6. Scan big buckets with `iterate()` (or `list({recursive:true})` paging), not
   unpaginated `list()`.
7. Round-trip `cursor` opaquely; don't persist cursors across restarts of a
   changed config.
8. Same-provider copies: `storage.copy()` (server-side). Different providers:
   `copyBetween()`.
9. Reach for `native()` / `nativeRequest()` for unsupported SDK features
   instead of bypassing the package.
10. Catch specific error subclasses; log `e.cause` for provider-side detail.
11. For local `getUrl()`, configure `baseUrl` (or use `native: { fileUrl: true }`).
12. Versioning differs per provider: `versionId` is normalized on
   download/stat/delete, but semantics (S3 vs Azure vs OCI) are not identical —
   don't assume one behavior.

## Custom drivers

Implement `StorageDriver<T>` (exported), register it, then create storages
with that type string. Builtin type names cannot be overridden; duplicate
registrations throw `StorageInvalidConfigError`.

```ts
import { registerStorageDriver, defineDriver, type StorageDriver } from '@mohamedhabibwork/storagekit';

const driver: StorageDriver<'memory'> = defineDriver({
  type: 'memory',
  upload(path, body, options) { /* returns { path, size?, etag?, versionId?, url?, provider } */ },
  download(path) { /* returns { stream, contentType?, contentLength?, buffer(), text(), json() } */ },
  delete(path) {},                                    // idempotent, like builtins
  deleteMany(paths, options) { /* { deleted, failed } */ },
  exists(path) { /* HEAD-style check */ },
  stat(path) { /* FileStat */ },
  list(options) { /* { files, directories (trailing '/'), cursor, hasMore } — one level unless options.recursive */ },
  copy(source, destination, options) {},
  move(source, destination, options) {},              // copy + delete
  getUrl(path) { /* unsigned URL or throw StorageUnsupportedOperationError */ },
  getSignedUrl(path, options) { /* or throw unsupported */ },
  native() { return client; },
  nativeRequest(fn) { return fn(native()); },
  capabilities() { return { signedUrls: false, multipartUpload: true, serverSideCopy: true, versioning: false, metadata: true, directories: false, bulkDelete: false }; },
});

registerStorageDriver('memory', (config, runtime) => driver);
const storage = await createStorage({ type: 'memory', /* custom fields */ });
```

Custom types get `unknown` native slots (`MapValueFor` fallback). Optional
async `ready()` on the driver is awaited by the factory. The registry is
shared process-wide (globalThis symbol), visible across entrypoints.
Missing methods (missing `type` string / non-function `upload`/`download`)
are rejected at registration/use with `StorageInvalidConfigError`.

## Testing custom drivers

```ts
import { defineDriverContractTests } from '@mohamedhabibwork/storagekit/testing';
defineDriverContractTests({ name: 'mine', createStorage: () => make(), capabilities: { signedUrls: true } });
```

## Fake driver for tests

`.../testing/fake` exports an in-memory driver with no vitest dependency:

```ts
import { createFakeStorage, FakeStorageDriver } from '@mohamedhabibwork/storagekit/testing/fake';
const storage = await createFakeStorage({ baseUrl, signedUrls: true, initialFiles, latencyMs });
// driver extras: seed(), reset(), failOnce(operation, error?), clearFailures(), files map
// registry path: registerStorageDriver('fake', (config) => new FakeStorageDriver(config as never));
```

## Runtime support

- Node.js >= 20 (primary; CI on 20/22), Bun >= 1.1, Deno >= 2 (npm compat).
- Every build runs `scripts/runtime-smoke.mjs` on all three runtimes in CI:
  local driver streams/listings/errors + custom-driver registry.
- Cloud SDKs are regular npm packages and work on all three; the local
  driver uses only `node:` built-ins that all three runtimes implement.

## Links

- npm: https://www.npmjs.com/package/@mohamedhabibwork/storagekit
- README with full examples: ./README.md (also in the published tarball)
- Per-driver guides: ./docs/local.md, ./docs/s3.md, ./docs/minio.md,
  ./docs/azure.md, ./docs/oracle.md, ./docs/rustfs.md, ./docs/gcs.md
  (shipped in the tarball)
- Framework upload recipes: ./docs/uploads.md (multer/Express/NestJS/Koa,
  Fastify, Hono, Elysia, Next.js, Remix, Nuxt, formidable, busboy)
- End-to-end examples: ./docs/examples.md
- Architecture & layer map: ./docs/architecture.md
- Custom driver guide: ./docs/custom-drivers.md