DL4S provides a high-level API for many accelerated operations common in neural networks and deep learning.
It furthermore has automatic differentiation builtin, which allows you to create and train neural networks without needing to manually
implement backpropagation - without needing a special Swift toolchain.
Features include implementations for many basic binary and unary operators,
broadcasting, matrix operations, convolutional and recurrent neural networks,
commonly used optimizers, second derivatives and much more.
DL4S provides implementations for common network architectures, such as VGG, AlexNet, ResNet and Transformers.
While its primary purpose is deep learning and optimization, DL4S can be used as a library for vectorized mathematical operations like numpy.
[Read the full documentation](https://palle-k.github.io/DL4S/)
## Overview
1. [Installation](#installation)
2. [Features](#features)
1. Layers
2. Optimizers
3. Losses
4. Tensor Operations
5. Engines
6. Architectures
3. [Examples](#examples)
## Installation
### iOS / tvOS / macOS
1. In Xcode, select "File" > "Swift Packages" > "Add Package Dependency"
2. Enter `https://github.com/palle-k/DL4S.git` into the Package URL field and click "Next".
3. Select "Branch", "master" and click "Next".
4. Enable the Package Product DL4S, your app in the "Add to Target" column and click "Next".
**Note**: Installation via CocoaPods is no longer supported for newer versions.
### Swift Package
Add the dependency to your `Package.swift` file:
```swift
.package(url: "https://github.com/palle-k/DL4S.git", .branch("master"))
```
Then add `DL4S` as a dependency to your target:
```swift
.target(name: "MyPackage", dependencies: ["DL4S"])
```
#### MKL / IPP / OpenMP Support
On Apple devices, DL4S uses the vectorized functions of the builtin Accelerate framework by default.
On x86_64 Linux (Intel / AMD), DL4S can use the Intel oneAPI Math Kernel Library (MKL) and Integrated Performance Primitives (IPP) through the `MKL` package trait (off by default).
If no acceleration library is available, a slower generic fallback is used.
To use MKL in a package that depends on DL4S, enable the trait in the dependency declaration:
```swift
.package(url: "https://github.com/palle-k/DL4S.git", branch: "master", traits: ["MKL"])
```
Make sure that oneAPI MKL and IPP are installed from the Intel apt repository following the ([installation guide](https://www.intel.com/content/www/us/en/docs/oneapi-toolkit/installation-guide-linux/latest/install-oneapi-toolkit-with-apt.html#INSTALL-ONEAPI-TOOLKIT-WITH-APT)).
Then export `PKG_CONFIG_PATH` and `CPATH` so that the build process discovers MKL and IPP:
```bash
# Sets PKG_CONFIG_PATH
source /opt/intel/oneapi/setvars.sh
export CPATH=${IPPROOT}/include${CPATH:+:${CPATH}}
swift build -c release --traits MKL
swift test --traits MKL
```
### TensorBoard Support
[DL4S-Tensorboard](https://github.com/palle-k/DL4S-Tensorboard) provides a summary writer that can write tensorboard compatible logs.
### LLDB Extension
DL4S includes a LLDB python script that provides custom descriptions for Tensors (`util/debugger_support/tensor.py`).
To use enhanced summaries, execute `command script import /path/to/DL4S/util/debugger_support/tensor.py`
either directly in LLDB or add the command to your `~/.lldbinit` file.
Then you can use the `print` or `frame variable` commands to print human-readable descriptions of tensors.
## Features
Layers
Core:
- [x] Convolution
- [x] Transposed Convolution
- [x] Dense/Linear/Fully Connected
- [x] LSTM
- [x] Gated Recurrent Unit (GRU)
- [x] Vanilla RNN
- [x] Embedding
- [x] Multi-head Attention
- [x] Transformer Block
Pooling:
- [x] Max Pooling
- [x] Average Pooling
- [x] Adaptive Max Pooling
- [x] Adaptive Average Pooling
Norm:
- [x] Batch Norm
- [x] Layer Norm
Utility:
- [x] Bidirectional RNNs
- [x] Sequential
- [x] Lambda
- [x] Dropout
- [x] Lambda
Activation:
- [x] Relu
- [x] LeakyRelu
- [x] Gelu
- [x] Tanh
- [x] Sigmoid
- [x] Softmax
- [x] Log Softmax
- [x] Dropout
- [x] Gelu
- [x] Swish
- [x] Mish
- [x] LiSHT
Transformer:
- [x] Positional Encoding
- [x] Scaled Dot Product Attention
- [x] Multihead Attention
- [x] Pointwise Feed Forward
- [x] Transformer Encoder Block
- [x] Transformer Decoder Block
Optimizers
- [x] SGD
- [x] Momentum
- [x] Adam
- [x] AMSGrad
- [x] AdaGrad
- [x] AdaDelta
- [x] RMSProp
Losses
- [x] Binary Cross-Entropy
- [x] Categorical Cross-Entropy
- [x] Negative Log Likelihood (NLL Loss)
- [x] MSE
- [x] L1 & L2 regularization
Tensor Operations
Behavior of broadcast operations is consistent with numpy rules.
- [x] broadcast-add
- [x] broadcast-sub
- [x] broadcast-mul
- [x] broadcast-div
- [x] matmul
- [x] neg
- [x] exp
- [x] pow
- [x] log
- [x] sqrt
- [x] sin
- [x] cos
- [x] tan
- [x] tanh
- [x] sum
- [x] max
- [x] relu
- [x] leaky relu
- [x] gelu
- [x] elu
- [x] elementwise min
- [x] elementwise max
- [x] reduce sum
- [x] reduce max
- [x] scatter
- [x] gather
- [x] conv2d
- [x] transposed conv2d
- [x] max pool
- [x] avg pool
- [x] subscript
- [x] subscript range
- [x] transpose
- [x] axis permute
- [x] reverse
- [x] im2col
- [x] col2im
- [x] stack / concat
- [x] swish activation
- [x] mish activation
- [x] lisht activation
- [x] diagonal matrix generation
- [x] diagonal extraction
- [x] band matrix generation
Engines
- [x] CPU (Accelerate framework for Apple Devices)
- [x] CPU (Intel Math Kernel Library and Integrated Performance Primitives)
- [x] CPU (Generic)
- [ ] GPU (ArrayFire: OpenCL, CUDA)
For an experimental, early stage GPU accelerated version, check out `feature/arrayfire`.
Architectures
Default implementations are provided for the following architectures:
- [x] ResNet18
- [x] VGG (11, 13, 16, 19)
- [x] AlexNet
- [x] Transformer
## Examples
Some high level examples have been implemented in other repositories:
- [Neural Machine Translation](https://github.com/palle-k/Seq2Seq-DL4S) based on seq2seq with Attention
- [Generative Adversarial Networks](https://github.com/palle-k/DL4S-WGAN-GP) - Wasserstein GAN with Gradient Penalty (WGAN-GP)
- [Reinforcement Learning](https://github.com/palle-k/REINFORCE-DL4S) - Trains an agent to find the exit in a 2D grid world.
### Arithmetic & Differentiation
DL4S provides a high-level interface to many vectorized operations on tensors.
```swift
let a = Tensor([[1,2],[3,4],[5,6]], requiresGradient: true)
let prod = a.transposed().matrixMultipled(with: a)
let s = prod.reduceSum()
let l = log(s)
print(l) // 5.1873856
```
When a tensor is marked to require a gradient, a compute graph will be captured.
The graph stores all operations, which use that tensor directly or indirectly as an operand.
It is then possible to backpropagate through that graph using the `gradients(of:)` function:
```swift
// Backpropagate
let dl_da = l.gradients(of: [a])[0]
print(dl_da)
/*
[[0.034, 0.034]
[0.078, 0.078]
[0.123, 0.123]]
*/
```
#### Second derivatives
The operations used during backpropagation are themselves differentiable.
Therefore, second derivatives can be computed by computing the gradient of the gradient.
When higher order derivatives are required, the compute graph of the backwards pass has to be explicitly retained.
```swift
let t = Tensor([1,2,3,4], requiresGradient: true)
let result = t * t * t
print(result) // [1, 8, 27, 64]
let grad = result.gradients(of: [t], retainBackwardsGraph: true)[0]
print(grad) // [3, 12, 27, 48]
let secondGrad = grad.gradients(of: [t], retainBackwardsGraph: true)[0]
print(secondGrad) // [6, 12, 18, 24]
let thirdGrad = secondGrad.gradients(of: [t])[0]
print(thirdGrad) // [6, 6, 6, 6]
```
### Convolutional Networks
Example for MNIST classification
```swift
// Input must be batchSizex1x28x28
var model = Sequential {
Convolution2D(inputChannels: 1, outputChannels: 6, kernelSize: (5, 5))
Relu()
MaxPool2D(windowSize: 2, stride: 2)
Convolution2D(inputChannels: 6, outputChannels: 16, kernelSize: (5, 5))
Relu()
MaxPool2D(windowSize: 2, stride: 2)
Flatten()
Dense(inputSize: 256, outputSize: 120)
Relu()
Dense(inputSize: 120, outputSize: 10)
LogSoftmax()
}
var optimizer = Adam(learningRate: 0.001)
// Single iteration of minibatch gradient descent
let batch: Tensor = ... // shape: [batchSize, 1, 28, 28]
let y_true: Tensor = ... // shape: [batchSize]
let pred = model(batch)
let loss = categoricalNegativeLogLikelihood(expected: y_true, actual: pred)
model.update { parameters in
optimizer.update(¶meters, along: loss.gradients(of: parameters))
}
```
### Recurrent Networks
Example for MNIST classification
The Gated Reccurent Unit scans the image from top to bottom and uses the final hidden state for classification.
```swift
var model = Sequential {
GRU(inputSize: 28, hiddenSize: 128, direction: .forward)
Lambda.Outputs, Tensor, Float, CPU> { inputs in
inputs.0
}
Dense(inputSize: 128, outputSize: 10)
LogSoftmax()
}
var optimizer = Adam(learningRate: 0.001)
let batch: Tensor = ... // shape: [batchSize, 28, 28]
let y_true: Tensor = ... // shape: [batchSize]
let x = batch.permuted(to: 1, 0, 2) // Swap first and second axis
let pred = model(x)
let loss = categoricalNegativeLogLikelihood(expected: y_true, actual: pred)
model.update { parameters in
optimizer.update(¶meters, along: loss.gradients(of: parameters))
}
```
### Custom Layers
Layers and entire models are declared with the `@Layer` macro. This auto-synthesizes any necessary utilities for model updates.
Parameters that are not updated must be declared as `let` or `@Frozen`.
```swift
@Layer
struct Classifier {
typealias Element = Float
typealias Device = CPU
@Frozen var backbone: ResNet18
var head: Dense
func callAsFunction(_ inputs: Tensor) -> Tensor {
head(backbone(inputs))
}
}
var model = Classifier(backbone: pretrained, head: Dense(inputSize: 512, outputSize: 10))
print(model.weightPaths) // ["head.weights", "head.bias"]
model.head.freeze() // stops the training of the head
model.head.unfreeze() // makes it trainable again
```
To get fine-grained control over the behavior of a layer, conformance to the `LayerType` protocol may be written by hand.
### Model Loading and Saving
DL4S uses the [safetensors](https://huggingface.co/docs/safetensors) format.
```swift
try SafetensorsEncoder().encode(model, to: URL(filePath: "model.safetensors"))
var restoredModel = MyModel()
try SafetensorsDecoder().load(into: &restoredModel, from: URL(filePath: "model.safetensors"))
```
Loading / persisting sharded safetensors is supported:
```swift
let encoder = SafetensorsEncoder(options: .init(sharding: .maximumBytes(2 << 30)))
try encoder.encode(transformer, to: URL(filePath: "checkpoint/", directoryHint: .isDirectory))
try SafetensorsDecoder().load(into: &transformer, from: URL(filePath: "checkpoint/", directoryHint: .isDirectory))
```
Optimizers store their state (excluding learning rates, momentums, and other hyperparameters) in safetensors as well.
```swift
try SafetensorsEncoder().encode(optimizer, to: URL(filePath: "optimizer.safetensors"))
var restoredOptimizer = Adam(learningRate: 0.001)
try SafetensorsDecoder().load(into: &restoredOptimizer, from: URL(filePath: "optimizer.safetensors"))
```