# Document Image Dewarping

Document image dewarping using text lines and line segments, implemented with classical computer-vision and numerical-optimization methods only. No machine learning model or model weights are required.

|Input|Debug|Output|
|:-:|:-:|:-:|
|<img width="3024" height="4032" alt="for_dewarping" src="https://github.com/user-attachments/assets/be2d27e9-cc0e-477b-bb8b-9e92e823d8f7" />|<img width="1650" height="2200" alt="for_dewarping_dewarped_debug" src="https://github.com/user-attachments/assets/5023e70c-8542-45b2-a079-fc243393ec9b" />|<img width="1596" height="2200" alt="for_dewarping_dewarped" src="https://github.com/user-attachments/assets/8436e695-f0c6-4656-8fee-b2af0dc115ca" />|

## Features

- Generalized cylindrical surface (GCS) model for curved pages
- Text-line extraction from connected components of both dark and light text
- Text and line-segment detection restricted to the detected paper region
- Structural text-line filtering to keep illustration strokes out of geometry fitting
- Automatic paper-boundary refinement for text-free page regions
- Joint optimization of surface shape, camera pose, and focal length
- Analytical Jacobian and a NumPy-only bounded Levenberg-Marquardt solver
- Iterative outlier removal for text lines and line segments
- EXIF-based focal-length initialization for JPEG images
- Optional real-world curved-object mode for bottles, cans, cups, and labels
- Configurable processing and rendering quality

## Requirements

- Python 3.12 or later
- [uv](https://docs.astral.sh/uv/) for the recommended setup

The only runtime dependencies are NumPy and OpenCV Headless. SciPy is not required; nonlinear least-squares optimization is implemented in `src/dewarping/lsq.py`.

## Installation

From a checkout of this repository, create the environment:

```bash
git clone https://github.com/PINTO0309/doc-dewarping.git && cd doc-dewarping
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync
source .venv/bin/activate
```

To display the available command-line options:

```bash
uv run dewarping --help
```

## Command-line usage

The positional arguments follow the original executable:

```text
dewarping INPUT OUTPUT [FLAG1] [FLAG2] [OPTIONS]
```

For example:

```bash
uv run dewarping input.jpg output.jpg 1 0
```

`FLAG1` and `FLAG2` were not documented by the original implementation. This package uses the following inferred behavior:

- `FLAG1`: `1` includes the line-segment terms from the full Kil 2017 method; `0` uses text lines only, similar to the Kim 2015 method. The default is `1`.
- `FLAG2`: `1` writes `<output_stem>_debug.png`. Final text inliers are green, rejected text candidates are gray, geometry-supported line segments are red, and sampled paper sides use distinct colors. The default is `0`.

The flags are optional, so the shortest invocation is:

```bash
uv run dewarping input.jpg output.jpg
```

### Quality presets

```bash
uv run dewarping input.jpg output.jpg 1 0 --quality high
```

| Preset | Processing side | Render grid | Interpolation | Max evaluations | Outlier iterations | Focal-length scan |
|---|---:|---:|---|---:|---:|---|
| `fast` | 1200 px | 65 | linear | 300 | 2 | No |
| `default` | 1600 px | 129 | linear | 600 | 3 | No |
| `high` | 2200 px | 257 | cubic | 1200 | 4 | Yes |

Individual settings can override a preset:

| Option | Description |
|---|---|
| `--proc-size N` | Maximum image side used for feature extraction and optimization |
| `--render-grid N` | Resolution per side of the inverse-mapping grid |
| `--interp {linear,cubic,lanczos}` | Output interpolation method |
| `--max-nfev N` | Maximum number of residual evaluations during optimization |
| `--outlier-iter N` | Number of outlier-removal iterations |
| `--no-exif` | Ignore an EXIF focal length |
| `--f-scan` | Try focal-length initializations corresponding to several fields of view when EXIF is unavailable |
| `--no-page-boundary` | Disable automatic refinement from a trusted, fully visible rectangular paper boundary |

For JPEG images containing `FocalLengthIn35mmFilm`, the value is used by default to initialize the focal length and narrow its optimization range. It is not fixed because EXIF focal lengths are not assumed to be exact.

### Curved-object scene mode

Use `--scene` to flatten labels or text on curved objects in real-world scenes:

```bash
uv run dewarping bottle.jpg bottle_flat.png 1 0 --scene
```

Scene mode detects a salient object using a SLIC-like segmentation and the graph-based manifold-ranking method of Yang et al., CVPR 2013. Text and line-segment extraction are restricted to the detected object. The surface model, optimization, and rendering stages are shared with document dewarping.

## Python API

The pipeline can also be called directly with an OpenCV image:

```python
import cv2

from dewarping.dewarp import dewarp_image
from dewarping.options import make_options

image = cv2.imread("input.jpg", cv2.IMREAD_COLOR)
if image is None:
    raise FileNotFoundError("input.jpg")

options = make_options("high")
result = dewarp_image(
    image,
    use_line_term=True,
    debug=False,
    opts=options,
)

if not cv2.imwrite("output.jpg", result.rectified):
    raise RuntimeError("Failed to write output.jpg")
```

`dewarp_image` returns a `DewarpOutput` containing:

- `rectified`: the dewarped image
- `debug_image`: the optional feature visualization
- `params`: the estimated surface and camera parameters, or no estimate when the input contains no usable features

## Tests

Run the NumPy/OpenCV unit tests, including page-boundary extraction and the
analytical boundary Jacobian check, with:

```bash
uv run python -m unittest discover -s tests -v
```

## How it works

1. Resize the image to a stable feature-extraction resolution.
2. Detect text connected components, assemble them into text lines and blocks, reject weak illustration-like line candidates when enough strong text is present, and detect line segments with LSD. With a trusted page boundary, retain only segments near strong text or the paper perimeter for geometry fitting.
3. Estimate a generalized cylindrical surface by minimizing text straightness, line spacing, block alignment, and line-segment orientation residuals.
4. Reclassify inliers and repeat bounded nonlinear least-squares optimization.
5. When all four paper sides are detected reliably, refine the text-first solution with page-boundary straightness constraints and retain it only if text alignment is preserved.
6. Apply a small residual deskew/warp correction and render the rectified image through an inverse map with OpenCV.

## Package structure

| Path | Purpose |
|---|---|
| `src/dewarping/textline.py` | Text binarization, connected components, text lines, and blocks |
| `src/dewarping/linesegs.py` | LSD line-segment detection and preprocessing |
| `src/dewarping/saliency.py` | Salient-object detection used by `--scene` |
| `src/dewarping/model.py` | GCS geometry, back-projection, and image mappings |
| `src/dewarping/optimize.py` | Cost function, alignment classification, and iterative optimization |
| `src/dewarping/lsq.py` | NumPy-only bounded Levenberg-Marquardt solver |
| `src/dewarping/dewarp.py` | End-to-end pipeline and rendering |
| `src/dewarping/options.py` | Quality presets and EXIF focal-length handling |
| `src/dewarping/cli.py` | Command-line interface |

## Limitations

- Pixel-identical output to the original executable is not guaranteed. In particular, its text-line extractor was replaced with a practical connected-component implementation because the original source is unavailable.
- The method depends on detectable text lines or line segments. Images with few usable structural features may be returned unchanged or produce a weak estimate.
- Highly irregular, folded, or non-cylindrical surfaces are outside the primary GCS model. A low-order residual warp can correct only small remaining errors.
- Boundary refinement requires a complete, approximately upright rectangular page. Cropped, torn, strongly non-convex, or scene-mode objects fall back to the text/line-segment result.
- Runtime depends on image complexity and the selected quality preset. `high` quality can be substantially slower, especially when it scans multiple focal lengths.

## License

This project is licensed under the [MIT License](LICENSE).

## Referenced

- Taeho Kil, Wonkyo Seo, Hyung Il Koo and Nam Ik Cho, "Robust Document Image Dewarping Using Text-Line and Line Segments", ICDAR 2017.
  ```bibtex
  @article{Kil2017RobustDI,
    title={Robust Document Image Dewarping Method Using Text-Lines and Line Segments},
    author={Tae Ho Kil and Wonkyo Seo and Hyung Il Koo and Nam Ik Cho},
    journal={2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)},
    year={2017},
    volume={01},
    pages={865-870},
    url={https://s-space.snu.ac.kr/handle/10371/136804}
  }
  ```
- DocRes: A Generalist Model Toward Unifying Document Image Restoration Tasks.
  ```bibtex
  @inproceedings{zhangdocres2024,
    Author={Jiaxin Zhang, Dezhi Peng, Chongyu Liu , Peirong Zhang and Lianwen Jin},
    Booktitle={In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition},
    Title={{DocRes: A Generalist Model Toward Unifying Document Image Restoration Tasks}},
    Year={2024}
  }
  ```