# Development

Here you'll find a contributing guide to get started with development.

## Environment

For local development, it is required to have Python 3.10 (or a later version) installed.

We use [uv](https://docs.astral.sh/uv/) for project management. Install it and set up your IDE accordingly.

We use [Poe the Poet](https://poethepoet.natn.io/) as a task runner, similar to npm scripts in `package.json`.
All tasks are defined in `pyproject.toml` under `[tool.poe.tasks]` and can be run with `uv run poe <task>`.

### Available tasks

| Task | Description |
| ---- | ----------- |
| `install-dev` | Install development dependencies |
| `check-code` | Run lint, type-check, and unit-tests |
| `lint` | Run linter |
| `format` | Fix lint issues and format code |
| `type-check` | Run type checker |
| `unit-tests` | Run unit tests |
| `unit-tests-cov` | Run unit tests with coverage |
| `e2e-templates-tests` | Run end-to-end template tests |
| `build-docs` | Build documentation website |
| `run-docs` | Run documentation website locally |
| `build` | Build package |
| `clean` | Remove build artifacts and clean caches |

## Dependencies

To install this package and its development dependencies, run:

```sh
uv run poe install-dev
```

## Code checking

To execute all code checking tools together, run:

```sh
uv run poe check-code
```

### Linting

We utilize [ruff](https://docs.astral.sh/ruff/) for linting, which analyzes code for potential issues and enforces consistent style. Refer to `pyproject.toml` for configuration details.

To run linting:

```sh
uv run poe lint
```

### Formatting

Our automated code formatting also leverages [ruff](https://docs.astral.sh/ruff/), ensuring uniform style and addressing fixable linting issues. Configuration specifics are outlined in `pyproject.toml`.

To run formatting:

```sh
uv run poe format
```

### Type checking

Type checking is handled by [ty](https://docs.astral.sh/ty/), verifying code against type annotations. Configuration settings can be found in `pyproject.toml`.

To run type checking:

```sh
uv run poe type-check
```

### Unit tests

We use [pytest](https://docs.pytest.org/) as a testing framework with many plugins. Check `pyproject.toml` for configuration details and installed plugins.

To run unit tests:

```sh
uv run poe unit-tests
```

To run unit tests with coverage report:

```sh
uv run poe unit-tests-cov
```

## End-to-end tests

Prerequisites:

- [apify-cli](https://docs.apify.com/cli/docs/installation) installed and available in `PATH`
- Set `APIFY_TEST_USER_API_TOKEN` to your [Apify API token](https://docs.apify.com/platform/integrations/api#api-token)

To run end-to-end tests:

```sh
uv run poe e2e-templates-tests
```

## Documentation

We follow the [Google docstring format](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) for code documentation. All user-facing classes and functions must be documented. Documentation standards are enforced using [Ruff](https://docs.astral.sh/ruff/).

Our API documentation is generated from these docstrings using [pydoc-markdown](https://pypi.org/project/pydoc-markdown/) with custom post-processing. Additional content is provided through markdown files in the `docs/` directory. The final documentation is rendered using [Docusaurus](https://docusaurus.io/) and published to GitHub Pages.

To run the documentation locally, ensure you have `Node.js` 20+ installed, then run:

```sh
uv run poe run-docs
```

### Linting the docs and website

Markdown content (this guide, `README.md`, and the `docs/` folder) is checked with
[markdownlint](https://github.com/DavidAnson/markdownlint). The Docusaurus website code is linted
with [oxlint](https://oxc.rs/) and formatted with [oxfmt](https://oxc.rs/). All of them run in CI.
To run them locally (requires Node.js 22.12 or newer and pnpm), from the `website/` directory:

```sh
pnpm lint          # lint Markdown and website code
pnpm lint:fix      # auto-fix both
pnpm format        # format the website code
```

Doc images are committed as optimized `.webp`. To convert a new image, run
`pnpm opt:images <path-to-image>` from the `website/` directory.

## Commits

We use [Conventional Commits](https://www.conventionalcommits.org/) format for commit messages. This convention is used to automatically determine version bumps during the release process.

### Available commit types

| Type | Description |
| ---- | ----------- |
| `feat` | A new feature |
| `fix` | A bug fix |
| `docs` | Documentation only changes |
| `style` | Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc) |
| `refactor` | A code change that neither fixes a bug nor adds a feature |
| `perf` | A code change that improves performance |
| `test` | Adding missing tests or correcting existing tests |
| `build` | Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm) |
| `ci` | Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs) |
| `chore` | Other changes that don't modify src or test files |
| `revert` | Reverts a previous commit |

## Release process

Publishing new versions to [PyPI](https://pypi.org/project/crawlee) is automated through GitHub Actions.

- **Beta releases**: On each commit to the master branch, a new beta release is automatically published. The version number is determined based on the latest release and conventional commits. The beta version suffix is incremented by 1 from the last beta release on PyPI.
- **Stable releases**: A stable version release may be created by triggering the `release` GitHub Actions workflow. The version number is determined based on the latest release and conventional commits (`auto` release type), or it may be overridden using the `custom` release type.

### Publishing to PyPI manually

1. **Do not do this unless absolutely necessary.** In all conceivable scenarios, you should use the `release` workflow instead.
2. **Make sure you know what you're doing.**
3. Update the version number by modifying the `version` field under `project` in `pyproject.toml`:

    ```toml
    [project]
    name = "crawlee"
    version = "x.z.y"
    ```

4. Build the package:

    ```sh
    uv run poe build
    ```

5. Upload to PyPI:

    ```sh
    uv publish --token YOUR_API_TOKEN
    ```