[![Test Status](https://github.com/opendatacube/odc-tools/actions/workflows/main.yml/badge.svg)](https://github.com/opendatacube/odc-tools/actions/workflows/main.yml) [![codecov](https://codecov.io/gh/opendatacube/odc-tools/branch/develop/graph/badge.svg?token=PovpVLRFwn)](https://codecov.io/gh/opendatacube/odc-tools) DEA Prototype Code ================== This repository provides developmental [libraries](https://github.com/opendatacube/odc-tools/tree/develop/libs) and [CLI tools](https://github.com/opendatacube/odc-tools/tree/develop/apps) for Open Datacube. - AWS S3 tools - CLIs for using ODC data from AWS S3 and SQS - Utilities for data visualizations in notebooks - Experiments on optimising Rasterio usage on AWS S3 Full list of libraries, and install instructions: - `odc.ui` tools for data visualization in notebook/lab - `odc.io` common IO utilities, used by apps mainly - `odc-cloud[ASYNC,AZURE,THREDDS]` cloud crawling support package - `odc.aws` AWS/S3 utilities, used by apps mainly - `odc.aio` faster concurrent fetching from S3 with async, used by apps `odc-cloud[ASYNC]` - `odc.{thredds,azure}` internal libs for cloud IO `odc-cloud[THREDDS,AZURE]` ## Promoted to their own repositories - `odc.stats` large scale processing framework (Moved to [odc-stats](http://github.com/opendatacube/odc-stats)) - `odc.stac` STAC to ODC conversion tools (Moved to [odc-stac](https://github.com/opendatacube/odc-stac)) - `odc.dscache` experimental key-value store where `key=UUID`, `value=Dataset` (moved to [odc-dscache](https://github.com/opendatacube/odc-dscache)) Installation ============ Libraries and applications in this repository are published to PyPI, and can be installed \ with `pip` like so: ``` pip install \ odc-ui \ odc-io \ odc-cloud[ASYNC] ``` For Conda Users --------------- Some **odc-tools** are available via `conda` from the `conda-forge` channel. ``` conda install -c conda-forge odc-apps-dc-tools odc-io odc-cloud ``` Cloud Tools =========== Installation ------------ Cloud tools depend on the `aiobotocore` package, which depends on specific versions of `botocore`. Another package we use, `boto3`, also depends on specific versions of `botocore`. As a result, having both `aiobotocore` and `boto3` in one environment can be a bit tricky. The way to solve this is to install `aiobotocore[awscli,boto3]` before anything else, which will install compatible versions of `boto3` and `awscli` into the environment. ``` pip install -U "aiobotocore[awscli,boto3]==1.3.3" # OR for conda setups conda install "aiobotocore==1.3.3" boto3 awscli ``` 1. For cloud (AWS only) ``` pip install odc-apps-dc-tools ``` 2. For cloud (AZURE, GCP, THREDDS and AWS) ``` pip install odc-apps-dc-tools[AZURE,GCP,THREDDS] ``` 2. For `dc-index-from-tar` (indexing to datacube from tar archive) ``` pip install odc-apps-dc-tools ``` Apps ---- 1. `s3-find` list S3 bucket with wildcard 2. `s3-to-tar` fetch documents from S3 and dump them to a tar archive 3. `gs-to-tar` search GS for documents and dump them to a tar archive 4. `dc-index-from-tar` read yaml documents from a tar archive and add them to datacube Example: ```bash #!/bin/bash s3_src='s3://dea-public-data/L2/sentinel-2-nrt/**/*.yaml' s3-find "${s3_src}" | \ s3-to-tar | \ dc-index-from-tar --env s2 --ignore-lineage ``` Fastest way to list regularly placed files is to use fixed depth listing: ```bash #!/bin/bash # only works when your metadata is same depth and has fixed file name s3_src='s3://dea-public-data/L2/sentinel-2-nrt/S2MSIARD/*/*/ARD-METADATA.yaml' s3-find --skip-check "${s3_src}" | \ s3-to-tar | \ dc-index-from-tar --env s2 --ignore-lineage ``` When using Google Storage: ```bash #!/bin/bash # Google Storage support gs-to-tar --bucket data.deadev.com --prefix mangrove_cover dc-index-from-tar --protocol gs --env mangroves --ignore-lineage metadata.tar.gz ``` Local Development ================= The following steps are used in the GitHub Actions workflow `main.yml` ```bash # install all packages in edit mode ./scripts/dev-install.sh --extra tests # setup database for testing ./scripts/setup-test-db.sh # run test echo "Running Tests" uv run pytest --cov=. \ --cov-report=html \ --cov-report=xml:coverage.xml \ --timeout=30 \ libs apps ``` Release Process =============== 1. Manually edit `{lib,app}/{pkg}/odc/{pkg}/_version.py` file to increase version number 2. Merge changes to the `develop` branch via a Pull Request 3. Fast-forward the `pypi/publish` branch to match `develop` 4. Push to GitHub Steps 3 and 4 can be done by an authorized user with `./scripts/sync-publish-branch.sh` script. Publishing to [PyPi](https://pypi.org/) happens automatically when changes are pushed to the protected `pypi/publish` branch. Only members of [Open Datacube Admins](https://github.com/orgs/opendatacube/teams/admins) group have the permission to push to this branch.