{ "cells": [ { "cell_type": "markdown", "source": [ "# Analyzing Bird Audio\n", "\n", "- toc: false\n", "- badges: true\n", "- comments: true\n", "- category: datascience\n", "- description: An analysis of bird audio from Xeno Canto to determine if it was a song or a call.\n", "- image: images/copied_from_nb/birds.png" ], "metadata": { "tags": [], "cell_id": "00000-b98ee5de-4796-4f03-9cd5-4523c8709e6c", "deepnote_cell_type": "markdown" } }, { "cell_type": "markdown", "source": [ "![Analyzing Bird Audio Cover Photo](birds.png)\n", "\n", "**Authors**: [Adithya Balaji](https://www.linkedin.com/in/adithyabsk/),\n", "[Malika Khurana](https://www.linkedin.com/in/malikakhurana/)\n", "\n", "> This report details our process for analyzing bird audio, with some snippets of code. You can find the full project\n", "> repo [on github](https://github.com/adithyabsk/bird_audio)." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "metadata": { "tags": [], "cell_id": "00000-3155da5a-5d99-4ec4-b125-8e5d4eecb1de", "deepnote_to_be_reexecuted": false, "source_hash": "8886d04a", "execution_start": 1621312147533, "execution_millis": 7, "deepnote_cell_type": "code" }, "source": [ "import IPython.display as ipd\n", "import requests" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "tags": [], "cell_id": "00002-36821c94-74e6-4e9a-8cc6-1e620079af98", "deepnote_to_be_reexecuted": false, "source_hash": "d2fb25ab", "execution_start": 1621312147545, "execution_millis": 15, "deepnote_cell_type": "code" }, "source": [ "# hide_input\n", "def display_code_block(url, start_line=1, end_line=-1):\n", " raw_audio = requests.get(url)\n", " text = \"\\n\".join(raw_audio.content.decode().split(\"\\n\")[start_line - 1 : end_line])\n", " ipd.display(\n", " ipd.Markdown(\n", " f\"\"\"\n", "```python\n", "{text}\n", "```\n", "\"\"\"\n", " )\n", " )" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "## Introduction\n", "\n", "We aim to accurately classify bird sounds as songs or calls. We used 3 different approaches and models based on\n", "recording metadata, the audio data itself, and spectrogram images of the recording to perform this classification task.\n", "\n", "
\n", "
\n", "\"Pipeline\n", "
\n", "
\n", "\n", "### Motivation\n", "\n", "The primary motivation to address this problem is to make it easier for scientists to collect data on bird populations\n", "and verify community-sourced labels.\n", "\n", "The other motivation is more open-ended: to understand the \"hidden\" insights in bird sounds. Bird calls reveal regional\n", "dialects, a sense of humor, information about predators in the area, indicators of ecosystem health—and inevitably also\n", "the threat on their ecosystems posed by human activity. Through the process of exploring bird call audio data, we hope\n", "we can build towards better understanding the impacts of the sounds produced by humans and become better listeners.\n", "\n", "### Songs vs Calls\n", "\n", "Bird sounds have a variety of different dimensions, but one of the first levels of categorizing bird sounds is\n", "classifying them as a song or a call, as each have distinct functions and reveal different aspects of the birds’\n", "ecology ([1](https://www.audubon.org/news/a-beginners-guide-common-bird-sounds-and-what-they-mean),\n", "[2](https://www.youtube.com/watch?v=4_1zIwEENt8))." ], "metadata": { "cell_id": "00001-bb8420c6-c1fe-4532-aa2c-b93684e8d62d", "tags": [], "deepnote_cell_type": "markdown" } }, { "cell_type": "markdown", "source": [ "#### Songs\n", "Songs tend to be longer, more melodic, and used for marking territory and attracting mates. Birds' song repertoire and\n", "song rate can indicate their health and the quality of their habitat, including pollutant levels and plant diversity\n", "([3](https://en.wikipedia.org/wiki/Bird_vocalization#Function), [4](https://www.jstor.org/stable/20062442),\n", "[5](https://www.fs.usda.gov/treesearch/pubs/46856)).\n" ], "metadata": { "tags": [], "cell_id": "00002-103f41dc-76c7-4eaf-b8a6-f926a930b69a", "deepnote_cell_type": "markdown" } }, { "cell_type": "code", "metadata": { "tags": [], "cell_id": "00002-cd34809b-a692-480f-80ba-9dc5f863eeea", "deepnote_to_be_reexecuted": false, "source_hash": "51ec5df", "execution_start": 1621312147564, "execution_millis": 29, "deepnote_output_heights": [ 56 ], "deepnote_cell_type": "code" }, "source": [ "# song sparrow - song\n", "ipd.Audio(\n", " requests.get(\n", " \"https://github.com/adithyabsk/bird_audio/blob/main/notebooks/assets/574080.mp3?raw=true\"\n", " ).content\n", ")" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "execution_count": 3, "data": { "text/plain": "", "text/html": "\n \n " }, "metadata": {} } ] }, { "cell_type": "markdown", "source": [ "#### Calls\n", "Calls are shorter than songs, and perform a wider range of functions like signalling food, maintaining social cohesion\n", "and contact, coordinating flight, resolving conflicts, and sounding alarms (distress, mobbing, hawk alarms) ([6](https://doi.org/10.1196/annals.1298.034)).\n", "Bird alarm calls can be understood and passed along across species, and have been found to encode information about the\n", "size and threat of a potential predator, so birds can respond accordingly - i.e. more intense mobbing for a higher\n", "threat ([7](https://www.nationalgeographic.com/animals/article/nuthatches-chickadees-communication-danger),\n", "[8](https://doi.org/10.1126/science.1108841)). Alarm calls can also give scientists an estimate of the number of\n", "predators in an area." ], "metadata": { "tags": [], "cell_id": "00004-e80cb207-d18b-40d6-8269-9db79d96b116", "deepnote_cell_type": "markdown" } }, { "cell_type": "code", "metadata": { "tags": [], "cell_id": "00003-0d793527-be8e-4939-a9e7-edd4df025c6e", "deepnote_to_be_reexecuted": false, "source_hash": "16381a37", "execution_start": 1621312147587, "execution_millis": 22, "deepnote_output_heights": [ 56 ], "deepnote_cell_type": "code" }, "source": [ "# song sparrow - call\n", "ipd.Audio(requests.get(\"https://github.com/adithyabsk/bird_audio/blob/main/notebooks/assets/585148.mp3?raw=true\".content)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "execution_count": 4, "data": { "text/plain": "", "text/html": "\n \n " }, "metadata": {} } ] }, { "cell_type": "markdown", "source": [ "## Related Work\n", "**Allometry of Alarm Calls: Black-Capped Chickadees Encode Information About Predator Size** ([8](https://doi.org/10.1126/science.1108841))\n", "The number of D-notes in chickadee alarm mobbing calls varies indirectly with the size of predator.\n", "\n", "**Gender identification using acoustic analysis in birds without external sexual dimorphism** ([9](https://doi.org/10.1186/s40657-015-0033-y))\n", "Bird sounds were analyzed to classify gender. Important acoustic features were: fundamental frequency (mean, max,\n", "count), note duration, syllable count and spacing, and amplitude modulation.\n", "\n", "**Regional dialects have been discovered among many bird species, and the Yellowhammer is a great example** ([10](http://www.yellowhammers.net/about), [11](https://doi.org/10.1093/beheco/arz114))\n", "Yellowhammer bird sounds in the Czech Republic and UK were studied to identify regional dialects, which differed in\n", "frequency and length of final syllables." ], "metadata": { "cell_id": "00002-0e69944b-7b03-4fc1-afaf-fb18c9b9839b", "tags": [], "deepnote_cell_type": "markdown" } }, { "cell_type": "markdown", "source": [ "## DVC\n", "\n", "[Data Version Control (DVC)](https://dvc.org/) is a useful tool for data science projects. You can think of it like git\n", "but for data. We built out our pipeline first in jupyter notebooks, and then in DVC, making it easy to change parameters\n", "and run the full pipeline from one place.\n", "\n", "
\n", "Note: Due to the size of the datasets, we chose not to include inline Jupyter snippets of code processing real\n", "data and instead opted to present only the outputs of the DVC scripts. (Python files, not notebooks)\n", "
" ], "metadata": { "tags": [], "cell_id": "00008-f5cecd3b-b746-4567-a0e6-25e3106a045f", "deepnote_cell_type": "markdown" } }, { "cell_type": "markdown", "source": [ "## Collecting Data\n", "\n", "For our analysis, we used audio files and metadata from [xeno-canto.org]. Xeno-canto (XC) is a website for collecting\n", "and sharing audio recordings of birds. Recordings and identifications on XC are sourced from the community (anyone can\n", "join).\n", "\n", "
\n", "
\n", "\"Xeno\n", "
\n", "
\n", "\n", "XC has a [straightforward API](https://www.xeno-canto.org/explore/api) that allows us to make RESTful queries, and\n", "specify a number of [filter parameters](https://www.xeno-canto.org/help/search) including country, species, recording\n", "quality, and duration. We used the XC API to get metadata and IDs for all recordings in the United States, and saved the\n", "JSON payload as a dataframe and csv. Below we see the main snippet of code from the DVC step that parallelizes data\n", "collection from XC." ], "metadata": { "cell_id": "00001-2e28dd94-127d-483c-9d7d-dc9b6fe29aaf", "tags": [], "deepnote_cell_type": "markdown" } }, { "cell_type": "code", "metadata": { "tags": [], "cell_id": "00011-00d2df97-550a-4a22-aee9-76e828027cba", "deepnote_to_be_reexecuted": false, "source_hash": "9987edd", "execution_start": 1621312147602, "execution_millis": 145, "deepnote_output_heights": [ 489.4375 ], "deepnote_cell_type": "code" }, "source": [ "# hide_input\n", "display_code_block(\n", " \"https://raw.githubusercontent.com/adithyabsk/bird_audio/main/pracds_final/data/build_meta.py\",\n", " 45,\n", " 80,\n", ")" ], "execution_count": null, "outputs": [ { "data": { "text/plain": "", "text/markdown": "\n```python\ndef search_recordings(**query_params) -> List[JSON]:\n \"\"\"Search for recordings using the Xeno Canto search API\n\n The keys in the return dictionaries are specified in the API docs\n\n https://www.xeno-canto.org/explore/api\n\n\n Args:\n **query_params: A dictionary with query and/or page as keys with values as specified in the\n API docs.\n\n Returns:\n A list of recording information (list of dictionaries)\n\n \"\"\"\n url = (BASE_URL / \"recordings\").with_query(query_params)\n resp = requests.get(str(url))\n if resp.status_code == 200:\n resp_json = resp.json()\n num_pages = resp_json[\"numPages\"]\n recordings = resp_json[\"recordings\"]\n if num_pages > 1:\n page_urls = [url.update_query(page=p) for p in range(2, num_pages + 1)]\n with ProcessPoolExecutor(max_workers=10) as ppe:\n recordings.extend(\n itertools.chain(\n *tqdm(\n ppe.map(get_page_recordings, page_urls), total=num_pages - 1\n )\n )\n )\n else:\n raise Exception(f\"Request failed with status code: {resp.status_code}\")\n\n return recordings\n```\n" }, "metadata": {}, "output_type": "display_data" } ] }, { "cell_type": "markdown", "source": [ "## Filtering & Labeling\n", "Through our DVC pipeline, we further filtered by the top 220 unique species, recordings under 20 seconds, recording\n", "quality A or B, and recordings with spectrograms available on XC. This reduced our dataset size from ~60,000 to get a\n", "dataframe of 5,800 recordings. We created labels (1 for call, 0 for song) by parsing the 'type' column of the df.\n", "\n", "The following scripts handle that process:\n", "\n", "1. [build_filter.py](https://github.com/adithyabsk/bird_audio/blob/main/pracds_final/data/build_filter.py)\n", "1. [build_song_vs_call.py](https://github.com/adithyabsk/bird_audio/blob/main/pracds_final/data/build_song_vs_call.py)\n", "1. [proc_svc_meta.py](https://github.com/adithyabsk/bird_audio/blob/main/pracds_final/features/proc_svc_meta.py)" ], "metadata": { "cell_id": "00004-5c180cbe-59b0-4882-aab3-66dea2ba570d", "deepnote_cell_type": "markdown" } }, { "cell_type": "markdown", "source": [ "## Exploring & Visualizing Data\n", "\n", "With our dataset assembled, we began exploring it visually. A distribution of recordings by genus, with song-call splits\n", "shows that the genus most represented in the dataset are warblers (*Setophaga*) with many more songs than call\n", "recordings. We can also see that, as expected, woodpeckers (*Melanerpes*), jays, magpies, and crows (*Cyanocitta*,\n", "*Corvus*) have almost no song recordings in the dataset.\n", "\n", "
\n", "
\n", "\"Count\n", "
\n", "
\n", "\n", "A map of recording density shows the regions most represented in the dataset which are, unsurprisingly, bird watching\n", "hot spots.\n", "\n", "
\n", "
\n", "\"Observation\n", "
\n", "
\n", "\n", "Given our domain knowledge that songs serve an important function in mating, we expected to see a higher proportion of\n", "songs in the spring, which is confirmed by the data.\n", "\n", "
\n", "
\n", "\"Song\n", "
\n", "
" ], "metadata": { "cell_id": "00005-12c735bf-8e1a-4347-8dd5-2328bc3e5475", "deepnote_cell_type": "markdown" } }, { "cell_type": "markdown", "source": [ "## Metadata Classification Model\n", "\n", "In our first model, we used the tabular metadata from XC entries to train a Gradient Boosted Decision Tree (GBDT) model using\n", "[XGBoost](https://xgboost.readthedocs.io/en/latest/). XGBoost, is a particular Python implementation of GBDTs that is\n", "designed to work on large amounts of data.\n", "\n", "
\n", "\n", "\"XGBoost\n", "\n", "
\n", "\n", "We used the genus, species, English name, and location (latitude and longitude) from XC metadata. These features were\n", "then all mapped and imputed using sklearn transformers to one-hot encoded form apart from latitude, longitude, and time\n", "(all mapped using standard or min-max scaling, and time features transformed with a sin function). We can see 10 rows of\n", "unprocessed data in the HTML table below." ], "metadata": { "cell_id": "00006-0f0fd362-cef4-4dc3-a785-6e8caaa53266", "deepnote_cell_type": "markdown" } }, { "cell_type": "code", "metadata": { "tags": [], "cell_id": "00015-9d18140d-330b-4490-9f71-eeeeef9898c6", "deepnote_to_be_reexecuted": false, "source_hash": "5be256b", "execution_start": 1621312147752, "execution_millis": 9, "deepnote_output_heights": [ 1 ], "deepnote_cell_type": "code" }, "source": [ "# hide_input\n", "ipd.display(\n", " ipd.HTML(\n", " \"
df_indexidgenspsspenreccntloclatlngalttypeurlfilefile-namesonolicqlengthtimedateuploadedalsormkbird-seenplayback-usedpredgenderagemonthdayhourminute
096454911BrantacanadensisNaNCanada GooseBruce LagerquistUnited StatesSedro-Woolley, Skagit County, Washington48.5237-122.018530call//www.xeno-canto.org/454911//www.xeno-canto.org/454911/downloadXC454911-190202_02 Canadian Geese.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/JHFICMRVUX/ffts/XC454911-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/JHFICMRVUX/ffts/XC454911-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/JHFICMRVUX/ffts/XC454911-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/JHFICMRVUX/ffts/XC454911-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:171900-01-01 11:30:002019-02-022019-02-04['Cygnus buccinator']Mixed flock of Trumpeter Swans and Canada Geese feeding in an agricultural field. Recording of Swan's here XC454910yesno1NaNNaN2.02.011.030.0
197418340BrantacanadensisNaNCanada GooseSue RiffeUnited StatesAu Sable SF - Big Creek Rd, Michigan44.0185-83.7560180song//www.xeno-canto.org/418340//www.xeno-canto.org/418340/downloadXC418340-Canada Goose on 5.11.18 at Au Sable SF MI at 11.20 for .14 _0908 .mp3{'small': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC418340-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC418340-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC418340-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC418340-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:141900-01-01 11:20:002018-05-112018-06-03['Agelaius phoeniceus']Natural vocalizationyesno0NaNNaN5.011.011.020.0
2107291051BrantacanadensisNaNCanada GooseEric HoughUnited StatesSan Juan River, Cottonwood Day-Use Area, Navajo Lake State Park, San Juan County, New Mexico36.8068-107.67891800call//www.xeno-canto.org/291051//www.xeno-canto.org/291051/downloadXC291051-CANG_11515_1730_SanJuanRiver-NavajoDam.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/BCFUZDOSJZ/ffts/XC291051-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/BCFUZDOSJZ/ffts/XC291051-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/BCFUZDOSJZ/ffts/XC291051-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/BCFUZDOSJZ/ffts/XC291051-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:151900-01-01 17:30:002015-11-152015-11-18['']Flock calling while flying over at dusk. Amplification, low and high pass filters used in Audacity.yesno1NaNNaN11.015.017.030.0
3108283618BrantacanadensisNaNCanada GooseGarrett MacDonaldUnited StatesBeluga--North Bog, Kenai Peninsula Borough, Alaska61.2089-151.010340call, flight call//www.xeno-canto.org/283618//www.xeno-canto.org/283618/downloadXC283618-LS100466.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/CDHIAMGTRT/ffts/XC283618-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/CDHIAMGTRT/ffts/XC283618-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/CDHIAMGTRT/ffts/XC283618-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/CDHIAMGTRT/ffts/XC283618-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:101900-01-01 11:00:002015-05-202015-10-03['']Natural vocalizations from a pair of birds in flight. Recording not modified.yesno1NaNNaN5.020.011.00.0
4110209702BrantacanadensisNaNCanada GooseAlbert @ Max lastukhinUnited StatesOyster Bay (near Lattingtown), Nassau, New York40.8881-73.585110call//www.xeno-canto.org/209702//www.xeno-canto.org/209702/downloadXC209702-Poecile atricapillus Dec_27,_2014,_4_05_PM,C1.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/LELYWQKUZX/ffts/XC209702-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/LELYWQKUZX/ffts/XC209702-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/LELYWQKUZX/ffts/XC209702-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/LELYWQKUZX/ffts/XC209702-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:111900-01-01 16:00:002014-12-272015-01-09['Poecile atricapillus']NaNyesno1NaNNaN12.027.016.00.0
5118165398BrantacanadensisparvipesCanada GooseTed FloydUnited StatesBoulder, Colorado40.0160-105.27651600call//www.xeno-canto.org/165398//www.xeno-canto.org/165398/downloadXC165398-CanG for Xeno-Canto.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/KADPGEQPZI/ffts/XC165398-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/KADPGEQPZI/ffts/XC165398-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/KADPGEQPZI/ffts/XC165398-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/KADPGEQPZI/ffts/XC165398-full.png'}//creativecommons.org/licenses/by-nc-sa/3.0/A0:191900-01-01 09:30:002014-01-242014-01-25['']A large flock of Canada Geese taking off. I believe most of the birds in this flock were parvipes (\\\"Lesser\\\") Canada Geese, but there were also larger (subspecies moffitti?) Canada Geese and a few Cackling Geese (several of the subspecies hutchinsii and possibly one of the subspecies minima) in the general vicinity. \\r\\n\\r\\nIn the old days this would have been an \\\"obvious\\\" or \\\"easy\\\" flock of \\\"Canada Geese.\\\" Now we're dealing with perhaps two species and probably two or three subspecies in the recording. Again, I believe most of the birds audible here are parvipes (\\\"Lesser\\\") Canada Geese.yesno1NaNNaN1.024.09.030.0
61291136BrantacanadensisNaNCanada GooseDon JonesUnited StatesBrace Road, Southampton, NJ39.9337-74.7170?song//www.xeno-canto.org/1136//www.xeno-canto.org/1136/downloadbird034.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/BCWZQTGMSO/ffts/XC1136-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/BCWZQTGMSO/ffts/XC1136-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/BCWZQTGMSO/ffts/XC1136-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/BCWZQTGMSO/ffts/XC1136-full.png'}//creativecommons.org/licenses/by-nc-nd/2.5/A0:10NaT1997-10-172008-11-20['']NaNunknownunknown0NaNNaN10.017.0NaNNaN
7132536877BrantacanadensisNaNCanada GooseSue RiffeUnited StatesS Cape May Meadows, Cape May Cty, New Jersey38.9381-74.94460adult, call, sex uncertain//www.xeno-canto.org/536877//www.xeno-canto.org/536877/downloadXC536877-Canada Goose on 10.18.19 at S Cape May Meadows NJ at 18.52 for .19.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC536877-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC536877-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC536877-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC536877-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/B0:191900-01-01 18:52:002019-10-182020-03-21['Charadrius vociferus']Natural vocalization of a flock of geese landing on the water near sunset. Windyyesno1NaNadult10.018.018.052.0
8133511453BrantacanadensisNaNCanada GoosePhoenix BirderUnited StatesGilbert, Maricopa County, Arizona33.3634-111.7341380adult, call, female, male//www.xeno-canto.org/511453//www.xeno-canto.org/511453/downloadXC511453-CaGo.2019.12.10.AZ.Maricopa.RiparianPreserve.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/UKNISVRBBF/ffts/XC511453-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/UKNISVRBBF/ffts/XC511453-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/UKNISVRBBF/ffts/XC511453-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/UKNISVRBBF/ffts/XC511453-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/B0:191900-01-01 08:52:002019-12-102019-12-10['Toxostoma curvirostre']Sound Devices MixPre-3 Wildtronics Stereo Model #WTPMMSA 22” Parabolic Reflector, phoenixbirder@gmail.comyesno1maleadult12.010.08.052.0
9134504983BrantacanadensiscanadensisCanada Goosenick talbotUnited StatesCentral Park, New York city,USA40.7740-73.971020call//www.xeno-canto.org/504983//www.xeno-canto.org/504983/downloadXC504983-2019_10_21 Branta canadensis2.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/CCUCXWCPSW/ffts/XC504983-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/CCUCXWCPSW/ffts/XC504983-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/CCUCXWCPSW/ffts/XC504983-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/CCUCXWCPSW/ffts/XC504983-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/B0:131900-01-01 13:00:002019-10-212019-10-30['']A pair of birds calling from a lakeyesno1NaNNaN10.021.013.00.0
\"\n", " )\n", ")" ], "execution_count": null, "outputs": [ { "data": { "text/plain": "", "text/html": "
df_indexidgenspsspenreccntloclatlngalttypeurlfilefile-namesonolicqlengthtimedateuploadedalsormkbird-seenplayback-usedpredgenderagemonthdayhourminute
096454911BrantacanadensisNaNCanada GooseBruce LagerquistUnited StatesSedro-Woolley, Skagit County, Washington48.5237-122.018530call//www.xeno-canto.org/454911//www.xeno-canto.org/454911/downloadXC454911-190202_02 Canadian Geese.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/JHFICMRVUX/ffts/XC454911-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/JHFICMRVUX/ffts/XC454911-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/JHFICMRVUX/ffts/XC454911-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/JHFICMRVUX/ffts/XC454911-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:171900-01-01 11:30:002019-02-022019-02-04['Cygnus buccinator']Mixed flock of Trumpeter Swans and Canada Geese feeding in an agricultural field. Recording of Swan's here XC454910yesno1NaNNaN2.02.011.030.0
197418340BrantacanadensisNaNCanada GooseSue RiffeUnited StatesAu Sable SF - Big Creek Rd, Michigan44.0185-83.7560180song//www.xeno-canto.org/418340//www.xeno-canto.org/418340/downloadXC418340-Canada Goose on 5.11.18 at Au Sable SF MI at 11.20 for .14 _0908 .mp3{'small': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC418340-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC418340-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC418340-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC418340-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:141900-01-01 11:20:002018-05-112018-06-03['Agelaius phoeniceus']Natural vocalizationyesno0NaNNaN5.011.011.020.0
2107291051BrantacanadensisNaNCanada GooseEric HoughUnited StatesSan Juan River, Cottonwood Day-Use Area, Navajo Lake State Park, San Juan County, New Mexico36.8068-107.67891800call//www.xeno-canto.org/291051//www.xeno-canto.org/291051/downloadXC291051-CANG_11515_1730_SanJuanRiver-NavajoDam.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/BCFUZDOSJZ/ffts/XC291051-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/BCFUZDOSJZ/ffts/XC291051-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/BCFUZDOSJZ/ffts/XC291051-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/BCFUZDOSJZ/ffts/XC291051-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:151900-01-01 17:30:002015-11-152015-11-18['']Flock calling while flying over at dusk. Amplification, low and high pass filters used in Audacity.yesno1NaNNaN11.015.017.030.0
3108283618BrantacanadensisNaNCanada GooseGarrett MacDonaldUnited StatesBeluga--North Bog, Kenai Peninsula Borough, Alaska61.2089-151.010340call, flight call//www.xeno-canto.org/283618//www.xeno-canto.org/283618/downloadXC283618-LS100466.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/CDHIAMGTRT/ffts/XC283618-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/CDHIAMGTRT/ffts/XC283618-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/CDHIAMGTRT/ffts/XC283618-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/CDHIAMGTRT/ffts/XC283618-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:101900-01-01 11:00:002015-05-202015-10-03['']Natural vocalizations from a pair of birds in flight. Recording not modified.yesno1NaNNaN5.020.011.00.0
4110209702BrantacanadensisNaNCanada GooseAlbert @ Max lastukhinUnited StatesOyster Bay (near Lattingtown), Nassau, New York40.8881-73.585110call//www.xeno-canto.org/209702//www.xeno-canto.org/209702/downloadXC209702-Poecile atricapillus Dec_27,_2014,_4_05_PM,C1.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/LELYWQKUZX/ffts/XC209702-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/LELYWQKUZX/ffts/XC209702-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/LELYWQKUZX/ffts/XC209702-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/LELYWQKUZX/ffts/XC209702-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:111900-01-01 16:00:002014-12-272015-01-09['Poecile atricapillus']NaNyesno1NaNNaN12.027.016.00.0
5118165398BrantacanadensisparvipesCanada GooseTed FloydUnited StatesBoulder, Colorado40.0160-105.27651600call//www.xeno-canto.org/165398//www.xeno-canto.org/165398/downloadXC165398-CanG for Xeno-Canto.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/KADPGEQPZI/ffts/XC165398-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/KADPGEQPZI/ffts/XC165398-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/KADPGEQPZI/ffts/XC165398-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/KADPGEQPZI/ffts/XC165398-full.png'}//creativecommons.org/licenses/by-nc-sa/3.0/A0:191900-01-01 09:30:002014-01-242014-01-25['']A large flock of Canada Geese taking off. I believe most of the birds in this flock were parvipes (\"Lesser\") Canada Geese, but there were also larger (subspecies moffitti?) Canada Geese and a few Cackling Geese (several of the subspecies hutchinsii and possibly one of the subspecies minima) in the general vicinity. \r\n\r\nIn the old days this would have been an \"obvious\" or \"easy\" flock of \"Canada Geese.\" Now we're dealing with perhaps two species and probably two or three subspecies in the recording. Again, I believe most of the birds audible here are parvipes (\"Lesser\") Canada Geese.yesno1NaNNaN1.024.09.030.0
61291136BrantacanadensisNaNCanada GooseDon JonesUnited StatesBrace Road, Southampton, NJ39.9337-74.7170?song//www.xeno-canto.org/1136//www.xeno-canto.org/1136/downloadbird034.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/BCWZQTGMSO/ffts/XC1136-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/BCWZQTGMSO/ffts/XC1136-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/BCWZQTGMSO/ffts/XC1136-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/BCWZQTGMSO/ffts/XC1136-full.png'}//creativecommons.org/licenses/by-nc-nd/2.5/A0:10NaT1997-10-172008-11-20['']NaNunknownunknown0NaNNaN10.017.0NaNNaN
7132536877BrantacanadensisNaNCanada GooseSue RiffeUnited StatesS Cape May Meadows, Cape May Cty, New Jersey38.9381-74.94460adult, call, sex uncertain//www.xeno-canto.org/536877//www.xeno-canto.org/536877/downloadXC536877-Canada Goose on 10.18.19 at S Cape May Meadows NJ at 18.52 for .19.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC536877-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC536877-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC536877-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC536877-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/B0:191900-01-01 18:52:002019-10-182020-03-21['Charadrius vociferus']Natural vocalization of a flock of geese landing on the water near sunset. Windyyesno1NaNadult10.018.018.052.0
8133511453BrantacanadensisNaNCanada GoosePhoenix BirderUnited StatesGilbert, Maricopa County, Arizona33.3634-111.7341380adult, call, female, male//www.xeno-canto.org/511453//www.xeno-canto.org/511453/downloadXC511453-CaGo.2019.12.10.AZ.Maricopa.RiparianPreserve.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/UKNISVRBBF/ffts/XC511453-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/UKNISVRBBF/ffts/XC511453-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/UKNISVRBBF/ffts/XC511453-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/UKNISVRBBF/ffts/XC511453-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/B0:191900-01-01 08:52:002019-12-102019-12-10['Toxostoma curvirostre']Sound Devices MixPre-3 Wildtronics Stereo Model #WTPMMSA 22” Parabolic Reflector, phoenixbirder@gmail.comyesno1maleadult12.010.08.052.0
9134504983BrantacanadensiscanadensisCanada Goosenick talbotUnited StatesCentral Park, New York city,USA40.7740-73.971020call//www.xeno-canto.org/504983//www.xeno-canto.org/504983/downloadXC504983-2019_10_21 Branta canadensis2.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/CCUCXWCPSW/ffts/XC504983-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/CCUCXWCPSW/ffts/XC504983-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/CCUCXWCPSW/ffts/XC504983-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/CCUCXWCPSW/ffts/XC504983-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/B0:131900-01-01 13:00:002019-10-212019-10-30['']A pair of birds calling from a lakeyesno1NaNNaN10.021.013.00.0
" }, "metadata": {}, "output_type": "display_data" } ] }, { "cell_type": "markdown", "source": [ "Here we also see a snippet of the data transformation pipeline and model training code which was done in\n", "[the following jupyter notebook](https://github.com/adithyabsk/bird_audio/blob/main/notebooks/4.0-ab-metadata-model.ipynb)." ], "metadata": { "tags": [], "cell_id": "00016-724f6cc1-fb6c-45f6-b9b5-3d999ad78565", "deepnote_cell_type": "markdown" } }, { "cell_type": "code", "metadata": { "tags": [], "cell_id": "00017-a783add5-e577-48bf-a8b7-2661612c6459", "deepnote_to_be_reexecuted": false, "source_hash": "50be9578", "execution_start": 1621312147758, "execution_millis": 120, "deepnote_output_heights": [ 611 ], "deepnote_cell_type": "code" }, "source": [ "# hide_input\n", "metadata_notebook = requests.get(\n", " \"https://raw.githubusercontent.com/adithyabsk/bird_audio/main/notebooks/4.0-ab-metadata-model.ipynb\"\n", ").json()\n", "mapping_snippet = \"\".join(metadata_notebook[\"cells\"][7][\"source\"][30:])\n", "traintest_snippet = \"\".join(metadata_notebook[\"cells\"][8][\"source\"])\n", "xgb_snippet = \"\".join(metadata_notebook[\"cells\"][10][\"source\"])\n", "ipd.display(\n", " ipd.Markdown(\n", " f\"\"\"\n", "```python\n", "{mapping_snippet}\n", "\n", "{traintest_snippet}\n", "\n", "{xgb_snippet}\n", "```\n", "\"\"\"\n", " )\n", ")" ], "execution_count": null, "outputs": [ { "data": { "text/plain": "", "text/markdown": "\n```python\nfeature_mapper = DataFrameMapper(\n [\n (\"id\", None),\n ([\"gen\"], OneHotEncoder(drop_invariant=True, use_cat_names=True)),\n ([\"sp\"], OneHotEncoder(drop_invariant=True, use_cat_names=True)),\n ([\"en\"], OneHotEncoder(drop_invariant=True, use_cat_names=True)),\n ([\"lat\"], [SimpleImputer(), StandardScaler()]), # gaussian\n ([\"lng\"], [SimpleImputer(), MinMaxScaler()]), # bi-modal --> MinMaxScaler\n # TODO: maybe later look into converting month / day into days since start of year\n (\n [\"month\"],\n [\n SimpleImputer(),\n FunctionTransformer(lambda X: np.sin((X - 1) * 2 * np.pi / 12)),\n StandardScaler(), # gaussian\n ],\n ),\n (\n [\"day\"],\n [\n SimpleImputer(),\n FunctionTransformer(lambda X: np.sin(X * 2 * np.pi / 31)),\n MinMaxScaler(), # uniform\n ],\n ),\n # TODO: maybe later look into converting hour / minute into seconds since start of day\n (\n [\"hour\"],\n [\n SimpleImputer(),\n FunctionTransformer(lambda X: np.sin(X * 2 * np.pi / 24)),\n StandardScaler(), # gaussian\n ],\n ),\n (\n [\"minute\"],\n [\n SimpleImputer(),\n FunctionTransformer(lambda X: np.sin(X * 2 * np.pi / 60)),\n MinMaxScaler(), # uniform\n ],\n ),\n ],\n df_out=True,\n)\n\nX_feat_df = feature_mapper.fit_transform(X_df, y_df[\"pred\"])\nX_train, X_test = (\n X_feat_df[X_feat_df.id.isin(train_ids)].drop(columns=[\"id\"]),\n X_feat_df[X_feat_df.id.isin(test_ids)].drop(columns=[\"id\"]),\n)\ny_train, y_test = (\n y_df[y_df.id.isin(train_ids)].drop(columns=[\"id\"]).squeeze(),\n y_df[y_df.id.isin(test_ids)].drop(columns=[\"id\"]).squeeze(),\n)\n\nxgb_clf = xgb.XGBClassifier()\neval_set = [(X_train, y_train), (X_test, y_test)]\nxgb_clf.fit(\n X_train, y_train, eval_metric=[\"error\", \"logloss\"], eval_set=eval_set, verbose=False\n)\n\nprint(xgb_clf.score(X_test, y_test))\n```\n" }, "metadata": {}, "output_type": "display_data" } ] }, { "cell_type": "markdown", "source": [ "## Audio Classification Model\n", "\n", "In one model we used the bird audio recordings themselves (mp3 and wav files), converted into time series arrays using\n", "[librosa](https://librosa.org/) and processed with [tsfresh](https://tsfresh.readthedocs.io/en/latest/index.html) to\n", "extract features, which we used to train a Gradient Boosted Tree model.\n", "\n", "
\n", "\n", "\"ts-fresh\n", "\n", "
" ], "metadata": { "cell_id": "00007-1aada947-ee4f-4c14-93c9-3c1735f917c2", "deepnote_cell_type": "markdown" } }, { "cell_type": "markdown", "source": [ "### Building Audio Features\n", "\n", "We ran audio data through a high-pass [Butterworth filter](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.butter.html)\n", "to take out background noise. We tested different parameters for Butterworth and Firwin filters, then examined resulting\n", "spectrograms and audio to determine which best reduced background noise without clipping bird sound frequencies.\n", "\n", "
\n", "
\n", "\"Filter\n", "
\n", "
\n", "\n", "The below code snippet shows the process of loading the `.mp3` file and performing the above filtering steps before\n", "saving as a `pd.DataFrame` which is what ts-fresh expects." ], "metadata": { "tags": [], "cell_id": "00017-c168c5c6-054f-4985-ad18-29c7c7aeb3f3", "deepnote_cell_type": "markdown" } }, { "cell_type": "code", "metadata": { "tags": [], "cell_id": "00018-f28125f0-d0a4-42d6-8b16-2a260199c1b1", "deepnote_to_be_reexecuted": false, "source_hash": "496b2af7", "execution_start": 1621312147870, "execution_millis": 92, "deepnote_output_heights": [ 398.546875 ], "deepnote_cell_type": "code", "pycharm": { "name": "#%%\n" } }, "source": [ "# hide_input\n", "display_code_block(\n", " \"https://raw.githubusercontent.com/adithyabsk/bird_audio/main/pracds_final/features/proc_audio.py\",\n", " 18,\n", " 46,\n", ")" ], "execution_count": null, "outputs": [ { "data": { "text/plain": "", "text/markdown": "\n```python\ndef unpack_audio(recordings_path: Path, id, filter_order, cutoff_freq):\n \"\"\"Load mp3 or wav into a floating point time series, then run a high-pass filter.\n\n Args:\n recordings_path: The path to the recordings dir\n id: The id for the audio recording to unpack\n filter_order: The order for the Butter filter\n cutoff_freq: The critical frequency for the Butter filter (below this is filtered out)\n\n Returns:\n A df of filtered time series data for the given id, with 'id', 'time', and 'val' columns\n \"\"\"\n try:\n audio_path = FILE_PATH / (\"data/raw/recordings/\" + str(id) + \".mp3\")\n # load mp3 as audio timeseries arr\n timeseries, sr = librosa.load(audio_path)\n except FileNotFoundError:\n audio_path = FILE_PATH / (\"data/raw/recordings/\" + str(id) + \".wav\")\n timeseries, sr = librosa.load(audio_path)\n\n # high-pass filter on audio timeseries\n timeseries_filt = highpass_filter(timeseries, sr, filter_order, cutoff_freq)\n\n df = pd.DataFrame(timeseries_filt, columns=[\"val\"])\n df.reset_index(inplace=True)\n df[\"id\"] = id # fill col with id\n df = df.reindex(columns=[\"id\", \"index\", \"val\"])\n df.columns = [\"id\", \"time\", \"val\"]\n return df\n```\n" }, "metadata": {}, "output_type": "display_data" } ] }, { "cell_type": "markdown", "source": [ "#### Feature Selection & Extraction\n", "\n", "We used ts-fresh to featurize each audio array after unpacking and filtering to avoid running out of memory. ts-fresh\n", "takes in dataframes with an id column, time column, and value column.\n", "\n", "
\n", "
\n", "\"Time\n", "
\n", "
\n", "\n", "ts-fresh provides feature calculator presets, but due to their and `librosa.load`'s long runtimes (13+ hours for 5% of\n", "the dataset), we manually specified the following small set of features based on our domain understanding of bird audio\n", "analysis.\n", "\n", "Lastly, we passed this \"static\" time series feature dataframe into a similar XGBoost model (from above) to predict the\n", "output class." ], "metadata": { "tags": [], "cell_id": "00017-2998f038-06c6-4c7f-8c1b-c4ddc702cb90", "deepnote_cell_type": "markdown" } }, { "cell_type": "code", "metadata": { "tags": [], "cell_id": "00022-43cdda12-dce5-466b-9fa8-146ddf835f04", "deepnote_to_be_reexecuted": false, "source_hash": "6d3af666", "execution_start": 1621312147958, "execution_millis": 19, "deepnote_output_heights": [ 437.5 ], "deepnote_cell_type": "code" }, "source": [ "# hide_input\n", "display_code_block(\n", " \"https://raw.githubusercontent.com/adithyabsk/bird_audio/main/pracds_final/features/proc_audio.py\",\n", " 71,\n", " 102,\n", ")" ], "execution_count": null, "outputs": [ { "data": { "text/plain": "", "text/markdown": "\n```python\nmanual_fc_params = {\n \"abs_energy\": None,\n \"fft_aggregated\": [{\"aggtype\": \"centroid\"}, {\"aggtype\": \"kurtosis\"}],\n \"root_mean_square\": None,\n \"spkt_welch_density\": [{\"coeff\": 2}, {\"coeff\": 5}, {\"coeff\": 8}],\n}\n\n# select features to calculate\n# features can be found here: https://tsfresh.readthedocs.io/en/latest/api/tsfresh.feature_extraction.html#tsfresh.feature_extraction.feature_calculators.fft_aggregated\ndef featurize_audio(id, fc_params):\n return extract_features(\n unpack_audio(id),\n column_id=\"id\",\n column_sort=\"time\",\n default_fc_parameters=fc_params,\n disable_progressbar=True,\n # we impute = remove all NaN features automatically\n impute_function=impute,\n # turn off parallelization\n n_jobs=0,\n )\n\n\n# featurize dataset\n# returns df of all combined\ndef featurize_set(ids, fc_params=None):\n if fc_params is None:\n fc_params = EfficientFCParameters()\n X_df = pd.DataFrame()\n for id in tqdm(ids):\n X_df = pd.concat([X_df, featurize_audio(id, fc_params)])\n return X_df\n```\n" }, "metadata": {}, "output_type": "display_data" } ] }, { "cell_type": "markdown", "source": [ "
\n", "
\n", "\"Feat\n", "
\n", "
" ], "metadata": { "cell_id": "00018-80daee04-77cd-4ffe-933d-fce90cb81112", "deepnote_cell_type": "markdown" } }, { "cell_type": "markdown", "source": [ "## Spectrogram Classification Model\n", "\n", "[Training Notebook Link](https://github.com/adithyabsk/bird_audio/blob/main/notebooks/5.0-ab-sonogram-model.ipynb)\n", "\n", "We used a computer vision approach to analyze spectrograms using [fast.ai](https://docs.fast.ai/) pre-trained model. We use an [`xresnet18`](https://github.com/fastai/fastai/blob/d7779196359c8e497a80e2f7f85c327318777c1a/fastai/vision/models/xresnet.py#L64) architecture pre-trained on ImageNet.\n", "\n", "
\n", "\n", "\"fast.ai\n", "\n", "
\n", "\n", "We load the data using fast.ai's `ImageDataLoader`. The model is then cut at the pooling layer (frozen weights) and\n", "then trained on its last layers to utilize transfer learning on our spectrogram images. A diagram of the architecture\n", "pulled directly from the original resnet paper is included below.\n", "\n", "
\n", "
\n", "\"ResNet50\n", "
\n", "
\n", "\n", "The model itself was trained on a Tesla K80 using [Google Colab](https://colab.research.google.com/signup) to speed up\n", "the training process. Additionally, we used [Weights and Biases](https://wandb.ai/site) to track the training and\n", "improve the model tuning. We've listed the main snippets of code below that handle the training process." ], "metadata": { "tags": [], "cell_id": "00021-6305e544-bfaf-448e-bc79-a3baa50ad2f8", "deepnote_cell_type": "markdown" } }, { "cell_type": "code", "metadata": { "tags": [], "cell_id": "00025-69ceacb1-3871-42bb-8226-048ff891c3fa", "deepnote_to_be_reexecuted": false, "source_hash": "b7bbbce4", "execution_start": 1621313567751, "execution_millis": 22, "deepnote_output_heights": [ 476.453125 ], "deepnote_cell_type": "code" }, "source": [ "# hide_input\n", "sonogram_notebook = requests.get(\n", " \"https://raw.githubusercontent.com/adithyabsk/bird_audio/main/notebooks/5.0-ab-sonogram-model.ipynb\"\n", ").json()\n", "datasetup_snippet = \"\".join(sonogram_notebook[\"cells\"][5][\"source\"])\n", "smodel_train_snippet = \"\".join(sonogram_notebook[\"cells\"][6][\"source\"])\n", "ipd.display(\n", " ipd.Markdown(\n", " f\"\"\"\n", "```python\n", "{datasetup_snippet}\n", "\n", "{smodel_train_snippet}\n", "```\n", "\"\"\"\n", " )\n", ")" ], "execution_count": null, "outputs": [ { "data": { "text/plain": "", "text/markdown": "\n```python\nbs = 128 # Batch size\nkwargs = {}\nif IS_COLAB:\n kwargs[\"num_workers\"] = 0\ndata = (\n # convert_mode is passed on intern|ally to the relevant function that will handle converting the images;\n # 'L' results in one color channel\n ImageDataLoaders.from_df(\n image_df,\n folder=ROOT_PATH / \"data/raw/sonograms\",\n valid_col=\"is_valid\",\n bs=bs,\n # num_works needs to be set to 0 for local evaluation to turn off multiprocessing\n **kwargs,\n )\n)\nlearn = cnn_learner(data, xresnet.xresnet18, pretrained=True)\n\n# Make sure this path exists on colab\nfname = \"sono_model.pth\"\nmodel_path = (ROOT_PATH / f\"models/{fname}\").resolve().absolute()\nif IS_COLAB and TRAIN:\n # Fine tune model\n wandb.init(project=\"sono-model\")\n learn.fit_one_cycle(1, cbs=WandbCallback())\n # GDrive fails when you try to use mkdir\n # so we manually call `save_model`\n save_path = f\"/home/{fname}\"\n save_model(save_path, learn.model, getattr(learn, \"opt\", None))\n %ls -al /home\n from google.colab import files\n\n files.download(save_path)\nelse:\n load_model(model_path, learn.model, learn.opt)\n```\n" }, "metadata": {}, "output_type": "display_data" } ] }, { "cell_type": "markdown", "source": [ "## Results\n", "\n", "Across our three models, we achieved scores in a range of 64-77%. This is above the baseline score of 55% (mean of\n", "labels), and we believe with more time to tune and ensemble the models, one could achieve an even more accurate\n", "classifier. We are encouraged by the amount of room both the time series based and sonogram based models have for\n", "improvement given that the metadata model wipes the floor in terms of accuracy.\n", "\n", "
\n", "\n", "| Model | Train Log Loss | Test Log Loss | Train Accuracy | Test Accuracy |\n", "|-|-|-|-|-|\n", "| Metadata Model | 0.331 | 0.507 | 0.879 | 0.773 |\n", "| Audio Model | 0.255 | 0.694 | 0.957 | 0.639 |\n", "| Spectrogram Model | 0.661 | 0.675 | 0.682 | 0.682 |\n", "| Baseline | 0.69 | 0.55 | 0.55 | 0.54 |\n", "\n", "
\n", "\n", "### Plots\n", "\n", "#### Metadata Model\n", "\n", "We note a plateau in the XGBoost validation accuracy which tends to suggest that further improvements in early stopping may\n", "be achieved.\n", "\n", "
\n", "
\n", "\"XGBoost\n", "
\n", "
\n", "\n", "Additionally, due to the nature of the decision tree based model we are able to compute feature importance. The most\n", "important features include the genera - this is not so surprising when we recall our genus-count distribution and see\n", "that the genera here are mostly those with recordings that are almost entirely songs or calls. The other important\n", "feature is month - again, we recall that in the spring the ratio of songs to calls goes up, so time of year is a \"good\"\n", "feature.\n", "\n", "
\n", "
\n", "\"XGBoost\n", "
\n", "
\n", "\n", "#### Time Series Model\n", "\n", "We can see that the test loss increases due to over-fitting, also evidenced by the very high training accuracy. This is\n", "a potential area of improvement in further research.\n", "\n", "
\n", "
\n", "\"LogReg\n", "
\n", "
\n", "\n", "#### Spectrogram Model\n", "\n", "This is the direct output from WandB which depicts the training process for the fine-tuned xresnet model. It is\n", "important to note that the X axis is steps and not epochs as this model was only trained for a single epoch (to save\n", "time and memory).\n", "\n", "
\n", "
\n", "\"Wandb\n", "
\n", "
" ], "metadata": { "cell_id": "00021-6151fea3-72b3-4168-9880-88c243f10ed6", "deepnote_cell_type": "markdown" } }, { "cell_type": "markdown", "source": [ "## Future Work\n", "\n", "We would like to note that there are a couple of immediate next steps that the project could take to dramatically\n", "improve the model performance\n", "\n", "- Ensembling the 3 models using a [`VotingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html)\n", "- More training time for the Spectrogram model (only 30 minutes was provided for fine-tuning)\n", " * Additional epochs (only 1 epoch was provided)\n", "- Filtering features in the audio classification model (ts-fresh likely generates more features than are needed)\n", "\n", "Long term: integrate model with Xeno Canto to provide tag suggestions based on the audio clip" ], "metadata": { "tags": [], "cell_id": "00027-a1d83e5d-3845-471e-a552-c760b2ca975d", "deepnote_cell_type": "markdown" } }, { "cell_type": "markdown", "source": [ "## Conclusion\n", "\n", "The classification of song vs call is the first distinction one can make in bird audio data across species, and on its\n", "own can give insights into the number of predators in an ecosystem, the timing of mating season, and other behaviors.\n", "It could also be valuable when part of a larger system of models. This report presents a promising start to tackle this\n", "problem with three separate machine learning models with reasonable accuracy. These models will likely prove quite handy\n", "in downstream classification tasks that look to find species, gender, location, and other parameters from the bird audio\n", "sample." ], "metadata": { "tags": [], "cell_id": "00026-74080aa6-cd0b-4563-8a9d-d0d68f85b704", "deepnote_cell_type": "markdown" } }, { "cell_type": "markdown", "source": [ "## References\n", "1. \"A Beginner’s Guide to Common Bird Sounds and What They Mean.\" [*Audubon.org.*](https://www.audubon.org/news/a-beginners-guide-common-bird-sounds-and-what-they-mean)\n", "2. \"Two Types of Communication Between Birds: Understanding Bird Language Songs And Calls.\" [*Youtube.*](https://www.youtube.com/watch?v=4_1zIwEENt8)\n", "3. \"Bird Vocalization.\" [*Wikipedia.*](https://en.wikipedia.org/wiki/Bird_vocalization#Function)\n", "4. Gorissen, Leen, et al. “Heavy Metal Pollution Affects Dawn Singing Behaviour in a Small Passerine Bird.” *Oecologia*, vol. 145, no. 3, 2005, pp. 504–509. [JSTOR](https://www.jstor.org/stable/20062442)\n", "5. Ortega, Yvette K.; Benson, Aubree; Greene, Erick. 2014. Invasive plant erodes local song diversity in a migratory passerine. *Ecology.* 95(2): 458-465. [Ecological Society of America](https://www.fs.usda.gov/treesearch/pubs/46856)\n", "6. Marler, P. (2004), Bird Calls: Their Potential for Behavioral Neurobiology. Annals of the New York Academy of Sciences, 1016: 31-44. [https://doi.org/10.1196/annals.1298.034](https://doi.org/10.1196/annals.1298.034)\n", "7. \"These birds 'retweet' alarm calls—but are careful about spreading rumors.\" [*National Geographic.*](https://www.nationalgeographic.com/animals/article/nuthatches-chickadees-communication-danger)\n", "8. Templeton, Christopher N., et al. “Allometry of Alarm Calls: Black-Capped Chickadees Encode Information About Predator Size.” Science, vol. 308, no. 5730, American Association for the Advancement of Science, 2005, pp. 1934–37, [doi:10.1126/science.1108841](https://doi.org/10.1126/science.1108841).\n", "9. Volodin, I.A., Volodina, E.V., Klenova, A.V. et al. Gender identification using acoustic analysis in birds without external sexual dimorphism. Avian Res 6, 20 (2015). [https://doi.org/10.1186/s40657-015-0033-y](https://doi.org/10.1186/s40657-015-0033-y)\n", "10. \"About yellowhammers.\" [*Yellowhammer Dialects.*](http://www.yellowhammers.net/about)\n", "11. Harry R Harding, Timothy A C Gordon, Emma Eastcott, Stephen D Simpson, Andrew N Radford, Causes and consequences of intraspecific variation in animal responses to anthropogenic noise, Behavioral Ecology, Volume 30, Issue 6, November/December 2019, Pages 1501–1511, [https://doi.org/10.1093/beheco/arz114](https://doi.org/10.1093/beheco/arz114)\n", "12. \"Open-source Version Control System for Machine Learning Projects.\" [*DVC.*](https://dvc.org/)\n", "13. [*xeno-canto.*](https://www.xeno-canto.org/explore/api)\n", "14. [*scikit-learn.*](https://scikit-learn.org/stable/index.html)\n", "15. [*xgboost.*](https://xgboost.readthedocs.io/en/latest/)\n", "16. [*fast.ai.*](https://docs.fast.ai/)\n", "\n", "### Metrics\n", "\n", "#### Word Count\n", "\n", "1753 words\n", "\n", "#### Code Line count\n", "\n", "We used [CLOC](https://github.com/AlDanial/cloc) to generate the code line counts\n", "\n", "| Language | Files | Code |\n", "|-|-|-|\n", "| Jupyter Notebook | 9 | 1195 |\n", "| Python | 8 | 397 |\n", "| Sum | **17** | **1592** |" ], "metadata": { "cell_id": "00022-1e1949a9-24c9-4dd3-b9bf-9ff82eedeb9d", "deepnote_cell_type": "markdown", "pycharm": { "name": "#%% md\n" } } } ], "nbformat": 4, "nbformat_minor": 4, "metadata": { "deepnote": { "is_reactive": false }, "deepnote_execution_queue": [], "deepnote_notebook_id": "bf38dd1c-03b4-4794-8429-f8b5a06d2638", "kernelspec": { "display_name": "Python 3.8.5 64-bit ('venv': venv)", "language": "python", "name": "python385jvsc74a57bd09d6766ad3736c29ebfe40ecf2d41a2944950e1cce237755c2a58ee0718f8bfc6" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } } }