{ "cells": [ { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "This example illustrates how to evaluate quality of audio synthesis by gathering answers to some quality questions about the audios with Crowdom.\n", "\n", "In our example, we ask workers to answer three quality questions about each given audio.\n", "\n", "In this example we will use conditional view for objects - so that for each question only corresponding answers will be displayed as an option. \n", "Moreover, this workflow shows how to use `Markdown` view for `Text` objects." ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Setup\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "%pip install crowdom" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "from datetime import timedelta\n", "import logging.config\n", "\n", "import markdown2\n", "import yaml\n", "import os\n", "from typing import Dict\n", "\n", "from crowdom import base, datasource, client, objects, pricing, control, classification, classification_loop, worker" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "with open('logging.yaml') as f:\n", " logging.config.dictConfig(yaml.full_load(f.read()))" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "toloka_client = client.create_toloka_client(token=os.getenv('TOLOKA_TOKEN') or input('Enter your token: '))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Questions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Define your questions and corresponding answers:\n", "- `question_label` contains the question you want to ask\n", "- possible answers are listed as `Enum` members:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "class Noise(objects.Question):\n", " NOISE = 'noise'\n", " CORRUPTION = 'corruption'\n", " BOTH = 'both'\n", " NONE = 'none'\n", " \n", "\n", " @classmethod\n", " def question_label(cls) -> Dict[str, str]:\n", " return {\n", " 'EN': '**Are there any noises or singal corruption?**',\n", " 'RU': '**Встречаются ли лишние шумы или искажения сигнала?**',\n", " }\n", " \n", " @classmethod\n", " def labels(cls) -> Dict['Noise', Dict[str, str]]:\n", " return {\n", " cls.NOISE: {'EN': 'There is noise', 'RU': 'На аудио есть лишний шум'},\n", " cls.CORRUPTION: {'EN': 'There is noise corruption', 'RU': 'Фрагменты сигнала искажены'},\n", " cls.BOTH: {'EN': 'Both problems present', 'RU': 'Есть и 1, и 2 проблема'},\n", " cls.NONE: {'EN': 'These problems are not present', 'RU': 'Подобных проблем нет'},\n", " } " ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "class Acoustics(objects.Question):\n", " INSIDE = 'inside'\n", " OUTDOORS = 'outdoors'\n", " STUDIO = 'studio'\n", " \n", " @classmethod\n", " def question_label(cls) -> Dict[str, str]:\n", " return {'RU': '**Где записывалось аудио?**', 'EN': '**Where was this audio recorded?**'}\n", " \n", " @classmethod\n", " def labels(cls) -> Dict['Acoustics', Dict[str, str]]:\n", " return {\n", " cls.INSIDE: {'EN': 'In living or industrial space', 'RU': 'В жилом / промышленном помещении'},\n", " cls.OUTDOORS: {'EN': 'Outdoors', 'RU': 'На улице'},\n", " cls.STUDIO: {'EN': 'In recording studio', 'RU': 'В профессиональной студии'},\n", " } " ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "class Intonation(objects.Question):\n", " YES = 'yes'\n", " NO = 'no'\n", " \n", " @classmethod\n", " def question_label(cls) -> Dict[str, str]:\n", " return {\n", " 'EN': '**Does intonation convey correct meaning of the text?**',\n", " 'RU': '**Верно ли интонация на аудио передаёт смысл написанного текста?**',\n", " }\n", " \n", " @classmethod\n", " def labels(cls) -> Dict['Intonation', Dict[str, str]]:\n", " return {\n", " cls.YES: {\n", " 'EN': 'Intonation conveys text meaning correctly', 'RU': 'Смысл текста передан интонацией верно',\n", " },\n", " cls.NO: {\n", " 'EN': 'Intonation does not convey text meaning', 'RU': 'Смысл текста передан интонацией неверно',\n", " },\n", " } " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create combined question and answer classes, which will contain all possible options for your task:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "Question, Answer, question_answers_list = objects.get_combined_classes([Noise, Acoustics, Intonation])\n", "question_answers = base.create_available_labels_if('question', question_answers_list)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this example, there are a couple of customization options specified:\n", "- `LabelsDisplayType.MONO` changes your question view from a radio-button list to text version\n", "- a title is specified for question to separate it visually from task's `Text`\n", "- `TextFormat.MARKDOWN` enables markdown rendering for question text and title\n", "- conditional view is enabled via `available_labels`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For some questions (e.g. `Intonation`), the `Text`, that is being spoken on the audio, matters, so it is also included in the `inputs`:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "function = base.ClassificationFunction(\n", " inputs=(\n", " base.ClassMeta(\n", " type=Question, \n", " name='question', \n", " input_display_type=base.LabelsDisplayType.MONO, \n", " title=base.Title(text={'EN': 'Question', 'RU': 'Вопрос'}, format=base.TextFormat.MARKDOWN),\n", " text_format=base.TextFormat.MARKDOWN),\n", " base.ObjectMeta(\n", " type=objects.Audio, \n", " name='audio'),\n", " objects.TextMeta(\n", " type=objects.Text, \n", " name='text',\n", " format=base.TextFormat.MARKDOWN),\n", " ),\n", " cls=base.ClassMeta(\n", " type=Answer, \n", " name='answer', \n", " available_labels=question_answers, \n", " title=base.Title(text={'EN': 'Answer', 'RU': 'Ответ'}, format=base.TextFormat.MARKDOWN)),\n", ")" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "task preview" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "audio_url = 'https://storage.yandexcloud.net/crowdom-public/examples/audio_questions/data/2ff85d99-f9fa-4297-8d58-0935de85515d.wav'\n", "sample_task = (Question.NOISE, objects.Audio(url=audio_url), objects.Text(text=''))\n", "\n", "client.TaskPreview(sample_task, task_function=function, lang='RU').display_link()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We need meaningful `Text` for the `Intonation` question:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "task preview" ], "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sample_task = (Question.INTONATION, objects.Audio(url=audio_url), objects.Text(text='Это модерн? Модерн!'))\n", "\n", "client.TaskPreview(sample_task, task_function=function, lang='RU').display_link()" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Task definition\n" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "lang = 'RU'" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "instruction = {}\n", "for worker_lang in ['EN', 'RU']:\n", " with open(f'instruction_{worker_lang}.md') as f:\n", " instruction[worker_lang] = markdown2.markdown(f.read())" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "task_spec = client.TaskSpec(\n", " id='qq',\n", " function=function,\n", " name={\n", " 'EN': 'Audio Questions', 'RU': 'Вопросы про аудио',\n", " },\n", " description={\n", " 'EN': 'Listen to a speech on audio and answer the question asked', \n", " 'RU': 'Прослушайте речь на аудио и ответьте на заданный вопрос',\n", " },\n", " instruction=instruction)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "task_spec_ru = client.PreparedTaskSpec(task_spec, lang)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "client.define_task(task_spec_ru, toloka_client)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "task_duration_hint = timedelta(seconds=10) # audios are about 1-5 seconds each" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Data preparation\n" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "input_objects = datasource.read_tasks('tasks.json', task_spec_ru.task_mapping)\n", "control_objects = datasource.read_tasks('control_tasks.json', task_spec_ru.task_mapping, has_solutions=True)" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Launch configuration\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As the task is non-standard, and total number of possible answers for all of the questions can be too high, it is better to specify parameters with code:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "pricing_config = pricing.PoolPricingConfig(assignment_price=.02, real_tasks_count=20, control_tasks_count=4)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "correct_control_task_ratio_for_acceptance = .7\n", "\n", "control_params = control.Control(\n", " rules=control.RuleBuilder().add_static_reward(\n", " threshold=correct_control_task_ratio_for_acceptance).add_speed_control(\n", " ratio_rand=.1,\n", " ratio_poor=.3,\n", " ).build())" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "params = client.Params(\n", " pricing_config=pricing_config,\n", " task_duration_hint=task_duration_hint,\n", " aggregation_algorithm=classification.AggregationAlgorithm.MAJORITY_VOTE,\n", " overlap=classification_loop.StaticOverlap(overlap=5),\n", " control=control_params,\n", " worker_filter=worker.WorkerFilter(\n", " filters=[\n", " worker.WorkerFilter.Params(\n", " langs={worker.LanguageRequirement(lang=lang)},\n", " regions=worker.lang_to_default_regions.get(lang, {}),\n", " age_range=(18, None),\n", " ),\n", " ],\n", " training_score=80,\n", " ),\n", ")" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Launch\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "artifacts = client.launch(\n", " task_spec_ru,\n", " params,\n", " input_objects,\n", " control_objects,\n", " toloka_client,\n", ")" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "results = artifacts.results" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Results study\n" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
questionaudiotextresultconfidenceoverlap
0noisehttps://storage.yandexcloud.net/crowdom-publ...*Вам доступны и рассрочка, и кредит, и наличны...noise__none1.01
1noisehttps://storage.yandexcloud.net/crowdom-publ...*И светло и зелено было вокруг.*noise__none1.01
2noisehttps://storage.yandexcloud.net/crowdom-publ...*Накормленные досыта котики играли с трудом.*noise__none1.01
3noisehttps://storage.yandexcloud.net/crowdom-publ...*Накормленные досыта, котики играли с трудом.*noise__none1.01
4noisehttps://storage.yandexcloud.net/crowdom-publ...*В доме - мыши.*noise__none1.01
\n", "
" ], "text/plain": [ " question audio \\\n", "0 noise https://storage.yandexcloud.net/crowdom-publ... \n", "1 noise https://storage.yandexcloud.net/crowdom-publ... \n", "2 noise https://storage.yandexcloud.net/crowdom-publ... \n", "3 noise https://storage.yandexcloud.net/crowdom-publ... \n", "4 noise https://storage.yandexcloud.net/crowdom-publ... \n", "\n", " text result confidence \\\n", "0 *Вам доступны и рассрочка, и кредит, и наличны... noise__none 1.0 \n", "1 *И светло и зелено было вокруг.* noise__none 1.0 \n", "2 *Накормленные досыта котики играли с трудом.* noise__none 1.0 \n", "3 *Накормленные досыта, котики играли с трудом.* noise__none 1.0 \n", "4 *В доме - мыши.* noise__none 1.0 \n", "\n", " overlap \n", "0 1 \n", "1 1 \n", "2 1 \n", "3 1 \n", "4 1 " ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results.predict()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.6" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }