{"cells": [{"cell_type": "markdown", "metadata": {}, "source": ["# How to Analysis Text Data of Traditional East Asian Medicine\n\n강의를 준비하며 데이터 분석에 필요한 내용들을 정리해 본다. \n\n이 곳에 소개된 방법은 자연어처리 및 기계학습 등 데이터 분석에서 일반적으로 널리 사용되는 방법이다. 다만 사용된 데이터는 한의학 자료로 예시를 두었다. \n\n기본 개념을 설명할 때에는 toy data를 가지고 코드를 써가며 소개하고, \n\n그런 뒤에 실제 활용을 설명할 때에는 전문 package들에 구비되어 있는 high level api를 이용하였다. \n\n코드는 파이썬3를 기준으로 하였다. \n\n내용과 코드는 앞으로 조금씩 계속 업데이트 될 것이다. \n\n[PROJECT PAGE](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/README.ipynb)\n\n\n## Contents\n\n\n### A. 데이터 분석 개요\n\n1. [데이터 분석이란]\n\n* 데이터 분석의 목적\n* 데이터 분석의 가정\n\n\n### B. 데이터 준비\n\n1. [데이터 모으기](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/B0100_Get_Data.ipynb)\n\n* 웹스크래핑\n* 데이터 불러오고 저장하기1 ( plain text / csv, tsv, json, yaml )\n* 데이터 불러오고 저장하기2 ( raw object / pickle, joblib )\n\n2. Pre-precessing\n\n* [text normalization](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/B0201_Text_Normalization1.ipynb), [lab](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/B0201_Text_Normalization_lab.ipynb)\n* [text manipulation & regular expression](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/B0202_Preprocessing.ipynb)\n* mapping & filtering\n\n3. Tokenization (or Segmentation)\n\n* [n-gram & others](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/B0301_Tokenization1.ipynb)\n* [score based segmentation](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/B0302_Tokenization2.ipynb), [lab](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/B0302_Tokenization_lab.ipynb)\n\n4. 데이터 살펴보기\n\n* item and feature ( Documents and term )\n* [token and type](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/B0401_Token_and_Type.ipynb)\n\n\n\n### B. 데이터 임베딩(정량화)\n\n* data structure ( vector, matrix, data_frame )\n\n1. [Vectorization](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/C0100_Vectorization.ipynb)\n\n* 기본 전제 : distribution hyperthesis\n* 1st order vector ( doc-feature matrix / tf, tfidf )\n* 2nd order vector ( feature-feature matrix / co-word )\n* special word embedding ( word2vec )\n\n2. [Similarity & Distance](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/C0200_Similarity_and_Distance.ipynb)\n\n\n### C. 암묵지 확인1\n\n1. [Observed value & Expected value](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/D0100_Observed_value_Expected_value.ipynb)\n\n* contingency table ( 2x2 table )\n\n2. [Co-occurrence (or Collocations)](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/D0200_Co-occurrence_Measures.ipynb)\n\n* association measure1 ( co-word )\n* association measure2 ( t-value, simple log likelyhood ratio, mutual information )\n\n3. [Comparing](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/D0300_Comparing.ipynb)\n\n* Pearson correlation coefficient\n* chi-square\n\n### D. 암묵지 확인2\n\n1. [Latent Analysis or Topic modeling](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/E0100_Latent_Analysis_or_Topic_Modeling.ipynb)\n\n* Latent Analysis or Topic modeling \n* Matrix Decomposition \n\n2. Clustering\n\n* hierarchical clustering\n* knn clustering\n\n3. Network\n\n* [Node and Edge ( Adjacency Matrix / Edge Lists )](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/E0301_Node_and_Edge.ipynb)\n* Visualize_network : [1. MDS](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/E0302_Visualize_network1.ipynb), [2. VOS](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/E0302_Visualize_network2.ipynb)\n* community detection\n\n\n### E. 학습과 예측\n\nModeling\n\n* modeling, training(fitting), prediction\n\nRegression\n\n* linear regression\n\nClassification\n\n* logistic regression\n* SVM\n\nNeural Networking\n\n* hyperthesis\n* objective functon\n* activation function\n* optimization\n\n### [F. Visualization](https://nbviewer.jupyter.org/github/pinedance/workshop-KM-data-analysis/blob/master/notebooks/G0200_Plots.ipynb)\n\ndimension reduction\n\nPlot 1\n\n* 수치형 → histogram\n\nPlot 2\n\n* 범주형 × 수치형  →  box plot\n* 수치형 × 수치형  →  scatter plot\n\nPlot 3\n\n* 수치형 × 수치형 × 범주형  →  scatter plot with color,  treemap\n* 수치형 × 수치형 × 수치형  →  bubble chart \n* 범주형 × 범주형 × 수치형  →  heatmap\n\nPlot 4 / Special\n\n* Star(spider) chart\n* Sankey Diagram\n\n\n## Citation\n\nplease cite it as:\n\n```yaml\nauthor : Junho Oh\ne-mail: pinedance@gmail.com / junho@kiom.re.kr\ntitle : How to Analysis Text Data of Traditional East Asian Medicine\nyear : 2019\nmonth : 02\nkeywords : Data Analysis, Traditional East Asian Medicine, Traditional Korean Medicine, Traditional Chinese Medicine\navailable from : https://github.com/pinedance/workshop-KM-data-analysis\n```\n\n© 2019 Junho Oh."]}], "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.6"}}, "nbformat": 4, "nbformat_minor": 2}