{"cells":[{"metadata":{"_uuid":"3741623e63c82436b813a3843d5ec70484105e94","_cell_guid":"00842a24-8c5b-4a25-8e20-8312df708d61"},"cell_type":"markdown","source":"# Motivation\n- Did you know that a lot of accidents happen every day in manufacturing plants? \n- Did you also know that it **kill thousands of people everywhere, globally**?\n\n# Objective\n   - Analyse real labor accident data aiming **to help manufacturing plants to save lives**.\n   - This notebook does not intend to use any machine learning technique, but to help people to **take valuable insights from few lines of data**\n   \n## Few simple Concepts to learn\n   - We will use two basic concepts about data exploration: **Highlight** and **Insight**\n   - **Highlight**: when we resume data information (you will see examples throught the notebook)\n   - **Insight**: ideas and questions that come from the Highlights\n    \n## Further details\n- Programming Language for data analysis: **Python**\n- Data handling tools: **pandas** helping with data-frame operations\n- Data Visualization tools: **Seaborn** and **Plotly**\n\nThe following data source is used: [here](https://kaggle.com/ihmstefanini/industrial-safety-and-health-analytics-database)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_kg_hide-input":true,"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, iplot\nfrom plotly.graph_objs import *\nimport plotly.plotly as py\nimport plotly.graph_objs as go\nimport seaborn as sns\ninit_notebook_mode(connected=True)\nimport os\nprint(os.listdir(\"../input\"))","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"109f95cbd6d122ec61f986cb1bb34947f5b58653","_cell_guid":"4a4e7a4a-2683-4c5e-80f3-1ed5cceb962c"},"cell_type":"markdown","source":"> # 1.0 Load Dataset (carregando os dados)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_kg_hide-input":true,"collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"data = pd.read_csv(\"../input/IHMStefanini_industrial_safety_and_health_database.csv\", delimiter=',', header=0, parse_dates = [\"Data\"], index_col =\"Data\")","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"f1d5dabda511a1a7bc92e01d4b10aea618754ce4","_cell_guid":"44fa4dbc-d0fe-485f-ac82-140a1d7004ef"},"cell_type":"markdown","source":"### 1.1 First look to the data shape (visualizando o formato da matriz de dados)"},{"metadata":{"_uuid":"e77eda655bb7126e2477e8a6f131516355775ad7","_kg_hide-input":true,"_cell_guid":"5cf6f400-aca9-4d4b-b5da-7dfadec366e6","trusted":true},"cell_type":"code","source":"data.shape","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"64c900471fdce31c7bc1742a2221967d899ee5b2","_cell_guid":"cf9037b6-25a2-450d-a235-0ed3cd2108be"},"cell_type":"markdown","source":"> ### **Outcome (saída)** : \n- small data set but with relevant information (let's see it)\n- base de dados pequena mas porém com informação relevante"},{"metadata":{"_uuid":"33703ec53a65f5dd864678343436e11958ec2876","_cell_guid":"9f497271-ed1f-4621-be9a-3ab7820ba33b"},"cell_type":"markdown","source":"  ### 1.2 First data vizualization (visualizando um pouco dos dados)"},{"metadata":{"_uuid":"71ea3a736bc3daee9d3038fbbc6dbcedb17190ff","_kg_hide-input":true,"scrolled":true,"_cell_guid":"2419560f-61f1-45f3-b98d-045d7bfa3cc1","trusted":true},"cell_type":"code","source":"data.head()","execution_count":4,"outputs":[]},{"metadata":{"_uuid":"f89b4cfb39ff4ad7c73fd3e4cc1997d0371e849c","_cell_guid":"5d6dcd39-3038-4acb-92e9-bbe8008c6d43"},"cell_type":"markdown","source":"### **Outcome (saída)** : \n- Some categories look generic (although they represent a real value), but they were modified to keep it anonymous and preserve company's identity\n- Algumas categorias parecem genéricas (embora representem um valor real), mas foram modificadas para mantê-las anônimas e preservar a identidade da empresa"},{"metadata":{"_uuid":"25c6a1b1a829e5b2d7c2a3e6afa4e2d799bdd859","_cell_guid":"1916da00-26d9-4266-9bb2-bb3029742264"},"cell_type":"markdown","source":"### 1.3 Checking data index format (verificando o formato do índice da matriz de dados)"},{"metadata":{"_uuid":"f67524be67466fd74e7a3de4172f4698797dc528","_kg_hide-input":true,"_cell_guid":"7ae0f43e-12d3-402f-818c-b886a74e783d","trusted":true},"cell_type":"code","source":"data.index","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"5bb847bdc7b54c79da2ea8de353db84f46c5acc6","_cell_guid":"a9fa010c-3fd2-481e-8dd4-409a1e8f1516"},"cell_type":"markdown","source":"### **Outcome (saída)** : \n- index is as expected (datetime)\n- o índice está no formato esperado (timestamp)"},{"metadata":{"_uuid":"57e01cbbad1883d8e33c9d1a296175d100129e4b","_cell_guid":"1dd15f4b-d164-45d6-9850-a368699a330a"},"cell_type":"markdown","source":"### 1.4 Identifing data types (identificando o \"tipo\" ou \"formato\" de cada coluna)"},{"metadata":{"_uuid":"d0d9bb9f7e420f08a6e1765eacea387a2f5cc208","_kg_hide-input":true,"_cell_guid":"70b4dce2-9cfc-4b4d-bc96-77e3ca734ca0","trusted":true},"cell_type":"code","source":"datadict = pd.DataFrame(data.dtypes)\ndatadict","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"27ebae35e1799f7fd44b694556d157c344823f5a","_cell_guid":"a5bee596-086f-4ad5-90eb-6e815a8179c7"},"cell_type":"markdown","source":"### 1.5 Checking null values (verificando a presença de valores nulos)"},{"metadata":{"_uuid":"93164a79eee48c9acab94174b70a9569bd8d8352","_kg_hide-input":true,"_cell_guid":"8c934ac5-1dc7-4d2e-adbd-314941f38a3a","trusted":true},"cell_type":"code","source":"datadict['MissingVal'] = data.isnull().sum()\ndatadict","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"e90e30672e44218f49ddfbbb2ed2c7dc1885e8f3","_cell_guid":"526bccb7-ef0a-40af-a8d0-5a0dc7133bcf"},"cell_type":"markdown","source":"### **Outcome (saída)** : \n- good, no need to worry about missing data!\n- legal, não precisa preocupar com valores faltantes"},{"metadata":{"_uuid":"170a9d00c15bf1a5e7ef45229e993d4b2240d8dd","_cell_guid":"28a13ebc-4953-46d9-8458-0c48642f0e12"},"cell_type":"markdown","source":"### 1.6 Identify number of unique values (identifica a quantidade de valores únicos presente em cada coluna)"},{"metadata":{"_uuid":"1152917ec054686150b2d367658f47a3cc68a06a","_kg_hide-input":true,"_cell_guid":"1ecfcec3-00bb-4d7c-97e0-ca9876c884a0","trusted":true},"cell_type":"code","source":"datadict['NUnique']=data.nunique()\ndatadict","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"025505e4c6c7974a78b7b1d61c43bf2f3efc8c79","_cell_guid":"455ef0b7-2986-4161-b55e-6135f463553c"},"cell_type":"markdown","source":"# 2.0 Basic Exploratory Data Analysis (análise exploratória de dados básica)\n\n   - In this part we will see how to get valuable Insights from a relevant data, which can help you to argue with about a such important issue: accidents in shop floor kill thousands of people everywere, globally"},{"metadata":{"_uuid":"daf6d3b4a906097a10897108a348954cadaa127d","_cell_guid":"d7f1a72d-a2a7-4385-beb7-5937420e0aab"},"cell_type":"markdown","source":"### 2.1 Basic descriptive statistics, but a lot of value on that!"},{"metadata":{"_uuid":"4f39abf1ddfcf9ddee857aefd330674f235bab92","_kg_hide-input":true,"scrolled":false,"_cell_guid":"1c6522e2-41dd-4120-a405-88a70119db21","trusted":true},"cell_type":"code","source":"data.describe(include=['object'])","execution_count":9,"outputs":[]},{"metadata":{"_uuid":"8e3cbc7d7dd95077ca1d5e1899da60d5107320a0","_cell_guid":"fcc6591d-b00f-4fd9-856a-298a5cbd931b"},"cell_type":"markdown","source":"Pandas *describe* command reference: [here](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html)"},{"metadata":{"_uuid":"16e9c5f11a9f860b04ce8daf8fa19e5d1045c862","_cell_guid":"5edca84b-f6c7-4076-afed-c588e5fb7535"},"cell_type":"markdown","source":"### Highlights:\n\n1. Country_01 is the country where most of the accidents happen (more than 50%)\n2. Local_03 (which also belongs to Country_01) is where most of the accidents happen \n3. Mining is also the most significant contributor to accidents\n4. Male (95%) and Third Party (43%) also counts for kind of people that suffers more accident\n\n### Insights:\n\n1. What makes Country_01 the most prominent contributor? Why is the amount of accidents higher there than in the other places?\n2. What also makes Local_03 the dangerous place to work?\n3. Why are Male and Third Party people suffering more accidents? Are they been trained correctly to prevent accidents?\n"},{"metadata":{"_uuid":"e47ced659e8da20f01e2628787f9635829becad6","_cell_guid":"39d3f216-b31d-45ab-8e42-46eee37d6cfe"},"cell_type":"markdown","source":"### 2.2 Some analysis as a kind of *excel dynamic table*"},{"metadata":{"_uuid":"484370a72d04b4522d347b7e4eee713b2c837613","_kg_hide-input":true,"_cell_guid":"7828fc49-232a-4432-bb58-6d2bc4367f10","trusted":true},"cell_type":"code","source":"data['Day of the Week'] = data.index.dayofweek\ngrouped_data = pd.DataFrame(data.groupby(['Countries','Day of the Week']).count())\ngrouped_data","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"2e6b3e018777c656724cfc42c6a891d0754c3ee4","_cell_guid":"8e0d8cf0-5928-4887-b13b-04bf0c2090fa"},"cell_type":"markdown","source":"### Highlights:\n\n1. Thursday is the day when most of the accidents happen in Country_01\n2. Wednesday is the day when most of the accidents happen in Country_02\n\n### Insights:\n\n1. What happens or what kind of routine leads to most of the accidents happen on Thursday on Country_01\n2. Same question for Country_02"},{"metadata":{"_uuid":"817aeb487652e9cb9b2b2da01435b29b0ebb0ac4","_cell_guid":"fb5279ba-0647-4ed0-8970-9bb674387e19"},"cell_type":"markdown","source":"### Changing the way of groupby data, its possible to see another patter:"},{"metadata":{"_uuid":"e98be6ad0a4b3fd72c74c1a88e6ff6c23c525a93","_kg_hide-input":true,"_cell_guid":"47cba80f-c562-4988-a030-613bfbf5ac7c","trusted":true},"cell_type":"code","source":"grouped_data = pd.DataFrame(data.groupby(['Industry Sector','Day of the Week']).count())\ngrouped_data","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"309df5833280f20f82959e56b94f76bf5bf92055","_cell_guid":"3aee6532-18f7-4b28-a168-fa8e0b515f20"},"cell_type":"markdown","source":"### Insights:\n\n1. What happens that most of the accidents in Mining happen on Saturday (45)?\n\n### Hint: \n - Change the way you group, and you will have different views/insights about the same data"},{"metadata":{"_uuid":"b3f2d432b9807fbdd68e166de3460f421e1b7bdc","_cell_guid":"d7e6c354-7268-46d3-bdd2-cdd25f6b2f22"},"cell_type":"markdown","source":"### 2.3 Basic trend plotting using plotly offline"},{"metadata":{"_uuid":"e20c9255baf9d00d5f83516aff78d2d6e43108b4","_kg_hide-input":true,"_cell_guid":"02189b62-5f27-4c9d-b8ec-c21f04251ab3","trusted":true},"cell_type":"code","source":"# Faz o resampling dos dados para 24h\ndf = data\ndf = df.Countries.resample('24H').count()\n\n#Plot o gráfico\ntrace_high = go.Scatter(\n                x=df.index,\n                y=df,\n                name = \"AAPL High\",\n                line = dict(color = '#17BECF'),\n                opacity = 0.8)\n\ndados= [trace_high]\n\nlayout = dict(\n    title = \"Number of Accidents/Day (all countries)\",\n\n)\n\nfig = dict(data=dados, layout=layout)\niplot(fig, filename = \"Manually Set Range\")","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"a51185b4c84133bfd15ad0e171f8f21c8077496e","_cell_guid":"632cfcc0-910a-469c-ac71-7ad3f0a9daa6","scrolled":false},"cell_type":"markdown","source":"### Highlight\n - It can be seen that there two peaks of accidents in February of 2016 and 2017.\n - Maybe a moving average filter would help us to know some tendencies\n\n### Insights:\n\n1. Why there are peaks of accidents at every beginning of the year? Is this because people are more relaxed, coming backing from vacations?"},{"metadata":{"_uuid":"49566657de52c9269fbe2b3f58a0593a01ba9899","_cell_guid":"bb3f53d9-0a0e-40b4-b10b-9c4f68dbdd98"},"cell_type":"markdown","source":"### Using a moving average filter to try to see some tendencies"},{"metadata":{"_uuid":"19ca14446d92931f99659bfcb1cbc4b1a40c3f14","_kg_hide-input":true,"_cell_guid":"b29232f7-7496-4a94-80c5-3b0cec1c823a","trusted":true},"cell_type":"code","source":"# Faz o resampling dos dados para 24h\ndf2 = data\ndf2 = df2.Countries.resample('24H').count()\ntemp = df2.rolling(window=30)\nb = temp.mean()\n\n#Plot o gráfico\ntrace_high = go.Scatter(\n                x=b.index,\n                y=b,\n                name = \"AAPL High\",\n                line = dict(color = '#17BECF'),\n                opacity = 0.8)\n\ndados= [trace_high]\n\nlayout = dict(\n    title = \"Moving Average of 30 Days of the number of accidents/Day (all countries)\",\n\n)\n\nfig = dict(data=dados, layout=layout)\niplot(fig, filename = \"Manually Set Range\")","execution_count":14,"outputs":[]},{"metadata":{"_uuid":"0ae4f6469214fd9c6bbe95692f9ff2ef6aa2238d","_cell_guid":"8b68a08c-eefb-4de7-98f2-d40c83146c49"},"cell_type":"markdown","source":"### Highlight\n - It is clear that there is an increase in the average of the number of accidents every beginning of the year.\n - It is also possible to see that in 2017 the mean is lower than in 2016 (which is good news).\n \n### Insights:\n\n1. Why this pattern persist (number of accidents increases every beginning of a year)?"},{"metadata":{"_uuid":"fa1bfaf4489a30de1bdd76233609e9355f065040","_cell_guid":"04e1e5b8-fbcf-43bf-9ceb-6432a1b17d08"},"cell_type":"markdown","source":"### 2.4 Basic bars plot using seaborn"},{"metadata":{"_uuid":"bc550e9c5d6f3a59f1e822167446649240b02c93","_cell_guid":"0ca5a4de-e734-4af0-a8a5-61c2f4d2e86d"},"cell_type":"markdown","source":"### Just a simple bar graph to show that is also lot of options to represent the same data"},{"metadata":{"_uuid":"ac756d1a428f9a0932298e114d7f6a6502fcfa95","_kg_hide-input":true,"_cell_guid":"9bc09be0-04f5-4a5c-aea8-7d8dbb8c3d38","trusted":true},"cell_type":"code","source":"g = sns.factorplot(data=data, kind=\"count\", x=\"Countries\", hue = \"Local\", size=8, aspect=1)","execution_count":15,"outputs":[]},{"metadata":{"_uuid":"be920bdaa08ee875bc86a0e9b119984fdb91c90e","_cell_guid":"c23afe52-2d71-4abf-a32c-d0c28ef6caa8"},"cell_type":"markdown","source":"### 2.5 A final pareto graph to analyze Crtical Risks using Plotly"},{"metadata":{"_uuid":"e2e64353fb1af8046971690eaf51396d47e97a12","_kg_hide-input":true,"_cell_guid":"79f9f01f-5c2b-4320-8531-49895a0675e4","scrolled":false,"collapsed":true,"trusted":true},"cell_type":"code","source":"columns = ['total','cumulative_sum', 'cumulative_perc','demarcation']\nparetodf = pd.DataFrame(columns=columns)\nparetodf = paretodf.fillna(0)\n\nparetodf['total'] = data[\"Risco Critico\"].value_counts()\n#print(paretodf)\n\nparetodf['cumulative_sum'] = paretodf.cumsum()\n#print(paretodf)\n\nparetodf['cumulative_perc'] = 100*paretodf.cumulative_sum/paretodf.total.sum()\n#print(paretodf)\n\nparetodf['demarcation'] = 80\n#print(paretodf)","execution_count":16,"outputs":[]},{"metadata":{"_uuid":"eed4e812a200cdac25b28c4ee441bc7485a83eec","_kg_hide-input":true,"scrolled":false,"_cell_guid":"88f6bc8e-4a16-40b2-8429-fe2531938142","trusted":true},"cell_type":"code","source":"trace1 = Bar(\n    x=paretodf.index[0:7],\n    y=paretodf.total[0:7],\n    name='Count',\n    marker=dict(\n        color='rgb(34,163,192)'\n               )\n)\ntrace2 = Scatter(\n    x=paretodf.index[0:7],\n    y=paretodf.cumulative_perc[0:7],\n    name='Cumulative Percentage',\n    yaxis='y2',\n    line=dict(\n        color='rgb(243,158,115)',\n        width=2.4\n       )\n)\ntrace3 = Scatter(\n    x=paretodf.index[0:7],\n    y=paretodf.demarcation[0:7],\n    name='80%',\n    yaxis='y2',\n    line=dict(\n        color='rgba(128,128,128,.45)',\n        dash = 'dash',\n        width=1.5\n       )\n)\ndataplot = [trace1, trace2,trace3]\nlayout = Layout(\n    title='Critical Risks Pareto',\n    titlefont=dict(\n        color='',\n        family='',\n        size=0\n    ),\n    font=Font(\n        color='rgb(128,128,128)',\n        family='Balto, sans-serif',\n        size=12\n    ),\n    width=623,\n    height=623,\n    paper_bgcolor='rgb(240, 240, 240)',\n    plot_bgcolor='rgb(240, 240, 240)',\n    hovermode='compare',\n    margin=dict(b=250,l=60,r=60,t=65),\n    showlegend=True,\n       legend=dict(\n          x=.83,\n          y=1.3,\n          font=dict(\n            family='Balto, sans-serif',\n            size=12,\n            color='rgba(128,128,128,.75)'\n        ),\n    ),\n    annotations=[ dict(\n                  text=\"Cumulative Percentage\",\n                  showarrow=False,\n                  xref=\"paper\", yref=\"paper\",\n                  textangle=90,\n                  x=1.100, y=.75,\n                  font=dict(\n                  family='Balto, sans-serif',\n                  size=14,\n                  color='rgba(243,158,115,.9)'\n            ),)],\n    xaxis=dict(\n      tickangle=-90\n    ),\n    yaxis=dict(\n        title='Count',\n        range=[0,300],\n      tickfont=dict(\n            color='rgba(34,163,192,.75)'\n        ),\n      tickvals = [0,6000,12000,18000,24000,30000],\n        titlefont=dict(\n                family='Balto, sans-serif',\n                size=14,\n                color='rgba(34,163,192,.75)')\n    ),\n    yaxis2=dict(\n        range=[0,101],\n        tickfont=dict(\n            color='rgba(243,158,115,.9)'\n        ),\n        tickvals = [0,20,40,60,80,100],\n        overlaying='y',\n        side='right'\n    )\n)\n\nfig = dict(data=dataplot, layout=layout)\niplot(fig)","execution_count":17,"outputs":[]},{"metadata":{"_uuid":"74acc43793eea9d0fd86cb7387c4d14d68de3557","_cell_guid":"79eebbb5-dde2-49b7-a169-5e03f2d9889f"},"cell_type":"markdown","source":"### Insight:\n - Its clear that this database needs attention as the column \"Others\" represent most of the Critical risks\n "},{"metadata":{"_uuid":"1d73fe19a25c9b0f4ba7ec9f23f3d48936990fff","_cell_guid":"90b19a77-1227-4e59-b73c-b7d5a9dbb958"},"cell_type":"markdown","source":"# Final outcomes:\n\n- Its possible to think that it could be done in Excel, yeah! But how fancy would it be using Excel? How much data would you be able to process in Excel instead of a Python Notebook?\n- Imagine the possibilities to communicate and show this Report to your boss or responsible for managing accidents in your plant?\n- You can also now explore **NLP** with this data..."},{"metadata":{"_uuid":"300574c0da2e5d6034029216c1547f3284695eb5","_cell_guid":"374f4038-6671-4ff3-853b-ba7cd7baa3fa","collapsed":true},"cell_type":"markdown","source":"## Thank you!"},{"metadata":{"_uuid":"5aa1dcc4d1744ad0ed7c40a6bea7e7cee72d91a5","_cell_guid":"a5044ae9-367e-46d1-a811-cc405f5af219","collapsed":true,"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}