{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "**_Columns Needed to import from MongoDB:_**\n", "\n", "file_list_columns = ['EIN', 'OrganizationName', 'URL', 'SubmittedOn', 'TaxPeriod']\n", "\n", "SOX_columns = ['WhistleblowerPolicy', 'WhistleblowerPolicyInd',\n", " 'ConflictOfInterestPolicyInd', 'ConflictOfInterestPolicy',\n", " 'DocumentRetentionPolicy', 'DocumentRetentionPolicyInd',]\n", "\n", "revenue_columns = ['CYTotalRevenueAmt', 'TotalRevenueCurrentYear'] \n", " \n", "efficiency_columns = ['CYTotalExpensesAmt', 'TotalExpensesCurrentYear'\n", " 'TotalProgramServiceExpense', 'TotalProgramServiceExpensesAmt']\n", " \n", "complexity_columns = ['CYContributionsGrantsAmt', \n", " 'FederatedCampaigns', 'FederatedCampaignsAmt', \n", " 'MembershipDues', 'MembershipDuesAmt',\n", " 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp',\n", " 'FundraisingAmt', 'FundraisingGrossIncomeAmt',\n", " 'RelatedOrganizations', 'RelatedOrganizationsAmt',\n", " 'GovernmentGrants', 'GovernmentGrantsAmt',\n", " 'PYProgramServiceRevenueAmt','CYProgramServiceRevenueAmt', \n", " 'ProgramServiceRevenueCY', 'TotalProgramServiceRevenue', 'TotalProgramServiceRevenueAmt', \n", " 'CYInvestmentIncomeAmt', 'InvestmentIncomeCurrentYear',\n", " 'OtherRevenueMiscGrp', 'OtherRevenueTotalAmt', \n", " 'CYOtherRevenueAmt', 'TotalOtherRevenue', \n", " ]\n", "\n", "cols = file_list_columns+SOX_columns+revenue_columns+efficiency_columns+complexity_columns\n", "\n", "\n", "\n", "\n", "**_Columns Needed to keep to merge with existing data:_**\n", "\n", "file_list_columns = ['EIN', 'OrganizationName', 'URL', 'SubmittedOn', 'TaxPeriod', 'FYE']\n", "\n", "SOX_columns = ['whistleblower_policy', 'conflict_of_interest_policy', 'records_retention_policy',\n", " 'SOX_policies', 'SOX_policies_binary', 'SOX_policies_all_binary']\n", "\n", "control_columns = ['tot_rev', 'total_expenses', 'program_expenses', 'complexity']\n", "\n", "unneeded_columns = ['CYTotalExpensesAmt', u'TotalExpensesCurrentYear', u'TotalFunctionalExpenses', \n", " 'TotalFunctionalExpensesGrp', u'TotalProgramServiceExpense', u'TotalProgramServiceExpensesAmt',\n", " 'CYContributionsGrantsAmt', u'FederatedCampaigns', u'FederatedCampaignsAmt', u'MembershipDues',\n", " 'MembershipDuesAmt', u'NetIncomeFromFundraisingEvents', u'NetIncmFromFundraisingEvtGrp', \n", " 'FundraisingAmt', 'FundraisingGrossIncomeAmt', u'RelatedOrganizations', \n", " 'RelatedOrganizationsAmt', u'GovernmentGrants', 'GovernmentGrantsAmt', \n", " 'PYProgramServiceRevenueAmt', u'CYProgramServiceRevenueAmt', 'ProgramServiceRevenue', \n", " 'ProgramServiceRevenueCY', u'ProgramServiceRevenueGrp', u'TotalProgramServiceRevenue', \n", " 'TotalProgramServiceRevenueAmt', u'CYInvestmentIncomeAmt', u'InvestmentIncome', \n", " 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', u'OtherRevenueTotalAmt', \n", " 'OtherRevenueMiscGrp', u'TotalOtherRevenue', 'CYOtherRevenueAmt', 'fundraising', \n", " 'program_revenue', 'OtherRevMisc', 'other_revenue_binary']\n", "\n", "cols = file_list_columns+SOX_columns+control_columns " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load Packages" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import sys\n", "import time\n", "import json" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import sqlite3 as lite\n", "\n", "from pandas import DataFrame\n", "from pandas import Series" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#http://pandas.pydata.org/pandas-docs/stable/options.html\n", "pd.set_option('display.max_columns', None)\n", "pd.set_option('max_colwidth', 250)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": true }, "outputs": [], "source": [ "plt.rcParams['figure.figsize'] = (15, 5)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": true }, "outputs": [], "source": [ "%matplotlib inline " ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### Set working directory" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/Users/gregorysaxton/Google Drive/SOX\n" ] } ], "source": [ "cd '/Users/gregorysaxton/Google Drive/SOX'" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false }, "outputs": [], "source": [ "dfx = pd.read_excel('advisories.xls')" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
org_idEINadvisory text - current advisoryclassification -- Gregclassification -- Danadvisory_text_0916level_0916current_ratings_urldate_0916name_0916
01672220503776On August 1, 2016, the New Hampshire Union Leader published an article titled, \"Former Portsmouth youth softball president accused of stealing thousands from nonprofit.\"potential material diversionNaNOn August 1, 2016, the New Hampshire Union Leader reported that Portsmouth Girls Softball Association was, or may continue to be, a subject of a material diversion of assets in an article titled, \"Former Portsmouth youth softball president accus...mediumhttp://www.charitynavigator.org/index.cfm?bay=search.summary&orgid=1672208/12/16Portsmouth Girls Softball Association
\n", "
" ], "text/plain": [ " org_id EIN \\\n", "0 16722 20503776 \n", "\n", " advisory text - current advisory \\\n", "0 On August 1, 2016, the New Hampshire Union Leader published an article titled, \"Former Portsmouth youth softball president accused of stealing thousands from nonprofit.\" \n", "\n", " classification -- Greg classification -- Dan \\\n", "0 potential material diversion NaN \n", "\n", " advisory_text_0916 \\\n", "0 On August 1, 2016, the New Hampshire Union Leader reported that Portsmouth Girls Softball Association was, or may continue to be, a subject of a material diversion of assets in an article titled, \"Former Portsmouth youth softball president accus... \n", "\n", " level_0916 \\\n", "0 medium \n", "\n", " current_ratings_url \\\n", "0 http://www.charitynavigator.org/index.cfm?bay=search.summary&orgid=16722 \n", "\n", " date_0916 name_0916 \n", "0 08/12/16 Portsmouth Girls Softball Association " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dfx[:1]" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "zero FR 15\n", "potential material diversion 12\n", "fundraising by non-charity 7\n", "potential corruption/impropriety 4\n", "potential financial violations 3\n", "potential fundraising impropriety 2\n", "tax-exempt status revoked 2\n", "potential fraud 2\n", "potential corruption 1\n", "misuse of federal grant money 1\n", "potential fraud/embezzlement 1\n", "fundraising by non-charity; charity shut down 1\n", "potential improper conduct 1\n", "under investigation 1\n", "tax-exempt status revoked; fraud 1\n", "non-filer 1\n", "potential bribery 1\n", "potential corruption/impropriety -- and charity potentially shut down 1\n", "potential corruption/impropriety/fraud -- and charity potentially shut down 1\n", "molestation lawsuit 1\n", "I DON'T THINK THIS ONE CAN BE THE CORRECT ADVISORY TEXT!!! 1\n", "whistleblower complaint of fraud 1\n", "Name: classification -- Greg, dtype: int64" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dfx['classification -- Greg'].value_counts()" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "321" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(dfx)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### MongoDB\n", "Depending on the project, I will store the data in SQLite or MongoDB. This time I'll use MongoDB -- it's great for storing JSON data where each observation could have different variables. Before we get to the interesting part the following code blocks set up the MongoDB environment and the new database we'll be using. \n", "\n", "**_Note:_** In a terminal we'll have to start MongoDB by running the command *mongod* or *sudo mongod*. Then we run the following code block here to access MongoDB." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import pymongo\n", "from pymongo import MongoClient\n", "client = MongoClient()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Read in Saved List of EINs with 'current' (2016) donor advisories" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "321\n", "[nan, nan, u'311660730', u'432059591', u'320022295', u'870804117', nan, u'223871535', u'620476294', u'311016441']\n" ] } ], "source": [ "f = open('2016 donor advisory EINs.json', 'r')\n", "advisories_2016 = json.load(f)\n", "print len(advisories_2016)\n", "print advisories_2016[:10]" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "256\n", "256\n", "['311660730', '432059591', '320022295', '870804117', '223871535']\n" ] } ], "source": [ "print len([str(x) for x in advisories_2016 if str(x) != 'nan'])\n", "advisories_2016 = [str(x) for x in advisories_2016 if str(x) != 'nan']\n", "print len(advisories_2016)\n", "print advisories_2016[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
There are 1,077 filings for those 256 EINs" ] }, { "cell_type": "code", "execution_count": 109, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "1077" ] }, "execution_count": 109, "metadata": {}, "output_type": "execute_result" } ], "source": [ "file_list.find({'EIN': { '$in': advisories_2016}}).count()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create Filings Collection" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# DEFINE MY COLLECTION (DATABASE TABLE) WHERE I'LL INSERT THE MESSAGES \n", "filings = db['filings']" ] }, { "cell_type": "code", "execution_count": 115, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0" ] }, "execution_count": 115, "metadata": {}, "output_type": "execute_result" } ], "source": [ "filings.count()" ] }, { "cell_type": "code", "execution_count": 116, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "u'URL_1'" ] }, "execution_count": 116, "metadata": {}, "output_type": "execute_result" } ], "source": [ "db.filings.create_index([('URL', pymongo.ASCENDING)], unique=True)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "[u'_id_', u'URL_1']" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "list(db.filings.index_information())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Read DB into PANDAS DF" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "# of columns: 671\n", "# of observations: 538\n" ] }, { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
@documentId@referenceDocumentId@referenceDocumentName@softwareId@softwareVersion@softwareVersionNumAccountantCompileOrReviewAccountantCompileOrReviewBasisAccountantCompileOrReviewIndAccountsPayableAccrExpnssGrpAccountsPayableAccruedExpensesAccountsReceivableAccountsReceivableGrpAcctCompileOrReviewBasisGrpActivitiesConductedPartnershipActivitiesConductedPrtshpIndActivity2Activity3ActivityOrMissionDescActivityOrMissionDescriptionActivityOtherAddressChangeAddressChangeIndAddressPrincipalOfficerUSAdvertisingAdvertisingGrpAllAffiliatesIncludedAllAffiliatesIncludedIndAllOtherContributionsAllOtherContributionsAmtAllOtherExpensesAllOtherExpensesGrpAmendedReturnAmendedReturnIndAnnualDisclosureCoveredPersonsAnnualDisclosureCoveredPrsnIndAuditCommitteeAuditCommitteeIndAuditedFinancialStmtAttIndBackupWthldComplianceIndBalanceSheetAmountsReportedBenefitsPaidToMembersCYBenefitsPaidToMembersPriorYearBenefitsToMembersBenefitsToMembersGrpBooksInCareOfDetailBsnssRltnshpThruFamilyMemberBsnssRltnshpWithOrganizationBusinessRlnWithFamMemIndBusinessRlnWithOfficerEntIndBusinessRlnWithOrgMemIndCYBenefitsPaidToMembersAmtCYContributionsGrantsAmtCYGrantsAndSimilarPaidAmtCYInvestmentIncomeAmtCYOtherExpensesAmtCYOtherRevenueAmtCYProgramServiceRevenueAmtCYRevenuesLessExpensesAmtCYSalariesCompEmpBnftPaidAmtCYTotalExpensesAmtCYTotalFundraisingExpenseAmtCYTotalProfFndrsngExpnsAmtCYTotalRevenueAmtCapStckTrstPrinCurrentFundsCapStkTrPrinCurrentFundsGrpCashNonInterestBearingCashNonInterestBearingGrpChangeToOrgDocumentsIndChangesToOrganizingDocsCntrbtnsRprtdFundraisingEventsCntrctRcvdGreaterThan100KCntCollectionsOfArtCollectionsOfArtIndCompCurrentOfcrDirectorsGrpCompCurrentOfficersDirectorsCompDisqualPersonsCompDisqualPersonsGrpCompensationFromOtherSourcesCompensationFromOtherSrcsIndCompensationProcessCEOCompensationProcessCEOIndCompensationProcessOtherCompensationProcessOtherIndComplianceWithBackupWitholdingConferencesMeetingsConferencesMeetingsGrpConflictOfInterestPolicyConflictOfInterestPolicyIndConservationEasementsConservationEasementsIndConsolidatedAuditFinancialStmtConsolidatedAuditFinclStmtIndContractorCompensationContractorCompensationGrpContriRptFundraisingEventAmtContributionsGrantsCurrentYearContributionsGrantsPriorYearCostOfGoodsSoldCostOfGoodsSoldAmtCreditCounselingCreditCounselingIndDAFExcessBusinessHoldingsIndDLNDecisionsSubjectToApprovaIndDecisionsSubjectToApprovalDeductibleArtContributionIndDeductibleContributionsOfArtDeductibleNonCashContriIndDeductibleNonCashContributionsDeferredRevenueDeferredRevenueGrpDelegationOfManagementDutiesDelegationOfMgmtDutiesIndDepreciationDepletionDepreciationDepletionGrpDescDescribedIn501C3DescribedInSection501c3IndDescriptionDisregardedEntityDisregardedEntityIndDistributionToDonorDistributionToDonorIndDoNotFollowSFAS117DocumentRetentionPolicyDocumentRetentionPolicyIndDoingBusinessAsDoingBusinessAsNameDonatedServicesAndUseFcltsAmtDonorAdvisedFundIndDonorAdvisedFundsEINElectionOfBoardMembersElectionOfBoardMembersIndEmployeeCntEmploymentTaxReturnsFiledEmploymentTaxReturnsFiledIndEngagedInExcessBenefitTransIndEscrowAccountEscrowAccountIndEscrowAccountLiabilityEscrowAccountLiabilityGrpExcessBenefitTransactionExcessBusinessHoldingsExcessBusinessHoldingsIndExpenseExpenseAmtFSAuditedFSAuditedBasisFSAuditedBasisGrpFSAuditedIndFamilyOrBusinessRelationshipFamilyOrBusinessRlnIndFederalGrantAuditPerformedFederalGrantAuditPerformedIndFederalGrantAuditRequiredFederalGrantAuditRequiredIndFederatedCampaignsFederatedCampaignsAmtFeesForServicesAccountingFeesForServicesAccountingGrpFeesForServicesInvstMgmntFeesFeesForServicesLegalFeesForServicesLegalGrpFeesForServicesLobbyingFeesForServicesLobbyingGrpFeesForServicesManagementFeesForServicesManagementGrpFeesForServicesOtherFeesForServicesOtherGrpFeesForServicesProfFundraisingFeesForSrvcInvstMgmntFeesGrpFiledLieu1041FinancialStatementBothFinancialStatementConsolidatedFinancialStatementSeparateFinancialStmtAttachedFollowSFAS117ForeignActivitiesForeignActivitiesIndForeignAddressForeignCountryCdForeignFinancialAccountForeignFinancialAccountIndForeignGrantsForeignGrantsGrpForeignOfficeForeignOfficeIndForm1098CFiledForm1098CFiledIndForm720FiledForm8282FiledCntForm8282PropertyDisposedOfForm8282PropertyDisposedOfIndForm8886TFiledForm8886TFiledIndForm8899FiledForm8899FiledindForm990-TFiledForm990PartVIISectionAForm990PartVIISectionAGrpForm990ProvidedToGoverningBodyForm990ProvidedToGvrnBodyIndForm990TFiledIndFormTypeFormationYrFormerOfcrEmployeesListedIndFormersListedFundraisingActivitiesFundraisingActivitiesIndFundraisingAmtFundraisingDirectExpensesFundraisingDirectExpensesAmtFundraisingEventsFundraisingGrossIncomeAmtFundsToPayPremiumsGainOrLossGainOrLossGrpGamingGamingActivitiesIndGoverningBodyVotingMembersCntGovernmentGrantsGovernmentGrantsAmtGrantAmtGrantToRelatedPersonGrantToRelatedPersonIndGrantsGrantsAndSimilarAmntsCYGrantsAndSimilarAmntsPriorYearGrantsPayableGrantsPayableGrpGrantsToDomesticIndividualsGrantsToDomesticIndividualsGrpGrantsToDomesticOrgsGrantsToDomesticOrgsGrpGrantsToIndividualsGrantsToIndividualsIndGrantsToOrganizationsGrantsToOrganizationsIndGrossAmountSalesAssetsGrossAmountSalesAssetsGrpGrossIncomeFundraisingEventsGrossReceiptsGrossReceiptsAmtGrossRentsGrossRentsGrpGrossSalesOfInventoryGrossSalesOfInventoryAmtGroupExemptionNumGroupExemptionNumberGroupReturnForAffiliatesGroupReturnForAffiliatesIndHospitalIRPDocumentCntIRPDocumentW2GCntIncludeFIN48FootnoteIndIncmFromInvestBondProceedsGrpIncomeFromInvestBondProceedsIndependentAuditFinancialStmtIndependentAuditFinclStmtIndIndependentVotingMemberCntIndivRcvdGreaterThan100KCntIndoorTanningServicesIndoorTanningServicesIndInfoInScheduleOPartIIIInfoInScheduleOPartIIIIndInfoInScheduleOPartIXInfoInScheduleOPartIXIndInfoInScheduleOPartVInfoInScheduleOPartVIInfoInScheduleOPartVIIInfoInScheduleOPartVIIIInfoInScheduleOPartVIIIIndInfoInScheduleOPartVIIIndInfoInScheduleOPartVIIndInfoInScheduleOPartVIndInfoInScheduleOPartXInfoInScheduleOPartXIInfoInScheduleOPartXIIInfoInScheduleOPartXIIIndInfoInScheduleOPartXIIndInfoInScheduleOPartXIndInformationTechnologyInformationTechnologyGrpInitialReturnInsuranceInsuranceGrpIntangibleAssetsIntangibleAssetsGrpInterestInterestGrpInventoriesForSaleOrUseInventoriesForSaleOrUseGrpInvestTaxExemptBondsInvestTaxExemptBondsIndInvestmentExpenseAmtInvestmentInJointVentureInvestmentInJointVentureIndInvestmentIncomeInvestmentIncomeCurrentYearInvestmentIncomeGrpInvestmentIncomePriorYearInvestmentsOtherSecuritiesInvestmentsOtherSecuritiesGrpInvestmentsProgramRelatedInvestmentsProgramRelatedGrpInvestmentsPubTradedSecGrpInvestmentsPubTradedSecuritiesIsAvailableIsElectronicJointCostsJointCostsIndLandBldgEquipAccumDeprecAmtLandBldgEquipBasisNetGrpLandBldgEquipCostOrOtherBssAmtLandBldgEquipmentAccumDeprecLandBuildingsEquipmentBasisLandBuildingsEquipmentBasisNetLastUpdatedLegalDomicileStateCdLessCostOthBasisSalesExpensesLessCostOthBasisSalesExpnssGrpLessRentalExpensesLessRentalExpensesGrpLicensedMoreThanOneStateLicensedMoreThanOneStateIndLoanOutstandingIndLoanToOfficerOrDQPLoansFromOfficersDirectorsLoansFromOfficersDirectorsGrpLobbyingActivitiesLobbyingActivitiesIndLocalChaptersLocalChaptersIndMaterialDiversionOrMisuseMaterialDiversionOrMisuseIndMembersOrStockholdersMembersOrStockholdersIndMembershipDuesMembershipDuesAmtMethodOfAccountingAccrualMethodOfAccountingAccrualIndMethodOfAccountingCashMethodOfAccountingCashIndMethodOfAccountingOtherMethodOfAccountingOtherIndMinutesOfCommitteesMinutesOfCommitteesIndMinutesOfGoverningBodyMinutesOfGoverningBodyIndMiscellaneousRevenueMiscellaneousRevenueGrpMissionDescMissionDescriptionMoreThan5000KToIndividualsMoreThan5000KToIndividualsIndMoreThan5000KToOrgIndMoreThan5000KToOrganizationsMortNotesPyblSecuredInvestPropMortgNotesPyblScrdInvstPropGrpNameOfForeignCountryNameOfPrincipalOfficerBusinessNameOfPrincipalOfficerPersonNbrIndependentVotingMembersNbrVotingGoverningBodyMembersNbrVotingMembersGoverningBodyNetAssetsOrFundBalancesBOYNetAssetsOrFundBalancesBOYAmtNetAssetsOrFundBalancesEOYNetAssetsOrFundBalancesEOYAmtNetGainOrLossInvestmentsNetGainOrLossInvestmentsGrpNetIncmFromFundraisingEvtGrpNetIncomeFromFundraisingEventsNetIncomeFromGamingNetIncomeFromGamingGrpNetIncomeOrLossNetIncomeOrLossGrpNetRentalIncomeOrLossNetRentalIncomeOrLossGrpNetUnrelatedBusTxblIncmAmtNetUnrelatedBusinessTxblIncomeNetUnrlzdGainsLossesInvstAmtNoListedPersonsCompensatedNoListedPersonsCompensatedIndNonDeductibleContributionsNonDeductibleDisclosureNoncashContributionsNoncashContributionsAmtNondeductibleContriDisclIndNondeductibleContributionsIndNumberFormsTransmittedWith1096NumberIndependentVotingMembersNumberIndividualsGT100KNumberOf8282FiledNumberOfContractorsGT100KNumberOfEmployeesNumberW2GIncludedObjectIdOccupancyOccupancyGrpOfficeExpensesOfficeExpensesGrpOfficerEntityWithBsnssRltnshpOfficerMailingAddressOfficerMailingAddressIndOnBehalfOfIssuerOnBehalfOfIssuerIndOperateHospitalIndOrgDoesNotFollowSFAS117IndOrgFiledInLieuOfForm1041IndOrganization501cOrganization501c3Organization501c3IndOrganization501cIndOrganizationFollowsSFAS117IndOrganizationNameOthNotesLoansReceivableNetGrpOtherAssetsTotalOtherAssetsTotalGrpOtherChangesInNetAssetsAmtOtherEmployeeBenefitsOtherEmployeeBenefitsGrpOtherExpensePriorYearOtherExpensesOtherExpensesCurrentYearOtherExpensesGrpOtherExplainInSchOOtherIndOtherLiabilitiesOtherLiabilitiesGrpOtherNotesLoansReceivableNetOtherRevenueCurrentYearOtherRevenueMiscOtherRevenueMiscGrpOtherRevenuePriorYearOtherRevenueTotalAmtOtherSalariesAndWagesOtherSalariesAndWagesGrpOtherWebsiteOtherWebsiteIndOwnWebsiteOwnWebsiteIndPYBenefitsPaidToMembersAmtPYContributionsGrantsAmtPYExcessBenefitTransIndPYGrantsAndSimilarPaidAmtPYInvestmentIncomeAmtPYOtherExpensesAmtPYOtherRevenueAmtPYProgramServiceRevenueAmtPYRevenuesLessExpensesAmtPYSalariesCompEmpBnftPaidAmtPYTotalExpensesAmtPYTotalProfFndrsngExpnsAmtPYTotalRevenueAmtPaidInCapSrplsLandBldgEqpFundPartialLiquidationPartialLiquidationIndPayPremiumsPrsnlBnftCntrctIndPaymentsToAffiliatesPaymentsToAffiliatesGrpPayrollTaxesPayrollTaxesGrpPdInCapSrplsLandBldgEqpFundGrpPensionPlanContributionsPensionPlanContributionsGrpPermanentlyRestrictedNetAssetsPermanentlyRstrNetAssetsGrpPledgesAndGrantsReceivablePledgesAndGrantsReceivableGrpPoliciesReferenceChaptersPoliciesReferenceChaptersIndPoliticalActivitiesPoliticalCampaignActyIndPremiumsPaidPrepaidExpensesDeferredChargesPrepaidExpensesDefrdChargesGrpPrincipalOfficerNmPriorExcessBenefitTransactionPriorPeriodAdjustmentsAmtProfessionalFundraisingProfessionalFundraisingIndProgSrvcAccomActy2GrpProgSrvcAccomActy3GrpProgSrvcAccomActyOtherGrpProgramServiceRevenueProgramServiceRevenueCYProgramServiceRevenueGrpProgramServiceRevenuePriorYearProhibitedTaxShelterTransProhibitedTaxShelterTransIndPymtTravelEntrtnmntPubOfclGrpQuidProQuoContriDisclIndQuidProQuoContributionsQuidProQuoContributionsIndQuidProQuoDisclosureRcvFndsToPayPrsnlBnftCntrctIndRcvblFromDisqualifiedPrsnGrpReceivablesFromDisqualPersonsReceivablesFromOfficersEtcReceivablesFromOfficersEtcGrpReconcilationDonatedServicesReconcilationInvestExpensesReconcilationOtherChangesReconcilationPriorAdjustmentReconcilationRevenueExpensesReconcilationRevenueExpnssAmtReconciliationUnrealizedInvestRegularMonitoringEnforcementRegularMonitoringEnfrcIndRelatedEntityRelatedEntityIndRelatedOrgControlledEntityRelatedOrganizationCtrlEntIndRelatedOrganizationsRelatedOrganizationsAmtRentalIncomeOrLossRentalIncomeOrLossGrpReportFin48FootnoteReportInvestOthSecuritiesReportInvestmentsOtherSecIndReportLandBldgEquipReportLandBuildingEquipmentIndReportOtherAssetsReportOtherAssetsIndReportOtherLiabilitiesReportOtherLiabilitiesIndReportProgRelInvestReportProgramRelatedInvstIndRetainedEarningsEndowmentEtcRevenueRevenueAmtRevenuesLessExpensesCYRevenuesLessExpensesPriorYearRoyaltiesRoyaltiesGrpRoyaltiesRevenueRoyaltiesRevenueGrpRtnEarnEndowmentIncmOthFndsGrpSalariesEtcCurrentYearSalariesEtcPriorYearSavingsAndTempCashInvestmentsSavingsAndTempCashInvstGrpScheduleBRequiredScheduleBRequiredIndScheduleJRequiredScheduleJRequiredIndScheduleORequiredScheduleORequiredIndSchoolSchoolOperatingIndSignificantChangeSignificantChangeIndSignificantNewProgramServicesSignificantNewProgramSrvcIndStateLegalDomicileStatesWhereCopyOfReturnIsFiledStatesWhereCopyOfReturnIsFldCdSubjectToProxyTaxSubjectToProxyTaxIndSubmittedOnTaxExemptBondLiabilitiesTaxExemptBondLiabilitiesGrpTaxExemptBondsTaxExemptBondsIndTaxPeriodTaxableDistributionsTaxableDistributionsIndTaxablePartyNotificationTaxablePartyNotificationIndTempOrPermanentEndowmentsIndTemporarilyRestrictedNetAssetsTemporarilyRstrNetAssetsGrpTermOrPermanentEndowmentsTerminateOperationsIndTerminatedTerminationOrContractionTheBooksAreInCareOfTotLiabNetAssetsFundBalanceGrpTotReportableCompRltdOrgAmtTotalAssetsTotalAssetsBOYTotalAssetsBOYAmtTotalAssetsEOYTotalAssetsEOYAmtTotalAssetsGrpTotalCompGT150KTotalCompGreaterThan150KIndTotalContributionsTotalContributionsAmtTotalEmployeeCntTotalExpensesCurrentYearTotalExpensesPriorYearTotalFunctionalExpensesTotalFunctionalExpensesGrpTotalFundrsngExpCurrentYearTotalGrossUBITotalGrossUBIAmtTotalJointCostsTotalJointCostsGrpTotalLiabNetAssetsFundBalancesTotalLiabilitiesTotalLiabilitiesBOYTotalLiabilitiesBOYAmtTotalLiabilitiesEOYTotalLiabilitiesEOYAmtTotalLiabilitiesGrpTotalNbrEmployeesTotalNbrVolunteersTotalNetAssetsFundBalanceGrpTotalNetAssetsFundBalancesTotalOfOtherProgramServiceExpTotalOfOtherProgramServiceGrntTotalOfOtherProgramServiceRevTotalOthProgramServiceRevGrpTotalOthProgramServiceRevenueTotalOtherCompensationTotalOtherCompensationAmtTotalOtherProgSrvcExpenseAmtTotalOtherProgSrvcGrantAmtTotalOtherProgSrvcRevenueAmtTotalOtherRevenueTotalProfFundrsngExpCYTotalProfFundrsngExpPriorYearTotalProgramServiceExpenseTotalProgramServiceExpensesAmtTotalProgramServiceRevenueTotalProgramServiceRevenueAmtTotalReportableCompFrmRltdOrgsTotalReportableCompFromOrgTotalReportableCompFromOrgAmtTotalRevenueTotalRevenueCurrentYearTotalRevenueGrpTotalRevenuePriorYearTotalVolunteersCntTransactionRelatedEntityTransactionWithControlEntIndTransfersToExemptNonChrtblOrgTravelTravelEntrtnmntPublicOfficialsTravelGrpTrnsfrExmptNonChrtblRltdOrgIndTypeOfOrganizationCorpIndTypeOfOrganizationCorporationTypeOfOrganizationOtherTypeOfOrganizationOtherIndTypeOfOrganizationTrustTypeOfOrganizationTrustIndURLUSAddressUnrelatedBusIncmOverLimitIndUnrelatedBusinessIncomeUnrestrictedNetAssetsUnrestrictedNetAssetsGrpUnsecuredNotesLoansPayableUnsecuredNotesLoansPayableGrpUponRequestUponRequestIndVotingMembersGoverningBodyCntVotingMembersIndependentCntWebSiteWebsiteAddressTxtWhistleblowerPolicyWhistleblowerPolicyIndWrittenPolicyOrProcedureWrittenPolicyOrProcedureIndYearFormation_id
0RetDoc1RetDoc2NaNNaNNaNNaNNaNNaNfalseNaNNaNNaNNaNNaNNaNfalseNaNNaNRAISE MONIES TO CONSTRUCT A FACILITY FOR YEAR-ROUND RECREATION PROGRAM DEDICATED TO CHILDREN AND ADULTS WITH SPECIAL NEEDS WHO RESIDE IN THE TOWN OF HEMPSTEAD.NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN{u'PhoneNum': u'5164316946', u'USAddress': {u'CityNm': u'LIDO BEACH', u'StateAbbreviationCd': u'NY', u'ZIPCd': u'11561', u'AddressLine1Txt': u'630 LIDO BLVD'}, u'BusinessName': {u'BusinessNameLine1Txt': u'ANCHOR BUILDING FUND INC'}}NaNNaNfalsefalsefalse000034400325000029060003440000325000NaNNaNNaN{u'EOYAmt': u'290600'}falseNaNNaNNaNNaNfalseNaNNaNNaNNaNNaNfalseNaNfalseNaNfalseNaNNaNNaNNaNfalseNaNfalseNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNfalseNaN93493316047805falseNaNfalseNaNfalseNaNNaNNaNNaNfalseNaNNaNTHE ORGANIZATIONS EXEMPT PURPOSE WAS TO RAISE MONIES TO CONSTRUCT A FACILTIY TO OPERATE ITS YEAR-ROUND PROGRAM.NaN{u'#text': u'true', u'@referenceDocumentId': u'RetDoc3'}NaNNaNfalseNaNNaNNaNNaNfalseNaNNaNNaNfalseNaN271317072NaNfalse0NaNNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNfalseNaNNaNNaNNaNNaNNaNNaN{u'TotalAmt': u'9400', u'ManagementAndGeneralAmt': u'9400'}NaNNaN{u'TotalAmt': u'25000', u'ManagementAndGeneralAmt': u'25000'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNNaNNaNfalseNaNNaNNaNfalseNaNfalseNaNNaNNaNfalseNaNNaNNaNfalseNaNNaN[{u'PersonNm': u'JOHN MALLOY', u'TitleTxt': u'PRESIDENT', u'AverageHoursPerWeekRt': u'2.00', u'ReportableCompFromRltdOrgAmt': u'0', u'OtherCompensationAmt': u'0', u'OfficerInd': u'X', u'ReportableCompFromOrgAmt': u'0'}, {u'PersonNm': u'ANNA DRAGO...NaNfalseNaN9902009falseNaNNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaNfalse2NaNNaNNaNNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNfalseNaNNaNNaNNaN325000NaNNaNNaNNaNNaNNaNNaNfalseNaN20falseNaNNaNNaNfalse2NaNNaNfalseNaNXNaNNaNNaNNaNNaNNaNNaNNaNXNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNTrueTrueNaNNaNNaNNaNNaNNaNNaNNaN2016-03-21T17:23:53NYNaNNaNNaNNaNNaNNaNfalseNaNNaNNaNNaNfalseNaNfalseNaNtrueNaNfalseNaNNaNNaNNaNNaNXNaNNaNNaNfalseNaNfalseNaNNaNRAISE MONIES TO CONSTRUCT A FACILITY FOR YEAR-ROUND RECREATION PROGRAM DEDICATED TO CHILDREN AND ADULTS WITH SPECIAL NEEDS WHO RESIDE IN THE TOWN OF HEMPSTEAD.NaNNaNfalsefalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN290600NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNXNaNNaNNaNNaNNaNfalseNaNNaNNaNNaNNaNNaNNaN201503169349304780NaNNaNNaNNaNNaNNaNfalseNaNNaNfalseNaNNaNNaNNaNXNaNXANCHOR BUILDING FUND INCNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN{u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000', u'Desc': u'SETTLEMENT'}NaN325000NaNNaNNaNNaNNaNNaNNaNNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalsefalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNNaNNaNJOHN MALLOYNaNNaNNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNNaNNaNfalseNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaN290600NaNNaNNaNNaNfalseNaNfalseNaNNaNNaNNaNNaNNaNfalseNaNfalseNaNfalseNaNfalseNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNfalseNaNtrueNaNfalseNaNtrueNaNfalseNaNNaNNYNaNfalse2016-01-30NaNNaNNaNfalse201412NaNNaNNaNfalsefalseNaNNaNNaNfalseNaNNaNNaN{u'BOYAmt': u'0', u'EOYAmt': u'290600'}NaNNaNNaNNaNNaN290600{u'BOYAmt': u'0', u'EOYAmt': u'290600'}NaNfalseNaNNaN0NaNNaNNaN{u'TotalAmt': u'34400', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'34400', u'FundraisingAmt': u'0'}NaNNaN0NaNNaNNaNNaNNaNNaNNaN0{u'BOYAmt': u'0', u'EOYAmt': u'0'}NaNNaN{u'BOYAmt': u'0', u'EOYAmt': u'290600'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN{u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000'}NaNNaNNaNNaNNaNNaNNaNNaNfalseXNaNNaNNaNNaNNaNhttps://s3.amazonaws.com/irs-form-990/201503169349304780_public.xml{u'CityNm': u'LIDO BEACH', u'StateAbbreviationCd': u'NY', u'ZIPCd': u'11561', u'AddressLine1Txt': u'630 LIDO BLVD'}falseNaNNaN{u'EOYAmt': u'290600'}NaNNaNNaNX22NaNWWW.CAMPANCHOR.ORGNaNfalseNaNNaNNaN57cce7823ffc5a55fe7f54b5
\n", "
" ], "text/plain": [ " @documentId @referenceDocumentId @referenceDocumentName @softwareId \\\n", "0 RetDoc1 RetDoc2 NaN NaN \n", "\n", " @softwareVersion @softwareVersionNum AccountantCompileOrReview \\\n", "0 NaN NaN NaN \n", "\n", " AccountantCompileOrReviewBasis AccountantCompileOrReviewInd \\\n", "0 NaN false \n", "\n", " AccountsPayableAccrExpnssGrp AccountsPayableAccruedExpenses \\\n", "0 NaN NaN \n", "\n", " AccountsReceivable AccountsReceivableGrp AcctCompileOrReviewBasisGrp \\\n", "0 NaN NaN NaN \n", "\n", " ActivitiesConductedPartnership ActivitiesConductedPrtshpInd Activity2 \\\n", "0 NaN false NaN \n", "\n", " Activity3 \\\n", "0 NaN \n", "\n", " ActivityOrMissionDesc \\\n", "0 RAISE MONIES TO CONSTRUCT A FACILITY FOR YEAR-ROUND RECREATION PROGRAM DEDICATED TO CHILDREN AND ADULTS WITH SPECIAL NEEDS WHO RESIDE IN THE TOWN OF HEMPSTEAD. \n", "\n", " ActivityOrMissionDescription ActivityOther AddressChange AddressChangeInd \\\n", "0 NaN NaN NaN NaN \n", "\n", " AddressPrincipalOfficerUS Advertising AdvertisingGrp AllAffiliatesIncluded \\\n", "0 NaN NaN NaN NaN \n", "\n", " AllAffiliatesIncludedInd AllOtherContributions AllOtherContributionsAmt \\\n", "0 NaN NaN NaN \n", "\n", " AllOtherExpenses AllOtherExpensesGrp AmendedReturn AmendedReturnInd \\\n", "0 NaN NaN NaN NaN \n", "\n", " AnnualDisclosureCoveredPersons AnnualDisclosureCoveredPrsnInd \\\n", "0 NaN NaN \n", "\n", " AuditCommittee AuditCommitteeInd AuditedFinancialStmtAttInd \\\n", "0 NaN NaN NaN \n", "\n", " BackupWthldComplianceInd BalanceSheetAmountsReported \\\n", "0 NaN NaN \n", "\n", " BenefitsPaidToMembersCY BenefitsPaidToMembersPriorYear BenefitsToMembers \\\n", "0 NaN NaN NaN \n", "\n", " BenefitsToMembersGrp \\\n", "0 NaN \n", "\n", " BooksInCareOfDetail \\\n", "0 {u'PhoneNum': u'5164316946', u'USAddress': {u'CityNm': u'LIDO BEACH', u'StateAbbreviationCd': u'NY', u'ZIPCd': u'11561', u'AddressLine1Txt': u'630 LIDO BLVD'}, u'BusinessName': {u'BusinessNameLine1Txt': u'ANCHOR BUILDING FUND INC'}} \n", "\n", " BsnssRltnshpThruFamilyMember BsnssRltnshpWithOrganization \\\n", "0 NaN NaN \n", "\n", " BusinessRlnWithFamMemInd BusinessRlnWithOfficerEntInd \\\n", "0 false false \n", "\n", " BusinessRlnWithOrgMemInd CYBenefitsPaidToMembersAmt \\\n", "0 false 0 \n", "\n", " CYContributionsGrantsAmt CYGrantsAndSimilarPaidAmt CYInvestmentIncomeAmt \\\n", "0 0 0 0 \n", "\n", " CYOtherExpensesAmt CYOtherRevenueAmt CYProgramServiceRevenueAmt \\\n", "0 34400 325000 0 \n", "\n", " CYRevenuesLessExpensesAmt CYSalariesCompEmpBnftPaidAmt CYTotalExpensesAmt \\\n", "0 290600 0 34400 \n", "\n", " CYTotalFundraisingExpenseAmt CYTotalProfFndrsngExpnsAmt CYTotalRevenueAmt \\\n", "0 0 0 325000 \n", "\n", " CapStckTrstPrinCurrentFunds CapStkTrPrinCurrentFundsGrp \\\n", "0 NaN NaN \n", "\n", " CashNonInterestBearing CashNonInterestBearingGrp ChangeToOrgDocumentsInd \\\n", "0 NaN {u'EOYAmt': u'290600'} false \n", "\n", " ChangesToOrganizingDocs CntrbtnsRprtdFundraisingEvents \\\n", "0 NaN NaN \n", "\n", " CntrctRcvdGreaterThan100KCnt CollectionsOfArt CollectionsOfArtInd \\\n", "0 NaN NaN false \n", "\n", " CompCurrentOfcrDirectorsGrp CompCurrentOfficersDirectors CompDisqualPersons \\\n", "0 NaN NaN NaN \n", "\n", " CompDisqualPersonsGrp CompensationFromOtherSources \\\n", "0 NaN NaN \n", "\n", " CompensationFromOtherSrcsInd CompensationProcessCEO \\\n", "0 false NaN \n", "\n", " CompensationProcessCEOInd CompensationProcessOther \\\n", "0 false NaN \n", "\n", " CompensationProcessOtherInd ComplianceWithBackupWitholding \\\n", "0 false NaN \n", "\n", " ConferencesMeetings ConferencesMeetingsGrp ConflictOfInterestPolicy \\\n", "0 NaN NaN NaN \n", "\n", " ConflictOfInterestPolicyInd ConservationEasements ConservationEasementsInd \\\n", "0 false NaN false \n", "\n", " ConsolidatedAuditFinancialStmt ConsolidatedAuditFinclStmtInd \\\n", "0 NaN false \n", "\n", " ContractorCompensation ContractorCompensationGrp \\\n", "0 NaN NaN \n", "\n", " ContriRptFundraisingEventAmt ContributionsGrantsCurrentYear \\\n", "0 NaN NaN \n", "\n", " ContributionsGrantsPriorYear CostOfGoodsSold CostOfGoodsSoldAmt \\\n", "0 NaN NaN NaN \n", "\n", " CreditCounseling CreditCounselingInd DAFExcessBusinessHoldingsInd \\\n", "0 NaN false NaN \n", "\n", " DLN DecisionsSubjectToApprovaInd DecisionsSubjectToApproval \\\n", "0 93493316047805 false NaN \n", "\n", " DeductibleArtContributionInd DeductibleContributionsOfArt \\\n", "0 false NaN \n", "\n", " DeductibleNonCashContriInd DeductibleNonCashContributions DeferredRevenue \\\n", "0 false NaN NaN \n", "\n", " DeferredRevenueGrp DelegationOfManagementDuties DelegationOfMgmtDutiesInd \\\n", "0 NaN NaN false \n", "\n", " DepreciationDepletion DepreciationDepletionGrp \\\n", "0 NaN NaN \n", "\n", " Desc \\\n", "0 THE ORGANIZATIONS EXEMPT PURPOSE WAS TO RAISE MONIES TO CONSTRUCT A FACILTIY TO OPERATE ITS YEAR-ROUND PROGRAM. \n", "\n", " DescribedIn501C3 DescribedInSection501c3Ind \\\n", "0 NaN {u'#text': u'true', u'@referenceDocumentId': u'RetDoc3'} \n", "\n", " Description DisregardedEntity DisregardedEntityInd DistributionToDonor \\\n", "0 NaN NaN false NaN \n", "\n", " DistributionToDonorInd DoNotFollowSFAS117 DocumentRetentionPolicy \\\n", "0 NaN NaN NaN \n", "\n", " DocumentRetentionPolicyInd DoingBusinessAs DoingBusinessAsName \\\n", "0 false NaN NaN \n", "\n", " DonatedServicesAndUseFcltsAmt DonorAdvisedFundInd DonorAdvisedFunds \\\n", "0 NaN false NaN \n", "\n", " EIN ElectionOfBoardMembers ElectionOfBoardMembersInd EmployeeCnt \\\n", "0 271317072 NaN false 0 \n", "\n", " EmploymentTaxReturnsFiled EmploymentTaxReturnsFiledInd \\\n", "0 NaN NaN \n", "\n", " EngagedInExcessBenefitTransInd EscrowAccount EscrowAccountInd \\\n", "0 false NaN NaN \n", "\n", " EscrowAccountLiability EscrowAccountLiabilityGrp ExcessBenefitTransaction \\\n", "0 NaN NaN NaN \n", "\n", " ExcessBusinessHoldings ExcessBusinessHoldingsInd Expense ExpenseAmt \\\n", "0 NaN NaN NaN NaN \n", "\n", " FSAudited FSAuditedBasis FSAuditedBasisGrp FSAuditedInd \\\n", "0 NaN NaN NaN false \n", "\n", " FamilyOrBusinessRelationship FamilyOrBusinessRlnInd \\\n", "0 NaN false \n", "\n", " FederalGrantAuditPerformed FederalGrantAuditPerformedInd \\\n", "0 NaN NaN \n", "\n", " FederalGrantAuditRequired FederalGrantAuditRequiredInd FederatedCampaigns \\\n", "0 NaN NaN NaN \n", "\n", " FederatedCampaignsAmt FeesForServicesAccounting \\\n", "0 NaN NaN \n", "\n", " FeesForServicesAccountingGrp \\\n", "0 {u'TotalAmt': u'9400', u'ManagementAndGeneralAmt': u'9400'} \n", "\n", " FeesForServicesInvstMgmntFees FeesForServicesLegal \\\n", "0 NaN NaN \n", "\n", " FeesForServicesLegalGrp \\\n", "0 {u'TotalAmt': u'25000', u'ManagementAndGeneralAmt': u'25000'} \n", "\n", " FeesForServicesLobbying FeesForServicesLobbyingGrp \\\n", "0 NaN NaN \n", "\n", " FeesForServicesManagement FeesForServicesManagementGrp FeesForServicesOther \\\n", "0 NaN NaN NaN \n", "\n", " FeesForServicesOtherGrp FeesForServicesProfFundraising \\\n", "0 NaN NaN \n", "\n", " FeesForSrvcInvstMgmntFeesGrp FiledLieu1041 FinancialStatementBoth \\\n", "0 NaN NaN NaN \n", "\n", " FinancialStatementConsolidated FinancialStatementSeparate \\\n", "0 NaN NaN \n", "\n", " FinancialStmtAttached FollowSFAS117 ForeignActivities ForeignActivitiesInd \\\n", "0 NaN NaN NaN false \n", "\n", " ForeignAddress ForeignCountryCd ForeignFinancialAccount \\\n", "0 NaN NaN NaN \n", "\n", " ForeignFinancialAccountInd ForeignGrants ForeignGrantsGrp ForeignOffice \\\n", "0 false NaN NaN NaN \n", "\n", " ForeignOfficeInd Form1098CFiled Form1098CFiledInd Form720Filed \\\n", "0 false NaN false NaN \n", "\n", " Form8282FiledCnt Form8282PropertyDisposedOf Form8282PropertyDisposedOfInd \\\n", "0 NaN NaN false \n", "\n", " Form8886TFiled Form8886TFiledInd Form8899Filed Form8899Filedind \\\n", "0 NaN NaN NaN false \n", "\n", " Form990-TFiled Form990PartVIISectionA \\\n", "0 NaN NaN \n", "\n", " Form990PartVIISectionAGrp \\\n", "0 [{u'PersonNm': u'JOHN MALLOY', u'TitleTxt': u'PRESIDENT', u'AverageHoursPerWeekRt': u'2.00', u'ReportableCompFromRltdOrgAmt': u'0', u'OtherCompensationAmt': u'0', u'OfficerInd': u'X', u'ReportableCompFromOrgAmt': u'0'}, {u'PersonNm': u'ANNA DRAGO... \n", "\n", " Form990ProvidedToGoverningBody Form990ProvidedToGvrnBodyInd \\\n", "0 NaN false \n", "\n", " Form990TFiledInd FormType FormationYr FormerOfcrEmployeesListedInd \\\n", "0 NaN 990 2009 false \n", "\n", " FormersListed FundraisingActivities FundraisingActivitiesInd FundraisingAmt \\\n", "0 NaN NaN false NaN \n", "\n", " FundraisingDirectExpenses FundraisingDirectExpensesAmt FundraisingEvents \\\n", "0 NaN NaN NaN \n", "\n", " FundraisingGrossIncomeAmt FundsToPayPremiums GainOrLoss GainOrLossGrp \\\n", "0 NaN NaN NaN NaN \n", "\n", " Gaming GamingActivitiesInd GoverningBodyVotingMembersCnt GovernmentGrants \\\n", "0 NaN false 2 NaN \n", "\n", " GovernmentGrantsAmt GrantAmt GrantToRelatedPerson GrantToRelatedPersonInd \\\n", "0 NaN NaN NaN false \n", "\n", " Grants GrantsAndSimilarAmntsCY GrantsAndSimilarAmntsPriorYear GrantsPayable \\\n", "0 NaN NaN NaN NaN \n", "\n", " GrantsPayableGrp GrantsToDomesticIndividuals GrantsToDomesticIndividualsGrp \\\n", "0 NaN NaN NaN \n", "\n", " GrantsToDomesticOrgs GrantsToDomesticOrgsGrp GrantsToIndividuals \\\n", "0 NaN NaN NaN \n", "\n", " GrantsToIndividualsInd GrantsToOrganizations GrantsToOrganizationsInd \\\n", "0 false NaN false \n", "\n", " GrossAmountSalesAssets GrossAmountSalesAssetsGrp \\\n", "0 NaN NaN \n", "\n", " GrossIncomeFundraisingEvents GrossReceipts GrossReceiptsAmt GrossRents \\\n", "0 NaN NaN 325000 NaN \n", "\n", " GrossRentsGrp GrossSalesOfInventory GrossSalesOfInventoryAmt \\\n", "0 NaN NaN NaN \n", "\n", " GroupExemptionNum GroupExemptionNumber GroupReturnForAffiliates \\\n", "0 NaN NaN NaN \n", "\n", " GroupReturnForAffiliatesInd Hospital IRPDocumentCnt IRPDocumentW2GCnt \\\n", "0 false NaN 2 0 \n", "\n", " IncludeFIN48FootnoteInd IncmFromInvestBondProceedsGrp \\\n", "0 false NaN \n", "\n", " IncomeFromInvestBondProceeds IndependentAuditFinancialStmt \\\n", "0 NaN NaN \n", "\n", " IndependentAuditFinclStmtInd IndependentVotingMemberCnt \\\n", "0 false 2 \n", "\n", " IndivRcvdGreaterThan100KCnt IndoorTanningServices IndoorTanningServicesInd \\\n", "0 NaN NaN false \n", "\n", " InfoInScheduleOPartIII InfoInScheduleOPartIIIInd InfoInScheduleOPartIX \\\n", "0 NaN X NaN \n", "\n", " InfoInScheduleOPartIXInd InfoInScheduleOPartV InfoInScheduleOPartVI \\\n", "0 NaN NaN NaN \n", "\n", " InfoInScheduleOPartVII InfoInScheduleOPartVIII InfoInScheduleOPartVIIIInd \\\n", "0 NaN NaN NaN \n", "\n", " InfoInScheduleOPartVIIInd InfoInScheduleOPartVIInd InfoInScheduleOPartVInd \\\n", "0 NaN X NaN \n", "\n", " InfoInScheduleOPartX InfoInScheduleOPartXI InfoInScheduleOPartXII \\\n", "0 NaN NaN NaN \n", "\n", " InfoInScheduleOPartXIIInd InfoInScheduleOPartXIInd InfoInScheduleOPartXInd \\\n", "0 NaN NaN NaN \n", "\n", " InformationTechnology InformationTechnologyGrp InitialReturn Insurance \\\n", "0 NaN NaN NaN NaN \n", "\n", " InsuranceGrp IntangibleAssets IntangibleAssetsGrp Interest InterestGrp \\\n", "0 NaN NaN NaN NaN NaN \n", "\n", " InventoriesForSaleOrUse InventoriesForSaleOrUseGrp InvestTaxExemptBonds \\\n", "0 NaN NaN NaN \n", "\n", " InvestTaxExemptBondsInd InvestmentExpenseAmt InvestmentInJointVenture \\\n", "0 NaN NaN NaN \n", "\n", " InvestmentInJointVentureInd InvestmentIncome InvestmentIncomeCurrentYear \\\n", "0 false NaN NaN \n", "\n", " InvestmentIncomeGrp InvestmentIncomePriorYear InvestmentsOtherSecurities \\\n", "0 NaN NaN NaN \n", "\n", " InvestmentsOtherSecuritiesGrp InvestmentsProgramRelated \\\n", "0 NaN NaN \n", "\n", " InvestmentsProgramRelatedGrp InvestmentsPubTradedSecGrp \\\n", "0 NaN NaN \n", "\n", " InvestmentsPubTradedSecurities IsAvailable IsElectronic JointCosts \\\n", "0 NaN True True NaN \n", "\n", " JointCostsInd LandBldgEquipAccumDeprecAmt LandBldgEquipBasisNetGrp \\\n", "0 NaN NaN NaN \n", "\n", " LandBldgEquipCostOrOtherBssAmt LandBldgEquipmentAccumDeprec \\\n", "0 NaN NaN \n", "\n", " LandBuildingsEquipmentBasis LandBuildingsEquipmentBasisNet \\\n", "0 NaN NaN \n", "\n", " LastUpdated LegalDomicileStateCd LessCostOthBasisSalesExpenses \\\n", "0 2016-03-21T17:23:53 NY NaN \n", "\n", " LessCostOthBasisSalesExpnssGrp LessRentalExpenses LessRentalExpensesGrp \\\n", "0 NaN NaN NaN \n", "\n", " LicensedMoreThanOneState LicensedMoreThanOneStateInd LoanOutstandingInd \\\n", "0 NaN NaN false \n", "\n", " LoanToOfficerOrDQP LoansFromOfficersDirectors LoansFromOfficersDirectorsGrp \\\n", "0 NaN NaN NaN \n", "\n", " LobbyingActivities LobbyingActivitiesInd LocalChapters LocalChaptersInd \\\n", "0 NaN false NaN false \n", "\n", " MaterialDiversionOrMisuse MaterialDiversionOrMisuseInd \\\n", "0 NaN true \n", "\n", " MembersOrStockholders MembersOrStockholdersInd MembershipDues \\\n", "0 NaN false NaN \n", "\n", " MembershipDuesAmt MethodOfAccountingAccrual MethodOfAccountingAccrualInd \\\n", "0 NaN NaN NaN \n", "\n", " MethodOfAccountingCash MethodOfAccountingCashInd MethodOfAccountingOther \\\n", "0 NaN X NaN \n", "\n", " MethodOfAccountingOtherInd MinutesOfCommittees MinutesOfCommitteesInd \\\n", "0 NaN NaN false \n", "\n", " MinutesOfGoverningBody MinutesOfGoverningBodyInd MiscellaneousRevenue \\\n", "0 NaN false NaN \n", "\n", " MiscellaneousRevenueGrp \\\n", "0 NaN \n", "\n", " MissionDesc \\\n", "0 RAISE MONIES TO CONSTRUCT A FACILITY FOR YEAR-ROUND RECREATION PROGRAM DEDICATED TO CHILDREN AND ADULTS WITH SPECIAL NEEDS WHO RESIDE IN THE TOWN OF HEMPSTEAD. \n", "\n", " MissionDescription MoreThan5000KToIndividuals MoreThan5000KToIndividualsInd \\\n", "0 NaN NaN false \n", "\n", " MoreThan5000KToOrgInd MoreThan5000KToOrganizations \\\n", "0 false NaN \n", "\n", " MortNotesPyblSecuredInvestProp MortgNotesPyblScrdInvstPropGrp \\\n", "0 NaN NaN \n", "\n", " NameOfForeignCountry NameOfPrincipalOfficerBusiness \\\n", "0 NaN NaN \n", "\n", " NameOfPrincipalOfficerPerson NbrIndependentVotingMembers \\\n", "0 NaN NaN \n", "\n", " NbrVotingGoverningBodyMembers NbrVotingMembersGoverningBody \\\n", "0 NaN NaN \n", "\n", " NetAssetsOrFundBalancesBOY NetAssetsOrFundBalancesBOYAmt \\\n", "0 NaN NaN \n", "\n", " NetAssetsOrFundBalancesEOY NetAssetsOrFundBalancesEOYAmt \\\n", "0 NaN 290600 \n", "\n", " NetGainOrLossInvestments NetGainOrLossInvestmentsGrp \\\n", "0 NaN NaN \n", "\n", " NetIncmFromFundraisingEvtGrp NetIncomeFromFundraisingEvents \\\n", "0 NaN NaN \n", "\n", " NetIncomeFromGaming NetIncomeFromGamingGrp NetIncomeOrLoss \\\n", "0 NaN NaN NaN \n", "\n", " NetIncomeOrLossGrp NetRentalIncomeOrLoss NetRentalIncomeOrLossGrp \\\n", "0 NaN NaN NaN \n", "\n", " NetUnrelatedBusTxblIncmAmt NetUnrelatedBusinessTxblIncome \\\n", "0 NaN NaN \n", "\n", " NetUnrlzdGainsLossesInvstAmt NoListedPersonsCompensated \\\n", "0 NaN NaN \n", "\n", " NoListedPersonsCompensatedInd NonDeductibleContributions \\\n", "0 X NaN \n", "\n", " NonDeductibleDisclosure NoncashContributions NoncashContributionsAmt \\\n", "0 NaN NaN NaN \n", "\n", " NondeductibleContriDisclInd NondeductibleContributionsInd \\\n", "0 NaN false \n", "\n", " NumberFormsTransmittedWith1096 NumberIndependentVotingMembers \\\n", "0 NaN NaN \n", "\n", " NumberIndividualsGT100K NumberOf8282Filed NumberOfContractorsGT100K \\\n", "0 NaN NaN NaN \n", "\n", " NumberOfEmployees NumberW2GIncluded ObjectId Occupancy \\\n", "0 NaN NaN 201503169349304780 NaN \n", "\n", " OccupancyGrp OfficeExpenses OfficeExpensesGrp OfficerEntityWithBsnssRltnshp \\\n", "0 NaN NaN NaN NaN \n", "\n", " OfficerMailingAddress OfficerMailingAddressInd OnBehalfOfIssuer \\\n", "0 NaN false NaN \n", "\n", " OnBehalfOfIssuerInd OperateHospitalInd OrgDoesNotFollowSFAS117Ind \\\n", "0 NaN false NaN \n", "\n", " OrgFiledInLieuOfForm1041Ind Organization501c Organization501c3 \\\n", "0 NaN NaN NaN \n", "\n", " Organization501c3Ind Organization501cInd OrganizationFollowsSFAS117Ind \\\n", "0 X NaN X \n", "\n", " OrganizationName OthNotesLoansReceivableNetGrp OtherAssetsTotal \\\n", "0 ANCHOR BUILDING FUND INC NaN NaN \n", "\n", " OtherAssetsTotalGrp OtherChangesInNetAssetsAmt OtherEmployeeBenefits \\\n", "0 NaN NaN NaN \n", "\n", " OtherEmployeeBenefitsGrp OtherExpensePriorYear OtherExpenses \\\n", "0 NaN NaN NaN \n", "\n", " OtherExpensesCurrentYear OtherExpensesGrp OtherExplainInSchO OtherInd \\\n", "0 NaN NaN NaN NaN \n", "\n", " OtherLiabilities OtherLiabilitiesGrp OtherNotesLoansReceivableNet \\\n", "0 NaN NaN NaN \n", "\n", " OtherRevenueCurrentYear OtherRevenueMisc \\\n", "0 NaN NaN \n", "\n", " OtherRevenueMiscGrp \\\n", "0 {u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000', u'Desc': u'SETTLEMENT'} \n", "\n", " OtherRevenuePriorYear OtherRevenueTotalAmt OtherSalariesAndWages \\\n", "0 NaN 325000 NaN \n", "\n", " OtherSalariesAndWagesGrp OtherWebsite OtherWebsiteInd OwnWebsite \\\n", "0 NaN NaN NaN NaN \n", "\n", " OwnWebsiteInd PYBenefitsPaidToMembersAmt PYContributionsGrantsAmt \\\n", "0 NaN NaN NaN \n", "\n", " PYExcessBenefitTransInd PYGrantsAndSimilarPaidAmt PYInvestmentIncomeAmt \\\n", "0 false NaN NaN \n", "\n", " PYOtherExpensesAmt PYOtherRevenueAmt PYProgramServiceRevenueAmt \\\n", "0 NaN NaN NaN \n", "\n", " PYRevenuesLessExpensesAmt PYSalariesCompEmpBnftPaidAmt PYTotalExpensesAmt \\\n", "0 NaN NaN NaN \n", "\n", " PYTotalProfFndrsngExpnsAmt PYTotalRevenueAmt PaidInCapSrplsLandBldgEqpFund \\\n", "0 NaN NaN NaN \n", "\n", " PartialLiquidation PartialLiquidationInd PayPremiumsPrsnlBnftCntrctInd \\\n", "0 NaN false false \n", "\n", " PaymentsToAffiliates PaymentsToAffiliatesGrp PayrollTaxes PayrollTaxesGrp \\\n", "0 NaN NaN NaN NaN \n", "\n", " PdInCapSrplsLandBldgEqpFundGrp PensionPlanContributions \\\n", "0 NaN NaN \n", "\n", " PensionPlanContributionsGrp PermanentlyRestrictedNetAssets \\\n", "0 NaN NaN \n", "\n", " PermanentlyRstrNetAssetsGrp PledgesAndGrantsReceivable \\\n", "0 NaN NaN \n", "\n", " PledgesAndGrantsReceivableGrp PoliciesReferenceChapters \\\n", "0 NaN NaN \n", "\n", " PoliciesReferenceChaptersInd PoliticalActivities PoliticalCampaignActyInd \\\n", "0 NaN NaN false \n", "\n", " PremiumsPaid PrepaidExpensesDeferredCharges PrepaidExpensesDefrdChargesGrp \\\n", "0 NaN NaN NaN \n", "\n", " PrincipalOfficerNm PriorExcessBenefitTransaction PriorPeriodAdjustmentsAmt \\\n", "0 JOHN MALLOY NaN NaN \n", "\n", " ProfessionalFundraising ProfessionalFundraisingInd ProgSrvcAccomActy2Grp \\\n", "0 NaN false NaN \n", "\n", " ProgSrvcAccomActy3Grp ProgSrvcAccomActyOtherGrp ProgramServiceRevenue \\\n", "0 NaN NaN NaN \n", "\n", " ProgramServiceRevenueCY ProgramServiceRevenueGrp \\\n", "0 NaN NaN \n", "\n", " ProgramServiceRevenuePriorYear ProhibitedTaxShelterTrans \\\n", "0 NaN NaN \n", "\n", " ProhibitedTaxShelterTransInd PymtTravelEntrtnmntPubOfclGrp \\\n", "0 false NaN \n", "\n", " QuidProQuoContriDisclInd QuidProQuoContributions QuidProQuoContributionsInd \\\n", "0 NaN NaN false \n", "\n", " QuidProQuoDisclosure RcvFndsToPayPrsnlBnftCntrctInd \\\n", "0 NaN false \n", "\n", " RcvblFromDisqualifiedPrsnGrp ReceivablesFromDisqualPersons \\\n", "0 NaN NaN \n", "\n", " ReceivablesFromOfficersEtc ReceivablesFromOfficersEtcGrp \\\n", "0 NaN NaN \n", "\n", " ReconcilationDonatedServices ReconcilationInvestExpenses \\\n", "0 NaN NaN \n", "\n", " ReconcilationOtherChanges ReconcilationPriorAdjustment \\\n", "0 NaN NaN \n", "\n", " ReconcilationRevenueExpenses ReconcilationRevenueExpnssAmt \\\n", "0 NaN 290600 \n", "\n", " ReconciliationUnrealizedInvest RegularMonitoringEnforcement \\\n", "0 NaN NaN \n", "\n", " RegularMonitoringEnfrcInd RelatedEntity RelatedEntityInd \\\n", "0 NaN NaN false \n", "\n", " RelatedOrgControlledEntity RelatedOrganizationCtrlEntInd \\\n", "0 NaN false \n", "\n", " RelatedOrganizations RelatedOrganizationsAmt RentalIncomeOrLoss \\\n", "0 NaN NaN NaN \n", "\n", " RentalIncomeOrLossGrp ReportFin48Footnote ReportInvestOthSecurities \\\n", "0 NaN NaN NaN \n", "\n", " ReportInvestmentsOtherSecInd ReportLandBldgEquip \\\n", "0 false NaN \n", "\n", " ReportLandBuildingEquipmentInd ReportOtherAssets ReportOtherAssetsInd \\\n", "0 false NaN false \n", "\n", " ReportOtherLiabilities ReportOtherLiabilitiesInd ReportProgRelInvest \\\n", "0 NaN false NaN \n", "\n", " ReportProgramRelatedInvstInd RetainedEarningsEndowmentEtc Revenue \\\n", "0 false NaN NaN \n", "\n", " RevenueAmt RevenuesLessExpensesCY RevenuesLessExpensesPriorYear Royalties \\\n", "0 NaN NaN NaN NaN \n", "\n", " RoyaltiesGrp RoyaltiesRevenue RoyaltiesRevenueGrp \\\n", "0 NaN NaN NaN \n", "\n", " RtnEarnEndowmentIncmOthFndsGrp SalariesEtcCurrentYear SalariesEtcPriorYear \\\n", "0 NaN NaN NaN \n", "\n", " SavingsAndTempCashInvestments SavingsAndTempCashInvstGrp ScheduleBRequired \\\n", "0 NaN NaN NaN \n", "\n", " ScheduleBRequiredInd ScheduleJRequired ScheduleJRequiredInd \\\n", "0 false NaN false \n", "\n", " ScheduleORequired ScheduleORequiredInd School SchoolOperatingInd \\\n", "0 NaN true NaN false \n", "\n", " SignificantChange SignificantChangeInd SignificantNewProgramServices \\\n", "0 NaN true NaN \n", "\n", " SignificantNewProgramSrvcInd StateLegalDomicile \\\n", "0 false NaN \n", "\n", " StatesWhereCopyOfReturnIsFiled StatesWhereCopyOfReturnIsFldCd \\\n", "0 NaN NY \n", "\n", " SubjectToProxyTax SubjectToProxyTaxInd SubmittedOn TaxExemptBondLiabilities \\\n", "0 NaN false 2016-01-30 NaN \n", "\n", " TaxExemptBondLiabilitiesGrp TaxExemptBonds TaxExemptBondsInd TaxPeriod \\\n", "0 NaN NaN false 201412 \n", "\n", " TaxableDistributions TaxableDistributionsInd TaxablePartyNotification \\\n", "0 NaN NaN NaN \n", "\n", " TaxablePartyNotificationInd TempOrPermanentEndowmentsInd \\\n", "0 false false \n", "\n", " TemporarilyRestrictedNetAssets TemporarilyRstrNetAssetsGrp \\\n", "0 NaN NaN \n", "\n", " TermOrPermanentEndowments TerminateOperationsInd Terminated \\\n", "0 NaN false NaN \n", "\n", " TerminationOrContraction TheBooksAreInCareOf \\\n", "0 NaN NaN \n", "\n", " TotLiabNetAssetsFundBalanceGrp TotReportableCompRltdOrgAmt \\\n", "0 {u'BOYAmt': u'0', u'EOYAmt': u'290600'} NaN \n", "\n", " TotalAssets TotalAssetsBOY TotalAssetsBOYAmt TotalAssetsEOY \\\n", "0 NaN NaN NaN NaN \n", "\n", " TotalAssetsEOYAmt TotalAssetsGrp TotalCompGT150K \\\n", "0 290600 {u'BOYAmt': u'0', u'EOYAmt': u'290600'} NaN \n", "\n", " TotalCompGreaterThan150KInd TotalContributions TotalContributionsAmt \\\n", "0 false NaN NaN \n", "\n", " TotalEmployeeCnt TotalExpensesCurrentYear TotalExpensesPriorYear \\\n", "0 0 NaN NaN \n", "\n", " TotalFunctionalExpenses \\\n", "0 NaN \n", "\n", " TotalFunctionalExpensesGrp \\\n", "0 {u'TotalAmt': u'34400', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'34400', u'FundraisingAmt': u'0'} \n", "\n", " TotalFundrsngExpCurrentYear TotalGrossUBI TotalGrossUBIAmt TotalJointCosts \\\n", "0 NaN NaN 0 NaN \n", "\n", " TotalJointCostsGrp TotalLiabNetAssetsFundBalances TotalLiabilities \\\n", "0 NaN NaN NaN \n", "\n", " TotalLiabilitiesBOY TotalLiabilitiesBOYAmt TotalLiabilitiesEOY \\\n", "0 NaN NaN NaN \n", "\n", " TotalLiabilitiesEOYAmt TotalLiabilitiesGrp \\\n", "0 0 {u'BOYAmt': u'0', u'EOYAmt': u'0'} \n", "\n", " TotalNbrEmployees TotalNbrVolunteers \\\n", "0 NaN NaN \n", "\n", " TotalNetAssetsFundBalanceGrp TotalNetAssetsFundBalances \\\n", "0 {u'BOYAmt': u'0', u'EOYAmt': u'290600'} NaN \n", "\n", " TotalOfOtherProgramServiceExp TotalOfOtherProgramServiceGrnt \\\n", "0 NaN NaN \n", "\n", " TotalOfOtherProgramServiceRev TotalOthProgramServiceRevGrp \\\n", "0 NaN NaN \n", "\n", " TotalOthProgramServiceRevenue TotalOtherCompensation \\\n", "0 NaN NaN \n", "\n", " TotalOtherCompensationAmt TotalOtherProgSrvcExpenseAmt \\\n", "0 NaN NaN \n", "\n", " TotalOtherProgSrvcGrantAmt TotalOtherProgSrvcRevenueAmt TotalOtherRevenue \\\n", "0 NaN NaN NaN \n", "\n", " TotalProfFundrsngExpCY TotalProfFundrsngExpPriorYear \\\n", "0 NaN NaN \n", "\n", " TotalProgramServiceExpense TotalProgramServiceExpensesAmt \\\n", "0 NaN NaN \n", "\n", " TotalProgramServiceRevenue TotalProgramServiceRevenueAmt \\\n", "0 NaN NaN \n", "\n", " TotalReportableCompFrmRltdOrgs TotalReportableCompFromOrg \\\n", "0 NaN NaN \n", "\n", " TotalReportableCompFromOrgAmt TotalRevenue TotalRevenueCurrentYear \\\n", "0 NaN NaN NaN \n", "\n", " TotalRevenueGrp \\\n", "0 {u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000'} \n", "\n", " TotalRevenuePriorYear TotalVolunteersCnt TransactionRelatedEntity \\\n", "0 NaN NaN NaN \n", "\n", " TransactionWithControlEntInd TransfersToExemptNonChrtblOrg Travel \\\n", "0 NaN NaN NaN \n", "\n", " TravelEntrtnmntPublicOfficials TravelGrp TrnsfrExmptNonChrtblRltdOrgInd \\\n", "0 NaN NaN false \n", "\n", " TypeOfOrganizationCorpInd TypeOfOrganizationCorporation \\\n", "0 X NaN \n", "\n", " TypeOfOrganizationOther TypeOfOrganizationOtherInd TypeOfOrganizationTrust \\\n", "0 NaN NaN NaN \n", "\n", " TypeOfOrganizationTrustInd \\\n", "0 NaN \n", "\n", " URL \\\n", "0 https://s3.amazonaws.com/irs-form-990/201503169349304780_public.xml \n", "\n", " USAddress \\\n", "0 {u'CityNm': u'LIDO BEACH', u'StateAbbreviationCd': u'NY', u'ZIPCd': u'11561', u'AddressLine1Txt': u'630 LIDO BLVD'} \n", "\n", " UnrelatedBusIncmOverLimitInd UnrelatedBusinessIncome UnrestrictedNetAssets \\\n", "0 false NaN NaN \n", "\n", " UnrestrictedNetAssetsGrp UnsecuredNotesLoansPayable \\\n", "0 {u'EOYAmt': u'290600'} NaN \n", "\n", " UnsecuredNotesLoansPayableGrp UponRequest UponRequestInd \\\n", "0 NaN NaN X \n", "\n", " VotingMembersGoverningBodyCnt VotingMembersIndependentCnt WebSite \\\n", "0 2 2 NaN \n", "\n", " WebsiteAddressTxt WhistleblowerPolicy WhistleblowerPolicyInd \\\n", "0 WWW.CAMPANCHOR.ORG NaN false \n", "\n", " WrittenPolicyOrProcedure WrittenPolicyOrProcedureInd YearFormation \\\n", "0 NaN NaN NaN \n", "\n", " _id \n", "0 57cce7823ffc5a55fe7f54b5 " ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.DataFrame(list(filings.find()))\n", "print '# of columns:', len(df.columns)\n", "print '# of observations:', len(df)\n", "df.head(1)" ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "538\n", "538\n", "FY2013 120\n", "FY2014 117\n", "FY2012 112\n", "FY2011 97\n", "FY2010 77\n", "FY2015 15\n", "Name: FYE, dtype: int64\n" ] } ], "source": [ "print df['TaxPeriod'].value_counts().sum()\n", "df['FYE'] = 'FY' + df['TaxPeriod'].str[:4]\n", "print df['FYE'].value_counts().sum()\n", "print df['FYE'].value_counts()" ] }, { "cell_type": "code", "execution_count": 57, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "538\n" ] } ], "source": [ "print df['EIN'].value_counts().sum()" ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "collapsed": true }, "outputs": [], "source": [ "columns = ['EIN', 'FYE', 'AuditCommittee', 'AuditCommitteeInd', 'CYContributionsGrantsAmt', \n", " 'CYTotalFundraisingExpenseAmt', 'GoverningBodyVotingMembersCnt', 'IndependentAuditFinancialStmt',\n", " 'IndependentAuditFinclStmtInd', 'IndependentVotingMemberCnt', 'LoanToOfficerOrDQP',\n", " 'NbrIndependentVotingMembers', 'TotalContributions', 'TotalContributionsAmt',\n", " 'TotalFundrsngExpCurrentYear', 'TotalFunctionalExpensesGrp', 'FundraisingAmt', \n", " 'ConsolidatedAuditFinancialStmt', 'ConsolidatedAuditFinclStmtInd',\n", " 'AuditedFinancialStmtAttInd', 'FSAudited', 'FSAuditedBasis', 'FSAuditedBasisGrp', 'FSAuditedInd', \n", " 'IndependentVotingMemberCnt', 'NumberIndependentVotingMembers', 'VotingMembersIndependentCnt'\n", " ]" ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "271\n", "138\n", "194\n", "194\n", "194\n", "344\n", "194 \n", "\n", "194\n", "344\n", "344\n", "323\n", "176\n", "344\n", "194\n", "32\n" ] } ], "source": [ "print df['AuditCommittee'].value_counts().sum()\n", "print df['AuditCommitteeInd'].value_counts().sum()\n", "print df['CYContributionsGrantsAmt'].value_counts().sum()\n", "print df['CYTotalFundraisingExpenseAmt'].value_counts().sum()\n", "print df['GoverningBodyVotingMembersCnt'].value_counts().sum()\n", "print len(df[df['IndependentAuditFinancialStmt'].notnull()])\n", "print len(df[df['IndependentAuditFinclStmtInd'].notnull()]), '\\n'\n", "print df['IndependentVotingMemberCnt'].value_counts().sum()\n", "#print df['LoanToOfficerOrDQP'].value_counts().sum()\n", "print len(df[df['LoanToOfficerOrDQP'].notnull()])\n", "print df['NbrIndependentVotingMembers'].value_counts().sum()\n", "print df['TotalContributions'].value_counts().sum()\n", "print df['TotalContributionsAmt'].value_counts().sum()\n", "print df['TotalFundrsngExpCurrentYear'].value_counts().sum()\n", "print len(df[df['TotalFunctionalExpensesGrp'].notnull()])\n", "print df['FundraisingAmt'].value_counts().sum()\n", "\n", "#'ConsolidatedAuditFinancialStmt', u'ConsolidatedAuditFinclStmtInd'\n", "# u'AuditedFinancialStmtAttInd',\n", "#FSAudited', u'FSAuditedBasis', u'FSAuditedBasisGrp', u'FSAuditedInd', \n", "#'IndependentVotingMemberCnt'\n", "#'NumberIndependentVotingMembers'\n", "#'VotingMembersIndependentCnt'" ] }, { "cell_type": "code", "execution_count": 82, "metadata": { "collapsed": true }, "outputs": [], "source": [ "columns = ['EIN', 'FYE', 'AuditCommittee', 'AuditCommitteeInd', \n", " 'GoverningBodyVotingMembersCnt', 'VotingMembersGoverningBodyCnt', #THESE TWO SEEM TO BE THE SAME\n", " 'IndependentVotingMemberCnt', 'VotingMembersIndependentCnt', #THESE TWO SEEM TO BE THE SAME\n", " 'NbrVotingGoverningBodyMembers', 'NbrVotingMembersGoverningBody', #THESE TWO SEEM TO BE THE SAME \n", " 'NumberIndependentVotingMembers', 'NbrIndependentVotingMembers', #THESE TWO SEEM TO BE THE SAME \n", " 'IndependentAuditFinancialStmt', 'ConsolidatedAuditFinancialStmt', 'FSAudited',\n", " 'IndependentAuditFinclStmtInd', 'ConsolidatedAuditFinclStmtInd', 'FSAuditedInd', \n", " 'AuditedFinancialStmtAttInd', 'FSAuditedBasis', 'FSAuditedBasisGrp', \n", " 'LoanToOfficerOrDQP', 'FundraisingAmt', 'TotalFundrsngExpCurrentYear', \n", " 'CYTotalFundraisingExpenseAmt', 'TotalFunctionalExpensesGrp', \n", " 'TotalContributions', 'TotalContributionsAmt', 'CYContributionsGrantsAmt', \n", " ]" ] }, { "cell_type": "code", "execution_count": 83, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "194\n", "194\n", "194\n", "194 \n", "\n", "344\n", "344\n", "344\n", "344\n" ] } ], "source": [ "print df[columns[4]].value_counts().sum()\n", "print df[columns[5]].value_counts().sum()\n", "print df[columns[6]].value_counts().sum()\n", "print df[columns[7]].value_counts().sum(), '\\n'\n", "print df[columns[8]].value_counts().sum()\n", "print df[columns[9]].value_counts().sum()\n", "print df[columns[10]].value_counts().sum()\n", "print df[columns[11]].value_counts().sum()" ] }, { "cell_type": "code", "execution_count": 84, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINFYEAuditCommitteeAuditCommitteeIndGoverningBodyVotingMembersCntVotingMembersGoverningBodyCntIndependentVotingMemberCntVotingMembersIndependentCntNbrVotingGoverningBodyMembersNbrVotingMembersGoverningBodyNumberIndependentVotingMembersNbrIndependentVotingMembersIndependentAuditFinancialStmtConsolidatedAuditFinancialStmtFSAuditedIndependentAuditFinclStmtIndConsolidatedAuditFinclStmtIndFSAuditedIndAuditedFinancialStmtAttIndFSAuditedBasisFSAuditedBasisGrpLoanToOfficerOrDQPFundraisingAmtTotalFundrsngExpCurrentYearCYTotalFundraisingExpenseAmtTotalFunctionalExpensesGrpTotalContributionsTotalContributionsAmtCYContributionsGrantsAmt
2581494135FY2014NaNfalse4400NaNNaNNaNNaNNaNNaNNaN{u'#text': u'false', u'@referenceDocumentId': u'IRS990ScheduleD'}{u'#text': u'false', u'@referenceDocumentId': u'IRS990ScheduleD'}trueNaNNaN{u'SeparateBasisFinclStmtInd': u'X'}NaNNaNNaN359301{u'TotalAmt': u'1382846', u'ProgramServicesAmt': u'830309', u'ManagementAndGeneralAmt': u'193236', u'FundraisingAmt': u'359301'}NaN12165361216536
3731284606FY20121NaNNaNNaNNaNNaN6644{u'#text': u'1', u'@referenceDocumentId': u'RetDoc1040000001'}{u'#text': u'0', u'@referenceDocumentId': u'RetDoc1040000001'}1NaNNaNNaNNaNNaNNaN0NaN392027NaNNaN9299628NaNNaN
4262224994FY2013NaN15544NaNNaNNaNNaNNaNNaNNaN{u'#text': u'1', u'@referenceDocumentId': u'RetDoc1040000001'}{u'#text': u'0', u'@referenceDocumentId': u'RetDoc1040000001'}1NaNNaN{u'SeparateBasisFinclStmtInd': u'X'}NaNNaNNaN17564{u'TotalAmt': u'608945', u'ProgramServicesAmt': u'495257', u'ManagementAndGeneralAmt': u'96124', u'FundraisingAmt': u'17564'}NaN359003359003
\n", "
" ], "text/plain": [ " EIN FYE AuditCommittee AuditCommitteeInd \\\n", "2 581494135 FY2014 NaN false \n", "3 731284606 FY2012 1 NaN \n", "4 262224994 FY2013 NaN 1 \n", "\n", " GoverningBodyVotingMembersCnt VotingMembersGoverningBodyCnt \\\n", "2 4 4 \n", "3 NaN NaN \n", "4 5 5 \n", "\n", " IndependentVotingMemberCnt VotingMembersIndependentCnt \\\n", "2 0 0 \n", "3 NaN NaN \n", "4 4 4 \n", "\n", " NbrVotingGoverningBodyMembers NbrVotingMembersGoverningBody \\\n", "2 NaN NaN \n", "3 6 6 \n", "4 NaN NaN \n", "\n", " NumberIndependentVotingMembers NbrIndependentVotingMembers \\\n", "2 NaN NaN \n", "3 4 4 \n", "4 NaN NaN \n", "\n", " IndependentAuditFinancialStmt \\\n", "2 NaN \n", "3 {u'#text': u'1', u'@referenceDocumentId': u'RetDoc1040000001'} \n", "4 NaN \n", "\n", " ConsolidatedAuditFinancialStmt FSAudited \\\n", "2 NaN NaN \n", "3 {u'#text': u'0', u'@referenceDocumentId': u'RetDoc1040000001'} 1 \n", "4 NaN NaN \n", "\n", " IndependentAuditFinclStmtInd \\\n", "2 {u'#text': u'false', u'@referenceDocumentId': u'IRS990ScheduleD'} \n", "3 NaN \n", "4 {u'#text': u'1', u'@referenceDocumentId': u'RetDoc1040000001'} \n", "\n", " ConsolidatedAuditFinclStmtInd \\\n", "2 {u'#text': u'false', u'@referenceDocumentId': u'IRS990ScheduleD'} \n", "3 NaN \n", "4 {u'#text': u'0', u'@referenceDocumentId': u'RetDoc1040000001'} \n", "\n", " FSAuditedInd AuditedFinancialStmtAttInd FSAuditedBasis \\\n", "2 true NaN NaN \n", "3 NaN NaN NaN \n", "4 1 NaN NaN \n", "\n", " FSAuditedBasisGrp LoanToOfficerOrDQP FundraisingAmt \\\n", "2 {u'SeparateBasisFinclStmtInd': u'X'} NaN NaN \n", "3 NaN 0 NaN \n", "4 {u'SeparateBasisFinclStmtInd': u'X'} NaN NaN \n", "\n", " TotalFundrsngExpCurrentYear CYTotalFundraisingExpenseAmt \\\n", "2 NaN 359301 \n", "3 392027 NaN \n", "4 NaN 17564 \n", "\n", " TotalFunctionalExpensesGrp \\\n", "2 {u'TotalAmt': u'1382846', u'ProgramServicesAmt': u'830309', u'ManagementAndGeneralAmt': u'193236', u'FundraisingAmt': u'359301'} \n", "3 NaN \n", "4 {u'TotalAmt': u'608945', u'ProgramServicesAmt': u'495257', u'ManagementAndGeneralAmt': u'96124', u'FundraisingAmt': u'17564'} \n", "\n", " TotalContributions TotalContributionsAmt CYContributionsGrantsAmt \n", "2 NaN 1216536 1216536 \n", "3 9299628 NaN NaN \n", "4 NaN 359003 359003 " ] }, "execution_count": 84, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[columns][2:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Audit Committee" ] }, { "cell_type": "code", "execution_count": 78, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "271\n", "409\n" ] }, { "data": { "text/plain": [ "1 212\n", "true 144\n", "false 32\n", "0 21\n", "Name: audit_committee, dtype: int64" ] }, "execution_count": 78, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['audit_committee'] = np.nan\n", "print len(df[df['audit_committee'].notnull()])\n", "df['audit_committee'] = np.where(df['AuditCommittee'].notnull(), df['AuditCommittee'], \n", " df['audit_committee'] )\n", "print len(df[df['audit_committee'].notnull()])\n", "df['audit_committee'] = np.where( ((df['audit_committee'].isnull()) & \n", " (df['AuditCommitteeInd'].notnull())), \n", " df['AuditCommitteeInd'], df['audit_committee'] )\n", "print len(df[df['audit_committee'].notnull()])\n", "df['audit_committee'].value_counts()" ] }, { "cell_type": "code", "execution_count": 79, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "409\n" ] }, { "data": { "text/plain": [ "1 356\n", "0 53\n", "Name: audit_committee, dtype: int64" ] }, "execution_count": 79, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['audit_committee'] = np.where( df['audit_committee']=='true', 1, df['audit_committee'] )\n", "df['audit_committee'] = np.where( df['audit_committee']=='1', 1, df['audit_committee'] )\n", "df['audit_committee'] = np.where( df['audit_committee']=='false', 0, df['audit_committee'] )\n", "df['audit_committee'] = np.where( df['audit_committee']=='0', 0, df['audit_committee'] )\n", "print len(df[df['audit_committee'].notnull()])\n", "df['audit_committee'].value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Board Size " ] }, { "cell_type": "code", "execution_count": 87, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "194\n", "538\n" ] }, { "data": { "text/plain": [ "3 92\n", "4 56\n", "5 44\n", "7 39\n", "6 27\n", "9 24\n", "8 23\n", "11 19\n", "10 18\n", "12 17\n", "17 17\n", "2 16\n", "30 12\n", "13 12\n", "23 12\n", "25 12\n", "15 12\n", "21 10\n", "16 8\n", "22 7\n", "18 6\n", "19 6\n", "27 5\n", "14 5\n", "36 4\n", "24 4\n", "29 3\n", "49 2\n", "60 2\n", "26 2\n", "82 2\n", "44 2\n", "0 2\n", "80 2\n", "31 2\n", "28 2\n", "83 1\n", "59 1\n", "62 1\n", "35 1\n", "34 1\n", "1 1\n", "20 1\n", "33 1\n", "40 1\n", "79 1\n", "Name: board_size, dtype: int64" ] }, "execution_count": 87, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['board_size'] = np.nan\n", "print len(df[df['board_size'].notnull()])\n", "df['board_size'] = np.where(df['GoverningBodyVotingMembersCnt'].notnull(), df['GoverningBodyVotingMembersCnt'], \n", " df['board_size'] )\n", "print len(df[df['board_size'].notnull()])\n", "df['board_size'] = np.where( ((df['board_size'].isnull()) & \n", " (df['NbrVotingGoverningBodyMembers'].notnull())), \n", " df['NbrVotingGoverningBodyMembers'], df['board_size'] )\n", "print len(df[df['board_size'].notnull()])\n", "df['board_size'].value_counts()\n", "#'GoverningBodyVotingMembersCnt', 'VotingMembersGoverningBodyCnt', #THESE TWO SEEM TO BE THE SAME\n", "#'IndependentVotingMemberCnt', 'VotingMembersIndependentCnt', #THESE TWO SEEM TO BE THE SAME\n", "#'NbrVotingGoverningBodyMembers', 'NbrVotingMembersGoverningBody', #THESE TWO SEEM TO BE THE SAME\n", "#'NumberIndependentVotingMembers', 'NbrIndependentVotingMembers', #THESE TWO SEEM TO BE THE SAME " ] }, { "cell_type": "code", "execution_count": 90, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "194\n", "538\n" ] }, { "data": { "text/plain": [ "0 88\n", "3 48\n", "4 42\n", "7 32\n", "2 30\n", "5 28\n", "9 28\n", "8 23\n", "6 18\n", "11 17\n", "16 16\n", "12 15\n", "25 11\n", "10 10\n", "15 9\n", "23 9\n", "30 8\n", "17 8\n", "19 8\n", "1 8\n", "13 8\n", "24 8\n", "14 7\n", "21 7\n", "18 7\n", "22 6\n", "28 5\n", "20 5\n", "27 4\n", "29 4\n", "36 3\n", "73 2\n", "59 2\n", "44 2\n", "34 2\n", "32 2\n", "72 1\n", "74 1\n", "83 1\n", "76 1\n", "58 1\n", "40 1\n", "31 1\n", "35 1\n", "Name: independent_members, dtype: int64" ] }, "execution_count": 90, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['independent_members'] = np.nan\n", "print len(df[df['independent_members'].notnull()])\n", "df['independent_members'] = np.where(df['IndependentVotingMemberCnt'].notnull(), df['IndependentVotingMemberCnt'], \n", " df['independent_members'] )\n", "print len(df[df['independent_members'].notnull()])\n", "df['independent_members'] = np.where( ((df['independent_members'].isnull()) & \n", " (df['NumberIndependentVotingMembers'].notnull())), \n", " df['NumberIndependentVotingMembers'], df['independent_members'] )\n", "print len(df[df['board_size'].notnull()])\n", "df['independent_members'].value_counts()\n", "#'GoverningBodyVotingMembersCnt', 'VotingMembersGoverningBodyCnt', #THESE TWO SEEM TO BE THE SAME\n", "#'IndependentVotingMemberCnt', 'VotingMembersIndependentCnt', #THESE TWO SEEM TO BE THE SAME\n", "#'NbrVotingGoverningBodyMembers', 'NbrVotingMembersGoverningBody', #THESE TWO SEEM TO BE THE SAME\n", "#'NumberIndependentVotingMembers', 'NbrIndependentVotingMembers', #THESE TWO SEEM TO BE THE SAME " ] }, { "cell_type": "code", "execution_count": 93, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINFYEaudit_committeeboard_sizeindependent_membersIndependentAuditFinancialStmtConsolidatedAuditFinancialStmtFSAuditedIndependentAuditFinclStmtIndConsolidatedAuditFinclStmtIndFSAuditedIndAuditedFinancialStmtAttIndFSAuditedBasisFSAuditedBasisGrpLoanToOfficerOrDQPFundraisingAmtTotalFundrsngExpCurrentYearCYTotalFundraisingExpenseAmtTotalFunctionalExpensesGrpTotalContributionsTotalContributionsAmtCYContributionsGrantsAmt
2581494135FY2014040NaNNaNNaN{u'#text': u'false', u'@referenceDocumentId': u'IRS990ScheduleD'}{u'#text': u'false', u'@referenceDocumentId': u'IRS990ScheduleD'}trueNaNNaN{u'SeparateBasisFinclStmtInd': u'X'}NaNNaNNaN359301{u'TotalAmt': u'1382846', u'ProgramServicesAmt': u'830309', u'ManagementAndGeneralAmt': u'193236', u'FundraisingAmt': u'359301'}NaN12165361216536
3731284606FY2012164{u'#text': u'1', u'@referenceDocumentId': u'RetDoc1040000001'}{u'#text': u'0', u'@referenceDocumentId': u'RetDoc1040000001'}1NaNNaNNaNNaNNaNNaN0NaN392027NaNNaN9299628NaNNaN
4262224994FY2013154NaNNaNNaN{u'#text': u'1', u'@referenceDocumentId': u'RetDoc1040000001'}{u'#text': u'0', u'@referenceDocumentId': u'RetDoc1040000001'}1NaNNaN{u'SeparateBasisFinclStmtInd': u'X'}NaNNaNNaN17564{u'TotalAmt': u'608945', u'ProgramServicesAmt': u'495257', u'ManagementAndGeneralAmt': u'96124', u'FundraisingAmt': u'17564'}NaN359003359003
\n", "
" ], "text/plain": [ " EIN FYE audit_committee board_size independent_members \\\n", "2 581494135 FY2014 0 4 0 \n", "3 731284606 FY2012 1 6 4 \n", "4 262224994 FY2013 1 5 4 \n", "\n", " IndependentAuditFinancialStmt \\\n", "2 NaN \n", "3 {u'#text': u'1', u'@referenceDocumentId': u'RetDoc1040000001'} \n", "4 NaN \n", "\n", " ConsolidatedAuditFinancialStmt FSAudited \\\n", "2 NaN NaN \n", "3 {u'#text': u'0', u'@referenceDocumentId': u'RetDoc1040000001'} 1 \n", "4 NaN NaN \n", "\n", " IndependentAuditFinclStmtInd \\\n", "2 {u'#text': u'false', u'@referenceDocumentId': u'IRS990ScheduleD'} \n", "3 NaN \n", "4 {u'#text': u'1', u'@referenceDocumentId': u'RetDoc1040000001'} \n", "\n", " ConsolidatedAuditFinclStmtInd \\\n", "2 {u'#text': u'false', u'@referenceDocumentId': u'IRS990ScheduleD'} \n", "3 NaN \n", "4 {u'#text': u'0', u'@referenceDocumentId': u'RetDoc1040000001'} \n", "\n", " FSAuditedInd AuditedFinancialStmtAttInd FSAuditedBasis \\\n", "2 true NaN NaN \n", "3 NaN NaN NaN \n", "4 1 NaN NaN \n", "\n", " FSAuditedBasisGrp LoanToOfficerOrDQP FundraisingAmt \\\n", "2 {u'SeparateBasisFinclStmtInd': u'X'} NaN NaN \n", "3 NaN 0 NaN \n", "4 {u'SeparateBasisFinclStmtInd': u'X'} NaN NaN \n", "\n", " TotalFundrsngExpCurrentYear CYTotalFundraisingExpenseAmt \\\n", "2 NaN 359301 \n", "3 392027 NaN \n", "4 NaN 17564 \n", "\n", " TotalFunctionalExpensesGrp \\\n", "2 {u'TotalAmt': u'1382846', u'ProgramServicesAmt': u'830309', u'ManagementAndGeneralAmt': u'193236', u'FundraisingAmt': u'359301'} \n", "3 NaN \n", "4 {u'TotalAmt': u'608945', u'ProgramServicesAmt': u'495257', u'ManagementAndGeneralAmt': u'96124', u'FundraisingAmt': u'17564'} \n", "\n", " TotalContributions TotalContributionsAmt CYContributionsGrantsAmt \n", "2 NaN 1216536 1216536 \n", "3 9299628 NaN NaN \n", "4 NaN 359003 359003 " ] }, "execution_count": 93, "metadata": {}, "output_type": "execute_result" } ], "source": [ "columns = ['EIN', 'FYE', \n", " #'AuditCommittee', 'AuditCommitteeInd', \n", " 'audit_committee',\n", " 'board_size', 'independent_members',\n", " #'GoverningBodyVotingMembersCnt', 'VotingMembersGoverningBodyCnt', #THESE TWO SEEM TO BE THE SAME\n", " #'IndependentVotingMemberCnt', 'VotingMembersIndependentCnt', #THESE TWO SEEM TO BE THE SAME\n", " #'NbrVotingGoverningBodyMembers', 'NbrVotingMembersGoverningBody', #THESE TWO SEEM TO BE THE SAME \n", " #'NumberIndependentVotingMembers', 'NbrIndependentVotingMembers', #THESE TWO SEEM TO BE THE SAME \n", " 'IndependentAuditFinancialStmt', 'ConsolidatedAuditFinancialStmt', 'FSAudited',\n", " 'IndependentAuditFinclStmtInd', 'ConsolidatedAuditFinclStmtInd', 'FSAuditedInd', \n", " 'AuditedFinancialStmtAttInd', 'FSAuditedBasis', 'FSAuditedBasisGrp', \n", " 'LoanToOfficerOrDQP', 'FundraisingAmt', 'TotalFundrsngExpCurrentYear', \n", " 'CYTotalFundraisingExpenseAmt', 'TotalFunctionalExpensesGrp', \n", " 'TotalContributions', 'TotalContributionsAmt', 'CYContributionsGrantsAmt', \n", " ]\n", "df[columns][2:5]" ] }, { "cell_type": "code", "execution_count": 95, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "344 \n", "\n", "194\n" ] } ], "source": [ "#print df[columns[5]].value_counts().sum()\n", "#print df[columns[6]].value_counts().sum()\n", "print df[columns[7]].value_counts().sum(), '\\n'\n", "#print df[columns[8]].value_counts().sum()\n", "#print df[columns[9]].value_counts().sum()\n", "print df[columns[10]].value_counts().sum()\n", "#print df[columns[11]].value_counts().sum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Audited Financial Statement" ] }, { "cell_type": "code", "execution_count": 98, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "344\n", "538\n" ] }, { "data": { "text/plain": [ "1 232\n", "true 148\n", "false 107\n", "0 51\n", "Name: audited_financials, dtype: int64" ] }, "execution_count": 98, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['audited_financials'] = np.nan\n", "print len(df[df['audited_financials'].notnull()])\n", "df['audited_financials'] = np.where(df['FSAudited'].notnull(), df['FSAudited'], \n", " df['audited_financials'] )\n", "print len(df[df['audited_financials'].notnull()])\n", "df['audited_financials'] = np.where( ((df['audited_financials'].isnull()) & \n", " (df['FSAuditedInd'].notnull())), \n", " df['FSAuditedInd'], df['audited_financials'] )\n", "print len(df[df['audited_financials'].notnull()])\n", "df['audited_financials'].value_counts()" ] }, { "cell_type": "code", "execution_count": 99, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "538\n" ] }, { "data": { "text/plain": [ "1 380\n", "0 158\n", "Name: audited_financials, dtype: int64" ] }, "execution_count": 99, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['audited_financials'] = np.where( df['audited_financials']=='true', 1, df['audited_financials'] )\n", "df['audited_financials'] = np.where( df['audited_financials']=='1', 1, df['audited_financials'] )\n", "df['audited_financials'] = np.where( df['audited_financials']=='false', 0, df['audited_financials'] )\n", "df['audited_financials'] = np.where( df['audited_financials']=='0', 0, df['audited_financials'] )\n", "print len(df[df['audited_financials'].notnull()])\n", "df['audited_financials'].value_counts()" ] }, { "cell_type": "code", "execution_count": 101, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINFYEaudit_committeeboard_sizeindependent_membersaudited_financialsTotalFundrsngExpCurrentYearCYTotalFundraisingExpenseAmtTotalFunctionalExpensesGrpTotalContributionsTotalContributionsAmtCYContributionsGrantsAmt
2581494135FY20140401NaN359301{u'TotalAmt': u'1382846', u'ProgramServicesAmt': u'830309', u'ManagementAndGeneralAmt': u'193236', u'FundraisingAmt': u'359301'}NaN12165361216536
3731284606FY20121641392027NaNNaN9299628NaNNaN
4262224994FY20131541NaN17564{u'TotalAmt': u'608945', u'ProgramServicesAmt': u'495257', u'ManagementAndGeneralAmt': u'96124', u'FundraisingAmt': u'17564'}NaN359003359003
\n", "
" ], "text/plain": [ " EIN FYE audit_committee board_size independent_members \\\n", "2 581494135 FY2014 0 4 0 \n", "3 731284606 FY2012 1 6 4 \n", "4 262224994 FY2013 1 5 4 \n", "\n", " audited_financials TotalFundrsngExpCurrentYear CYTotalFundraisingExpenseAmt \\\n", "2 1 NaN 359301 \n", "3 1 392027 NaN \n", "4 1 NaN 17564 \n", "\n", " TotalFunctionalExpensesGrp \\\n", "2 {u'TotalAmt': u'1382846', u'ProgramServicesAmt': u'830309', u'ManagementAndGeneralAmt': u'193236', u'FundraisingAmt': u'359301'} \n", "3 NaN \n", "4 {u'TotalAmt': u'608945', u'ProgramServicesAmt': u'495257', u'ManagementAndGeneralAmt': u'96124', u'FundraisingAmt': u'17564'} \n", "\n", " TotalContributions TotalContributionsAmt CYContributionsGrantsAmt \n", "2 NaN 1216536 1216536 \n", "3 9299628 NaN NaN \n", "4 NaN 359003 359003 " ] }, "execution_count": 101, "metadata": {}, "output_type": "execute_result" } ], "source": [ "columns = ['EIN', 'FYE', \n", " #'AuditCommittee', 'AuditCommitteeInd', \n", " 'audit_committee',\n", " 'board_size', 'independent_members',\n", " #'GoverningBodyVotingMembersCnt', 'VotingMembersGoverningBodyCnt', #THESE TWO SEEM TO BE THE SAME\n", " #'IndependentVotingMemberCnt', 'VotingMembersIndependentCnt', #THESE TWO SEEM TO BE THE SAME\n", " #'NbrVotingGoverningBodyMembers', 'NbrVotingMembersGoverningBody', #THESE TWO SEEM TO BE THE SAME \n", " #'NumberIndependentVotingMembers', 'NbrIndependentVotingMembers', #THESE TWO SEEM TO BE THE SAME \n", " 'audited_financials',\n", " #'IndependentAuditFinancialStmt', 'ConsolidatedAuditFinancialStmt', 'FSAudited',\n", " #'IndependentAuditFinclStmtInd', 'ConsolidatedAuditFinclStmtInd', 'FSAuditedInd', \n", " #'AuditedFinancialStmtAttInd', 'FSAuditedBasis', 'FSAuditedBasisGrp', \n", " \n", " #'LoanToOfficerOrDQP', \n", " #'FundraisingAmt', \n", " 'TotalFundrsngExpCurrentYear', 'CYTotalFundraisingExpenseAmt', 'TotalFunctionalExpensesGrp', \n", " 'TotalContributions', 'TotalContributionsAmt', 'CYContributionsGrantsAmt', \n", " ]\n", "df[columns][2:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Zero Fundraising" ] }, { "cell_type": "code", "execution_count": 108, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "344\n", "194\n", "194\n" ] } ], "source": [ "print df[columns[6]].value_counts().sum()\n", "print df[columns[7]].value_counts().sum()\n", "print len(df[df[columns[8]].notnull()])" ] }, { "cell_type": "code", "execution_count": 109, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "344\n", "538\n" ] }, { "data": { "text/plain": [ "0 238\n", "930602 2\n", "150000 2\n", "12107156 1\n", "1218686 1\n", "250988 1\n", "4426103 1\n", "1182027 1\n", "48180 1\n", "28125 1\n", "7735 1\n", "1127595 1\n", "890703 1\n", "1432842 1\n", "11357773 1\n", "234280 1\n", "3588 1\n", "4267087 1\n", "583844 1\n", "57597 1\n", "456843 1\n", "99370 1\n", "53125 1\n", "178386 1\n", "73831 1\n", "1203637 1\n", "277828 1\n", "659270 1\n", "4685328 1\n", "159980 1\n", " ... \n", "10710 1\n", "11730 1\n", "677021 1\n", "36836 1\n", "68477 1\n", "12342 1\n", "8585 1\n", "180000 1\n", "150 1\n", "359301 1\n", "74632 1\n", "289015 1\n", "2925264 1\n", "76621 1\n", "26529 1\n", "1170772 1\n", "132025 1\n", "5417144 1\n", "330771 1\n", "304506 1\n", "86011 1\n", "43224 1\n", "57971 1\n", "17757 1\n", "21434 1\n", "32860 1\n", "401830 1\n", "593922 1\n", "635121 1\n", "110725 1\n", "Name: fundraising_exp, dtype: int64" ] }, "execution_count": 109, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['fundraising_exp'] = np.nan\n", "print len(df[df['fundraising_exp'].notnull()])\n", "df['fundraising_exp'] = np.where(df['TotalFundrsngExpCurrentYear'].notnull(), df['TotalFundrsngExpCurrentYear'], \n", " df['fundraising_exp'] )\n", "print len(df[df['fundraising_exp'].notnull()])\n", "df['fundraising_exp'] = np.where( ((df['fundraising_exp'].isnull()) & \n", " (df['CYTotalFundraisingExpenseAmt'].notnull())), \n", " df['CYTotalFundraisingExpenseAmt'], df['fundraising_exp'] )\n", "print len(df[df['fundraising_exp'].notnull()])\n", "df['fundraising_exp'].value_counts()" ] }, { "cell_type": "code", "execution_count": 111, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0 300\n", "1 238\n", "Name: low_quality_reporting, dtype: int64" ] }, "execution_count": 111, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['low_quality_reporting'] = np.where(df['fundraising_exp']=='0', 1,0)\n", "df['low_quality_reporting'].value_counts()" ] }, { "cell_type": "code", "execution_count": 112, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINFYEaudit_committeeboard_sizeindependent_membersaudited_financialsfundraising_explow_quality_reportingTotalContributionsTotalContributionsAmtCYContributionsGrantsAmt
2581494135FY201404013593010NaN12165361216536
3731284606FY2012164139202709299628NaNNaN
4262224994FY20131541175640NaN359003359003
\n", "
" ], "text/plain": [ " EIN FYE audit_committee board_size independent_members \\\n", "2 581494135 FY2014 0 4 0 \n", "3 731284606 FY2012 1 6 4 \n", "4 262224994 FY2013 1 5 4 \n", "\n", " audited_financials fundraising_exp low_quality_reporting \\\n", "2 1 359301 0 \n", "3 1 392027 0 \n", "4 1 17564 0 \n", "\n", " TotalContributions TotalContributionsAmt CYContributionsGrantsAmt \n", "2 NaN 1216536 1216536 \n", "3 9299628 NaN NaN \n", "4 NaN 359003 359003 " ] }, "execution_count": 112, "metadata": {}, "output_type": "execute_result" } ], "source": [ "columns = ['EIN', 'FYE', \n", " #'AuditCommittee', 'AuditCommitteeInd', \n", " 'audit_committee',\n", " 'board_size', 'independent_members',\n", " #'GoverningBodyVotingMembersCnt', 'VotingMembersGoverningBodyCnt', #THESE TWO SEEM TO BE THE SAME\n", " #'IndependentVotingMemberCnt', 'VotingMembersIndependentCnt', #THESE TWO SEEM TO BE THE SAME\n", " #'NbrVotingGoverningBodyMembers', 'NbrVotingMembersGoverningBody', #THESE TWO SEEM TO BE THE SAME \n", " #'NumberIndependentVotingMembers', 'NbrIndependentVotingMembers', #THESE TWO SEEM TO BE THE SAME \n", " 'audited_financials',\n", " #'IndependentAuditFinancialStmt', 'ConsolidatedAuditFinancialStmt', 'FSAudited',\n", " #'IndependentAuditFinclStmtInd', 'ConsolidatedAuditFinclStmtInd', 'FSAuditedInd', \n", " #'AuditedFinancialStmtAttInd', 'FSAuditedBasis', 'FSAuditedBasisGrp', \n", " \n", " #'LoanToOfficerOrDQP', \n", " #'FundraisingAmt', \n", " #'TotalFundrsngExpCurrentYear', 'CYTotalFundraisingExpenseAmt', 'TotalFunctionalExpensesGrp', \n", " 'fundraising_exp', 'low_quality_reporting',\n", " 'TotalContributions', 'TotalContributionsAmt', 'CYContributionsGrantsAmt', \n", " ]\n", "df[columns][2:5]" ] }, { "cell_type": "code", "execution_count": 114, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINFYEaudit_committeeboard_sizeindependent_membersaudited_financialsfundraising_explow_quality_reportingTotalContributionsTotalContributionsAmtCYContributionsGrantsAmt
5141631995FY2012130159297603328598NaNNaN
6581925867FY20150991260700NaN480315480315
7431196717FY2015077101NaN425909425909
8131624000FY2012136361267236097671159NaNNaN
9390806314FY201213534191969804041706NaNNaN
10591561180FY20111595814685328015298518NaNNaN
11741152597FY2010NaN493200118821365NaNNaN
12042129889FY2013127241442668023922926NaNNaN
13261636099FY20131191816770210NaN1555716615557166
14300038297FY20101108125751005704220NaNNaN
15390806314FY2010133321100385906320334NaNNaN
16042129889FY2011125251890550025889793NaNNaN
17300038297FY20111119125819906828824NaNNaN
18521007153FY201112222101NaNNaNNaN
19941347046FY20111222212913100013093432NaNNaN
20251892177FY201112524145684303850700NaNNaN
21134147912FY201217717869102578781NaNNaN
22132738818FY20121303011170772024067624NaNNaN
23205183267FY2014111101475030NaN3604299236042992
24912053374FY2015NaN2300386160NaN197727197727
\n", "
" ], "text/plain": [ " EIN FYE audit_committee board_size independent_members \\\n", "5 141631995 FY2012 1 3 0 \n", "6 581925867 FY2015 0 9 9 \n", "7 431196717 FY2015 0 7 7 \n", "8 131624000 FY2012 1 36 36 \n", "9 390806314 FY2012 1 35 34 \n", "10 591561180 FY2011 1 59 58 \n", "11 741152597 FY2010 NaN 49 32 \n", "12 042129889 FY2013 1 27 24 \n", "13 261636099 FY2013 1 19 18 \n", "14 300038297 FY2010 1 10 8 \n", "15 390806314 FY2010 1 33 32 \n", "16 042129889 FY2011 1 25 25 \n", "17 300038297 FY2011 1 11 9 \n", "18 521007153 FY2011 1 22 22 \n", "19 941347046 FY2011 1 22 22 \n", "20 251892177 FY2011 1 25 24 \n", "21 134147912 FY2012 1 7 7 \n", "22 132738818 FY2012 1 30 30 \n", "23 205183267 FY2014 1 11 10 \n", "24 912053374 FY2015 NaN 23 0 \n", "\n", " audited_financials fundraising_exp low_quality_reporting \\\n", "5 1 592976 0 \n", "6 1 26070 0 \n", "7 1 0 1 \n", "8 1 267236 0 \n", "9 1 919698 0 \n", "10 1 4685328 0 \n", "11 0 0 1 \n", "12 1 442668 0 \n", "13 1 677021 0 \n", "14 1 257510 0 \n", "15 1 1003859 0 \n", "16 1 890550 0 \n", "17 1 258199 0 \n", "18 1 0 1 \n", "19 1 2913100 0 \n", "20 1 456843 0 \n", "21 1 78691 0 \n", "22 1 1170772 0 \n", "23 1 47503 0 \n", "24 0 38616 0 \n", "\n", " TotalContributions TotalContributionsAmt CYContributionsGrantsAmt \n", "5 3328598 NaN NaN \n", "6 NaN 480315 480315 \n", "7 NaN 425909 425909 \n", "8 97671159 NaN NaN \n", "9 4041706 NaN NaN \n", "10 15298518 NaN NaN \n", "11 18821365 NaN NaN \n", "12 23922926 NaN NaN \n", "13 NaN 15557166 15557166 \n", "14 5704220 NaN NaN \n", "15 6320334 NaN NaN \n", "16 25889793 NaN NaN \n", "17 6828824 NaN NaN \n", "18 NaN NaN NaN \n", "19 13093432 NaN NaN \n", "20 3850700 NaN NaN \n", "21 2578781 NaN NaN \n", "22 24067624 NaN NaN \n", "23 NaN 36042992 36042992 \n", "24 NaN 197727 197727 " ] }, "execution_count": 114, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[columns][5:25]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Save DF " ] }, { "cell_type": "code", "execution_count": 118, "metadata": { "collapsed": false }, "outputs": [], "source": [ "df[columns].to_pickle('additional robustness variables for advisory orgs from e-file data.pkl')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Save DF\n", "# Take last" ] }, { "cell_type": "code", "execution_count": 120, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINFYEaudit_committeeboard_sizeindependent_membersaudited_financialsfundraising_explow_quality_reportingTotalContributionsTotalContributionsAmtCYContributionsGrantsAmt
533570789483FY20151161613213340NaN65210666521066
534362169124FY201511716116508490NaN51956455195645
535630932057FY2014199101NaN50985395098539
536941347046FY2014NaN2219031461370NaN1171752511717525
537562439708FY2014177101NaN35484613548461
\n", "
" ], "text/plain": [ " EIN FYE audit_committee board_size independent_members \\\n", "533 570789483 FY2015 1 16 16 \n", "534 362169124 FY2015 1 17 16 \n", "535 630932057 FY2014 1 9 9 \n", "536 941347046 FY2014 NaN 22 19 \n", "537 562439708 FY2014 1 7 7 \n", "\n", " audited_financials fundraising_exp low_quality_reporting \\\n", "533 1 321334 0 \n", "534 1 1650849 0 \n", "535 1 0 1 \n", "536 0 3146137 0 \n", "537 1 0 1 \n", "\n", " TotalContributions TotalContributionsAmt CYContributionsGrantsAmt \n", "533 NaN 6521066 6521066 \n", "534 NaN 5195645 5195645 \n", "535 NaN 5098539 5098539 \n", "536 NaN 11717525 11717525 \n", "537 NaN 3548461 3548461 " ] }, "execution_count": 120, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[columns].tail()" ] }, { "cell_type": "code", "execution_count": 121, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countmeanstdmin25%50%75%max
low_quality_reporting538.00.4423790.4971310.00.00.01.01.0
\n", "
" ], "text/plain": [ " count mean std min 25% 50% 75% max\n", "low_quality_reporting 538.0 0.442379 0.497131 0.0 0.0 0.0 1.0 1.0" ] }, "execution_count": 121, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[columns].describe().T" ] }, { "cell_type": "code", "execution_count": 124, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['EIN', 'FYE', 'audit_committee', 'board_size', 'independent_members', 'audited_financials', 'fundraising_exp', 'low_quality_reporting', 'TotalContributions', 'TotalContributionsAmt', 'CYContributionsGrantsAmt']\n" ] } ], "source": [ "print columns" ] }, { "cell_type": "code", "execution_count": 126, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "323\n", "176\n", "194\n" ] } ], "source": [ "print len(df[df['TotalContributions'].notnull()])\n", "print len(df[df['TotalContributionsAmt'].notnull()])\n", "print len(df[df['CYContributionsGrantsAmt'].notnull()])" ] }, { "cell_type": "code", "execution_count": 127, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "370" ] }, "execution_count": 127, "metadata": {}, "output_type": "execute_result" } ], "source": [ "176+194" ] }, { "cell_type": "code", "execution_count": 128, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINFYEaudit_committeeboard_sizeindependent_membersaudited_financialsfundraising_explow_quality_reporting
0271317072FY2014NaN22001
1953246624FY2014199101
2581494135FY201404013593010
3731284606FY201216413920270
4262224994FY20131541175640
\n", "
" ], "text/plain": [ " EIN FYE audit_committee board_size independent_members \\\n", "0 271317072 FY2014 NaN 2 2 \n", "1 953246624 FY2014 1 9 9 \n", "2 581494135 FY2014 0 4 0 \n", "3 731284606 FY2012 1 6 4 \n", "4 262224994 FY2013 1 5 4 \n", "\n", " audited_financials fundraising_exp low_quality_reporting \n", "0 0 0 1 \n", "1 1 0 1 \n", "2 1 359301 0 \n", "3 1 392027 0 \n", "4 1 17564 0 " ] }, "execution_count": 128, "metadata": {}, "output_type": "execute_result" } ], "source": [ "columns = ['EIN', 'FYE', 'audit_committee', 'board_size', 'independent_members', 'audited_financials', \n", " 'fundraising_exp', 'low_quality_reporting']\n", "# 'TotalContributions', 'TotalContributionsAmt', 'CYContributionsGrantsAmt']\n", "df[columns][:5]" ] }, { "cell_type": "code", "execution_count": 129, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "EIN object\n", "FYE object\n", "audit_committee object\n", "board_size object\n", "independent_members object\n", "audited_financials object\n", "fundraising_exp object\n", "low_quality_reporting int64\n", "dtype: object" ] }, "execution_count": 129, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[columns].dtypes" ] }, { "cell_type": "code", "execution_count": 136, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "0\n", "129\n", "0\n", "0\n", "0\n", "0\n", "0\n" ] }, { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINFYEaudit_committeeboard_sizeindependent_membersaudited_financialsfundraising_explow_quality_reporting
0271317072FY2014NaN22001
11741152597FY2010NaN4932001
\n", "
" ], "text/plain": [ " EIN FYE audit_committee board_size independent_members \\\n", "0 271317072 FY2014 NaN 2 2 \n", "11 741152597 FY2010 NaN 49 32 \n", "\n", " audited_financials fundraising_exp low_quality_reporting \n", "0 0 0 1 \n", "11 0 0 1 " ] }, "execution_count": 136, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print len(df[df[columns[0]].isnull()])\n", "print len(df[df[columns[1]].isnull()])\n", "print len(df[df[columns[2]].isnull()])\n", "print len(df[df[columns[3]].isnull()])\n", "print len(df[df[columns[4]].isnull()])\n", "print len(df[df[columns[5]].isnull()])\n", "print len(df[df[columns[6]].isnull()])\n", "print len(df[df[columns[7]].isnull()])" ] }, { "cell_type": "code", "execution_count": 141, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINFYEaudit_committeeboard_sizeindependent_membersaudited_financialsfundraising_explow_quality_reporting
252030498214FY2012NaN6601401830
433030498214FY201106602375680
78030498214FY20100220733340
333042129889FY201412319101
12042129889FY20131272414426680
\n", "
" ], "text/plain": [ " EIN FYE audit_committee board_size independent_members \\\n", "252 030498214 FY2012 NaN 6 6 \n", "433 030498214 FY2011 0 6 6 \n", "78 030498214 FY2010 0 2 2 \n", "333 042129889 FY2014 1 23 19 \n", "12 042129889 FY2013 1 27 24 \n", "\n", " audited_financials fundraising_exp low_quality_reporting \n", "252 0 140183 0 \n", "433 0 237568 0 \n", "78 0 73334 0 \n", "333 1 0 1 \n", "12 1 442668 0 " ] }, "execution_count": 141, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = df[columns]\n", "df = df.sort_values(by=['EIN', 'FYE'], ascending=[1,0])\n", "df[:5]" ] }, { "cell_type": "code", "execution_count": 147, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "152\n", "152\n" ] }, { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINFYEaudit_committeeboard_sizeindependent_membersaudited_financialsfundraising_explow_quality_reporting
0030498214FY20120.06601401830
1042129889FY20141.02319101
2042701694FY20141.019191261010
3050516084FY20141.02101877250
4061741467FY20140.010100121080
\n", "
" ], "text/plain": [ " EIN FYE audit_committee board_size independent_members \\\n", "0 030498214 FY2012 0.0 6 6 \n", "1 042129889 FY2014 1.0 23 19 \n", "2 042701694 FY2014 1.0 19 19 \n", "3 050516084 FY2014 1.0 2 1 \n", "4 061741467 FY2014 0.0 10 10 \n", "\n", " audited_financials fundraising_exp low_quality_reporting \n", "0 0 140183 0 \n", "1 1 0 1 \n", "2 1 26101 0 \n", "3 0 187725 0 \n", "4 0 12108 0 " ] }, "execution_count": 147, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print len(df.groupby('EIN').agg('first'))\n", "df_advisories = df.groupby('EIN').agg('first')\n", "print len(df_advisories)\n", "df_advisories = df_advisories.reset_index()\n", "df_advisories[:5]" ] }, { "cell_type": "code", "execution_count": 151, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "u'/Users/gregorysaxton/SkyDrive/Mining_Firms'" ] }, "execution_count": 151, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pwd" ] }, { "cell_type": "code", "execution_count": 152, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/Users/gregorysaxton/Google Drive/SOX\n" ] } ], "source": [ "cd '/Users/gregorysaxton/Google Drive/SOX'" ] }, { "cell_type": "code", "execution_count": 153, "metadata": { "collapsed": true }, "outputs": [], "source": [ "df_advisories.to_pickle('Additional Variables for Robustness Tests - 152 advisory orgs.pkl')" ] }, { "cell_type": "code", "execution_count": 157, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['EIN', 'FYE', 'audit_committee', 'board_size', 'independent_members', 'audited_financials', 'fundraising_exp', 'low_quality_reporting']\n" ] } ], "source": [ "print df_advisories.columns.tolist()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": 116, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.0386560693642\n" ] } ], "source": [ "from __future__ import division\n", "print 321/8304" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": 52, "metadata": { "collapsed": false }, "outputs": [], "source": [ "#columns = ['fund_expense_percent_2016', 'program_expense_growth_2016', 'independent_board_2016', \n", "# 'audited_financials_2016', 'no_loans_related_2016', 'contributions_gifts_grants_2016', \n", "# 'total_contributions_2016', 'fund_expense_percent_2011', 'fund_expense_2011', \n", "# 'primary_revenue_growth_2011', 'program_expense_growth_2011', 'independent_board_2011', \n", "# 'audited_financials_2011', 'no_loans_related_2011', 'pt1_num_vtng_gvrn_bdy_mems', 'pt1_num_ind_vtng_mems', \n", "# 'num_vtng_gvrn_bdy_mems', 'num_ind_vtng_mems', 'contri_grnts_cy', 'tot_fndrsng_exp_cy', 'aud_fincl_stmts', \n", "# 'fs_audited', 'audit_committee']\n", "#print df[columns[17]].value_counts(), '\\n'" ] }, { "cell_type": "code", "execution_count": 636, "metadata": { "collapsed": false }, "outputs": [], "source": [ "#df2 = pd.DataFrame(list(filings.find()))\n", "#print '# of columns:', len(df2.columns)\n", "#print '# of observations:', len(df2)\n", "#df2.head(1)" ] }, { "cell_type": "code", "execution_count": 637, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "\"\\ndf2[['CYContributionsGrantsAmt', 'FederatedCampaigns', 'FederatedCampaignsAmt', \\n 'MembershipDues', 'MembershipDuesAmt',\\n 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp',\\n 'FundraisingActivities', 'FundraisingActivitiesInd', 'FundraisingAmt', \\n 'FundraisingDirectExpenses', 'FundraisingDirectExpensesAmt', 'FundraisingEvents', \\n 'FundraisingGrossIncomeAmt',\\n ]][:5]\\n\\n \\n 'RelatedOrganizations', 'RelatedOrganizationsAmt',\\n 'GovernmentGrants', 'GovernmentGrantsAmt',\\n 'PYProgramServiceRevenueAmt', 'CYProgramServiceRevenueAmt',\\n 'ProgramServiceRevenue', 'ProgramServiceRevenueCY', 'ProgramServiceRevenueGrp',\\n 'TotalProgramServiceRevenue', 'TotalProgramServiceRevenueAmt', \\n 'CYInvestmentIncomeAmt', 'InvestmentIncome', 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', \\n 'OtherRevenueTotalAmt', 'OtherRevenueMiscGrp',\\n 'TotalOtherRevenue', 'CYOtherRevenueAmt'\\n ]\\n\"" ] }, "execution_count": 637, "metadata": {}, "output_type": "execute_result" } ], "source": [ "'''\n", "df2[['CYContributionsGrantsAmt', 'FederatedCampaigns', 'FederatedCampaignsAmt', \n", " 'MembershipDues', 'MembershipDuesAmt',\n", " 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp',\n", " 'FundraisingActivities', 'FundraisingActivitiesInd', 'FundraisingAmt', \n", " 'FundraisingDirectExpenses', 'FundraisingDirectExpensesAmt', 'FundraisingEvents', \n", " 'FundraisingGrossIncomeAmt',\n", " ]][:5]\n", "\n", " \n", " 'RelatedOrganizations', 'RelatedOrganizationsAmt',\n", " 'GovernmentGrants', 'GovernmentGrantsAmt',\n", " 'PYProgramServiceRevenueAmt', 'CYProgramServiceRevenueAmt',\n", " 'ProgramServiceRevenue', 'ProgramServiceRevenueCY', 'ProgramServiceRevenueGrp',\n", " 'TotalProgramServiceRevenue', 'TotalProgramServiceRevenueAmt', \n", " 'CYInvestmentIncomeAmt', 'InvestmentIncome', 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', \n", " 'OtherRevenueTotalAmt', 'OtherRevenueMiscGrp',\n", " 'TotalOtherRevenue', 'CYOtherRevenueAmt'\n", " ]\n", "'''" ] }, { "cell_type": "code", "execution_count": 638, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "\"\\ndf2[[ 'TaxPeriod',\\n #'CYContributionsGrantsAmt', 'FederatedCampaigns', 'FederatedCampaignsAmt', \\n #'MembershipDues', 'MembershipDuesAmt',\\n 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp',\\n #'FundraisingActivities', \\n #'FundraisingActivitiesInd', \\n 'FundraisingAmt', \\n 'FundraisingDirectExpenses', 'FundraisingDirectExpensesAmt', 'FundraisingEvents', \\n 'FundraisingGrossIncomeAmt',\\n ]][5:20]\\n\"" ] }, "execution_count": 638, "metadata": {}, "output_type": "execute_result" } ], "source": [ "'''\n", "df2[[ 'TaxPeriod',\n", " #'CYContributionsGrantsAmt', 'FederatedCampaigns', 'FederatedCampaignsAmt', \n", " #'MembershipDues', 'MembershipDuesAmt',\n", " 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp',\n", " #'FundraisingActivities', \n", " #'FundraisingActivitiesInd', \n", " 'FundraisingAmt', \n", " 'FundraisingDirectExpenses', 'FundraisingDirectExpensesAmt', 'FundraisingEvents', \n", " 'FundraisingGrossIncomeAmt',\n", " ]][5:20]\n", "'''" ] }, { "cell_type": "code", "execution_count": 639, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
OrganizationNameObjectIdURLSubmittedOnDLNLastUpdatedTaxPeriodIsElectronicFormType_idIsAvailableEINWhistleblowerPolicyWhistleblowerPolicyIndConflictOfInterestPolicyIndConflictOfInterestPolicyDocumentRetentionPolicyDocumentRetentionPolicyIndCYTotalRevenueAmtTotalRevenueTotalRevenueCurrentYearCYTotalExpensesAmtTotalExpensesCurrentYearTotalFunctionalExpensesTotalFunctionalExpensesGrpTotalProgramServiceExpenseTotalProgramServiceExpensesAmtCYContributionsGrantsAmtFederatedCampaignsFederatedCampaignsAmtMembershipDuesMembershipDuesAmtNetIncomeFromFundraisingEventsNetIncmFromFundraisingEvtGrpFundraisingAmtFundraisingGrossIncomeAmtRelatedOrganizationsRelatedOrganizationsAmtGovernmentGrantsGovernmentGrantsAmtPYProgramServiceRevenueAmtCYProgramServiceRevenueAmtProgramServiceRevenueProgramServiceRevenueCYProgramServiceRevenueGrpTotalProgramServiceRevenueTotalProgramServiceRevenueAmtCYInvestmentIncomeAmtInvestmentIncomeInvestmentIncomeCurrentYearInvestmentIncomeGrpOtherRevenueTotalAmtOtherRevenueMiscGrpTotalOtherRevenueCYOtherRevenueAmt
0ANCHOR BUILDING FUND INC201503169349304780https://s3.amazonaws.com/irs-form-990/201503169349304780_public.xml2016-01-30934933160478052016-03-21T17:23:53201412True99057cce7823ffc5a55fe7f54b5True271317072NaNfalsefalseNaNNaNfalse325000NaNNaN34400NaNNaN{u'TotalAmt': u'34400', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'34400', u'FundraisingAmt': u'0'}NaNNaN0NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaNNaNNaNNaNNaN0NaNNaNNaN325000{u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000', u'Desc': u'SETTLEMENT'}NaN325000
1INLAND BEHAVIORAL AND HEALTH SERVICES INC201501339349302635https://s3.amazonaws.com/irs-form-990/201501339349302635_public.xml2015-07-15934931330263552016-03-21T17:23:53201406True99057cce87d3ffc5a55fe7f568cTrue953246624NaNtruetrueNaNNaNtrue8825153NaNNaN9016469NaNNaN{u'TotalAmt': u'9016469', u'ProgramServicesAmt': u'7477089', u'ManagementAndGeneralAmt': u'1539380', u'FundraisingAmt': u'0'}NaN747708932937NaNNaNNaNNaNNaN{u'TotalRevenueColumnAmt': u'0'}NaNNaNNaNNaNNaNNaN94579808758017NaNNaN[{u'TotalRevenueColumnAmt': u'265216', u'BusinessCd': u'624100', u'RelatedOrExemptFuncIncomeAmt': u'265216', u'Desc': u'Drug/Alcohol Treatment'}, {u'TotalRevenueColumnAmt': u'3987366', u'BusinessCd': u'624100', u'RelatedOrExemptFuncIncomeAmt': u'...NaN8758017107NaNNaN{u'TotalRevenueColumnAmt': u'107', u'ExclusionAmt': u'107'}34092[{u'TotalRevenueColumnAmt': u'7299', u'ExclusionAmt': u'7299', u'Desc': u'Gain on A/C ins. claim'}, {u'TotalRevenueColumnAmt': u'26793', u'ExclusionAmt': u'26793', u'Desc': u'Other income'}]NaN34092
\n", "
" ], "text/plain": [ " OrganizationName ObjectId \\\n", "0 ANCHOR BUILDING FUND INC 201503169349304780 \n", "1 INLAND BEHAVIORAL AND HEALTH SERVICES INC 201501339349302635 \n", "\n", " URL \\\n", "0 https://s3.amazonaws.com/irs-form-990/201503169349304780_public.xml \n", "1 https://s3.amazonaws.com/irs-form-990/201501339349302635_public.xml \n", "\n", " SubmittedOn DLN LastUpdated TaxPeriod IsElectronic \\\n", "0 2016-01-30 93493316047805 2016-03-21T17:23:53 201412 True \n", "1 2015-07-15 93493133026355 2016-03-21T17:23:53 201406 True \n", "\n", " FormType _id IsAvailable EIN \\\n", "0 990 57cce7823ffc5a55fe7f54b5 True 271317072 \n", "1 990 57cce87d3ffc5a55fe7f568c True 953246624 \n", "\n", " WhistleblowerPolicy WhistleblowerPolicyInd ConflictOfInterestPolicyInd \\\n", "0 NaN false false \n", "1 NaN true true \n", "\n", " ConflictOfInterestPolicy DocumentRetentionPolicy DocumentRetentionPolicyInd \\\n", "0 NaN NaN false \n", "1 NaN NaN true \n", "\n", " CYTotalRevenueAmt TotalRevenue TotalRevenueCurrentYear CYTotalExpensesAmt \\\n", "0 325000 NaN NaN 34400 \n", "1 8825153 NaN NaN 9016469 \n", "\n", " TotalExpensesCurrentYear TotalFunctionalExpenses \\\n", "0 NaN NaN \n", "1 NaN NaN \n", "\n", " TotalFunctionalExpensesGrp \\\n", "0 {u'TotalAmt': u'34400', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'34400', u'FundraisingAmt': u'0'} \n", "1 {u'TotalAmt': u'9016469', u'ProgramServicesAmt': u'7477089', u'ManagementAndGeneralAmt': u'1539380', u'FundraisingAmt': u'0'} \n", "\n", " TotalProgramServiceExpense TotalProgramServiceExpensesAmt \\\n", "0 NaN NaN \n", "1 NaN 7477089 \n", "\n", " CYContributionsGrantsAmt FederatedCampaigns FederatedCampaignsAmt \\\n", "0 0 NaN NaN \n", "1 32937 NaN NaN \n", "\n", " MembershipDues MembershipDuesAmt NetIncomeFromFundraisingEvents \\\n", "0 NaN NaN NaN \n", "1 NaN NaN NaN \n", "\n", " NetIncmFromFundraisingEvtGrp FundraisingAmt FundraisingGrossIncomeAmt \\\n", "0 NaN NaN NaN \n", "1 {u'TotalRevenueColumnAmt': u'0'} NaN NaN \n", "\n", " RelatedOrganizations RelatedOrganizationsAmt GovernmentGrants \\\n", "0 NaN NaN NaN \n", "1 NaN NaN NaN \n", "\n", " GovernmentGrantsAmt PYProgramServiceRevenueAmt CYProgramServiceRevenueAmt \\\n", "0 NaN NaN 0 \n", "1 NaN 9457980 8758017 \n", "\n", " ProgramServiceRevenue ProgramServiceRevenueCY \\\n", "0 NaN NaN \n", "1 NaN NaN \n", "\n", " ProgramServiceRevenueGrp \\\n", "0 NaN \n", "1 [{u'TotalRevenueColumnAmt': u'265216', u'BusinessCd': u'624100', u'RelatedOrExemptFuncIncomeAmt': u'265216', u'Desc': u'Drug/Alcohol Treatment'}, {u'TotalRevenueColumnAmt': u'3987366', u'BusinessCd': u'624100', u'RelatedOrExemptFuncIncomeAmt': u'... \n", "\n", " TotalProgramServiceRevenue TotalProgramServiceRevenueAmt \\\n", "0 NaN NaN \n", "1 NaN 8758017 \n", "\n", " CYInvestmentIncomeAmt InvestmentIncome InvestmentIncomeCurrentYear \\\n", "0 0 NaN NaN \n", "1 107 NaN NaN \n", "\n", " InvestmentIncomeGrp \\\n", "0 NaN \n", "1 {u'TotalRevenueColumnAmt': u'107', u'ExclusionAmt': u'107'} \n", "\n", " OtherRevenueTotalAmt \\\n", "0 325000 \n", "1 34092 \n", "\n", " OtherRevenueMiscGrp \\\n", "0 {u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000', u'Desc': u'SETTLEMENT'} \n", "1 [{u'TotalRevenueColumnAmt': u'7299', u'ExclusionAmt': u'7299', u'Desc': u'Gain on A/C ins. claim'}, {u'TotalRevenueColumnAmt': u'26793', u'ExclusionAmt': u'26793', u'Desc': u'Other income'}] \n", "\n", " TotalOtherRevenue CYOtherRevenueAmt \n", "0 NaN 325000 \n", "1 NaN 34092 " ] }, "execution_count": 639, "metadata": {}, "output_type": "execute_result" } ], "source": [ "file_list_columns = ['OrganizationName', 'ObjectId', 'URL', 'SubmittedOn', 'DLN', 'LastUpdated', 'TaxPeriod',\n", " 'IsElectronic', 'FormType', '_id', 'IsAvailable', 'EIN']\n", "SOX_columns = ['WhistleblowerPolicy', 'WhistleblowerPolicyInd',\n", " 'ConflictOfInterestPolicyInd', 'ConflictOfInterestPolicy',\n", " 'DocumentRetentionPolicy', 'DocumentRetentionPolicyInd',]\n", "revenue_columns = ['CYTotalRevenueAmt', 'TotalRevenue', 'TotalRevenueCurrentYear']\n", "efficiency_columns = ['CYTotalExpensesAmt', 'TotalExpensesCurrentYear',\n", " 'TotalFunctionalExpenses', 'TotalFunctionalExpensesGrp',\n", " 'TotalProgramServiceExpense', 'TotalProgramServiceExpensesAmt',]\n", "#df['program_efficiency'] = df['program_expenses']/df['total_expenses']\n", "complexity_columns = ['CYContributionsGrantsAmt', 'FederatedCampaigns', 'FederatedCampaignsAmt', \n", " 'MembershipDues', 'MembershipDuesAmt',\n", " 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp',\n", " #'FundraisingActivities', 'FundraisingActivitiesInd', \n", " 'FundraisingAmt', \n", " #'FundraisingDirectExpenses', 'FundraisingDirectExpensesAmt', 'FundraisingEvents', \n", " 'FundraisingGrossIncomeAmt', \n", " 'RelatedOrganizations', 'RelatedOrganizationsAmt',\n", " 'GovernmentGrants', 'GovernmentGrantsAmt',\n", " 'PYProgramServiceRevenueAmt', 'CYProgramServiceRevenueAmt',\n", " 'ProgramServiceRevenue', 'ProgramServiceRevenueCY', 'ProgramServiceRevenueGrp',\n", " 'TotalProgramServiceRevenue', 'TotalProgramServiceRevenueAmt', \n", " 'CYInvestmentIncomeAmt', 'InvestmentIncome', 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', \n", " 'OtherRevenueTotalAmt', 'OtherRevenueMiscGrp',\n", " 'TotalOtherRevenue', 'CYOtherRevenueAmt'\n", " ]\n", " \n", "#[contributions_gifts_grants_2016', 'federated_campaigns_2016', 'membership_dues_2016', \n", "# 'fundraising_events_2016', 'related_organizations_2016', 'government_grants_2016', \n", "# 'program_service_revenue_2016', 'other_revenue_2016']\n", "\n", "cols = file_list_columns+SOX_columns+revenue_columns+efficiency_columns+complexity_columns\n", "#total_revenue_logged + program_efficiency + age + complexity_2011\n", "df[cols][:2]" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
OrganizationNameObjectIdURLSubmittedOnDLNLastUpdatedTaxPeriodIsElectronicFormType_idIsAvailableEINMaterialDiversionOrMisuseMaterialDiversionOrMisuseInd
0ANCHOR BUILDING FUND INC201503169349304780https://s3.amazonaws.com/irs-form-990/201503169349304780_public.xml2016-01-30934933160478052016-03-21T17:23:53201412True99057cce7823ffc5a55fe7f54b5True271317072NaNtrue
\n", "
" ], "text/plain": [ " OrganizationName ObjectId \\\n", "0 ANCHOR BUILDING FUND INC 201503169349304780 \n", "\n", " URL \\\n", "0 https://s3.amazonaws.com/irs-form-990/201503169349304780_public.xml \n", "\n", " SubmittedOn DLN LastUpdated TaxPeriod IsElectronic \\\n", "0 2016-01-30 93493316047805 2016-03-21T17:23:53 201412 True \n", "\n", " FormType _id IsAvailable EIN \\\n", "0 990 57cce7823ffc5a55fe7f54b5 True 271317072 \n", "\n", " MaterialDiversionOrMisuse MaterialDiversionOrMisuseInd \n", "0 NaN true " ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#file_list_columns = ['OrganizationName', 'ObjectId', 'URL', 'SubmittedOn', 'DLN', 'LastUpdated', 'TaxPeriod',\n", "# 'IsElectronic', 'FormType', '_id', 'IsAvailable', 'EIN']\n", "#cols = file_list_columns + ['MaterialDiversionOrMisuse', 'MaterialDiversionOrMisuseInd']\n", "#df[cols][:1]" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 93\n", "false 89\n", "true 8\n", "1 4\n", "Name: MaterialDiversionOrMisuseInd, dtype: int64 \n", "\n", "0 185\n", "false 157\n", "1 1\n", "true 1\n", "Name: MaterialDiversionOrMisuse, dtype: int64 \n", "\n" ] } ], "source": [ "#print df[cols[-1]].value_counts(), '\\n'\n", "#print df[cols[-2]].value_counts(), '\\n'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
152 unique EINs in dataframe." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "538\n", "152\n" ] } ], "source": [ "print len(df['EIN'].tolist())\n", "print len(set(df['EIN'].tolist()))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
All except one are 990s." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "990 537\n", "990EZ 1\n", "Name: FormType, dtype: int64" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['FormType'].value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Create FYE variable" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "538\n", "538 \n", "\n", "True 538\n", "Name: IsAvailable, dtype: int64 \n", "\n", "True 538\n", "Name: IsElectronic, dtype: int64 \n", "\n" ] } ], "source": [ "print len(df)\n", "print df['TaxPeriod'].value_counts().sum(), '\\n'\n", "print df['IsAvailable'].value_counts(), '\\n'\n", "print df['IsElectronic'].value_counts(), '\\n'" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0 FY2014\n", "1 FY2014\n", "Name: FYE, dtype: object" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['FYE'] = 'FY' + df['TaxPeriod'].str[:4]\n", "df['FYE'][:2]" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "FY2013 120\n", "FY2014 117\n", "FY2012 112\n", "FY2011 97\n", "FY2010 77\n", "FY2015 15\n", "Name: FYE, dtype: int64" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['FYE'].value_counts()" ] }, { "cell_type": "code", "execution_count": 644, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
FYEEINOrganizationNameURLSubmittedOnTaxPeriodWhistleblowerPolicyWhistleblowerPolicyIndConflictOfInterestPolicyIndConflictOfInterestPolicyDocumentRetentionPolicyDocumentRetentionPolicyIndCYTotalRevenueAmtTotalRevenueTotalRevenueCurrentYearCYTotalExpensesAmtTotalExpensesCurrentYearTotalFunctionalExpensesTotalFunctionalExpensesGrpTotalProgramServiceExpenseTotalProgramServiceExpensesAmtCYContributionsGrantsAmtFederatedCampaignsFederatedCampaignsAmtMembershipDuesMembershipDuesAmtNetIncomeFromFundraisingEventsNetIncmFromFundraisingEvtGrpFundraisingAmtFundraisingGrossIncomeAmtRelatedOrganizationsRelatedOrganizationsAmtGovernmentGrantsGovernmentGrantsAmtPYProgramServiceRevenueAmtCYProgramServiceRevenueAmtProgramServiceRevenueProgramServiceRevenueCYProgramServiceRevenueGrpTotalProgramServiceRevenueTotalProgramServiceRevenueAmtCYInvestmentIncomeAmtInvestmentIncomeInvestmentIncomeCurrentYearInvestmentIncomeGrpOtherRevenueTotalAmtOtherRevenueMiscGrpTotalOtherRevenueCYOtherRevenueAmt
0FY2014271317072ANCHOR BUILDING FUND INChttps://s3.amazonaws.com/irs-form-990/201503169349304780_public.xml2016-01-30201412NaNfalsefalseNaNNaNfalse325000NaNNaN34400NaNNaN{u'TotalAmt': u'34400', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'34400', u'FundraisingAmt': u'0'}NaNNaN0NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaNNaNNaNNaNNaN0NaNNaNNaN325000{u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000', u'Desc': u'SETTLEMENT'}NaN325000
1FY2014953246624INLAND BEHAVIORAL AND HEALTH SERVICES INChttps://s3.amazonaws.com/irs-form-990/201501339349302635_public.xml2015-07-15201406NaNtruetrueNaNNaNtrue8825153NaNNaN9016469NaNNaN{u'TotalAmt': u'9016469', u'ProgramServicesAmt': u'7477089', u'ManagementAndGeneralAmt': u'1539380', u'FundraisingAmt': u'0'}NaN747708932937NaNNaNNaNNaNNaN{u'TotalRevenueColumnAmt': u'0'}NaNNaNNaNNaNNaNNaN94579808758017NaNNaN[{u'TotalRevenueColumnAmt': u'265216', u'BusinessCd': u'624100', u'RelatedOrExemptFuncIncomeAmt': u'265216', u'Desc': u'Drug/Alcohol Treatment'}, {u'TotalRevenueColumnAmt': u'3987366', u'BusinessCd': u'624100', u'RelatedOrExemptFuncIncomeAmt': u'...NaN8758017107NaNNaN{u'TotalRevenueColumnAmt': u'107', u'ExclusionAmt': u'107'}34092[{u'TotalRevenueColumnAmt': u'7299', u'ExclusionAmt': u'7299', u'Desc': u'Gain on A/C ins. claim'}, {u'TotalRevenueColumnAmt': u'26793', u'ExclusionAmt': u'26793', u'Desc': u'Other income'}]NaN34092
\n", "
" ], "text/plain": [ " FYE EIN OrganizationName \\\n", "0 FY2014 271317072 ANCHOR BUILDING FUND INC \n", "1 FY2014 953246624 INLAND BEHAVIORAL AND HEALTH SERVICES INC \n", "\n", " URL \\\n", "0 https://s3.amazonaws.com/irs-form-990/201503169349304780_public.xml \n", "1 https://s3.amazonaws.com/irs-form-990/201501339349302635_public.xml \n", "\n", " SubmittedOn TaxPeriod WhistleblowerPolicy WhistleblowerPolicyInd \\\n", "0 2016-01-30 201412 NaN false \n", "1 2015-07-15 201406 NaN true \n", "\n", " ConflictOfInterestPolicyInd ConflictOfInterestPolicy \\\n", "0 false NaN \n", "1 true NaN \n", "\n", " DocumentRetentionPolicy DocumentRetentionPolicyInd CYTotalRevenueAmt \\\n", "0 NaN false 325000 \n", "1 NaN true 8825153 \n", "\n", " TotalRevenue TotalRevenueCurrentYear CYTotalExpensesAmt \\\n", "0 NaN NaN 34400 \n", "1 NaN NaN 9016469 \n", "\n", " TotalExpensesCurrentYear TotalFunctionalExpenses \\\n", "0 NaN NaN \n", "1 NaN NaN \n", "\n", " TotalFunctionalExpensesGrp \\\n", "0 {u'TotalAmt': u'34400', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'34400', u'FundraisingAmt': u'0'} \n", "1 {u'TotalAmt': u'9016469', u'ProgramServicesAmt': u'7477089', u'ManagementAndGeneralAmt': u'1539380', u'FundraisingAmt': u'0'} \n", "\n", " TotalProgramServiceExpense TotalProgramServiceExpensesAmt \\\n", "0 NaN NaN \n", "1 NaN 7477089 \n", "\n", " CYContributionsGrantsAmt FederatedCampaigns FederatedCampaignsAmt \\\n", "0 0 NaN NaN \n", "1 32937 NaN NaN \n", "\n", " MembershipDues MembershipDuesAmt NetIncomeFromFundraisingEvents \\\n", "0 NaN NaN NaN \n", "1 NaN NaN NaN \n", "\n", " NetIncmFromFundraisingEvtGrp FundraisingAmt FundraisingGrossIncomeAmt \\\n", "0 NaN NaN NaN \n", "1 {u'TotalRevenueColumnAmt': u'0'} NaN NaN \n", "\n", " RelatedOrganizations RelatedOrganizationsAmt GovernmentGrants \\\n", "0 NaN NaN NaN \n", "1 NaN NaN NaN \n", "\n", " GovernmentGrantsAmt PYProgramServiceRevenueAmt CYProgramServiceRevenueAmt \\\n", "0 NaN NaN 0 \n", "1 NaN 9457980 8758017 \n", "\n", " ProgramServiceRevenue ProgramServiceRevenueCY \\\n", "0 NaN NaN \n", "1 NaN NaN \n", "\n", " ProgramServiceRevenueGrp \\\n", "0 NaN \n", "1 [{u'TotalRevenueColumnAmt': u'265216', u'BusinessCd': u'624100', u'RelatedOrExemptFuncIncomeAmt': u'265216', u'Desc': u'Drug/Alcohol Treatment'}, {u'TotalRevenueColumnAmt': u'3987366', u'BusinessCd': u'624100', u'RelatedOrExemptFuncIncomeAmt': u'... \n", "\n", " TotalProgramServiceRevenue TotalProgramServiceRevenueAmt \\\n", "0 NaN NaN \n", "1 NaN 8758017 \n", "\n", " CYInvestmentIncomeAmt InvestmentIncome InvestmentIncomeCurrentYear \\\n", "0 0 NaN NaN \n", "1 107 NaN NaN \n", "\n", " InvestmentIncomeGrp \\\n", "0 NaN \n", "1 {u'TotalRevenueColumnAmt': u'107', u'ExclusionAmt': u'107'} \n", "\n", " OtherRevenueTotalAmt \\\n", "0 325000 \n", "1 34092 \n", "\n", " OtherRevenueMiscGrp \\\n", "0 {u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000', u'Desc': u'SETTLEMENT'} \n", "1 [{u'TotalRevenueColumnAmt': u'7299', u'ExclusionAmt': u'7299', u'Desc': u'Gain on A/C ins. claim'}, {u'TotalRevenueColumnAmt': u'26793', u'ExclusionAmt': u'26793', u'Desc': u'Other income'}] \n", "\n", " TotalOtherRevenue CYOtherRevenueAmt \n", "0 NaN 325000 \n", "1 NaN 34092 " ] }, "execution_count": 644, "metadata": {}, "output_type": "execute_result" } ], "source": [ "file_list_columns = ['EIN', 'OrganizationName', 'URL', 'SubmittedOn', 'TaxPeriod',\n", " #'ObjectId', '_id', 'IsAvailable', 'DLN', 'LastUpdated', 'IsElectronic', 'FormType', \n", " ]\n", "SOX_columns = ['WhistleblowerPolicy', 'WhistleblowerPolicyInd',\n", " 'ConflictOfInterestPolicyInd', 'ConflictOfInterestPolicy',\n", " 'DocumentRetentionPolicy', 'DocumentRetentionPolicyInd',]\n", "revenue_columns = ['CYTotalRevenueAmt', 'TotalRevenue', 'TotalRevenueCurrentYear']\n", "efficiency_columns = ['CYTotalExpensesAmt', 'TotalExpensesCurrentYear',\n", " 'TotalFunctionalExpenses', 'TotalFunctionalExpensesGrp',\n", " 'TotalProgramServiceExpense', 'TotalProgramServiceExpensesAmt',]\n", "#df['program_efficiency'] = df['program_expenses']/df['total_expenses']\n", "complexity_columns = ['CYContributionsGrantsAmt', 'FederatedCampaigns', 'FederatedCampaignsAmt', \n", " 'MembershipDues', 'MembershipDuesAmt',\n", " 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp',\n", " #'FundraisingActivities', 'FundraisingActivitiesInd', \n", " 'FundraisingAmt', \n", " #'FundraisingDirectExpenses', 'FundraisingDirectExpensesAmt', 'FundraisingEvents', \n", " 'FundraisingGrossIncomeAmt', \n", " 'RelatedOrganizations', 'RelatedOrganizationsAmt',\n", " 'GovernmentGrants', 'GovernmentGrantsAmt',\n", " 'PYProgramServiceRevenueAmt', 'CYProgramServiceRevenueAmt',\n", " 'ProgramServiceRevenue', 'ProgramServiceRevenueCY', 'ProgramServiceRevenueGrp',\n", " 'TotalProgramServiceRevenue', 'TotalProgramServiceRevenueAmt', \n", " 'CYInvestmentIncomeAmt', 'InvestmentIncome', 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', \n", " 'OtherRevenueTotalAmt', 'OtherRevenueMiscGrp',\n", " 'TotalOtherRevenue', 'CYOtherRevenueAmt'\n", " ]\n", " \n", "#[contributions_gifts_grants_2016', 'federated_campaigns_2016', 'membership_dues_2016', \n", "# 'fundraising_events_2016', 'related_organizations_2016', 'government_grants_2016', \n", "# 'program_service_revenue_2016', 'other_revenue_2016']\n", "\n", "cols = file_list_columns+SOX_columns+revenue_columns+efficiency_columns+complexity_columns\n", "#total_revenue_logged + program_efficiency + age + complexity_2011\n", "df[cols][:2]\n", "cols = ['FYE']+file_list_columns+SOX_columns+revenue_columns+efficiency_columns+complexity_columns\n", "#total_revenue_logged + program_efficiency + age + complexity_2011\n", "df[cols][:2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Limit Dataset to Columns Listed Above" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#cols = ['FYE'] + file_list_columns + ['MaterialDiversionOrMisuse', 'MaterialDiversionOrMisuseInd']\n", "#df = df.sort_values(by=['EIN', 'FYE'], ascending=[1, 0])[cols]" ] }, { "cell_type": "code", "execution_count": 645, "metadata": { "collapsed": false }, "outputs": [], "source": [ "df = df.sort_values(by=['EIN', 'FYE'], ascending=[1, 0])[cols]" ] }, { "cell_type": "code", "execution_count": 646, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "538\n" ] }, { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
FYEEINOrganizationNameURLSubmittedOnTaxPeriodWhistleblowerPolicyWhistleblowerPolicyIndConflictOfInterestPolicyIndConflictOfInterestPolicyDocumentRetentionPolicyDocumentRetentionPolicyIndCYTotalRevenueAmtTotalRevenueTotalRevenueCurrentYearCYTotalExpensesAmtTotalExpensesCurrentYearTotalFunctionalExpensesTotalFunctionalExpensesGrpTotalProgramServiceExpenseTotalProgramServiceExpensesAmtCYContributionsGrantsAmtFederatedCampaignsFederatedCampaignsAmtMembershipDuesMembershipDuesAmtNetIncomeFromFundraisingEventsNetIncmFromFundraisingEvtGrpFundraisingAmtFundraisingGrossIncomeAmtRelatedOrganizationsRelatedOrganizationsAmtGovernmentGrantsGovernmentGrantsAmtPYProgramServiceRevenueAmtCYProgramServiceRevenueAmtProgramServiceRevenueProgramServiceRevenueCYProgramServiceRevenueGrpTotalProgramServiceRevenueTotalProgramServiceRevenueAmtCYInvestmentIncomeAmtInvestmentIncomeInvestmentIncomeCurrentYearInvestmentIncomeGrpOtherRevenueTotalAmtOtherRevenueMiscGrpTotalOtherRevenueCYOtherRevenueAmt
252FY2012030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml2013-12-31201212falseNaNNaNtruetrueNaNNaN{u'TotalRevenueColumn': u'2515399', u'ExclusionAmount': u'2063'}2515399NaN2100922{u'ManagementAndGeneral': u'362113', u'ProgramServices': u'1598626', u'Fundraising': u'140183', u'Total': u'2100922'}NaN1598626NaNNaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaNNaNNaN185408NaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'63', u'ExclusionAmount': u'63'}63NaNNaNNaN0NaN
433FY2011030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml2013-02-14201112falseNaNNaNtruetrueNaNNaN{u'TotalRevenueColumn': u'2161209'}2161209NaN2481310{u'ManagementAndGeneral': u'527949', u'ProgramServices': u'1715793', u'Fundraising': u'237568', u'Total': u'2481310'}NaN1715793NaNNaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'0'}0NaNNaNNaN0NaN
\n", "
" ], "text/plain": [ " FYE EIN OrganizationName \\\n", "252 FY2012 030498214 NEWARK NOW INC \n", "433 FY2011 030498214 NEWARK NOW INC \n", "\n", " URL \\\n", "252 https://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml \n", "433 https://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml \n", "\n", " SubmittedOn TaxPeriod WhistleblowerPolicy WhistleblowerPolicyInd \\\n", "252 2013-12-31 201212 false NaN \n", "433 2013-02-14 201112 false NaN \n", "\n", " ConflictOfInterestPolicyInd ConflictOfInterestPolicy \\\n", "252 NaN true \n", "433 NaN true \n", "\n", " DocumentRetentionPolicy DocumentRetentionPolicyInd CYTotalRevenueAmt \\\n", "252 true NaN NaN \n", "433 true NaN NaN \n", "\n", " TotalRevenue \\\n", "252 {u'TotalRevenueColumn': u'2515399', u'ExclusionAmount': u'2063'} \n", "433 {u'TotalRevenueColumn': u'2161209'} \n", "\n", " TotalRevenueCurrentYear CYTotalExpensesAmt TotalExpensesCurrentYear \\\n", "252 2515399 NaN 2100922 \n", "433 2161209 NaN 2481310 \n", "\n", " TotalFunctionalExpenses \\\n", "252 {u'ManagementAndGeneral': u'362113', u'ProgramServices': u'1598626', u'Fundraising': u'140183', u'Total': u'2100922'} \n", "433 {u'ManagementAndGeneral': u'527949', u'ProgramServices': u'1715793', u'Fundraising': u'237568', u'Total': u'2481310'} \n", "\n", " TotalFunctionalExpensesGrp TotalProgramServiceExpense \\\n", "252 NaN 1598626 \n", "433 NaN 1715793 \n", "\n", " TotalProgramServiceExpensesAmt CYContributionsGrantsAmt \\\n", "252 NaN NaN \n", "433 NaN NaN \n", "\n", " FederatedCampaigns FederatedCampaignsAmt MembershipDues MembershipDuesAmt \\\n", "252 NaN NaN NaN NaN \n", "433 NaN NaN NaN NaN \n", "\n", " NetIncomeFromFundraisingEvents NetIncmFromFundraisingEvtGrp \\\n", "252 {u'TotalRevenueColumn': u'0'} NaN \n", "433 {u'TotalRevenueColumn': u'0'} NaN \n", "\n", " FundraisingAmt FundraisingGrossIncomeAmt RelatedOrganizations \\\n", "252 NaN NaN NaN \n", "433 NaN NaN NaN \n", "\n", " RelatedOrganizationsAmt GovernmentGrants GovernmentGrantsAmt \\\n", "252 NaN 185408 NaN \n", "433 NaN NaN NaN \n", "\n", " PYProgramServiceRevenueAmt CYProgramServiceRevenueAmt \\\n", "252 NaN NaN \n", "433 NaN NaN \n", "\n", " ProgramServiceRevenue ProgramServiceRevenueCY ProgramServiceRevenueGrp \\\n", "252 NaN 0 NaN \n", "433 NaN 0 NaN \n", "\n", " TotalProgramServiceRevenue TotalProgramServiceRevenueAmt \\\n", "252 0 NaN \n", "433 0 NaN \n", "\n", " CYInvestmentIncomeAmt \\\n", "252 NaN \n", "433 NaN \n", "\n", " InvestmentIncome \\\n", "252 {u'TotalRevenueColumn': u'63', u'ExclusionAmount': u'63'} \n", "433 {u'TotalRevenueColumn': u'0'} \n", "\n", " InvestmentIncomeCurrentYear InvestmentIncomeGrp OtherRevenueTotalAmt \\\n", "252 63 NaN NaN \n", "433 0 NaN NaN \n", "\n", " OtherRevenueMiscGrp TotalOtherRevenue CYOtherRevenueAmt \n", "252 NaN 0 NaN \n", "433 NaN 0 NaN " ] }, "execution_count": 646, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print len(df)\n", "df[:2]" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "538\n" ] }, { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
FYEOrganizationNameObjectIdURLSubmittedOnDLNLastUpdatedTaxPeriodIsElectronicFormType_idIsAvailableEINMaterialDiversionOrMisuseMaterialDiversionOrMisuseInd
252FY2012NEWARK NOW INC201303199349303505https://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml2013-12-31934933190350532016-03-21T17:23:53201212True99057cce8673ffc5a55fe7f5664True030498214falseNaN
433FY2011NEWARK NOW INC201320249349300417https://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml2013-02-14934930240041732016-03-21T17:23:53201112True99057cce7d63ffc5a55fe7f554dTrue030498214falseNaN
\n", "
" ], "text/plain": [ " FYE OrganizationName ObjectId \\\n", "252 FY2012 NEWARK NOW INC 201303199349303505 \n", "433 FY2011 NEWARK NOW INC 201320249349300417 \n", "\n", " URL \\\n", "252 https://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml \n", "433 https://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml \n", "\n", " SubmittedOn DLN LastUpdated TaxPeriod IsElectronic \\\n", "252 2013-12-31 93493319035053 2016-03-21T17:23:53 201212 True \n", "433 2013-02-14 93493024004173 2016-03-21T17:23:53 201112 True \n", "\n", " FormType _id IsAvailable EIN \\\n", "252 990 57cce8673ffc5a55fe7f5664 True 030498214 \n", "433 990 57cce7d63ffc5a55fe7f554d True 030498214 \n", "\n", " MaterialDiversionOrMisuse MaterialDiversionOrMisuseInd \n", "252 false NaN \n", "433 false NaN " ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#print len(df)\n", "#df[:2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Combine/Fix Revenue" ] }, { "cell_type": "code", "execution_count": 647, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "194\n", "344\n", "344\n" ] } ], "source": [ "print df[revenue_columns[0]].value_counts().sum()\n", "print df[revenue_columns[1]].value_counts().sum()\n", "print df[revenue_columns[2]].value_counts().sum()" ] }, { "cell_type": "code", "execution_count": 648, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CYTotalRevenueAmtTotalRevenueTotalRevenueCurrentYear
252NaN{u'TotalRevenueColumn': u'2515399', u'ExclusionAmount': u'2063'}2515399
433NaN{u'TotalRevenueColumn': u'2161209'}2161209
78NaN{u'TotalRevenueColumn': u'3990564'}3990564
33322995526NaNNaN
12NaN{u'TotalRevenueColumn': u'25923449', u'RelatedOrExemptFunctionIncome': u'451157', u'ExclusionAmount': u'1549366'}25923449
\n", "
" ], "text/plain": [ " CYTotalRevenueAmt \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 22995526 \n", "12 NaN \n", "\n", " TotalRevenue \\\n", "252 {u'TotalRevenueColumn': u'2515399', u'ExclusionAmount': u'2063'} \n", "433 {u'TotalRevenueColumn': u'2161209'} \n", "78 {u'TotalRevenueColumn': u'3990564'} \n", "333 NaN \n", "12 {u'TotalRevenueColumn': u'25923449', u'RelatedOrExemptFunctionIncome': u'451157', u'ExclusionAmount': u'1549366'} \n", "\n", " TotalRevenueCurrentYear \n", "252 2515399 \n", "433 2161209 \n", "78 3990564 \n", "333 NaN \n", "12 25923449 " ] }, "execution_count": 648, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[revenue_columns][:5]" ] }, { "cell_type": "code", "execution_count": 649, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "344\n", "538\n", "538\n" ] } ], "source": [ "df['tot_rev'] = np.nan\n", "print len(df[df['tot_rev'].notnull()])\n", "df['tot_rev'] = np.where(df['TotalRevenueCurrentYear'].notnull(), df['TotalRevenueCurrentYear'], df['tot_rev'] )\n", "print len(df[df['tot_rev'].notnull()])\n", "df['tot_rev'] = np.where( ((df['tot_rev'].isnull()) & (df['CYTotalRevenueAmt'].notnull())), \n", " df['CYTotalRevenueAmt'], df['tot_rev'] )\n", "print len(df[df['tot_rev'].notnull()])\n", "df['tot_rev'] = df['tot_rev'].astype('int')\n", "print len(df[df['tot_rev'].notnull()])" ] }, { "cell_type": "code", "execution_count": 650, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['FYE', u'EIN', u'OrganizationName', u'URL', u'SubmittedOn', u'TaxPeriod', u'WhistleblowerPolicy', u'WhistleblowerPolicyInd', u'ConflictOfInterestPolicyInd', u'ConflictOfInterestPolicy', u'DocumentRetentionPolicy', u'DocumentRetentionPolicyInd', u'CYTotalRevenueAmt', u'TotalRevenue', u'TotalRevenueCurrentYear', u'CYTotalExpensesAmt', u'TotalExpensesCurrentYear', u'TotalFunctionalExpenses', u'TotalFunctionalExpensesGrp', u'TotalProgramServiceExpense', u'TotalProgramServiceExpensesAmt', u'CYContributionsGrantsAmt', u'FederatedCampaigns', u'FederatedCampaignsAmt', u'MembershipDues', u'MembershipDuesAmt', u'NetIncomeFromFundraisingEvents', u'NetIncmFromFundraisingEvtGrp', u'FundraisingAmt', u'FundraisingGrossIncomeAmt', u'RelatedOrganizations', u'RelatedOrganizationsAmt', u'GovernmentGrants', u'GovernmentGrantsAmt', u'PYProgramServiceRevenueAmt', u'CYProgramServiceRevenueAmt', u'ProgramServiceRevenue', u'ProgramServiceRevenueCY', u'ProgramServiceRevenueGrp', u'TotalProgramServiceRevenue', u'TotalProgramServiceRevenueAmt', u'CYInvestmentIncomeAmt', u'InvestmentIncome', u'InvestmentIncomeCurrentYear', u'InvestmentIncomeGrp', u'OtherRevenueTotalAmt', u'OtherRevenueMiscGrp', u'TotalOtherRevenue', u'CYOtherRevenueAmt', 'tot_rev']\n" ] } ], "source": [ "print df.columns.tolist()" ] }, { "cell_type": "code", "execution_count": 587, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
FYEEINOrganizationNameURLSubmittedOnTaxPeriodtot_revWhistleblowerPolicyWhistleblowerPolicyIndConflictOfInterestPolicyIndConflictOfInterestPolicyDocumentRetentionPolicyDocumentRetentionPolicyIndCYTotalExpensesAmtTotalExpensesCurrentYearTotalFunctionalExpensesTotalFunctionalExpensesGrpTotalProgramServiceExpenseTotalProgramServiceExpensesAmtCYContributionsGrantsAmtFederatedCampaignsFederatedCampaignsAmtMembershipDuesMembershipDuesAmtNetIncomeFromFundraisingEventsNetIncmFromFundraisingEvtGrpFundraisingAmtFundraisingGrossIncomeAmtRelatedOrganizationsRelatedOrganizationsAmtGovernmentGrantsGovernmentGrantsAmtPYProgramServiceRevenueAmtCYProgramServiceRevenueAmtProgramServiceRevenueProgramServiceRevenueCYProgramServiceRevenueGrpTotalProgramServiceRevenueTotalProgramServiceRevenueAmtCYInvestmentIncomeAmtInvestmentIncomeInvestmentIncomeCurrentYearInvestmentIncomeGrpOtherRevenueTotalAmtOtherRevenueMiscGrpTotalOtherRevenueCYOtherRevenueAmt
252FY2012030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml2013-12-312012122515399falseNaNNaNtruetrueNaNNaN2100922{u'ManagementAndGeneral': u'362113', u'ProgramServices': u'1598626', u'Fundraising': u'140183', u'Total': u'2100922'}NaN1598626NaNNaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaNNaNNaN185408NaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'63', u'ExclusionAmount': u'63'}63NaNNaNNaN0NaN
433FY2011030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml2013-02-142011122161209falseNaNNaNtruetrueNaNNaN2481310{u'ManagementAndGeneral': u'527949', u'ProgramServices': u'1715793', u'Fundraising': u'237568', u'Total': u'2481310'}NaN1715793NaNNaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'0'}0NaNNaNNaN0NaN
78FY2010030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201220909349300327_public.xml2012-05-252010123990564falseNaNNaNtruetrueNaNNaN3912643{u'ManagementAndGeneral': u'908524', u'ProgramServices': u'2930785', u'Fundraising': u'73334', u'Total': u'3912643'}NaN2930785NaNNaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'0'}0NaNNaNNaN0NaN
333FY2014042129889SCHEPENS EYE RESEARCH INSTITUTE INChttps://s3.amazonaws.com/irs-form-990/201502299349301700_public.xml2015-09-2120140922995526NaNtruetrueNaNNaNtrue25626262NaNNaN{u'TotalAmt': u'25626262', u'ProgramServicesAmt': u'18485740', u'ManagementAndGeneralAmt': u'7140522', u'FundraisingAmt': u'0'}NaN1848574020437991NaNNaNNaNNaNNaN{u'TotalRevenueColumnAmt': u'0'}NaNNaNNaN2053228NaN1612401600NaNNaNNaNNaN01539746NaNNaN{u'TotalRevenueColumnAmt': u'362651', u'ExclusionAmt': u'362651'}428192[{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'...NaN1017789
12FY2013042129889SCHEPENS EYE RESEARCH INSTITUTE INChttps://s3.amazonaws.com/irs-form-990/201442269349304504_public.xml2014-10-2820130925923449trueNaNNaNtruetrueNaNNaN26224173{u'ManagementAndGeneral': u'6501012', u'ProgramServices': u'19280493', u'Fundraising': u'442668', u'Total': u'26224173'}NaN19280493NaNNaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaN2466016NaN16660756NaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'398381', u'ExclusionAmount': u'398381'}1195606NaNNaNNaN451157NaN
\n", "
" ], "text/plain": [ " FYE EIN OrganizationName \\\n", "252 FY2012 030498214 NEWARK NOW INC \n", "433 FY2011 030498214 NEWARK NOW INC \n", "78 FY2010 030498214 NEWARK NOW INC \n", "333 FY2014 042129889 SCHEPENS EYE RESEARCH INSTITUTE INC \n", "12 FY2013 042129889 SCHEPENS EYE RESEARCH INSTITUTE INC \n", "\n", " URL \\\n", "252 https://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml \n", "433 https://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml \n", "78 https://s3.amazonaws.com/irs-form-990/201220909349300327_public.xml \n", "333 https://s3.amazonaws.com/irs-form-990/201502299349301700_public.xml \n", "12 https://s3.amazonaws.com/irs-form-990/201442269349304504_public.xml \n", "\n", " SubmittedOn TaxPeriod tot_rev WhistleblowerPolicy \\\n", "252 2013-12-31 201212 2515399 false \n", "433 2013-02-14 201112 2161209 false \n", "78 2012-05-25 201012 3990564 false \n", "333 2015-09-21 201409 22995526 NaN \n", "12 2014-10-28 201309 25923449 true \n", "\n", " WhistleblowerPolicyInd ConflictOfInterestPolicyInd \\\n", "252 NaN NaN \n", "433 NaN NaN \n", "78 NaN NaN \n", "333 true true \n", "12 NaN NaN \n", "\n", " ConflictOfInterestPolicy DocumentRetentionPolicy \\\n", "252 true true \n", "433 true true \n", "78 true true \n", "333 NaN NaN \n", "12 true true \n", "\n", " DocumentRetentionPolicyInd CYTotalExpensesAmt TotalExpensesCurrentYear \\\n", "252 NaN NaN 2100922 \n", "433 NaN NaN 2481310 \n", "78 NaN NaN 3912643 \n", "333 true 25626262 NaN \n", "12 NaN NaN 26224173 \n", "\n", " TotalFunctionalExpenses \\\n", "252 {u'ManagementAndGeneral': u'362113', u'ProgramServices': u'1598626', u'Fundraising': u'140183', u'Total': u'2100922'} \n", "433 {u'ManagementAndGeneral': u'527949', u'ProgramServices': u'1715793', u'Fundraising': u'237568', u'Total': u'2481310'} \n", "78 {u'ManagementAndGeneral': u'908524', u'ProgramServices': u'2930785', u'Fundraising': u'73334', u'Total': u'3912643'} \n", "333 NaN \n", "12 {u'ManagementAndGeneral': u'6501012', u'ProgramServices': u'19280493', u'Fundraising': u'442668', u'Total': u'26224173'} \n", "\n", " TotalFunctionalExpensesGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 {u'TotalAmt': u'25626262', u'ProgramServicesAmt': u'18485740', u'ManagementAndGeneralAmt': u'7140522', u'FundraisingAmt': u'0'} \n", "12 NaN \n", "\n", " TotalProgramServiceExpense TotalProgramServiceExpensesAmt \\\n", "252 1598626 NaN \n", "433 1715793 NaN \n", "78 2930785 NaN \n", "333 NaN 18485740 \n", "12 19280493 NaN \n", "\n", " CYContributionsGrantsAmt FederatedCampaigns FederatedCampaignsAmt \\\n", "252 NaN NaN NaN \n", "433 NaN NaN NaN \n", "78 NaN NaN NaN \n", "333 20437991 NaN NaN \n", "12 NaN NaN NaN \n", "\n", " MembershipDues MembershipDuesAmt NetIncomeFromFundraisingEvents \\\n", "252 NaN NaN {u'TotalRevenueColumn': u'0'} \n", "433 NaN NaN {u'TotalRevenueColumn': u'0'} \n", "78 NaN NaN {u'TotalRevenueColumn': u'0'} \n", "333 NaN NaN NaN \n", "12 NaN NaN {u'TotalRevenueColumn': u'0'} \n", "\n", " NetIncmFromFundraisingEvtGrp FundraisingAmt \\\n", "252 NaN NaN \n", "433 NaN NaN \n", "78 NaN NaN \n", "333 {u'TotalRevenueColumnAmt': u'0'} NaN \n", "12 NaN NaN \n", "\n", " FundraisingGrossIncomeAmt RelatedOrganizations RelatedOrganizationsAmt \\\n", "252 NaN NaN NaN \n", "433 NaN NaN NaN \n", "78 NaN NaN NaN \n", "333 NaN NaN 2053228 \n", "12 NaN 2466016 NaN \n", "\n", " GovernmentGrants GovernmentGrantsAmt PYProgramServiceRevenueAmt \\\n", "252 185408 NaN NaN \n", "433 NaN NaN NaN \n", "78 NaN NaN NaN \n", "333 NaN 16124016 0 \n", "12 16660756 NaN NaN \n", "\n", " CYProgramServiceRevenueAmt ProgramServiceRevenue ProgramServiceRevenueCY \\\n", "252 NaN NaN 0 \n", "433 NaN NaN 0 \n", "78 NaN NaN 0 \n", "333 0 NaN NaN \n", "12 NaN NaN 0 \n", "\n", " ProgramServiceRevenueGrp TotalProgramServiceRevenue \\\n", "252 NaN 0 \n", "433 NaN 0 \n", "78 NaN 0 \n", "333 NaN NaN \n", "12 NaN 0 \n", "\n", " TotalProgramServiceRevenueAmt CYInvestmentIncomeAmt \\\n", "252 NaN NaN \n", "433 NaN NaN \n", "78 NaN NaN \n", "333 0 1539746 \n", "12 NaN NaN \n", "\n", " InvestmentIncome \\\n", "252 {u'TotalRevenueColumn': u'63', u'ExclusionAmount': u'63'} \n", "433 {u'TotalRevenueColumn': u'0'} \n", "78 {u'TotalRevenueColumn': u'0'} \n", "333 NaN \n", "12 {u'TotalRevenueColumn': u'398381', u'ExclusionAmount': u'398381'} \n", "\n", " InvestmentIncomeCurrentYear \\\n", "252 63 \n", "433 0 \n", "78 0 \n", "333 NaN \n", "12 1195606 \n", "\n", " InvestmentIncomeGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 {u'TotalRevenueColumnAmt': u'362651', u'ExclusionAmt': u'362651'} \n", "12 NaN \n", "\n", " OtherRevenueTotalAmt \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 428192 \n", "12 NaN \n", "\n", " OtherRevenueMiscGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 [{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'... \n", "12 NaN \n", "\n", " TotalOtherRevenue CYOtherRevenueAmt \n", "252 0 NaN \n", "433 0 NaN \n", "78 0 NaN \n", "333 NaN 1017789 \n", "12 451157 NaN " ] }, "execution_count": 587, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_cols = ['FYE', 'EIN', 'OrganizationName', 'URL', 'SubmittedOn', 'TaxPeriod', \n", " #'CYTotalRevenueAmt', 'TotalRevenue', 'TotalRevenueCurrentYear', \n", " 'tot_rev',\n", " 'WhistleblowerPolicy', 'WhistleblowerPolicyInd', \n", " 'ConflictOfInterestPolicyInd', 'ConflictOfInterestPolicy', \n", " 'DocumentRetentionPolicy', 'DocumentRetentionPolicyInd', \n", " 'CYTotalExpensesAmt', 'TotalExpensesCurrentYear', \n", " 'TotalFunctionalExpenses', 'TotalFunctionalExpensesGrp', \n", " 'TotalProgramServiceExpense', 'TotalProgramServiceExpensesAmt', \n", " 'CYContributionsGrantsAmt', \n", " 'FederatedCampaigns', 'FederatedCampaignsAmt', \n", " 'MembershipDues', 'MembershipDuesAmt', \n", " 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp', \n", " 'FundraisingAmt', 'FundraisingGrossIncomeAmt',\n", " 'RelatedOrganizations', 'RelatedOrganizationsAmt', \n", " 'GovernmentGrants', 'GovernmentGrantsAmt', \n", " 'PYProgramServiceRevenueAmt', 'CYProgramServiceRevenueAmt', 'ProgramServiceRevenue', \n", " 'ProgramServiceRevenueCY', 'ProgramServiceRevenueGrp', 'TotalProgramServiceRevenue', \n", " 'TotalProgramServiceRevenueAmt', \n", " 'CYInvestmentIncomeAmt', 'InvestmentIncome', 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', \n", " 'OtherRevenueTotalAmt', 'OtherRevenueMiscGrp', 'TotalOtherRevenue', 'CYOtherRevenueAmt']\n", "df[new_cols][:5]" ] }, { "cell_type": "code", "execution_count": 651, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countmeanstdmin25%50%75%max
tot_rev538.05.857063e+073.234413e+08-218265025.01320586.254261393.513341627.03.741635e+09
\n", "
" ], "text/plain": [ " count mean std min 25% \\\n", "tot_rev 538.0 5.857063e+07 3.234413e+08 -218265025.0 1320586.25 \n", "\n", " 50% 75% max \n", "tot_rev 4261393.5 13341627.0 3.741635e+09 " ] }, "execution_count": 651, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.describe().T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fix/Combine SOX Policy Columns" ] }, { "cell_type": "code", "execution_count": 652, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "344\n", "194 \n", "\n", "344\n", "194 \n", "\n", "344\n", "194 \n", "\n" ] } ], "source": [ "print df['WhistleblowerPolicy'].value_counts().sum()\n", "print df['WhistleblowerPolicyInd'].value_counts().sum(), '\\n'\n", "print df['ConflictOfInterestPolicy'].value_counts().sum()\n", "print df['ConflictOfInterestPolicyInd'].value_counts().sum(), '\\n'\n", "print df['DocumentRetentionPolicy'].value_counts().sum()\n", "print df['DocumentRetentionPolicyInd'].value_counts().sum(), '\\n'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Material Diversion\n", "NOTE: THIS VARIABLE JUST ADDED IN AD HOC -- IT WASN'T SAVED IN INITIAL ROUND." ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "344\n", "538\n" ] }, { "data": { "text/plain": [ "0 278\n", "false 246\n", "true 9\n", "1 5\n", "Name: material_diversion, dtype: int64" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['material_diversion'] = np.nan\n", "print len(df[df['material_diversion'].notnull()])\n", "df['material_diversion'] = np.where(df['MaterialDiversionOrMisuse'].notnull(), df['MaterialDiversionOrMisuse'], \n", " df['material_diversion'] )\n", "print len(df[df['material_diversion'].notnull()])\n", "df['material_diversion'] = np.where( ((df['material_diversion'].isnull()) & \n", " (df['MaterialDiversionOrMisuseInd'].notnull())), \n", " df['MaterialDiversionOrMisuseInd'], df['material_diversion'] )\n", "print len(df[df['material_diversion'].notnull()])\n", "df['material_diversion'].value_counts()" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "538\n" ] }, { "data": { "text/plain": [ "0 524\n", "1 14\n", "Name: material_diversion, dtype: int64" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['material_diversion'] = np.where( df['material_diversion']=='true', 1, df['material_diversion'] )\n", "df['material_diversion'] = np.where( df['material_diversion']=='1', 1, df['material_diversion'] )\n", "df['material_diversion'] = np.where( df['material_diversion']=='false', 0, df['material_diversion'] )\n", "df['material_diversion'] = np.where( df['material_diversion']=='0', 0, df['material_diversion'] )\n", "print len(df[df['material_diversion'].notnull()])\n", "df['material_diversion'].value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Whistleblower Policy" ] }, { "cell_type": "code", "execution_count": 653, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "344\n", "538\n" ] }, { "data": { "text/plain": [ "1 179\n", "true 138\n", "false 117\n", "0 104\n", "Name: whistleblower_policy, dtype: int64" ] }, "execution_count": 653, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['whistleblower_policy'] = np.nan\n", "print len(df[df['whistleblower_policy'].notnull()])\n", "df['whistleblower_policy'] = np.where(df['WhistleblowerPolicy'].notnull(), df['WhistleblowerPolicy'], \n", " df['whistleblower_policy'] )\n", "print len(df[df['whistleblower_policy'].notnull()])\n", "df['whistleblower_policy'] = np.where( ((df['whistleblower_policy'].isnull()) & (df['WhistleblowerPolicyInd'].notnull())), \n", " df['WhistleblowerPolicyInd'], df['whistleblower_policy'] )\n", "print len(df[df['whistleblower_policy'].notnull()])\n", "df['whistleblower_policy'].value_counts()" ] }, { "cell_type": "code", "execution_count": 654, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "538\n" ] }, { "data": { "text/plain": [ "1 317\n", "0 221\n", "Name: whistleblower_policy, dtype: int64" ] }, "execution_count": 654, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['whistleblower_policy'] = np.where( df['whistleblower_policy']=='true', 1, df['whistleblower_policy'] )\n", "df['whistleblower_policy'] = np.where( df['whistleblower_policy']=='1', 1, df['whistleblower_policy'] )\n", "df['whistleblower_policy'] = np.where( df['whistleblower_policy']=='false', 0, df['whistleblower_policy'] )\n", "df['whistleblower_policy'] = np.where( df['whistleblower_policy']=='0', 0, df['whistleblower_policy'] )\n", "print len(df[df['whistleblower_policy'].notnull()])\n", "df['whistleblower_policy'].value_counts()" ] }, { "cell_type": "code", "execution_count": 655, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
whistleblower_policyWhistleblowerPolicyIndWhistleblowerPolicy
2520NaNfalse
4330NaNfalse
780NaNfalse
3331trueNaN
121NaNtrue
\n", "
" ], "text/plain": [ " whistleblower_policy WhistleblowerPolicyInd WhistleblowerPolicy\n", "252 0 NaN false\n", "433 0 NaN false\n", "78 0 NaN false\n", "333 1 true NaN\n", "12 1 NaN true" ] }, "execution_count": 655, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[['whistleblower_policy', 'WhistleblowerPolicyInd', 'WhistleblowerPolicy' ]][:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Conflict of Interest Policy" ] }, { "cell_type": "code", "execution_count": 656, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "344\n", "538\n" ] }, { "data": { "text/plain": [ "1 262\n", "true 192\n", "false 63\n", "0 21\n", "Name: conflict_of_interest_policy, dtype: int64" ] }, "execution_count": 656, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['conflict_of_interest_policy'] = np.nan\n", "print len(df[df['conflict_of_interest_policy'].notnull()])\n", "df['conflict_of_interest_policy'] = np.where(df['ConflictOfInterestPolicy'].notnull(), df['ConflictOfInterestPolicy'], \n", " df['conflict_of_interest_policy'] )\n", "print len(df[df['conflict_of_interest_policy'].notnull()])\n", "df['conflict_of_interest_policy'] = np.where( ((df['conflict_of_interest_policy'].isnull()) \n", " & (df['ConflictOfInterestPolicyInd'].notnull())), \n", " df['ConflictOfInterestPolicyInd'], df['conflict_of_interest_policy'] )\n", "print len(df[df['conflict_of_interest_policy'].notnull()])\n", "df['conflict_of_interest_policy'].value_counts()" ] }, { "cell_type": "code", "execution_count": 657, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "538\n" ] }, { "data": { "text/plain": [ "1 454\n", "0 84\n", "Name: conflict_of_interest_policy, dtype: int64" ] }, "execution_count": 657, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['conflict_of_interest_policy'] = np.where( df['conflict_of_interest_policy']=='true', 1, df['conflict_of_interest_policy'] )\n", "df['conflict_of_interest_policy'] = np.where( df['conflict_of_interest_policy']=='1', 1, df['conflict_of_interest_policy'] )\n", "df['conflict_of_interest_policy'] = np.where( df['conflict_of_interest_policy']=='false', 0, df['conflict_of_interest_policy'] )\n", "df['conflict_of_interest_policy'] = np.where( df['conflict_of_interest_policy']=='0', 0, df['conflict_of_interest_policy'] )\n", "print len(df[df['conflict_of_interest_policy'].notnull()])\n", "df['conflict_of_interest_policy'].value_counts()" ] }, { "cell_type": "code", "execution_count": 658, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
conflict_of_interest_policyConflictOfInterestPolicyIndConflictOfInterestPolicy
2521NaNtrue
4331NaNtrue
781NaNtrue
3331trueNaN
121NaNtrue
\n", "
" ], "text/plain": [ " conflict_of_interest_policy ConflictOfInterestPolicyInd \\\n", "252 1 NaN \n", "433 1 NaN \n", "78 1 NaN \n", "333 1 true \n", "12 1 NaN \n", "\n", " ConflictOfInterestPolicy \n", "252 true \n", "433 true \n", "78 true \n", "333 NaN \n", "12 true " ] }, "execution_count": 658, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[['conflict_of_interest_policy', 'ConflictOfInterestPolicyInd', 'ConflictOfInterestPolicy' ]][:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Document Retention Policy" ] }, { "cell_type": "code", "execution_count": 659, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "344\n", "538\n" ] }, { "data": { "text/plain": [ "1 224\n", "true 159\n", "false 96\n", "0 59\n", "Name: records_retention_policy, dtype: int64" ] }, "execution_count": 659, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['records_retention_policy'] = np.nan\n", "print len(df[df['records_retention_policy'].notnull()])\n", "df['records_retention_policy'] = np.where(df['DocumentRetentionPolicy'].notnull(), df['DocumentRetentionPolicy'], \n", " df['records_retention_policy'] )\n", "print len(df[df['records_retention_policy'].notnull()])\n", "df['records_retention_policy'] = np.where( ((df['records_retention_policy'].isnull()) \n", " & (df['DocumentRetentionPolicyInd'].notnull())), \n", " df['DocumentRetentionPolicyInd'], df['records_retention_policy'] )\n", "print len(df[df['records_retention_policy'].notnull()])\n", "df['records_retention_policy'].value_counts()" ] }, { "cell_type": "code", "execution_count": 660, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "538\n" ] }, { "data": { "text/plain": [ "1 383\n", "0 155\n", "Name: records_retention_policy, dtype: int64" ] }, "execution_count": 660, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['records_retention_policy'] = np.where( df['records_retention_policy']=='true', 1, df['records_retention_policy'] )\n", "df['records_retention_policy'] = np.where( df['records_retention_policy']=='1', 1, df['records_retention_policy'] )\n", "df['records_retention_policy'] = np.where( df['records_retention_policy']=='false', 0, df['records_retention_policy'] )\n", "df['records_retention_policy'] = np.where( df['records_retention_policy']=='0', 0, df['records_retention_policy'] )\n", "print len(df[df['records_retention_policy'].notnull()])\n", "df['records_retention_policy'].value_counts()" ] }, { "cell_type": "code", "execution_count": 661, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
records_retention_policyDocumentRetentionPolicyIndDocumentRetentionPolicy
2521NaNtrue
4331NaNtrue
781NaNtrue
3331trueNaN
121NaNtrue
\n", "
" ], "text/plain": [ " records_retention_policy DocumentRetentionPolicyInd \\\n", "252 1 NaN \n", "433 1 NaN \n", "78 1 NaN \n", "333 1 true \n", "12 1 NaN \n", "\n", " DocumentRetentionPolicy \n", "252 true \n", "433 true \n", "78 true \n", "333 NaN \n", "12 true " ] }, "execution_count": 661, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[['records_retention_policy', 'DocumentRetentionPolicyInd', 'DocumentRetentionPolicy' ]][:5]" ] }, { "cell_type": "code", "execution_count": 662, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['FYE', u'EIN', u'OrganizationName', u'URL', u'SubmittedOn', u'TaxPeriod', u'WhistleblowerPolicy', u'WhistleblowerPolicyInd', u'ConflictOfInterestPolicyInd', u'ConflictOfInterestPolicy', u'DocumentRetentionPolicy', u'DocumentRetentionPolicyInd', u'CYTotalRevenueAmt', u'TotalRevenue', u'TotalRevenueCurrentYear', u'CYTotalExpensesAmt', u'TotalExpensesCurrentYear', u'TotalFunctionalExpenses', u'TotalFunctionalExpensesGrp', u'TotalProgramServiceExpense', u'TotalProgramServiceExpensesAmt', u'CYContributionsGrantsAmt', u'FederatedCampaigns', u'FederatedCampaignsAmt', u'MembershipDues', u'MembershipDuesAmt', u'NetIncomeFromFundraisingEvents', u'NetIncmFromFundraisingEvtGrp', u'FundraisingAmt', u'FundraisingGrossIncomeAmt', u'RelatedOrganizations', u'RelatedOrganizationsAmt', u'GovernmentGrants', u'GovernmentGrantsAmt', u'PYProgramServiceRevenueAmt', u'CYProgramServiceRevenueAmt', u'ProgramServiceRevenue', u'ProgramServiceRevenueCY', u'ProgramServiceRevenueGrp', u'TotalProgramServiceRevenue', u'TotalProgramServiceRevenueAmt', u'CYInvestmentIncomeAmt', u'InvestmentIncome', u'InvestmentIncomeCurrentYear', u'InvestmentIncomeGrp', u'OtherRevenueTotalAmt', u'OtherRevenueMiscGrp', u'TotalOtherRevenue', u'CYOtherRevenueAmt', 'tot_rev', 'whistleblower_policy', 'conflict_of_interest_policy', 'records_retention_policy']\n" ] } ], "source": [ "print df.columns.tolist()" ] }, { "cell_type": "code", "execution_count": 663, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
FYEEINOrganizationNameURLSubmittedOnTaxPeriodtot_revwhistleblower_policyconflict_of_interest_policyrecords_retention_policyCYTotalExpensesAmtTotalExpensesCurrentYearTotalFunctionalExpensesTotalFunctionalExpensesGrpTotalProgramServiceExpenseTotalProgramServiceExpensesAmtCYContributionsGrantsAmtFederatedCampaignsFederatedCampaignsAmtMembershipDuesMembershipDuesAmtNetIncomeFromFundraisingEventsNetIncmFromFundraisingEvtGrpFundraisingAmtFundraisingGrossIncomeAmtRelatedOrganizationsRelatedOrganizationsAmtGovernmentGrantsGovernmentGrantsAmtPYProgramServiceRevenueAmtCYProgramServiceRevenueAmtProgramServiceRevenueProgramServiceRevenueCYProgramServiceRevenueGrpTotalProgramServiceRevenueTotalProgramServiceRevenueAmtCYInvestmentIncomeAmtInvestmentIncomeInvestmentIncomeCurrentYearInvestmentIncomeGrpOtherRevenueTotalAmtOtherRevenueMiscGrpTotalOtherRevenueCYOtherRevenueAmt
252FY2012030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml2013-12-312012122515399011NaN2100922{u'ManagementAndGeneral': u'362113', u'ProgramServices': u'1598626', u'Fundraising': u'140183', u'Total': u'2100922'}NaN1598626NaNNaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaNNaNNaN185408NaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'63', u'ExclusionAmount': u'63'}63NaNNaNNaN0NaN
433FY2011030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml2013-02-142011122161209011NaN2481310{u'ManagementAndGeneral': u'527949', u'ProgramServices': u'1715793', u'Fundraising': u'237568', u'Total': u'2481310'}NaN1715793NaNNaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'0'}0NaNNaNNaN0NaN
78FY2010030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201220909349300327_public.xml2012-05-252010123990564011NaN3912643{u'ManagementAndGeneral': u'908524', u'ProgramServices': u'2930785', u'Fundraising': u'73334', u'Total': u'3912643'}NaN2930785NaNNaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'0'}0NaNNaNNaN0NaN
333FY2014042129889SCHEPENS EYE RESEARCH INSTITUTE INChttps://s3.amazonaws.com/irs-form-990/201502299349301700_public.xml2015-09-212014092299552611125626262NaNNaN{u'TotalAmt': u'25626262', u'ProgramServicesAmt': u'18485740', u'ManagementAndGeneralAmt': u'7140522', u'FundraisingAmt': u'0'}NaN1848574020437991NaNNaNNaNNaNNaN{u'TotalRevenueColumnAmt': u'0'}NaNNaNNaN2053228NaN1612401600NaNNaNNaNNaN01539746NaNNaN{u'TotalRevenueColumnAmt': u'362651', u'ExclusionAmt': u'362651'}428192[{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'...NaN1017789
12FY2013042129889SCHEPENS EYE RESEARCH INSTITUTE INChttps://s3.amazonaws.com/irs-form-990/201442269349304504_public.xml2014-10-2820130925923449111NaN26224173{u'ManagementAndGeneral': u'6501012', u'ProgramServices': u'19280493', u'Fundraising': u'442668', u'Total': u'26224173'}NaN19280493NaNNaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaN2466016NaN16660756NaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'398381', u'ExclusionAmount': u'398381'}1195606NaNNaNNaN451157NaN
\n", "
" ], "text/plain": [ " FYE EIN OrganizationName \\\n", "252 FY2012 030498214 NEWARK NOW INC \n", "433 FY2011 030498214 NEWARK NOW INC \n", "78 FY2010 030498214 NEWARK NOW INC \n", "333 FY2014 042129889 SCHEPENS EYE RESEARCH INSTITUTE INC \n", "12 FY2013 042129889 SCHEPENS EYE RESEARCH INSTITUTE INC \n", "\n", " URL \\\n", "252 https://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml \n", "433 https://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml \n", "78 https://s3.amazonaws.com/irs-form-990/201220909349300327_public.xml \n", "333 https://s3.amazonaws.com/irs-form-990/201502299349301700_public.xml \n", "12 https://s3.amazonaws.com/irs-form-990/201442269349304504_public.xml \n", "\n", " SubmittedOn TaxPeriod tot_rev whistleblower_policy \\\n", "252 2013-12-31 201212 2515399 0 \n", "433 2013-02-14 201112 2161209 0 \n", "78 2012-05-25 201012 3990564 0 \n", "333 2015-09-21 201409 22995526 1 \n", "12 2014-10-28 201309 25923449 1 \n", "\n", " conflict_of_interest_policy records_retention_policy CYTotalExpensesAmt \\\n", "252 1 1 NaN \n", "433 1 1 NaN \n", "78 1 1 NaN \n", "333 1 1 25626262 \n", "12 1 1 NaN \n", "\n", " TotalExpensesCurrentYear \\\n", "252 2100922 \n", "433 2481310 \n", "78 3912643 \n", "333 NaN \n", "12 26224173 \n", "\n", " TotalFunctionalExpenses \\\n", "252 {u'ManagementAndGeneral': u'362113', u'ProgramServices': u'1598626', u'Fundraising': u'140183', u'Total': u'2100922'} \n", "433 {u'ManagementAndGeneral': u'527949', u'ProgramServices': u'1715793', u'Fundraising': u'237568', u'Total': u'2481310'} \n", "78 {u'ManagementAndGeneral': u'908524', u'ProgramServices': u'2930785', u'Fundraising': u'73334', u'Total': u'3912643'} \n", "333 NaN \n", "12 {u'ManagementAndGeneral': u'6501012', u'ProgramServices': u'19280493', u'Fundraising': u'442668', u'Total': u'26224173'} \n", "\n", " TotalFunctionalExpensesGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 {u'TotalAmt': u'25626262', u'ProgramServicesAmt': u'18485740', u'ManagementAndGeneralAmt': u'7140522', u'FundraisingAmt': u'0'} \n", "12 NaN \n", "\n", " TotalProgramServiceExpense TotalProgramServiceExpensesAmt \\\n", "252 1598626 NaN \n", "433 1715793 NaN \n", "78 2930785 NaN \n", "333 NaN 18485740 \n", "12 19280493 NaN \n", "\n", " CYContributionsGrantsAmt FederatedCampaigns FederatedCampaignsAmt \\\n", "252 NaN NaN NaN \n", "433 NaN NaN NaN \n", "78 NaN NaN NaN \n", "333 20437991 NaN NaN \n", "12 NaN NaN NaN \n", "\n", " MembershipDues MembershipDuesAmt NetIncomeFromFundraisingEvents \\\n", "252 NaN NaN {u'TotalRevenueColumn': u'0'} \n", "433 NaN NaN {u'TotalRevenueColumn': u'0'} \n", "78 NaN NaN {u'TotalRevenueColumn': u'0'} \n", "333 NaN NaN NaN \n", "12 NaN NaN {u'TotalRevenueColumn': u'0'} \n", "\n", " NetIncmFromFundraisingEvtGrp FundraisingAmt \\\n", "252 NaN NaN \n", "433 NaN NaN \n", "78 NaN NaN \n", "333 {u'TotalRevenueColumnAmt': u'0'} NaN \n", "12 NaN NaN \n", "\n", " FundraisingGrossIncomeAmt RelatedOrganizations RelatedOrganizationsAmt \\\n", "252 NaN NaN NaN \n", "433 NaN NaN NaN \n", "78 NaN NaN NaN \n", "333 NaN NaN 2053228 \n", "12 NaN 2466016 NaN \n", "\n", " GovernmentGrants GovernmentGrantsAmt PYProgramServiceRevenueAmt \\\n", "252 185408 NaN NaN \n", "433 NaN NaN NaN \n", "78 NaN NaN NaN \n", "333 NaN 16124016 0 \n", "12 16660756 NaN NaN \n", "\n", " CYProgramServiceRevenueAmt ProgramServiceRevenue ProgramServiceRevenueCY \\\n", "252 NaN NaN 0 \n", "433 NaN NaN 0 \n", "78 NaN NaN 0 \n", "333 0 NaN NaN \n", "12 NaN NaN 0 \n", "\n", " ProgramServiceRevenueGrp TotalProgramServiceRevenue \\\n", "252 NaN 0 \n", "433 NaN 0 \n", "78 NaN 0 \n", "333 NaN NaN \n", "12 NaN 0 \n", "\n", " TotalProgramServiceRevenueAmt CYInvestmentIncomeAmt \\\n", "252 NaN NaN \n", "433 NaN NaN \n", "78 NaN NaN \n", "333 0 1539746 \n", "12 NaN NaN \n", "\n", " InvestmentIncome \\\n", "252 {u'TotalRevenueColumn': u'63', u'ExclusionAmount': u'63'} \n", "433 {u'TotalRevenueColumn': u'0'} \n", "78 {u'TotalRevenueColumn': u'0'} \n", "333 NaN \n", "12 {u'TotalRevenueColumn': u'398381', u'ExclusionAmount': u'398381'} \n", "\n", " InvestmentIncomeCurrentYear \\\n", "252 63 \n", "433 0 \n", "78 0 \n", "333 NaN \n", "12 1195606 \n", "\n", " InvestmentIncomeGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 {u'TotalRevenueColumnAmt': u'362651', u'ExclusionAmt': u'362651'} \n", "12 NaN \n", "\n", " OtherRevenueTotalAmt \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 428192 \n", "12 NaN \n", "\n", " OtherRevenueMiscGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 [{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'... \n", "12 NaN \n", "\n", " TotalOtherRevenue CYOtherRevenueAmt \n", "252 0 NaN \n", "433 0 NaN \n", "78 0 NaN \n", "333 NaN 1017789 \n", "12 451157 NaN " ] }, "execution_count": 663, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_cols = ['FYE', 'EIN', 'OrganizationName', 'URL', 'SubmittedOn', 'TaxPeriod', \n", " 'tot_rev', 'whistleblower_policy', 'conflict_of_interest_policy', 'records_retention_policy',\n", " #'WhistleblowerPolicy', u'WhistleblowerPolicyInd', u'ConflictOfInterestPolicyInd', \n", " #'ConflictOfInterestPolicy', u'DocumentRetentionPolicy', u'DocumentRetentionPolicyInd', \n", " #'CYTotalRevenueAmt', 'TotalRevenue', 'TotalRevenueCurrentYear', \n", " 'CYTotalExpensesAmt', \n", " 'TotalExpensesCurrentYear', 'TotalFunctionalExpenses', 'TotalFunctionalExpensesGrp', \n", " 'TotalProgramServiceExpense', 'TotalProgramServiceExpensesAmt', 'CYContributionsGrantsAmt', \n", " 'FederatedCampaigns', 'FederatedCampaignsAmt', u'MembershipDues', 'MembershipDuesAmt', \n", " 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp', \n", " 'FundraisingAmt', 'FundraisingGrossIncomeAmt',\n", " 'RelatedOrganizations', \n", " 'RelatedOrganizationsAmt', 'GovernmentGrants', 'GovernmentGrantsAmt', 'PYProgramServiceRevenueAmt', \n", " 'CYProgramServiceRevenueAmt', 'ProgramServiceRevenue', 'ProgramServiceRevenueCY', \n", " 'ProgramServiceRevenueGrp', 'TotalProgramServiceRevenue', 'TotalProgramServiceRevenueAmt', \n", " 'CYInvestmentIncomeAmt', 'InvestmentIncome', 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', \n", " 'OtherRevenueTotalAmt', 'OtherRevenueMiscGrp', 'TotalOtherRevenue', 'CYOtherRevenueAmt', \n", " ]\n", "df = df[new_cols]\n", "df[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Save DF\n", "152 unique EINs in this dataset -- roughly half of the 321 '2016' donor advisories." ] }, { "cell_type": "code", "execution_count": 381, "metadata": { "collapsed": true }, "outputs": [], "source": [ "df.to_pickle('538 e-file 990s for 2016 donor advisory organizations, v2.pkl')" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#cols = ['FYE'] + file_list_columns + ['MaterialDiversionOrMisuse', 'MaterialDiversionOrMisuseInd'] + ['material_diversion']\n", "#df[cols].to_pickle('538 e-file 990s for 2016 donor advisory organizations (material diversions only).pkl')" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#df['material_diversion'] = df['material_diversion'].astype('int')" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countmeanstdmin25%50%75%max
material_diversion538.00.0260220.159350.00.00.00.01.0
\n", "
" ], "text/plain": [ " count mean std min 25% 50% 75% max\n", "material_diversion 538.0 0.026022 0.15935 0.0 0.0 0.0 0.0 1.0" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#df[cols].describe().T" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#cols = ['FYE'] + file_list_columns + ['MaterialDiversionOrMisuse', 'MaterialDiversionOrMisuseInd'] + ['material_diversion']\n", "#df[cols].to_pickle('538 e-file 990s for 2016 donor advisory organizations (material diversions only).pkl')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fix/Combine Efficiency Variables" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Total Expenses" ] }, { "cell_type": "code", "execution_count": 664, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CYTotalExpensesAmtTotalExpensesCurrentYearTotalFunctionalExpensesTotalFunctionalExpensesGrpTotalProgramServiceExpenseTotalProgramServiceExpensesAmt
252NaN2100922{u'ManagementAndGeneral': u'362113', u'ProgramServices': u'1598626', u'Fundraising': u'140183', u'Total': u'2100922'}NaN1598626NaN
433NaN2481310{u'ManagementAndGeneral': u'527949', u'ProgramServices': u'1715793', u'Fundraising': u'237568', u'Total': u'2481310'}NaN1715793NaN
78NaN3912643{u'ManagementAndGeneral': u'908524', u'ProgramServices': u'2930785', u'Fundraising': u'73334', u'Total': u'3912643'}NaN2930785NaN
33325626262NaNNaN{u'TotalAmt': u'25626262', u'ProgramServicesAmt': u'18485740', u'ManagementAndGeneralAmt': u'7140522', u'FundraisingAmt': u'0'}NaN18485740
\n", "
" ], "text/plain": [ " CYTotalExpensesAmt TotalExpensesCurrentYear \\\n", "252 NaN 2100922 \n", "433 NaN 2481310 \n", "78 NaN 3912643 \n", "333 25626262 NaN \n", "\n", " TotalFunctionalExpenses \\\n", "252 {u'ManagementAndGeneral': u'362113', u'ProgramServices': u'1598626', u'Fundraising': u'140183', u'Total': u'2100922'} \n", "433 {u'ManagementAndGeneral': u'527949', u'ProgramServices': u'1715793', u'Fundraising': u'237568', u'Total': u'2481310'} \n", "78 {u'ManagementAndGeneral': u'908524', u'ProgramServices': u'2930785', u'Fundraising': u'73334', u'Total': u'3912643'} \n", "333 NaN \n", "\n", " TotalFunctionalExpensesGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 {u'TotalAmt': u'25626262', u'ProgramServicesAmt': u'18485740', u'ManagementAndGeneralAmt': u'7140522', u'FundraisingAmt': u'0'} \n", "\n", " TotalProgramServiceExpense TotalProgramServiceExpensesAmt \n", "252 1598626 NaN \n", "433 1715793 NaN \n", "78 2930785 NaN \n", "333 NaN 18485740 " ] }, "execution_count": 664, "metadata": {}, "output_type": "execute_result" } ], "source": [ "efficiency_columns = ['CYTotalExpensesAmt', 'TotalExpensesCurrentYear',\n", " 'TotalFunctionalExpenses', 'TotalFunctionalExpensesGrp',\n", " 'TotalProgramServiceExpense', 'TotalProgramServiceExpensesAmt',]\n", "df[efficiency_columns][:4]" ] }, { "cell_type": "code", "execution_count": 665, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "194\n", "538\n" ] }, { "data": { "text/plain": [ "count 538\n", "unique 536\n", "top 271781\n", "freq 2\n", "Name: total_expenses, dtype: object" ] }, "execution_count": 665, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['total_expenses'] = np.nan\n", "print len(df[df['total_expenses'].notnull()])\n", "df['total_expenses'] = np.where(df['CYTotalExpensesAmt'].notnull(), df['CYTotalExpensesAmt'], \n", " df['total_expenses'] )\n", "print len(df[df['total_expenses'].notnull()])\n", "df['total_expenses'] = np.where( ((df['total_expenses'].isnull()) & (df['TotalExpensesCurrentYear'].notnull())), \n", " df['TotalExpensesCurrentYear'], df['total_expenses'] )\n", "print len(df[df['total_expenses'].notnull()])\n", "df['total_expenses'].describe().T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Program Service Expenses" ] }, { "cell_type": "code", "execution_count": 666, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "336\n", "524\n" ] }, { "data": { "text/plain": [ "count 524\n", "unique 522\n", "top 31689530\n", "freq 2\n", "Name: program_expenses, dtype: object" ] }, "execution_count": 666, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['program_expenses'] = np.nan\n", "print len(df[df['program_expenses'].notnull()])\n", "df['program_expenses'] = np.where(df['TotalProgramServiceExpense'].notnull(), df['TotalProgramServiceExpense'], \n", " df['program_expenses'] )\n", "print len(df[df['program_expenses'].notnull()])\n", "df['program_expenses'] = np.where( ((df['program_expenses'].isnull()) & (df['TotalProgramServiceExpensesAmt'].notnull())), \n", " df['TotalProgramServiceExpensesAmt'], df['program_expenses'] )\n", "print len(df[df['program_expenses'].notnull()])\n", "df['program_expenses'].describe().T" ] }, { "cell_type": "code", "execution_count": 667, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "14\n" ] }, { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
FYEEINOrganizationNameURLSubmittedOnTaxPeriodtot_revwhistleblower_policyconflict_of_interest_policyrecords_retention_policyCYTotalExpensesAmtTotalExpensesCurrentYearTotalFunctionalExpensesTotalFunctionalExpensesGrpTotalProgramServiceExpenseTotalProgramServiceExpensesAmtCYContributionsGrantsAmtFederatedCampaignsFederatedCampaignsAmtMembershipDuesMembershipDuesAmtNetIncomeFromFundraisingEventsNetIncmFromFundraisingEvtGrpFundraisingAmtFundraisingGrossIncomeAmtRelatedOrganizationsRelatedOrganizationsAmtGovernmentGrantsGovernmentGrantsAmtPYProgramServiceRevenueAmtCYProgramServiceRevenueAmtProgramServiceRevenueProgramServiceRevenueCYProgramServiceRevenueGrpTotalProgramServiceRevenueTotalProgramServiceRevenueAmtCYInvestmentIncomeAmtInvestmentIncomeInvestmentIncomeCurrentYearInvestmentIncomeGrpOtherRevenueTotalAmtOtherRevenueMiscGrpTotalOtherRevenueCYOtherRevenueAmttotal_expensesprogram_expenses
172FY2014132644507WORLD TRADE CENTERS ASSOCIATION INChttps://s3.amazonaws.com/irs-form-990/201543149349303504_public.xml2016-02-0820141279131041117763846NaNNaN{u'TotalAmt': u'7763846'}NaNNaN0NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN88344996812087NaNNaN[{u'TotalRevenueColumnAmt': u'3242920', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'3242920', u'Desc': u'MEMBERSHIP DUES'}, {u'TotalRevenueColumnAmt': u'2890000', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'28900...NaN68120871075687NaNNaN{u'TotalRevenueColumnAmt': u'334806', u'ExclusionAmt': u'334806'}25330{u'TotalRevenueColumnAmt': u'25330', u'BusinessCd': u'900099', u'ExclusionAmt': u'25330', u'Desc': u'MISCELLANEOUS'}NaN253307763846NaN
404FY2013132644507WORLD TRADE CENTERS ASSOCIATION INChttps://s3.amazonaws.com/irs-form-990/201433149349302383_public.xml2014-12-1020131292283951117795021NaNNaN{u'TotalAmt': u'7795021'}NaNNaN0NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN75202458834499NaNNaN[{u'TotalRevenueColumnAmt': u'4774000', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'4774000', u'Desc': u'INITIATION FEES'}, {u'TotalRevenueColumnAmt': u'3347499', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'33474...NaN8834499391315NaNNaN{u'TotalRevenueColumnAmt': u'275100', u'ExclusionAmt': u'275100'}2581{u'TotalRevenueColumnAmt': u'2581', u'BusinessCd': u'900099', u'ExclusionAmt': u'2581', u'Desc': u'MISCELLANEOUS'}NaN25817795021NaN
431FY2012132644507WORLD TRADE CENTERS ASSOCIATION INChttps://s3.amazonaws.com/irs-form-990/201313169349303646_public.xml2013-12-242012127749706111NaN6237509{u'Total': u'6237509'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN[{u'BusinessCode': u'900099', u'TotalRevenueColumn': u'3333245', u'RelatedOrExemptFunctionIncome': u'3333245', u'Description': u'MEMBERSHIP DUES'}, {u'BusinessCode': u'900099', u'TotalRevenueColumn': u'3278000', u'RelatedOrExemptFunctionIncome': ...7520245NaN7520245NaNNaN{u'TotalRevenueColumn': u'234964', u'ExclusionAmount': u'234964'}211074NaNNaNNaN18387NaN6237509NaN
106FY2011132644507WORLD TRADE CENTERS ASSOCIATION INChttps://s3.amazonaws.com/irs-form-990/201222279349301852_public.xml2012-11-152011127072954111NaN5880789{u'Total': u'5880789'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN[{u'BusinessCode': u'900099', u'TotalRevenueColumn': u'3313485', u'RelatedOrExemptFunctionIncome': u'3313485', u'Description': u'MEMBERSHIP DUES'}, {u'BusinessCode': u'900099', u'TotalRevenueColumn': u'3183000', u'RelatedOrExemptFunctionIncome': ...6926485NaN6926485NaNNaN{u'TotalRevenueColumn': u'237277', u'ExclusionAmount': u'237277'}144021NaNNaNNaN2448NaN5880789NaN
72FY2010132644507WORLD TRADE CENTERS ASSOCIATION INChttps://s3.amazonaws.com/irs-form-990/201123199349305807_public.xml2012-01-032010126858206111NaN6024086{u'Total': u'6024086'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN[{u'BusinessCode': u'900099', u'TotalRevenueColumn': u'3452000', u'RelatedOrExemptFunctionIncome': u'3452000', u'Description': u'INITIATION FEES'}, {u'BusinessCode': u'900099', u'TotalRevenueColumn': u'3134807', u'RelatedOrExemptFunctionIncome': ...6586807NaN6586807NaNNaN{u'TotalRevenueColumn': u'217696', u'ExclusionAmount': u'217696'}192714NaNNaNNaN78685NaN6024086NaN
52FY2014135579302UNITED HEBREW CEMETERY INChttps://s3.amazonaws.com/irs-form-990/201531949349300508_public.xml2015-07-2920141242238261112023750NaNNaN{u'TotalAmt': u'2023750'}NaNNaN0NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN25723052466379NaNNaN[{u'TotalRevenueColumnAmt': u'1235180', u'RelatedOrExemptFuncIncomeAmt': u'1235180', u'Desc': u'SALES OF GRAVES & PLOTS'}, {u'TotalRevenueColumnAmt': u'601571', u'RelatedOrExemptFuncIncomeAmt': u'601571', u'Desc': u'INTERMENTS'}, {u'TotalRevenueC...NaN24663791299578NaNNaN{u'TotalRevenueColumnAmt': u'481123', u'RelatedOrExemptFuncIncomeAmt': u'481123'}457869{u'TotalRevenueColumnAmt': u'457869', u'RelatedOrExemptFuncIncomeAmt': u'457869', u'Desc': u'PERPETUAL CARE ENDOWMENTS'}NaN4578692023750NaN
406FY2013135579302UNITED HEBREW CEMETERY INChttps://s3.amazonaws.com/irs-form-990/201433039349300503_public.xml2014-11-2420131249361780004133761NaNNaN{u'TotalAmt': u'4133761', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'0', u'FundraisingAmt': u'0'}NaNNaN0NaNNaNNaNNaNNaN{u'TotalRevenueColumnAmt': u'0'}NaNNaNNaNNaNNaNNaN21965462572305NaNNaN[{u'TotalRevenueColumnAmt': u'43547', u'RelatedOrExemptFuncIncomeAmt': u'43547', u'Desc': u'CELL TOWER INCOME'}, {u'TotalRevenueColumnAmt': u'371441', u'RelatedOrExemptFuncIncomeAmt': u'371441', u'Desc': u'FOUNDATIONS'}, {u'TotalRevenueColumnAmt'...NaN2572305917828NaNNaN{u'TotalRevenueColumnAmt': u'584869', u'RelatedOrExemptFuncIncomeAmt': u'584869'}1446045[{u'TotalRevenueColumnAmt': u'1000000', u'RelatedOrExemptFuncIncomeAmt': u'1000000', u'Desc': u'LEGAL RECOVERY SETTLEMENT'}, {u'TotalRevenueColumnAmt': u'446045', u'RelatedOrExemptFuncIncomeAmt': u'446045', u'Desc': u'PERPETUAL CARE ENDOWMENTS'}]NaN14460454133761NaN
500FY2010201737542GREATER CORNERSTONE COMMUNITY DEVhttps://s3.amazonaws.com/irs-form-990/201111369349303781_public.xml2011-06-08201006548759111NaN177901{u'ManagementAndGeneral': u'111025', u'ProgramServices': u'0', u'Fundraising': u'66876', u'Total': u'177901'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaNNaNNaNNaNNaN0NaNNaNNaNNaNNaN177901NaN
0FY2014271317072ANCHOR BUILDING FUND INChttps://s3.amazonaws.com/irs-form-990/201503169349304780_public.xml2016-01-3020141232500000034400NaNNaN{u'TotalAmt': u'34400', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'34400', u'FundraisingAmt': u'0'}NaNNaN0NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaNNaNNaNNaNNaN0NaNNaNNaN325000{u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000', u'Desc': u'SETTLEMENT'}NaN32500034400NaN
277FY2014330837780INTERNATIONAL CHRISTIAN BROADCASTING INChttps://s3.amazonaws.com/irs-form-990/201543209349314539_public.xml2016-02-2220141299261113330NaNNaN{u'TotalAmt': u'3330', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'3330', u'FundraisingAmt': u'0'}NaNNaN9225NaNNaNNaNNaNNaNNaNNaNNaNNaN9025NaNNaNNaN0NaNNaNNaNNaNNaN701NaNNaN{u'TotalRevenueColumnAmt': u'701', u'ExclusionAmt': u'701'}NaNNaNNaN03330NaN
226FY2011591951577ALLIED VETERANS OF THE WORLD INChttps://s3.amazonaws.com/irs-form-990/201243209349307874_public.xml2012-12-312011121736929111NaN1690623{u'Total': u'1690623'}NaNNaNNaNNaNNaNNaN26929NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaNNaNNaNNaN{u'TotalRevenueColumn': u'1710000'}1710000NaNNaNNaNNaNNaN1690623NaN
213FY2010591951577ALLIED VETERANS OF THE WORLD INChttps://s3.amazonaws.com/irs-form-990/201132279349301923_public.xml2011-09-072010122496382111NaN1161939{u'Total': u'1161939'}NaNNaNNaNNaNNaNNaN8056NaNNaNNaNNaNNaN2482845NaNNaNNaNNaNNaNNaN0NaNNaNNaNNaNNaN0NaNNaNNaN5481NaN1161939NaN
495FY2010591951577ALLIED VETERANS OF THE WORLD INChttps://s3.amazonaws.com/irs-form-990/201243219349300739_public.xml2012-12-312010122496382111NaN1282149{u'Total': u'1282149'}NaNNaNNaNNaNNaNNaN8056NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaNNaNNaNNaN{u'TotalRevenueColumn': u'2482845'}2482845NaNNaNNaN5481NaN1282149NaN
496FY2011953553530NATIONAL MINORITY TV INChttps://s3.amazonaws.com/irs-form-990/201243219349300444_public.xml2012-12-312011120111NaN0{u'ManagementAndGeneral': u'0', u'ProgramServices': u'0', u'Fundraising': u'0', u'Total': u'0'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaNNaNNaNNaNNaN0NaNNaNNaNNaNNaN0NaN
\n", "
" ], "text/plain": [ " FYE EIN OrganizationName \\\n", "172 FY2014 132644507 WORLD TRADE CENTERS ASSOCIATION INC \n", "404 FY2013 132644507 WORLD TRADE CENTERS ASSOCIATION INC \n", "431 FY2012 132644507 WORLD TRADE CENTERS ASSOCIATION INC \n", "106 FY2011 132644507 WORLD TRADE CENTERS ASSOCIATION INC \n", "72 FY2010 132644507 WORLD TRADE CENTERS ASSOCIATION INC \n", "52 FY2014 135579302 UNITED HEBREW CEMETERY INC \n", "406 FY2013 135579302 UNITED HEBREW CEMETERY INC \n", "500 FY2010 201737542 GREATER CORNERSTONE COMMUNITY DEV \n", "0 FY2014 271317072 ANCHOR BUILDING FUND INC \n", "277 FY2014 330837780 INTERNATIONAL CHRISTIAN BROADCASTING INC \n", "226 FY2011 591951577 ALLIED VETERANS OF THE WORLD INC \n", "213 FY2010 591951577 ALLIED VETERANS OF THE WORLD INC \n", "495 FY2010 591951577 ALLIED VETERANS OF THE WORLD INC \n", "496 FY2011 953553530 NATIONAL MINORITY TV INC \n", "\n", " URL \\\n", "172 https://s3.amazonaws.com/irs-form-990/201543149349303504_public.xml \n", "404 https://s3.amazonaws.com/irs-form-990/201433149349302383_public.xml \n", "431 https://s3.amazonaws.com/irs-form-990/201313169349303646_public.xml \n", "106 https://s3.amazonaws.com/irs-form-990/201222279349301852_public.xml \n", "72 https://s3.amazonaws.com/irs-form-990/201123199349305807_public.xml \n", "52 https://s3.amazonaws.com/irs-form-990/201531949349300508_public.xml \n", "406 https://s3.amazonaws.com/irs-form-990/201433039349300503_public.xml \n", "500 https://s3.amazonaws.com/irs-form-990/201111369349303781_public.xml \n", "0 https://s3.amazonaws.com/irs-form-990/201503169349304780_public.xml \n", "277 https://s3.amazonaws.com/irs-form-990/201543209349314539_public.xml \n", "226 https://s3.amazonaws.com/irs-form-990/201243209349307874_public.xml \n", "213 https://s3.amazonaws.com/irs-form-990/201132279349301923_public.xml \n", "495 https://s3.amazonaws.com/irs-form-990/201243219349300739_public.xml \n", "496 https://s3.amazonaws.com/irs-form-990/201243219349300444_public.xml \n", "\n", " SubmittedOn TaxPeriod tot_rev whistleblower_policy \\\n", "172 2016-02-08 201412 7913104 1 \n", "404 2014-12-10 201312 9228395 1 \n", "431 2013-12-24 201212 7749706 1 \n", "106 2012-11-15 201112 7072954 1 \n", "72 2012-01-03 201012 6858206 1 \n", "52 2015-07-29 201412 4223826 1 \n", "406 2014-11-24 201312 4936178 0 \n", "500 2011-06-08 201006 548759 1 \n", "0 2016-01-30 201412 325000 0 \n", "277 2016-02-22 201412 9926 1 \n", "226 2012-12-31 201112 1736929 1 \n", "213 2011-09-07 201012 2496382 1 \n", "495 2012-12-31 201012 2496382 1 \n", "496 2012-12-31 201112 0 1 \n", "\n", " conflict_of_interest_policy records_retention_policy CYTotalExpensesAmt \\\n", "172 1 1 7763846 \n", "404 1 1 7795021 \n", "431 1 1 NaN \n", "106 1 1 NaN \n", "72 1 1 NaN \n", "52 1 1 2023750 \n", "406 0 0 4133761 \n", "500 1 1 NaN \n", "0 0 0 34400 \n", "277 1 1 3330 \n", "226 1 1 NaN \n", "213 1 1 NaN \n", "495 1 1 NaN \n", "496 1 1 NaN \n", "\n", " TotalExpensesCurrentYear \\\n", "172 NaN \n", "404 NaN \n", "431 6237509 \n", "106 5880789 \n", "72 6024086 \n", "52 NaN \n", "406 NaN \n", "500 177901 \n", "0 NaN \n", "277 NaN \n", "226 1690623 \n", "213 1161939 \n", "495 1282149 \n", "496 0 \n", "\n", " TotalFunctionalExpenses \\\n", "172 NaN \n", "404 NaN \n", "431 {u'Total': u'6237509'} \n", "106 {u'Total': u'5880789'} \n", "72 {u'Total': u'6024086'} \n", "52 NaN \n", "406 NaN \n", "500 {u'ManagementAndGeneral': u'111025', u'ProgramServices': u'0', u'Fundraising': u'66876', u'Total': u'177901'} \n", "0 NaN \n", "277 NaN \n", "226 {u'Total': u'1690623'} \n", "213 {u'Total': u'1161939'} \n", "495 {u'Total': u'1282149'} \n", "496 {u'ManagementAndGeneral': u'0', u'ProgramServices': u'0', u'Fundraising': u'0', u'Total': u'0'} \n", "\n", " TotalFunctionalExpensesGrp \\\n", "172 {u'TotalAmt': u'7763846'} \n", "404 {u'TotalAmt': u'7795021'} \n", "431 NaN \n", "106 NaN \n", "72 NaN \n", "52 {u'TotalAmt': u'2023750'} \n", "406 {u'TotalAmt': u'4133761', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'0', u'FundraisingAmt': u'0'} \n", "500 NaN \n", "0 {u'TotalAmt': u'34400', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'34400', u'FundraisingAmt': u'0'} \n", "277 {u'TotalAmt': u'3330', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'3330', u'FundraisingAmt': u'0'} \n", "226 NaN \n", "213 NaN \n", "495 NaN \n", "496 NaN \n", "\n", " TotalProgramServiceExpense TotalProgramServiceExpensesAmt \\\n", "172 NaN NaN \n", "404 NaN NaN \n", "431 NaN NaN \n", "106 NaN NaN \n", "72 NaN NaN \n", "52 NaN NaN \n", "406 NaN NaN \n", "500 NaN NaN \n", "0 NaN NaN \n", "277 NaN NaN \n", "226 NaN NaN \n", "213 NaN NaN \n", "495 NaN NaN \n", "496 NaN NaN \n", "\n", " CYContributionsGrantsAmt FederatedCampaigns FederatedCampaignsAmt \\\n", "172 0 NaN NaN \n", "404 0 NaN NaN \n", "431 NaN NaN NaN \n", "106 NaN NaN NaN \n", "72 NaN NaN NaN \n", "52 0 NaN NaN \n", "406 0 NaN NaN \n", "500 NaN NaN NaN \n", "0 0 NaN NaN \n", "277 9225 NaN NaN \n", "226 NaN NaN NaN \n", "213 NaN NaN NaN \n", "495 NaN NaN NaN \n", "496 NaN NaN NaN \n", "\n", " MembershipDues MembershipDuesAmt NetIncomeFromFundraisingEvents \\\n", "172 NaN NaN NaN \n", "404 NaN NaN NaN \n", "431 NaN NaN NaN \n", "106 NaN NaN NaN \n", "72 NaN NaN NaN \n", "52 NaN NaN NaN \n", "406 NaN NaN NaN \n", "500 NaN NaN NaN \n", "0 NaN NaN NaN \n", "277 NaN NaN NaN \n", "226 26929 NaN NaN \n", "213 8056 NaN NaN \n", "495 8056 NaN NaN \n", "496 NaN NaN NaN \n", "\n", " NetIncmFromFundraisingEvtGrp FundraisingAmt \\\n", "172 NaN NaN \n", "404 NaN NaN \n", "431 NaN NaN \n", "106 NaN NaN \n", "72 NaN NaN \n", "52 NaN NaN \n", "406 {u'TotalRevenueColumnAmt': u'0'} NaN \n", "500 NaN NaN \n", "0 NaN NaN \n", "277 NaN NaN \n", "226 NaN NaN \n", "213 NaN NaN \n", "495 NaN NaN \n", "496 NaN NaN \n", "\n", " FundraisingGrossIncomeAmt RelatedOrganizations RelatedOrganizationsAmt \\\n", "172 NaN NaN NaN \n", "404 NaN NaN NaN \n", "431 NaN NaN NaN \n", "106 NaN NaN NaN \n", "72 NaN NaN NaN \n", "52 NaN NaN NaN \n", "406 NaN NaN NaN \n", "500 NaN NaN NaN \n", "0 NaN NaN NaN \n", "277 NaN NaN 9025 \n", "226 NaN NaN NaN \n", "213 NaN 2482845 NaN \n", "495 NaN NaN NaN \n", "496 NaN NaN NaN \n", "\n", " GovernmentGrants GovernmentGrantsAmt PYProgramServiceRevenueAmt \\\n", "172 NaN NaN 8834499 \n", "404 NaN NaN 7520245 \n", "431 NaN NaN NaN \n", "106 NaN NaN NaN \n", "72 NaN NaN NaN \n", "52 NaN NaN 2572305 \n", "406 NaN NaN 2196546 \n", "500 NaN NaN NaN \n", "0 NaN NaN NaN \n", "277 NaN NaN NaN \n", "226 NaN NaN NaN \n", "213 NaN NaN NaN \n", "495 NaN NaN NaN \n", "496 NaN NaN NaN \n", "\n", " CYProgramServiceRevenueAmt \\\n", "172 6812087 \n", "404 8834499 \n", "431 NaN \n", "106 NaN \n", "72 NaN \n", "52 2466379 \n", "406 2572305 \n", "500 NaN \n", "0 0 \n", "277 0 \n", "226 NaN \n", "213 NaN \n", "495 NaN \n", "496 NaN \n", "\n", " ProgramServiceRevenue \\\n", "172 NaN \n", "404 NaN \n", "431 [{u'BusinessCode': u'900099', u'TotalRevenueColumn': u'3333245', u'RelatedOrExemptFunctionIncome': u'3333245', u'Description': u'MEMBERSHIP DUES'}, {u'BusinessCode': u'900099', u'TotalRevenueColumn': u'3278000', u'RelatedOrExemptFunctionIncome': ... \n", "106 [{u'BusinessCode': u'900099', u'TotalRevenueColumn': u'3313485', u'RelatedOrExemptFunctionIncome': u'3313485', u'Description': u'MEMBERSHIP DUES'}, {u'BusinessCode': u'900099', u'TotalRevenueColumn': u'3183000', u'RelatedOrExemptFunctionIncome': ... \n", "72 [{u'BusinessCode': u'900099', u'TotalRevenueColumn': u'3452000', u'RelatedOrExemptFunctionIncome': u'3452000', u'Description': u'INITIATION FEES'}, {u'BusinessCode': u'900099', u'TotalRevenueColumn': u'3134807', u'RelatedOrExemptFunctionIncome': ... \n", "52 NaN \n", "406 NaN \n", "500 NaN \n", "0 NaN \n", "277 NaN \n", "226 NaN \n", "213 NaN \n", "495 NaN \n", "496 NaN \n", "\n", " ProgramServiceRevenueCY \\\n", "172 NaN \n", "404 NaN \n", "431 7520245 \n", "106 6926485 \n", "72 6586807 \n", "52 NaN \n", "406 NaN \n", "500 0 \n", "0 NaN \n", "277 NaN \n", "226 0 \n", "213 0 \n", "495 0 \n", "496 0 \n", "\n", " ProgramServiceRevenueGrp \\\n", "172 [{u'TotalRevenueColumnAmt': u'3242920', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'3242920', u'Desc': u'MEMBERSHIP DUES'}, {u'TotalRevenueColumnAmt': u'2890000', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'28900... \n", "404 [{u'TotalRevenueColumnAmt': u'4774000', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'4774000', u'Desc': u'INITIATION FEES'}, {u'TotalRevenueColumnAmt': u'3347499', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'33474... \n", "431 NaN \n", "106 NaN \n", "72 NaN \n", "52 [{u'TotalRevenueColumnAmt': u'1235180', u'RelatedOrExemptFuncIncomeAmt': u'1235180', u'Desc': u'SALES OF GRAVES & PLOTS'}, {u'TotalRevenueColumnAmt': u'601571', u'RelatedOrExemptFuncIncomeAmt': u'601571', u'Desc': u'INTERMENTS'}, {u'TotalRevenueC... \n", "406 [{u'TotalRevenueColumnAmt': u'43547', u'RelatedOrExemptFuncIncomeAmt': u'43547', u'Desc': u'CELL TOWER INCOME'}, {u'TotalRevenueColumnAmt': u'371441', u'RelatedOrExemptFuncIncomeAmt': u'371441', u'Desc': u'FOUNDATIONS'}, {u'TotalRevenueColumnAmt'... \n", "500 NaN \n", "0 NaN \n", "277 NaN \n", "226 NaN \n", "213 NaN \n", "495 NaN \n", "496 NaN \n", "\n", " TotalProgramServiceRevenue TotalProgramServiceRevenueAmt \\\n", "172 NaN 6812087 \n", "404 NaN 8834499 \n", "431 7520245 NaN \n", "106 6926485 NaN \n", "72 6586807 NaN \n", "52 NaN 2466379 \n", "406 NaN 2572305 \n", "500 NaN NaN \n", "0 NaN NaN \n", "277 NaN NaN \n", "226 NaN NaN \n", "213 NaN NaN \n", "495 NaN NaN \n", "496 NaN NaN \n", "\n", " CYInvestmentIncomeAmt \\\n", "172 1075687 \n", "404 391315 \n", "431 NaN \n", "106 NaN \n", "72 NaN \n", "52 1299578 \n", "406 917828 \n", "500 NaN \n", "0 0 \n", "277 701 \n", "226 NaN \n", "213 NaN \n", "495 NaN \n", "496 NaN \n", "\n", " InvestmentIncome \\\n", "172 NaN \n", "404 NaN \n", "431 {u'TotalRevenueColumn': u'234964', u'ExclusionAmount': u'234964'} \n", "106 {u'TotalRevenueColumn': u'237277', u'ExclusionAmount': u'237277'} \n", "72 {u'TotalRevenueColumn': u'217696', u'ExclusionAmount': u'217696'} \n", "52 NaN \n", "406 NaN \n", "500 NaN \n", "0 NaN \n", "277 NaN \n", "226 {u'TotalRevenueColumn': u'1710000'} \n", "213 NaN \n", "495 {u'TotalRevenueColumn': u'2482845'} \n", "496 NaN \n", "\n", " InvestmentIncomeCurrentYear \\\n", "172 NaN \n", "404 NaN \n", "431 211074 \n", "106 144021 \n", "72 192714 \n", "52 NaN \n", "406 NaN \n", "500 0 \n", "0 NaN \n", "277 NaN \n", "226 1710000 \n", "213 0 \n", "495 2482845 \n", "496 0 \n", "\n", " InvestmentIncomeGrp \\\n", "172 {u'TotalRevenueColumnAmt': u'334806', u'ExclusionAmt': u'334806'} \n", "404 {u'TotalRevenueColumnAmt': u'275100', u'ExclusionAmt': u'275100'} \n", "431 NaN \n", "106 NaN \n", "72 NaN \n", "52 {u'TotalRevenueColumnAmt': u'481123', u'RelatedOrExemptFuncIncomeAmt': u'481123'} \n", "406 {u'TotalRevenueColumnAmt': u'584869', u'RelatedOrExemptFuncIncomeAmt': u'584869'} \n", "500 NaN \n", "0 NaN \n", "277 {u'TotalRevenueColumnAmt': u'701', u'ExclusionAmt': u'701'} \n", "226 NaN \n", "213 NaN \n", "495 NaN \n", "496 NaN \n", "\n", " OtherRevenueTotalAmt \\\n", "172 25330 \n", "404 2581 \n", "431 NaN \n", "106 NaN \n", "72 NaN \n", "52 457869 \n", "406 1446045 \n", "500 NaN \n", "0 325000 \n", "277 NaN \n", "226 NaN \n", "213 NaN \n", "495 NaN \n", "496 NaN \n", "\n", " OtherRevenueMiscGrp \\\n", "172 {u'TotalRevenueColumnAmt': u'25330', u'BusinessCd': u'900099', u'ExclusionAmt': u'25330', u'Desc': u'MISCELLANEOUS'} \n", "404 {u'TotalRevenueColumnAmt': u'2581', u'BusinessCd': u'900099', u'ExclusionAmt': u'2581', u'Desc': u'MISCELLANEOUS'} \n", "431 NaN \n", "106 NaN \n", "72 NaN \n", "52 {u'TotalRevenueColumnAmt': u'457869', u'RelatedOrExemptFuncIncomeAmt': u'457869', u'Desc': u'PERPETUAL CARE ENDOWMENTS'} \n", "406 [{u'TotalRevenueColumnAmt': u'1000000', u'RelatedOrExemptFuncIncomeAmt': u'1000000', u'Desc': u'LEGAL RECOVERY SETTLEMENT'}, {u'TotalRevenueColumnAmt': u'446045', u'RelatedOrExemptFuncIncomeAmt': u'446045', u'Desc': u'PERPETUAL CARE ENDOWMENTS'}] \n", "500 NaN \n", "0 {u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000', u'Desc': u'SETTLEMENT'} \n", "277 NaN \n", "226 NaN \n", "213 NaN \n", "495 NaN \n", "496 NaN \n", "\n", " TotalOtherRevenue CYOtherRevenueAmt total_expenses program_expenses \n", "172 NaN 25330 7763846 NaN \n", "404 NaN 2581 7795021 NaN \n", "431 18387 NaN 6237509 NaN \n", "106 2448 NaN 5880789 NaN \n", "72 78685 NaN 6024086 NaN \n", "52 NaN 457869 2023750 NaN \n", "406 NaN 1446045 4133761 NaN \n", "500 NaN NaN 177901 NaN \n", "0 NaN 325000 34400 NaN \n", "277 NaN 0 3330 NaN \n", "226 NaN NaN 1690623 NaN \n", "213 5481 NaN 1161939 NaN \n", "495 5481 NaN 1282149 NaN \n", "496 NaN NaN 0 NaN " ] }, "execution_count": 667, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print len(df[df['program_expenses'].isnull()])\n", "df[df['program_expenses'].isnull()]" ] }, { "cell_type": "code", "execution_count": 336, "metadata": { "collapsed": true }, "outputs": [], "source": [ "df[df['program_expenses'].isnull()].to_csv('14 filings with zero program expenses.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Create efficiency variable" ] }, { "cell_type": "code", "execution_count": 731, "metadata": { "collapsed": true }, "outputs": [], "source": [ "df['program_expenses'] = df['program_expenses'].astype('float')\n", "df['total_expenses'] = df['total_expenses'].astype('float')" ] }, { "cell_type": "code", "execution_count": 732, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "count 524.000000\n", "mean 0.779227\n", "std 0.191070\n", "min 0.023817\n", "25% NaN\n", "50% NaN\n", "75% NaN\n", "max 1.000000\n", "Name: program_efficiency, dtype: float64\n" ] } ], "source": [ "df['program_efficiency'] = df['program_expenses']/df['total_expenses']\n", "print df['program_efficiency'].describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fix/Combine Complexity Variables" ] }, { "cell_type": "code", "execution_count": 668, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CYContributionsGrantsAmtFederatedCampaignsFederatedCampaignsAmtMembershipDuesMembershipDuesAmtNetIncomeFromFundraisingEventsNetIncmFromFundraisingEvtGrpFundraisingAmtFundraisingGrossIncomeAmtRelatedOrganizationsRelatedOrganizationsAmtGovernmentGrantsGovernmentGrantsAmtPYProgramServiceRevenueAmtCYProgramServiceRevenueAmtProgramServiceRevenueProgramServiceRevenueCYProgramServiceRevenueGrpTotalProgramServiceRevenueTotalProgramServiceRevenueAmtCYInvestmentIncomeAmtInvestmentIncomeInvestmentIncomeCurrentYearInvestmentIncomeGrpOtherRevenueTotalAmtOtherRevenueMiscGrpTotalOtherRevenueCYOtherRevenueAmt
252NaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaNNaNNaN185408NaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'63', u'ExclusionAmount': u'63'}63NaNNaNNaN0NaN
433NaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'0'}0NaNNaNNaN0NaN
78NaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'0'}0NaNNaNNaN0NaN
33320437991NaNNaNNaNNaNNaN{u'TotalRevenueColumnAmt': u'0'}NaNNaNNaN2053228NaN1612401600NaNNaNNaNNaN01539746NaNNaN{u'TotalRevenueColumnAmt': u'362651', u'ExclusionAmt': u'362651'}428192[{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'...NaN1017789
12NaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaN2466016NaN16660756NaNNaNNaNNaN0NaN0NaNNaN{u'TotalRevenueColumn': u'398381', u'ExclusionAmount': u'398381'}1195606NaNNaNNaN451157NaN
\n", "
" ], "text/plain": [ " CYContributionsGrantsAmt FederatedCampaigns FederatedCampaignsAmt \\\n", "252 NaN NaN NaN \n", "433 NaN NaN NaN \n", "78 NaN NaN NaN \n", "333 20437991 NaN NaN \n", "12 NaN NaN NaN \n", "\n", " MembershipDues MembershipDuesAmt NetIncomeFromFundraisingEvents \\\n", "252 NaN NaN {u'TotalRevenueColumn': u'0'} \n", "433 NaN NaN {u'TotalRevenueColumn': u'0'} \n", "78 NaN NaN {u'TotalRevenueColumn': u'0'} \n", "333 NaN NaN NaN \n", "12 NaN NaN {u'TotalRevenueColumn': u'0'} \n", "\n", " NetIncmFromFundraisingEvtGrp FundraisingAmt \\\n", "252 NaN NaN \n", "433 NaN NaN \n", "78 NaN NaN \n", "333 {u'TotalRevenueColumnAmt': u'0'} NaN \n", "12 NaN NaN \n", "\n", " FundraisingGrossIncomeAmt RelatedOrganizations RelatedOrganizationsAmt \\\n", "252 NaN NaN NaN \n", "433 NaN NaN NaN \n", "78 NaN NaN NaN \n", "333 NaN NaN 2053228 \n", "12 NaN 2466016 NaN \n", "\n", " GovernmentGrants GovernmentGrantsAmt PYProgramServiceRevenueAmt \\\n", "252 185408 NaN NaN \n", "433 NaN NaN NaN \n", "78 NaN NaN NaN \n", "333 NaN 16124016 0 \n", "12 16660756 NaN NaN \n", "\n", " CYProgramServiceRevenueAmt ProgramServiceRevenue ProgramServiceRevenueCY \\\n", "252 NaN NaN 0 \n", "433 NaN NaN 0 \n", "78 NaN NaN 0 \n", "333 0 NaN NaN \n", "12 NaN NaN 0 \n", "\n", " ProgramServiceRevenueGrp TotalProgramServiceRevenue \\\n", "252 NaN 0 \n", "433 NaN 0 \n", "78 NaN 0 \n", "333 NaN NaN \n", "12 NaN 0 \n", "\n", " TotalProgramServiceRevenueAmt CYInvestmentIncomeAmt \\\n", "252 NaN NaN \n", "433 NaN NaN \n", "78 NaN NaN \n", "333 0 1539746 \n", "12 NaN NaN \n", "\n", " InvestmentIncome \\\n", "252 {u'TotalRevenueColumn': u'63', u'ExclusionAmount': u'63'} \n", "433 {u'TotalRevenueColumn': u'0'} \n", "78 {u'TotalRevenueColumn': u'0'} \n", "333 NaN \n", "12 {u'TotalRevenueColumn': u'398381', u'ExclusionAmount': u'398381'} \n", "\n", " InvestmentIncomeCurrentYear \\\n", "252 63 \n", "433 0 \n", "78 0 \n", "333 NaN \n", "12 1195606 \n", "\n", " InvestmentIncomeGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 {u'TotalRevenueColumnAmt': u'362651', u'ExclusionAmt': u'362651'} \n", "12 NaN \n", "\n", " OtherRevenueTotalAmt \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 428192 \n", "12 NaN \n", "\n", " OtherRevenueMiscGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 [{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'... \n", "12 NaN \n", "\n", " TotalOtherRevenue CYOtherRevenueAmt \n", "252 0 NaN \n", "433 0 NaN \n", "78 0 NaN \n", "333 NaN 1017789 \n", "12 451157 NaN " ] }, "execution_count": 668, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[complexity_columns][:5]" ] }, { "cell_type": "code", "execution_count": 669, "metadata": { "collapsed": true }, "outputs": [], "source": [ "complexity_columns = ['CYContributionsGrantsAmt', 'FederatedCampaigns', 'FederatedCampaignsAmt', \n", " 'MembershipDues', 'MembershipDuesAmt',\n", " 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp',\n", " #'FundraisingActivities', 'FundraisingActivitiesInd', \n", " 'FundraisingAmt', \n", " #'FundraisingDirectExpenses', 'FundraisingDirectExpensesAmt', 'FundraisingEvents', \n", " 'FundraisingGrossIncomeAmt', \n", " 'RelatedOrganizations', 'RelatedOrganizationsAmt',\n", " 'GovernmentGrants', 'GovernmentGrantsAmt',\n", " 'PYProgramServiceRevenueAmt', 'CYProgramServiceRevenueAmt',\n", " 'ProgramServiceRevenue', 'ProgramServiceRevenueCY', 'ProgramServiceRevenueGrp',\n", " 'TotalProgramServiceRevenue', 'TotalProgramServiceRevenueAmt', \n", " 'CYInvestmentIncomeAmt', 'InvestmentIncome', 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', \n", " 'OtherRevenueTotalAmt', 'OtherRevenueMiscGrp',\n", " 'TotalOtherRevenue', 'CYOtherRevenueAmt'\n", " ]\n", " \n", "#[contributions_gifts_grants_2016', 'federated_campaigns_2016', 'membership_dues_2016', \n", "# 'fundraising_events_2016', 'related_organizations_2016', 'government_grants_2016', \n", "# 'program_service_revenue_2016', 'other_revenue_2016']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Federated Campaigns" ] }, { "cell_type": "code", "execution_count": 670, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "21\n", "40\n" ] }, { "data": { "text/plain": [ "count 40\n", "unique 34\n", "top 0\n", "freq 6\n", "Name: FederatedCampaigns, dtype: object" ] }, "execution_count": 670, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print len(df[df['FederatedCampaigns'].notnull()])\n", "df['FederatedCampaigns'] = np.where( ((df['FederatedCampaigns'].isnull()) & (df['FederatedCampaignsAmt'].notnull())), \n", " df['FederatedCampaignsAmt'], df['FederatedCampaigns'] )\n", "print len(df[df['FederatedCampaigns'].notnull()])\n", "df['FederatedCampaigns'].describe().T" ] }, { "cell_type": "code", "execution_count": 671, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CYContributionsGrantsAmtFederatedCampaignsFederatedCampaignsAmtMembershipDuesMembershipDuesAmtNetIncomeFromFundraisingEventsNetIncmFromFundraisingEvtGrpFundraisingAmtFundraisingGrossIncomeAmtRelatedOrganizationsRelatedOrganizationsAmtGovernmentGrantsGovernmentGrantsAmtPYProgramServiceRevenueAmtCYProgramServiceRevenueAmtProgramServiceRevenueProgramServiceRevenueCYProgramServiceRevenueGrpTotalProgramServiceRevenueTotalProgramServiceRevenueAmtCYInvestmentIncomeAmtInvestmentIncomeInvestmentIncomeCurrentYearInvestmentIncomeGrpOtherRevenueTotalAmtOtherRevenueMiscGrpTotalOtherRevenueCYOtherRevenueAmt
101NaNNaNNaNNaNNaN{u'TotalRevenueColumn': u'0'}NaNNaNNaNNaNNaNNaNNaNNaNNaN{u'BusinessCode': u'611600', u'TotalRevenueColumn': u'194006', u'RelatedOrExemptFunctionIncome': u'194006', u'Description': u'TUITION'}194006NaN194006NaNNaN{u'TotalRevenueColumn': u'7', u'RelatedOrExemptFunctionIncome': u'7'}7NaNNaNNaN0NaN
52812294982NaNNaNNaNNaNNaNNaN1817567NaNNaNNaNNaN646173NaN0NaNNaNNaNNaNNaN1549NaNNaN{u'TotalRevenueColumnAmt': u'1549'}NaNNaNNaN0
40112741996NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN629939NaN0NaNNaNNaNNaNNaN1656NaNNaN{u'TotalRevenueColumnAmt': u'1656'}NaNNaNNaN0
361NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN437339NaNNaNNaNNaN0NaNNaNNaNNaN{u'TotalRevenueColumn': u'2695'}2695NaNNaNNaNNaNNaN
36NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN461226NaNNaNNaNNaN0NaNNaNNaNNaN{u'TotalRevenueColumn': u'2317'}2317NaNNaNNaNNaNNaN
\n", "
" ], "text/plain": [ " CYContributionsGrantsAmt FederatedCampaigns FederatedCampaignsAmt \\\n", "101 NaN NaN NaN \n", "528 12294982 NaN NaN \n", "401 12741996 NaN NaN \n", "361 NaN NaN NaN \n", "36 NaN NaN NaN \n", "\n", " MembershipDues MembershipDuesAmt NetIncomeFromFundraisingEvents \\\n", "101 NaN NaN {u'TotalRevenueColumn': u'0'} \n", "528 NaN NaN NaN \n", "401 NaN NaN NaN \n", "361 NaN NaN NaN \n", "36 NaN NaN NaN \n", "\n", " NetIncmFromFundraisingEvtGrp FundraisingAmt FundraisingGrossIncomeAmt \\\n", "101 NaN NaN NaN \n", "528 NaN 1817567 NaN \n", "401 NaN NaN NaN \n", "361 NaN NaN NaN \n", "36 NaN NaN NaN \n", "\n", " RelatedOrganizations RelatedOrganizationsAmt GovernmentGrants \\\n", "101 NaN NaN NaN \n", "528 NaN NaN NaN \n", "401 NaN NaN NaN \n", "361 NaN NaN 437339 \n", "36 NaN NaN 461226 \n", "\n", " GovernmentGrantsAmt PYProgramServiceRevenueAmt CYProgramServiceRevenueAmt \\\n", "101 NaN NaN NaN \n", "528 646173 NaN 0 \n", "401 629939 NaN 0 \n", "361 NaN NaN NaN \n", "36 NaN NaN NaN \n", "\n", " ProgramServiceRevenue \\\n", "101 {u'BusinessCode': u'611600', u'TotalRevenueColumn': u'194006', u'RelatedOrExemptFunctionIncome': u'194006', u'Description': u'TUITION'} \n", "528 NaN \n", "401 NaN \n", "361 NaN \n", "36 NaN \n", "\n", " ProgramServiceRevenueCY ProgramServiceRevenueGrp \\\n", "101 194006 NaN \n", "528 NaN NaN \n", "401 NaN NaN \n", "361 0 NaN \n", "36 0 NaN \n", "\n", " TotalProgramServiceRevenue TotalProgramServiceRevenueAmt \\\n", "101 194006 NaN \n", "528 NaN NaN \n", "401 NaN NaN \n", "361 NaN NaN \n", "36 NaN NaN \n", "\n", " CYInvestmentIncomeAmt \\\n", "101 NaN \n", "528 1549 \n", "401 1656 \n", "361 NaN \n", "36 NaN \n", "\n", " InvestmentIncome \\\n", "101 {u'TotalRevenueColumn': u'7', u'RelatedOrExemptFunctionIncome': u'7'} \n", "528 NaN \n", "401 NaN \n", "361 {u'TotalRevenueColumn': u'2695'} \n", "36 {u'TotalRevenueColumn': u'2317'} \n", "\n", " InvestmentIncomeCurrentYear InvestmentIncomeGrp \\\n", "101 7 NaN \n", "528 NaN {u'TotalRevenueColumnAmt': u'1549'} \n", "401 NaN {u'TotalRevenueColumnAmt': u'1656'} \n", "361 2695 NaN \n", "36 2317 NaN \n", "\n", " OtherRevenueTotalAmt OtherRevenueMiscGrp TotalOtherRevenue \\\n", "101 NaN NaN 0 \n", "528 NaN NaN NaN \n", "401 NaN NaN NaN \n", "361 NaN NaN NaN \n", "36 NaN NaN NaN \n", "\n", " CYOtherRevenueAmt \n", "101 NaN \n", "528 0 \n", "401 0 \n", "361 NaN \n", "36 NaN " ] }, "execution_count": 671, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[complexity_columns][20:25]" ] }, { "cell_type": "code", "execution_count": 672, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "32\n", "12\n", "44\n" ] }, { "data": { "text/plain": [ "count 44\n", "unique 37\n", "top 0\n", "freq 7\n", "Name: MembershipDues, dtype: object" ] }, "execution_count": 672, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print len(df[df['MembershipDues'].notnull()])\n", "print len(df[df['MembershipDuesAmt'].notnull()])\n", "df['MembershipDues'] = np.where( ((df['MembershipDues'].isnull()) & (df['MembershipDuesAmt'].notnull())), \n", " df['MembershipDuesAmt'], df['MembershipDues'] )\n", "print len(df[df['MembershipDues'].notnull()])\n", "df['MembershipDues'].describe().T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Fundraising " ] }, { "cell_type": "code", "execution_count": 673, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['CYContributionsGrantsAmt', 'FederatedCampaigns', 'FederatedCampaignsAmt', 'MembershipDues', 'MembershipDuesAmt', 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp', 'FundraisingAmt', 'FundraisingGrossIncomeAmt', 'RelatedOrganizations', 'RelatedOrganizationsAmt', 'GovernmentGrants', 'GovernmentGrantsAmt', 'PYProgramServiceRevenueAmt', 'CYProgramServiceRevenueAmt', 'ProgramServiceRevenue', 'ProgramServiceRevenueCY', 'ProgramServiceRevenueGrp', 'TotalProgramServiceRevenue', 'TotalProgramServiceRevenueAmt', 'CYInvestmentIncomeAmt', 'InvestmentIncome', 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', 'OtherRevenueTotalAmt', 'OtherRevenueMiscGrp', 'TotalOtherRevenue', 'CYOtherRevenueAmt']\n" ] } ], "source": [ "print complexity_columns" ] }, { "cell_type": "code", "execution_count": 674, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import ast" ] }, { "cell_type": "code", "execution_count": 675, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NetIncomeFromFundraisingEventsNetIncmFromFundraisingEvtGrpFundraisingAmtFundraisingGrossIncomeAmt
12{u'TotalRevenueColumn': u'0'}NaNNaNNaN
481{u'TotalRevenueColumn': u'221449', u'ExclusionAmount': u'221449'}NaNNaNNaN
16{u'TotalRevenueColumn': u'23265', u'ExclusionAmount': u'23265'}NaNNaNNaN
130{u'TotalRevenueColumn': u'1322', u'RelatedOrExemptFunctionIncome': u'1322', u'UnrelatedBusinessRevenue': u'0', u'ExclusionAmount': u'0'}NaNNaNNaN
434NaN{u'TotalRevenueColumnAmt': u'9429', u'ExclusionAmt': u'9429'}298528106
374{u'TotalRevenueColumn': u'2923', u'ExclusionAmount': u'2923'}NaNNaNNaN
118{u'TotalRevenueColumn': u'47210', u'ExclusionAmount': u'47210'}NaNNaNNaN
124{u'TotalRevenueColumn': u'99073', u'RelatedOrExemptFunctionIncome': u'99073'}NaNNaNNaN
\n", "
" ], "text/plain": [ " NetIncomeFromFundraisingEvents \\\n", "12 {u'TotalRevenueColumn': u'0'} \n", "481 {u'TotalRevenueColumn': u'221449', u'ExclusionAmount': u'221449'} \n", "16 {u'TotalRevenueColumn': u'23265', u'ExclusionAmount': u'23265'} \n", "130 {u'TotalRevenueColumn': u'1322', u'RelatedOrExemptFunctionIncome': u'1322', u'UnrelatedBusinessRevenue': u'0', u'ExclusionAmount': u'0'} \n", "434 NaN \n", "374 {u'TotalRevenueColumn': u'2923', u'ExclusionAmount': u'2923'} \n", "118 {u'TotalRevenueColumn': u'47210', u'ExclusionAmount': u'47210'} \n", "124 {u'TotalRevenueColumn': u'99073', u'RelatedOrExemptFunctionIncome': u'99073'} \n", "\n", " NetIncmFromFundraisingEvtGrp \\\n", "12 NaN \n", "481 NaN \n", "16 NaN \n", "130 NaN \n", "434 {u'TotalRevenueColumnAmt': u'9429', u'ExclusionAmt': u'9429'} \n", "374 NaN \n", "118 NaN \n", "124 NaN \n", "\n", " FundraisingAmt FundraisingGrossIncomeAmt \n", "12 NaN NaN \n", "481 NaN NaN \n", "16 NaN NaN \n", "130 NaN NaN \n", "434 2985 28106 \n", "374 NaN NaN \n", "118 NaN NaN \n", "124 NaN NaN " ] }, "execution_count": 675, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[['NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp', 'FundraisingAmt', \n", " 'FundraisingGrossIncomeAmt']][4:12]" ] }, { "cell_type": "code", "execution_count": 676, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "144\n", "84\n", "228\n" ] }, { "data": { "text/plain": [ "count 228\n", "unique 129\n", "top {u'TotalRevenueColumn': u'0'}\n", "freq 66\n", "Name: NetIncomeFromFundraisingEvents, dtype: object" ] }, "execution_count": 676, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print len(df[df['NetIncomeFromFundraisingEvents'].notnull()])\n", "print len(df[df['NetIncmFromFundraisingEvtGrp'].notnull()])\n", "df['NetIncomeFromFundraisingEvents'] = np.where( ((df['NetIncomeFromFundraisingEvents'].isnull()) \n", " & (df['NetIncmFromFundraisingEvtGrp'].notnull())), \n", " df['NetIncmFromFundraisingEvtGrp'], df['NetIncomeFromFundraisingEvents'] )\n", "print len(df[df['NetIncomeFromFundraisingEvents'].notnull()])\n", "df['NetIncomeFromFundraisingEvents'].describe().T" ] }, { "cell_type": "code", "execution_count": 677, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NetIncomeFromFundraisingEventsNetIncmFromFundraisingEvtGrpFundraisingAmtFundraisingGrossIncomeAmt
12{u'TotalRevenueColumn': u'0'}NaNNaNNaN
481{u'TotalRevenueColumn': u'221449', u'ExclusionAmount': u'221449'}NaNNaNNaN
16{u'TotalRevenueColumn': u'23265', u'ExclusionAmount': u'23265'}NaNNaNNaN
130{u'TotalRevenueColumn': u'1322', u'RelatedOrExemptFunctionIncome': u'1322', u'UnrelatedBusinessRevenue': u'0', u'ExclusionAmount': u'0'}NaNNaNNaN
434{u'TotalRevenueColumnAmt': u'9429', u'ExclusionAmt': u'9429'}{u'TotalRevenueColumnAmt': u'9429', u'ExclusionAmt': u'9429'}298528106
374{u'TotalRevenueColumn': u'2923', u'ExclusionAmount': u'2923'}NaNNaNNaN
118{u'TotalRevenueColumn': u'47210', u'ExclusionAmount': u'47210'}NaNNaNNaN
124{u'TotalRevenueColumn': u'99073', u'RelatedOrExemptFunctionIncome': u'99073'}NaNNaNNaN
\n", "
" ], "text/plain": [ " NetIncomeFromFundraisingEvents \\\n", "12 {u'TotalRevenueColumn': u'0'} \n", "481 {u'TotalRevenueColumn': u'221449', u'ExclusionAmount': u'221449'} \n", "16 {u'TotalRevenueColumn': u'23265', u'ExclusionAmount': u'23265'} \n", "130 {u'TotalRevenueColumn': u'1322', u'RelatedOrExemptFunctionIncome': u'1322', u'UnrelatedBusinessRevenue': u'0', u'ExclusionAmount': u'0'} \n", "434 {u'TotalRevenueColumnAmt': u'9429', u'ExclusionAmt': u'9429'} \n", "374 {u'TotalRevenueColumn': u'2923', u'ExclusionAmount': u'2923'} \n", "118 {u'TotalRevenueColumn': u'47210', u'ExclusionAmount': u'47210'} \n", "124 {u'TotalRevenueColumn': u'99073', u'RelatedOrExemptFunctionIncome': u'99073'} \n", "\n", " NetIncmFromFundraisingEvtGrp \\\n", "12 NaN \n", "481 NaN \n", "16 NaN \n", "130 NaN \n", "434 {u'TotalRevenueColumnAmt': u'9429', u'ExclusionAmt': u'9429'} \n", "374 NaN \n", "118 NaN \n", "124 NaN \n", "\n", " FundraisingAmt FundraisingGrossIncomeAmt \n", "12 NaN NaN \n", "481 NaN NaN \n", "16 NaN NaN \n", "130 NaN NaN \n", "434 2985 28106 \n", "374 NaN NaN \n", "118 NaN NaN \n", "124 NaN NaN " ] }, "execution_count": 677, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[['NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp', 'FundraisingAmt', \n", " 'FundraisingGrossIncomeAmt']][4:12]" ] }, { "cell_type": "code", "execution_count": 678, "metadata": { "collapsed": false }, "outputs": [], "source": [ "counter = 1\n", "df['fundraising'] = np.nan\n", "for index, row in df.iterrows():\n", " #print row['NetIncomeFromFundraisingEvents']\n", " #print type(ast.literal_eval(str(row['NetIncomeFromFundraisingEvents'])))\n", " if pd.notnull(row['NetIncomeFromFundraisingEvents']):\n", " \n", " try:\n", " fr_revs = ast.literal_eval(str(row['NetIncomeFromFundraisingEvents']))\n", " except:\n", " df.ix[index, 'fundraising'] = np.nan \n", " if 'TotalRevenueColumn' in fr_revs:\n", " #print 'yes', fr_revs['TotalRevenueColumn']\n", " df.ix[index, 'fundraising'] = fr_revs['TotalRevenueColumn']\n", " elif 'TotalRevenueColumnAmt' in fr_revs:\n", " #print 'yes', fr_revs['TotalRevenueColumnAmt']\n", " df.ix[index, 'fundraising'] = fr_revs['TotalRevenueColumnAmt']\n", " \n", " else:\n", " df.ix[index, 'fundraising'] = np.nan\n", " #counter +=1\n", " #print counter" ] }, { "cell_type": "code", "execution_count": 453, "metadata": { "collapsed": false }, "outputs": [], "source": [ "'''\n", "counter = 1\n", "df['fundraising'] = np.nan\n", "for index, row in df.iterrows():\n", " #print row['NetIncomeFromFundraisingEvents']\n", " #print type(ast.literal_eval(str(row['NetIncomeFromFundraisingEvents'])))\n", " if pd.notnull(row['NetIncomeFromFundraisingEvents']):\n", " \n", " try:\n", " fr_revs = ast.literal_eval(str(row['NetIncomeFromFundraisingEvents']))\n", " except:\n", " df[index, 'fundraising'] = np.nan\n", " if 'TotalRevenueColumn' in fr_revs:\n", " #print 'yes', fr_revs['TotalRevenueColumn']\n", " df.ix[index, 'fundraising'] = fr_revs['TotalRevenueColumn']\n", " elif 'TotalRevenueColumnAmt' in fr_revs:\n", " #print 'yes', fr_revs['TotalRevenueColumnAmt']\n", " df.ix[index, 'fundraising'] = fr_revs['TotalRevenueColumnAmt']\n", " else:\n", " df[index, 'fundraising'] = np.nan\n", " #counter +=1\n", " #print counter\n", "'''" ] }, { "cell_type": "code", "execution_count": 679, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
fundraisingNetIncomeFromFundraisingEventsNetIncmFromFundraisingEvtGrpFundraisingAmtFundraisingGrossIncomeAmt
120{u'TotalRevenueColumn': u'0'}NaNNaNNaN
481221449{u'TotalRevenueColumn': u'221449', u'ExclusionAmount': u'221449'}NaNNaNNaN
1623265{u'TotalRevenueColumn': u'23265', u'ExclusionAmount': u'23265'}NaNNaNNaN
1301322{u'TotalRevenueColumn': u'1322', u'RelatedOrExemptFunctionIncome': u'1322', u'UnrelatedBusinessRevenue': u'0', u'ExclusionAmount': u'0'}NaNNaNNaN
\n", "
" ], "text/plain": [ " fundraising \\\n", "12 0 \n", "481 221449 \n", "16 23265 \n", "130 1322 \n", "\n", " NetIncomeFromFundraisingEvents \\\n", "12 {u'TotalRevenueColumn': u'0'} \n", "481 {u'TotalRevenueColumn': u'221449', u'ExclusionAmount': u'221449'} \n", "16 {u'TotalRevenueColumn': u'23265', u'ExclusionAmount': u'23265'} \n", "130 {u'TotalRevenueColumn': u'1322', u'RelatedOrExemptFunctionIncome': u'1322', u'UnrelatedBusinessRevenue': u'0', u'ExclusionAmount': u'0'} \n", "\n", " NetIncmFromFundraisingEvtGrp FundraisingAmt FundraisingGrossIncomeAmt \n", "12 NaN NaN NaN \n", "481 NaN NaN NaN \n", "16 NaN NaN NaN \n", "130 NaN NaN NaN " ] }, "execution_count": 679, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[['fundraising', 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp', 'FundraisingAmt', \n", " 'FundraisingGrossIncomeAmt']][4:8]" ] }, { "cell_type": "code", "execution_count": 680, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
fundraisingFundraisingAmtFundraisingGrossIncomeAmt
2520NaNNaN
4330NaNNaN
780NaNNaN
3330NaNNaN
120NaNNaN
481221449NaNNaN
1623265NaNNaN
1301322NaNNaN
4349429298528106
3742923NaNNaN
11847210NaNNaN
12499073NaNNaN
\n", "
" ], "text/plain": [ " fundraising FundraisingAmt FundraisingGrossIncomeAmt\n", "252 0 NaN NaN\n", "433 0 NaN NaN\n", "78 0 NaN NaN\n", "333 0 NaN NaN\n", "12 0 NaN NaN\n", "481 221449 NaN NaN\n", "16 23265 NaN NaN\n", "130 1322 NaN NaN\n", "434 9429 2985 28106\n", "374 2923 NaN NaN\n", "118 47210 NaN NaN\n", "124 99073 NaN NaN" ] }, "execution_count": 680, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[['fundraising', 'FundraisingAmt', 'FundraisingGrossIncomeAmt']][:12]" ] }, { "cell_type": "code", "execution_count": 682, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "228\n", "32\n", "53\n", "228\n" ] }, { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
fundraisingFundraisingAmtFundraisingGrossIncomeAmt
2520NaNNaN
4330NaNNaN
780NaNNaN
3330NaNNaN
120NaNNaN
481221449NaNNaN
1623265NaNNaN
1301322NaNNaN
4349429298528106
3742923NaNNaN
11847210NaNNaN
12499073NaNNaN
\n", "
" ], "text/plain": [ " fundraising FundraisingAmt FundraisingGrossIncomeAmt\n", "252 0 NaN NaN\n", "433 0 NaN NaN\n", "78 0 NaN NaN\n", "333 0 NaN NaN\n", "12 0 NaN NaN\n", "481 221449 NaN NaN\n", "16 23265 NaN NaN\n", "130 1322 NaN NaN\n", "434 9429 2985 28106\n", "374 2923 NaN NaN\n", "118 47210 NaN NaN\n", "124 99073 NaN NaN" ] }, "execution_count": 682, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print len(df[df['fundraising'].notnull()])\n", "print len(df[df['FundraisingAmt'].notnull()])\n", "print len(df[df['FundraisingGrossIncomeAmt'].notnull()])\n", "df['fundraising'] = np.where( ((df['fundraising']=='0') & (df['FundraisingAmt'].notnull())), \n", " df['FundraisingAmt'], df['fundraising'] )\n", "df['fundraising'] = np.where( ((df['fundraising']=='0') & (df['FundraisingGrossIncomeAmt'].notnull())), \n", " df['FundraisingGrossIncomeAmt'], df['fundraising'] )\n", "print len(df[df['fundraising'].notnull()])\n", "df[['fundraising', 'FundraisingAmt', 'FundraisingGrossIncomeAmt']][:12]" ] }, { "cell_type": "code", "execution_count": 683, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "count 228\n", "unique 132\n", "top 0\n", "freq 96\n", "Name: fundraising, dtype: object" ] }, "execution_count": 683, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['fundraising'].describe().T" ] }, { "cell_type": "code", "execution_count": 469, "metadata": { "collapsed": false }, "outputs": [], "source": [ "#df[df['fundraising']=='-38198'][['EIN', 'FYE', 'fundraising', 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp', \n", "# 'FundraisingAmt', 'FundraisingGrossIncomeAmt']]" ] }, { "cell_type": "code", "execution_count": 684, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINFYEfundraisingNetIncomeFromFundraisingEventsNetIncmFromFundraisingEvtGrpFundraisingAmtFundraisingGrossIncomeAmt
106132644507FY2011NaNNaNNaNNaNNaN
72132644507FY2010NaNNaNNaNNaNNaN
336132690403FY2013NaNNaNNaNNaNNaN
430132690403FY2012NaNNaNNaNNaNNaN
473132690403FY2011NaNNaNNaNNaNNaN
33132690403FY2010NaNNaNNaNNaNNaN
531132738818FY2014-18373{u'TotalRevenueColumnAmt': u'-18373', u'ExclusionAmt': u'-18373'}{u'TotalRevenueColumnAmt': u'-18373', u'ExclusionAmt': u'-18373'}21743432760
162132738818FY20130{u'TotalRevenueColumn': u'0'}NaNNaNNaN
22132738818FY20120{u'TotalRevenueColumn': u'0'}NaNNaNNaN
82132738818FY2011NaNNaNNaNNaNNaN
140132738818FY20100{u'TotalRevenueColumn': u'0'}NaNNaNNaN
442133119118FY201221748{u'TotalRevenueColumn': u'21748', u'ExclusionAmount': u'21748'}NaNNaNNaN
491133119118FY2011176541{u'TotalRevenueColumn': u'176541', u'ExclusionAmount': u'176541'}NaNNaNNaN
499133119118FY2010236763{u'TotalRevenueColumn': u'236763', u'ExclusionAmount': u'236763'}NaNNaNNaN
70133552154FY2011853901{u'TotalRevenueColumn': u'853901', u'ExclusionAmount': u'853901'}NaNNaNNaN
123133552154FY2010418298{u'TotalRevenueColumn': u'418298', u'ExclusionAmount': u'418298'}NaNNaNNaN
296133600807FY2014NaNNaNNaNNaNNaN
407133600807FY2013NaNNaNNaNNaNNaN
428133600807FY2012NaNNaNNaNNaNNaN
180133600807FY2010NaNNaNNaNNaNNaN
323134147912FY2014NaNNaNNaNNaNNaN
384134147912FY2013NaNNaNNaNNaNNaN
21134147912FY20120{u'TotalRevenueColumn': u'0'}NaNNaNNaN
246134147912FY20110{u'TotalRevenueColumn': u'0'}NaNNaNNaN
\n", "
" ], "text/plain": [ " EIN FYE fundraising \\\n", "106 132644507 FY2011 NaN \n", "72 132644507 FY2010 NaN \n", "336 132690403 FY2013 NaN \n", "430 132690403 FY2012 NaN \n", "473 132690403 FY2011 NaN \n", "33 132690403 FY2010 NaN \n", "531 132738818 FY2014 -18373 \n", "162 132738818 FY2013 0 \n", "22 132738818 FY2012 0 \n", "82 132738818 FY2011 NaN \n", "140 132738818 FY2010 0 \n", "442 133119118 FY2012 21748 \n", "491 133119118 FY2011 176541 \n", "499 133119118 FY2010 236763 \n", "70 133552154 FY2011 853901 \n", "123 133552154 FY2010 418298 \n", "296 133600807 FY2014 NaN \n", "407 133600807 FY2013 NaN \n", "428 133600807 FY2012 NaN \n", "180 133600807 FY2010 NaN \n", "323 134147912 FY2014 NaN \n", "384 134147912 FY2013 NaN \n", "21 134147912 FY2012 0 \n", "246 134147912 FY2011 0 \n", "\n", " NetIncomeFromFundraisingEvents \\\n", "106 NaN \n", "72 NaN \n", "336 NaN \n", "430 NaN \n", "473 NaN \n", "33 NaN \n", "531 {u'TotalRevenueColumnAmt': u'-18373', u'ExclusionAmt': u'-18373'} \n", "162 {u'TotalRevenueColumn': u'0'} \n", "22 {u'TotalRevenueColumn': u'0'} \n", "82 NaN \n", "140 {u'TotalRevenueColumn': u'0'} \n", "442 {u'TotalRevenueColumn': u'21748', u'ExclusionAmount': u'21748'} \n", "491 {u'TotalRevenueColumn': u'176541', u'ExclusionAmount': u'176541'} \n", "499 {u'TotalRevenueColumn': u'236763', u'ExclusionAmount': u'236763'} \n", "70 {u'TotalRevenueColumn': u'853901', u'ExclusionAmount': u'853901'} \n", "123 {u'TotalRevenueColumn': u'418298', u'ExclusionAmount': u'418298'} \n", "296 NaN \n", "407 NaN \n", "428 NaN \n", "180 NaN \n", "323 NaN \n", "384 NaN \n", "21 {u'TotalRevenueColumn': u'0'} \n", "246 {u'TotalRevenueColumn': u'0'} \n", "\n", " NetIncmFromFundraisingEvtGrp \\\n", "106 NaN \n", "72 NaN \n", "336 NaN \n", "430 NaN \n", "473 NaN \n", "33 NaN \n", "531 {u'TotalRevenueColumnAmt': u'-18373', u'ExclusionAmt': u'-18373'} \n", "162 NaN \n", "22 NaN \n", "82 NaN \n", "140 NaN \n", "442 NaN \n", "491 NaN \n", "499 NaN \n", "70 NaN \n", "123 NaN \n", "296 NaN \n", "407 NaN \n", "428 NaN \n", "180 NaN \n", "323 NaN \n", "384 NaN \n", "21 NaN \n", "246 NaN \n", "\n", " FundraisingAmt FundraisingGrossIncomeAmt \n", "106 NaN NaN \n", "72 NaN NaN \n", "336 NaN NaN \n", "430 NaN NaN \n", "473 NaN NaN \n", "33 NaN NaN \n", "531 217434 32760 \n", "162 NaN NaN \n", "22 NaN NaN \n", "82 NaN NaN \n", "140 NaN NaN \n", "442 NaN NaN \n", "491 NaN NaN \n", "499 NaN NaN \n", "70 NaN NaN \n", "123 NaN NaN \n", "296 NaN NaN \n", "407 NaN NaN \n", "428 NaN NaN \n", "180 NaN NaN \n", "323 NaN NaN \n", "384 NaN NaN \n", "21 NaN NaN \n", "246 NaN NaN " ] }, "execution_count": 684, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[['EIN', 'FYE', 'fundraising', 'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp', \n", " 'FundraisingAmt', 'FundraisingGrossIncomeAmt']][50:74]" ] }, { "cell_type": "code", "execution_count": 685, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "float64 \n", "\n", "count 2.280000e+02\n", "mean 8.138974e+04\n", "std 5.064406e+05\n", "min -2.839000e+05\n", "25% NaN\n", "50% NaN\n", "75% NaN\n", "max 7.283318e+06\n", "Name: fundraising, dtype: float64\n" ] } ], "source": [ "df['fundraising'] = df['fundraising'].astype('float')\n", "print df['fundraising'].dtype, '\\n'\n", "print df['fundraising'].describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Related Orgs" ] }, { "cell_type": "code", "execution_count": 687, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "33\n", "19\n", "52\n" ] }, { "data": { "text/plain": [ "count 52\n", "unique 46\n", "top 0\n", "freq 7\n", "Name: RelatedOrganizations, dtype: object" ] }, "execution_count": 687, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print len(df[df['RelatedOrganizations'].notnull()])\n", "print len(df[df['RelatedOrganizationsAmt'].notnull()])\n", "df['RelatedOrganizations'] = np.where( ((df['RelatedOrganizations'].isnull()) \n", " & (df['RelatedOrganizationsAmt'].notnull())), \n", " df['RelatedOrganizationsAmt'], df['RelatedOrganizations'] )\n", "print len(df[df['RelatedOrganizations'].notnull()])\n", "df['RelatedOrganizations'].describe().T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Government Grants" ] }, { "cell_type": "code", "execution_count": 690, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "213\n", "70\n", "213\n" ] }, { "data": { "text/plain": [ "count 213\n", "unique 207\n", "top 548987\n", "freq 3\n", "Name: GovernmentGrants, dtype: object" ] }, "execution_count": 690, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print len(df[df['GovernmentGrants'].notnull()])\n", "print len(df[df['GovernmentGrantsAmt'].notnull()])\n", "df['GovernmentGrants'] = np.where( ((df['GovernmentGrants'].isnull()) & (df['GovernmentGrantsAmt'].notnull())), \n", " df['GovernmentGrantsAmt'], df['GovernmentGrants'] )\n", "print len(df[df['GovernmentGrants'].notnull()])\n", "df['GovernmentGrants'].describe().T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Program Service Revenue" ] }, { "cell_type": "code", "execution_count": 691, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "151\n", "194\n", "344\n", "233\n", "130 \n", "\n", "0\n", "151\n", "194\n", "538\n", "538\n", "538\n" ] }, { "data": { "text/plain": [ "count 538\n", "unique 259\n", "top 0\n", "freq 217\n", "Name: program_revenue, dtype: object" ] }, "execution_count": 691, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print len(df[df['PYProgramServiceRevenueAmt'].notnull()])\n", "print len(df[df['CYProgramServiceRevenueAmt'].notnull()])\n", "print len(df[df['ProgramServiceRevenueCY'].notnull()])\n", "print len(df[df['TotalProgramServiceRevenue'].notnull()])\n", "print len(df[df['TotalProgramServiceRevenueAmt'].notnull()]), '\\n'\n", "df['program_revenue'] = np.nan\n", "print len(df[df['program_revenue'].notnull()])\n", "df['program_revenue'] = np.where( df['PYProgramServiceRevenueAmt'].notnull(), \n", " df['PYProgramServiceRevenueAmt'], df['program_revenue'] )\n", "print len(df[df['program_revenue'].notnull()])\n", "df['program_revenue'] = np.where( ((df['program_revenue'].isnull()) & (df['CYProgramServiceRevenueAmt'].notnull())), \n", " df['CYProgramServiceRevenueAmt'], df['program_revenue'] )\n", "print len(df[df['program_revenue'].notnull()])\n", "df['program_revenue'] = np.where( ((df['program_revenue'].isnull()) & (df['ProgramServiceRevenueCY'].notnull())), \n", " df['ProgramServiceRevenueCY'], df['program_revenue'] )\n", "print len(df[df['program_revenue'].notnull()])\n", "df['program_revenue'] = np.where( ((df['program_revenue'].isnull()) & (df['TotalProgramServiceRevenue'].notnull())), \n", " df['TotalProgramServiceRevenue'], df['program_revenue'] )\n", "print len(df[df['program_revenue'].notnull()])\n", "df['program_revenue'] = np.where( ((df['program_revenue'].isnull()) \n", " & (df['TotalProgramServiceRevenueAmt'].notnull())), \n", " df['TotalProgramServiceRevenueAmt'], df['program_revenue'] )\n", "print len(df[df['program_revenue'].notnull()])\n", "df['program_revenue'].describe().T" ] }, { "cell_type": "code", "execution_count": 692, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
program_revenuePYProgramServiceRevenueAmtCYProgramServiceRevenueAmtProgramServiceRevenueProgramServiceRevenueCYProgramServiceRevenueGrpTotalProgramServiceRevenueTotalProgramServiceRevenueAmt
2520NaNNaNNaN0NaN0NaN
4330NaNNaNNaN0NaN0NaN
780NaNNaNNaN0NaN0NaN
333000NaNNaNNaNNaN0
120NaNNaNNaN0NaN0NaN
4810NaNNaNNaN0NaN0NaN
160NaNNaNNaN0NaNNaNNaN
1300NaNNaNNaN0NaN0NaN
434000NaNNaNNaNNaNNaN
3740NaNNaNNaN0NaNNaNNaN
1180NaNNaNNaN0NaNNaNNaN
1240NaNNaNNaN0NaNNaNNaN
26376449764490NaNNaNNaNNaNNaN
19676449NaNNaN{u'TotalRevenueColumn': u'76449', u'RelatedOrExemptFunctionIncome': u'76449', u'Description': u'RENTAL INCOME'}76449NaN76449NaN
256205125NaNNaN{u'TotalRevenueColumn': u'205125', u'RelatedOrExemptFunctionIncome': u'205125', u'Description': u'RENTAL INCOME'}205125NaN205125NaN
233256109NaNNaN[{u'TotalRevenueColumn': u'174799', u'RelatedOrExemptFunctionIncome': u'174799', u'Description': u'RENTAL INCOME'}, {u'TotalRevenueColumn': u'81310', u'RelatedOrExemptFunctionIncome': u'81310', u'Description': u'THRIFT STORE REVENUE'}]256109NaN256109NaN
129107536NaNNaNNaN107536NaN107536NaN
319224006224006249554NaNNaN{u'TotalRevenueColumnAmt': u'249554', u'BusinessCd': u'611600', u'RelatedOrExemptFuncIncomeAmt': u'249554', u'Desc': u'TUITION'}NaN249554
394206873206873224006NaNNaN{u'TotalRevenueColumnAmt': u'224006', u'BusinessCd': u'611600', u'RelatedOrExemptFuncIncomeAmt': u'224006', u'Desc': u'TUITION'}NaN224006
462206873NaNNaN{u'BusinessCode': u'611600', u'TotalRevenueColumn': u'206873', u'RelatedOrExemptFunctionIncome': u'206873', u'Description': u'TUITION'}206873NaN206873NaN
\n", "
" ], "text/plain": [ " program_revenue PYProgramServiceRevenueAmt CYProgramServiceRevenueAmt \\\n", "252 0 NaN NaN \n", "433 0 NaN NaN \n", "78 0 NaN NaN \n", "333 0 0 0 \n", "12 0 NaN NaN \n", "481 0 NaN NaN \n", "16 0 NaN NaN \n", "130 0 NaN NaN \n", "434 0 0 0 \n", "374 0 NaN NaN \n", "118 0 NaN NaN \n", "124 0 NaN NaN \n", "263 76449 76449 0 \n", "196 76449 NaN NaN \n", "256 205125 NaN NaN \n", "233 256109 NaN NaN \n", "129 107536 NaN NaN \n", "319 224006 224006 249554 \n", "394 206873 206873 224006 \n", "462 206873 NaN NaN \n", "\n", " ProgramServiceRevenue \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 NaN \n", "12 NaN \n", "481 NaN \n", "16 NaN \n", "130 NaN \n", "434 NaN \n", "374 NaN \n", "118 NaN \n", "124 NaN \n", "263 NaN \n", "196 {u'TotalRevenueColumn': u'76449', u'RelatedOrExemptFunctionIncome': u'76449', u'Description': u'RENTAL INCOME'} \n", "256 {u'TotalRevenueColumn': u'205125', u'RelatedOrExemptFunctionIncome': u'205125', u'Description': u'RENTAL INCOME'} \n", "233 [{u'TotalRevenueColumn': u'174799', u'RelatedOrExemptFunctionIncome': u'174799', u'Description': u'RENTAL INCOME'}, {u'TotalRevenueColumn': u'81310', u'RelatedOrExemptFunctionIncome': u'81310', u'Description': u'THRIFT STORE REVENUE'}] \n", "129 NaN \n", "319 NaN \n", "394 NaN \n", "462 {u'BusinessCode': u'611600', u'TotalRevenueColumn': u'206873', u'RelatedOrExemptFunctionIncome': u'206873', u'Description': u'TUITION'} \n", "\n", " ProgramServiceRevenueCY \\\n", "252 0 \n", "433 0 \n", "78 0 \n", "333 NaN \n", "12 0 \n", "481 0 \n", "16 0 \n", "130 0 \n", "434 NaN \n", "374 0 \n", "118 0 \n", "124 0 \n", "263 NaN \n", "196 76449 \n", "256 205125 \n", "233 256109 \n", "129 107536 \n", "319 NaN \n", "394 NaN \n", "462 206873 \n", "\n", " ProgramServiceRevenueGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 NaN \n", "12 NaN \n", "481 NaN \n", "16 NaN \n", "130 NaN \n", "434 NaN \n", "374 NaN \n", "118 NaN \n", "124 NaN \n", "263 NaN \n", "196 NaN \n", "256 NaN \n", "233 NaN \n", "129 NaN \n", "319 {u'TotalRevenueColumnAmt': u'249554', u'BusinessCd': u'611600', u'RelatedOrExemptFuncIncomeAmt': u'249554', u'Desc': u'TUITION'} \n", "394 {u'TotalRevenueColumnAmt': u'224006', u'BusinessCd': u'611600', u'RelatedOrExemptFuncIncomeAmt': u'224006', u'Desc': u'TUITION'} \n", "462 NaN \n", "\n", " TotalProgramServiceRevenue TotalProgramServiceRevenueAmt \n", "252 0 NaN \n", "433 0 NaN \n", "78 0 NaN \n", "333 NaN 0 \n", "12 0 NaN \n", "481 0 NaN \n", "16 NaN NaN \n", "130 0 NaN \n", "434 NaN NaN \n", "374 NaN NaN \n", "118 NaN NaN \n", "124 NaN NaN \n", "263 NaN NaN \n", "196 76449 NaN \n", "256 205125 NaN \n", "233 256109 NaN \n", "129 107536 NaN \n", "319 NaN 249554 \n", "394 NaN 224006 \n", "462 206873 NaN " ] }, "execution_count": 692, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[['program_revenue', 'PYProgramServiceRevenueAmt', 'CYProgramServiceRevenueAmt',\n", " 'ProgramServiceRevenue', 'ProgramServiceRevenueCY', 'ProgramServiceRevenueGrp',\n", " 'TotalProgramServiceRevenue', 'TotalProgramServiceRevenueAmt']][:20]" ] }, { "cell_type": "code", "execution_count": 693, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CYContributionsGrantsAmtFederatedCampaignsMembershipDuesfundraisingRelatedOrganizationsGovernmentGrantsprogram_revenueCYInvestmentIncomeAmtInvestmentIncomeInvestmentIncomeCurrentYearInvestmentIncomeGrpOtherRevenueTotalAmtOtherRevenueMiscGrpTotalOtherRevenueCYOtherRevenueAmt
252NaNNaNNaN0.0NaN1854080NaN{u'TotalRevenueColumn': u'63', u'ExclusionAmount': u'63'}63NaNNaNNaN0NaN
433NaNNaNNaN0.0NaNNaN0NaN{u'TotalRevenueColumn': u'0'}0NaNNaNNaN0NaN
78NaNNaNNaN0.0NaNNaN0NaN{u'TotalRevenueColumn': u'0'}0NaNNaNNaN0NaN
33320437991NaNNaN0.020532281612401601539746NaNNaN{u'TotalRevenueColumnAmt': u'362651', u'ExclusionAmt': u'362651'}428192[{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'...NaN1017789
12NaNNaNNaN0.02466016166607560NaN{u'TotalRevenueColumn': u'398381', u'ExclusionAmount': u'398381'}1195606NaNNaNNaN451157NaN
481NaNNaNNaN221449.0954610213806660NaN{u'TotalRevenueColumn': u'494644', u'ExclusionAmount': u'494644'}1249229NaNNaNNaN458249NaN
16NaNNaNNaN23265.0NaN193629780NaN{u'TotalRevenueColumn': u'646166', u'ExclusionAmount': u'646166'}703373NaNNaNNaN58199NaN
130NaN001322.00204390970NaN{u'TotalRevenueColumn': u'761762', u'RelatedOrExemptFunctionIncome': u'0', u'UnrelatedBusinessRevenue': u'0', u'ExclusionAmount': u'761762'}1195892NaNNaNNaN91122NaN
434556113NaNNaN9429.0NaNNaN0132915NaNNaN{u'TotalRevenueColumnAmt': u'132915', u'ExclusionAmt': u'132915'}15142{u'TotalRevenueColumnAmt': u'15142', u'BusinessCd': u'900099', u'ExclusionAmt': u'15142', u'Desc': u'OTHER INCOME'}NaN75409
374NaNNaNNaN2923.0NaNNaN0NaN{u'TotalRevenueColumn': u'141660', u'ExclusionAmount': u'141660'}141660NaNNaNNaN14151NaN
\n", "
" ], "text/plain": [ " CYContributionsGrantsAmt FederatedCampaigns MembershipDues fundraising \\\n", "252 NaN NaN NaN 0.0 \n", "433 NaN NaN NaN 0.0 \n", "78 NaN NaN NaN 0.0 \n", "333 20437991 NaN NaN 0.0 \n", "12 NaN NaN NaN 0.0 \n", "481 NaN NaN NaN 221449.0 \n", "16 NaN NaN NaN 23265.0 \n", "130 NaN 0 0 1322.0 \n", "434 556113 NaN NaN 9429.0 \n", "374 NaN NaN NaN 2923.0 \n", "\n", " RelatedOrganizations GovernmentGrants program_revenue \\\n", "252 NaN 185408 0 \n", "433 NaN NaN 0 \n", "78 NaN NaN 0 \n", "333 2053228 16124016 0 \n", "12 2466016 16660756 0 \n", "481 954610 21380666 0 \n", "16 NaN 19362978 0 \n", "130 0 20439097 0 \n", "434 NaN NaN 0 \n", "374 NaN NaN 0 \n", "\n", " CYInvestmentIncomeAmt \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 1539746 \n", "12 NaN \n", "481 NaN \n", "16 NaN \n", "130 NaN \n", "434 132915 \n", "374 NaN \n", "\n", " InvestmentIncome \\\n", "252 {u'TotalRevenueColumn': u'63', u'ExclusionAmount': u'63'} \n", "433 {u'TotalRevenueColumn': u'0'} \n", "78 {u'TotalRevenueColumn': u'0'} \n", "333 NaN \n", "12 {u'TotalRevenueColumn': u'398381', u'ExclusionAmount': u'398381'} \n", "481 {u'TotalRevenueColumn': u'494644', u'ExclusionAmount': u'494644'} \n", "16 {u'TotalRevenueColumn': u'646166', u'ExclusionAmount': u'646166'} \n", "130 {u'TotalRevenueColumn': u'761762', u'RelatedOrExemptFunctionIncome': u'0', u'UnrelatedBusinessRevenue': u'0', u'ExclusionAmount': u'761762'} \n", "434 NaN \n", "374 {u'TotalRevenueColumn': u'141660', u'ExclusionAmount': u'141660'} \n", "\n", " InvestmentIncomeCurrentYear \\\n", "252 63 \n", "433 0 \n", "78 0 \n", "333 NaN \n", "12 1195606 \n", "481 1249229 \n", "16 703373 \n", "130 1195892 \n", "434 NaN \n", "374 141660 \n", "\n", " InvestmentIncomeGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 {u'TotalRevenueColumnAmt': u'362651', u'ExclusionAmt': u'362651'} \n", "12 NaN \n", "481 NaN \n", "16 NaN \n", "130 NaN \n", "434 {u'TotalRevenueColumnAmt': u'132915', u'ExclusionAmt': u'132915'} \n", "374 NaN \n", "\n", " OtherRevenueTotalAmt \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 428192 \n", "12 NaN \n", "481 NaN \n", "16 NaN \n", "130 NaN \n", "434 15142 \n", "374 NaN \n", "\n", " OtherRevenueMiscGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 [{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'... \n", "12 NaN \n", "481 NaN \n", "16 NaN \n", "130 NaN \n", "434 {u'TotalRevenueColumnAmt': u'15142', u'BusinessCd': u'900099', u'ExclusionAmt': u'15142', u'Desc': u'OTHER INCOME'} \n", "374 NaN \n", "\n", " TotalOtherRevenue CYOtherRevenueAmt \n", "252 0 NaN \n", "433 0 NaN \n", "78 0 NaN \n", "333 NaN 1017789 \n", "12 451157 NaN \n", "481 458249 NaN \n", "16 58199 NaN \n", "130 91122 NaN \n", "434 NaN 75409 \n", "374 14151 NaN " ] }, "execution_count": 693, "metadata": {}, "output_type": "execute_result" } ], "source": [ "complexity_columns = ['CYContributionsGrantsAmt', 'FederatedCampaigns', #'FederatedCampaignsAmt', \n", " 'MembershipDues', #'MembershipDuesAmt',\n", " #'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp',\n", " #'FundraisingActivities', 'FundraisingActivitiesInd', \n", " #'FundraisingAmt', \n", " #'FundraisingDirectExpenses', 'FundraisingDirectExpensesAmt', 'FundraisingEvents', \n", " #'FundraisingGrossIncomeAmt', \n", " 'fundraising',\n", " 'RelatedOrganizations', #'RelatedOrganizationsAmt',\n", " 'GovernmentGrants', #'GovernmentGrantsAmt',\n", " #'PYProgramServiceRevenueAmt', 'CYProgramServiceRevenueAmt',\n", " #'ProgramServiceRevenue', 'ProgramServiceRevenueCY', 'ProgramServiceRevenueGrp',\n", " #'TotalProgramServiceRevenue', 'TotalProgramServiceRevenueAmt', \n", " 'program_revenue',\n", " 'CYInvestmentIncomeAmt', 'InvestmentIncome', 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', \n", " 'OtherRevenueTotalAmt', 'OtherRevenueMiscGrp',\n", " 'TotalOtherRevenue', 'CYOtherRevenueAmt'\n", " ]\n", " \n", "#[contributions_gifts_grants_2016', 'federated_campaigns_2016', 'membership_dues_2016', \n", "# 'fundraising_events_2016', 'related_organizations_2016', 'government_grants_2016', \n", "# 'program_service_revenue_2016', 'other_revenue_2016']\n", "\n", "df[complexity_columns][:10]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Investment Income" ] }, { "cell_type": "code", "execution_count": 694, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CYInvestmentIncomeAmtInvestmentIncomeCurrentYear
118NaN182952
124NaN92923
263-141807NaN
196NaN0
256NaN0
233NaN-52721
\n", "
" ], "text/plain": [ " CYInvestmentIncomeAmt InvestmentIncomeCurrentYear\n", "118 NaN 182952\n", "124 NaN 92923\n", "263 -141807 NaN\n", "196 NaN 0\n", "256 NaN 0\n", "233 NaN -52721" ] }, "execution_count": 694, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[['CYInvestmentIncomeAmt', 'InvestmentIncomeCurrentYear']][10:16]" ] }, { "cell_type": "code", "execution_count": 695, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "194\n", "344\n", "538\n" ] }, { "data": { "text/plain": [ "count 538\n", "unique 397\n", "top 0\n", "freq 123\n", "Name: CYInvestmentIncomeAmt, dtype: object" ] }, "execution_count": 695, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print len(df[df['CYInvestmentIncomeAmt'].notnull()])\n", "print len(df[df['InvestmentIncomeCurrentYear'].notnull()])\n", "df['CYInvestmentIncomeAmt'] = np.where( ((df['CYInvestmentIncomeAmt'].isnull()) \n", " & (df['InvestmentIncomeCurrentYear'].notnull())), \n", " df['InvestmentIncomeCurrentYear'], df['CYInvestmentIncomeAmt'] )\n", "print len(df[df['CYInvestmentIncomeAmt'].notnull()])\n", "df['CYInvestmentIncomeAmt'].describe().T" ] }, { "cell_type": "code", "execution_count": 696, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CYContributionsGrantsAmtFederatedCampaignsMembershipDuesfundraisingRelatedOrganizationsGovernmentGrantsprogram_revenueCYInvestmentIncomeAmtOtherRevenueTotalAmtOtherRevenueMiscGrpTotalOtherRevenueCYOtherRevenueAmt
252NaNNaNNaN0.0NaN185408063NaNNaN0NaN
433NaNNaNNaN0.0NaNNaN00NaNNaN0NaN
78NaNNaNNaN0.0NaNNaN00NaNNaN0NaN
33320437991NaNNaN0.020532281612401601539746428192[{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'...NaN1017789
12NaNNaNNaN0.024660161666075601195606NaNNaN451157NaN
481NaNNaNNaN221449.09546102138066601249229NaNNaN458249NaN
16NaNNaNNaN23265.0NaN193629780703373NaNNaN58199NaN
130NaN001322.002043909701195892NaNNaN91122NaN
434556113NaNNaN9429.0NaNNaN013291515142{u'TotalRevenueColumnAmt': u'15142', u'BusinessCd': u'900099', u'ExclusionAmt': u'15142', u'Desc': u'OTHER INCOME'}NaN75409
374NaNNaNNaN2923.0NaNNaN0141660NaNNaN14151NaN
\n", "
" ], "text/plain": [ " CYContributionsGrantsAmt FederatedCampaigns MembershipDues fundraising \\\n", "252 NaN NaN NaN 0.0 \n", "433 NaN NaN NaN 0.0 \n", "78 NaN NaN NaN 0.0 \n", "333 20437991 NaN NaN 0.0 \n", "12 NaN NaN NaN 0.0 \n", "481 NaN NaN NaN 221449.0 \n", "16 NaN NaN NaN 23265.0 \n", "130 NaN 0 0 1322.0 \n", "434 556113 NaN NaN 9429.0 \n", "374 NaN NaN NaN 2923.0 \n", "\n", " RelatedOrganizations GovernmentGrants program_revenue \\\n", "252 NaN 185408 0 \n", "433 NaN NaN 0 \n", "78 NaN NaN 0 \n", "333 2053228 16124016 0 \n", "12 2466016 16660756 0 \n", "481 954610 21380666 0 \n", "16 NaN 19362978 0 \n", "130 0 20439097 0 \n", "434 NaN NaN 0 \n", "374 NaN NaN 0 \n", "\n", " CYInvestmentIncomeAmt OtherRevenueTotalAmt \\\n", "252 63 NaN \n", "433 0 NaN \n", "78 0 NaN \n", "333 1539746 428192 \n", "12 1195606 NaN \n", "481 1249229 NaN \n", "16 703373 NaN \n", "130 1195892 NaN \n", "434 132915 15142 \n", "374 141660 NaN \n", "\n", " OtherRevenueMiscGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 [{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'... \n", "12 NaN \n", "481 NaN \n", "16 NaN \n", "130 NaN \n", "434 {u'TotalRevenueColumnAmt': u'15142', u'BusinessCd': u'900099', u'ExclusionAmt': u'15142', u'Desc': u'OTHER INCOME'} \n", "374 NaN \n", "\n", " TotalOtherRevenue CYOtherRevenueAmt \n", "252 0 NaN \n", "433 0 NaN \n", "78 0 NaN \n", "333 NaN 1017789 \n", "12 451157 NaN \n", "481 458249 NaN \n", "16 58199 NaN \n", "130 91122 NaN \n", "434 NaN 75409 \n", "374 14151 NaN " ] }, "execution_count": 696, "metadata": {}, "output_type": "execute_result" } ], "source": [ "complexity_columns = ['CYContributionsGrantsAmt', 'FederatedCampaigns', #'FederatedCampaignsAmt', \n", " 'MembershipDues', #'MembershipDuesAmt',\n", " #'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp',\n", " #'FundraisingActivities', 'FundraisingActivitiesInd', \n", " #'FundraisingAmt', \n", " #'FundraisingDirectExpenses', 'FundraisingDirectExpensesAmt', 'FundraisingEvents', \n", " #'FundraisingGrossIncomeAmt', \n", " 'fundraising',\n", " 'RelatedOrganizations', #'RelatedOrganizationsAmt',\n", " 'GovernmentGrants', #'GovernmentGrantsAmt',\n", " #'PYProgramServiceRevenueAmt', 'CYProgramServiceRevenueAmt',\n", " #'ProgramServiceRevenue', 'ProgramServiceRevenueCY', 'ProgramServiceRevenueGrp',\n", " #'TotalProgramServiceRevenue', 'TotalProgramServiceRevenueAmt', \n", " 'program_revenue',\n", " 'CYInvestmentIncomeAmt', \n", " #'InvestmentIncome', 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', \n", " 'OtherRevenueTotalAmt', 'OtherRevenueMiscGrp',\n", " 'TotalOtherRevenue', 'CYOtherRevenueAmt'\n", " ]\n", " \n", "#[contributions_gifts_grants_2016', 'federated_campaigns_2016', 'membership_dues_2016', \n", "# 'fundraising_events_2016', 'related_organizations_2016', 'government_grants_2016', \n", "# 'program_service_revenue_2016', 'other_revenue_2016']\n", "\n", "df[complexity_columns][:10]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Other Revenue" ] }, { "cell_type": "code", "execution_count": 697, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
OtherRevenueTotalAmtOtherRevenueMiscGrpTotalOtherRevenueCYOtherRevenueAmt
252NaNNaN0NaN
433NaNNaN0NaN
78NaNNaN0NaN
333428192[{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'...NaN1017789
12NaNNaN451157NaN
481NaNNaN458249NaN
16NaNNaN58199NaN
130NaNNaN91122NaN
43415142{u'TotalRevenueColumnAmt': u'15142', u'BusinessCd': u'900099', u'ExclusionAmt': u'15142', u'Desc': u'OTHER INCOME'}NaN75409
374NaNNaN14151NaN
\n", "
" ], "text/plain": [ " OtherRevenueTotalAmt \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 428192 \n", "12 NaN \n", "481 NaN \n", "16 NaN \n", "130 NaN \n", "434 15142 \n", "374 NaN \n", "\n", " OtherRevenueMiscGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 [{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'... \n", "12 NaN \n", "481 NaN \n", "16 NaN \n", "130 NaN \n", "434 {u'TotalRevenueColumnAmt': u'15142', u'BusinessCd': u'900099', u'ExclusionAmt': u'15142', u'Desc': u'OTHER INCOME'} \n", "374 NaN \n", "\n", " TotalOtherRevenue CYOtherRevenueAmt \n", "252 0 NaN \n", "433 0 NaN \n", "78 0 NaN \n", "333 NaN 1017789 \n", "12 451157 NaN \n", "481 458249 NaN \n", "16 58199 NaN \n", "130 91122 NaN \n", "434 NaN 75409 \n", "374 14151 NaN " ] }, "execution_count": 697, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[['OtherRevenueTotalAmt', 'OtherRevenueMiscGrp','TotalOtherRevenue', 'CYOtherRevenueAmt']][:10]" ] }, { "cell_type": "code", "execution_count": 698, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'} \n", "\n", "{u'TotalRevenueColumnAmt': u'15142', u'BusinessCd': u'900099', u'ExclusionAmt': u'15142', u'Desc': u'OTHER INCOME'} \n", "\n", "{u'TotalRevenueColumnAmt': u'155000', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'155000', u'Desc': u'REFUND OF PRIOR-YEAR GRANT'} \n", "\n", "{u'TotalRevenueColumnAmt': u'4740', u'BusinessCd': u'624100', u'RelatedOrExemptFuncIncomeAmt': u'4740', u'Desc': u'CONFERENCE REGISTRATION'} \n", "\n", "{u'TotalRevenueColumnAmt': u'2027768', u'BusinessCd': u'900099', u'ExclusionAmt': u'2027768', u'Desc': u'INSURANCE PROCEEDS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'25330', u'BusinessCd': u'900099', u'ExclusionAmt': u'25330', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'2581', u'BusinessCd': u'900099', u'ExclusionAmt': u'2581', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'1480447', u'RelatedOrExemptFuncIncomeAmt': u'1480447'} \n", "\n", "{u'TotalRevenueColumnAmt': u'3837835', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'3837835', u'Desc': u'RESTITUTION'} \n", "\n", "{u'TotalRevenueColumnAmt': u'458878', u'BusinessCd': u'900099', u'ExclusionAmt': u'458878', u'Desc': u'MEALS REVENUE'} \n", "\n", "{u'TotalRevenueColumnAmt': u'850981', u'BusinessCd': u'900099', u'ExclusionAmt': u'850981', u'Desc': u'REINSURANCE RECOVERY'} \n", "\n", "{u'TotalRevenueColumnAmt': u'600', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'600', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'11140', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'11140', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'457869', u'RelatedOrExemptFuncIncomeAmt': u'457869', u'Desc': u'PERPETUAL CARE ENDOWMENTS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'1000000', u'RelatedOrExemptFuncIncomeAmt': u'1000000', u'Desc': u'LEGAL RECOVERY SETTLEMENT'} \n", "\n", "{u'TotalRevenueColumnAmt': u'7316917', u'BusinessCd': u'561000', u'RelatedOrExemptFuncIncomeAmt': u'7316917', u'Desc': u'AGENCY FEES'} \n", "\n", "{u'TotalRevenueColumnAmt': u'6937816', u'BusinessCd': u'561000', u'RelatedOrExemptFuncIncomeAmt': u'6937816', u'Desc': u'AGENCY FEES'} \n", "\n", "{u'TotalRevenueColumnAmt': u'74698', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'74698', u'Desc': u'OTHER INCOME'} \n", "\n", "{u'TotalRevenueColumnAmt': u'-2193', u'BusinessCd': u'900099', u'ExclusionAmt': u'-2193', u'Desc': u'MISCELLANEOUS INCOME'} \n", "\n", "{u'TotalRevenueColumnAmt': u'6750977', u'BusinessCd': u'900099', u'ExclusionAmt': u'6750977', u'Desc': u'DEBT FORGIVENESS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'749391', u'BusinessCd': u'900099', u'ExclusionAmt': u'749391', u'Desc': u'DEBT FORGIVENESS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'517', u'BusinessCd': u'713990', u'RelatedOrExemptFuncIncomeAmt': u'517', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'121400', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'121400', u'Desc': u'BAD DEBT RECOVERY'} \n", "\n", "{u'TotalRevenueColumnAmt': u'292707', u'BusinessCd': u'900099', u'ExclusionAmt': u'292707', u'Desc': u'DEBT FORGIVENESS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'2653040', u'BusinessCd': u'900099', u'ExclusionAmt': u'2653040', u'Desc': u'DEBT FORGIVENESS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'5297259', u'BusinessCd': u'900099', u'ExclusionAmt': u'5297259', u'Desc': u'DEBT FORGIVENESS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'9096', u'BusinessCd': u'624200', u'ExclusionAmt': u'9096', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'1143351', u'ExclusionAmt': u'1143351', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'667098', u'ExclusionAmt': u'667098', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'147143', u'BusinessCd': u'900099', u'ExclusionAmt': u'147143', u'Desc': u'REFUNDED GRANTS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'81840', u'BusinessCd': u'900099', u'ExclusionAmt': u'81840', u'Desc': u'REFUNDED GRANTS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'15652', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'15652', u'Desc': u'OTHER REVENUE'} \n", "\n", "{u'TotalRevenueColumnAmt': u'5100', u'BusinessCd': u'900099', u'ExclusionAmt': u'5100', u'Desc': u'MISCELLANEOUS INCOME'} \n", "\n", "{u'TotalRevenueColumnAmt': u'2993', u'RelatedOrExemptFuncIncomeAmt': u'2993', u'Desc': u'OTHER'} \n", "\n", "{u'TotalRevenueColumnAmt': u'2407', u'RelatedOrExemptFuncIncomeAmt': u'2407', u'Desc': u'OTHER'} \n", "\n", "{u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000', u'Desc': u'SETTLEMENT'} \n", "\n", "{u'TotalRevenueColumnAmt': u'1050', u'BusinessCd': u'900099', u'ExclusionAmt': u'1050', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'100', u'BusinessCd': u'900099', u'ExclusionAmt': u'100', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'561', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'561', u'Desc': u'OTHER INCOME'} \n", "\n", "{u'TotalRevenueColumnAmt': u'629', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'629', u'Desc': u'OTHER INCOME'} \n", "\n", "{u'TotalRevenueColumnAmt': u'2111004', u'BusinessCd': u'900099', u'ExclusionAmt': u'2111004', u'Desc': u'DEBT FORGIVENESS'} \n", "\n", "{u'UnrelatedBusinessRevenueAmt': u'336417', u'TotalRevenueColumnAmt': u'336417', u'BusinessCd': u'900004', u'Desc': u'JOB BOARD'} \n", "\n", "{u'UnrelatedBusinessRevenueAmt': u'247673', u'TotalRevenueColumnAmt': u'247673', u'BusinessCd': u'900004', u'Desc': u'JOB PLACEMENT'} \n", "\n", "{u'TotalRevenueColumnAmt': u'3990', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'3990', u'Desc': u'CONVENTIONS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'1057365', u'BusinessCd': u'900099', u'ExclusionAmt': u'1057365', u'Desc': u'DEBT FORGIVENESS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'27153', u'BusinessCd': u'900099', u'ExclusionAmt': u'27153', u'Desc': u'MISCELLANEOUS REVENUE'} \n", "\n", "{u'TotalRevenueColumnAmt': u'13039', u'ExclusionAmt': u'13039', u'Desc': u'MISC. REVENUE'} \n", "\n", "{u'TotalRevenueColumnAmt': u'13039', u'BusinessCd': u'900099', u'ExclusionAmt': u'13039', u'Desc': u'Miscellaneous'} \n", "\n", "{u'TotalRevenueColumnAmt': u'1262179', u'BusinessCd': u'900099', u'ExclusionAmt': u'1262179', u'Desc': u'DEBT FORGIVENESS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'4985', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'4985', u'Desc': u'MISCELLANEOUS INCOME'} \n", "\n", "{u'UnrelatedBusinessRevenueAmt': u'165000', u'TotalRevenueColumnAmt': u'165000', u'BusinessCd': u'541800', u'Desc': u'ADVERTISING'} \n", "\n", "{u'UnrelatedBusinessRevenueAmt': u'130000', u'TotalRevenueColumnAmt': u'130000', u'BusinessCd': u'541800', u'Desc': u'ADVERTISING'} \n", "\n", "{u'TotalRevenueColumnAmt': u'165533', u'BusinessCd': u'900099', u'ExclusionAmt': u'165533', u'Desc': u'RECOVERY ON THEFT'} \n", "\n", "{u'TotalRevenueColumnAmt': u'45246', u'BusinessCd': u'999999', u'RelatedOrExemptFuncIncomeAmt': u'45246', u'Desc': u'MANAGEMENT SERVICE'} \n", "\n", "{u'TotalRevenueColumnAmt': u'1235', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'1235', u'Desc': u'MISCELLANEOUS INCOME'} \n", "\n", "{u'TotalRevenueColumnAmt': u'205555', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'205555', u'Desc': u'INCOME FROM FIRE INSURANCE'} \n", "\n", "{u'TotalRevenueColumnAmt': u'183015', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'183015', u'Desc': u'DEBT WRITE-OFF'} \n", "\n", "{u'TotalRevenueColumnAmt': u'13241', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'13241', u'Desc': u'RETURN ON FCC INVESTME'} \n", "\n", "{u'TotalRevenueColumnAmt': u'13481', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'13481', u'Desc': u'MISCELLANEOUS INCOME'} \n", "\n", "{u'TotalRevenueColumnAmt': u'55430', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'55430', u'Desc': u'MISCELLANEOUS INCOME'} \n", "\n", "{u'TotalRevenueColumnAmt': u'6436', u'ExclusionAmt': u'6436', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'BusinessCd': u'531190', u'Desc': u'LOSS FROM PARTNERSHIP'} \n", "\n", "{u'TotalRevenueColumnAmt': u'25911', u'ExclusionAmt': u'25911', u'Desc': u'GAIN ON CSV OF LIFE INSURANCE'} \n", "\n", "{u'TotalRevenueColumnAmt': u'2737', u'ExclusionAmt': u'2737', u'Desc': u'Other revenue'} \n", "\n", "{u'TotalRevenueColumnAmt': u'6480', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'6480', u'Desc': u'MISC REFUNDS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'33', u'ExclusionAmt': u'33', u'Desc': u'Other'} \n", "\n", "{u'TotalRevenueColumnAmt': u'5171923', u'BusinessCd': u'900099', u'ExclusionAmt': u'5171923', u'Desc': u'MULTIMEDIA CONTRACTUAL INCOME'} \n", "\n", "{u'UnrelatedBusinessRevenueAmt': u'16379', u'TotalRevenueColumnAmt': u'25900', u'BusinessCd': u'900099', u'ExclusionAmt': u'9521', u'Desc': u'FDOT ENCROACHMENT REMOVAL'} \n", "\n", "{u'TotalRevenueColumnAmt': u'16893', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'16893', u'Desc': u'SMALL BUSINESS HEALTH'} \n", "\n", "{u'TotalRevenueColumnAmt': u'11214', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'11214', u'Desc': u'FEDERAL TAX REFUND'} \n", "\n", "{u'TotalRevenueColumnAmt': u'427458', u'RelatedOrExemptFuncIncomeAmt': u'159837', u'BusinessCd': u'900199', u'ExclusionAmt': u'267621', u'Desc': u'OTHER REVENUE'} \n", "\n", "{u'TotalRevenueColumnAmt': u'301470', u'BusinessCd': u'621110', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'738003', u'BusinessCd': u'621112', u'RelatedOrExemptFuncIncomeAmt': u'738003', u'Desc': u'Miscellaneous Revenue'} \n", "\n", "{u'TotalRevenueColumnAmt': u'130433', u'BusinessCd': u'900099', u'ExclusionAmt': u'130433', u'Desc': u'FLOOD INSURANCE PAYOUT'} \n", "\n", "{u'TotalRevenueColumnAmt': u'772420', u'BusinessCd': u'900099', u'ExclusionAmt': u'772420', u'Desc': u'BABC REGISTRATION FEES'} \n", "\n", "{u'TotalRevenueColumnAmt': u'1284814', u'BusinessCd': u'900099', u'ExclusionAmt': u'1284814', u'Desc': u'DEBT FORGIVENESS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'1271392', u'BusinessCd': u'900099', u'ExclusionAmt': u'1271392', u'Desc': u'DEBT FORGIVENESS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'87550', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'87550', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'93778', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'93778', u'Desc': u'INSURANCE PROCEEDS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'378405', u'BusinessCd': u'541800', u'RelatedOrExemptFuncIncomeAmt': u'378405', u'Desc': u'ADMINISTRATIVE INCOME'} \n", "\n", "{u'TotalRevenueColumnAmt': u'134786', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'134786', u'Desc': u'INSURANCE PROCEEDS FRO'} \n", "\n", "{u'TotalRevenueColumnAmt': u'105205', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'105205', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'UnrelatedBusinessRevenueAmt': u'7929062', u'TotalRevenueColumnAmt': u'7929062', u'BusinessCd': u'812300', u'Desc': u'LAUNDRY SERVICES'} \n", "\n", "{u'TotalRevenueColumnAmt': u'5706', u'RelatedOrExemptFuncIncomeAmt': u'5706', u'Desc': u'OTHER RECEIPTS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'2380', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'2380', u'Desc': u'GAIN/LOSS ON CURRENCY'} \n", "\n", "{u'TotalRevenueColumnAmt': u'-1340', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'-1340', u'Desc': u'GAIN/LOSS ON CURRENCY'} \n", "\n", "{u'TotalRevenueColumnAmt': u'1613', u'RelatedOrExemptFuncIncomeAmt': u'1613', u'Desc': u'DAF Mgmt Fees'} \n", "\n", "{u'TotalRevenueColumnAmt': u'11566', u'RelatedOrExemptFuncIncomeAmt': u'11566', u'Desc': u'OTHER REVENUE'} \n", "\n", "{u'TotalRevenueColumnAmt': u'37665', u'RelatedOrExemptFuncIncomeAmt': u'37665', u'Desc': u'OTHER REVENUE'} \n", "\n", "{u'TotalRevenueColumnAmt': u'5005', u'ExclusionAmt': u'5005', u'Desc': u'MISC. REFUNDS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'66924', u'BusinessCd': u'515100', u'ExclusionAmt': u'66924', u'Desc': u'MISCELLANEOUS'} \n", "\n", "{u'TotalRevenueColumnAmt': u'1548749', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'1548749', u'Desc': u'VALUE ADDED TAX REBATE'} \n", "\n", "{u'TotalRevenueColumnAmt': u'137059', u'BusinessCd': u'900099', u'ExclusionAmt': u'137059', u'Desc': u'OTHER'} \n", "\n", "{u'TotalRevenueColumnAmt': u'7299', u'ExclusionAmt': u'7299', u'Desc': u'Gain on A/C ins. claim'} \n", "\n", "{u'TotalRevenueColumnAmt': u'50340', u'BusinessCd': u'900099', u'ExclusionAmt': u'50340', u'Desc': u'List Rental'} \n", "\n", "{u'TotalRevenueColumnAmt': u'19820777', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'19820777', u'Desc': u'OTHER INCOME- FROM AFFILIATES'} \n", "\n", "{u'TotalRevenueColumnAmt': u'15369743', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'15369743', u'Desc': u'OTHER INCOME- FROM AFFILIATES'} \n", "\n" ] } ], "source": [ "counter = 1\n", "df['OtherRevMisc'] = np.nan\n", "for index, row in df.iterrows():\n", " #print row['NetIncomeFromFundraisingEvents']\n", " #print type(ast.literal_eval(str(row['NetIncomeFromFundraisingEvents'])))\n", " \n", " try:\n", " fr_revs = str(row['OtherRevenueMiscGrp']) \n", " if fr_revs.startswith('['):\n", " #print 'yes'\n", " fr_revs = fr_revs.replace('[', '').replace(']', '')\n", " fr_revs = ast.literal_eval(fr_revs)\n", " fr_revs = fr_revs[0]\n", " else:\n", " fr_revs = ast.literal_eval(fr_revs)\n", " print fr_revs, '\\n'\n", " if 'TotalRevenueColumn' in fr_revs:\n", " #print 'yes', fr_revs['TotalRevenueColumn']\n", " df.ix[index, 'OtherRevMisc'] = fr_revs['TotalRevenueColumn']\n", " elif 'TotalRevenueColumnAmt' in fr_revs:\n", " #print 'yes', fr_revs['TotalRevenueColumnAmt']\n", " df.ix[index, 'OtherRevMisc'] = fr_revs['TotalRevenueColumnAmt']\n", " except:\n", " df.ix[index, 'OtherRevMisc'] = np.nan \n", " else:\n", " df.ix[index, 'OtherRevMisc'] = np.nan\n", " #counter +=1\n", " #print counter" ] }, { "cell_type": "code", "execution_count": 699, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
OtherRevenueTotalAmtOtherRevMiscOtherRevenueMiscGrpTotalOtherRevenueCYOtherRevenueAmt
252NaNNaNNaN0NaN
433NaNNaNNaN0NaN
78NaNNaNNaN0NaN
333428192NaN[{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'...NaN1017789
12NaNNaNNaN451157NaN
481NaNNaNNaN458249NaN
16NaNNaNNaN58199NaN
130NaNNaNNaN91122NaN
43415142NaN{u'TotalRevenueColumnAmt': u'15142', u'BusinessCd': u'900099', u'ExclusionAmt': u'15142', u'Desc': u'OTHER INCOME'}NaN75409
374NaNNaNNaN14151NaN
\n", "
" ], "text/plain": [ " OtherRevenueTotalAmt OtherRevMisc \\\n", "252 NaN NaN \n", "433 NaN NaN \n", "78 NaN NaN \n", "333 428192 NaN \n", "12 NaN NaN \n", "481 NaN NaN \n", "16 NaN NaN \n", "130 NaN NaN \n", "434 15142 NaN \n", "374 NaN NaN \n", "\n", " OtherRevenueMiscGrp \\\n", "252 NaN \n", "433 NaN \n", "78 NaN \n", "333 [{u'TotalRevenueColumnAmt': u'33550', u'BusinessCd': u'531390', u'RelatedOrExemptFuncIncomeAmt': u'33550', u'Desc': u'PARKING INCOME'}, {u'TotalRevenueColumnAmt': u'394642', u'BusinessCd': u'900099', u'RelatedOrExemptFuncIncomeAmt': u'394642', u'... \n", "12 NaN \n", "481 NaN \n", "16 NaN \n", "130 NaN \n", "434 {u'TotalRevenueColumnAmt': u'15142', u'BusinessCd': u'900099', u'ExclusionAmt': u'15142', u'Desc': u'OTHER INCOME'} \n", "374 NaN \n", "\n", " TotalOtherRevenue CYOtherRevenueAmt \n", "252 0 NaN \n", "433 0 NaN \n", "78 0 NaN \n", "333 NaN 1017789 \n", "12 451157 NaN \n", "481 458249 NaN \n", "16 58199 NaN \n", "130 91122 NaN \n", "434 NaN 75409 \n", "374 14151 NaN " ] }, "execution_count": 699, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[['OtherRevenueTotalAmt', 'OtherRevMisc', 'OtherRevenueMiscGrp','TotalOtherRevenue', 'CYOtherRevenueAmt']][:10]" ] }, { "cell_type": "code", "execution_count": 700, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "120\n", "344\n", "194\n", "414\n", "414\n" ] }, { "data": { "text/plain": [ "count 414\n", "unique 294\n", "top 0\n", "freq 119\n", "Name: OtherRevenueTotalAmt, dtype: object" ] }, "execution_count": 700, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print len(df[df['OtherRevenueTotalAmt'].notnull()])\n", "print len(df[df['InvestmentIncomeCurrentYear'].notnull()])\n", "df['OtherRevenueTotalAmt'] = np.where( ((df['OtherRevenueTotalAmt'].isnull()) & (df['CYOtherRevenueAmt'].notnull())), \n", " df['CYOtherRevenueAmt'], df['OtherRevenueTotalAmt'] )\n", "print len(df[df['OtherRevenueTotalAmt'].notnull()])\n", "df['OtherRevenueTotalAmt'] = np.where( ((df['OtherRevenueTotalAmt'].isnull()) & (df['TotalOtherRevenue'].notnull())), \n", " df['TotalOtherRevenue'], df['OtherRevenueTotalAmt'] )\n", "print len(df[df['OtherRevenueTotalAmt'].notnull()])\n", "df['OtherRevenueTotalAmt'] = np.where( ((df['OtherRevenueTotalAmt'].isnull()) & (df['OtherRevMisc'].notnull())), \n", " df['OtherRevMisc'], df['OtherRevenueTotalAmt'] )\n", "print len(df[df['OtherRevenueTotalAmt'].notnull()])\n", "df['OtherRevenueTotalAmt'].describe().T" ] }, { "cell_type": "code", "execution_count": 701, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CYContributionsGrantsAmtFederatedCampaignsMembershipDuesfundraisingRelatedOrganizationsGovernmentGrantsprogram_revenueCYInvestmentIncomeAmtOtherRevenueTotalAmt
252NaNNaNNaN0.0NaN1854080630
433NaNNaNNaN0.0NaNNaN000
78NaNNaNNaN0.0NaNNaN000
33320437991NaNNaN0.020532281612401601539746428192
12NaNNaNNaN0.024660161666075601195606451157
481NaNNaNNaN221449.09546102138066601249229458249
16NaNNaNNaN23265.0NaN19362978070337358199
130NaN001322.00204390970119589291122
434556113NaNNaN9429.0NaNNaN013291515142
374NaNNaNNaN2923.0NaNNaN014166014151
\n", "
" ], "text/plain": [ " CYContributionsGrantsAmt FederatedCampaigns MembershipDues fundraising \\\n", "252 NaN NaN NaN 0.0 \n", "433 NaN NaN NaN 0.0 \n", "78 NaN NaN NaN 0.0 \n", "333 20437991 NaN NaN 0.0 \n", "12 NaN NaN NaN 0.0 \n", "481 NaN NaN NaN 221449.0 \n", "16 NaN NaN NaN 23265.0 \n", "130 NaN 0 0 1322.0 \n", "434 556113 NaN NaN 9429.0 \n", "374 NaN NaN NaN 2923.0 \n", "\n", " RelatedOrganizations GovernmentGrants program_revenue \\\n", "252 NaN 185408 0 \n", "433 NaN NaN 0 \n", "78 NaN NaN 0 \n", "333 2053228 16124016 0 \n", "12 2466016 16660756 0 \n", "481 954610 21380666 0 \n", "16 NaN 19362978 0 \n", "130 0 20439097 0 \n", "434 NaN NaN 0 \n", "374 NaN NaN 0 \n", "\n", " CYInvestmentIncomeAmt OtherRevenueTotalAmt \n", "252 63 0 \n", "433 0 0 \n", "78 0 0 \n", "333 1539746 428192 \n", "12 1195606 451157 \n", "481 1249229 458249 \n", "16 703373 58199 \n", "130 1195892 91122 \n", "434 132915 15142 \n", "374 141660 14151 " ] }, "execution_count": 701, "metadata": {}, "output_type": "execute_result" } ], "source": [ "complexity_columns = ['CYContributionsGrantsAmt', 'FederatedCampaigns', #'FederatedCampaignsAmt', \n", " 'MembershipDues', #'MembershipDuesAmt',\n", " #'NetIncomeFromFundraisingEvents', 'NetIncmFromFundraisingEvtGrp',\n", " #'FundraisingActivities', 'FundraisingActivitiesInd', \n", " #'FundraisingAmt', \n", " #'FundraisingDirectExpenses', 'FundraisingDirectExpensesAmt', 'FundraisingEvents', \n", " #'FundraisingGrossIncomeAmt', \n", " 'fundraising',\n", " 'RelatedOrganizations', #'RelatedOrganizationsAmt',\n", " 'GovernmentGrants', #'GovernmentGrantsAmt',\n", " #'PYProgramServiceRevenueAmt', 'CYProgramServiceRevenueAmt',\n", " #'ProgramServiceRevenue', 'ProgramServiceRevenueCY', 'ProgramServiceRevenueGrp',\n", " #'TotalProgramServiceRevenue', 'TotalProgramServiceRevenueAmt', \n", " 'program_revenue',\n", " 'CYInvestmentIncomeAmt', \n", " #'InvestmentIncome', 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', \n", " 'OtherRevenueTotalAmt', #'OtherRevenueMiscGrp',\n", " #'TotalOtherRevenue', 'CYOtherRevenueAmt'\n", " ]\n", " \n", "#[contributions_gifts_grants_2016', 'federated_campaigns_2016', 'membership_dues_2016', \n", "# 'fundraising_events_2016', 'related_organizations_2016', 'government_grants_2016', \n", "# 'program_service_revenue_2016', 'other_revenue_2016']\n", "\n", "df[complexity_columns][:10]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Change to float" ] }, { "cell_type": "code", "execution_count": 702, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['CYContributionsGrantsAmt', 'FederatedCampaigns', 'MembershipDues', 'fundraising', 'RelatedOrganizations', 'GovernmentGrants', 'program_revenue', 'CYInvestmentIncomeAmt', 'OtherRevenueTotalAmt']\n" ] } ], "source": [ "print complexity_columns" ] }, { "cell_type": "code", "execution_count": 703, "metadata": { "collapsed": true }, "outputs": [], "source": [ "df['CYContributionsGrantsAmt'] = df['CYContributionsGrantsAmt'].astype('float')\n", "df['FederatedCampaigns'] = df['FederatedCampaigns'].astype('float')\n", "df['MembershipDues'] = df['MembershipDues'].astype('float')\n", "df['fundraising'] = df['fundraising'].astype('float')\n", "df['RelatedOrganizations'] = df['RelatedOrganizations'].astype('float')\n", "df['GovernmentGrants'] = df['GovernmentGrants'].astype('float')\n", "df['program_revenue'] = df['program_revenue'].astype('float')\n", "df['CYInvestmentIncomeAmt'] = df['CYInvestmentIncomeAmt'].astype('float')\n", "df['OtherRevenueTotalAmt'] = df['OtherRevenueTotalAmt'].astype('float')" ] }, { "cell_type": "code", "execution_count": 704, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "CYContributionsGrantsAmt float64\n", "FederatedCampaigns float64\n", "MembershipDues float64\n", "fundraising float64\n", "RelatedOrganizations float64\n", "GovernmentGrants float64\n", "program_revenue float64\n", "CYInvestmentIncomeAmt float64\n", "OtherRevenueTotalAmt float64\n", "dtype: object" ] }, "execution_count": 704, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[complexity_columns].dtypes" ] }, { "cell_type": "code", "execution_count": 705, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countmeanstdmin25%50%75%max
CYContributionsGrantsAmt194.01.462879e+079.210089e+070.0NaNNaNNaN9.182532e+08
FederatedCampaigns40.08.356717e+051.710843e+060.0NaNNaNNaN6.051068e+06
MembershipDues44.09.858459e+041.330675e+050.0NaNNaNNaN6.000830e+05
fundraising228.08.138974e+045.064406e+05-283900.0NaNNaNNaN7.283318e+06
RelatedOrganizations52.03.521487e+065.394679e+060.0NaNNaNNaN2.086632e+07
GovernmentGrants213.01.975853e+079.133767e+070.0NaNNaNNaN7.549167e+08
program_revenue538.04.042349e+072.827323e+080.00.0172176.03712362.03.090399e+09
CYInvestmentIncomeAmt538.09.094292e+051.338435e+07-218265025.00.01222.036411.51.583775e+08
OtherRevenueTotalAmt414.02.131831e+061.163631e+07-162277.0NaNNaNNaN1.463759e+08
\n", "
" ], "text/plain": [ " count mean std min 25% \\\n", "CYContributionsGrantsAmt 194.0 1.462879e+07 9.210089e+07 0.0 NaN \n", "FederatedCampaigns 40.0 8.356717e+05 1.710843e+06 0.0 NaN \n", "MembershipDues 44.0 9.858459e+04 1.330675e+05 0.0 NaN \n", "fundraising 228.0 8.138974e+04 5.064406e+05 -283900.0 NaN \n", "RelatedOrganizations 52.0 3.521487e+06 5.394679e+06 0.0 NaN \n", "GovernmentGrants 213.0 1.975853e+07 9.133767e+07 0.0 NaN \n", "program_revenue 538.0 4.042349e+07 2.827323e+08 0.0 0.0 \n", "CYInvestmentIncomeAmt 538.0 9.094292e+05 1.338435e+07 -218265025.0 0.0 \n", "OtherRevenueTotalAmt 414.0 2.131831e+06 1.163631e+07 -162277.0 NaN \n", "\n", " 50% 75% max \n", "CYContributionsGrantsAmt NaN NaN 9.182532e+08 \n", "FederatedCampaigns NaN NaN 6.051068e+06 \n", "MembershipDues NaN NaN 6.000830e+05 \n", "fundraising NaN NaN 7.283318e+06 \n", "RelatedOrganizations NaN NaN 2.086632e+07 \n", "GovernmentGrants NaN NaN 7.549167e+08 \n", "program_revenue 172176.0 3712362.0 3.090399e+09 \n", "CYInvestmentIncomeAmt 1222.0 36411.5 1.583775e+08 \n", "OtherRevenueTotalAmt NaN NaN 1.463759e+08 " ] }, "execution_count": 705, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[complexity_columns].describe().T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### combined other revenue" ] }, { "cell_type": "code", "execution_count": 543, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#df = df.drop('other_revenue', 1)" ] }, { "cell_type": "code", "execution_count": 707, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "538\n", "414\n", "538\n" ] } ], "source": [ "print len(df[df['CYInvestmentIncomeAmt'].notnull()])\n", "print len(df[df['OtherRevenueTotalAmt'].notnull()])\n", "df['other_revenue_binary'] = np.nan\n", "df['other_revenue_binary'] = np.where(df['CYInvestmentIncomeAmt'].notnull(), df['CYInvestmentIncomeAmt'],\n", " df['OtherRevenueTotalAmt'])\n", "print len(df[df['other_revenue_binary'].notnull()])" ] }, { "cell_type": "code", "execution_count": 708, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['CYContributionsGrantsAmt', 'FederatedCampaigns', 'MembershipDues', 'fundraising', 'RelatedOrganizations', 'GovernmentGrants', 'program_revenue', 'CYInvestmentIncomeAmt', 'OtherRevenueTotalAmt']\n" ] } ], "source": [ "print complexity_columns" ] }, { "cell_type": "code", "execution_count": 709, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CYContributionsGrantsAmtFederatedCampaignsMembershipDuesfundraisingRelatedOrganizationsGovernmentGrantsprogram_revenueother_revenue_binaryCYInvestmentIncomeAmtOtherRevenueTotalAmt
118NaNNaNNaN47210.0NaNNaN0.0182952.0182952.0124761.0
124NaNNaNNaN99073.0NaNNaN0.092923.092923.09015.0
263994088.0NaNNaNNaNNaNNaN76449.0-141807.0-141807.00.0
196NaNNaNNaNNaNNaNNaN76449.00.00.0NaN
256NaNNaNNaNNaNNaNNaN205125.00.00.0NaN
233NaNNaNNaNNaNNaNNaN256109.0-52721.0-52721.0NaN
129NaNNaNNaNNaNNaNNaN107536.00.00.0NaN
319104931.0NaNNaN0.0NaNNaN224006.023.023.00.0
39486418.0NaNNaN0.0NaNNaN206873.030.030.00.0
462NaNNaNNaN0.0NaNNaN206873.04.04.00.0
101NaNNaNNaN0.0NaNNaN194006.07.07.00.0
52812294982.0NaNNaNNaNNaN646173.00.01549.01549.00.0
40112741996.0NaNNaNNaNNaN629939.00.01656.01656.00.0
361NaNNaNNaNNaNNaN437339.00.02695.02695.0NaN
36NaNNaNNaNNaNNaN461226.00.02317.02317.0NaN
41NaNNaNNaNNaNNaN373496.00.02984.02984.0NaN
63NaNNaNNaNNaNNaNNaN0.00.00.0NaN
93NaNNaNNaNNaNNaNNaN0.0642.0642.0NaN
504NaNNaNNaNNaNNaNNaN0.0616.0616.0NaN
31307743.0NaNNaNNaNNaNNaN0.05902.05902.0155000.0
\n", "
" ], "text/plain": [ " CYContributionsGrantsAmt FederatedCampaigns MembershipDues \\\n", "118 NaN NaN NaN \n", "124 NaN NaN NaN \n", "263 994088.0 NaN NaN \n", "196 NaN NaN NaN \n", "256 NaN NaN NaN \n", "233 NaN NaN NaN \n", "129 NaN NaN NaN \n", "319 104931.0 NaN NaN \n", "394 86418.0 NaN NaN \n", "462 NaN NaN NaN \n", "101 NaN NaN NaN \n", "528 12294982.0 NaN NaN \n", "401 12741996.0 NaN NaN \n", "361 NaN NaN NaN \n", "36 NaN NaN NaN \n", "41 NaN NaN NaN \n", "63 NaN NaN NaN \n", "93 NaN NaN NaN \n", "504 NaN NaN NaN \n", "31 307743.0 NaN NaN \n", "\n", " fundraising RelatedOrganizations GovernmentGrants program_revenue \\\n", "118 47210.0 NaN NaN 0.0 \n", "124 99073.0 NaN NaN 0.0 \n", "263 NaN NaN NaN 76449.0 \n", "196 NaN NaN NaN 76449.0 \n", "256 NaN NaN NaN 205125.0 \n", "233 NaN NaN NaN 256109.0 \n", "129 NaN NaN NaN 107536.0 \n", "319 0.0 NaN NaN 224006.0 \n", "394 0.0 NaN NaN 206873.0 \n", "462 0.0 NaN NaN 206873.0 \n", "101 0.0 NaN NaN 194006.0 \n", "528 NaN NaN 646173.0 0.0 \n", "401 NaN NaN 629939.0 0.0 \n", "361 NaN NaN 437339.0 0.0 \n", "36 NaN NaN 461226.0 0.0 \n", "41 NaN NaN 373496.0 0.0 \n", "63 NaN NaN NaN 0.0 \n", "93 NaN NaN NaN 0.0 \n", "504 NaN NaN NaN 0.0 \n", "31 NaN NaN NaN 0.0 \n", "\n", " other_revenue_binary CYInvestmentIncomeAmt OtherRevenueTotalAmt \n", "118 182952.0 182952.0 124761.0 \n", "124 92923.0 92923.0 9015.0 \n", "263 -141807.0 -141807.0 0.0 \n", "196 0.0 0.0 NaN \n", "256 0.0 0.0 NaN \n", "233 -52721.0 -52721.0 NaN \n", "129 0.0 0.0 NaN \n", "319 23.0 23.0 0.0 \n", "394 30.0 30.0 0.0 \n", "462 4.0 4.0 0.0 \n", "101 7.0 7.0 0.0 \n", "528 1549.0 1549.0 0.0 \n", "401 1656.0 1656.0 0.0 \n", "361 2695.0 2695.0 NaN \n", "36 2317.0 2317.0 NaN \n", "41 2984.0 2984.0 NaN \n", "63 0.0 0.0 NaN \n", "93 642.0 642.0 NaN \n", "504 616.0 616.0 NaN \n", "31 5902.0 5902.0 155000.0 " ] }, "execution_count": 709, "metadata": {}, "output_type": "execute_result" } ], "source": [ "complexity_columns = ['CYContributionsGrantsAmt', 'FederatedCampaigns', 'MembershipDues', 'fundraising', \n", " 'RelatedOrganizations', 'GovernmentGrants', 'program_revenue', 'other_revenue_binary',\n", " 'CYInvestmentIncomeAmt', 'OtherRevenueTotalAmt']\n", "df[complexity_columns][10:30]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#[contributions_gifts_grants_2016', 'federated_campaigns_2016', 'membership_dues_2016', \n", "# 'fundraising_events_2016', 'related_organizations_2016', 'government_grants_2016', \n", "# 'program_service_revenue_2016', 'other_revenue_2016']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Complexity" ] }, { "cell_type": "code", "execution_count": 710, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countmeanstdmin25%50%75%max
CYContributionsGrantsAmt194.01.462879e+079.210089e+070.0NaNNaNNaN9.182532e+08
FederatedCampaigns40.08.356717e+051.710843e+060.0NaNNaNNaN6.051068e+06
MembershipDues44.09.858459e+041.330675e+050.0NaNNaNNaN6.000830e+05
fundraising228.08.138974e+045.064406e+05-283900.0NaNNaNNaN7.283318e+06
RelatedOrganizations52.03.521487e+065.394679e+060.0NaNNaNNaN2.086632e+07
GovernmentGrants213.01.975853e+079.133767e+070.0NaNNaNNaN7.549167e+08
program_revenue538.04.042349e+072.827323e+080.00.0172176.03712362.03.090399e+09
other_revenue_binary538.09.094292e+051.338435e+07-218265025.00.01222.036411.51.583775e+08
CYInvestmentIncomeAmt538.09.094292e+051.338435e+07-218265025.00.01222.036411.51.583775e+08
OtherRevenueTotalAmt414.02.131831e+061.163631e+07-162277.0NaNNaNNaN1.463759e+08
\n", "
" ], "text/plain": [ " count mean std min 25% \\\n", "CYContributionsGrantsAmt 194.0 1.462879e+07 9.210089e+07 0.0 NaN \n", "FederatedCampaigns 40.0 8.356717e+05 1.710843e+06 0.0 NaN \n", "MembershipDues 44.0 9.858459e+04 1.330675e+05 0.0 NaN \n", "fundraising 228.0 8.138974e+04 5.064406e+05 -283900.0 NaN \n", "RelatedOrganizations 52.0 3.521487e+06 5.394679e+06 0.0 NaN \n", "GovernmentGrants 213.0 1.975853e+07 9.133767e+07 0.0 NaN \n", "program_revenue 538.0 4.042349e+07 2.827323e+08 0.0 0.0 \n", "other_revenue_binary 538.0 9.094292e+05 1.338435e+07 -218265025.0 0.0 \n", "CYInvestmentIncomeAmt 538.0 9.094292e+05 1.338435e+07 -218265025.0 0.0 \n", "OtherRevenueTotalAmt 414.0 2.131831e+06 1.163631e+07 -162277.0 NaN \n", "\n", " 50% 75% max \n", "CYContributionsGrantsAmt NaN NaN 9.182532e+08 \n", "FederatedCampaigns NaN NaN 6.051068e+06 \n", "MembershipDues NaN NaN 6.000830e+05 \n", "fundraising NaN NaN 7.283318e+06 \n", "RelatedOrganizations NaN NaN 2.086632e+07 \n", "GovernmentGrants NaN NaN 7.549167e+08 \n", "program_revenue 172176.0 3712362.0 3.090399e+09 \n", "other_revenue_binary 1222.0 36411.5 1.583775e+08 \n", "CYInvestmentIncomeAmt 1222.0 36411.5 1.583775e+08 \n", "OtherRevenueTotalAmt NaN NaN 1.463759e+08 " ] }, "execution_count": 710, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[complexity_columns].describe().T" ] }, { "cell_type": "code", "execution_count": 711, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['CYContributionsGrantsAmt', 'FederatedCampaigns', 'MembershipDues', 'fundraising', 'RelatedOrganizations', 'GovernmentGrants', 'program_revenue', 'other_revenue_binary', 'CYInvestmentIncomeAmt', 'OtherRevenueTotalAmt']\n" ] } ], "source": [ "print complexity_columns" ] }, { "cell_type": "code", "execution_count": 712, "metadata": { "collapsed": true }, "outputs": [], "source": [ "complexity_columns = ['CYContributionsGrantsAmt', 'FederatedCampaigns', 'MembershipDues', 'fundraising', \n", " 'RelatedOrganizations', 'GovernmentGrants', 'program_revenue', 'other_revenue_binary']" ] }, { "cell_type": "code", "execution_count": 713, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "252 2\n", "433 0\n", "78 0\n", "Name: complexity, dtype: int64" ] }, "execution_count": 713, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['complexity'] = (df[complexity_columns] > 0).sum(1)\n", "df['complexity'][:3]" ] }, { "cell_type": "code", "execution_count": 714, "metadata": { "collapsed": true }, "outputs": [], "source": [ "complexity_columns = ['complexity'] + complexity_columns" ] }, { "cell_type": "code", "execution_count": 715, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "count 538.000000\n", "mean 2.431227\n", "std 1.278642\n", "min 0.000000\n", "25% 2.000000\n", "50% 2.000000\n", "75% 3.000000\n", "max 7.000000\n", "Name: complexity, dtype: float64" ] }, "execution_count": 715, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['complexity'].describe()" ] }, { "cell_type": "code", "execution_count": 716, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countmeanstdmin25%50%75%max
complexity538.02.431227e+001.278642e+000.02.02.03.07.000000e+00
CYContributionsGrantsAmt194.01.462879e+079.210089e+070.0NaNNaNNaN9.182532e+08
FederatedCampaigns40.08.356717e+051.710843e+060.0NaNNaNNaN6.051068e+06
MembershipDues44.09.858459e+041.330675e+050.0NaNNaNNaN6.000830e+05
fundraising228.08.138974e+045.064406e+05-283900.0NaNNaNNaN7.283318e+06
RelatedOrganizations52.03.521487e+065.394679e+060.0NaNNaNNaN2.086632e+07
GovernmentGrants213.01.975853e+079.133767e+070.0NaNNaNNaN7.549167e+08
program_revenue538.04.042349e+072.827323e+080.00.0172176.03712362.03.090399e+09
other_revenue_binary538.09.094292e+051.338435e+07-218265025.00.01222.036411.51.583775e+08
\n", "
" ], "text/plain": [ " count mean std min 25% \\\n", "complexity 538.0 2.431227e+00 1.278642e+00 0.0 2.0 \n", "CYContributionsGrantsAmt 194.0 1.462879e+07 9.210089e+07 0.0 NaN \n", "FederatedCampaigns 40.0 8.356717e+05 1.710843e+06 0.0 NaN \n", "MembershipDues 44.0 9.858459e+04 1.330675e+05 0.0 NaN \n", "fundraising 228.0 8.138974e+04 5.064406e+05 -283900.0 NaN \n", "RelatedOrganizations 52.0 3.521487e+06 5.394679e+06 0.0 NaN \n", "GovernmentGrants 213.0 1.975853e+07 9.133767e+07 0.0 NaN \n", "program_revenue 538.0 4.042349e+07 2.827323e+08 0.0 0.0 \n", "other_revenue_binary 538.0 9.094292e+05 1.338435e+07 -218265025.0 0.0 \n", "\n", " 50% 75% max \n", "complexity 2.0 3.0 7.000000e+00 \n", "CYContributionsGrantsAmt NaN NaN 9.182532e+08 \n", "FederatedCampaigns NaN NaN 6.051068e+06 \n", "MembershipDues NaN NaN 6.000830e+05 \n", "fundraising NaN NaN 7.283318e+06 \n", "RelatedOrganizations NaN NaN 2.086632e+07 \n", "GovernmentGrants NaN NaN 7.549167e+08 \n", "program_revenue 172176.0 3712362.0 3.090399e+09 \n", "other_revenue_binary 1222.0 36411.5 1.583775e+08 " ] }, "execution_count": 716, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[complexity_columns].describe().T" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#[contributions_gifts_grants_2016', 'federated_campaigns_2016', 'membership_dues_2016', \n", "# 'fundraising_events_2016', 'related_organizations_2016', 'government_grants_2016', \n", "# 'program_service_revenue_2016', 'other_revenue_2016']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Combined SOX policies variables" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Create combined *SOX_policy* variable" ] }, { "cell_type": "code", "execution_count": 717, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "252 2.0\n", "433 2.0\n", "78 2.0\n", "Name: SOX_policies, dtype: float64" ] }, "execution_count": 717, "metadata": {}, "output_type": "execute_result" } ], "source": [ "SOX_columns = ['conflict_of_interest_policy', 'whistleblower_policy', 'records_retention_policy']\n", "#df['SOX_policies'] = (df[SOX_columns] > 0).sum(1)\n", "df['SOX_policies'] = np.where(df['conflict_of_interest_policy'].notnull(),\n", " (df[SOX_columns] > 0).sum(1), np.nan)\n", "df['SOX_policies'][:3]" ] }, { "cell_type": "code", "execution_count": 718, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SOX_policiesconflict_of_interest_policywhistleblower_policyrecords_retention_policy
2522.0101
4332.0101
782.0101
3333.0111
123.0111
4813.0111
163.0111
1303.0111
\n", "
" ], "text/plain": [ " SOX_policies conflict_of_interest_policy whistleblower_policy \\\n", "252 2.0 1 0 \n", "433 2.0 1 0 \n", "78 2.0 1 0 \n", "333 3.0 1 1 \n", "12 3.0 1 1 \n", "481 3.0 1 1 \n", "16 3.0 1 1 \n", "130 3.0 1 1 \n", "\n", " records_retention_policy \n", "252 1 \n", "433 1 \n", "78 1 \n", "333 1 \n", "12 1 \n", "481 1 \n", "16 1 \n", "130 1 " ] }, "execution_count": 718, "metadata": {}, "output_type": "execute_result" } ], "source": [ "SOX_columns = ['SOX_policies'] + SOX_columns\n", "df[SOX_columns][:8]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Binary version ('1' equals any SOX policy)." ] }, { "cell_type": "code", "execution_count": 719, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "538\n" ] }, { "data": { "text/plain": [ "1.0 463\n", "0.0 75\n", "Name: SOX_policies_binary, dtype: int64" ] }, "execution_count": 719, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['SOX_policies_binary'] = df['SOX_policies']\n", "df['SOX_policies_binary'] = np.where(df['SOX_policies_binary']>=1, 1, df['SOX_policies'])\n", "print df['SOX_policies_binary'].value_counts().sum()\n", "df['SOX_policies_binary'].value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Create binary version for *all three policies*" ] }, { "cell_type": "code", "execution_count": 720, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1.0 293\n", "0.0 245\n", "Name: SOX_policies_all_binary, dtype: int64\n" ] } ], "source": [ "df['SOX_policies_all_binary'] = np.nan\n", "df['SOX_policies_all_binary'] = df['SOX_policies']\n", "df['SOX_policies_all_binary'] = np.where( ((df['SOX_policies_all_binary']==1) | (df['SOX_policies_all_binary']==2)),\n", " 0, df['SOX_policies_all_binary'])\n", "df['SOX_policies_all_binary'] = np.where(df['SOX_policies_all_binary']==3, 1, df['SOX_policies_all_binary'])\n", "print df['SOX_policies_all_binary'].value_counts()" ] }, { "cell_type": "code", "execution_count": 721, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3.0 293\n", "2.0 105\n", "0.0 75\n", "1.0 65\n", "Name: SOX_policies, dtype: int64 \n", "\n", "1.0 463\n", "0.0 75\n", "Name: SOX_policies_binary, dtype: int64 \n", "\n", "1.0 293\n", "0.0 245\n", "Name: SOX_policies_all_binary, dtype: int64\n" ] } ], "source": [ "print df['SOX_policies'].value_counts(), '\\n'\n", "print df['SOX_policies_binary'].value_counts(), '\\n'\n", "print df['SOX_policies_all_binary'].value_counts()" ] }, { "cell_type": "code", "execution_count": 724, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "54\n", "['FYE', u'EIN', u'OrganizationName', u'URL', u'SubmittedOn', u'TaxPeriod', 'tot_rev', 'whistleblower_policy', 'conflict_of_interest_policy', 'records_retention_policy', u'CYTotalExpensesAmt', u'TotalExpensesCurrentYear', u'TotalFunctionalExpenses', u'TotalFunctionalExpensesGrp', u'TotalProgramServiceExpense', u'TotalProgramServiceExpensesAmt', u'CYContributionsGrantsAmt', u'FederatedCampaigns', u'FederatedCampaignsAmt', u'MembershipDues', u'MembershipDuesAmt', u'NetIncomeFromFundraisingEvents', u'NetIncmFromFundraisingEvtGrp', u'FundraisingAmt', u'FundraisingGrossIncomeAmt', u'RelatedOrganizations', u'RelatedOrganizationsAmt', u'GovernmentGrants', u'GovernmentGrantsAmt', u'PYProgramServiceRevenueAmt', u'CYProgramServiceRevenueAmt', u'ProgramServiceRevenue', u'ProgramServiceRevenueCY', u'ProgramServiceRevenueGrp', u'TotalProgramServiceRevenue', u'TotalProgramServiceRevenueAmt', u'CYInvestmentIncomeAmt', u'InvestmentIncome', u'InvestmentIncomeCurrentYear', u'InvestmentIncomeGrp', u'OtherRevenueTotalAmt', u'OtherRevenueMiscGrp', u'TotalOtherRevenue', u'CYOtherRevenueAmt', 'total_expenses', 'program_expenses', 'fundraising', 'program_revenue', 'OtherRevMisc', 'other_revenue_binary', 'complexity', 'SOX_policies', 'SOX_policies_binary', 'SOX_policies_all_binary']\n" ] } ], "source": [ "print len(df.columns)\n", "print df.columns.tolist()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Save DF" ] }, { "cell_type": "code", "execution_count": 727, "metadata": { "collapsed": true }, "outputs": [], "source": [ "df.to_pickle('538 e-file 990s for 2016 donor advisory organizations, v3 (all calc columns).pkl')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Columns to keep to merge with existing data" ] }, { "cell_type": "code", "execution_count": 733, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINOrganizationNameURLSubmittedOnTaxPeriodFYEwhistleblower_policyconflict_of_interest_policyrecords_retention_policySOX_policiesSOX_policies_binarySOX_policies_all_binarytot_revtotal_expensesprogram_expensesprogram_efficiencycomplexity
252030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml2013-12-31201212FY20120112.01.00.025153992100922.01598626.00.7609162
433030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml2013-02-14201112FY20110112.01.00.021612092481310.01715793.00.6914870
78030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201220909349300327_public.xml2012-05-25201012FY20100112.01.00.039905643912643.02930785.00.7490550
333042129889SCHEPENS EYE RESEARCH INSTITUTE INChttps://s3.amazonaws.com/irs-form-990/201502299349301700_public.xml2015-09-21201409FY20141113.01.01.02299552625626262.018485740.00.7213594
12042129889SCHEPENS EYE RESEARCH INSTITUTE INChttps://s3.amazonaws.com/irs-form-990/201442269349304504_public.xml2014-10-28201309FY20131113.01.01.02592344926224173.019280493.00.7352183
\n", "
" ], "text/plain": [ " EIN OrganizationName \\\n", "252 030498214 NEWARK NOW INC \n", "433 030498214 NEWARK NOW INC \n", "78 030498214 NEWARK NOW INC \n", "333 042129889 SCHEPENS EYE RESEARCH INSTITUTE INC \n", "12 042129889 SCHEPENS EYE RESEARCH INSTITUTE INC \n", "\n", " URL \\\n", "252 https://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml \n", "433 https://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml \n", "78 https://s3.amazonaws.com/irs-form-990/201220909349300327_public.xml \n", "333 https://s3.amazonaws.com/irs-form-990/201502299349301700_public.xml \n", "12 https://s3.amazonaws.com/irs-form-990/201442269349304504_public.xml \n", "\n", " SubmittedOn TaxPeriod FYE whistleblower_policy \\\n", "252 2013-12-31 201212 FY2012 0 \n", "433 2013-02-14 201112 FY2011 0 \n", "78 2012-05-25 201012 FY2010 0 \n", "333 2015-09-21 201409 FY2014 1 \n", "12 2014-10-28 201309 FY2013 1 \n", "\n", " conflict_of_interest_policy records_retention_policy SOX_policies \\\n", "252 1 1 2.0 \n", "433 1 1 2.0 \n", "78 1 1 2.0 \n", "333 1 1 3.0 \n", "12 1 1 3.0 \n", "\n", " SOX_policies_binary SOX_policies_all_binary tot_rev total_expenses \\\n", "252 1.0 0.0 2515399 2100922.0 \n", "433 1.0 0.0 2161209 2481310.0 \n", "78 1.0 0.0 3990564 3912643.0 \n", "333 1.0 1.0 22995526 25626262.0 \n", "12 1.0 1.0 25923449 26224173.0 \n", "\n", " program_expenses program_efficiency complexity \n", "252 1598626.0 0.760916 2 \n", "433 1715793.0 0.691487 0 \n", "78 2930785.0 0.749055 0 \n", "333 18485740.0 0.721359 4 \n", "12 19280493.0 0.735218 3 " ] }, "execution_count": 733, "metadata": {}, "output_type": "execute_result" } ], "source": [ "file_list_columns = ['EIN', 'OrganizationName', 'URL', 'SubmittedOn', 'TaxPeriod', 'FYE']\n", "\n", "SOX_columns = ['whistleblower_policy', 'conflict_of_interest_policy', 'records_retention_policy',\n", " 'SOX_policies', 'SOX_policies_binary', 'SOX_policies_all_binary']\n", "\n", "control_columns = ['tot_rev', 'total_expenses', 'program_expenses', 'program_efficiency', 'complexity']\n", "\n", "unneeded_columns = ['CYTotalExpensesAmt', u'TotalExpensesCurrentYear', u'TotalFunctionalExpenses', \n", " 'TotalFunctionalExpensesGrp', u'TotalProgramServiceExpense', u'TotalProgramServiceExpensesAmt',\n", " 'CYContributionsGrantsAmt', u'FederatedCampaigns', u'FederatedCampaignsAmt', u'MembershipDues',\n", " 'MembershipDuesAmt', u'NetIncomeFromFundraisingEvents', u'NetIncmFromFundraisingEvtGrp', \n", " 'FundraisingAmt', 'FundraisingGrossIncomeAmt', u'RelatedOrganizations', \n", " 'RelatedOrganizationsAmt', u'GovernmentGrants', 'GovernmentGrantsAmt', \n", " 'PYProgramServiceRevenueAmt', u'CYProgramServiceRevenueAmt', 'ProgramServiceRevenue', \n", " 'ProgramServiceRevenueCY', u'ProgramServiceRevenueGrp', u'TotalProgramServiceRevenue', \n", " 'TotalProgramServiceRevenueAmt', u'CYInvestmentIncomeAmt', u'InvestmentIncome', \n", " 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', u'OtherRevenueTotalAmt', \n", " 'OtherRevenueMiscGrp', u'TotalOtherRevenue', 'CYOtherRevenueAmt', 'fundraising', \n", " 'program_revenue', 'OtherRevMisc', 'other_revenue_binary']\n", "\n", "cols = file_list_columns+SOX_columns+control_columns\n", "df = df[cols]\n", "df[cols][:5] " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
14 orgs are missing *program_expenses*. Dan and I agreed to count these as '0' (see email 9/5/16)" ] }, { "cell_type": "code", "execution_count": 735, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINFYEtotal_expensesprogram_expensesprogram_efficiency
172132644507FY20147763846.0NaNNaN
404132644507FY20137795021.0NaNNaN
431132644507FY20126237509.0NaNNaN
106132644507FY20115880789.0NaNNaN
72132644507FY20106024086.0NaNNaN
52135579302FY20142023750.0NaNNaN
406135579302FY20134133761.0NaNNaN
500201737542FY2010177901.0NaNNaN
0271317072FY201434400.0NaNNaN
277330837780FY20143330.0NaNNaN
226591951577FY20111690623.0NaNNaN
213591951577FY20101161939.0NaNNaN
495591951577FY20101282149.0NaNNaN
496953553530FY20110.0NaNNaN
\n", "
" ], "text/plain": [ " EIN FYE total_expenses program_expenses program_efficiency\n", "172 132644507 FY2014 7763846.0 NaN NaN\n", "404 132644507 FY2013 7795021.0 NaN NaN\n", "431 132644507 FY2012 6237509.0 NaN NaN\n", "106 132644507 FY2011 5880789.0 NaN NaN\n", "72 132644507 FY2010 6024086.0 NaN NaN\n", "52 135579302 FY2014 2023750.0 NaN NaN\n", "406 135579302 FY2013 4133761.0 NaN NaN\n", "500 201737542 FY2010 177901.0 NaN NaN\n", "0 271317072 FY2014 34400.0 NaN NaN\n", "277 330837780 FY2014 3330.0 NaN NaN\n", "226 591951577 FY2011 1690623.0 NaN NaN\n", "213 591951577 FY2010 1161939.0 NaN NaN\n", "495 591951577 FY2010 1282149.0 NaN NaN\n", "496 953553530 FY2011 0.0 NaN NaN" ] }, "execution_count": 735, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cols2 = ['EIN', 'FYE', 'total_expenses', 'program_expenses', 'program_efficiency']\n", "df[df['program_efficiency'].isnull()][cols2]" ] }, { "cell_type": "code", "execution_count": 737, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "# of columns: 671\n", "# of observations: 538\n" ] }, { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
@documentId@referenceDocumentId@referenceDocumentName@softwareId@softwareVersion@softwareVersionNumAccountantCompileOrReviewAccountantCompileOrReviewBasisAccountantCompileOrReviewIndAccountsPayableAccrExpnssGrpAccountsPayableAccruedExpensesAccountsReceivableAccountsReceivableGrpAcctCompileOrReviewBasisGrpActivitiesConductedPartnershipActivitiesConductedPrtshpIndActivity2Activity3ActivityOrMissionDescActivityOrMissionDescriptionActivityOtherAddressChangeAddressChangeIndAddressPrincipalOfficerUSAdvertisingAdvertisingGrpAllAffiliatesIncludedAllAffiliatesIncludedIndAllOtherContributionsAllOtherContributionsAmtAllOtherExpensesAllOtherExpensesGrpAmendedReturnAmendedReturnIndAnnualDisclosureCoveredPersonsAnnualDisclosureCoveredPrsnIndAuditCommitteeAuditCommitteeIndAuditedFinancialStmtAttIndBackupWthldComplianceIndBalanceSheetAmountsReportedBenefitsPaidToMembersCYBenefitsPaidToMembersPriorYearBenefitsToMembersBenefitsToMembersGrpBooksInCareOfDetailBsnssRltnshpThruFamilyMemberBsnssRltnshpWithOrganizationBusinessRlnWithFamMemIndBusinessRlnWithOfficerEntIndBusinessRlnWithOrgMemIndCYBenefitsPaidToMembersAmtCYContributionsGrantsAmtCYGrantsAndSimilarPaidAmtCYInvestmentIncomeAmtCYOtherExpensesAmtCYOtherRevenueAmtCYProgramServiceRevenueAmtCYRevenuesLessExpensesAmtCYSalariesCompEmpBnftPaidAmtCYTotalExpensesAmtCYTotalFundraisingExpenseAmtCYTotalProfFndrsngExpnsAmtCYTotalRevenueAmtCapStckTrstPrinCurrentFundsCapStkTrPrinCurrentFundsGrpCashNonInterestBearingCashNonInterestBearingGrpChangeToOrgDocumentsIndChangesToOrganizingDocsCntrbtnsRprtdFundraisingEventsCntrctRcvdGreaterThan100KCntCollectionsOfArtCollectionsOfArtIndCompCurrentOfcrDirectorsGrpCompCurrentOfficersDirectorsCompDisqualPersonsCompDisqualPersonsGrpCompensationFromOtherSourcesCompensationFromOtherSrcsIndCompensationProcessCEOCompensationProcessCEOIndCompensationProcessOtherCompensationProcessOtherIndComplianceWithBackupWitholdingConferencesMeetingsConferencesMeetingsGrpConflictOfInterestPolicyConflictOfInterestPolicyIndConservationEasementsConservationEasementsIndConsolidatedAuditFinancialStmtConsolidatedAuditFinclStmtIndContractorCompensationContractorCompensationGrpContriRptFundraisingEventAmtContributionsGrantsCurrentYearContributionsGrantsPriorYearCostOfGoodsSoldCostOfGoodsSoldAmtCreditCounselingCreditCounselingIndDAFExcessBusinessHoldingsIndDLNDecisionsSubjectToApprovaIndDecisionsSubjectToApprovalDeductibleArtContributionIndDeductibleContributionsOfArtDeductibleNonCashContriIndDeductibleNonCashContributionsDeferredRevenueDeferredRevenueGrpDelegationOfManagementDutiesDelegationOfMgmtDutiesIndDepreciationDepletionDepreciationDepletionGrpDescDescribedIn501C3DescribedInSection501c3IndDescriptionDisregardedEntityDisregardedEntityIndDistributionToDonorDistributionToDonorIndDoNotFollowSFAS117DocumentRetentionPolicyDocumentRetentionPolicyIndDoingBusinessAsDoingBusinessAsNameDonatedServicesAndUseFcltsAmtDonorAdvisedFundIndDonorAdvisedFundsEINElectionOfBoardMembersElectionOfBoardMembersIndEmployeeCntEmploymentTaxReturnsFiledEmploymentTaxReturnsFiledIndEngagedInExcessBenefitTransIndEscrowAccountEscrowAccountIndEscrowAccountLiabilityEscrowAccountLiabilityGrpExcessBenefitTransactionExcessBusinessHoldingsExcessBusinessHoldingsIndExpenseExpenseAmtFSAuditedFSAuditedBasisFSAuditedBasisGrpFSAuditedIndFamilyOrBusinessRelationshipFamilyOrBusinessRlnIndFederalGrantAuditPerformedFederalGrantAuditPerformedIndFederalGrantAuditRequiredFederalGrantAuditRequiredIndFederatedCampaignsFederatedCampaignsAmtFeesForServicesAccountingFeesForServicesAccountingGrpFeesForServicesInvstMgmntFeesFeesForServicesLegalFeesForServicesLegalGrpFeesForServicesLobbyingFeesForServicesLobbyingGrpFeesForServicesManagementFeesForServicesManagementGrpFeesForServicesOtherFeesForServicesOtherGrpFeesForServicesProfFundraisingFeesForSrvcInvstMgmntFeesGrpFiledLieu1041FinancialStatementBothFinancialStatementConsolidatedFinancialStatementSeparateFinancialStmtAttachedFollowSFAS117ForeignActivitiesForeignActivitiesIndForeignAddressForeignCountryCdForeignFinancialAccountForeignFinancialAccountIndForeignGrantsForeignGrantsGrpForeignOfficeForeignOfficeIndForm1098CFiledForm1098CFiledIndForm720FiledForm8282FiledCntForm8282PropertyDisposedOfForm8282PropertyDisposedOfIndForm8886TFiledForm8886TFiledIndForm8899FiledForm8899FiledindForm990-TFiledForm990PartVIISectionAForm990PartVIISectionAGrpForm990ProvidedToGoverningBodyForm990ProvidedToGvrnBodyIndForm990TFiledIndFormTypeFormationYrFormerOfcrEmployeesListedIndFormersListedFundraisingActivitiesFundraisingActivitiesIndFundraisingAmtFundraisingDirectExpensesFundraisingDirectExpensesAmtFundraisingEventsFundraisingGrossIncomeAmtFundsToPayPremiumsGainOrLossGainOrLossGrpGamingGamingActivitiesIndGoverningBodyVotingMembersCntGovernmentGrantsGovernmentGrantsAmtGrantAmtGrantToRelatedPersonGrantToRelatedPersonIndGrantsGrantsAndSimilarAmntsCYGrantsAndSimilarAmntsPriorYearGrantsPayableGrantsPayableGrpGrantsToDomesticIndividualsGrantsToDomesticIndividualsGrpGrantsToDomesticOrgsGrantsToDomesticOrgsGrpGrantsToIndividualsGrantsToIndividualsIndGrantsToOrganizationsGrantsToOrganizationsIndGrossAmountSalesAssetsGrossAmountSalesAssetsGrpGrossIncomeFundraisingEventsGrossReceiptsGrossReceiptsAmtGrossRentsGrossRentsGrpGrossSalesOfInventoryGrossSalesOfInventoryAmtGroupExemptionNumGroupExemptionNumberGroupReturnForAffiliatesGroupReturnForAffiliatesIndHospitalIRPDocumentCntIRPDocumentW2GCntIncludeFIN48FootnoteIndIncmFromInvestBondProceedsGrpIncomeFromInvestBondProceedsIndependentAuditFinancialStmtIndependentAuditFinclStmtIndIndependentVotingMemberCntIndivRcvdGreaterThan100KCntIndoorTanningServicesIndoorTanningServicesIndInfoInScheduleOPartIIIInfoInScheduleOPartIIIIndInfoInScheduleOPartIXInfoInScheduleOPartIXIndInfoInScheduleOPartVInfoInScheduleOPartVIInfoInScheduleOPartVIIInfoInScheduleOPartVIIIInfoInScheduleOPartVIIIIndInfoInScheduleOPartVIIIndInfoInScheduleOPartVIIndInfoInScheduleOPartVIndInfoInScheduleOPartXInfoInScheduleOPartXIInfoInScheduleOPartXIIInfoInScheduleOPartXIIIndInfoInScheduleOPartXIIndInfoInScheduleOPartXIndInformationTechnologyInformationTechnologyGrpInitialReturnInsuranceInsuranceGrpIntangibleAssetsIntangibleAssetsGrpInterestInterestGrpInventoriesForSaleOrUseInventoriesForSaleOrUseGrpInvestTaxExemptBondsInvestTaxExemptBondsIndInvestmentExpenseAmtInvestmentInJointVentureInvestmentInJointVentureIndInvestmentIncomeInvestmentIncomeCurrentYearInvestmentIncomeGrpInvestmentIncomePriorYearInvestmentsOtherSecuritiesInvestmentsOtherSecuritiesGrpInvestmentsProgramRelatedInvestmentsProgramRelatedGrpInvestmentsPubTradedSecGrpInvestmentsPubTradedSecuritiesIsAvailableIsElectronicJointCostsJointCostsIndLandBldgEquipAccumDeprecAmtLandBldgEquipBasisNetGrpLandBldgEquipCostOrOtherBssAmtLandBldgEquipmentAccumDeprecLandBuildingsEquipmentBasisLandBuildingsEquipmentBasisNetLastUpdatedLegalDomicileStateCdLessCostOthBasisSalesExpensesLessCostOthBasisSalesExpnssGrpLessRentalExpensesLessRentalExpensesGrpLicensedMoreThanOneStateLicensedMoreThanOneStateIndLoanOutstandingIndLoanToOfficerOrDQPLoansFromOfficersDirectorsLoansFromOfficersDirectorsGrpLobbyingActivitiesLobbyingActivitiesIndLocalChaptersLocalChaptersIndMaterialDiversionOrMisuseMaterialDiversionOrMisuseIndMembersOrStockholdersMembersOrStockholdersIndMembershipDuesMembershipDuesAmtMethodOfAccountingAccrualMethodOfAccountingAccrualIndMethodOfAccountingCashMethodOfAccountingCashIndMethodOfAccountingOtherMethodOfAccountingOtherIndMinutesOfCommitteesMinutesOfCommitteesIndMinutesOfGoverningBodyMinutesOfGoverningBodyIndMiscellaneousRevenueMiscellaneousRevenueGrpMissionDescMissionDescriptionMoreThan5000KToIndividualsMoreThan5000KToIndividualsIndMoreThan5000KToOrgIndMoreThan5000KToOrganizationsMortNotesPyblSecuredInvestPropMortgNotesPyblScrdInvstPropGrpNameOfForeignCountryNameOfPrincipalOfficerBusinessNameOfPrincipalOfficerPersonNbrIndependentVotingMembersNbrVotingGoverningBodyMembersNbrVotingMembersGoverningBodyNetAssetsOrFundBalancesBOYNetAssetsOrFundBalancesBOYAmtNetAssetsOrFundBalancesEOYNetAssetsOrFundBalancesEOYAmtNetGainOrLossInvestmentsNetGainOrLossInvestmentsGrpNetIncmFromFundraisingEvtGrpNetIncomeFromFundraisingEventsNetIncomeFromGamingNetIncomeFromGamingGrpNetIncomeOrLossNetIncomeOrLossGrpNetRentalIncomeOrLossNetRentalIncomeOrLossGrpNetUnrelatedBusTxblIncmAmtNetUnrelatedBusinessTxblIncomeNetUnrlzdGainsLossesInvstAmtNoListedPersonsCompensatedNoListedPersonsCompensatedIndNonDeductibleContributionsNonDeductibleDisclosureNoncashContributionsNoncashContributionsAmtNondeductibleContriDisclIndNondeductibleContributionsIndNumberFormsTransmittedWith1096NumberIndependentVotingMembersNumberIndividualsGT100KNumberOf8282FiledNumberOfContractorsGT100KNumberOfEmployeesNumberW2GIncludedObjectIdOccupancyOccupancyGrpOfficeExpensesOfficeExpensesGrpOfficerEntityWithBsnssRltnshpOfficerMailingAddressOfficerMailingAddressIndOnBehalfOfIssuerOnBehalfOfIssuerIndOperateHospitalIndOrgDoesNotFollowSFAS117IndOrgFiledInLieuOfForm1041IndOrganization501cOrganization501c3Organization501c3IndOrganization501cIndOrganizationFollowsSFAS117IndOrganizationNameOthNotesLoansReceivableNetGrpOtherAssetsTotalOtherAssetsTotalGrpOtherChangesInNetAssetsAmtOtherEmployeeBenefitsOtherEmployeeBenefitsGrpOtherExpensePriorYearOtherExpensesOtherExpensesCurrentYearOtherExpensesGrpOtherExplainInSchOOtherIndOtherLiabilitiesOtherLiabilitiesGrpOtherNotesLoansReceivableNetOtherRevenueCurrentYearOtherRevenueMiscOtherRevenueMiscGrpOtherRevenuePriorYearOtherRevenueTotalAmtOtherSalariesAndWagesOtherSalariesAndWagesGrpOtherWebsiteOtherWebsiteIndOwnWebsiteOwnWebsiteIndPYBenefitsPaidToMembersAmtPYContributionsGrantsAmtPYExcessBenefitTransIndPYGrantsAndSimilarPaidAmtPYInvestmentIncomeAmtPYOtherExpensesAmtPYOtherRevenueAmtPYProgramServiceRevenueAmtPYRevenuesLessExpensesAmtPYSalariesCompEmpBnftPaidAmtPYTotalExpensesAmtPYTotalProfFndrsngExpnsAmtPYTotalRevenueAmtPaidInCapSrplsLandBldgEqpFundPartialLiquidationPartialLiquidationIndPayPremiumsPrsnlBnftCntrctIndPaymentsToAffiliatesPaymentsToAffiliatesGrpPayrollTaxesPayrollTaxesGrpPdInCapSrplsLandBldgEqpFundGrpPensionPlanContributionsPensionPlanContributionsGrpPermanentlyRestrictedNetAssetsPermanentlyRstrNetAssetsGrpPledgesAndGrantsReceivablePledgesAndGrantsReceivableGrpPoliciesReferenceChaptersPoliciesReferenceChaptersIndPoliticalActivitiesPoliticalCampaignActyIndPremiumsPaidPrepaidExpensesDeferredChargesPrepaidExpensesDefrdChargesGrpPrincipalOfficerNmPriorExcessBenefitTransactionPriorPeriodAdjustmentsAmtProfessionalFundraisingProfessionalFundraisingIndProgSrvcAccomActy2GrpProgSrvcAccomActy3GrpProgSrvcAccomActyOtherGrpProgramServiceRevenueProgramServiceRevenueCYProgramServiceRevenueGrpProgramServiceRevenuePriorYearProhibitedTaxShelterTransProhibitedTaxShelterTransIndPymtTravelEntrtnmntPubOfclGrpQuidProQuoContriDisclIndQuidProQuoContributionsQuidProQuoContributionsIndQuidProQuoDisclosureRcvFndsToPayPrsnlBnftCntrctIndRcvblFromDisqualifiedPrsnGrpReceivablesFromDisqualPersonsReceivablesFromOfficersEtcReceivablesFromOfficersEtcGrpReconcilationDonatedServicesReconcilationInvestExpensesReconcilationOtherChangesReconcilationPriorAdjustmentReconcilationRevenueExpensesReconcilationRevenueExpnssAmtReconciliationUnrealizedInvestRegularMonitoringEnforcementRegularMonitoringEnfrcIndRelatedEntityRelatedEntityIndRelatedOrgControlledEntityRelatedOrganizationCtrlEntIndRelatedOrganizationsRelatedOrganizationsAmtRentalIncomeOrLossRentalIncomeOrLossGrpReportFin48FootnoteReportInvestOthSecuritiesReportInvestmentsOtherSecIndReportLandBldgEquipReportLandBuildingEquipmentIndReportOtherAssetsReportOtherAssetsIndReportOtherLiabilitiesReportOtherLiabilitiesIndReportProgRelInvestReportProgramRelatedInvstIndRetainedEarningsEndowmentEtcRevenueRevenueAmtRevenuesLessExpensesCYRevenuesLessExpensesPriorYearRoyaltiesRoyaltiesGrpRoyaltiesRevenueRoyaltiesRevenueGrpRtnEarnEndowmentIncmOthFndsGrpSalariesEtcCurrentYearSalariesEtcPriorYearSavingsAndTempCashInvestmentsSavingsAndTempCashInvstGrpScheduleBRequiredScheduleBRequiredIndScheduleJRequiredScheduleJRequiredIndScheduleORequiredScheduleORequiredIndSchoolSchoolOperatingIndSignificantChangeSignificantChangeIndSignificantNewProgramServicesSignificantNewProgramSrvcIndStateLegalDomicileStatesWhereCopyOfReturnIsFiledStatesWhereCopyOfReturnIsFldCdSubjectToProxyTaxSubjectToProxyTaxIndSubmittedOnTaxExemptBondLiabilitiesTaxExemptBondLiabilitiesGrpTaxExemptBondsTaxExemptBondsIndTaxPeriodTaxableDistributionsTaxableDistributionsIndTaxablePartyNotificationTaxablePartyNotificationIndTempOrPermanentEndowmentsIndTemporarilyRestrictedNetAssetsTemporarilyRstrNetAssetsGrpTermOrPermanentEndowmentsTerminateOperationsIndTerminatedTerminationOrContractionTheBooksAreInCareOfTotLiabNetAssetsFundBalanceGrpTotReportableCompRltdOrgAmtTotalAssetsTotalAssetsBOYTotalAssetsBOYAmtTotalAssetsEOYTotalAssetsEOYAmtTotalAssetsGrpTotalCompGT150KTotalCompGreaterThan150KIndTotalContributionsTotalContributionsAmtTotalEmployeeCntTotalExpensesCurrentYearTotalExpensesPriorYearTotalFunctionalExpensesTotalFunctionalExpensesGrpTotalFundrsngExpCurrentYearTotalGrossUBITotalGrossUBIAmtTotalJointCostsTotalJointCostsGrpTotalLiabNetAssetsFundBalancesTotalLiabilitiesTotalLiabilitiesBOYTotalLiabilitiesBOYAmtTotalLiabilitiesEOYTotalLiabilitiesEOYAmtTotalLiabilitiesGrpTotalNbrEmployeesTotalNbrVolunteersTotalNetAssetsFundBalanceGrpTotalNetAssetsFundBalancesTotalOfOtherProgramServiceExpTotalOfOtherProgramServiceGrntTotalOfOtherProgramServiceRevTotalOthProgramServiceRevGrpTotalOthProgramServiceRevenueTotalOtherCompensationTotalOtherCompensationAmtTotalOtherProgSrvcExpenseAmtTotalOtherProgSrvcGrantAmtTotalOtherProgSrvcRevenueAmtTotalOtherRevenueTotalProfFundrsngExpCYTotalProfFundrsngExpPriorYearTotalProgramServiceExpenseTotalProgramServiceExpensesAmtTotalProgramServiceRevenueTotalProgramServiceRevenueAmtTotalReportableCompFrmRltdOrgsTotalReportableCompFromOrgTotalReportableCompFromOrgAmtTotalRevenueTotalRevenueCurrentYearTotalRevenueGrpTotalRevenuePriorYearTotalVolunteersCntTransactionRelatedEntityTransactionWithControlEntIndTransfersToExemptNonChrtblOrgTravelTravelEntrtnmntPublicOfficialsTravelGrpTrnsfrExmptNonChrtblRltdOrgIndTypeOfOrganizationCorpIndTypeOfOrganizationCorporationTypeOfOrganizationOtherTypeOfOrganizationOtherIndTypeOfOrganizationTrustTypeOfOrganizationTrustIndURLUSAddressUnrelatedBusIncmOverLimitIndUnrelatedBusinessIncomeUnrestrictedNetAssetsUnrestrictedNetAssetsGrpUnsecuredNotesLoansPayableUnsecuredNotesLoansPayableGrpUponRequestUponRequestIndVotingMembersGoverningBodyCntVotingMembersIndependentCntWebSiteWebsiteAddressTxtWhistleblowerPolicyWhistleblowerPolicyIndWrittenPolicyOrProcedureWrittenPolicyOrProcedureIndYearFormation_id
0RetDoc1RetDoc2NaNNaNNaNNaNNaNNaNfalseNaNNaNNaNNaNNaNNaNfalseNaNNaNRAISE MONIES TO CONSTRUCT A FACILITY FOR YEAR-ROUND RECREATION PROGRAM DEDICATED TO CHILDREN AND ADULTS WITH SPECIAL NEEDS WHO RESIDE IN THE TOWN OF HEMPSTEAD.NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN{u'PhoneNum': u'5164316946', u'USAddress': {u'CityNm': u'LIDO BEACH', u'StateAbbreviationCd': u'NY', u'ZIPCd': u'11561', u'AddressLine1Txt': u'630 LIDO BLVD'}, u'BusinessName': {u'BusinessNameLine1Txt': u'ANCHOR BUILDING FUND INC'}}NaNNaNfalsefalsefalse000034400325000029060003440000325000NaNNaNNaN{u'EOYAmt': u'290600'}falseNaNNaNNaNNaNfalseNaNNaNNaNNaNNaNfalseNaNfalseNaNfalseNaNNaNNaNNaNfalseNaNfalseNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNfalseNaN93493316047805falseNaNfalseNaNfalseNaNNaNNaNNaNfalseNaNNaNTHE ORGANIZATIONS EXEMPT PURPOSE WAS TO RAISE MONIES TO CONSTRUCT A FACILTIY TO OPERATE ITS YEAR-ROUND PROGRAM.NaN{u'#text': u'true', u'@referenceDocumentId': u'RetDoc3'}NaNNaNfalseNaNNaNNaNNaNfalseNaNNaNNaNfalseNaN271317072NaNfalse0NaNNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNfalseNaNNaNNaNNaNNaNNaNNaN{u'TotalAmt': u'9400', u'ManagementAndGeneralAmt': u'9400'}NaNNaN{u'TotalAmt': u'25000', u'ManagementAndGeneralAmt': u'25000'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNNaNNaNfalseNaNNaNNaNfalseNaNfalseNaNNaNNaNfalseNaNNaNNaNfalseNaNNaN[{u'PersonNm': u'JOHN MALLOY', u'TitleTxt': u'PRESIDENT', u'AverageHoursPerWeekRt': u'2.00', u'ReportableCompFromRltdOrgAmt': u'0', u'OtherCompensationAmt': u'0', u'OfficerInd': u'X', u'ReportableCompFromOrgAmt': u'0'}, {u'PersonNm': u'ANNA DRAGO...NaNfalseNaN9902009falseNaNNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaNfalse2NaNNaNNaNNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNfalseNaNNaNNaNNaN325000NaNNaNNaNNaNNaNNaNNaNfalseNaN20falseNaNNaNNaNfalse2NaNNaNfalseNaNXNaNNaNNaNNaNNaNNaNNaNNaNXNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNTrueTrueNaNNaNNaNNaNNaNNaNNaNNaN2016-03-21T17:23:53NYNaNNaNNaNNaNNaNNaNfalseNaNNaNNaNNaNfalseNaNfalseNaNtrueNaNfalseNaNNaNNaNNaNNaNXNaNNaNNaNfalseNaNfalseNaNNaNRAISE MONIES TO CONSTRUCT A FACILITY FOR YEAR-ROUND RECREATION PROGRAM DEDICATED TO CHILDREN AND ADULTS WITH SPECIAL NEEDS WHO RESIDE IN THE TOWN OF HEMPSTEAD.NaNNaNfalsefalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN290600NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNXNaNNaNNaNNaNNaNfalseNaNNaNNaNNaNNaNNaNNaN201503169349304780NaNNaNNaNNaNNaNNaNfalseNaNNaNfalseNaNNaNNaNNaNXNaNXANCHOR BUILDING FUND INCNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN{u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000', u'Desc': u'SETTLEMENT'}NaN325000NaNNaNNaNNaNNaNNaNNaNNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalsefalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNNaNNaNJOHN MALLOYNaNNaNNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNNaNNaNfalseNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaN290600NaNNaNNaNNaNfalseNaNfalseNaNNaNNaNNaNNaNNaNfalseNaNfalseNaNfalseNaNfalseNaNfalseNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNfalseNaNfalseNaNtrueNaNfalseNaNtrueNaNfalseNaNNaNNYNaNfalse2016-01-30NaNNaNNaNfalse201412NaNNaNNaNfalsefalseNaNNaNNaNfalseNaNNaNNaN{u'BOYAmt': u'0', u'EOYAmt': u'290600'}NaNNaNNaNNaNNaN290600{u'BOYAmt': u'0', u'EOYAmt': u'290600'}NaNfalseNaNNaN0NaNNaNNaN{u'TotalAmt': u'34400', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'34400', u'FundraisingAmt': u'0'}NaNNaN0NaNNaNNaNNaNNaNNaNNaN0{u'BOYAmt': u'0', u'EOYAmt': u'0'}NaNNaN{u'BOYAmt': u'0', u'EOYAmt': u'290600'}NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN{u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000'}NaNNaNNaNNaNNaNNaNNaNNaNfalseXNaNNaNNaNNaNNaNhttps://s3.amazonaws.com/irs-form-990/201503169349304780_public.xml{u'CityNm': u'LIDO BEACH', u'StateAbbreviationCd': u'NY', u'ZIPCd': u'11561', u'AddressLine1Txt': u'630 LIDO BLVD'}falseNaNNaN{u'EOYAmt': u'290600'}NaNNaNNaNX22NaNWWW.CAMPANCHOR.ORGNaNfalseNaNNaNNaN57cce7823ffc5a55fe7f54b5
\n", "
" ], "text/plain": [ " @documentId @referenceDocumentId @referenceDocumentName @softwareId \\\n", "0 RetDoc1 RetDoc2 NaN NaN \n", "\n", " @softwareVersion @softwareVersionNum AccountantCompileOrReview \\\n", "0 NaN NaN NaN \n", "\n", " AccountantCompileOrReviewBasis AccountantCompileOrReviewInd \\\n", "0 NaN false \n", "\n", " AccountsPayableAccrExpnssGrp AccountsPayableAccruedExpenses \\\n", "0 NaN NaN \n", "\n", " AccountsReceivable AccountsReceivableGrp AcctCompileOrReviewBasisGrp \\\n", "0 NaN NaN NaN \n", "\n", " ActivitiesConductedPartnership ActivitiesConductedPrtshpInd Activity2 \\\n", "0 NaN false NaN \n", "\n", " Activity3 \\\n", "0 NaN \n", "\n", " ActivityOrMissionDesc \\\n", "0 RAISE MONIES TO CONSTRUCT A FACILITY FOR YEAR-ROUND RECREATION PROGRAM DEDICATED TO CHILDREN AND ADULTS WITH SPECIAL NEEDS WHO RESIDE IN THE TOWN OF HEMPSTEAD. \n", "\n", " ActivityOrMissionDescription ActivityOther AddressChange AddressChangeInd \\\n", "0 NaN NaN NaN NaN \n", "\n", " AddressPrincipalOfficerUS Advertising AdvertisingGrp AllAffiliatesIncluded \\\n", "0 NaN NaN NaN NaN \n", "\n", " AllAffiliatesIncludedInd AllOtherContributions AllOtherContributionsAmt \\\n", "0 NaN NaN NaN \n", "\n", " AllOtherExpenses AllOtherExpensesGrp AmendedReturn AmendedReturnInd \\\n", "0 NaN NaN NaN NaN \n", "\n", " AnnualDisclosureCoveredPersons AnnualDisclosureCoveredPrsnInd \\\n", "0 NaN NaN \n", "\n", " AuditCommittee AuditCommitteeInd AuditedFinancialStmtAttInd \\\n", "0 NaN NaN NaN \n", "\n", " BackupWthldComplianceInd BalanceSheetAmountsReported \\\n", "0 NaN NaN \n", "\n", " BenefitsPaidToMembersCY BenefitsPaidToMembersPriorYear BenefitsToMembers \\\n", "0 NaN NaN NaN \n", "\n", " BenefitsToMembersGrp \\\n", "0 NaN \n", "\n", " BooksInCareOfDetail \\\n", "0 {u'PhoneNum': u'5164316946', u'USAddress': {u'CityNm': u'LIDO BEACH', u'StateAbbreviationCd': u'NY', u'ZIPCd': u'11561', u'AddressLine1Txt': u'630 LIDO BLVD'}, u'BusinessName': {u'BusinessNameLine1Txt': u'ANCHOR BUILDING FUND INC'}} \n", "\n", " BsnssRltnshpThruFamilyMember BsnssRltnshpWithOrganization \\\n", "0 NaN NaN \n", "\n", " BusinessRlnWithFamMemInd BusinessRlnWithOfficerEntInd \\\n", "0 false false \n", "\n", " BusinessRlnWithOrgMemInd CYBenefitsPaidToMembersAmt \\\n", "0 false 0 \n", "\n", " CYContributionsGrantsAmt CYGrantsAndSimilarPaidAmt CYInvestmentIncomeAmt \\\n", "0 0 0 0 \n", "\n", " CYOtherExpensesAmt CYOtherRevenueAmt CYProgramServiceRevenueAmt \\\n", "0 34400 325000 0 \n", "\n", " CYRevenuesLessExpensesAmt CYSalariesCompEmpBnftPaidAmt CYTotalExpensesAmt \\\n", "0 290600 0 34400 \n", "\n", " CYTotalFundraisingExpenseAmt CYTotalProfFndrsngExpnsAmt CYTotalRevenueAmt \\\n", "0 0 0 325000 \n", "\n", " CapStckTrstPrinCurrentFunds CapStkTrPrinCurrentFundsGrp \\\n", "0 NaN NaN \n", "\n", " CashNonInterestBearing CashNonInterestBearingGrp ChangeToOrgDocumentsInd \\\n", "0 NaN {u'EOYAmt': u'290600'} false \n", "\n", " ChangesToOrganizingDocs CntrbtnsRprtdFundraisingEvents \\\n", "0 NaN NaN \n", "\n", " CntrctRcvdGreaterThan100KCnt CollectionsOfArt CollectionsOfArtInd \\\n", "0 NaN NaN false \n", "\n", " CompCurrentOfcrDirectorsGrp CompCurrentOfficersDirectors CompDisqualPersons \\\n", "0 NaN NaN NaN \n", "\n", " CompDisqualPersonsGrp CompensationFromOtherSources \\\n", "0 NaN NaN \n", "\n", " CompensationFromOtherSrcsInd CompensationProcessCEO \\\n", "0 false NaN \n", "\n", " CompensationProcessCEOInd CompensationProcessOther \\\n", "0 false NaN \n", "\n", " CompensationProcessOtherInd ComplianceWithBackupWitholding \\\n", "0 false NaN \n", "\n", " ConferencesMeetings ConferencesMeetingsGrp ConflictOfInterestPolicy \\\n", "0 NaN NaN NaN \n", "\n", " ConflictOfInterestPolicyInd ConservationEasements ConservationEasementsInd \\\n", "0 false NaN false \n", "\n", " ConsolidatedAuditFinancialStmt ConsolidatedAuditFinclStmtInd \\\n", "0 NaN false \n", "\n", " ContractorCompensation ContractorCompensationGrp \\\n", "0 NaN NaN \n", "\n", " ContriRptFundraisingEventAmt ContributionsGrantsCurrentYear \\\n", "0 NaN NaN \n", "\n", " ContributionsGrantsPriorYear CostOfGoodsSold CostOfGoodsSoldAmt \\\n", "0 NaN NaN NaN \n", "\n", " CreditCounseling CreditCounselingInd DAFExcessBusinessHoldingsInd \\\n", "0 NaN false NaN \n", "\n", " DLN DecisionsSubjectToApprovaInd DecisionsSubjectToApproval \\\n", "0 93493316047805 false NaN \n", "\n", " DeductibleArtContributionInd DeductibleContributionsOfArt \\\n", "0 false NaN \n", "\n", " DeductibleNonCashContriInd DeductibleNonCashContributions DeferredRevenue \\\n", "0 false NaN NaN \n", "\n", " DeferredRevenueGrp DelegationOfManagementDuties DelegationOfMgmtDutiesInd \\\n", "0 NaN NaN false \n", "\n", " DepreciationDepletion DepreciationDepletionGrp \\\n", "0 NaN NaN \n", "\n", " Desc \\\n", "0 THE ORGANIZATIONS EXEMPT PURPOSE WAS TO RAISE MONIES TO CONSTRUCT A FACILTIY TO OPERATE ITS YEAR-ROUND PROGRAM. \n", "\n", " DescribedIn501C3 DescribedInSection501c3Ind \\\n", "0 NaN {u'#text': u'true', u'@referenceDocumentId': u'RetDoc3'} \n", "\n", " Description DisregardedEntity DisregardedEntityInd DistributionToDonor \\\n", "0 NaN NaN false NaN \n", "\n", " DistributionToDonorInd DoNotFollowSFAS117 DocumentRetentionPolicy \\\n", "0 NaN NaN NaN \n", "\n", " DocumentRetentionPolicyInd DoingBusinessAs DoingBusinessAsName \\\n", "0 false NaN NaN \n", "\n", " DonatedServicesAndUseFcltsAmt DonorAdvisedFundInd DonorAdvisedFunds \\\n", "0 NaN false NaN \n", "\n", " EIN ElectionOfBoardMembers ElectionOfBoardMembersInd EmployeeCnt \\\n", "0 271317072 NaN false 0 \n", "\n", " EmploymentTaxReturnsFiled EmploymentTaxReturnsFiledInd \\\n", "0 NaN NaN \n", "\n", " EngagedInExcessBenefitTransInd EscrowAccount EscrowAccountInd \\\n", "0 false NaN NaN \n", "\n", " EscrowAccountLiability EscrowAccountLiabilityGrp ExcessBenefitTransaction \\\n", "0 NaN NaN NaN \n", "\n", " ExcessBusinessHoldings ExcessBusinessHoldingsInd Expense ExpenseAmt \\\n", "0 NaN NaN NaN NaN \n", "\n", " FSAudited FSAuditedBasis FSAuditedBasisGrp FSAuditedInd \\\n", "0 NaN NaN NaN false \n", "\n", " FamilyOrBusinessRelationship FamilyOrBusinessRlnInd \\\n", "0 NaN false \n", "\n", " FederalGrantAuditPerformed FederalGrantAuditPerformedInd \\\n", "0 NaN NaN \n", "\n", " FederalGrantAuditRequired FederalGrantAuditRequiredInd FederatedCampaigns \\\n", "0 NaN NaN NaN \n", "\n", " FederatedCampaignsAmt FeesForServicesAccounting \\\n", "0 NaN NaN \n", "\n", " FeesForServicesAccountingGrp \\\n", "0 {u'TotalAmt': u'9400', u'ManagementAndGeneralAmt': u'9400'} \n", "\n", " FeesForServicesInvstMgmntFees FeesForServicesLegal \\\n", "0 NaN NaN \n", "\n", " FeesForServicesLegalGrp \\\n", "0 {u'TotalAmt': u'25000', u'ManagementAndGeneralAmt': u'25000'} \n", "\n", " FeesForServicesLobbying FeesForServicesLobbyingGrp \\\n", "0 NaN NaN \n", "\n", " FeesForServicesManagement FeesForServicesManagementGrp FeesForServicesOther \\\n", "0 NaN NaN NaN \n", "\n", " FeesForServicesOtherGrp FeesForServicesProfFundraising \\\n", "0 NaN NaN \n", "\n", " FeesForSrvcInvstMgmntFeesGrp FiledLieu1041 FinancialStatementBoth \\\n", "0 NaN NaN NaN \n", "\n", " FinancialStatementConsolidated FinancialStatementSeparate \\\n", "0 NaN NaN \n", "\n", " FinancialStmtAttached FollowSFAS117 ForeignActivities ForeignActivitiesInd \\\n", "0 NaN NaN NaN false \n", "\n", " ForeignAddress ForeignCountryCd ForeignFinancialAccount \\\n", "0 NaN NaN NaN \n", "\n", " ForeignFinancialAccountInd ForeignGrants ForeignGrantsGrp ForeignOffice \\\n", "0 false NaN NaN NaN \n", "\n", " ForeignOfficeInd Form1098CFiled Form1098CFiledInd Form720Filed \\\n", "0 false NaN false NaN \n", "\n", " Form8282FiledCnt Form8282PropertyDisposedOf Form8282PropertyDisposedOfInd \\\n", "0 NaN NaN false \n", "\n", " Form8886TFiled Form8886TFiledInd Form8899Filed Form8899Filedind \\\n", "0 NaN NaN NaN false \n", "\n", " Form990-TFiled Form990PartVIISectionA \\\n", "0 NaN NaN \n", "\n", " Form990PartVIISectionAGrp \\\n", "0 [{u'PersonNm': u'JOHN MALLOY', u'TitleTxt': u'PRESIDENT', u'AverageHoursPerWeekRt': u'2.00', u'ReportableCompFromRltdOrgAmt': u'0', u'OtherCompensationAmt': u'0', u'OfficerInd': u'X', u'ReportableCompFromOrgAmt': u'0'}, {u'PersonNm': u'ANNA DRAGO... \n", "\n", " Form990ProvidedToGoverningBody Form990ProvidedToGvrnBodyInd \\\n", "0 NaN false \n", "\n", " Form990TFiledInd FormType FormationYr FormerOfcrEmployeesListedInd \\\n", "0 NaN 990 2009 false \n", "\n", " FormersListed FundraisingActivities FundraisingActivitiesInd FundraisingAmt \\\n", "0 NaN NaN false NaN \n", "\n", " FundraisingDirectExpenses FundraisingDirectExpensesAmt FundraisingEvents \\\n", "0 NaN NaN NaN \n", "\n", " FundraisingGrossIncomeAmt FundsToPayPremiums GainOrLoss GainOrLossGrp \\\n", "0 NaN NaN NaN NaN \n", "\n", " Gaming GamingActivitiesInd GoverningBodyVotingMembersCnt GovernmentGrants \\\n", "0 NaN false 2 NaN \n", "\n", " GovernmentGrantsAmt GrantAmt GrantToRelatedPerson GrantToRelatedPersonInd \\\n", "0 NaN NaN NaN false \n", "\n", " Grants GrantsAndSimilarAmntsCY GrantsAndSimilarAmntsPriorYear GrantsPayable \\\n", "0 NaN NaN NaN NaN \n", "\n", " GrantsPayableGrp GrantsToDomesticIndividuals GrantsToDomesticIndividualsGrp \\\n", "0 NaN NaN NaN \n", "\n", " GrantsToDomesticOrgs GrantsToDomesticOrgsGrp GrantsToIndividuals \\\n", "0 NaN NaN NaN \n", "\n", " GrantsToIndividualsInd GrantsToOrganizations GrantsToOrganizationsInd \\\n", "0 false NaN false \n", "\n", " GrossAmountSalesAssets GrossAmountSalesAssetsGrp \\\n", "0 NaN NaN \n", "\n", " GrossIncomeFundraisingEvents GrossReceipts GrossReceiptsAmt GrossRents \\\n", "0 NaN NaN 325000 NaN \n", "\n", " GrossRentsGrp GrossSalesOfInventory GrossSalesOfInventoryAmt \\\n", "0 NaN NaN NaN \n", "\n", " GroupExemptionNum GroupExemptionNumber GroupReturnForAffiliates \\\n", "0 NaN NaN NaN \n", "\n", " GroupReturnForAffiliatesInd Hospital IRPDocumentCnt IRPDocumentW2GCnt \\\n", "0 false NaN 2 0 \n", "\n", " IncludeFIN48FootnoteInd IncmFromInvestBondProceedsGrp \\\n", "0 false NaN \n", "\n", " IncomeFromInvestBondProceeds IndependentAuditFinancialStmt \\\n", "0 NaN NaN \n", "\n", " IndependentAuditFinclStmtInd IndependentVotingMemberCnt \\\n", "0 false 2 \n", "\n", " IndivRcvdGreaterThan100KCnt IndoorTanningServices IndoorTanningServicesInd \\\n", "0 NaN NaN false \n", "\n", " InfoInScheduleOPartIII InfoInScheduleOPartIIIInd InfoInScheduleOPartIX \\\n", "0 NaN X NaN \n", "\n", " InfoInScheduleOPartIXInd InfoInScheduleOPartV InfoInScheduleOPartVI \\\n", "0 NaN NaN NaN \n", "\n", " InfoInScheduleOPartVII InfoInScheduleOPartVIII InfoInScheduleOPartVIIIInd \\\n", "0 NaN NaN NaN \n", "\n", " InfoInScheduleOPartVIIInd InfoInScheduleOPartVIInd InfoInScheduleOPartVInd \\\n", "0 NaN X NaN \n", "\n", " InfoInScheduleOPartX InfoInScheduleOPartXI InfoInScheduleOPartXII \\\n", "0 NaN NaN NaN \n", "\n", " InfoInScheduleOPartXIIInd InfoInScheduleOPartXIInd InfoInScheduleOPartXInd \\\n", "0 NaN NaN NaN \n", "\n", " InformationTechnology InformationTechnologyGrp InitialReturn Insurance \\\n", "0 NaN NaN NaN NaN \n", "\n", " InsuranceGrp IntangibleAssets IntangibleAssetsGrp Interest InterestGrp \\\n", "0 NaN NaN NaN NaN NaN \n", "\n", " InventoriesForSaleOrUse InventoriesForSaleOrUseGrp InvestTaxExemptBonds \\\n", "0 NaN NaN NaN \n", "\n", " InvestTaxExemptBondsInd InvestmentExpenseAmt InvestmentInJointVenture \\\n", "0 NaN NaN NaN \n", "\n", " InvestmentInJointVentureInd InvestmentIncome InvestmentIncomeCurrentYear \\\n", "0 false NaN NaN \n", "\n", " InvestmentIncomeGrp InvestmentIncomePriorYear InvestmentsOtherSecurities \\\n", "0 NaN NaN NaN \n", "\n", " InvestmentsOtherSecuritiesGrp InvestmentsProgramRelated \\\n", "0 NaN NaN \n", "\n", " InvestmentsProgramRelatedGrp InvestmentsPubTradedSecGrp \\\n", "0 NaN NaN \n", "\n", " InvestmentsPubTradedSecurities IsAvailable IsElectronic JointCosts \\\n", "0 NaN True True NaN \n", "\n", " JointCostsInd LandBldgEquipAccumDeprecAmt LandBldgEquipBasisNetGrp \\\n", "0 NaN NaN NaN \n", "\n", " LandBldgEquipCostOrOtherBssAmt LandBldgEquipmentAccumDeprec \\\n", "0 NaN NaN \n", "\n", " LandBuildingsEquipmentBasis LandBuildingsEquipmentBasisNet \\\n", "0 NaN NaN \n", "\n", " LastUpdated LegalDomicileStateCd LessCostOthBasisSalesExpenses \\\n", "0 2016-03-21T17:23:53 NY NaN \n", "\n", " LessCostOthBasisSalesExpnssGrp LessRentalExpenses LessRentalExpensesGrp \\\n", "0 NaN NaN NaN \n", "\n", " LicensedMoreThanOneState LicensedMoreThanOneStateInd LoanOutstandingInd \\\n", "0 NaN NaN false \n", "\n", " LoanToOfficerOrDQP LoansFromOfficersDirectors LoansFromOfficersDirectorsGrp \\\n", "0 NaN NaN NaN \n", "\n", " LobbyingActivities LobbyingActivitiesInd LocalChapters LocalChaptersInd \\\n", "0 NaN false NaN false \n", "\n", " MaterialDiversionOrMisuse MaterialDiversionOrMisuseInd \\\n", "0 NaN true \n", "\n", " MembersOrStockholders MembersOrStockholdersInd MembershipDues \\\n", "0 NaN false NaN \n", "\n", " MembershipDuesAmt MethodOfAccountingAccrual MethodOfAccountingAccrualInd \\\n", "0 NaN NaN NaN \n", "\n", " MethodOfAccountingCash MethodOfAccountingCashInd MethodOfAccountingOther \\\n", "0 NaN X NaN \n", "\n", " MethodOfAccountingOtherInd MinutesOfCommittees MinutesOfCommitteesInd \\\n", "0 NaN NaN false \n", "\n", " MinutesOfGoverningBody MinutesOfGoverningBodyInd MiscellaneousRevenue \\\n", "0 NaN false NaN \n", "\n", " MiscellaneousRevenueGrp \\\n", "0 NaN \n", "\n", " MissionDesc \\\n", "0 RAISE MONIES TO CONSTRUCT A FACILITY FOR YEAR-ROUND RECREATION PROGRAM DEDICATED TO CHILDREN AND ADULTS WITH SPECIAL NEEDS WHO RESIDE IN THE TOWN OF HEMPSTEAD. \n", "\n", " MissionDescription MoreThan5000KToIndividuals MoreThan5000KToIndividualsInd \\\n", "0 NaN NaN false \n", "\n", " MoreThan5000KToOrgInd MoreThan5000KToOrganizations \\\n", "0 false NaN \n", "\n", " MortNotesPyblSecuredInvestProp MortgNotesPyblScrdInvstPropGrp \\\n", "0 NaN NaN \n", "\n", " NameOfForeignCountry NameOfPrincipalOfficerBusiness \\\n", "0 NaN NaN \n", "\n", " NameOfPrincipalOfficerPerson NbrIndependentVotingMembers \\\n", "0 NaN NaN \n", "\n", " NbrVotingGoverningBodyMembers NbrVotingMembersGoverningBody \\\n", "0 NaN NaN \n", "\n", " NetAssetsOrFundBalancesBOY NetAssetsOrFundBalancesBOYAmt \\\n", "0 NaN NaN \n", "\n", " NetAssetsOrFundBalancesEOY NetAssetsOrFundBalancesEOYAmt \\\n", "0 NaN 290600 \n", "\n", " NetGainOrLossInvestments NetGainOrLossInvestmentsGrp \\\n", "0 NaN NaN \n", "\n", " NetIncmFromFundraisingEvtGrp NetIncomeFromFundraisingEvents \\\n", "0 NaN NaN \n", "\n", " NetIncomeFromGaming NetIncomeFromGamingGrp NetIncomeOrLoss \\\n", "0 NaN NaN NaN \n", "\n", " NetIncomeOrLossGrp NetRentalIncomeOrLoss NetRentalIncomeOrLossGrp \\\n", "0 NaN NaN NaN \n", "\n", " NetUnrelatedBusTxblIncmAmt NetUnrelatedBusinessTxblIncome \\\n", "0 NaN NaN \n", "\n", " NetUnrlzdGainsLossesInvstAmt NoListedPersonsCompensated \\\n", "0 NaN NaN \n", "\n", " NoListedPersonsCompensatedInd NonDeductibleContributions \\\n", "0 X NaN \n", "\n", " NonDeductibleDisclosure NoncashContributions NoncashContributionsAmt \\\n", "0 NaN NaN NaN \n", "\n", " NondeductibleContriDisclInd NondeductibleContributionsInd \\\n", "0 NaN false \n", "\n", " NumberFormsTransmittedWith1096 NumberIndependentVotingMembers \\\n", "0 NaN NaN \n", "\n", " NumberIndividualsGT100K NumberOf8282Filed NumberOfContractorsGT100K \\\n", "0 NaN NaN NaN \n", "\n", " NumberOfEmployees NumberW2GIncluded ObjectId Occupancy \\\n", "0 NaN NaN 201503169349304780 NaN \n", "\n", " OccupancyGrp OfficeExpenses OfficeExpensesGrp OfficerEntityWithBsnssRltnshp \\\n", "0 NaN NaN NaN NaN \n", "\n", " OfficerMailingAddress OfficerMailingAddressInd OnBehalfOfIssuer \\\n", "0 NaN false NaN \n", "\n", " OnBehalfOfIssuerInd OperateHospitalInd OrgDoesNotFollowSFAS117Ind \\\n", "0 NaN false NaN \n", "\n", " OrgFiledInLieuOfForm1041Ind Organization501c Organization501c3 \\\n", "0 NaN NaN NaN \n", "\n", " Organization501c3Ind Organization501cInd OrganizationFollowsSFAS117Ind \\\n", "0 X NaN X \n", "\n", " OrganizationName OthNotesLoansReceivableNetGrp OtherAssetsTotal \\\n", "0 ANCHOR BUILDING FUND INC NaN NaN \n", "\n", " OtherAssetsTotalGrp OtherChangesInNetAssetsAmt OtherEmployeeBenefits \\\n", "0 NaN NaN NaN \n", "\n", " OtherEmployeeBenefitsGrp OtherExpensePriorYear OtherExpenses \\\n", "0 NaN NaN NaN \n", "\n", " OtherExpensesCurrentYear OtherExpensesGrp OtherExplainInSchO OtherInd \\\n", "0 NaN NaN NaN NaN \n", "\n", " OtherLiabilities OtherLiabilitiesGrp OtherNotesLoansReceivableNet \\\n", "0 NaN NaN NaN \n", "\n", " OtherRevenueCurrentYear OtherRevenueMisc \\\n", "0 NaN NaN \n", "\n", " OtherRevenueMiscGrp \\\n", "0 {u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000', u'Desc': u'SETTLEMENT'} \n", "\n", " OtherRevenuePriorYear OtherRevenueTotalAmt OtherSalariesAndWages \\\n", "0 NaN 325000 NaN \n", "\n", " OtherSalariesAndWagesGrp OtherWebsite OtherWebsiteInd OwnWebsite \\\n", "0 NaN NaN NaN NaN \n", "\n", " OwnWebsiteInd PYBenefitsPaidToMembersAmt PYContributionsGrantsAmt \\\n", "0 NaN NaN NaN \n", "\n", " PYExcessBenefitTransInd PYGrantsAndSimilarPaidAmt PYInvestmentIncomeAmt \\\n", "0 false NaN NaN \n", "\n", " PYOtherExpensesAmt PYOtherRevenueAmt PYProgramServiceRevenueAmt \\\n", "0 NaN NaN NaN \n", "\n", " PYRevenuesLessExpensesAmt PYSalariesCompEmpBnftPaidAmt PYTotalExpensesAmt \\\n", "0 NaN NaN NaN \n", "\n", " PYTotalProfFndrsngExpnsAmt PYTotalRevenueAmt PaidInCapSrplsLandBldgEqpFund \\\n", "0 NaN NaN NaN \n", "\n", " PartialLiquidation PartialLiquidationInd PayPremiumsPrsnlBnftCntrctInd \\\n", "0 NaN false false \n", "\n", " PaymentsToAffiliates PaymentsToAffiliatesGrp PayrollTaxes PayrollTaxesGrp \\\n", "0 NaN NaN NaN NaN \n", "\n", " PdInCapSrplsLandBldgEqpFundGrp PensionPlanContributions \\\n", "0 NaN NaN \n", "\n", " PensionPlanContributionsGrp PermanentlyRestrictedNetAssets \\\n", "0 NaN NaN \n", "\n", " PermanentlyRstrNetAssetsGrp PledgesAndGrantsReceivable \\\n", "0 NaN NaN \n", "\n", " PledgesAndGrantsReceivableGrp PoliciesReferenceChapters \\\n", "0 NaN NaN \n", "\n", " PoliciesReferenceChaptersInd PoliticalActivities PoliticalCampaignActyInd \\\n", "0 NaN NaN false \n", "\n", " PremiumsPaid PrepaidExpensesDeferredCharges PrepaidExpensesDefrdChargesGrp \\\n", "0 NaN NaN NaN \n", "\n", " PrincipalOfficerNm PriorExcessBenefitTransaction PriorPeriodAdjustmentsAmt \\\n", "0 JOHN MALLOY NaN NaN \n", "\n", " ProfessionalFundraising ProfessionalFundraisingInd ProgSrvcAccomActy2Grp \\\n", "0 NaN false NaN \n", "\n", " ProgSrvcAccomActy3Grp ProgSrvcAccomActyOtherGrp ProgramServiceRevenue \\\n", "0 NaN NaN NaN \n", "\n", " ProgramServiceRevenueCY ProgramServiceRevenueGrp \\\n", "0 NaN NaN \n", "\n", " ProgramServiceRevenuePriorYear ProhibitedTaxShelterTrans \\\n", "0 NaN NaN \n", "\n", " ProhibitedTaxShelterTransInd PymtTravelEntrtnmntPubOfclGrp \\\n", "0 false NaN \n", "\n", " QuidProQuoContriDisclInd QuidProQuoContributions QuidProQuoContributionsInd \\\n", "0 NaN NaN false \n", "\n", " QuidProQuoDisclosure RcvFndsToPayPrsnlBnftCntrctInd \\\n", "0 NaN false \n", "\n", " RcvblFromDisqualifiedPrsnGrp ReceivablesFromDisqualPersons \\\n", "0 NaN NaN \n", "\n", " ReceivablesFromOfficersEtc ReceivablesFromOfficersEtcGrp \\\n", "0 NaN NaN \n", "\n", " ReconcilationDonatedServices ReconcilationInvestExpenses \\\n", "0 NaN NaN \n", "\n", " ReconcilationOtherChanges ReconcilationPriorAdjustment \\\n", "0 NaN NaN \n", "\n", " ReconcilationRevenueExpenses ReconcilationRevenueExpnssAmt \\\n", "0 NaN 290600 \n", "\n", " ReconciliationUnrealizedInvest RegularMonitoringEnforcement \\\n", "0 NaN NaN \n", "\n", " RegularMonitoringEnfrcInd RelatedEntity RelatedEntityInd \\\n", "0 NaN NaN false \n", "\n", " RelatedOrgControlledEntity RelatedOrganizationCtrlEntInd \\\n", "0 NaN false \n", "\n", " RelatedOrganizations RelatedOrganizationsAmt RentalIncomeOrLoss \\\n", "0 NaN NaN NaN \n", "\n", " RentalIncomeOrLossGrp ReportFin48Footnote ReportInvestOthSecurities \\\n", "0 NaN NaN NaN \n", "\n", " ReportInvestmentsOtherSecInd ReportLandBldgEquip \\\n", "0 false NaN \n", "\n", " ReportLandBuildingEquipmentInd ReportOtherAssets ReportOtherAssetsInd \\\n", "0 false NaN false \n", "\n", " ReportOtherLiabilities ReportOtherLiabilitiesInd ReportProgRelInvest \\\n", "0 NaN false NaN \n", "\n", " ReportProgramRelatedInvstInd RetainedEarningsEndowmentEtc Revenue \\\n", "0 false NaN NaN \n", "\n", " RevenueAmt RevenuesLessExpensesCY RevenuesLessExpensesPriorYear Royalties \\\n", "0 NaN NaN NaN NaN \n", "\n", " RoyaltiesGrp RoyaltiesRevenue RoyaltiesRevenueGrp \\\n", "0 NaN NaN NaN \n", "\n", " RtnEarnEndowmentIncmOthFndsGrp SalariesEtcCurrentYear SalariesEtcPriorYear \\\n", "0 NaN NaN NaN \n", "\n", " SavingsAndTempCashInvestments SavingsAndTempCashInvstGrp ScheduleBRequired \\\n", "0 NaN NaN NaN \n", "\n", " ScheduleBRequiredInd ScheduleJRequired ScheduleJRequiredInd \\\n", "0 false NaN false \n", "\n", " ScheduleORequired ScheduleORequiredInd School SchoolOperatingInd \\\n", "0 NaN true NaN false \n", "\n", " SignificantChange SignificantChangeInd SignificantNewProgramServices \\\n", "0 NaN true NaN \n", "\n", " SignificantNewProgramSrvcInd StateLegalDomicile \\\n", "0 false NaN \n", "\n", " StatesWhereCopyOfReturnIsFiled StatesWhereCopyOfReturnIsFldCd \\\n", "0 NaN NY \n", "\n", " SubjectToProxyTax SubjectToProxyTaxInd SubmittedOn TaxExemptBondLiabilities \\\n", "0 NaN false 2016-01-30 NaN \n", "\n", " TaxExemptBondLiabilitiesGrp TaxExemptBonds TaxExemptBondsInd TaxPeriod \\\n", "0 NaN NaN false 201412 \n", "\n", " TaxableDistributions TaxableDistributionsInd TaxablePartyNotification \\\n", "0 NaN NaN NaN \n", "\n", " TaxablePartyNotificationInd TempOrPermanentEndowmentsInd \\\n", "0 false false \n", "\n", " TemporarilyRestrictedNetAssets TemporarilyRstrNetAssetsGrp \\\n", "0 NaN NaN \n", "\n", " TermOrPermanentEndowments TerminateOperationsInd Terminated \\\n", "0 NaN false NaN \n", "\n", " TerminationOrContraction TheBooksAreInCareOf \\\n", "0 NaN NaN \n", "\n", " TotLiabNetAssetsFundBalanceGrp TotReportableCompRltdOrgAmt \\\n", "0 {u'BOYAmt': u'0', u'EOYAmt': u'290600'} NaN \n", "\n", " TotalAssets TotalAssetsBOY TotalAssetsBOYAmt TotalAssetsEOY \\\n", "0 NaN NaN NaN NaN \n", "\n", " TotalAssetsEOYAmt TotalAssetsGrp TotalCompGT150K \\\n", "0 290600 {u'BOYAmt': u'0', u'EOYAmt': u'290600'} NaN \n", "\n", " TotalCompGreaterThan150KInd TotalContributions TotalContributionsAmt \\\n", "0 false NaN NaN \n", "\n", " TotalEmployeeCnt TotalExpensesCurrentYear TotalExpensesPriorYear \\\n", "0 0 NaN NaN \n", "\n", " TotalFunctionalExpenses \\\n", "0 NaN \n", "\n", " TotalFunctionalExpensesGrp \\\n", "0 {u'TotalAmt': u'34400', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'34400', u'FundraisingAmt': u'0'} \n", "\n", " TotalFundrsngExpCurrentYear TotalGrossUBI TotalGrossUBIAmt TotalJointCosts \\\n", "0 NaN NaN 0 NaN \n", "\n", " TotalJointCostsGrp TotalLiabNetAssetsFundBalances TotalLiabilities \\\n", "0 NaN NaN NaN \n", "\n", " TotalLiabilitiesBOY TotalLiabilitiesBOYAmt TotalLiabilitiesEOY \\\n", "0 NaN NaN NaN \n", "\n", " TotalLiabilitiesEOYAmt TotalLiabilitiesGrp \\\n", "0 0 {u'BOYAmt': u'0', u'EOYAmt': u'0'} \n", "\n", " TotalNbrEmployees TotalNbrVolunteers \\\n", "0 NaN NaN \n", "\n", " TotalNetAssetsFundBalanceGrp TotalNetAssetsFundBalances \\\n", "0 {u'BOYAmt': u'0', u'EOYAmt': u'290600'} NaN \n", "\n", " TotalOfOtherProgramServiceExp TotalOfOtherProgramServiceGrnt \\\n", "0 NaN NaN \n", "\n", " TotalOfOtherProgramServiceRev TotalOthProgramServiceRevGrp \\\n", "0 NaN NaN \n", "\n", " TotalOthProgramServiceRevenue TotalOtherCompensation \\\n", "0 NaN NaN \n", "\n", " TotalOtherCompensationAmt TotalOtherProgSrvcExpenseAmt \\\n", "0 NaN NaN \n", "\n", " TotalOtherProgSrvcGrantAmt TotalOtherProgSrvcRevenueAmt TotalOtherRevenue \\\n", "0 NaN NaN NaN \n", "\n", " TotalProfFundrsngExpCY TotalProfFundrsngExpPriorYear \\\n", "0 NaN NaN \n", "\n", " TotalProgramServiceExpense TotalProgramServiceExpensesAmt \\\n", "0 NaN NaN \n", "\n", " TotalProgramServiceRevenue TotalProgramServiceRevenueAmt \\\n", "0 NaN NaN \n", "\n", " TotalReportableCompFrmRltdOrgs TotalReportableCompFromOrg \\\n", "0 NaN NaN \n", "\n", " TotalReportableCompFromOrgAmt TotalRevenue TotalRevenueCurrentYear \\\n", "0 NaN NaN NaN \n", "\n", " TotalRevenueGrp \\\n", "0 {u'TotalRevenueColumnAmt': u'325000', u'ExclusionAmt': u'325000'} \n", "\n", " TotalRevenuePriorYear TotalVolunteersCnt TransactionRelatedEntity \\\n", "0 NaN NaN NaN \n", "\n", " TransactionWithControlEntInd TransfersToExemptNonChrtblOrg Travel \\\n", "0 NaN NaN NaN \n", "\n", " TravelEntrtnmntPublicOfficials TravelGrp TrnsfrExmptNonChrtblRltdOrgInd \\\n", "0 NaN NaN false \n", "\n", " TypeOfOrganizationCorpInd TypeOfOrganizationCorporation \\\n", "0 X NaN \n", "\n", " TypeOfOrganizationOther TypeOfOrganizationOtherInd TypeOfOrganizationTrust \\\n", "0 NaN NaN NaN \n", "\n", " TypeOfOrganizationTrustInd \\\n", "0 NaN \n", "\n", " URL \\\n", "0 https://s3.amazonaws.com/irs-form-990/201503169349304780_public.xml \n", "\n", " USAddress \\\n", "0 {u'CityNm': u'LIDO BEACH', u'StateAbbreviationCd': u'NY', u'ZIPCd': u'11561', u'AddressLine1Txt': u'630 LIDO BLVD'} \n", "\n", " UnrelatedBusIncmOverLimitInd UnrelatedBusinessIncome UnrestrictedNetAssets \\\n", "0 false NaN NaN \n", "\n", " UnrestrictedNetAssetsGrp UnsecuredNotesLoansPayable \\\n", "0 {u'EOYAmt': u'290600'} NaN \n", "\n", " UnsecuredNotesLoansPayableGrp UponRequest UponRequestInd \\\n", "0 NaN NaN X \n", "\n", " VotingMembersGoverningBodyCnt VotingMembersIndependentCnt WebSite \\\n", "0 2 2 NaN \n", "\n", " WebsiteAddressTxt WhistleblowerPolicy WhistleblowerPolicyInd \\\n", "0 WWW.CAMPANCHOR.ORG NaN false \n", "\n", " WrittenPolicyOrProcedure WrittenPolicyOrProcedureInd YearFormation \\\n", "0 NaN NaN NaN \n", "\n", " _id \n", "0 57cce7823ffc5a55fe7f54b5 " ] }, "execution_count": 737, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#df2 = pd.DataFrame(list(filings.find()))\n", "#print '# of columns:', len(df2.columns)\n", "#print '# of observations:', len(df2)\n", "#df2.head(1)" ] }, { "cell_type": "code", "execution_count": 744, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINTaxPeriodCYTotalExpensesAmtTotalExpensesCurrentYearTotalProgramServiceExpenseTotalProgramServiceExpensesAmtTotalFunctionalExpensesTotalFunctionalExpensesGrp
4041326445072013127795021NaNNaNNaNNaN{u'TotalAmt': u'7795021'}
106132644507201112NaN5880789NaNNaN{u'Total': u'5880789'}NaN
431132644507201212NaN6237509NaNNaN{u'Total': u'6237509'}NaN
1721326445072014127763846NaNNaNNaNNaN{u'TotalAmt': u'7763846'}
72132644507201012NaN6024086NaNNaN{u'Total': u'6024086'}NaN
521355793022014122023750NaNNaNNaNNaN{u'TotalAmt': u'2023750'}
4061355793022013124133761NaNNaNNaNNaN{u'TotalAmt': u'4133761', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'0', u'FundraisingAmt': u'0'}
500201737542201006NaN177901NaNNaN{u'ManagementAndGeneral': u'111025', u'ProgramServices': u'0', u'Fundraising': u'66876', u'Total': u'177901'}NaN
027131707220141234400NaNNaNNaNNaN{u'TotalAmt': u'34400', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'34400', u'FundraisingAmt': u'0'}
3393308377802013122151741NaNNaN2151482NaN{u'TotalAmt': u'2151741', u'ProgramServicesAmt': u'2151482', u'ManagementAndGeneralAmt': u'259', u'FundraisingAmt': u'0'}
345330837780201212NaN211124211124NaN{u'ManagementAndGeneral': u'0', u'ProgramServices': u'211124', u'Fundraising': u'0', u'Total': u'211124'}NaN
216330837780201112NaN2853826728NaN{u'ManagementAndGeneral': u'1810', u'ProgramServices': u'26728', u'Fundraising': u'0', u'Total': u'28538'}NaN
2773308377802014123330NaNNaNNaNNaN{u'TotalAmt': u'3330', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'3330', u'FundraisingAmt': u'0'}
213591951577201012NaN1161939NaNNaN{u'Total': u'1161939'}NaN
226591951577201112NaN1690623NaNNaN{u'Total': u'1690623'}NaN
495591951577201012NaN1282149NaNNaN{u'Total': u'1282149'}NaN
496953553530201112NaN0NaNNaN{u'ManagementAndGeneral': u'0', u'ProgramServices': u'0', u'Fundraising': u'0', u'Total': u'0'}NaN
\n", "
" ], "text/plain": [ " EIN TaxPeriod CYTotalExpensesAmt TotalExpensesCurrentYear \\\n", "404 132644507 201312 7795021 NaN \n", "106 132644507 201112 NaN 5880789 \n", "431 132644507 201212 NaN 6237509 \n", "172 132644507 201412 7763846 NaN \n", "72 132644507 201012 NaN 6024086 \n", "52 135579302 201412 2023750 NaN \n", "406 135579302 201312 4133761 NaN \n", "500 201737542 201006 NaN 177901 \n", "0 271317072 201412 34400 NaN \n", "339 330837780 201312 2151741 NaN \n", "345 330837780 201212 NaN 211124 \n", "216 330837780 201112 NaN 28538 \n", "277 330837780 201412 3330 NaN \n", "213 591951577 201012 NaN 1161939 \n", "226 591951577 201112 NaN 1690623 \n", "495 591951577 201012 NaN 1282149 \n", "496 953553530 201112 NaN 0 \n", "\n", " TotalProgramServiceExpense TotalProgramServiceExpensesAmt \\\n", "404 NaN NaN \n", "106 NaN NaN \n", "431 NaN NaN \n", "172 NaN NaN \n", "72 NaN NaN \n", "52 NaN NaN \n", "406 NaN NaN \n", "500 NaN NaN \n", "0 NaN NaN \n", "339 NaN 2151482 \n", "345 211124 NaN \n", "216 26728 NaN \n", "277 NaN NaN \n", "213 NaN NaN \n", "226 NaN NaN \n", "495 NaN NaN \n", "496 NaN NaN \n", "\n", " TotalFunctionalExpenses \\\n", "404 NaN \n", "106 {u'Total': u'5880789'} \n", "431 {u'Total': u'6237509'} \n", "172 NaN \n", "72 {u'Total': u'6024086'} \n", "52 NaN \n", "406 NaN \n", "500 {u'ManagementAndGeneral': u'111025', u'ProgramServices': u'0', u'Fundraising': u'66876', u'Total': u'177901'} \n", "0 NaN \n", "339 NaN \n", "345 {u'ManagementAndGeneral': u'0', u'ProgramServices': u'211124', u'Fundraising': u'0', u'Total': u'211124'} \n", "216 {u'ManagementAndGeneral': u'1810', u'ProgramServices': u'26728', u'Fundraising': u'0', u'Total': u'28538'} \n", "277 NaN \n", "213 {u'Total': u'1161939'} \n", "226 {u'Total': u'1690623'} \n", "495 {u'Total': u'1282149'} \n", "496 {u'ManagementAndGeneral': u'0', u'ProgramServices': u'0', u'Fundraising': u'0', u'Total': u'0'} \n", "\n", " TotalFunctionalExpensesGrp \n", "404 {u'TotalAmt': u'7795021'} \n", "106 NaN \n", "431 NaN \n", "172 {u'TotalAmt': u'7763846'} \n", "72 NaN \n", "52 {u'TotalAmt': u'2023750'} \n", "406 {u'TotalAmt': u'4133761', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'0', u'FundraisingAmt': u'0'} \n", "500 NaN \n", "0 {u'TotalAmt': u'34400', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'34400', u'FundraisingAmt': u'0'} \n", "339 {u'TotalAmt': u'2151741', u'ProgramServicesAmt': u'2151482', u'ManagementAndGeneralAmt': u'259', u'FundraisingAmt': u'0'} \n", "345 NaN \n", "216 NaN \n", "277 {u'TotalAmt': u'3330', u'ProgramServicesAmt': u'0', u'ManagementAndGeneralAmt': u'3330', u'FundraisingAmt': u'0'} \n", "213 NaN \n", "226 NaN \n", "495 NaN \n", "496 NaN " ] }, "execution_count": 744, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#efficiency_columns = ['EIN', 'TaxPeriod', 'CYTotalExpensesAmt', 'TotalExpensesCurrentYear',\n", "# 'TotalProgramServiceExpense', 'TotalProgramServiceExpensesAmt',\n", "# 'TotalFunctionalExpenses', 'TotalFunctionalExpensesGrp']\n", "#df2 = df2.sort_values(by=['EIN'], ascending=[1])\n", "#df2[df2['EIN'].isin(zero_prog_exp)][efficiency_columns]" ] }, { "cell_type": "code", "execution_count": 736, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINFYEtotal_expensesprogram_expensesprogram_efficiency
172132644507FY20147763846.0NaNNaN
404132644507FY20137795021.0NaNNaN
431132644507FY20126237509.0NaNNaN
106132644507FY20115880789.0NaNNaN
72132644507FY20106024086.0NaNNaN
52135579302FY20142023750.0NaNNaN
406135579302FY20134133761.0NaNNaN
500201737542FY2010177901.0NaNNaN
0271317072FY201434400.0NaNNaN
277330837780FY20143330.0NaNNaN
339330837780FY20132151741.02151482.00.999880
345330837780FY2012211124.0211124.01.000000
216330837780FY201128538.026728.00.936576
226591951577FY20111690623.0NaNNaN
213591951577FY20101161939.0NaNNaN
495591951577FY20101282149.0NaNNaN
496953553530FY20110.0NaNNaN
\n", "
" ], "text/plain": [ " EIN FYE total_expenses program_expenses program_efficiency\n", "172 132644507 FY2014 7763846.0 NaN NaN\n", "404 132644507 FY2013 7795021.0 NaN NaN\n", "431 132644507 FY2012 6237509.0 NaN NaN\n", "106 132644507 FY2011 5880789.0 NaN NaN\n", "72 132644507 FY2010 6024086.0 NaN NaN\n", "52 135579302 FY2014 2023750.0 NaN NaN\n", "406 135579302 FY2013 4133761.0 NaN NaN\n", "500 201737542 FY2010 177901.0 NaN NaN\n", "0 271317072 FY2014 34400.0 NaN NaN\n", "277 330837780 FY2014 3330.0 NaN NaN\n", "339 330837780 FY2013 2151741.0 2151482.0 0.999880\n", "345 330837780 FY2012 211124.0 211124.0 1.000000\n", "216 330837780 FY2011 28538.0 26728.0 0.936576\n", "226 591951577 FY2011 1690623.0 NaN NaN\n", "213 591951577 FY2010 1161939.0 NaN NaN\n", "495 591951577 FY2010 1282149.0 NaN NaN\n", "496 953553530 FY2011 0.0 NaN NaN" ] }, "execution_count": 736, "metadata": {}, "output_type": "execute_result" } ], "source": [ "zero_prog_exp = ['132644507', '135579302', '201737542', '271317072', '330837780', '591951577', '953553530']\n", "df[df['EIN'].isin(zero_prog_exp)][cols2]" ] }, { "cell_type": "code", "execution_count": 748, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "14\n", "0\n", "14\n", "14\n" ] } ], "source": [ "print len(df[df['program_efficiency'].isnull()])\n", "print len(df[df['total_expenses'].isnull()])\n", "print len(df[df['program_expenses'].isnull()])\n", "print len(df[df['program_efficiency'].isnull()])" ] }, { "cell_type": "code", "execution_count": 749, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n" ] } ], "source": [ "df['program_efficiency'] = np.where(df['program_efficiency'].isnull(), 0, df['program_efficiency'])\n", "print len(df[df['program_efficiency'].isnull()])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Create logged revenues" ] }, { "cell_type": "code", "execution_count": 768, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1\n", "2\n", "0\n", "0\n" ] } ], "source": [ "print len(df[df['tot_rev']==0])\n", "print len(df[df['tot_rev']<0])\n", "df['tot_rev_no_neg'] = df['tot_rev']\n", "df['tot_rev_no_neg'] = np.where(df['tot_rev_no_neg']<=0, 1, df['tot_rev_no_neg'])\n", "print len(df[df['tot_rev_no_neg']==0])\n", "print len(df[df['tot_rev_no_neg']<0])" ] }, { "cell_type": "code", "execution_count": 769, "metadata": { "collapsed": true }, "outputs": [], "source": [ "df['total_revenue_logged'] = np.log(df['tot_rev_no_neg'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Columns Needed to keep to merge with existing data:" ] }, { "cell_type": "code", "execution_count": 770, "metadata": { "collapsed": false }, "outputs": [], "source": [ "file_list_columns = ['EIN', 'OrganizationName', 'URL', 'SubmittedOn', 'TaxPeriod', 'FYE']\n", "\n", "SOX_columns = ['whistleblower_policy', 'conflict_of_interest_policy', 'records_retention_policy',\n", " 'SOX_policies', 'SOX_policies_binary', 'SOX_policies_all_binary']\n", "\n", "control_columns = ['tot_rev', 'tot_rev_no_neg', 'total_revenue_logged', 'total_expenses', 'program_expenses', \n", " 'program_efficiency', 'complexity']\n", "\n", "unneeded_columns = ['CYTotalExpensesAmt', u'TotalExpensesCurrentYear', u'TotalFunctionalExpenses', \n", " 'TotalFunctionalExpensesGrp', u'TotalProgramServiceExpense', u'TotalProgramServiceExpensesAmt',\n", " 'CYContributionsGrantsAmt', u'FederatedCampaigns', u'FederatedCampaignsAmt', u'MembershipDues',\n", " 'MembershipDuesAmt', u'NetIncomeFromFundraisingEvents', u'NetIncmFromFundraisingEvtGrp', \n", " 'FundraisingAmt', 'FundraisingGrossIncomeAmt', u'RelatedOrganizations', \n", " 'RelatedOrganizationsAmt', u'GovernmentGrants', 'GovernmentGrantsAmt', \n", " 'PYProgramServiceRevenueAmt', u'CYProgramServiceRevenueAmt', 'ProgramServiceRevenue', \n", " 'ProgramServiceRevenueCY', u'ProgramServiceRevenueGrp', u'TotalProgramServiceRevenue', \n", " 'TotalProgramServiceRevenueAmt', u'CYInvestmentIncomeAmt', u'InvestmentIncome', \n", " 'InvestmentIncomeCurrentYear', 'InvestmentIncomeGrp', u'OtherRevenueTotalAmt', \n", " 'OtherRevenueMiscGrp', u'TotalOtherRevenue', 'CYOtherRevenueAmt', 'fundraising', \n", " 'program_revenue', 'OtherRevMisc', 'other_revenue_binary']\n", "\n", "cols = file_list_columns+SOX_columns+control_columns \n", "\n", "df = df[cols]" ] }, { "cell_type": "code", "execution_count": 762, "metadata": { "collapsed": false }, "outputs": [], "source": [ "df['whistleblower_policy'] = df['whistleblower_policy'].astype('int')\n", "df['conflict_of_interest_policy'] = df['conflict_of_interest_policy'].astype('int')\n", "df['records_retention_policy'] = df['records_retention_policy'].astype('int')" ] }, { "cell_type": "code", "execution_count": 771, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINOrganizationNameURLSubmittedOnTaxPeriodFYEwhistleblower_policyconflict_of_interest_policyrecords_retention_policySOX_policiesSOX_policies_binarySOX_policies_all_binarytot_revtot_rev_no_negtotal_revenue_loggedtotal_expensesprogram_expensesprogram_efficiencycomplexity
252030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml2013-12-31201212FY20120112.01.00.02515399251539914.7379422100922.01598626.00.7609162
433030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml2013-02-14201112FY20110112.01.00.02161209216120914.5861782481310.01715793.00.6914870
78030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201220909349300327_public.xml2012-05-25201012FY20100112.01.00.03990564399056415.1994433912643.02930785.00.7490550
333042129889SCHEPENS EYE RESEARCH INSTITUTE INChttps://s3.amazonaws.com/irs-form-990/201502299349301700_public.xml2015-09-21201409FY20141113.01.01.0229955262299552616.95081025626262.018485740.00.7213594
12042129889SCHEPENS EYE RESEARCH INSTITUTE INChttps://s3.amazonaws.com/irs-form-990/201442269349304504_public.xml2014-10-28201309FY20131113.01.01.0259234492592344917.07065826224173.019280493.00.7352183
481042129889SCHEPENS EYE RESEARCH INSTITUTE INChttps://s3.amazonaws.com/irs-form-990/201312269349301491_public.xml2013-12-17201209FY20121113.01.01.0285795922857959217.16820325481942.019252739.00.7555444
16042129889SCHEPENS EYE RESEARCH INSTITUTE INChttps://s3.amazonaws.com/irs-form-990/201212219349301091_public.xml2012-08-16201106FY20111113.01.01.0264100022641000217.08925328367393.022102327.00.7791463
130042129889SCHEPENS EYE RESEARCH INSTITUTE INChttps://s3.amazonaws.com/irs-form-990/201121199349301122_public.xml2011-05-04201006FY20101113.01.01.0284931552849315517.16517429094188.024121623.00.8290873
434042701694WESTFIELD STATE FOUNDATION INChttps://s3.amazonaws.com/irs-form-990/201403119349300995_public.xml2014-12-03201406FY20141113.01.01.076443776443713.546895636346.0478219.00.7515083
374042701694WESTFIELD STATE FOUNDATION INChttps://s3.amazonaws.com/irs-form-990/201401339349305050_public.xml2014-06-27201306FY20131113.01.01.074520474520413.521413766452.0446064.00.5819862
\n", "
" ], "text/plain": [ " EIN OrganizationName \\\n", "252 030498214 NEWARK NOW INC \n", "433 030498214 NEWARK NOW INC \n", "78 030498214 NEWARK NOW INC \n", "333 042129889 SCHEPENS EYE RESEARCH INSTITUTE INC \n", "12 042129889 SCHEPENS EYE RESEARCH INSTITUTE INC \n", "481 042129889 SCHEPENS EYE RESEARCH INSTITUTE INC \n", "16 042129889 SCHEPENS EYE RESEARCH INSTITUTE INC \n", "130 042129889 SCHEPENS EYE RESEARCH INSTITUTE INC \n", "434 042701694 WESTFIELD STATE FOUNDATION INC \n", "374 042701694 WESTFIELD STATE FOUNDATION INC \n", "\n", " URL \\\n", "252 https://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml \n", "433 https://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml \n", "78 https://s3.amazonaws.com/irs-form-990/201220909349300327_public.xml \n", "333 https://s3.amazonaws.com/irs-form-990/201502299349301700_public.xml \n", "12 https://s3.amazonaws.com/irs-form-990/201442269349304504_public.xml \n", "481 https://s3.amazonaws.com/irs-form-990/201312269349301491_public.xml \n", "16 https://s3.amazonaws.com/irs-form-990/201212219349301091_public.xml \n", "130 https://s3.amazonaws.com/irs-form-990/201121199349301122_public.xml \n", "434 https://s3.amazonaws.com/irs-form-990/201403119349300995_public.xml \n", "374 https://s3.amazonaws.com/irs-form-990/201401339349305050_public.xml \n", "\n", " SubmittedOn TaxPeriod FYE whistleblower_policy \\\n", "252 2013-12-31 201212 FY2012 0 \n", "433 2013-02-14 201112 FY2011 0 \n", "78 2012-05-25 201012 FY2010 0 \n", "333 2015-09-21 201409 FY2014 1 \n", "12 2014-10-28 201309 FY2013 1 \n", "481 2013-12-17 201209 FY2012 1 \n", "16 2012-08-16 201106 FY2011 1 \n", "130 2011-05-04 201006 FY2010 1 \n", "434 2014-12-03 201406 FY2014 1 \n", "374 2014-06-27 201306 FY2013 1 \n", "\n", " conflict_of_interest_policy records_retention_policy SOX_policies \\\n", "252 1 1 2.0 \n", "433 1 1 2.0 \n", "78 1 1 2.0 \n", "333 1 1 3.0 \n", "12 1 1 3.0 \n", "481 1 1 3.0 \n", "16 1 1 3.0 \n", "130 1 1 3.0 \n", "434 1 1 3.0 \n", "374 1 1 3.0 \n", "\n", " SOX_policies_binary SOX_policies_all_binary tot_rev tot_rev_no_neg \\\n", "252 1.0 0.0 2515399 2515399 \n", "433 1.0 0.0 2161209 2161209 \n", "78 1.0 0.0 3990564 3990564 \n", "333 1.0 1.0 22995526 22995526 \n", "12 1.0 1.0 25923449 25923449 \n", "481 1.0 1.0 28579592 28579592 \n", "16 1.0 1.0 26410002 26410002 \n", "130 1.0 1.0 28493155 28493155 \n", "434 1.0 1.0 764437 764437 \n", "374 1.0 1.0 745204 745204 \n", "\n", " total_revenue_logged total_expenses program_expenses \\\n", "252 14.737942 2100922.0 1598626.0 \n", "433 14.586178 2481310.0 1715793.0 \n", "78 15.199443 3912643.0 2930785.0 \n", "333 16.950810 25626262.0 18485740.0 \n", "12 17.070658 26224173.0 19280493.0 \n", "481 17.168203 25481942.0 19252739.0 \n", "16 17.089253 28367393.0 22102327.0 \n", "130 17.165174 29094188.0 24121623.0 \n", "434 13.546895 636346.0 478219.0 \n", "374 13.521413 766452.0 446064.0 \n", "\n", " program_efficiency complexity \n", "252 0.760916 2 \n", "433 0.691487 0 \n", "78 0.749055 0 \n", "333 0.721359 4 \n", "12 0.735218 3 \n", "481 0.755544 4 \n", "16 0.779146 3 \n", "130 0.829087 3 \n", "434 0.751508 3 \n", "374 0.581986 2 " ] }, "execution_count": 771, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[:10]" ] }, { "cell_type": "code", "execution_count": 772, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countmeanstdmin25%50%75%max
whistleblower_policy538.05.892193e-014.924334e-010.00.000000e+001.000000e+001.000000e+001.000000e+00
conflict_of_interest_policy538.08.438662e-013.633200e-010.01.000000e+001.000000e+001.000000e+001.000000e+00
records_retention_policy538.07.118959e-014.533013e-010.00.000000e+001.000000e+001.000000e+001.000000e+00
SOX_policies538.02.144981e+001.097391e+000.01.000000e+003.000000e+003.000000e+003.000000e+00
SOX_policies_binary538.08.605948e-013.466912e-010.01.000000e+001.000000e+001.000000e+001.000000e+00
SOX_policies_all_binary538.05.446097e-014.984695e-010.00.000000e+001.000000e+001.000000e+001.000000e+00
tot_rev538.05.857063e+073.234413e+08-218265025.01.320586e+064.261394e+061.334163e+073.741635e+09
tot_rev_no_neg538.05.897644e+073.232302e+081.01.320586e+064.261394e+061.334163e+073.741635e+09
total_revenue_logged538.01.524967e+012.287752e+000.01.409358e+011.526510e+011.640638e+012.204279e+01
total_expenses538.05.632274e+073.003909e+080.01.375426e+064.421644e+061.285109e+073.287631e+09
program_expenses524.04.994483e+072.653191e+083884.0NaNNaNNaN2.884879e+09
program_efficiency538.07.589501e-012.257745e-010.06.945523e-018.398745e-019.007387e-011.000000e+00
complexity538.02.431227e+001.278642e+000.02.000000e+002.000000e+003.000000e+007.000000e+00
\n", "
" ], "text/plain": [ " count mean std min \\\n", "whistleblower_policy 538.0 5.892193e-01 4.924334e-01 0.0 \n", "conflict_of_interest_policy 538.0 8.438662e-01 3.633200e-01 0.0 \n", "records_retention_policy 538.0 7.118959e-01 4.533013e-01 0.0 \n", "SOX_policies 538.0 2.144981e+00 1.097391e+00 0.0 \n", "SOX_policies_binary 538.0 8.605948e-01 3.466912e-01 0.0 \n", "SOX_policies_all_binary 538.0 5.446097e-01 4.984695e-01 0.0 \n", "tot_rev 538.0 5.857063e+07 3.234413e+08 -218265025.0 \n", "tot_rev_no_neg 538.0 5.897644e+07 3.232302e+08 1.0 \n", "total_revenue_logged 538.0 1.524967e+01 2.287752e+00 0.0 \n", "total_expenses 538.0 5.632274e+07 3.003909e+08 0.0 \n", "program_expenses 524.0 4.994483e+07 2.653191e+08 3884.0 \n", "program_efficiency 538.0 7.589501e-01 2.257745e-01 0.0 \n", "complexity 538.0 2.431227e+00 1.278642e+00 0.0 \n", "\n", " 25% 50% 75% \\\n", "whistleblower_policy 0.000000e+00 1.000000e+00 1.000000e+00 \n", "conflict_of_interest_policy 1.000000e+00 1.000000e+00 1.000000e+00 \n", "records_retention_policy 0.000000e+00 1.000000e+00 1.000000e+00 \n", "SOX_policies 1.000000e+00 3.000000e+00 3.000000e+00 \n", "SOX_policies_binary 1.000000e+00 1.000000e+00 1.000000e+00 \n", "SOX_policies_all_binary 0.000000e+00 1.000000e+00 1.000000e+00 \n", "tot_rev 1.320586e+06 4.261394e+06 1.334163e+07 \n", "tot_rev_no_neg 1.320586e+06 4.261394e+06 1.334163e+07 \n", "total_revenue_logged 1.409358e+01 1.526510e+01 1.640638e+01 \n", "total_expenses 1.375426e+06 4.421644e+06 1.285109e+07 \n", "program_expenses NaN NaN NaN \n", "program_efficiency 6.945523e-01 8.398745e-01 9.007387e-01 \n", "complexity 2.000000e+00 2.000000e+00 3.000000e+00 \n", "\n", " max \n", "whistleblower_policy 1.000000e+00 \n", "conflict_of_interest_policy 1.000000e+00 \n", "records_retention_policy 1.000000e+00 \n", "SOX_policies 3.000000e+00 \n", "SOX_policies_binary 1.000000e+00 \n", "SOX_policies_all_binary 1.000000e+00 \n", "tot_rev 3.741635e+09 \n", "tot_rev_no_neg 3.741635e+09 \n", "total_revenue_logged 2.204279e+01 \n", "total_expenses 3.287631e+09 \n", "program_expenses 2.884879e+09 \n", "program_efficiency 1.000000e+00 \n", "complexity 7.000000e+00 " ] }, "execution_count": 772, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.describe().T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Save DF" ] }, { "cell_type": "code", "execution_count": 787, "metadata": { "collapsed": true }, "outputs": [], "source": [ "df.to_pickle('e-file 990s for 2016 donor advisory organizations, v4 (key columns only).pkl')\n", "df.to_excel('e-file 990s for 2016 donor advisory organizations, v4 (key columns only).xls')" ] }, { "cell_type": "code", "execution_count": 786, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EINOrganizationNameURLSubmittedOnTaxPeriodFYEwhistleblower_policyconflict_of_interest_policyrecords_retention_policySOX_policiesSOX_policies_binarySOX_policies_all_binarytot_revtot_rev_no_negtotal_revenue_loggedtotal_expensesprogram_expensesprogram_efficiencycomplexity
252030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml2013-12-31201212FY20120112.01.00.02515399251539914.7379422100922.01598626.00.7609162
433030498214NEWARK NOW INChttps://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml2013-02-14201112FY20110112.01.00.02161209216120914.5861782481310.01715793.00.6914870
\n", "
" ], "text/plain": [ " EIN OrganizationName \\\n", "252 030498214 NEWARK NOW INC \n", "433 030498214 NEWARK NOW INC \n", "\n", " URL \\\n", "252 https://s3.amazonaws.com/irs-form-990/201303199349303505_public.xml \n", "433 https://s3.amazonaws.com/irs-form-990/201320249349300417_public.xml \n", "\n", " SubmittedOn TaxPeriod FYE whistleblower_policy \\\n", "252 2013-12-31 201212 FY2012 0 \n", "433 2013-02-14 201112 FY2011 0 \n", "\n", " conflict_of_interest_policy records_retention_policy SOX_policies \\\n", "252 1 1 2.0 \n", "433 1 1 2.0 \n", "\n", " SOX_policies_binary SOX_policies_all_binary tot_rev tot_rev_no_neg \\\n", "252 1.0 0.0 2515399 2515399 \n", "433 1.0 0.0 2161209 2161209 \n", "\n", " total_revenue_logged total_expenses program_expenses \\\n", "252 14.737942 2100922.0 1598626.0 \n", "433 14.586178 2481310.0 1715793.0 \n", "\n", " program_efficiency complexity \n", "252 0.760916 2 \n", "433 0.691487 0 " ] }, "execution_count": 786, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[:2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Compare to 47 missing" ] }, { "cell_type": "code", "execution_count": 774, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "47\n" ] }, { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CN org_idEINFYE (choose latest available on CN)donor_advisoryconflict_of_interestrecords_retentionwhistleblowercomplexitytotal_revenueprogram_efficiencyprogram_expensetotal_expenseUnnamed: 12
010087581925867201511.00.00.05.0445953.0NaN229316.0370526.0NaN
110552942719901201211.00.01.03.03935913.0NaN3842824.04134682.0NaN
210902262224994201411.01.01.03.0706895.0NaN356046.0565973.0NaN
311009953523852201411.01.01.02.03620634.0NaN490708.03455917.0NaN
411327720760857201411.01.01.05.04244456.0NaN4210946.04942239.0NaN
\n", "
" ], "text/plain": [ " CN org_id EIN FYE (choose latest available on CN) donor_advisory \\\n", "0 10087 581925867 2015 1 \n", "1 10552 942719901 2012 1 \n", "2 10902 262224994 2014 1 \n", "3 11009 953523852 2014 1 \n", "4 11327 720760857 2014 1 \n", "\n", " conflict_of_interest records_retention whistleblower complexity \\\n", "0 1.0 0.0 0.0 5.0 \n", "1 1.0 0.0 1.0 3.0 \n", "2 1.0 1.0 1.0 3.0 \n", "3 1.0 1.0 1.0 2.0 \n", "4 1.0 1.0 1.0 5.0 \n", "\n", " total_revenue program_efficiency program_expense total_expense \\\n", "0 445953.0 NaN 229316.0 370526.0 \n", "1 3935913.0 NaN 3842824.0 4134682.0 \n", "2 706895.0 NaN 356046.0 565973.0 \n", "3 3620634.0 NaN 490708.0 3455917.0 \n", "4 4244456.0 NaN 4210946.0 4942239.0 \n", "\n", " Unnamed: 12 \n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 NaN \n", "4 NaN " ] }, "execution_count": 774, "metadata": {}, "output_type": "execute_result" } ], "source": [ "missing_47 = pd.read_excel('47 missing SOX_updated.xls')\n", "print len(missing_47)\n", "missing_47.head()" ] }, { "cell_type": "code", "execution_count": 779, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "47\n", "47\n" ] }, { "data": { "text/plain": [ "['581925867', '942719901', '262224994', '953523852', '720760857']" ] }, "execution_count": 779, "metadata": {}, "output_type": "execute_result" } ], "source": [ "missing_47_EINs = missing_47['EIN'].tolist()\n", "missing_47_EINs = [str(x) for x in missing_47_EINs]\n", "print len(missing_47_EINs)\n", "print len(set(missing_47_EINs))\n", "missing_47_EINs[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
42 of the 47 EINs are in the e-file database." ] }, { "cell_type": "code", "execution_count": 782, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "164\n", "164\n", "42\n" ] } ], "source": [ "print len(df[df['EIN'].isin(missing_47_EINs)])\n", "print len(df[df['EIN'].isin(missing_47_EINs)]['EIN'].tolist())\n", "print len(set(df[df['EIN'].isin(missing_47_EINs)]['EIN'].tolist()))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.12" } }, "nbformat": 4, "nbformat_minor": 0 }