{ "cells": [ { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "# Analyzing replicability of connectivity-based multivariate BWAS on the Human Connectome Project dataset\n", "\n", "### Re-compute certain results for more sample sizes.\n", "\n", "## Imports" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2021-08-03T20:04:15.431840Z", "start_time": "2021-08-03T20:04:14.753565Z" }, "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "from sklearn.linear_model import Ridge\n", "from sklearn.svm import SVR\n", "from sklearn.model_selection import KFold, train_test_split, cross_val_predict, GridSearchCV\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.decomposition import PCA\n", "from joblib import Parallel, delayed\n", "from mlxtend.evaluate import permutation_test\n", "sns.set(rc={\"figure.figsize\":(4, 2)})\n", "sns.set_style(\"whitegrid\")" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Load HCP data\n", "\n", "We load functional network matrices (netmats) from the HCP1200-release, as published on connectomeDB: https://db.humanconnectome.org/\n", "Due to licensoing issues, data is not supplied with the repository, but can be downloaded from the ConnectomeDB.\n", "See [hcp_data/readme.md](hcp_data/readme.md) for more details." ] }, { "cell_type": "code", "execution_count": 2, "outputs": [ { "data": { "text/plain": " Release Acquisition Gender Age 3T_Full_MR_Compl T1_Count \\\nSubject \n100004 S900 Q06 M 22-25 False 0 \n100206 S900 Q11 M 26-30 True 1 \n100307 Q1 Q01 F 26-30 True 1 \n100408 Q3 Q03 M 31-35 True 1 \n100610 S900 Q08 M 26-30 True 2 \n... ... ... ... ... ... ... \n992774 Q2 Q02 M 31-35 True 2 \n993675 S900 Q09 F 26-30 True 2 \n994273 S500 Q06 M 26-30 True 1 \n995174 S1200 Q13 M 22-25 False 1 \n996782 S900 Q08 F 26-30 True 2 \n\n T2_Count 3T_RS-fMRI_Count 3T_RS-fMRI_PctCompl 3T_Full_Task_fMRI \\\nSubject \n100004 0 0 0.0 False \n100206 1 4 100.0 True \n100307 1 4 100.0 True \n100408 1 4 100.0 True \n100610 1 4 100.0 True \n... ... ... ... ... \n992774 2 4 100.0 True \n993675 2 4 100.0 True \n994273 1 4 100.0 True \n995174 1 2 0.0 True \n996782 2 4 100.0 True \n\n ... Odor_Unadj Odor_AgeAdj PainIntens_RawScore PainInterf_Tscore \\\nSubject ... \n100004 ... 101.12 86.45 2.0 45.9 \n100206 ... 108.79 97.19 1.0 49.7 \n100307 ... 101.12 86.45 0.0 38.6 \n100408 ... 108.79 98.04 2.0 52.6 \n100610 ... 122.25 110.45 0.0 38.6 \n... ... ... ... ... ... \n992774 ... 122.25 111.41 4.0 50.1 \n993675 ... 122.25 110.45 0.0 38.6 \n994273 ... 122.25 111.41 7.0 63.8 \n995174 ... 88.61 64.58 3.0 50.1 \n996782 ... 108.79 97.19 0.0 38.6 \n\n Taste_Unadj Taste_AgeAdj Mars_Log_Score Mars_Errs Mars_Final \\\nSubject \n100004 107.17 105.31 1.80 0.0 1.80 \n100206 72.63 72.03 1.84 0.0 1.84 \n100307 71.69 71.76 1.76 0.0 1.76 \n100408 114.01 113.59 1.76 2.0 1.68 \n100610 84.84 85.31 1.92 1.0 1.88 \n... ... ... ... ... ... \n992774 107.17 103.55 1.76 0.0 1.76 \n993675 84.07 84.25 1.80 1.0 1.76 \n994273 110.65 109.73 1.80 1.0 1.76 \n995174 117.16 117.40 1.80 0.0 1.80 \n996782 75.43 73.72 1.84 0.0 1.84 \n\n age \nSubject \n100004 23.5 \n100206 28.0 \n100307 28.0 \n100408 33.0 \n100610 28.0 \n... ... \n992774 33.0 \n993675 28.0 \n994273 28.0 \n995174 23.5 \n996782 28.0 \n\n[1206 rows x 582 columns]", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
ReleaseAcquisitionGenderAge3T_Full_MR_ComplT1_CountT2_Count3T_RS-fMRI_Count3T_RS-fMRI_PctCompl3T_Full_Task_fMRI...Odor_UnadjOdor_AgeAdjPainIntens_RawScorePainInterf_TscoreTaste_UnadjTaste_AgeAdjMars_Log_ScoreMars_ErrsMars_Finalage
Subject
100004S900Q06M22-25False0000.0False...101.1286.452.045.9107.17105.311.800.01.8023.5
100206S900Q11M26-30True114100.0True...108.7997.191.049.772.6372.031.840.01.8428.0
100307Q1Q01F26-30True114100.0True...101.1286.450.038.671.6971.761.760.01.7628.0
100408Q3Q03M31-35True114100.0True...108.7998.042.052.6114.01113.591.762.01.6833.0
100610S900Q08M26-30True214100.0True...122.25110.450.038.684.8485.311.921.01.8828.0
..................................................................
992774Q2Q02M31-35True224100.0True...122.25111.414.050.1107.17103.551.760.01.7633.0
993675S900Q09F26-30True224100.0True...122.25110.450.038.684.0784.251.801.01.7628.0
994273S500Q06M26-30True114100.0True...122.25111.417.063.8110.65109.731.801.01.7628.0
995174S1200Q13M22-25False1120.0True...88.6164.583.050.1117.16117.401.800.01.8023.5
996782S900Q08F26-30True224100.0True...108.7997.190.038.675.4373.721.840.01.8428.0
\n

1206 rows × 582 columns

\n
" }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# HCP data can be obtained from the connectomeDB\n", "# data is not part of this repository\n", "subjectIDs = pd.read_csv('hcp_data/subjectIDs.txt', header=None)\n", "\n", "netmats_pearson = pd.read_csv('hcp_data/netmats1_correlationZ.txt',\n", " sep=' ',\n", " header=None)\n", "netmats_pearson['ID'] = subjectIDs[0]\n", "netmats_pearson.set_index('ID', drop=True, inplace=True)\n", "\n", "\n", "netmats_parcor = pd.read_csv('hcp_data/netmats2_partial-correlation.txt',\n", " sep=' ',\n", " header=None)\n", "netmats_parcor['ID'] = subjectIDs[0]\n", "netmats_parcor.set_index('ID', drop=True, inplace=True)\n", "\n", "behavior = pd.read_csv('hcp_data/hcp1200_behavioral_data.csv')\n", "behavior = behavior.set_index('Subject', drop=True)\n", "\n", "# convert age to numeric\n", "age = []\n", "for s in behavior['Age']:\n", " if s == '36+':\n", " age.append(36)\n", " else:\n", " split = s.split(sep='-')\n", " age.append(np.mean((float(split[0]), float(split[1]))))\n", "\n", "behavior['age'] = age\n", "behavior" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "# Function to prepare target variable\n" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 3, "outputs": [], "source": [ "def create_data(target='CogTotalComp_AgeAdj', feature_data=netmats_parcor):\n", " # it's a good practice to use pandas for merging, messing up subject order can be painful\n", " features = feature_data.columns\n", " df = behavior\n", " df = df.merge(feature_data, left_index=True, right_index=True, how='left')\n", "\n", " df = df.dropna(subset = [target] + features.values.tolist())\n", " y = df[target].values\n", " X = df[features].values\n", " return X, y" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "# Function implementing a single bootstrap iteration\n", "\n", "We define a workhorse function which:\n", "- randomly samples the discovery and the replication datasets,\n", "- creates cross-validated estimates of predictive performance within the discovery sample\n", "- finalizes the model by fitting it to the whole discovery sample (overfits the discovery but not the replication sample)\n", "- use it to predict the replication sample" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 4, "outputs": [], "source": [ "def bootstrap_workhorse(X, y, sample_size, model, random_state, shuffle_y=False):\n", "\n", " #create discovery and replication samples by random sampling from the whole dataset (without replacement)\n", "\n", " if shuffle_y:\n", " rng = np.random.default_rng(random_state)\n", " y = rng.permutation(y)\n", "\n", " X_discovery, X_replication, y_discovery, y_replication = train_test_split(X, y, train_size=sample_size, test_size=sample_size, shuffle=True, random_state=random_state)\n", "\n", " cv = KFold(10)\n", " # obtain cross-validated predictions in the discovery sample\n", "\n", " predicted_discovery_cv = np.zeros_like(y_discovery)\n", " cor_per_fold = np.zeros(cv.n_splits)\n", " i = 0\n", " for train, test in cv.split(X=X_discovery, y=y_discovery):\n", " model.fit(X=X_discovery[train], y=y_discovery[train])\n", " predicted_discovery_cv[test] = model.predict(X=X_discovery[test])\n", " cor_per_fold[i] = np.corrcoef(y_discovery[test], predicted_discovery_cv[test])[0,1]\n", " i += 1\n", " # correlation between the cross-validated predictions and observations in the discovery sample\n", " # this is the correct, unbiased estimate!\n", " # calculated as mean test performance across all folds\n", " r_disc_cv = np.mean(cor_per_fold)\n", " # finalize model by training it on the full discovery sample (without cross-validation)\n", " final_model = model.fit(X=X_discovery, y=y_discovery)\n", " # obtain predictions with the final model on the discovery sample, note that this model actually overfits this sample.\n", " # we do this only to demonstrate biased estimates\n", " predicted_discovery_overfit = final_model.predict(X=X_discovery)\n", " # here we obtain the biased effect size (r) estimates for demonstrational purposes\n", " r_disc_overfit = np.corrcoef(predicted_discovery_overfit, y_discovery)[0, 1]\n", "\n", " # We use the final model to predict the replication sample\n", " # This is correct (no overfitting here), the final model did not see this data during training\n", " predicted_replication = final_model.predict(X=X_replication)\n", " # we obtain the out-of-sample prediction performance estimates\n", " r_rep = np.corrcoef(predicted_replication, y_replication)[0, 1]\n", "\n", " # below we calculate permutation-based p-values for all three effect size estimates (in-sample unbiased, in-sample biased, out-of-sample)\n", " # (one sided tests, testing for positive correlation)\n", " p_disc_cv = permutation_test(predicted_discovery_cv, y_discovery, method='approximate', num_rounds=1000, func=lambda x, y: np.corrcoef(x, y)[1][0],seed=random_state)\n", " p_disc_overfit = permutation_test(predicted_discovery_overfit, y_discovery, method='approximate', num_rounds=1000, func=lambda x, y: np.corrcoef(x, y)[1][0],seed=random_state)\n", " p_rep = permutation_test(predicted_replication, y_replication, method='approximate', num_rounds=1000, func=lambda x, y: np.corrcoef(x, y)[1][0],seed=random_state)\n", " # return results\n", " return r_disc_cv, r_disc_overfit, r_rep, p_disc_cv, p_disc_overfit, p_rep" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "All set, now we start the analysis." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "markdown", "source": [ "# Replicability with sample sizes n=25, 50, 75, 100, 125, 150, 175, 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475 and max\n", "Here we train a few different models on 100 bootstrap samples.\n", "\n", "We aggregate the results of our workhorse function in `n_bootstrap`=100 bootstrap cases (run in parallel).\n", "\n", "The whole process is repeated for all sample sizes, fetaure_sets and target variables.\n", "\n", "## Here we test age and 5 cognitive variables, including 'cognitive ability' (the main target variable in the target paper)\n", "- age: age group of the participants\n", "- CogTotalComp_AgeAdj: total cognitive ability\n", "- PMAT24_A_CR, : Fluid Intelligence (Penn Progressive Matrices)\n", "- CardSort_AgeAdj: Executive Function/Cognitive Flexibility (Dimensional Change Card Sort)\n", "- Flanker_AgeAdj: Executive Function/Inhibition (Flanker Task)\n", "- PicSeq_AgeAdj: Episodic Memory (Picture Sequence Memory)" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "markdown", "source": [ "# Reproducing the PCA+SVR-based model from the target paper\n", "### Like in the target paper:\n", "- Both PCA and SVR are done inside the cross-validation\n", "- PCA reatains the firts k principal components that together explain 50% of the variance\n", "- scikit-learn makes sure that PCA is only fit for the training samples\n", "- both for the test sets (in the cross-validation) and the replication sample PCA is not re-fit, bt features are simply transformed with the already fit PCA" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 8, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "*****************************************************************\n", "netmats_pearson PCA_SVR age 25\n", "0.07814693256464597 0.06311660130821281\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 50\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.0034788424473824646 0.05507531631070878\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 75\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":59: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.05767555332048385 0.06778677053397866\n", "Replicability at alpha = 0.05 : 33.33333333333333 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 100\n", "0.06510214673673408 0.07992169563215587\n", "Replicability at alpha = 0.05 : 10.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 125\n", "0.08398882831710941 0.10534949872581784\n", "Replicability at alpha = 0.05 : 37.5 %\n", "Replicability at alpha = 0.01 : 6.25 %\n", "Replicability at alpha = 0.005 : 6.25 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 150\n", "0.11257418326571461 0.11541960536894891\n", "Replicability at alpha = 0.05 : 52.0 %\n", "Replicability at alpha = 0.01 : 4.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 175\n", "0.1155368141107679 0.12651956981630005\n", "Replicability at alpha = 0.05 : 55.172413793103445 %\n", "Replicability at alpha = 0.01 : 17.24137931034483 %\n", "Replicability at alpha = 0.005 : 13.793103448275861 %\n", "Replicability at alpha = 0.001 : 3.4482758620689653 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 200\n", "0.1114303048333668 0.1398647058199516\n", "Replicability at alpha = 0.05 : 64.70588235294117 %\n", "Replicability at alpha = 0.01 : 26.47058823529412 %\n", "Replicability at alpha = 0.005 : 11.76470588235294 %\n", "Replicability at alpha = 0.001 : 2.941176470588235 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 225\n", "0.11751236076225453 0.1413404590814415\n", "Replicability at alpha = 0.05 : 69.04761904761905 %\n", "Replicability at alpha = 0.01 : 21.428571428571427 %\n", "Replicability at alpha = 0.005 : 16.666666666666664 %\n", "Replicability at alpha = 0.001 : 4.761904761904762 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 250\n", "0.13499620393072262 0.15429199592664644\n", "Replicability at alpha = 0.05 : 78.26086956521739 %\n", "Replicability at alpha = 0.01 : 36.95652173913043 %\n", "Replicability at alpha = 0.005 : 28.26086956521739 %\n", "Replicability at alpha = 0.001 : 13.043478260869565 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 275\n", "0.14903193208079654 0.15447041039381318\n", "Replicability at alpha = 0.05 : 83.33333333333334 %\n", "Replicability at alpha = 0.01 : 25.757575757575758 %\n", "Replicability at alpha = 0.005 : 19.696969696969695 %\n", "Replicability at alpha = 0.001 : 7.575757575757576 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 300\n", "0.15601435339697584 0.15810854528637436\n", "Replicability at alpha = 0.05 : 89.1891891891892 %\n", "Replicability at alpha = 0.01 : 40.54054054054054 %\n", "Replicability at alpha = 0.005 : 31.08108108108108 %\n", "Replicability at alpha = 0.001 : 12.162162162162163 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 325\n", "0.1671812924407429 0.16278343644645824\n", "Replicability at alpha = 0.05 : 91.02564102564102 %\n", "Replicability at alpha = 0.01 : 58.97435897435898 %\n", "Replicability at alpha = 0.005 : 41.02564102564102 %\n", "Replicability at alpha = 0.001 : 14.102564102564102 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 350\n", "0.1718316632124011 0.1731928713530674\n", "Replicability at alpha = 0.05 : 94.11764705882352 %\n", "Replicability at alpha = 0.01 : 63.52941176470588 %\n", "Replicability at alpha = 0.005 : 52.94117647058824 %\n", "Replicability at alpha = 0.001 : 18.823529411764707 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 375\n", "0.17122668000377325 0.17928790504832592\n", "Replicability at alpha = 0.05 : 97.6470588235294 %\n", "Replicability at alpha = 0.01 : 76.47058823529412 %\n", "Replicability at alpha = 0.005 : 63.52941176470588 %\n", "Replicability at alpha = 0.001 : 23.52941176470588 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 400\n", "0.17065337167611985 0.18509558942644763\n", "Replicability at alpha = 0.05 : 97.72727272727273 %\n", "Replicability at alpha = 0.01 : 79.54545454545455 %\n", "Replicability at alpha = 0.005 : 64.77272727272727 %\n", "Replicability at alpha = 0.001 : 32.95454545454545 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 425\n", "0.17076394986094715 0.1902224982376579\n", "Replicability at alpha = 0.05 : 98.82352941176471 %\n", "Replicability at alpha = 0.01 : 78.82352941176471 %\n", "Replicability at alpha = 0.005 : 69.41176470588235 %\n", "Replicability at alpha = 0.001 : 41.17647058823529 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 450\n", "0.17308250538661116 0.19597602750039833\n", "Replicability at alpha = 0.05 : 100.0 %\n", "Replicability at alpha = 0.01 : 81.52173913043478 %\n", "Replicability at alpha = 0.005 : 71.73913043478261 %\n", "Replicability at alpha = 0.001 : 45.65217391304348 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age 475\n", "0.18081606562456337 0.19381850490149558\n", "Replicability at alpha = 0.05 : 98.93617021276596 %\n", "Replicability at alpha = 0.01 : 91.48936170212765 %\n", "Replicability at alpha = 0.005 : 88.29787234042553 %\n", "Replicability at alpha = 0.001 : 60.63829787234043 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR age max\n", "0.18282912012661387 0.19084783442850575\n", "Replicability at alpha = 0.05 : 100.0 %\n", "Replicability at alpha = 0.01 : 85.56701030927834 %\n", "Replicability at alpha = 0.005 : 82.4742268041237 %\n", "Replicability at alpha = 0.001 : 56.70103092783505 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 25\n", "-0.012765746580277846 0.052533016946524666\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 50\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":59: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.02613076180866329 0.07257458992180708\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 75\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":59: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.035862602309382 0.06095291179965927\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 100\n", "0.04254487476856877 0.06349813361022945\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 125\n", "0.04149677352346028 0.07303200757265954\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 150\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":59: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.06587736162122486 0.07685151622875608\n", "Replicability at alpha = 0.05 : 16.666666666666664 %\n", "Replicability at alpha = 0.01 : 16.666666666666664 %\n", "Replicability at alpha = 0.005 : 16.666666666666664 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 175\n", "0.09249691698008317 0.09297984386331187\n", "Replicability at alpha = 0.05 : 50.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 200\n", "0.0801394538604939 0.1061294190223969\n", "Replicability at alpha = 0.05 : 25.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 225\n", "0.08863558904442431 0.11237818873176193\n", "Replicability at alpha = 0.05 : 33.33333333333333 %\n", "Replicability at alpha = 0.01 : 14.285714285714285 %\n", "Replicability at alpha = 0.005 : 4.761904761904762 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 250\n", "0.1044939349453288 0.12434596149490698\n", "Replicability at alpha = 0.05 : 51.85185185185185 %\n", "Replicability at alpha = 0.01 : 11.11111111111111 %\n", "Replicability at alpha = 0.005 : 3.7037037037037033 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 275\n", "0.11955052745942898 0.1359523201921851\n", "Replicability at alpha = 0.05 : 64.86486486486487 %\n", "Replicability at alpha = 0.01 : 18.91891891891892 %\n", "Replicability at alpha = 0.005 : 8.108108108108109 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 300\n", "0.12371681381416577 0.13372831832326426\n", "Replicability at alpha = 0.05 : 62.16216216216216 %\n", "Replicability at alpha = 0.01 : 18.91891891891892 %\n", "Replicability at alpha = 0.005 : 10.81081081081081 %\n", "Replicability at alpha = 0.001 : 2.7027027027027026 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 325\n", "0.13093104271229364 0.14009832266371144\n", "Replicability at alpha = 0.05 : 72.3404255319149 %\n", "Replicability at alpha = 0.01 : 23.404255319148938 %\n", "Replicability at alpha = 0.005 : 14.893617021276595 %\n", "Replicability at alpha = 0.001 : 4.25531914893617 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 350\n", "0.15088912196146678 0.15036264409074854\n", "Replicability at alpha = 0.05 : 80.0 %\n", "Replicability at alpha = 0.01 : 36.666666666666664 %\n", "Replicability at alpha = 0.005 : 23.333333333333332 %\n", "Replicability at alpha = 0.001 : 10.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 375\n", "0.15597119585148086 0.1576965059117357\n", "Replicability at alpha = 0.05 : 90.9090909090909 %\n", "Replicability at alpha = 0.01 : 43.93939393939394 %\n", "Replicability at alpha = 0.005 : 33.33333333333333 %\n", "Replicability at alpha = 0.001 : 6.0606060606060606 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 400\n", "0.15569714936227072 0.16274303113694116\n", "Replicability at alpha = 0.05 : 95.65217391304348 %\n", "Replicability at alpha = 0.01 : 50.72463768115942 %\n", "Replicability at alpha = 0.005 : 34.78260869565217 %\n", "Replicability at alpha = 0.001 : 8.695652173913043 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 425\n", "0.1588211600842311 0.1620844233266087\n", "Replicability at alpha = 0.05 : 93.05555555555556 %\n", "Replicability at alpha = 0.01 : 66.66666666666666 %\n", "Replicability at alpha = 0.005 : 48.61111111111111 %\n", "Replicability at alpha = 0.001 : 11.11111111111111 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 450\n", "0.1660449095766733 0.16863492603140337\n", "Replicability at alpha = 0.05 : 95.34883720930233 %\n", "Replicability at alpha = 0.01 : 67.44186046511628 %\n", "Replicability at alpha = 0.005 : 51.162790697674424 %\n", "Replicability at alpha = 0.001 : 18.6046511627907 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 475\n", "0.1639794666728734 0.17409341882316898\n", "Replicability at alpha = 0.05 : 98.82352941176471 %\n", "Replicability at alpha = 0.01 : 74.11764705882354 %\n", "Replicability at alpha = 0.005 : 55.294117647058826 %\n", "Replicability at alpha = 0.001 : 25.882352941176475 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj max\n", "0.1657456085840359 0.1759580210438174\n", "Replicability at alpha = 0.05 : 100.0 %\n", "Replicability at alpha = 0.01 : 72.82608695652173 %\n", "Replicability at alpha = 0.005 : 58.69565217391305 %\n", "Replicability at alpha = 0.001 : 29.347826086956523 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 25\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.009622651031827124 0.006527399552072927\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 50\n", "0.01140704603289156 0.03937358302729429\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 75\n", "0.03562457718858567 0.03201261112244902\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 100\n", "0.0654201392870136 0.05283547742503512\n", "Replicability at alpha = 0.05 : 18.75 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 125\n", "0.06948704868304105 0.07003839749164883\n", "Replicability at alpha = 0.05 : 10.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 150\n", "0.07875099797108849 0.08901475342266543\n", "Replicability at alpha = 0.05 : 26.666666666666668 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 175\n", "0.079165374473938 0.09873289660325801\n", "Replicability at alpha = 0.05 : 41.17647058823529 %\n", "Replicability at alpha = 0.01 : 11.76470588235294 %\n", "Replicability at alpha = 0.005 : 5.88235294117647 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 200\n", "0.08398499336458215 0.09991315410566606\n", "Replicability at alpha = 0.05 : 57.14285714285714 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 225\n", "0.09610454017613254 0.1140036312432465\n", "Replicability at alpha = 0.05 : 55.55555555555556 %\n", "Replicability at alpha = 0.01 : 18.51851851851852 %\n", "Replicability at alpha = 0.005 : 11.11111111111111 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 250\n", "0.1088488350425472 0.12228373369465263\n", "Replicability at alpha = 0.05 : 58.333333333333336 %\n", "Replicability at alpha = 0.01 : 25.0 %\n", "Replicability at alpha = 0.005 : 11.11111111111111 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 275\n", "0.11805418247347252 0.12985218319730554\n", "Replicability at alpha = 0.05 : 68.18181818181817 %\n", "Replicability at alpha = 0.01 : 20.454545454545457 %\n", "Replicability at alpha = 0.005 : 11.363636363636363 %\n", "Replicability at alpha = 0.001 : 4.545454545454546 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 300\n", "0.11497738950538995 0.14153398369488263\n", "Replicability at alpha = 0.05 : 78.72340425531915 %\n", "Replicability at alpha = 0.01 : 34.04255319148936 %\n", "Replicability at alpha = 0.005 : 27.659574468085108 %\n", "Replicability at alpha = 0.001 : 2.127659574468085 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 325\n", "0.12332124492463431 0.14300465290889006\n", "Replicability at alpha = 0.05 : 82.35294117647058 %\n", "Replicability at alpha = 0.01 : 27.450980392156865 %\n", "Replicability at alpha = 0.005 : 21.568627450980394 %\n", "Replicability at alpha = 0.001 : 1.9607843137254901 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 350\n", "0.13352263698943231 0.16051680473137572\n", "Replicability at alpha = 0.05 : 87.87878787878788 %\n", "Replicability at alpha = 0.01 : 31.818181818181817 %\n", "Replicability at alpha = 0.005 : 18.181818181818183 %\n", "Replicability at alpha = 0.001 : 6.0606060606060606 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 375\n", "0.14291474184848016 0.16718204008058787\n", "Replicability at alpha = 0.05 : 94.87179487179486 %\n", "Replicability at alpha = 0.01 : 44.871794871794876 %\n", "Replicability at alpha = 0.005 : 25.64102564102564 %\n", "Replicability at alpha = 0.001 : 10.256410256410255 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 400\n", "0.15238717524605863 0.17155022123378255\n", "Replicability at alpha = 0.05 : 98.78048780487805 %\n", "Replicability at alpha = 0.01 : 60.97560975609756 %\n", "Replicability at alpha = 0.005 : 40.243902439024396 %\n", "Replicability at alpha = 0.001 : 14.634146341463413 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 425\n", "0.1530500523323691 0.1725480868980855\n", "Replicability at alpha = 0.05 : 97.61904761904762 %\n", "Replicability at alpha = 0.01 : 77.38095238095238 %\n", "Replicability at alpha = 0.005 : 53.57142857142857 %\n", "Replicability at alpha = 0.001 : 16.666666666666664 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 450\n", "0.15706365287150292 0.17360298447608316\n", "Replicability at alpha = 0.05 : 98.86363636363636 %\n", "Replicability at alpha = 0.01 : 72.72727272727273 %\n", "Replicability at alpha = 0.005 : 60.22727272727273 %\n", "Replicability at alpha = 0.001 : 25.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 475\n", "0.1630312924190221 0.17825945122880046\n", "Replicability at alpha = 0.05 : 98.88888888888889 %\n", "Replicability at alpha = 0.01 : 81.11111111111111 %\n", "Replicability at alpha = 0.005 : 67.77777777777779 %\n", "Replicability at alpha = 0.001 : 31.11111111111111 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR max\n", "0.16937260790985878 0.17931030150814908\n", "Replicability at alpha = 0.05 : 100.0 %\n", "Replicability at alpha = 0.01 : 81.91489361702128 %\n", "Replicability at alpha = 0.005 : 76.59574468085107 %\n", "Replicability at alpha = 0.001 : 42.5531914893617 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 25\n", "-0.0019028872213924125 -0.008753460718655168\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 50\n", "0.0041877588769700476 0.026446681706768983\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 75\n", "0.026825092375948947 0.03767071318825717\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 100\n", "0.017319046912004265 0.037796860369966837\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 125\n", "0.032955243766298 0.04020703484126933\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 150\n", "0.020533910962801338 0.03458890797600347\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 175\n", "0.025742836566290585 0.039566210794864866\n", "Replicability at alpha = 0.05 : 16.666666666666664 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 200\n", "0.04000162013366887 0.04785239767856528\n", "Replicability at alpha = 0.05 : 16.666666666666664 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 225\n", "0.049234767880881325 0.050938982294854905\n", "Replicability at alpha = 0.05 : 25.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 250\n", "0.06096902073007601 0.060499437456216644\n", "Replicability at alpha = 0.05 : 6.666666666666667 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 275\n", "0.06269480460047201 0.06623663681059051\n", "Replicability at alpha = 0.05 : 25.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 300\n", "0.06640491375871052 0.07393753936413099\n", "Replicability at alpha = 0.05 : 18.181818181818183 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 325\n", "0.06410086612576932 0.07828881514077404\n", "Replicability at alpha = 0.05 : 16.666666666666664 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 350\n", "0.07516133328180917 0.087532863488003\n", "Replicability at alpha = 0.05 : 28.57142857142857 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 375\n", "0.07251952508522898 0.09425589704444427\n", "Replicability at alpha = 0.05 : 44.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 400\n", "0.07964488965307073 0.0927428634369964\n", "Replicability at alpha = 0.05 : 56.666666666666664 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 425\n", "0.08959860531327472 0.0938875787742953\n", "Replicability at alpha = 0.05 : 55.55555555555556 %\n", "Replicability at alpha = 0.01 : 11.11111111111111 %\n", "Replicability at alpha = 0.005 : 2.7777777777777777 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 450\n", "0.09214835841629018 0.09710394400944428\n", "Replicability at alpha = 0.05 : 68.18181818181817 %\n", "Replicability at alpha = 0.01 : 4.545454545454546 %\n", "Replicability at alpha = 0.005 : 2.272727272727273 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 475\n", "0.09315563823606367 0.0981352592158315\n", "Replicability at alpha = 0.05 : 62.7906976744186 %\n", "Replicability at alpha = 0.01 : 4.651162790697675 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj max\n", "0.0947098828552236 0.10203836949356383\n", "Replicability at alpha = 0.05 : 61.224489795918366 %\n", "Replicability at alpha = 0.01 : 8.16326530612245 %\n", "Replicability at alpha = 0.005 : 4.081632653061225 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 25\n", "0.017427440494622005 0.04635309528134357\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 50\n", "0.013298678784128739 0.052864559935567525\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 75\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":59: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.06975060074320828 0.033568783195742524\n", "Replicability at alpha = 0.05 : 20.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 100\n", "0.06624026266001 0.05511154998176417\n", "Replicability at alpha = 0.05 : 60.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 125\n", "0.06402370847302795 0.07293905202224414\n", "Replicability at alpha = 0.05 : 20.0 %\n", "Replicability at alpha = 0.01 : 10.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 150\n", "0.07582578628232152 0.07617880223036112\n", "Replicability at alpha = 0.05 : 28.57142857142857 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 175\n", "0.08214592649616946 0.07228731176326164\n", "Replicability at alpha = 0.05 : 13.333333333333334 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 200\n", "0.0753583027418951 0.07325950038420452\n", "Replicability at alpha = 0.05 : 45.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 225\n", "0.08637113927170104 0.07568098334978508\n", "Replicability at alpha = 0.05 : 38.095238095238095 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 250\n", "0.08213338572959444 0.07938886872508781\n", "Replicability at alpha = 0.05 : 25.0 %\n", "Replicability at alpha = 0.01 : 4.166666666666666 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 275\n", "0.07926130021302251 0.09196331693681625\n", "Replicability at alpha = 0.05 : 35.714285714285715 %\n", "Replicability at alpha = 0.01 : 7.142857142857142 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 300\n", "0.08376531584402028 0.09085835147403382\n", "Replicability at alpha = 0.05 : 38.46153846153847 %\n", "Replicability at alpha = 0.01 : 3.8461538461538463 %\n", "Replicability at alpha = 0.005 : 3.8461538461538463 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 325\n", "0.07707530022995264 0.09320999915319664\n", "Replicability at alpha = 0.05 : 38.095238095238095 %\n", "Replicability at alpha = 0.01 : 14.285714285714285 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 350\n", "0.0841396096406637 0.09349802960081627\n", "Replicability at alpha = 0.05 : 43.333333333333336 %\n", "Replicability at alpha = 0.01 : 6.666666666666667 %\n", "Replicability at alpha = 0.005 : 3.3333333333333335 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 375\n", "0.08814404029494786 0.09530660548395765\n", "Replicability at alpha = 0.05 : 42.10526315789473 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 400\n", "0.09240300272718435 0.09578858550347505\n", "Replicability at alpha = 0.05 : 48.717948717948715 %\n", "Replicability at alpha = 0.01 : 5.128205128205128 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 425\n", "0.09544017127530559 0.09794898999887712\n", "Replicability at alpha = 0.05 : 60.416666666666664 %\n", "Replicability at alpha = 0.01 : 6.25 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 450\n", "0.0994420545269816 0.10088886217363773\n", "Replicability at alpha = 0.05 : 63.63636363636363 %\n", "Replicability at alpha = 0.01 : 5.454545454545454 %\n", "Replicability at alpha = 0.005 : 1.8181818181818181 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 475\n", "0.09993961752844732 0.10419426452299732\n", "Replicability at alpha = 0.05 : 64.28571428571429 %\n", "Replicability at alpha = 0.01 : 10.714285714285714 %\n", "Replicability at alpha = 0.005 : 3.571428571428571 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj max\n", "0.10509717705029202 0.10638876826754434\n", "Replicability at alpha = 0.05 : 71.66666666666667 %\n", "Replicability at alpha = 0.01 : 10.0 %\n", "Replicability at alpha = 0.005 : 5.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 25\n", "0.046214651785071836 0.003473316915260759\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 50\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":59: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.030554472440857624 0.03493508595660751\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 75\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":59: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.04865325647225168 0.05459780590717687\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 100\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":59: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.047759076983809795 0.06147545482107202\n", "Replicability at alpha = 0.05 : 20.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 125\n", "0.04080505903681604 0.06367701849523064\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 150\n", "0.045843724317538435 0.06137533503879337\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 175\n", "0.032719090843903745 0.06389907075091782\n", "Replicability at alpha = 0.05 : 16.666666666666664 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 200\n", "0.05474657301765336 0.06749833856170852\n", "Replicability at alpha = 0.05 : 42.857142857142854 %\n", "Replicability at alpha = 0.01 : 14.285714285714285 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 225\n", "0.0654340978662934 0.07025910568145566\n", "Replicability at alpha = 0.05 : 50.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 250\n", "0.07194870263258932 0.07938495413482728\n", "Replicability at alpha = 0.05 : 10.0 %\n", "Replicability at alpha = 0.01 : 10.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 275\n", "0.07097419489735 0.07551583367253861\n", "Replicability at alpha = 0.05 : 25.0 %\n", "Replicability at alpha = 0.01 : 8.333333333333332 %\n", "Replicability at alpha = 0.005 : 8.333333333333332 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 300\n", "0.06991029950177208 0.07458636299063129\n", "Replicability at alpha = 0.05 : 41.66666666666667 %\n", "Replicability at alpha = 0.01 : 8.333333333333332 %\n", "Replicability at alpha = 0.005 : 8.333333333333332 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 325\n", "0.08308090703915044 0.081171907729014\n", "Replicability at alpha = 0.05 : 47.61904761904761 %\n", "Replicability at alpha = 0.01 : 14.285714285714285 %\n", "Replicability at alpha = 0.005 : 4.761904761904762 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 350\n", "0.08221829092335195 0.08688769951430177\n", "Replicability at alpha = 0.05 : 56.25 %\n", "Replicability at alpha = 0.01 : 31.25 %\n", "Replicability at alpha = 0.005 : 12.5 %\n", "Replicability at alpha = 0.001 : 6.25 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 375\n", "0.08468767921209663 0.09236588860311817\n", "Replicability at alpha = 0.05 : 68.75 %\n", "Replicability at alpha = 0.01 : 12.5 %\n", "Replicability at alpha = 0.005 : 12.5 %\n", "Replicability at alpha = 0.001 : 3.125 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 400\n", "0.0869689899820891 0.09318486322526578\n", "Replicability at alpha = 0.05 : 65.625 %\n", "Replicability at alpha = 0.01 : 9.375 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 425\n", "0.08270536184135027 0.10051028430101452\n", "Replicability at alpha = 0.05 : 61.29032258064516 %\n", "Replicability at alpha = 0.01 : 9.67741935483871 %\n", "Replicability at alpha = 0.005 : 6.451612903225806 %\n", "Replicability at alpha = 0.001 : 3.225806451612903 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 450\n", "0.08636275270970561 0.09730143031198002\n", "Replicability at alpha = 0.05 : 57.57575757575758 %\n", "Replicability at alpha = 0.01 : 9.090909090909092 %\n", "Replicability at alpha = 0.005 : 3.0303030303030303 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 475\n", "0.08497287378671342 0.10125962454994882\n", "Replicability at alpha = 0.05 : 62.5 %\n", "Replicability at alpha = 0.01 : 18.75 %\n", "Replicability at alpha = 0.005 : 3.125 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj max\n", "0.09227559512484998 0.10077547292317074\n", "Replicability at alpha = 0.05 : 56.81818181818182 %\n", "Replicability at alpha = 0.01 : 11.363636363636363 %\n", "Replicability at alpha = 0.005 : 2.272727272727273 %\n", "Replicability at alpha = 0.001 : 2.272727272727273 %\n", "CPU times: user 53.4 s, sys: 9.85 s, total: 1min 3s\n", "Wall time: 7h 35min 52s\n" ] }, { "data": { "text/plain": " connectivity model target n r_discovery_cv \\\n0 netmats_pearson PCA_SVR age 25 NaN \n1 netmats_pearson PCA_SVR age 25 NaN \n2 netmats_pearson PCA_SVR age 25 NaN \n3 netmats_pearson PCA_SVR age 25 NaN \n4 netmats_pearson PCA_SVR age 25 NaN \n... ... ... ... ... ... \n11995 netmats_pearson PCA_SVR PicSeq_AgeAdj 501 0.170381 \n11996 netmats_pearson PCA_SVR PicSeq_AgeAdj 501 0.095436 \n11997 netmats_pearson PCA_SVR PicSeq_AgeAdj 501 0.098129 \n11998 netmats_pearson PCA_SVR PicSeq_AgeAdj 501 0.12842 \n11999 netmats_pearson PCA_SVR PicSeq_AgeAdj 501 0.07409 \n\n r_discovery_overfit r_replication p_discovery_cv p_discovery_overfit \\\n0 0.466156 -0.139314 0.995005 0.017982 \n1 0.689387 0.196194 0.94006 0.000999 \n2 0.681033 0.064105 0.944056 0.000999 \n3 0.513247 0.003161 0.998002 0.008991 \n4 0.870617 0.063318 1.0 0.000999 \n... ... ... ... ... \n11995 0.360809 0.062856 0.004995 0.000999 \n11996 0.352693 0.142834 0.085914 0.000999 \n11997 0.373386 0.163908 0.028971 0.000999 \n11998 0.376163 0.109268 0.065934 0.000999 \n11999 0.308118 0.032076 0.347652 0.000999 \n\n p_replication \n0 0.757243 \n1 0.180819 \n2 0.313686 \n3 0.526474 \n4 0.376623 \n... ... \n11995 0.085914 \n11996 0.000999 \n11997 0.000999 \n11998 0.012987 \n11999 0.225774 \n\n[12000 rows x 10 columns]", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
connectivitymodeltargetnr_discovery_cvr_discovery_overfitr_replicationp_discovery_cvp_discovery_overfitp_replication
0netmats_pearsonPCA_SVRage25NaN0.466156-0.1393140.9950050.0179820.757243
1netmats_pearsonPCA_SVRage25NaN0.6893870.1961940.940060.0009990.180819
2netmats_pearsonPCA_SVRage25NaN0.6810330.0641050.9440560.0009990.313686
3netmats_pearsonPCA_SVRage25NaN0.5132470.0031610.9980020.0089910.526474
4netmats_pearsonPCA_SVRage25NaN0.8706170.0633181.00.0009990.376623
.................................
11995netmats_pearsonPCA_SVRPicSeq_AgeAdj5010.1703810.3608090.0628560.0049950.0009990.085914
11996netmats_pearsonPCA_SVRPicSeq_AgeAdj5010.0954360.3526930.1428340.0859140.0009990.000999
11997netmats_pearsonPCA_SVRPicSeq_AgeAdj5010.0981290.3733860.1639080.0289710.0009990.000999
11998netmats_pearsonPCA_SVRPicSeq_AgeAdj5010.128420.3761630.1092680.0659340.0009990.012987
11999netmats_pearsonPCA_SVRPicSeq_AgeAdj5010.074090.3081180.0320760.3476520.0009990.225774
\n

12000 rows × 10 columns

\n
" }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time\n", "\n", "random_state = 42\n", "n_bootstrap = 100\n", "\n", "features = {\n", " 'netmats_pearson': netmats_pearson\n", "}\n", "\n", "models = {\n", " 'PCA_SVR': Pipeline([('pca', PCA(n_components=0.5)),\n", " ('svr', SVR())])\n", "\n", "}\n", "\n", "# We aggregate all results here:\n", "df = pd.DataFrame(columns=['connectivity','model','target','n','r_discovery_cv','r_discovery_overfit','r_replication','p_discovery_cv','p_discovery_overfit','p_replication'])\n", "\n", "for feature_set in features:\n", " for model in models:\n", " for target_var in ['age', 'CogTotalComp_AgeAdj', 'PMAT24_A_CR', 'Flanker_AgeAdj', 'CardSort_AgeAdj', 'PicSeq_AgeAdj']:\n", " for sample_size in [25, 50, 75, 100, 125, 150, 175, 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475, 'max']:\n", "\n", " print('*****************************************************************')\n", " print(feature_set, model, target_var, sample_size)\n", "\n", " X, y = create_data(target=target_var, feature_data=features[feature_set])\n", "\n", " if sample_size=='max':\n", " sample_size = int(len(y)/2)\n", "\n", " # create random seeds for each bootstrap iteration for reproducibility\n", " rng = np.random.default_rng(random_state)\n", " random_sates = rng.integers(np.iinfo(np.int32).max, size=n_bootstrap)\n", "\n", " # run bootstrap iterations in parallel\n", " r_discovery_cv, r_discovery_overfit, r_replication, p_discovery_cv, p_discovery_overfit, p_replication = zip(\n", " *Parallel(n_jobs=-1)(\n", " delayed(bootstrap_workhorse)(X, y, sample_size, models[model], seed) for seed in random_sates))\n", "\n", " tmp_data_frame = pd.DataFrame({\n", " 'connectivity' : feature_set,\n", " 'model' : model,\n", " 'target' : target_var,\n", " 'n' : sample_size,\n", " 'r_discovery_cv': r_discovery_cv,\n", " 'r_discovery_overfit': r_discovery_overfit,\n", " 'r_replication': r_replication,\n", " 'p_discovery_cv': p_discovery_cv,\n", " 'p_discovery_overfit': p_discovery_overfit,\n", " 'p_replication': p_replication\n", " })\n", " #sns.scatterplot(x='r_replication', y='r_discovery_cv', data=tmp_data_frame)\n", " #plt.ylabel('in-sample (r)')\n", " #plt.xlabel('out-of-sample (r_pred)')\n", " #plt.show()\n", " print(tmp_data_frame.r_discovery_cv.mean(), tmp_data_frame.r_replication.mean())\n", "\n", " for alpha in [0.05, 0.01, 0.005, 0.001]:\n", " print('Replicability at alpha =', alpha, ':',\n", " (tmp_data_frame.loc[tmp_data_frame['p_discovery_cv']\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
connectivitymodeltargetnr_discovery_cvr_discovery_overfitr_replicationp_discovery_cvp_discovery_overfitp_replication
0netmats_parcorridgeage25NaN1.00.4207250.39960.0009990.01998
1netmats_parcorridgeage25NaN1.00.1833040.0059940.0009990.197802
2netmats_parcorridgeage25NaN1.00.2611310.339660.0009990.106893
3netmats_parcorridgeage25NaN1.00.0101330.3166830.0009990.518482
4netmats_parcorridgeage25NaN1.00.2674321.00.0009990.107892
.................................
11995netmats_parcorridgePicSeq_AgeAdj5010.1692421.00.2145010.0019980.0009990.000999
11996netmats_parcorridgePicSeq_AgeAdj5010.2361931.00.174450.0009990.0009990.000999
11997netmats_parcorridgePicSeq_AgeAdj5010.2012981.00.2655690.0009990.0009990.000999
11998netmats_parcorridgePicSeq_AgeAdj5010.2610041.00.197190.0009990.0009990.000999
11999netmats_parcorridgePicSeq_AgeAdj5010.1081831.00.2036050.0119880.0009990.000999
\n

12000 rows × 10 columns

\n" }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time\n", "\n", "random_state = 42\n", "n_bootstrap = 100\n", "\n", "features = {\n", " 'netmats_parcor': netmats_parcor,\n", "}\n", "\n", "models = {\n", " 'ridge': Ridge()\n", "}\n", "\n", "# We aggregate all results here:\n", "df = pd.DataFrame(columns=['connectivity','model','target','n','r_discovery_cv','r_discovery_overfit','r_replication','p_discovery_cv','p_discovery_overfit','p_replication'])\n", "\n", "for feature_set in features:\n", " for model in models:\n", " for target_var in ['age', 'CogTotalComp_AgeAdj', 'PMAT24_A_CR', 'Flanker_AgeAdj', 'CardSort_AgeAdj', 'PicSeq_AgeAdj']:\n", " for sample_size in [25, 50, 75, 100, 125, 150, 175, 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475, 'max']:\n", "\n", " print('*****************************************************************')\n", " print(feature_set, model, target_var, sample_size)\n", "\n", " X, y = create_data(target=target_var, feature_data=features[feature_set])\n", "\n", " if sample_size=='max':\n", " sample_size = int(len(y)/2)\n", "\n", " # create random seeds for each bootstrap iteration for reproducibility\n", " rng = np.random.default_rng(random_state)\n", " random_sates = rng.integers(np.iinfo(np.int32).max, size=n_bootstrap)\n", "\n", " # run bootstrap iterations in parallel\n", " r_discovery_cv, r_discovery_overfit, r_replication, p_discovery_cv, p_discovery_overfit, p_replication = zip(\n", " *Parallel(n_jobs=-1)(\n", " delayed(bootstrap_workhorse)(X, y, sample_size, models[model], seed) for seed in random_sates))\n", "\n", " tmp_data_frame = pd.DataFrame({\n", " 'connectivity' : feature_set,\n", " 'model' : model,\n", " 'target' : target_var,\n", " 'n' : sample_size,\n", " 'r_discovery_cv': r_discovery_cv,\n", " 'r_discovery_overfit': r_discovery_overfit,\n", " 'r_replication': r_replication,\n", " 'p_discovery_cv': p_discovery_cv,\n", " 'p_discovery_overfit': p_discovery_overfit,\n", " 'p_replication': p_replication\n", " })\n", " #sns.scatterplot(x='r_replication', y='r_discovery_cv', data=tmp_data_frame)\n", " #plt.ylabel('in-sample (r)')\n", " #plt.xlabel('out-of-sample (r_pred)')\n", " #plt.show()\n", " print(tmp_data_frame.r_discovery_cv.mean(), tmp_data_frame.r_replication.mean())\n", "\n", " for alpha in [0.05, 0.01, 0.005, 0.001]:\n", " print('Replicability at alpha =', alpha, ':',\n", " (tmp_data_frame.loc[tmp_data_frame['p_discovery_cv']:60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.008057963444189749 0.0007617908898002353\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 125\n", "-0.0055081698082560734 0.005587961002894532\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 150\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "-0.007513323395307536 0.007795039076033006\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 175\n", "-0.024426284422064924 0.0012732099168818367\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 200\n", "-0.009799381612001118 -0.0007998371652269578\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 225\n", "-0.006800965416406829 0.0036268724292571873\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 250\n", "-0.005643844886017258 0.005905116934532512\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 275\n", "-0.0077719657112639015 0.005089051475198677\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 300\n", "-5.560961358049732e-05 0.004031677602922873\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 325\n", "0.009858336998598637 0.0012551202601687702\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 350\n", "0.008354480479978098 0.002851850940927076\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 375\n", "-0.0009925765445703652 0.002565885599065646\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 400\n", "-0.0063230559073241275 0.009341029456317186\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 425\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "-0.0026882419873370074 0.002663124216213178\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 450\n", "-0.0033108931357782594 0.0017600493742229478\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj 475\n", "-0.006009179108780085 0.0003451398423298031\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CogTotalComp_AgeAdj max\n", "-0.00432381390725913 0.0009409815772123078\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 25\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[:, None]\n", "/home/tspisak/src/RPN-signature/venv/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n", " c /= stddev[None, :]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "-0.0020690067725892386 0.01791656637314636\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 50\n", "0.008028813068835057 -0.007791174824071893\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 75\n", "0.0030038526863170965 -0.024410976025185813\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 100\n", "0.017448432133557602 -0.01142796342196336\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 125\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "-0.011745913705172827 0.0015701689428313892\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 150\n", "0.004682130921237872 0.008077423750979749\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 175\n", "0.008377111073047287 0.0007396947990444197\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 200\n", "0.00762089856315215 0.007601195768630712\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 225\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.004539626282972586 0.011146561293955184\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 250\n", "0.004534000426500338 0.0069876445099758475\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 275\n", "0.015687029961596998 0.0045546164690225864\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 300\n", "0.013826396717998823 0.00028625928905271636\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 325\n", "0.015537331992997666 0.002657362747722801\n", "Replicability at alpha = 0.05 : 10.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 350\n", "0.014122297559363537 -0.00123750809209937\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 375\n", "0.004394764699869901 0.00018156931476631826\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 400\n", "-0.0003213337712836753 -0.0005133020306968125\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 425\n", "0.0012179850109901964 -0.0007894193213929146\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 450\n", "0.0032211375592086666 -0.0024186464334604316\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR 475\n", "0.0023798146533590835 0.004755574941689076\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PMAT24_A_CR max\n", "0.0012442268418254698 0.0006788357811788294\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 25\n", "-0.010368062081003675 0.008046507525408711\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 50\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "-0.006951145029068247 0.01310051805653064\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 75\n", "-0.02211813055997961 -0.0015327321930223173\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 100\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.0026547027339089086 -0.011287055848290308\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 125\n", "-0.004085185456582049 -0.005436574090510735\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 150\n", "0.006351054316329651 -0.0016043281951136107\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 175\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "-0.0013170165851853078 0.007054488190669121\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 200\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.0032305958927370975 0.010932189862339538\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 225\n", "0.0038702843026217053 0.00670751467688789\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 250\n", "-0.0017338995664610177 0.005023723215510912\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 275\n", "-0.009579410471613963 0.004009035978402434\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 300\n", "-0.009364873140989646 0.004417393188958138\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 325\n", "-0.012730551302338316 0.005035249963531995\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 350\n", "-0.005674608954688538 0.0020981463684660944\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 375\n", "-0.003085373611425863 -0.0026181464434469736\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 400\n", "-0.009200141103222157 -0.0011760124196208207\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 425\n", "-0.007516357185878566 0.001823702086658912\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 450\n", "0.001775273635506997 0.00091871517489634\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj 475\n", "0.0016362261472975654 0.003645912001952621\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR Flanker_AgeAdj max\n", "-0.008707439356889543 0.00429709444279815\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 25\n", "-0.007471260589129269 -0.010523105186374403\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 50\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "-0.0066247411916624775 0.005565188055752858\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 75\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.024732081332919202 -0.003098713166931714\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 100\n", "-0.0066700364064274445 -0.0012550854807044523\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 125\n", "0.0008748149725552831 -0.012503410705501568\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 150\n", "0.005235158565729553 -0.018725812586150054\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 175\n", "-0.019576041430061128 -0.0030081661077301074\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 200\n", "-0.007011765164321644 -0.001359026986178459\n", "Replicability at alpha = 0.05 : 33.33333333333333 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 225\n", "0.006687170659119923 -0.008361755894203625\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 250\n", "0.007348686796235472 -0.004097072465235302\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 275\n", "0.017662855497993714 -0.0017398534040184322\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 300\n", "0.014291142572627707 -0.00043760039646580204\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 325\n", "0.0072005012818622885 -0.0036804590726560254\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 350\n", "0.008410705760970236 -0.00142359911083541\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 375\n", "0.0015822124814592726 4.872542063388736e-05\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 400\n", "0.004983089733263311 -0.0005793603050996731\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 425\n", "0.006205343159989503 -0.0013925897505949916\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 450\n", "-0.0006616110163891906 0.0018772721336790116\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj 475\n", "-0.002363247042333287 0.0025257077980559146\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR CardSort_AgeAdj max\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "7.973534426850258e-05 0.0009197645946550262\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 25\n", "0.0035313618245588975 -0.018715779861556075\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 50\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.017903733881327325 0.01480521119830485\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 75\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.0022159051889437216 0.0007679572783647408\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 100\n", "-0.006270139331591028 0.001866435500016127\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 125\n", "-0.010451597387826961 0.01194470589032671\n", "Replicability at alpha = 0.05 : nan %\n", "Replicability at alpha = 0.01 : nan %\n", "Replicability at alpha = 0.005 : nan %\n", "Replicability at alpha = 0.001 : nan %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 150\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ ":60: RuntimeWarning: invalid value encountered in long_scalars\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "-0.007515165387502061 0.007867010232362288\n", "Replicability at alpha = 0.05 : 100.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 175\n", "0.005441196332331857 0.0017350151574608147\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 200\n", "-0.005344849015655386 0.0013732274285362875\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 225\n", "-0.0005694410745183176 -0.0024720961888230718\n", "Replicability at alpha = 0.05 : 20.0 %\n", "Replicability at alpha = 0.01 : 20.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 250\n", "0.008677089454720035 -0.0022289683420963586\n", "Replicability at alpha = 0.05 : 25.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 275\n", "0.015069852332234686 -0.006518455917347051\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 300\n", "0.010393117348453423 0.0005262481184145984\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 325\n", "0.003260407561043856 -0.0016700013638583978\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 350\n", "0.0030032189168597147 -0.0007761166008547188\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 375\n", "0.0030616232094884377 0.0022756316401759086\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 400\n", "0.0015794251181664178 -0.0001193252110228156\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 425\n", "-9.28339141746343e-05 -0.0015291014008270657\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 450\n", "0.0011448443762844137 0.0025144700972381463\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj 475\n", "-0.002755252616117111 0.0007572032310020428\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "*****************************************************************\n", "netmats_pearson PCA_SVR PicSeq_AgeAdj max\n", "-0.00039147984484469634 0.0016130406528689545\n", "Replicability at alpha = 0.05 : 0.0 %\n", "Replicability at alpha = 0.01 : 0.0 %\n", "Replicability at alpha = 0.005 : 0.0 %\n", "Replicability at alpha = 0.001 : 0.0 %\n", "CPU times: user 53.7 s, sys: 10.3 s, total: 1min 3s\n", "Wall time: 7h 37min 58s\n" ] }, { "data": { "text/plain": " connectivity model target n r_discovery_cv \\\n0 netmats_pearson PCA_SVR age 25 NaN \n1 netmats_pearson PCA_SVR age 25 NaN \n2 netmats_pearson PCA_SVR age 25 NaN \n3 netmats_pearson PCA_SVR age 25 NaN \n4 netmats_pearson PCA_SVR age 25 NaN \n... ... ... ... ... ... \n11995 netmats_pearson PCA_SVR PicSeq_AgeAdj 501 0.017149 \n11996 netmats_pearson PCA_SVR PicSeq_AgeAdj 501 -0.090964 \n11997 netmats_pearson PCA_SVR PicSeq_AgeAdj 501 0.073699 \n11998 netmats_pearson PCA_SVR PicSeq_AgeAdj 501 0.039265 \n11999 netmats_pearson PCA_SVR PicSeq_AgeAdj 501 -0.068938 \n\n r_discovery_overfit r_replication p_discovery_cv p_discovery_overfit \\\n0 0.640453 0.249772 0.904096 0.000999 \n1 0.674393 -0.05573 0.959041 0.000999 \n2 0.656076 -0.149824 0.908092 0.000999 \n3 0.691142 0.022858 0.40959 0.000999 \n4 0.723121 -0.185205 1.0 0.000999 \n... ... ... ... ... \n11995 0.334397 -0.004167 0.796204 0.000999 \n11996 0.325431 -0.060089 0.99001 0.000999 \n11997 0.307215 -0.083865 0.23976 0.000999 \n11998 0.311807 -0.081005 0.305694 0.000999 \n11999 0.278715 0.065742 0.987013 0.000999 \n\n p_replication \n0 0.110889 \n1 0.606394 \n2 0.724276 \n3 0.458541 \n4 0.833167 \n... ... \n11995 0.542458 \n11996 0.899101 \n11997 0.973027 \n11998 0.962038 \n11999 0.08991 \n\n[12000 rows x 10 columns]", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
connectivitymodeltargetnr_discovery_cvr_discovery_overfitr_replicationp_discovery_cvp_discovery_overfitp_replication
0netmats_pearsonPCA_SVRage25NaN0.6404530.2497720.9040960.0009990.110889
1netmats_pearsonPCA_SVRage25NaN0.674393-0.055730.9590410.0009990.606394
2netmats_pearsonPCA_SVRage25NaN0.656076-0.1498240.9080920.0009990.724276
3netmats_pearsonPCA_SVRage25NaN0.6911420.0228580.409590.0009990.458541
4netmats_pearsonPCA_SVRage25NaN0.723121-0.1852051.00.0009990.833167
.................................
11995netmats_pearsonPCA_SVRPicSeq_AgeAdj5010.0171490.334397-0.0041670.7962040.0009990.542458
11996netmats_pearsonPCA_SVRPicSeq_AgeAdj501-0.0909640.325431-0.0600890.990010.0009990.899101
11997netmats_pearsonPCA_SVRPicSeq_AgeAdj5010.0736990.307215-0.0838650.239760.0009990.973027
11998netmats_pearsonPCA_SVRPicSeq_AgeAdj5010.0392650.311807-0.0810050.3056940.0009990.962038
11999netmats_pearsonPCA_SVRPicSeq_AgeAdj501-0.0689380.2787150.0657420.9870130.0009990.08991
\n

12000 rows × 10 columns

\n
" }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time\n", "\n", "random_state = 42\n", "n_bootstrap = 100\n", "\n", "features = {\n", " 'netmats_pearson': netmats_pearson\n", "}\n", "\n", "models = {\n", " 'PCA_SVR': Pipeline([('pca', PCA(n_components=0.5)),\n", " ('svr', SVR())])\n", "\n", "}\n", "\n", "# We aggregate all results here:\n", "df = pd.DataFrame(columns=['connectivity','model','target','n','r_discovery_cv','r_discovery_overfit','r_replication','p_discovery_cv','p_discovery_overfit','p_replication'])\n", "\n", "for feature_set in features:\n", " for model in models:\n", " for target_var in ['age', 'CogTotalComp_AgeAdj', 'PMAT24_A_CR', 'Flanker_AgeAdj', 'CardSort_AgeAdj', 'PicSeq_AgeAdj']:\n", " for sample_size in [25, 50, 75, 100, 125, 150, 175, 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475, 'max']:\n", "\n", " print('*****************************************************************')\n", " print(feature_set, model, target_var, sample_size)\n", "\n", " X, y = create_data(target=target_var, feature_data=features[feature_set]) # gives a random y when target is None\n", "\n", " if sample_size=='max':\n", " sample_size = int(len(y)/2)\n", "\n", " # create random seeds for each bootstrap iteration for reproducibility\n", " rng = np.random.default_rng(random_state)\n", " random_sates = rng.integers(np.iinfo(np.int32).max, size=n_bootstrap)\n", "\n", " # run bootstrap iterations in parallel, with shuffle_y=True\n", " r_discovery_cv, r_discovery_overfit, r_replication, p_discovery_cv, p_discovery_overfit, p_replication = zip(\n", " *Parallel(n_jobs=-1)(\n", " delayed(bootstrap_workhorse)(X, y, sample_size, models[model], seed, shuffle_y=True) for seed in random_sates))\n", "\n", " tmp_data_frame = pd.DataFrame({\n", " 'connectivity' : feature_set,\n", " 'model' : model,\n", " 'target' : target_var,\n", " 'n' : sample_size,\n", " 'r_discovery_cv': r_discovery_cv,\n", " 'r_discovery_overfit': r_discovery_overfit,\n", " 'r_replication': r_replication,\n", " 'p_discovery_cv': p_discovery_cv,\n", " 'p_discovery_overfit': p_discovery_overfit,\n", " 'p_replication': p_replication\n", " })\n", "\n", " #sns.scatterplot(x='r_replication', y='r_discovery_cv', data=tmp_data_frame)\n", " #plt.ylabel('in-sample (r)')\n", " #plt.xlabel('out-of-sample (r_pred)')\n", " #plt.show()\n", " print(tmp_data_frame.r_discovery_cv.mean(), tmp_data_frame.r_replication.mean())\n", "\n", " for alpha in [0.05, 0.01, 0.005, 0.001]:\n", " print('Replicability at alpha =', alpha, ':',\n", " (tmp_data_frame.loc[tmp_data_frame['p_discovery_cv']\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
connectivitymodeltargetnr_discovery_cvr_discovery_overfitr_replicationp_discovery_cvp_discovery_overfitp_replication
0netmats_parcorRidgeage25NaN1.0-0.1073220.9660340.0009990.716284
1netmats_parcorRidgeage25NaN1.00.2186410.9970030.0009990.144855
2netmats_parcorRidgeage25NaN1.00.1355090.5054950.0009990.273726
3netmats_parcorRidgeage25NaN1.0-0.1860920.9880120.0009990.812188
4netmats_parcorRidgeage25NaN1.00.2414570.8661340.0009990.130869
.................................
11995netmats_parcorRidgePicSeq_AgeAdj5010.0014561.0-0.0698810.5124880.0009990.941059
11996netmats_parcorRidgePicSeq_AgeAdj501-0.0735831.0-0.0044220.9430570.0009990.526474
11997netmats_parcorRidgePicSeq_AgeAdj5010.045751.0-0.0402730.0879120.0009990.824176
11998netmats_parcorRidgePicSeq_AgeAdj5010.0373591.00.0302450.2677320.0009990.256743
11999netmats_parcorRidgePicSeq_AgeAdj5010.1809131.0-0.0073980.0009990.0009990.551449
\n

12000 rows × 10 columns

\n" }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time\n", "\n", "random_state = 42\n", "n_bootstrap = 100\n", "\n", "features = {\n", " 'netmats_parcor': netmats_parcor\n", "}\n", "\n", "models = {\n", " 'Ridge': Ridge()\n", "\n", "}\n", "\n", "# We aggregate all results here:\n", "df = pd.DataFrame(columns=['connectivity','model','target','n','r_discovery_cv','r_discovery_overfit','r_replication','p_discovery_cv','p_discovery_overfit','p_replication'])\n", "\n", "for feature_set in features:\n", " for model in models:\n", " for target_var in ['age', 'CogTotalComp_AgeAdj', 'PMAT24_A_CR', 'Flanker_AgeAdj', 'CardSort_AgeAdj', 'PicSeq_AgeAdj']:\n", " for sample_size in [25, 50, 75, 100, 125, 150, 175, 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475, 'max']:\n", "\n", " print('*****************************************************************')\n", " print(feature_set, model, target_var, sample_size)\n", "\n", " X, y = create_data(target=target_var, feature_data=features[feature_set]) # gives a random y when target is None\n", "\n", " if sample_size=='max':\n", " sample_size = int(len(y)/2)\n", "\n", " # create random seeds for each bootstrap iteration for reproducibility\n", " rng = np.random.default_rng(random_state)\n", " random_sates = rng.integers(np.iinfo(np.int32).max, size=n_bootstrap)\n", "\n", " # run bootstrap iterations in parallel, with shuffle_y=True\n", " r_discovery_cv, r_discovery_overfit, r_replication, p_discovery_cv, p_discovery_overfit, p_replication = zip(\n", " *Parallel(n_jobs=-1)(\n", " delayed(bootstrap_workhorse)(X, y, sample_size, models[model], seed, shuffle_y=True) for seed in random_sates))\n", "\n", " tmp_data_frame = pd.DataFrame({\n", " 'connectivity' : feature_set,\n", " 'model' : model,\n", " 'target' : target_var,\n", " 'n' : sample_size,\n", " 'r_discovery_cv': r_discovery_cv,\n", " 'r_discovery_overfit': r_discovery_overfit,\n", " 'r_replication': r_replication,\n", " 'p_discovery_cv': p_discovery_cv,\n", " 'p_discovery_overfit': p_discovery_overfit,\n", " 'p_replication': p_replication\n", " })\n", "\n", " #sns.scatterplot(x='r_replication', y='r_discovery_cv', data=tmp_data_frame)\n", " #plt.ylabel('in-sample (r)')\n", " #plt.xlabel('out-of-sample (r_pred)')\n", " #plt.show()\n", " print(tmp_data_frame.r_discovery_cv.mean(), tmp_data_frame.r_replication.mean())\n", "\n", " for alpha in [0.05, 0.01, 0.005, 0.001]:\n", " print('Replicability at alpha =', alpha, ':',\n", " (tmp_data_frame.loc[tmp_data_frame['p_discovery_cv']