{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "
\n", " \n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Evaluate methods \n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [ { "data": { "application/javascript": [ "\n", "require(['notebook'],\n", " function() {\n", " IPython.CodeCell.config_defaults.highlight_modes['magic_text/x-c++src'] = {'reg':[/^%%cpp/]};\n", " console.log(\"JupyROOT - %%cpp magic configured\");\n", " }\n", ");\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Welcome to JupyROOT 6.07/07\n" ] } ], "source": [ "import ROOT\n", "from ROOT import TFile, TMVA, TCut" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Enable JS visualization" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%jsmva on" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## Declarations, building training and testing trees " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For more details please see this notebook." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
DataSetInfo
Dataset: tmva_class_exampleAdded class \"Signal\"
Add Tree TreeS of type Signal with 6000 events
DataSetInfo
Dataset: tmva_class_exampleAdded class \"Background\"
Add Tree TreeB of type Background with 6000 events
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "SysError in : could not delete TMVA.root (errno: 26) (Text file busy)\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "outputFile = TFile( \"TMVA.root\", 'RECREATE' )\n", "\n", "TMVA.Tools.Instance()\n", "\n", "factory = TMVA.Factory(JobName=\"TMVAClassification\", TargetFile=outputFile,\n", " V=False, Color=True, DrawProgressBar=True, Transformations=[\"I\", \"D\", \"P\", \"G\",\"D\"],\n", " AnalysisType=\"Classification\")\n", "\n", "dataset = \"tmva_class_example\"\n", "loader = TMVA.DataLoader(dataset)\n", "\n", "loader.AddVariable( \"myvar1 := var1+var2\", 'F' )\n", "loader.AddVariable( \"myvar2 := var1-var2\", \"Expression 2\", 'F' )\n", "loader.AddVariable( \"var3\", \"Variable 3\", 'F' )\n", "loader.AddVariable( \"var4\", \"Variable 4\", 'F' )\n", "\n", "loader.AddSpectator( \"spec1:=var1*2\", \"Spectator 1\", 'F' )\n", "loader.AddSpectator( \"spec2:=var1*3\", \"Spectator 2\", 'F' )\n", "\n", "if ROOT.gSystem.AccessPathName( \"./tmva_class_example.root\" ) != 0: \n", " ROOT.gSystem.Exec( \"wget https://root.cern.ch/files/tmva_class_example.root\")\n", " \n", "input = TFile.Open( \"./tmva_class_example.root\" )\n", "\n", "# Get the signal and background trees for training\n", "signal = input.Get( \"TreeS\" )\n", "background = input.Get( \"TreeB\" )\n", " \n", "# Global event weights (see below for setting event-wise weights)\n", "signalWeight = 1.0\n", "backgroundWeight = 1.0\n", "\n", "mycuts = TCut(\"\")\n", "mycutb = TCut(\"\")\n", "\n", "loader.AddSignalTree(signal, signalWeight)\n", "loader.AddBackgroundTree(background, backgroundWeight)\n", "loader.fSignalWeight = signalWeight\n", "loader.fBackgroundWeight = backgroundWeight\n", "loader.fTreeS = signal\n", "loader.fTreeB = background\n", "\n", "loader.PrepareTrainingAndTestTree(SigCut=mycuts, BkgCut=mycutb,\n", " nTrain_Signal=0, nTrain_Background=0, SplitMode=\"Random\", NormMode=\"NumEvents\", V=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Booking methods" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For more details please see this notebook." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false, "scrolled": false }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
Factory Booking method: SVM
SVM
Dataset: tmva_class_exampleCreate Transformation \"Norm\" with events from all classes.
Norm Transformation, Variable selection :
Input : variable 'myvar1' <---> Output : variable 'myvar1'
Input : variable 'myvar2' <---> Output : variable 'myvar2'
Input : variable 'var3' <---> Output : variable 'var3'
Input : variable 'var4' <---> Output : variable 'var4'
Factory Booking method: MLP
MLP
Dataset: tmva_class_exampleCreate Transformation \"N\" with events from all classes.
Norm Transformation, Variable selection :
Input : variable 'myvar1' <---> Output : variable 'myvar1'
Input : variable 'myvar2' <---> Output : variable 'myvar2'
Input : variable 'var3' <---> Output : variable 'var3'
Input : variable 'var4' <---> Output : variable 'var4'
MLP Building Network.
Initializing weights
Factory Booking method: LD
DataSetFactory
Dataset: tmva_class_exampleNumber of events in input trees
Number of training and testing events
Signaltraining events3000
testing events3000
training and testing events6000
Backgroundtraining events3000
testing events3000
training and testing events6000
DataSetInfo Correlation matrix (Signal)
DataSetInfo Correlation matrix (Background)
DataSetFactory
Dataset: tmva_class_example
Factory Booking method: DNN
DNN
Dataset: tmva_class_exampleCreate Transformation \"Normalize\" with events from all classes.
Norm Transformation, Variable selection :
Input : variable 'myvar1' <---> Output : variable 'myvar1'
Input : variable 'myvar2' <---> Output : variable 'myvar2'
Input : variable 'var3' <---> Output : variable 'var3'
Input : variable 'var4' <---> Output : variable 'var4'
Factory Booking method: Likelihood
Factory Booking method: BDT
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "factory.BookMethod( DataLoader=loader, Method=TMVA.Types.kSVM, MethodTitle=\"SVM\", \n", " Gamma=0.25, Tol=0.001, VarTransform=\"Norm\" )\n", "\n", "factory.BookMethod( DataLoader=loader, Method=TMVA.Types.kMLP, MethodTitle=\"MLP\", \n", " H=False, V=False, NeuronType=\"tanh\", VarTransform=\"N\", NCycles=600, HiddenLayers=\"N+5\",\n", " TestRate=5, UseRegulator=False )\n", "\n", "factory.BookMethod( DataLoader=loader, Method=TMVA.Types.kLD, MethodTitle=\"LD\", \n", " H=False, V=False, VarTransform=\"None\", CreateMVAPdfs=True, PDFInterpolMVAPdf=\"Spline2\",\n", " NbinsMVAPdf=50, NsmoothMVAPdf=10 )\n", "\n", "trainingStrategy = [{\n", " \"LearningRate\": 1e-1,\n", " \"Momentum\": 0.0,\n", " \"Repetitions\": 1,\n", " \"ConvergenceSteps\": 300,\n", " \"BatchSize\": 20,\n", " \"TestRepetitions\": 15,\n", " \"WeightDecay\": 0.001,\n", " \"Regularization\": \"NONE\",\n", " \"DropConfig\": \"0.0+0.5+0.5+0.5\",\n", " \"DropRepetitions\": 1,\n", " \"Multithreading\": True\n", " \n", " }, {\n", " \"LearningRate\": 1e-2,\n", " \"Momentum\": 0.5,\n", " \"Repetitions\": 1,\n", " \"ConvergenceSteps\": 300,\n", " \"BatchSize\": 30,\n", " \"TestRepetitions\": 7,\n", " \"WeightDecay\": 0.001,\n", " \"Regularization\": \"L2\",\n", " \"DropConfig\": \"0.0+0.1+0.1+0.1\",\n", " \"DropRepetitions\": 1,\n", " \"Multithreading\": True\n", " \n", " }, {\n", " \"LearningRate\": 1e-2,\n", " \"Momentum\": 0.3,\n", " \"Repetitions\": 1,\n", " \"ConvergenceSteps\": 300,\n", " \"BatchSize\": 40,\n", " \"TestRepetitions\": 7,\n", " \"WeightDecay\": 0.001,\n", " \"Regularization\": \"L2\",\n", " \"Multithreading\": True\n", " \n", " },{\n", " \"LearningRate\": 1e-3,\n", " \"Momentum\": 0.1,\n", " \"Repetitions\": 1,\n", " \"ConvergenceSteps\": 200,\n", " \"BatchSize\": 70,\n", " \"TestRepetitions\": 7,\n", " \"WeightDecay\": 0.001,\n", " \"Regularization\": \"NONE\",\n", " \"Multithreading\": True\n", " \n", "}]\n", "\n", "factory.BookMethod(DataLoader=loader, Method=TMVA.Types.kDNN, MethodTitle=\"DNN\", \n", " H = False, V=False, VarTransform=\"Normalize\", ErrorStrategy=\"CROSSENTROPY\",\n", " Layout=[\"TANH|100\", \"TANH|50\", \"TANH|10\", \"LINEAR\"],\n", " TrainingStrategy=trainingStrategy)\n", "\n", "factory.BookMethod(loader, TMVA.Types.kLikelihood, \"Likelihood\", \n", " \"NSmoothSig[0]=20:NSmoothBkg[0]=20:NSmoothBkg[1]=10:NSmooth=1:NAvEvtPerBin=50\",\n", " H=True, V=False,TransformOutput=True,PDFInterpol=\"Spline2\")\n", "\n", "factory.BookMethod(DataLoader= loader, Method=TMVA.Types.kBDT, MethodTitle=\"BDT\",\n", " H=False,V=False,NTrees=850,MinNodeSize=\"2.5%\",MaxDepth=3,BoostType=\"AdaBoost\", AdaBoostBeta=0.5,\n", " UseBaggedBoost=True,BaggedSampleFraction=0.5, SeparationType=\"GiniIndex\", nCuts=20 )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Train Methods" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false, "scrolled": false }, "outputs": [ { "data": { "text/html": [ "

Dataset: tmva_class_example

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "

Train method: SVM

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", "
\n", "
\n", "
0%
\n", "
\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "

Train method: MLP

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", "
\n", "
\n", "
0%
\n", "
\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "

Train method: LD

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "Training..." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "End" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "

Train method: DNN

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", "
\n", "
\n", "
0%
\n", "
\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "

Train method: Likelihood

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "Training..." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "End" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "

Train method: BDT

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", "
\n", "
\n", "
0%
\n", "
\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
TFHandler_SVM
VariableMeanRMSMinMax
myvar10.0839890.36407-1.00001.0000
myvar20.00947780.27696-1.00001.0000
var30.0802790.36720-1.00001.0000
var40.129860.39603-1.00001.0000
Building SVM Working Set...with 6000 event instances
Elapsed time for Working Set build: 1.24 sec
Sorry, no computing time forecast available for SVM, please wait ...
Elapsed time: 7.25 sec
Elapsed time for training with 6000 events: 8.52 sec
SVM
Dataset: tmva_class_exampleEvaluation of SVM on training sample (6000 events)
Elapsed time for evaluation of 6000 events: 1.04 sec
Creating xml weight file: tmva_class_example/weights/TMVAClassification_SVM.weights.xml
Creating standalone class: tmva_class_example/weights/TMVAClassification_SVM.class.C
TFHandler_MLP
VariableMeanRMSMinMax
myvar10.0839890.36407-1.00001.0000
myvar20.00947780.27696-1.00001.0000
var30.0802790.36720-1.00001.0000
var40.129860.39603-1.00001.0000
Training Network
Elapsed time for training with 6000 events: 4.01 sec
MLP
Dataset: tmva_class_exampleEvaluation of MLP on training sample (6000 events)
Elapsed time for evaluation of 6000 events: 0.012 sec
Creating xml weight file: tmva_class_example/weights/TMVAClassification_MLP.weights.xml
Creating standalone class: tmva_class_example/weights/TMVAClassification_MLP.class.C
Write special histos to file: TMVA.root:/tmva_class_example/Method_MLP/MLP
LD Results for LD coefficients:
Variable:Coefficient:
myvar1:-0.359
myvar2:-0.109
var3:-0.211
var4:+0.722
(offset):-0.054
Elapsed time for training with 6000 events: 0.00764 sec
LD
Dataset: tmva_class_exampleEvaluation of LD on training sample (6000 events)
Elapsed time for evaluation of 6000 events: 0.003 sec
Dataset: tmva_class_example Separation from histogram (PDF): 0.452 (0.000)
Evaluation of LD on training sample
Creating xml weight file: tmva_class_example/weights/TMVAClassification_LD.weights.xml
Creating standalone class: tmva_class_example/weights/TMVAClassification_LD.class.C
TFHandler_DNN
VariableMeanRMSMinMax
myvar10.0839890.36407-1.00001.0000
myvar20.00947780.27696-1.00001.0000
var30.0802790.36720-1.00001.0000
var40.129860.39603-1.00001.0000
TFHandler_DNN
VariableMeanRMSMinMax
myvar10.0839890.36407-1.00001.0000
myvar20.00947780.27696-1.00001.0000
var30.0802790.36720-1.00001.0000
var40.129860.39603-1.00001.0000
TFHandler_DNN
VariableMeanRMSMinMax
myvar10.0751130.36776-1.10741.0251
myvar20.00755950.27349-0.906631.0008
var30.0702280.37106-1.06491.0602
var40.120900.39854-1.18711.0199
Add Layer with 100 nodes.
Add Layer with 50 nodes.
Add Layer with 10 nodes.
Add Layer with 1 nodes.
Total number of Synapses = 6010
Training with learning rate = 0.1, momentum = 0, repetitions = 1
Drop configuration
drop repetitions = 1
Layer 0 = 0
Layer 1 = 0.5
Layer 2 = 0.5
Layer 3 = 0.5
learning rate reduced to 0.05
Training with learning rate = 0.01, momentum = 0.5, repetitions = 1
Drop configuration
drop repetitions = 1
Layer 0 = 0
Layer 1 = 0.1
Layer 2 = 0.1
Layer 3 = 0.1
Training with learning rate = 0.01, momentum = 0.3, repetitions = 1
Training with learning rate = 0.001, momentum = 0.1, repetitions = 1
Elapsed time for training with 6000 events: 18.2 sec
DNN
Dataset: tmva_class_exampleEvaluation of DNN on training sample (6000 events)
Elapsed time for evaluation of 6000 events: 0.223 sec
Creating xml weight file: tmva_class_example/weights/TMVAClassification_DNN.weights.xml
Creating standalone class: tmva_class_example/weights/TMVAClassification_DNN.class.C
Write monitoring histograms to file: TMVA.root:/tmva_class_example/Method_DNN/DNN
================================================================
Dataset: Likelihood
--- Short description:
The maximum-likelihood classifier models the data with probability
density functions (PDF) reproducing the signal and background
distributions of the input variables. Correlations among the
variables are ignored.
--- Performance optimisation:
Required for good performance are decorrelated input variables
(PCA transformation via the option \"VarTransform=Decorrelate\"
may be tried). Irreducible non-linear correlations may be reduced
by precombining strongly correlated input variables, or by simply
removing one of the variables.
--- Performance tuning via configuration options:
High fidelity PDF estimates are mandatory, i.e., sufficient training
statistics is required to populate the tails of the distributions
It would be a surprise if the default Spline or KDE kernel parameters
provide a satisfying fit to the data. The user is advised to properly
tune the events per bin and smooth options in the spline cases
individually per variable. If the KDE kernel is used, the adaptive
Gaussian kernel may lead to artefacts, so please always also try
the non-adaptive one.
All tuning parameters must be adjusted individually for each input
variable!
================================================================
Filling reference histograms
Building PDF out of reference histograms
Elapsed time for training with 6000 events: 0.0384 sec
Likelihood
Dataset: tmva_class_exampleEvaluation of Likelihood on training sample (6000 events)
Elapsed time for evaluation of 6000 events: 0.00767 sec
Creating xml weight file: tmva_class_example/weights/TMVAClassification_Likelihood.weights.xml
Creating standalone class: tmva_class_example/weights/TMVAClassification_Likelihood.class.C
Write monitoring histograms to file: TMVA.root:/tmva_class_example/Method_Likelihood/Likelihood
BDT #events: (reweighted) sig: 3000 bkg: 3000
#events: (unweighted) sig: 3000 bkg: 3000
Training 850 Decision Trees ... patience please
Elapsed time for training with 6000 events: 2.13 sec
BDT
Dataset: tmva_class_exampleEvaluation of BDT on training sample (6000 events)
Elapsed time for evaluation of 6000 events: 0.824 sec
Creating xml weight file: tmva_class_example/weights/TMVAClassification_BDT.weights.xml
Creating standalone class: tmva_class_example/weights/TMVAClassification_BDT.class.C
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "factory.TrainAllMethods()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Test end evaluate the methods" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To test test the methods and evaluate the performance we need to run Factory.TestAllMethods and Factory.EvaluateAllMethods functions." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false, "scrolled": false }, "outputs": [ { "data": { "text/html": [ "
Factory Test method: SVM for Classification performance
SVM
Dataset: tmva_class_exampleEvaluation of SVM on testing sample (6000 events)
Elapsed time for evaluation of 6000 events: 0.994 sec
Factory Test method: MLP for Classification performance
MLP
Dataset: tmva_class_exampleEvaluation of MLP on testing sample (6000 events)
Elapsed time for evaluation of 6000 events: 0.00914 sec
Factory Test method: LD for Classification performance
LD
Dataset: tmva_class_exampleEvaluation of LD on testing sample (6000 events)
Elapsed time for evaluation of 6000 events: 0.00178 sec
Dataset: tmva_class_exampleEvaluation of LD on testing sample
Factory Test method: DNN for Classification performance
DNN
Dataset: tmva_class_exampleEvaluation of DNN on testing sample (6000 events)
Elapsed time for evaluation of 6000 events: 0.205 sec
Factory Test method: Likelihood for Classification performance
Likelihood
Dataset: tmva_class_exampleEvaluation of Likelihood on testing sample (6000 events)
Elapsed time for evaluation of 6000 events: 0.0074 sec
Factory Test method: BDT for Classification performance
BDT
Dataset: tmva_class_exampleEvaluation of BDT on testing sample (6000 events)
Elapsed time for evaluation of 6000 events: 0.758 sec
Factory Evaluate classifier: SVM
TFHandler_SVM
VariableMeanRMSMinMax
myvar10.0751130.36776-1.10741.0251
myvar20.00755950.27349-0.906631.0008
var30.0702280.37106-1.06491.0602
var40.120900.39854-1.18711.0199
SVM
Dataset: tmva_class_exampleLoop over test events and fill histograms with classifier response...
TFHandler_SVM
VariableMeanRMSMinMax
myvar10.0751130.36776-1.10741.0251
myvar20.00755950.27349-0.906631.0008
var30.0702280.37106-1.06491.0602
var40.120900.39854-1.18711.0199
Factory Evaluate classifier: MLP
TFHandler_MLP
VariableMeanRMSMinMax
myvar10.0751130.36776-1.10741.0251
myvar20.00755950.27349-0.906631.0008
var30.0702280.37106-1.06491.0602
var40.120900.39854-1.18711.0199
MLP
Dataset: tmva_class_exampleLoop over test events and fill histograms with classifier response...
TFHandler_MLP
VariableMeanRMSMinMax
myvar10.0751130.36776-1.10741.0251
myvar20.00755950.27349-0.906631.0008
var30.0702280.37106-1.06491.0602
var40.120900.39854-1.18711.0199
Factory Evaluate classifier: LD
LD
Dataset: tmva_class_exampleLoop over test events and fill histograms with classifier response...
Also filling probability and rarity histograms (on request)...
TFHandler_LD
VariableMeanRMSMinMax
myvar1-0.0108143.0633-9.86057.9024
myvar20.000905521.1092-3.70674.0291
var3-0.0151181.7459-5.35634.6430
var40.143312.1667-6.96755.0307
Factory Evaluate classifier: DNN
DNN
Dataset: tmva_class_exampleLoop over test events and fill histograms with classifier response...
TFHandler_DNN
VariableMeanRMSMinMax
myvar10.0751130.36776-1.10741.0251
myvar20.00755950.27349-0.906631.0008
var30.0702280.37106-1.06491.0602
var40.120900.39854-1.18711.0199
Factory Evaluate classifier: Likelihood
Likelihood
Dataset: tmva_class_exampleLoop over test events and fill histograms with classifier response...
TFHandler_Likelihood
VariableMeanRMSMinMax
myvar1-0.0108143.0633-9.86057.9024
myvar20.000905521.1092-3.70674.0291
var3-0.0151181.7459-5.35634.6430
var40.143312.1667-6.96755.0307
Factory Evaluate classifier: BDT
BDT
Dataset: tmva_class_exampleLoop over test events and fill histograms with classifier response...
TFHandler_BDT
VariableMeanRMSMinMax
myvar1-0.0108143.0633-9.86057.9024
myvar20.000905521.1092-3.70674.0291
var3-0.0151181.7459-5.35634.6430
var40.143312.1667-6.96755.0307
Evaluation results ranked by best signal efficiency and purity (area)
DataSet MVA
Name: Method: ROC-integ
tmva_class_example MLP : 0.939
tmva_class_example DNN : 0.937
tmva_class_example SVM : 0.937
tmva_class_example BDT : 0.931
tmva_class_example LD : 0.895
tmva_class_example Likelihood : 0.827
Testing efficiency compared to training efficiency (overtraining check)
DataSet MVA Signal efficiency: from test sample (from training sample)
Name: Method: @B=0.01 @B=0.10 @B=0.30
tmva_class_example MLP : 0.395 (0.345) 0.805 (0.795) 0.964 (0.966)
tmva_class_example DNN : 0.362 (0.305) 0.801 (0.786) 0.962 (0.962)
tmva_class_example SVM : 0.397 (0.323) 0.801 (0.791) 0.960 (0.960)
tmva_class_example BDT : 0.350 (0.380) 0.778 (0.805) 0.955 (0.959)
tmva_class_example LD : 0.261 (0.242) 0.679 (0.662) 0.901 (0.903)
tmva_class_example Likelihood : 0.106 (0.101) 0.400 (0.371) 0.812 (0.813)
Dataset:tmva_class_exa...: Created tree 'TestTree' with 6000 events
Dataset:tmva_class_exa...: Created tree 'TrainTree' with 6000 events
Factory Thank you for using TMVA!
For citation information, please visit: http://tmva.sf.net/citeTMVA.html
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "factory.TestAllMethods()\n", "factory.EvaluateAllMethods()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Classifier Output Distributions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To draw the classifier output distribution we have to use Factory.DrawOutputDistribution function which is inserted by invoking jsmva magic. The parameters of the function are the following:\n", "The options string can contain the following options:\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", "
KeywordCan be used as positional argumentDefaultPredefined valuesDescription
datasetNameyes, 1.-- The name of dataset
methodNameyes, 2.-- The name of method
" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "factory.DrawOutputDistribution(dataset, \"MLP\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Classifier Probability Distributions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To draw the classifier probability distribution we have to use Factory.DrawProbabilityDistribution function which is inserted by invoking jsmva magic. The parameters of the function are the following:\n", "The options string can contain the following options:\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", "
KeywordCan be used as positional argumentDefaultPredefined valuesDescription
datasetNameyes, 1.-- The name of dataset
" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "factory.DrawProbabilityDistribution(dataset, \"LD\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ROC curve" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To draw the ROC (receiver operating characteristic) curve we have to use Factory.DrawROCCurve function which is inserted by invoking jsmva magic. The parameters of the function are the following:\n", "The options string can contain the following options:\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", "
KeywordCan be used as positional argumentDefaultPredefined valuesDescription
datasetNameyes, 1.-- The name of dataset
" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false, "scrolled": false }, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "Warning in : Deleting canvas with same name: ROCCurve tmva_class_example\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "factory.DrawROCCurve(dataset)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Classifier Cut Efficiencies" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To draw the classifier cut efficiencies we have to use Factory.DrawCutEfficiencies function which is inserted by invoking jsmva magic. The parameters of the function are the following:\n", "The options string can contain the following options:\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", "
KeywordCan be used as positional argumentDefaultPredefined valuesDescription
datasetNameyes, 1.-- The name of dataset
methodNameyes, 2.-- The name of method
" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "Warning in : Replacing existing TH1: purS_MLP (Potential memory leak).\n", "Warning in : Replacing existing TH1: effpurS_MLP (Potential memory leak).\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "factory.DrawCutEfficiencies(dataset, \"MLP\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Close the factory's output file" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "outputFile.Close()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.11" } }, "nbformat": 4, "nbformat_minor": 0 }