{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Visualizing the stock market structure\n\nThis example employs several unsupervised learning techniques to extract\nthe stock market structure from variations in historical quotes.\n\nThe quantity that we use is the daily variation in quote price: quotes\nthat are linked tend to fluctuate in relation to each other during a day.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Authors: The scikit-learn developers\n# SPDX-License-Identifier: BSD-3-Clause"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Retrieve the data from Internet\n\nThe data is from 2003 - 2008. This is reasonably calm: (not too long ago so\nthat we get high-tech firms, and before the 2008 crash). This kind of\nhistorical data can be obtained from APIs like the\n[data.nasdaq.com](https://data.nasdaq.com/) and\n[alphavantage.co](https://www.alphavantage.co/).\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import sys\n\nimport numpy as np\nimport pandas as pd\n\nsymbol_dict = {\n    \"TOT\": \"Total\",\n    \"XOM\": \"Exxon\",\n    \"CVX\": \"Chevron\",\n    \"COP\": \"ConocoPhillips\",\n    \"VLO\": \"Valero Energy\",\n    \"MSFT\": \"Microsoft\",\n    \"IBM\": \"IBM\",\n    \"TWX\": \"Time Warner\",\n    \"CMCSA\": \"Comcast\",\n    \"CVC\": \"Cablevision\",\n    \"YHOO\": \"Yahoo\",\n    \"DELL\": \"Dell\",\n    \"HPQ\": \"HP\",\n    \"AMZN\": \"Amazon\",\n    \"TM\": \"Toyota\",\n    \"CAJ\": \"Canon\",\n    \"SNE\": \"Sony\",\n    \"F\": \"Ford\",\n    \"HMC\": \"Honda\",\n    \"NAV\": \"Navistar\",\n    \"NOC\": \"Northrop Grumman\",\n    \"BA\": \"Boeing\",\n    \"KO\": \"Coca Cola\",\n    \"MMM\": \"3M\",\n    \"MCD\": \"McDonald's\",\n    \"PEP\": \"Pepsi\",\n    \"K\": \"Kellogg\",\n    \"UN\": \"Unilever\",\n    \"MAR\": \"Marriott\",\n    \"PG\": \"Procter Gamble\",\n    \"CL\": \"Colgate-Palmolive\",\n    \"GE\": \"General Electrics\",\n    \"WFC\": \"Wells Fargo\",\n    \"JPM\": \"JPMorgan Chase\",\n    \"AIG\": \"AIG\",\n    \"AXP\": \"American express\",\n    \"BAC\": \"Bank of America\",\n    \"GS\": \"Goldman Sachs\",\n    \"AAPL\": \"Apple\",\n    \"SAP\": \"SAP\",\n    \"CSCO\": \"Cisco\",\n    \"TXN\": \"Texas Instruments\",\n    \"XRX\": \"Xerox\",\n    \"WMT\": \"Wal-Mart\",\n    \"HD\": \"Home Depot\",\n    \"GSK\": \"GlaxoSmithKline\",\n    \"PFE\": \"Pfizer\",\n    \"SNY\": \"Sanofi-Aventis\",\n    \"NVS\": \"Novartis\",\n    \"KMB\": \"Kimberly-Clark\",\n    \"R\": \"Ryder\",\n    \"GD\": \"General Dynamics\",\n    \"RTN\": \"Raytheon\",\n    \"CVS\": \"CVS\",\n    \"CAT\": \"Caterpillar\",\n    \"DD\": \"DuPont de Nemours\",\n}\n\n\nsymbols, names = np.array(sorted(symbol_dict.items())).T\n\nquotes = []\n\nfor symbol in symbols:\n    print(\"Fetching quote history for %r\" % symbol, file=sys.stderr)\n    url = (\n        \"https://raw.githubusercontent.com/scikit-learn/examples-data/\"\n        \"master/financial-data/{}.csv\"\n    )\n    quotes.append(pd.read_csv(url.format(symbol)))\n\nclose_prices = np.vstack([q[\"close\"] for q in quotes])\nopen_prices = np.vstack([q[\"open\"] for q in quotes])\n\n# The daily variations of the quotes are what carry the most information\nvariation = close_prices - open_prices"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n## Learning a graph structure\n\nWe use sparse inverse covariance estimation to find which quotes are\ncorrelated conditionally on the others. Specifically, sparse inverse\ncovariance gives us a graph, that is a list of connections. For each\nsymbol, the symbols that it is connected to are those useful to explain\nits fluctuations.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn import covariance\n\nalphas = np.logspace(-1.5, 1, num=10)\nedge_model = covariance.GraphicalLassoCV(alphas=alphas)\n\n# standardize the time series: using correlations rather than covariance\n# former is more efficient for structure recovery\nX = variation.copy().T\nX /= X.std(axis=0)\nedge_model.fit(X)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Clustering using affinity propagation\n\nWe use clustering to group together quotes that behave similarly. Here,\namongst the `various clustering techniques <clustering>` available\nin the scikit-learn, we use `affinity_propagation` as it does\nnot enforce equal-size clusters, and it can choose automatically the\nnumber of clusters from the data.\n\nNote that this gives us a different indication than the graph, as the\ngraph reflects conditional relations between variables, while the\nclustering reflects marginal properties: variables clustered together can\nbe considered as having a similar impact at the level of the full stock\nmarket.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn import cluster\n\n_, labels = cluster.affinity_propagation(edge_model.covariance_, random_state=0)\nn_labels = labels.max()\n\nfor i in range(n_labels + 1):\n    print(f\"Cluster {i + 1}: {', '.join(names[labels == i])}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Embedding in 2D space\n\nFor visualization purposes, we need to lay out the different symbols on a\n2D canvas. For this we use `manifold` techniques to retrieve 2D\nembedding.\nWe use a dense eigen_solver to achieve reproducibility (arpack is initiated\nwith the random vectors that we don't control). In addition, we use a large\nnumber of neighbors to capture the large-scale structure.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Finding a low-dimension embedding for visualization: find the best position of\n# the nodes (the stocks) on a 2D plane\n\nfrom sklearn import manifold\n\nnode_position_model = manifold.LocallyLinearEmbedding(\n    n_components=2, eigen_solver=\"dense\", n_neighbors=6\n)\n\nembedding = node_position_model.fit_transform(X.T).T"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Visualization\n\nThe output of the 3 models are combined in a 2D graph where nodes\nrepresents the stocks and edges the:\n\n- cluster labels are used to define the color of the nodes\n- the sparse covariance model is used to display the strength of the edges\n- the 2D embedding is used to position the nodes in the plan\n\nThis example has a fair amount of visualization-related code, as\nvisualization is crucial here to display the graph. One of the challenge\nis to position the labels minimizing overlap. For this we use an\nheuristic based on the direction of the nearest neighbor along each\naxis.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\nfrom matplotlib.collections import LineCollection\n\nplt.figure(1, facecolor=\"w\", figsize=(10, 8))\nplt.clf()\nax = plt.axes([0.0, 0.0, 1.0, 1.0])\nplt.axis(\"off\")\n\n# Plot the graph of partial correlations\npartial_correlations = edge_model.precision_.copy()\nd = 1 / np.sqrt(np.diag(partial_correlations))\npartial_correlations *= d\npartial_correlations *= d[:, np.newaxis]\nnon_zero = np.abs(np.triu(partial_correlations, k=1)) > 0.02\n\n# Plot the nodes using the coordinates of our embedding\nplt.scatter(\n    embedding[0], embedding[1], s=100 * d**2, c=labels, cmap=plt.cm.nipy_spectral\n)\n\n# Plot the edges\nstart_idx, end_idx = np.where(non_zero)\n# a sequence of (*line0*, *line1*, *line2*), where::\n#            linen = (x0, y0), (x1, y1), ... (xm, ym)\nsegments = [\n    [embedding[:, start], embedding[:, stop]] for start, stop in zip(start_idx, end_idx)\n]\nvalues = np.abs(partial_correlations[non_zero])\nlc = LineCollection(\n    segments, zorder=0, cmap=plt.cm.hot_r, norm=plt.Normalize(0, 0.7 * values.max())\n)\nlc.set_array(values)\nlc.set_linewidths(15 * values)\nax.add_collection(lc)\n\n# Add a label to each node. The challenge here is that we want to\n# position the labels to avoid overlap with other labels\nfor index, (name, label, (x, y)) in enumerate(zip(names, labels, embedding.T)):\n    dx = x - embedding[0]\n    dx[index] = 1\n    dy = y - embedding[1]\n    dy[index] = 1\n    this_dx = dx[np.argmin(np.abs(dy))]\n    this_dy = dy[np.argmin(np.abs(dx))]\n    if this_dx > 0:\n        horizontalalignment = \"left\"\n        x = x + 0.002\n    else:\n        horizontalalignment = \"right\"\n        x = x - 0.002\n    if this_dy > 0:\n        verticalalignment = \"bottom\"\n        y = y + 0.002\n    else:\n        verticalalignment = \"top\"\n        y = y - 0.002\n    plt.text(\n        x,\n        y,\n        name,\n        size=10,\n        horizontalalignment=horizontalalignment,\n        verticalalignment=verticalalignment,\n        bbox=dict(\n            facecolor=\"w\",\n            edgecolor=plt.cm.nipy_spectral(label / float(n_labels)),\n            alpha=0.6,\n        ),\n    )\n\nplt.xlim(\n    embedding[0].min() - 0.15 * np.ptp(embedding[0]),\n    embedding[0].max() + 0.10 * np.ptp(embedding[0]),\n)\nplt.ylim(\n    embedding[1].min() - 0.03 * np.ptp(embedding[1]),\n    embedding[1].max() + 0.03 * np.ptp(embedding[1]),\n)\n\nplt.show()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.21"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}