{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 8.1 Parsing Unix timestamps" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It's not obvious how to deal with Unix timestamps in pandas -- it took me quite a while to figure this out. The file we're using here is a popularity-contest file I found on my system at `/var/log/popularity-contest`.\n", "\n", "Here's an [explanation of how this file works](http://popcon.ubuntu.com/README).\n", "\n", "I'm going to hope that nothing in it is sensitive :)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "# Read it, and remove the last row\n", "popcon = pd.read_csv('../data/popularity-contest', sep=' ', )[:-1]\n", "popcon.columns = ['atime', 'ctime', 'package-name', 'mru-program', 'tag']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The colums are the access time, created time, package name, recently used program, and a tag" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
atimectimepackage-namemru-programtag
013872957971367633260perl-base/usr/bin/perlNaN
113872957961354370480login/bin/suNaN
213872957431354341275libtalloc2/usr/lib/x86_64-linux-gnu/libtalloc.so.2.0.7NaN
313872957431387224204libwbclient0/usr/lib/x86_64-linux-gnu/libwbclient.so.0<RECENT-CTIME>
413872957421354341253libselinux1/lib/x86_64-linux-gnu/libselinux.so.1NaN
\n", "
" ], "text/plain": [ " atime ctime package-name \\\n", "0 1387295797 1367633260 perl-base \n", "1 1387295796 1354370480 login \n", "2 1387295743 1354341275 libtalloc2 \n", "3 1387295743 1387224204 libwbclient0 \n", "4 1387295742 1354341253 libselinux1 \n", "\n", " mru-program tag \n", "0 /usr/bin/perl NaN \n", "1 /bin/su NaN \n", "2 /usr/lib/x86_64-linux-gnu/libtalloc.so.2.0.7 NaN \n", "3 /usr/lib/x86_64-linux-gnu/libwbclient.so.0 \n", "4 /lib/x86_64-linux-gnu/libselinux.so.1 NaN " ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "popcon[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The magical part about parsing timestamps in pandas is that numpy datetimes are already stored as Unix timestamps. So all we need to do is tell pandas that these integers are actually datetimes -- it doesn't need to do any conversion at all.\n", "\n", "We need to convert these to ints to start:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "popcon['atime'] = popcon['atime'].astype(int)\n", "popcon['ctime'] = popcon['ctime'].astype(int)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Every numpy array and pandas series has a dtype -- this is usually `int64`, `float64`, or `object`. Some of the time types available are `datetime64[s]`, `datetime64[ms]`, and `datetime64[us]`. There are also `timedelta` types, similarly.\n", "\n", "We can use the `pd.to_datetime` function to convert our integer timestamps into datetimes. This is a constant-time operation -- we're not actually changing any of the data, just how pandas thinks about it." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "popcon['atime'] = pd.to_datetime(popcon['atime'], unit='s')\n", "popcon['ctime'] = pd.to_datetime(popcon['ctime'], unit='s')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we look at the dtype now, it's `\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
atimectimepackage-namemru-programtag
02013-12-17 15:56:372013-05-04 02:07:40perl-base/usr/bin/perlNaN
12013-12-17 15:56:362012-12-01 14:01:20login/bin/suNaN
22013-12-17 15:55:432012-12-01 05:54:35libtalloc2/usr/lib/x86_64-linux-gnu/libtalloc.so.2.0.7NaN
32013-12-17 15:55:432013-12-16 20:03:24libwbclient0/usr/lib/x86_64-linux-gnu/libwbclient.so.0<RECENT-CTIME>
42013-12-17 15:55:422012-12-01 05:54:13libselinux1/lib/x86_64-linux-gnu/libselinux.so.1NaN
\n", "" ], "text/plain": [ " atime ctime package-name \\\n", "0 2013-12-17 15:56:37 2013-05-04 02:07:40 perl-base \n", "1 2013-12-17 15:56:36 2012-12-01 14:01:20 login \n", "2 2013-12-17 15:55:43 2012-12-01 05:54:35 libtalloc2 \n", "3 2013-12-17 15:55:43 2013-12-16 20:03:24 libwbclient0 \n", "4 2013-12-17 15:55:42 2012-12-01 05:54:13 libselinux1 \n", "\n", " mru-program tag \n", "0 /usr/bin/perl NaN \n", "1 /bin/su NaN \n", "2 /usr/lib/x86_64-linux-gnu/libtalloc.so.2.0.7 NaN \n", "3 /usr/lib/x86_64-linux-gnu/libwbclient.so.0 \n", "4 /lib/x86_64-linux-gnu/libselinux.so.1 NaN " ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "popcon[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now suppose we want to look at all packages that aren't libraries.\n", "\n", "First, I want to get rid of everything with timestamp 0. Notice how we can just use a string in this comparison, even though it's actually a timestamp on the inside? That is because pandas is amazing." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "popcon = popcon[popcon['atime'] > '1970-01-01']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can use pandas' magical string abilities to just look at rows where the package name doesn't contain 'lib'." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "nonlibraries = popcon[~popcon['package-name'].str.contains('lib')]" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
atimectimepackage-namemru-programtag
572013-12-17 04:55:392013-12-17 04:55:42ddd/usr/bin/ddd<RECENT-CTIME>
4502013-12-16 20:03:202013-12-16 20:05:13nodejs/usr/bin/npm<RECENT-CTIME>
4542013-12-16 20:03:202013-12-16 20:05:04switchboard-plug-keyboard/usr/lib/plugs/pantheon/keyboard/options.txt<RECENT-CTIME>
4452013-12-16 20:03:202013-12-16 20:05:04thunderbird-locale-en/usr/lib/thunderbird-addons/extensions/langpac...<RECENT-CTIME>
3962013-12-16 20:08:272013-12-16 20:05:03software-center/usr/sbin/update-software-center<RECENT-CTIME>
4492013-12-16 20:03:202013-12-16 20:05:00samba-common-bin/usr/bin/net.samba3<RECENT-CTIME>
3972013-12-16 20:08:252013-12-16 20:04:59postgresql-client-9.1/usr/lib/postgresql/9.1/bin/psql<RECENT-CTIME>
3982013-12-16 20:08:232013-12-16 20:04:58postgresql-9.1/usr/lib/postgresql/9.1/bin/postmaster<RECENT-CTIME>
4522013-12-16 20:03:202013-12-16 20:04:55php5-dev/usr/include/php5/main/snprintf.h<RECENT-CTIME>
4402013-12-16 20:03:202013-12-16 20:04:54php-pear/usr/share/php/XML/Util.php<RECENT-CTIME>
\n", "
" ], "text/plain": [ " atime ctime package-name \\\n", "57 2013-12-17 04:55:39 2013-12-17 04:55:42 ddd \n", "450 2013-12-16 20:03:20 2013-12-16 20:05:13 nodejs \n", "454 2013-12-16 20:03:20 2013-12-16 20:05:04 switchboard-plug-keyboard \n", "445 2013-12-16 20:03:20 2013-12-16 20:05:04 thunderbird-locale-en \n", "396 2013-12-16 20:08:27 2013-12-16 20:05:03 software-center \n", "449 2013-12-16 20:03:20 2013-12-16 20:05:00 samba-common-bin \n", "397 2013-12-16 20:08:25 2013-12-16 20:04:59 postgresql-client-9.1 \n", "398 2013-12-16 20:08:23 2013-12-16 20:04:58 postgresql-9.1 \n", "452 2013-12-16 20:03:20 2013-12-16 20:04:55 php5-dev \n", "440 2013-12-16 20:03:20 2013-12-16 20:04:54 php-pear \n", "\n", " mru-program tag \n", "57 /usr/bin/ddd \n", "450 /usr/bin/npm \n", "454 /usr/lib/plugs/pantheon/keyboard/options.txt \n", "445 /usr/lib/thunderbird-addons/extensions/langpac... \n", "396 /usr/sbin/update-software-center \n", "449 /usr/bin/net.samba3 \n", "397 /usr/lib/postgresql/9.1/bin/psql \n", "398 /usr/lib/postgresql/9.1/bin/postmaster \n", "452 /usr/include/php5/main/snprintf.h \n", "440 /usr/share/php/XML/Util.php " ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nonlibraries.sort_values('ctime', ascending=False)[:10]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Okay, cool, it says that I I installed ddd recently. And postgresql! I remember installing those things. Neat." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The whole message here is that if you have a timestamp in seconds or milliseconds or nanoseconds, then you can just \"cast\" it to a `'datetime64[the-right-thing]'` and pandas/numpy will take care of the rest." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "