\n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"for c in aircraft_types:\n",
" print(c)\n",
" verify_images(path/c, delete=True, max_size=500)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Visual Inspection"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Clean out the images that don't belong. This is done manually in the file explorer (faster than displaying in jupyter as I did the first time on this project).\n",
"\n",
"I noticed I didn't do the mapping the best way. I should've done a {key: {key:val}} mapping of {class: {int_name: url}}. Instead I did {int_name: url}. This means I have to do a full lookup of every key:value pair in the dictionary for each class. This is not ideal.\n",
"\n",
"Actually one additional mistake means I have to redo the whole download: I didn't save filepaths, I saved *filenames* only, as keys. This means there's no way tell which class a filename belongs to on the dictionary's side. In fact it's worse: because there are going to be at most `n_classes` identical copies of *each filename*... meaning the dictionary is useless because entries are just getting rewritten.\n",
"\n",
"So this forces a chance to correct the original mistake."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Update urls"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All this work was done on my Mac for 2 reasons: I'm not burning GCP credits, and I can review images fastest through macOS's GUI. With the dataset now fully cleaned, I need to transfer those changes to the remote machine. I'm not going to move the images because that won't scale. The dataset was originall 2.27 GB; 150MB after resizing to max(500x500) w/ the fastai image verifier, but still.\n",
"\n",
"Instead I'm going to use the filename-url mapping I worked on creating earlier to find the images that are no longer in the dataset, and remove them from the url files. I already have the code to do the removals. All I need to do is update the file containing urls to remove."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"# load urls to remove\n",
"removal_urls = torch.load(path/'removal_urls.pkl')"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"# update the values - since I forgot to remove the 'Error ' part:\n",
"for k in removal_urls.keys():\n",
" removes = removal_urls[k]\n",
" # cut off the 'Error ' part\n",
" for i in range(len(removes)):\n",
" removes[i] = removes[i].split('Error ')[-1]"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"# load filename-url mappings. {class : {filepath : url}}\n",
"file_mapping = torch.load(path/'file_mapping.pkl')"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"PosixPath('data/aircraft')"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"path"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[PosixPath('data/aircraft/mig21/00000366.JPG'),\n",
" PosixPath('data/aircraft/mig21/00000158.jpg'),\n",
" PosixPath('data/aircraft/mig21/00000170.jpg'),\n",
" PosixPath('data/aircraft/mig21/00000038.jpg'),\n",
" PosixPath('data/aircraft/mig21/00000010.jpg')]"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"flist[:5]"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['https://www.lockheedmartin.com/content/dam/lockheed-martin/aero/photo/f22/f-22.jpg.pc-adaptive.full.medium.jpeg',\n",
" 'https://www.lockheedmartin.com/content/dam/lockheed-martin/aero/photo/f22/F-22%20Speedline%20aircraft_10-31-2016_(Lockheed%20Martin%20photo%20by%20Andrew%20McMurtrie).jpg.pc-adaptive.full.medium.',\n",
" 'https://www.lockheedmartin.com/content/dam/lockheed-martin/aero/photo/f22/F-22-Squadron.png']"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"removal_urls['f22']"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [],
"source": [
"# run through directory, lookup urls of missing files in file_mapping & add to removal_urls\n",
"for clas in aircraft_types:\n",
" flist = (path/clas).ls() # pull all filepaths in class folder\n",
" # I keep getting ideas about better ways to do this; which is great, \n",
" # but for now, the focus is just to get it done. ie dict lookups vs array searches\n",
" for fpath in file_mapping[clas].keys():\n",
" if Path(fpath) not in flist:# remember flist consists of Posix-paths, not strings\n",
" removal_urls[clas].append(file_mapping[clas][fpath])"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [],
"source": [
"# remove links from the url files\n",
"for aircraft_type in removal_urls.keys():\n",
" fpath = path/'fighterjet-urls'/(aircraft_type + '.txt')\n",
" with open(fpath) as f: text_file = [line for line in f] # open file; read lines\n",
" for i,line in enumerate(text_file):\n",
" line = line.rstrip() # remove trailing /n for searching\n",
" if line in removal_urls[aircraft_type]: text_file.pop(i) # remove line from text file\n",
" with open(fpath, mode='wt') as f: # this deletes the original file *I think*: https://stackoverflow.com/a/11469328\n",
" for line in text_file: f.write(line)"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [],
"source": [
"# add contents of removal_urls to the master broken links file\n",
"with open(path/'fighterjet-failed-links.txt', mode='a') as f:\n",
" for c in removal_urls.keys():\n",
" f.writelines(f'{c}\\n')\n",
" for line in removal_urls[c]:\n",
" f.writelines(f'{line}\\n')"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [],
"source": [
"# save removal_urls to disk (not sure if I'll keep this or the other file)\n",
"torch.save(removal_urls, path/'removal_urls.pkl')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The filemapping is no longer relevant since the images will be redownloaded on the other machine, and will have new mappings."
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"6373"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tot = 0\n",
"for clas in aircraft_types: tot += len((path/clas).ls())\n",
"tot"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The original dataset size was 10,241 images, this's been cleaned down to 6,373."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (FastAI)",
"language": "python",
"name": "fastai"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}