Classification of Paintings Using Fast AI

In [1]:
## Imports
import pandas as pd
import numpy as np
import os
from os import listdir
from os.path import isfile, join

from fastai import *
from fastai.vision import *

# function I will use to create folders for my classes in directory
def createFolder(directory):
    try:
        if not os.path.exists(directory):
            os.makedirs(directory)
    except OSError:
        print ('Error: Creating directory. ' +  directory)

warnings.filterwarnings("ignore")

Create a Training and Testing dataset with Google Images

In [2]:
# this list contains a set of lists containing the name of the folder 
# and the name of the text file that will contain the path of the images to download.

folder_paths_list = [['classic', 'classic_art.txt'],['contemporary', 'contemporary_art.txt'],
                    ['medieval','medieval_art.txt'], ['modern','modern_art.txt'],
                    ['renaissance', 'renaissance_art.txt'], ['romanticism','romanticism_art.txt']]

# create folders in the data folder. each folder will contain the pictures that belong to that category.
for i in folder_paths_list:
    createFolder('./data/art/'+i[0])

Get images url from google images by:

  • go to google image and search for your category.
  • scroll untill you have seen enough pictures
  • Press CtrlShiftJ in Windows/Linux and CmdOptJ in Mac and copy the script below into the command line that appears.
urls = Array.from(document.querySelectorAll('.rg_di .rg_meta')).map(el=>JSON.parse(el.textContent).ou);
window.open('data:text/csv;charset=utf-8,' + escape(urls.join('\n')));
  • save files as .txt files and move them into data folder for its category. name the text file as was decided in the list above
In [3]:
# download pictures in to the folders. 
path = Path('data/art')
for i in folder_paths_list:
    dest = path/i[0]
    dest.mkdir(parents=True, exist_ok=True)
    download_images(path/i[0]/i[1], dest, max_pics=200)
In [4]:
# verify images
classes = [i[0] for i in folder_paths_list]
for c in classes:
    print(c)
    verify_images(path/c, delete=True, max_size=500)
In [5]:
# create data set and split it into training and testing datasets
np.random.seed(42)
data = ImageDataBunch.from_folder(path, train=".", valid_pct=0.2,
        ds_tfms=get_transforms(), size=224, num_workers=4).normalize(imagenet_stats)
In [6]:
data.classes
Out[6]:
['classic', 'contemporary', 'medieval', 'modern', 'renaissance', 'romanticism']
In [8]:
data.show_batch(rows=3, figsize=(7,8))
In [9]:
print("these are my data classes", data.classes)
print('There are in total {} classes'.format(data.c))
print('{} pictures are in my training dataset, {} are in my testing dataset'.format(len(data.train_ds), len(data.valid_ds)))

these are my data classes ['classic', 'contemporary', 'medieval', 'modern', 'renaissance', 'romanticism']
There are in total 6 classes
896 pictures are in my training dataset, 224 are in my testing dataset

Train Model

First, let's create out convolutional neural network using the data we have created and by using resnet34. we also want to print out the error rate at each iteration.

In [10]:
learn = cnn_learner(data, models.resnet34, metrics=error_rate)

Now let's fit the model onto the data and iterate n times

In [11]:
learn.fit_one_cycle(4)
epoch train_loss valid_loss error_rate time
0 2.315930 1.132018 0.375000 00:05
1 1.676171 1.185606 0.361607 00:05
2 1.364731 1.203666 0.375000 00:05
3 1.150863 1.184371 0.375000 00:05

The table shows that the model is wrong ~37% of the times

In [12]:
# save model
learn.save('stage-1')

Improve the model

To improve the model we want to unfreeze the rest of our model and run the learning rate finder

In [13]:
learn.unfreeze()
In [14]:
learn.lr_find()

LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.
In [15]:
# plot learning rate
learn.recorder.plot()
In [16]:
learn.fit_one_cycle(5, max_lr=slice(1e-5,1e-3))
epoch train_loss valid_loss error_rate time
0 0.805280 1.072682 0.348214 00:05
1 0.735670 1.025120 0.370536 00:05
2 0.632514 0.980505 0.339286 00:05
3 0.537677 0.978524 0.339286 00:05
4 0.472308 0.980168 0.348214 00:05

The error rate is now slightle better. This tells us that our model is wrong ~33% of the times.

In [17]:
# save model
learn.save('stage-2')

Interpretation of Results

In [18]:
# load the latest model
learn.load('stage-2');
In [19]:
interp = ClassificationInterpretation.from_learner(learn)
losses,idxs = interp.top_losses()
len(data.valid_ds)==len(losses)==len(idxs)
Out[19]:
True
In [20]:
# look at top images that are classfied wrong
interp.plot_top_losses(6, figsize=(15,11))
In [21]:
# plot confusion matrix. 
interp.plot_confusion_matrix()

The confusion matrix tells us that the most commor error that our model makes is between modern art and contemporary art.

We can do better if the dataset if less noisy.

Clean Up data - Optional

In [30]:
# import FastAI widgets
from fastai.widgets import *
In [31]:
# we are going to create a dataset that contains all images together (test + train)
db = (ImageList.from_folder(path)
                   .split_none()
                   .label_from_folder()
                   .transform(get_transforms(), size=224)
                   .databunch()
     )
In [32]:
# create model on new data, load state2 model and check the top losses
learn_cln = cnn_learner(db, models.resnet34, metrics=error_rate)

learn_cln.load('stage-2');
In [33]:
ds, idxs = DatasetFormatter().from_toplosses(learn_cln)
In [34]:
ImageCleaner(ds, idxs, path)
Out[34]:
<fastai.widgets.image_cleaner.ImageCleaner at 0x7f0382b72150>
In [35]:
ds, idxs = DatasetFormatter().from_similars(learn_cln)

Getting activations...
100.00% [18/18 01:17<00:00] 
Computing similarities...
In [36]:
ImageCleaner(ds, idxs, path, duplicates=True)
Out[36]:
<fastai.widgets.image_cleaner.ImageCleaner at 0x7f038178de90>

Put model into production

Create folder with test pictures

In [22]:
createFolder('./data/test_pics')
# manually add pictures in the test folder that you want to test
path_test = Path('data/test_pics')
In [23]:
# list all of items in the test directory
pictures = [f for f in listdir('./data/test_pics') if isfile(join('./data/test_pics', f))]
pictures
Out[23]:
['pic1.jpg', 'pic2.jpg']
In [24]:
# export model 
learn.export()
defaults.device = torch.device('cpu')

Show test pictures that I have selected

In [25]:
img = open_image(path_test/pictures[0])
img
Out[25]:

A painting by Piero della Francesca that is entitled: Double Portrait of Battista Sforza, Duchess of Urbino and Federico da Duke of Urbino Montefeltro, It is showing a husband and wife looking at each other over a frame. This is Northern Renaissance art made with oil and wood.

In [26]:
img = open_image(path_test/pictures[1])
img
Out[26]:

Pablo Picasso’s 1937 painting “Femme au Béret et à la Robe Quadrillée (Marie-Thérèse Walter)” is considered modern art as modern art period spans from 1860s to the 1970s

In [29]:
for p in pictures:
    img = open_image(path_test/p)
    learn = load_learner(path)
    pred_class,pred_idx,outputs = learn.predict(img)
    print('picture {} is classified as {}'.format(pictures.index(p)+1, pred_class))

picture 1 is classified as renaissance
picture 2 is classified as modern