Image Caption

You could download Microsoft COCO dataset from here

VGG 16 pretrained model could download from here

如果你是中国人，可以从百度网盘的这里下载

Then unzip COCO and specify path in image_caption_keras.py :

vgg_model_weights = '/home/qhduan/Downloads/COCO/vgg16_weights.h5'
coco_train = '/home/qhduan/Downloads/COCO/train2014'
coco_caption = '/home/qhduan/Downloads/COCO/annotations/captions_train2014.json'

You could open preview.ipynb to view the train result and test result.

$ python3 image_caption_keras.py
Using Theano backend.
Using gpu device 0: GeForce GTX 1070 (CNMeM is disabled, cuDNN 5105)
/home/qhduan/.local/lib/python3.5/site-packages/theano/sandbox/cuda/__init__.py:600: UserWarning: Your cuDNN version is more recent than the one Theano officially supports. If you see any problems, try updating Theano or downgrading cuDNN to version 5.
  warnings.warn(warn)
[nltk_data] Downloading package punkt to /home/qhduan/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
train_size 82783
100%|████████████████████████████████████████████████████████████████████████████████████████| 82783/82783 [00:06<00:00, 12363.22it/s]
vocabulary_size 8679
max_len 55
train_words_size 935568
(256, 4096) (256, 55) (256, 8679)
(256, 4096) (256, 55) (256, 8679)
(256, 4096) (256, 55) (256, 8679)
(256, 4096) (256, 55) (256, 8679)
(256, 4096) (256, 55) (256, 8679)
(256, 4096) (256, 55) (256, 8679)
(256, 4096) (256, 55) (256, 8679)
(256, 4096) (256, 55) (256, 8679)
(256, 4096) (256, 55) (256, 8679)
(256, 4096) (256, 55) (256, 8679)
(256, 4096) (256, 55) (256, 8679)
Epoch 1/20
935680/935680 [==============================] - 980s - loss: 4.1896 - acc: 0.2899          
Epoch 2/20
935680/935680 [==============================] - 980s - loss: 3.1131 - acc: 0.4088     
Epoch 3/20
935680/935680 [==============================] - 1024s - loss: 2.8843 - acc: 0.4295    
Epoch 4/20
935680/935680 [==============================] - 1053s - loss: 2.7526 - acc: 0.4415     
Epoch 5/20
935680/935680 [==============================] - 1053s - loss: 2.6622 - acc: 0.4504     
Epoch 6/20
935680/935680 [==============================] - 1001s - loss: 2.5747 - acc: 0.4587     
Epoch 7/20
935680/935680 [==============================] - 988s - loss: 2.4988 - acc: 0.4663      
Epoch 8/20
935680/935680 [==============================] - 1060s - loss: 2.4339 - acc: 0.4740     
Epoch 9/20
935680/935680 [==============================] - 1032s - loss: 2.3833 - acc: 0.4802     
Epoch 10/20
935680/935680 [==============================] - 1005s - loss: 2.3305 - acc: 0.4866     
Epoch 11/20
935680/935680 [==============================] - 1007s - loss: 2.2816 - acc: 0.4927     
Epoch 12/20
935680/935680 [==============================] - 1063s - loss: 2.2408 - acc: 0.4987     
Epoch 13/20
935680/935680 [==============================] - 1031s - loss: 2.1983 - acc: 0.5048     
Epoch 14/20
935680/935680 [==============================] - 995s - loss: 2.1705 - acc: 0.5086      
Epoch 15/20
935680/935680 [==============================] - 991s - loss: 2.1432 - acc: 0.5122     
Epoch 16/20
935680/935680 [==============================] - 984s - loss: 2.1109 - acc: 0.5173      
Epoch 17/20
935680/935680 [==============================] - 978s - loss: 2.0837 - acc: 0.5222      
Epoch 18/20
935680/935680 [==============================] - 978s - loss: 2.0582 - acc: 0.5257     
Epoch 19/20
935680/935680 [==============================] - 978s - loss: 2.0487 - acc: 0.5281     
Epoch 20/20
935680/935680 [==============================] - 978s - loss: 2.0179 - acc: 0.5325