やりたいこと

  • google colabを使用してYOLO-v3において画像データを学習させる。

すでにやったこと

  • 教師画像になるデータを20枚集めてリサイズ
  • 集めた画像をVottでアノテーションしzipとしてcolab上にあげ、unzip
  • YOLO用のデータに変換
  • Kerasで使えるように変換

詰まったところ

上記のことをやった上でtrain.pyを実行したところ以下のエラーが出ました。

2019-09-09 08:03:13.567901: W tensorflow/core/common_runtime/bfc_allocator.cc:319] ****************************************************************************************************
2019-09-09 08:03:13.568544: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at fused_batch_norm_op.cc:487 : Resource exhausted: OOM when allocating tensor with shape[32,512,20,20] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "train.py", line 195, in <module>
    _main()
  File "train.py", line 89, in _main
    callbacks=[logging, checkpoint, reduce_lr, early_stopping])
  File "/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1418, in fit_generator
    initial_epoch=initial_epoch)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py", line 217, in fit_generator
    class_weight=class_weight)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1217, in train_on_batch
    outputs = self.train_function(ins)
  File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__
    return self._call(inputs)
  File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2675, in _call
    fetched = self._callable_fn(*array_vals)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1458, in __call__
    run_metadata_ptr)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[32,512,20,20] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[{{node batch_normalization_41/FusedBatchNorm}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[loss_1/add_74/_5299]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[32,512,20,20] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[{{node batch_normalization_41/FusedBatchNorm}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

python初学者なのでとりあえずテストで動かしてみる事を現状のマイルストーンとしています。
よろしくお願いします。

使用したpkg
https://github.com/sleepless-se/keras-yolo3