tf_distributeKeras

케라스를 사용한 분산형 학습입니다.

이 글에서 학습한 부분은 상당히 중요하다고 생각합니다.

 

먼저 콜백에 관한 내용입니다.

에포크마다 체크포인트를 저장하고 이 저장된 체크포인트에서 가중치를 불러와 시험(test)할 수 있습니다.

그리고 학습률도 에포크가 끝난 후 조정하는 내용입니다.

 

단순 훈련, 검증하는 것을 넘어서, 모델을 저장하고

저장한 모델을 다시 불러와 다른 시험(test)을 하는 내용입니다.

 

이번에 학습한 내용은 해커톤을 할 때 많은 도움을 준 내용입니다. 

해커톤 전에 미리 봤더라면 하는 아쉬움이 남습니다.

 

튜토리얼은 주피터 노트북 첫 번째 셀에 링크가 있으니 그곳에 가서 학습하셔도 됩니다.

 

<참조 : https://www.tensorflow.org/tutorials/distribute/keras?hl=ko>

In [ ]:

케라스를 사용한 분산훈련¶

• tf.distribute.Strategy : 훈련을 여러 처리 장치들로 분산시키는 것을 추상화한 것입니다
  • 기존 모델이나 훈련 코드를 조금 바꾸어 분산 훈련을 할 수 있게 하는 것이 분산 전략 API의 목표
• tf.distribute.MirroredStrategy 사용
  • 동기화된 훈련 방식을 활용하여 한 장비에 여러 개의 GPU로 그래프 내 복제를 수행합니다
  • 모델의 모든 변수를 각 프로세서에 복사합니다
  • 각 프로세서의 그래디언트(gradient)를 올 리듀스(all-reduce)를 사용하여 모읍니다
  • 계산한 값을 각 프로세서의 모델 복사본에 적용합니다

In [ ]:

01. 필요한 패키지 가져오기¶

In [2]:

# 텐서플로와 텐서플로 데이터셋 패키지 가져오기
import tensorflow_datasets as tfds
import tensorflow as tf

tfds.disable_progress_bar()

import os

In [ ]:

02. Dataset Download¶

In [3]:

datasets, info = tfds.load(name='mnist', with_info=True, as_supervised=True)

mnist_train, mnist_test = datasets['train'], datasets['test']

Downloading and preparing dataset Unknown size (download: Unknown size, generated: Unknown size, total: Unknown size) to C:\Users\jjanh_000\tensorflow_datasets\mnist\3.0.1...
Dataset mnist downloaded and prepared to C:\Users\jjanh_000\tensorflow_datasets\mnist\3.0.1. Subsequent calls will reuse this data.

In [ ]:

03. 분산 전략 정의하기¶

In [4]:

strategy = tf.distribute.MirroredStrategy()

WARNING:tensorflow:There are non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.

WARNING:tensorflow:There are non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.

INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',)

INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',)

In [5]:

print('장치 수: {}'.format(strategy.num_replicas_in_sync))

장치 수: 1

In [ ]:

04. 입력 파이프라인 구성하기¶

• 다중 GPU로 모델을 훈련할 때는 배치 크기를 늘려야 컴퓨팅 자원을 효과적으로 사용할 수 있습니다
• GPU 메모리에 맞추어 가능한 가장 큰 배치 크기를 사용합니다
  • 이에 맞게 학습률도 조정해야 합니다

In [7]:

# 데이터셋 내 샘플 수는 info.splits.total_num_examples로도 얻을 수 있습니다

num_train_examples = info.splits['train'].num_examples
num_test_examples = info.splits['test'].num_examples

BUFFER_SIZE = 10000

BATCH_SIZE_PER_REPLICA = 62
BATCH_SIZE = BATCH_SIZE_PER_REPLICA * strategy.num_replicas_in_sync

In [8]:

# 픽셀 값은 0~255 : 0~1 범위로 정규화

def scale(image, label):
    image = tf.cast(image, tf.float32)
    image /= 255
    
    return image, label

In [9]:

train_dataset = mnist_train.map(scale).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)
eval_dataset = mnist_test.map(scale).batch(BATCH_SIZE)

In [ ]:

05. 모델 만들기¶

In [10]:

with strategy.scope():
    model = tf.keras.Sequential([
        tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(28, 28, 1)),
        tf.keras.layers.MaxPooling2D(),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(64, activation='relu'),
        tf.keras.layers.Dense(10, activation='softmax')
    ])

In [11]:

model.compile(loss='sparse_categorical_crossentropy',
             optimizer=tf.keras.optimizers.Adam(),
             metrics=['accuracy'])

In [ ]:

06. 콜백 정의하기¶

• 텐서보드(TensorBoard) : 텐서보드용 로그를 남겨서, 텐서보드에서 그래프를 그릴 수 있게 합니다
• 모델 체크포인트(Checkpoint) : 매 에포크(epoch)가 끝난 후 모델을 저장합니다
• 학습률 스케줄러 : 매 에포크 혹은 배치가 끝난 후 학습률을 바꿀 수 있습니다

In [12]:

# 체크포인트를 저장할 체크포인트 디렉토리를 지정합니다
checkpoint_dir = './dK_training_checkpoints'

# 체크포인트 파일의 이름
checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt_{epoch}')

In [13]:

# 학습률을 점점 줄이기 위한 함수
# 필요한 함수를 직접 정의하여 사용할 수 있습ㄴ다

def decay(epoch):
    if epoch < 3:
        return 1e-3
    if epoch >3 and epoch < 7:
        return 1e-4
    else:
        return 1e-5

In [14]:

# 에포크가 끝날 때마다 학습률을 출력하는 콜백
class PrintLR(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs=None):
        print('\n에포크 {}의 학습률은 {}입니다.'.format(epoch+1, model.optimizer.lr.numpy()))

In [15]:

callbacks = [
    tf.keras.callbacks.TensorBoard(log_dir='./dK_logs'),
    tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_prefix,
                                      save_weights_only=True),
    tf.keras.callbacks.LearningRateScheduler(decay),
    PrintLR()
]

In [ ]:

07. 훈련과 평가¶

In [16]:

model.fit(train_dataset, epochs=12, callbacks=callbacks)

Epoch 1/12
  6/968 [..............................] - ETA: 6:53 - loss: 2.0570 - accuracy: 0.4113WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0864s vs `on_train_batch_end` time: 0.2818s). Check your callbacks.

WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0864s vs `on_train_batch_end` time: 0.2818s). Check your callbacks.

968/968 [==============================] - 103s 99ms/step - loss: 0.2078 - accuracy: 0.9394s - loss: 0.2108 - ac

에포크 1의 학습률은 0.0010000000474974513입니다.
Epoch 2/12
968/968 [==============================] - 65s 66ms/step - loss: 0.0689 - accuracy: 0.9795

에포크 2의 학습률은 0.0010000000474974513입니다.
Epoch 3/12
968/968 [==============================] - 78s 80ms/step - loss: 0.0471 - accuracy: 0.9861

에포크 3의 학습률은 0.0010000000474974513입니다.
Epoch 4/12
968/968 [==============================] - 82s 84ms/step - loss: 0.0301 - accuracy: 0.9917

에포크 4의 학습률은 9.999999747378752e-06입니다.
Epoch 5/12
968/968 [==============================] - 77s 78ms/step - loss: 0.0254 - accuracy: 0.9931

에포크 5의 학습률은 9.999999747378752e-05입니다.
Epoch 6/12
968/968 [==============================] - 73s 74ms/step - loss: 0.0227 - accuracy: 0.9943

에포크 6의 학습률은 9.999999747378752e-05입니다.
Epoch 7/12
968/968 [==============================] - 74s 75ms/step - loss: 0.0208 - accuracy: 0.9947

에포크 7의 학습률은 9.999999747378752e-05입니다.
Epoch 8/12
968/968 [==============================] - 74s 75ms/step - loss: 0.0183 - accuracy: 0.9956

에포크 8의 학습률은 9.999999747378752e-06입니다.
Epoch 9/12
968/968 [==============================] - 76s 78ms/step - loss: 0.0179 - accuracy: 0.9958

에포크 9의 학습률은 9.999999747378752e-06입니다.
Epoch 10/12
968/968 [==============================] - 92s 94ms/step - loss: 0.0177 - accuracy: 0.9958

에포크 10의 학습률은 9.999999747378752e-06입니다.
Epoch 11/12
968/968 [==============================] - 80s 81ms/step - loss: 0.0175 - accuracy: 0.9959

에포크 11의 학습률은 9.999999747378752e-06입니다.
Epoch 12/12
968/968 [==============================] - 83s 85ms/step - loss: 0.0173 - accuracy: 0.9960

에포크 12의 학습률은 9.999999747378752e-06입니다.

Out[16]:

<keras.callbacks.History at 0x1012abc1f0>

In [ ]:

In [17]:

# 체크포인트 디렉토리 확인하기
!ls {checkpoint_dir}

'ls'은(는) 내부 또는 외부 명령, 실행할 수 있는 프로그램, 또는
배치 파일이 아닙니다.

goole 검색 : python directory file list -> https://stackoverflow.com/questions/3207219/how-do-i-list-all-files-of-a-directory¶

from os import listdir
from os.path import isfile, join
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

---

from os import walk

f = []
for (dirpath, dirnames, filenames) in walk(mypath):
    f.extend(filenames)
    break

---

from os import walk

filenames = next(walk(mypath), (None, None, []))[2]  # [] if no file

---

for (dirpath, dirnames, filenames) in os.walk(mypath):
    checksum_files.extend(os.path.join(dirpath, filename) for filename in filenames) 
    break

https://careerkarma.com/blog/python-list-files-in-directory/¶

import os

path = '/home/data_analysis/netflix'

files = os.listdir(path)

for f in files:
    print(f)

---

import os

path = '/home/data_analysis/netflix'

for root, directories, files in os.walk(path, topdown=False):
    for name in files:
        print(os.path.join(root, name))
    for name in directories:
        print(os.path.join(root, name))

다른 표현식¶

• (, , filenames) = walk(mypath).next()
• f.extend(filenames) : f += filenames
  • not : f = f + filenames
• , , filenames = next(walk(mypath), (None, None, []))

In [18]:

files = os.listdir(checkpoint_dir)

for f in files:
    print(f)

checkpoint
ckpt_1.data-00000-of-00001
ckpt_1.index
ckpt_10.data-00000-of-00001
ckpt_10.index
ckpt_11.data-00000-of-00001
ckpt_11.index
ckpt_12.data-00000-of-00001
ckpt_12.index
ckpt_2.data-00000-of-00001
ckpt_2.index
ckpt_3.data-00000-of-00001
ckpt_3.index
ckpt_4.data-00000-of-00001
ckpt_4.index
ckpt_5.data-00000-of-00001
ckpt_5.index
ckpt_6.data-00000-of-00001
ckpt_6.index
ckpt_7.data-00000-of-00001
ckpt_7.index
ckpt_8.data-00000-of-00001
ckpt_8.index
ckpt_9.data-00000-of-00001
ckpt_9.index

In [ ]:

• 모델의 성능 확인 : 가장 최근 체크포인트를 불러온 후 테스트 데이터에 대하여 evaluate를 호출합니다

In [19]:

model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))

eval_loss, eval_acc = model.evaluate(eval_dataset)

print('평가 손실: {}, 평가 정확도: {}'.format(eval_loss, eval_acc))

162/162 [==============================] - 8s 31ms/step - loss: 0.0395 - accuracy: 0.9871
평가 손실: 0.03950142487883568, 평가 정확도: 0.9871000051498413

In [ ]:

08. SavedModel로 내보내기¶

• 플랫폼과 무관한 SavedModel 형식으로 그래프와 변수들을 내보냅니다.

In [21]:

path = 'saved_model/'

In [22]:

tf.keras.experimental.export_saved_model(model, path)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-22-7f22af6799f5> in <module>
----> 1 tf.keras.experimental.export_saved_model(model, path)

AttributeError: module 'keras.api._v2.keras.experimental' has no attribute 'export_saved_model'

오류 발생하여 검색 : https://www.tensorflow.org/api_docs/python/tf/compat/v1/keras/experimental/export_saved_model¶

• 튜토리얼 코드 수정하여 성공

• 모델을 저장하고 저장한 모델을 불러와서 사용
• 다시 훈련시키지 않아도 됩니다

• 블로그 링크 : https://hwiyong.tistory.com/116

In [23]:

tf.compat.v1.keras.experimental.export_saved_model(model, path)

C:\Users\jjanh_000\anaconda3\lib\site-packages\keras\saving\saved_model_experimental.py:117: UserWarning: `tf.keras.experimental.export_saved_model` is deprecatedand will be removed in a future version. Please use `model.save(..., save_format="tf")` or `tf.keras.models.save_model(..., save_format="tf")`.
  warnings.warn('`tf.keras.experimental.export_saved_model` is deprecated'
C:\Users\jjanh_000\anaconda3\lib\site-packages\keras\backend.py:459: UserWarning: `tf.keras.backend.learning_phase_scope` is deprecated and will be removed after 2020-10-11. To update it, simply pass a True/False value to the `training` argument of the `__call__` method of your layer or model.
  warnings.warn('`tf.keras.backend.learning_phase_scope` is deprecated and '
C:\Users\jjanh_000\anaconda3\lib\site-packages\keras\engine\training.py:2427: UserWarning: `Model.state_updates` will be removed in a future version. This property should not be used in TensorFlow 2.0, as `updates` are applied automatically.
  warnings.warn('`Model.state_updates` will be removed in a future version. '

WARNING:tensorflow:From C:\Users\jjanh_000\anaconda3\lib\site-packages\keras\saving\utils_v1\signature_def_utils.py:63: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.

WARNING:tensorflow:From C:\Users\jjanh_000\anaconda3\lib\site-packages\keras\saving\utils_v1\signature_def_utils.py:63: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Train: ['train']

INFO:tensorflow:Signatures INCLUDED in export for Train: ['train']

INFO:tensorflow:Signatures INCLUDED in export for Eval: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: None

WARNING:tensorflow:Export includes no default signature!

WARNING:tensorflow:Export includes no default signature!

INFO:tensorflow:No assets to save.

INFO:tensorflow:No assets to save.

INFO:tensorflow:No assets to write.

INFO:tensorflow:No assets to write.

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

WARNING:tensorflow:Export includes no default signature!

WARNING:tensorflow:Export includes no default signature!

INFO:tensorflow:No assets to save.

INFO:tensorflow:No assets to save.

INFO:tensorflow:No assets to write.

INFO:tensorflow:No assets to write.

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: ['serving_default']

INFO:tensorflow:Signatures INCLUDED in export for Predict: ['serving_default']

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: None

INFO:tensorflow:No assets to save.

INFO:tensorflow:No assets to save.

INFO:tensorflow:No assets to write.

INFO:tensorflow:No assets to write.

INFO:tensorflow:SavedModel written to: saved_model/saved_model.pb

INFO:tensorflow:SavedModel written to: saved_model/saved_model.pb

In [ ]:

In [26]:

unreplicated_model = tf.compat.v1.keras.experimental.load_from_saved_model(path)

unreplicated_model.compile(
    loss='sparse_categorical_crossentropy',
    optimizer=tf.keras.optimizers.Adam(),
    metrics=['accuracy']
)

eval_loss, eval_acc = unreplicated_model.evaluate(eval_dataset)

print('평가 손실: {}, 평가 정확도: {}'.format(eval_loss, eval_acc))

C:\Users\jjanh_000\anaconda3\lib\site-packages\keras\saving\saved_model_experimental.py:400: UserWarning: `tf.keras.experimental.load_from_saved_model` is deprecatedand will be removed in a future version. Please switch to `tf.keras.models.load_model`.
  warnings.warn('`tf.keras.experimental.load_from_saved_model` is deprecated'

162/162 [==============================] - 5s 28ms/step - loss: 0.0395 - accuracy: 0.9871
평가 손실: 0.03950142487883568, 평가 정확도: 0.9871000051498413

In [ ]: