210721 tf_imageSegmentation

image Segmentation : 이미지에서 사물이나 물건의 모습을 구분하여 나누는 것입니다.

이렇게 나누어진 부분을 다른 이미지로 대체할 수도 있고, 각각 나뉘어진 부분에 대한 다른 처리를 할 수도 있습니다.

이 segmentation에 대한 학습입니다.

다음에 학습할 부분인 Image Detection과 같이 활용하여 사용합니다.

 

<참조 : https://www.tensorflow.org/tutorials/images/segmentation>

In [1]:

Image Segmentation¶

In [2]:

!pip install git+https://github.com/tensorflow/examples.git

Collecting git+https://github.com/tensorflow/examples.git
  Cloning https://github.com/tensorflow/examples.git to /tmp/pip-req-build-ri0pq71z
  Running command git clone -q https://github.com/tensorflow/examples.git /tmp/pip-req-build-ri0pq71z
Requirement already satisfied: absl-py in /usr/local/lib/python3.7/dist-packages (from tensorflow-examples===b31b32ffd4c501af6c00fa4a7e073643b8ad6e6f-) (0.12.0)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from tensorflow-examples===b31b32ffd4c501af6c00fa4a7e073643b8ad6e6f-) (1.15.0)
Building wheels for collected packages: tensorflow-examples
  Building wheel for tensorflow-examples (setup.py) ... done
  Created wheel for tensorflow-examples: filename=tensorflow_examples-b31b32ffd4c501af6c00fa4a7e073643b8ad6e6f_-py3-none-any.whl size=270922 sha256=27cdf018b427a15a57a01ada09899ddc61194f7a0abfd4c3a00be297bf46e67d
  Stored in directory: /tmp/pip-ephem-wheel-cache-mfkw9xlf/wheels/eb/19/50/2a4363c831fa12b400af86325a6f26ade5d2cdc5b406d552ca
  WARNING: Built wheel for tensorflow-examples is invalid: Metadata 1.2 mandates PEP 440 version, but 'b31b32ffd4c501af6c00fa4a7e073643b8ad6e6f-' is not
Failed to build tensorflow-examples
Installing collected packages: tensorflow-examples
    Running setup.py install for tensorflow-examples ... done
  DEPRECATION: tensorflow-examples was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
Successfully installed tensorflow-examples-b31b32ffd4c501af6c00fa4a7e073643b8ad6e6f-

In [2]:

01. Import¶

In [3]:

import tensorflow as tf

from tensorflow_examples.models.pix2pix import pix2pix

import tensorflow_datasets as tfds

from IPython.display import clear_output
import matplotlib.pyplot as plt

In [3]:

02. Download the Oxford-IIIT Pets dataset¶

In [4]:

dataset, info = tfds.load('oxford_iiit_pet:3.*.*', with_info=True)

Downloading and preparing dataset oxford_iiit_pet/3.2.0 (download: 773.52 MiB, generated: 774.69 MiB, total: 1.51 GiB) to /root/tensorflow_datasets/oxford_iiit_pet/3.2.0...

Shuffling and writing examples to /root/tensorflow_datasets/oxford_iiit_pet/3.2.0.incompleteFL7WRY/oxford_iiit_pet-train.tfrecord

Shuffling and writing examples to /root/tensorflow_datasets/oxford_iiit_pet/3.2.0.incompleteFL7WRY/oxford_iiit_pet-test.tfrecord

Dataset oxford_iiit_pet downloaded and prepared to /root/tensorflow_datasets/oxford_iiit_pet/3.2.0. Subsequent calls will reuse this data.

In [5]:

def normalize(input_image, input_mask):
    input_image = tf.cast(input_image, tf.float32) / 255.0
    input_mask -= 1
    return input_image, input_mask

In [6]:

@tf.function
def load_image_train(datapoint):
    input_image = tf.image.resize(datapoint['image'], (128, 128))
    input_mask = tf.image.resize(datapoint['segmentation_mask'], (128, 128))

    if tf.random.uniform(()) > 0.5:
        input_image = tf.image.flip_left_right(input_image)
        input_mask = tf.image.flip_left_right(input_mask)

    input_image, input_mask = normalize(input_image, input_mask)

    return input_image, input_mask

In [7]:

def load_image_test(datapoint):
    input_image = tf.image.resize(datapoint['image'], (128, 128))
    input_mask = tf.image.resize(datapoint['segmentation_mask'], (128, 128))

    input_image, input_mask = normalize(input_image, input_mask)

    return input_image, input_mask

In [7]:

• The dataset already contains the required splits of test and train.

In [8]:

TRAIN_LENGTH = info.splits['train'].num_examples
BATCH_SIZE = 64
BUFFER_SIZE = 1000
STEPS_PER_EPOCH = TRAIN_LENGTH // BATCH_SIZE

In [9]:

train = dataset['train'].map(load_image_train, num_parallel_calls=tf.data.AUTOTUNE)
test = dataset['test'].map(load_image_test)

In [10]:

train_dataset = train.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()
train_dataset = train_dataset.prefetch(buffer_size=tf.data.AUTOTUNE)
test_dataset = test.batch(BATCH_SIZE)

In [11]:

def display(display_list):
    plt.figure(figsize=(15, 15))

    title = ['Input Image', 'True Mask', 'Predicted Mask']

    for i in range(len(display_list)):
        plt.subplot(1, len(display_list), i+1)
        plt.title(title[i])
        plt.imshow(tf.keras.preprocessing.image.array_to_img(display_list[i]))
        plt.axis('off')

    plt.show()

In [12]:

for image, mask in train.take(1):
    sample_image, sample_mask = image, mask
    display([sample_image, sample_mask])

In [12]:

03. Define the model¶

• The model : a modified U-Net
• U-Net consists of
  • encoder(downsampler) : a pretrained MobileNetV2 model
    • tf.keras.applications
  • decoder(upsampler)

In [13]:

OUTPUT_CHANNELS = 3

In [14]:

base_model = tf.keras.applications.MobileNetV2(input_shape=[128, 128, 3], include_top=False)

# Use the activations of these layers
layer_names = [
        'block_1_expand_relu',      # 64 x 64
        'block_3_expand_relu',      # 32 x 32
        'block_6_expand_relu',      # 16 x 16
        'block_13_expand_relu',     # 8 x 8
        'block_16_project',         # 4 x 4
]
base_model_outputs = [base_model.get_layer(name).output for name in layer_names]

# Create the feature extraction model
down_stack = tf.keras.Model(inputs=base_model.input, outputs=base_model_outputs)

down_stack.trainable = False

Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/mobilenet_v2/mobilenet_v2_weights_tf_dim_ordering_tf_kernels_1.0_128_no_top.h5
9412608/9406464 [==============================] - 0s 0us/step

In [14]:

• decoder/upsampler

In [15]:

up_stack = [
        pix2pix.upsample(512, 3),       # 4 x 4 -> 8 x 8
        pix2pix.upsample(256, 3),       # 8 x 8 -> 16 x 16
        pix2pix.upsample(128, 3),       # 16 x 16 -> 32 x 32
        pix2pix.upsample(64, 3),        # 32 x 32 -> 64 x 64
]

In [16]:

def unet_model(output_channels):
    inputs = tf.keras.layers.Input(shape=[128, 128, 3])

    # Downsampling through the model
    skips = down_stack(inputs)
    x = skips[-1]
    skips = reversed(skips[:-1])

    # Upsampling and establishing the skip connections
    for up, skip in zip(up_stack, skips):
        x = up(x)
        concat = tf.keras.layers.Concatenate()
        x = concat([x, skip])

    # This is the last layer of the model
    last = tf.keras.layers.Conv2DTranspose(
        output_channels, 3, strides=2,
        padding='same'
    )               # 64 x 64 -> 128 x 128

    x = last(x)

    return tf.keras.Model(inputs=inputs, outputs=x)

In [16]:

04. Train the model¶

In [17]:

model = unet_model(OUTPUT_CHANNELS)

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

In [18]:

tf.keras.utils.plot_model(model, show_shapes=True)

Out[18]:

In [18]:

In [19]:

def create_mask(pred_mask):
    pred_mask = tf.argmax(pred_mask, axis=-1)
    pred_mask = pred_mask[..., tf.newaxis]
    return pred_mask[0]

In [20]:

def show_predictions(dataset=None, num=1):
    if dataset:
        for image, mask in dataset.take(num):
            pred_mask = model.predict(image)
            display([image[0], mask[0], create_mask(pred_mask)])
    else:
        display([sample_image, sample_mask,
                 create_mask(model.predict(sample_image[tf.newaxis, ...]))])

In [21]:

show_predictions()

In [21]:

• how the model improves while it is training

In [22]:

class DisplayCallback(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs=None):
        clear_output(wait=True)
        show_predictions()
        print('\nSample Prediction after epoch {}\n'.format(epoch+1))

In [23]:

EPOCHS = 20
VAL_SUBSPLITS = 5
VALIDATION_STEPS = info.splits['test'].num_examples // BATCH_SIZE // VAL_SUBSPLITS

model_history = model.fit(train_dataset, epochs=EPOCHS,
                          steps_per_epoch=STEPS_PER_EPOCH,
                          validation_steps=VALIDATION_STEPS,
                          validation_data=test_dataset,
                          callbacks=[DisplayCallback()])

Sample Prediction after epoch 20

In [23]:

In [24]:

loss = model_history.history['loss']
val_loss = model_history.history['val_loss']

plt.figure()
plt.plot(model_history.epoch, loss, 'r', label='Training loss')
plt.plot(model_history.epoch, val_loss, 'bo', label='Validation loss')
plt.title('Training and Validation Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss Value')
plt.ylim([0, 1])
plt.legend()
plt.show()

In [ ]:

05. Make predictions¶

In [25]:

show_predictions(test_dataset, 3)

In [25]:

In [25]:

In [ ]:

06. Optional : Imbalanced classes and class weights¶

In [26]:

try:
    model_history = model.fit(train_dataset, epochs=EPOCHS,
                              steps_per_epoch=STEPS_PER_EPOCH,
                              class_weight={0: 2.0, 1: 2.0, 2: 1.0})
    assert False
except Exception as e:
    print(f'{type(e).__name__}: {e}')

ValueError: `class_weight` not supported for 3+ dimensional targets.

• So in this case you need to implement the weighting yourself.
• In addition to (data, label) pairs, Model.fit also accepts (data, label, sample_weight) triples.
• Model.fit propagates the sample_weight to the losses and metrics, which also accept a sample_weight argument. The sample weight is multiplied by the sample's value before the reduction step. For example:

In [27]:

label = [0, 0]
prediction = [[-3., 0], [-3, 0]]
sample_weight = [1, 10]

loss = tf.losses.SparseCategoricalCrossentropy(from_logits=True,
                                        reduction=tf.losses.Reduction.NONE)
loss(label, prediction, sample_weight).numpy()

Out[27]:

array([ 3.0485873, 30.485874 ], dtype=float32)

In [28]:

def add_sample_weights(image, label):
    # The weights for each class, with the constraint that:
    #     sum(class_weights) == 1.0
    class_weights = tf.constant([2.0, 2.0, 1.0])
    class_weights = class_weights / tf.reduce_sum(class_weights)

    # Create an image of 'sample_weights' by using the label at each pixel as an
    # index into the 'class weights'.
    sample_weights = tf.gather(class_weights, indices=tf.cast(label, tf.int32))

    return image, label, sample_weights

In [29]:

train_dataset.map(add_sample_weights).element_spec

WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/array_ops.py:5049: calling gather (from tensorflow.python.ops.array_ops) with validate_indices is deprecated and will be removed in a future version.
Instructions for updating:
The `validate_indices` argument has no effect. Indices are always validated on CPU and never validated on GPU.

WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/array_ops.py:5049: calling gather (from tensorflow.python.ops.array_ops) with validate_indices is deprecated and will be removed in a future version.
Instructions for updating:
The `validate_indices` argument has no effect. Indices are always validated on CPU and never validated on GPU.

Out[29]:

(TensorSpec(shape=(None, 128, 128, 3), dtype=tf.float32, name=None),
 TensorSpec(shape=(None, 128, 128, 1), dtype=tf.float32, name=None),
 TensorSpec(shape=(None, 128, 128, 1), dtype=tf.float32, name=None))

In [ ]:

• Now you can train a model on this weighted dataset.

In [30]:

weighted_model = unet_model(OUTPUT_CHANNELS)

weighted_model.compile(
    optimizer='adam',
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=['accuracy']
)

In [31]:

weighted_model_history = weighted_model.fit(
    train_dataset.map(add_sample_weights),
    epochs=10,
    steps_per_epoch=10
)

Epoch 1/10
10/10 [==============================] - 4s 127ms/step - loss: 0.2580 - accuracy: 0.6676
Epoch 2/10
10/10 [==============================] - 1s 126ms/step - loss: 0.1351 - accuracy: 0.8446
Epoch 3/10
10/10 [==============================] - 1s 128ms/step - loss: 0.0991 - accuracy: 0.8759
Epoch 4/10
10/10 [==============================] - 1s 129ms/step - loss: 0.0783 - accuracy: 0.8838
Epoch 5/10
10/10 [==============================] - 1s 127ms/step - loss: 0.0660 - accuracy: 0.9009
Epoch 6/10
10/10 [==============================] - 1s 121ms/step - loss: 0.0593 - accuracy: 0.9091
Epoch 7/10
10/10 [==============================] - 1s 127ms/step - loss: 0.0540 - accuracy: 0.9140
Epoch 8/10
10/10 [==============================] - 1s 130ms/step - loss: 0.0494 - accuracy: 0.9212
Epoch 9/10
10/10 [==============================] - 1s 128ms/step - loss: 0.0481 - accuracy: 0.9235
Epoch 10/10
10/10 [==============================] - 1s 132ms/step - loss: 0.0464 - accuracy: 0.9259

In [33]:

loss = weighted_model_history.history['loss']
# val_loss = weighted_model_history.history['val_loss']

plt.figure()
plt.plot(weighted_model_history.epoch, loss, 'r', label='Training loss')
# plt.plot(weighted_model_history.epoch, val_loss, 'bo', label='Validation loss')
plt.title('Weighted_Model Training  Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss Value')
plt.ylim([0, 1])
plt.legend()
plt.show()

In [ ]: