Random seeds fixed in TensorFlow

Roughly speaking

background

When implementing machine learning code in TensorFlow, we often use pseudo-random numbers (hereinafter simply "random numbers") for weight initialization and so on. As a result, the result changes each time the code is re-executed, and it is not possible to distinguish between the change in the result due to changes in data and parameters and the change in the result due to random numbers. Random numbers are generated based on a value called a "seed", which, unless otherwise specified, uses a somewhat variable default value (often a time or process ID). TensorFlow also provides a way to fix the random seed, but I was a little worried about its behavior, so I tried to organize it.

In addition to TensorFlow, we will also discuss fixing random number seeds in the random module and NumPy package.

version

The content of this article is based on the following versions:

random module

In the Python standard random module, you can set the random seed with the random.seed function. The experimental code is as follows.

seed_random.py


import random
random.seed(0)
print("1:", random.random())
print("2:", random.random())
print("3:", random.random())
random.seed(0)
print("4:", random.random())
print("5:", random.random())
print("6:", random.random())

The result when executed 3 times is as follows. The same value is output each time a random number seed is set or executed.

$ python3 seed_random.py
1: 0.8444218515250481
2: 0.7579544029403025
3: 0.420571580830845
4: 0.8444218515250481
5: 0.7579544029403025
6: 0.420571580830845
$ python3 seed_random.py
1: 0.8444218515250481
2: 0.7579544029403025
3: 0.420571580830845
4: 0.8444218515250481
5: 0.7579544029403025
6: 0.420571580830845
$ python3 seed_random.py
1: 0.8444218515250481
2: 0.7579544029403025
3: 0.420571580830845
4: 0.8444218515250481
5: 0.7579544029403025
6: 0.420571580830845

NumPy package

In the NumPy package, you can set the random seed with the numpy.random.seed function. The experimental code is as follows.

seed_numpy.py


import numpy as np
np.random.seed(0)
print("1:", np.random.rand(3))
print("2:", np.random.rand(3))
print("3:", np.random.rand(3))
np.random.seed(0)
print("4:", np.random.rand(3))
print("5:", np.random.rand(3))
print("6:", np.random.rand(3))

The result when executed 3 times is as follows. The same value is output each time a random number seed is set or executed.

$ python3 seed_numpy.py
1: [ 0.5488135   0.71518937  0.60276338]
2: [ 0.54488318  0.4236548   0.64589411]
3: [ 0.43758721  0.891773    0.96366276]
4: [ 0.5488135   0.71518937  0.60276338]
5: [ 0.54488318  0.4236548   0.64589411]
6: [ 0.43758721  0.891773    0.96366276]
$ python3 seed_numpy.py
1: [ 0.5488135   0.71518937  0.60276338]
2: [ 0.54488318  0.4236548   0.64589411]
3: [ 0.43758721  0.891773    0.96366276]
4: [ 0.5488135   0.71518937  0.60276338]
5: [ 0.54488318  0.4236548   0.64589411]
6: [ 0.43758721  0.891773    0.96366276]
$ python3 seed_numpy.py
1: [ 0.5488135   0.71518937  0.60276338]
2: [ 0.54488318  0.4236548   0.64589411]
3: [ 0.43758721  0.891773    0.96366276]
4: [ 0.5488135   0.71518937  0.60276338]
5: [ 0.54488318  0.4236548   0.64589411]
6: [ 0.43758721  0.891773    0.96366276]

TensorFlow package

In the TensorFlow package, you can set the random seed with the tensorflow.set_random_seed function or the seed argument of the random number operator. There are two types of random number seeds in TensorFlow called "graph level" and "operation level". You also need to be careful about the order of the settings, and you need to set the random seeds "after creating the graph" and "before generating the operator".

Reference: Constants, Sequences, and Random Values --tf.set_random_seed (seed)

An example that works as expected

The experimental code is as follows.

seed_tensorflow1.py


import tensorflow as tf
with tf.Graph().as_default():
    tf.set_random_seed(0)
    rand_op1 = tf.random_normal([3])
    rand_op2 = tf.random_normal([3], seed=1)
    print("session1")
    with tf.Session() as sess1:
        print("op1-1:", sess1.run(rand_op1))
        print("op1-2:", sess1.run(rand_op1))
        print("op2-1:", sess1.run(rand_op2))
        print("op2-2:", sess1.run(rand_op2))
    print("session2")
    with tf.Session() as sess2:
        print("op1-1:", sess2.run(rand_op1))
        print("op1-2:", sess2.run(rand_op1))
        print("op2-1:", sess2.run(rand_op2))
        print("op2-2:", sess2.run(rand_op2))

The result when executed 3 times is as follows.

$ python3 seed_tensorflow1.py
session1
op1-1: [-1.40955448 -0.53668278 -0.56523788]
op1-2: [-1.07107699  0.4139019   2.29180121]
op2-1: [ 0.68034536  0.8777824  -0.64773595]
op2-2: [-1.21607268  0.95542693 -0.16866584]
session2
op1-1: [-1.40955448 -0.53668278 -0.56523788]
op1-2: [-1.07107699  0.4139019   2.29180121]
op2-1: [ 0.68034536  0.8777824  -0.64773595]
op2-2: [-1.21607268  0.95542693 -0.16866584]
$ python3 seed_tensorflow1.py
session1
op1-1: [-1.40955448 -0.53668278 -0.56523788]
op1-2: [-1.07107699  0.4139019   2.29180121]
op2-1: [ 0.68034536  0.8777824  -0.64773595]
op2-2: [-1.21607268  0.95542693 -0.16866584]
session2
op1-1: [-1.40955448 -0.53668278 -0.56523788]
op1-2: [-1.07107699  0.4139019   2.29180121]
op2-1: [ 0.68034536  0.8777824  -0.64773595]
op2-2: [-1.21607268  0.95542693 -0.16866584]
$ python3 seed_tensorflow1.py
session1
op1-1: [-1.40955448 -0.53668278 -0.56523788]
op1-2: [-1.07107699  0.4139019   2.29180121]
op2-1: [ 0.68034536  0.8777824  -0.64773595]
op2-2: [-1.21607268  0.95542693 -0.16866584]
session2
op1-1: [-1.40955448 -0.53668278 -0.56523788]
op1-2: [-1.07107699  0.4139019   2.29180121]
op2-1: [ 0.68034536  0.8777824  -0.64773595]
op2-2: [-1.21607268  0.95542693 -0.16866584]

Doesn't work as expected Example 1: Set a random seed before creating the graph

Here is an example of setting a random seed before creating a graph. The experimental code is as follows.

seed_tensorflow2.py


import tensorflow as tf
tf.set_random_seed(0) #Set random seeds before creating the graph
with tf.Graph().as_default():
    rand_op1 = tf.random_normal([3])
    rand_op2 = tf.random_normal([3], seed=1)
    with tf.Session() as sess1:
        print("op1-1:", sess1.run(rand_op1))
        print("op1-2:", sess1.run(rand_op1))
        print("op2-1:", sess1.run(rand_op2))
        print("op2-2:", sess1.run(rand_op2))

The result when executed 3 times is as follows. Rand_op2, which sets the random number seed at the operation level, outputs the same value three times, but rand_op1 outputs a different value three times.

$ python3 seed_tensorflow2.py
op1-1: [ 0.22292495  0.67142487  0.80927771]
op1-2: [-1.11188841  0.57819426  0.19088539]
op2-1: [-0.81131822  1.48459876  0.06532937]
op2-2: [ 0.55171245 -0.13107552 -0.04481386]
$ python3 seed_tensorflow2.py
op1-1: [-0.96930581  1.48854125  0.52752507]
op1-2: [ 0.91646689  0.85830265 -0.18069462]
op2-1: [-0.81131822  1.48459876  0.06532937]
op2-2: [ 0.55171245 -0.13107552 -0.04481386]
$ python3 seed_tensorflow2.py
op1-1: [ 0.79890805  1.35702407 -0.12329593]
op1-2: [ 1.69212222  0.22590902 -0.73435217]
op2-1: [-0.81131822  1.48459876  0.06532937]
op2-2: [ 0.55171245 -0.13107552 -0.04481386]

Doesn't work as expected Example 2: Set random seed after generating random operator

The following example shows how to set the random seed after generating the random operator. The experimental code is as follows.

seed_tensorflow3.py


import tensorflow as tf
with tf.Graph().as_default():
    rand_op1 = tf.random_normal([3])
    rand_op2 = tf.random_normal([3], seed=1)
    tf.set_random_seed(0) #Set random number seed after generating random number operator
    with tf.Session() as sess:
        print("op1-1:", sess.run(rand_op1))
        print("op1-2:", sess.run(rand_op1))
        print("op2-1:", sess.run(rand_op2))
        print("op2-2:", sess.run(rand_op2))

The result when executed 3 times is as follows. Similar to the above example, rand_op2, which sets the random number seed at the operation level, outputs the same value three times, but rand_op1 outputs a different value three times.

$ python3 seed_tensorflow3.py
op1-1: [ 0.03802272  1.69988739 -0.59952497]
op1-2: [ 0.62614048  0.07537607 -1.19501412]
op2-1: [-0.81131822  1.48459876  0.06532937]
op2-2: [ 0.55171245 -0.13107552 -0.04481386]
$ python3 seed_tensorflow3.py
op1-1: [ 1.02020776  1.70896292  0.45571503]
op1-2: [-0.46230376 -1.22950804  0.51038951]
op2-1: [-0.81131822  1.48459876  0.06532937]
op2-2: [ 0.55171245 -0.13107552 -0.04481386]
$ python3 seed_tensorflow3.py
op1-1: [ 1.31134415 -1.12688231  0.1805287 ]
op1-2: [-0.57391566  0.94440365 -1.07441545]
op2-1: [-0.81131822  1.48459876  0.06532937]
op2-2: [ 0.55171245 -0.13107552 -0.04481386]

Other

It is unconfirmed information, but it seems that it will not be the same value even if you set a random number seed when running on GPU.

Reference: Mention that GPU reductions are nondeterministic in docs · Issue # 2732 · tensorflow / tensorflow

Acknowledgments

We received very useful information from Mr. Hirai, a fellow study group on TensorFlow. Thank you here.

Recommended Posts

Random seeds fixed in TensorFlow
Random number seed fixed in TensorFlow and Numpy A little different
Random walk in Python
Image normalization in TensorFlow
[TensorFlow 2.x (tf.keras)] Fixed random number seed to improve reproducibility
Balanced Random Forest in python
Clipping and normalization in TensorFlow
Use Random Forest in Python
Weighted random choice in python
Ensure reproducibility with tf.keras in Tensorflow 2.3
Testing with random numbers in Python
Try Embedding Visualization added in TensorFlow 0.12
Install tensorflow in Docker (LINUX) (memo)
Fixed Apscheduler running twice in Flask
Summary of various operations in Tensorflow
install tensorflow in anaconda + python3.5 environment
How to run TensorFlow 1.0 code in 2.0
Random seed research in machine learning
Create a random string in Python