Question:
How to connect LSTM layers in Keras, RepeatVector or return_sequence=True? I'm trying to develop an Encoder model in keras for timeseries. The shape of data is (5039, 28, 1), meaning that my seq_len is 28 and I have one feature. For the first layer of the encoder, I'm using 112 hunits, second layer will have 56 and to be able to get back to the input shape for decoder, I had to add 3rd layer with 28 hunits (this autoencoder is supposed to reconstruct its input). But I don't know what is the correct approach to connect the LSTM layers together. AFAIK, I can either add RepeatVector or return_seq=True. You can see both of my models in the following code. I wonder what will be the difference and which approach is the correct one? First model using return_sequence=True:
```
inputEncoder = Input(shape=(28, 1))
firstEncLayer = LSTM(112, return_sequences=True)(inputEncoder)
snd = LSTM(56, return_sequences=True)(firstEncLayer)
outEncoder = LSTM(28)(snd)

context = RepeatVector(1)(outEncoder)
context_reshaped = Reshape((28,1))(context)

encoder_model = Model(inputEncoder, outEncoder)
firstDecoder = LSTM(112, return_sequences=True)(context_reshaped)
outDecoder = LSTM(1, return_sequences=True)(firstDecoder)

autoencoder = Model(inputEncoder, outDecoder)
```
Second model with RepeatVector:
```
inputEncoder = Input(shape=(28, 1))
firstEncLayer = LSTM(112)(inputEncoder)
firstEncLayer = RepeatVector(1)(firstEncLayer)
snd = LSTM(56)(firstEncLayer)
snd = RepeatVector(1)(snd)
outEncoder = LSTM(28)(snd)
encoder_model = Model(inputEncoder, outEncoder)

context = RepeatVector(1)(outEncoder)
context_reshaped = Reshape((28, 1))(context)

firstDecoder = LSTM(112)(context_reshaped)
firstDecoder = RepeatVector(1)(firstDecoder)
sndDecoder = LSTM(28)(firstDecoder)

outDecoder = RepeatVector(1)(sndDecoder)
outDecoder = Reshape((28, 1))(outDecoder)

autoencoder = Model(inputEncoder, outDecoder)
```

Response: You will probably have to see for yourself which one is better because it depends on the problem you're solving. However, I'm giving you the difference between the two approaches. Essentially, return_sequences=True returns all the outputs the encoder observed in the past, while RepeatVector repeats the very last output of the encoder.


Question:
How to calculate the Cosine similarity between two tensors? I have two normalized tensors and I need to calculate the cosine similarity between these tensors. How do I do it with TensorFlow?
```
cosine(normalize_a,normalize_b)

    a = tf.placeholder(tf.float32, shape=[None], name="input_placeholder_a")
    b = tf.placeholder(tf.float32, shape=[None], name="input_placeholder_b")
    normalize_a = tf.nn.l2_normalize(a,0)
    normalize_b = tf.nn.l2_normalize(b,0)
```

Response:
This will do the job:
```
a = tf.placeholder(tf.float32, shape=[None], name="input_placeholder_a")
b = tf.placeholder(tf.float32, shape=[None], name="input_placeholder_b")
normalize_a = tf.nn.l2_normalize(a,0)
normalize_b = tf.nn.l2_normalize(b,0)
cos_similarity=tf.reduce_sum(tf.multiply(normalize_a,normalize_b))
sess=tf.Session()
cos_sim=sess.run(cos_similarity,feed_dict={a:[1,2,3],b:[2,4,6]})
```
This prints 0.99999988


Question:
ValueError: Trying to share variable rnn/multi_rnn_cell/cell_0/basic_lstm_cell/kernel This it the code:
```
X = tf.placeholder(tf.float32, [batch_size, seq_len_1, 1], name='X')
labels = tf.placeholder(tf.float32, [None, alpha_size], name='labels')

rnn_cell = tf.contrib.rnn.BasicLSTMCell(512)
m_rnn_cell = tf.contrib.rnn.MultiRNNCell([rnn_cell] * 3, state_is_tuple=True)
pre_prediction, state = tf.nn.dynamic_rnn(m_rnn_cell, X, dtype=tf.float32)
```
This is full error: ValueError: Trying to share variable rnn/multi_rnn_cell/cell_0/basic_lstm_cell/kernel, but specified shape (1024, 2048) and found shape (513, 2048). I'm using a GPU version of tensorflow.

Response:
I encountered a similar problem when I upgraded to v1.2 (tensorflow-gpu). Instead of using [rnn_cell]*3, I created 3 rnn_cells (stacked_rnn) by a loop (so that they don't share variables) and fed MultiRNNCell with stacked_rnn and the problem goes away. I'm not sure it is the right way to do it.
```
stacked_rnn = []
for iiLyr in range(3):
    stacked_rnn.append(tf.nn.rnn_cell.LSTMCell(num_units=512, state_is_tuple=True))
MultiLyr_cell = tf.nn.rnn_cell.MultiRNNCell(cells=stacked_rnn, state_is_tuple=True)
```


Question:
TensorFlow: “Attempting to use uninitialized value” in variable initialization Here's my code.
```
import tensorflow as tf

a=tf.Variable(tf.constant([0,1,2],dtype=tf.int32))
b=tf.Variable(tf.constant([1,1,1],dtype=tf.int32))
recall=tf.metrics.recall(b,a)

init=tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    rec=sess.run(recall)
    print(rec)
```
I tried to test tf.metrics.precision and got the following error message.
```
FailedPreconditionError (see above for traceback): Attempting to use uninitialized value recall/true_positives/count
     [[Node: recall/true_positives/count/read = Identity[T=DT_FLOAT, _class=["loc:@recall/true_positives/count"], _device="/job:localhost/replica:0/task:0/gpu:0"](recall/true_positives/count)]]
     [[Node: recall/value/_15 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_73_recall/value", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
```

Response:
You also need to initialise the local variables hidden in the tf.metrics.recallmethod. For example, this piece of code would work:
```
init_g = tf.global_variables_initializer()
init_l = tf.local_variables_initializer()
with tf.Session() as sess:
    sess.run(init_g)
    sess.run(init_l)
```


Question: Why use tensorflow gfile? (for file I/O) Tensorflow code uses methods for file I/O that are different than python builtin methods. According to the source code, it is useful as "File I/O wrappers without thread locking" I am not sure on what occasions it is useful and when it shouldn't be used. Any idea? Thank you

Response: This comment: File I/O wrappers without thread locking ...is a particularly unhelpful description for TensorFlow's tf.gfile module! The main roles of the tf.gfile module are: To provide an API that is close to Python's file objects, and To provide an implementation based on TensorFlow's C++ FileSystem API. The C++ FileSystem API supports multiple file system implementations, including local files, Google Cloud Storage (using a gs:// prefix), and HDFS (using an hdfs:// prefix). TensorFlow exports these as tf.gfile so that you can uses these implementations for saving and loading checkpoints, writing TensorBoard logs, and accessing training data (among other uses). However, if all of your files are local, you can use the regular Python file API without any problem.


Question:
Tensorflow, multi label accuracy calculation I am working on a multi label problem and i am trying to determine the accuracy of my model. My model:
```py
NUM_CLASSES = 361

x = tf.placeholder(tf.float32, [None, IMAGE_PIXELS])
y_ = tf.placeholder(tf.float32, [None, NUM_CLASSES])

# create the network
pred = conv_net( x )

# loss
cost = tf.reduce_mean( tf.nn.sigmoid_cross_entropy_with_logits( pred, y_) )

# train step
train_step   = tf.train.AdamOptimizer().minimize( cost )
```
i want to calculate the accuracy in two different ways - % of all labels that are predicted correctly - % of images where ALL labels are predicted correctly unfortunately i am only able to calculate the % of all labels that are predicted correctly. I thought this code would calculate % of images where ALL labels are predicted correctly
```py
correct_prediction = tf.equal( tf.round( pred ), tf.round( y_ ) )

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
```
and this code % of all labels that are predicted correctly
```py
pred_reshape = tf.reshape( pred, [ BATCH_SIZE * NUM_CLASSES, 1 ] )
y_reshape = tf.reshape( y_, [ BATCH_SIZE * NUM_CLASSES, 1 ] )

correct_prediction_all = tf.equal( tf.round( pred_reshape ), tf.round( y_reshape ) )

accuracy_all = tf.reduce_mean( tf.cast(correct_prediction_all, tf.float32 ) )
```
somehow the coherency of the labels belonging to one image is lost and i am not sure why.

Response:
I believe the bug in your code is in: correct_prediction = tf.equal( tf.round( pred ), tf.round( y_ ) ). pred should be unscaled logits (i.e. without a final sigmoid). Here you want to compare the output of sigmoid(pred) and y_ (both in the interval [0, 1]) so you have to write:
```py
correct_prediction = tf.equal(tf.round(tf.nn.sigmoid(pred)), tf.round(y_))
```
Then to compute: Mean accuracy over all labels:
```py
accuracy1 = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
```
Accuracy where all labels need to be correct:
```py
all_labels_true = tf.reduce_min(tf.cast(correct_prediction), tf.float32), 1)
accuracy2 = tf.reduce_mean(all_labels_true)
```


Question:
can anyone give a tiny example to explain the params of tf.random.categorical? tensorflow's site gives this example
```
tf.random.categorical(tf.log([[10., 10.]]), 5)
```
produces a tensor that "has shape [1, 5], where each value is either 0 or 1 with equal probability" I have already known, the basic demo, the meaning of tf.log([[10., 10.]]). what I want to know is what does [batch_size, num_classes] do, can anyone give a tiny example to explain the params?

Response:
As you note, tf.random.categorical takes two parameters: logits, a 2D float tensor with shape [batch_size, num_classes] num_samples, an integer scalar. The output is a 2D integer tensor with shape [batch_size, num_samples]. Each "row" of the logits tensor (logits[0, :], logits[1, :], ...) represents the event probabilities of a different categorical distribution. The function does not expect actual probability values, though, but unnormalized log-probabilities; so the actual probabilities would be softmax(logits[0, :]), softmax(logits[1, :]), etc. The benefit of this is that you can give basically any real values as input (e.g. the output of a neural network) and they will be valid. Also, it's trivial to use specific probability values, or proportions, using logarithms. For example, both [log(0.1), log(0.3), log(0.6)] and [log(1), log(3), log(6)] represent the same probability, where the second class is three times as likely as the first one but only half as likely as the third one. For each row of (unnormalized log-)probabilities, you get num_samples samples from the distribution. Each sample is an integer between 0 and num_classes - 1, drawn according to the given probabilities. So, the result is the 2D tensor with shape [batch_size, num_samples] with the sampled integers for each distribution. EDIT: A small example of the function.
```
import tensorflow as tf

with tf.Graph().as_default(), tf.Session() as sess:
    tf.random.set_random_seed(123)
    logits = tf.log([[1., 1., 1., 1.],
                     [0., 1., 2., 3.]])
    num_samples = 30
    cat = tf.random.categorical(logits, num_samples)
    print(sess.run(cat))
    # [[3 3 1 1 0 3 3 0 2 3 1 3 3 3 1 1 0 2 2 0 3 1 3 0 1 1 0 1 3 3]
    #  [2 2 3 3 2 3 3 3 2 2 3 3 2 2 2 1 3 3 3 2 3 2 2 1 3 3 3 3 3 2]]
```
In this case, the result is an array with two rows and 30 columns. The values in the first row are sampled from a categorical distribution where every class ([0, 1, 2, 3]) has the same probability. In the second row, the class 3 is the most likely one, and class 0 has just no probability of being sampled.


Question:
Custom weighted loss function in Keras for weighing each element I'm trying to create a simple weighted loss function. Say, I have input dimensions 100 * 5, and output dimensions also 100 * 5. I also have a weight matrix of the same dimension. Something like the following:
```
import numpy as np
train_X = np.random.randn(100, 5)
train_Y = np.random.randn(100, 5)*0.01 + train_X

weights = np.random.randn(*train_X.shape)
```
Defining the custom loss function
```
def custom_loss_1(y_true, y_pred):
    return K.mean(K.abs(y_true-y_pred)*weights)
```
Defining the model
```
from keras.layers import Dense, Input
from keras import Model
import keras.backend as K

input_layer = Input(shape=(5,))
out = Dense(5)(input_layer)
model = Model(input_layer, out)
```
Testing with existing metrics works fine
```
model.compile('adam','mean_absolute_error')
model.fit(train_X, train_Y, epochs=1)
```
Testing with our custom loss function doesn't work
```
model.compile('adam',custom_loss_1)
model.fit(train_X, train_Y, epochs=10)
```
It gives the following stack trace:
```
InvalidArgumentError (see above for traceback): Incompatible shapes: [32,5] vs. [100,5]
 [[Node: loss_9/dense_8_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](loss_9/dense_8_loss/Abs, loss_9/dense_8_loss/mul/y)]]
```
Where is the number 32 coming from? Testing a loss function with weights as Keras tensors
```
def custom_loss_2(y_true, y_pred):
    return K.mean(K.abs(y_true-y_pred)*K.ones_like(y_true))
```
This function seems to do the work. So, probably suggests that a Keras tensor as a weight matrix would work. So, I created another version of the loss function. Loss function try 3
```
from functools import partial

def custom_loss_3(y_true, y_pred, weights):
    return K.mean(K.abs(y_true-y_pred)*K.variable(weights, dtype=y_true.dtype))

cl3 = partial(custom_loss_3, weights=weights)
```
Fitting data using cl3 gives the same error as above.
```
InvalidArgumentError (see above for traceback): Incompatible shapes: [32,5] vs. [100,5]
     [[Node: loss_11/dense_8_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](loss_11/dense_8_loss/Abs, loss_11/dense_8_loss/Variable/read)]]
```
I wonder what I'm missing! I could have used the notion of sample_weight in Keras; but then I'd have to reshape my inputs to a 3d vector. I thought that this custom loss function should really have been trivial.

Response:
In model.fit the batch size is 32 by default, that's where this number is coming from. Here's what's happening: In custom_loss_1 the tensor K.abs(y_true-y_pred) has shape (batch_size=32, 5), while the numpy array weights has shape (100, 5). This is an invalid multiplication, since the dimensions don't agree and broadcasting can't be applied. In custom_loss_2 this problem doesn't exist because you're multiplying 2 tensors with the same shape (batch_size=32, 5). In custom_loss_3 the problem is the same as in custom_loss_1, because converting weights into a Keras variable doesn't change their shape. UPDATE: It seems you want to give a different weight to each element in each training sample, so the weights array should have shape (100, 5) indeed. In this case, I would input your weights' array into your model and then use this tensor within the loss function:
```
import numpy as np
from keras.layers import Dense, Input
from keras import Model
import keras.backend as K
from functools import partial

def custom_loss_4(y_true, y_pred, weights):
    return K.mean(K.abs(y_true - y_pred) * weights)

train_X = np.random.randn(100, 5)
train_Y = np.random.randn(100, 5) * 0.01 + train_X
weights = np.random.randn(*train_X.shape)

input_layer = Input(shape=(5,))
weights_tensor = Input(shape=(5,))
out = Dense(5)(input_layer)
cl4 = partial(custom_loss_4, weights=weights_tensor)
model = Model([input_layer, weights_tensor], out)
model.compile('adam', cl4)
model.fit(x=[train_X, weights], y=train_Y, epochs=10)
```


Question:
Compute gradient norm of each part of composite loss function Assume I have the following loss function:
```
loss_a = tf.reduce_mean(my_loss_fn(model_output, targets))
loss_b = tf.reduce_mean(my_other_loss_fn(model_output, targets))
loss_final = loss_a + tf.multiply(alpha, loss_b)
```
To visualize the norm of the gradients w.r.t to loss_final one could do this:
```
optimizer = tf.train.AdamOptimizer(learning_rate=0.001)
grads_and_vars = optimizer.compute_gradients(loss_final)
grads, _ = list(zip(*grads_and_vars))
norms = tf.global_norm(grads)
gradnorm_s = tf.summary.scalar('gradient norm', norms)
train_op = optimizer.apply_gradients(grads_and_vars, name='train_op')
```
However, I would like to plot the norm of the gradients w.r.t to loss_a and loss_b separately. How can I do this in the most efficient way? Do I have to call compute_gradients(..) on both loss_a and loss_b separately and then add those two gradients together before passing them to optimizer.apply_gradients(..)? I know that this would mathematically be correct due to the summation rule, but it just seems a bit cumbersome and I also don't know how you would implement the summation of the gradients correctly. Also, loss_final is rather simple, because it's just a summation. What if loss_final was more complicated, e.g. a division? I'm using Tensorflow 0.12.

Response:
You are right that combining gradients could get messy. Instead just compute the gradients of each of the losses as well as the final loss. Because tensorflow optimizes the directed acyclic graph (DAG) before compilation, this doesn't result in duplication of work. For example:
```
import tensorflow as tf

with tf.name_scope('inputs'):
    W = tf.Variable(dtype=tf.float32, initial_value=tf.random_normal((4, 1), dtype=tf.float32), name='W')
    x = tf.random_uniform((6, 4), dtype=tf.float32, name='x')

with tf.name_scope('outputs'):
    y = tf.matmul(x, W, name='y')

def my_loss_fn(output, targets, name):
    return tf.reduce_mean(tf.abs(output - targets), name=name)

def my_other_loss_fn(output, targets, name):
    return tf.sqrt(tf.reduce_mean((output - targets) ** 2), name=name)

def get_tensors(loss_fn):

    loss = loss_fn(y, targets, 'loss')
    grads = tf.gradients(loss, W, name='gradients')
    norm = tf.norm(grads, name='norm')

    return loss, grads, norm

targets = tf.random_uniform((6, 1))

with tf.name_scope('a'):
    loss_a, grads_a, norm_a = get_tensors(my_loss_fn)

with tf.name_scope('b'):
    loss_b, grads_b, norm_b = get_tensors(my_loss_fn)

with tf.name_scope('combined'):
    loss = tf.add(loss_a, loss_b, name='loss')
    grad = tf.gradients(loss, W, name='gradients')

with tf.Session() as sess:
    tf.global_variables_initializer().run(session=sess)

    writer = tf.summary.FileWriter('./tensorboard_results', sess.graph)
    res = sess.run([norm_a, norm_b, grad])

    print(*res, sep='\n')
```
Edit: In response to your comment... You can check the DAG of a tensorflow model using tensorboard. I've updated the code to store the graph. Run tensorboard --logdir $PWD/tensorboard_results in a terminal and navigate to the url printed on the commandline (typically http://localhost:6006/). Then click on GRAPH tab to view the DAG. You can recursively expand the tensors, ops, namespaces to see subgraphs to see individual operations and their inputs.


Question:
Quantize a Keras neural network model Recently, I've started creating neural networks with Tensorflow + Keras and I would like to try the quantization feature available in Tensorflow. So far, experimenting with examples from TF tutorials worked just fine and I have this basic working example (from https://www.tensorflow.org/tutorials/keras/basic_classification):
```
import tensorflow as tf
from tensorflow import keras

fashion_mnist = keras.datasets.fashion_mnist
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()

# fashion mnist data labels (indexes related to their respective labelling in the data set)
class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

# preprocess the train and test images
train_images = train_images / 255.0
test_images = test_images / 255.0

# settings variables
input_shape = (train_images.shape[1], train_images.shape[2])

# create the model layers
model = keras.Sequential([
keras.layers.Flatten(input_shape=input_shape),
keras.layers.Dense(128, activation=tf.nn.relu),
keras.layers.Dense(10, activation=tf.nn.softmax)
])

# compile the model with added settings
model.compile(optimizer=tf.train.AdamOptimizer(),
          loss='sparse_categorical_crossentropy',
          metrics=['accuracy'])

# train the model
epochs = 3
model.fit(train_images, train_labels, epochs=epochs)

# evaluate the accuracy of model on test data
test_loss, test_acc = model.evaluate(test_images, test_labels)
print('Test accuracy:', test_acc)
```
Now, I would like to employ quantization in the learning and classification process. The quantization documentation (https://www.tensorflow.org/performance/quantization) (the page is no longer available since cca September 15, 2018) suggests to use this piece of code:
```
loss = tf.losses.get_total_loss()
tf.contrib.quantize.create_training_graph(quant_delay=2000000)
optimizer = tf.train.GradientDescentOptimizer(0.00001)
optimizer.minimize(loss)
```
However, it does not contain any information about where this code should be utilized or how it should be connected to a TF code (not even mentioning a high level model created with Keras). I have no idea how this quantization part relates to the previously created neural network model. Just inserting it following the neural network code runs into the following error:
```
Traceback (most recent call last):
  File "so.py", line 41, in <module>
    loss = tf.losses.get_total_loss()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/util.py", line 112, in get_total_loss
    return math_ops.add_n(losses, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py", line 2119, in add_n
    raise ValueError("inputs must be a list of at least one Tensor with the "
ValueError: inputs must be a list of at least one Tensor with the same dtype and shape
```
Is it possible to quantize a Keras NN model in this way or am I missing something basic? A possible solution that crossed my mind could be using low level TF API instead of Keras (needing to do quite a bit of work to construct the model), or maybe trying to extract some of the lower level methods from the Keras models.

Response:
As mentioned in other answers, TensorFlow Lite can help you with network quantization. TensorFlow Lite provides several levels of support for quantization. Tensorflow Lite post-training quantization quantizes weights and activations post training easily. Quantization-aware training allows for training of networks that can be quantized with minimal accuracy drop; this is only available for a subset of convolutional neural network architectures. So first, you need to decide whether you need post-training quantization or quantization-aware training. For example, if you already saved the model as *.h5 files, you would probably want to follow @Mitiku's instruction and do the post-training quantization. If you prefer to achieve higher performance by simulating the effect of quantization in training (using the method you quoted in the question), and your model is in the subset of CNN architecture supported by quantization-aware training, this example may help you in terms of interaction between Keras and TensorFlow. Basically, you only need to add this code between model definition and its fitting:
```
sess = tf.keras.backend.get_session()
tf.contrib.quantize.create_training_graph(sess.graph)
sess.run(tf.global_variables_initializer())
```


Question: What is a local variable in tensorflow? Tensorflow has this API defined: tf.local_variables() Returns all variables created with collection=[LOCAL_VARIABLES]. Returns: A list of local Variable objects. What exactly is a local variable in TensorFlow? Can someone give me an example?

Response:
Short answer: a local variable in TF is any variable which was created with collections=[tf.GraphKeys.LOCAL_VARIABLES]. For example:
```
e = tf.Variable(6, name='var_e', collections=[tf.GraphKeys.LOCAL_VARIABLES])
```
LOCAL_VARIABLES: the subset of Variable objects that are local to each machine. Usually used for temporarily variables, like counters. Note: use tf.contrib.framework.local_variable to add to this collection. They are usually not saved/restored to checkpoint and used for temporary or intermediate values. Long answer: this was a source of confusion for me as well. In the beginning I thought that local variables mean the same thing as local variable in almost any programming language, but it is not the same thing:
```
import tensorflow as tf

def some_func():
    z = tf.Variable(1, name='var_z')

a = tf.Variable(1, name='var_a')
b = tf.get_variable('var_b', 2)
with tf.name_scope('aaa'):
    c = tf.Variable(3, name='var_c')

with tf.variable_scope('bbb'):
    d = tf.Variable(3, name='var_d')

some_func()
some_func()

print [str(i.name) for i in tf.global_variables()]
print [str(i.name) for i in tf.local_variables()]
```
No matter what I tried, I always recieved only global variables:
```
['var_a:0', 'var_b:0', 'aaa/var_c:0', 'bbb/var_d:0', 'var_z:0', 'var_z_1:0']
[]
```
The documentation for tf.local_variables have not provided a lot of details: Local variables - per process variables, usually not saved/restored to checkpoint and used for temporary or intermediate values. For example, they can be used as counters for metrics computation or number of epochs this machine has read data. The local_variable() automatically adds new variable to GraphKeys.LOCAL_VARIABLES. This convenience function returns the contents of that collection. But reading docs for the init method in tf.Variable class, I found that while creating a variable, you can provide what kind of a variable do you want it to be by assigning a list of collections. The list of possible collection elements is here. So to create a local variable you need to do something like this. You will see it in the list of local_variables:
```
e = tf.Variable(6, name='var_e', collections=[tf.GraphKeys.LOCAL_VARIABLES])
print [str(i.name) for i in tf.local_variables()]
```


Question:
Tensorflow: Attempting to use uninitialized value beta1_power I got the following error when I try to run the code at the end of the post. But it is not clear to me what is wrong with my code. Could anybody let me know the tricks in debugging a tensorflow program?
```
$ ./main.py
Extracting /tmp/data/train-images-idx3-ubyte.gz
Extracting /tmp/data/train-labels-idx1-ubyte.gz
Extracting /tmp/data/t10k-images-idx3-ubyte.gz
Extracting /tmp/data/t10k-labels-idx1-ubyte.gz
2017-12-11 22:53:16.061163: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
Traceback (most recent call last):
  File "./main.py", line 55, in <module>
    sess.run(opt, feed_dict={x: batch_x, y: batch_y})
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 889, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value beta1_power
     [[Node: beta1_power/read = Identity[T=DT_FLOAT, _class=["loc:@Variable"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](beta1_power)]]

Caused by op u'beta1_power/read', defined at:
  File "./main.py", line 46, in <module>
    opt=tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 353, in minimize
    name=name)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 474, in apply_gradients
    self._create_slots([_get_variable_for(v) for v in var_list])
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/adam.py", line 130, in _create_slots
    trainable=False)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1927, in variable
    caching_device=caching_device, name=name, dtype=dtype)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 213, in __init__
    constraint=constraint)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 356, in _init_from_args
    self._snapshot = array_ops.identity(self._variable, name="read")
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 125, in identity
    return gen_array_ops.identity(input, name=name)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2071, in identity
    "Identity", input=input, name=name)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value beta1_power
     [[Node: beta1_power/read = Identity[T=DT_FLOAT, _class=["loc:@Variable"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](beta1_power)]]
```
The code is here. It uses LSTM.
```
#!/usr/bin/env python
# vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1 fileencoding=utf-8:

import tensorflow as tf
from tensorflow.contrib import rnn

#import mnist dataset
from tensorflow.examples.tutorials.mnist import input_data
mnist=input_data.read_data_sets("/tmp/data/", one_hot=True)

learning_rate=0.001

#defining placeholders
#input image placeholder
time_steps=28
n_input=28
x=tf.placeholder("float", [None, time_steps, n_input])

#processing the input tensor from [batch_size,n_steps,n_input] to "time_steps" number of [batch_size,n_input] tensors
input=tf.unstack(x, time_steps, 1)

#defining the network
num_units=128
lstm_layer = rnn.BasicLSTMCell(num_units, forget_bias=1)
outputs,_ = rnn.static_rnn(lstm_layer, input, dtype="float32")

#weights and biases of appropriate shape to accomplish above task
n_classes=10
out_weights=tf.Variable(tf.random_normal([num_units, n_classes]))
out_bias=tf.Variable(tf.random_normal([n_classes]))

#converting last output of dimension [batch_size,num_units] to [batch_size,n_classes] by out_weight multiplication
prediction=tf.matmul(outputs[-1], out_weights) + out_bias

y=tf.placeholder("float", [None, n_classes])
loss=tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=prediction, labels=y))
#optimization

#model evaluation
correct_prediction=tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1))
accuracy=tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

#initialize variables
init=tf.global_variables_initializer()
batch_size=128
opt=tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)
with tf.Session() as sess:
    sess.run(init)
    iter=1
    while iter<800:
        batch_x, batch_y = mnist.train.next_batch(batch_size=batch_size)

        batch_x=batch_x.reshape((batch_size, time_steps, n_input))

        sess.run(opt, feed_dict={x: batch_x, y: batch_y})

        if iter %10==0:
            acc=sess.run(accuracy,feed_dict={x:batch_x,y:batch_y})
            los=sess.run(loss,feed_dict={x:batch_x,y:batch_y})
            print("For iter ",iter)
            print("Accuracy ",acc)
            print("Loss ",los)
            print("__________________")

        iter=iter+1

#calculating test accuracy
test_data = mnist.test.images[:128].reshape((-1, time_steps, n_input))
test_label = mnist.test.labels[:128]
print("Testing Accuracy:", sess.run(accuracy, feed_dict={x: test_data, y: test_label}))
```

Response:
Change the order of these two lines:
```py
opt=tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)
init=tf.global_variables_initializer()
```
Since AdamOptimizer has it's own variables, you should define the initilizer init after opt, not before.


Question:
Tensorflow Deep MNIST: Resource exhausted: OOM when allocating tensor with shape[10000,32,28,28] This is the sample MNIST code I am running:
```
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)

import tensorflow as tf
sess = tf.InteractiveSession()

x = tf.placeholder(tf.float32, shape=[None, 784])
y_ = tf.placeholder(tf.float32, shape=[None, 10])

W = tf.Variable(tf.zeros([784,10]))
b = tf.Variable(tf.zeros([10]))

y = tf.nn.softmax(tf.matmul(x,W) + b)

def weight_variable(shape):
  initial = tf.truncated_normal(shape, stddev=0.1)
  return tf.Variable(initial)

def bias_variable(shape):
  initial = tf.constant(0.1, shape=shape)
  return tf.Variable(initial)

def conv2d(x, W):
  return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')

def max_pool_2x2(x):
  return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],
                        strides=[1, 2, 2, 1], padding='SAME')

W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])

x_image = tf.reshape(x, [-1,28,28,1])

h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)

W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])

h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
h_pool2 = max_pool_2x2(h_conv2)

W_fc1 = weight_variable([7 * 7 * 64, 1024])
b_fc1 = bias_variable([1024])

h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)

keep_prob = tf.placeholder(tf.float32)
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)

W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])

y_conv=tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_conv), reduction_indices=[1]))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

init = tf.initialize_all_variables()
config = tf.ConfigProto()
config.gpu_options.allocator_type = 'BFC'
with tf.Session(config = config) as s:
  sess.run(init)

for i in range(20000):
  batch = mnist.train.next_batch(50)
  if i%100 == 0:
    train_accuracy = accuracy.eval(feed_dict={
        x:batch[0], y_: batch[1], keep_prob: 1.0})
    print("step %d, training accuracy %g"%(i, train_accuracy))
  train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})

print("test accuracy %g"%accuracy.eval(feed_dict={
    x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))
```
The GPU I am using is: GeForce GTX 750 Ti Error:
```
...
...
...
step 19900, training accuracy 1
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (256):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (512):   Total Chunks: 1, Chunks in use: 0 768B allocated for chunks. 1.20MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (1024):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (2048):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (4096):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (8192):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (16384):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (32768):     Total Chunks: 1, Chunks in use: 0 36.8KiB allocated for chunks. 4.79MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (65536):     Total Chunks: 1, Chunks in use: 0 78.5KiB allocated for chunks. 4.79MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (131072):    Total Chunks: 1, Chunks in use: 0 200.0KiB allocated for chunks. 153.1KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (262144):    Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (524288):    Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (1048576):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (2097152):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (4194304):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (8388608):   Total Chunks: 1, Chunks in use: 0 11.86MiB allocated for chunks. 390.6KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (16777216):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (33554432):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (67108864):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (134217728):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (268435456):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:656] Bin for 957.03MiB was 256.00MiB, Chunk State:
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a40000 of size 1280
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a40500 of size 1280
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a40a00 of size 31488
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a48500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a48600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a48700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a48800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a48900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a48a00 of size 4096
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a49a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a49b00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a49c00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a49d00 of size 3328
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a4aa00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a4ab00 of size 204800
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a7cb00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x601a7cc00 of size 12845056
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026bcc00 of size 4096
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026bdc00 of size 40960
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026c7c00 of size 31488
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026cf700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026cf800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026cf900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026cfa00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026cfb00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026cfc00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026cfd00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026cfe00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026cff00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026d0000 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026d0100 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026d0500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026d0600 of size 3328
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026d1300 of size 40960
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6026db300 of size 80128
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x602702600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x602734700 of size 204800
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x603342700 of size 4096
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x603343700 of size 3328
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x60334d700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x60334d800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x60334d900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x60334da00 of size 3328
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x60334e700 of size 3328
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x60334f400 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x60334f500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x60334f600 of size 204800
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x603381600 of size 204800
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6033b3600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6033b3700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6033b3800 of size 12845056
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x603ff3800 of size 12845056
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x604c33800 of size 4096
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x604c34800 of size 4096
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x604c35800 of size 40960
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x604c3f800 of size 40960
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x604c49800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x604c49900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x604c49a00 of size 13053184
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6058bc700 of size 31360000
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x6076a4b00 of size 1801385216
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x6026d0200 of size 768
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x6026eec00 of size 80384
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x602702700 of size 204800
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x602766700 of size 12435456
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x603344400 of size 37632
I tensorflow/core/common_runtime/bfc_allocator.cc:689]      Summary of in-use Chunks by size:
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 32 Chunks of size 256 totalling 8.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 2 Chunks of size 1280 totalling 2.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 5 Chunks of size 3328 totalling 16.2KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 5 Chunks of size 4096 totalling 20.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 2 Chunks of size 31488 totalling 61.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 4 Chunks of size 40960 totalling 160.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 80128 totalling 78.2KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 4 Chunks of size 204800 totalling 800.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 3 Chunks of size 12845056 totalling 36.75MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 13053184 totalling 12.45MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 31360000 totalling 29.91MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 1801385216 totalling 1.68GiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] Sum Total of in-use chunks: 1.76GiB
I tensorflow/core/common_runtime/bfc_allocator.cc:698] Stats:
Limit:                  1898266624
InUse:                  1885507584
MaxInUse:               1885907712
NumAllocs:                 2387902
MaxAllocSize:           1801385216

W tensorflow/core/common_runtime/bfc_allocator.cc:270] **********************************************************xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
W tensorflow/core/common_runtime/bfc_allocator.cc:271] Ran out of memory trying to allocate 957.03MiB.  See logs for memory state.
W tensorflow/core/framework/op_kernel.cc:968] Resource exhausted: OOM when allocating tensor with shape[10000,32,28,28]
Traceback (most recent call last):
  File "trainer_deepMnist.py", line 109, in <module>
    x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 559, in eval
    return _eval_using_default_session(self, feed_dict, self.graph, session)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3648, in _eval_using_default_session
    return session.run(tensors, feed_dict)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 710, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 908, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 958, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 978, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.ResourceExhaustedError: OOM when allocating tensor with shape[10000,32,28,28]
     [[Node: Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](Reshape, Variable_2/read)]]
Caused by op u'Conv2D', defined at:
  File "trainer_deepMnist.py", line 61, in <module>
    h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
  File "trainer_deepMnist.py", line 46, in conv2d
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 394, in conv2d
    data_format=data_format, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2320, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1239, in __init__
    self._traceback = _extract_stack()
```
I read some github issues (here, here) related to the same problem but could not understand how I should change my code to solve this problem.

Response:
Here is how I solved this problem: the error means that the GPU runs out of memory during accuracy evaluation. Hence it needs a smaller sized dataset, which can be achieved by using data in batches. So, instead of running the code on the whole test dataset it needs to be run in batches as mentioned in this post: How to read data in batches when using TensorFlow Hence, for accuracy evaluation on test dataset, instead of this loc :
```
print("test accuracy %g"%accuracy.eval(feed_dict={ x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))
```
this can be used :
```
for i in xrange(10):
    testSet = mnist.test.next_batch(50)
    print("test accuracy %g"%accuracy.eval(feed_dict={ x: testSet[0], y_: testSet[1], keep_prob: 1.0}))
```
When i ran 1000 epochs for training and used 10 batches of batch_size = 50 for accuracy evaluation, I got the following results:
```
step 0, training accuracy 0.04
step 100, training accuracy 0.88
step 200, training accuracy 0.9
step 300, training accuracy 0.88
step 400, training accuracy 0.94
step 500, training accuracy 0.96
step 600, training accuracy 0.94
step 700, training accuracy 0.96
step 800, training accuracy 0.9
step 900, training accuracy 1
test accuracy 1
test accuracy 0.92
test accuracy 1
test accuracy 1
test accuracy 0.94
test accuracy 0.96
test accuracy 0.92
test accuracy 0.96
test accuracy 0.92
test accuracy 0.94
```


Question:
Feeding image data in tensorflow for transfer learning I am trying to use tensorflow for transfer learning. I downloaded the pre-trained model inception3 from the tutorial. In the code, for prediction:
```
prediction = sess.run(softmax_tensor,{'DecodeJpeg/contents:0'}:image_data})
```
Is there a way to feed the png image. I tried changing DecodeJpeg to DecodePng but it did not work. Beside, what should I change if I want to feed decoded image file like a numpy array or a batch of arrays? Thanks!!

Response:
The shipped InceptionV3 graph used in classify_image.py only supports JPEG images out-of-the-box. There are two ways you could use this graph with PNG images: Convert the PNG image to a height x width x 3 (channels) Numpy array, for example using PIL, then feed the 'DecodeJpeg:0' tensor:
```py
import numpy as np
from PIL import Image
# ...

image = Image.open("example.png")
image_array = np.array(image)[:, :, 0:3]  # Select RGB channels only.

prediction = sess.run(softmax_tensor, {'DecodeJpeg:0': image_array})
```
Perhaps confusingly, 'DecodeJpeg:0' is the output of the DecodeJpeg op, so by feeding this tensor, you are able to feed raw image data. Add a tf.image.decode_png() op to the imported graph. Simply switching the name of the fed tensor from 'DecodeJpeg/contents:0' to 'DecodePng/contents:0' does not work because there is no 'DecodePng' op in the shipped graph. You can add such a node to the graph by using the input_map argument to tf.import_graph_def():
```py
png_data = tf.placeholder(tf.string, shape=[])
decoded_png = tf.image.decode_png(png_data, channels=3)
# ...

graph_def = ...
softmax_tensor = tf.import_graph_def(
    graph_def,
    input_map={'DecodeJpeg:0': decoded_png},
    return_elements=['softmax:0'])

sess.run(softmax_tensor, {png_data: ...})
```


Question: How to accumulate gradients for large batch sizes in Keras I am working with a very memory demanding CNN model for a task of classification. This poses a big limit on the batch size that I can use during training. One solution is to accumulate the gradients during training, meaning that the weights of the model are not updated after every single batch. Instead the same weights are used for several batches, while the gradients from each batch are accumulated and than averaged for a single weight-update action. I'm using a Tensorflow backend Keras and I'm pretty sure that Keras has no off-the-shelf function/method to achieve this. How can it be done for a Keras/tensorflow model?

Response:
As was mentioned in the question, there is no off-the-shelf function/method to achieve this with Keras/Tensorflow. However this can be done by writing a custom optimizer for Keras. The main idea is to use a flag to determine whether to update the weights during each batch. The following implementation is based on this github post by "alexeydevederkin" and it is an accumulating Adam optimizer:
```
import keras.backend as K
from keras.legacy import interfaces
from keras.optimizers import Optimizer

class AdamAccumulate(Optimizer):

    def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999,
                 epsilon=None, decay=0., amsgrad=False, accum_iters=1, **kwargs):
        if accum_iters < 1:
            raise ValueError('accum_iters must be >= 1')
        super(AdamAccumulate, self).__init__(**kwargs)
        with K.name_scope(self.__class__.__name__):
            self.iterations = K.variable(0, dtype='int64', name='iterations')
            self.lr = K.variable(lr, name='lr')
            self.beta_1 = K.variable(beta_1, name='beta_1')
            self.beta_2 = K.variable(beta_2, name='beta_2')
            self.decay = K.variable(decay, name='decay')
        if epsilon is None:
            epsilon = K.epsilon()
        self.epsilon = epsilon
        self.initial_decay = decay
        self.amsgrad = amsgrad
        self.accum_iters = K.variable(accum_iters, K.dtype(self.iterations))
        self.accum_iters_float = K.cast(self.accum_iters, K.floatx())

    @interfaces.legacy_get_updates_support
    def get_updates(self, loss, params):
        grads = self.get_gradients(loss, params)
        self.updates = [K.update_add(self.iterations, 1)]

        lr = self.lr

        completed_updates = K.cast(K.tf.floordiv(self.iterations, self.accum_iters), K.floatx())

        if self.initial_decay > 0:
            lr = lr * (1. / (1. + self.decay * completed_updates))

        t = completed_updates + 1

        lr_t = lr * (K.sqrt(1. - K.pow(self.beta_2, t)) / (1. - K.pow(self.beta_1, t)))

        # self.iterations incremented after processing a batch
        # batch:              1 2 3 4 5 6 7 8 9
        # self.iterations:    0 1 2 3 4 5 6 7 8
        # update_switch = 1:        x       x    (if accum_iters=4)
        update_switch = K.equal((self.iterations + 1) % self.accum_iters, 0)
        update_switch = K.cast(update_switch, K.floatx())

        ms = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]
        vs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]
        gs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]

        if self.amsgrad:
            vhats = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]
        else:
            vhats = [K.zeros(1) for _ in params]

        self.weights = [self.iterations] + ms + vs + vhats

        for p, g, m, v, vhat, tg in zip(params, grads, ms, vs, vhats, gs):

            sum_grad = tg + g
            avg_grad = sum_grad / self.accum_iters_float

            m_t = (self.beta_1 * m) + (1. - self.beta_1) * avg_grad
            v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(avg_grad)

            if self.amsgrad:
                vhat_t = K.maximum(vhat, v_t)
                p_t = p - lr_t * m_t / (K.sqrt(vhat_t) + self.epsilon)
                self.updates.append(K.update(vhat, (1 - update_switch) * vhat + update_switch * vhat_t))
            else:
                p_t = p - lr_t * m_t / (K.sqrt(v_t) + self.epsilon)

            self.updates.append(K.update(m, (1 - update_switch) * m + update_switch * m_t))
            self.updates.append(K.update(v, (1 - update_switch) * v + update_switch * v_t))
            self.updates.append(K.update(tg, (1 - update_switch) * sum_grad))
            new_p = p_t

            # Apply constraints.
            if getattr(p, 'constraint', None) is not None:
                new_p = p.constraint(new_p)

            self.updates.append(K.update(p, (1 - update_switch) * p + update_switch * new_p))
        return self.updates

    def get_config(self):
        config = {'lr': float(K.get_value(self.lr)),
                  'beta_1': float(K.get_value(self.beta_1)),
                  'beta_2': float(K.get_value(self.beta_2)),
                  'decay': float(K.get_value(self.decay)),
                  'epsilon': self.epsilon,
                  'amsgrad': self.amsgrad}
        base_config = super(AdamAccumulate, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))
```
It can be used in the following way:
```
opt = AdamAccumulate(lr=0.001, decay=1e-5, accum_iters=5)
model.compile( loss='categorical_crossentropy',   # Loss function
                            optimizer=opt,        # Optimization technique
                            metrics=['accuracy']) # Accuracy matrix
model.fit(X_train, y_train, batch_size = 10)
```
In this example, the model processes 10 samples in every iteration ("batch_size"), but the update to the weights only happens after accumulating 5 such batches ("accum_iters"). So the actual batch size for updating the weights is 50.


Question:
TensorFlow: how is dataset.train.next_batch defined? I am trying to learn TensorFlow and studying the example at: https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/autoencoder.ipynb I then have some questions in the code below:
```
for epoch in range(training_epochs):
    # Loop over all batches
    for i in range(total_batch):
        batch_xs, batch_ys = mnist.train.next_batch(batch_size)
        # Run optimization op (backprop) and cost op (to get loss value)
        _, c = sess.run([optimizer, cost], feed_dict={X: batch_xs})
    # Display logs per epoch step
    if epoch % display_step == 0:
        print("Epoch:", '%04d' % (epoch+1),
              "cost=", "{:.9f}".format(c))
```
Since mnist is just a dataset, what exactly does mnist.train.next_batch mean? How was the dataset.train.next_batch defined? Thanks!

Response: The mnist object is returned from the read_data_sets() function defined in the tf.contrib.learn module. The mnist.train.next_batch(batch_size) method is implemented here, and it returns a tuple of two arrays, where the first represents a batch of batch_size MNIST images, and the second represents a batch of batch-size labels corresponding to those images. The images are returned as a 2-D NumPy array of size [batch_size, 784] (since there are 784 pixels in an MNIST image), and the labels are returned as either a 1-D NumPy array of size [batch_size] (if read_data_sets() was called with one_hot=False) or a 2-D NumPy array of size [batch_size, 10] (if read_data_sets() was called with one_hot=True).


Question:
What is tensorflow.compat.as_str()? In the Google/Udemy Tensorflow tutorial there is the following code:
```
import tensorflow as tf
...
def read_data(filename):
    """Extract the first file enclosed in a zip file as a list of words"""
    with zipfile.ZipFile(filename) as f:
    data = tf.compat.as_str(f.read(f.namelist()[0])).split()
return data
```
This executes fine but I cannot find compat.as_str in the Tensorflow documentation or anywhere else. Q1: What does compat.as_str do? Q2: Is this tensorflow compat library documented somewhere? Q3: This is a call to the tensorflow library, so how and why does it work in normal python code, rather than inside a tensorflow graph? I.e. I thought tensorflow library calls had to be inside a tensorflow graph defintion block:
```
graph = tf.Graph()
with graph.as_default()
    ... tensorflow function calls here ...
```
I am running python 2.7.

Response:
Basically, it comes from the fact that in Python 2, strings were dealt with primarily as bytes, not unicode. In Python 3, all strings are natively unicode. The purpose of the function is to ensure that whichever Python version you're using, you won't be bothered, hence the compat module name standing for compatibility. Under the hood, tensorflow.compat.as_str converts both bytes and unicode strings to unicode strings.
```
Signature: tensorflow.compat.as_str(bytes_or_text, encoding='utf-8')
Docstring:
Returns the given argument as a unicode string.

Args:
  bytes_or_text: A `bytes`, `str, or `unicode` object.
  encoding: A string indicating the charset for decoding unicode.

Returns:
  A `unicode` (Python 2) or `str` (Python 3) object.

Raises:
  TypeError: If `bytes_or_text` is not a binary or unicode string.
```
The library is documented here.


Question: EM score in SQuAD Challenge The SQuAD Challenge ranks the results against the F1 and EM scores. There is a lot of information about the F1 score (a function of precision and recall). But what would the EM score be?

Response: Exact match. This metric measures the percentage of predictions that match any one of the ground truth answers exactly. According to here.


Question: How to acquire tf.data.dataset's shape? I know dataset has output_shapes, but it shows like below: data_set: DatasetV1Adapter shapes: {item_id_hist: (?, ?), tags: (?, ?), client_platform: (?,), entrance: (?,), item_id: (?,), lable: (?,), mode: (?,), time: (?,), user_id: (?,)}, types: {item_id_hist: tf.int64, tags: tf.int64, client_platform: tf.string, entrance: tf.string, item_id: tf.int64, lable: tf.int64, mode: tf.int64, time: tf.int64, user_id: tf.int64} How can I get the total number of my data?

Response:
Where the length is known you can call:
```
tf.data.experimental.cardinality(dataset)
```
but if this fails then, it's important to know that a TensorFlow Dataset is (in general) lazily evaluated so this means that in the general case we may need to iterate over every record before we can find the length of the dataset. For example, assuming you have eager execution enabled and its a small 'toy' dataset that fits comfortably in memory you could just enumerate it into a new list and grab the last index (then add 1 because lists are zero-indexed):
```
dataset_length = [i for i,_ in enumerate(dataset)][-1] + 1
```
Of course this is inefficient at best and, for large datasets, will fail entirely because everything needs to fit into memory for the list. in such circumstances I can't see any alternative other than to iterate through the records keeping a manual count.


Question:
How to use tensorflow to implement deconvolution? I want to take use of tensorflow to implement fully convolutional network. There is a function
```
tf.nn.conv2d_transpose(value, filter, output_shape, strides, padding, name),
```
which could be used to take bilinear upsampling. However, I am confused as to how to use it? The input is an image with a single channel, and the output is also an image with a single channel, whose size is two times the one of the input. I tried to use the function as follows, but got an IndexError: list index out of range:
```
with tf.name_scope('deconv') as scope:
    deconv = tf.nn.conv2d_transpose(conv6, [3, 3, 1, 1],
        [1, 26, 20, 1], 2, padding='SAME', name=None)
```

Response:
Got it! (assuming input_size = [1, 13, 10,1])
```
with tf.name_scope('deconv') as scope:
    deconv = tf.nn.conv2d_transpose(input_layer, [3, 3, 1, 1],
         [1, 26, 20, 1], [1, 2, 2, 1], padding='SAME', name=None)
```


Question:
Is there a tensorflow equivalent to np.empty? Numpy has this helper function, np.empty, which will: Return a new array of given shape and type, without initializing entries. I find it pretty useful when I want to create a tensor using tf.concat since: The number of dimensions of the input tensors must match, and all dimensions except axis must be equal. So it comes in handy to start with an empty tensor of an expected shape. Is there any way to achieve this in tensorflow? [edit] A simplified example of why I want this
```
netInput = np.empty([0, 4])
    netTarget = np.empty([0, 4])
    inputWidth = 2

    for step in range(data.shape.as_list()[-2]-frames_width-1):
        netInput = tf.concat([netInput, data[0, step:step + frames_width, :]], -2)
        target = tf.concat([target, data[0, step + frames_width + 1:step + frames_width + 2, :]], -2)
```
In this example, if netInput or netTarget are initialized, I'll be concatenating an extra example with that initialization. And to initialize them with the first value, I need to hack the loop. Nothing mayor, I just wondered if there is a 'tensorflow' way to solve this.

Response:
If you're creating an empty tensor, tf.zeros will do
```

```python
a = tf.zeros([0, 4])
```

```python
tf.concat([a, [[1, 2, 3, 4], [5, 6, 7, 8]]], axis=0)
#Output
#<tf.Tensor: shape=(2, 4), dtype=float32, numpy=
#array([[1., 2., 3., 4.],
#       [5., 6., 7., 8.]], dtype=float32)>
#```
```


Question:
How to inform class weights when using `tensorflow.python.keras.estimator.model_to_estimator` to convert Keras Models to Estimator API? I'm having some trouble to convert a pure Keras model to TensorFlow Estimator API on an unbalanced dataset. When using pure Keras API, the class_weight parameter is available at model.fit method, but when converting a Keras model to TensorFlow Estimator with tensorflow.python.keras.estimator.model_to_estimator there is no place to inform class_weights. How can overcome this? I'm using TF 1.12 on Ubuntu 18, Cuda 9, Cudnn 7 Pure Keras model:
```
def keras_model(n_classes=None, model_dir='./tmp-model/', config=None):
    with tf.device('/gpu:0'):

        # Inputs
        inp_raw = Input(shape=(max_len,), name='word_raw')

        # raw text LSTM network
        word_raw_emb = Embedding(
            input_dim=nunique_chars_raw,
            output_dim=EMBED_SIZE,
            input_length=MAX_WORD_LENGTH,
            trainable=True,
            name='word_raw_emb')(inp_raw)

        word_raw_emb = Dropout(rate=dropout_rate)(word_raw_emb)

        word_raw_emb_lstm = Bidirectional(
            LSTM(48, return_sequences=True))(word_raw_emb)
        word_raw_emb_gru = Bidirectional(
            GRU(48, return_sequences=False))(word_raw_emb_lstm)

        word_raw_net = Dense(16, activation='relu')(word_raw_emb_gru)
        output_raw_net = Dense(n_classes, activation='softmax')(word_raw_net)

        model = Model(inputs=inp_raw, outputs=output_raw_net)
        optz = keras.optimizers.RMSprop(
            lr=0.002, rho=0.9, epsilon=None, decay=0.0)
        model.compile(loss='categorical_crossentropy',
                      optimizer=optz, metrics=['categorical_accuracy'])
        return model

model = keras_model(5)

model.fit(train_X, train_Y_onehot,
        batch_size=128,
        epochs=10,
        validation_data=(eval_X,eval_Y_onehot),
        class_weight=class_weights,
        verbose=1)
```
Keras model to TensorFlow Estimator:
```
def keras_estimator_model(n_classes=None, model_dir='./tmp-model/', config=None):
    with tf.device('/gpu:0'):

        # Inputs
        inp_raw = Input(shape=(max_len,), name='word_raw')

        # raw text LSTM network
        word_raw_emb = Embedding(
            input_dim=nunique_chars_raw,
            output_dim=EMBED_SIZE,
            input_length=MAX_WORD_LENGTH,
            trainable=True,
            name='word_raw_emb')(inp_raw)

        word_raw_emb = Dropout(rate=dropout_rate)(word_raw_emb)

        word_raw_emb_lstm = Bidirectional(
            LSTM(48, return_sequences=True))(word_raw_emb)
        word_raw_emb_gru = Bidirectional(
            GRU(48, return_sequences=False))(word_raw_emb_lstm)

        word_raw_net = Dense(16, activation='relu')(word_raw_emb_gru)
        output_raw_net = Dense(n_classes, activation='softmax')(word_raw_net)

        model = Model(inputs=inp_raw, outputs=output_raw_net)
        optz = keras.optimizers.RMSprop(
            lr=0.002, rho=0.9, epsilon=None, decay=0.0)
        model.compile(loss='categorical_crossentropy',
                      optimizer=optz, metrics=['categorical_accuracy'])

        model_estimator = model_to_estimator(keras_model=model, model_dir=model_dir, config=config)

        return model_estimator

estimator_model = keras_estimator_model(5)

train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn,max_steps=10)

eval_spec = tf.estimator.EvalSpec(
        input_fn=eval_input_fn,
        steps=None,
        start_delay_secs=10,
        throttle_secs=10)

tf.estimator.train_and_evaluate(estimator_model, train_spec, eval_spec)
```

Response:
I've codded a workaround and it seems to work. I
```
import tensorflow as tf
from tensorflow.python.keras import backend as K

def weighted_loss_fn(class_weights):

    def _loss_fn(y_true, y_pred):
        class_weights_tensor = K.variable(class_weights)
        y_true_labels = K.argmax(y_true,axis=1)
        weights = K.gather(class_weights_tensor,y_true_labels)

        return tf.losses.softmax_cross_entropy(onehot_labels=y_true, logits=y_pred, weights=weights)
    return _loss_fn

def keras_estimator_model(n_classes=None, model_dir='./tmp-model/', config=None, class_weights=None):
    with tf.device('/gpu:0'):

        # Inputs
        inp_raw = Input(shape=(max_len,), name='word_raw')

        # raw text LSTM network
        word_raw_emb = Embedding(
            input_dim=nunique_chars_raw,
            output_dim=EMBED_SIZE,
            input_length=MAX_WORD_LENGTH,
            trainable=True,
            name='word_raw_emb')(inp_raw)

        word_raw_emb = Dropout(rate=dropout_rate)(word_raw_emb)

        word_raw_emb_lstm = Bidirectional(
            LSTM(48, return_sequences=True))(word_raw_emb)
        word_raw_emb_gru = Bidirectional(
            GRU(48, return_sequences=False))(word_raw_emb_lstm)

        word_raw_net = Dense(16, activation='relu')(word_raw_emb_gru)
        output_raw_net = Dense(n_classes, activation='softmax')(word_raw_net)

        model = Model(inputs=inp_raw, outputs=output_raw_net)
        optz = keras.optimizers.RMSprop(
            lr=0.002, rho=0.9, epsilon=None, decay=0.0)

        loss_fn = weighted_loss_fn(class_weights)
        model.compile(loss=loss_fn,
                      optimizer=optz, metrics=['categorical_accuracy'])

        model_estimator = model_to_estimator(keras_model=model, model_dir=model_dir, config=config)

        return model_estimator

estimator_model = keras_estimator_model(5)
train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn,max_steps=10)
eval_spec = tf.estimator.EvalSpec(
        input_fn=eval_input_fn,
        steps=None,
        start_delay_secs=10,
        throttle_secs=10)

tf.estimator.train_and_evaluate(estimator_model, train_spec, eval_spec)
```
In my case class_weights = [ 0.17041813 42.00318471 35.26470588 29.70495495 42.00318471 44.55743243]


Question: Storing tensorflow models in memory The program I'm writing involves switching between models during run-time. I am currently using Saver to save/load models from the disk as specified here: https://www.tensorflow.org/api_docs/python/state_ops/saving_and_restoring_variables#Saver. The models are fairly small and can be stored in memory, so I was wondering if anyone knows of a way to store and restore these models in-memory instead of saving them to disk. I tried to modify the tensorflow source to save the model to memory however gen_io_ops seems to be generated during compile time. Another possible way is to use memory mapped files. Does anyone know of an easier way?

Response: I would just have two different sessions with their own computation graphs. Alternatively, you could just duplicate the computation graph (two copies of the variables, operations, etc) in the same session. Then you would call sess.run(comp1 if useCompOne else comp2), however you'd like to set it up.


Question:
TypeError: Fetch argument has invalid type float32, must be a string or Tensor I'm training a CNN quite similar to the one in this example, for image segmentation. The images are 1500x1500x1, and labels are of the same size. After defining the CNN structure, and in launching the session as in this code sample: (conv_net_test.py)
```
with tf.Session() as sess:
sess.run(init)
summ = tf.train.SummaryWriter('/tmp/logdir/', sess.graph_def)
step = 1
print ("import data, read from read_data_sets()...")

#Data defined by me, returns a DataSet object with testing and training images and labels for segmentation problem.
data = import_data_test.read_data_sets('Dataset')

# Keep training until reach max iterations
while step * batch_size < training_iters:
    batch_x, batch_y = data.train.next_batch(batch_size)
    print ("running backprop for step %d" % step)
    batch_x = batch_x.reshape(batch_size, n_input, n_input, n_channels)
    batch_y = batch_y.reshape(batch_size, n_input, n_input, n_channels)
    batch_y = np.int64(batch_y)
    sess.run(optimizer, feed_dict={x: batch_x, y: batch_y, keep_prob: dropout})
    if step % display_step == 0:
        # Calculate batch loss and accuracy
        #pdb.set_trace()
        loss, acc = sess.run([loss, accuracy], feed_dict={x: batch_x, y: batch_y, keep_prob: 1.})
    step += 1
print "Optimization Finished"
```
I hit upon the following TypeError (stacktrace below):
```
conv_net_test.py in <module>()
    178             #pdb.set_trace()
--> 179             loss, acc = sess.run([loss, accuracy], feed_dict={x: batch_x, y: batch_y, keep_prob: 1.})
    180         step += 1
    181     print "Optimization Finished!"

tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
    370     try:
    371       result = self._run(None, fetches, feed_dict, options_ptr,
--> 372                          run_metadata_ptr)
    373       if run_metadata:
    374         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
    582
    583     # Validate and process fetches.
--> 584     processed_fetches = self._process_fetches(fetches)
    585     unique_fetches = processed_fetches[0]
    586     target_list = processed_fetches[1]

tensorflow/python/client/session.pyc in _process_fetches(self, fetches)
    538           raise TypeError('Fetch argument %r of %r has invalid type %r, '
    539                           'must be a string or Tensor. (%s)'
--> 540                           % (subfetch, fetch, type(subfetch), str(e)))

TypeError: Fetch argument 1.4415792e+2 of 1.4415792e+2 has invalid type <type 'numpy.float32'>, must be a string or Tensor. (Can not convert a float32 into a Tensor or Operation.)
```
I am stumped at this point. Maybe this is a simple case of converting the type, but I'm not sure how/where. Also, why does the loss have to be a string? (Assuming the same error will pop up for the accuracy as well, once this is fixed). Any help appreciated!

Response:
Where you use loss = sess.run(loss), you redefine in python the variable loss. The first time it will run fine. The second time, you will try to do:
```
sess.run(1.4415792e+2)
```
Because loss is now a float. You should use different names like:
```
loss_val, acc = sess.run([loss, accuracy], feed_dict={x: batch_x, y: batch_y, keep_prob: 1.})
```


Question:
Using Tensorflow Huber loss in Keras I am trying to use huber loss in a keras model (writing DQN), but I am getting bad result, I think I am something doing wrong. My is code is below.
```
model = Sequential()
model.add(Dense(output_dim=64, activation='relu', input_dim=state_dim))
model.add(Dense(output_dim=number_of_actions, activation='linear'))
loss = tf.losses.huber_loss(delta=1.0)
model.compile(loss=loss, opt='sgd')
return model
```

Response:
I came here with the exact same question. The accepted answer uses logcosh which may have similar properties, but it isn't exactly Huber Loss. Here's how I implemented Huber Loss for Keras (note that I'm using Keras from Tensorflow 1.5).
```
import numpy as np
import tensorflow as tf

'''
 ' Huber loss.
 ' https://jaromiru.com/2017/05/27/on-using-huber-loss-in-deep-q-learning/
 ' https://en.wikipedia.org/wiki/Huber_loss
'''
def huber_loss(y_true, y_pred, clip_delta=1.0):
  error = y_true - y_pred
  cond  = tf.keras.backend.abs(error) < clip_delta

  squared_loss = 0.5 * tf.keras.backend.square(error)
  linear_loss  = clip_delta * (tf.keras.backend.abs(error) - 0.5 * clip_delta)

  return tf.where(cond, squared_loss, linear_loss)

'''
 ' Same as above but returns the mean loss.
'''
def huber_loss_mean(y_true, y_pred, clip_delta=1.0):
  return tf.keras.backend.mean(huber_loss(y_true, y_pred, clip_delta))
```
Depending if you want to reduce the loss or the mean of the loss, use the corresponding function above.


Question: How do backpropagation works in tensorflow In tensorflow it seems that the entire backpropagation algorithm is performed by a single running of an optimizer on a certain cost function, which is the output of some MLP or a CNN. I do not fully understand how tensorflow knows from the cost that it is indeed an output of a certain NN? A cost function can be defined for any model. How should I "tell" it that a certain cost function derives from a NN?

Response:
Question How should I "tell" tf that a certain cost function derives from a NN? (short) Answer This is done by simply configuring your optimizer to minimize (or maximize) a tensor. For example, if I have a loss function like so
```
loss = tf.reduce_sum( tf.square( y0 - y_out ) )
```
where y0 is the ground truth (or desired output) and y_out is the calculated output, then I could minimize the loss by defining my training function like so
```
train = tf.train.GradientDescentOptimizer(1.0).minimize(loss)
```
This tells Tensorflow that when train is calculated, it is to apply gradient descent on loss to minimize it, and loss is calculated using y0 and y_out, and so gradient descent will also affect those (if they are trainable variables), and so on. The variable y0, y_out, loss, and train are not standard python variables but instead descriptions of a computation graph. Tensorflow uses information about that computation graph to unroll it while applying gradient descent. Specifically how it does that is beyond the scope of this answer. Here and here are two good starting points for more information about more specifics. Code Example Let's walk through a code example. First the code.
```
### imports
import tensorflow as tf

### constant data
x  = [[0.,0.],[1.,1.],[1.,0.],[0.,1.]]
y_ = [[0.],[0.],[1.],[1.]]

### induction
# 1x2 input -> 2x3 hidden sigmoid -> 3x1 sigmoid output

# Layer 0 = the x2 inputs
x0 = tf.constant( x  , dtype=tf.float32 )
y0 = tf.constant( y_ , dtype=tf.float32 )

# Layer 1 = the 2x3 hidden sigmoid
m1 = tf.Variable( tf.random_uniform( [2,3] , minval=0.1 , maxval=0.9 , dtype=tf.float32  ))
b1 = tf.Variable( tf.random_uniform( [3]   , minval=0.1 , maxval=0.9 , dtype=tf.float32  ))
h1 = tf.sigmoid( tf.matmul( x0,m1 ) + b1 )

# Layer 2 = the 3x1 sigmoid output
m2 = tf.Variable( tf.random_uniform( [3,1] , minval=0.1 , maxval=0.9 , dtype=tf.float32  ))
b2 = tf.Variable( tf.random_uniform( [1]   , minval=0.1 , maxval=0.9 , dtype=tf.float32  ))
y_out = tf.sigmoid( tf.matmul( h1,m2 ) + b2 )

### loss
# loss : sum of the squares of y0 - y_out
loss = tf.reduce_sum( tf.square( y0 - y_out ) )

# training step : gradient decent (1.0) to minimize loss
train = tf.train.GradientDescentOptimizer(1.0).minimize(loss)

### training
# run 500 times using all the X and Y
# print out the loss and any other interesting info
with tf.Session() as sess:
  sess.run( tf.global_variables_initializer() )
  for step in range(500) :
    sess.run(train)

  results = sess.run([m1,b1,m2,b2,y_out,loss])
  labels  = "m1,b1,m2,b2,y_out,loss".split(",")
  for label,result in zip(*(labels,results)) :
    print ""
    print label
    print result

print ""
```
Let's go through it, but in reverse order starting with
```
sess.run(train)
```
This tells tensorflow to look up the graph node defined by train and calculate it. Train is defined as
```
train = tf.train.GradientDescentOptimizer(1.0).minimize(loss)
```
To calculate this tensorflow must compute the automatic differentiation for loss, which means walking the graph. loss is defined as
```
loss = tf.reduce_sum( tf.square( y0 - y_out ) )
```
Which is really tensorflow applying automatic differentiation to unroll first tf.reduce_sum, then tf.square, then y0 - y_out, which leads to then having to walk the graph for both y0 and y_out.
```
y0 = tf.constant( y_ , dtype=tf.float32 )
```
y0 is a constant and will not be updated.
```
y_out = tf.sigmoid( tf.matmul( h1,m2 ) + b2 )
```
y_out will be processed similar to loss, first tf.sigmoid will be processed, etc... All in all, each operation ( such as tf.sigmoid, tf.square ) not only defines the forward operation ( apply sigmoid or square ) but also information necessary for automatic differentiation. This is different than standard python math such as
```
x = 7 + 9
```
The above equation encodes nothing except how to update x, where as
```
z = y0 - y_out
```
encodes the graph of subtracting y_out from y0 and stores both the forward operation and enough to do automatic differentiation in z


Question:
Python_io in tensorflow I'm having trouble working with tensorflow. I want to use TFRecordWriter() as below:
```
with tf.python_io.TFRecordWriter(testing_filename) as tfrecord_writer:
    # do sth
```
but I get the error: AttributeError: module 'tensorflow' has no attribute 'python_io' I'm working with tensorflow 1.2 and python 3. How can I fix the problem? Thanks.

Response:
The problem in the (python_io) :
```
with tf.python_io.TFRecordWriter(testing_filename) as tfrecord_writer:
# do sth
```
changed it to:
```
with tf.io.TFRecordWriter(testing_filename) as tfrecord_writer:
# do sth
```


Question:
tensorflow dataset shuffle then batch or batch then shuffle I recently began learning tensorflow. I am unsure about whether there is a difference
```
x = np.array([[1],[2],[3],[4],[5]])
dataset = tf.data.Dataset.from_tensor_slices(x)
ds.shuffle(buffer_size=4)
ds.batch(4)
```
and
```
x = np.array([[1],[2],[3],[4],[5]])
dataset = tf.data.Dataset.from_tensor_slices(x)
ds.batch(4)
ds.shuffle(buffer_size=4)
```
Also, I am not sure why I cannot use
```
dataset = dataset.shuffle_batch(buffer_size=2,batch_size=BATCH_SIZE)
```
as it gives the error
```
dataset = dataset.shuffle_batch(buffer_size=2,batch_size=BATCH_SIZE)
AttributeError: 'TensorSliceDataset' object has no attribute 'shuffle_batch'
```
Thank you!

Response:
TL;DR: Yes, there is a difference. Almost always, you will want to call Dataset.shuffle() before Dataset.batch(). There is no shuffle_batch() method on the tf.data.Dataset class, and you must call the two methods separately to shuffle and batch a dataset. The transformations of a tf.data.Dataset are applied in the same sequence that they are called. Dataset.batch() combines consecutive elements of its input into a single, batched element in the output. We can see the effect of the order of operations by considering the following two datasets:
```
tf.enable_eager_execution()  # To simplify the example code.

# Batch before shuffle.
dataset = tf.data.Dataset.from_tensor_slices([0, 0, 0, 1, 1, 1, 2, 2, 2])
dataset = dataset.batch(3)
dataset = dataset.shuffle(9)

for elem in dataset:
  print(elem)

# Prints:
# tf.Tensor([1 1 1], shape=(3,), dtype=int32)
# tf.Tensor([2 2 2], shape=(3,), dtype=int32)
# tf.Tensor([0 0 0], shape=(3,), dtype=int32)

# Shuffle before batch.
dataset = tf.data.Dataset.from_tensor_slices([0, 0, 0, 1, 1, 1, 2, 2, 2])
dataset = dataset.shuffle(9)
dataset = dataset.batch(3)

for elem in dataset:
  print(elem)

# Prints:
# tf.Tensor([2 0 2], shape=(3,), dtype=int32)
# tf.Tensor([2 1 0], shape=(3,), dtype=int32)
# tf.Tensor([0 1 1], shape=(3,), dtype=int32)
```
In the first version (batch before shuffle), the elements of each batch are 3 consecutive elements from the input; whereas in the second version (shuffle before batch), they are randomly sampled from the input. Typically, when training by (some variant of) mini-batch stochastic gradient descent, the elements of each batch should be sampled as uniformly as possible from the total input. Otherwise, it is possible that the network will overfit to whatever structure was in the input data, and the resulting network will not achieve as high an accuracy.


Question:
What are c_state and m_state in Tensorflow LSTM? Tensorflow r0.12's documentation for tf.nn.rnn_cell.LSTMCell describes this as the init:
```
tf.nn.rnn_cell.LSTMCell.__call__(inputs, state, scope=None)
```
where state is as follows: state: if state_is_tuple is False, this must be a state Tensor, 2-D, batch x state_size. If state_is_tuple is True, this must be a tuple of state Tensors, both 2-D, with column sizes c_state and m_state. What aare c_state and m_state and how do they fit into LSTMs? I cannot find reference to them anywhere in the documentation. Here is a link to that page in the documentation.

Response:
I agree that the documentation is unclear. Looking at tf.nn.rnn_cell.LSTMCell.__call__ clarifies (I took the code from TensorFlow 1.0.0):
```
def __call__(self, inputs, state, scope=None):
    """Run one step of LSTM.

    Args:
      inputs: input Tensor, 2D, batch x num_units.
      state: if `state_is_tuple` is False, this must be a state Tensor,
        `2-D, batch x state_size`.  If `state_is_tuple` is True, this must be a
        tuple of state Tensors, both `2-D`, with column sizes `c_state` and
        `m_state`.
      scope: VariableScope for the created subgraph; defaults to "lstm_cell".

    Returns:
      A tuple containing:

      - A `2-D, [batch x output_dim]`, Tensor representing the output of the
        LSTM after reading `inputs` when previous state was `state`.
        Here output_dim is:
           num_proj if num_proj was set,
           num_units otherwise.
      - Tensor(s) representing the new state of LSTM after reading `inputs` when
        the previous state was `state`.  Same type and shape(s) as `state`.

    Raises:
      ValueError: If input size cannot be inferred from inputs via
        static shape inference.
    """
    num_proj = self._num_units if self._num_proj is None else self._num_proj

    if self._state_is_tuple:
      (c_prev, m_prev) = state
    else:
      c_prev = array_ops.slice(state, [0, 0], [-1, self._num_units])
      m_prev = array_ops.slice(state, [0, self._num_units], [-1, num_proj])

    dtype = inputs.dtype
    input_size = inputs.get_shape().with_rank(2)[1]
    if input_size.value is None:
      raise ValueError("Could not infer input size from inputs.get_shape()[-1]")
    with vs.variable_scope(scope or "lstm_cell",
                           initializer=self._initializer) as unit_scope:
      if self._num_unit_shards is not None:
        unit_scope.set_partitioner(
            partitioned_variables.fixed_size_partitioner(
                self._num_unit_shards))
      # i = input_gate, j = new_input, f = forget_gate, o = output_gate
      lstm_matrix = _linear([inputs, m_prev], 4 * self._num_units, bias=True,
                            scope=scope)
      i, j, f, o = array_ops.split(
          value=lstm_matrix, num_or_size_splits=4, axis=1)

      # Diagonal connections
      if self._use_peepholes:
        with vs.variable_scope(unit_scope) as projection_scope:
          if self._num_unit_shards is not None:
            projection_scope.set_partitioner(None)
          w_f_diag = vs.get_variable(
              "w_f_diag", shape=[self._num_units], dtype=dtype)
          w_i_diag = vs.get_variable(
              "w_i_diag", shape=[self._num_units], dtype=dtype)
          w_o_diag = vs.get_variable(
              "w_o_diag", shape=[self._num_units], dtype=dtype)

      if self._use_peepholes:
        c = (sigmoid(f + self._forget_bias + w_f_diag * c_prev) * c_prev +
             sigmoid(i + w_i_diag * c_prev) * self._activation(j))
      else:
        c = (sigmoid(f + self._forget_bias) * c_prev + sigmoid(i) *
             self._activation(j))

      if self._cell_clip is not None:
        # pylint: disable=invalid-unary-operand-type
        c = clip_ops.clip_by_value(c, -self._cell_clip, self._cell_clip)
        # pylint: enable=invalid-unary-operand-type

      if self._use_peepholes:
        m = sigmoid(o + w_o_diag * c) * self._activation(c)
      else:
        m = sigmoid(o) * self._activation(c)

      if self._num_proj is not None:
        with vs.variable_scope("projection") as proj_scope:
          if self._num_proj_shards is not None:
            proj_scope.set_partitioner(
                partitioned_variables.fixed_size_partitioner(
                    self._num_proj_shards))
          m = _linear(m, self._num_proj, bias=False, scope=scope)

        if self._proj_clip is not None:
          # pylint: disable=invalid-unary-operand-type
          m = clip_ops.clip_by_value(m, -self._proj_clip, self._proj_clip)
          # pylint: enable=invalid-unary-operand-type

    new_state = (LSTMStateTuple(c, m) if self._state_is_tuple else
                 array_ops.concat([c, m], 1))
    return m, new_state
```
The key lines are:
```
c = (sigmoid(f + self._forget_bias) * c_prev + sigmoid(i) *
         self._activation(j))
```
and
```
m = sigmoid(o) * self._activation(c)
```
and
```
new_state = (LSTMStateTuple(c, m)
```
If you compare the code to compute c and m with the LSTM equations (see below), you can see it corresponds to the cell state (typically denoted with c) and hidden state (typically denoted with h), respectively: new_state = (LSTMStateTuple(c, m) indicates that the first element of the returned state tuple is c (cell state a.k.a. c_state), and the second element of the returned state tuple is m (hidden state a.k.a. m_state).


Question: How Can I Define Only the Gradient for a Tensorflow Subgraph? First: I am only a few days in with Tensorflow, so please bear with me. I started out from the cifar10 tutorial code and I am now using a combination of convolutions and eigenvalue decompositions that break the symbolic differentiation. I.e. the graph gets built, then upon calling train() the script halts with "No gradient defined for operation [...] (op type: SelfAdjointEig)". No surprise there. The inputs to the subgraph in question are still only the input feature maps and the filters being used, and I have the formulas for the gradients at hand and they should be straight-forward to implement given the inputs to the subgraph and the gradient with respect to its output. From what I can see in the docs, I can register a gradient method for custom Ops with RegisterGradient or override them with the experimental gradient_override_map. Both of those should give me access to exactly the things I need. For example, searching on Github I find a lot of examples that access the op's inputs as op.input[0] or such. The problem I have is that I want to essentially "shortcut" a whole subgraph, not a single op, so I have no single op to decorate. Since this is happening in one of the convolutional layers of the cifar example I tried using the scope object for that layer. Conceptually, what enters and exits that scope's graph is exactly what I want so if I could somehow override the whole scope's gradients that would "already" do it. I saw tf.Graph.create_op which (I think) I could use to register a new type of operation and I could then override that Operation type's gradient computation with aforementioned methods. But I don't see a way of defining that op's forward pass without writing it in C++... Maybe I am approaching this the wrong way entirely? Since all of my forward or backward operations can be implemented with the python interface I obviously want to avoid implementing anything in C++.

Response:
Here's a trick from Sergey Ioffe: Suppose you want group of ops that behave as f(x) in forward mode, but as g(x) in the backward mode. You implement it as
```
t = g(x)
y = t + tf.stop_gradient(f(x) - t)
```
So in your case your g(x) could be an identity op, with a custom gradient using gradient_override_map


Question:
Is gradient in the tensorflow's graph calculated incorrectly? A very simple example in tensorflow: min (x + 1)^2 where x is a scalar. The code is:
```py
import tensorflow as tf

x = tf.Variable(initial_value=3.0)
add = tf.add(x, 1)
y = tf.square(add)
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01)
train = optimizer.minimize(y)
```
Then write the graph to disk:
```py
graph = tf.get_default_graph()
writer = tf.summary.FileWriter("some/dir/to/write/events")
writer.add_graph(graph=graph)
```
Finally, visualizing it in tensorboard, it looks like this: The question is, why is the "Add" node connected with the gradients? I think since I am trying to minimize y, the "Square" node should be. Is this a bug? Can anyone explain it?

Response:
There is no bug involved. You just need to understand what is a gradient and know how to compute one yourself. So (x+1)^2' = 2*(x+1). Which means that you do not need to calculate (x+1)^2 to calculate the gradient. If you will zoom in the gradient part you will see that it calculated the gradient of your square and figured out that it does which part of the graph is needed there : Here is a more interesting and more intuitive example:
```py
import tensorflow as tf

x = tf.Variable(initial_value=3.0)
y = tf.cos(x)

train = tf.train.GradientDescentOptimizer(learning_rate=0.01).minimize(y)

with tf.Session() as sess:
    writer = tf.summary.FileWriter('logs', sess.graph)
    writer.close()
```
You should know that cos(x)' = - sin(x). Which means that only x is needed to calculate gradient. And this is what you see in the graph:


Question:
Tensorflow ValueError: No variables to save from I have written a tensorflow CNN and it is already trained. I wish to restore it to run it on a few samples but unfortunately its spitting out: ValueError: No variables to save My eval code can be found here:
```py
import tensorflow as tf

import main
import Process
import Input

eval_dir = "/Users/Zanhuang/Desktop/NNP/model.ckpt-30"
checkpoint_dir = "/Users/Zanhuang/Desktop/NNP/checkpoint"

init_op = tf.initialize_all_variables()
saver = tf.train.Saver()

def evaluate():
  with tf.Graph().as_default() as g:
    sess.run(init_op)

    ckpt = tf.train.get_checkpoint_state(checkpoint_dir)

    saver.restore(sess, eval_dir)

    images, labels = Process.eval_inputs(eval_data = eval_data)

    forward_propgation_results = Process.forward_propagation(images)

    top_k_op = tf.nn.in_top_k(forward_propgation_results, labels, 1)

    print(top_k_op)

def main(argv=None):
    evaluate()

if __name__ == '__main__':
  tf.app.run()
```

Response:
The tf.train.Saver must be created after the variables that you want to restore (or save). Additionally it must be created in the same graph as those variables. Assuming that Process.forward_propagation(…) also creates the variables in your model, adding the saver creation after this line should work:
```
forward_propgation_results = Process.forward_propagation(images)
```
In addition, you must pass the new tf.Graph that you created to the tf.Session constructor so you'll need to move the creation of sess inside that with block as well. The resulting function will be something like:
```py
def evaluate():
  with tf.Graph().as_default() as g:
    images, labels = Process.eval_inputs(eval_data = eval_data)
    forward_propgation_results = Process.forward_propagation(images)
    init_op = tf.initialize_all_variables()
    saver = tf.train.Saver()
    top_k_op = tf.nn.in_top_k(forward_propgation_results, labels, 1)

  with tf.Session(graph=g) as sess:
    sess.run(init_op)
    saver.restore(sess, eval_dir)
    print(sess.run(top_k_op))
```