Question:
How do I pass an OpenCV Mat into a C++ Tensorflow graph? In Tensorflow C++ I can load an image file into the graph using
```
tensorflow::Node* file_reader =  tensorflow::ops::ReadFile(tensorflow::ops::Const(IMAGE_FILE_NAME, b.opts()),b.opts().WithName(input_name));
tensorflow::Node* image_reader = tensorflow::ops::DecodePng(file_reader, b.opts().WithAttr("channels", 3).WithName("png_reader"));
tensorflow::Node* float_caster = tensorflow::ops::Cast(image_reader, tensorflow::DT_FLOAT, b.opts().WithName("float_caster"));
tensorflow::Node* dims_expander = tensorflow::ops::ExpandDims(float_caster, tensorflow::ops::Const(0, b.opts()), b.opts());
tensorflow::Node* resized = tensorflow::ops::ResizeBilinear(dims_expander, tensorflow::ops::Const({input_height, input_width},b.opts().WithName("size")),b.opts());
```
For an embedded application I would like to instead pass an OpenCV Mat into this graph. How would I convert the Mat to a tensor that could be used as input to tensorflow::ops::Cast or tensorflow::ops::ExpandDims?

Response:
It's not directly from a CvMat, but you can see an example of how to initialize a Tensor from an in-memory array in the TensorFlow Android example: https://github.com/tensorflow/tensorflow/blob/0.6.0/tensorflow/examples/android/jni/tensorflow_jni.cc#L173 You would start off by creating a new tensorflow::Tensor object, with something like this (all code untested): tensorflow::Tensor input_tensor(tensorflow::DT_FLOAT, tensorflow::TensorShape({1, height, width, depth})); This creates a Tensor object with float values, with a batch size of 1, and a size of widthxheight, and with depth channels. For example a 128 wide by 64 high image with 3 channels would pass in a shape of {1, 64, 128, 3}. The batch size is just used when you need to pass in multiple images in a single call, and for simple uses you can leave it as 1. Then you would get the underlying array behind the tensor using a line like this: auto input_tensor_mapped = input_tensor.tensor(); The input_tensor_mapped object is an interface to the data in your newly-created tensor, and you can then copy your own data into it. Here I'm assuming you've set source_data as a pointer to your source data, for example: const float* source_data = some_structure.imageData; You can then loop through your data and copy it over:
```
for (int y = 0; y < height; ++y) {
    const float* source_row = source_data + (y * width * depth);
    for (int x = 0; x < width; ++x) {
        const float* source_pixel = source_row + (x * depth);
        for (int c = 0; c < depth; ++c) {
           const float* source_value = source_pixel + c;
           input_tensor_mapped(0, y, x, c) = *source_value;
        }
    }
}
```
There are obvious opportunities to optimize this naive approach, and I don't have sample code on hand to show how to deal with the OpenCV side of getting the source data, but hopefully this is helpful to get you started.


Question:
How to find an index of the first matching element in TensorFlow I am looking for a TensorFlow way of implementing something similar to Python's list.index() function. Given a matrix and a value to find, I want to know the first occurrence of the value in each row of the matrix. For example,
```
m is a <batch_size, 100> matrix of integers
val = 23

result = [0] * batch_size
for i, row_elems in enumerate(m):
  result[i] = row_elems.index(val)
```
I cannot assume that 'val' appears only once in each row, otherwise I would have implemented it using tf.argmax(m == val). In my case, it is important to get the index of the first occurrence of 'val' and not any.

Response:
It seems that tf.argmax works like np.argmax (according to the test), which will return the first index when there are multiple occurrences of the max value. You can use tf.argmax(tf.cast(tf.equal(m, val), tf.int32), axis=1) to get what you want. However, currently the behavior of tf.argmax is undefined in case of multiple occurrences of the max value. If you are worried about undefined behavior, you can apply tf.argmin on the return value of tf.where as @Igor Tsvetkov suggested. For example,
```
# test with tensorflow r1.0
import tensorflow as tf

val = 3
m = tf.placeholder(tf.int32)
m_feed = [[0  ,   0, val,   0, val],
          [val,   0, val, val,   0],
          [0  , val,   0,   0,   0]]

tmp_indices = tf.where(tf.equal(m, val))
result = tf.segment_min(tmp_indices[:, 1], tmp_indices[:, 0])

with tf.Session() as sess:
    print(sess.run(result, feed_dict={m: m_feed})) # [2, 0, 1]
```
Note that tf.segment_min will raise InvalidArgumentError when there is some row containing no val. In your code row_elems.index(val) will raise exception too when row_elems don't contain val.


Question: What's the differences between tf.GraphKeys.TRAINABLE_VARIABLES and tf.GraphKeys.UPDATE_OPS in tensorflow? Here is doc of tf.GraphKeys in tensorflow, such as TRAINABLE_VARIABLES: the subset of Variable objects that will be trained by an optimizer. And i know tf.get_collection(), which can find some tensor that you want. When use tensorflow.contrib.layers.batch_norm(), the parameter updates_collections default value is GraphKeys.UPDATE_OPS. How can we understand those collections, and difference in them. Besides, we can find more in ops.py.

Response:
These are two different things. TRAINABLE_VARIABLES TRAINABLE_VARIABLES is the collection of variables or training parameters which should be modified when minimizing the loss. For example, these can be the weights determining the function performed by each node in the network. How do variables get added to this collection? This happens automatically when you define a new variable with tf.get_variable, unless you specify
```
tf.get_variable(..., trainable=False)
```
When would you want a variable to be untrainable? This happens from time to time. For example, occasionally you will want to use a two-step approach in which you first train the entire network on a large, generic dataset, then fine-tune the network on a smaller dataset which is specifically related to your problem. In such cases, you might want to fine-tune only part of the network, e.g., the last layer. Specifying some variables as untrainable is one of the ways to do this. UPDATE_OPS UPDATE_OPS is a collection of ops (operations performed when the graph runs, like multiplication, ReLU, etc.), not variables. Specifically, this collection maintains a list of ops which need to run before each training step. How do ops get added to this collection? By definition, update_ops occur outside the regular flow of training by loss minimization, so generally you will be adding ops to this collection only under special circumstances. For example, when performing batch normalization, you want to recompute the batch mean and variance before each training step, and this is how it's done. The mechanics of batch normalization using tf.contrib.layers.batch_norm are described in more detail in this article.


Question:
Purpose of using "with tf.Session()"? I am practicing the keras method called concatenate. And use of with statement in this example kind of got me thinking the purpose of this statement The example code looks like:
```
import numpy as np
import keras.backend as K
import tensorflow as tf

t1 = K.variable(np.array([ [[1, 2], [2, 3]], [[4, 4], [5, 3]]]))
t2 = K.variable(np.array([[[7, 4], [8, 4]], [[2, 10], [15, 11]]]))

d0 = K.concatenate([t1 , t2] , axis=-2)

init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)
    print(sess.run(d0))
```
Then I check document from: tensorflow and says that: A session may own resources, such as tf.Variable, tf.QueueBase, and tf.ReaderBase. It is important to release these resources when they are no longer required. To do this, either invoke the tf.Session.close method on the session, or use the session as a context manager. Which I believe has already explained all of it,but can somebody give me more intuitive explanation. Thanks in advance and have a nice day!

Response: tf.Session() initiates a TensorFlow Graph object in which tensors are processed through operations (or ops). The with block terminates the session as soon as the operations are completed. Hence, there is no need for calling Session.close. Also, a session contains variables, global variables, placeholders, and ops. These have to be initiated once the session is created. Hence we call tf.global_variables_initializer().run() A graph contains tensors and operations. To initiate a graph, a session is created which runs the graph. In other words, graph provides a schema whereas a session processes a graph to compute values( tensors ).


Question: Can we use Yolo to detect and recognize text in a image Currently I am using a deep learing model which is called "Yolov2" for object detection, and I want to use it to extract text and use save it in disk, but i don't know how to do that, if anyone know more about that, please advice me I use Tensorflow Thanks

Response: If you use the pretrained model, you would need to save those outputs and input the images into a character recognition network, if using neural net, or another approach. What you are doing is "scene text recognition". You can check out the Reading Text in the Wild with Convolutional Neural Networks paper, here's a demo and homepage. Github user chongyangtao has a whole list of resources on the topic.


Question: Where is the tensorflow session in Keras I'm new to keras and tensorflow. When I write programs with tensorflow, I must bulid a session to run the graph. However, when I use keras, although the backend is obviously tensorflow, I don't see session in the keras code. It seems all thing is done after the model.compile and model.fit. So, how does Keras work? where is the tensorflow session? and instead of session, can I use eager execution with keras? Thanks in advance and sorry for my English

Response:
Keras doesn't directly have a session because it supports multiple backends. Assuming you use TF as backend, you can get the global session as:
```
from keras import backend as K
sess = K.get_session()
```
If, on the other hand, yo already have an open Session and want to set it as the session Keras should use, you can do so via:
```
K.set_session(sess)
```


Question: what's the difference between tf.constant and tf.convert_to_tensor tf.to_float(tf.convert_to_tensor(python_object)) used many times in Tensorflow object detection api like grid_anchor_generator. normaly I'll use tf.constant(python_object, dtype=tf.float32). I'm wondering the difference between them. Thanks

Response: For tf.constant, the input value must be a static non-tensor type. For example a numpy array. For tf.convert_to_tensor, the value "an object whose type has a registered Tensor conversion function." This means input types like Tensors or tf.Variables can also be provided as inputs. For example, see the tensor conversion function for Variables here: https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/python/ops/variables.py#L762


Question:
Will dropout automatically be deactivated in Keras model? I have trained a model in Keras. The model contains dropout layers and I want to be absolutely sure nothing is dropped when doing this. According to the documentation, a layer's output can be extracted like this:
```
layer_name = 'my_layer'
intermediate_layer_model = Model(inputs=model.input,
                                 outputs=model.get_layer(layer_name).output)
intermediate_output = intermediate_layer_model.predict(data)
```
However, docs also show how to do so with a Keras function:
```
get_3rd_layer_output = K.function([model.layers[0].input, K.learning_phase()],
                                  [model.layers[3].output])

# output in test mode = 0
layer_output = get_3rd_layer_output([x, 0])[0]

# output in train mode = 1
layer_output = get_3rd_layer_output([x, 1])[0]
```
Here, the learning_phase() flag tells keras whether to actually use dropout and similar things that are only used during training. My question is, if I use the first approach, will dropout automatically be deactivated, or do I need to do something similar to setting the learning phase flag (as is done in the second approach).

Response: Yes, Model knows when it's training or testing, the flag is auto set when you call train() or predict().


Question:
how to convert logits to probability in binary classification in tensorflow?
```
logits= tf.matmul(inputs, weight) + bias
```
After matmul operation, the logits are two values derive from the MLP layer. My target is binary classification, how to convert the two values, logits, into probabilities, which include positive prob and negative prob and the sum of them is 1 ?

Response:
I am writing this answer for anyone who needs further clarifications: If it is a binary classification, it should be:
```
prediction = tf.round(tf.nn.sigmoid(logit))
```
If it is a multi-class classification:
```
prediction = tf.nn.softmax(logit)
```
then using the argmax function you can get the index of the class that has the highest probability score.
```
np.argmax(prediction, 0)
```


Question: using leaky relu in Tensorflow How can I change G_h1 = tf.nn.relu(tf.matmul(z, G_W1) + G_b1) to leaky relu? I have tried looping over the tensor using max(value, 0,01*value) but I get TypeError: Using a tf.Tensor as a Python bool is not allowed. I also tried to find the source code on relu on Tensorflow github so that I can modify it to leaky relu but I couldn't find it..

Response:
You could write one based on tf.relu, something like:
```
def lrelu(x, alpha):
  return tf.nn.relu(x) - alpha * tf.nn.relu(-x)
```
EDIT Tensorflow 1.4 now has a native tf.nn.leaky_relu.


Question:
TypeError: concat() got multiple values for argument 'axis' This is my convolution neural net:
```
def convolutional_neural_network(frame):
    wts = {'conv1': tf.random_normal([5, 5, 3, 32]),
            'conv2': tf.random_normal([5, 5, 32, 64]),
            'fc': tf.random_normal([158*117*64 + 4, 128]),
            'out': tf.random_normal([128, n_classes])
            }
    biases = {'fc': tf.random_normal([128]),
                'out': tf.random_normal([n_classes])
            }

    conv1 = conv2d(frame, wts['conv1'])
    # print(conv1)
    conv1 = maxpool2d(conv1)
    # print(conv1)
    conv2 = conv2d(conv1, wts['conv2'])
    conv2 = maxpool2d(conv2)
    # print(conv2)
    conv2 = tf.reshape(conv2, shape=[-1,158*117*64])
    print(conv2)
    print(controls_at_each_frame)
    conv2 = tf.concat(conv2, controls_at_each_frame, axis=1)
    fc = tf.add(tf.matmul(conv2, wts['fc']), biases['fc'])

    output = tf.nn.relu(tf.add(tf.matmul(fc, wts['out']), biases['out']))

    return output
```
where
```
frame = tf.placeholder('float', [None, 640-10, 465, 3])
controls_at_each_frame = tf.placeholder('float', [None, 4]) # [w, a, s, d] (1/0)
```
are the used placeholder. I am making a self driving car in GTA San Andreas. What I want to do is concatenate frame and controls_at_each_frame into a single layer which will be then sent to an fully connected layer. When I run I get an error TypeError: concat() got multiple values for argument 'axis' at
```
conv2 = tf.concat(conv2, controls_at_each_frame, axis=1)
```
Could you explain why this happening?

Response: Try conv2 = tf.concat((conv2, controls_at_each_frame), axis=1). Note I'm putting the two frames that you want to concatenate within parentheses, as specified here.


Question:
How to do a column sum in Tensorflow? What is the equivalent of the following in Tensorflow?
```
np.sum(A, axis=1)
```

Response:
There is tf.reduce_sum which is a bit more powerfull tool for doing so.
```
# 'x' is [[1, 1, 1]
#         [1, 1, 1]]
tf.reduce_sum(x) ==> 6
tf.reduce_sum(x, 0) ==> [2, 2, 2]
tf.reduce_sum(x, 1) ==> [3, 3]
tf.reduce_sum(x, 1, keep_dims=True) ==> [[3], [3]]
tf.reduce_sum(x, [0, 1]) ==> 6
```


Question:
tensorflow constant with variable size I have a variable batch size, so all of my inputs are of the form
```
tf.placeholder(tf.float32, shape=(None, ...)
```
to accept the variable batch sizes. However, how might you create a constant value with variable batch size? The issue is with this line:
```
log_probs = tf.constant(0.0, dtype=tf.float32, shape=[None, 1])
```
It is giving me an error:
```
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
```
I'm sure it is possible to initialize a constant tensor with variable batch size, how might I do so? I've also tried the following:
```
tf.constant(0.0, dtype=tf.float32, shape=[-1, 1])
```
I get this error:
```
ValueError: Too many elements provided. Needed at most -1, but received 1
```

Response:
A tf.constant() has fixed size and value at graph construction time, so it probably isn't the right op for your application. If you are trying to create a tensor with a dynamic size and the same (constant) value for every element, you can use tf.fill() and tf.shape() to create an appropriately-shaped tensor. For example, to create a tensor t that has the same shape as input and the value 0.5 everywhere:
```
input = tf.placeholder(tf.float32, shape=(None, ...))

# `tf.shape(input)` takes the dynamic shape of `input`.
t = tf.fill(tf.shape(input), 0.5)
```
As Yaroslav mentions in his comment, you may also be able to use (NumPy-style) broadcasting to avoid materializing a tensor with dynamic shape. For example, if input has shape (None, 32) and t has shape (1, 32) then computing tf.mul(input, t) will broadcast t on the first dimension to match the shape of input.


Question: Custom padding for convolutions in TensorFlow In tensorflow function tf.nn.conv2d, the padding option just has 'SAME' and 'VALID'. But in the conv layer of Caffe, there is pad option can define the number of pixels to (implicitly) add to each side of the input. How to achieve that in Tensorflow? Thank you very much.

Response:
You can use tf.pad() (see the doc) to pad the Tensor before applying tf.nn.conv2d(..., padding="VALID") (valid padding means no padding). For instance, if you want to pad the image with 2 pixels in height, and 1 pixel in width, and then apply a convolution with a 5x5 kernel:
```py
input = tf.placeholder(tf.float32, [None, 28, 28, 3])
padded_input = tf.pad(input, [[0, 0], [2, 2], [1, 1], [0, 0]], "CONSTANT")

filter = tf.placeholder(tf.float32, [5, 5, 3, 16])
output = tf.nn.conv2d(padded_input, filter, strides=[1, 1, 1, 1], padding="VALID")
```
output will have shape [None, 28, 26, 16], because you have only a padding of 1 in width.


Question:
How to rename a variable which respects the name scope? Given x, y are tensors, I know I can do
```
with tf.name_scope("abc"):
    z = tf.add(x, y, name="z")
```
So that z is named "abc/z". I am wondering if there exists a function f which assign the name directly in the following case:
```
with tf.name_scope("abc"):
    z = x + y
    f(z, name="z")
```
The stupid f I am using now is z = tf.add(0, z, name="z")

Response:
If you want to "rename" an op, there is no way to do that directly, because a tf.Operation (or tf.Tensor) is immutable once it has been created. The typical way to rename an op is therefore to use tf.identity(), which has almost no runtime cost:
```
with tf.name_scope("abc"):
    z = x + y
    z = tf.identity(z, name="z")
```
Note however that the recommended way to structure your name scope is to assign the name of the scope itself to the "output" from the scope (if there is a single output op):
```
with tf.name_scope("abc") as scope:
    # z will get the name "abc". x and y will have names in "abc/..." if they
    # are converted to tensors.
    z = tf.add(x, y, name=scope)
```
This is how the TensorFlow libraries are structured, and it tends to give the best visualization in TensorBoard.


Question:
Removing then Inserting a New Middle Layer in a Keras Model Given a predefined Keras model, I am trying to first load in pre-trained weights, then remove one to three of the models internal (non-last few) layers, and then replace it with another layer. I can't seem to find any documentation on keras.io about to do such a thing or remove layers from a predefined model at all. The model I am using is a good ole VGG-16 network which is instantiated in a function as shown below:
```
def model(self, output_shape):

    # Prepare image for input to model
    img_input = Input(shape=self._input_shape)

    # Block 1
    x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv1')(img_input)
    x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2')(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool')(x)

    # Block 2
    x = Conv2D(128, (3, 3), activation='relu', padding='same', name='block2_conv1')(x)
    x = Conv2D(128, (3, 3), activation='relu', padding='same', name='block2_conv2')(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block2_pool')(x)

    # Block 3
    x = Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv1')(x)
    x = Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv2')(x)
    x = Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv3')(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block3_pool')(x)

    # Block 4
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv1')(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv2')(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv3')(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block4_pool')(x)

    # Block 5
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv1')(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv2')(x)
    x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv3')(x)
    x = MaxPooling2D((2, 2), strides=(2, 2), name='block5_pool')(x)

    # Classification block
    x = Flatten(name='flatten')(x)
    x = Dense(4096, activation='relu', name='fc1')(x)
    x = Dropout(0.5)(x)
    x = Dense(4096, activation='relu', name='fc2')(x)
    x = Dropout(0.5)(x)
    x = Dense(output_shape, activation='softmax', name='predictions')(x)

    inputs = img_input

    # Create model.
    model = Model(inputs, x, name=self._name)

    return model
```
So as an example, I'd like to take the two Conv layers in Block 1 and replace them with just one Conv layer, after loading the original weights into all of the other layers. Any ideas?

Response:
Assuming that you have a model vgg16_model, initialized either by your function above or by keras.applications.VGG16(weights='imagenet'). Now, you need to insert a new layer in the middle in such a way that the weights of other layers will be saved. The idea is to disassemble the whole network to separate layers, then assemble it back. Here is the code specifically for your task:
```python
vgg_model = applications.VGG16(include_top=True, weights='imagenet')

# Disassemble layers
layers = [l for l in vgg_model.layers]

# Defining new convolutional layer.
# Important: the number of filters should be the same!
# Note: the receiptive field of two 3x3 convolutions is 5x5.
new_conv = Conv2D(filters=64,
                  kernel_size=(5, 5),
                  name='new_conv',
                  padding='same')(layers[0].output)

# Now stack everything back
# Note: If you are going to fine tune the model, do not forget to
#       mark other layers as un-trainable

x = new_conv
for i in range(3, len(layers)):
    layers[i].trainable = False
    x = layers[i](x)

# Final touch
result_model = Model(inputs=layer[0].input, outputs=x)
result_model.summary()
```
And the output of the above code is:
```python
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_50 (InputLayer)        (None, 224, 224, 3)       0
_________________________________________________________________
new_conv (Conv2D)            (None, 224, 224, 64)      1792
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 56, 56, 128)       0
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 56, 56, 256)       295168
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 56, 56, 256)       590080
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 56, 56, 256)       590080
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 28, 28, 256)       0
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 28, 28, 512)       1180160
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 28, 28, 512)       2359808
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 28, 28, 512)       2359808
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 14, 14, 512)       0
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 14, 14, 512)       2359808
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 14, 14, 512)       2359808
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 14, 14, 512)       2359808
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0
_________________________________________________________________
flatten (Flatten)            (None, 25088)             0
_________________________________________________________________
fc1 (Dense)                  (None, 4096)              102764544
_________________________________________________________________
fc2 (Dense)                  (None, 4096)              16781312
_________________________________________________________________
predictions (Dense)          (None, 1000)              4097000
=================================================================
Total params: 138,320,616
Trainable params: 1,792
Non-trainable params: 138,318,824
_________________________________________________________________
```


Question:
Running Adam Optimizer I am attempting to run an AdamOptimizer for one step of training, unsuccesfully.
```
optimizer = tf.train.AdamOptimizer(learning_rate)
init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)
    sess.run(optimizer.minimize(cost), feed_dict={X:X_data, Y: Y_data})
```
The console is spitting an ugly looking error:
```
FailedPreconditionError (see above for traceback): Attempting to use uninitialized value beta1_power
 [[Node: beta1_power/read = Identity[T=DT_FLOAT, _class=["loc:@W1"], _device="/job:localhost/replica:0/task:0/cpu:0"](beta1_power)]]
```
In the code, cost is a well defined function implementing a conv NN plus a logistic loss function, using two parameters X, Y (the entry of the NN and the training labels respectively) Any ideas on what could possibly be wrong?

Response:
optimizer.minimize(cost) is creating new values & variables in your graph. When you call sess.run(init) the variables that the .minimize method creates are not yet defined: from this your error. You just have to declare your minimization operation before invoking tf.global_variables_initializer():
```
optimizer = tf.train.AdamOptimizer(learning_rate)
minimize = optimizer.minimize(cost)
init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)
    sess.run(minimize, feed_dict={X:X_data, Y: Y_data})
```


Question:
What's the difference between tf.expand_dims and tf.newaxis in Tensorflow? Hi I am new to Tensorflow. I want to change the dimention of Tensor, and I found 3 types of method to implement this, like below:
```
a = tf.constant([[1,2,3],[4,5,6]]) # shape (2,3)

# change dimention of a to (2,3,1)
b = tf.expand_dims(a,2)   # shape(2,3,1)
c = a[:,:,tf.newaxis]     # shape(2,3,1)
d = tf.reshape(a,(2,3,1)) # shape(2,3,1)
```
Is there any difference among the 3 methods, e.g. in terms of performance? Which method should I use?

Response: There is no real difference between the three, but sometimes one or the other may be more convenient: tf.expand_dims(a, 2): Convenient when you want to add one dimension and its index is variable (for example the result of another TensorFlow operation, or some function parameter). Depending on your style you may find it more readable, since it clearly expresses the intention of adding a dimension. a[:,:,tf.newaxis]: Personally I use this a lot because I find it readable (maybe because I'm used to it from NumPy), although not in every case. Especially convenient if you want to add multiple dimensions (instead of calling tf.expand_dims multiple times). Also (obviously) if you want to take a slice and add new dimensions at the same time. However it is not usable with variable axis indices, and if you have many dimensions tf.expand_dims may be less confusing. tf.reshape(a,(2,3,1)): Personally I rarely or never use this to just add a dimension, because it requires me to know and specify all (or all but one) the remaining dimension sizes, and also it may be misleading when reading the code. However, if I need to reshape and add a dimension, I usually do it in the same operation.


Question:
Tensorflow "map operation" for tensor? I am adapting the cifar10 convolution example to my problem. I'd like to change the data input from a design that reads images one-at-a-time from a file to a design that operates on an already-in-memory set of images. The original inputs() function looks like this:
```
read_input = cifar10_input.read_cifar10(filename_queue)
reshaped_image = tf.cast(read_input.uint8image, tf.float32)
# Crop the central [height, width] of the image.
resized_image = tf.image.resize_image_with_crop_or_pad(reshaped_image,
                                                     width, height)
```
In the original version, read_input is a tensor containing one image. I keep all my images in RAM, so instead of using filename_queue, I have one huge images_tensor = tf.constant(images), where images_tensor.shape is (something, 32, 32, 3). My question is very-very basic: what is the best way to apply some function (tf.image.resize_image_with_crop_or_pad in my case) to all elements of images_tensor? Iterating is problematic in tensorflow, with limited slices(TensorFlow - numpy-like tensor indexing). Is there a solution to achieving this using just one command?

Response:
As of version 0.8 there is map_fn. From the documentation: map_fn(fn, elems, dtype=None, parallel_iterations=10, back_prop=True, swap_memory=False, name=None) map on the list of tensors unpacked from elems on dimension 0. This map operator repeatedly applies the callable fn to a sequence of elements from first to last. The elements are made of the tensors unpacked from elems. dtype is the data type of the return value of fn. Users must provide dtype if it is different from the data type of elems. Suppose that elems is unpacked into values, a list of tensors. The shape of the result tensor is [len(values)] + fn(values[0]).shape. Args: fn: The callable to be performed. elems: A tensor to be unpacked to apply fn. dtype: (optional) The output type of fn. parallel_iterations: (optional) The number of iterations allowed to run in parallel. back_prop: (optional) True enables back propagation. swap_memory: (optional) True enables GPU-CPU memory swapping. name: (optional) Name prefix for the returned tensors. Returns: A tensor that packs the results of applying fn to the list of tensors unpacked from elems, from first to last. Raises: TypeError: if fn is not callable. Example:
```
elems = [1, 2, 3, 4, 5, 6]
  squares = map_fn(lambda x: x * x, elems)
  # squares == [1, 4, 9, 16, 25, 36]
  ```
```


Question: How much faster is NCHW compared to NHWC in TensorFlow/cuDNN? The official TensorFlow performance guide states: Most TensorFlow operations used by a CNN support both NHWC and NCHW data format. On GPU, NCHW is faster. But on CPU, NHWC is sometimes faster. How much faster is NCHW compared to NHWC in TensorFlow/cuDNN, for convolution? Are there any references or benchmarks for this? Also, why is it faster? As I understand (see here), TensorFlow for NHWC on GPU will internally always transpose to NCHW, then calls the cuDNN conv kernel for NCHW, then transpose it back. But why does it do that? The cuDNN conv kernel also works for NHWC. Maybe at some point they did the comparison and the cuDNN conv kernel for NHWC was very slow. But is that up-to-date? And how big was the difference? What are the technical reasons that NHWC is so much slower? Or is the cuDNN kernel for this case just not well optimized?

Response: The reason is that most implementations of simple convolutions (not talking winograd or fft here), end up doing some kind of simple matrix multiplication, which means that in their inner loop they multiply some values from both tensors and sum the results. On a CPU implementation, using SSE or AVX optimization, it's faster to do this along the C dimension, because you just multiply-add the values 4 by 4 or 8 by 8, and then do the reduction (sum your 4 or 8 accumulations) at the end once you added all the C dimension. On a GPU however, doing a reduction across threads is a more costly operation (at least it was until Kepler introduced wrap-level atomic operations), so historically it has been optimized so that each thread in a wrap reads consecutive (in memory) HW values, and do the accumulation over parts of C with a loop. Note though that the latest nvidia cards (RTX), now have tensor multiplication cores, that can process small blocks in one operation, including the reduction over a small portion of C, so on these cards it's actually faster to use NHWC (or hybrid NCHWC formats).


Question: Why is tf.Variable uppercase but tf.constant lowercase? There is a little hard to guess why the case (I mean, upper case versus lower case) of constructors for variable, placeholder and constant is not the same, as below: tf.Variable(), tf.placeholder(), tf.constant(). What is the inherent difference between the variable method and the rest, that is start with an upper case letter?

Response: tf.constant() and tf.placeholder() are nodes in the graph (ops or operations). On the other hand tf.Variable() is a class. And in PEP8 python style guide: Class names should normally use the CapWords convention.


Question: What is the default variable initializer in Tensorflow? What is the default method of variable initialization used when tf.get_variable() is called without any specification for the initializer? The Docs just says 'None'.

Response: From the documentation: If initializer is None (the default), the default initializer passed in the variable scope will be used. If that one is None too, a glorot_uniform_initializer will be used. The glorot_uniform_initializer function initializes values from a uniform distribution. This function is documented as: The Glorot uniform initializer, also called Xavier uniform initializer. It draws samples from a uniform distribution within [-limit, limit], where limit is sqrt(6 / (fan_in + fan_out)) where fan_in is the number of input units in the weight tensor and fan_out is the number of output units in the weight tensor. Reference: http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf


Question:
Inconsistency between image resizing with Keras (PIL) and TensorFlow? I am bugged by an apparent inconsistency between: image resizing functionalities from keras.preprocessing, which are wrappers around PIL functions image resizing functions in TensorFlow's tf.image. I am training a deep learning model for a computer vision task with Keras (actually, with tf.keras, but that doesn't matter here). I then serve the model with TF Serving, which requires me to send images to the model as encoded byte-strings, where they are decoded using tf.image.decode_png before going through the model graph. The problem occurs when I resize the images. Resizing with bilinear interpolation (or any other method) gives different results with PIL compared to with tf.image, to such extent that the classification by the model changes depending on which function I use. The code below provides a reproducible example.
```
import numpy as np
from PIL import Image
from keras.preprocessing.image import load_img, img_to_array
import tensorflow as tf

# Generate an 'image' with numpy, save as png
np.random.seed(42)
image = np.random.randint(0, 255, size=(256, 256, 3)).astype(np.uint8)
Image.fromarray(image).convert("RGB").save('my_image.png')
```
Now, let's load the image in two ways. First with the PIL wrappers from Keras, as during the model training, then encoded as a binary string and decoded with TensorFlow functions, as in my model server.
```
# Using Keras PIL wrappers
keras_image = img_to_array(load_img('./my_image.png'))

# Using TF functionalities
with tf.Session() as sess:
    with open('./my_image.png', 'rb') as f:
        tf_image_ = tf.image.decode_png(f.read())
    tf_image = sess.run(tf_image_)
```
So far so good, as both images are exactly the same (apart from the dtype, as Keras has casted the image to float32):
```
# Assert equality
np.array_equal(keras_image, tf_image)
> True
```
Repeating this code with resizing however gives a different result:
```
# Using Keras PIL wrappers, with resizing
keras_image_rs = img_to_array(load_img('./my_image.png',
                             target_size=(224, 224),
                             interpolation='bilinear'))

# Using TF functionalities, with resizing
with tf.Session() as sess:
    with open('./my_image.png', 'rb') as f:
        tf_image_ = tf.image.decode_png(f.read())
        # Add and remove dimension
        # As tf.image.resize_* requires a batch dimension
        tf_image_ = tf.expand_dims(tf_image_, 0)
        tf_image_ = tf.image.resize_bilinear(tf_image_,
                                            [224, 224],
                                             align_corners=True)
        tf_image_ = tf.squeeze(tf_image_, axis=[0])

    tf_image_rs = sess.run(tf_image_)

# Assert equality
np.array_equal(keras_image_rs, tf_image_rs)
> False
```
The mean absolute difference between the two images is non-negligible:
```
np.mean(np.abs(keras_image_rs - tf_image_rs))
7.982703
```
I played with the align_corners argument, and tried other available interpolation methods as well. None give the same output as when resizing the image with PIL. This is quite annoying as it gives me a skew between training and testing results. Does anyone have an idea as to what causes this behavior, or on how to fix it?

Response: The behavior described completely matches what's written in this article: How Tensorflow’s tf.image.resize stole 60 days of my life In short: yes, PIL/sklearn/OpenCV and other common libraries for image manipulation have the correct behavior, while tf.image.resize has a different behavior that won't be changed in order to not break old trained models. Hence, you should always preprocess your image using the same library outside the computational graph. Link to the relevant github thread: https://github.com/tensorflow/tensorflow/issues/6720


Question:
Your kernel may have been built without NUMA support I have Jetson TX2, python 2.7, Tensorflow 1.5, CUDA 9.0 Tensorflow seems to be working but everytime, I run the program, I get this warning:
```
with tf.Session() as sess:
    print (sess.run(y,feed_dict))
```
...
```
2018-08-07 18:07:53.200320: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:881] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node Your kernel may have been built without NUMA support.

2018-08-07 18:07:53.200427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] Found device 0 with properties:

name: NVIDIA Tegra X2
major: 6
minor: 2
memoryClockRate(GHz): 1.3005

pciBusID: 0000:00:00.0

totalMemory: 7.66GiB
freeMemory: 1.79GiB

2018-08-07 18:07:53.200474: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)

2018-08-07 18:07:53.878574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:859] Could not identify NUMA node of /job:localhost/replica:0/task:0/device:GPU:0, defaulting to 0. Your kernel may not have been built with NUMA support.
```
Should I be worried? Or is it something negligible?

Response: It shouldn't be a problem for you, since you don't need NUMA support for this board (it has only one memory controller, so memory accesses are uniform). Also, I found this post on nvidia forum that seems to confirm this.


Question: How to deploy and serve prediction using TensorFlow from API? From google tutorial we know how to train a model in TensorFlow. But what is the best way to save a trained model, then serve the prediction using a basic minimal python api in production server. My question is basically for TensorFlow best practices to save the model and serve prediction on live server without compromising speed and memory issue. Since the API server will be running on the background for forever. A small snippet of python code will be appreciated.

Response: TensorFlow Serving is a high performance, open source serving system for machine learning models, designed for production environments and optimized for TensorFlow. The initial release contains C++ server and Python client examples based on gRPC. The basic architecture is shown in the diagram below. To get started quickly, check out the tutorial.


Question:
Use attribute and target matrices for TensorFlow Linear Regression Python I'm trying to follow this tutorial. TensorFlow just came out and I'm really trying to understand it. I'm familiar with penalized linear regression like Lasso, Ridge, and ElasticNet and its usage in scikit-learn. For scikit-learn Lasso regression, all I need to input into the regression algorithm is DF_X [an M x N dimensional attribute matrix (pd.DataFrame)] and SR_y [an M dimensional target vector (pd.Series)]. The Variable structure in TensorFlow is a bit new to me and I'm not sure how to structure my input data into what it wants. It seems as if softmax regression is for classification. How can I restructure my DF_X (M x N attribute matrix) and SR_y (M dimensional target vector) to input into tensorflow for linear regression? My current method for doing a Linear Regression uses pandas, numpy, and sklearn and it's shown below. I think this question will be really helpful for people getting familiar with TensorFlow:
```
#!/usr/bin/python
import pandas as pd
import numpy as np
import tensorflow as tf
from sklearn.linear_model import LassoCV

#Create DataFrames for attribute and target matrices
DF_X = pd.DataFrame(np.array([[0,0,1],[2,3,1],[4,5,1],[3,4,1]]),columns=["att1","att2","att3"],index=["s1","s2","s3","s4"])
SR_y = pd.Series(np.array([3,2,5,8]),index=["s1","s2","s3","s4"],name="target")

print DF_X
#att1  att2  att3
#s1     0     0     1
#s2     2     3     1
#s3     4     5     1
#s4     3     4     1

print SR_y
#s1    3
#s2    2
#s3    5
#s4    8
#Name: target, dtype: int64

#Create Linear Model (Lasso Regression)
model = LassoCV()
model.fit(DF_X,SR_y)

print model
#LassoCV(alphas=None, copy_X=True, cv=None, eps=0.001, fit_intercept=True,
#max_iter=1000, n_alphas=100, n_jobs=1, normalize=False, positive=False,
#precompute='auto', random_state=None, selection='cyclic', tol=0.0001,
#verbose=False)

print model.coef_
#[ 0.         0.3833346  0.       ]
```

Response:
Softmax is an only addition function (in logistic regression for example), it is not a model like
```
model = LassoCV()
model.fit(DF_X,SR_y)
```
Therefore you can't simply give it data with fit method. However, you can simply create your model with the help of TensorFlow functions. First of all, you have to create a computational graph, for example for linear regression you will create tensors with the size of your data. They are only tensors and you will give them your array in another part of the program.
```
import tensorflow as tf
x = tf.placeholder("float", [4, 3])
y_ = tf.placeholder("float",[4])
```
When you create two variables, that will contain initial weights of our model
```
W = tf.Variable(tf.zeros([3,1]))
b = tf.Variable(tf.zeros([1]))
```
And now you can create the model (you want to create regression, not classification therefore you don't need to use tf.nn.softmax )
```
y=tf.matmul(x,W) + b
```
As you have regression and linear model you will use
```
loss=tf.reduce_sum(tf.square(y_ - y))
```
Then we will train our model with the same step as in the tutorial
```
train_step = tf.train.GradientDescentOptimizer(0.01).minimize(loss)
```
Now that you created the computational graph you have to write one more part of the program, where you will use this graph to work with your data.
```
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
sess.run(train_step, feed_dict={x:np.asarray(DF_X),y_:np.asarray(SR_y)})
```
Here you give your data to this computational graph with the help of feed_dict. In TensorFlow you provide information in numpy arrays. If you want to see your mistake you can write
```
sess.run(loss,feed_dict={x:np.asarray(DF_X),y_:np.asarray(SR_y)})
```


Question:
TF 2.0 print tensor values I'm learning the newest release of Tensorflow (2.0) and I have tried to run a simple code to slice a matrix. Using the decorator @tf.function I made the following class:
```
class Data:
def __init__(self):
    pass

def back_to_zero(self, input):
    time = tf.slice(input, [0,0], [-1,1])
    new_time = time - time[0][0]
    return new_time

@tf.function
def load_data(self, inputs):
    new_x = self.back_to_zero(inputs)
    print(new_x)
```
So, when run the code using a numpy matrix, I can't retrieve the numbers.
```
time = np.linspace(0,10,20)
magntiudes = np.random.normal(0,1,size=20)
x = np.vstack([time, magntiudes]).T

d = Data()
d.load_data(x)
```
Output:
```
Tensor("sub:0", shape=(20, 1), dtype=float64)
```
I need to get this tensor in a numpy format, but TF 2.0 has not the class tf.Session to use run() or eval() methods. Thanks for any help that you can offer me!

Response:
Inside the graph indicated by the decorator @tf.function, you can use tf.print to print the values of your tensor.
```
tf.print(new_x)
```
Here is how the code can be rewritten
```
class Data:
    def __init__(self):
        pass

    def back_to_zero(self, input):
        time = tf.slice(input, [0,0], [-1,1])
        new_time = time - time[0][0]
        return new_time

    @tf.function
    def load_data(self, inputs):
        new_x = self.back_to_zero(inputs)
        tf.print(new_x) # print inside the graph context
        return new_x

time = np.linspace(0,10,20)
magntiudes = np.random.normal(0,1,size=20)
x = np.vstack([time, magntiudes]).T

d = Data()
data = d.load_data(x)
print(data) # print outside the graph context
```
the tensor type outside the tf.decorator context is of type tensorflow.python.framework.ops.EagerTensor. To convert it to a numpy array, you can use data.numpy()


Question:
Tensorflow `set_random_seed` not working [duplicate] This question already has answers here: Reproducible results in Tensorflow with tf.set_random_seed (9 answers) Closed 5 years ago. Calling tf.set_random_seed(SEED) has no effect that I can tell... For example, running the code below several times inside an IPython notebook produces different output each time:
```
import tensorflow as tf
tf.set_random_seed(42)
sess = tf.InteractiveSession()
a = tf.constant([1, 2, 3, 4, 5])
tf.initialize_all_variables().run()
a_shuf = tf.random_shuffle(a)
print(a.eval())
print(a_shuf.eval())
sess.close()
```
If I set the seed explicitly: a_shuf = tf.random_shuffle(a, seed=42), the output is the same after each run. But why do I need to set the seed if I already call tf.set_random_seed(42)? The equivalent code using numpy just works:
```
import numpy as np
np.random.seed(42)
a = [1,2,3,4,5]
np.random.shuffle(a)
print(a)
```

Response: That only sets the graph-level random seed. If you execute this snippet several times in a row, the graph will change, and two shuffle statements will get different operation-level seeds. The details are described in the doc string for set_random_seed To get deterministic a_shuf you can either Call tf.reset_default_graph() between invocations or Set operation-level seed for shuffle: a_shuf = tf.random_shuffle(a, seed=42)


Question: Is it meaningless to use ReduceLROnPlateau with Adam optimizer? This question is basically for the working of Keras or tf.keras for people who have the verty deep knowledge of the framework According to my knowledge, tf.keras.optimizers.Adam is an optimizer which has already an Adaptive Learning rate scheme. So if we are using from keras.callbacks.ReduceLROnPlateau with the Adam optimizer or any other, isn't it meaningless to do so? I don't have the very inner workings of Keras based Optimizer but it looks natural to me that if we are using the adaptive optimizer, why to to use this and If we use this given callback, what would be the effect on the training?

Response: Conceptually, consider the gradient a fixed, mathematical value from automatic differentiation. What every optimizer other than pure SGD does is to take the gradient and apply some statistical analysis to create a better gradient. In the simplest case, momentum, the gradient is averaged with previous gradients. In RMSProp, the variance of the gradient across batches is measured - the noisier it is, the less RMSProp "trusts" the gradient and so the gradient is reduced (divided by the stdev of the gradient for that weight). Adam does both. Then, all optimizers multiply the statistically adjusted gradient by a learning rate. So although one colloquial description of Adam is that it automatically tunes a learning rate... a more informative description is that Adam statistically adjusts gradients to be more reliable, but you still need to decide on a learning rate and how it changes during training (e.g. a LR policy). ReduceLROnPlateau, cosine decay, warmup, etc are examples of an LR policy. Whether you program TF or PyTorch, the psuedocode on PyTorch's optimizers are my go to to understand the optimizer algorithms. Looks like a wall of greek letters as first, but you'll grok it if you stare at it for a few minutes. https://pytorch.org/docs/stable/optim.html


Question: How to append data to TensorFlow tfrecords file How to append new data (e.g. pairs of images and labels) to an already existing tfrecord file? The class tf.python_io.TFRecordWriter does not seem to have any option for that. This question may also be reformulated to how to concatenate tfrecord files.

Response: According to the comments in the ticket I opened this won't be implemented, soon.


Question:
How to select half precision (BFLOAT16 vs FLOAT16) for your trained model? how will you decide what precision works best for your inference model? Both BF16 and F16 takes two bytes but they use different number of bits for fraction and exponent. Range will be different but I am trying to understand why one chose one over other. Thank you
```
|--------+------+----------+----------|
    | Format | Bits | Exponent | Fraction |
    |--------+------+----------+----------|
    | FP32   |   32 |        8 |       23 |
    | FP16   |   16 |        5 |       10 |
    | BF16   |   16 |        8 |        7 |
    |--------+------+----------+----------|

Range
bfloat16: ~1.18e-38 … ~3.40e38 with 3 significant decimal digits.
float16:  ~5.96e−8 (6.10e−5) … 65504 with 4 significant decimal digits precision.
```

Response: bfloat16 is generally easier to use, because it works as a drop-in replacement for float32. If your code doesn't create nan/inf numbers or turn a non-0 into a 0 with float32, then it shouldn't do it with bfloat16 either, roughly speaking. So, if your hardware supports it, I'd pick that. Check out AMP if you choose float16.


Question: Is TensorFlow suitable for Recommendation Systems [closed] Closed. This question is opinion-based. It is not currently accepting answers. Want to improve this question? Because this question may lead to opinionated discussion, debate, and answers, it has been closed. You may edit the question if you feel you can improve it so that it requires answers that include facts and citations or a detailed explanation of the proposed solution. If edited, the question will be reviewed and might be reopened. Closed 9 years ago. Improve this question I have read blogpost about TensorFlow is being open sourced. In the tutorials and the examples on the TensorFlow website, I see that they are mostly classification problems. (e.g. given an image, classify the number written in it) I am curious about it the software also suitable for solving problems in recommendation system? For example, is it good for solving problems on collaborative filtering / content-based filtering?

Response: Tensorflow is great for deep learning, or training large neural nets. Although, it can be used for several other mathematical applications such as PDEs, various classifiers, recommendation systems etc, there doesn't seem to have a lot of support for them as yet. This reddit thread might be a good place to start for searching libraries which are centred around recommendation systems.


Question:
Tensorflow - casting from int to float strange behavior I am working on tensorflow 0.12 and am having problem with casting. The following snippet of code does a strange thing:
```
sess = tf.InteractiveSession()
a = tf.constant(1)
b = tf.cast(a, tf.float32)
print b.eval()
```
I get a value: 6.86574233e-36 I also tried using tf.to_float() and tf.saturate_cast. Both gave the same result. Please help.

Response:
```
sess = tf.InteractiveSession()
a = tf.constant(1, tf.int64)  <--------
b = tf.cast(a, tf.float32)
print b.eval()  # 1.0
```
You need to declare the dtype for your tf.constant: https://www.tensorflow.org/api_docs/python/tf/constant


Question: Save Keras model at specific epochs I am using Keras to do some training on my dataset and it is time consuming to keep running every time to locate the number of epochs needed to get the best results. I tried using callbacks to get the best model, but it just does not work and usually stops too early. Also, saving every N epochs is not an option for me. What I am trying to do is save the model after some specific epochs are done. Let's say for example, after epoch = 150 is over, it will be saved as model.save(model_1.h5) and after epoch = 152, it will be saved as model.save(model_2.h5) etc... for few specific epochs. Is there a way to implement this in Keras ? I already searched for a method but no luck so far. Thank you for any help/suggestion.

Response:
Edit In most cases it's enough to use name formatting suggested by @Toan Tran in his answer. But if you need some sophisticated logic, you can use a callback, for example
```
import keras

class CustomSaver(keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs={}):
        if epoch == 2:  # or save after some epoch, each k-th epoch etc.
            self.model.save("model_{}.hd5".format(epoch))
```
on_epoch_end is called at the end of each epoch; epoch is a number of epoch, latter argument is a logs (you can read about other callback methods in docs). Put the logic into this method (in example it's as simple as possible). Create saver object and put it into fit method:
```
import keras
import numpy as np

inp = keras.layers.Input(shape=(10,))
dense = keras.layers.Dense(10, activation='relu')(inp)
out = keras.layers.Dense(1, activation='sigmoid')(dense)
model = keras.models.Model(inp, out)
model.compile(optimizer="adam", loss="binary_crossentropy",)

# Just a noise data for fast working example
X = np.random.normal(0, 1, (1000, 10))
y = np.random.randint(0, 2, 1000)

# create and use callback:
saver = CustomSaver()
model.fit(X, y, callbacks=[saver], epochs=5)
```
In the bash:
```
!ls
Out:
model_2.hd5
```
So, it works.


Question:
How does Keras 1d convolution layer work with word embeddings - text classification problem? (Filters, kernel size, and all hyperparameter) I am currently developing a text classification tool using Keras. It works (it works fine and I got up to 98.7 validation accuracy) but I can't wrap my head around about how exactly 1D-convolution layer works with text data. What hyper-parameters should I use? I have the following sentences (input data): Maximum words in the sentence: 951 (if it's less - the paddings are added) Vocabulary size: ~32000 Amount of sentences (for training): 9800 embedding_vecor_length: 32 (how many relations each word has in word embeddings) batch_size: 37 (it doesn't matter for this question) Number of labels (classes): 4 It's a very simple model (I have made more complicated structures but, strangely it works better - even without using LSTM):
```
model = Sequential()
model.add(Embedding(top_words, embedding_vecor_length, input_length=max_review_length))
model.add(Conv1D(filters=32, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(labels_count, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
print(model.summary())
```
My main question is: What hyper-parameters should I use for Conv1D layer?
```
model.add(Conv1D(filters=32, kernel_size=2, padding='same', activation='relu'))
```
If I have following input data: Max word count: 951 Word-embeddings dimension: 32 Does it mean that filters=32 will only scan first 32 words completely discarding the rest (with kernel_size=2)? And I should set filters to 951 (max amount of words in the sentence)? Examples on images: So for instance this is an input data: http://joxi.ru/krDGDBBiEByPJA It's the first step of a convoulution layer (stride 2): http://joxi.ru/Y2LB099C9dWkOr It's the second step (stride 2): http://joxi.ru/brRG699iJ3Ra1m And if filters = 32, layer repeats it 32 times? Am I correct? So I won't get to say 156-th word in the sentence, and thus this information will be lost?

Response:
I would try to explain how 1D-Convolution is applied on a sequence data. I just use the example of a sentence consisting of words but obviously it is not specific to text data and it is the same with other sequence data and timeseries. Suppose we have a sentence consisting of m words where each word has been represented using word embeddings: Now we would like to apply a 1D convolution layer consisting of n different filters with kernel size of k on this data. To do so, sliding windows of length k are extracted from the data and then each filter is applied on each of those extracted windows. Here is an illustration of what happens (here I have assumed k=3 and removed the bias parameter of each filter for simplicity): As you can see in the figure above, the response of each filter is equivalent to the result of its convolution (i.e. element-wise multiplication and then summing all the results) with the extracted window of length k (i.e. i-th to (i+k-1)-th words in the given sentence). Further, note that each filter has the same number of channels as the number of features (i.e. word-embeddings dimension) of the training sample (hence performing convolution, i.e. element-wise multiplication, is possible). Essentially, each filter is detecting the presence of a particular feature of pattern in a local window of training data (e.g. whether a couple of specific words exist in this window or not). After all the filters have been applied on all the windows of length k we would have an output of like this which is the result of convolution: As you can see, there are m-k+1 windows in the figure since we have assumed that the padding='valid' and stride=1 (default behavior of Conv1D layer in Keras). The stride argument determines how much the window should slide (i.e. shift) to extract the next window (e.g. in our example above, a stride of 2 would extract windows of words: (1,2,3), (3,4,5), (5,6,7), ... instead). The padding argument determines whether the window should entirely consists of the words in training sample or there should be paddings at the beginning and at the end; this way, the convolution response may have the same length (i.e. m and not m-k+1) as the training sample (e.g. in our example above, padding='same' would extract windows of words: (PAD,1,2), (1,2,3), (2,3,4), ..., (m-2,m-1,m), (m-1,m, PAD)). You can verify some of the things I mentioned using Keras:
```
from keras import models
from keras import layers

n = 32  # number of filters
m = 20  # number of words in a sentence
k = 3   # kernel size of filters
emb_dim = 100  # embedding dimension

model = models.Sequential()
model.add(layers.Conv1D(n, k, input_shape=(m, emb_dim)))

model.summary()
```
Model summary:
```
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv1d_2 (Conv1D)            (None, 18, 32)            9632
=================================================================
Total params: 9,632
Trainable params: 9,632
Non-trainable params: 0
_________________________________________________________________
```
As you can see the output of convolution layer has a shape of (m-k+1,n) = (18, 32) and the number of parameters (i.e. filters weights) in the convolution layer is equal to: num_filters * (kernel_size * n_features) + one_bias_per_filter = n * (k * emb_dim) + n = 32 * (3 * 100) + 32 = 9632.


Question:
Get the last output of a dynamic_rnn in TensorFlow I have a 3-D tensor of shape [batch, None, dim] where the second dimension, i.e. the timesteps, is unknown. I use dynamic_rnn to process such input, like in the following snippet:
```
import numpy as np
import tensorflow as tf

batch = 2
dim = 3
hidden = 4

lengths = tf.placeholder(dtype=tf.int32, shape=[batch])
inputs = tf.placeholder(dtype=tf.float32, shape=[batch, None, dim])
cell = tf.nn.rnn_cell.GRUCell(hidden)
cell_state = cell.zero_state(batch, tf.float32)
output, _ = tf.nn.dynamic_rnn(cell, inputs, lengths, initial_state=cell_state)
```
Actually, running this snipped with some actual numbers, I have some reasonable results:
```
inputs_ = np.asarray([[[0, 0, 0], [1, 1, 1], [2, 2, 2], [3, 3, 3]],
                    [[6, 6, 6], [7, 7, 7], [8, 8, 8], [9, 9, 9]]],
                    dtype=np.int32)
lengths_ = np.asarray([3, 1], dtype=np.int32)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    output_ = sess.run(output, {inputs: inputs_, lengths: lengths_})
    print(output_)
```
And the output is:
```
[[[ 0.          0.          0.          0.        ]
  [ 0.02188676 -0.01294564  0.05340237 -0.47148666]
  [ 0.0343586  -0.02243731  0.0870839  -0.89869428]
  [ 0.          0.          0.          0.        ]]

 [[ 0.00284752 -0.00315077  0.00108094 -0.99883419]
  [ 0.          0.          0.          0.        ]
  [ 0.          0.          0.          0.        ]
  [ 0.          0.          0.          0.        ]]]
```
Is there a way to get a 3-D tensor of shape [batch, 1, hidden] with the last relevant output of the dynamic RNN? Thanks!

Response:
This is what gather_nd is for!
```
def extract_axis_1(data, ind):
    """
    Get specified elements along the first axis of tensor.
    :param data: Tensorflow tensor that will be subsetted.
    :param ind: Indices to take (one for each element along axis 0 of data).
    :return: Subsetted tensor.
    """

    batch_range = tf.range(tf.shape(data)[0])
    indices = tf.stack([batch_range, ind], axis=1)
    res = tf.gather_nd(data, indices)

    return res
```
In your case:
```
output = extract_axis_1(output, lengths - 1)
```
Now output is a tensor of dimension [batch_size, num_cells].


Question:
Keras input_shape for conv2d and manually loaded images I am manually creating my dataset from a number of 384x286 b/w images. I load an image like this:
```
x = []
for f in files:
        img = Image.open(f)
        img.load()
        data = np.asarray(img, dtype="int32")
        x.append(data)
x = np.array(x)
```
this results in x being an array (num_samples, 286, 384)
```
print(x.shape) => (100, 286, 384)
```
reading the keras documentation, and checking my backend, i should provide to the convolution step an input_shape composed by ( rows, cols, channels ) since i don't arbitrarily know the sample size, i would have expected to pass as an input size, something similar to
```
( None, 286, 384, 1 )
```
the model is built as follows:
```
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))
# other steps...
```
passing as input_shape (286, 384, 1) causes: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (85, 286, 384) passing as_input_shape (None, 286, 384, 1 ) causes: Input 0 is incompatible with layer conv2d_1: expected ndim=4, found ndim=5 what am i doing wrong ? how do i have to reshape the input array?

Response:
Set the input_shape to (286,384,1). Now the model expects an input with 4 dimensions. This means that you have to reshape your image with .reshape(n_images, 286, 384, 1). Now you have added an extra dimension without changing the data and your model is ready to run. Basically, you need to reshape your data to (n_images, x_shape, y_shape, channels). The cool thing is that you also can use an RGB-image as input. Just change channels to 3. Check also this answer: Keras input explanation: input_shape, units, batch_size, dim, etc Example
```
import numpy as np
from keras.models import Sequential
from keras.layers.convolutional import Convolution2D
from keras.layers.core import Flatten, Dense, Activation
from keras.utils import np_utils

#Create model
model = Sequential()
model.add(Convolution2D(32, kernel_size=(3, 3), activation='relu', input_shape=(286,384,1)))
model.add(Flatten())
model.add(Dense(2))
model.add(Activation('softmax'))

model.compile(loss='binary_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])

#Create random data
n_images=100
data = np.random.randint(0,2,n_images*286*384)
labels = np.random.randint(0,2,n_images)
labels = np_utils.to_categorical(list(labels))

#add dimension to images
data = data.reshape(n_images,286,384,1)

#Fit model
model.fit(data, labels, verbose=1)
```


Question:
How can I convert TFRecords into numpy arrays? The main idea is to convert TFRecords into numpy arrays. Assume that the TFRecord stores images. Specifically: Read a TFRecord File and convert each image into a numpy array. Write the image into 1.jpg, 2.jpg, etc. At the same time, write the file name and label to the text file like this:
```
1.jpg 2
2.jpg 4
3.jpg 5
```
I currently use the following code:
```python
import tensorflow as tf
import os

def read_and_decode(filename_queue):
  reader = tf.TFRecordReader()
  _, serialized_example = reader.read(filename_queue)
  features = tf.parse_single_example(
      serialized_example,
      # Defaults are not specified since both keys are required.
      features={
          'image_raw': tf.FixedLenFeature([], tf.string),
          'label': tf.FixedLenFeature([], tf.int64),
          'height': tf.FixedLenFeature([], tf.int64),
          'width': tf.FixedLenFeature([], tf.int64),
          'depth': tf.FixedLenFeature([], tf.int64)
      })
  image = tf.decode_raw(features['image_raw'], tf.uint8)
  label = tf.cast(features['label'], tf.int32)
  height = tf.cast(features['height'], tf.int32)
  width = tf.cast(features['width'], tf.int32)
  depth = tf.cast(features['depth'], tf.int32)
  return image, label, height, width, depth

with tf.Session() as sess:
  filename_queue = tf.train.string_input_producer(["../data/svhn/svhn_train.tfrecords"])
  image, label, height, width, depth = read_and_decode(filename_queue)
  image = tf.reshape(image, tf.pack([height, width, 3]))
  image.set_shape([32,32,3])
  init_op = tf.initialize_all_variables()
  sess.run(init_op)
  print (image.eval())
```
I'm just reading trying to get at least one image for starters. The code just gets stuck when I run this.

Response:
Oops, it was a silly mistake on my part. I used a string_input_producer but forgot to run the queue_runners.
```python
with tf.Session() as sess:
  filename_queue = tf.train.string_input_producer(["../data/svhn/svhn_train.tfrecords"])
  image, label, height, width, depth = read_and_decode(filename_queue)
  image = tf.reshape(image, tf.pack([height, width, 3]))
  image.set_shape([32,32,3])
  init_op = tf.initialize_all_variables()
  sess.run(init_op)
  coord = tf.train.Coordinator()
  threads = tf.train.start_queue_runners(coord=coord)
  for i in range(1000):
    example, l = sess.run([image, label])
    print (example,l)
  coord.request_stop()
  coord.join(threads)
```


Question:
What are the uses of TimeDistributed wrapper for LSTM or any other layers I am trying to understand the use of TimeDistributed layer in keras/tensorflow. I have read some threads and articles but still I didn't get it properly. The threads that gave me some understanding of what the TImeDistributed layer does are - What is the role of TimeDistributed layer in Keras? TimeDistributed(Dense) vs Dense in Keras - Same number of parameters But I still don't know why the layer is actually used! For example, both the below codes will provide same output (& output_shape):
```
model = Sequential()
model.add(TimeDistributed(LSTM(5, input_shape = (10, 20), return_sequences = True)))
print(model.output_shape)

model = Sequential()
model.add(LSTM(5, input_shape = (10, 20), return_sequences = True))
print(model.output_shape)
```
And the output shape will be (according to my knowledge) -
```
(None, 10, 5)
```
So, if both the models provide same output, what is actually the use of TimeDistributed Layer? And I also had one other question. TimeDistributed layer applies time related data to separate layers (sharing same weights). So, how is it different from unrolling the LSTM layer which is provided in keras API as: unroll: Boolean (default False). If True, the network will be unrolled, else a symbolic loop will be used. Unrolling can speed-up a RNN, although it tends to be more memory-intensive. Unrolling is only suitable for short sequences. What is the difference between these two? Thank you.. I am still a newbie and so have many questions.

Response: As Keras documentation suggests TimeDistributed is a wrapper that applies a layer to every temporal slice of an input. Here is an example which might help: Let's say that you have video samples of cats and your task is a simple video classification problem, returning 0 if the cat is not moving or 1 if the cat is moving. Let's assume your input dim is (None, 50, 25, 25, 3) which means you have 50 time steps or frames per sample, and your frames are 25 by 25 and have 3 channels, rgb. Well, one aporoach would be to extract some "features" from each frame using CNN, like Conv2D, and then pass them to an LSTM layer. But the feature extraction would be the same for each frame. Now TimeDistributed comes to the rescue. You can wrap your Conv2D with it, then pass the output to a Flatten layer wrapped also by TimeDistributed. So after applying TimeDistributed(Conv2D(...)), the output would be something of dim like (None, 50, 5, 5, 16), and after TimeDistributed(Flatten()), the output would be of dim (None, 50, 400). (The actual dim would depend on Conv2D parameters.) The output at this layer now can be passes through LSTM. So obviously, LSTM itself does not need a TimeDistributed wrapper.


Question: significance of "trainable" and "training" flag in tf.layers.batch_normalization What is the significance of "trainable" and "training" flag in tf.layers.batch_normalization? How are these two different during training and prediction?

Response:
The batch norm has two phases:
```
1. Training:
   -  Normalize layer activations using `moving_avg`, `moving_var`, `beta` and `gamma`
     (`training`* should be `True`.)
   -  update the `moving_avg` and `moving_var` statistics.
     (`trainable` should be `True`)
2. Inference:
   -  Normalize layer activations using `beta` and `gamma`.
      (`training` should be `False`)
```
Example code to illustrate few cases:
```
#random image
img = np.random.randint(0,10,(2,2,4)).astype(np.float32)

# batch norm params initialized
beta = np.ones((4)).astype(np.float32)*1 # all ones
gamma = np.ones((4)).astype(np.float32)*2 # all twos
moving_mean = np.zeros((4)).astype(np.float32) # all zeros
moving_var = np.ones((4)).astype(np.float32) # all ones

#Placeholders for input image
_input = tf.placeholder(tf.float32, shape=(1,2,2,4), name='input')

#batch Norm
out = tf.layers.batch_normalization(
       _input,
       beta_initializer=tf.constant_initializer(beta),
       gamma_initializer=tf.constant_initializer(gamma),
       moving_mean_initializer=tf.constant_initializer(moving_mean),
       moving_variance_initializer=tf.constant_initializer(moving_var),
       training=False, trainable=False)

update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
init_op = tf.global_variables_initializer()

 ## 2. Run the graph in a session

 with tf.Session() as sess:

    # init the variables
    sess.run(init_op)

    for i in range(2):
        ops, o = sess.run([update_ops, out], feed_dict={_input: np.expand_dims(img, 0)})
        print('beta', sess.run('batch_normalization/beta:0'))
        print('gamma', sess.run('batch_normalization/gamma:0'))
        print('moving_avg',sess.run('batch_normalization/moving_mean:0'))
        print('moving_variance',sess.run('batch_normalization/moving_variance:0'))
        print('out', np.round(o))
        print('')
```
When training=False and trainable=False:
```
img = [[[4., 5., 9., 0.]...
  out = [[ 9. 11. 19.  1.]...
  The activation is scaled/shifted using gamma and beta.
```
When training=True and trainable=False:
```
out = [[ 2.  2.  3. -1.] ...
  The activation is normalized using `moving_avg`, `moving_var`, `gamma` and `beta`.
  The averages are not updated.
```
When traning=True and trainable=True:
```
The out is same as above, but the `moving_avg` and `moving_var` gets updated to new values.

  moving_avg [0.03249997 0.03499997 0.06499994 0.02749997]
  moving_variance [1.0791667 1.1266665 1.0999999 1.0925]
```


Question:
How to explicitly broadcast a tensor to match another's shape in tensorflow? I have three tensors, A, B and C in tensorflow, A and B are both of shape (m, n, r), C is a binary tensor of shape (m, n, 1). I want to select elements from either A or B based on the value of C. The obvious tool is tf.select, however that does not have broadcasting semantics, so I need to first explicitly broadcast C to the same shape as A and B. This would be my first attempt at how to do this, but it doesn't like me mixing a tensor (tf.shape(A)[2]) into the shape list.
```
import tensorflow as tf
A = tf.random_normal([20, 100, 10])
B = tf.random_normal([20, 100, 10])
C = tf.random_normal([20, 100, 1])
C = tf.greater_equal(C, tf.zeros_like(C))

C = tf.tile(C, [1,1,tf.shape(A)[2]])
D = tf.select(C, A, B)
```
What's the correct approach here?

Response:
EDIT: In all versions of TensorFlow since 0.12rc0, the code in the question works directly. TensorFlow will automatically stack tensors and Python numbers into a tensor argument. The solution below using tf.pack() is only needed in versions prior to 0.12rc0. Note that tf.pack() was renamed to tf.stack() in TensorFlow 1.0. Your solution is very close to working. You should replace the line:
```
C = tf.tile(C, [1,1,tf.shape(C)[2]])
```
...with the following:
```
C = tf.tile(C, tf.pack([1, 1, tf.shape(A)[2]]))
```
(The reason for the issue is that TensorFlow won't implicitly convert a list of tensors and Python literals into a tensor. tf.pack() takes a list of tensors, so it will convert each of the elements in its input (1, 1, and tf.shape(C)[2]) to a tensor. Since each element is a scalar, the result will be a vector.)


Question:
How to apply data augmentation in TensorFlow 2.0 after tfds.load() I'm following this guide. It shows how to download datasets from the new TensorFlow Datasets using tfds.load() method:
```
import tensorflow_datasets as tfds
SPLIT_WEIGHTS = (8, 1, 1)
splits = tfds.Split.TRAIN.subsplit(weighted=SPLIT_WEIGHTS)

(raw_train, raw_validation, raw_test), metadata = tfds.load(
    'cats_vs_dogs', split=list(splits),
    with_info=True, as_supervised=True)
```
The next steps shows how to apply a function to each item in the dataset using map method:
```
def format_example(image, label):
    image = tf.cast(image, tf.float32)
    image = image / 255.0
    # Resize the image if required
    image = tf.image.resize(image, (IMG_SIZE, IMG_SIZE))
    return image, label

train = raw_train.map(format_example)
validation = raw_validation.map(format_example)
test = raw_test.map(format_example)
```
Then to access the elements we can use:
```
for features in ds_train.take(1):
  image, label = features["image"], features["label"]
```
OR
```
for example in tfds.as_numpy(train_ds):
  numpy_images, numpy_labels = example["image"], example["label"]
```
However, the guide doesn't mention anything about data augmentation. I want to use real time data augmentation similar to that of Keras's ImageDataGenerator Class. I tried using:
```
if np.random.rand() > 0.5:
    image = tf.image.flip_left_right(image)
```
and other similar augmentation functions in format_example() but, how can I verify that it's performing real time augmentation and not replacing the original image in the dataset? I could convert the complete dataset to Numpy array by passing batch_size=-1 to tfds.load() and then use tfds.as_numpy() but, that would load all the images in memory which is not needed. I should be able to use train = train.prefetch(tf.data.experimental.AUTOTUNE) to load just enough data for next training loop.

Response:
You are approaching the problem from a wrong direction. First, download data using tfds.load, cifar10 for example (for simplicity we will use default TRAIN and TEST splits):
```
import tensorflow_datasets as tfds

dataloader = tfds.load("cifar10", as_supervised=True)
train, test = dataloader["train"], dataloader["test"]
```
(you can use custom tfds.Split objects to create validations datasets or other, see documentation) train and test are tf.data.Dataset objects so you can use map, apply, batch and similar functions to each of those. Below is an example, where I will (using tf.image mostly): convert each image to tf.float64 in the 0-1 range (don't use this stupid snippet from official docs, this way ensures correct image format) cache() results as those can be re-used after each repeat randomly flip left_to_right each image randomly change contrast of image shuffle data and batch IMPORTANT: repeat all the steps when dataset is exhausted. This means that after one epoch all of the above transformations are applied again (except for the ones which were cached). Here is the code doing the above (you can change lambdas to functors or functions):
```
train = train.map(
    lambda image, label: (tf.image.convert_image_dtype(image, tf.float32), label)
).cache().map(
    lambda image, label: (tf.image.random_flip_left_right(image), label)
).map(
    lambda image, label: (tf.image.random_contrast(image, lower=0.0, upper=1.0), label)
).shuffle(
    100
).batch(
    64
).repeat()
```
Such tf.data.Dataset can be passed directly to Keras's fit, evaluate and predict methods. Verifying it actually works like that I see you are highly suspicious of my explanation, let's go through an example: 1. Get small subset of data Here is one way to take a single element, admittedly unreadable and unintuitive, but you should be fine with it if you do anything with Tensorflow:
```
# Horrible API is horrible
element = tfds.load(
    # Take one percent of test and take 1 element from it
    "cifar10",
    as_supervised=True,
    split=tfds.Split.TEST.subsplit(tfds.percent[:1]),
).take(1)
```
2. Repeat data and check whether it is the same: Using Tensorflow 2.0 one can actually do it without stupid workarounds (almost):
```
element = element.repeat(2)
# You can iterate through tf.data.Dataset now, finally...
images = [image[0] for image in element]
print(f"Are the same: {tf.reduce_all(tf.equal(images[0], images[1]))}")
```
And it unsurprisingly returns:
```
Are the same: True
```
3. Check whether data differs after each repeat with random augmentation Below snippet repeats single element 5 times and checks which are equal and which are different.
```
element = (
    tfds.load(
        # Take one percent of test and take 1 element
        "cifar10",
        as_supervised=True,
        split=tfds.Split.TEST.subsplit(tfds.percent[:1]),
    )
    .take(1)
    .map(lambda image, label: (tf.image.random_flip_left_right(image), label))
    .repeat(5)
)

images = [image[0] for image in element]

for i in range(len(images)):
    for j in range(i, len(images)):
        print(
            f"{i} same as {j}: {tf.reduce_all(tf.equal(images[i], images[j]))}"
        )
```
Output (in mine case, each run would be different):
```
0 same as 0: True
0 same as 1: False
0 same as 2: True
0 same as 3: False
0 same as 4: False
1 same as 1: True
1 same as 2: False
1 same as 3: True
1 same as 4: True
2 same as 2: True
2 same as 3: False
2 same as 4: False
3 same as 3: True
3 same as 4: True
4 same as 4: True
```
You could cast each of those images to numpy as well and see the images for yourself using skimage.io.imshow, matplotlib.pyplot.imshow or other alternatives. Another example of visualization of real-time data augmentation This answer provides a more comprehensive and readable view on data augmentation using Tensorboard and MNIST, might want to check that one out (yeah, shameless plug, but useful I guess).


Question:
Tensorflow: Hierarchical Softmax Implementation I'm currently having text inputs represented by vector, and I want to classify their categories. Because they are multi-level categories, I meant to use Hierarchical Softmax.
```
Example:

 - Computer Science
     - Machine Learning
     - NLP
 - Economics
 - Maths
     - Algebra
     - Geometry
```
I don't know how to implement it in Tensorflow. All examples I've met is using other frameworks. Thanks

Response: Practically if your total number of categories is in the range of hundreds to thousands (less than 50K), you don't need to consider using hierarchical softmax, which is designed to run training faster for classifying into millions of categories (for example, the number of words in a vocabulary). In my experience (with Naive Bayesian and neural networks), utilizing the hierarchical structure at training time does not necessarily improve your classification quality. However, if you are interested to implement Hierarchical Softmax anyway, that's another story.


Question: Input to LSTM network tensorflow I have a time series of length t (x0, ...,xt) each of the xi is a d-dimension vector i.e. xi=(x0i, x1i, ...., xdi). Thus my input X is of shape [batch_size, d] The input for the tensorflow LSTM should be of size [batchSize, hidden_size]. My question is how should i input my time series to the LSTM. One possible solution that i thought of is to have additional weight matrix, W, of size [d,hidden_size] and to input the LSTM with X*W + B. Is this correct or should i input something else to the netwoרk? Thanks

Response: Your intuition is correct; what you need (and what you have described) is an embedding to translate your input vector to the dimension of your LSTM's input. There are three primary ways that I know of to accomplish that. You could do this manually with an additional weight matrix W and bias vector b as you described. You could create the weight matrix and bias vectors automatically using the linear() function from TensorFlow's rnn_cell.py library. Then pass the output of that linear layer as the input of your LSTM when you create your LSTM via the rnn_decoder() function in Tensorflow's seq2seq.py library or otherwise. Or you could have Tensorflow create this embedding and hook it up to the inputs of your LSTM automatically, by creating the LSTM via the embedding_rnn_decoder() function at line 141 of the same seq2seq library. (If you trace through the code for this function without any optional arguments, you'll see that it is simply creating a linear embedding layer for the input as well as the LSTM and hooking them together.) Unless you need access to the individual components that you're creating for some reason, I would recommend the third option to keep your code at a high level.


Question:
Use tf.scatter_update in a two dimensional tf.Variable I' m following this Manipulating matrix elements in tensorflow. using tf.scatter_update. But my problem is: What happens if my tf.Variable is 2D? Let's say:
```
a = tf.Variable(initial_value=[[0, 0, 0, 0],[0, 0, 0, 0]])
```
How can i update for example the first element of every row and assign to that the value 1? I tried something like
```
for line in range(2):
    sess.run(tf.scatter_update(a[line],[0],[1]))
```
but it fails (i was expecting that) and gives me the error: TypeError: Input 'ref' of 'ScatterUpdate' Op requires l-value input How can i fix that kind of problems? `

Response:
In tensorflow you cannot update a Tensor but you can update a Variable. The scatter_update operator can update only the first dimension of the variable. You have to pass always a reference tensor to the scatter update (a instead of a[line]). This is how you can update the first element of the variable:
```
import tensorflow as tf

g = tf.Graph()
with g.as_default():
    a = tf.Variable(initial_value=[[0, 0, 0, 0],[0, 0, 0, 0]])
    b = tf.scatter_update(a, [0, 1], [[1, 0, 0, 0], [1, 0, 0, 0]])

with tf.Session(graph=g) as sess:
   sess.run(tf.initialize_all_variables())
   print sess.run(a)
   print sess.run(b)
```
Output:
```
[[0 0 0 0]
 [0 0 0 0]]
[[1 0 0 0]
 [1 0 0 0]]
```
But having to change again the whole tensor it might be faster to just assign a completely new one.


Question: Tensorflow warning: The graph couldn't be sorted in topological order? When I run tensorflow training (with custom defined graph, closed source), it outputs the warning: 2018-10-03 14:29:24.352895: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 0, topological sort failed with message: The graph couldn't be sorted in topological order. What does it mean? What could likely cause this problem and how to avoid it? Update: For the record, in my case, tensorflow still works correctly despite this warning. So I think it just means there are more loops in the computation graph than what tensorflow expects, not necessarily infinite loop. I fixed this by avoid using manual loop in code, but instead using tensor manipulation (stack, concat, slice, reshape...)

Response: A topological ordering of a directed graph is an ordering of its vertices in such a way that whenever there is an edge from vertex u to vertex v, vertex u comes before vertex v in the ordering. This kind of ordering is possible for every directed acyclic graph but not for arbitrary graphs. There is most likely some kind of cycle in your graph that prevents the sorting algorithm from succeeding. So, the way to go is searching for the cycle and removing it in some way. As an example, consider a very small graph with two vertices, u and v, and two edges, u -> v and v -> u. There is no way to sort u and v in accordance with the requirements given above. Without further information on your code it is hard to say exactly what is going on.


Question:
Is it possible to visualize a tensorflow graph without a training op? I know how to visualize a tensorflow graph after training with tensorboard. Now, is it possible to visualize just the forward part of the graph, i.e., with no training operator defined? The reason I'm asking this is that I'm getting this error:
```
No gradients provided for any variable, check your graph for ops that do not support gradients, between variables [ ... list of model variables here ... ] and loss Tensor("Mean:0", dtype=float32).
```
I'd like to inspect the graph to find out where the gradient tensor flow (pun intended) is broken.

Response:
Yes, you can visualize any graph. Try this simple script:
```
import tensorflow as tf

a = tf.add(1, 2, name="Add_these_numbers")
b = tf.multiply(a, 3)
c = tf.add(4, 5, name="And_These_ones")
d = tf.multiply(c, 6, name="Multiply_these_numbers")
e = tf.multiply(4, 5, name="B_add")
f = tf.div(c, 6, name="B_mul")
g = tf.add(b, d)
h = tf.multiply(g, f)

with tf.Session() as sess:
    writer = tf.summary.FileWriter("output", sess.graph)
    print(sess.run(h))
    writer.close()
```
Then run...
```
tensorboard --logdir=output
```
... and you'll see: So you can simply create a session just to write the graph to the FileWriter and not do anything else.


Question: Holding variables constant during optimizer I have a TensorFlow computational graph for a loss tensor L that depends on 2 tf.Variables, A and B. I'd like to run gradient ascent on variable A (A+=gradient of L wrt A) while holding B fixed, and vice versa - running gradient ascent on B (B+=gradient of L wrt B) while holding A fixed. How do I do this?

Response:
tf.stop_gradient(tensor) might be what you are looking for. The tensor will be treated as constant for gradient computation purposes. You can create two losses with different parts treated as constants. The other option (and often better) would be to create 2 optimizers but explicitly optimize only subsets of variables, e.g.
```py
train_a = tf.train.GradientDescentOptimizer(0.1).minimize(loss_a, var_list=[A])
train_b = tf.train.GradientDescentOptimizer(0.1).minimize(loss_b, var_list=[B])
```
and you can iterate between them on the updates.


Question:
How tf.transpose works in tensorflow?
```
tf.transpose(a, perm=None, name='transpose')
```
transposes a. It permutes the dimensions according to perm. So if I am using this matrix to transform:
```
import tensorflow as tt
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"]="3"
import numpy as bb
ab=([[[1,2,3],[6,5,4]],[[4,5,6],[3,6,3]]])
v=bb.array(ab)
fg=tt.transpose(v)
print(v)

with tt.Session() as df:
    print("\n New tranformed matrix is: \n\n{}".format(df.run(fg)))
```
Result is :
```
[[[1 2 3]
  [6 5 4]]

 [[4 5 6]
  [3 6 3]]]

 New tranformed matrix is:

[[[1 4]
  [6 3]]

 [[2 5]
  [5 6]]

 [[3 6]
  [4 3]]]

Process finished with exit code 0
```
now if i use perm argument then :
```
import tensorflow as tt
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"]="3"
import numpy as bb
ab=([[[1,2,3],[6,5,4]],[[4,5,6],[3,6,3]]])
v=bb.array(ab)
fg=tt.transpose(v,perm=[0,2,1])
print(v)

with tt.Session() as df:
    print("\n New tranformed matrix is: \n\n{}".format(df.run(fg)))
```
Result is :
```
[[[1 2 3]
  [6 5 4]]

 [[4 5 6]
  [3 6 3]]]

 New tranformed matrix is:

[[[1 6]
  [2 5]
  [3 4]]

 [[4 3]
  [5 6]
  [6 3]]]

Process finished with exit code 0
```
Due to this, I am confused and I have two questions : Whenever I want to transpose a matrix I have to give perm[0,2,1] as default ? What is 0,2,1 here ?

Response:
Looking at the numpy.transpose documentation, we find that transpose takes the argument axes : list of ints, optional By default, reverse the dimensions, otherwise permute the axes according to the values given. So the default call to transpose translates into np.transpose(a, axes=[1,0]) for the 2D case, or np.transpose(a, axes=[2,1,0]). The operation you want to have here, is one that leaves the "depth" dimension unchanged. Therefore in the axes argument, the depth axes, which is the 0th axes, needs to stay unchanged. The axes 1 and 2 (where 1 is the vertical axis), need to change positions. So you change the axes order from the initial [0,1,2] to [0,2,1] ([stays the same, changes with other, changes with other]). In tensorflow, they have for some reason renamed axes to perm. The argument from above stays the same. images Concerning images, they differ from the arrays in the question. Images normally have their x and y stored in the first two dimensions and the channel in the last, [y,x,channel]. In order to "transpose" an image in the sense of a 2D transposition, where horizontal and vertical axes are exchanged, you would need to use
```
np.transpose(a, axes=[1,0,2])
```
(channel stays the same, x and y are exchanged).


Question:
eval() and run() in tensorflow I'm referring to Deep MNIST for Experts tutorial given by the tensorflow. I have a problem in Train and Evaluate part of that tutorial. There they have given a sample code as follows.
```
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_conv),reduction_indices=[1]))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
sess.run(tf.initialize_all_variables())
for i in range(20000):
  batch = mnist.train.next_batch(50)
  if i%100 == 0:
    train_accuracy = accuracy.eval(feed_dict={x:batch[0], y_: batch[1], keep_prob: 1.0})
    print("step %d, training accuracy %g"%(i, train_accuracy))
  train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})

print("test accuracy %g"%accuracy.eval(feed_dict={x: mnist.test.images,
                       y_: mnist.test.labels, keep_prob: 1.0}))
```
So in these code segment they have used accuracy.eval() at one time. And other time train_step.run(). As I know of both of them are tensor variables. And in some cases, I have seen like
```
sess.run(variable, feed_dict)
```
So my question is what are the differences between these 3 implementations. And how can I know what to use when..? Thank You!!

Response: If you have only one default session, they are basically the same. From https://github.com/tensorflow/tensorflow/blob/v1.12.0/tensorflow/python/framework/ops.py#L2351: op.run() is a shortcut for calling tf.get_default_session().run(op) From https://github.com/tensorflow/tensorflow/blob/v1.12.0/tensorflow/python/framework/ops.py#L691: t.eval() is a shortcut for calling tf.get_default_session().run(t) Difference between Tensor and Operation: Tensor: https://www.tensorflow.org/api_docs/python/tf/Tensor Operation: https://www.tensorflow.org/api_docs/python/tf/Operation Note: the Tensor class will be replaced by Output in the future. Currently these two are aliases for each other.


Question:
Keras multiple binary outputs Can someone help me understand a bit better this problem? I must train a neural network which should output 200 mutually independent categories, each of these categories is a percentage ranging from 0 to 1. This seems to me like a binary_crossentropy problem, but every example I see on the internet uses binary_crossentropy with a single output. Since my output should be 200, if I apply binary_crossentropy, would that be correct? This is what I have in mind, is that a correct approach or should I change it?
```
inputs = Input(shape=(input_shape,))
hidden = Dense(2048, activation='relu')(inputs)
hidden = Dense(2048, activation='relu')(hidden)
output = Dense(200, name='output_cat', activation='sigmoid')(hidden)
model = Model(inputs=inputs, outputs=[output])
loss_map = {'output_cat': 'binary_crossentropy'}
model.compile(loss=loss_map, optimizer="sgd", metrics=['mae', 'accuracy'])
```

Response:
To optimize for multiple independent binary classification problems (and not multiple category problem where you can use categorical_crossentropy) using Keras, you could do the following (here I take the example of 2 independent binary outputs, but you can extend that as much as needed):
```
inputs = Input(shape=(input_shape,))
    hidden = Dense(2048, activation='relu')(inputs)
    hidden = Dense(2048, activation='relu')(hidden)
    output = Dense(units = 2, activation='sigmoid')(hidden )
```
here you split your output using Keras's Lambda layer:
```
output_1 = Lambda(lambda x: x[...,:1])(output)
    output_2 = Lambda(lambda x: x[...,1:])(output)

    adad = optimizers.Adadelta()
```
your model output becomes a list of the different independent outputs
```
model = Model(inputs, [output_1, output_2])
```
you compile the model using one loss function for each output, in a list. (In fact, if you give only one kind of loss function, I believe it will apply it to all the outputs independently)
```
model.compile(optimizer=adad, loss=['binary_crossentropy','binary_crossentropy'])
```