Question:
Extract target from Tensorflow PrefetchDataset I am still learning tensorflow and keras, and I suspect this question has a very easy answer I'm just missing due to lack of familiarity. I have a PrefetchDataset object:
```
> print(tf_test)
$ <PrefetchDataset shapes: ((None, 99), (None,)), types: (tf.float32, tf.int64)>
```
...made up of features and a target. I can iterate over it using a for loop:
```
> for example in tf_test:
>     print(example[0].numpy())
>     print(example[1].numpy())
>     exit()
$ [[-0.31 -0.94 -1.12 ... 0.18 -0.27]
   [-0.22 -0.54 -0.14 ... 0.33 -0.55]
   [-0.60 -0.02 -1.41 ... 0.21 -0.63]
   ...
   [-0.03 -0.91 -0.12 ... 0.77 -0.23]
   [-0.76 -1.48 -0.15 ... 0.38 -0.35]
   [-0.55 -0.08 -0.69 ... 0.44 -0.36]]
  [0 0 1 0 1 0 0 0 1 0 1 1 0 1 0 0 0
   ...
   0 1 1 0]
```
However, this is very slow. What I'd like to do is access the tensor corresponding to the class labels and turn that into a numpy array, or a list, or any sort of iterable that can be fed into scikit-learn's classification report and/or confusion matrix:
```
> y_pred = model.predict(tf_test)
> print(y_pred)
$ [[0.01]
   [0.14]
   [0.00]
   ...
   [0.32]
   [0.03]
   [0.00]]
> y_pred_list = [int(x[0]) for x in y_pred]             # assumes value >= 0.5 is positive prediction
> y_true = []                                           # what I need help with
> print(sklearn.metrics.confusion_matrix(y_true, y_pred_list)
```
...OR access the data such that it could be used in tensorflow's confusion matrix:
```
> labels = []                                           # what I need help with
> predictions = y_pred_list                             # could we just use a tensor?
> print(tf.math.confusion_matrix(labels, predictions)
```
In both cases, the general ability to grab the target data from the original object in a manner that isn't computationally expensive would be very helpful (and might help with my underlying intuitions re: tensorflow and keras). Any advice would be greatly appreciated.

Response: You can convert it to a list with list(ds) and then recompile it as a normal Dataset with tf.data.Dataset.from_tensor_slices(list(ds)). From there your nightmare begins again but at least it's a nightmare that other people have had before. Note that for more complex datasets (e.g. nested dictionaries) you will need more preprocessing after calling list(ds), but this should work for the example you asked about. This is far from a satisfying answer but unfortunately the class is entirely undocumented and none of the standard Dataset tricks work.


Question:
Multilabel Text Classification using TensorFlow The text data is organized as vector with 20,000 elements, like [2, 1, 0, 0, 5, ...., 0]. i-th element indicates the frequency of the i-th word in a text. The ground truth label data is also represented as vector with 4,000 elements, like [0, 0, 1, 0, 1, ...., 0]. i-th element indicates whether the i-th label is a positive label for a text. The number of labels for a text differs depending on texts. I have a code for single-label text classification. How can I edit the following code for multilabel text classification? Especially, I would like to know following points. How to compute accuracy using TensorFlow. How to set a threshold which judges whether a label is positive or negative. For instance, if the output is [0.80, 0.43, 0.21, 0.01, 0.32] and the ground truth is [1, 1, 0, 0, 1], the labels with scores over 0.25 should be judged as positive. Thank you.
```
import tensorflow as tf

# hidden Layer
class HiddenLayer(object):
    def __init__(self, input, n_in, n_out):
        self.input = input

        w_h = tf.Variable(tf.random_normal([n_in, n_out],mean = 0.0,stddev = 0.05))
        b_h = tf.Variable(tf.zeros([n_out]))

        self.w = w_h
        self.b = b_h
        self.params = [self.w, self.b]

    def output(self):
        linarg = tf.matmul(self.input, self.w) + self.b
        self.output = tf.nn.relu(linarg)

        return self.output

# output Layer
class OutputLayer(object):
    def __init__(self, input, n_in, n_out):
        self.input = input

        w_o = tf.Variable(tf.random_normal([n_in, n_out], mean = 0.0, stddev = 0.05))
        b_o = tf.Variable(tf.zeros([n_out]))

        self.w = w_o
        self.b = b_o
        self.params = [self.w, self.b]

    def output(self):
        linarg = tf.matmul(self.input, self.w) + self.b
        self.output = tf.nn.relu(linarg)

        return self.output

# model
def model():
    h_layer = HiddenLayer(input = x, n_in = 20000, n_out = 1000)
    o_layer = OutputLayer(input = h_layer.output(), n_in = 1000, n_out = 4000)

    # loss function
    out = o_layer.output()
    cross_entropy = -tf.reduce_sum(y_*tf.log(out + 1e-9), name='xentropy')

    # regularization
    l2 = (tf.nn.l2_loss(h_layer.w) + tf.nn.l2_loss(o_layer.w))
    lambda_2 = 0.01

    # compute loss
    loss = cross_entropy + lambda_2 * l2

    # compute accuracy for single label classification task
    correct_pred = tf.equal(tf.argmax(out, 1), tf.argmax(y, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_pred, "float"))

    return loss, accuracy
```

Response:
Change relu to sigmoid of output layer. Modify cross entropy loss to explicit mathematical formula of sigmoid cross entropy loss (explicit loss was working in my case/version of tensorflow )
```
import tensorflow as tf

# hidden Layer
class HiddenLayer(object):
    def __init__(self, input, n_in, n_out):
        self.input = input

        w_h = tf.Variable(tf.random_normal([n_in, n_out],mean = 0.0,stddev = 0.05))
        b_h = tf.Variable(tf.zeros([n_out]))

        self.w = w_h
        self.b = b_h
        self.params = [self.w, self.b]

    def output(self):
        linarg = tf.matmul(self.input, self.w) + self.b
        self.output = tf.nn.relu(linarg)

        return self.output

# output Layer
class OutputLayer(object):
    def __init__(self, input, n_in, n_out):
        self.input = input

        w_o = tf.Variable(tf.random_normal([n_in, n_out], mean = 0.0, stddev = 0.05))
        b_o = tf.Variable(tf.zeros([n_out]))

        self.w = w_o
        self.b = b_o
        self.params = [self.w, self.b]

    def output(self):
        linarg = tf.matmul(self.input, self.w) + self.b
        #changed relu to sigmoid
        self.output = tf.nn.sigmoid(linarg)

        return self.output

# model
def model():
    h_layer = HiddenLayer(input = x, n_in = 20000, n_out = 1000)
    o_layer = OutputLayer(input = h_layer.output(), n_in = 1000, n_out = 4000)

    # loss function
    out = o_layer.output()
    # modified cross entropy to explicit mathematical formula of sigmoid cross entropy loss
    cross_entropy = -tf.reduce_sum( (  (y_*tf.log(out + 1e-9)) + ((1-y_) * tf.log(1 - out + 1e-9)) )  , name='xentropy' )

    # regularization
    l2 = (tf.nn.l2_loss(h_layer.w) + tf.nn.l2_loss(o_layer.w))
    lambda_2 = 0.01

    # compute loss
    loss = cross_entropy + lambda_2 * l2

    # compute accuracy for single label classification task
    correct_pred = tf.equal(tf.argmax(out, 1), tf.argmax(y, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_pred, "float"))

    return loss, accuracy
```


Question:
How to fix MatMul Op has type float64 that does not match type float32 TypeError? I am trying to save Nueral Network weights into a file and then restoring those weights by initializing the network instead of random initialization. My code works fine with random initialization. But, when i initialize weights from file it is showing me an error TypeError: Input 'b' of 'MatMul' Op has type float64 that does not match type float32 of argument 'a'. I don't know how do i solve this issue.Here is my code: Model Initialization
```
# Parameters
training_epochs = 5
batch_size = 64
display_step = 5
batch = tf.Variable(0, trainable=False)
regualarization =  0.008

# Network Parameters
n_hidden_1 = 300 # 1st layer num features
n_hidden_2 = 250 # 2nd layer num features

n_input = model.layer1_size # Vector input (sentence shape: 30*10)
n_classes = 12 # Sentence Category detection total classes (0-11 categories)

#History storing variables for plots
loss_history = []
train_acc_history = []
val_acc_history = []

# tf Graph input
x = tf.placeholder("float", [None, n_input])
y = tf.placeholder("float", [None, n_classes])
```
Model parameters
```
#loading Weights
def weight_variable(fan_in, fan_out, filename):
    stddev = np.sqrt(2.0/fan_in)
    if (filename == ""):
        initial  = tf.random_normal([fan_in,fan_out], stddev=stddev)
    else:
        initial  = np.loadtxt(filename)
    print initial.shape
    return tf.Variable(initial)

#loading Biases
def bias_variable(shape, filename):
    if (filename == ""):
     initial = tf.constant(0.1, shape=shape)
    else:
     initial  = np.loadtxt(filename)
    print initial.shape
    return tf.Variable(initial)

# Create model
def multilayer_perceptron(_X, _weights, _biases):
    layer_1 = tf.nn.relu(tf.add(tf.matmul(_X, _weights['h1']), _biases['b1']))
    layer_2 = tf.nn.relu(tf.add(tf.matmul(layer_1, _weights['h2']), _biases['b2']))
    return tf.matmul(layer_2, weights['out']) + biases['out']

# Store layers weight & bias
weights = {
'h1':  w2v_utils.weight_variable(n_input, n_hidden_1,    filename="weights_h1.txt"),
'h2':  w2v_utils.weight_variable(n_hidden_1, n_hidden_2, filename="weights_h2.txt"),
'out': w2v_utils.weight_variable(n_hidden_2, n_classes,  filename="weights_out.txt")
}

 biases = {
'b1': w2v_utils.bias_variable([n_hidden_1], filename="biases_b1.txt"),
'b2': w2v_utils.bias_variable([n_hidden_2], filename="biases_b2.txt"),
'out': w2v_utils.bias_variable([n_classes], filename="biases_out.txt")
}

# Define loss and optimizer
#learning rate
# Optimizer: set up a variable that's incremented once per batch and
# controls the learning rate decay.
learning_rate = tf.train.exponential_decay(
    0.02*0.01,           # Base learning rate. #0.002
    batch * batch_size,  # Current index into the dataset.
    X_train.shape[0],    # Decay step.
    0.96,                # Decay rate.
    staircase=True)

# Construct model
pred = tf.nn.relu(multilayer_perceptron(x, weights, biases))

#L2 regularization
l2_loss = tf.add_n([tf.nn.l2_loss(v) for v in tf.trainable_variables()])

#Softmax loss
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, y))

#Total_cost
cost = cost+ (regualarization*0.5*l2_loss)

# Adam Optimizer
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost,global_step=batch)

# Add ops to save and restore all the variables.
saver = tf.train.Saver()

# Initializing the variables
init = tf.initialize_all_variables()

print "Network Initialized!"
```
ERROR DETAILS

Response:
The tf.matmul() op does not perform automatic type conversions, so both of its inputs must have the same element type. The error message you are seeing indicates that you have a call to tf.matmul() where the first argument has type tf.float32, and the second argument has type tf.float64. You must convert one of the inputs to match the other, for example using tf.cast(x, tf.float32). Looking at your code, I don't see anywhere that a tf.float64 tensor is explicitly created (the default dtype for floating-point values in the TensorFlow Python API—e.g. for tf.constant(37.0)—is tf.float32). I would guess that the errors are caused by the np.loadtxt(filename) calls, which might be loading an np.float64 array. You can explicitly change them to load np.float32 arrays (which are converted to tf.float32 tensors) as follows:
```
initial = np.loadtxt(filename).astype(np.float32)
```


Question:
Tensor with unspecified dimension in tensorflow I'm playing around with tensorflow and ran into a problem with the following code:
```
def _init_parameters(self, input_data, labels):

    # the input shape is (batch_size, input_size)
    input_size = tf.shape(input_data)[1]

    # labels in one-hot format have shape (batch_size, num_classes)
    num_classes = tf.shape(labels)[1]

    stddev = 1.0 / tf.cast(input_size, tf.float32)

    w_shape = tf.pack([input_size, num_classes], 'w-shape')
    normal_dist = tf.truncated_normal(w_shape, stddev=stddev, name='normaldist')
    self.w = tf.Variable(normal_dist, name='weights')
```
(I'm using tf.pack as suggested in this question, since I was getting the same error) When I run it (from a larger script that invokes this one), I get this error:
```
ValueError: initial_value must have a shape specified: Tensor("normaldist:0", shape=TensorShape([Dimension(None), Dimension(None)]), dtype=float32)
```
I tried to replicate the process in the interactive shell. Indeed, the dimensions of normal_dist are unspecified, although the supplied values do exist:
```

```python
input_size.eval()
#Output
#4
```

```python
num_classes.eval()
#Output
#3
```

```python
w_shape.eval()
#Output
#array([4, 3], dtype=int32)
```

```python
normal_dist.eval()
#Output
#array([[-0.27035281, -0.223277  ,  0.14694688],
#       [-0.16527176,  0.02180306,  0.00807841],
#       [ 0.22624688,  0.36425814, -0.03099642],
#       [ 0.25575709, -0.02765726, -0.26169327]], dtype=float32)
```

```python
normal_dist.get_shape()
#Output
#TensorShape([Dimension(None), Dimension(None)])
#```
#This is weird. Tensorflow generates the vector but can't say its shape. Am I doing something wrong?
```

Response:
As Ishamael says, all tensors have a static shape, which is known at graph construction time and accessible using Tensor.get_shape(); and a dynamic shape, which is only known at runtime and is accessible by fetching the value of the tensor, or passing it to an operator like tf.shape. In many cases, the static and dynamic shapes are the same, but they can be different - the static shape can be partially defined - in order allow the dynamic shape to vary from one step to the next. In your code normal_dist has a partially-defined static shape, because w_shape is a computed value. (TensorFlow sometimes attempts to evaluate these computed values at graph construction time, but it gets stuck at tf.pack.) It infers the shape TensorShape([Dimension(None), Dimension(None)]), which means "a matrix with an unknown number of rows and columns," because it knowns that w_shape is a vector of length 2, so the resulting normal_dist must be 2-dimensional. You have two options to deal with this. You can set the static shape as Ishamael suggests, but this requires you to know the shape at graph construction time. For example, the following may work:
```
normal_dist.set_shape([input_data.get_shape()[1], labels.get_shape()[1]])
```
Alternatively, you can pass validate_shape=False to the tf.Variable constructor. This allows you to create a variable with a partially-defined shape, but it limits the amount of static shape information that can be inferred later on in the graph.


Question:
What do the options in ConfigProto like allow_soft_placement and log_device_placement mean? We see this quite often in many of the TensorFlow tutorials:
```py
sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True,
                                        log_device_placement=True))
```
What does allow_soft_placement and log_device_placement mean?

Response:
If you look at the API of ConfigProto, on line 278, you will see this:
```cpp
// Whether soft placement is allowed. If allow_soft_placement is true,
  // an op will be placed on CPU if
  //   1. there's no GPU implementation for the OP
  // or
  //   2. no GPU devices are known or registered
  // or
  //   3. need to co-locate with reftype input(s) which are from CPU.
  bool allow_soft_placement = 7;
```
What this really means is that if you do something like this without allow_soft_placement=True, TensorFlow will throw an error.
```py
with tf.device('/gpu:0'):
    # some op that doesn't have a GPU implementation
```
Right below it, you will see on line 281:
```cpp
// Whether device placements should be logged.
  bool log_device_placement = 8;
```
When log_device_placement=True, you will get a verbose output of something like this:
```
2017-07-03 01:13:59.466748: I tensorflow/core/common_runtime/simple_placer.cc:841] Placeholder_1: (Placeholder)/job:localhost/replica:0/task:0/cpu:0
Placeholder: (Placeholder): /job:localhost/replica:0/task:0/cpu:0
2017-07-03 01:13:59.466765: I tensorflow/core/common_runtime/simple_placer.cc:841] Placeholder: (Placeholder)/job:localhost/replica:0/task:0/cpu:0
Variable/initial_value: (Const): /job:localhost/replica:0/task:0/cpu:0
2017-07-03 01:13:59.466783: I tensorflow/core/common_runtime/simple_placer.cc:841] Variable/initial_value: (Const)/job:localhost/replica:0/task:0/cpu:0
```
You can see where each operation is mapped to. For this case, they are all mapped to /cpu:0, but if you're in a distributed setting, there would be many more devices.


Question:
Output from TensorFlow `py_func` has unknown rank/shape I am trying to create a simple neural net in TensorFlow. The only tricky part is I have a custom operation that I have implemented with py_func. When I pass the output from py_func to a Dense layer, TensorFlow complains that the rank should be known. The specific error is:
```py
ValueError: Inputs to `Dense` should have known rank.
```
I don't know how to preserve the shape of my data when I pass it through py_func. My question is how do I get the correct shape? I have a simple example below to illustrate the problem.
```py
def my_func(x):
    return np.sinh(x).astype('float32')

inp = tf.convert_to_tensor(np.arange(5))
y = tf.py_func(my_func, [inp], tf.float32, False)

with tf.Session() as sess:
    with sess.as_default():
        print(inp.shape)
        print(inp.eval())
        print(y.shape)
        print(y.eval())
```
The output from this snippet is:
```py
(5,)
[0 1 2 3 4]
<unknown>
[  0.
1.17520118   3.62686038  10.01787472  27.28991699]
```
Why is y.shape ? I want the shape to be (5,) the same as inp. Thanks!

Response:
Since py_func can execute arbitrary Python code and output anything, TensorFlow can't figure out the shape (it would require analyzing Python code of function body) You can instead give the shape manually
```
y.set_shape(inp.get_shape())
```


Question:
Printing the loss during TensorFlow training I am looking at the TensorFlow "MNIST For ML Beginners" tutorial, and I want to print out the training loss after every training step. My training loop currently looks like this:
```
for i in range(100):
    batch_xs, batch_ys = mnist.train.next_batch(100)
    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
```
Now, train_step is defined as:
```
train_step = tf.train.GradientDescentOptimizer(0.01).minimize(cross_entropy)
```
Where cross_entropy is the loss which I want to print out:
```
cross_entropy = -tf.reduce_sum(y_ * tf.log(y))
```
One way to print this would be to explicitly compute cross_entropy in the training loop:
```
for i in range(100):
    batch_xs, batch_ys = mnist.train.next_batch(100)
    cross_entropy = -tf.reduce_sum(y_ * tf.log(y))
    print 'loss = ' + str(cross_entropy)
    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
```
I now have two questions regarding this: Given that cross_entropy is already computed during sess.run(train_step, ...), it seems inefficient to compute it twice, requiring twice the number of forward passes of all the training data. Is there a way to access the value of cross_entropy when it was computed during sess.run(train_step, ...)? How do I even print a tf.Variable? Using str(cross_entropy) gives me an error... Thank you!

Response:
You can fetch the value of cross_entropy by adding it to the list of arguments to sess.run(...). For example, your for-loop could be rewritten as follows:
```
for i in range(100):
    batch_xs, batch_ys = mnist.train.next_batch(100)
    cross_entropy = -tf.reduce_sum(y_ * tf.log(y))
    _, loss_val = sess.run([train_step, cross_entropy],
                           feed_dict={x: batch_xs, y_: batch_ys})
    print 'loss = ' + loss_val
```
The same approach can be used to print the current value of a variable. Let's say, in addition to the value of cross_entropy, you wanted to print the value of a tf.Variable called W, you could do the following:
```
for i in range(100):
    batch_xs, batch_ys = mnist.train.next_batch(100)
    cross_entropy = -tf.reduce_sum(y_ * tf.log(y))
    _, loss_val, W_val = sess.run([train_step, cross_entropy, W],
                                  feed_dict={x: batch_xs, y_: batch_ys})
    print 'loss = %s' % loss_val
    print 'W = %s' % W_val
```


Question: What is the difference of static Computational Graphs in tensorflow and dynamic Computational Graphs in Pytorch? When I was learning tensorflow, one basic concept of tensorflow was computational graphs, and the graphs was said to be static. And I found in Pytorch, the graphs was said to be dynamic. What's the difference of static Computational Graphs in tensorflow and dynamic Computational Graphs in Pytorch?

Response: Both frameworks operate on tensors and view any model as a directed acyclic graph (DAG), but they differ drastically on how you can define them. TensorFlow follows ‘data as code and code is data’ idiom. In TensorFlow you define graph statically before a model can run. All communication with outer world is performed via tf.Session object and tf.Placeholder which are tensors that will be substituted by external data at runtime. In PyTorch things are way more imperative and dynamic: you can define, change and execute nodes as you go, no special session interfaces or placeholders. Overall, the framework is more tightly integrated with Python language and feels more native most of the times. When you write in TensorFlow sometimes you feel that your model is behind a brick wall with several tiny holes to communicate over. Anyways, this still sounds like a matter of taste more or less. However, those approaches differ not only in a software engineering perspective: there are several dynamic neural network architectures that can benefit from the dynamic approach. Recall RNNs: with static graphs, the input sequence length will stay constant. This means that if you develop a sentiment analysis model for English sentences you must fix the sentence length to some maximum value and pad all smaller sequences with zeros. Not too convenient, huh. And you will get more problems in the domain of recursive RNNs and tree-RNNs. Currently Tensorflow has limited support for dynamic inputs via Tensorflow Fold. PyTorch has it by-default. Reference: https://medium.com/towards-data-science/pytorch-vs-tensorflow-spotting-the-difference-25c75777377b https://www.reddit.com/r/MachineLearning/comments/5w3q74/d_so_pytorch_vs_tensorflow_whats_the_verdict_on/


Question:
ValueError: Shapes (None, 1) and (None, 3) are incompatible I have a 3 dimensional dataset of audio files where X.shape is (329,20,85). I want to have a simpl bare-bones model running, so please don't nitpick and address only the issue at hand. Here is the code:
```
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.LSTM(32, return_sequences=True, stateful=False, input_shape = (20,85,1)))
model.add(tf.keras.layers.LSTM(20))
model.add(tf.keras.layers.Dense(nb_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=["accuracy"])
model.summary()
print("Train...")
model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=50, validation_data=(X_test, y_test))
```
But then I had the error mentioned in the title: ValueError: Shapes (None, 1) and (None, 3) are incompatible Here is the model.summary()
```
Model: "sequential_13"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
lstm_21 (LSTM)               (None, 20, 32)            15104
_________________________________________________________________
lstm_22 (LSTM)               (None, 20)                4240
_________________________________________________________________
dense_8 (Dense)              (None, 3)                 63
=================================================================
Total params: 19,407
Trainable params: 19,407
Non-trainable params: 0
_________________________________________________________________
Train...
```
For this, I followed this post and updated Tensorflow to the latest version, but the issue persists. This post is completely unrelated and highly unreliable.This post although a bit relatable is unanswered for a while now. Update 1.0: I strongly think the problem has something to do with the final Dense layer where I pass nb_classes as 3, since I am classifying for 3 categories in y. So I changed the Dense layer's nb_classes to 1, which ran the model and gives me this output, which I am positive is wrong.
```
Train...
9/9 [==============================] - 2s 177ms/step - loss: 0.0000e+00 - accuracy: 0.1520 - val_loss: 0.0000e+00 - val_accuracy: 0.3418

<tensorflow.python.keras.callbacks.History at 0x7f50f1dcebe0>
```
Update 2.0: I one hot encoded the ys and resolved the shape issue. But now the above output with  persists. Any help with this? Or should I post a new question for this? Thanks for all the help. How should I proceed, or what should I be changing?

Response:
The first problem is with the LSTM input_shape. input_shape = (20,85,1). From the doc: https://keras.io/layers/recurrent/ LSTM layer expects 3D tensor with shape (batch_size, timesteps, input_dim). model.add(tf.keras.layers.Dense(nb_classes, activation='softmax')) - this suggets you're doing a multi-class classification. So, you need your y_train and y_test have to be one-hot-encoded. That means they must have dimension (number_of_samples, 3), where 3 denotes number of classes. You need to apply tensorflow.keras.utils.to_categorical to them.
```
y_train = to_categorical(y_train, 3)
y_test = to_categorical(y_test, 3)
```
ref: https://www.tensorflow.org/api_docs/python/tf/keras/utils/to_categorical tf.keras.callbacks.History() - this callback is automatically applied to every Keras model. The History object gets returned by the fit method of models. ref: https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/History


Question:
Confused by the behavior of `tf.cond` I need a conditional control flow in my graph. If pred is True, the graph should call an op that updates a variable and then returns it, otherwise it returns the variable unchanged. A simplified version is:
```py
pred = tf.constant(True)
x = tf.Variable([1])
assign_x_2 = tf.assign(x, [2])
def update_x_2():
  with tf.control_dependencies([assign_x_2]):
    return tf.identity(x)
y = tf.cond(pred, update_x_2, lambda: tf.identity(x))
with tf.Session() as session:
  session.run(tf.initialize_all_variables())
  print(y.eval())
```
However, I find that both pred=True and pred=False lead to the same result y=[2], which means the assign op is also called when update_x_2 is not selected by tf.cond. How to explain this? And how to solve this problem?

Response:
TL;DR: If you want tf.cond() to perform a side effect (like an assignment) in one of the branches, you must create the op that performs the side effect inside the function that you pass to tf.cond(). The behavior of tf.cond() is a little unintuitive. Because execution in a TensorFlow graph flows forward through the graph, all operations that you refer to in either branch must execute before the conditional is evaluated. This means that both the true and the false branches receive a control dependency on the tf.assign() op, and so y always gets set to 2, even if pred is False. The solution is to create the tf.assign() op inside the function that defines the true branch. For example, you could structure your code as follows:
```py
pred = tf.placeholder(tf.bool, shape=[])
x = tf.Variable([1])
def update_x_2():
  with tf.control_dependencies([tf.assign(x, [2])]):
    return tf.identity(x)
y = tf.cond(pred, update_x_2, lambda: tf.identity(x))
with tf.Session() as session:
  session.run(tf.initialize_all_variables())
  print(y.eval(feed_dict={pred: False}))  # ==> [1]
  print(y.eval(feed_dict={pred: True}))   # ==> [2]
```


Question:
what does the question mark in tensorflow shape mean? I'm looking into magenta code, and printing its tensor object. I got this result:
```
Tensor("fully_connected/BiasAdd:0", shape=(?, 38), dtype=float32)
```
What does this question mark in shape mean?

Response: It means that first dimension is not fixed in the graph and it can vary between run calls


Question:
You must feed a value for placeholder tensor 'Placeholder' with dtype float I'm a newer to tensorflow, I really don't know how to solve the problem. The code is like: Feed the train with values:
```
sess.run(train_op, feed_dict={images: e, labels: l, keep_prob_fc2: 0.5})
```
Use the value in CNN:
```
x = tf.placeholder(tf.float32, [None, 10 * 1024])
```
Then have the error
```
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'Placeholder' with dtype float
     [[Node: Placeholder = Placeholder[dtype=DT_FLOAT, shape=[], _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
```
I print the input valuetypes using print(e.dtype) and the result is float32 and e.shape:(10, 32, 32, 1). I really don't know why this error is happening. The code format First:
```
define the CNN model
       "image = tf.placeholder(tf.float32, [FLAGS.batch_size, 32,32,1])" is here
```
Second:
```
loss funtion and train_op is here
       "label = tf.placeholder(tf.float32, [None, FLAGS.batch_size])" is here
```
Third is the session:
```
images, labels = getShuffleimage()#here will get shuffle data
num_examples = 0
init = tf.initialize_local_variables()

with tf.Session() as sess:
    # Start populating the filename queue.
    sess.run(init)
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord, sess=sess)

    try:
        step = 0
        while not coord.should_stop():
            start_time = time.time()
            image, label = sess.run([images, labels])#get shuffle images
            print(image.shape)
            print(image.dtype)
            sess.run(train_op, feed_dict={image: image, label: label , keep_prob_fc2: 0.5})
            duration = time.time() - start_time

    except tf.errors.OutOfRangeError:
        print('Done training after reading all data')
    finally:
        # When done, ask the threads to stop.
        coord.request_stop()

        # Wait for threads to finish.
        coord.join(threads)
        sess.close()
```

Response: Some questions first why you use sess = tf.InteractiveSession() and with tf.Session() as sess: at same time, just curious second what is your placeholder name x or images? if name is x, {images: x_data...} won't feed x_data to x, it override(?) images I think feed_dict should be {x: x_data...} if name is images,do you have two images in your program, placeholder and shuffle data, try to modify name of variable


Question: Why does keras model predict slower after compile? In theory, the prediction should be constant as the weights have a fixed size. How do I get my speed back after compile (without the need to remove optimizer)? See associated experiment: https://nbviewer.jupyter.org/github/offchan42/TensorFlowExperiments/blob/master/test-prediction-speed-after-compile.ipynb?flush_cache=true

Response:
UPDATE - 1/15/2020: the current best practice for small batch sizes should be to feed inputs to the model directly - i.e. preds = model(x), and if layers behave differently at train / inference, model(x, training=False). Per latest commit, this is now documented. I haven't benchmarked these, but per the Git discussion, it's also worth trying predict_on_batch() - especially with improvements in TF 2.1. ULTIMATE CULPRIT: self._experimental_run_tf_function = True. It's experimental. But it's not actually bad. To any TensorFlow devs reading: clean up your code. It's a mess. And it violates important coding practices, such as one function does one thing; _process_inputs does a lot more than "process inputs", same for _standardize_user_data. "I'm not paid enough" - but you do pay, in extra time spent understanding your own stuff, and in users filling your Issues page with bugs easier resolved with a clearer code. SUMMARY: it's only a little slower with compile(). compile() sets an internal flag which assigns a different prediction function to predict. This function constructs a new graph upon each call, slowing it down relative to uncompiled. However, the difference is only pronounced when train time is much shorter than data processing time. If we increase the model size to at least mid-sized, the two become equal. See code at the bottom. This slight increase in data processing time is more than compensated by amplified graph capability. Since it's more efficient to keep only one model graph around, the one pre-compile is discarded. Nonetheless: if your model is small relative to data, you are better off without compile() for model inference. See my other answer for a workaround. WHAT SHOULD I DO? Compare model performance compiled vs uncompiled as I have in code at the bottom. Compiled is faster: run predict on a compiled model. Compiled is slower: run predict on an uncompiled model. Yes, both are possible, and it will depend on (1) data size; (2) model size; (3) hardware. Code at the bottom actually shows compiled model being faster, but 10 iterations is a small sample. See "workarounds" in my other answer for the "how-to". DETAILS: This took a while to debug, but was fun. Below I describe the key culprits I discovered, cite some relevant documentation, and show profiler results that led to the ultimate bottleneck. (FLAG == self.experimental_run_tf_function, for brevity) Model by default instantiates with FLAG=False. compile() sets it to True. predict() involves acquiring the prediction function, func = self._select_training_loop(x) Without any special kwargs passed to predict and compile, all other flags are such that: (A) FLAG==True --> func = training_v2.Loop() (B) FLAG==False --> func = training_arrays.ArrayLikeTrainingLoop() From source code docstring, (A) is heavily graph-reliant, uses more distribution strategy, and ops are prone to creating & destroying graph elements, which "may" (do) impact performance. True culprit: _process_inputs(), accounting for 81% of runtime. Its major component? _create_graph_function(), 72% of runtime. This method does not even exist for (B). Using a mid-sized model, however, _process_inputs comprises less than 1% of runtime. Code at bottom, and profiling results follow. DATA PROCESSORS: (A): , used in _process_inputs() . Relevant source code (B): numpy.ndarray, returned by convert_eager_tensors_to_numpy. Relevant source code, and here MODEL EXECUTION FUNCTION (e.g. predict) (A): distribution function, and here (B): distribution function (different), and here PROFILER: results for code in my other answer, "tiny model", and in this answer, "medium model": Tiny model: 1000 iterations, compile() Tiny model: 1000 iterations, no compile() Medium model: 10 iterations DOCUMENTATION (indirectly) on effects of compile(): source Unlike other TensorFlow operations, we don't convert python numerical inputs to tensors. Moreover, a new graph is generated for each distinct python numerical value, for example calling g(2) and g(3) will generate two new graphs function instantiates a separate graph for every unique set of input shapes and datatypes. For example, the following code snippet will result in three distinct graphs being traced, as each input has a different shape A single tf.function object might need to map to multiple computation graphs under the hood. This should be visible only as performance (tracing graphs has a nonzero computational and memory cost) but should not affect the correctness of the program COUNTEREXAMPLE:
```py
from tensorflow.keras.layers import Input, Dense, LSTM, Bidirectional, Conv1D
from tensorflow.keras.layers import Flatten, Dropout
from tensorflow.keras.models import Model
import numpy as np
from time import time

def timeit(func, arg, iterations):
    t0 = time()
    for _ in range(iterations):
        func(arg)
    print("%.4f sec" % (time() - t0))

batch_size = 32
batch_shape = (batch_size, 400, 16)
ipt   = Input(batch_shape=batch_shape)
x     = Bidirectional(LSTM(512, activation='relu', return_sequences=True))(ipt)
x     = LSTM(512, activation='relu', return_sequences=True)(ipt)
x     = Conv1D(128, 400, 1, padding='same')(x)
x     = Flatten()(x)
x     = Dense(256, activation='relu')(x)
x     = Dropout(0.5)(x)
x     = Dense(128, activation='relu')(x)
x     = Dense(64,  activation='relu')(x)
out   = Dense(1,  activation='sigmoid')(x)
model = Model(ipt, out)

X = np.random.randn(*batch_shape)
timeit(model.predict, X, 10)
model.compile('adam', loss='binary_crossentropy')
timeit(model.predict, X, 10)
```
Outputs:
```py
34.8542 sec
34.7435 sec
```


Question:
How to import keras.engine.topology in Tensorflow? I want to import keras.engine.topology in Tensorflow. I used to add the word tensorflow at the beginning of every Keras import if I want to use the Tensorflow version of Keras. For example: instead of writing:
```
from keras.layers import Dense, Dropout, Input
```
I just write the following code and it works fine :
```
from tensorflow.keras.layers import Dense, Dropout, Input
```
But that's not the case for this specific import:
```
from tensorflow.keras.engine.topology import Layer, InputSpec
```
And I m getting the following error message:
```
No module named 'tensorflow.keras.engine'
```

Response:
You can import Layer and InputSpec from TensorFlow as follows:
```
from tensorflow.python.keras.layers import Layer, InputSpec
```
UPDATE: 30/10/2019
```
from tensorflow.keras.layers import Layer, InputSpec
```


Question: Tensorflow VarLenFeature vs FixedLenFeature I was trying to save images of different sizes into tf-records. I found that even though the images have different sizes, I can still load them with FixedLenFeature. By checking the docs on FixedLenFeature and VarLenFeature, I found the difference seems to be that VarLenFeauture returns a sparse tensor. Could anyone illustrate some situations one should use FixedLenFeature or VarLenFeature?

Response:
You can load images probably beacause you saved them using feature type tf.train.BytesList() and whole image data is one big byte value inside a list. If I'm right you're using tf.decode_raw to get the data out of the image you load from TFRecord. Regarding example use cases: I use VarLenFeature for saving datasets for object detection task: There's variable amount of bounding boxes per image (equal to object in image) therefore I need another feature objects_number to track amount of objects (and bboxes). Each bounding box itself is a list of 4 float coordinates I'm using following code to load it:
```
features = tf.parse_single_example(
    serialized_example,
    features={
        # We know the length of both fields. If not the
        # tf.VarLenFeature could be used
        'height': tf.FixedLenFeature([], tf.int64),
        'width': tf.FixedLenFeature([], tf.int64),
        'depth': tf.FixedLenFeature([], tf.int64),
        # Label part
        'objects_number': tf.FixedLenFeature([], tf.int64),
        'bboxes': tf.VarLenFeature(tf.float32),
        'labels': tf.VarLenFeature(tf.int64),
        # Dense data
        'image_raw': tf.FixedLenFeature([],tf.string)

    })

# Get metadata
objects_number = tf.cast(features['objects_number'], tf.int32)
height = tf.cast(features['height'], tf.int32)
width = tf.cast(features['width'], tf.int32)
depth = tf.cast(features['depth'], tf.int32)

# Actual data
image_shape = tf.parallel_stack([height, width, depth])
bboxes_shape = tf.parallel_stack([objects_number, 4])

# BBOX data is actually dense convert it to dense tensor
bboxes = tf.sparse_tensor_to_dense(features['bboxes'], default_value=0)

# Since information about shape is lost reshape it
bboxes = tf.reshape(bboxes, bboxes_shape)
image = tf.decode_raw(features['image_raw'], tf.uint8)
image = tf.reshape(image, image_shape)
```
Notice that "image_raw" is fixed length Feature (has one element) and holds values of type "bytes", however a value of "bytes" type can itself have variable size (its a string of bytes, and can have many symbols within it). So "image_raw" is a list with ONE element of type "bytes", which can be super big. To further elaborate on how it works: Features are lists of values, those values have specific "type". Datatypes for features are subset of data types for tensors, you have: int64 (64 bit space in memory) bytes (occupies as many bytes in memory as you want) float (occupies 32-64 bits in memory idk how much) You can check here tensors data types. So you can store variable length data without VarLenFeatures at all (actually well you do it), but first you would need to convert it into bytes/string feature, and then decode it. And this is most common method.


Question: How can I test a .tflite model to prove that it behaves as the original model using the same Test Data? I have generated a .tflite model based on a trained model, I would like to test that the tfilte model gives the same results as the original model. Giving both the same test data and obtaining the same result.

Response: You may use TensorFlow Lite Python interpreter to test your tflite model. It allows you to feed input data in python shell and read the output directly like you are just using a normal tensorflow model. I have answered this question here. And you can read this TensorFlow lite official guide for detailed information. You can also use Netron to visualize your model. It allows you to load your .tflite file directly and inspect your model architecture and model weights.


Question:
What does opt.apply_gradients() do in TensorFlow? The documentation is not quite clear about this. I suppose the gradients one can obtain by opt.compute_gradients(E, [v]) contain the ∂E/∂x = g(x) for each element x of the tensor that v stores. Does opt.apply_gradients(grads_and_vars) essentially execute x ← -η·g(x), where η is the learning rate? That would imply that if I want to add a positive additive change p to the variable, I would need to need to change g(x) ← g(x) - (1/η)p, e.g. like this:
```py
opt = tf.train.GradientDescentOptimizer(learning_rate=l)
grads_and_vars = opt.compute_gradients(loss, var_list)

for l, gv in enumerate(grads_and_vars):
    grads_and_vars[l] = (gv[0] - (1/l) * p, gv[1])

train_op = opt.apply_gradients(grads_and_vars)
```
Is there a better way to do this?

Response:
The update rule that the apply_gradients method actually applies depends on the specific optimizer. Take a look at the implementation of apply_gradients in the tf.train.Optimizer class here. It relies on the derived classes implementing the update rule in the methods _apply_dense and _apply_spares. The update rule you are referring to is implemented by the GradientDescentOptimizer. Regarding your desired positive additive update: If what you are calling opt is an instantiation of GradientDescentOptimizer, then you could indeed achieve what you want to do by
```
grads_and_vars = opt.compute_gradients(E, [v])
eta = opt._learning_rate
my_grads_and_vars = [(g-(1/eta)*p, v) for g, v in grads_and_vars]
opt.apply_gradients(my_grads_and_vars)
```
The more elegant way to do this is probably to write a new optimizer (inheriting from tf.train.Optimizer) that implements your desired update rule directly.


Question: Run Tensorflow unit tests Is there any way to run Tensorflow unit tests manually? I want to perform sanity checks while modifying TF source code. I see there are many _test.py files with classes that perform many test operations and I can't figure out how to run them. There should be an easy way?

Response:
The easiest way to run the TensorFlow unit tests is using Bazel, assuming you have downloaded the source from Git:
```
# All tests (for C++ changes).
$ bazel test //tensorflow/...

# All Python tests (for Python front-end changes).
$ bazel test //tensorflow/python/...

# All tests (with GPU support).
$ bazel test -c opt --config=cuda //tensorflow/...
$ bazel test -c opt --config=cuda //tensorflow/python/...
```


Question:
Cannot convert a partially converted tensor in TensorFlow There are many methods in TensorFlow that requires specifying a shape, for example truncated_normal:
```
tf.truncated_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None)
```
I have a placeholder for the input of shape [None, 784], where the first dimension is None because the batch size can vary. I could use a fixed batch size but it still would be different from the test/validation set size. I cannot feed this placeholder to tf.truncated_normal because it requires a fully specified tensor shape. What is a simple way to having tf.truncated_normal accept different tensor shapes?

Response:
You just need to feed it in as a single example but in the batched shape. So that means adding an extra dimension to the shape e.g.
```
batch_size = 32 # set this to the actual size of your batch
tf.truncated_normal((batch_size, 784), mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None)
```
This way it will "fit" into the placeholder. If you expect batch_size to change you can also use:
```
tf.truncated_normal(tf.shape(input_tensor), mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None)
```
Where input_tensor could be a placeholder or just whatever tensor is going to have this noise added to it.


Question:
How to properly use tf.metrics.accuracy? I have some trouble using the accuracy function from tf.metrics for a multiple classification problem with logits as input. My model output looks like:
```
logits = [[0.1, 0.5, 0.4],
          [0.8, 0.1, 0.1],
          [0.6, 0.3, 0.2]]
```
And my labels are one hot encoded vectors:
```
labels = [[0, 1, 0],
          [1, 0, 0],
          [0, 0, 1]]
```
When I try to do something like tf.metrics.accuracy(labels, logits) it never gives the correct result. I am obviously doing something wrong but I can't figure what it is.

Response:
TL;DR The accuracy function tf.metrics.accuracy calculates how often predictions matches labels based on two local variables it creates: total and count, that are used to compute the frequency with which logits matches labels.
```
acc, acc_op = tf.metrics.accuracy(labels=tf.argmax(labels, 1),
                                  predictions=tf.argmax(logits,1))

print(sess.run([acc, acc_op]))
print(sess.run([acc]))
# Output
#[0.0, 0.66666669]
#[0.66666669]
```
acc (accuracy): simply returns the metrics using total and count, doesnt update the metrics. acc_op (update up): updates the metrics. To understand why the acc returns 0.0, go through the details below. Details using a simple example:
```
logits = tf.placeholder(tf.int64, [2,3])
labels = tf.Variable([[0, 1, 0], [1, 0, 1]])

acc, acc_op = tf.metrics.accuracy(labels=tf.argmax(labels, 1),
                                  predictions=tf.argmax(logits,1))
```
Initialize the variables: Since metrics.accuracy creates two local variables total and count, we need to call local_variables_initializer() to initialize them.
```
sess = tf.Session()

sess.run(tf.local_variables_initializer())
sess.run(tf.global_variables_initializer())

stream_vars = [i for i in tf.local_variables()]
print(stream_vars)

#[<tf.Variable 'accuracy/total:0' shape=() dtype=float32_ref>,
# <tf.Variable 'accuracy/count:0' shape=() dtype=float32_ref>]
```
Understanding update ops and accuracy calculation:
```
print('acc:',sess.run(acc, {logits:[[0,1,0],[1,0,1]]}))
#acc: 0.0

print('[total, count]:',sess.run(stream_vars))
#[total, count]: [0.0, 0.0]
```
The above returns 0.0 for accuracy as total and count are zeros, inspite of giving matching inputs.
```
print('ops:', sess.run(acc_op, {logits:[[0,1,0],[1,0,1]]}))
#ops: 1.0

print('[total, count]:',sess.run(stream_vars))
#[total, count]: [2.0, 2.0]
```
With the new inputs, the accuracy is calculated when the update op is called. Note: since all the logits and labels match, we get accuracy of 1.0 and the local variables total and count actually give total correctly predicted and the total comparisons made. Now we call accuracy with the new inputs (not the update ops):
```
print('acc:', sess.run(acc,{logits:[[1,0,0],[0,1,0]]}))
#acc: 1.0
```
Accuracy call doesnt update the metrics with the new inputs, it just returns the value using the two local variables. Note: the logits and labels dont match in this case. Now calling update ops again:
```
print('op:',sess.run(acc_op,{logits:[[0,1,0],[0,1,0]]}))
#op: 0.75
print('[total, count]:',sess.run(stream_vars))
#[total, count]: [3.0, 4.0]
```
The metrics are updated to new inputs For more information on how to use the metrics during training and how to reset them during validation, can be found here.


Question:
Darknet YOLO image size I am trying to train custom object classifier in Darknet YOLO v2 https://pjreddie.com/darknet/yolo/ I gathered a dataset for images most of them are 6000 x 4000 px and some lower resolutions as well. Do I need to resize the images before training to be squared ? I found that the config uses:
```
[net]
batch=64
subdivisions=8
height=416
width=416
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1
```
thats why I was wondering how to use it for different sizes of data sets.

Response: You don't have to resize it, because Darknet will do it instead of you! It means you really don't need to do that and you can use different image sizes during your training. What you posted above is just network configuration. There should be full network definition as well. And the height and the width tell you what's the network resolution. And it also keeps aspect ratio, check e.g this.


Question: What is the difference between MaxPool and MaxPooling layers in Keras? I just started working with keras and noticed that there are two layers with very similar names for max-pooling: MaxPool and MaxPooling. I was surprised that I couldn't find the difference between these two on Google; so I am wondering what the difference is between the two if any.

Response: They are basically the same thing (i.e. aliases of each other). For future readers who might want to know how this could be determined: go to the documentation page of the layer (you can use the list here) and click on "View aliases". This is then accompanied by a blue plus sign (+). For example, if you go to MaxPool2D documentation and do this, you will find MaxPooling2D in the list of aliases of this layer as follow:


Question:
Tensorflow: How does tf.get_variable work? I have read about tf.get_variable from this question and also a bit from the documentation available at the tensorflow website. However, I am still not clear and was unable to find an answer online. How does tf.get_variable work? For example:
```
var1 = tf.Variable(3.,dtype=float64)
var2 = tf.get_variable("var1",[],dtype=tf.float64)
```
Does it mean that var2 is another variable with initialization similar to var1? Or is var2 an alias for var1 (I tried and it doesn't seem to)? How are var1 and var2 related? How is a variable constructed when the variable we are getting doesn't really exist?

Response:
tf.get_variable(name) creates a new variable called name (or add _ if name already exists in the current scope) in the tensorflow graph. In your example, you're creating a python variable called var1. The name of that variable in the tensorflow graph is not ** var1, but is Variable:0. Every node you define has its own name that you can specify or let tensorflow give a default (and always different) one. You can see the name value accessing the name property of the python variable. (ie print(var1.name)). On your second line, you're defining a Python variable var2 whose name in the tensorflow graph is var1. The script
```
import tensorflow as tf

var1 = tf.Variable(3.,dtype=tf.float64)
print(var1.name)
var2 = tf.get_variable("var1",[],dtype=tf.float64)
print(var2.name)
```
In fact prints:
```
Variable:0
var1:0
```
If you, instead, want to define a variable (node) called var1 in the tensorflow graph and then getting a reference to that node, you cannot simply use tf.get_variable("var1"), because it will create a new different variable valled var1_1. This script
```
var1 = tf.Variable(3.,dtype=tf.float64, name="var1")
print(var1.name)
var2 = tf.get_variable("var1",[],dtype=tf.float64)
print(var2.name)
```
prints:
```
var1:0
var1_1:0
```
If you want to create a reference to the node var1, you first: Have to replace tf.Variable with tf.get_variable. The variables created with tf.Variable can't be shared, while the latter can. Know what the scope of the var1 is and allow the reuse of that scope when declaring the reference. Looking at the code is the better way for understanding
```
import tensorflow as tf

#var1 = tf.Variable(3.,dtype=tf.float64, name="var1")
var1 = tf.get_variable(initializer=tf.constant_initializer(3.), dtype=tf.float64, name="var1", shape=())
current_scope = tf.contrib.framework.get_name_scope()
print(var1.name)
with tf.variable_scope(current_scope, reuse=True):
    var2 = tf.get_variable("var1",[],dtype=tf.float64)
    print(var2.name)
```
outputs:
```
var1:0
var1:0
```


Question: Is there a way of determining how much GPU memory is in use by TensorFlow? Tensorflow tends to preallocate the entire available memory on it's GPUs. For debugging, is there a way of telling how much of that memory is actually in use?

Response:
(1) There is some limited support with Timeline for logging memory allocations. Here is an example for its usage:
```
run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
    run_metadata = tf.RunMetadata()
    summary, _ = sess.run([merged, train_step],
                          feed_dict=feed_dict(True),
                          options=run_options,
                          run_metadata=run_metadata)
    train_writer.add_run_metadata(run_metadata, 'step%03d' % i)
    train_writer.add_summary(summary, i)
    print('Adding run metadata for', i)
    tl = timeline.Timeline(run_metadata.step_stats)
    print(tl.generate_chrome_trace_format(show_memory=True))
    trace_file = tf.gfile.Open(name='timeline', mode='w')
    trace_file.write(tl.generate_chrome_trace_format(show_memory=True))
```
You can give this code a try with the MNIST example (mnist with summaries) This will generate a tracing file named timeline, which you can open with chrome://tracing. Note that this only gives an approximated GPU memory usage statistics. It basically simulated a GPU execution, but doesn't have access to the full graph metadata. It also can't know how many variables have been assigned to the GPU. (2) For a very coarse measure of GPU memory usage, nvidia-smi will show the total device memory usage at the time you run the command. nvprof can show the on-chip shared memory usage and register usage at the CUDA kernel level, but doesn't show the global/device memory usage. Here is an example command: nvprof --print-gpu-trace matrixMul And more details here: http://docs.nvidia.com/cuda/profiler-users-guide/#abstract


Question: How to add report_tensor_allocations_upon_oom to RunOptions in Keras I'm trying to train a neural net on a GPU using Keras and am getting a "Resource exhausted: OOM when allocating tensor" error. The specific tensor it's trying to allocate isn't very big, so I assume some previous tensor consumed almost all the VRAM. The error message comes with a hint that suggests this: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. That sounds good, but how do I do it? RunOptions appears to be a Tensorflow thing, and what little documentation I can find for it associates it with a "session". I'm using Keras, so Tensorflow is hidden under a layer of abstraction and its sessions under another layer below that. How do I dig underneath everything to set this option in such a way that it will take effect?

Response:
TF1 solution: Its not as hard as it seems, what you need to know is that according to the documentation, the **kwargs parameter passed to model.compile will be passed to session.run So you can do something like:
```
import tensorflow as tf
run_opts = tf.RunOptions(report_tensor_allocations_upon_oom = True)

model.compile(loss = "...", optimizer = "...", metrics = "..", options = run_opts)
```
And it should be passed directly each time session.run is called. TF2: The solution above works only for tf1. For tf2, unfortunately, it appears there is no easy solution yet.


Question:
ValueError: Variable rnn/basic_rnn_cell/kernel already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Any ideas how can I solve problem shown below? With the information that I found on the web it is associated with problem of reusing tensorflow scope however nothing works.
```python
ValueError: Variable rnn/basic_rnn_cell/kernel already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:

  File "/code/backend/management/commands/RNN.py", line 370, in predict
    states_series, current_state = tf.nn.dynamic_rnn(cell=cell, inputs=batchX_placeholder, dtype=tf.float32)
  File "/code/backend/management/commands/RNN.py", line 499, in Command
    predict("string")
  File "/code/backend/management/commands/RNN.py", line 12, in <module>
    class Command(BaseCommand):
```
I tried for instance something like this
```python
with tf.variable_scope('scope'):
 states_series, current_state = tf.nn.dynamic_rnn(cell=cell, inputs=batchX_placeholder, dtype=tf.float32)
```
and this
```python
with tf.variable_scope('scope', reuse = True ):
 states_series, current_state = tf.nn.dynamic_rnn(cell=cell, inputs=batchX_placeholder, dtype=tf.float32)
```
and this
```python
with tf.variable_scope('scope', reuse = tf.AUTO_REUSE ):
 states_series, current_state = tf.nn.dynamic_rnn(cell=cell, inputs=batchX_placeholder, dtype=tf.float32)
```
Any ideas?

Response:
Does this happen when you run the model for the first time (upon opening a new python console)? If not, you need to clear you computational graph. You can do that by putting this line at the beginning of your script.
```
tf.reset_default_graph()
```


Question:
numpy random choice in Tensorflow Is there an equivalent function to numpy random choice in Tensorflow. In numpy we can get an item randomly from the given list with its weights.
```
np.random.choice([1,2,3,5], 1, p=[0.1, 0, 0.3, 0.6, 0])
```
This code will select an item from the given list with p weights.

Response:
No, but you can achieve the same result using tf.multinomial:
```
elems = tf.convert_to_tensor([1,2,3,5])
samples = tf.multinomial(tf.log([[1, 0, 0.3, 0.6]]), 1) # note log-prob
elems[tf.cast(samples[0][0], tf.int32)].eval()
Out: 1
elems[tf.cast(samples[0][0], tf.int32)].eval()
Out: 5
```
The [0][0] part is here, as multinomial expects a row of unnormalized log-probabilities for each element of the batch and also has another dimension for the number of samples.


Question:
Get the bounding box coordinates in the TensorFlow object detection API tutorial I am new to both Python and Tensorflow. I am trying to run the object detection tutorial file from the Tensorflow Object Detection API, but I cannot find where I can get the coordinates of the bounding boxes when objects are detected. Relevant code:
```
# The following processing is only for single image
 detection_boxes = tf.squeeze(tensor_dict['detection_boxes'], [0])
 detection_masks = tf.squeeze(tensor_dict['detection_masks'], [0])
```
The place where I assume bounding boxes are drawn is like this:
```
# Visualization of the results of detection.
 vis_util.visualize_boxes_and_labels_on_image_array(
      image_np,
      output_dict['detection_boxes'],
      output_dict['detection_classes'],
      output_dict['detection_scores'],
      category_index,
      instance_masks=output_dict.get('detection_masks'),
      use_normalized_coordinates=True,
      line_thickness=8)
 plt.figure(figsize=IMAGE_SIZE)
 plt.imshow(image_np)
```
I tried printing output_dict['detection_boxes'] but I am not sure what the numbers mean. There are a lot.
```
array([[ 0.56213236,  0.2780568 ,  0.91445708,  0.69120586],
       [ 0.56261235,  0.86368728,  0.59286624,  0.8893863 ],
       [ 0.57073039,  0.87096912,  0.61292225,  0.90354401],
       [ 0.51422435,  0.78449738,  0.53994244,  0.79437423],
......

       [ 0.32784131,  0.5461576 ,  0.36972913,  0.56903434],
       [ 0.03005961,  0.02714229,  0.47211722,  0.44683522],
       [ 0.43143299, 0.09211366,  0.58121657,  0.3509962 ]], dtype=float32)
```
I found answers for similar questions, but I don't have a variable called boxes as they do. How can I get the coordinates?

Response:
I tried printing output_dict['detection_boxes'] but I am not sure what the numbers mean You can check out the code for yourself. visualize_boxes_and_labels_on_image_array is defined here. Note that you are passing use_normalized_coordinates=True. If you trace the function calls, you will see your numbers [ 0.56213236, 0.2780568 , 0.91445708, 0.69120586] etc. are the values [ymin, xmin, ymax, xmax] where the image coordinates:
```
(left, right, top, bottom) = (xmin * im_width, xmax * im_width,
                              ymin * im_height, ymax * im_height)
```
are computed by the function:
```
def draw_bounding_box_on_image(image,
                           ymin,
                           xmin,
                           ymax,
                           xmax,
                           color='red',
                           thickness=4,
                           display_str_list=(),
                           use_normalized_coordinates=True):
  """Adds a bounding box to an image.
  Bounding box coordinates can be specified in either absolute (pixel) or
  normalized coordinates by setting the use_normalized_coordinates argument.
  Each string in display_str_list is displayed on a separate line above the
  bounding box in black text on a rectangle filled with the input 'color'.
  If the top of the bounding box extends to the edge of the image, the strings
  are displayed below the bounding box.
  Args:
    image: a PIL.Image object.
    ymin: ymin of bounding box.
    xmin: xmin of bounding box.
    ymax: ymax of bounding box.
    xmax: xmax of bounding box.
    color: color to draw bounding box. Default is red.
    thickness: line thickness. Default value is 4.
    display_str_list: list of strings to display in box
                      (each to be shown on its own line).
    use_normalized_coordinates: If True (default), treat coordinates
      ymin, xmin, ymax, xmax as relative to the image.  Otherwise treat
      coordinates as absolute.
  """
  draw = ImageDraw.Draw(image)
  im_width, im_height = image.size
  if use_normalized_coordinates:
    (left, right, top, bottom) = (xmin * im_width, xmax * im_width,
                                  ymin * im_height, ymax * im_height)
```


Question: How does TensorFlow's MultiRnnCell work? Could someone help to explain the inner mechanism of TensorFlow's tf.contrib.rnn.MultiRnnCell? For example, if I wanted to stack up two basic RNN cells into a MultiRnnCell, what would be the input and output of each basic RNN cell? I would like to know the details of how it works.

Response: Study this blog post as well as the provided implementation. It describes in detail how use MultiRNNCell to stack multiple RNN cells.


Question:
What is the difference between [], [None], None and () for the shape of a placeholder? I have seen pieces of code using either [], [None], None or () as the shape for a placeholder, that is
```
x = tf.placeholder(..., shape=[], ...)
y = tf.placeholder(..., shape=[None], ...)
z = tf.placeholder(..., shape=None, ...)
w = tf.placeholder(..., shape=(), ...)
```
What's the difference between these?

Response:
TensorFlow uses arrays rather than tuples. It converts tuples to arrays. Therefore [] and () are equivalent. Now, consider this code example:
```
x = tf.placeholder(dtype=tf.int32, shape=[], name="foo1")
y = tf.placeholder(dtype=tf.int32, shape=[None], name="foo2")
z = tf.placeholder(dtype=tf.int32, shape=None, name="foo3")

val1 = np.array((1, 2, 3))
val2 = 45

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    #print(sess.run(x, feed_dict = {x: val1}))  # Fails
    print(sess.run(y, feed_dict = {y: val1}))
    print(sess.run(z, feed_dict = {z: val1}))

    print(sess.run(x, feed_dict = {x: val2}))
    #print(sess.run(y, feed_dict = {y: val2}))  # Fails
    print(sess.run(z, feed_dict = {z: val2}))
```
As can be seen, placeholder with [] shape takes a single scalar value directly. Placeholder with [None] shape takes a 1-dimensional array and placeholder with None shape can take in any value while computation takes place.


Question:
TensorFlow/Keras multi-threaded model fitting I'm attempting to train multiple keras models with different parameter values using multiple threads (and the tensorflow backend). I've seen a few examples of using the same model within multiple threads, but in this particular case, I run into various errors regarding conflicting graphs, etc. Here's a simple example of what I'd like to be able to do:
```py
from concurrent.futures import ThreadPoolExecutor
import numpy as np
import tensorflow as tf
from keras import backend as K
from keras.layers import Dense
from keras.models import Sequential

sess = tf.Session()

def example_model(size):
    model = Sequential()
    model.add(Dense(size, input_shape=(5,)))
    model.add(Dense(1))
    model.compile(optimizer='sgd', loss='mse')
    return model

if __name__ == '__main__':
    K.set_session(sess)
    X = np.random.random((10, 5))
    y = np.random.random((10, 1))
    models = [example_model(i) for i in range(5, 10)]

    e = ThreadPoolExecutor(4)
    res_list = [e.submit(model.fit, X, y) for model in models]

    for res in res_list:
        print(res.result())
```
The resulting error is ValueError: Tensor("Variable:0", shape=(5, 5), dtype=float32_ref) must be from the same graph as Tensor("Variable_2/read:0", shape=(), dtype=float32).. I've also tried initializing the models within the threads which gives a similar failure. Any thoughts on the best way to go about this? I'm not at all attached to this exact structure, but I'd prefer to be able to use multiple threads rather than processes so all the models are trained within the same GPU memory allocation.

Response:
Tensorflow Graphs are not threadsafe (see https://www.tensorflow.org/api_docs/python/tf/Graph) and when you create a new Tensorflow Session, it by default uses the default graph. You can get around this by creating a new session with a new graph in your parallelized function and constructing your keras model there. Here is some code that creates and fits a model on each available gpu in parallel:
```py
import concurrent.futures
import numpy as np

import keras.backend as K
from keras.layers import Dense
from keras.models import Sequential

import tensorflow as tf
from tensorflow.python.client import device_lib

def get_available_gpus():
    local_device_protos = device_lib.list_local_devices()
    return [x.name for x in local_device_protos if x.device_type == 'GPU']

xdata = np.random.randn(100, 8)
ytrue = np.random.randint(0, 2, 100)

def fit(gpu):
    with tf.Session(graph=tf.Graph()) as sess:
        K.set_session(sess)
        with tf.device(gpu):
            model = Sequential()
            model.add(Dense(12, input_dim=8, activation='relu'))
            model.add(Dense(8, activation='relu'))
            model.add(Dense(1, activation='sigmoid'))

            model.compile(loss='binary_crossentropy', optimizer='adam')
            model.fit(xdata, ytrue, verbose=0)

            return model.evaluate(xdata, ytrue, verbose=0)

gpus = get_available_gpus()
with concurrent.futures.ThreadPoolExecutor(len(gpus)) as executor:
    results = [x for x in executor.map(fit, gpus)]
print('results: ', results)
```


Question:
Is sparse tensor multiplication implemented in TensorFlow? Multiplication of sparse tensors with themselves or with dense tensors does not seem to work in TensorFlow. The following example
```py
from __future__ import print_function
import tensorflow as tf

x = tf.constant([[1.0,2.0],
                 [3.0,4.0]])
y = tf.SparseTensor(indices=[[0,0],[1,1]], values=[1.0,1.0], shape=[2,2])
z = tf.matmul(x,y)

sess = tf.Session()
sess.run(tf.initialize_all_variables())
print(sess.run([x, y, z]))
```
fails with the error message
```py
TypeError: Input 'b' of 'MatMul' Op has type string that does not match type
float32 of argument 'a'
```
Both tensors have values of type float32 as seen by evaluating them without the multiplication op. Multiplication of y with itself returns a similar error message. Multipication of x with itself works fine.

Response:
General-purpose multiplication for tf.SparseTensor is not currently implemented in TensorFlow. However, there are three partial solutions, and the right one to choose will depend on the characteristics of your data: If you have a tf.SparseTensor and a tf.Tensor, you can use tf.sparse_tensor_dense_matmul() to multiply them. This is more efficient than the next approach if one of the tensors is too large to fit in memory when densified: the documentation has more guidance about how to decide between these two methods. Note that it accepts a tf.SparseTensor as the first argument, so to solve your exact problem you will need to use the adjoint_a and adjoint_b arguments, and transpose the result. If you have two sparse tensors and need to multiply them, the simplest (if not the most performant) way is to convert them to dense and use tf.matmul:
```
a = tf.SparseTensor(...)
b = tf.SparseTensor(...)

c = tf.matmul(tf.sparse_tensor_to_dense(a, 0.0),
              tf.sparse_tensor_to_dense(b, 0.0),
              a_is_sparse=True, b_is_sparse=True)
```
Note that the optional a_is_sparse and b_is_sparse arguments mean that "a (or b) has a dense representation but a large number of its entries are zero", which triggers the use of a different multiplication algorithm. For the special case of sparse vector by (potentially large and sharded) dense matrix multiplication, and the values in the vector are 0 or 1, the tf.nn.embedding_lookup operator may be more appropriate. This tutorial discusses when you might use embeddings and how to invoke the operator in more detail. For the special case of sparse matrix by (potentially large and sharded) dense matrix, tf.nn.embedding_lookup_sparse() may be appropriate. This function accepts one or two tf.SparseTensor objects, with sp_ids representing the non-zero values, and the optional sp_weights representing their values (which otherwise default to one).


Question: Machine Learning (tensorflow / sklearn) in Django? I have a django form, which is collecting user response. I also have a tensorflow sentences classification model. What is the best/standard way to put these two together. Details: tensorflow model was trained on the Movie Review data from Rotten Tomatoes. Everytime a new row is made in my response model , i want the tensorflow code to classify it( + or - ). Basically I have a django project directory and two .py files for classification. Before going ahead myself , i wanted to know what is the standard way to implement machine learning algorithms to a web app. It'd be awesome if you could suggest a tutorial or a repo. Thank you !

Response: Asynchronous processing If you don't need the classification result from the ML code to pass immediately to the user (e.g. as a response to the same POST request that submtted), then you can always queue the classification job to be ran in the background or even a different server with more CPU/memory resources (e.g. with django-background-tasks or Celery) A queued task would be for example to populate the field UserResponse.class_name (positive, negative) on the database rows that have that field blank (not yet classified) Real time notification If the ML code is slow and want to return that result to the user as soon as it is available, you can use the asynchronous approach described above, and pair with the real time notification (e.g. socket.io to the browser (this can be triggered from the queued task) This becomes necessary if ML execution time is so long that it might time-out the HTTP request in the synchronous approach described below. Synchronous processing, if ML code is not CPU intensive (fast enough) If you need that classification result returned immediately, and the ML classification is fast enough *, you can do so within the HTTP request-response cycle (the POST request returns after the ML code is done, synchronously) *Fast enough here means it wouldn't time-out the HTTP request/response, and the user wouldn't lose patience.


Question:
ValueError: Tensor conversion requested dtype float32 for Tensor with dtype int32 I get following error
```
ValueError: Tensor conversion requested dtype float32 for Tensor with dtype int32: 'Tensor("Placeholder_1:0", shape=TensorShape([Dimension(128), Dimension(2)]), dtype=int32)'
```
when I try to calculate cross entropy loss
```
losses = tf.nn.softmax_cross_entropy_with_logits(scores, input_y)
```
I use Python 3.4.3. Any ideas why?

Response: It sounds like you have defined input_y—which I am assuming is a tf.placeholder()—as having type tf.int32. Either change this to tf.float32 or add a cast: tf.cast(input_y, tf.float32) or tf.to_float(input_y).


Question:
Should the custom loss function in Keras return a single loss value for the batch or an arrary of losses for every sample in the training batch? I'm learning keras API in tensorflow(2.3). In this guide on tensorflow website, I found an example of custom loss funciton:
```
def custom_mean_squared_error(y_true, y_pred):
        return tf.math.reduce_mean(tf.square(y_true - y_pred))
```
The reduce_mean function in this custom loss function will return an scalar. Is it right to define loss function like this? As far as I know, the first dimension of the shapes of y_true and y_pred is the batch size. I think the loss function should return loss values for every sample in the batch. So the loss function shoud give an array of shape (batch_size,). But the above function gives a single value for the whole batch. Maybe the above example is wrong? Could anyone give me some help on this problem? p.s. Why do I think the loss function should return an array rather than a single value? I read the source code of Model class. When you provide a loss function (please note it's a function, not a loss class) to Model.compile() method, ths loss function is used to construct a LossesContainer object, which is stored in Model.compiled_loss. This loss function passed to the constructor of LossesContainer class is used once again to construct a LossFunctionWrapper object, which is stored in LossesContainer._losses. According to the source code of LossFunctionWrapper class, the overall loss value for a training batch is calculated by the LossFunctionWrapper.__call__() method (inherited from Loss class), i.e. it returns a single loss value for the whole batch. But the LossFunctionWrapper.__call__() first calls the LossFunctionWrapper.call() method to obtain an array of losses for every sample in the training batch. Then these losses are fianlly averaged to get the single loss value for the whole batch. It's in the LossFunctionWrapper.call() method that the loss function provided to the Model.compile() method is called. That's why I think the custom loss funciton should return an array of losses, insead of a single scalar value. Besides, if we write a custom Loss class for the Model.compile() method, the call() method of our custom Loss class should also return an array, rather than a signal value. I opened an issue on github. It's confirmed that custom loss function is required to return one loss value per sample. The example will need to be updated to reflect this.

Response:
Actually, as far as I know, the shape of return value of the loss function is not important, i.e. it could be a scalar tensor or a tensor of one or multiple values per sample. The important thing is how it should be reduced to a scalar value so that it could be used in optimization process or shown to the user. For that, you can check the reduction types in Reduction documentation. Further, here is what the compile method documentation says about the loss argument, partially addressing this point: loss: String (name of objective function), objective function or tf.keras.losses.Loss instance. See tf.keras.losses. An objective function is any callable with the signature loss = fn(y_true,y_pred), where y_true = ground truth values with shape = [batch_size, d0, .. dN], except sparse loss functions such as sparse categorical crossentropy where shape = [batch_size, d0, .. dN-1]. y_pred = predicted values with shape = [batch_size, d0, .. dN]. It returns a weighted loss float tensor. If a custom Loss instance is used and reduction is set to NONE, return value has the shape [batch_size, d0, .. dN-1] ie. per-sample or per-timestep loss values; otherwise, it is a scalar. If the model has multiple outputs, you can use a different loss on each output by passing a dictionary or a list of losses. The loss value that will be minimized by the model will then be the sum of all individual losses. In addition, it's worth noting that most of the built-in loss functions in TF/Keras are usually reduced over the last dimension (i.e. axis=-1). For those who doubt that a custom loss function which returns a scalar value would work: you can run the following snippet and you will see that the model would train and converge properly.
```
import tensorflow as tf
import numpy as np

def custom_loss(y_true, y_pred):
    return tf.reduce_sum(tf.square(y_true - y_pred))

inp = tf.keras.layers.Input(shape=(3,))
out = tf.keras.layers.Dense(3)(inp)

model = tf.keras.Model(inp, out)
model.compile(loss=custom_loss, optimizer=tf.keras.optimizers.Adam(lr=0.1))

x = np.random.rand(1000, 3)
y = x * 10 + 2.5
model.fit(x, y, epochs=20)
```


Question:
Add L2 regularization when using high level tf.layers Is it possible to add an L2 regularization when using the layers defined in tf.layers? It seems to me that since tf.layers is an high level wrapper, there is no easy way to get access to the filter weights. With tf.nn.conv2d
```
regularizer = tf.contrib.layers.l2_regularizer(scale=0.1)

weights = tf.get_variable(
    name="weights",
    regularizer=regularizer
)

#Previous layers

...

#Second layer
layer 2 = tf.nn.conv2d(
input,
weights,
[1,1,1,1],
[1,1,1,1])

#More layers
...

#Loss
loss = #some loss

reg_variables = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
reg_term = tf.contrib.layers.apply_regularization(regularizer, reg_variables)
loss += reg_term
```
Now what would that look like with tf.layers.conv2d? Thanks!

Response:
You can pass them into tf.layers.conv2d as arguments:
```
regularizer = tf.contrib.layers.l2_regularizer(scale=0.1)
layer2 = tf.layers.conv2d(
    inputs,
    filters,
    kernel_size,
    kernel_regularizer=regularizer)
```
Then you should add the regularization loss to your loss like this:
```
l2_loss = tf.losses.get_regularization_loss()
loss += l2_loss
```
Edit: Thanks Zeke Arneodo, Tom and srcolinas I added, the last bit on your feedback so that the accepted answer provides the complete solution.


Question: For what reason Convolution 1x1 is used in deep neural networks? I'm looking at InceptionV3 (GoogLeNet) architecture and cannot understand why do we need conv1x1 layers? I know how convolution works, but I see a profit with patch size > 1.

Response: You can think about 1x1xD convolution as a dimensionality reduction technique when it's placed somewhere into a network. If you have an input volume of 100x100x512 and you convolve it with a set of D filters each one with size 1x1x512 you reduce the number of features from 512 to D. The output volume is, therefore, 100x100xD. As you can see this (1x1x512)xD convolution is mathematically equivalent to a fully connected layer. The main difference is that whilst FC layer requires the input to have a fixed size, the convolutional layer can accept in input every volume with spatial extent greater or equal than 100x100. A 1x1xD convolution can substitute any fully connected layer because of this equivalence. In addition, 1x1xD convolutions not only reduce the features in input to the next layer, but also introduces new parameters and new non-linearity into the network that will help to increase model accuracy. When the 1x1xD convolution is placed at the end of a classification network, it acts exactly as a FC layer, but instead of thinking about it as a dimensionality reduction technique it's more intuitive to think about it as a layer that will output a tensor with shape WxHxnum_classes. The spatial extent of the output tensor (identified by W and H) is dynamic and is determined by the locations of the input image that the network analyzed. If the network has been defined with an input of 200x200x3 and we give it in input an image with this size, the output will be a map with W = H = 1 and depth = num_classes. But, if the input image have a spatial extent greater than 200x200 than the convolutional network will analyze different locations of the input image (just like a standard convolution does) and will produce a tensor with W > 1 and H > 1. This is not possibile with a FC layer that constrains the network to accept fixed size input and produce fixed size output.


Question:
Tensorflow2 warning using @tffunction This example code from Tensorflow 2
```
writer = tf.summary.create_file_writer("/tmp/mylogs/tf_function")

@tf.function
def my_func(step):
  with writer.as_default():
    # other model code would go here
    tf.summary.scalar("my_metric", 0.5, step=step)

for step in range(100):
  my_func(step)
  writer.flush()
```
but it is throwing warnings. WARNING:tensorflow:5 out of the last 5 calls to triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/beta/tutorials/eager/tf_function#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details. Is there a better way to do this?

Response:
tf.function has some "peculiarities". I highly recommend reading this article: https://www.tensorflow.org/tutorials/customization/performance In this case, the problem is that the function is "retraced" (i.e. a new graph is built) every time you call with a different input signature. For tensors, input signature refers to shape and dtype, but for Python numbers, every new value is interpreted as "different". In this case, because you call the function with a step variable that changes every time, the function is retraced every single time as well. This will be extremely slow for "real" code (e.g. calling a model inside the function). You can fix it by simply converting step to a tensor, in which case the different values will not count as a new input signature:
```
for step in range(100):
    step = tf.convert_to_tensor(step, dtype=tf.int64)
    my_func(step)
    writer.flush()
```
or use tf.range to get tensors directly:
```
for step in tf.range(100):
    step = tf.cast(step, tf.int64)
    my_func(step)
    writer.flush()
```
This should not produce warnings (and be much faster).


Question: tensorflow: what's the difference between tf.nn.dropout and tf.layers.dropout I'm quite confused about whether to use tf.nn.dropout or tf.layers.dropout. many MNIST CNN examples seems to use tf.nn.droput, with keep_prop as one of params. but how is it different with tf.layers.dropout? is the "rate" params in tf.layers.dropout similar to tf.nn.dropout? Or generally speaking, is the difference between tf.nn.dropout and tf.layers.dropout applies to all other similar situations, like similar functions in tf.nn and tf.layers.

Response: A quick glance through tensorflow/python/layers/core.py and tensorflow/python/ops/nn_ops.py reveals that tf.layers.dropout is a wrapper for tf.nn.dropout. The only differences in the two functions are: The tf.nn.dropout has parameter keep_prob: "Probability that each element is kept" tf.layers.dropout has parameter rate: "The dropout rate" Thus, keep_prob = 1 - rate as defined here The tf.layers.dropout has training parameter: "Whether to return the output in training mode (apply dropout) or in inference mode (return the input untouched)."


Question: Why would I ever use tf.concat instead of tf.stack? Is there a good reason to use tf.concat instead of tf.stack? They seem very similar. Is it just to guarantee that the resulting tensor will have the same number of dimensions as the input list of tensors?

Response:
Actually, I've misunderstood how tf.stack works. If the axis parameter is within the range of the existing dimensions, a new axis will be inserted at that index. Example:
```
import tensorflow as tf

t1 = tf.random_normal([1, 3])
t2 = tf.random_normal([1, 3])

tf.stack([t1, t2], axis=1).shape.as_list() == [1, 2, 3]
tf.concat([t1, t2], axis=1).shape.as_list() == [1, 6]
```


Question: How does asynchronous training work in distributed Tensorflow? I've read Distributed Tensorflow Doc, and it mentions that in asynchronous training, each replica of the graph has an independent training loop that executes without coordination. From what I understand, if we use parameter-server with data parallelism architecture, it means each worker computes gradients and updates its own weights without caring about other workers updates for distributed training Neural Network. As all weights are shared on parameter server (ps), I think ps still has to coordinate (or aggregate) weight updates from all workers in some way. I wonder how does the aggregation work in asynchronous training. Or in more general words, how does asynchronous training work in distributed Tensorflow?

Response: When you train asynchronously in Distributed TensorFlow, a particular worker does the following: The worker reads all of the shared model parameters in parallel from the PS task(s), and copies them to the worker task. These reads are uncoordinated with any concurrent writes, and no locks are acquired: in particular the worker may see partial updates from one or more other workers (e.g. a subset of the updates from another worker may have been applied, or a subset of the elements in a variable may have been updated). The worker computes gradients locally, based on a batch of input data and the parameter values that it read in step 1. The worker sends the gradients for each variable to the appropriate PS task, and applies the gradients to their respective variable, using an update rule that is determined by the optimization algorithm (e.g. SGD, SGD with Momentum, Adagrad, Adam, etc.). The update rules typically use (approximately) commutative operations, so they may be applied independently on the updates from each worker, and the state of each variable will be a running aggregate of the sequence of updates received. In asynchronous training, each update from the worker is applied concurrently, and the updates may be somewhat coordinated if the optional use_locking=True flag was set when the respective optimizer (e.g. tf.train.GradientDescentOptimizer) was initialized. Note however that the locking here only provides mutual exclusion for two concurrent updates, and (as noted above) reads do not acquire locks; the locking does not provide atomicity across the entire set of updates. (By contrast, in synchronous training, a utility like tf.train.SyncReplicasOptimizer will ensure that all of the workers read the same, up-to-date values for each model parameter; and that all of the updates for a synchronous step are aggregated before they are applied to the underlying variables. To do this, the workers are synchronized by a barrier, which they enter after sending their gradient update, and leave after the aggregated update has been applied to all variables.)


Question:
TensorFlow: using a tensor to index another tensor I have a basic question about how to do indexing in TensorFlow. In numpy:
```
x = np.asarray([1,2,3,3,2,5,6,7,1,3])
e = np.asarray([0,1,0,1,1,1,0,1])
#numpy
print x * e[x]
```
I can get
```
[1 0 3 3 0 5 0 7 1 3]
```
How can I do this in TensorFlow?
```
x = np.asarray([1,2,3,3,2,5,6,7,1,3])
e = np.asarray([0,1,0,1,1,1,0,1])
x_t = tf.constant(x)
e_t = tf.constant(e)
with tf.Session():
    ????
```
Thanks!

Response:
Fortunately, the exact case you're asking about is supported in TensorFlow by tf.gather():
```
result = x_t * tf.gather(e_t, x_t)

with tf.Session() as sess:
    print sess.run(result)  # ==> 'array([1, 0, 3, 3, 0, 5, 0, 7, 1, 3])'
```
The tf.gather() op is less powerful than NumPy's advanced indexing: it only supports extracting full slices of a tensor on its 0th dimension. Support for more general indexing has been requested, and is being tracked in this GitHub issue.


Question:
TensorFlow Training Assuming I have a very simple neural network, like multilayer perceptron. For each layer the activation function is sigmoid and the network are fully connected. In TensorFlow this might be defined like this:
```py
sess = tf.InteractiveSession()

    # Training Tensor
    x = tf.placeholder(tf.float32, shape = [None, n_fft])
    # Label Tensor
    y_ = tf.placeholder(tf.float32, shape = [None, n_fft])

    # Declaring variable buffer for weights W and bias b
    # Layer structure [n_fft, n_fft, n_fft, n_fft]
    # Input -> Layer 1
    struct_w = [n_fft, n_fft]
    struct_b = [n_fft]
    W1 = weight_variable(struct_w, 'W1')
    b1 = bias_variable(struct_b, 'b1')
    h1 = tf.nn.sigmoid(tf.matmul(x, W1) + b1)

    # Layer1 -> Layer 2
    W2 = weight_variable(struct_w, 'W2')
    b2 = bias_variable(struct_b, 'b2')
    h2 = tf.nn.sigmoid(tf.matmul(h1, W2) + b2)

    # Layer2 -> output
    W3 = weight_variable(struct_w, 'W3')
    b3 = bias_variable(struct_b, 'b3')
    y = tf.nn.sigmoid(tf.matmul(h2, W3) + b3)

    # Calculating difference between label and output using mean square error
    mse = tf.reduce_mean(tf.square(y - y_))

    # Train the Model
    # Gradient Descent
    train_step = tf.train.GradientDescentOptimizer(0.3).minimize(mse)
```
The design target for this model is to map a n_fft points fft spectrogram to another n_fft target spectrogram. Let's assume both the training data and target data are of size [3000, n_fft]. They are stored in variables spec_train and spec_target. Now here comes the question. For TensorFlow is there any difference between these two trainings? Training 1:
```py
for i in xrange(200):
        train_step.run(feed_dict = {x: spec_train, y_: spec_target})
```
Training 2:
```py
for i in xrange(200):
        for j in xrange(3000):
            train = spec_train[j, :].reshape(1, n_fft)
            label = spec_target[j, :].reshape(1, n_fft)
            train_step.run(feed_dict = {x: train, y_: label})
```
Thank you very much!

Response: In the first training version, you are training the entire batch of training data at once, which means that the first and the 3000th element of spec_train will be processed using the same model parameters in a single step. This is known as (Batch) Gradient Descent. In the second training version, you are training a single example from the training data at once, which means that the 3000th element of spec_train will be processed using model parameters that have been updated 2999 times since the first element was most recently processed. This is known as Stochastic Gradient Descent (or it would be if the element was selected at random). In general, TensorFlow is used with datasets that are too large to process in one batch, so mini-batch SGD (where a subset of the examples are processed in one step) is favored. Processing a single element at a time is theoretically desirable, but is inherently sequential and has high fixed costs because the matrix multiplications and other operations are not as computationally dense. Therefore, processing a small batch (e.g. 32 or 128) of examples at once is the usual approach, with multiple replicas training on different batches in parallel. See this Stats StackExchange question for a more theoretical discussion of when you should use one approach versus the other.


Question:
What are the constraints for tensorflow scope names? I'm running a tensorflow model and getting the following error:
```none
ValueError: 'Cement (component 1)(kg in a m^3 mixture)' is not a valid scope name.
```
I get that tensorflow probably doesn't like special chars and spaces in its scope names, but I'm trying to find an actual doc on what chars are allowed. Does anyone know where I could find this?

Response:
From the TF source: NOTE: This constructor validates the given name. Valid scope names match one of the following regular expressions:
```
[A-Za-z0-9.][A-Za-z0-9_.\\-/]* (for scopes at the root)
[A-Za-z0-9_.\\-/]* (for other scopes)
```


Question: What is the difference between tensors and sparse tensors? I am having troubles understanding the meaning and usages for Tensorflow Tensors and Sparse Tensors. According to the documentation Tensor Tensor is a typed multi-dimensional array. For example, you can represent a mini-batch of images as a 4-D array of floating point numbers with dimensions [batch, height, width, channels]. Sparse Tensor TensorFlow represents a sparse tensor as three separate dense tensors: indices, values, and shape. In Python, the three tensors are collected into a SparseTensor class for ease of use. If you have separate indices, values, and shape tensors, wrap them in a SparseTensor object before passing to the ops below. My understandings are Tensors are used for operations, input and output. And Sparse Tensor is just another representation of a Tensor(dense?). Hope someone can further explain the differences, and the use cases for them.

Response:
Matthew did a great job but I would love to give an example to shed more light on Sparse tensors with a example. If a tensor has lots of values that are zero, it can be called sparse. Lets consider a sparse 1-D Tensor
```
[0, 7, 0, 0, 8, 0, 0, 0, 0]
```
A sparse representation of the same tensor will focus only on the non-zero values
```
values = [7,8]
```
We also have to remember where those values occurs, by their indices
```
indices = [1,4]
```
The one-dimensional indices form will work with some methods, for this one-dimensional example, but in general indices have multiple dimensions, so it will be more consistent (and work everywhere) to represent indices like this:
```
indices = [[1], [4]]
```
With values and indices, we don't have quite enough information yet. How many zeros are there? We represent dense shape of a tensor.
```
dense_shape = [9]
```
These three things together, values, indices, and dense_shape, are a sparse representation of the tensor In tensorflow 2.0 it can be implemented as
```
x = tf.SparseTensor(values=[7,8],indices=[[1],[4]],dense_shape=[9])
x
#o/p: <tensorflow.python.framework.sparse_tensor.SparseTensor at 0x7ff04a58c4a8>

print(x.values)
print(x.dense_shape)
print(x.indices)
#o/p:
tf.Tensor([7 8], shape=(2,), dtype=int32)
tf.Tensor([9], shape=(1,), dtype=int64)
tf.Tensor(
[[1]
 [4]], shape=(2, 1), dtype=int64)
```
EDITED to correct indices as pointed out in the comments.


Question:
How to initialise only optimizer variables in Tensorflow? I want to use MomentumOptimizer in Tensorflow. However, since this optimizer uses some internal variable, attempting to use it without initializing this variable yields an error: FailedPreconditionError (see above for traceback): Attempting to use uninitialized value Variable_2/Momentum This can be easily solved by initializing all variables, using for example
```
tf.global_variables_initializer().run()
```
However, I do not want to initialize all the variables - only those of optimizer. Is there any way to do this?

Response:
Both current answers kinda work by filtering the variable name using the 'Momentum' string. But that is very brittle on two sides: It could silently (re-)initialize some other variables you don't actually want to reset! Either simply because of a name-clash, or because you have a more complex graph and optimize different parts separately, for example. It will only work for one specific optimizer, and how do you know the names to look out for for others? Bonus: an update to tensorflow might silently break your code. Fortunately, tensorflow's abstract Optimizer class has a mechanism for that, these extra optimizer variables are called "slots", and you can get all slot names of an optimizer using the get_slot_names() method:
```
opt = tf.train.MomentumOptimizer(...)
print(opt.get_slot_names())
# prints ['momentum']
```
And you can get the variable corresponding to the slot for a specific (trainable) variable v using the get_slot(var, slot_name) method:
```
opt.get_slot(some_var, 'momentum')
```
Putting all this together, you can create an op that initializes the optimizer's state as follows:
```
var_list = # list of vars to optimize, e.g.
           # tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)
opt = tf.train.MomentumOptimizer(0.1, 0.95)
step_op = opt.minimize(loss, var_list=var_list)
reset_opt_op = tf.variables_initializer([opt.get_slot(var, name) for name in opt.get_slot_names() for var in var_list])
```
This will really only reset the correct variables, and be robust across optimizers. Except for one unfortunate caveat: AdamOptimizer. That one also keeps a counter for how often it's been called. That means you should really think hard about what you're doing here anyways, but for completeness' sake, you can get its extra states as opt._get_beta_accumulators(). The returned list should be added to the list in the above reset_opt_op line.