Question: Parallelization strategies for deep learning What strategies and forms of parallelization are feasible and available for training and serving a neural network?: inside a machine across cores (e.g. GPU / TPU / CPU) across machines on a network or a rack I'm also looking for evidence for how they may also be used in e.g. TensorFlow, PyTorch or MXNet. Training To my knowledge, when training large neural networks on large datasets, one could at least have: Different cores or machines operate on different parts of the graph ("graph splitting"). E.g. backpropagation through the graph itself can be parallelized e.g. by having different layers hosted on different machines since (I think?) the autodiff graph is always a DAG. Different cores or machines operate on different samples of data ("data splitting"). In SGD, the computation of gradients across batches or samples can also be parallelized (e.g. the gradients can be combined after computing them independently on different batches). I believe this is also called gradient accumulation (?). When is each strategy better for what type of problem or neural network? Which modes are supported by modern libraries? and can one combine all four (2x2) strategies? On top of that, I have read about: Asynchronous training Synchronous training but I don't know what exactly that refers to, e.g. is it the computation of gradients on different data batches or the computation of gradients on different subgraphs? Or perhaps it refers to something else altogether? Serving If the network is huge, prediction / inference may also be slow, and the model may not fit on a single machine in memory at serving time. Are there any known multi-core and multi-node prediction solutions that work that can handle such models?

Response:
Training In general, there are two strategies of parallelizing model training: data parallelism and model parallelism. 1. Data parallelism This strategy splits training data into N partitions, each of which will be trained on different “devices” (different CPU cores, GPUs, or even machines). In contrast to training without data parallelism which produces one gradient per minibatch, we now have N gradients for each minibatch step. The next question is how we should combine these N gradients. One way to do it is by averaging all the N gradients and then updating the model parameters once based on the average. This technique is called synchronous distributed SGD. By doing the average, we have a more accurate gradient, but with a cost of waiting all the devices to finish computing its own local gradient. Another way is by not combining the gradients — each gradient will instead be used to update the model parameters independently. So, there will be N parameter updates for each minibatch step, in contrast to only one for the previous technique. This technique is called asynchronous distributed SGD. Because it doesn't have to wait other devices to finish, the async approach will take less time to complete a minibatch step than the sync approach will do. However, the async approach will produce a more noisy gradient, so it might need to complete more minibatch steps to catch up with the performance (in terms of loss) of the sync approach. There are many papers proposing some improvements and optimizations on either approach, but the main idea is generally the same as described above. In the literature there's been some disagreement on which technique is better in practice. At the end most people now settle on the synchronous approach. Data Parallelism in PyTorch To do synchronous SGD, we can wrap our model with torch.nn.parallel.DistributedDataParallel:
```
from torch.nn.parallel import DistributedDataParallel as DDP

# `model` is the model we previously initialized
model = ...

# `rank` is a device number starting from 0
model = model.to(rank)
ddp_model = DDP(model, device_ids=[rank])
```
Then we can train it similarly. For more details, you can refer to the official tutorial. For doing asynchronous SGD in PyTorch, we need to implement it more manually since there is no wrapper similar to DistributedDataParallel for it. Data Parallelism in TensorFlow/Keras For synchronous SGD, we can use tf.distribute.MirroredStrategy to wrap the model initalization:
```
import tensorflow as tf

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    model = Model(...)
    model.compile(...)
```
Then we can train it as usual. For more details, you can refer to the official guides on Keras website and TensorFlow website. For asynchronous SGD, we can use tf.distribute.experimental.ParameterServerStrategy similarly. 2. Model Parallelism This strategy splits the model into N parts, each of which will be computed on different devices. A common way to split the model is based on layers: different sets of layers are placed on different devices. But we can also split it more intricately depending on the model architecture. Model Parallelism in TensorFlow and PyTorch To implement model parallelism in either TensorFlow or PyTorch, the idea is the same: to move some model parameters into a different device. In PyTorch we can use torch.nn.Module.to method to move a module into a different device. For example, suppose we want to create two linear layers each of which is placed on a different GPU:
```
import torch.nn as nn

linear1 = nn.Linear(16, 8).to('cuda:0')
linear2 = nn.Linear(8, 4).to('cuda:1')
```
In TensorFlow we can use tf.device to place an operation into a specific device. To implement the PyTorch example above in TensorFlow:
```
import tensorflow as tf
from tensorflow.keras import layers

with tf.device('/GPU:0'):
    linear1 = layers.Dense(8, input_dim=16)
with tf.device('/GPU:1'):
    linear2 = layers.Dense(4, input_dim=8)
```
For more details you can refer to the official PyTorch tutorial; or if you use TensorFlow you can even use a more high-level library like mesh. 3. Hybrid: Data and Model Parallelism Recall that data parallelism only splits the training data, whereas model parallelism only splits the model structures. If we have a model so large that even after using either parallelism strategy it still doesn't fit in the memory, we can always do both. In practice most people prefer data parallelism to model parallelism since the former is more decoupled (in fact, independent) from the model architecture than the latter. That is, by using data parallelism they can change the model architecture as they like, without worrying which part of the model should be parallelized. Model Inference / Serving Parallelizing model serving is easier than parallelizing model training since the model parameters are already fixed and each request can be processed independently. Similar to scaling a regular Python web service, we can scale model serving by spawning more processes (to workaround Python's GIL) in a single machine, or even spawning more machine instances. When we use a GPU to serve the model, though, we need to do more work to scale it. Because of how concurrency is handled differently by a GPU compared to a CPU, in order to maximize the performance, we need to do inference request batching. The idea is when a request comes, instead of immediately processing it, we wait some timeout duration for other requests to come. When the timeout is up, even if the number of requests is only one, we batch them all to be processed on the GPU. In order to minimize the average request latency, we need to find the optimal timeout duration. To find it we need to observe that there is a trade-off between minimizing the timeout duration and maximizing the number of batch size. If the timeout is too low, the batch size will be small, so the GPU will be underutilized. But if the timeout is too high, the requests that come early will wait too long before they get processed. So, the optimal timeout duration depends on the model complexity (hence, the inference duration) and the average requests per second to receive. Implementing a scheduler to do request batching is not a trivial task, so instead of doing it manually, we'd better use TensorFlow Serving or PyTorch Serve which already supports it. To learn more about parallel and distributed learning, you can read this review paper.


Question:
Tensorflow median value How can I calculate the median value of a list in tensorflow? Like
```
node = tf.median(X)
```
X is the placeholder In numpy, I can directly use np.median to get the median value. How can I use the numpy operation in tensorflow?

Response:
For calculating median of an array with tensorflow you can use the percentile function, since the 50th percentile is the median.
```
import tensorflow as tf
import tensorflow_probability as tfp
import numpy as np

np.random.seed(0)
x = np.random.normal(3.0, .1, 100)

median = tfp.stats.percentile(x, 50.0, interpolation='midpoint')

tf.Session().run(median)
```
The code above is equivalent to np.percentile(x, 50, interpolation='midpoint').


Question: Convert Keras model to C++ [closed] Closed. This question is seeking recommendations for software libraries, tutorials, tools, books, or other off-site resources. It does not meet Stack Overflow guidelines. It is not currently accepting answers. We don’t allow questions seeking recommendations for software libraries, tutorials, tools, books, or other off-site resources. You can edit the question so it can be answered with facts and citations. Closed 6 years ago. Improve this question I am using Keras (with Theano) to train my CNN model. Does anyone has idea how can I use it in my C++ application? Does anyone tried something similar? I have idea to write some python code that will generate a c++ code with network functions - any suggestion on it? I found a similar question here how to use Tensorflow Keras model in C++ but without answer.

Response: To answer my own question and have a solution - I wrote a plain c++ solution called keras2cpp (its code available on github). In this solution you store network architecture (in json) and weights (in hdf5). Then you can dump a network to a plain text file with provided script. You can use obtained text file with network in pure c++ code. There are no dependencies on python libraries or hdf5. It should work for theano and tensorflow backend.


Question: What is freezing/unfreezing a layer in neural networks? I have been playing around with neural networks for quite a while now, and recently came across the terms freezing & unfreezing the layers before training a neural network while reading about transfer learning & am struggling with understanding their usage. When is one supposed to use freezing/unfreezing? Which layers are to freeze/unfreeze? For instance, when I import a pre-trained model & train it on my data, is my entire neural-net except the output layer freezed? How do I determine if I need to unfreeze? If so how do I determine which layers to unfreeze & train to improve model performance?

Response: I would just add to the other answer that this is most commonly used with CNNs and the amount of layers that you want to freeze (not train) is "given" by the amount of similarity between the task that you are solving and the original one (the one that the original network is solving). If the tasks are very similar, let's say that you are using CNN pretrained on imagenet and you just want to add some other "general" objects that the network should recognize then you might get away with training just the dense top of the network. The more dissimilar the tasks are, the more layers of the original network you will need to unfreeze during the training.


Question: How to download graphs from tensorboard? I have trained a model and checking its log in tensorboard. It shows a few graphs and gives me the option to download CSV or JSON, but I want the exact graph that they show on their portal. Here is how it looks: Is there any way to download it in PNG or SVG format?

Response: Click the checkbox "Show data download links" at the top left. This will add a download icon below the chart to download the chart in SVG format


Question:
Printing extra training metrics with Tensorflow Estimator Is there a way to let Tensorflow print extra training metrics (e.g. batch accuracy) when using the Estimator API? One can add summaries and view the result in Tensorboard (see another post), but I was wondering if there is an elegant way to get the scalar summary values printed while training. This already happens for training loss, e.g.:
```
loss = 0.672677, step = 2901 (52.995 sec)
```
but it would be nice to have e.g.
```
loss = 0.672677, accuracy = 0.54678, step = 2901 (52.995 sec)
```
without to much trouble. I am aware that most of the time it is more useful to plot test set accuracy (I am already doing this with a validation monitor), but in this case I am also interested in training batch accuracy.

Response:
From what I've read it is not possible to change it by passing parameter. You can try to do by creating a logging hook and passing it into to estimator run. In the body of model_fn function for your estimator:
```
logging_hook = tf.train.LoggingTensorHook({"loss" : loss,
    "accuracy" : accuracy}, every_n_iter=10)

# Rest of the function

return tf.estimator.EstimatorSpec(
    ...params...
    training_hooks = [logging_hook])
```
EDIT: To see the output you must also set logging verbosity high enough (unless its your default): tf.logging.set_verbosity(tf.logging.INFO)


Question:
ValueError: Input 0 of layer sequential is incompatible with the layer: : expected min_ndim=4, found ndim=3. Full shape received: [8, 28, 28] I keep on getting this error related to input shape. Any help would be highly appreciated. Thanks!
```
import tensorflow as tf

(xtrain, ytrain), (xtest, ytest) = tf.keras.datasets.mnist.load_data()

model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(16, kernel_size=3, activation='relu'),
    tf.keras.layers.MaxPooling2D(pool_size=2),
    tf.keras.layers.Conv2D(32, kernel_size=3, activation='relu'),
    tf.keras.layers.MaxPooling2D(pool_size=2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
    ])

model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics='accuracy')

history = model.fit(xtrain, ytrain,
                    validation_data=(xtest, ytest),
                    epochs=10, batch_size=8)
```
ValueError: Input 0 of layer sequential is incompatible with the layer: : expected min_ndim=4, found ndim=3. Full shape received: [8, 28, 28]

Response:
The input layers of the model you created needs a 4 dimension tensor to work with but the x_train tensor you are passing to it has only 3 dimensions This means that you have to reshape your training set with .reshape(n_images, 286, 384, 1). Now you have added an extra dimension without changing the data and your model is ready to run. you need to reshape your x_train tensor to a 4 dimension before training your model. for example:
```
x_train = x_train.reshape(-1, 28, 28, 1)
```
for more info on keras inputs Check this answer


Question: Keras : How should I prepare input data for RNN? I'm having trouble with preparing input data for RNN on Keras. Currently, my training data dimension is: (6752, 600, 13) 6752: number of training data 600: number of time steps 13: size of feature vectors (the vector is in float) X_train and Y_train are both in this dimension. I want to prepare this data to be fed into SimpleRNN on Keras. Suppose that we're going through time steps, from step #0 to step #599. Let's say I want to use input_length = 5, which means that I want to use recent 5 inputs. (e.g. step #10, #11,#12,#13,#14 @ step #14). How should I reshape X_train? should it be (6752, 5, 600, 13) or should it be (6752, 600, 5, 13)? And what shape should Y_train be in? Should it be (6752, 600, 13) or (6752, 1, 600, 13) or (6752, 600, 1, 13)?

Response:
If you only want to predict the output using the most recent 5 inputs, there is no need to ever provide the full 600 time steps of any training sample. My suggestion would be to pass the training data in the following manner:
```
t=0  t=1  t=2  t=3  t=4  t=5  ...  t=598  t=599
sample0      |---------------------|
sample0           |---------------------|
sample0                |-----------------
...
sample0                                         ----|
sample0                                         ----------|
sample1      |---------------------|
sample1           |---------------------|
sample1                |-----------------
....
....
sample6751                                      ----|
sample6751                                      ----------|
```
The total number of training sequences will sum up to
```
(600 - 4) * 6752 = 4024192    # (nb_timesteps - discarded_tailing_timesteps) * nb_samples
```
Each training sequence consists of 5 time steps. At each time step of every sequence you pass all 13 elements of the feature vector. Subsequently, the shape of the training data will be (4024192, 5, 13). This loop can reshape your data:
```py
input = np.random.rand(6752,600,13)
nb_timesteps = 5

flag = 0

for sample in range(input.shape[0]):
    tmp = np.array([input[sample,i:i+nb_timesteps,:] for i in range(input.shape[1] - nb_timesteps + 1)])

    if flag==0:
        new_input = tmp
        flag = 1

    else:
        new_input = np.concatenate((new_input,tmp))
```


Question: Is using batch size as 'powers of 2' faster on tensorflow? I read from somewhere that if you choose a batch size that is a power 2, training will be faster. What is this rule? Is this applicable to other applications? Can you provide a reference paper?

Response: The notion comes from aligning computations (C) onto the physical processors (PP) of the GPU. Since the number of PP is often a power of 2, using a number of C different from a power of 2 leads to poor performance. You can see the mapping of the C onto the PP as a pile of slices of size the number of PP. Say you've got 16 PP. You can map 16 C on them : 1 C is mapped onto 1 PP. You can map 32 C on them : 2 slices of 16 C , 1 PP will be responsible for 2 C. This is due to the SIMD paradigm used by GPUs. This is often called Data Parallelism : all the PP do the same thing at the same time but on different data.


Question:
Tensorflow: Using tf.slice to split the input I'm trying to split my input layer into different sized parts. I'm trying to use tf.slice to do that but it's not working. Some sample code:
```
import tensorflow as tf
import numpy as np

ph = tf.placeholder(shape=[None,3], dtype=tf.int32)

x = tf.slice(ph, [0, 0], [3, 2])

input_ = np.array([[1,2,3],
                   [3,4,5],
                   [5,6,7]])

with tf.Session() as sess:
        sess.run(tf.initialize_all_variables())
        print sess.run(x, feed_dict={ph: input_})
```
Output:
```
[[1 2]
 [3 4]
 [5 6]]
```
This works and is roughly what I want to happen, but I have to specify the first dimension (3 in this case). I can't know though how many vectors I'll be inputting, that's why I'm using a placeholder with None in the first place! Is it possible to use slice in such a way that it will work when a dimension is unknown until runtime? I've tried using a placeholder that takes its value from ph.get_shape()[0] like so: x = tf.slice(ph, [0, 0], [num_input, 2]). but that didn't work either.

Response:
You can specify one negative dimension in the size parameter of tf.slice. The negative dimension tells Tensorflow to dynamically determine the right value basing its decision on the other dimensions.
```
import tensorflow as tf
import numpy as np

ph = tf.placeholder(shape=[None,3], dtype=tf.int32)

# look the -1 in the first position
x = tf.slice(ph, [0, 0], [-1, 2])

input_ = np.array([[1,2,3],
                   [3,4,5],
                   [5,6,7]])

with tf.Session() as sess:
        sess.run(tf.initialize_all_variables())
        print(sess.run(x, feed_dict={ph: input_}))
```


Question: tensorflow store training data on GPU memory I am pretty new to tensorflow. I used to use theano for deep learning development. I notice a difference between these two, that is where input data can be stored. In Theano, it supports shared variable to store input data on GPU memory to reduce the data transfer between CPU and GPU. In tensorflow, we need to feed data into placeholder, and the data can come from CPU memory or files. My question is: is it possible to store input data on GPU memory for tensorflow? or does it already do it in some magic way? Thanks.

Response:
If your data fits on the GPU, you can load it into a constant on GPU from e.g. a numpy array:
```
with tf.device('/gpu:0'):
  tensorflow_dataset = tf.constant(numpy_dataset)
```
One way to extract minibatches would be to slice that array at each step instead of feeding it using tf.slice:
```
batch = tf.slice(tensorflow_dataset, [index, 0], [batch_size, -1])
```
There are many possible variations around that theme, including using queues to prefetch the data to GPU dynamically.


Question: Getting the current learning rate from a tf.train.AdamOptimizer I'd like print out the learning rate for each training step of my nn. I know that Adam has an adaptive learning rate, but is there a way i can see this (for visualization in tensorboard)

Response: All the optimizers have a private variable that holds the value of a learning rate. In adagrad and gradient descent it is called self._learning_rate. In adam it is self._lr. So you will just need to print sess.run(optimizer._lr) to get this value. Sess.run is needed because they are tensors.


Question:
How can I print the intermediate variables in the loss function in TensorFlow and Keras? I'm writing a custom objective to train a Keras (with TensorFlow backend) model but I need to debug some intermediate computation. For simplicity, let's say I have:
```
def custom_loss(y_pred, y_true):
    diff = y_pred - y_true
    return K.square(diff)
```
I could not find an easy way to access, for example, the intermediate variable diff or its shape during training. In this simple example, I know that I could return diff to print its values, but my actual loss is more complex and I can't return intermediate values without getting compiling errors. Is there an easy way to debug intermediate variables in Keras?

Response:
This is not something that is solved in Keras as far as I know, so you have to resort to backend-specific functionality. Both Theano and TensorFlow have Print nodes that are identity nodes (i.e., they return the input node) and have the side-effect of printing the input (or some tensor of the input). Example for Theano:
```
diff = y_pred - y_true
diff = theano.printing.Print('shape of diff', attrs=['shape'])(diff)
return K.square(diff)
```
Example for TensorFlow:
```
diff = y_pred - y_true
diff = tf.Print(diff, [tf.shape(diff)])
return K.square(diff)
```
Note that this only works for intermediate values. Keras expects tensors that are passed to other layers to have specific attributes such as _keras_shape. Values processed by the backend, i.e. through Print, usually do not have that attribute. To solve this, you can wrap debug statements in a Lambda layer for example.


Question: How does Keras define "accuracy" and "loss"? I can't find how Keras defines "accuracy" and "loss". I know I can specify different metrics (e.g. mse, cross entropy) but Keras prints out a standard "accuracy". How is that defined? Likewise for loss; I know I can specify different types of regularization; are those in the loss? Ideally, I'd like to print out the equation used to define it; if not, I'll settle for an answer here.

Response: Have a look at metrics.py, there you can find definition of all available metrics including different types of accuracy. Accuracy is not printed unless you add it to the list of desired metrics when you compile your model. Regularizers are by definition added to the loss. For example, see add_loss method of the Layerclass. Update The type of accuracy is determined based on the objective function, see training.py. The default choice is categorical_accuracy. Other types like binary_accuracy and sparse_categorical_accuracy are selected when the objective function is either binary or sparse.


Question: How to graph tf.keras model in Tensorflow-2.0? I upgraded to Tensorflow 2.0 and there is no tf.summary.FileWriter("tf_graphs", sess.graph). I was looking through some other StackOverflow questions on this and they said to use tf.compat.v1.summary etc. Surely there must be a way to graph and visualize a tf.keras model in Tensorflow version 2. What is it? I'm looking for a tensorboard output like the one below. Thank you!

Response:
You can visualize the graph of any tf.function decorated function, but first, you have to trace its execution. Visualizing the graph of a Keras model means to visualize it's call method. By default, this method is not tf.function decorated and therefore you have to wrap the model call in a function correctly decorated and execute it.
```py
import tensorflow as tf

model = tf.keras.Sequential(
    [
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(32, activation="relu"),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10, activation="softmax"),
    ]
)

@tf.function
def traceme(x):
    return model(x)

logdir = "log"
writer = tf.summary.create_file_writer(logdir)
tf.summary.trace_on(graph=True, profiler=True)
# Forward pass
traceme(tf.zeros((1, 28, 28, 1)))
with writer.as_default():
    tf.summary.trace_export(name="model_trace", step=0, profiler_outdir=logdir)
```


Question:
TensorFlow: argmax (-min) I just noticed an unexpected (at least for me) behavior in TensorFlow. I thought tf.argmax (-argmin) operates on the ranks of a Tensor from outer to inner, but apparently it does not?! Example:
```
import numpy as np
import tensorflow as tf

sess = tf.InteractiveSession()

arr = np.array([[31, 23,  4, 24, 27, 34],
                [18,  3, 25,  0,  6, 35],
                [28, 14, 33, 22, 20,  8],
                [13, 30, 21, 19,  7,  9],
                [16,  1, 26, 32,  2, 29],
                [17, 12,  5, 11, 10, 15]])

# arr has rank 2 and shape (6, 6)
tf.rank(arr).eval()
> 2
tf.shape(arr).eval()
> array([6, 6], dtype=int32)
```
tf.argmax takes two arguments: input and dimension. Since the indices of array arr are arr[rows, columns], I would expect tf.argmax(arr, 0) to return the index of the maximum element per row, while I would have expected tf.argmax(arr, 1) to return the maximum element per column. Likewise for tf.argmin. However, the opposite is true:
```
tf.argmax(arr, 0).eval()
> array([0, 3, 2, 4, 0, 1])

# 0 -> 31 (arr[0, 0])
# 3 -> 30 (arr[3, 1])
# 2 -> 33 (arr[2, 2])
# ...
# thus, this is clearly searching for the maximum element
# for every column, and *not* for every row

tf.argmax(arr, 1).eval()
> array([5, 5, 2, 1, 3, 0])

# 5 -> 34 (arr[0, 5])
# 5 -> 35 (arr[1, 5])
# 2 -> 33 (arr[2, 2])
# ...
# this clearly returns the maximum element per row,
# albeit 'dimension' was set to 1
```
Can someone explain this behavior? Generalized every n-dimensional Tensor t is indexed by t[i, j, k, ...]. Thus, t has rank n and shape (i, j, k, ...). Since dimension 0 corresponds to i, dimension 1 to j, and so forth. Why does tf.argmax (& -argmin) ignore this scheme?

Response: Think of the dimension argument of tf.argmax as the axis across which you reduce. tf.argmax(arr, 0) reduces across dimension 0, i.e. the rows. Reducing across rows means that you will get the argmax of each individual column. This might be counterintuitive, but it falls in line with the conventions used in tf.reduce_max and so on.


Question:
How to stack multiple lstm in keras? I am using deep learning library keras and trying to stack multiple LSTM with no luck. Below is my code
```
model = Sequential()
model.add(LSTM(100,input_shape =(time_steps,vector_size)))
model.add(LSTM(100))
```
The above code returns error in the third line Exception: Input 0 is incompatible with layer lstm_28: expected ndim=3, found ndim=2 The input X is a tensor of shape (100,250,50). I am running keras on tensorflow backend

Response:
You need to add return_sequences=True to the first layer so that its output tensor has ndim=3 (i.e. batch size, timesteps, hidden state). Please see the following example:
```
# expected input data shape: (batch_size, timesteps, data_dim)
model = Sequential()
model.add(LSTM(32, return_sequences=True,
               input_shape=(timesteps, data_dim)))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32, return_sequences=True))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32))  # return a single vector of dimension 32
model.add(Dense(10, activation='softmax'))
```
From: https://keras.io/getting-started/sequential-model-guide/ (search for "stacked lstm")


Question: How to get the count of an element in a tensor in TensorFlow? I want to get the count of an element in a tensor, for example, t = [1, 2, 0, 0, 0, 0] (t is a tensor). I can get the amount 4 of zeros by calling t.count(0) in Python, but in TensorFlow, I can't find any functions to do this. How can I get the count of zeros?

Response:
There isn't a built in count method in TensorFlow right now. But you could do it using the existing tools in a method like so:
```
def tf_count(t, val):
    elements_equal_to_value = tf.equal(t, val)
    as_ints = tf.cast(elements_equal_to_value, tf.int32)
    count = tf.reduce_sum(as_ints)
    return count
```


Question:
Save Tensorflow graph for viewing in Tensorboard without summary operations I have a rather complicated Tensorflow graph that I'd like to visualize for optimization purposes. Is there a function that I can call that will simply save the graph for viewing in Tensorboard without needing to annotate variables? I Tried this:
```
merged = tf.merge_all_summaries()
writer = tf.train.SummaryWriter("/Users/Name/Desktop/tf_logs", session.graph_def)
```
But no output was produced. This is using the 0.6 wheel. This appears to be related: Graph visualisaton is not showing in tensorboard for seq2seq model

Response:
You can also dump the graph as a GraphDef protobuf and load that directly in TensorBoard. You can do this without starting a session or running the model.
```
## ... create graph ...

```python
graph_def = tf.get_default_graph().as_graph_def()
```

```python
graphpb_txt = str(graph_def)
```

```python
with open('graphpb.txt', 'w') as f: f.write(graphpb_txt)
#Output
#```
#This will output a file that looks something like this, depending on the specifics of your model.
#```
#node {
#  name: "W"
#  op: "Const"
#  attr {
#    key: "dtype"
#    value {
#      type: DT_FLOAT
#    }
#  }
#...
#version 1
#```
#In TensorBoard you can then use the "Upload" button to load it from disk.
```


Question: Get the value of some weights in a model trained by TensorFlow I have trained a ConvNet model with TensorFlow, and I want to get a particular weight in layer. For example in torch7 I would simply access model.modules[2].weights. to get the weights of layer 2. How would I do the same thing in TensorFlow?

Response:
In TensorFlow, trained weights are represented by tf.Variable objects. If you created a tf.Variable—e.g. called v—yourself, you can get its value as a NumPy array by calling sess.run(v) (where sess is a tf.Session). If you do not currently have a pointer to the tf.Variable, you can get a list of the trainable variables in the current graph by calling tf.trainable_variables(). This function returns a list of all trainable tf.Variable objects in the current graph, and you can select the one that you want by matching the v.name property. For example:
```
# Desired variable is called "tower_2/filter:0".
var = [v for v in tf.trainable_variables() if v.name == "tower_2/filter:0"][0]
```


Question:
Tensorflow minimise with respect to only some elements of a variable Is it possible to minimise a loss function by changing only some elements of a variable? In other words, if I have a variable X of length 2, how can I minimise my loss function by changing X[0] and keeping X[1] constant? Hopefully this code I have attempted will describe my problem:
```
import tensorflow as tf
import tensorflow.contrib.opt as opt

X = tf.Variable([1.0, 2.0])
X0 = tf.Variable([3.0])

Y = tf.constant([2.0, -3.0])

scatter = tf.scatter_update(X, [0], X0)

with tf.control_dependencies([scatter]):
    loss = tf.reduce_sum(tf.squared_difference(X, Y))

opt = opt.ScipyOptimizerInterface(loss, [X0])

init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)
    opt.minimize(sess)

    print("X: {}".format(X.eval()))
    print("X0: {}".format(X0.eval()))
```
which outputs:
```
INFO:tensorflow:Optimization terminated with:
  Message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'
  Objective function value: 26.000000
  Number of iterations: 0
  Number of functions evaluations: 1
X: [3. 2.]
X0: [3.]
```
where I would like to to find the optimal value of X0 = 2 and thus X = [2, 2] edit Motivation for doing this: I would like to import a trained graph/model and then tweak various elements of some of the variables depending on some new data I have.

Response:
You can use this trick to restrict the gradient calculation to one index:
```
import tensorflow as tf
import tensorflow.contrib.opt as opt

X = tf.Variable([1.0, 2.0])

part_X = tf.scatter_nd([[0]], [X[0]], [2])

X_2 = part_X + tf.stop_gradient(-part_X + X)

Y = tf.constant([2.0, -3.0])

loss = tf.reduce_sum(tf.squared_difference(X_2, Y))

opt = opt.ScipyOptimizerInterface(loss, [X])

init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)
    opt.minimize(sess)

    print("X: {}".format(X.eval()))
```
part_X becomes the value you want to change in a one-hot vector of the same shape as X. part_X + tf.stop_gradient(-part_X + X) is the same as X in the forward pass, since part_X - part_X is 0. However in the backward pass the tf.stop_gradient prevents all unnecessary gradient calculations.


Question: What is the purpose of graph collections in TensorFlow? The API discusses Graph Collections which judging from the code are a general purpose key/data storage. What is the purpose of those collections?

Response: Remember that under the hood, Tensorflow is a system for specifying and then executing computational data flow graphs. The graph collections are used as part of keeping track of the constructed graphs and how they must be executed. For example, when you create certain kinds of ops, such as tf.train.batch_join, the code that adds the op will also add some queue runners to the QUEUE_RUNNERS graph collection. Later, when you call start_queue_runners(), by default, it will look at the QUEUE_RUNNERS collection to know which runners to start.


Question: YOLO object detection: how does the algorithm predict bounding boxes larger than a grid cell? I am trying to better understand how the YOLO2 & 3 algorithms works. The algorithm processes a series of convolutions until it gets down to a 13x13 grid. Then it is able to classify objects within each grid cell as well as the bounding boxes for those objects. If you look at this picture, you see that the bounding box in red is larger than any individual grid cell. Also the bounding box is centered at the center of the object. My questions of to do with how do the predicted bounding boxes exceed the size of the grid cell, when the network activations are based upon the individual grid cell. I mean everything outside of the grid cell should be unknown to the neurons predicting the bounding boxes for an object detected in that cell right. More precisely here are my questions: 1. How does the algorithm predict bounding boxes that are larger than the grid cell? 2. How does the algorithm know in which cell the center of the object is located?

Response: everything outside of the grid cell should be unknown to the neurons predicting the bounding boxes for an object detected in that cell right. It's not quite right. The cells correspond to a partition of the image where the neuron have learned to respond if the center of an object is located within. However, the receptive field of those output neurons is much larger than the cell and actually cover the entire image. It is therefore able to recognize and draw a bounding box around an object much larger than its assigned "center cell". So a cell is centered on the center of the receptive field of the output neuron but is a much smaller part. It is also somewhat arbitrary, and one could image for example to have overlapping cells -- in which case you would expect neighboring neurons to fire simultaneously when an object is centered in the overlapping zone of their cells.


Question: How do I swap tensor's axes in TensorFlow? I have a tensor of shape (30, 116, 10), and I want to swap the first two dimensions, so that I have a tensor of shape (116, 30, 10) I saw that numpy as such a function implemented (np.swapaxes) and I searched for something similar in tensorflow but I found nothing. Do you have any idea?

Response: tf.transpose provides the same functionality as np.swapaxes, although in a more generalized form. In your case, you can do tf.transpose(orig_tensor, [1, 0, 2]) which would be equivalent to np.swapaxes(orig_np_array, 0, 1).


Question:
How does TensorFlow SparseCategoricalCrossentropy work? I'm trying to understand this loss function in TensorFlow but I don't get it. It's SparseCategoricalCrossentropy. All other loss functions need outputs and labels of the same shape, this specific loss function doesn't. Source code:
```
import tensorflow as tf;

scce = tf.keras.losses.SparseCategoricalCrossentropy();
Loss = scce(
  tf.constant([ 1,    1,    1,    2   ], tf.float32),
  tf.constant([[1,2],[3,4],[5,6],[7,8]], tf.float32)
);
print("Loss:", Loss.numpy());
```
The error is:
```
InvalidArgumentError: Received a label value of 2 which is outside the valid range of [0, 2).
Label values: 1 1 1 2 [Op:SparseSoftmaxCrossEntropyWithLogits]
```
How to provide proper params to the loss function SparseCategoricalCrossentropy?

Response:
SparseCategoricalCrossentropy and CategoricalCrossentropy both compute categorical cross-entropy. The only difference is in how the targets/labels should be encoded. When using SparseCategoricalCrossentropy the targets are represented by the index of the category (starting from 0). Your outputs have shape 4x2, which means you have two categories. Therefore, the targets should be a 4 dimensional vector with entries that are either 0 or 1. For example:
```
scce = tf.keras.losses.SparseCategoricalCrossentropy();
Loss = scce(
  tf.constant([ 0,    0,    0,    1   ], tf.float32),
  tf.constant([[1,2],[3,4],[5,6],[7,8]], tf.float32))
```
This in contrast to CategoricalCrossentropy where the labels should be one-hot encoded:
```
cce = tf.keras.losses.CategoricalCrossentropy();
Loss = cce(
  tf.constant([ [1,0]    [1,0],    [1, 0],   [0, 1]   ], tf.float32),
  tf.constant([[1,2],[3,4],[5,6],[7,8]], tf.float32))
```
SparseCategoricalCrossentropy is more efficient when you have a lot of categories.


Question: How to add regularizations in TensorFlow? I found in many available neural network code implemented using TensorFlow that regularization terms are often implemented by manually adding an additional term to loss value. My questions are: Is there a more elegant or recommended way of regularization than doing it manually? I also find that get_variable has an argument regularizer. How should it be used? According to my observation, if we pass a regularizer to it (such as tf.contrib.layers.l2_regularizer, a tensor representing regularized term will be computed and added to a graph collection named tf.GraphKeys.REGULARIZATOIN_LOSSES. Will that collection be automatically used by TensorFlow (e.g. used by optimizers when training)? Or is it expected that I should use that collection by myself?

Response:
As you say in the second point, using the regularizer argument is the recommended way. You can use it in get_variable, or set it once in your variable_scope and have all your variables regularized. The losses are collected in the graph, and you need to manually add them to your cost function like this.
```
reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
  reg_constant = 0.01  # Choose an appropriate one.
  loss = my_normal_loss + reg_constant * sum(reg_losses)
```


Question:
Tensorflow negative sampling I am trying to follow the udacity tutorial on tensorflow where I came across the following two lines for word embedding models:
```
# Look up embeddings for inputs.
  embed = tf.nn.embedding_lookup(embeddings, train_dataset)
  # Compute the softmax loss, using a sample of the negative labels each time.
  loss = tf.reduce_mean(tf.nn.sampled_softmax_loss(softmax_weights, softmax_biases,
                        embed, train_labels, num_sampled, vocabulary_size))
```
Now I understand that the second statement is for sampling negative labels. But the question is how does it know what the negative labels are? All I am providing the second function is the current input and its corresponding labels along with number of labels that I want to (negatively) sample from. Isn't there the risk of sampling from the input set in itself? This is the full example: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/udacity/5_word2vec.ipynb

Response: You can find the documentation for tf.nn.sampled_softmax_loss() here. There is even a good explanation of Candidate Sampling provided by TensorFlow here (pdf). How does it know what the negative labels are? TensorFlow will randomly select negative classes among all the possible classes (for you, all the possible words). Isn't there the risk of sampling from the input set in itself? When you want to compute the softmax probability for your true label, you compute: logits[true_label] / sum(logits[negative_sampled_labels]. As the number of classes is huge (the vocabulary size), there is very little probability to sample the true_label as a negative label. Anyway, I think TensorFlow removes this possibility altogether when randomly sampling. (EDIT: @Alex confirms TensorFlow does this by default)


Question: List of Differentiable Ops in Tensorflow Is there a master list of Tensorflow ops that are differentiable (i.e., will auto-differentiate)? Two other ways to phrase this: List of ops that do not have ops.NoGradient set. List of ops that will not trigger LookupError. For example, I'd assume that all the Control Flow ops are not differentiable (e.g., tf.where). How would I find this other than by manually running them all through tf.gradients to see if they throw the LookupError. "Commonsense" is not a valid answer. Thanks. EDIT: tf.where is differentiable so my intuitions are wrong. Perhaps the correct question here is which ops in Tensorflow are not differentiable. Thanks.

Response: I have devised the entire list of Differentiable and Non-Differentiable Ops using python code. You will find the compact list here. Also the code which generated it. https://github.com/Mainak431/List-of-Differentiable--OPs-and-Non-differentiable-OPs--in-Tensorflow


Question: What does train_on_batch() do in keras model? I saw a sample of code (too big to paste here) where the author used model.train_on_batch(in, out) instead of model.fit(in, out). The official documentation of Keras says: Single gradient update over one batch of samples. But I don't get it. Is it the same as fit(), but instead of doing many feed-forward and backprop steps, it does it once? Or am I wrong?

Response: Yes, train_on_batch trains using a single batch only and once. While fit trains many batches for many epochs. (Each batch causes an update in weights). The idea of using train_on_batch is probably to do more things yourself between each batch.


Question: Difference between tf.nn_conv2d and tf.nn.depthwise_conv2d What is the difference between tf.nn_conv2d and tf.nn.depthwise_conv2d in Tensorflow?

Response: I am no expert on this, but as far as I understand the difference is this: Lets say you have an input colour image with length 100, width 100. So the dimensions are 100x100x3. For both examples we use the same filter of width and height 5. Lets say we want the next layer to have a depth of 8. In tf.nn.conv2d you define the kernel shape as [width, height, in_channels, out_channels]. In our case this means the kernel has shape [5,5,3,out_channels]. The weight-kernel that is strided over the image has a shape of 5x5x3, and it is strided over the whole image 8 times to produce 8 different feature maps. In tf.nn.depthwise_conv2d you define the kernel shape as [width, height, in_channels, channel_multiplier]. Now the output is produced differently. Separate filters of 5x5x1 are strided over each dimension of the input image, one filter per dimension, each producing one feature map per dimension. So here, a kernel size [5,5,3,1] would produce an output with depth 3. The channel_multiplier tells you how many different filters you want to apply per dimension. So the original desired output of depth 8 is not possible with 3 input dimensions. Only multiples of 3 are possible.


Question: Tensorflow equivalent to numpy.diff Is there a tensorflow equivalent to numpy.diff? Calculate the n-th discrete difference along given axis. For my project I only need n=1

Response:
Try this:
```
def tf_diff_axis_0(a):
    return a[1:]-a[:-1]

def tf_diff_axis_1(a):
    return a[:,1:]-a[:,:-1]
```
To check:
```
import numpy as np
import tensorflow as tf

x0=np.arange(5)+np.zeros((5,5))
sess = tf.Session()
np.diff(x0, axis=0) == sess.run(tf_diff_axis_0(tf.constant(x0)))
np.diff(x0, axis=1) == sess.run(tf_diff_axis_1(tf.constant(x0)))
```


Question: batch size in model.fit and model.predict In keras, both model.fit and model.predict has a parameter of batch_size. My understanding is that batch size in model.fit is related to batch optimization, what's the physical meaning of batch_size in model_predict? Does it need to be equal to the one used by model.fit?

Response: No it doesn‘t. Imagine inside your model there is a function which increases the amount of memory significantly. Therefore, you might run into resource errors if you try to predict all your data in one go. This is often the case when you use gpu with limited gpu memory for predicting. So instead you choose to predict only small batches at the same time. The batch_size parameter in the predict function will not alter your results in any way. So you can choose any batch_size you want for prediction.


Question:
Tensorflow - Using tf.summary with 1.2 Estimator API I'm trying to add some TensorBoard logging to a model which uses the new tf.estimator API. I have a hook set up like so:
```
summary_hook = tf.train.SummarySaverHook(
    save_secs=2,
    output_dir=MODEL_DIR,
    summary_op=tf.summary.merge_all())

# ...

classifier.train(
    input_fn,
    steps=1000,
    hooks=[summary_hook])
```
In my model_fn, I am also creating a summary -
```
def model_fn(features, labels, mode):
    # ... model stuff, calculate the value of loss
    tf.summary.scalar("loss", loss)
    # ...
```
However, when I run this code, I get the following error from the summary_hook: Exactly one of scaffold or summary_op must be provided. This is probably because tf.summary.merge_all() is not finding any summaries and is returning None, despite the tf.summary.scalar I declared in the model_fn. Any ideas why this wouldn't be working?

Response:
Use tf.train.Scaffold() and pass tf.merge_all as following
```
summary_hook = tf.train.SummarySaverHook(
    save_secs=2,
    output_dir=MODEL_DIR,
    scaffold=tf.train.Scaffold(summary_op=tf.summary.merge_all()))
```


Question: How to get the value of a tensor? Python While doing some calculations I end up calculating an average_acc. When I try to print it, it outputs: tf.Tensor(0.982349, shape=(), dtype=float32). How do I get the 0.98.. value of it and use it as a normal float? What I'm trying to do is get a bunch of those in an array and plot some graphs, but for that, I need simple floats as far as I can tell.

Response:
The easiest and best way to do it is using tf.keras.backend.get_value API.
```
print(average_acc)
>>tf.Tensor(0.982349, shape=(), dtype=float32)
print(tf.keras.backend.get_value(average_acc))
>>0.982349
```


Question:
How do I pass a scalar via a TensorFlow feed dictionary My TensorFlow model uses tf.random_uniform to initialize a variable. I would like to specify the range when I begin training, so I created a placeholder for the initialization value.
```
init = tf.placeholder(tf.float32, name="init")
v = tf.Variable(tf.random_uniform((100, 300), -init, init), dtype=tf.float32)
initialize = tf.initialize_all_variables()
```
I initialize variables at the start of training like so.
```
session.run(initialize, feed_dict={init: 0.5})
```
This gives me the following error:
```
ValueError: initial_value must have a shape specified: Tensor("Embedding/random_uniform:0", dtype=float32)
```
I cannot figure out the correct shape parameter to pass to tf.placeholder. I would think for a scalar I should do init = tf.placeholder(tf.float32, shape=0, name="init") but this gives the following error:
```
ValueError: Incompatible shapes for broadcasting: (100, 300) and (0,)
```
If I replace init with the literal value 0.5 in the call to tf.random_uniform it works. How do I pass this scalar initial value via the feed dictionary?

Response:
TL;DR: Define init with a scalar shape as follows:
```
init = tf.placeholder(tf.float32, shape=(), name="init")
```
This looks like an unfortunate implementation detail of tf.random_uniform(): it currently uses tf.add() and tf.multiply() to rescale the random value from [-1, +1] to [minval, maxval], but if the shape of minval or maxval is unknown, tf.add() and tf.multiply() can't infer the proper shapes, because there might be broadcasting involved. By defining init with a known shape (where a scalar is () or [], not 0), TensorFlow can draw the proper inferences about the shape of the result of tf.random_uniform(), and your program should work as intended.


Question: How is Hard Sigmoid defined I am working on Deep Nets using keras. There is an activation "hard sigmoid". Whats its mathematical definition ? I know what is Sigmoid. Someone asked similar question on Quora: https://www.quora.com/What-is-hard-sigmoid-in-artificial-neural-networks-Why-is-it-faster-than-standard-sigmoid-Are-there-any-disadvantages-over-the-standard-sigmoid But I could not find the precise mathematical definition anywhere ?

Response:
Since Keras supports both Tensorflow and Theano, the exact implementation might be different for each backend - I'll cover Theano only. For Theano backend Keras uses T.nnet.hard_sigmoid, which is in turn linearly approximated standard sigmoid:
```py
slope = tensor.constant(0.2, dtype=out_dtype)
shift = tensor.constant(0.5, dtype=out_dtype)
x = (x * slope) + shift
x = tensor.clip(x, 0, 1)
```
i.e. it is: max(0, min(1, x*0.2 + 0.5))


Question:
In Tensorflow, how to use tf.gather() for the last dimension? I am trying to gather slices of a tensor in terms of the last dimension for partial connection between layers. Because the output tensor's shape is [batch_size, h, w, depth], I want to select slices based on the last dimension, such as
```
# L is intermediate tensor
partL = L[:, :, :, [0,2,3,8]]
```
However, tf.gather(L, [0, 2,3,8]) seems to only work for the first dimension (right?) Can anyone tell me how to do it?

Response: As of TensorFlow 1.3 tf.gather has an axis parameter, so the various workarounds here are no longer necessary. https://www.tensorflow.org/versions/r1.3/api_docs/python/tf/gather https://github.com/tensorflow/tensorflow/issues/11223


Question: How to do Xavier initialization on TensorFlow I'm porting my Caffe network over to TensorFlow but it doesn't seem to have xavier initialization. I'm using truncated_normal but this seems to be making it a lot harder to train.

Response:
Since version 0.8 there is a Xavier initializer, see here for the docs. You can use something like this:
```
W = tf.get_variable("W", shape=[784, 256],
           initializer=tf.contrib.layers.xavier_initializer())
```


Question:
Tensorflow Precision / Recall / F1 score and Confusion matrix I would like to know if there is a way to implement the different score function from the scikit learn package like this one :
```
from sklearn.metrics import confusion_matrix
confusion_matrix(y_true, y_pred)
```
into a tensorflow model to get the different score.
```
with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as sess:
init = tf.initialize_all_variables()
sess.run(init)
for epoch in xrange(1):
        avg_cost = 0.
        total_batch = len(train_arrays) / batch_size
        for batch in range(total_batch):
                train_step.run(feed_dict = {x: train_arrays, y: train_labels})
                avg_cost += sess.run(cost, feed_dict={x: train_arrays, y: train_labels})/total_batch
        if epoch % display_step == 0:
                print "Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(avg_cost)

print "Optimization Finished!"
correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
# Calculate accuracy
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
print "Accuracy:", batch, accuracy.eval({x: test_arrays, y: test_labels})
```
Will i have to run the session again to get the prediction ?

Response:
You do not really need sklearn to calculate precision/recall/f1 score. You can easily express them in TF-ish way by looking at the formulas: Now if you have your actual and predicted values as vectors of 0/1, you can calculate TP, TN, FP, FN using tf.count_nonzero:
```
TP = tf.count_nonzero(predicted * actual)
TN = tf.count_nonzero((predicted - 1) * (actual - 1))
FP = tf.count_nonzero(predicted * (actual - 1))
FN = tf.count_nonzero((predicted - 1) * actual)
```
Now your metrics are easy to calculate:
```
precision = TP / (TP + FP)
recall = TP / (TP + FN)
f1 = 2 * precision * recall / (precision + recall)
```


Question: What are the differences between tf.initialize_all_variables() and tf.global_variables_initializer() On Tensorflow official website, it gives explantions of the tf.initialize_all_variables() and tf.global_variables_initializer() functions as follow tf.initialize_all_variables(): Returns an op that initializes all variables. tf.global_variables_initializer(): Adds an op to initialize all variables in the model It seems like both can be used to initialize all variables in graphs. Can we use these two functions exchangbly? If not, what would be the differences?

Response: Unfortunately, you forgot to read an important line in the documentation of tf.initialize_all_variables. THIS FUNCTION IS DEPRECATED. It will be removed after 2017-03-02. Instructions for updating: Use tf.global_variables_initializer instead.


Question: What's the difference between Variable and ResourceVariable in Tensorflow In Tensorflow, Variable is a resource, inherited from ResourceBase and managed by ResourceMgr. But why is there another named ResourceVariable? Both of them can be used for optimizers like gradient_descent (see this example). What's the difference? I know the former is well documented and most often used. What's the purpose of the latter?

Response: ResourceVariable is the replacement for Variable, that aims to clean up some of the messier aspects of the semantics of Variable. ResourceVariable is the default in TF 2.0 and you very likely don't care about the differences between the two unless you are working on details deep inside the Tensorflow implementation. When eager execution is enabled tf.Variable also creates resource variables. So just use tf.Variable for now, it's almost certainly what you want; if you experience issues which look like race conditions or bugs from inconsistent values of variables in code you can try enabling resource variables (by either passing use_resource=True to your variable-creating code or calling tf.enable_resource_variables() in TF 1.x).


Question:
How to fix "AttributeError: module 'tensorflow' has no attribute 'get_default_graph'"? I am trying to run some code to create an LSTM model but i get an error: AttributeError: module 'tensorflow' has no attribute 'get_default_graph' My code is as follows:
```
from keras.models import Sequential

model = Sequential()
model.add(Dense(32, input_dim=784))
model.add(Activation('relu'))
model.add(LSTM(17))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
```
I have found someone else with a similar problem and they updated tensorflow and it works; but mine is up to date and still does not work. I am new to using keras and machine learning so I apologise if this is something silly!

Response: Please try: from tensorflow.keras.models import Sequential instead of from keras.models import Sequential


Question:
Keras - Difference between categorical_accuracy and sparse_categorical_accuracy What is the difference between categorical_accuracy and sparse_categorical_accuracy in Keras? There is no hint in the documentation for these metrics, and by asking Dr. Google, I did not find answers for that either. The source code can be found here:
```
def categorical_accuracy(y_true, y_pred):
    return K.cast(K.equal(K.argmax(y_true, axis=-1),
                          K.argmax(y_pred, axis=-1)),
                  K.floatx())

def sparse_categorical_accuracy(y_true, y_pred):
    return K.cast(K.equal(K.max(y_true, axis=-1),
                          K.cast(K.argmax(y_pred, axis=-1), K.floatx())),
                  K.floatx())
```

Response: So in categorical_accuracy you need to specify your target (y) as one-hot encoded vector (e.g. in case of 3 classes, when a true class is second class, y should be (0, 1, 0). In sparse_categorical_accuracy you need should only provide an integer of the true class (in the case from previous example - it would be 1 as classes indexing is 0-based).


Question: SGD with momentum in TensorFlow In Caffe, the SGD solver has a momentum parameter (link). In TensorFlow, I see that tf.train.GradientDescentOptimizer does not have an explicit momentum parameter. However, I can see that there is tf.train.MomentumOptimizer optimizer. Is it the equivalent of Caffe SGD with momentum optimizer?

Response: Yes it is. tf.train.MomentumOptimizer = SGD + momentum


Question:
Keras replacing input layer The code that I have (that I can't change) uses the Resnet with my_input_tensor as the input_tensor.
```
model1 = keras.applications.resnet50.ResNet50(input_tensor=my_input_tensor, weights='imagenet')
```
Investigating the source code, ResNet50 function creates a new keras Input Layer with my_input_tensor and then create the rest of the model. This is the behavior that I want to copy with my own model. I load my model from h5 file.
```
model2 = keras.models.load_model('my_model.h5')
```
Since this model already has an Input Layer, I want to replace it with a new Input Layer defined with my_input_tensor. How can I replace an input layer?

Response:
When you saved your model using:
```
old_model.save('my_model.h5')
```
it will save following: The architecture of the model, allowing to create the model. The weights of the model. The training configuration of the model (loss, optimizer). The state of the optimizer, allowing training to resume from where you left before. So then, when you load the model:
```
res50_model = load_model('my_model.h5')
```
you should get the same model back, you can verify the same using:
```
res50_model.summary()
res50_model.get_weights()
```
Now you can, pop the input layer and add your own using:
```
res50_model.layers.pop(0)
res50_model.summary()
```
add new input layer:
```
newInput = Input(batch_shape=(0,299,299,3))    # let us say this new InputLayer
newOutputs = res50_model(newInput)
newModel = Model(newInput, newOutputs)

newModel.summary()
res50_model.summary()
```


Question:
How to Calculate R^2 in Tensorflow I am trying to do regression in Tensorflow. I'm not positive I am calculating R^2 correctly as Tensorflow gives me a different answer than sklearn.metrics.r2_score Can someone please look at my below code and let me know if I implemented the pictured equation correctly. Thanks
```
total_error = tf.square(tf.sub(y, tf.reduce_mean(y)))
unexplained_error = tf.square(tf.sub(y, prediction))
R_squared = tf.reduce_mean(tf.sub(tf.div(unexplained_error, total_error), 1.0))
R = tf.mul(tf.sign(R_squared),tf.sqrt(tf.abs(R_squared)))
```

Response:
What you are computing the "R^2" is compared to the given expression, you are computing the mean at the wrong place. You should take the mean when computing the errors, before doing the division.
```
unexplained_error = tf.reduce_sum(tf.square(tf.sub(y, prediction)))
total_error = tf.reduce_sum(tf.square(tf.sub(y, tf.reduce_mean(y))))
R_squared = tf.sub(1, tf.div(unexplained_error, total_error))
```


Question: tf.newaxis operation in TensorFlow x_train = x_train[..., tf.newaxis].astype("float32") x_test = x_test[..., tf.newaxis].astype("float32") Can someone please explain how tf.newaxis works ? I found a brief mention in the documentation https://www.tensorflow.org/api_docs/python/tf/strided_slice but I could not properly understand.

Response:
Check this example:
```
a = tf.constant([100])
print(a.shape) ## (1)
expanded_1 = tf.expand_dims(a,axis=1)
print(expanded_1.shape) ## (1,1)
expanded_2 = a[:, tf.newaxis]
print(expanded_2.shape) ## (1,1)
```
It is similar to expand_dims() which adds a new axis. If you want to add a new axis at the beginning of the tensor, use
```
expanded_2 = a[tf.newaxis, :]
```
otherwise (at the end)
```
expanded_2 = a[:,tf.newaxis]
```


Question:
Best way to flatten a 2D tensor containing a vector in TensorFlow? What is the most efficient way to flatten a 2D tensor which is actually a horizontal or vertical vector into a 1D tensor? Is there a difference in terms of performance between:
```
tf.reshape(w, [-1])
```
and
```
tf.squeeze(w)
```
?

Response: Both tf.reshape(w, [-1]) and tf.squeeze(w) are "cheap" in that they operate only on the metadata (i.e. the shape) of the given tensor, and don't modify the data itself. Of the two tf.reshape() has slightly simpler logic internally, but the performance of the two should be indistinguishable.


Question: TensorFlow: How to handle void labeled data in image segmentation? I was wondering how to handle not labeled parts of an image in image segmentation using TensorFlow. For example, my input is an image of height * width * channels. The labels are too of the size height * width, with one label for every pixel. Some parts of the image are annotated, other parts are not. I would wish that those parts have no influence on the gradient computation whatsoever. Furthermore, I am not interested in the network predicting this “void” label. Is there a label or a function for this? At the moment I am using tf.nn.sparse_softmax_cross_entropy_with_logits.

Response: I'm not 100% familiar with TF. However, have you considered using the weights parameter of the loss? Looking at tf.loses.sparse_softmax_cross_entropy it has a parameter weights weights: Coefficients for the loss. This must be scalar or of same rank as labels You can set weightof "void" pixels to zero, thus making the loss ignore them. You can also remove the reduction from tf.nn.sparse_softmax_cross_entropy_with_logits and use tf.losses.compute_weighted_loss to perform the weighting.


Question:
Custom weight initialization tensorflow tf.layers.dense I'm trying to set up custom initializer to tf.layers.dense where I initialize kernel_initializer with a weight matrix I already have.
```
u_1 = tf.placeholder(tf.float32, [784, 784])
first_layer_u = tf.layers.dense(X_, n_params, activation=None,
                              kernel_initializer=u_1,
                              bias_initializer=tf.keras.initializers.he_normal())
```
This is throwing error saying ValueError: If initializer is a constant, do not specify shape. Is it a problem to assign placeholder to kernel_initializer or am I missing something?

Response:
There are at least two ways to achieve this: 1 Create your own layer
```
W1 = tf.Variable(YOUR_WEIGHT_MATRIX, name='Weights')
  b1 = tf.Variable(tf.zeros([YOUR_LAYER_SIZE]), name='Biases') #or pass your own
  h1 = tf.add(tf.matmul(X, W1), b1)
```
2 Use the tf.constant_initializer
```
init = tf.constant_initializer(YOUR_WEIGHT_MATRIX)
l1 = tf.layers.dense(X, o, kernel_initializer=init)
```


Question:
Keras Sequential model with multiple inputs I am making a MLP model which takes two inputs and produces a single output. I have two input arrays (one for each input) and 1 output array. The neural network has 1 hidden layer with 2 neurons. Each array has 336 elements.
```
model0 = keras.Sequential([
keras.layers.Dense(2, input_dim=2, activation=keras.activations.sigmoid, use_bias=True),
keras.layers.Dense(1, activation=keras.activations.relu, use_bias=True),
])

# Compile the neural network #
model0.compile(
    optimizer = keras.optimizers.RMSprop(lr=0.02,rho=0.9,epsilon=None,decay=0),
    loss = 'mean_squared_error',
    metrics=['accuracy']
)
```
I tried two ways, both of them are giving errors.
```
model0.fit(numpy.array([array_1, array_2]),output, batch_size=16, epochs=100)
```
ValueError: Error when checking input: expected dense_input to have shape (2,) but got array with shape (336,) The second way:
```
model0.fit([array_1, array_2],output, batch_size=16, epochs=100)
```
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 2 arrays: Similar question. But not using sequential model.

Response:
To solve this problem you have two options. 1. Using a sequential model You can concatenate both arrays into one before feeding to the network. Let's assume the two arrays have a shape of (Number_data_points, ), now the arrays can be merged using numpy.stack method.
```py
merged_array = np.stack([array_1, array_2], axis=1)
```
```py
model0 = keras.Sequential([
keras.layers.Dense(2, input_dim=2, activation=keras.activations.sigmoid, use_bias=True),
keras.layers.Dense(1, activation=keras.activations.relu, use_bias=True),
])

model0.fit(merged_array,output, batch_size=16, epochs=100)
```
2. Using Functional API. This is the most recommened way to use when there are multiple inputs to the model.
```py
input1 = keras.layers.Input(shape=(1, ))
input2 = keras.layers.Input(shape=(1,))
merged = keras.layers.Concatenate(axis=1)([input1, input2])
dense1 = keras.layers.Dense(2, input_dim=2, activation=keras.activations.sigmoid, use_bias=True)(merged)
output = keras.layers.Dense(1, activation=keras.activations.relu, use_bias=True)(dense1)
model10 = keras.models.Model(inputs=[input1, input2], output=output)
```
Now you can use the second method you have trying to fit to the model
```py
model0.fit([array_1, array_2],output, batch_size=16, epochs=100)
```


Question:
How to do slice assignment in Tensorflow I found that Tensorflow provides scatter_update() to assign values to the slice of a tensor in the 0 dimension. For example, if the tensor T is three dimensional, I can assign value v[1, :, :] to T[i, :, :].
```
a = tf.Variable(tf.zeros([10,36,36]))
value = np.ones([1,36,36])
d = tf.scatter_update(a,[0],value)

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    print a.eval()
    sess.run(d)
    print a.eval()
```
But how to assign values v[1,1,:] to T[i,j,:]?
```
a = tf.Variable(tf.zeros([10,36,36]))
value1 = np.random.randn(1,1,36)
e = tf.scatter_update(a,[0],value1) #Error

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    print a.eval()
    sess.rum(e)
    print a.eval()
```
Is there any other function that TF provide or a simple way to do this?

Response:
Currently, you can do slice assignment for variables in TensorFlow. There is no specific named function for it, but you can select a slice and call assign on it:
```
my_var = my_var[4:8].assign(tf.zeros(4))
```
First, note that (after having looked at the documentation) it seems that the return value of assign, even when applied to a slice, is always a reference to the whole variable after applying the update. EDIT: The information below is either deprecated, imprecise or was always wrong. The fact is that the returned value of assign is a tensor that can be readily used and already incorporates the dependency to the assignment, so simply evaluating that or using it in further operations will ensure it gets executed without need for an explicit tf.control_dependencies block. Note, also, that this will only add the assignment op to the graph, but will not run it unless it is explicitly executed or set as a dependency of some other operation. A good practice is to use it in a tf.control_dependencies context:
```
with tf.control_dependencies([my_var[4:8].assign(tf.zeros(4))]):
    my_var = tf.identity(my_var)
```
You can read more about it in TensorFlow issue #4638.


Question:
What are logits? What is the difference between softmax and softmax_cross_entropy_with_logits? In the tensorflow API docs they use a keyword called logits. What is it? A lot of methods are written like:
```
tf.nn.softmax(logits, name=None)
```
If logits is just a generic Tensor input, why is it named logits? Secondly, what is the difference between the following two methods?
```
tf.nn.softmax(logits, name=None)
tf.nn.softmax_cross_entropy_with_logits(logits, labels, name=None)
```
I know what tf.nn.softmax does, but not the other. An example would be really helpful.

Response:
The softmax+logits simply means that the function operates on the unscaled output of earlier layers and that the relative scale to understand the units is linear. It means, in particular, the sum of the inputs may not equal 1, that the values are not probabilities (you might have an input of 5). Internally, it first applies softmax to the unscaled output, and then computes the cross entropy of those values vs. what they "should" be as defined by the labels. tf.nn.softmax produces the result of applying the softmax function to an input tensor. The softmax "squishes" the inputs so that sum(input) = 1, and it does the mapping by interpreting the inputs as log-probabilities (logits) and then converting them back into raw probabilities between 0 and 1. The shape of output of a softmax is the same as the input:
```
a = tf.constant(np.array([[.1, .3, .5, .9]]))
print s.run(tf.nn.softmax(a))
[[ 0.16838508  0.205666    0.25120102  0.37474789]]
```
See this answer for more about why softmax is used extensively in DNNs. tf.nn.softmax_cross_entropy_with_logits combines the softmax step with the calculation of the cross-entropy loss after applying the softmax function, but it does it all together in a more mathematically careful way. It's similar to the result of:
```
sm = tf.nn.softmax(x)
ce = cross_entropy(sm)
```
The cross entropy is a summary metric: it sums across the elements. The output of tf.nn.softmax_cross_entropy_with_logits on a shape [2,5] tensor is of shape [2,1] (the first dimension is treated as the batch). If you want to do optimization to minimize the cross entropy AND you're softmaxing after your last layer, you should use tf.nn.softmax_cross_entropy_with_logits instead of doing it yourself, because it covers numerically unstable corner cases in the mathematically right way. Otherwise, you'll end up hacking it by adding little epsilons here and there. Edited 2016-02-07: If you have single-class labels, where an object can only belong to one class, you might now consider using tf.nn.sparse_softmax_cross_entropy_with_logits so that you don't have to convert your labels to a dense one-hot array. This function was added after release 0.6.0.


Question: Obtaining total number of records from .tfrecords file in Tensorflow Is it possible for obtain the total number of records from a .tfrecords file ? Related to this, how does one generally keep track of the number of epochs that have elapsed while training models? While it is possible for us to specify the batch_size and num_of_epochs, I am not sure if it is straightforward to obtain values such as current epoch, number of batches per epoch etc - just so that I could have more control of how the training is progressing. Currently, I'm just using a dirty hack to compute this as I know before hand how many records there are in my .tfrecords file and the size of my minibatches. Appreciate any help..

Response:
To count the number of records, you should be able to use tf.python_io.tf_record_iterator.
```py
c = 0
for fn in tf_records_filenames:
  for record in tf.python_io.tf_record_iterator(fn):
     c += 1
```
To just keep track of the model training, tensorboard comes in handy.


Question: Tensorboard graph recall I am training an object detector and I ran the evaluation job. I see certain graphs in the tensorboard. What is DetectionBoxes_Recall/AR@10 vs AR@100 vs AR@100(medium) in the tensorflowboard as shown. And what is the difference between DetectionBoxes_Precision/mAP, mAP(large), mAP(medium), mAP(small), mAP(0.50IOU) and mAP(0.75IOU)? Please help I am very new to this thank you.

Response: 'DetectionBoxes_Precision/mAP': mean average precision over classes averaged over IOU thresholds ranging from .5 to .95 with .05 increments. 'DetectionBoxes_Precision/[email protected]': mean average precision at 50% IOU 'DetectionBoxes_Precision/[email protected]': mean average precision at 75% IOU 'DetectionBoxes_Precision/mAP (small)': mean average precision for small objects (area < 32^2 pixels). 'DetectionBoxes_Precision/mAP (medium)': mean average precision for medium sized objects (32^2 pixels < area < 96^2 pixels). 'DetectionBoxes_Precision/mAP (large)': mean average precision for large objects (96^2 pixels < area < 10000^2 pixels). 'DetectionBoxes_Recall/AR@1': average recall with 1 detection. 'DetectionBoxes_Recall/AR@10': average recall with 10 detections. 'DetectionBoxes_Recall/AR@100': average recall with 100 detections. 'DetectionBoxes_Recall/AR@100 (small)': average recall for small objects with 100. 'DetectionBoxes_Recall/AR@100 (medium)': average recall for medium objects with 100. 'DetectionBoxes_Recall/AR@100 (large)': average recall for large objects with 100 detections.


Question:
Tensorflow: save the model with smallest validation error I ran a training job with tensorflow and got the following curve for loss on validation set. The net starts to overfit after 6000-th iteration. So I'd like to get the model before overfitting. My training code is something like below:
```
train_step = ......
summary = tf.scalar_summary(l1_loss.op.name, l1_loss)
summary_writer = tf.train.SummaryWriter("checkpoint", sess.graph)
saver = tf.train.Saver()
for i in xrange(20000):
    batch = get_next_batch(batch_size)
    sess.run(train_step, feed_dict = {x: batch.x, y:batch.y})
    if (i+1) % 100 == 0:
        saver.save(sess, "checkpoint/net", global_step = i+1)
        summary_str = sess.run(summary, feed_dict=validation_feed_dict)
        summary_writer.add_summary(summary_str, i+1)
        summary_writer.flush()
```
After training finishes, there is only five checkpoints saved (19600, 19700, 19800, 19900, 20000). Is there any way to let tensorflow save checkpoint according to the validation error? P.S. I know that tf.train.Saver has a max_to_keep argument, which in principal could save all the checkpoints. But that's not I wanted (unless it's the only option). I want the saver keep the checkpoint with the smallest validation loss so far. Is that possible?

Response: You need to calculate the classification accuracy on the validation-set and keep track of the best one seen so far, and only write the checkpoint once an improvement has been found to the validation accuracy. If the data-set and/or model is large, then you may have to split the validation-set into batches to fit the computation in memory. This tutorial shows exactly how to do what you want: https://github.com/Hvass-Labs/TensorFlow-Tutorials/blob/master/04_Save_Restore.ipynb It is also available as a short video: https://www.youtube.com/watch?v=Lx8JUJROkh0


Question:
Get Gradients with Keras Tensorflow 2.0 I would like to keep track of the gradients over tensorboard. However, since session run statements are not a thing anymore and the write_grads argument of tf.keras.callbacks.TensorBoard is deprecated, I would like to know how to keep track of gradients during training with Keras or tensorflow 2.0. My current approach is to create a new callback class for this purpose, but without success. Maybe someone else knows how to accomplish this kind of advanced stuff. The code created for testing is shown below, but runs into errors independently of printing a gradient value to console or tensorboard.
```
import tensorflow as tf
from tensorflow.python.keras import backend as K

mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu', name='dense128'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax', name='dense10')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

class GradientCallback(tf.keras.callbacks.Callback):
    console = True

    def on_epoch_end(self, epoch, logs=None):
        weights = [w for w in self.model.trainable_weights if 'dense' in w.name and 'bias' in w.name]
        loss = self.model.total_loss
        optimizer = self.model.optimizer
        gradients = optimizer.get_gradients(loss, weights)
        for t in gradients:
            if self.console:
                print('Tensor: {}'.format(t.name))
                print('{}\n'.format(K.get_value(t)[:10]))
            else:
                tf.summary.histogram(t.name, data=t)

file_writer = tf.summary.create_file_writer("./metrics")
file_writer.set_as_default()

# write_grads has been removed
tensorboard_cb = tf.keras.callbacks.TensorBoard(histogram_freq=1, write_grads=True)
gradient_cb = GradientCallback()

model.fit(x_train, y_train, epochs=5, callbacks=[gradient_cb, tensorboard_cb])
```
Priniting bias gradients to console (console parameter = True) leads to: AttributeError: 'Tensor' object has no attribute 'numpy' Writing to tensorboard (console parameter = False) creates: TypeError: Using a tf.Tensor as a Python bool is not allowed. Use if t is not None: instead of if t: to test if a tensor is defined, and use TensorFlow ops such as tf.cond to execute subgraphs conditioned on the value of a tensor.

Response:
To compute the gradients of the loss against the weights, use
```
with tf.GradientTape() as tape:
    loss = model(model.trainable_weights)

tape.gradient(loss, model.trainable_weights)
```
This is (arguably poorly) documented on GradientTape. We do not need to tape.watch the variable because trainable parameters are watched by default. As a function, it can be written as
```
def gradient(model, x):
    x_tensor = tf.convert_to_tensor(x, dtype=tf.float32)
    with tf.GradientTape() as t:
        t.watch(x_tensor)
        loss = model(x_tensor)
    return t.gradient(loss, x_tensor).numpy()
```


Question: How do I split Tensorflow datasets? I have a tensorflow dataset based on one .tfrecord file. How do I split the dataset into test and train datasets? E.g. 70% Train and 30% test? Edit: My Tensorflow Version: 1.8 I've checked, there is no "split_v" function as mentioned in the possible duplicate. Also I am working with a tfrecord file.

Response:
You may use Dataset.take() and Dataset.skip():
```
train_size = int(0.7 * DATASET_SIZE)
val_size = int(0.15 * DATASET_SIZE)
test_size = int(0.15 * DATASET_SIZE)

full_dataset = tf.data.TFRecordDataset(FLAGS.input_file)
full_dataset = full_dataset.shuffle()
train_dataset = full_dataset.take(train_size)
test_dataset = full_dataset.skip(train_size)
val_dataset = test_dataset.skip(test_size)
test_dataset = test_dataset.take(test_size)
```
For more generality, I gave an example using a 70/15/15 train/val/test split but if you don't need a test or a val set, just ignore the last 2 lines. Take: Creates a Dataset with at most count elements from this dataset. Skip: Creates a Dataset that skips count elements from this dataset. You may also want to look into Dataset.shard(): Creates a Dataset that includes only 1/num_shards of this dataset.


Question:
how to set rmse cost function in tensorflow I have cost function in tensorflow.
```
activation = tf.add(tf.mul(X, W), b)
cost = (tf.pow(Y-y_model, 2)) # use sqr error for cost function
```
I am trying out this example. How can I change it to rmse cost function?

Response:
```
tf.sqrt(tf.reduce_mean(tf.square(tf.subtract(targets, outputs))))
```
And slightly simplified (TensorFlow overloads the most important operators):
```
tf.sqrt(tf.reduce_mean((targets - outputs)**2))
```


Question:
How do I select certain columns of a 2D tensor in TensorFlow? As generalized slicing is being worked on in this issue, what would be the best way to achieve an op gathering columns of a 2D tensor (matrix)? For example, for tensor t:
```
1 2 3 4
5 6 7 8
```
and indices [1,3], I would like to get:
```
2 4
6 8
```
which is equivalent to numpy t[:, [1,3]].

Response:
Meanwhile the gather method has an axis parameter.
```
import tensorflow as tf
params = tf.constant([[1,2,3],[4,5,6]])
indices = [0,2]
op = tf.gather(params, indices, axis=1)
```
produces the output
```
[[1 3]
 [4 6]]
```


Question: Tensorflow create a tfrecords file from csv I am trying to write a csv file (all columns are floats) to a tfrecords file then read them back out. All the examples I have seen pack the csv columns then feed it to sess.run() directly but I can't figure out how to write the feature columns and label column to a tfrecord instead. How could I do this?

Response:
You will need a separate script to convert your csv file to TFRecords. Imagine you have a CSV with the following header:
```
feature_1, feature_2, ..., feature_n, label
```
You need to read your CSV with something like pandas, construct tf.train.Example manually and then write it to file with TFRecordWriter
```
csv = pandas.read_csv("your.csv").values
with tf.python_io.TFRecordWriter("csv.tfrecords") as writer:
    for row in csv:
        features, label = row[:-1], row[-1]
        example = tf.train.Example()
        example.features.feature["features"].float_list.value.extend(features)
        example.features.feature["label"].int64_list.value.append(label)
        writer.write(example.SerializeToString())
```


Question:
Tensorflow: How to convert scalar tensor to scalar variable in python? In Tensorflow, I'd like to convert a scalar tensor to an integer. Is it possible to do? I need to create a loop and the index of the loop is a scalar tensor, and inside the loop body, I want to use the index to access an entry in a tensor array. For example:
```
idx = tf.constant(0)
c = lambda i : tf.less(i, 10)
def body(idx) :
  i = # convert idx to int
  b = weights[i]  # access an entry in a tensor array, tensor cannot be used directly
  ....
  return idx+1
tf.while_loop(c, body, [idx])
```

Response:
In Tensorflow 2.0+, it's as simple as:
```py
my_tensor.numpy()
```


Question:
What does the .numpy() function do? I tried searching for the documentation online but I can't find anything that gives me an answer. What does .numpy() function do? The example code given is:
```
y_true = []
for X_batch, y_batch in mnist_test:
    y_true.append(y_batch.numpy()[0].tolist())
```

Response: Both in Pytorch and Tensorflow, the .numpy() method is pretty much straightforward. It converts a tensor object into an numpy.ndarray object. This implicitly means that the converted tensor will be now processed on the CPU.


Question: softmax and sigmoid function for the output layer In the deep learning implementations related to object detection and semantic segmentation, I have seen the output layers using either sigmoid or softmax. I am not very clear when to use which? It seems to me both of them can support these tasks. Are there any guidelines for this choice?

Response: softmax() helps when you want a probability distribution, which sums up to 1. sigmoid is used when you want the output to be ranging from 0 to 1, but need not sum to 1. In your case, you wish to classify and choose between two alternatives. I would recommend using softmax() as you will get a probability distribution which you can apply cross entropy loss function on.


Question: Tensorboard scalar plotting with epoch number on the horizontal axis I am new to TensorFlow, and I recently started to play around a little bit with data visualization using Tensorboard. I was wondering if it is possible to convert the horizontal axis of the monitoring scalars (I monitor accuracy and loss on train and validation) to show epoch number instead of iteration number. the only way I can think of how to do it is to change the sampling frequency to one per epoch, but I am interested in keeping the original sampling resolution. is there a better way?

Response:
Yes, you can do this by passing the epoch number to the global_step parameter of the add_summary() method:
```
summary_writer = tf.summary.FileWriter(log_dir)

my_summary = session.run(my_summary_op, feed_dict)
summary_writer.add_summary(my_summary, global_step=epoch_number)
```


Question: Understanding Tensorflow LSTM Input shape I have a dataset X which consists N = 4000 samples, each sample consists of d = 2 features (continuous values) spanning back t = 10 time steps. I also have the corresponding 'labels' of each sample which are also continuous values, at time step 11. At the moment my dataset is in the shape X: [4000,20], Y: [4000]. I want to train an LSTM using TensorFlow to predict the value of Y (regression), given the 10 previous inputs of d features, but I am having a tough time implementing this in TensorFlow. The main problem I have at the moment is understanding how TensorFlow is expecting the input to be formatted. I have seen various examples such as this, but these examples deal with one big string of continuous time series data. My data is different samples, each an independent time series.

Response:
The documentation of tf.nn.dynamic_rnn states: inputs: The RNN inputs. If time_major == False (default), this must be a Tensor of shape: [batch_size, max_time, ...], or a nested tuple of such elements. In your case, this means that the input should have a shape of [batch_size, 10, 2]. Instead of training on all 4000 sequences at once, you'd use only batch_size many of them in each training iteration. Something like the following should work (added reshape for clarity):
```
batch_size = 32
# batch_size sequences of length 10 with 2 values for each timestep
input = get_batch(X, batch_size).reshape([batch_size, 10, 2])
# Create LSTM cell with state size 256. Could also use GRUCell, ...
# Note: state_is_tuple=False is deprecated;
# the option might be completely removed in the future
cell = tf.nn.rnn_cell.LSTMCell(256, state_is_tuple=True)
outputs, state = tf.nn.dynamic_rnn(cell,
                                   input,
                                   sequence_length=[10]*batch_size,
                                   dtype=tf.float32)
```
From the documentation, outputs will be of shape [batch_size, 10, 256], i.e. one 256-output for each timestep. state will be a tuple of shapes [batch_size, 256]. You could predict your final value, one for each sequence, from that:
```
predictions = tf.contrib.layers.fully_connected(state.h,
                                                num_outputs=1,
                                                activation_fn=None)
loss = get_loss(get_batch(Y).reshape([batch_size, 1]), predictions)
```
The number 256 in the shapes of outputs and state is determined by cell.output_size resp. cell.state_size. When creating the LSTMCell like above, these are the same. Also see the LSTMCell documentation.


Question: Tensorflow object detection API not displaying global steps I am new here. I recently started working with object detection and decided to use the Tensorflow object detection API. But, when I start training the model, it does not display the global step like it should, although it's still training in the background. Details: I am training on a server and accessing it using OpenSSH on Windows. I trained a custom dataset, by collecting pictures and labeling them. I trained it using model_main.py. Also, until a couple of months back, the API was a little different, and only recently they changed to the latest version. For instance, earlier it used to use train.py for training, instead of model_main.py. All the online tutorial I can find use train.py, so it might be a problem with the latest commit. But I don't find anyone else fining this problem. Thanks in advance!

Response: Add tf.logging.set_verbosity(tf.logging.INFO) after the import section of the model_main.py script. It will display a summary after every 100th step.


Question:
How does tensorflow batch_matmul work? Tensorflow has a function called batch_matmul which multiplies higher dimensional tensors. But I'm having a hard time understanding how it works, perhaps partially because I'm having a hard time visualizing it. What I want to do is multiply a matrix by each slice of a 3D tensor, but I don't quite understand what the shape of tensor a is. Is z the innermost dimension? Which of the following is correct? I would most prefer the first to be correct -- it's most intuitive to me and easy to see in the .eval() output. But I suspect the second is correct. Tensorflow says that batch_matmul performs:
```
out[..., :, :] = matrix(x[..., :, :]) * matrix(y[..., :, :])
```
What does that mean? What does that mean in the context of my example? What is being multiplied with with what? And why aren't I getting a 3D tensor the way I expected?

Response:
You can imagine it as doing a matmul over each training example in the batch. For example, if you have two tensors with the following dimensions:
```
a.shape = [100, 2, 5]
b.shape = [100, 5, 2]
```
and you do a batch tf.matmul(a, b), your output will have the shape [100, 2, 2]. 100 is your batch size, the other two dimensions are the dimensions of your data.


Question: Remove nodes from graph or reset entire default graph When working with the default global graph, is it possible to remove nodes after they've been added, or alternatively to reset the default graph to empty? When working with TF interactively in IPython, I find myself having to restart the kernel repeatedly. I would like to be able to experiment with graphs more easily if possible.

Response:
Update 11/2/2016 tf.reset_default_graph() Old stuff There's reset_default_graph, but not part of public API (I think it should be, does someone wants to file an issue on GitHub?) My work-around to reset things is this:
```
from tensorflow.python.framework import ops
ops.reset_default_graph()
sess = tf.InteractiveSession()
```


Question:
Tensorflow - matmul of input matrix with batch data I have some data represented by input_x. It is a tensor of unknown size (should be inputted by batch) and each item there is of size n. input_x undergoes tf.nn.embedding_lookup, so that embed now has dimensions [?, n, m] where m is the embedding size and ? refers to the unknown batch size. This is described here:
```
input_x = tf.placeholder(tf.int32, [None, n], name="input_x")
embed = tf.nn.embedding_lookup(W, input_x)
```
I'm now trying to multiply each sample in my input data (which is now expanded by embedding dimension) by a matrix variable, U, and I can't seem to get how to do that. I first tried using tf.matmul but it gives an error due to mismatch in shapes. I then tried the following, by expanding the dimension of U and applying batch_matmul (I also tried the function from tf.nn.math_ops., the result was the same):
```
U = tf.Variable( ... )
U1 = tf.expand_dims(U,0)
h=tf.batch_matmul(embed, U1)
```
This passes the initial compilation, but then when actual data is applied, I get the following error: In[0].dim(0) and In[1].dim(0) must be the same: [64,58,128] vs [1,128,128] I also know why this is happening - I replicated the dimension of U and it is now 1, but the minibatch size, 64, doesn't fit. How can I do that matrix multiplication on my tensor-matrix input correctly (for unknown batch size)?

Response:
Previous answers are obsolete. Currently tf.matmul() support tensors with rank > 2: The inputs must be matrices (or tensors of rank > 2, representing batches of matrices), with matching inner dimensions, possibly after transposition. Also tf.batch_matmul() was removed and tf.matmul() is the right way to do batch multiplication. The main idea can be understood from the following code:
```
import tensorflow as tf
batch_size, n, m, k = 10, 3, 5, 2
A = tf.Variable(tf.random_normal(shape=(batch_size, n, m)))
B = tf.Variable(tf.random_normal(shape=(batch_size, m, k)))
tf.matmul(A, B)
```
Now you will receive a tensor of the shape (batch_size, n, k). Here is what is going on here. Assume you have batch_size of matrices nxm and batch_size of matrices mxk. Now for each pair of them you calculate nxm X mxk which gives you an nxk matrix. You will have batch_size of them. Notice that something like this is also valid:
```
A = tf.Variable(tf.random_normal(shape=(a, b, n, m)))
B = tf.Variable(tf.random_normal(shape=(a, b, m, k)))
tf.matmul(A, B)
```
and will give you a shape (a, b, n, k)


Question:
Keras early stopping callback error, val_loss metric not available I am training a Keras (Tensorflow backend, Python, on MacBook) and am getting an error in the early stopping callback in fit_generator function. The error is as follows:
```
RuntimeWarning: Early stopping conditioned on metric `val_loss` which is not available. Available metrics are:
  (self.monitor, ','.join(list(logs.keys()))),
RuntimeWarning: Can save best model only with val_acc available, skipping.

'skipping.' % (self.monitor), RuntimeWarning
[local-dir]/lib/python3.6/site-packages/keras/callbacks.py:497: RuntimeWarning: Early stopping conditioned on metric `val_loss` which is not available. Available metrics are:
  (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning
[local-dir]/lib/python3.6/site-packages/keras/callbacks.py:406: RuntimeWarning: Can save best model only with val_acc available, skipping.
  'skipping.' % (self.monitor), RuntimeWarning)
Traceback (most recent call last):
  :
  [my-code]
  :
  File "[local-dir]/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
  File "[local-dir]/lib/python3.6/site-packages/keras/engine/training.py", line 2213, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
  File "[local-dir]/lib/python3.6/site-packages/keras/callbacks.py", line 76, in on_epoch_end
callback.on_epoch_end(epoch, logs)
  File "[local-dir]/lib/python3.6/site-packages/keras/callbacks.py", line 310, in on_epoch_end
self.progbar.update(self.seen, self.log_values, force=True)
AttributeError: 'ProgbarLogger' object has no attribute 'log_values'
```
My code is as follows (which looks OK):
```
:
ES = EarlyStopping(monitor="val_loss", min_delta=0.001, patience=3, mode="min", verbose=1)
:
self.model.fit_generator(
        generator        = train_batch,
        validation_data  = valid_batch,
        validation_steps = validation_steps,
        steps_per_epoch  = steps_per_epoch,
        epochs           = epochs,
        callbacks        = [ES],
        verbose          = 1,
        workers          = 3,
        max_queue_size   = 8)
```
The error message appears to relate to the early stopping callback but the callback looks OK. Also the error states that the val_loss is not appropriate, but I am not sure why... one more unusual thing about this is that the error only occurs when I use smaller data sets. Any help is appreciated.

Response: If the error only occurs when you use smaller datasets, you're very likely using datasets small enough to not have a single sample in the validation set. Thus it cannot calculate a validation loss.


Question:
'tf' is not defined on load_model() - using lambda I have a Keras model that I am trying to export and use in a different python code. Here is my code:
```
from keras.models import Sequential
from keras.layers import Dense, Embedding, LSTM, GRU, Flatten, Dropout, Lambda
from keras.layers.embeddings import Embedding
import tensorflow as tf

EMBEDDING_DIM = 100

model = Sequential()
model.add(Embedding(vocab_size, 300, weights=[embedding_matrix], input_length=max_length, trainable=False))
model.add(Lambda(lambda x: tf.reduce_mean(x, axis=1)))
model.add(Dense(8, input_dim=4, activation='relu'))
model.add(Dense(3, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X_train_pad, y_train, batch_size=128, epochs=25, validation_data=(X_val_pad, y_val), verbose=2)
model.save('my_model.h5')
```
In another file, when I import my_model.h5 :
```
from keras.models import load_model
from keras.layers import Lambda
import tensorflow as tf

def learning(test_samples):
    model = load_model('my_model.h5')
    #ERROR HERE
    #rest of the code
```
The error is the following:
```
in <lambda>
    model.add(Lambda(lambda x: tf.reduce_mean(x, axis=1)))
NameError: name 'tf' is not defined
```
After research, I got that the fact that I used lambda in my model is the reason for this problem, but I added these references and it didn't help:
```
from keras.models import load_model
from keras.layers import Lambda
import tensorflow as tf
```
What could be the problem? Thank you

Response:
When loading the model, you need to explicitly handle custom objects or custom layers (CTRL+f the docs for Handling custom layers):
```
import tensorflow as tf
import keras
model = keras.models.load_model('my_model.h5', custom_objects={'tf': tf})
```


Question:
what is the difference between Flatten() and GlobalAveragePooling2D() in keras I want to pass the output of ConvLSTM and Conv2D to a Dense Layer in Keras, what is the difference between using global average pooling and flatten Both is working in my case.
```python
model.add(ConvLSTM2D(filters=256,kernel_size=(3,3)))
model.add(Flatten())
# or model.add(GlobalAveragePooling2D())
model.add(Dense(256,activation='relu'))
```

Response: That both seem to work doesn't mean they do the same. Flatten will take a tensor of any shape and transform it into a one dimensional tensor (plus the samples dimension) but keeping all values in the tensor. For example a tensor (samples, 10, 20, 1) will be flattened to (samples, 10 * 20 * 1). GlobalAveragePooling2D does something different. It applies average pooling on the spatial dimensions until each spatial dimension is one, and leaves other dimensions unchanged. In this case values are not kept as they are averaged. For example a tensor (samples, 10, 20, 1) would be output as (samples, 1, 1, 1), assuming the 2nd and 3rd dimensions were spatial (channels last).