Question: Log accuracy metric while training a tf.estimator What's the simplest way to print accuracy metrics along with the loss when training a pre-canned estimator? Most tutorials and documentations seem to address the issue of when you're creating a custom estimator -- which seems overkill if the intention is to use one of the available ones. tf.contrib.learn had a few (now deprecated) Monitor hooks. TF now suggests using the hook API, but it appears that it doesn't actually come with anything that can utilize the labels and predictions to generate an accuracy number.

Response:
Have you tried tf.contrib.estimator.add_metrics(estimator, metric_fn) (doc)? It takes an initialized estimator (can be pre-canned) and adds to it the metrics defined by metric_fn. Usage Example:
```python
def custom_metric(labels, predictions):
    # This function will be called by the Estimator, passing its predictions.
    # Let's suppose you want to add the "mean" metric...

    # Accessing the class predictions (careful, the key name may change from one canned Estimator to another)
    predicted_classes = predictions["class_ids"]

    # Defining the metric (value and update tensors):
    custom_metric = tf.metrics.mean(labels, predicted_classes, name="custom_metric")

    # Returning as a dict:
    return {"custom_metric": custom_metric}

# Initializing your canned Estimator:
classifier = tf.estimator.DNNClassifier(feature_columns=columns_feat, hidden_units=[10, 10], n_classes=NUM_CLASSES)

# Adding your custom metrics:
classifier = tf.contrib.estimator.add_metrics(classifier, custom_metric)

# Training/Evaluating:
tf.logging.set_verbosity(tf.logging.INFO) # Just to have some logs to display for demonstration

train_spec = tf.estimator.TrainSpec(input_fn=lambda:your_train_dataset_function(),
                                    max_steps=TRAIN_STEPS)
eval_spec=tf.estimator.EvalSpec(input_fn=lambda:your_test_dataset_function(),
                                steps=EVAL_STEPS,
                                start_delay_secs=EVAL_DELAY,
                                throttle_secs=EVAL_INTERVAL)
tf.estimator.train_and_evaluate(classifier, train_spec, eval_spec)
```
Logs:
```
...
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Evaluation [20/200]
INFO:tensorflow:Evaluation [40/200]
...
INFO:tensorflow:Evaluation [200/200]
INFO:tensorflow:Finished evaluation at 2018-04-19-09:23:03
INFO:tensorflow:Saving dict for global step 1: accuracy = 0.5668, average_loss = 0.951766, custom_metric = 1.2442, global_step = 1, loss = 95.1766
...
```
As you can see, the custom_metric is returned along the default metrics and loss.


Question:
Tensorflow allocating GPU memory when using tf.device('/cpu:0') System Info: 1.1.0, GPU, Windows, Python 3.5, code runs in ipython consoles. I am trying to run two different Tensorflow sessions, one on the GPU (that does some batch work) and one on the CPU that I use for quick tests while the other works. The problem is that when I spawn the second session specifying with tf.device('/cpu:0') the session tries to allocate GPU memory and crashes my other session. My code:
```
import os
os.environ["CUDA_VISIBLE_DEVICES"] = ""
import time

import tensorflow as tf

with tf.device('/cpu:0'):
  with tf.Session() as sess:
    # Here 6 GBs of GPU RAM are allocated.
    time.sleep(5)
```
How do I force Tensorflow to ignore the GPU? UPDATE: As suggested in a comment by @Nicolas, I took a look at this answer and ran
```
import os
os.environ["CUDA_VISIBLE_DEVICES"] = ""
import tensorflow as tf

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())
```
which prints:
```
[name: "/cpu:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 2215045474989189346
, name: "/gpu:0"
device_type: "GPU"
memory_limit: 6787871540
locality {
  bus_id: 1
}
incarnation: 13663872143510826785
physical_device_desc: "device: 0, name: GeForce GTX 1080, pci bus id: 0000:02:00.0"
]
```
It seems to me that even if I explicitly tell the script to ignore any CUDA devices, it still finds and uses them. Could this be a bug of TF 1.1?

Response:
It turns out that setting CUDA_VISIBLE_DEVICES to the empty string does not mask the CUDA devices visible to the script. From the documentation of CUDA_VISIBLE_DEVICES (emphasis added by me): Only the devices whose index is present in the sequence are visible to CUDA applications and they are enumerated in the order of the sequence. If one of the indices is invalid, only the devices whose index precedes the invalid index are visible to CUDA applications. For example, setting CUDA_VISIBLE_DEVICES to 2,1 causes device 0 to be invisible and device 2 to be enumerated before device 1. Setting CUDA_VISIBLE_DEVICES to 0,2,-1,1 causes devices 0 and 2 to be visible and device 1 to be invisible. It seems like the empty string used to be handled as "no valid devices exist" but changed meaning, as it is not mentioned in the documentation. Changing the code to os.environ["CUDA_VISIBLE_DEVICES"] = "-1" fixes the problem. Running
```
import os
os.environ["CUDA_VISIBLE_DEVICES"]="-1"
import tensorflow as tf

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())
```
now prints
```
[name: "/cpu:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 14097726166554667970
]
```
and instantiating a tf.Session does not hog GPU memory anymore.


Question: How to visualize learned filters on tensorflow Similarly to the Caffe framework, where it is possible to watch the learned filters during CNNs training and it's resulting convolution with input images, I wonder if is it possible to do the same with TensorFlow? A Caffe example can be viewed in this link: http://nbviewer.jupyter.org/github/BVLC/caffe/blob/master/examples/00-classification.ipynb Grateful for your help!

Response:
To see just a few conv1 filters in Tensorboard, you can use this code (it works for cifar10)
```
# this should be a part of the inference(images) function in cifar10.py file

# conv1
with tf.variable_scope('conv1') as scope:
  kernel = _variable_with_weight_decay('weights', shape=[5, 5, 3, 64],
                                       stddev=1e-4, wd=0.0)
  conv = tf.nn.conv2d(images, kernel, [1, 1, 1, 1], padding='SAME')
  biases = _variable_on_cpu('biases', [64], tf.constant_initializer(0.0))
  bias = tf.nn.bias_add(conv, biases)
  conv1 = tf.nn.relu(bias, name=scope.name)
  _activation_summary(conv1)

  with tf.variable_scope('visualization'):
    # scale weights to [0 1], type is still float
    x_min = tf.reduce_min(kernel)
    x_max = tf.reduce_max(kernel)
    kernel_0_to_1 = (kernel - x_min) / (x_max - x_min)

    # to tf.image_summary format [batch_size, height, width, channels]
    kernel_transposed = tf.transpose (kernel_0_to_1, [3, 0, 1, 2])

    # this will display random 3 filters from the 64 in conv1
    tf.image_summary('conv1/filters', kernel_transposed, max_images=3)
```
I also wrote a simple gist to display all 64 conv1 filters in a grid.


Question: module 'tensorflow.python.keras.api._v2.keras.layers' has no attribute 'CuDNNLSTM' When I write tf.keras.layers.LSTM, I get the warning Note that this layer is not optimized for performance. Please use tf.keras.layers.CuDNNLSTM for better performance on GPU. But when I change the layer to tf.keras.layers.CuDNNLSTM, I get the error AttributeError: module 'tensorflow.python.keras.api._v2.keras.layers' has no attribute 'CuDNNLSTM' Tensorflow version is 2.0.0-alpha0, Keras version is 2.2.4-tf. How can I fix this problem?

Response:
In general, in TensorFlow 2.0 we should just use:
```py
tf.keras.layers.LSTM
```
which, despite the warning, will use the GPU. The warning message incorrectly existed in the 2.0.0-alpha0 version but has since been removed in 2.0.0-beta1 If for some reason you specifically need the original implementation of tf.keras.layers.CuDNNLSTM then you can use tf.compat.v1.keras.layers.CuDNNLSTM but this would be an edge case.


Question:
How to write a custom f1 loss function with weighted average for keras? I am trying to do a multiclass classification in keras. Till now I am using categorical_crossentropy as the loss function. But since the metric required is weighted-f1, I am not sure if categorical_crossentropy is the best loss choice. I was trying to implement a weighted-f1 score in keras using sklearn.metrics.f1_score, but due to the problems in conversion between a tensor and a scalar, I am running into errors. Something like this:
```
def f1_loss(y_true, y_pred):
   return 1 - f1_score(np.argmax(y_true, axis=1), np.argmax(y_pred, axis=1), average='weighted')
```
Followed by
```
model.compile(loss=f1_loss, optimizer=opt)
```
How do I write this loss function in keras? Edit: Shape for y_true and y_pred is (n_samples, n_classes) in my case it is (n_samples, 4) y_true and y_pred both are tensors so sklearn's f1_score cannot work directly on them. I need a function that calculates weighted f1 on tensors.

Response:
The variables are self explained:
```
def f1_weighted(true, pred): #shapes (batch, 4)

    #for metrics include these two lines, for loss, don't include them
    #these are meant to round 'pred' to exactly zeros and ones
    #predLabels = K.argmax(pred, axis=-1)
    #pred = K.one_hot(predLabels, 4)

    ground_positives = K.sum(true, axis=0) + K.epsilon()       # = TP + FN
    pred_positives = K.sum(pred, axis=0) + K.epsilon()         # = TP + FP
    true_positives = K.sum(true * pred, axis=0) + K.epsilon()  # = TP
        #all with shape (4,)

    precision = true_positives / pred_positives
    recall = true_positives / ground_positives
        #both = 1 if ground_positives == 0 or pred_positives == 0
        #shape (4,)

    f1 = 2 * (precision * recall) / (precision + recall + K.epsilon())
        #still with shape (4,)

    weighted_f1 = f1 * ground_positives / K.sum(ground_positives)
    weighted_f1 = K.sum(weighted_f1)

    return 1 - weighted_f1 #for metrics, return only 'weighted_f1'
```
Important notes: This loss will work batchwise (as any Keras loss). So if you are working with small batch sizes, the results will be unstable between each batch, and you may get a bad result. Use big batch sizes, enough to include a significant number of samples for all classes. Since this loss collapses the batch size, you will not be able to use some Keras features that depend on the batch size, such as sample weights, for instance.


Question: how do I use a very large (>2M) word embedding in tensorflow? I am running a model with a very big word embedding (>2M words). When I use tf.embedding_lookup, it expects the matrix, which is big. When I run, I subsequently get out of GPU memory error. If I reduce the size of the embedding, everything works fine. Is there a way to deal with larger embedding?

Response:
The recommended way is to use a partitioner to shard this large tensor across several parts:
```py
embedding = tf.get_variable("embedding", [1000000000, 20],
                            partitioner=tf.fixed_size_partitioner(3))
```
This will split the tensor into 3 shards along 0 axis, but the rest of the program will see it as an ordinary tensor. The biggest benefit is to use a partitioner along with parameter server replication, like this:
```py
with tf.device(tf.train.replica_device_setter(ps_tasks=3)):
  embedding = tf.get_variable("embedding", [1000000000, 20],
                              partitioner=tf.fixed_size_partitioner(3))
```
The key function here is tf.train.replica_device_setter. It allows you to run 3 different processes, called parameter servers, that store all of model variables. The large embedding tensor will be split across these servers like on this picture.


Question:
Does SHAP in Python support Keras or TensorFlow models while using DeepExplainer? I am currently using SHAP Package to determine the feature contributions. I have used the approach for XGBoost and RandomForest and it worked really well. Since the data I am working on is a sequential data I tried using LSTM and CNN to train the model and then get the feature importance using the SHAP's DeepExplainer; but it is continuously throwing error. The error I am getting is:
```
AssertionError: <class 'keras.callbacks.History'> is not currently a supported model type!.
```
I am attaching the sample code as well (LSTM). It would be helpful if someone could help me with it.
```
shap.initjs()
model = Sequential()
model.add(LSTM(n_neurons, input_shape=(X.shape[1],X.shape[2]), return_sequences=True))
model.add(LSTM(n_neurons, return_sequences=False))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
h=model.fit(X, y, epochs=nb_epochs, batch_size=n_batch, verbose=1, shuffle=True)
background = X[np.random.choice(X.shape[0],100, replace=False)]
explainer = shap.DeepExplainer(h,background)
```

Response:
The returned value of model.fit is not the model instance; rather, it's the history of training (i.e. stats like loss and metric values) as an instance of keras.callbacks.History class. That's why you get the mentioned error when you pass the returned History object to shap.DeepExplainer. Instead, you should pass the model instance itself:
```
explainer = shap.DeepExplainer(model, background)
```


Question: GPU is lost during execution of either Tensorflow or Theano code When training either one of two different neural networks, one with Tensorflow and the other with Theano, sometimes after a random amount of time (could be a few hours or minutes, mostly a few hours), the execution freezes and I get this message by running "nvidia-smi": "Unable to determine the device handle for GPU 0000:02:00.0: GPU is lost. Reboot the system to recover this GPU" I tried to monitor the GPU performance for 13-hours execution, and everything seems stable: I'm working with: Ubuntu 14.04.5 LTS GPUs are Nvidia Titan Xp (this behavior repeats on another GPU on the same machine) CUDA 8.0 CuDNN 5.1 Tensorflow 1.3 Theano 0.8.2 I'm not sure how to approach this problem, can anyone please suggest ideas of what can cause this and how to diagnose/fix this?

Response: I posted this question a while ago, but after some investigation back then that took a few weeks, we managed to find the problem (and a solution). I don't remember all the details now, but I'm posting our main conclusion, in case someone will find it useful. Bottom line is - the hardware we had was not strong enough to support high load GPU-CPU communication. We observed these issues on a rack server with 1 CPU and 4 GPU devices, There was simply an overload on the PCI bus. The problem was solved by adding another CPU to the rack server.


Question: LSTMStateTuple vs cell.zero_state() for RNN in Tensorflow I am confused on what is the correct way to use the initial state tensor in Tensorflow for RNNs. There is almost a 50/50 split between turtorials that either use LSTMStateTuple or cell.zero_state. Are the two the same? If so, why are there two ways? In one example they use tf.nn.rnn_cell.LSTMStateTuple to set the initial state and in the other they use cell.zero_state(). Why are there two methods? When to prefer the one or the other? Can you only use LSTMStateTuple when you set state_is_tuple? If so, does cell.zero_state() no longer work?

Response:
The two are different things. state_is_tuple is used on LSTM cells because the state of LSTM cells is a tuple. cell.zero_state is the initializer of the state for all RNN cells. You will generally prefer cell.zero_state function as it will initialize the required state class depending on whether state_is_tuple is true or not. See this GitHub issue where you can see the cell.zero_state recommended - "use the zero_state function on the cell object". Another reason why you may want cell.zero_state is because it is agnostic of the type of the cell (LSTM, GRU, RNN) and you can do something like this:
```
if type == 'GRU':
   cell = BasicGRUCell
else:
   cell = BasicLSTMCell(state_is_tuple=True)

init_state = cell.zero_state(batch_size)
```
with the initial state being set up OK. LSTMStateTuple will work only on cells that have the state as a tuple. When to use LSTMStateTuple? You'll want to use LSTMStateTuple when you're initializing your state with custom values (passed by the trainer). cell.zero_state() will return the state with all the values equal to 0.0. If you want to keep state between batches than you'll have to get it after each batch and add it to your feed_dict the next batch. See this for an explanation on why LSTM state is a tuple.


Question:
Validation and Test with TensorFlow I have created a one hidden layer neural network with a pyramidal architecture using TensorFlow. Here is the code:
```
num_classes = 10
image_size = 28

#Read the data
train_dataset, train_labels, valid_dataset, valid_labels, test_dataset, test_labels = OpenDataSets("...")
#Create and convert what is needed.
tf_train_dataset = tf.placeholder(tf.float32, shape=(batch_size, image_size * image_size))
tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
tf_valid_dataset = tf.constant(valid_dataset)
tf_test_dataset = tf.constant(test_dataset)

#Then I create the NN.
Wh = tf.Variable(tf.truncated_normal([image_size * image_size, image_size * image_size / 2]))
bh = tf.Variable(tf.truncated_normal([image_size * image_size / 2]))
hidden = tf.nn.relu(tf.matmul(tf_train_dataset, Wh) + bh)

Wout = tf.Variable(tf.truncated_normal([image_size * image_size / 2, num_labels]))
bout = tf.Variable(tf.truncated_normal([num_labels]))
logits = tf.nn.relu(tf.matmul(hidden, Wout) + bout)

loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels))
optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)
train_prediction = tf.nn.softmax(logits)
```
And now I train my NN:
```
with tf.Session(graph=graph) as session:
    tf.initialize_all_variables().run()
    for step in range(1000):
        offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]
        feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
        _, l, predictions = session.run([optimizer, loss, train_prediction], feed_dict=feed_dict)
```
Now I would like to validate and test my NN after training. But I don't know how to create the new feed_dict and use session.run in order to validate/test. Thanks for your help!

Response:
You will first have to create appropriate validation/test tensor functions. For one-layer MPL, it involves nested multiply with weights and addition of biases (and also Relu's since you have them in your original model). Define these right below your train predictions
```
valid_prediction = tf.nn.softmax(
                      tf.nn.relu(tf.matmul(
                         tf.nn.relu(tf.matmul(tf_valid_dataset, Wh) + bh)), Wout) + bout)))
test_prediction = tf.nn.softmax(
                      tf.nn.relu(tf.matmul(
                         tf.nn.relu(tf.matmul(tf_test_dataset, Wh) + bh)), Wout) + bout)))
```
These expressions are in fact quite identical to logit variable defined in your code, only using tf_valid_dataset and tf_test_dataset respectively. You can create intermediate variables to simplify them. You will then have to create some validation/test function to test accuracy. Simplest would be to test most likely predicted class (Misclassification error roughly). Define this outside your graph/session.
```
def accuracy(predictions, labels):
      pred_class = np.argmax(predictions, 1)
      true_class = np.argmax(labels, 1)
      return (100.0 * np.sum(pred_class == true_class) / predictions.shape[0])
```
After that, you can simply pass this accuracy function inside same session/feed_dict to compute validation/test score.
```
print 'Validation accuracy: %.1f%%' % accuracy(valid_prediction.eval(), valid_labels)
print 'Test accuracy: %.1f%%' % accuracy(test_prediction.eval(), test_labels)
```


Question:
How to use tf.data.Dataset.apply() for reshaping the dataset I am working with time series models in tensorflow. My dataset contains physics signals. I need to divide this signals into windows as give this sliced windows as input to my model. Here is how I am reading the data and slicing it:
```py
import tensorflow as tf
import numpy as np
def _ds_slicer(data):
    win_len = 768
    return {"mix":(tf.stack(tf.split(data["mix"],win_len))),
            "pure":(tf.stack(tf.split(data["pure"],win_len)))}
dataset = tf.data.Dataset.from_tensor_slices({
    "mix" : np.random.uniform(0,1,[1000,24576]),
    "pure" : np.random.uniform(0,1,[1000,24576])
})
dataset = dataset.map(_ds_slicer)
print dataset.output_shapes
# {'mix': TensorShape([Dimension(768), Dimension(32)]), 'pure': TensorShape([Dimension(768), Dimension(32)])}
```
I want to reshape this dataset to # {'mix': TensorShape([Dimension(32)]), 'pure': TensorShape([Dimension(32))} Equivalent transformation in numpy would be something like following:
```py
signal  = np.random.uniform(0,1,[1000,24576])
sliced_sig = np.stack(np.split(signal,768,axis=1),axis=1)
print sliced_sig.shape #(1000, 768, 32)
sliced_sig=sliced_sig.reshape(-1, sliced_sig.shape[-1])
print sliced_sig.shape #(768000, 32)
```
I thought of using tf.contrib.data.group_by_window as an input to dataset.apply() but couldn't figure out exactly how to use it. Is there a way I can use any custom transformation to reshape the dataset?

Response:
I think you're just looking for the transformation tf.contrib.data.unbatch. This does exactly what you want:
```py
x = np.zeros((1000, 768, 32))

dataset = tf.data.Dataset.from_tensor_slices(x)
print(dataset.output_shapes)  # (768, 32)
dataset = dataset.apply(tf.contrib.data.unbatch())
print(dataset.output_shapes)  # (32,)
```
From the documentation: If elements of the dataset are shaped [B, a0, a1, ...], where B may vary from element to element, then for each element in the dataset, the unbatched dataset will contain B consecutive elements of shape [a0, a1, ...]. Edit for TF 2.0 (Thanks @DavidParks) From TF 2.0, you can use directly tf.data.Dataset.unbatch:
```py
x = np.zeros((1000, 768, 32))

dataset = tf.data.Dataset.from_tensor_slices(x)
print(dataset.output_shapes)  # (768, 32)
dataset = dataset.unbatch()
print(dataset.output_shapes)  # (32,)
```


Question: Bounding boxes using tensorflow and inception-v3 Is it possible to have bounding boxes prediction using TensorFlow? I found TensorBox on github but I'm looking for a better supported or maybe official way to address this problem. I need to retrain the model for my own classes.

Response: It is unclear what exactly do you mean. Do you need object detection? I assume it from the 'bounding boxes'. If so, inception networks are not directly applicable for your task, they are classification networks. You should look for object detection models, like Single Shot Detector (SSD) or You Only Look Once (YOLO). They often use pre-trained convolutional layers from classification networks, but have additional layers on the top of it. If you want Inception (aka GoogLeNet), YOLO is based on that. Take a look at this implementation: https://github.com/thtrieu/darkflow or any other you can find in Google.


Question:
How do I use Batch Normalization during test time in Keras? I am currently trying to implement a model using Batch Normalization in Keras. I have succesfully implemented it for the training phase. However, for testing, Batch Normalization calculates the statistics (mean and variance) of the entire population before doing a forward pass through the network (the BN mean and variance are pre-calculated, and then kept static; this is in contrast to the training phase, where the mean and variance are determined by the batch). My question regarding Keras is: Assume (X, y) is the entire population. Assume (X_batch, y_batch) is a batch (a subset of that entire population) If I use the
```
model.test_on_batch(X_batch, y_batch)
```
how can I pass on to the batch-normalization layer the mean and variance of the entire population for X and y? Is there any way I can let keras handle this automatically?

Response: how can I pass on to the batch-normalization layer the mean and variance of the entire population for X and y? Is there any way I can let keras handle this automatically? Keras should just do it (in sufficiently recent versions): https://github.com/fchollet/keras/issues/81 To double-check, you may want to try batch_size=1 at test/prediction time, and if Keras fails to use the global statistics, you'll probably see very bad results.


Question: Use Tensorflow Object Detection API to detect small objects in images I'd like to use the Tensorflow Object Detection API to identify objects in a series of webcam images. The Faster RCNN models pre-trained on the COCO dataset appear to be suitable, as they contain all the object categories I need. However, I'd like to improve the performance of the model at identifying fairly small objects within each image. If I understand correctly, I need to edit the anchor scales parameter in the config file to get the model to use smaller bounding boxes. My questions are: Do I need to re-train the model on the entire COCO dataset after I adjust this parameter? Or is there a way to change the model just for inference and avoid any re-training? Are there any other tips/tricks to successfully identifying small objects, short of cropping the image into sections and running inference on each one separately? Background info I'm currently feeding 1280x720 images to the model. At around 200x150 pixels I'm finding it harder to detect objects.

Response: You'll need to retrain completely unfortunately, since the weights do depend on the shape of the anchor. Having a feature map with higher resolution should help (but slow down the process), so changing the feature extractor to get one with less input size reduction (max poolings with stride >1 is usually what reduces the space size) or upscaling the image a bit in the initial image resizer.


Question: CTC Loss InvalidArgumentError: sequence_length(b) <= time I am running into this error while trying to use tf.nn.ctc_loss through keras (ctc_batch_cost): InvalidArgumentError (see above for traceback): sequence_length(4) <= 471 According to the documentation for tf.nn.ctc_loss, Input requirements are: sequence_length(b) <= time for all b max(labels.indices(labels.indices[:, 1] == b, 2)) <= sequence_length(b) for all b. I am having a hard time understanding what this means-- what is b and what is sequence_length(b)?

Response: In this case b is each example in a minibatch. sequence_length(b) is the number of time stamps you have for that example. This is specified in the sequence_length argument passed to tf.nn.ctc_loss which is a 1-d tensor of sequence lengths.


Question: Visualizing attention activation in Tensorflow Is there a way to visualize the attention weights on some input like the figure in the link above(from Bahdanau et al., 2014), in TensorFlow's seq2seq models? I have found TensorFlow's github issue regarding this, but I couldn't find out how to fetch the attention mask during the session.

Response:
I also want to visualize the attention weights of Tensorflow seq2seq ops for my text summarization task. And I think the temporary solution is to use session.run() to evaluate the attention mask tensor as mentioned above. Interestingly, the original seq2seq.py ops is considered legacy version and can’t be found in github easily so I just used the seq2seq.py file in the 0.12.0 wheel distribution and modified it. To draw the heatmap, I used the 'Matplotlib' package, which is very convenient. The final output of attention visualization for news headline textsum looks like this: I modified the code as below: https://github.com/rockingdingo/deepnlp/tree/master/deepnlp/textsum#attention-visualization seq2seq_attn.py
```
# Find the attention mask tensor in function attention_decoder()-> attention()
# Add the attention mask tensor to ‘return’ statement of all the function that calls the attention_decoder(),
# all the way up to model_with_buckets() function, which is the final function I use for bucket training.

def attention(query):
  """Put attention masks on hidden using hidden_features and query."""
  ds = []  # Results of attention reads will be stored here.

  # some code

  for a in xrange(num_heads):
    with variable_scope.variable_scope("Attention_%d" % a):
      # some code

      s = math_ops.reduce_sum(v[a] * math_ops.tanh(hidden_features[a] + y),
                              [2, 3])
      # This is the attention mask tensor we want to extract
      a = nn_ops.softmax(s)

      # some code

  # add 'a' to return function
  return ds, a
```
seq2seq_model_attn.py
```
# modified model.step() function and return masks tensor
self.outputs, self.losses, self.attn_masks = seq2seq_attn.model_with_buckets(…)

# use session.run() to evaluate attn masks
attn_out = session.run(self.attn_masks[bucket_id], input_feed)
attn_matrix = ...
```
predict_attn.py and eval.py
```
# Use the plot_attention function in eval.py to visual the 2D ndarray during prediction.

eval.plot_attention(attn_matrix[0:ty_cut, 0:tx_cut], X_label = X_label, Y_label = Y_label)
```
And probably in the future tensorflow will have better way to extract and visualize the attention weight map. Any thoughts?


Question:
Multivariate Regression Neural Network Loss Function I am doing multivariate regression with a fully connected multilayer neural network in Tensorflow. The network predicts 2 continuous float variables (y1,y2) given an input vector (x1,x2,...xN), i.e. the network has 2 output nodes. With 2 outputs the network does not seem to converge. My loss function is essentially the L2 distance between the prediction and truth vectors (each contains 2 scalars):
```
loss = tf.nn.l2_loss(tf.sub(prediction, truthValues_placeholder)) + L2regularizationLoss
```
I am using L2 regularization, dropout regularization, and my activation functions are tanh. My questions: Is L2 distance the proper way to calculate loss for a multivariate network output? Are there some tricks needed to get multivariate regression networks to converge (as opposed to single-variable networks and classifiers)?

Response: Yes, you can use L2 distance for multivariate regression. But I would recommended experimenting with an absolute L1 distance as well. One of the problems with L2 is its susceptibility to outliers, and with L1 is the non-smooth nature at the origin. You can fix both of these issues using the Huber Loss, which acts like L2 near the origin, and like absolute L1 as you move away from the origin.


Question:
tf.data.Dataset: The `batch_size` argument must not be specified for the given input type I'm using Talos and Google colab TPU to run hyperparameter tuning of a Keras model. Note that I'm using Tensorflow 1.15.0 and Keras 2.2.4-tf.
```py
import os
import tensorflow as tf
import talos as ta
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.optimizers import Adam
from sklearn.model_selection import train_test_split

def iris_model(x_train, y_train, x_val, y_val, params):

    # Specify a distributed strategy to use TPU
    resolver = tf.contrib.cluster_resolver.TPUClusterResolver(tpu='grpc://' + os.environ['COLAB_TPU_ADDR'])
    tf.contrib.distribute.initialize_tpu_system(resolver)
    strategy = tf.contrib.distribute.TPUStrategy(resolver)

    # Use the strategy to create and compile a Keras model
    with strategy.scope():
      model = Sequential()
      model.add(Dense(32, input_shape=(4,), activation=tf.nn.relu, name="relu"))
      model.add(Dense(3, activation=tf.nn.softmax, name="softmax"))
      model.compile(optimizer=Adam(learning_rate=0.1), loss=params['losses'])

    # Convert data type to use TPU
    x_train = x_train.astype('float32')
    x_val = x_val.astype('float32')

    dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
    dataset = dataset.cache()
    dataset = dataset.shuffle(1000, reshuffle_each_iteration=True).repeat()
    dataset = dataset.batch(params['batch_size'], drop_remainder=True)

    # Fit the Keras model on the dataset
    out = model.fit(dataset, batch_size=params['batch_size'], epochs=params['epochs'], validation_data=[x_val, y_val], verbose=0, steps_per_epoch=2)

    return out, model

# Load dataset
X, y = ta.templates.datasets.iris()

# Train and test set
x_train, x_val, y_train, y_val = train_test_split(X, y, test_size=0.30, shuffle=False)

# Create a hyperparameter distributions
p = {'losses': ['logcosh'], 'batch_size': [128, 256, 384, 512, 1024], 'epochs': [10, 20]}

# Use Talos to scan the best hyperparameters of the Keras model
scan_object = ta.Scan(x_train, y_train, params=p, model=iris_model, experiment_name='test', x_val=x_val, y_val=y_val, fraction_limit=0.1)
```
After converting the train set to a Dataset using tf.data.Dataset, I get the following error when fitting the model with out = model.fit:
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-c812209b95d0> in <module>()
      8
      9 # Use Talos to scan the best hyperparameters of the Keras model
---> 10 scan_object = ta.Scan(x_train, y_train, params=p, model=iris_model, experiment_name='test', x_val=x_val, y_val=y_val, fraction_limit=0.1)

8 frames
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py in _validate_or_infer_batch_size(self, batch_size, steps, x)
   1813             'The `batch_size` argument must not be specified for the given '
   1814             'input type. Received input: {}, batch_size: {}'.format(
-> 1815                 x, batch_size))
   1816       return
   1817

ValueError: The `batch_size` argument must not be specified for the given input type. Received input: <DatasetV1Adapter shapes: ((512, 4), (512, 3)), types: (tf.float32, tf.float32)>, batch_size: 512
```
Then, if I follow those instructions and don't set the batch-size argument to model.fit. I get another error in:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-c812209b95d0> in <module>()
      8
      9 # Use Talos to scan the best hyperparameters of the Keras model
---> 10 scan_object = ta.Scan(x_train, y_train, params=p, model=iris_model, experiment_name='test', x_val=x_val, y_val=y_val, fraction_limit=0.1)

8 frames
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py in _distribution_standardize_user_data(self, x, y, sample_weight, class_weight, batch_size, validation_split, shuffle, epochs, allow_partial_batch)
   2307             strategy) and not drop_remainder:
   2308           dataset_size = first_x_value.shape[0]
-> 2309           if dataset_size % batch_size == 0:
   2310             drop_remainder = True
   2311

TypeError: unsupported operand type(s) for %: 'int' and 'NoneType'
```

Response:
It looks to me that the problem with your code is that the training and validation data is not in the same format. You are batching the training data but not the validation examples. You can ensure that they are in the same format by replacing the bottom half of your iris_model function with this:
```
def fix_data(x, y):
    x = x.astype('float32')
    ds = Dataset.from_tensor_slices((x, y))
    ds = ds.cache()
    ds = ds.shuffle(1000, reshuffle_each_iteration = True)
    ds = ds.repeat()
    ds = ds.batch(params['batch_size'], drop_remainder = True)
    return ds
train = fix_data(x_train, y_train)
val = fix_data(x_val, y_val)

# Fit the Keras model on the dataset
out = model.fit(x = train, epochs = params['epochs'],
                steps_per_epoch = 2,
                validation_data = val,
                validation_steps = 2)
```
At least this works for me and your code runs without error.


Question:
Efficiently Finding Closest Word In TensorFlow Embedding Recently, I've been trying to find the closest word to an embedding. The two most notable ways of doing this is by cosine distance or euclidean distance. I'm trying to find how to efficiently compute the cosine distance for a tensor of shape [batch_size x embedding_size] One approach is to unpack the tensor and the compute the cosine distance
```
#embedding is shape [vocab_size x embedding size]
  array_list = tf.unpack(batch_array)
  word_class_list = tf.unpack(embedding)
  index_list_of_closest_word = []
  for eacharray in array_list:
    list_of_distances = []
    for eachwordclass in word_class_list:
      list_of_distances.append(cosine_distance(eacharray, eachwordclass))
    index_list_of_closest_word.append(tf.argmax(tf.pack(list_of_distances)))
```
However, this approach is terribly inefficient. Is there perhaps a more efficient manner to do this? I know word2vec does this pretty fast and tensorflow, with the power of a gpu, should be able to do these batch calculations in parallel. Thanks!

Response:
The cosine similarity formula is: The inputs you have are: embedding: the embedding matrix, of shape [vocab_size, embedding_size] batch_array: a batch of embeddings, to which you want to find the closest words, of shape [batch_size, embedding_size]
```py
embedding = tf.placeholder(tf.float32, [vocab_size, embedding_size])
batch_array = tf.placeholder(tf.float32, [batch_size, embedding_size])
```
To compute the cosine similarity, you can first L2 normalize both inputs: (you may want to store the normed embedding, as you are going to reuse it a lot)
```py
normed_embedding = tf.nn.l2_normalize(embedding, dim=1)
normed_array = tf.nn.l2_normalize(batch_array, dim=1)
```
Then you have to compute the dot products of all words (vocab_size in total) vs. all arrays from the batch (batch_size in total):
```py
cosine_similarity = tf.matmul(normed_array, tf.transpose(normed_embedding, [1, 0]))
```
You can finally compute the argmax for each element of the batch:
```py
closest_words = tf.argmax(cosine_similarity, 1)  # shape [batch_size], type int64
```


Question:
Is there a way to check if mxnet uses my gpu? Can I see what are the available GPUs with mxnet? Is there something similar for TensorFlow's
```
tf.test.gpu_device_name()
```
in mxnet?

Response:
The definitive way to check if your GPU is being utilized is by using nvidia-smi command. My favorite arguments are:
```
nvidia-smi --query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used --format=csv -l 1
```
If you just want to test if gpu support is available (which is what tf.test.gpu_device_name()) does, the following function can help:
```
import mxnet as mx
def gpu_device(gpu_number=0):
    try:
        _ = mx.nd.array([1, 2, 3], ctx=mx.gpu(gpu_number))
    except mx.MXNetError:
        return None
    return mx.gpu(gpu_number)
```
This function returns None if the requested gpu device isn't available, or returns the relevant context if gpu device is available. You can also use this function to check if there is any support for GPU on this system:
```
if not gpu_device():
    print('No GPU device found!')
```


Question:
Could not satisfy explicit device specification '/device:GPU:0' because no devices matching I want to use TensorFlow 0.12 for GPU on my Ubuntu 14.04 machine. But when assigning a device to a node I am getting the following error.
```
InvalidArgumentError (see above for traceback): Cannot assign a device to
node 'my_model/RNN/zeros': Could not satisfy explicit device specification
'/device:GPU:0' because no devices matching that specification are registered
in this process; available devices: /job:localhost/replica:0/task:0/cpu:0
    [[Node: my_model/RNN/zeros = Fill[T=DT_FLOAT, _device="/device:GPU:0"]
(my_model/RNN/pack, my_model/RNN/zeros/Const)]]
```
My tensorflow seems to be set up correctly, since this simple program works:
```
import tensorflow as tf
# Creates a graph.
with tf.device('/gpu:0'):
  a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
  b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
  c = tf.matmul(a, b)
# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
# Runs the op.
print(sess.run(c))
```
Which outputs:
```
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA
library libcublas.so locally I tensorflow/stream_executor/dso_loader.cc:128]
successfully opened CUDA library libcudnn.so locally I
tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library
 libcufft.so locally I tensorflow/stream_executor/dso_loader.cc:128]
successfully opened CUDA library libcuda.so.1 locally I
tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library
 libcurand.so locally I tensorflow/core/common_runtime/gpu/gpu_device.cc:885]
Found device 0 with properties:  name: Tesla K40m major: 3 minor: 5
memoryClockRate (GHz) 0.745 pciBusID 0000:08:00.0 Total memory: 11.17GiB Free
memory:

    11.10GiB I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0  I
 tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y  I
tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow
device (/gpu:0) -> (device: 0, name: Tesla K40m, pci bus id: 0000:08:00.0)
Device mapping: /job:localhost/replica:0/task:0/gpu:0 -> device: 0, name:
Tesla K40m, pci bus id: 0000:08:00.0 I tensorflow/core/common_runtime
/direct_session.cc:255] Device mapping: /job:localhost/replica:0/task:0/gpu:0
-> device: 0, name: Tesla K40m, pci bus id: 0000:08:00.0

    MatMul: (MatMul): /job:localhost/replica:0/task:0/gpu:0 I tensorflow/core
/common_runtime/simple_placer.cc:827] MatMul: (MatMul)/job:localhost/replica:0
/task:0/gpu:0 b: (Const): /job:localhost/replica:0/task:0/gpu:0 I
tensorflow/core/common_runtime/simple_placer.cc:827] b: (Const)/job:localhost
/replica:0/task:0/gpu:0 a: (Const): /job:localhost/replica:0/task:0/gpu:0 I
tensorflow/core/common_runtime/simple_placer.cc:827] a: (Const)/job:localhost
/replica:0/task:0/gpu:0 [[ 22.  28.]  [ 49.
    64.]]
```
How can I assign a device to a node correctly?

Response: Try using sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)). This will resolve the problem if it couldn't place an operation on the GPU. Since some operations have only CPU implementation. Using allow_soft_placement=True will allow TensorFlow to fall back to CPU when no GPU implementation is available.


Question:
Do not use tf.reset_default_graph() to clear nested graphs I have a bunch of functions, which create portions of computation graph. In some of such functions I do
```
with tf.name_scope("my_scope_name"):
        self._eye_n_components = tf.eye(se...
```
At the beginning of topmost function I call
```
tf.reset_default_graph()
```
and then call those partial functions and also they can call each other. Unfortunately, I get an error Error: Do not use tf.reset_default_graph() to clear nested graphs. If you need a cleared graph, exit the nesting and create a new graph. Several questions. 1) What is nesting and how to "exit nesting"? 2) How to create new graph? 3) How to catch, where I am entering the nesting? 4) How to clear entire graph so that tensorflow does not think I am trying to clear nested one?

Response: This error message is displayed when you call tf.reset_default_graph() in one of the following scenarios: Inside a with graph.as_default(): block. Inside a with tf.Session(): block. Between creating a tf.InteractiveSession and calling sess.close(). Each of these scenarios involves registering a default (and potentially "nested") tf.Graph object, which will be unregistered when you exit the block (or close the tf.InteractiveSession). Resetting the default graph in those scenarios would leave the system in an inconsistent state, so you should ensure to exit the block (or close the tf.InteractiveSession) before calling tf.reset_default_graph().


Question:
Create an int list feature to save as tfrecord in tensorflow? How can I create a tensorflow record from a list? From the documentation here it seems possible. There's also this example where they convert a numpy array into a byte array using the .tostring() from numpy. However when I try to pass in:
```
labels = np.asarray([[1,2,3],[4,5,6]])
...
example = tf.train.Example(features=tf.train.Features(feature={
    'height': _int64_feature(rows),
    'width': _int64_feature(cols),
    'depth': _int64_feature(depth),
    'label': _int64_feature(labels[index]),
    'image_raw': _bytes_feature(image_raw)}))
writer.write(example.SerializeToString())
```
I get the error:
```
TypeError: array([1, 2, 3]) has type type 'numpy.ndarray', but expected one of: (type 'int', type 'long')
```
Which doesn't help me to figure out how to store a list of integers into the tfrecord. I've tried looking through the docs.

Response:
After a while of messing around with it and looking further in the documentation I found my own answer. In the above function using the example code as a base:
```
def _int64_feature(value):
  return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))
...
'label': _int64_feature(labels[index]),
```
labels[index] is being cast to a list as [value] so you have [np.array([1,2,3])] which causes the error. The above cast was necessary in the example because tf.train.Int64List() expects either a list or numpy array and the example was passing in a single integer so they typecasted it to a list as so. In the example it was like this
```
label = [1,2,3,4]
...
'label': _int64_feature(label[index])

tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))
#Where value = [1] in this case
```
If you want to pass in a list do this
```
labels = np.asarray([[1,2,3],[4,5,6]])
...
def _int64_feature(value):
  return tf.train.Feature(int64_list=tf.train.Int64List(value=value))
...
'label': _int64_feature(labels[index]),
```
I'll probably do a pull request because I found the original documentation for tf.train.Feature to be almost non-existent. TL;DR Pass either a list or numpy array to tf.train.Int64List() but not a list of lists or list of numpy arrays.


Question: TensorBoard could not bind to port 6006, it was already in use I already closed the Tensorboard before, but after I open new terminal and re-run the command tensorboard --logdir /tmp/retrain_logs, it produces following error: E0128 12:24:08.469702 MainThread program.py:201] TensorBoard could not bind to port 6006, it was already in use E0128 12:24:08.469702 140707008960320 program.py:201] TensorBoard could not bind to port 6006, it was already in use ERROR: TensorBoard could not bind to port 6006, it was already in use Anybody knows the how could this happen? I can easily restart the PC, but now I'm in the middle of training my dataset (unfortunately I haven't set checkpoint) so I don't want to terminate it.

Response:
Sometimes, the port continues to be occupied in spite of ending the instance. In that case, you can first check if your port is still occupied by:
```
lsof -i:6006
```
and then, using the PID listed above to kill the tcp instance:
```
kill -9 PID
```


Question:
Minimize a function of one variable in Tensorflow I am new to Tensorflow and was wondering whether it would be possible to minimize a function of one variable using Tensorflow. For example, can we use Tensorflow to minimize 2*x^2 - 5^x + 4 using an initial guess (say x = 1)? I am trying the following:
```
import tensorflow as tf
import numpy as np

X = tf.placeholder(tf.float32, shape = ())
xvar = tf.Variable(np.random.randn())
f = 2*mul(X,X) - 5*X + 4

opt = tf.train.GradientDescentOptimizer(0.5).minimize(f)

with tf.Session() as sess:
    tf.global_variables_initializer().run()
    y = sess.run(opt, feed_dict = {X : 5.0}) #initial guess = 5.0
    print(y)
```
But this gives the following error:
```
ValueError: No gradients provided for any variable, check your graph for ops that do not support gradients, between variables
```
Please help me understand what am I doing wrong here.

Response:
If you want to minimize a single parameter you could do the following (I've avoided using a placeholder since you are trying to train a parameter - placeholders are often used for hyper-parameters and input and aren't considered trainable parameters):
```
import tensorflow as tf

x = tf.Variable(10.0, trainable=True)
f_x = 2 * x* x - 5 *x + 4

loss = f_x
opt = tf.train.GradientDescentOptimizer(0.1).minimize(f_x)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for i in range(100):
        print(sess.run([x,loss]))
        sess.run(opt)
```
This will output the following list of pairs (x,loss):
```
[10.0, 154.0]
[6.5, 56.0]
[4.4000001, 20.720001]
[3.1400001, 8.0192013]
[2.3840001, 3.4469128]
[1.9304, 1.8008881]
[1.65824, 1.2083197]
[1.494944, 0.99499512]
[1.3969663, 0.91819811]
[1.3381798, 0.89055157]
[1.3029079, 0.88059855]
[1.2817447, 0.87701511]
[1.2690468, 0.87572551]
[1.2614281, 0.87526155]
[1.2568569, 0.87509394]
[1.2541142, 0.87503386]
[1.2524685, 0.87501216]
[1.2514811, 0.87500429]
[1.2508886, 0.87500143]
[1.2505331, 0.87500048]
[1.2503198, 0.875]
[1.2501919, 0.87500024]
[1.2501152, 0.87499976]
[1.2500691, 0.875]
[1.2500415, 0.875]
[1.2500249, 0.87500024]
[1.2500149, 0.87500024]
[1.2500089, 0.875]
[1.2500054, 0.87500024]
[1.2500032, 0.875]
[1.2500019, 0.875]
[1.2500012, 0.87500024]
[1.2500007, 0.87499976]
[1.2500005, 0.875]
[1.2500002, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
[1.2500001, 0.87500024]
```


Question:
Tensorflow 2.0 doesn't compute the gradient I want to visualize the patterns that a given feature map in a CNN has learned (in this example I'm using vgg16). To do so I create a random image, feed through the network up to the desired convolutional layer, choose the feature map and find the gradients with the respect to the input. The idea is to change the input in such a way that will maximize the activation of the desired feature map. Using tensorflow 2.0 I have a GradientTape that follows the function and then computes the gradient, however the gradient returns None, why is it unable to compute the gradient?
```
import tensorflow as tf
import matplotlib.pyplot as plt
import time
import numpy as np
from tensorflow.keras.applications import vgg16

class maxFeatureMap():

    def __init__(self, model):

        self.model = model
        self.optimizer = tf.keras.optimizers.Adam()

    def getNumLayers(self, layer_name):

        for layer in self.model.layers:
            if layer.name == layer_name:
                weights = layer.get_weights()
                num = weights[1].shape[0]
        return ("There are {} feature maps in {}".format(num, layer_name))

    def getGradient(self, layer, feature_map):

        pic = vgg16.preprocess_input(np.random.uniform(size=(1,96,96,3))) ## Creates values between 0 and 1
        pic = tf.convert_to_tensor(pic)

        model = tf.keras.Model(inputs=self.model.inputs,
                               outputs=self.model.layers[layer].output)
        with tf.GradientTape() as tape:
            ## predicts the output of the model and only chooses the feature_map indicated
            predictions = model.predict(pic, steps=1)[0][:,:,feature_map]
            loss = tf.reduce_mean(predictions)
        print(loss)
        gradients = tape.gradient(loss, pic[0])
        print(gradients)
        self.optimizer.apply_gradients(zip(gradients, pic))

model = vgg16.VGG16(weights='imagenet', include_top=False)

x = maxFeatureMap(model)
x.getGradient(1, 24)
```

Response: This is a common pitfall with GradientTape; the tape only traces tensors that are set to be "watched" and by default tapes will watch only trainable variables (meaning tf.Variable objects created with trainable=True). To watch the pic tensor, you should add tape.watch(pic) as the very first line inside the tape context. Also, I'm not sure if the indexing (pic[0]) will work, so you might want to remove that -- since pic has just one entry in the first dimension it shouldn't matter anyway. Furthermore, you cannot use model.predict because this returns a numpy array, which basically "destroys" the computation graph chain so gradients won't be backpropagated. You should simply use the model as a callable, i.e. predictions = model(pic).


Question:
TensorFlow in_top_k evaluation input argumants I am following the tutorial in this link and trying to change the evaluation method for the model (at the bottom). I would like to get a top-5 evaluation and I'm trying to use to following code:
```
topFiver=tf.nn.in_top_k(y, y_, 5, name=None)
```
However, this yields the following error:
```
File "AlexNet.py", line 111, in <module>
    topFiver = tf.nn.in_top_k(pred, y, 5, name=None)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 346, in in_top_k
    targets=targets, k=k, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 486, in apply_op
    _Attr(op_def, input_arg.type_attr))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 59, in _SatisfiesTypeConstraint
    ", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
TypeError: DataType float32 for attr 'T' not in list of allowed values: int32, int64
```
As far as I can tell, the problem is that tf.nn.in_top_k() only works for tf.int32 or tf.int64 data, but my data is in tf.float32 format. Is there any workaround for this?

Response:
The targets argument to tf.nn.in_top_k(predictions, targets, k) must be a vector of class IDs (i.e. indices of columns in the predictions matrix). This means that it only works for single-class classification problems. If your problem is a single-class problem, then I assume that your y_ tensor is a one-hot encoding of the true labels for your examples (for example because you also pass them to an op like tf.nn.softmax_cross_entropy_with_logits(). In that case, you have two options: If the labels were originally stored as integer labels, pass them directly to tf.nn.in_top_k() without converting them to one-hot. (Also, consider using tf.nn.sparse_softmax_cross_entropy_with_logits() as your loss function, because it may be more efficient.) If the labels were originally stored in the one-hot format, you can convert them to integers using tf.argmax():
```
labels = tf.argmax(y_, 1)
topFiver = tf.nn.in_top_k(y, labels, 5)
```


Question: How can I get the number of CUDA cores in my GPU using Python and Numba? I would like to know how to obtain the total number of CUDA Cores in my GPU using Python, Numba and cudatoolkit.

Response:
Most of what you need can be found by combining the information in this answer along with the information in this answer. We'll use the first answer to indicate how to get the device compute capability and also the number of streaming multiprocessors. We'll use the second answer (converted to python) to use the compute capability to get the "core" count per SM, then multiply that by the number of SMs. Here is a full example:
```
$ cat t36.py
from numba import cuda

cc_cores_per_SM_dict = {
    (2,0) : 32,
    (2,1) : 48,
    (3,0) : 192,
    (3,5) : 192,
    (3,7) : 192,
    (5,0) : 128,
    (5,2) : 128,
    (6,0) : 64,
    (6,1) : 128,
    (7,0) : 64,
    (7,5) : 64,
    (8,0) : 64,
    (8,6) : 128,
    (8,9) : 128,
    (9,0) : 128,
    (10,0) : 128,
    (12,0) : 128
    }
# the above dictionary should result in a value of "None" if a cc match
# is not found.  The dictionary needs to be extended as new devices become
# available, and currently does not account for all Jetson devices
device = cuda.get_current_device()
my_sms = getattr(device, 'MULTIPROCESSOR_COUNT')
my_cc = device.compute_capability
cores_per_sm = cc_cores_per_SM_dict.get(my_cc)
total_cores = cores_per_sm*my_sms
print("GPU compute capability: " , my_cc)
print("GPU total number of SMs: " , my_sms)
print("total cores: " , total_cores)

$ python t36.py
GPU compute capability:  (5, 2)
GPU total number of SMs:  8
total cores:  1024
$
```


Question: Batch normalization when batch size=1 What will happen when I use batch normalization but set batch_size = 1? Because I am using 3D medical images as training dataset, the batch size can only be set to 1 because of GPU limitation. Normally, I know, when batch_size = 1, variance will be 0. And (x-mean)/variance will lead to error because of division by 0. But why did errors not occur when I set batch_size = 1? Why my network was trained as good as I expected? Could anyone explain it? Some people argued that: The ZeroDivisionError may not be encountered because of two cases. First, the exception is caught in a try catch block. Second, a small rational number is added ( 1e-19 ) to the variance term so that it is never zero. But some people disagree. They said that: You should calculate mean and std across all pixels in the images of the batch. (So even batch_size = 1, there are still a lot of pixels in the batch. So the reason why batch_size=1 can still work is not because of 1e-19) I have checked the Pytorch source code, and from the code I think the latter one is right. Does anyone have different opinion???

Response:
variance will be 0 No, it won't; BatchNormalization computes statistics only with respect to a single axis (usually the channels axis, =-1 (last) by default); every other axis is collapsed, i.e. summed over for averaging; details below. More importantly, however, unless you can explicitly justify it, I advise against using BatchNormalization with batch_size=1; there are strong theoretical reasons against it, and multiple publications have shown BN performance degrade for batch_size under 32, and severely for <=8. In a nutshell, batch statistics "averaged" over a single sample vary greatly sample-to-sample (high variance), and BN mechanisms don't work as intended. Small mini-batch alternatives: Batch Renormalization -- Layer Normalization -- Weight Normalization Implementation details: from source code:
```py
reduction_axes = list(range(len(input_shape)))
del reduction_axes[self.axis]
```
Eventually, tf.nn.monents is called with axes=reduction_axes, which performs a reduce_sum to compute variance. Then, in the TensorFlow backend, mean and variance are passed to tf.nn.batch_normalization to return train- or inference-normalized inputs. In other words, if your input is (batch_size, height, width, depth, channels), or (1, height, width, depth, channels), then BN will run calculations over the 1, height, width, and depth dimensions. Can variance ever be zero? - yes, if every single datapoint for any given channel slice (along every dimension) is the same. But this should be near-impossible for real data. Other answers: first one is misleading: a small rational number is added (1e-19) to the variance This doesn't happen in computing variance, but it is added to variance when normalizing; nonetheless, it is rarely necessary, as variance is far from zero. Also, the epsilon term is actually defaulted to 1e-3 by Keras; it serves roles in regularizing, beyond mere avoiding zero-division. Update: I failed to address an important piece of intuition with suspecting variance to be 0; indeed, the batch statistics variance is zero, since there is only one statistic - but the "statistic" itself concerns the mean & variance of the channel + spatial dimensions. In other words, the variance of the mean & variance (of the single train sample) is zero, but the mean & variance themselves aren't.


Question:
Tensorflow while loop : dealing with lists
```
import tensorflow as tf

array = tf.Variable(tf.random_normal([10]))
i = tf.constant(0)
l = []

def cond(i,l):
   return i < 10

def body(i,l):
   temp = tf.gather(array,i)
   l.append(temp)
   return i+1,l

index,list_vals = tf.while_loop(cond, body, [i,l])
```
I want to process a tensor array in the similar way as described in the above code. In the body of the while loop I want to process the array by element by element basis to apply some function. For demonstration, I have given a small code snippet. However, it is giving an error message as follows.
```
ValueError: Number of inputs and outputs of body must match loop_vars: 1, 2
```
Any help in resolving this is appreciated. Thanks

Response:
Citing the documentation: loop_vars is a (possibly nested) tuple, namedtuple or list of tensors that is passed to both cond and body You cannot pass regular python array as a tensor. What you can do, is:
```
i = tf.constant(0)
l = tf.Variable([])

def body(i, l):
    temp = tf.gather(array,i)
    l = tf.concat([l, [temp]], 0)
    return i+1, l

index, list_vals = tf.while_loop(cond, body, [i, l],
                                 shape_invariants=[i.get_shape(),
                                                   tf.TensorShape([None])])
```
The shape invariants are there, because normally tf.while_loop expects the shapes of tensors inside while loop won't change.
```
sess = tf.Session()
sess.run(tf.global_variables_initializer())
sess.run(list_vals)
Out: array([-0.38367489, -1.76104736,  0.26266089, -2.74720812,  1.48196387,
            -0.23357525, -1.07429159, -1.79547787, -0.74316853,  0.15982138],
           dtype=float32)
```


Question:
What is the rule to know how many LSTM cells and how many units in each LSTM cell do you need in Keras? I know that a LSTM cell has a number of ANNs inside. But when defining the hidden layer for the same problem, I have seen some people using only 1 LSTM cell and others use 2, 3 LSTM cells like this -
```
model = Sequential()
model.add(LSTM(256, input_shape=(n_prev, 1), return_sequences=True))
model.add(Dropout(0.3))
model.add(LSTM(128, input_shape=(n_prev, 1), return_sequences=True))
model.add(Dropout(0.3))
model.add(LSTM(64, input_shape=(n_prev, 1), return_sequences=False))
model.add(Dropout(0.3))
model.add(Dense(1))
model.add(Activation('linear'))
```
Is there any rule as to how many LSTM cells you should take? Or its just manual experimenting? Another question following this is, how many units you should take in an LSTM cell. Like some people take 256, some take 64 for the same problem.

Response: There are no "rules", but there are guidelines; in practice, you'd experiment with depth vs. width, each of which works differently: RNN width is defined by (1) # of input channels; (2) # of cell's filters (output channels/units). As with CNN, each RNN filter is an independent feature extractor: more is suited for higher-complexity information, including but not limited to: dimensionality, modality, noise, frequency. RNN depth is defined by (1) # of stacked layers; (2) # of timesteps. Specifics will vary by architecture, but from information standpoint, unlike CNNs, RNNs are dense: every timestep influences the ultimate output of a layer, hence the ultimate output of the next layer - so it again isn't as simple as "more nonlinearity"; stacked RNNs exploit both spatial and temporal information. In general, width extracts more features, whereas depth extracts richer features - but if there aren't many features to extract from given data, width should be lessened - and the "simpler" the data/problem, the less layers are suitable. Ultimately, however, it may be best to spare extensive analysis and try different combinations of each -- see this SO for more info. Lastly, avoid Dropout and use LSTM(recurrent_dropout=...) instead (see linked SO).


Question: What is the difference between concatenate and add in keras? I would like to add skip connections for my inner layers of a fully convolutional network in keras, there is a keras.layers.Add option and there is a keras.layers.concatenate option. What is the difference? and which one I should use?

Response:
What is the difference? Add layer adds two input tensor while concatenate appends two tensors. You can refer this documentation for more info. Example:
```
import keras
import tensorflow as tf
import keras.backend as K

a = tf.constant([1,2,3])
b = tf.constant([4,5,6])

add = keras.layers.Add()
print(K.eval(add([a,b])))
#output: [5 7 9]

concat = keras.layers.Concatenate()
print(K.eval(concat([a,b])))
#output: array([1, 2, 3, 4, 5, 6], dtype=int32)
```
which one I should use? You can use Add for skip connections.


Question: How is the categorical_crossentropy implemented in keras? I'm trying to apply the concept of distillation, basically to train a new smaller network to do the same as the original one but with less computation. I have the softmax outputs for every sample instead of the logits. My question is, how is the categorical cross entropy loss function implemented? Like it takes the maximum value of the original labels and multiply it with the corresponded predicted value in the same index, or it does the summation all over the logits (One Hot encoding) as the formula says:

Response:
I see that you used the tensorflow tag, so I guess this is the backend you are using?
```
def categorical_crossentropy(output, target, from_logits=False):
"""Categorical crossentropy between an output tensor and a target tensor.
# Arguments
    output: A tensor resulting from a softmax
        (unless `from_logits` is True, in which
        case `output` is expected to be the logits).
    target: A tensor of the same shape as `output`.
    from_logits: Boolean, whether `output` is the
        result of a softmax, or is a tensor of logits.
# Returns
    Output tensor.
```
This code comes from the keras source code. Looking directly at the code should answer all your questions :) If you need more info just ask ! EDIT : Here is the code that interests you :
```
# Note: tf.nn.softmax_cross_entropy_with_logits
# expects logits, Keras expects probabilities.
if not from_logits:
    # scale preds so that the class probas of each sample sum to 1
    output /= tf.reduce_sum(output,
                            reduction_indices=len(output.get_shape()) - 1,
                            keep_dims=True)
    # manual computation of crossentropy
    epsilon = _to_tensor(_EPSILON, output.dtype.base_dtype)
    output = tf.clip_by_value(output, epsilon, 1. - epsilon)
    return - tf.reduce_sum(target * tf.log(output),
                          reduction_indices=len(output.get_shape()) - 1)
```
If you look at the return, they sum it... :)


Question: How to concatenate two tensors horizontally in TensorFlow? I have 2 tensors of shape (100, 4) and (100, 2). I would like to perform a concatenation operation, in TensorFlow, similar to np.hstack, in NumPy, so that the output would be of shape (100, 6). Is there a TensorFlow function to do that?

Response:
You can use tf.concat for this purpose as follows:
```
sess=tf.Session()
t1 = [[1, 2], [4, 5]]
t2 = [[7, 8, 9], [10, 11, 12]]
res=tf.concat(concat_dim=1,values=[t1, t2])
print(res.eval(session=sess))
```
This prints
```
[[ 1  2  7  8  9]
 [ 4  5 10 11 12]]
```


Question: Machine learning: how to identify if there is no object of trained classes in image if I train a deep neural network with (say) 10 classes and I feed the network a completely different image, is it reasonable to expect that the output layer's cells will not activate much, so I will know there is no object of the trained classes in the image? My intuition says "Yes", but is it so? And what would be the best approach to this? Thanks

Response: During the supervised training you usually assume that during training you get complete representation of the future types of objects. Typically - these are all labeled instances. In your case - there are also "noise" instances, thus there are basically two main approaches: As multi-class neural network has K output neurons, each representing the probability of being a member of a particular class, you can simply condition on these distribution to say that the new object does not belong to any of them. One particular approach is to check if min(p(y|x))<T (where p(y|x) is the activation of output neuron) for some threshold T. You can either set this value by hand, or through analysis of you "noise" instances (with you do have some for training). Simply pass them through your network and compare what value of T gives you best recognition rate Add another one-class classifier (anomaly detector) before your network - so you will end up with the sequence of two classifiers, first is able to recognize if it is a noise or element of any of your classes (notice, that this can be trained without access to noise samples, see one-class classification or anomaly detection techniques. You could also add another output to the network to represent noise, but this probably will not work well, as you will force your network to generate consistent internal representation for both noise vs data and inter-class decisions.


Question:
`return_sequences = False` equivalent in pytorch LSTM In tensorflow/keras, we can simply set return_sequences = False for the last LSTM layer before the classification/fully connected/activation (softmax/sigmoid) layer to get rid of the temporal dimension. In PyTorch, I don't find anything similar. For the classification task, I don't need a sequence to sequence model but many to one architecture like this: Here's my simple bi-LSTM model.
```
import torch
from torch import nn

class BiLSTMClassifier(nn.Module):
    def __init__(self):
        super(BiLSTMClassifier, self).__init__()
        self.embedding = torch.nn.Embedding(num_embeddings = 65000, embedding_dim = 64)
        self.bilstm = torch.nn.LSTM(input_size = 64, hidden_size = 8, num_layers = 2,
                                    batch_first = True, dropout = 0.2, bidirectional = True)
        # as we have 5 classes
        self.linear = nn.Linear(8*2*512, 5) # last dimension
    def forward(self, x):
        x = self.embedding(x)
        print(x.shape)
        x, _ = self.bilstm(x)
        print(x.shape)
        x = self.linear(x.reshape(x.shape[0], -1))
        print(x.shape)

# create our model

bilstmclassifier = BiLSTMClassifier()
```
If I observe the shapes after each layer,
```
xx = torch.tensor(X_encoded[0]).reshape(1,512)
print(xx.shape)
# torch.Size([1, 512])
bilstmclassifier(xx)
#torch.Size([1, 512, 64])
#torch.Size([1, 512, 16])
#torch.Size([1, 5])
```
What can I do so that the last LSTM returns a tensor with shape (1, 16) instead of (1, 512, 16)?

Response:
The simplest way to do this is by indexing into the tensor:
```
x = x[:, -1, :]
```
where x is the RNN output. Of course, if batch_first is False, one would have to use x[-1, :, :] (or just x[-1]) to index into the time axis instead. Turns out this is the same thing Tensorflow/Keras do. The relevant code can be found in K.rnn here:
```
last_output = tuple(o[-1] for o in outputs)
```
Note that the code at this point uses time_major data format, so the index is into the first axis. Also, outputs is a tuple because it can be multiple layers, state/cell pairs etc., but it is generally the sequence of outputs for all time steps. This is then used in the RNN class as follows:
```
if self.return_sequences:
    output = K.maybe_convert_to_ragged(is_ragged_input, outputs, row_lengths)
else:
    output = last_output
```
So in total, we can see that return_sequences=False just uses outputs[-1].


Question:
'tensorflow.python.framework.ops.EagerTensor' object has no attribute '_in_graph_mode' I am trying to visualize CNN filters by optimizing a random 'image' so that it produces a high mean activation on that filter which is somehow similar to the neural style transfer algorithm. For that purpose, I am using TensorFlow==2.2.0-rc. But during the optimization process, an error occurs saying 'tensorflow.python.framework.ops.EagerTensor' object has no attribute '_in_graph_mode'. I tried debugging it and it somehow works when I don't use the opt.apply_gradients() and instead, apply its gradient manually like img = img - lr * grads but I want to use the "Adam" optimizer rather than the simple SGD. Here is my code for the optimization part
```
opt = tf.optimizers.Adam(learning_rate=lr, decay = 1e-6)
for _ in range(epoch):
    with tf.GradientTape() as tape:
        tape.watch(img)
        y = model(img)[:, :, :, filter]
        loss = -tf.math.reduce_mean(y)
        grads = tape.gradient(loss, img)
        opt.apply_gradients(zip([grads], [img]))
```

Response:
The reason for the bug is that the tf.keras optimizers apply gradients to variable objects (of type tf.Variable), while you are trying to apply gradients to tensors (of type tf.Tensor). Tensor objects are not mutable in TensorFlow, thus the optimizer cannot apply gradients to it. You should initialize the variable img as a tf.Variable. This is how your code should be:
```py
# NOTE: The original image is lost here. If this is not desired, then you can
# rename the variable to something like img_var.
img = tf.Variable(img)
opt = tf.optimizers.Adam(learning_rate=lr, decay = 1e-6)

for _ in range(epoch):
    with tf.GradientTape() as tape:
        tape.watch(img)
        y = model(img.value())[:, :, :, filter]
        loss = -tf.math.reduce_mean(y)

    grads = tape.gradient(loss, img)
    opt.apply_gradients(zip([grads], [img]))
```
Also, it is recommended to calculate the gradients outside the tape's context. This is because keeping it in will lead to the tape tracking the gradient calculation itself, leading to higher memory usage. This is only desirable if you want to calculate higher-order gradients. Since you don't need those, I have kept them outside. Note I have changed the line y = model(img)[:, :, :, filter] to y = model(img.value())[:, :, :, filter]. This is because tf.keras models need tensors as input, not variables (bug, or feature?).


Question: What is the purpose of tf.compat? What's the purpose of tf.compat module? It looks like just the entire Tensorflow API is replicated inside this module. The documentation states Functions for Python 2 vs. 3 compatibility. So why there is a "v1" and a "v2" submodule? What are the compatibility problems address by tf.compat specifically?

Response:
tf.compat allows you to write code that works both in TensorFlow 1.x and 2.x. For example, the following piece of code:
```
import tensorflow as tf

tf.compat.v1.disable_v2_behavior()
with tf.compat.v1.Session() as sess:
    x = tf.compat.v1.placeholder(tf.float32, [2])
    x2 = tf.square(x)
    print(sess.run(x2, feed_dict={x: [2, 3]}))
    # [4. 9.]
```
Runs the same on TensorFlow 1.15.0 and 2.0.0, even though session and placeholders were deprecated in 2.x. Likewise, tf.compat.v2 allows you to use things introduced in 2.x from 1.x. Also, these APIs provide also backwards compatibility for the future too, so if at some point a 3.x version is released, the mechanism to write version-independent code will already be there since the first version of 2.x. EDIT: The documentation for the module about Python should actually be changed. Originally, tf.compat only held functions for that purpose (and it was like that until 1.13, see all module documentation). However, it was later repurposed for TensorFlow version compatibility.


Question: What is a tensorflow session actually? What does the sentence below quoted from getting started with tensorflow mean? A session encapsulates the control and state of the TensorFlow runtime. I know encapsulation in object oriented programming, and also have played with session a bit with success. Still I do not get this sentence very well. Can someone rephrase it in simple words?

Response:
This encapsulation has nothing to do with OOP encapsulation. A slightly better (in terms of understanding for a new-comer) definition is in session documentation. A Session object encapsulates the environment in which Operation objects are executed, and Tensor objects are evaluated. Which means that none the operators and variables defined in the graph-definition part are being executed. For example nothing is being executed/calculated here
```
a = tf.Variable(tf.random_normal([3, 3], stddev=1.)
b = tf.Variable(tf.random_normal([3, 3], stddev=1.)
c = a + b
```
You will not get the values of a tensors a/b/c now. There values will be evaluated only inside of the Session.


Question:
What is the expected input range for working with Keras VGG models? I'm trying to use a pretrained VGG 16 from keras. But I'm really unsure about what the input range should be. Quick answer, which of these color orders? RGB BGR And which range? 0 to 255? balanced from about -125 to about +130? 0 to 1? -1 to 1? I notice the file where the model is defined imports an input preprocessor:
```
from .imagenet_utils import preprocess_input
```
But this preprocessor is never used in the rest of the file. Also, when I check the code for this preprocessor, it has two modes: caffe and tf (tensorflow). Each mode works differently. Finally, I can't find consistent documentation on the internet. So, what is the best range for working? To what range are the model weights trained?

Response: The model weights were ported from caffe, so it's in BGR format. Caffe uses a BGR color channel scheme for reading image files. This is due to the underlying OpenCV implementation of imread. The assumption of RGB is a common mistake. You can find the original caffe model weight files on VGG website. This link can also be found on Keras documentation. I think the second range would be the closest one. There's no scaling during training, but the authors have subtracted the mean value of the ILSVRC2014 training set. As stated in the original VGG paper, section 2.1: The only preprocessing we do is subtracting the mean RGB value, computed on the training set, from each pixel. This sentence is actually what imagenet_utils.preprocess_input(mode='caffe') does. Convert from RGB to BGR: because keras.preprocessing.image.load_img() loads images in RGB format, this conversion is required for VGG16 (and all models ported from caffe). Subtract the mean BGR values: (103.939, 116.779, 123.68) is subtracted from the image array. The preprocessor is not used in vgg16.py. It's imported in the file so that users can use the preprocess function by calling keras.applications.vgg16.preprocess_input(rgb_img_array), without caring about where model weights come from. The argument for preprocess_input() is always an image array in RGB format. If the model was trained with caffe, preprocess_input() will convert the array into BGR format. Note that the function preprocess_input() is not intended to be called from imagenet_utils module. If you are using VGG16, call keras.applications.vgg16.preprocess_input() and the images will be converted to a suitable format and range that VGG16 was trained on. Similarly, if you are using Inception V3, call keras.applications.inception_v3.preprocess_input() and the images will be converted to the range that Inception V3 was trained on.


Question:
How do I generate a random vector in TensorFlow and maintain it for further use? I am trying to generate a random variable and use it twice. However, when I use it the second time, the generator creates a second random variable that is not identical to the first. Here is code to demonstrate:
```
import numpy as np
import tensorflow as tf

# A random variable
rand_var_1 = tf.random_uniform([5],0,10, dtype = tf.int32, seed = 0)
rand_var_2 = tf.random_uniform([5],0,10, dtype = tf.int32, seed = 0)

#Op1
z1 = tf.add(rand_var_1,rand_var_2)

#Op2
z2 = tf.add(rand_var_1,rand_var_2)

init = tf.initialize_all_variables()

with tf.Session() as sess:
    sess.run(init)
    z1_op = sess.run(z1)
    z2_op = sess.run(z2)
    print(z1_op,z2_op)
```
I want z1_op and z2_op to be equal. I think this is because the random_uniform op gets called twice. Is there a way to use TensorFlow (without using NumPy) to achieve this? (My use case is more complicated, but this is the distilled question.)

Response:
The current version of your code will randomly generate a new value for rand_var_1 and rand_var_2 on each call to sess.run() (although since you set the seed to 0, they will have the same value within a single call to sess.run()). If you want to retain the value of a randomly-generated tensor for later use, you should assign it to a tf.Variable:
```
rand_var_1 = tf.Variable(tf.random_uniform([5], 0, 10, dtype=tf.int32, seed=0))
rand_var_2 = tf.Variable(tf.random_uniform([5], 0, 10, dtype=tf.int32, seed=0))

# Or, alternatively:
rand_var_1 = tf.Variable(tf.random_uniform([5], 0, 10, dtype=tf.int32, seed=0))
rand_var_2 = tf.Variable(rand_var_1.initialized_value())

# Or, alternatively:
rand_t = tf.random_uniform([5], 0, 10, dtype=tf.int32, seed=0)
rand_var_1 = tf.Variable(rand_t)
rand_var_2 = tf.Variable(rand_t)
```
...then tf.initialize_all_variables() will have the desired effect:
```
# Op 1
z1 = tf.add(rand_var_1, rand_var_2)

# Op 2
z2 = tf.add(rand_var_1, rand_var_2)

init = tf.initialize_all_variables()

with tf.Session() as sess:
    sess.run(init)        # Random numbers generated here and cached.
    z1_op = sess.run(z1)  # Reuses cached values for rand_var_1, rand_var_2.
    z2_op = sess.run(z2)  # Reuses cached values for rand_var_1, rand_var_2.
    print(z1_op, z2_op)   # Will print two identical vectors.
```


Question:
What does it mean for a tensor to have shape [None, x] in TensorFlow? [duplicate] This question already has an answer here: Tensorflow batch size in input placholder (1 answer) Closed 7 years ago. If I define a tf placeholder as:
```
tf.placeholder("float", [None, 10])
```
what does the None do? What is the linear algebra interpretation of this? A horizontal vector?

Response:
A None value in the shape of a tensor means that the tensor can be of any size (large than or equal to 1) in that dimension. E.g., for the tf.placeholder you defined in your question, you could pass a arrays with shapes like
```
[1234, 10]
[10, 10]
[1, 10]
```
but not an array of shape [10,] (i.e. a vector).


Question:
Determining if A Value is in a Set in TensorFlow The tf.logical_or, tf.logical_and, and tf.select functions are very useful. However, suppose you have value x, and you wanted to see if it was in a set(a, b, c, d, e). In python you would simply write:
```
if x in set([a, b, c, d, e]):
  # Do some action.
```
As far as I can tell, the only way to do this in TensorFlow, is to have nested 'tf.logical_or' along with 'tf.equal'. I provided just one iteration of this concept below:
```
tf.logical_or(
    tf.logical_or(tf.equal(x, a), tf.equal(x, b)),
    tf.logical_or(tf.equal(x, c), tf.equal(x, d))
)
```
I feel that there must be an easier way to do this in TensorFlow. Is there?

Response:
To provide a more concrete answer, say you want to check whether the last dimension of the tensor x contains any value from a 1D tensor s, you could do the following:
```
tile_multiples = tf.concat([tf.ones(tf.shape(tf.shape(x)), dtype=tf.int32), tf.shape(s)], axis=0)
x_tile = tf.tile(tf.expand_dims(x, -1), tile_multiples)
x_in_s = tf.reduce_any(tf.equal(x_tile, s), -1))
```
For example, for s and x:
```
s = tf.constant([3, 4])
x = tf.constant([[[1, 2, 3, 0, 0],
                  [4, 4, 4, 0, 0]],
                 [[3, 5, 5, 6, 4],
                  [4, 7, 3, 8, 9]]])
```
x has shape [2, 2, 5] and s has shape [2] so tile_multiples = [1, 1, 1, 2], meaning we will tile the last dimension of x 2 times (once for each element in s) along a new dimension. So, x_tile will look like:
```
[[[[1 1]
   [2 2]
   [3 3]
   [0 0]
   [0 0]]

  [[4 4]
   [4 4]
   [4 4]
   [0 0]
   [0 0]]]

 [[[3 3]
   [5 5]
   [5 5]
   [6 6]
   [4 4]]

  [[4 4]
   [7 7]
   [3 3]
   [8 8]
   [9 9]]]]
```
and x_in_s will compare each of the tiled values to one of the values in s. tf.reduce_any along the last dim will return true if any of the tiled values was in s, giving the final result:
```
[[[False False  True False False]
  [ True  True  True False False]]

 [[ True False False False  True]
  [ True False  True False False]]]
```


Question:
Tensorflow 2.2.0 error: [Predictions must be > 0] [Condition x >= y did not hold element-wise:] while using Bidirectional LSTM layer I get the following error message when working on a named-entity-recognition task:
```
tensorflow.python.framework.errors_impl.InvalidArgumentError:  assertion failed: [predictions must be >= 0] [Condition x >= y did not hold element-wise:] [x (bidirectional_lstm_model/time_distributed/Reshape_1:0) = ] [[[-0.100267865 -0.104010895 0.04090859...]]...] [y (Cast_2/x:0) = ] [0]
     [[{{node assert_greater_equal/Assert/AssertGuard/else/_1/Assert}}]] [Op:__inference_train_function_6216]
Function call stack:
train_function
```
How can I troubleshoot this? I have checked my input train_x and train_y tensors and they seem fine (Some examples provided towards the end). I was originally using a Conditional Random Field decoder. I replaced that with a Dense layer instead, to see if that changes the error message. The error remains the same though, and is somehow related to the RNN component of the model. In general, what strategy do you use to troubleshoot such errors deep from within the guts of TF? I tried to set up a debugging session on PyCharm and jumped through a bunch of TF files, without learning anything useful about how to solve my problem. The following is my network architecture:
```
Model: "bidirectional_lstm_model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
encoder_input (InputLayer)   [(None, None)]            0
_________________________________________________________________
encoder_embedding (Embedding (None, None, 300)         2013300
_________________________________________________________________
encoder_bidirectional_rnn (B (None, None, 32)          40576
_________________________________________________________________
time_distributed (TimeDistri (None, None, 25)          825
=================================================================
Total params: 2,054,701
Trainable params: 41,401
Non-trainable params: 2,013,300
_________________________________________________________________
```
Above + more details (losses, optimizer etc):
```
# Create model
encoder_input = keras.Input(shape=(None,), name='encoder_input')
encoder_embedding = layers.Embedding(input_dim=input_vocabulary,
                                     output_dim=embedding_vector_len,
                                     embeddings_initializer=tf.keras.initializers.Constant(embedding_matrix),
                                     trainable=False, name='encoder_embedding')(encoder_input)
encoder_rnn = layers.LSTM(16, return_sequences=True, name='encoder_rnn')
encoder_bidirectional_rnn = layers.Bidirectional(encoder_rnn, name='encoder_bidirectional_rnn')(encoder_embedding)
decoder_dense = layers.TimeDistributed(layers.Dense(number_of_tags, name='decoder_dense'))(encoder_bidirectional_rnn)
model = keras.Model(inputs=encoder_input, outputs=decoder_dense, name='bidirectional_lstm_model')
model.summary()

metrics_precision = tf.keras.metrics.Precision()
metrics_recall = tf.keras.metrics.Recall()
model.compile(
    loss=tf.keras.losses.categorical_crossentropy,
    optimizer='adam',
    metrics=[metrics_precision, metrics_recall]
)
```
Here is what my train_x and train_y arrays look like:
```
# Shapes
train_x.shape  # (9775, 47)  (np.ndarray type)
train_y.shape  # TensorShape([9775, 47, 25])  (Obtained from tf.one_hot)

# Sample (Zero-padded from the right)
train_x[0, :]

# array([4917, 2806, 6357, 2287, 6059,    0,    0,    0,    0,    0,    0,
#      0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
#      0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
#      0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
#      0,    0,    0])

train_y[0, :, :]

# array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],  # Non "O" tag
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],  # Non "O" tag
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#   [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]], dtype=float32)
```

Response:
you are missing the last layer activation:
```py
decoder_dense = layers.TimeDistributed(layers.Dense(number_of_tags, name='decoder_dense'))(encoder_bidirectional_rnn)
```
You should specify that you want a softmax, leaving the activation as default is actually a linear activation, meaning that you can have any value, therefore the negative ones. You should create the last Dense layer as follows:
```py
decoder_dense = layers.TimeDistributed(layers.Dense(number_of_tags, activation='softmax', name='decoder_dense'))(encoder_bidirectional_rnn)
```


Question: Tensorflow js VS Tensorflow Lite Quite an open-ended question. Just pretty curious whats the current difference if I want to deploy a machine learning (object detection) model on the browser, perhaps on a webapp to begin with (to be viewed on a phone). From what I know, both tensorflowjs and tensorflowlite are compatible for such a deployment. (I've heard tensorflowlite is superior but, curious to find the pros and cons if any) What are the main differences between them? Will tensorflowjs be a good choice too?

Response: main difference of tensorflow-lite against tensorflow.js is that you can not train new or existing model using tensorflow lite,but with tensorflow.js you can. Hence tensorflowlite is only for inference to do on small capacity device(e.g. IOT) moreover tensorflow.js work with nodejs/typescript and/or browser environment only.


Question:
How does tf.keras.layers.Conv2D with padding='same' and strides > 1 behave? I read What is the difference between 'SAME' and 'VALID' padding in tf.nn.max_pool of tensorflow? but this is not true to my experiment.
```
import tensorflow as tf

inputs = tf.random_normal([1, 64, 64, 3])
print(inputs.shape)
conv = tf.keras.layers.Conv2D(6, 4, strides=2, padding='same')
outputs = conv(inputs)
print(outputs.shape)
```
produces
```
(1, 64, 64, 3)
(1, 32, 32, 6)
```
. However following the above link produces (1, 31, 31, 6) because there is no extra values outside filter ranges without any padding. How does tf.keras.layers.Conv2D with padding='same' and strides > 1 behave? I want to know the exact answer and its evidence.

Response:
Keras uses TensorFlow implementation of padding. All the details are available in the documentation here First, consider the 'SAME' padding scheme. A detailed explanation of the reasoning behind it is given in these notes. Here, we summarize the mechanics of this padding scheme. When using 'SAME', the output height and width are computed as:
```
out_height = ceil(float(in_height) / float(strides[1]))
out_width  = ceil(float(in_width) / float(strides[2]))
```
The total padding applied along the height and width is computed as:
```
if (in_height % strides[1] == 0):
  pad_along_height = max(filter_height - strides[1], 0)
else:
  pad_along_height = max(filter_height - (in_height % strides[1]), 0)
if (in_width % strides[2] == 0):
  pad_along_width = max(filter_width - strides[2], 0)
else:
  pad_along_width = max(filter_width - (in_width % strides[2]), 0)
```
Finally, the padding on the top, bottom, left and right are:
```
pad_top = pad_along_height // 2
pad_bottom = pad_along_height - pad_top
pad_left = pad_along_width // 2
pad_right = pad_along_width - pad_left
```
Note that the division by 2 means that there might be cases when the padding on both sides (top vs bottom, right vs left) are off by one. In this case, the bottom and right sides always get the one additional padded pixel. For example, when pad_along_height is 5, we pad 2 pixels at the top and 3 pixels at the bottom. Note that this is different from existing libraries such as cuDNN and Caffe, which explicitly specify the number of padded pixels and always pad the same number of pixels on both sides. For the 'VALID' scheme, the output height and width are computed as:
```
out_height = ceil(float(in_height - filter_height + 1) / float(strides[1]))
out_width  = ceil(float(in_width - filter_width + 1) / float(strides[2]))
```
and no padding is used.