Question:
Issue feeding a list into feed_dict in TensorFlow I'm trying to pass a list into feed_dict, however I'm having trouble doing so. Say I have:
```
inputs = 10 * [tf.placeholder(tf.float32, shape=(batch_size, input_size))]
```
where inputs is fed into some function outputs that I want to compute. So to run this in tensorflow, I created a session and ran the following:
```
sess.run(outputs, feed_dict = {inputs: data})
#data is my list of inputs, which is also of length 10
```
but I get an error, TypeError: unhashable type: 'list'. However, I'm able to pass the data element-wise like so:
```
sess.run(outputs, feed_dict = {inputs[0]: data[0], ..., inputs[9]: data[9]})
```
So I'm wondering if there's a way I can solve this issue. I've also tried to construct a dictionary(using a for loop), however this results in a dictionary with a single element, where they key is: tensorflow.python.framework.ops.Tensor at 0x107594a10

Response:
There are two issues that are causing problems here: The first issue is that the Session.run() call only accepts a small number of types as the keys of the feed_dict. In particular, lists of tensors are not supported as keys, so you have to put each tensor as a separate key.* One convenient way to do this is using a dictionary comprehension:
```
inputs = [tf.placeholder(...), ...]
data = [np.array(...), ...]
sess.run(y, feed_dict={i: d for i, d in zip(inputs, data)})
```
The second issue is that the 10 * [tf.placeholder(...)] syntax in Python creates a list with ten elements, where each element is the same tensor object (i.e. has the same name property, the same id property, and is reference-identical if you compare two elements from the list using inputs[i] is inputs[j]). This explains why, when you tried to create a dictionary using the list elements as keys, you ended up with a dictionary with a single element - because all of the list elements were identical. To create 10 different placeholder tensors, as you intended, you should instead do the following:
```
inputs = [tf.placeholder(tf.float32, shape=(batch_size, input_size))
          for _ in xrange(10)]
```
If you print the elements of this list, you'll see that each element is a tensor with a different name. EDIT: * You can now pass tuples as the keys of a feed_dict, because these may be used as dictionary keys.


Question:
Split tensor into training and test sets Let's say I've read in a textfile using a TextLineReader. Is there some way to split this into train and test sets in Tensorflow? Something like:
```
def read_my_file_format(filename_queue):
  reader = tf.TextLineReader()
  key, record_string = reader.read(filename_queue)
  raw_features, label = tf.decode_csv(record_string)
  features = some_processing(raw_features)
  features_train, labels_train, features_test, labels_test = tf.train_split(features,
                                                                            labels,
                                                                            frac=.1)
  return features_train, labels_train, features_test, labels_test
```

Response:
As elham mentioned, you can use scikit-learn to do this easily. scikit-learn is an open source library for machine learning. There are tons of tools for data preparation including the model_selection module, which handles comparing, validating and choosing parameters. The model_selection.train_test_split() method is specifically designed to split your data into train and test sets randomly and by percentage.
```
X_train, X_test, y_train, y_test = train_test_split(features,
                                                    labels,
                                                    test_size=0.33,
                                                    random_state=42)
```
test_size is the percentage to reserve for testing and random_state is to seed the random sampling. I typically use this to provide train and validation data sets, and keep true test data separately. You could just run train_test_split twice to do this as well. I.e. split the data into (Train + Validation) and Test, then split Train + Validation into two separate tensors.


Question: Keras - Validation Loss and Accuracy stuck at 0 I am trying to train a simple 2 layer Fully Connected neural net for Binary Classification in Tensorflow keras. I have split my data into Training and Validation sets with a 80-20 split using sklearn's train_test_split(). When I call model.fit(X_train, y_train, validation_data=[X_val, y_val]), it shows 0 validation loss and accuracy for all epochs, but it trains just fine. Also, when I try to evaluate it on the validation set, the output is non-zero. Can someone please explain why I am facing this 0 loss 0 accuracy error on validation. Thanks for your help. Here is the complete sample code (MCVE) for this error: https://colab.research.google.com/drive/1P8iCUlnD87vqtuS5YTdoePcDOVEKpBHr?usp=sharing

Response:
If you use keras instead of tf.keras everything works fine. With tf.keras, I even tried validation_data = [X_train, y_train], this also gives zero accuracy. Here is a demonstration:
```
model.fit(X_train, y_train, validation_data=[X_train.to_numpy(), y_train.to_numpy()],
epochs=10, batch_size=64)

Epoch 1/10
8/8 [==============================] - 0s 6ms/step - loss: 0.7898 - accuracy: 0.6087 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Epoch 2/10
8/8 [==============================] - 0s 6ms/step - loss: 0.6710 - accuracy: 0.6500 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Epoch 3/10
8/8 [==============================] - 0s 5ms/step - loss: 0.6748 - accuracy: 0.6500 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Epoch 4/10
8/8 [==============================] - 0s 6ms/step - loss: 0.6716 - accuracy: 0.6370 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Epoch 5/10
8/8 [==============================] - 0s 6ms/step - loss: 0.6085 - accuracy: 0.6326 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Epoch 6/10
8/8 [==============================] - 0s 6ms/step - loss: 0.6744 - accuracy: 0.6326 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Epoch 7/10
8/8 [==============================] - 0s 6ms/step - loss: 0.6102 - accuracy: 0.6522 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Epoch 8/10
8/8 [==============================] - 0s 6ms/step - loss: 0.7032 - accuracy: 0.6109 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Epoch 9/10
8/8 [==============================] - 0s 5ms/step - loss: 0.6283 - accuracy: 0.6717 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Epoch 10/10
8/8 [==============================] - 0s 5ms/step - loss: 0.6120 - accuracy: 0.6652 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
```
So, definitely there is some issue with tensorflow implementation of fit. I dug up the source, and it seems the part responsible for validation_data:
```
...
...
        # Run validation.
        if validation_data and self._should_eval(epoch, validation_freq):
          val_x, val_y, val_sample_weight = (
              data_adapter.unpack_x_y_sample_weight(validation_data))
          val_logs = self.evaluate(
              x=val_x,
              y=val_y,
              sample_weight=val_sample_weight,
              batch_size=validation_batch_size or batch_size,
              steps=validation_steps,
              callbacks=callbacks,
              max_queue_size=max_queue_size,
              workers=workers,
              use_multiprocessing=use_multiprocessing,
              return_dict=True)
          val_logs = {'val_' + name: val for name, val in val_logs.items()}
          epoch_logs.update(val_logs)
```
internally calls model.evaluate, as we have already established evaluate works fine, I realized the only culprit could be unpack_x_y_sample_weight. So, I looked into the implementation:
```
def unpack_x_y_sample_weight(data):
  """Unpacks user-provided data tuple."""
  if not isinstance(data, tuple):
    return (data, None, None)
  elif len(data) == 1:
    return (data[0], None, None)
  elif len(data) == 2:
    return (data[0], data[1], None)
  elif len(data) == 3:
    return (data[0], data[1], data[2])

  raise ValueError("Data not understood.")
```
It's crazy, but if you just pass a tuple instead of a list, everything works fine due to the check inside unpack_x_y_sample_weight. (Your labels are missing after this step and somehow the data is getting fixed inside evaluate, so you're training with no reasonable labels, this seems like a bug but the documentation clearly states to pass tuple) The following code gives correct validation accuracy and loss:
```
model.fit(X_train, y_train, validation_data=(X_train.to_numpy(), y_train.to_numpy()),
epochs=10, batch_size=64)

Epoch 1/10
8/8 [==============================] - 0s 7ms/step - loss: 0.5832 - accuracy: 0.6696 - val_loss: 0.6892 - val_accuracy: 0.6674
Epoch 2/10
8/8 [==============================] - 0s 7ms/step - loss: 0.6385 - accuracy: 0.6804 - val_loss: 0.8984 - val_accuracy: 0.5565
Epoch 3/10
8/8 [==============================] - 0s 7ms/step - loss: 0.6822 - accuracy: 0.6391 - val_loss: 0.6556 - val_accuracy: 0.6739
Epoch 4/10
8/8 [==============================] - 0s 6ms/step - loss: 0.6276 - accuracy: 0.6609 - val_loss: 1.0691 - val_accuracy: 0.5630
Epoch 5/10
8/8 [==============================] - 0s 7ms/step - loss: 0.7048 - accuracy: 0.6239 - val_loss: 0.6474 - val_accuracy: 0.6326
Epoch 6/10
8/8 [==============================] - 0s 7ms/step - loss: 0.6545 - accuracy: 0.6500 - val_loss: 0.6659 - val_accuracy: 0.6043
Epoch 7/10
8/8 [==============================] - 0s 7ms/step - loss: 0.5796 - accuracy: 0.6913 - val_loss: 0.6891 - val_accuracy: 0.6435
Epoch 8/10
8/8 [==============================] - 0s 7ms/step - loss: 0.5915 - accuracy: 0.6891 - val_loss: 0.5307 - val_accuracy: 0.7152
Epoch 9/10
8/8 [==============================] - 0s 7ms/step - loss: 0.5571 - accuracy: 0.7000 - val_loss: 0.5465 - val_accuracy: 0.6957
Epoch 10/10
8/8 [==============================] - 0s 7ms/step - loss: 0.7133 - accuracy: 0.6283 - val_loss: 0.7046 - val_accuracy: 0.6413
```
So, as this seems to be a bug, I have just opened a relevant issue at Tensorflow Github repo: https://github.com/tensorflow/tensorflow/issues/39370


Question:
When global_variables_initializer() is actually required
```
import tensorflow as tf
x = tf.constant(35, name='x')
y = tf.Variable(x + 5, name='y')
# model = tf.global_variables_initializer()
with tf.Session() as session:
        print("x = ", session.run(x))
        # session.run(model)
        print("y = ", session.run(y))
```
I was not able to understand when global_variables_initializer() is actually required. In the above code, if we uncomment lines 4 & 7, I can execute the code and see the values. If I run as-is, I see a crash. My question is which variables it is initializing. x is a constant which does not need initialization and y is variable which is not being initialized but is used as an arithmetic operation.

Response:
tf.global_variables_initializer is a shortcut to initialize all global variables. It is not required, and you can use other ways to initialize your variables or in case of easy scripts sometimes you do not need to initialize them at all. Everything except of variables do not require initialization (constants and placeholders). But every used variable (even if it is a constant) should be initialized. This will give you an error, although z is just 0-d tensor with only one number.
```
import tensorflow as tf
z = tf.Variable(4)
with tf.Session() as session:
        print(session.run(z))
```
I highlighted the word used, because if you just have variables which are not run (or non of the runs depends on them) you do not need to initialize them. For example this code will execute without any problems, nonetheless it has 2 variables and one operation which depends on them. But the run does not require them.
```
import tensorflow as tf
x = tf.constant(35, name='x')
y = tf.Variable(x + 5, name='y')
z = tf.Variable(4)
a = y + z
with tf.Session() as session:
        print("x = ", session.run(x))
```


Question: Difference between Tensorflow's tf.keras.layers.Dense and PyTorch's torch.nn.Linear? I have a quick (and possibly silly) question about how Tensorflow defines its Linear layer. Within PyTorch, a Linear (or Dense) layer is defined as, y = x A^T + b where A and b are the weight matrix and bias vector for a Linear layer (see here). However, I can't precisely find an equivalent equation for Tensorflow! Is it the same as PyTorch or is it just y = x A + b ? Thank you in advance!

Response:
If we set activation to None in the dense layer in keras API, then they are technically equivalent. Tensorflow's
```
tf.keras.layers.Dense(..., activation=None)
```
According to the doc, more study here. activation: Activation function to use. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x). And in PyTorch's src.
```
torch.nn.Linear
```
They are now equal at this point. A linear transformation to the incoming data: y = x*W^T + b. See the following more concrete equivalent implementation of these two. In PyTorch, we do
```
class Network(torch.nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.fc1 = torch.nn.Linear(5, 30)
    def forward(self, state):
        return self.fc1(state)
```
or,
```
trd = torch.nn.Linear(in_features = 3, out_features = 30)
y = trd(torch.ones(5, 3))
print(y.size())
# torch.Size([5, 30])
```
Its equivalent tf implementation would be
```
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(30, input_shape=(5,), activation=None))
```
or,
```
tfd = tf.keras.layers.Dense(30, input_shape=(3,), activation=None)
x = tfd(tf.ones(shape=(5, 3)))
print(x.shape)
# (5, 30)
```


Question: TensorFlow: Unpooling Is there TensorFlow native function that does unpooling for Deconvolutional Networks ? I have written this in normal python, but it is getting complicated when want to translate it to TensorFlow as it's objects does not even support item assignment at the moment, and I think this is a great inconvenience with TF.

Response: I don't think there is an official unpooling layer yet which is frustrating because you have to use image resize (bilinear interpolation or nearest neighbor) which is like an average unpooling operation and it's reaaaly slow. Look at the tf api in the section 'image' and you will find it. Tensorflow has a maxpooling_with_argmax thing where you get you maxpooled output as well as the activation map which is nice as you could use it in an unpooling layer to preserve the 'lost' spacial information but it seems as there isn't such an unpooling operation that does it. I guess that they are planning to add it ... soon. Edit: I found some guy on google discuss a week ago who seems to have implemented something like this but I personally haven't tried it yet. https://github.com/ppwwyyxx/tensorpack/blob/master/tensorpack/models/pool.py#L66


Question:
In TensorFlow,what's the meaning of ":0" in a Variable's name?
```
import tensorflow as tf
with tf.device('/gpu:0'):
    foo = tf.Variable(1, name='foo')
    assert foo.name == "foo:0"
with tf.device('/gpu:1'):
    bar = tf.Variable(1, name='bar')
    assert bar.name == "bar:0"
```
The above code returns true.I use with tf.device here to illustrate that the ":0" doesn't mean the variable lie on the specific device.So what's the meaning of the ":0" in the variable's name(foo and bar in this example)?

Response:
It has to do with representation of tensors in underlying API. A tensor is a value associated with output of some op. In case of variables, there's a Variable op with one output. An op can have more than one output, so those tensors get referenced to as :0, :1 etc. For instance if you use tf.nn.top_k, there are two values created by this op, so you may see TopKV2:0 and TopKV2:1
```
a,b=tf.nn.top_k([1], 1)
print a.name # => 'TopKV2:0'
print b.name # => 'TopKV2:1'
```
How to understand the term `tensor` in TensorFlow?


Question:
How to get the type of a Tensor? I'm looking for something similar to the effects of:
```
x.get_shape()
```
that will give the type of x. Is there is any function for this?

Response:
You can use get_shape() to get the shape of a tensorflow variable.
```

```python
x = tf.Variable(tf.random_normal([256, 100]))
```

```python
x.get_shape()
#Output
#(256, 100)
#```
#You can use dtype property to get the type of a tensorflow variable.
#```
```

```python
x = tf.Variable(tf.random_normal([256, 100]))
```

```python
x.dtype
#Output
#<dtype: 'float32_ref'>
#```
#You can use as_numpy_dtype property of dtype to convert from tf.dtype to numpy dtype.
#```
```

```python
x = tf.Variable(tf.random_normal([256, 100]))
```

```python
x.dtype.as_numpy_dtype
#Output
#<class 'numpy.float32'>
#```
```


Question: replicate a row tensor using tf.tile? I have a tensor which is simply a vector, vector = [0.5 0.4] and tf.shape indicates that it has shape=(1,), I would like to replicate the vector m times and have the shape of [m, 2], so for m = 2, matrix = [[0.5 0.4], [0.5 0.4]]. How do I achieve that using tf.tile?

Response:
Take the following, vec is a vector, multiply is your m, the number of times to repeat the vector. tf.tile is performed on the vector and then using tf.reshape it is reshaped into the desired structure.
```
import tensorflow as tf

vec = tf.constant([1, 2, 3, 4])
multiply = tf.constant([3])

matrix = tf.reshape(tf.tile(vec, multiply), [ multiply[0], tf.shape(vec)[0]])
with tf.Session() as sess:
    print(sess.run([matrix]))
```
This results in:
```
[array([[1, 2, 3, 4],
       [1, 2, 3, 4],
       [1, 2, 3, 4]], dtype=int32)]
```


Question: Difference between tf.clip_by_value and tf.clip_by_global_norm for RNN's and how to decide max value to clip on? Want to understand the difference in roles of tf.clip_by_value and tf.clip_by_global_norm during the implementation of Gradient Clipping in TensorFlow. Which one is preferred and how to decide the max value to clip on?

Response:
TL;DR: use tf.clip_by_global_norm for gradient clipping, with "some high value" as max value. clip_by_value tf.clip_by_value clips each value inside one tensor, regardless of the other values in the tensor. For instance,
```
tf.clip_by_value([-1, 2, 10], 0, 3)  -> [0, 2, 3]  # Only the values below 0 or above 3 are changed
```
Consequently, it can change the direction of the tensor, so it should be used if the values in the tensor are decorrelated one from another (which is not the case for gradient clipping), or to avoid zero / infinite values in a tensor that could lead to Nan / infinite values elsewhere (by clipping with a minimum of epsilon=1e-8 and a very big max value for instance). clip_by_norm tf.clip_by_norm rescales one tensor if necessary, so that its L2 norm does not exceed a certain threshold. It's useful typically to avoid exploding gradient on one tensor, because you keep the gradient direction. For instance:
```
tf.clip_by_norm([-2, 3, 6], 5)  -> [-2, 3, 6]*5/7  # The original L2 norm is 7, which is >5, so the final one is 5
tf.clip_by_norm([-2, 3, 6], 9)  -> [-2, 3, 6]  # The original L2 norm is 7, which is <9, so it is left unchanged
```
However, clip_by_norm works on only one gradient, so if you use it on all your gradient tensors, you'll unbalance them (some will be rescaled, others not, and not all with the same scale). Note that the two first ones work on only one tensor, while the last one is used on a list of tensors. clip_by_global_norm tf.clip_by_global_norm rescales a list of tensors so that the total norm of the vector of all their norms does not exceed a threshold. The goal is the same as clip_by_norm (avoid exploding gradient, keep the gradient directions), but it works on all the gradients at once rather than on each one separately (that is, all of them are rescaled by the same factor if necessary, or none of them are rescaled). This is better, because the balance between the different gradients is maintained. For instance:
```
tf.clip_by_global_norm([tf.constant([-2, 3, 6]),tf.constant([-4, 6, 12])] , 14.5)
```
will rescale both tensors by a factor 14.5/sqrt(49 + 196), because the first tensor has a L2 norm of 7, the second one 14, and sqrt(7^2+ 14^2)>14.5 This (tf.clip_by_global_norm) is the one that you should use for gradient clipping. See this for instance for more information. Choosing the value Choosing the max value is the hardest part. You should use the biggest value such that you don't have exploding gradient (whose effects can be Nans or infinite values appearing in your tensors, constant loss /accuracy after a few training steps). The value should be bigger for tf.clip_by_global_norm than for the others, since the global L2 norm will be mechanically bigger than the other ones due to the number of tensors implied.


Question: Does TensorFlow view all CPUs of one machine as ONE device? From the experiments I run, it seems like TensorFlow uses automatically all CPUs on one machine. Furthermore, it seems like TensorFlow refers to all CPUs as /cpu:0. Am I right, that only the different GPUs of one machine get indexed and viewed as separate devices, but all the CPUs on one machine get viewed as a single device? Is there any way that a machine can have multiple CPUs viewing it from TensorFlows perspective?

Response: By default all CPUs available to the process are aggregated under cpu:0 device. There's answer by mrry here showing how to create logical devices like /cpu:1, /cpu:2 There doesn't seem to be working functionality to pin logical devices to specific physical cores or be able to use NUMA nodes in tensorflow. A possible work-around is to use distributed TensorFlow with multiple processes on one machine and use taskset on Linux to pin specific processes to specific cores


Question: Convolution2D + LSTM versus ConvLSTM2D Are 1 and 2 the same? Use Convolution2D layers and LSTM layers Use ConvLSTM2D If there is any difference, could you explain it for me?

Response:
They are not exactly the same, here is why: 1. Use Convolution2D layers and LSTM layers As it is known, Convolution2D serves well for capturing image or spatial features, whilst LSTM are used to detect correlations over time. However, by stacking these kind of layers, the correlation between space and time features may not be captured properly. 2. Use ConvLSTM2D To solve this, Xingjian Shi et al. proposed a network structure able to capture spatiotemporal correlations, namely ConvLSTM. In Keras, this is reflected in the ConvLSTM2D class, which computes convolutional operations in both the input and the recurrent transformations. Keras code Too illustrate this, you can see here the LSTM code, if you go to the call method from LSTMCell, you'd only see:
```python
x_i = K.dot(inputs_i, self.kernel_i)
    x_f = K.dot(inputs_f, self.kernel_f)
    x_c = K.dot(inputs_c, self.kernel_c)
    x_o = K.dot(inputs_o, self.kernel_o)
```
Instead, the ConvLSTM2DCell class calls:
```python
x_i = self.input_conv(inputs_i, self.kernel_i, self.bias_i, padding=self.padding)
    x_f = self.input_conv(inputs_f, self.kernel_f, self.bias_f, padding=self.padding)
    x_c = self.input_conv(inputs_c, self.kernel_c, self.bias_c, padding=self.padding)
    x_o = self.input_conv(inputs_o, self.kernel_o, self.bias_o, padding=self.padding)
    h_i = self.recurrent_conv(h_tm1_i, self.recurrent_kernel_i)
    h_f = self.recurrent_conv(h_tm1_f, self.recurrent_kernel_f)
    h_c = self.recurrent_conv(h_tm1_c, self.recurrent_kernel_c)
    h_o = self.recurrent_conv(h_tm1_o, self.recurrent_kernel_o)
```
Where:
```python
def input_conv(self, x, w, b=None, padding='valid'):
    conv_out = K.conv2d(x, w, strides=self.strides,
                        padding=padding,
                        data_format=self.data_format,
                        dilation_rate=self.dilation_rate)
    if b is not None:
        conv_out = K.bias_add(conv_out, b,
                              data_format=self.data_format)
    return conv_out

def recurrent_conv(self, x, w):
    conv_out = K.conv2d(x, w, strides=(1, 1),
                        padding='same',
                        data_format=self.data_format)
    return conv_out
```
In LSTM, the equivalent for h_x (recurrent transformations) would be:
```python
K.dot(h_tm1_x, self.recurrent_kernel_x)
```
Instead of ConvLSTM2D's:
```python
self.recurrent_conv(h_tm1_x, self.recurrent_kernel_x)
```
These kind of transformations could not be computed with stacked Conv2D and LSTM layers.


Question:
Keras: how to get tensor dimensions inside custom loss? I'm trying to write my custom loss function: I want to apply categorical_crossentropy to the parts of input vector and then sum. Assume y_true, y_pred are 1D vectors. Code:
```
def custom_loss(y_true, y_pred):

    loss_sum= 0.0
    for i in range(0,y_true.shape[0],dictionary_dims):
        loss_sum+= keras.backend.categorical_crossentropy(y_true[i*dictionary_dims:(i+1)*dictionary_dims], y_pred[i*dictionary_dims:(i+1)*dictionary_dims])

    return loss_sum
```
But I get an error:
```
for i in range(0,y_true.shape[0],dictionary_dims):
TypeError: __index__ returned non-int (type NoneType)
```
So how to access shape of input tensors to get subset of tensor? Update: Also tried to write loss via tensorflow directly:
```
def custom_loss_tf(y_true, y_pred):

    print('tf.shape(y_true)',tf.shape(y_true)) #
    print('type(tf.shape(y_true))',type(tf.shape(y_true))) #

    sys.exit()

    loss_sum= 0.0
    for i in range(0,y_true.shape[0],dictionary_dims):
        loss_sum+= keras.backend.categorical_crossentropy(y_true[i*dictionary_dims:(i+1)*dictionary_dims], y_pred[i*dictionary_dims:(i+1)*dictionary_dims])

    return loss_sum
```
Output:
```
tf.shape(y_true) Tensor("Shape:0", shape=(2,), dtype=int32)
type(tf.shape(y_true)) <class 'tensorflow.python.framework.ops.Tensor'>
```
Not sure what is shape=(2,) mean, but this is not what I'm expecting, because model.summary() shows that last layer is (None, 26):
```
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         (None, 80, 120, 3)        0
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 80, 120, 32)       896
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 40, 60, 32)        0
_________________________________________________________________
activation_1 (Activation)    (None, 40, 60, 32)        0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 40, 60, 32)        9248
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 20, 30, 32)        0
_________________________________________________________________
activation_2 (Activation)    (None, 20, 30, 32)        0
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 20, 30, 64)        18496
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 10, 15, 64)        0
_________________________________________________________________
activation_3 (Activation)    (None, 10, 15, 64)        0
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 10, 15, 64)        36928
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 5, 7, 64)          0
_________________________________________________________________
activation_4 (Activation)    (None, 5, 7, 64)          0
_________________________________________________________________
flatten_1 (Flatten)          (None, 2240)              0
_________________________________________________________________
head (Dense)                 (None, 26)                58266
=================================================================
```

Response: Two things here: If you want to get a tensor shape you should use int_shape function from keras.backend. The first dimension is set to be a batch dimension so int_shape(y_true)[0] will return you a batch size. You should use int_shape(y_true)[1].


Question: tensorflow einsum vs. matmul vs. tensordot In tensorflow, the functions tf.einsum, tf.matmul, and tf.tensordot can all be used for the same tasks. (I realize that tf.einsum and tf.tensordot have more general definitions; I also realize that tf.matmul has batch functionality.) In a situation where any of the three could be used, does one function tend to be fastest? Are there other recommendation rules? For example, suppose that A is a rank-2 tensor, and b is rank-1 tensor, and you want to compute the product c_j = A_ij b_j. Of the three options: c = tf.einsum('ij,j->i', A, b) c = tf.matmul(A, tf.expand_dims(b,1)) c = tf.tensordot(A, b, 1) Is any generally preferable to the others?

Response: Both tf.tensordot() and tf.einsum() are syntactic sugar that wrap one or more invocations of tf.matmul() (although in some special cases tf.einsum() can reduce to the simpler elementwise tf.multiply()). In the limit, I'd expect all three functions to have equivalent performance for the same computation. However, for smaller matrices, it may be more efficient to use tf.matmul() directly, because it would yield a simpler TensorFlow graph with fewer operations, and hence the pre-operation invocation costs will be lower.


Question:
How is tf.summary.tensor_summary meant to be used? TensorFlow provides a tf.summary.tensor_summary() function that appears to be a multidimensional variant of tf.summary.scalar():
```py
tf.summary.tensor_summary(name, tensor, summary_description=None, collections=None)
```
I thought it could be useful for summarizing inferred probabilities per class ... somewhat like
```py
op_summary = tf.summary.tensor_summary('classes', some_tensor)
# ...
summary = sess.run(op_summary)
writer.add_summary(summary)
```
However it appears that TensorBoard doesn't provide a way to display these summaries at all. How are they meant to be used?

Response: I cannot get it to work either. It seems like that feature is still under development. See this video from the TensorFlow Dev Summit that states that the tensor_summary is still under development (starting at 9:17): https://youtu.be/eBbEDRsCmv4?t=9m17s. It will probably be better defined and examples should be provided in the future.


Question: What is the difference between keras and tf.keras? I'm learning TensorFlow and Keras. I'd like to try https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438/, and it seems to be written in Keras. Would it be fairly straightforward to convert code to tf.keras? I'm not more interested in the portability of the code, rather than the true difference between the two.

Response: The difference between tf.keras and keras is the Tensorflow specific enhancement to the framework. keras is an API specification that describes how a Deep Learning framework should implement certain part, related to the model definition and training. Is framework agnostic and supports different backends (Theano, Tensorflow, ...) tf.keras is the Tensorflow specific implementation of the Keras API specification. It adds the framework the support for many Tensorflow specific features like: perfect support for tf.data.Dataset as input objects, support for eager execution, ... In Tensorflow 2.0 tf.keras will be the default and I highly recommend to start working using tf.keras


Question:
tf.multiply vs tf.matmul to calculate the dot product I have a matrix (of vectors) X with shape [3,4], and I want to calculate the dot product between each pair of vectors (X[1].X[1]) and (X[1].X[2])...etc. I saw a cosine similarity code were they use tf.reduce_sum(tf.multyply(X, X),axis=1) to calculate the dot product between the vectors in a matrix of vectors.However, this result in only calculates the dot product between (X[i], X[i]). I used tf.matmul(X, X, transpose_b=True) which calculate the dot product between every two vectors but I am still confused why tf.multiply didn't do this I think the problem with my code. the code is:
```
data=[[1.0,2.0,4.0,5.0],[0.0,6.0,7.0,8.0],[8.0,1.0,1.0,1.0]]
X=tf.constant(data)
matResult=tf.matmul(X, X, transpose_b=True)

multiplyResult=tf.reduce_sum(tf.multiply(X,X),axis=1)
with tf.Session() as sess:
   print('matResult')
   print(sess.run([matResult]))
   print()
   print('multiplyResult')
   print(sess.run([multiplyResult]))
```
The output is:
```
matResult
[array([[  46.,   80.,   19.],
       [  80.,  149.,   21.],
       [  19.,   21.,   67.]], dtype=float32)]

multiplyResult
 [array([  46.,  149.,   67.], dtype=float32)]
```
I would appreciate any advise

Response:
tf.multiply(X, Y) or the * operator does element-wise multiplication so that:
```
[[1 2]    [[1 3]      [[1 6]
 [3 4]] .  [2 1]]  =   [6 4]]
```
wheras tf.matmul does matrix multiplication so that:
```
[[1 0]    [[1 3]      [[1 3]
 [0 1]] .  [2 1]]  =   [2 1]]
```
using tf.matmul(X, X, transpose_b=True) means that you are calculating X . X^T where ^T indicates the transposing of the matrix and . is the matrix multiplication. tf.reduce_sum(_, axis=1) takes the sum along 1st axis (starting counting with 0) which means you are suming the rows:
```
tf.reduce_sum([[a, b], [c, d]], axis=1) = [a+b, c+d]
```
This means that:
```
tf.reduce_sum(tf.multiply(X, X), axis=1) = [X[1].X[1], ..., X[n].X[n]]
```
so that is the one you want if you only want the norms of each rows. On the other hand:
```
tf.matmul(X, X, transpose_b=True) = [
                                      [ X[1].X[1], X[1].X[2], ..., X[1].X[n] ],
                                      [ X[2].X[1], ..., X[2].X[n] ],
                                       ...
                                      [ X[n].X[1], ..., X[n].X[n] ]
                                   ]
```
so that is what you need if you want the similarity between all pairs of rows.


Question: Implementation difference between TensorFlow Variable and TensorFlow Tensor First of all, I am aware that a related question has been asked here. However, this question is about the implementation and internals. I was reading the paper "A Tour of TensorFlow". The following two points are quoted from there: 1. A tensor itself does not hold or store values in memory, but provides only an interface for retrieving the value referenced by the tensor. This suggests to me that a Tensor is an object that simply stores the pointer to a result of an operation and, on retrieving the result or value of the tensor, it simply dereferences that pointer. 2. Variables can be described as persistent, mutable handles to in-memory buffers storing tensors. As such, variables are characterized by a certain shape and a fixed type. At this I get confused because I thought, based on the previous point, that Tensors simply store a pointer. If they were simply pointers, they could be mutable as well. To be precise these are my questions: What is the meaning of "in-memory buffers"? What is the meaning of a "handle"? Is my initial assumption about the internals of a tensor correct? What is the essential internal implementation difference between a tensor and a variable? Why are they declared differently and why is that difference essential to TensorFlow?

Response: Before explaining the distinction between tensors and variables, we should be precise about what the word "tensor" means in the context of TensorFlow: In the Python API, a tf.Tensor object represents the symbolic result of a TensorFlow operation. For example, in the expression t = tf.matmul(x, y), t is a tf.Tensor object representing the result of multiplying x and y (which may themselves be symbolic results of other operations, concrete values such as NumPy arrays, or variables). In this context, a "symbolic result" is more complicated than a pointer to the result of an operation. It is more analogous to a function object that, when called (i.e. passed to tf.Session.run()) will run the necessary computation to produce the result of that operation, and return it to you as a concrete value (e.g. a NumPy array). In the C++ API, a tensorflow::Tensor object represents the concrete value of a multi-dimensional array. For example, the MatMul kernel takes two two-dimensional tensorflow::Tensor objects as inputs, and produces a single two-dimensional tensorflow::Tensor object as its output. This distinction is a little confusing, and we might choose different names if we started over (in other language APIs, we prefer the name Output for a symbolic result and Tensor for a concrete value). A similar distinction exists for variables. In the Python API, a tf.Variable is the symbolic representation of a variable, which has methods for creating operations that read the current value of the variable, and assign values to it. In the C++ implementation, a tensorflow::Var object is a wrapper around a shared, mutable tensorflow::Tensor object. With that context out the way, we can address your specific questions: What is the meaning of "in-memory buffers"? An in-memory buffer is simply a contiguous region of memory that has been allocated with a TensorFlow allocator. tensorflow::Tensor objects contain a pointer to an in-memory buffer, which holds the values of that tensor. The buffer could be in host memory (i.e. accessible from the CPU) or device memory (e.g. accessible only from a GPU), and TensorFlow has operations to move data between these memory spaces. What is the meaning of a "handle"? In the explanation in the paper, the word "handle" is used in a couple of different ways, which are slightly different from how TensorFlow uses the term. The paper uses "symbolic handle" to refer to a tf.Tensor object, and "persistent, mutable handle" to refer to a tf.Variable object. The TensorFlow codebase uses "handle" to refer to a name for a stateful object (like a tf.FIFOQueue or tf.TensorArray) that can be passed around without copying all of the values (i.e. call-by-reference). Is my initial assumption about the internal of a tensor correct? Your assumption most closely matches the definition of a (C++) tensorflow::Tensor object. The (Python) tf.Tensor object is more complicated because it refers to a function for computing a value, rather than the value itself. What is the essential internal implementation difference between a tensor and a variable? In C++, a tensorflow::Tensor and tensorflow::Var are very similar; the only different is that tensorflow::Var also has a mutex that can be used to lock the variable when it is being updated. In Python, the essential difference is that a tf.Tensor is implemented as a dataflow graph, and it is read-only (i.e. by calling tf.Session.run()). A tf.Variable can be both read (i.e. by evaluating its read operation) and written (e.g. by running an assign operation). Why are they declared differently and why is that difference essential to TensorFlow? Tensors and variables serve different purposes. Tensors (tf.Tensor objects) can represent complex compositions of mathematical expressions, like loss functions in a neural network, or symbolic gradients. Variables represent state that is updated over time, like weight matrices and convolutional filters during training. While in principle you could represent the evolving state of a model without variables, you would end up with a very large (and repetetive) mathematical expression, so variables provide a convenient way to materialize the state of the model, and—for example—share it with other machines for parallel training.


Question:
Keras custom loss function: Accessing current input pattern In Keras (with Tensorflow backend), is the current input pattern available to my custom loss function? The current input pattern is defined as the input vector used to produce the prediction. For example, consider the following: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42, shuffle=False). Then the current input pattern is the current X_train vector associated with the y_train (which is termed y_true in the loss function). When designing a custom loss function, I intend to optimize/minimize a value that requires access to the current input pattern, not just the current prediction. I've taken a look through https://github.com/fchollet/keras/blob/master/keras/losses.py I've also looked through "Cost function that isn't just y_pred, y_true?" I am also familiar with previous examples to produce a customized loss function:
```
import keras.backend as K

def customLoss(y_true,y_pred):
    return K.sum(K.log(y_true) - K.log(y_pred))
```
Presumably (y_true,y_pred) are defined elsewhere. I've taken a look through the source code without success and I'm wondering whether I need to define the current input pattern myself or whether this is already accessible to my loss function.

Response:
You can wrap the loss function as a inner function and pass your input tensor to it (as commonly done when passing additional arguments to the loss function).
```
def custom_loss_wrapper(input_tensor):
    def custom_loss(y_true, y_pred):
        return K.binary_crossentropy(y_true, y_pred) + K.mean(input_tensor)
    return custom_loss

input_tensor = Input(shape=(10,))
hidden = Dense(100, activation='relu')(input_tensor)
out = Dense(1, activation='sigmoid')(hidden)
model = Model(input_tensor, out)
model.compile(loss=custom_loss_wrapper(input_tensor), optimizer='adam')
```
You can verify that input_tensor and the loss value (mostly, the K.mean(input_tensor) part) will change as different X is passed to the model.
```
X = np.random.rand(1000, 10)
y = np.random.randint(2, size=1000)
model.test_on_batch(X, y)  # => 1.1974642

X *= 1000
model.test_on_batch(X, y)  # => 511.15466
```


Question:
TensorFlow: AttributeError: 'Tensor' object has no attribute 'shape' I have the following code which uses TensorFlow. After I reshape a list, it says AttributeError: 'Tensor' object has no attribute 'shape' when I try to print its shape.
```
# Get the shape of the training data.
print "train_data.shape: " + str(train_data.shape)
train_data = tf.reshape(train_data, [400, 1])
print "train_data.shape: " + str(train_data.shape)
train_size,num_features = train_data.shape
```
Output: train_data.shape: (400,) Traceback (most recent call last): File "", line 1, in File "/home/shehab/Downloads/tools/python/pycharm-edu-2.0.4/helpers/pydev/pydev_import_hook.py", line 21, in do_import module = self._system_import(name, *args, **kwargs) File "/home/shehab/Dropbox/py-projects/try-tf/logistic_regression.py", line 77, in print "train_data.shape: " + str(train_data.shape) AttributeError: 'Tensor' object has no attribute 'shape' Could anyone please tell me what I am missing?

Response:
UPDATE: Since TensorFlow 1.0, tf.Tensor now has a tf.Tensor.shape property, which returns the same value as tf.Tensor.get_shape(). Indeed, in versions prior to TensorFlow 1.0 tf.Tensor doesn't have a .shape property. You should use the Tensor.get_shape() method instead:
```
train_data = tf.reshape(train_data, [400, 1])
print "train_data.shape: " + str(train_data.get_shape())
```
Note that in general you might not be able to get the actual shape of the result of a TensorFlow operation. In some cases, the shape will be a computed value that depends on running the computation to find its value; and it may even vary from one run to the next (e.g. the shape of tf.unique()). In that case, the result of get_shape() for some dimensions may be None (or "?").


Question: What is right batch normalization function in Tensorflow? In tensorflow 1.4, I found two functions that do batch normalization and they look same: tf.layers.batch_normalization (link) tf.contrib.layers.batch_norm (link) Which function should I use? Which one is more stable?

Response: Just to add to the list, there're several more ways to do batch-norm in tensorflow: tf.nn.batch_normalization is a low-level op. The caller is responsible to handle mean and variance tensors themselves. tf.nn.fused_batch_norm is another low-level op, similar to the previous one. The difference is that it's optimized for 4D input tensors, which is the usual case in convolutional neural networks. tf.nn.batch_normalization accepts tensors of any rank greater than 1. tf.layers.batch_normalization is a high-level wrapper over the previous ops. The biggest difference is that it takes care of creating and managing the running mean and variance tensors, and calls a fast fused op when possible. Usually, this should be the default choice for you. tf.contrib.layers.batch_norm is the early implementation of batch norm, before it's graduated to the core API (i.e., tf.layers). The use of it is not recommended because it may be dropped in the future releases. tf.nn.batch_norm_with_global_normalization is another deprecated op. Currently, delegates the call to tf.nn.batch_normalization, but likely to be dropped in the future. Finally, there's also Keras layer keras.layers.BatchNormalization, which in case of tensorflow backend invokes tf.nn.batch_normalization.


Question: what does arg_scope actually do? I am a beginner in neural nets and TensorFlow, and I am trying to understand the role of arg_scope. It seems to me that it is a way to put together a dictionary of "things you want to do" to a certain layer with certain variables. Please correct me if I am wrong. How would you explain exactly what it is for, to a beginner?

Response:
When defining convolution layers, you may always use the same padding type and the same initializer, and maybe even the same convolution size. For you pooling, maybe you are also always using the same 2x2 pooling size. And so on. arg_scope is a way to avoid repeating providing the same arguments over and over again to the same layer types. Examples from the source documentation: Example of how to use tf.contrib.framework.arg_scope:
```
from third_party.tensorflow.contrib.layers.python import layers
  arg_scope = tf.contrib.framework.arg_scope
  with arg_scope([layers.conv2d], padding='SAME',
                 initializer=layers.variance_scaling_initializer(),
                 regularizer=layers.l2_regularizer(0.05)):
    net = layers.conv2d(inputs, 64, [11, 11], 4, padding='VALID', scope='conv1')
    net = layers.conv2d(net, 256, [5, 5], scope='conv2')
```
The first call to conv2d will behave as follows:
```
layers.conv2d(inputs, 64, [11, 11], 4, padding='VALID',
                  initializer=layers.variance_scaling_initializer(),
                  regularizer=layers.l2_regularizer(0.05), scope='conv1')
```
The second call to conv2d will also use the arg_scope's default for padding:
```
layers.conv2d(inputs, 256, [5, 5], padding='SAME',
                  initializer=layers.variance_scaling_initializer(),
                  regularizer=layers.l2_regularizer(0.05), scope='conv2')
```
Example of how to reuse an arg_scope:
```
with arg_scope([layers.conv2d], padding='SAME',
                 initializer=layers.variance_scaling_initializer(),
                 regularizer=layers.l2_regularizer(0.05)) as sc:
    net = layers.conv2d(net, 256, [5, 5], scope='conv1')
    ....
  with arg_scope(sc):
    net = layers.conv2d(net, 256, [5, 5], scope='conv2')
```


Question: How to accumulate gradients in tensorflow? I have a question similar to this one. Because I have limited resources and I work with a deep model (VGG-16) - used to train a triplet network - I want to accumulate gradients for 128 batches of size one training example, and then propagate the error and update the weights. It's not clear to me how do I do this. I work with tensorflow but any implementation/pseudocode is welcome.

Response:
Let's walk through the code proposed in one of the answers you linked to:
```
## Optimizer definition - nothing different from any classical example
opt = tf.train.AdamOptimizer()

## Retrieve all trainable variables you defined in your graph
tvs = tf.trainable_variables()
## Creation of a list of variables with the same shape as the trainable ones
# initialized with 0s
accum_vars = [tf.Variable(tf.zeros_like(tv.initialized_value()), trainable=False) for tv in tvs]
zero_ops = [tv.assign(tf.zeros_like(tv)) for tv in accum_vars]

## Calls the compute_gradients function of the optimizer to obtain... the list of gradients
gvs = opt.compute_gradients(rmse, tvs)

## Adds to each element from the list you initialized earlier with zeros its gradient (works because accum_vars and gvs are in the same order)
accum_ops = [accum_vars[i].assign_add(gv[0]) for i, gv in enumerate(gvs)]

## Define the training step (part with variable value update)
train_step = opt.apply_gradients([(accum_vars[i], gv[1]) for i, gv in enumerate(gvs)])
```
This first part basically adds new variables and ops to your graph which will allow you to Accumulate the gradient with ops accum_ops in (the list of) variable accum_vars Update the model weights with ops train_step Then, to use it when training, you have to follow these steps (still from the answer you linked):
```
## The while loop for training
while ...:
    # Run the zero_ops to initialize it
    sess.run(zero_ops)
    # Accumulate the gradients 'n_minibatches' times in accum_vars using accum_ops
    for i in xrange(n_minibatches):
        sess.run(accum_ops, feed_dict=dict(X: Xs[i], y: ys[i]))
    # Run the train_step ops to update the weights based on your accumulated gradients
    sess.run(train_step)
```


Question: Difference between model(x) and model.predict(x) in Keras? What is the difference between using model(x) and model.predict(x) for predicting the outcome of a model in Keras?

Response:
Keras with tensorflow backend was using underlying tensorflow objects, but mostly was providing high level outputs which could be understood outside the tensorflow environment (as an example it could output numpy arrays or python lists). Today given a model in tensorflow 2.0 (built using the keras library),
```
out_np = model.predict(x)
```
provides a numpy array which can, as an example, be printed with print(out_np). On the other hand,
```
out_tf = model(x)
```
results into a tensorflow object, wich can be converted to a numpy array with .numpy() The two results are equivalent, as an example, we have that the following is True,
```
out_np.max() == out_tf.numpy().max()
```
The format may be different, but the meaning of model(x) and model.predict(x) is the same: given an input x, it is the value of the output nodes of a network characterized by its structure, weights and biases.


Question: Need To Compile Keras Model Before `model.evaluate()` I load a Keras model from .json and .hdf5 files. When I call model.evaluate(), it returns an error: You must compile a model before training/testing. Use `model.compile(optimizer, loss) Why do I need to compile to run evaluate()? To add, the model can be passed predict() with no problem.

Response:
Because evaluate will calculate the loss function and the metrics. You don't have any of them until you compile the model. They're parameters to the compile method:
```
model.compile(optimizer=..., loss=..., metrics=...)
```
On the other hand, predict doesn't evaluate any metric or loss, it just passes the input data through the model and gets its output. You need the "loss" for training too, so you can't train without compiling. And you can compile a model as many times as you want, and even change the parameters. The outputs and the loss function: The model's outputs depend on it being defined with weights. That is automatic and you can predict from any model, even without any training. Every model in Keras is already born with weights (either initialized by you or randomly initialized) You input something, the model calculates the output. At the end of everything, this is all that matters. A good model has proper weights and outputs things correctly. But before getting to that end, your model needs to be trained. Now, the loss function takes the current output and compares it with the expected/true result. It's a function supposed to be minimized. The less the loss, the closer your results are to the expected. This is the function from which the derivatives will be taken so the backpropagation algorithm can update the weights. The loss function is not useful for the final purpose of the model, but it's necessary for training. That's probably why you can have models without loss functions (and consequently, there is no way to evaluate them).


Question: How can I change the shape of a variable in TensorFlow? TensorFlow tutorial says that at creation time we need to specify the shape of tensors. That shape automatically becomes the shape of the tensor. It also says that TensorFlow provides advanced mechanisms to reshape variables. How can I do that? Any code example?

Response:
The tf.Variable class is the recommended way to create variables, but it restricts your ability to change the shape of the variable once it has been created. If you need to change the shape of a variable, you can do the following (e.g. for a 32-bit floating point tensor):
```
var = tf.Variable(tf.placeholder(tf.float32))
# ...
new_value = ...  # Tensor or numpy array.
change_shape_op = tf.assign(var, new_value, validate_shape=False)
# ...
sess.run(change_shape_op)  # Changes the shape of `var` to new_value's shape.
```
Note that this feature is not in the documented public API, so it is subject to change. If you do find yourself needing to use this feature, let us know, and we can investigate a way to support it moving forward.


Question:
Tensorflow: When should I use or not use `feed_dict`? I am kind of confused why are we using feed_dict? According to my friend, you commonly use feed_dict when you use placeholder, and this is probably something bad for production. I have seen code like this, in which feed_dict is not involved:
```
for j in range(n_batches):
    X_batch, Y_batch = mnist.train.next_batch(batch_size)
    _, loss_batch = sess.run([optimizer, loss], {X: X_batch, Y:Y_batch})
```
I have also seen code like this, in which feed_dict is involved:
```
for i in range(100):
    for x, y in data:
        # Session execute optimizer and fetch values of loss
        _, l = sess.run([optimizer, loss], feed_dict={X: x, Y:y})
        total_loss += l
```
I understand feed_dict is that you are feeding in data and try X as the key as if in the dictionary. But here I don't see any difference. So, what exactly is the difference and why do we need feed_dict?

Response:
In a tensorflow model you can define a placeholder such as x = tf.placeholder(tf.float32), then you will use x in your model. For example, I define a simple set of operations as:
```
x = tf.placeholder(tf.float32)
y = x * 42
```
Now when I ask tensorflow to compute y, it's clear that y depends on x.
```
with tf.Session() as sess:
  sess.run(y)
```
This will produce an error because I did not give it a value for x. In this case, because x is a placeholder, if it gets used in a computation you must pass it in via feed_dict. If you don't it's an error. Let's fix that:
```
with tf.Session() as sess:
  sess.run(y, feed_dict={x: 2})
```
The result this time will be 84. Great. Now let's look at a trivial case where feed_dict is not needed:
```
x = tf.constant(2)
y = x * 42
```
Now there are no placeholders (x is a constant) and so nothing needs to be fed to the model. This works now:
```
with tf.Session() as sess:
  sess.run(y)
```


Question:
TensorFlow Object Detection API Weird Behavior I was playing with TensorFlow's brand new Object Detection API and decided to train it on some other publicly available datasets. I happened to stumble upon this grocery dataset which consists of images of various brands of cigarette boxes on the supermarket shelf along with a text file which lists out the bounding boxes of each cigarette box in each image. 10 major brands have been labeled in the dataset and all other brands fall into the 11th "miscellaneous" category. I followed their tutorial and managed to train the model on this dataset. Due to limitations on processing power, I used only a third of the dataset and performed a 70:30 split for training and testing data. I used the faster_rcnn_resnet101 model. All parameters in my config file are the same as the default parameters provided by TF. After 16491 global steps, I tested the model on some images but I am not too happy with the results - Failed to detect the Camels in top-shelf whereas it detects the product in other images Why does it fail to detect the Marlboros in the top row? Another issue I had is that the model never detected any other label except for label 1 Doesn't detected a crop instance of the product from the training data It detects cigarette boxes with 99% confidence even in negative images! Can somebody help me with what is going wrong? What can I do to improve the accuracy? And why does it detect all products to belong in category 1 even though I have mentioned that there are 11 classes in total? Edit Added my label map:
```
item {
  id: 1
  name: '1'
}

item {
  id: 2
  name: '2'
}

item {
  id: 3
  name: '3'
}

item {
  id: 4
  name: '4'
}

item {
  id: 5
  name: '5'
}

item {
  id: 6
  name: '6'
}

item {
  id: 7
  name: '7'
}

item {
  id: 8
  name: '8'
}

item {
  id: 9
  name: '9'
}

item {
  id: 10
  name: '10'
}

item {
  id: 11
  name: '11'
}
```

Response:
So I think I figured out what is going on. I did some analysis on the dataset and found out that it is skewed towards objects of category 1. This is the frequency distribution of the each category from 1 to 11 (in 0 based indexing)
```
0 10440
1 304
2 998
3 67
4 412
5 114
6 190
7 311
8 195
9 78
10 75
```
I guess the model is hitting a local minima where just labelling everything as category 1 is good enough. About the problem of not detecting some boxes : I tried training again, but this time I didn't differentiate between brands. Instead, I tried to teach the model what a cigarette box is. It still wasn't detecting all the boxes. Then I decided to crop the input image and provide that as an input. Just to see if the results improve and it did! It turns out that the dimensions of the input image were much larger than the 600 x 1024 that is accepted by the model. So, it was scaling down these images to 600 x 1024 which meant that the cigarette boxes were losing their details :) So, I decided to test the original model which was trained on all classes on cropped images and it works like a charm :) This was the output of the model on the original image This is the output of the model when I crop out the top left quarter and provide it as input. Thanks everyone who helped! And congrats to the TensorFlow team for an amazing job for the API :) Now everybody can train object detection models!


Question:
Running trained tensorflow model in C++ I have trained an image classification network in python using tensorflow. The trained model was saved as a .pb. Now, I want to test the model, I need this to be done in C++. I had used numpy in manipulating and handling data. During training phase the image is passed in as a numpy array. The image is stretched out as a 1D array and the class label is prepended to this array. I'm confused as to how to pass the image data while running the model in C++, where numpy isn't available to me. I use numpy operations to manipulate and handle the data. In what format should I pass in the data if I have to execute it in C++. Below is how I train and save my model
```
def trainModel(data):
    global_step = tf.Variable(0, name='global_step', trainable=False)
    X, y,keep_prob = modelInputs((741, 620, 1),4)
    logits = cnnModel(X,keep_prob)
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y), name="cost")
    optimizer = tf.train.AdamOptimizer(.0001, name='Adam').minimize(cost)
    prediction = tf.argmax(logits, 1, name="prediction")
    correct_pred = tf.equal(prediction, tf.argmax(y, 1), name="correct_pred")
    accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        saver = tf.train.Saver()
        batch_size = 30
        for e in range(11):
            batch_x, batch_y = data.next_batch(batch_size)
            batch_y = batch_y.astype('int32')
            x = np.reshape(batch_x, [batch_size, 741, 620, 1])
            labels = np.zeros(shape=(batch_size,4))
            labels[np.arange(len(labels)),batch_y]=1
            sess.run(optimizer, feed_dict={X: x, y: labels,keep_prob:0.5})
            global_step.assign(e).eval()
        saver.save(sess, './my_test_model',global_step=global_step)
```
*741x620 is the size of the image

Response:
Instructions for using a graph in C++ can be found here. Here is some code to use your image as input:
```
tensorflow::Tensor keep_prob = tensorflow::Tensor(tensorflow::DT_FLOAT, tensorflow::TensorShape());
keep_prob.scalar<float>()() = 1.0;

tensorflow::Tensor input_tensor(tensorflow::DT_FLOAT, tensorflow::TensorShape({1,height,width,depth}));
auto input_tensor_mapped = input_tensor.tensor<float, 4>();
const float * source_data = (float*) img.data;  // here img is an opencv image, but if it's just a float array this code is very easy to adapt
// copying the image data into the corresponding tensor
for (int y = 0; y < height; ++y) {
    const float* source_row = source_data + (y * width * depth);
    for (int x = 0; x < width; ++x) {
        const float* source_pixel = source_row + (x * depth);
        for (int c = 0; c < depth; ++c) {
            const float* source_value = source_pixel + c;
            input_tensor_mapped(0, y, x, c) = *source_value;
        }
    }
}
std::vector<tensorflow::Tensor> finalOutput;

tensorflow::Status run_status = this->tf_session->Run({{InputName,input_tensor},
                                                       {dropoutPlaceHolderName, keep_prob}},
                                                      {OutputName},
                                                      {},
                                                      &finalOutput);
```


Question: TensorFlow: how to log GPU memory (VRAM) utilization? TensorFlow always (pre-)allocates all free memory (VRAM) on my graphics card, which is ok since I want my simulations to run as fast as possible on my workstation. However, I would like to log how much memory (in sum) TensorFlow really uses. Additionally it would be really nice, if I could also log how much memory single tensors use. This information is important to measure and compare the memory size that different ML/AI architectures need. Any tips?

Response:
Update, can use TensorFlow ops to query allocator:
```
# maximum across all sessions and .run calls so far
sess.run(tf.contrib.memory_stats.MaxBytesInUse())
# current usage
sess.run(tf.contrib.memory_stats.BytesInUse())
```
Also you can get detailed information about session.run call including all memory being allocations during run call by looking at RunMetadata. IE something like this
```
run_metadata = tf.RunMetadata()
sess.run(c, options=tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE, output_partition_graphs=True), run_metadata=run_metadata)
```
Here's an end-to-end example -- take column vector, row vector and add them to get a matrix of additions:
```
import tensorflow as tf

no_opt = tf.OptimizerOptions(opt_level=tf.OptimizerOptions.L0,
                             do_common_subexpression_elimination=False,
                             do_function_inlining=False,
                             do_constant_folding=False)
config = tf.ConfigProto(graph_options=tf.GraphOptions(optimizer_options=no_opt),
                        log_device_placement=True, allow_soft_placement=False,
                        device_count={"CPU": 3},
                        inter_op_parallelism_threads=3,
                        intra_op_parallelism_threads=1)
sess = tf.Session(config=config)

with tf.device("cpu:0"):
    a = tf.ones((13, 1))
with tf.device("cpu:1"):
    b = tf.ones((1, 13))
with tf.device("cpu:2"):
    c = a+b

sess = tf.Session(config=config)
run_metadata = tf.RunMetadata()
sess.run(c, options=tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE, output_partition_graphs=True), run_metadata=run_metadata)
with open("/tmp/run2.txt", "w") as out:
  out.write(str(run_metadata))
```
If you open run.txt you'll see messages like this:
```
node_name: "ones"

      allocation_description {
        requested_bytes: 52
        allocator_name: "cpu"
        ptr: 4322108320
      }
  ....

  node_name: "ones_1"

      allocation_description {
        requested_bytes: 52
        allocator_name: "cpu"
        ptr: 4322092992
      }
  ...
  node_name: "add"
      allocation_description {
        requested_bytes: 676
        allocator_name: "cpu"
        ptr: 4492163840
```
So here you can see that a and b allocated 52 bytes each (13*4), and the result allocated 676 bytes.


Question:
TensorFlow: cast a float64 tensor to float32 I am trying to use: train = optimizer.minimize(loss) but the standard optimizers do not work with tf.float64. Therefore I want to truncate my loss from tf.float64 to only tf.float32.
```
Traceback (most recent call last):
  File "q4.py", line 85, in <module>
    train = optimizer.minimize(loss)
  File "/Library/Python/2.7/site-packages/tensorflow/python/training/optimizer.py", line 190, in minimize
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/Library/Python/2.7/site-packages/tensorflow/python/training/optimizer.py", line 229, in compute_gradients
    self._assert_valid_dtypes([loss])
  File "/Library/Python/2.7/site-packages/tensorflow/python/training/optimizer.py", line 354, in _assert_valid_dtypes
    dtype, t.name, [v for v in valid_dtypes]))
ValueError: Invalid type tf.float64 for Add_1:0, expected: [tf.float32].
```

Response:
The short answer is that you can convert a tensor from tf.float64 to tf.float32 using the tf.cast() op:
```
loss = tf.cast(loss, tf.float32)
```
The longer answer is that this will not solve all of your problems with the optimizers. (The lack of support for tf.float64 is a known issue.) The optimizers require that all of the tf.Variable objects that you are trying to optimize must also have type tf.float32.


Question:
Neural network for square (x^2) approximation I made a simple module that should figure out the relationship between input and output numbers, in this case, x and x squared. The code in Python:
```
import numpy as np
import tensorflow as tf

# TensorFlow only log error messages.
tf.logging.set_verbosity(tf.logging.ERROR)

features = np.array([-10, -9, -8, -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8,
                    9, 10], dtype = float)
labels = np.array([100, 81, 64, 49, 36, 25, 16, 9, 4, 1, 0, 1, 4, 9, 16, 25, 36, 49, 64,
                    81, 100], dtype = float)

model = tf.keras.Sequential([
    tf.keras.layers.Dense(units = 1, input_shape = [1])
])

model.compile(loss = "mean_squared_error", optimizer = tf.keras.optimizers.Adam(0.0001))
model.fit(features, labels, epochs = 50000, verbose = False)
print(model.predict([4, 11, 20]))
```
I tried a different number of units, and adding more layers, and even using the relu activation function, but the results were always wrong. It works with other relationships like x and 2x. What is the problem here?

Response:
You are making two very basic mistakes: Your ultra-simple model (a single-layer network with a single unit) hardly qualifies as a neural network at all, let alone a "deep learning" one (as your question is tagged) Similarly, your dataset (just 20 samples) is also ultra-small It is certainly understood that neural networks need to be of some complexity if they are to solve problems even as "simple" as x*x; and where they really shine is when fed with large training datasets. The methodology when trying to solve such function approximations is not to just list the (few possible) inputs and then fed to the model, along with the desired outputs; remember, NNs learn through examples, and not through symbolic reasoning. And the more examples the better. What we usually do in similar cases is to generate a large number of examples, which we subsequently feed to the model for training. Having said that, here is a rather simple demonstration of a 3-layer neural network in Keras for approximating the function x*x, using as input 10,000 random numbers generated in [-50, 50]:
```
import numpy as np
import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import Adam
from keras import regularizers
import matplotlib.pyplot as plt

model = Sequential()
model.add(Dense(8, activation='relu', kernel_regularizer=regularizers.l2(0.001), input_shape = (1,)))
model.add(Dense(8, activation='relu', kernel_regularizer=regularizers.l2(0.001)))
model.add(Dense(1))

model.compile(optimizer=Adam(),loss='mse')

# generate 10,000 random numbers in [-50, 50], along with their squares
x = np.random.random((10000,1))*100-50
y = x**2

# fit the model, keeping 2,000 samples as validation set
hist = model.fit(x,y,validation_split=0.2,
             epochs= 15000,
             batch_size=256)

# check some predictions:
print(model.predict([4, -4, 11, 20, 8, -5]))
# result:
[[ 16.633354]
 [ 15.031291]
 [121.26833 ]
 [397.78638 ]
 [ 65.70035 ]
 [ 27.040245]]
```
Well, not that bad! Remember that NNs are function approximators: we should expect them neither to exactly reproduce the functional relationship nor to "know" that the results for 4 and -4 should be identical. Let's generate some new random data in [-50,50] (remember, for all practical purposes, these are unseen data for the model) and plot them, along with the original ones, to get a more general picture:
```
plt.figure(figsize=(14,5))
plt.subplot(1,2,1)
p = np.random.random((1000,1))*100-50 # new random data in [-50, 50]
plt.plot(p,model.predict(p), '.')
plt.xlabel('x')
plt.ylabel('prediction')
plt.title('Predictions on NEW data in [-50,50]')

plt.subplot(1,2,2)
plt.xlabel('x')
plt.ylabel('y')
plt.plot(x,y,'.')
plt.title('Original data')
```
Result: Well, it arguably does look like a good approximation indeed... You could also take a look at this thread for a sine approximation. The last thing to keep in mind is that, although we did get a decent approximation even with our relatively simple model, what we should not expect is extrapolation, i.e. good performance outside [-50, 50]; for details, see my answer in Is deep learning bad at fitting simple non linear functions outside training scope?


Question: Deep neural network skip connection implemented as summation vs concatenation? [closed] Closed. This question needs to be more focused. It is not currently accepting answers. Want to improve this question? Guide the asker to update the question so it focuses on a single, specific problem. Narrowing the question will help others answer the question concisely. You may edit the question if you feel you can improve it yourself. If edited, the question will be reviewed and might be reopened. Closed 7 years ago. Improve this question In deep neural network, we can implement the skip connections to help: Solve problem of vanishing gradient, training faster The network learns a combination of low level and high level features Recover info loss during downsampling like max pooling. https://medium.com/@mikeliao/deep-layer-aggregation-combining-layers-in-nn-architectures-2744d29cab8 However, i read some source code, some implemented skip connections as concatenation, some as summation. So my question is what are the benefits of each of these implementations?

Response: Basically, the difference relies on the different way in which the final layer is influenced by middle features. Standard architectures with skip-connection using element-wise summation (e.g. ResNet) can be viewed as an iterative estimation procedure to some extent (see for instance this work), where the features are refined through the various layers of the network. The main benefits of this choice are that it works and is a compact solution (it keeps the number of features fixed across a block). Architectures with concatenated skip-connections (e.g. DenseNet), allow the subsequent layers to re-use middle representations, maintaining more information which can lead to better performances. Apart from the feature re-use, another consequence is the implicit deep supervision (as in this work) which allow better gradient propagation across the network, especially for deep ones (in fact it has been used for the Inception architecture). Obviously, if not properly designed, concatenating features can lead to an exponential growth of the parameters (this explains, in part, the hierarchical aggregation used in the work you pointed out) and, depending on the problem, using a lot of information could lead to overfitting.


Question:
Confused about conv2d_transpose I'm getting this error message when using conv2d_transpose:
```
W tensorflow/core/common_runtime/executor.cc:1102] 0x7fc81f0d6250 Compute status: Invalid argument: Conv2DBackpropInput: Number of rows of out_backprop doesn't match computed: actual = 32, computed = 4
 [[Node: generator/g_h1/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, padding="SAME", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/cpu:0"](generator/g_h1/conv2d_transpose/output_shape, generator/g_h1/w/read, _recv_l_0)]]
```
However, it occurs after the graph is built while compiling the loss function (Adam). Any ideas on what would cause this? I suspect it's related to the input dimensions but I'm not sure exactly why. Full error: https://gist.github.com/jimfleming/75d88e888044615dd6e3 Relevant code:
```
# l shape: [batch_size, 32, 32, 4]

output_shape = [self.batch_size, 8, 8, 128]
filter_shape = [7, 7, 128, l.get_shape()[-1]]
strides = [1, 2, 2, 1]
with tf.variable_scope("g_h1"):
    w = tf.get_variable('w', filter_shape, initializer=tf.random_normal_initializer(stddev=0.02))
    h1 = tf.nn.conv2d_transpose(l, w, output_shape=output_shape, strides=strides, padding='SAME')
    h1 = tf.nn.relu(h1)

output_shape = [self.batch_size, 16, 16, 128]
filter_shape = [7, 7, 128, h1.get_shape()[-1]]
strides = [1, 2, 2, 1]
with tf.variable_scope("g_h2"):
    w = tf.get_variable('w', filter_shape, initializer=tf.random_normal_initializer(stddev=0.02))
    h2 = tf.nn.conv2d_transpose(h1, w,output_shape=output_shape,  strides=strides, padding='SAME')
    h2 = tf.nn.relu(h2)

output_shape = [self.batch_size, 32, 32, 3]
filter_shape = [5, 5, 3, h2.get_shape()[-1]]
strides = [1, 2, 2, 1]
with tf.variable_scope("g_h3"):
    w = tf.get_variable('w', filter_shape, initializer=tf.random_normal_initializer(stddev=0.02))
    h3 = tf.nn.conv2d_transpose(h2, w,output_shape=output_shape,  strides=strides, padding='SAME')
    h3 = tf.nn.tanh(h3)
```

Response:
Thanks for the question! You're exactly right---the problem is that the input and output dimensions being passed to tf.nn.conv2d_transpose don't agree. (The error may be detected when computing gradients, but the gradient computation isn't the problem.) Let's look at just the first part of your code, and simplify it a little bit:
```
sess = tf.Session()
batch_size = 3
output_shape = [batch_size, 8, 8, 128]
strides = [1, 2, 2, 1]

l = tf.constant(0.1, shape=[batch_size, 32, 32, 4])
w = tf.constant(0.1, shape=[7, 7, 128, 4])

h1 = tf.nn.conv2d_transpose(l, w, output_shape=output_shape, strides=strides, padding='SAME')
print sess.run(h1)
```
I replaced the variables with constants --- it's easier to see what's going on. If you try to run this code, you get a similar error:
```
InvalidArgumentError: Conv2DCustomBackpropInput: Size of out_backprop doesn't match computed: actual = 32, computed = 4
  [[Node: conv2d_transpose_6 = Conv2DBackpropInput[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/cpu:0"](conv2d_transpose_6/output_shape, Const_25, Const_24)]]
```
Now, the error is a little misleading --- it talks about the 'out_backprop' argument to 'Conv2DCustomBackpropInput'. The key is that tf.nn.conv2d_transpose is actually just the gradient of tf.nn.conv2d, so Tensorflow uses the same code internally (Conv2DCustomBackpropInput) to compute the gradient of tf.nn.conv2d and to compute tf.nn.conv2d_transpose. The error means that the 'output_shape' you requested is not possible, given the shapes of 'l' and 'w'. Since tf.nn.conv2d_transpose is the backward (gradient) counterpart of tf.nn.conv2d, one way to see what the correct shapes should be is to use the corresponding forward operation:
```
output = tf.constant(0.1, shape=output_shape)
expected_l = tf.nn.conv2d(output, w, strides=strides, padding='SAME')
print expected_l.get_shape()
# Prints (3, 4, 4, 4)
```
That is, in the forward direction, if you provided a tensor of shape 'output_shape', you would get out a tensor of shape (3, 4, 4, 4). So one way to fix the problem is to change the shape of 'l' to (3, 4, 4, 4); if you change the code above to:
```
l = tf.constant(0.1, shape=[batch_size, 4, 4, 4])
```
everything works fine. In general, try using tf.nn.conv2d to get a feel for what the relationship between the tensor shapes is. Since tf.nn.conv2d_transpose is its backward counterpart, it has the same relationship between input, output and filter shapes (but with the roles of the input and output reversed.) Hope that helps!


Question:
expected ndim=3, found ndim=2 I'm new with Keras and I'm trying to implement a Sequence to Sequence LSTM. Particularly, I have a dataset with 9 features and I want to predict 5 continuous values. I split the training and the test set and their shape are respectively:
```
X TRAIN (59010, 9)

X TEST (25291, 9)

Y TRAIN (59010, 5)

Y TEST (25291, 5)
```
The LSTM is extremely simple at the moment:
```
model = Sequential()
model.add(LSTM(100, input_shape=(9,), return_sequences=True))
model.compile(loss="mean_absolute_error", optimizer="adam", metrics= ['accuracy'])

history = model.fit(X_train,y_train,epochs=100, validation_data=(X_test,y_test))
```
But I have the following error: ValueError: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=2 Can anyone help me?

Response:
LSTM layer expects inputs to have shape of (batch_size, timesteps, input_dim). In keras you need to pass (timesteps, input_dim) for input_shape argument. But you are setting input_shape (9,). This shape does not include timesteps dimension. The problem can be solved by adding extra dimension to input_shape for time dimension. E.g adding extra dimension with value 1 could be simple solution. For this you have to reshape input dataset( X Train) and Y shape. But this might be problematic because the time resolution is 1 and you are feeding length one sequence. With length one sequence as input, using LSTM does not seem the right option.
```
x_train = x_train.reshape(-1, 1, 9)
x_test  = x_test.reshape(-1, 1, 9)
y_train = y_train.reshape(-1, 1, 5)
y_test = y_test.reshape(-1, 1, 5)

model = Sequential()
model.add(LSTM(100, input_shape=(1, 9), return_sequences=True))
model.add(LSTM(5, input_shape=(1, 9), return_sequences=True))
model.compile(loss="mean_absolute_error", optimizer="adam", metrics= ['accuracy'])

history = model.fit(X_train,y_train,epochs=100, validation_data=(X_test,y_test))
```


Question:
Is Intel based graphic card compatible with tensorflow/GPU? Is this graphic card compatible with tensorflow/GPU ?
```
*-display
   description: VGA compatible controller
   product: Haswell-ULT Integrated Graphics Controller
   vendor: Intel Corporation
   physical id: 2
   bus info: pci@0000:00:02.0
   version: 09
   width: 64 bits
   clock: 33MHz
   capabilities: msi pm vga_controller bus_master cap_list rom
   configuration: driver=i915 latency=0
   resources: irq:44 memory:c2000000-c23fffff memory:b0000000-bfffffff ioport:7000(size=64)
```

Response: At the moment no. Only Nvidia GPUs and (intel/amd) CPU versions are available. They are working on an openCL compatible version of tensorflow that would be compatible and you can follow its progress here


Question:
tensorflow: Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob` I get this warning most of the time when i define a model using Keras. It seems to somehow come from tensorflow though:
```
WARNING:tensorflow:From C:\Users\lenik\AppData\Local\Programs\Python\Python37\lib\site-packages\keras\backend\tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
```
Is this warning something to worry about? If yes, how do i solve this problem?

Response: This depreciation warning is due to the Dropout layer in tf.keras.layers.Dropout.To avoid this warning, you need to clearly specify rate= in Dropout as: Dropout(rate=0.2). Earlier it was keep_prob and it is now deprecated to rate i.e. rate = 1-keep_prob. For more, you can check out this tensorflow documentation.


Question:
What is tf.nn.max_pool's ksize parameter used for? In the definition of tf.nn.max_pool, what is ksize used for?
```
tf.nn.max_pool(value, ksize, strides, padding, data_format='NHWC', name=None)

Performs the max pooling on the input.

Args:

value: A 4-D Tensor with shape [batch, height, width, channels] and type    tf.float32.
ksize: A list of ints that has length >= 4. The size of the window for each dimension of the input tensor.
```
For instance, if an input value is of tensor : [1, 64, 64, 3] and ksize=3.what does that mean?

Response: The documentation states: ksize: A list of ints that has length >= 4. The size of the window for each dimension of the input tensor. In general for images, your input is of shape [batch_size, 64, 64, 3] for an RGB image of 64x64 pixels. The kernel size ksize will typically be [1, 2, 2, 1] if you have a 2x2 window over which you take the maximum. On the batch size dimension and the channels dimension, ksize is 1 because we don't want to take the maximum over multiple examples, or over multiples channels.


Question: Why can GPU do matrix multiplication faster than CPU? I've been using GPU for a while without questioning it but now I'm curious. Why can GPU do matrix multiplication much faster than CPU? Is it because of parallel processing? But I didn't write any parallel processing code. Does it do it automatically by itself? Any intuition / high-level explanation will be appreciated!

Response:
How do you parallelize the computations? GPU's are able to do a lot of parallel computations. A Lot more than a CPU could do. Look at this example of vector addition of let's say 1M elements. Using a CPU let's say you have 100 maximum threads you can run : (100 is lot more but let's assume for a while) In a typical multi-threading example let's say you parallelized additions on all threads. Here is what I mean by it :
```
c[0] = a[0] + b[0] # let's do it on thread 0
c[1] = a[1] + b[1] # let's do it on thread 1
c[101] = a[101] + b[101] # let's do it on thread 1
```
We are able to do it because value of c[0], doesn't depend upon any other values except a[0] and b[0]. So each addition is independent of others. Hence, we were able to easily parallelize the task. As you see in above example that simultaneously all the addition of 100 different elements take place saving you time. In this way it takes 1M/100 = 10,000 steps to add all the elements. How Efficient does GPU Parallelizes? Now consider today's GPU with about 2048 threads, all threads can independently do 2048 different operations in constant time. Hence giving a boost up. In your case of matrix multiplication. You can parallelize the computations, Because GPU have much more threads and in each thread you have multiple blocks. So a lot of computations are parallelized, resulting quick computations. But I didn't write any parallel processing for my GTX1080! Does it do it by itself? Almost all the framework for machine learning uses parallelized implementation of all the possible operations. This is achieved by CUDA programming, NVIDIA API to do parallel computations on NVIDIA GPU's. You don't write it explicitly, it's all done at low level, and you do not even get to know. Yes it doesn't mean that a C++ program you wrote will automatically be parallelized, just because you have a GPU. No, you need to write it using CUDA, only then it will be parallelized, but most programming framework have it, So it is not required from your end.


Question:
what does x = tf.placeholder(tf.float32, [None, 784]) means? I know basic use for tf.placeholder:
```
x = tf.placeholder(tf.float32, shape=(1024, 1024))
y = tf.matmul(x, x)

with tf.Session() as sess:
   print(sess.run(y))  # ERROR: will fail because x was not fed.

   rand_array = np.random.rand(1024, 1024)
   print(sess.run(y, feed_dict={x: rand_array}))  # Will succeed.
```
I know the second parameter is about shape. However I don't know what is that mean when the first one is None in the shape. ex:[None,784].

Response: From the tutorial: Deep MNIST for Experts Here we assign it a shape of [None, 784], where 784 is the dimensionality of a single flattened 28 by 28 pixel MNIST image, and None indicates that the first dimension, corresponding to the batch size, can be of any size.


Question:
How to feed a placeholder? I am trying to implement a simple feed forward network. However, I can't figure out how to feed a Placeholder. This example:
```
import tensorflow as tf

num_input  = 2
num_hidden = 3
num_output = 2

x  = tf.placeholder("float", [num_input, 1])
W_hidden = tf.Variable(tf.zeros([num_hidden, num_input]))
W_out    = tf.Variable(tf.zeros([num_output, num_hidden]))
b_hidden = tf.Variable(tf.zeros([num_hidden]))
b_out    = tf.Variable(tf.zeros([num_output]))

h = tf.nn.softmax(tf.matmul(W_hidden,x) + b_hidden)

sess = tf.Session()

with sess.as_default():
    print h.eval()
```
Gives me the following error:
```
...
    results = self._do_run(target_list, unique_fetch_targets, feed_dict_string)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 419, in _do_run
    e.code)
tensorflow.python.framework.errors.InvalidArgumentError: You must feed a value for placeholder tensor 'Placeholder' with dtype float and shape dim { size: 2 } dim { size: 1 }
     [[Node: Placeholder = Placeholder[dtype=DT_FLOAT, shape=[2,1], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op u'Placeholder', defined at:
  File "/home/sfalk/workspace/SemEval2016/java/semeval2016-python/slot1_tf.py", line 8, in <module>
    x  = tf.placeholder("float", [num_input, 1])
  ...
```
I have tried
```
tf.assign([tf.Variable(1.0), tf.Variable(1.0)], x)
tf.assign([1.0, 1.0], x)
```
but that does not work apparently.

Response:
To feed a placeholder, you use the feed_dict argument to Session.run() (or Tensor.eval()). Let's say you have the following graph, with a placeholder:
```
x = tf.placeholder(tf.float32, shape=[2, 2])
y = tf.constant([[1.0, 1.0], [0.0, 1.0]])
z = tf.matmul(x, y)
```
If you want to evaluate z, you must feed a value for x. You can do this as follows:
```
sess = tf.Session()
print sess.run(z, feed_dict={x: [[3.0, 4.0], [5.0, 6.0]]})
```
For more information, see the documentation on feeding.


Question:
Looping over a tensor I am trying to process a tensor of variable size, in a python way that would be something like:
```
# X is of shape [m, n]
for x in X:
    process(x)
```
I have tried to use tf.scan, the thing is that I want to process every sub-tensor, so I have tried to use a nested scan, but I was enable to do it, because tf.scan work with the accumulator, if not found it will take the first entry of the elems as initializer, which I don't want to do. As an example, suppose I want to add one to every element of my tensor (this is just an example), and I want to process it element by element. If I run the code bellow, I will only have one added to a sub-tensor, because scan consider the first tensor as initializer, along with the first element of every sub-tensor.
```
import numpy as np
import tensorflow as tf

batch_x = np.random.randint(0, 10, size=(5, 10))
x = tf.placeholder(tf.float32, shape=[None, 10])

def inner_loop(x_in):
    return tf.scan(lambda _, x_: x_ + 1, x_in)

outer_loop = tf.scan(lambda _, input_: inner_loop(input_), x, back_prop=True)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    rs = sess.run(outer_loop, feed_dict={x: batch_x})
```
Any suggestions ?

Response:
To loop over a tensor you could try tf.unstack Unpacks the given dimension of a rank-R tensor into rank-(R-1) tensors. So adding 1 to each tensor would look something like:
```
import tensorflow as tf
x = tf.placeholder(tf.float32, shape=(None, 10))
x_unpacked = tf.unstack(x) # defaults to axis 0, returns a list of tensors

processed = [] # this will be the list of processed tensors
for t in x_unpacked:
    # do whatever
    result_tensor = t + 1
    processed.append(result_tensor)

output = tf.concat(processed, 0)

with tf.Session() as sess:
    print(sess.run([output], feed_dict={x: np.zeros((5, 10))}))
```
Obviously you can further unpack each tensor from the list to process it, down to single elements. To avoid lots of nested unpacking though, you could maybe try flattening x with tf.reshape(x, [-1]) first, and then loop over it like
```
flattened_unpacked = tf.unstack(tf.reshape(x, [-1])
for elem in flattened_unpacked:
    process(elem)
```
In this case elem is a scalar.


Question: Why was Eigen chosen for TensorFlow? [closed] Closed. This question is seeking recommendations for software libraries, tutorials, tools, books, or other off-site resources. It does not meet Stack Overflow guidelines. It is not currently accepting answers. We don’t allow questions seeking recommendations for software libraries, tutorials, tools, books, or other off-site resources. You can edit the question so it can be answered with facts and citations. Closed 8 years ago. Improve this question The TensorFlow white paper mentions that Eigen is used. Are there public explanations for how Eigen was chosen, and are they motivation for using Eigen in TensorFlow C++ op kernels?

Response: I think that one of the key feature that drove the use of Eigen in the first place is because Eigen features its own highly optimized matrix product kernels whereas all other competitors have to be linked to some BLAS libraries. Moreover, the code of Eigen's product kernel is C++ with easy access to low-level internal kernels, so it was 'easy' for them to tweak and extend it to match their needs. This way Google has been able to develop the Tensor module with high CPU performance in a pure header-only fashion. The support for CUDA and now OpenCL via SyCL came later, those are not intrinsic features of Eigen that drove the initial choice.


Question:
TensorFlow strings: what they are and how to work with them When I read file with tf.read_file I get something with type tf.string. Documentation says only that it is "Variable length byte arrays. Each element of a Tensor is a byte array." (https://www.tensorflow.org/versions/r0.10/resources/dims_types.html). I have no idea how to interpret this. I can do nothing with this type. In usual python you can get elements by index like my_string[:4], but when I run following code I get an error.
```
import tensorflow as tf
import numpy as np

x = tf.constant("This is string")
y = x[:4]

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
result = sess.run(y)
print result
```
It says
```
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.py", line 621, in assert_has_rank
    raise ValueError("Shape %s must have rank %d" % (self, rank))
ValueError: Shape () must have rank 1
```
Also I cannot convert my string to tf.float32 tensor. It is .flo file and it has magic header "PIEH". This numpy code successfuly convert such header into number (see example here https://stackoverflow.com/a/28016469/4744283) but I can't do that with tensorflow. I tried tf.string_to_number(string, out_type=tf.float32) but it says
```
tensorflow.python.framework.errors.InvalidArgumentError: StringToNumberOp could not correctly convert string: PIEH
```
So, what string is? What it's shape is? How can I at least get part of the string? I suppose that if I can get part of it I can just skip "PIEH" part. UPD: I forgot to say that tf.slice(string, [0], [4]) also doesn't work with same error.

Response:
Unlike Python, where a string can be treated as a list of characters for the purposes of slicing and such, TensorFlow's tf.strings are indivisible values. For instance, x below is a Tensor with shape (2,) whose each element is a variable length string.
```
x = tf.constant(["This is a string", "This is another string"])
```
However, to achieve what you want, TensorFlow provides the tf.decode_raw operator. It takes a tf.string tensor as input, but can decode the string into any other primitive data type. For instance, to interpret the string as a tensor of characters, you can do the following :
```
x = tf.constant("This is string")
x = tf.decode_raw(x, tf.uint8)
y = x[:4]
sess = tf.InteractiveSession()
print(y.eval())
# prints [ 84 104 105 115]
```


Question:
How to set the input of a Keras layer with a Tensorflow tensor? In my previous question, I used Keras' Layer.set_input() to connect my Tensorflow pre-processing output tensor to my Keras model's input. However, this method has been removed after Keras version 1.1.1. How can I achieve this in newer Keras versions? Example:
```
# Tensorflow pre-processing
raw_input = tf.placeholder(tf.string)
### some TF operations on raw_input ###
tf_embedding_input = ...    # pre-processing output tensor

# Keras model
model = Sequential()
e = Embedding(max_features, 128, input_length=maxlen)

### THIS DOESN'T WORK ANYMORE ###
e.set_input(tf_embedding_input)
################################

model.add(e)
model.add(LSTM(128, activation='sigmoid'))
model.add(Dense(num_classes, activation='softmax'))
```

Response:
After you are done with pre-processing, You can add the tensor as input layer by calling tensor param of Input So in your case:
```
tf_embedding_input = ...    # pre-processing output tensor

# Keras model
model = Sequential()
model.add(Input(tensor=tf_embedding_input))
model.add(Embedding(max_features, 128, input_length=maxlen))
```


Question:
How to deal with UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape I am having the following warning in Tensorflow: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. The reason I am getting this is:
```py
import tensorflow as tf
# Flatten batch elements to rank-2 tensor where 1st max_length rows
    #belong to first batch element and so forth
all_timesteps = tf.reshape(raw_output, [-1, n_dim])  # (batch_size*max_length, n_dim)
# Indices to last element of each sequence.
# Index to first element is the sequence order number times max
    #sequence length.
# Index to last element is the index to first element plus sequence
    #length.
row_inds = tf.range(0, batch_size) * max_length + (seq_len - 1)
# Gather rows with indices to last elements of sequences
# http://stackoverflow.com/questions/35892412/tensorflow-dense-gradient-explanation
# This is due to gather returning IndexedSlice which is later
    #converted into a Tensor for gradient
# calculation.
last_timesteps = tf.gather(all_timesteps, row_inds)  # (batch_size,n_dim)
```
tf.gather is causing the issue. I have been ignoring it until now because my architectures were not really big. However, now, I have bigger architectures and a lot of data. I am facing Out of memory issues when training with batch sizes bigger than 10. I believe that dealing with this warning would allow me to fit my models inside the GPU. Please note that I am using Tensorflow 1.3.

Response:
I managed to solve the issue by using tf.dynnamic_partition instead of tf.gather . I replaced the above code like this:
```py
# Flatten batch elements to rank-2 tensor where 1st max_length rows belong to first batch element and so forth
all_timesteps = tf.reshape(raw_output, [-1, n_dim])  # (batch_size*max_length, n_dim)
# Indices to last element of each sequence.
# Index to first element is the sequence order number times max sequence length.
# Index to last element is the index to first element plus sequence length.
row_inds = tf.range(0, batch_size) * max_length + (seq_len - 1)
# Creating a vector of 0s and 1s that will specify what timesteps to choose.
partitions = tf.reduce_sum(tf.one_hot(row_inds, tf.shape(all_timesteps)[0], dtype='int32'), 0)
# Selecting the elements we want to choose.
last_timesteps = tf.dynamic_partition(all_timesteps, partitions, 2)  # (batch_size, n_dim)
last_timesteps = last_timesteps[1]
```


Question:
How to implement Tensorflow batch normalization in LSTM My current LSTM network looks like this.
```py
rnn_cell = tf.contrib.rnn.BasicRNNCell(num_units=CELL_SIZE)
init_s = rnn_cell.zero_state(batch_size=1, dtype=tf.float32)  # very first hidden state
outputs, final_s = tf.nn.dynamic_rnn(
    rnn_cell,              # cell you have chosen
    tf_x,                  # input
    initial_state=init_s,  # the initial hidden state
    time_major=False,      # False: (batch, time step, input); True: (time step, batch, input)
)

# reshape 3D output to 2D for fully connected layer
outs2D = tf.reshape(outputs, [-1, CELL_SIZE])
net_outs2D = tf.layers.dense(outs2D, INPUT_SIZE)

# reshape back to 3D
outs = tf.reshape(net_outs2D, [-1, TIME_STEP, INPUT_SIZE])
```
Usually, I apply tf.layers.batch_normalization as batch normalization. But I am not sure if this works in a LSTM network.
```py
b1 = tf.layers.batch_normalization(outputs, momentum=0.4, training=True)
d1 = tf.layers.dropout(b1, rate=0.4, training=True)

# reshape 3D output to 2D for fully connected layer
outs2D = tf.reshape(d1, [-1, CELL_SIZE])
net_outs2D = tf.layers.dense(outs2D, INPUT_SIZE)

# reshape back to 3D
outs = tf.reshape(net_outs2D, [-1, TIME_STEP, INPUT_SIZE])
```

Response: If you want to use batch norm for RNN (LSTM or GRU), you can check out this implementation , or read the full description from blog post. However, the layer-normalization has more advantage than batch norm in sequence data. Specifically, "the effect of batch normalization is dependent on the mini-batch size and it is not obvious how to apply it to recurrent networks" (from the paper Ba, et al. Layer normalization). For layer normalization, it normalizes the summed inputs within each layer. You can check out the implementation of layer-normalization for GRU cell: