Question:
Difference between `apply_gradients` and `minimize` of optimizer in tensorflow I am confused about the difference between apply_gradients and minimize of optimizer in tensorflow. For example,
```py
optimizer = tf.train.AdamOptimizer(1e-3)
grads_and_vars = optimizer.compute_gradients(cnn.loss)
train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)
```
and
```py
optimizer = tf.train.AdamOptimizer(1e-3)
train_op = optimizer.minimize(cnn.loss, global_step=global_step)
```
Are they the same indeed? If I want to decay the learning rate, can I use the following codes?
```py
global_step = tf.Variable(0, name="global_step", trainable=False)
starter_learning_rate = 1e-3
learning_rate = tf.train.exponential_decay(starter_learning_rate, global_step,
                                       100, FLAGS.decay_rate, staircase=True)
# Passing global_step to minimize() will increment it at each step.
learning_step = (
    optimizer = tf.train.AdamOptimizer(learning_rate)
    grads_and_vars = optimizer.compute_gradients(cnn.loss)
    train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)
)
```
Thanks for your help!

Response: You can easily know from the link : https://www.tensorflow.org/get_started/get_started (tf.train API part) that they actually do the same job. The difference it that: if you use the separated functions( tf.gradients, tf.apply_gradients), you can apply other mechanism between them, such as gradient clipping.


Question:
Higher validation accuracy, than training accurracy using Tensorflow and Keras [closed] Closed. This question is not about programming or software development. It is not currently accepting answers. This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered. Closed last year. The community reviewed whether to reopen this question last year and left it closed: Original close reason(s) were not resolved Improve this question I'm trying to use deep learning to predict income from 15 self reported attributes from a dating site. We're getting rather odd results, where our validation data is getting better accuracy and lower loss, than our training data. And this is consistent across different sizes of hidden layers. This is our model:
```py
for hl1 in [250, 200, 150, 100, 75, 50, 25, 15, 10, 7]:
    def baseline_model():
        model = Sequential()
        model.add(Dense(hl1, input_dim=299, kernel_initializer='normal', activation='relu', kernel_regularizer=regularizers.l1_l2(0.001)))
        model.add(Dropout(0.5, seed=seed))
        model.add(Dense(3, kernel_initializer='normal', activation='sigmoid'))

        model.compile(loss='categorical_crossentropy', optimizer='adamax', metrics=['accuracy'])
        return model

    history_logs = LossHistory()
    model = baseline_model()
    history = model.fit(X, Y, validation_split=0.3, shuffle=False, epochs=50, batch_size=10, verbose=2, callbacks=[history_logs])
```
And this is an example of the accuracy and losses: and . We've tried to remove regularization and dropout, which, as expected, ended in overfitting (training acc: ~85%). We've even tried to decrease the learning rate drastically, with similiar results. Has anyone seen similar results?

Response: This happens when you use Dropout, since the behaviour when training and testing are different. When training, a percentage of the features are set to zero (50% in your case since you are using Dropout(0.5)). When testing, all features are used (and are scaled appropriately). So the model at test time is more robust - and can lead to higher testing accuracies.


Question: How to convert one-hot encodings into integers? I have a numpy array data set with shape (100,10). Each row is a one-hot encoding. I want to transfer it into a nd-array with shape (100,) such that I transferred each vector row into a integer that denote the index of the nonzero index. Is there a quick way of doing this using numpy or tensorflow?

Response:
You can use numpy.argmax or tf.argmax. Example:
```
import numpy as np
a  = np.array([[0,1,0,0],[1,0,0,0],[0,0,0,1]])
print('np.argmax(a, axis=1): {0}'.format(np.argmax(a, axis=1)))
```
output:
```
np.argmax(a, axis=1): [1 0 3]
```
You may also want to look at sklearn.preprocessing.LabelBinarizer.inverse_transform.


Question:
Display MNIST image using matplotlib [duplicate] This question already has answers here: Show image from MNIST DataSet (6 answers) Closed 7 years ago. I am using tensorflow to import some MNIST input data. I followed this tutorial...https://www.tensorflow.org/get_started/mnist/beginners I am importing them as so...
```
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
```
I want to be able to display any of the images from the training set. I know the location of the images is mnist.train.images, so I try to access the first images and display it like so...
```
with tf.Session() as sess:
    #access first image
    first_image = mnist.train.images[0]

    first_image = np.array(first_image, dtype='uint8')
    pixels = first_image.reshape((28, 28))
    plt.imshow(pixels, cmap='gray')
```
I a attempt to convert the image to a 28 by 28 numpy array because I know that each image is 28 by 28 pixels. However, when I run the code all I get is the following... Clearly I am doing something wrong. When I print out the matrix, everything seems to look good, but I think I am incorrectly reshaping it.

Response:
Here is the complete code for showing image using matplotlib
```
from matplotlib import pyplot as plt
import numpy as np
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets('MNIST_data', one_hot = True)
first_image = mnist.test.images[0]
first_image = np.array(first_image, dtype='float')
pixels = first_image.reshape((28, 28))
plt.imshow(pixels, cmap='gray')
plt.show()
```


Question: What is the difference between Dataset.from_tensors and Dataset.from_tensor_slices? I have a dataset represented as a NumPy matrix of shape (num_features, num_examples) and I wish to convert it to TensorFlow type tf.Dataset. I am struggling trying to understand the difference between these two methods: Dataset.from_tensors and Dataset.from_tensor_slices. What is the right one and why? TensorFlow documentation (link) says that both method accept a nested structure of tensor although when using from_tensor_slices the tensor should have same size in the 0-th dimension.

Response:
from_tensors combines the input and returns a dataset with a single element:
```

```python
t = tf.constant([[1, 2], [3, 4]])
```

```python
ds = tf.data.Dataset.from_tensors(t)
```

```python
[x for x in ds]
#Output
#[<tf.Tensor: shape=(2, 2), dtype=int32, numpy=
# array([[1, 2],
#        [3, 4]], dtype=int32)>]
#```
#from_tensor_slices creates a dataset with a separate element for each row of the input tensor:
#```
```

```python
t = tf.constant([[1, 2], [3, 4]])
```

```python
ds = tf.data.Dataset.from_tensor_slices(t)
```

```python
[x for x in ds]
#Output
#[<tf.Tensor: shape=(2,), dtype=int32, numpy=array([1, 2], dtype=int32)>,
# <tf.Tensor: shape=(2,), dtype=int32, numpy=array([3, 4], dtype=int32)>]
#```
```


Question:
Where is gen_math_ops script in tensorflow? I read the source code of tensorflow on github and find gen_math_ops is imported.
```
from tensorflow.python.ops import gen_math_ops
```
However, i cannot find this script in the whole project and it's also not under the ops folder. search result No gen_math_ops under ops

Response: It's automatically generated by tf_gen_op_wrapper_* rules here. Also you can use ?? in your IPython notebook to find location


Question:
Error when trying to rename a pretrained model on tf.keras I trained two models in order to ensemble them, when I try to load them with this code:
```
from tensorflow.keras.models import load_model
  models=[]
  modelTemp=load_model('models/full.h5')
  modelTemp.name = "inception1"
  models.append(modelTemp)
```
error occur :
```
AttributeError: Can't set the attribute "name", likely because it conflicts with an existing read-only @property of the object. Please choose a different name.
```
full error message:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py in __setattr__(self, name, value)
   1968       try:
-> 1969         super(tracking.AutoTrackable, self).__setattr__(name, value)
   1970       except AttributeError:

AttributeError: can't set attribute

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
2 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py in __setattr__(self, name, value)
   1972             ('Can\'t set the attribute "{}", likely because it conflicts with '
   1973              'an existing read-only @property of the object. Please choose a '
-> 1974              'different name.').format(name))
   1975       return
   1976

AttributeError: Can't set the attribute "name", likely because it conflicts with an existing read-only @property of the object. Please choose a different name.
```

Response:
According to this question here on StackOverflow you need to use:
```
modelTemp._name = 'inception'
```


Question:
How to compute all second derivatives (only the diagonal of the Hessian matrix) in Tensorflow? I have a loss value/function and I would like to compute all the second derivatives with respect to a tensor f (of size n). I managed to use tf.gradients twice, but when applying it for the second time, it sums the derivatives across the first input (see second_derivatives in my code). Also I managed to retrieve the Hessian matrix, but I would like to only compute its diagonal to avoid extra-computation.
```
import tensorflow as tf
import numpy as np

f = tf.Variable(np.array([[1., 2., 0]]).T)
loss = tf.reduce_prod(f ** 2 - 3 * f + 1)

first_derivatives = tf.gradients(loss, f)[0]

second_derivatives = tf.gradients(first_derivatives, f)[0]

hessian = [tf.gradients(first_derivatives[i,0], f)[0][:,0] for i in range(3)]

model = tf.initialize_all_variables()
with tf.Session() as sess:
    sess.run(model)
    print "\nloss\n", sess.run(loss)
    print "\nloss'\n", sess.run(first_derivatives)
    print "\nloss''\n", sess.run(second_derivatives)
    hessian_value = np.array(map(list, sess.run(hessian)))
    print "\nHessian\n", hessian_value
```
My thinking was that tf.gradients(first_derivatives, f[0, 0])[0] would work to retrieve for instance the second derivative with respect to f_0 but it seems that tensorflow doesn't allow to derive from a slice of a tensor.

Response:
tf.gradients([f1,f2,f3],...) computes gradient of f=f1+f2+f3 Also, differentiating with respect to x[0] is problematic because x[0] refers to a new Slice node which is not an ancestor of your loss, so derivative with respect to it will be None. You could get around it by using pack to glue x[0], x[1], ... together into xx and have your loss depend on xx instead of x . An alternative is to use separate variables for individual components, in which case computing Hessian would look something like this.
```
def replace_none_with_zero(l):
  return [0 if i==None else i for i in l]

tf.reset_default_graph()

x = tf.Variable(1.)
y = tf.Variable(1.)
loss = tf.square(x) + tf.square(y)
grads = tf.gradients([loss], [x, y])
hess0 = replace_none_with_zero(tf.gradients([grads[0]], [x, y]))
hess1 = replace_none_with_zero(tf.gradients([grads[1]], [x, y]))
hessian = tf.pack([tf.pack(hess0), tf.pack(hess1)])
sess = tf.InteractiveSession()
sess.run(tf.initialize_all_variables())
print hessian.eval()
```
You'll see
```
[[ 2.  0.]
 [ 0.  2.]]
```


Question: What is the difference between tf.group and tf.control_dependencies? Aside from tf.control_dependencies being a context manager (i.e. used with Python with), what's the difference between tf.group and tf.control_dependencies? When should which be used? Is it that tf.group doesn't have any particular order of operations? I'd assume tf.group([op_1, op_2, op_3]) executes ops in the list's order, but maybe that's not the case? The docstring doesn't specify a behaviour.

Response:
If you look at the graphdef, the c=tf.group(a, b) produces the same graph as
```
with tf.control_dependencies([a, b]):
    c = tf.no_op()
```
There's no specific order in which ops will run, TensorFlow tries to execute operations as soon as it can (i.e. in parallel).


Question: Dynamically tile a tensor depending on the batch size I have a 1D tensor a that I want to stack/pack/tile into a 2D tensor like y=[a, a, a]. If I knew how many times I wanted it repeated, I could use tf.tile along with reshape. But I don't because the size is dependent on the batch size. The placeholder value is None which isn't a valid input. I know for tf.slice one can input -1 and let tensorflow figure it out, but I don't see how tensorflow could infer the correct size. I do have a tensor x that would be equal in shape to y, but I don't see a tile_like function. Any suggestions?

Response:
You can use tf.shape to find out the runtime shape of a tensor, and use it as the basis for the argument to tf.tile:
```
import tensorflow as tf
import numpy as np

x = tf.placeholder(tf.float32, shape=[None, 3])

y = tf.tile([2, 3], tf.shape(x)[0:1])

sess = tf.Session()
print(sess.run(y, feed_dict={x: np.zeros([11, 3])}))
```
I verified this code works with the Tensorflow 1.0 release candidiate. Hope that helps!


Question:
Tensorflow get all variables in scope I have some variables created within a certain scope like this:
```
with tf.variable_scope("my_scope"):
  createSomeVariables()
  ...
```
I then want to get the list of all the variables in "my_scope" so I can pass it to an optimizer. What is the right way to do this?

Response: I think you want tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='my_scope'). This will get all variables in a scope. To pass to an optimizer you do not want all variables you would just want the trainable variables. Those are also kept in a default collection, which is tf.GraphKeys.TRAINABLE_VARIABLES.


Question:
The added layer must be an instance of class Layer. Found:  I am new to machine learning. I was following this tutorial on fine-tuning VGG16 models. The model loaded fine with this code:
```
vgg_model = tensorflow.keras.applications.vgg16.VGG16()
```
but gets this ERROR:
```
TypeError: The added layer must be an instance of class Layer. Found: <tensorflow.python.keras.engine.input_layer.InputLayer object at 0x000001FA104CBB70>
```
When running this code:
```
model = Sequential()
for layer in vgg_model.layers[:-1]:
    model.add(layer)
```
Dependencies: Keras 2.2.3 Tensorflow 1.12.0 tensorflow-gpu1.12.0 Python 3.6.0 I am following this blog but instead, I want to use VGG16. Any help to fix this would be appreciated. Thank you so much.

Response:
This won't work because a tensorflow.keras layer is getting added to a keras Model.
```
vgg_model = tensorflow.keras.applications.vgg16.VGG16()
model = keras.Sequential()
model.add(vgg_model.layers[0])
```
Instantiate tensorflow.keras.Sequential(). This will work.
```
model = tensorflow.keras.Sequential()
model.add(vgg_model.layers[0])
```


Question:
Find Unique values in a 2D Tensor using Tensorflow tf.unique currently only works on 1D tensors. How can I find unique values in a 2D tensor.
```
ip=tf.constant([[1,2,1],[3,4,1],[5,6,1],[1,2,1]])
#op should be  = [[1,2,1],[3,4,1],[5,6,1]]
```

Response:
Answer in April 2022 Since version 2.2, Tensorflow has supported find unique 1-D tensor in a 2D tensor using tf.raw_ops.UniqueV2.
```py

```python
import tensorflow as tf
```

```python
a = tf.constant([[1, 2, 1], [3, 4, 1], [5, 6, 1], [1, 2, 1]])
```

```python
y, idx = tf.raw_ops.UniqueV2(x=a, axis=[0])
```

```python
y
#Output
#<tf.Tensor: shape=(3, 3), dtype=int32, numpy=
#array([[1, 2, 1],
#       [3, 4, 1],
#       [5, 6, 1]], dtype=int32)>
```

```python
idx
#Output
#<tf.Tensor: shape=(4,), dtype=int32, numpy=array([0, 1, 2, 0], dtype=int32)>
#```
#Check the docs for more details.
```


Question: Split .tfrecords file into many .tfrecords files Is there any way to split .tfrecords file into many .tfrecords files directly, without writing back each Dataset example ?

Response:
In tensorflow 2.0.0, this will work:
```
import tensorflow as tf

raw_dataset = tf.data.TFRecordDataset("input_file.tfrecord")

shards = 10

for i in range(shards):
    writer = tf.data.experimental.TFRecordWriter(f"output_file-part-{i}.tfrecord")
    writer.write(raw_dataset.shard(shards, i))
```


Question:
Google Storage (gs) wrapper file input/out for Cloud ML? Google recently announced the Clould ML, https://cloud.google.com/ml/ and it's very useful. However, one limitation is that the input/out of a Tensorflow program should support gs://. If we use all tensorflow APIS to read/write files, it should OK, since these APIs support gs://. However, if we use native file IO APIs such as open, it does not work, because they don't understand gs:// For example:
```
with open(vocab_file, 'wb') as f:
        cPickle.dump(self.words, f)
```
This code won't work in Google Cloud ML. However, modifying all native file IO APIs to tensorflow APIs or Google Storage Python APIs is really tedious. Is there any simple way to do this? Any wrappers to support google storage systems, gs:// on top of the native file IO? As suggested here Pickled scipy sparse matrix as input data?, perhaps we can use file_io.read_file_to_string('gs://...'), but still this requrements significant code modifcation.

Response:
Do it like this:
```
from tensorflow.python.lib.io import file_io

with file_io.FileIO('gs://.....', mode='w+') as f:
    cPickle.dump(self.words, f)
```
Or you can read pickle file in like this:
```
file_stream = file_io.FileIO(train_file, mode='r')
x_train, y_train, x_test, y_test  = pickle.load(file_stream)
```


Question:
CuDNNLSTM: Failed to call ThenRnnForward I am facing an issue when trying to use CuDNNLSTM instead of keras.layers.LSTM. This is the error I am getting: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, seq_length, batch_size]: [1, 300, 512, 1, 5521, 128] [[{{node bidirectional_1/CudnnRNN_1}} = CudnnRNN[T=DT_FLOAT, _class=["loc:@train...NNBackprop"], direction="unidirectional", dropout=0, input_mode="linear_input", is_training=true, rnn_mode="lstm", seed=87654321, seed2=0, _device="/job:localhost/replica:0/task:0/device:GPU:0"](bidirectional_1/transpose_1, bidirectional_1/ExpandDims_1, bidirectional_1/ExpandDims_1, bidirectional_1/concat_1)]] [[{{node loss/mul/_75}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1209_loss/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]] Also, I got this error in one of the runs: InternalError: GPU sync failed And the kernel kept dying after each run. I only started getting this error when I tried to run it on a VM instance on google cloud with CuDNNLSTM. my code is:
```
MAX_LEN = max(len(article) for article in X_train_tokens)
EMBEDDING_DIM=300
vocab_size = len(word_to_id)
classes = 2
# Text input
text_input = Input(shape=(MAX_LEN,))
embedding = Embedding(vocab_size, EMBEDDING_DIM, input_length=MAX_LEN)(text_input)
x = Bidirectional(LSTM(512, return_sequences=False))(embedding)
pred = Dense(2, activation='softmax')(x)
model = Model(inputs=[text_input],outputs=pred)
model.compile(loss='categorical_crossentropy', optimizer='RMSprop',     metrics=['accuracy'])
batch_size = 128
generator = text_training_generator(batch_size)
steps = len(X_train)/ batch_size

model.fit_generator(generator, steps_per_epoch=steps, verbose=True, epochs=10)
```
The model summary:
```
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         (None, 5521)              0
_________________________________________________________________
embedding_1 (Embedding)      (None, 5521, 300)         8099100
_________________________________________________________________
bidirectional_1 (Bidirection (None, 1024)              3330048
_________________________________________________________________
dense_1 (Dense)              (None, 2)                 2050
=================================================================
Total params: 11,431,198
Trainable params: 11,431,198
Non-trainable params: 0
_________________________________________________________________
```

Response:
Probably your are running out of memory on the gpu. Your network is very large with 11 million trainable parameters. Do you really need a 512*2 Output of your recurrent layer? Furthermore your embedding_dim is also quite large, while your vocabulary is quite small with 5k words. I guess your network is too complex for your problem. I would suggest to try an embedding size of 32 and a LSTM size of 32 as a start. If your accuracy is still bad, you can increase complexity.
```
EMBEDDING_DIM = 32
Bidirectional(LSTM(32, return_sequences=False))(embedding)
```


Question: Can ReLU handle a negative input? I'm training a neural network on data that comes in as negative & positive values. Is there any way to feed the data into a ReLU network without converting it all to positive and having a separate input which says if the data is negative or positive? The problem I see is that a negative input at the input layer means that unless you have initialised your weights to be negative, the ReLU node isn't ever activated and is forever dead.

Response: I'm not really 100% sure what you're asking, as there are many activation functions and you can easy code your own. If you dont want to code your own, maybe try some alternatives: Leaky ReLU Parameteric ReLU Basically, take a look here


Question:
Unable to feed value for placeholder tensor I have written a simple version bidirectional lstm for sentence classification. But it keeps giving me "You must feed a value for placeholder tensor 'train_x'" error and it seems this come from the variable initialization step.
```
data = load_data(FLAGS.data)
model = RNNClassifier(FLAGS)
init = tf.initialize_all_variables()

with tf.Session() as sess:
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(sess=sess, coord=coord)
    sess.run(init)
    print("Graph initialized..")
    print()
    np.random.seed(FLAGS.random_state)
    for epoch in range(FLAGS.max_max_epoch):

        loss = sess.run(model.cost, feed_dict={model.train_x: data.train_x, model.train_y: data.train_y,
                                        model.embedding_placeholder: data.glove_vec})
        print("Epoch {:2d}: Loss = {:.6f} = {:.5f}".format(epoch+1, loss))
    coord.request_stop()
    coord.join(threads)
```
And the RNNClassifier class code (in a different directory):
```
class RNNClassifier:

    def __init__(self, FLAGS):
        self.params = FLAGS
        with tf.device("/cpu:0"):
            self.train_x = tf.placeholder(tf.int32, [6248, 42], name='train_x')
            self.train_y = tf.placeholder(tf.int32, [6248, 3], name='train_y')
            self.embedding_placeholder = tf.placeholder(tf.float32, [1193515, 100])

        with tf.variable_scope('forward_lstm'):
            lstm_fw_cell = tf.nn.rnn_cell.LSTMCell(num_units=self.params.num_hidden, use_peepholes=False,
                                                activation=tf.nn.relu, forget_bias=0.0,
                                                state_is_tuple=True)
        with tf.variable_scope('backward_lstm'):
            lstm_bw_cell = tf.nn.rnn_cell.LSTMCell(num_units=self.params.num_hidden, use_peepholes=False,
                                                activation=tf.nn.relu, forget_bias=0.0,
                                                state_is_tuple=True)

        fw_initial_state = lstm_fw_cell.zero_state(self.params.batch_size, tf.float32)
        bw_initial_state = lstm_bw_cell.zero_state(self.params.batch_size, tf.float32)
        self._initial_state = [fw_initial_state, bw_initial_state]

        with tf.device("/cpu:0"), tf.variable_scope('softmax'):
            self.W = tf.get_variable('W', [self.params.num_hidden*2, self.params.num_classes])
            self.b = tf.get_variable('b', [self.params.num_classes], initializer=tf.constant_initializer(0.0))

        batched_inputs, batched_labels = self.batch_data()
        embed_inputs = self.use_embedding(batched_inputs)

        rnn_outputs, output_state_fw, output_state_bw  = tf.nn.bidirectional_rnn(
            cell_fw=lstm_fw_cell,
            cell_bw=lstm_bw_cell,
            inputs=embed_inputs,
            initial_state_fw=fw_initial_state,
            initial_state_bw=bw_initial_state
            )

        logits = tf.matmul(rnn_outputs[-1], self.W) + self.b

        self._cost = cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, tf.cast(batched_labels, tf.float32)))
        optimizer = tf.train.AdamOptimizer(learning_rate=0.05).minimize(cost)

    def batch_data(self):
        # inputs = tf.convert_to_tensor(train_x, dtype=tf.int32)
        # labels = tf.convert_to_tensor(train_y, dtype=tf.int32)
        batched_inputs, batched_labels = tf.train.batch(
            tensors=[self._train_x, self._train_y],
            batch_size=self.params.batch_size,
            dynamic_pad=True,
            enqueue_many=True,
            name='batching'
    )
    return batched_inputs, batched_labels

    def use_embedding(self, batched_inputs):
        with tf.device("/cpu:0"), tf.name_scope("input_embedding"):
            embedding = tf.get_variable("embedding", shape=[1193515, 100], trainable=False)
            embedding_init = embedding.assign(self.embedding_placeholder)
            embed_inputs = tf.split(1, self.params.seq_len, tf.nn.embedding_lookup(embedding_init, batched_inputs))
            embed_inputs = [tf.squeeze(input_, [1]) for input_ in embed_inputs]
    return embed_inputs

    @property
    def cost(self):
        return self._cost
```
The output (including the error):
```
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GTX 750 Ti
major: 5 minor: 0 memoryClockRate (GHz) 1.0845
pciBusID 0000:01:00.0
Total memory: 2.00GiB
Free memory: 1.41GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:839] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0)
E tensorflow/core/client/tensor_c_api.cc:485] You must feed a value for placeholder tensor 'train_x' with dtype int32 and shape [6248,42]
     [[Node: train_x = Placeholder[dtype=DT_INT32, shape=[6248,42], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Graph initialized..

W tensorflow/core/framework/op_kernel.cc:936] Out of range: PaddingFIFOQueue '_0_batching/padding_fifo_queue' is closed and has insufficient elements (requested 50, current size 0)
     [[Node: batching = QueueDequeueMany[_class=["loc:@batching/padding_fifo_queue"], component_types=[DT_INT32, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batching/padding_fifo_queue, batching/n)]]
W tensorflow/core/framework/op_kernel.cc:936] Out of range: PaddingFIFOQueue '_0_batching/padding_fifo_queue' is closed and has insufficient elements (requested 50, current size 0)
     [[Node: batching = QueueDequeueMany[_class=["loc:@batching/padding_fifo_queue"], component_types=[DT_INT32, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batching/padding_fifo_queue, batching/n)]]
E tensorflow/core/client/tensor_c_api.cc:485] PaddingFIFOQueue '_0_batching/padding_fifo_queue' is closed and has insufficient elements (requested 50, current size 0)
     [[Node: batching = QueueDequeueMany[_class=["loc:@batching/padding_fifo_queue"], component_types=[DT_INT32, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batching/padding_fifo_queue, batching/n)]]
     [[Node: batching/_9 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1191_batching", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
Traceback (most recent call last):
  File "train_lstm.py", line 66, in <module>
    model.embedding_placeholder: data.glove_vec})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 382, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 655, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 723, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 743, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.OutOfRangeError: PaddingFIFOQueue '_0_batching/padding_fifo_queue' is closed and has insufficient elements (requested 50, current size 0)
     [[Node: batching = QueueDequeueMany[_class=["loc:@batching/padding_fifo_queue"], component_types=[DT_INT32, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batching/padding_fifo_queue, batching/n)]]
     [[Node: batching/_9 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1191_batching", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
Caused by op u'batching', defined at:
  File "train_lstm.py", line 49, in <module>
    model = RNNClassifier(FLAGS)
  File "/home/ccrmad/Code/TDLSTM/models/rnn_classifier.py", line 34, in __init__
    batched_inputs, batched_labels = self.batch_data()
  File "/home/ccrmad/Code/TDLSTM/models/rnn_classifier.py", line 74, in batch_data
    name='batching'
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py", line 595, in batch
    dequeued = queue.dequeue_many(batch_size, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/data_flow_ops.py", line 435, in dequeue_many
    self._queue_ref, n=n, component_types=self._dtypes, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 867, in _queue_dequeue_many
    timeout_ms=timeout_ms, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2310, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1232, in __init__
    self._traceback = _extract_stack()
```
I have tried move the train_x and train_y placeholder initialization before init = tf.initialize_all_variables() and feed them to RNNClassifier() as two args but it still give the same error. Why?

Response:
It looks like the problem is in this method, in RNNClassifier:
```
def batch_data(self):
    # ...
    batched_inputs, batched_labels = tf.train.batch(
        tensors=[self._train_x, self._train_y],
        batch_size=self.params.batch_size,
        dynamic_pad=True,
        enqueue_many=True,
        name='batching')
```
The two tensor arguments to tf.train.batch() are self._train_x and self._train_y. In the RNNClassifier constructor, it seems like you create these as tf.placeholder() tensors:
```
def __init__(self, FLAGS):
    # ...
    with tf.device("/cpu:0"):
        self.train_x = tf.placeholder(tf.int32, [6248, 42], name='train_x')
        self.train_y = tf.placeholder(tf.int32, [6248, 3], name='train_y')
```
...though I'm assuming that the difference between self._train_x and self.train_x is a copy-paste error, because self._train_x doesn't seem to be defined anywhere else. Now, one of the surprising things about tf.train.batch() is that it consumes its inputs in a completely separate thread, called a "queue runner", and started when you call tf.train.start_queue_runners(). This thread calls Session.run() on a subgraph that depends on your placeholders, but doesn't know how to feed these placeholders, so it is this call that fails, leading to the error you are seeing. How then should you fix it? One option would be to use a "feeding queue runner", for which there is experimental support. A simpler option would be to use the tf.train.slice_input_producer() to produce slices from your input data, as follows:
```
def batch_data(self, train_x, train_y):
    input_slice, label_slice = tf.train.slice_input_producer([train_x, train_y])
    batched_inputs, batched_labels = tf.train.batch(
        tensors=[input_slice, label_slice],
        batch_size=self.params.batch_size,
        dynamic_pad=False,   # All rows are the same shape.
        enqueue_many=False,  # tf.train.slice_input_producer() produces one row at a time.
        name='batching')
    return batched_inputs, batched_labels

# ...
# Create batches from the entire training data, where `array_input` and
# `array_labels` are two NumPy arrays.
batched_inputs, batched_labels = model.batch_data(array_input, array_labels)
```


Question:
TensorFlow for binary classification I am trying to adapt this MNIST example to binary classification. But when changing my NLABELS from NLABELS=2 to NLABELS=1, the loss function always returns 0 (and accuracy 1).
```
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

# Import data
mnist = input_data.read_data_sets('data', one_hot=True)
NLABELS = 2

sess = tf.InteractiveSession()

# Create the model
x = tf.placeholder(tf.float32, [None, 784], name='x-input')
W = tf.Variable(tf.zeros([784, NLABELS]), name='weights')
b = tf.Variable(tf.zeros([NLABELS], name='bias'))

y = tf.nn.softmax(tf.matmul(x, W) + b)

# Add summary ops to collect data
_ = tf.histogram_summary('weights', W)
_ = tf.histogram_summary('biases', b)
_ = tf.histogram_summary('y', y)

# Define loss and optimizer
y_ = tf.placeholder(tf.float32, [None, NLABELS], name='y-input')

# More name scopes will clean up the graph representation
with tf.name_scope('cross_entropy'):
    cross_entropy = -tf.reduce_mean(y_ * tf.log(y))
    _ = tf.scalar_summary('cross entropy', cross_entropy)
with tf.name_scope('train'):
    train_step = tf.train.GradientDescentOptimizer(10.).minimize(cross_entropy)

with tf.name_scope('test'):
    correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
    _ = tf.scalar_summary('accuracy', accuracy)

# Merge all the summaries and write them out to /tmp/mnist_logs
merged = tf.merge_all_summaries()
writer = tf.train.SummaryWriter('logs', sess.graph_def)
tf.initialize_all_variables().run()

# Train the model, and feed in test data and record summaries every 10 steps

for i in range(1000):
    if i % 10 == 0:  # Record summary data and the accuracy
        labels = mnist.test.labels[:, 0:NLABELS]
        feed = {x: mnist.test.images, y_: labels}

        result = sess.run([merged, accuracy, cross_entropy], feed_dict=feed)
        summary_str = result[0]
        acc = result[1]
        loss = result[2]
        writer.add_summary(summary_str, i)
        print('Accuracy at step %s: %s - loss: %f' % (i, acc, loss))
   else:
        batch_xs, batch_ys = mnist.train.next_batch(100)
        batch_ys = batch_ys[:, 0:NLABELS]
        feed = {x: batch_xs, y_: batch_ys}
    sess.run(train_step, feed_dict=feed)
```
I have checked the dimensions of both batch_ys (fed into y) and _y and they are both 1xN matrices when NLABELS=1 so the problem seems to be prior to that. Maybe something to do with the matrix multiplication? I actually have got this same problem in a real project, so any help would be appreciated... Thanks!

Response: The original MNIST example uses a one-hot encoding to represent the labels in the data: this means that if there are NLABELS = 10 classes (as in MNIST), the target output is [1 0 0 0 0 0 0 0 0 0] for class 0, [0 1 0 0 0 0 0 0 0 0] for class 1, etc. The tf.nn.softmax() operator converts the logits computed by tf.matmul(x, W) + b into a probability distribution across the different output classes, which is then compared to the fed-in value for y_. If NLABELS = 1, this acts as if there were only a single class, and the tf.nn.softmax() op would compute a probability of 1.0 for that class, leading to a cross-entropy of 0.0, since tf.log(1.0) is 0.0 for all of the examples. There are (at least) two approaches you could try for binary classification: The simplest would be to set NLABELS = 2 for the two possible classes, and encode your training data as [1 0] for label 0 and [0 1] for label 1. This answer has a suggestion for how to do that. You could keep the labels as integers 0 and 1 and use tf.nn.sparse_softmax_cross_entropy_with_logits(), as suggested in this answer.


Question: Is there a way to stack two tensorflow datasets? I want to stack two datasets objects in Tensorflow (rbind function in R). I have created one dataset A from tfRecord files and one dataset B from numpy arrays. Both have same variables. Do you know if there is a way to stack these two datasets to create a bigger one ? Or to create an iterrator that will randomly read data from this two sources ? Thanks

Response:
The tf.data.Dataset.concatenate() method is the closest analog of tf.stack() when working with datasets. If you have two datasets with the same structure (i.e. same types for each component, but possibly different shapes):
```py
dataset_1 = tf.data.Dataset.range(10, 20)
dataset_2 = tf.data.Dataset.range(60, 70)
```
then you can concatenate them as follows:
```py
combined_dataset = dataset_1.concatenate(dataset_2)
```


Question: How to calculate input_dim for a keras sequential model? Keras Dense layer needs an input_dim or input_shape to be specified. What value do I put in there? My input is a matrix of 1,000,000 rows and only 3 columns. My output is 1,600 classes. What do I put there? dimensionality of the inputs (1000000, 1600) 2 because it's a 2D matrix

Response: input_dim is the number of dimensions of the features, in your case that is just 3. The equivalent notation for input_shape, which is an actual dimensional shape, is (3,)


Question: Custom loss function in Keras I'm working on a image class-incremental classifier approach using a CNN as a feature extractor and a fully-connected block for classifying. First, I did a fine-tuning of a VGG per-trained network to do a new task. Once the net is trained for the new task, i store some examples for every class in order to avoid forgetting when new classes are available. When some classes are available, i have to compute every output of the exemplars included the exemplars for the new classes. Now adding zeros to the outputs for old classes and adding the label corresponding to each new class on the new classes outputs i have my new labels, i.e: if 3 new classes enter.... Old class type output: [0.1, 0.05, 0.79, ..., 0 0 0] New class type output: [0.1, 0.09, 0.3, 0.4, ..., 1 0 0] **the last outputs correspond to the class. My question is, how i can change the loss function for a custom one to train for the new classes? The loss function that i want to implement is defined as: where distillation loss corresponds to the outputs for old classes to avoid forgetting, and classification loss corresponds to the new classes. If you can provide me a sample of code to change the loss function in keras would be nice. Thanks!!!!!

Response:
All you have to do is define a function for that, using keras backend functions for calculations. The function must take the true values and the model predicted values. Now, since I'm not sure about what are g, q, x an y in your function, I'll just create a basic example here without caring about what it means or whether it's an actual useful function:
```
import keras.backend as K

def customLoss(yTrue,yPred):
    return K.sum(K.log(yTrue) - K.log(yPred))
```
All backend functions can be seen here. After that, compile your model using that function instead of a regular one:
```
model.compile(loss=customLoss, optimizer = .....)
```


Question:
TensorFlow: "Attempting to use uninitialized value" in variable initialization I am trying to implement multivariate linear regression in Python using TensorFlow, but have run into some logical and implementation issues. My code throws the following error:
```
Attempting to use uninitialized value Variable
Caused by op u'Variable/read'
```
Ideally the weights output should be [2, 3]
```
def hypothesis_function(input_2d_matrix_trainingexamples,
                        output_matrix_of_trainingexamples,
                        initial_parameters_of_hypothesis_function,
                        learning_rate, num_steps):
    # calculate num attributes and num examples
    number_of_attributes = len(input_2d_matrix_trainingexamples[0])
    number_of_trainingexamples = len(input_2d_matrix_trainingexamples)

    #Graph inputs
    x = []
    for i in range(0, number_of_attributes, 1):
        x.append(tf.placeholder("float"))
    y_input = tf.placeholder("float")

    # Create Model and Set Model weights
    parameters = []
    for i in range(0, number_of_attributes, 1):
        parameters.append(
            tf.Variable(initial_parameters_of_hypothesis_function[i]))

    #Contruct linear model
    y = tf.Variable(parameters[0], "float")
    for i in range(1, number_of_attributes, 1):
        y = tf.add(y, tf.multiply(x[i], parameters[i]))

    # Minimize the mean squared errors
    loss = tf.reduce_mean(tf.square(y - y_input))
    optimizer = tf.train.GradientDescentOptimizer(learning_rate)
    train = optimizer.minimize(loss)

    #Initialize the variables
    init = tf.initialize_all_variables()

    # launch the graph
    session = tf.Session()
    session.run(init)
    for step in range(1, num_steps + 1, 1):
        for i in range(0, number_of_trainingexamples, 1):
            feed = {}
            for j in range(0, number_of_attributes, 1):
                array = [input_2d_matrix_trainingexamples[i][j]]
                feed[j] = array
            array1 = [output_matrix_of_trainingexamples[i]]
            feed[number_of_attributes] = array1
            session.run(train, feed_dict=feed)

    for i in range(0, number_of_attributes - 1, 1):
        print (session.run(parameters[i]))

array = [[0.0, 1.0, 2.0], [0.0, 2.0, 3.0], [0.0, 4.0, 5.0]]
hypothesis_function(array, [8.0, 13.0, 23.0], [1.0, 1.0, 1.0], 0.01, 200)
```

Response:
Run this:
```
init = tf.global_variables_initializer()
sess.run(init)
```
Or (depending on the version of TF that you have):
```
init = tf.initialize_all_variables()
sess.run(init)
```


Question: TensorBoard - Plot training and validation losses on the same graph? Is there a way to plot both the training losses and validation losses on the same graph? It's easy to have two separate scalar summaries for each of them individually, but this puts them on separate graphs. If both are displayed in the same graph it's much easier to see the gap between them and whether or not they have begin to diverge due to overfitting. Is there a built in way to do this? If not, a work around way? Thank you much!

Response: The work-around I have been doing is to use two SummaryWriter with different log dir for training set and cross-validation set respectively. And you will see something like this:


Question:
Evaluate all pair combinations of rows of two tensors in tensorflow I am trying to define a custom op in tensorflow, in which at one point I need to construct a matrix (z) that would contain sums of all combinations of pairs of rows of two matrices (x and y). In general, the numbers of rows of x and y are dynamical. In numpy it's fairly simple:
```
import numpy as np
from itertools import product

rows_x = 4
rows_y = 2
dim = 2

x = np.arange(dim*rows_x).reshape(rows_x, dim)
y = np.arange(dim*rows_y).reshape(rows_y, dim)

print('x:\n{},\ny:\n{}\n'.format(x, y))

z = np.zeros((rows_x*rows_y, dim))
print('for loop:')
for i, (x_id, y_id) in enumerate(product(range(rows_x), range(rows_y))):
    print('row {}: {} + {}'.format(i, x[x_id, ], y[y_id, ]))
    z[i, ] = x[x_id, ] + y[y_id, ]

print('\nz:\n{}'.format(z))
```
returns:
```
x:
[[0 1]
 [2 3]
 [4 5]
 [6 7]],
y:
[[0 1]
 [2 3]]

for loop:
row 0: [0 1] + [0 1]
row 1: [0 1] + [2 3]
row 2: [2 3] + [0 1]
row 3: [2 3] + [2 3]
row 4: [4 5] + [0 1]
row 5: [4 5] + [2 3]
row 6: [6 7] + [0 1]
row 7: [6 7] + [2 3]

z:
[[  0.   2.]
 [  2.   4.]
 [  2.   4.]
 [  4.   6.]
 [  4.   6.]
 [  6.   8.]
 [  6.   8.]
 [  8.  10.]]
```
However, I haven't got a clue how to implement anything similar in tensorflow. I was mainly going through SO and the tensorflow API in hopes of finding a function that would yield combinations of elements of two tensors, or a function that would give permutations of elements of a tensor, but to no avail. Any suggestions are most welcome.

Response:
You could simply use the broadcasting ability of tensorflow.
```
import tensorflow as tf

x = tf.constant([[0, 1],[2, 3],[4, 5],[6, 7]], dtype=tf.float32)
y = tf.constant([[0, 1],[2, 3]], dtype=tf.float32)

x_ = tf.expand_dims(x, 0)
y_ = tf.expand_dims(y, 1)
z = tf.reshape(tf.add(x_, y_), [-1, 2])
# or more succinctly
z = tf.reshape(x[None] + y[:, None], [-1, 2])

sess = tf.Session()
sess.run(z)
```


Question:
AttributeError: module 'tensorflow' has no attribute 'get_variable' I am trying to run this line of code :
```
var_init_1 = tf.get_variable("var_init_1", [1, 2], dtype=tf.int32,  initializer=tf.zeros_initializer)
print(var_init_1.shape)
```
It should give an output the shape of tensor of zeros. But why I'm getting an error like this:
```
AttributeError                            Traceback (most recent call last)
<ipython-input-37-3cc73aa1818e> in <module>
----> 1 var_init_1 = tf.get_variable("var_init_1", [1, 2], dtype=tf.int32,  initializer=tf.zeros_initializer)
      2 print(var_init_1.shape)

AttributeError: module 'tensorflow' has no attribute 'get_variable'
```

Response: Replace tf.get_variable with tf.Variable.


Question: What is a loss function in simple words? Can anyone please explain in simple words and possibly with some examples what is a loss function in the field of machine learning/neural networks? This came out while I was following a Tensorflow tutorial: https://www.tensorflow.org/get_started/get_started

Response: It describes how far off the result your network produced is from the expected result - it indicates the magnitude of error your model made on its prediciton. You can then take that error and 'backpropagate' it through your model, adjusting its weights and making it get closer to the truth the next time around.


Question:
Why do we use tf.name_scope() I've been reading the tutorials on TensorFlow where they have written
```
with tf.name_scope('read_inputs') as scope:
    # something
```
The example
```
a = tf.constant(5)
```
and
```
with tf.name_scope('s1') as scope:
    a = tf.constant(5)
```
seem to have the same effect. So, why do we use name_scope?

Response:
They are not the same thing.
```
import tensorflow as tf
c1 = tf.constant(42)
with tf.name_scope('s1'):
    c2 = tf.constant(42)
print(c1.name)
print(c2.name)
```
prints
```
Const:0
s1/Const:0
```
So as the name suggests, the scope functions create a scope for the names of the ops you create inside. This has an effect on how you refer to tensors, on reuse, on how the graph shows in TensorBoard and so on.


Question:
How to use Model.fit which supports generators (after fit_generator deprecation) I have got this deprecation warning while using Model.fit_generator in tensorflow:
```
WARNING:tensorflow: Model.fit_generator (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
Instructions for updating:
Please use Model.fit, which supports generators.
```
How can I use Model.fit instead of Model.fit_generator?

Response: Model.fit_generator is deprecated starting from tensorflow 2.1.0 which is currently is in rc1. You can find the documentation for tf-2.1.0-rc1 here: https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/Model#fit As you can see the first argument of the Model.fit can take a generator so just pass it your generator.


Question:
Custom metric based on tensorflow's streaming metrics returns NaN I'm trying to define the F1-score as a custom metric in TensorFlow for a DNNClassifier. To do that, I wrote a function
```
def metric_fn(predictions=[], labels=[], weights=[]):
    P, _ = tf.contrib.metrics.streaming_precision(predictions, labels)
    R, _ = tf.contrib.metrics.streaming_recall(predictions, labels)
    if P + R == 0:
        return 0
    return 2*(P*R)/(P+R)
```
that uses streaming_precision and streaming_recall from TensorFlow to calulate the F1 score. After that I made a new entry to the validation_metrics:
```
validation_metrics = {
    "accuracy":
        tf.contrib.learn.MetricSpec(
            metric_fn=tf.contrib.metrics.streaming_accuracy,
            prediction_key=tf.contrib.learn.PredictionKey.CLASSES),
    "precision":
        tf.contrib.learn.MetricSpec(
            metric_fn=tf.contrib.metrics.streaming_precision,
            prediction_key=tf.contrib.learn.PredictionKey.CLASSES),
    "recall":
        tf.contrib.learn.MetricSpec(
            metric_fn=tf.contrib.metrics.streaming_recall,
            prediction_key=tf.contrib.learn.PredictionKey.CLASSES),
    "f1score":
        tf.contrib.learn.MetricSpec(
            metric_fn=metric_fn,
            prediction_key=tf.contrib.learn.PredictionKey.CLASSES)
}
```
However, although I get correct precision and recall values, f1score is always nan:
```none
INFO:tensorflow:Saving dict for global step 151: accuracy = 0.982456, accuracy/baseline_label_mean = 0.397661, accuracy/threshold_0.500000_mean = 0.982456, auc = 0.982867, f1score = nan, global_step = 151, labels/actual_label_mean = 0.397661, labels/prediction_mean = 0.406118, loss = 0.310612, precision = 0.971014, precision/positive_threshold_0.500000_mean = 0.971014, recall = 0.985294, recall/positive_threshold_0.500000_mean = 0.985294
```
Something is wrong with my metric_fn, but I can't figure it out. The values P and R obtained by metric_fn are of the form Tensor("precision/value:0", shape=(), dtype=float32). I find this a bit strange. I was expecting a scalar tensor. Any help is appreciated.

Response:
I think the problem may come from the fact that the streaming metrics you use within your metric_fn do not get any update. Try the following (I also included minor modifications to my taste):
```
def metric_fn(predictions=None, labels=None, weights=None):
    P, update_op1 = tf.contrib.metrics.streaming_precision(predictions, labels)
    R, update_op2 = tf.contrib.metrics.streaming_recall(predictions, labels)
    eps = 1e-5;
    return (2*(P*R)/(P+R+eps), tf.group(update_op1, update_op2))
```


Question:
Tensorflow NaN bug? I'm using TensorFlow and I modified the tutorial example to take my RGB images. The algorithm works flawlessly out of the box on the new image set, until suddenly (still converging, it's around 92% accuracy usually), it crashes with the error that ReluGrad received non-finite values. Debugging shows that nothing unusual happens with the numbers until very suddenly, for unknown reason, the error is thrown. Adding
```
print "max W vales: %g %g %g %g"%(tf.reduce_max(tf.abs(W_conv1)).eval(),tf.reduce_max(tf.abs(W_conv2)).eval(),tf.reduce_max(tf.abs(W_fc1)).eval(),tf.reduce_max(tf.abs(W_fc2)).eval())
print "max b vales: %g %g %g %g"%(tf.reduce_max(tf.abs(b_conv1)).eval(),tf.reduce_max(tf.abs(b_conv2)).eval(),tf.reduce_max(tf.abs(b_fc1)).eval(),tf.reduce_max(tf.abs(b_fc2)).eval())
```
as debug code to each loop, yields the following output:
```
Step 8600
max W vales: 0.759422 0.295087 0.344725 0.583884
max b vales: 0.110509 0.111748 0.115327 0.124324
Step 8601
max W vales: 0.75947 0.295084 0.344723 0.583893
max b vales: 0.110516 0.111753 0.115322 0.124332
Step 8602
max W vales: 0.759521 0.295101 0.34472 0.5839
max b vales: 0.110521 0.111747 0.115312 0.124365
Step 8603
max W vales: -3.40282e+38 -3.40282e+38 -3.40282e+38 -3.40282e+38
max b vales: -3.40282e+38 -3.40282e+38 -3.40282e+38 -3.40282e+38
```
Since none of my values is very high, the only way a NaN can happen is by a badly handled 0/0, but since this tutorial code doesn't do any divisions or similar operations, I see no other explanation than that this comes from the internal TF code. I'm clueless on what to do with this. Any suggestions? The algorithm is converging nicely, its accuracy on my validation set was steadily climbing and just reached 92.5% at iteration 8600.

Response:
Actually, it turned out to be something stupid. I'm posting this in case anyone else would run into a similar error.
```
cross_entropy = -tf.reduce_sum(y_*tf.log(y_conv))
```
is actually a horrible way of computing the cross-entropy. In some samples, certain classes could be excluded with certainty after a while, resulting in y_conv=0 for that sample. That's normally not a problem since you're not interested in those, but in the way cross_entropy is written there, it yields 0*log(0) for that particular sample/class. Hence the NaN. Replacing it with
```
cross_entropy = -tf.reduce_sum(y_*tf.log(tf.clip_by_value(y_conv,1e-10,1.0)))
```
solved all my problems.


Question: Getting around tf.argmax which is not differentiable I've written a custom loss function for my neural network but it can't compute any gradients. I thinks it is because I need the index of the highest value and are therefore using argmax to get this index. As argmax is not differentiable I to get around this but I don't know how it is possible. Can anyone help?

Response:
As aidan suggested, it's just a softargmax stretched to the limits by beta. We can use tf.nn.softmax to get around the numerical issues:
```
def softargmax(x, beta=1e10):
  x = tf.convert_to_tensor(x)
  x_range = tf.range(x.shape.as_list()[-1], dtype=x.dtype)
  return tf.reduce_sum(tf.nn.softmax(x*beta) * x_range, axis=-1)
```


Question:
TensorFlow 'module' object has no attribute 'global_variables_initializer' I'm new to Tensorflow I'm running a Deep learning Assignment from Udacity on iPython notebook. link And it has an error.
```
AttributeError                            Traceback (most recent call last)
`<ipython-input-18-3446420b5935>` in `<module>`()
  2
  3 with tf.Session(graph=graph) as session:
----> 4   tf.global_variables_initializer().run()

AttributeError: 'module' object has no attribute 'global_variables_initializer'
```
Please help! How can I fix this? Thank you.

Response: In older versions, it was called tf.initialize_all_variables.


Question:
Bilinear Tensor Product in TensorFlow I'm working on re-implementing this paper and the key operation is a bilinear tensor product. I hardly know what that means, but the paper has a nice little graphic, which I understand. The key operation is e_1 * W * e_2, and I want to know how to implement it in tensorflow, because the rest should be easy. Basically, given 3D tensor W, slice it into matrices, and for the j'th slice (a matrix), multiply it on each side by e_1 and e_2, resulting in a scalar, which is the jth entry in the resulting vector (the output of this operation). So I want to perform a product of e_1, a d-dimensional vector, W, the d x d x k tensor, and e_2, another d-dimensional vector. Could this product be expressed concisely in TensorFlow as it is now, or would I have to define my own op somehow? EARLIER EDITS Why doesn't multiplying these tensors work, and is there some way to define it more explicitly so that it works?
```py

```python
import tensorflow as tf
```

```python
tf.InteractiveSession()
```

```python
a = tf.ones([3, 3, 3])
```

```python
a.eval()
#Output
#array([[[ 1.,  1.,  1.],
#        [ 1.,  1.,  1.],
#        [ 1.,  1.,  1.]],
```

[[ 1.,  1.,  1.],
        [ 1.,  1.,  1.],
        [ 1.,  1.,  1.]],

       [[ 1.,  1.,  1.],
        [ 1.,  1.,  1.],
        [ 1.,  1.,  1.]]], dtype=float32)

```python
b = tf.ones([3, 1, 1])
```

```python
b.eval()
#Output
#array([[[ 1.]],
```

[[ 1.]],

       [[ 1.]]], dtype=float32)
>>>
```
The error message is
```py
ValueError: Shapes TensorShape([Dimension(3), Dimension(3), Dimension(3)]) and TensorShape([Dimension(None), Dimension(None)]) must have the same rank
```
CURRENTLY Turns out that multiplying two 3D tensors doesn't work either with tf.matmul, so but tf.batch_matmul does. tf.batch_matmul will also do 3D tensors and matrices. Then I tried 3D and a vector:
```py
ValueError: Dimensions Dimension(3) and Dimension(1) are not compatible
```

Response:
You can do this with a simple reshape. For the first of the two matrix multiplies, you have k*d, length d vectors to dot product with. This should be close:
```
temp = tf.matmul(E1,tf.reshape(Wddk,[d,d*k]))
result = tf.matmul(E2,tf.reshape(temp,[d,k]))
```


Question:
About names of variable scope in tensorflow Recently I have been trying to learn to use TensorFlow, and I do not understand how variable scopes work exactly. In particular, I have the following problem:
```py
import tensorflow as tf
from tensorflow.models.rnn import rnn_cell
from tensorflow.models.rnn import rnn

inputs = [tf.placeholder(tf.float32,shape=[10,10]) for _ in range(5)]
cell = rnn_cell.BasicLSTMCell(10)
outpts, states = rnn.rnn(cell, inputs, dtype=tf.float32)

print outpts[2].name
# ==> u'RNN/BasicLSTMCell_2/mul_2:0'
```
Where does the '_2' in 'BasicLSTMCell_2' come from? How does it work when later using tf.get_variable(reuse=True) to get the same variable again? edit: I think I find a related problem:
```py
def creating(s):
    with tf.variable_scope('test'):
        with tf.variable_scope('inner'):
            a=tf.get_variable(s,[1])
    return a

def creating_mod(s):
    with tf.variable_scope('test'):
        with tf.variable_scope('inner'):
            a=tf.Variable(0.0, name=s)
    return a

tf.ops.reset_default_graph()
a=creating('a')
b=creating_mod('b')
c=creating('c')
d=creating_mod('d')

print a.name, '\n', b.name,'\n', c.name,'\n', d.name
```
The output is
```py
test/inner/a:0
test_1/inner/b:0
test/inner/c:0
test_3/inner/d:0
```
I'm confused...

Response: The answer above is somehow misguiding. Let me answer why you got two different scope names, even though it looks like that you defined two identical functions: creating and creating_mod. This is simply because you used tf.Variable(0.0, name=s) to create the variable in the function creating_mod. ALWAYS use tf.get_variable, if you want your variable to be recognized by scope! Check out this issue for more details. Thanks!


Question: Tensorflow Tensorboard default port Is there a way to change the default port (6006) on TensorBoard so we could open multiple TensorBoards? Maybe an option like --port="8008"?

Response:
In fact there is an option to change the default port ...
```
tensorboard --logdir=/tmp  --port=8008
```


Question:
Tensorflow (python): "ValueError: setting an array element with a sequence" in train_step.run(...) I'm trying to implement a simple logistic regression model trained with my own set of images, but I am getting this error when I try to train the model:
```
Traceback (most recent call last):
File "main.py", line 26, in <module>
model.entrenar_modelo(sess, training_images, training_labels)
File "/home/jr/Desktop/Dropbox/Machine_Learning/TF/Míos/Hip/model_log_reg.py", line 24, in entrenar_modelo
train_step.run({x: batch_xs, y_: batch_ys})
File "/home/jr/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1267, in run
_run_using_default_session(self, feed_dict, self.graph, session)
File "/home/jr/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2763, in _run_using_default_session
session.run(operation, feed_dict)
File "/home/jr/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 334, in run
np_val = np.array(subfeed_val, dtype=subfeed_t.dtype.as_numpy_dtype)
ValueError: setting an array element with a sequence.
```
The data I'm feeding to train_step.run({x: batch_xs, y_: batch_ys}) is like this: batch_xs: list of tensor objects representing images of 100x100 (10,000 long tensors) batch_ys: list of labels as floats (1.0 or 0.0) What am I doing wrong? Edits It seems the problem was that I had to evaluate the tensors in batch_xs before passing them to train_step.run(...). I thought the run method would take care of that, but I guess I was wrong? Anyway, so once I did this before calling the function:
```
for i, x in enumerate(batch_xs):
    batch_xs[i] = x.eval()
    #print batch_xs[i].shape
    #assert all(x.shape == (100, 100, 3) for x in batch_xs)
# Now I can call the function
```
I had several issues even after doing what is suggested in the answers below. I finally fixed everything by ditching tensors and using numpy arrays.

Response:
This particular error is coming out of numpy. Calling np.array on a sequence with a inconsistant dimensions can throw it.
```

```python
np.array([1,2,3,[4,5,6]])
```

ValueError: setting an array element with a sequence.
```
It looks like it's failing at the point where tf ensures that all the elements of the feed_dict are numpy.arrays. Check your feed_dict.


Question: How do you de-normalise? Once you do normalisation of your data so the values are between 0-1, how do you de-normalise it so you can interpret the result? So when you normalise your data, and feed it to your network and get an output which is normalised data. How do you reverse normalisation to get the original data?

Response:
If you have some data d that you normalize to 0-1 by doing (something like)
```
min_d = np.min(d)
max_d = np.max(d)
normalized_d = (d - min_d) / (max_d - min_d)
```
you can de-normalize this by inverting the normalization. In this case
```
denormalized_d = normalized_d * (max_d - min_d) + min_d
```


Question:
Keras + Tensorflow: Prediction on multiple gpus I'm using Keras with tensorflow as backend. I have one compiled/trained model. My prediction loop is slow so I would like to find a way to parallelize the predict_proba calls to speed things up. I would like to take a list of batches (of data) and then per available gpu, run model.predict_proba() over a subset of those batches. Essentially:
```
data = [ batch_0, batch_1, ... , batch_N ]
on gpu_0 => return predict_proba(batch_0)
on gpu_1 => return predict_proba(batch_1)
...
on gpu_N => return predict_proba(batch_N)
```
I know that it's possible in pure Tensorflow to assign ops to a given gpu (https://www.tensorflow.org/tutorials/using_gpu). However, I don't know how this translates to my situation since I've built/compiled/trained my model using Keras' api. I had thought that maybe I just needed to use python's multiprocessing module and start a process per gpu that would run predict_proba(batch_n). I know this is theoretically possible given another SO post of mine: Keras + Tensorflow and Multiprocessing in Python. However, this still leaves me with the dilemma of not knowing how to actually "choose" a gpu to operate the process on. My question boils down to: how does one parallelize prediction for one model in Keras across multiple gpus when using Tensorflow as Keras' backend? Additionally I am curious if similar parallelization for prediction is possible with only one gpu. A high level description or code example would be greatly appreciated! Thanks!

Response: I created one simple example to show how to run keras model across multiple gpus. Basically, multiple processes are created and each of process owns a gpu. To specify the gpu id in process, setting env variable CUDA_VISIBLE_DEVICES is a very straightforward way (os.environ["CUDA_VISIBLE_DEVICES"]). Hope this git repo can help you. https://github.com/yuanyuanli85/Keras-Multiple-Process-Prediction


Question:
Get the diagonal of a matrix in TensorFlow Is there a way to extract the diagonal of a square matrix in TensorFlow? That is, for a matrix like this:
```
[
 [0, 1, 2],
 [3, 4, 5],
 [6, 7, 8]
]
```
I want to fetch the elements: [0, 4, 8] In numpy, this is pretty straight-forward via np.diag: In TensorFlow, there is a diag function, but it only forms a new matrix with the elements specified in the argument on the diagonal, which is not what I want. I could imagine how this could be done via striding... but I don't see striding for tensors in TensorFlow.

Response: with tensorflow 0.8 its possible to extract the diagonal elements with tf.diag_part() (see documentation) UPDATE for tensorflow >= r1.12 its tf.linalg.tensor_diag_part (see documentation)


Question: Difference between tf.data.Dataset.map() and tf.data.Dataset.apply() With the recent upgrade to version 1.4, Tensorflow included tf.data in the library core. One "major new feature" described in the version 1.4 release notes is tf.data.Dataset.apply(), which is a "method for applying custom transformation functions". How is this different from the already existing tf.data.Dataset.map()?

Response: The difference is that map will execute one function on every element of the Dataset separately, whereas apply will execute one function on the whole Dataset at once (such as group_by_window given as example in the documentation). The argument of apply is a function that takes a Dataset and returns a Dataset when the argument of map is a function that takes one element and returns one transformed element.


Question:
TensorFlow: Saver has 5 models limit I wanted to save multiple models for my experiment but I noticed that tf.train.Saver() constructor could not save more than 5 models. Here is a simple code:
```py
import tensorflow as tf

x = tf.Variable(tf.zeros([1]))
saver = tf.train.Saver()
sess = tf.Session()

for i in range(10):
  sess.run(tf.initialize_all_variables())
  saver.save( sess, '/home/eneskocabey/Desktop/model' + str(i) )
```
When I ran this code, I saw only 5 models on my Desktop. Why is this? How can I save more than 5 models with the same tf.train.Saver() constructor?

Response:
The tf.train.Saver() constructor takes an optional argument called max_to_keep, which defaults to keeping the 5 most recent checkpoints of your model. To save more models, simply specify a value for that argument:
```
import tensorflow as tf

x = tf.Variable(tf.zeros([1]))
saver = tf.train.Saver(max_to_keep=10)
sess = tf.Session()

for i in range(10):
  sess.run(tf.initialize_all_variables())
  saver.save(sess, '/home/eneskocabey/Desktop/model' + str(i))
```
To keep all checkpoints, pass the argument max_to_keep=None to the saver constructor.


Question:
How to get accuracy of model using keras? After fitting the model (which was running for a couple of hours), I wanted to get the accuracy with the following code:
```
train_loss=hist.history['loss']
val_loss=hist.history['val_loss']
train_acc=hist.history['acc']
val_acc=hist.history['val_acc']
xc=range(nb_epoch)
```
of the trained model, but was getting an error, which is caused by the deprecated methods I was using.
```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-233-081ed5e89aa4> in <module>()
      3 train_loss=hist.history['loss']
      4 val_loss=hist.history['val_loss']
----> 5 train_acc=hist.history['acc']
      6 val_acc=hist.history['val_acc']
      7 xc=range(nb_epoch)

KeyError: 'acc'
```
The code I used to fit the model before trying to read the accuracy, is the following:
```
hist = model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
            verbose=1, validation_data=(X_test, Y_test))

hist = model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
            verbose=1, validation_split=0.2)
```
Which produces this output when running it:
```
Epoch 1/20
237/237 [==============================] - 104s 440ms/step - loss: 6.2802 - val_loss: 2.4209
    .....
    .....
    .....
Epoch 19/20
    189/189 [==============================] - 91s 480ms/step - loss: 0.0590 - val_loss: 0.2193
    Epoch 20/20
    189/189 [==============================] - 85s 451ms/step - loss: 0.0201 - val_loss: 0.2312
```
I've noticed that I was running deprecated methods & arguments. So how can I read the accuracy and val_accuracy without having to fit again, and waiting for a couple of hours again? I tried to replace train_acc=hist.history['acc'] with train_acc=hist.history['accuracy'] but it didn't help.

Response:
You probably didn't add "acc" as a metric when compiling the model.
```
model.compile(optimizer=..., loss=..., metrics=['accuracy',...])
```
You can get the metrics and loss from any data without training again with:
```
model.evaluate(X, Y)
```


Question:
What's the difference between tf.cond and if-else? What difference between tf.cond and if-else? Scenario 1
```
import tensorflow as tf

x = 'x'
y = tf.cond(tf.equal(x, 'x'), lambda: 1, lambda: 0)
with tf.Session() as sess:
    print(sess.run(y))
x = 'y'
with tf.Session() as sess:
    print(sess.run(y))
```
Scenario 2
```
import tensorflow as tf

x = tf.Variable('x')
y = tf.cond(tf.equal(x, 'x'), lambda: 1, lambda: 0)
init = tf.global_variables_initializer()
with tf.Session() as sess:
    init.run()
    print(sess.run(y))

tf.assign(x, 'y')
with tf.Session() as sess:
    init.run()
    print(sess.run(y))
```
The outputs are both 1. Does it mean only tf.placeholder can work, and not all the tensor, such as tf.variable? When should I choose if-else condition and when to use tf.cond? What are the diffences between them?

Response: tf.cond is evaluated at the runtime, whereas if-else is evaluated at the graph construction time. If you want to evaluate your condition depending on the value of the tensor at the runtime, tf.cond is the best option.


Question: Where can I find tensorflow.contrib.layers for TensorFlow 2.0 I have been developing machine learning codes in TensorFlow with contrib.layers. I am happy with the module; it works very well and provides me sufficient control over my model. However, TensorFlow 2.0 will completely remove contrib module and the new keras module does not provide me enough flexibility and control without extra efforts (personal experience, correct me if I am wrong), even though it is easier to write. I want to know where the contrib module will be in TensorFlow 2.0. Thanks in advance.

Response: According to the RFC document from August 2018, tf.contrib will be deleted with some of its parts becoming standalone projects (such as tensorflow/probability). This is the case for tf.conrib.layers. Even tf.layers (which was distilled from tf.contrib.layers) will no longer be supported. A detailed description of how to use the Keras API instead is provided in the migration guide.


Question: In Tensorflow, get the names of all the Tensors in a graph I am creating neural nets with Tensorflow and skflow; for some reason I want to get the values of some inner tensors for a given input, so I am using myClassifier.get_layer_value(input, "tensorName"), myClassifier being a skflow.estimators.TensorFlowEstimator. However, I find it difficult to find the correct syntax of the tensor name, even knowing its name (and I'm getting confused between operation and tensors), so I'm using tensorboard to plot the graph and look for the name. Is there a way to enumerate all the tensors in a graph without using tensorboard?

Response:
You can do
```
[n.name for n in tf.get_default_graph().as_graph_def().node]
```
Also, if you are prototyping in an IPython notebook, you can show the graph directly in notebook, see show_graph function in Alexander's Deep Dream notebook


Question: How to read weights saved in tensorflow checkpoint file? I'd like to read the weights and visualize them as images. But I don't see any documentation about model format and how to read the trained weights.

Response: There's this utility which has on print_tensors_in_checkpoint_file method http://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/inspect_checkpoint.py Alternatively, you can use Saver to restore the model and use session.run on variable tensors to get values as numpy arrays


Question:
what does the tf.nn.lrn() method do? Here is the code-snipped from the cifar10-tutorial. It's from the cifar10.py.
```
# conv1
with tf.variable_scope('conv1') as scope:
kernel = _variable_with_weight_decay('weights', shape=[5, 5, 3, 64],
                                     stddev=1e-4, wd=0.0)
conv = tf.nn.conv2d(images, kernel, [1, 1, 1, 1], padding='SAME')
biases = _variable_on_cpu('biases', [64], tf.constant_initializer(0.0))
bias = tf.nn.bias_add(conv, biases)
conv1 = tf.nn.relu(bias, name=scope.name)
_activation_summary(conv1)

# pool1
pool1 = tf.nn.max_pool(conv1, ksize=[1, 3, 3, 1], strides=[1, 2, 2, 1],
                     padding='SAME', name='pool1')
# norm1
norm1 = tf.nn.lrn(pool1, 4, bias=1.0, alpha=0.001 / 9.0, beta=0.75,
                name='norm1')
```
What does the tf.nn.lrn-Method do? I can't find a definition in the API Documentation on https://www.tensorflow.org/versions/r0.8/api_docs/python/index.html

Response: tf.nn.lrn is a short for tf.nn.local_response_normalization. Therefore, the documentation you may want to look at is: https://www.tensorflow.org/api_docs/python/tf/nn/local_response_normalization


Question: Tensorflow - ValueError: Parent directory of trained_variables.ckpt doesn't exist, can't save I want to save my tensorflow session sess but i have the following error ValueError: Parent directory of trained_variables.ckpt doesn't exist, can't save. This is my line of code : saver.save(sess, "trained_variables.ckpt") I've also tried to change the file name and put model instead of trained_variables.ckpt but i get the same problem. Following this tutorial A TensorFlow Tutorial: Email Classification

Response:
```
saver.save(sess, "./trained_variables.ckpt")
```


Question:
Tensorflow: 'module' object has no attribute 'scalar_summary' I tried to run the following code to test my TensorBoard, however, when I ran the program, there is an error said:
```
'module' object has no attribute 'scalar_summary'
```
I want to know how can I fix this issue, thanks. The following is the system info: Operating System: Ubuntu 16.04 LTS Tensorflow version: 0.12rc (master) Running environment: Jupyter Notebook Test program and Output:

Response: The tf.scalar_summary() function was moved in the master branch, after the 0.12 release. You can now find it as tf.summary.scalar().


Question: Randomly sample from multiple tf.data.Datasets in Tensorflow suppose I have N tf.data.Datasets and a list of N probabilities (summing to 1), now I would like to create dataset such that the examples are sampled from the N datasets with the given probabilities. I would like this to work for arbitrary probabilities -> simple zip/concat/flatmap with fixed number of examples from each dataset is probably not what I am looking for. Is it possible to do this in TF? Thanks!

Response: As of 1.12, tf.data.experimental.sample_from_datasets provides this functionality: https://www.tensorflow.org/api_docs/python/tf/data/experimental/sample_from_datasets EDIT: Looks like in earlier versions this can be accessed by tf.contrib.data.sample_from_datasets


Question: How to keep lookup tables initialized for prediction (and not just training)? I create a lookup table from tf.contrib.lookup, using the training data (as input). Then, I pass every input through that lookup table, before passing it through my model. This works for training, but when it comes to online prediction from this same model, it raises the error: Table not initialized I'm using SavedModel to save the model. I run the prediction from this saved model. How can I initialize this table so that it stays initialized? Or is there a better way to save the model so that the table is always initialized?

Response: I think you would be better off using tf.tables_initializer() as the legacy_init_op. tf.saved_model.main_op.main_op() also adds local and global initialization ops in addition to table initialization. when you load the saved model and it runs the legacy_init_op, it would reset your variables, which is not what you want.


Question: Return number of epochs for EarlyStopping callback in Keras Is there any way to return the number of epochs after which the training was stopped in Keras when using the EarlyStopping callback? I can get the log of the training and validation loss and compute the number of epochs myself using the patience parameter, but is there a more direct way?

Response: Use EarlyStopping.stopped_epoch attribute: remember the callback in a separate variable, say callback, and check callback.stopped_epoch after the training stopped.


Question:
Tensorflow tf.data.Dataset API, dataset unzip function? In tensorflow 1.12 there is the Dataset.zip function: documented here. However, I was wondering if there is a dataset unzip function which will return back the original two datasets.
```
# NOTE: The following examples use `{ ... }` to represent the
# contents of a dataset.
a = { 1, 2, 3 }
b = { 4, 5, 6 }
c = { (7, 8), (9, 10), (11, 12) }
d = { 13, 14 }

# The nested structure of the `datasets` argument determines the
# structure of elements in the resulting dataset.
Dataset.zip((a, b)) == { (1, 4), (2, 5), (3, 6) }
Dataset.zip((b, a)) == { (4, 1), (5, 2), (6, 3) }

# The `datasets` argument may contain an arbitrary number of
# datasets.
Dataset.zip((a, b, c)) == { (1, 4, (7, 8)),
                            (2, 5, (9, 10)),
                            (3, 6, (11, 12)) }

# The number of elements in the resulting dataset is the same as
# the size of the smallest dataset in `datasets`.
Dataset.zip((a, d)) == { (1, 13), (2, 14) }
```
I would like to have the following
```
dataset = Dataset.zip((a, d)) == { (1, 13), (2, 14) }
a, d = dataset.unzip()
```

Response:
My workaround was to just use map, not sure if there might be interest in a syntax sugar function for unzip later though.
```
a = dataset.map(lambda a, b: a)
b = dataset.map(lambda a, b: b)
```


Question: What does TensorFlow's `conv2d_transpose()` operation do? The documentation for the conv2d_transpose() operation does not clearly explain what it does: The transpose of conv2d. This operation is sometimes called "deconvolution" after Deconvolutional Networks, but is actually the transpose (gradient) of conv2d rather than an actual deconvolution. I went through the paper that the doc points to, but it did not help. What does this operation do and what are examples of why you would want to use it?

Response: This is the best explanation I've seen online how convolution transpose works is here. I'll give my own short description. It applies convolution with a fractional stride. In other words spacing out the input values (with zeroes) to apply the filter over a region that's potentially smaller than the filter size. As for the why one would want to use it. It can be used as a sort of upsampling with learned weights as opposed to bilinear interpolation or some other fixed form of upsampling.


Question:
Tensorflow GPU Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found When i run
```
import tensorflow as tf
tf.test.is_gpu_available(
    cuda_only=False, min_cuda_compute_capability=None
)
```
I get the following error

Response:
Step 1
```
Move to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\bin
```
Step 2
```
Rename file cusolver64_11.dll  To  cusolver64_10.dll
```
```
cusolver64_10.dll
```


Question: What is the equivalent of np.std() in TensorFlow? Just looking for the equivalent of np.std() in TensorFlow to calculate the standard deviation of a tensor.

Response:
To get the mean and variance just use tf.nn.moments.
```
mean, var = tf.nn.moments(x, axes=[1])
```
For more on tf.nn.moments params see docs


Question:
How to disable printing reports after each epoch in Keras? After each epoch I have printout like below:
```
Train on 102 samples, validate on 26 samples
Epoch 1/1
Epoch 00000: val_acc did not improve
102/102 [==============================] - 3s - loss: 0.4934 - acc: 0.8997 - val_loss: 0.4984 - val_acc: 0.9231
```
I am not using built-in epochs, so I would like to disable these printouts and print something myself. How to do that? I am using tensorflow backend if it matters.

Response: Set verbose=0 to the fit method of your model.


Question: How can I do scatter and gather operations in NumPy? I want to implement the scatter and gather operations of Tensorflow or PyTorch in Numpy. torch.scatter torch.gather

Response: There are two built-in numpy functions that suit your request: Use np.take_along_axis to implement torch.gather Use np.put_along_axis to implement torch.scatter


Question: tensorflow: how to rotate an image for data augmentation? In tensorflow, I would like to rotate an image from a random angle, for data augmentation. But I don't find this transformation in the tf.image module.

Response:
This can be done in tensorflow now:
```
tf.contrib.image.rotate(images, degrees * math.pi / 180, interpolation='BILINEAR')
```