Question:
How to use tf.reset_default_graph() Whenever I try to use tf.reset_default_graph(), I get this error: IndexError: list index out of range or ``. At which part of my code should I use this? When should I be using this? Edit: I updated the code, but the error still occurs.
```py
def evaluate():
    with tf.name_scope("loss"):
        global x # x is a tf.placeholder()
        xentropy = tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=neural_network(x))
        loss = tf.reduce_mean(xentropy, name="loss")

    with tf.name_scope("train"):
        optimizer = tf.train.AdamOptimizer()
        training_op = optimizer.minimize(loss)

    with tf.name_scope("exec"):
        with tf.Session() as sess:
            for i in range(1, 2):
                sess.run(tf.global_variables_initializer())
                sess.run(training_op, feed_dict={x: np.array(train_data).reshape([-1, 1]), y: label})
                print "Training " + str(i)
                saver = tf.train.Saver()
                saver.save(sess, "saved_models/testing")
                print "Model Saved."

def predict():
    with tf.name_scope("predict"):
        tf.reset_default_graph()
        with tf.Session() as sess:
            saver = tf.train.import_meta_graph("saved_models/testing.meta")
            saver.restore(sess, "saved_models/testing")
            output_ = tf.get_default_graph().get_tensor_by_name('output_layer:0')
            print sess.run(output_, feed_dict={x: np.array([12003]).reshape([-1, 1])})

def main():
    print "Starting Program..."
    evaluate()
    writer = tf.summary.FileWriter("mygraph/logs", tf.get_default_graph())
    predict()
```
If I remove the tf.reset_default_graph() from the updated code, I get this error: ValueError: cannot add op with name hidden_layer1/kernel/Adam as that name is already used From my current understanding, tf.reset_default_graph() removes all graphs, hence I avoided the error I mention above(ValueError: cannot add op with name hidden_layer1/kernel/Adam as that name is already used)

Response:
This is probably how you use it:
```
import tensorflow as tf
a = tf.constant(1)
with tf.Session() as sess:
    tf.reset_default_graph()
```
You get an error because you use it in a session. From the tf.reset_default_graph() documentation: Calling this function while a tf.Session or tf.InteractiveSession is active will result in undefined behavior. Using any previously created tf.Operation or tf.Tensor objects after calling this function will result in undefined behavior tf.reset_default_graph() can be helpful (at least for me) during the testing phase while I experiment in jupyter notebook. However, I have never used it in production and do not see how it would be helpful there. Here is an example that could be in a notebook:
```
import tensorflow as tf
# create some graph
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print sess.run(...)
```
Now I do not need this stuff anymore, but if I create another graph and visualize it in tensorboard I will see old nodes and the new nodes. To solve this, I could restart the kernel and run only the next cell. However, I can just do:
```
tf.reset_default_graph()
# create a new graph
with tf.Session() as sess:
    print sess.run(...)
```
Edit after OP added his code:
```
with tf.name_scope("predict"):
    tf.reset_default_graph()
```
Here is what approximately happens. Your code fails because tf.name_scope already added something to a graph. While being inside of this "adding something to the graph", you tell TF to remove the graph completely, but it can't because it is busy adding something.


Question: Tensorflow model zoo? One of the main advantages of caffe for me was the possibility of doing transfer learning on freely distributed pretrained models. Is there a place to get trained models from papers/competitions in tensorflow format? If not, is there a possibility to convert existing caffe(or any other) models into tensorflow models?

Response: You can likely use the caffe to tensorflow model converter to convert model zoo models. If you try it and report back, it would be great to know. There's a potential issue with converting maxpooling and padding, but it seems to work for many models.


Question:
Tensorflow feature column for variable list of values From the TensorFlow docs it's clear how to use tf.feature_column.categorical_column_with_vocabulary_list to create a feature column which takes as input some string and outputs a one-hot vector. For example
```
vocabulary_feature_column =
    tf.feature_column.categorical_column_with_vocabulary_list(
        key="vocab_feature",
        vocabulary_list=["kitchenware", "electronics", "sports"])
```
Let's say "kitchenware" maps to [1,0,0] and "electronics" maps to [0,1,0]. My question is related to having a list of strings as a feature. For example, if the feature value was ["kitchenware","electronics"] then the desired output would be [1,1,0]. The input list length is not fixed but the output dimension is. The use case is a straight bag-of-words type model (obviously with a much larger vocabulary list!). What is the correct way to implement this?

Response:
Here is an example how to feed data to the indicator column:
```
features = {'letter': [['A','A'], ['C','D'], ['E','F'], ['G','A'], ['X','R']]}

letter_feature = tf.feature_column.categorical_column_with_vocabulary_list(
                "letter", ["A", "B", "C"], dtype=tf.string)

indicator = tf.feature_column.indicator_column(letter_feature)
tensor = tf.feature_column.input_layer(features, [indicator])

with tf.Session() as session:
    session.run(tf.global_variables_initializer())
    session.run(tf.tables_initializer())
    print(session.run([tensor]))
```
Which outputs:
```
[array([[2., 0., 0.],
       [0., 0., 1.],
       [0., 0., 0.],
       [1., 0., 0.],
       [0., 0., 0.]], dtype=float32)]
```


Question: What is a "cell class" in Keras? Or, more specific: what is the difference between ConvLSTM2D and ConvLSTM2DCell? What is the difference between SimpleRNN and SimpleRNNCell? Same question for GRU and GRUCell Keras manuals are not very verbose here. I can see from RTFS (reading those fine sources) that these classes are descendants of different base classes. Those, with names, ending with Cell, are subclasses of Layer. In my task I need to classify video sequences. That is, my classifier's input is a sequence of video frames, and the output is a single label (one-hot encoded vector). What class should I use?

Response: The difference is the same for every cell layer combo: Cell: The cell is the actual computation component, they take a single input, a past state and produce an output, new states. These actually perform the step function which contain the computation of a GRU cell for example. RNN Layer: These layers wrap the corresponding cells to apply the same cell to multiple timesteps. So the cell is iterated over the input sequence and collect the output(s) based on extra options such as return_sequences.


Question: Mean Euclidean distance in Tensorflow I have two tensors of sequences of size [batch_size, seq_length, 2]. I want to compute mean Euclidean distance between tensors. What is the elegant way to do this?

Response:
Given the two tensors A & B each with shape [batch_size, seq_length, 2], you can compute the Euclidean distance (L2 norm) using tf.norm:
```
l2_norm = tf.norm(A-B, ord='euclidean')
```


Question:
Using pre-trained inception_resnet_v2 with Tensorflow I have been trying to use the pre-trained inception_resnet_v2 model released by Google. I am using their model definition(https://github.com/tensorflow/models/blob/master/slim/nets/inception_resnet_v2.py) and given checkpoint(http://download.tensorflow.org/models/inception_resnet_v2_2016_08_30.tar.gz) to load the model in tensorflow as below [Download a extract the checkpoint file and download sample images dog.jpg and panda.jpg to test this code]-
```
import tensorflow as tf
slim = tf.contrib.slim
from PIL import Image
from inception_resnet_v2 import *
import numpy as np

checkpoint_file = 'inception_resnet_v2_2016_08_30.ckpt'
sample_images = ['dog.jpg', 'panda.jpg']
#Load the model
sess = tf.Session()
arg_scope = inception_resnet_v2_arg_scope()
with slim.arg_scope(arg_scope):
  logits, end_points = inception_resnet_v2(input_tensor, is_training=False)
saver = tf.train.Saver()
saver.restore(sess, checkpoint_file)
for image in sample_images:
  im = Image.open(image).resize((299,299))
  im = np.array(im)
  im = im.reshape(-1,299,299,3)
  predict_values, logit_values = sess.run([end_points['Predictions'], logits], feed_dict={input_tensor: im})
  print (np.max(predict_values), np.max(logit_values))
  print (np.argmax(predict_values), np.argmax(logit_values))
```
However, the results from this model code does not give the expected results (class no 918 is predicted irrespective of the input image). Can someone help me understand where I am going wrong?

Response: The Inception networks expect the input image to have color channels scaled from [-1, 1]. As seen here. You could either use the existing preprocessing, or in your example just scale the images yourself: im = 2*(im/255.0)-1.0 before feeding them to the network. Without scaling the input [0-255] is much larger than the network expects and the biases all work to very strongly predict category 918 (comic books).


Question:
Understanding LSTM model using tensorflow for sentiment analysis I am trying to learn LSTM model for sentiment analysis using Tensorflow, I have gone through the LSTM model. Following code (create_sentiment_featuresets.py) generates the lexicon from 5000 positive sentences and 5000 negative sentences.
```
import nltk
from nltk.tokenize import word_tokenize
import numpy as np
import random
from collections import Counter
from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

def create_lexicon(pos, neg):
    lexicon = []
    with open(pos, 'r') as f:
        contents = f.readlines()
        for l in contents[:len(contents)]:
            l= l.decode('utf-8')
            all_words = word_tokenize(l)
            lexicon += list(all_words)
    f.close()

    with open(neg, 'r') as f:
        contents = f.readlines()
        for l in contents[:len(contents)]:
            l= l.decode('utf-8')
            all_words = word_tokenize(l)
            lexicon += list(all_words)
    f.close()

    lexicon = [lemmatizer.lemmatize(i) for i in lexicon]
    w_counts = Counter(lexicon)
    l2 = []
    for w in w_counts:
        if 1000 > w_counts[w] > 50:
            l2.append(w)
    print("Lexicon length create_lexicon: ",len(lexicon))
    return l2

def sample_handling(sample, lexicon, classification):
    featureset = []
    print("Lexicon length Sample handling: ",len(lexicon))
    with open(sample, 'r') as f:
        contents = f.readlines()
        for l in contents[:len(contents)]:
            l= l.decode('utf-8')
            current_words = word_tokenize(l.lower())
            current_words= [lemmatizer.lemmatize(i) for i in current_words]
            features = np.zeros(len(lexicon))
            for word in current_words:
                if word.lower() in lexicon:
                    index_value = lexicon.index(word.lower())
                    features[index_value] +=1
            features = list(features)
            featureset.append([features, classification])
    f.close()
    print("Feature SET------")
    print(len(featureset))
    return featureset

def create_feature_sets_and_labels(pos, neg, test_size = 0.1):
    global m_lexicon
    m_lexicon = create_lexicon(pos, neg)
    features = []
    features += sample_handling(pos, m_lexicon, [1,0])
    features += sample_handling(neg, m_lexicon, [0,1])
    random.shuffle(features)
    features = np.array(features)

    testing_size = int(test_size * len(features))

    train_x = list(features[:,0][:-testing_size])
    train_y = list(features[:,1][:-testing_size])
    test_x = list(features[:,0][-testing_size:])
    test_y = list(features[:,1][-testing_size:])
    return train_x, train_y, test_x, test_y

def get_lexicon():
    global m_lexicon
    return m_lexicon
```
The following code (sentiment_analysis.py) is for sentiment analysis using simple neural network model and is working fine
```
from create_sentiment_featuresets import create_feature_sets_and_labels
from create_sentiment_featuresets import get_lexicon
import tensorflow as tf
import numpy as np
# extras for testing
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
#- end extras

train_x, train_y, test_x, test_y = create_feature_sets_and_labels('pos.txt', 'neg.txt')

# pt A-------------

n_nodes_hl1 = 1500
n_nodes_hl2 = 1500
n_nodes_hl3 = 1500

n_classes = 2
batch_size = 100
hm_epochs = 10

x = tf.placeholder(tf.float32)
y = tf.placeholder(tf.float32)

hidden_1_layer = {'f_fum': n_nodes_hl1,
                'weight': tf.Variable(tf.random_normal([len(train_x[0]), n_nodes_hl1])),
                'bias': tf.Variable(tf.random_normal([n_nodes_hl1]))}
hidden_2_layer = {'f_fum': n_nodes_hl2,
                'weight': tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2])),
                'bias': tf.Variable(tf.random_normal([n_nodes_hl2]))}
hidden_3_layer = {'f_fum': n_nodes_hl3,
                'weight': tf.Variable(tf.random_normal([n_nodes_hl2, n_nodes_hl3])),
                'bias': tf.Variable(tf.random_normal([n_nodes_hl3]))}
output_layer = {'f_fum': None,
                'weight': tf.Variable(tf.random_normal([n_nodes_hl3, n_classes])),
                'bias': tf.Variable(tf.random_normal([n_classes]))}

def nueral_network_model(data):
    l1 = tf.add(tf.matmul(data, hidden_1_layer['weight']), hidden_1_layer['bias'])
    l1 = tf.nn.relu(l1)
    l2 = tf.add(tf.matmul(l1, hidden_2_layer['weight']), hidden_2_layer['bias'])
    l2 = tf.nn.relu(l2)
    l3 = tf.add(tf.matmul(l2, hidden_3_layer['weight']), hidden_3_layer['bias'])
    l3 = tf.nn.relu(l3)
    output = tf.matmul(l3, output_layer['weight']) + output_layer['bias']
    return output

# pt B--------------

def train_neural_network(x):
    prediction = nueral_network_model(x)
    cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(logits= prediction, labels= y))
    optimizer = tf.train.AdamOptimizer(learning_rate= 0.001).minimize(cost)

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        for epoch in range(hm_epochs):
            epoch_loss = 0
            i = 0
            while i < len(train_x):
                start = i
                end = i+ batch_size
                batch_x = np.array(train_x[start: end])
                batch_y = np.array(train_y[start: end])
                _, c = sess.run([optimizer, cost], feed_dict= {x: batch_x, y: batch_y})
                epoch_loss += c
                i+= batch_size
            print('Epoch', epoch+ 1, 'completed out of ', hm_epochs, 'loss:', epoch_loss)

        correct= tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1))
        accuracy = tf.reduce_mean(tf.cast(correct, 'float'))
        print('Accuracy:', accuracy.eval({x:test_x, y:test_y}))

        # testing --------------
        m_lexicon= get_lexicon()
        print('Lexicon length: ',len(m_lexicon))
        input_data= "David likes to go out with Kary"
        current_words= word_tokenize(input_data.lower())
        current_words = [lemmatizer.lemmatize(i) for i in current_words]
        features = np.zeros(len(m_lexicon))
        for word in current_words:
            if word.lower() in m_lexicon:
                index_value = m_lexicon.index(word.lower())
                features[index_value] +=1

        features = np.array(list(features)).reshape(1,-1)
        print('features length: ',len(features))
        result = sess.run(tf.argmax(prediction.eval(feed_dict={x:features}), 1))
        print(prediction.eval(feed_dict={x:features}))
        if result[0] == 0:
            print('Positive: ', input_data)
        elif result[0] == 1:
            print('Negative: ', input_data)

train_neural_network(x)
```
I am trying to modify the above (sentiment_analysis.py) for LSTM model after reading the RNN w/ LSTM cell example in TensorFlow and Python which is for LSTM on mnist image dataset: Some how through many hit and run trails, I was able to get the below running code (sentiment_demo_lstm.py) :
```
import tensorflow as tf
from tensorflow.contrib import rnn
from create_sentiment_featuresets import create_feature_sets_and_labels
from create_sentiment_featuresets import get_lexicon

import numpy as np

# extras for testing
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
#- end extras

train_x, train_y, test_x, test_y = create_feature_sets_and_labels('pos.txt', 'neg.txt')

n_steps= 100
input_vec_size= len(train_x[0])
hm_epochs = 8
n_classes = 2
batch_size = 128
n_hidden = 128

x = tf.placeholder('float', [None, input_vec_size, 1])
y = tf.placeholder('float')

def recurrent_neural_network(x):
    layer = {'weights': tf.Variable(tf.random_normal([n_hidden, n_classes])),   # hidden_layer, n_classes
            'biases': tf.Variable(tf.random_normal([n_classes]))}

    h_layer = {'weights': tf.Variable(tf.random_normal([1, n_hidden])), # hidden_layer, n_classes
            'biases': tf.Variable(tf.random_normal([n_hidden], mean = 1.0))}

    x = tf.transpose(x, [1,0,2])
    x = tf.reshape(x, [-1, 1])
    x= tf.nn.relu(tf.matmul(x, h_layer['weights']) + h_layer['biases'])

    x = tf.split(x, input_vec_size, 0)

    lstm_cell = rnn.BasicLSTMCell(n_hidden, state_is_tuple=True)
    outputs, states = rnn.static_rnn(lstm_cell, x, dtype= tf.float32)
    output = tf.matmul(outputs[-1], layer['weights']) + layer['biases']

    return output

def train_neural_network(x):
    prediction = recurrent_neural_network(x)
    cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(logits= prediction, labels= y))
    optimizer = tf.train.AdamOptimizer(learning_rate= 0.001).minimize(cost)

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())

        for epoch in range(hm_epochs):
            epoch_loss = 0
            i = 0
            while (i+ batch_size) < len(train_x):
                start = i
                end = i+ batch_size
                batch_x = np.array(train_x[start: end])
                batch_y = np.array(train_y[start: end])
                batch_x = batch_x.reshape(batch_size ,input_vec_size, 1)
                _, c = sess.run([optimizer, cost], feed_dict= {x: batch_x, y: batch_y})
                epoch_loss += c
                i+= batch_size
            print('--------Epoch', epoch+ 1, 'completed out of ', hm_epochs, 'loss:', epoch_loss)

        correct= tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1))
        accuracy = tf.reduce_mean(tf.cast(correct, 'float'))

        print('Accuracy:', accuracy.eval({x:np.array(test_x).reshape(-1, input_vec_size, 1), y:test_y}))

        # testing --------------
        m_lexicon= get_lexicon()
        print('Lexicon length: ',len(m_lexicon))
        input_data= "Mary does not like pizza"  #"he seems to to be healthy today"  #"David likes to go out with Kary"

        current_words= word_tokenize(input_data.lower())
        current_words = [lemmatizer.lemmatize(i) for i in current_words]
        features = np.zeros(len(m_lexicon))
        for word in current_words:
            if word.lower() in m_lexicon:
                index_value = m_lexicon.index(word.lower())
                features[index_value] +=1
        features = np.array(list(features)).reshape(-1, input_vec_size, 1)
        print('features length: ',len(features))

        result = sess.run(tf.argmax(prediction.eval(feed_dict={x:features}), 1))
        print('RESULT: ', result)
        print(prediction.eval(feed_dict={x:features}))
        if result[0] == 0:
            print('Positive: ', input_data)
        elif result[0] == 1:
            print('Negative: ', input_data)

train_neural_network(x)
```
Output of
```
print(train_x[0])
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

print(train_y[0])
[0, 1]
```
len(train_x)= 9596, len(train_x[0]) = 423 meaning train_x is a list of 9596x423 ? Tough I have a running code now, I still have lots of doubts. In sentiment_demo_lstm, I am not able to understand the following part
```
x = tf.transpose(x, [1,0,2])
x = tf.reshape(x, [-1, 1])
x = tf.split(x, input_vec_size, 0)
```
I have print the following shapes:
```
x = tf.placeholder('float', [None, input_vec_size, 1]) ==> TensorShape([Dimension(None), Dimension(423), Dimension(1)]))
x = tf.transpose(x, [1,0,2]) ==> TensorShape([Dimension(423), Dimension(None), Dimension(1)]))
x = tf.reshape(x, [-1, 1]) ==> TensorShape([Dimension(None), Dimension(1)]))
x = tf.split(x, input_vec_size, 0) ==> ?
```
Here I took the number of hidden layers as 128, does it need to be same as the number of inputs i.e. len(train_x)= 9596 The value 1 in
```
x = tf.placeholder('float', [None, input_vec_size, 1])
```
and
```
x = tf.reshape(x, [-1, 1])
```
is because train_x[0] is 428x1 ? The following is in order to match the placeholder
```
batch_x = np.array(train_x[start: end]) ==> (128, 423)
batch_x = batch_x.reshape(batch_size ,input_vec_size, 1) ==> (128, 423, 1)
```
x = tf.placeholder('float', [None, input_vec_size, 1]) dimensions, right? If I modified the code:
```
while (i+ batch_size) < len(train_x):
```
as
```
while i < len(train_x):
```
I get the following error:
```
Traceback (most recent call last):
  File "sentiment_demo_lstm.py", line 131, in <module>
    train_neural_network(x)
  File "sentiment_demo_lstm.py", line 86, in train_neural_network
    batch_x = batch_x.reshape(batch_size ,input_vec_size, 1)
ValueError: cannot reshape array of size 52452 into shape (128,423,1)
```
=> I can't include the last 124 records/feature-sets while training?

Response:
This is loaded question. Let me try to put it in simple English hiding all the complicated inner details: A simple Unrolled LSTM model with 3 steps is shown below. Each LSTM cell takes an input vector and the hidden output vector of the previous LSTM cell and produces an output vector and the hidden output for the next LSTM cell. A concise representation of the same model is shown below. LSTM models are sequence to sequence models, i.e, they are used for problems when a sequence has to be labeled with an another sequence, like POS tagging or NER tagging of each word in a sentence. You seem to be using it for classification problem. There are two possible ways to use LSTM model for classification 1) Take the output of all the states (O1, O2 and O3 in our example) and apply a softmax layer with softmax layer output size being equal to number of classes (2 in your case) 2) Take the output of the last state (O3) and apply a softmax layer to it. (This is what you are doing in your cod. outputs[-1] return the last row in the outputs) So we back propagate (Backpropagation Through Time - BTT) on the error of the softmax output. Coming to the implementation using Tensorflow, lets see what is the input and output to the LSTM model. Each LSTM takes an input, but we have 3 such LSTM cells, So the input (X placeholder) should be of size (inputsize * time steps). But we don't calculate error for single input and BTT for it, but instead we do it on a batch of input - output combinations. So the Input of LSTM will be (batchsize * inputsize * time steps). A LSTM cells is defined with the size of hidden state. The size of output and the hidden output vector of the LSTM cell will be same as the size of the hidden states (Check LSTM internal calcuations for why!). We then define an LSTM Model using a list of these LSTM cells where the size of the list will be equal to the number of unrolling of the model. So we define the number of unrolling to be done and the size of input during each unrolling. I have skipped lots of things like how to handle variable length sequence, sequence to sequence error calcuations, How LSTM calcuates output and hidden output etc. Coming to your implementation, you are applying a relu layer before the input of each LSTM cell. I dont understand why you are doing that but I guess you are doing it to map your input size to that of the LSTM input size. Coming to your questions: x is the placeholder (tensor/matrix/ndarray) of size [None, input_vec_size, 1]. i.e it can take variable number of rows but each row with input_vec_size columns and each element being a vector is size 1. Normally placeholders are defined with "None" in the rows so that we can vary the batch size of the input. lets say input_vec_size = 3 You are passing a ndarray of size [128 * 3 * 1] x = tf.transpose(x, [1,0,2]) --> [3*128*1] x = tf.reshape(x, [-1, 1]) --> [384*1] h_layer['weights'] --> [1, 128] x= tf.nn.relu(tf.matmul(x, h_layer['weights']) + h_layer['biases']) --> [384 * 128] No input size are hidden size are different. LSTM does a set of operations on the input and previous hidden output and given an output and next hidden output both of which are of size hidden size. x = tf.placeholder('float', [None, input_vec_size, 1]) It defines a tensor or ndarray or variable number of rows, each rows has input_vec_size columns an and each value is a single value vector. x = tf.reshape(x, [-1, 1]) --> reshapes the input x into a matrix of size fixed to 1 column and any number of rows. batch_x = batch_x.reshape(batch_size ,input_vec_size, 1) batch_x.reshape will fail if number of values in batch_x != batch_size*input_vec_size*1. This might be the case for last batch because len(train_x) might not be a multiple of batch_size resulting in the non fully filled last batch. You can avoid this problem by using
```
batch_x = batch_x.reshape(-1 ,input_vec_size, 1)
```
But I am still not sure why you are using Relu in front of the input layer. You are applying logistic regression at the output of the last cell which is fine. You can look my toy example which is a classifier using bidirectional LSTM for classifying if a sequence is increasing or decreasing or mixed. Toy sequence_classifier using LSTM in Tensorflow


Question:
Sorting an Array in TensorFlow Let's assume I have an array in TensorFlow:
```
[ 0.12300211,  0.51767069,  0.13886075,  0.55363625],
[ 0.47279349,  0.50432992,  0.48080254,  0.51576483],
[ 0.84347934,  0.44505221,  0.88839239,  0.48857492],
[ 0.93650454,  0.43652734,  0.96464157,  0.47236174], ..
```
I would like to sort this array by the third column. How do I do this? I am able to sort each column individually using tf.nn.top_k(), which gives me the sorted values and the respective indices. I could use the indices of this third column to reorder the others, but I cannot find a reordering Op. Assuming I want to keep things in-graph (no Python shenanigans): How do I sort (the above array) in TensorFlow? How do I re-order in TensorFlow when I have indices for re-ordering?

Response:
The following works :
```
a = tf.constant(...) # the array
reordered = tf.gather(a, tf.nn.top_k(a[:, 2], k=4).indices)
```


Question:
Drop a dimension of a tensor in Tensorflow I have a tensor that have shape (50, 100, 1, 512) and i want to reshape it or drop the third dimension so that the new tensor have shape (50, 100, 512). I have tried tf.slice with tf.squeeze:
```
a = tf.slice(a, [50, 100, 1, 512], [50, 100, 1, 512])
b = tf.squeeze(a)
```
Everything seem working when i tried to print the shape of a and b but when i start training my model this error came
```
tensorflow.python.framework.errors_impl.InvalidArgumentError: Expected size[0] in [0, 0], but got 50
     [[Node: Slice = Slice[Index=DT_INT32, T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](MaxPool_2, Slice/begin, Slice/size)]]
```
Are there any problem with my slice. How can i fix it. Thanks

Response:
Generally tf.squeeze will drop the dimensions.
```
a = tf.constant([[[1,2,3],[3,4,5]]])
```
The above tensor shape is [1,2,3]. After performing squeeze operation,
```
b = tf.squeeze(a)
```
Now, Tensor shape is [2,3]


Question:
on colab - class_weight is causing a ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() i'm running a CNN with keras sequential on google colab. i'm getting the following error: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() when i remove the class_weight argument from the model.fit function, the error is gone and the network is trained succesfully. however, i really want to account for unbalanced data i checked the shape of my class_weights vector and it's good (and nd.array, just like you would get when generating class_Weights from sklearn compute class weights function ) not sure what details are relevant but i wil gladly provide more details regarding version and all that mess. p.s a fact that might be important - my data is the FER2013 data and i'm using FERplus labels. meaning, my samples are not associated with one unique class, rather each sample has it's own probability distribution for each class. bottom line, my labels are vectors of size class_names with all elements adding up to one. just to be super clear, an example: img1 label = [0,0,0,0,0.2,0,0.3,0,0,0.5] anyhow, i computed class_weights as an nd.array of size 10 with elements ranging between 0 and 1, supposed to balance down the more represented classes. i was not sure if that is relevant to the error, but i'm bringing it up just in case. my code:
```
def create_model_plus():
  return tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(filters=32,kernel_size=5,strides=1,input_shape=(48, 48, 1),padding='same',use_bias=True,kernel_initializer='normal',bias_initializer=tf.keras.initializers.Constant(0.1),activation=tf.nn.relu),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.MaxPooling2D((2, 2), strides=2),
    tf.keras.layers.Conv2D(filters=64,kernel_size=5,strides=1,padding='same',use_bias=True,kernel_initializer='normal',bias_initializer=tf.keras.initializers.Constant(0.1),activation=tf.nn.relu),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.MaxPooling2D((2, 2), strides=1),
    tf.keras.layers.Conv2D(filters=128,kernel_size=5,strides=1,padding='same',use_bias=True,kernel_initializer='normal',bias_initializer=tf.keras.initializers.Constant(0.1),activation=tf.nn.relu),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.MaxPooling2D((2, 2), strides=1),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(1008, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(512, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
  ])

history_df=[]
history_object=tf.keras.callbacks.History()
#save_best_object=tf.keras.callbacks.ModelCheckpoint('/Users/nimrodros', monitor='val_loss', verbose=1, save_best_only=True, save_weights_only=False, mode='auto', period=1)

early_stop_object=tf.keras.callbacks.EarlyStopping(monitor='val_loss',min_delta=0.001, patience=4)
gony_adam=tf.keras.optimizers.Adam(
    lr=0.001
)
reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.3,patience=3, min_lr=0.0001, verbose=1)

#log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
#tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

datagen = tf.keras.preprocessing.image.ImageDataGenerator(rotation_range=8, width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    validation_split=0.3
    )
datagen.fit(images.reshape(28709,48,48,1))
model = create_model_plus()
model.compile(optimizer=gony_adam,
        loss='categorical_crossentropy',
        metrics=['accuracy'])
history = model.fit(x=datagen.flow(images.reshape(28709,48,48,1), FER_train_labels, batch_size=32,subset='training'),validation_data=datagen.flow(images.reshape(28709,48,48,1), FER_train_labels, batch_size=32,subset='validation'),steps_per_epoch=600,validation_steps=250,epochs=60,callbacks=[history_object,early_stop_object,reduce_lr],class_weight=cl_weigh)
history_df=pd.DataFrame(history.history)
```
hope someone knows what to do! thanks!!!

Response:
The problem is that the sklearn API returns a numpy array but the keras requires a dictionary as an input for class_weight (see here). You can resolve the error using below method:
```
from sklearn.utils import class_weight
weight = class_weight.compute_class_weight('balanced', np.unique(y_train), y_train)
weight = {i : weight[i] for i in range(5)}
```


Question:
How to use several summary collections in Tensorflow? I have 2 distinctive groups of summaries. One is collected once per batch another one is collected once per epoch. How can I use merge_all_summaries(key='???') to collect summaries in this two groups separately? Doing it manually is always an option but there seems to be a better way. Illustration of how i think it should work:
```py
# once per batch
      tf.scalar_summary("loss", graph.loss)
      tf.scalar_summary("batch_acc", batch_accuracy)
      # once per epoch
      gradients = tf.gradients(graph.loss, [W, D])
      tf.histogram_summary("embedding/W", W, collections='per_epoch')
      tf.histogram_summary("embedding/D", D, collections='per_epoch')

      tf.merge_all_summaries()                # -> (MergeSummary...) :)
      tf.merge_all_summaries(key='per_epoch') # -> NONE              :(
```

Response:
Problem solved. collections parameter of a summary is supposed to be a list. Solution:
```
# once per batch
  tf.scalar_summary("loss", graph.loss)
  tf.scalar_summary("batch_acc", batch_accuracy)
  # once per epoch
  tf.histogram_summary("embedding/W", W, collections=['per_epoch'])
  tf.histogram_summary("embedding/D", D, collections=['per_epoch'])

  tf.merge_all_summaries()                # -> (MergeSummary...) :)
  tf.merge_all_summaries(key='per_epoch') # -> (MergeSummary...) :)
```
Edit. Syntactical change in TF:
```
# once per batch
  tf.summary.scalar("loss", graph.loss)
  tf.summary.scalar("batch_acc", batch_accuracy)
  # once per epoch
  tf.summary.histogram("embedding/W", W, collections=['per_epoch'])
  tf.summary.histogram("embedding/D", D, collections=['per_epoch'])

  tf.summary.merge_all()                # -> (MergeSummary...) :)
  tf.summary.merge_all(key='per_epoch') # -> (MergeSummary...) :)
```


Question:
swap tensor axis in keras I want to swap tensor axis of image batches from (batch_size, row, col, ch) to (batch_size, ch, row, col). in numpy, this can be done with
```
X_batch = np.moveaxis( X_batch, 3, 1)
```
How would I do that in Keras?

Response:
You can use K.permute_dimensions() which is exactly similar to np.transpose(). Example:
```
import numpy as np
from keras import backend as K

A = np.random.random((1000,32,64,3))
# B = np.moveaxis( A, 3, 1)
C = np.transpose( A, (0,3,1,2))

print A.shape
print C.shape

A_t = K.variable(A)
C_t = K.permute_dimensions(A_t, (0,3,1,2))

print K.eval(A_t).shape
print K.eval(C_t).shape
```


Question:
Where does next_batch in the TensorFlow tutorial batch_xs, batch_ys = mnist.train.next_batch(100) come from? I am trying out the TensorFlow tutorial and don't understand where does next_batch in this line come from?
```
batch_xs, batch_ys = mnist.train.next_batch(100)
```
I looked at
```
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
```
And didn't see next_batch there either. Now when trying out next_batch in my own code, I am getting
```
AttributeError: 'numpy.ndarray' object has no attribute 'next_batch'
```
So I would like to understand where does next_batch come from?

Response:
next_batch is a method of the DataSet class (see https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/learn/python/learn/datasets/mnist.py for more information on what's in the class). When you load the mnist data and assign it to the variable mnist with:
```
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
```
look at the class of mnist.train. You can see it by typing:
```
print mnist.train.__class__
```
You'll see the following:
```
<class 'tensorflow.contrib.learn.python.learn.datasets.mnist.Dataset'>
```
Because mnist.train is an instance of class DataSet, you can use the class's function next_batch. For more information on classes, check out the documentation.


Question:
Regularization for LSTM in tensorflow Tensorflow offers a nice LSTM wrapper.
```
rnn_cell.BasicLSTM(num_units, forget_bias=1.0, input_size=None,
           state_is_tuple=False, activation=tanh)
```
I would like to use regularization, say L2 regularization. However, I don't have direct access to the different weight matrices used in the LSTM cell, so I cannot explicitly do something like
```
loss = something + beta * tf.reduce_sum(tf.nn.l2_loss(weights))
```
Is there a way to access the matrices or use regularization somehow with LSTM?

Response:
I like to do the following, yet the only thing I know is that some parameters prefers not to be regularized with L2, such as batch norm parameters and biases. LSTMs contains one Bias tensor (despite conceptually it has many biases, they seem to be concatenated or something, for performance), and for the batch normalization I add "noreg" in the variables' name to ignore it too.
```
loss = your regular output loss
l2 = lambda_l2_reg * sum(
    tf.nn.l2_loss(tf_var)
        for tf_var in tf.trainable_variables()
        if not ("noreg" in tf_var.name or "Bias" in tf_var.name)
)
loss += l2
```
Where lambda_l2_reg is the small multiplier, e.g.: float(0.005) Doing this selection (which is the full if in the loop discarding some variables in the regularization) once made me jump from 0.879 F1 score to 0.890 in one shot of testing the code without readjusting the value of the config's lambda, well this was including both the changes for the batch normalisation and the Biases and I had other biases in the neural network. According to this paper, regularizing the recurrent weights may help with exploding gradients. Also, according to this other paper, dropout would be better used between stacked cells and not inside cells if you use some. About the exploding gradient problem, if you use gradient clipping with the loss that has the L2 regularization already added to it, that regularization will be taken into account too during the clipping process. P.S. Here is the neural network I was working on: https://github.com/guillaume-chevalier/HAR-stacked-residual-bidir-LSTMs


Question:
How to freeze weights in certain layer with Keras? I am trying to freeze the weights of certain layer in a prediction model with Keras and mnist dataset, but it does not work. The code is like:
```python
from keras.layers import Dense, Flatten
from keras.utils import to_categorical
from keras.models import Sequential, load_model
from keras.datasets import mnist
from keras.losses import categorical_crossentropy

import numpy as np

def load_data():
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_train = x_train.astype('float32')
    x_test = x_test.astype('float32')
    x_train /= 255
    x_test /= 255
    y_train = to_categorical(y_train, num_classes=10)
    y_test = to_categorical(y_test, num_classes=10)
    return x_train, y_train, x_test, y_test

def run():
    x_train, y_train, x_test, y_test = load_data()
    model = Sequential([Flatten(input_shape=(28, 28)),
                        Dense(300, name='dense1', activation='relu'),
                        Dense(100, name='dense2', activation='relu'),
                        Dense(10, name='dense3', activation='softmax')])
    model.trainable = True
    model.compile(optimizer='Adam',
                  metrics=['accuracy'],
                  loss=categorical_crossentropy)

    print(model.summary())
    model.fit(x_train, y_train, epochs=5, verbose=2)
    print(model.evaluate(x_test, y_test))
    return model

def freeze(model):
    x_train, y_train, x_test, y_test = load_data()

    name = 'dense1'

    weightsAndBias = model.get_layer(name=name).get_weights()

    # freeze the weights of this layer
    model.get_layer(name=name).trainable = False

    # record the weights before retrain
    weights_before = weightsAndBias[0]
    # retrain
    model.fit(x_train, y_train, verbose=2, epochs=1)
    weights_after = model.get_layer(name=name).get_weights()[0]

    if (weights_before == weights_after).all():
        print('the weights did not change!!!')
    else:
        print('the weights changed!!!!')

if __name__ == '__main__':
    model = run()
    freeze(model)
```
The program outputs 'the weights changed!!!!'. I do not understand why the weights of the layer named 'dense1' changes after setting model.get_layer(name=name).trainable = False.

Response:
You can do it by using:
```
model=Sequential()
layer=Dense(64,init='glorot_uniform',input_shape=(784,))
layer.trainable=False
model.add(layer)
layer2=Dense(784, activation='sigmoid',init='glorot_uniform')
layer2.trainable=True
model.add(layer2)
model.compile(loss='relu', optimizer=sgd,metrics = ['mae'])
```


Question:
what is meaning of hook that used in tensorflow I couldn't understand the exact meaning of Hook in python, tensorflow
```
_LearningRateSetterHook(tf.train.SessionRun**Hook**):
```
I would greatly appreciate it if you explain it to me. Thank you

Response: This could be a more general question about what hooks are. Hooks are named appropriately in that they allow a way to 'hook into' certain points of the execution of a program. Thus you could trigger a function or logging after a certain part of the code executes. To give an example I have listed the description of the SessionRunHook that you mentioned along with a link to its documentation. It specifically allows you to 'hook' into the mentioned points. SessionRunHooks are useful to track training, report progress, request early stopping and more. SessionRunHooks use the observer pattern and notify at the following points: when a session starts being used before a call to the session.run() after a call to the session.run() when the session closed A SessionRunHook encapsulates a piece of reusable/composable computation that can piggyback a call to MonitoredSession.run(). A hook can add any ops-or-tensor/feeds to the run call, and when the run call finishes with success gets the outputs it requested. Hooks are allowed to add ops to the graph in hook.begin(). The graph is finalized after the begin() method is called. https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/session_run_hook.py


Question: How does tf.multinomial work? How does tf.multinomial work? Here is stated that it "Draws samples from a multinomial distribution". What does that mean?

Response:
If you perform an experiment n times that can have only two outcomes (either success or failure, head or tail, etc.), then the number of times you obtain one of the two outcomes (success) is a binomial random variable. In other words, If you perform an experiment that can have only two outcomes (either success or failure, head or tail, etc.), then a random variable that takes value 1 in case of success and value 0 in case of failure is a Bernoulli random variable. If you perform an experiment n times that can have K outcomes (where K can be any natural number) and you denote by X_i the number of times that you obtain the i-th outcome, then the random vector X defined as X = [X_1, X_2, X_3, ..., X_K] is a multinomial random vector. In other words, if you perform an experiment that can have K outcomes and you denote by X_i a random variable that takes value 1 if you obtain the i-th outcome and 0 otherwise, then the random vector X defined as X = [X_1, X_2, X_3, ..., X_K] is a Multinoulli random vector. In other words, when the i-th outcome is obtained, the i-th entry of the Multinoulli random vector X takes value 1, while all other entries take value 0. So, a multinomial distribution can be seen as a sum of mutually independent Multinoulli random variables. And the probabilities of the K possible outcomes will be denoted by p_1, p_2, p_3, ..., p_K An example in Tensorflow,
```

```python
isess = tf.InteractiveSession()
```

```python
prob = [[.1, .2, .7], [.3, .3, .4]]  # Shape [2, 3]
     ...: dist = tf.distributions.Multinomial(total_count=[4., 5], probs=prob)
     ...:
     ...: counts = [[2., 1, 1], [3, 1, 1]]
     ...: isess.run(dist.prob(counts))  # Shape [2]
     ...:
#Output
#array([ 0.0168    ,  0.06479999], dtype=float32)
#```
#Note: The Multinomial is identical to the Binomial distribution when K = 2. For more detailed information please refer either tf.compat.v1.distributions.Multinomial or the latest docs of tensorflow_probability.distributions.Multinomial
```


Question:
AxisError: axis 1 is out of bounds for array of dimension 1 when calculating accuracy of classes I try to predict 10 classes using this code
```
#Predicting the Test set rules
y_pred = model.predict(traindata)
y_pred = np.argmax(y_pred, axis=1)
y_true = np.argmax(testdata, axis=1)

target_names = ["akLembut","akMundur","akTajam","caMenaik", "caMenurun", "coretanTengah", "garisAtas", "garisBawah", "garisBawahBanyak", "ttdCangkang"]
print("\n"+ classification_report(y_true, y_pred, target_names=target_names))
```
But then I got an error message like this
```
AxisError                                 Traceback (most recent call last)
<ipython-input-13-a2b02b251547> in <module>()
      2 y_pred = model.predict(traindata)
      3 y_pred = np.argmax(y_pred, axis=1)
----> 4 y_true = np.argmax(testdata, axis=1)
      5
      6 target_names = ["akLembut","akMundur","akTajam","caMenaik", "caMenurun", "coretanTengah", "garisAtas", "garisBawah", "garisBawahBanyak", "ttdCangkang"]

<__array_function__ internals> in argmax(*args, **kwargs)

2 frames
/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py in _wrapit(obj, method, *args, **kwds)
     45     except AttributeError:
     46         wrap = None
---> 47     result = getattr(asarray(obj), method)(*args, **kwds)
     48     if wrap:
     49         if not isinstance(result, mu.ndarray):

AxisError: axis 1 is out of bounds for array of dimension 1
```
I already train the data and I need to know each accuracy.

Response:
My guess is that your test_data array is only one-dimensional, so change to
```
y_true = np.argmax(testdata, axis=0)
```


Question: How to perform tf.image.per_image_standardization on a batch of images in tensorflow I would like to know how to perform image whitening on a batch of images. According to the documentation in https://www.tensorflow.org/api_docs/python/tf/image/per_image_standardization, it is said that tf.image.per_image_standardization takes as input a 3D tensor, that is an image, of shape: [height, width, channels]. Is it a missing feature or there is a different method? Any help is much appreciated.

Response: This is how to perform this operation on a batch of images. tf.map_fn(lambda frame: tf.image.per_image_standardization(frame), frames)


Question: Is it thread-safe when using tf.Session in inference service? Now we have used TensorFlow to train and export an model. We can implement the inference service with this model just like how tensorflow/serving does. I have a question about whether the tf.Session object is thread-safe or not. If it's true, we may initialize the object after starting and use the singleton object to process the concurrent requests.

Response: The tf.Session object is thread-safe for Session.run() calls from multiple threads. Before TensorFlow 0.10 graph modification was not thread-safe. This was fixed in the 0.10 release, so you can add nodes to the graph concurrently with Session.run() calls, although this is not advised for performance reasons; instead, it is recommended to call sess.graph.finalize() before using the session from multiple threads, to prevent accidental memory leaks.


Question:
Tensorflow: `batch_size` or `steps` is required for `Tensor` or `NumPy` input data Consider the following TensorFlow code:
```
import numpy as np
import tensorflow as tf

import tensorflow_datasets as tfds

mnist_dataset, mnist_info = tfds.load(name = 'mnist', with_info=True, as_supervised=True)

mnist_train, mnist_test = mnist_dataset['train'], mnist_dataset['test']

num_validation_samples = 0.1 * mnist_info.splits['train'].num_examples
num_validation_samples = tf.cast(num_validation_samples, tf.int64)

num_test_samples = mnist_info.splits['test'].num_examples
num_test_samples = tf.cast(num_test_samples, tf.int64)

def scale(image, label):
    image = tf.cast(image, tf.float32)
    image /= 255.
    return image, label

scaled_train_and_validation_data = mnist_train.map(scale)
test_data = mnist_test.map(scale)

BUFFER_SIZE = 10_000

shuffled_train_and_validation_data = scaled_train_and_validation_data.shuffle(BUFFER_SIZE)

validation_data = shuffled_train_and_validation_data.take(num_validation_samples)
train_data = shuffled_train_and_validation_data.skip(num_validation_samples)

BATCH_SIZE = 100
train_data = train_data.batch(BATCH_SIZE)
validation_data = validation_data.batch(num_validation_samples) # Single batch, having size equal to number of validation samples
test_data = test_data.batch(num_test_samples)

validation_inputs, validation_targets = next(iter(validation_data))

input_size = 784 # One for each pixel of the 28 * 28 image
output_size = 10
hidden_layer_size = 50 # Arbitrary chosen

model = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28,28,1)),
    tf.keras.layers.Dense(hidden_layer_size, activation='relu'), # First hidden layer
    tf.keras.layers.Dense(hidden_layer_size, activation='relu'),
    tf.keras.layers.Dense(output_size, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
NUM_EPOCHS = 5
model.fit(train_data, epochs = NUM_EPOCHS, validation_data=(validation_inputs, validation_targets), verbose=2)
```
On running it tf gives the error: ValueError: batch_size or steps is required for Tensor or NumPy input data. When batch_size is added in the call to fit():
```
model.fit(train_data, batch_size = BATCH_SIZE, epochs = NUM_EPOCHS, validation_data=(validation_inputs, validation_targets), verbose=2)
```
It then complains: ValueError: The batch_size argument must not be specified for the given input type. Received input: , batch_size: 100 What is the error here?

Response:
The error happens because a tf.Dataset is provided to the argument validation_data of Model.fit, but Keras does not know how many steps to validate for. To solve this problem, you can just set the argument validation_steps. For example:
```
model.fit(train_data,
    batch_size=BATCH_SIZE,
    epochs=NUM_EPOCHS,
    validation_data=(validation_inputs, validation_targets),
    validation_steps=10)
```


Question: In Tensorflow, what is the difference between a tensor that has a type ending in _ref and a tensor that does not? The docs say: In addition, variants of these types with the _ref suffix are defined for reference-typed tensors. What exactly does this mean? What are reference-typed tensors and how do they differ from standard ones?

Response: A reference-typed tensor is mutable. The most common way to create a reference-typed tensor is to define a tf.Variable: defining a tf.Variable whose initial value has dtype tf.float32 will create a reference-typed tensor with dtype tf.float32_ref. You can mutate a reference-typed tensor by passing it as the first argument to tf.assign(). (Note that reference-typed tensors are something of an implementation detail in the present version of TensorFlow. We'd encourage you to use higher-level wrappers like tf.Variable, which may migrate to alternative representations for mutable state in the future.)


Question:
Is tensorflow lazy? Let's say you have some piece of code like this
```
import tensorflow as tf
...
f = h*y + z*t  #Just some expression involving other tensors.
e = ... # some expression that does not involve f.
result = tf.select(b, e, f)

sess.run(result)
```
b is a boolean tensor of the same shape as e and f. If all the elements of b evaluate to true, we don't need f and the result will just be (or be equal to) e. The question: when the session is run with result, and the elements of e are all true, is f evaluated?

Response: TL;DR: TensorFlow is strict, so both e and f will be evaluated before the tf.select() node executes. This has caused some confusion. TensorFlow first prunes the dataflow graph based on which operations are statically required to produce the values that are fetched (i.e. the arguments to sess.run()). Once the graph has been pruned, however, the runtime uses strict execution, whereby all of the inputs to an operation (such as tf.select()) must have been computed before that operation can execute. There is experimental support for conditional execution in the tf.control_flow_ops module, using the tf.control_flow_ops.cond() function, but this is sparsely documented at present.


Question:
Dropout layer before or after LSTM. What is the difference? Suppose that we have an LSTM model for time series forecasting. Also, this is a multivariate case, so we're using more than one feature for training the model.
```
ipt   = Input(shape = (shape[0], shape[1])
x     = Dropout(0.3)(ipt) ## Dropout before LSTM.
x     = CuDNNLSTM(10, return_sequences = False)(x)
out   = Dense(1, activation='relu')(x)
```
We can add Dropout layer before LSTM (like the above code) or after LSTM. If we add it before LSTM, is it applying dropout on timesteps (different lags of time series), or different input features, or both of them? If we add it after LSTM and because return_sequences is False, what is dropout doing here? Is there any different between dropout option in LSTM and dropout layer before LSTM layer?

Response: As default, Dropout creates a random tensor of zeros an ones. No pattern, no privileged axis. So, you can't say a specific thing is being dropped, just random coordinates in the tensor. (Well, it drops features, but different features for each step, and differently for each sample) You can, if you want, use the noise_shape property, which will define the shape of the random tensor. Then you can select if you want to drop steps, features or samples, or maybe a combination. Dropping time steps: noise_shape = (1,steps,1) Dropping features: noise_shape = (1,1, features) Dropping samples: noise_shape = (None, 1, 1) There is also the SpatialDropout1D layer, which uses noise_shape = (input_shape[0], 1, input_shape[2]) automatically. This drops the same feature for all time steps, but treats each sample individually (each sample will drop a different group of features). After the LSTM you have shape = (None, 10). So, you use Dropout the same way you would use in any fully connected network. It drops a different group of features for each sample. A dropout as an argument to the LSTM has a lot of differences. It generates 4 different dropout masks, for creating different inputs for each of the different gates. (You can see the LSTMCell code to check this). Also, there is the option of recurrent_dropout, which will generate 4 dropout masks, but to be applied to the states instead of the inputs, each step of the recurrent calculations.


Question:
Why in Keras subclassing API, the call method is never called and as an alternative the input is passed by calling the object of this class? When creating a model using Keras subclassing API, we write a custom model class and define a function named call(self, x)(mostly to write the forward pass) which expects an input. However, this method is never called and instead of passing the input to call, it is passed to the object of this class as model(images). How are we able to call this model object and pass values when we haven't implemented Python special method, __call__ in the class
```
class MyModel(Model):
  def __init__(self):
    super(MyModel, self).__init__()
    self.conv1 = Conv2D(32, 3, activation='relu')
    self.flatten = Flatten()
    self.d1 = Dense(128, activation='relu')
    self.d2 = Dense(10, activation='softmax')

  def call(self, x):
    x = self.conv1(x)
    x = self.flatten(x)
    x = self.d1(x)
    return self.d2(x)

# Create an instance of the model
model = MyModel()
```
Use tf.GradientTape to train the model:
```
@tf.function
def train_step(images, labels):
  with tf.GradientTape() as tape:
    predictions = model(images)
    loss = loss_object(labels, predictions)
  gradients = tape.gradient(loss, model.trainable_variables)
  optimizer.apply_gradients(zip(gradients, model.trainable_variables))

  train_loss(loss)
  train_accuracy(labels, predictions)
```
Shouldn't the input be passed like below:
```
model = MyModel()
model.call(images)
```

Response:
Actually __call__ method is implemented in the Layer class, which is inherited by Network class, which is inherited by Model class:
```
class Layer(module.Module):
    def __call__(self, inputs, *args, **kwargs):

class Network(base_layer.Layer):

class Model(network.Network):
```
So MyClass will inherit this __call__ method. Additional info: So actually what we do is overriding the inherited call method, which new call method will be then called from the inherited __call__ method. That is why we don't need to do a model.call(). So when we call our model instance, it's inherited __call__ method will be executed automatically, which calls our own call method.


Question:
Batch_size in tensorflow? Understanding the concept My question is simple and stright forward. What does a batch size specify while training and predicting a neural network. How to visualize it so as to get a clear picture of how data is being feed to the network. Suppose I have an autoencoder
```
encoder = tflearn.input_data(shape=[None, 41])
encoder = tflearn.fully_connected(encoder, 41,activation='relu')
```
and I am taking an input as csv file with 41 features, So as to what I understand is it will take each feature from csv file and feed it to the 41 neurons of the first layer when my batch size is 1. But when I increase the batch size to 100 how is this 41 features of 100 batches are going to be feed to this network?
```
model.fit(test_set, test_labels_set, n_epoch=1, validation_set=(valid_set, valid_labels_set),
          run_id="auto_encoder", batch_size=100,show_metric=True, snapshot_epoch=False)
```
Will there be any normalization on the batches or some operations on them? The number of epocs are same for both the cases

Response: The batch size is the amount of samples you feed in your network. For your input encoder you specify that you enter an unspecified(None) amount of samples with 41 values per sample. The advantage of using None is that you can now train with batches of 100 values at once (which is good for your gradient), and test with a batch of only one value (one sample for which you want a prediction). If you don't specify normalization per batch there is no normalization per batch ;) Hope I explained it well enough! If you have more questions feel free to ask them!


Question:
What does from_logits = True or False mean in sparse_categorical_crossentropy of Tensorflow? In Tensorflow 2.0, there is a loss function called
```
tf.keras.losses.sparse_categorical_crossentropy(labels, targets, from_logits = False)
```
What are the differences between setting from_logits = True or False? My guess was that when incoming values are logits, you set from_logits = True, and if incoming values are probabilities(output by softmax etc.) then you just set from_logits = False (which is a default setting). But why? loss is just some calculation. Why does it need to differ by its incoming values? I also saw in google's tensorflow tutorial https://www.tensorflow.org/alpha/tutorials/sequences/text_generation that it doesnt set from_logits = True even if incoming values of the last layer are logits. Here is the code
```
@tf.function
def train_step(inp, target):
  with tf.GradientTape() as tape:
    predictions = model(inp)
    loss = tf.reduce_mean(
        tf.keras.losses.sparse_categorical_crossentropy(target, predictions))
  grads = tape.gradient(loss, model.trainable_variables)
  optimizer.apply_gradients(zip(grads, model.trainable_variables))

  return loss
```
where the model is
```
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim,
                              batch_input_shape=[batch_size, None]),
    tf.keras.layers.LSTM(rnn_units,
                        return_sequences=True,
                        stateful=True,
                        recurrent_initializer='glorot_uniform'),
    tf.keras.layers.Dense(vocab_size)
  ])
```
which does not have the last layer of softmax. (Also, in another part of the tutorial, it set from_logits = True) So, doesn't it matter whether I set it True or not?

Response: The post Deepak mentioned has some math background. But for simplicity, from_logits=True means the input to crossEntropy layer is normal tensor/logits, while if from_logits=False, means the input is a probability and usually you should have some softmax activation in your last layer.


Question:
Weighted mse custom loss function in keras I'm working with time series data, outputting 60 predicted days ahead. I'm currently using mean squared error as my loss function and the results are bad I want to implement a weighted mean squared error such that the early outputs are much more important than later ones. Weighted Mean Square Root formula: So I need some way to iterate over a tensor's elements, with an index (since I need to iterate over both the predicted and the true values at the same time, then write the results to a tensor with only one element. They're both (?,60) but really (1,60) lists. And nothing I'm trying is working. Here's the code for the broken version
```
def weighted_mse(y_true,y_pred):
    wmse = K.cast(0.0,'float')

    size = K.shape(y_true)[0]
    for i in range(0,K.eval(size)):
        wmse += 1/(i+1)*K.square((y_true[i]-y_pred)[i])

    wmse /= K.eval(size)
    return wmse
```
I am currently getting this error as a result:
```
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'dense_2_target' with dtype float
 [[Node: dense_2_target = Placeholder[dtype=DT_FLOAT, shape=[], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
```
Having read the replies to similar posts, I don't think a mask can accomplish the task, and looping over elements in one tensor would also not work since I'd not be able to access the corresponding element in the other tensor. Any suggestions would be appreciated

Response:
You can use this approach:
```
def weighted_mse(yTrue,yPred):

    ones = K.ones_like(yTrue[0,:]) #a simple vector with ones shaped as (60,)
    idx = K.cumsum(ones) #similar to a 'range(1,61)'

    return K.mean((1/idx)*K.square(yTrue-yPred))
```
The use of ones_like with cumsum allows you to use this loss function to any kind of (samples,classes) outputs. Hint: always use backend functions when working with tensors. You can use slices, but avoid iterating.


Question:
How to visualize a tensor summary in tensorboard I'm trying to visualize a tensor summary in tensorboard. However I can't see the tensor summary at all in the board. Here is my code:
```
out = tf.strided_slice(logits, begin=[self.args.uttWindowSize-1, 0], end=[-self.args.uttWindowSize+1, self.args.numClasses],
                               strides=[1, 1], name='softmax_truncated')
        tf.summary.tensor_summary('softmax_input', out)
```
where out is a multi-dimensional tensor. I guess there must be something wrong with my code. Probably I used the tensor_summary function incorrectly.

Response:
What you do is you create a summary op, but you don't invoke it and don't write the summary (see documentation). To actually create a summary you need to do the following:
```
# Create a summary operation
summary_op = tf.summary.tensor_summary('softmax_input', out)

# Create the summary
summary_str = sess.run(summary_op)

# Create a summary writer
writer = tf.train.SummaryWriter(...)

# Write the summary
writer.add_summary(summary_str)
```
Explicitly writing a summary (last two lines) is only necessary if you don't have a higher level helper like a Supervisor. Otherwise you invoke
```
sv.summary_computed(sess, summary_str)
```
and the Supervisor will handle it. More info, also see: How to manually create a tf.Summary()


Question:
What is the correct way to create representative dataset for TFliteconverter? I am trying to infer tinyYOLO-V2 with INT8 weights and activation. I can convert the weights to INT8 with TFliteConverter. For INT8 activation, I have to give representative dataset to estimate the scaling factor. My method of creating such dataset seems wrong. What is the correct procedure ?
```
def rep_data_gen():
    a = []
    for i in range(160):
        inst = anns[i]
        file_name = inst['filename']
        img = cv2.imread(img_dir + file_name)
        img = cv2.resize(img, (NORM_H, NORM_W))
        img = img / 255.0
        img = img.astype('float32')
        a.append(img)
    a = np.array(a)
    print(a.shape) # a is np array of 160 3D images
    img = tf.data.Dataset.from_tensor_slices(a).batch(1)
    for i in img.take(BATCH_SIZE):
        print(i)
        yield [i]
# https://www.tensorflow.org/lite/performance/post_training_quantization
converter = tf.lite.TFLiteConverter.from_keras_model_file("./yolo.h5")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = [tf.int8]
converter.inference_output_type = [tf.int8]
converter.representative_dataset=rep_data_gen
tflite_quant_model = converter.convert()
```
ValueError: Cannot set tensor: Got tensor of type STRING but expected type FLOAT32 for input 27, name: input_1

Response: I used your code for reading in a dataset and found the error: img = img.astype('float32') should be img = img.astype(np.float32) Hope this helps


Question:
Tensorflow: What exact formula is applied in `tf.nn.sparse_softmax_cross_entropy_with_logits`? I tried to manually recompute the outputs of this function so I created a minimal example:
```
logits = tf.pack(np.array([[[[0,1,2]]]],dtype=np.float32)) # img of shape (1, 1, 1, 3)
labels = tf.pack(np.array([[[1]]],dtype=np.int32)) # gt of shape (1, 1, 1)

softmaxCrossEntropie = tf.nn.sparse_softmax_cross_entropy_with_logits(logits,labels)
softmaxCrossEntropie.eval() # --> output is [1.41]
```
Now according to my own calculation I only get [1.23] When manually calculating, I'm simply applying softmax and cross-entropy: where q(x) = sigma(x_j) or (1-sigma(x_j)) depending whether j is the correct ground truth class or not and p(x) = labels which are then one-hot-encoded I'm not sure where the difference might originate from. I cannot really imagine that some epsilon causes such a big difference. Does someone know where I can lookup, which exact formula is used by tensorflow? Is the source code of that exact part available? I could only find nn_ops.py, but it only uses another function called gen_nn_ops._sparse_softmax_cross_entropy_with_logits which I couldn't find on github...

Response: Well, usually p(x) in cross-entropy equation is true distribution, while q(x) is the distribution obtained from softmax. So, if p(x) is one-hot (and this is so, otherwise sparse cross-entropy could not be applied), cross entropy is just negative log for probability of true category. In your example, softmax(logits) is a vector with values [0.09003057, 0.24472847, 0.66524096], so the loss is -log(0.24472847) = 1.4076059 which is exactly what you got as output.


Question:
Mean average precision (mAP) in tensorflow I need to calculate the mAP described in this question for object detection using Tensorflow. Average precision(AP) is a typical performance measure used for ranked sets. AveragePrecision is defined as the average of the precision scores after each true positive, TP in the scope S. Given a scope S = 7,and a ranked list (gain vector) G = [1,1,0,1,1,0,0,1,1,0,1,0,0,..] where 1/0 indicate the gains associated to relevant/non-‐relevant items, respectively: AP = (1/1 + 2/2 + 3/4 + 4/5) / 4 = 0.8875. Mean Average Precision (mAP): average of the average precision value for a set of queries. i got 5 One-Hot tensors with the predictions:
```
prediction_A
prediction_B
prediction_C
prediction_D
prediction_E
```
where a single prediction tensor has this structure (for example prediction_A):
```
00100
01000
00001
00010
00010
```
Then i've got the correct labels (one-hot) tensors, with the same structure:
```
y_A
y_B
y_C
y_D
y_E
```
i want compute mAP using tensorflow, cause i want summarize that, how i can do it? i found this function but i can't use it, cause i have a multidimensional vector. I also write a python function that compute AP but it doesn't use Tensorflow
```
def compute_av_precision(match_list):
    n = len(match_list)
    tp_counter = 0

    cumulate_precision = 0
    for i in range(0,n):
        if match_list[i] == True:

            tp_counter += 1

            cumulate_precision += (float(tp_counter)/float(i+1))

    if tp_counter != 0:
        av_precision = cumulate_precision/float(tp_counter)
        return av_precision
    return 0
```

Response:
I think you may need this one:
```
tf.metrics.average_precision_at_k
```
this method takes labels and prediction to calcualte the AP@K you mentioned below are the referenced links https://www.tensorflow.org/api_docs/python/tf/metrics/average_precision_at_k which implemented AP@K metric defined here: https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Average_precision BTW, if you need a metric in Tensorflow, firstly you should search inside their official documents. Here is a list of all implemented metrics https://www.tensorflow.org/api_docs/python/tf/metrics cheers


Question:
Difference between tf.assign and assignment operator (=) I'm trying to understand the difference between tf.assign and the assignment operator(=). I have three sets of code First, using simple tf.assign
```
import tensorflow as tf

with tf.Graph().as_default():
  a = tf.Variable(1, name="a")
  assign_op = tf.assign(a, tf.add(a,1))
  with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print sess.run(assign_op)
    print a.eval()
    print a.eval()
```
The output is expected as
```
2
2
2
```
Second, using assignment operator
```
import tensorflow as tf

with tf.Graph().as_default():
  a = tf.Variable(1, name="a")
  a = a + 1
  with tf.Session() as sess:
   sess.run(tf.global_variables_initializer())
   print sess.run(a)
   print a.eval()
   print a.eval()
```
The results are still 2, 2, 2. Third, I use both tf.assign and assignment operator
```
import tensorflow as tf

with tf.Graph().as_default():
  a = tf.Variable(1, name="a")
  a = tf.assign(a, tf.add(a,1))
  with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print sess.run(a)
    print a.eval()
    print a.eval()
```
Now, the output becomes 2, 3, 4. My questions are In the 2nd snippet using (=), when I have sess.run(a), it seems I'm running an assign op. So does "a = a+1" internally create an assignment op like assign_op = tf.assign(a, a+1)? Is the op run by the session really just the assign_op? But when I run a.eval(), it doesn't continue to increment a, hence it appears eval is evaluating a "static" variable. I'm not sure how to explain the 3rd snippet. Why the two evals increment a, but the two evals in the 2nd snippet doesn't? Thanks.

Response:
The main confusion here is that doing a = a + 1 will reassign the Python variable a to the resulting tensor of the addition operation a + 1. tf.assign, on the other hand, is an operation for setting the value of a TensorFlow variable.
```
a = tf.Variable(1, name="a")
a = a + 1
```
This is equivalent to:
```
a = tf.add(tf.Variable(1, name="a"), 1)
```
With that in mind: In the 2nd snippet using (=), when I have sess.run(a), it seems I'm running an assign op. So does "a = a+1" internally create an assignment op like assign_op = tf.assign(a, a+1)? [...] It might look so, but not true. As explained above, this will only reassign the Python variable. And without tf.assign or any other operation that changes the variable, it stays with the value 1. Each time a is evaluated, the program will always calculate a + 1 => 1 + 1. I'm not sure how to explain the 3rd snippet. Why the two evals increment a, but the two evals in the 2nd snippet doesn't? That's because calling eval() on the assignment tensor in the third snippet also triggers the variable assignment (note that this isn't much different from doing session.run(a) with the current session).


Question: Tensorflow Serving - Stateful LSTM Is there a canonical way to maintain a stateful LSTM, etc. with Tensorflow Serving? Using the Tensorflow API directly this is straightforward - but I'm not certain how best to accomplish persisting LSTM state between calls after exporting the model to Serving. Are there any examples out there which accomplish the above? The samples within the repo are very basic.

Response: From Martin Wicke on the TF mailing list: "There's no good integration for stateful models in the model server yet. As you noted, it basically assumes models are pure functions. We're working on this, and you should see this functionality appear eventually, but it's too far out to promise a time. So in the meantime, you can write a simple wrapper which keeps state on the server (and assigns some sort of session ID which is passed around in requests), or you can write your own server which maintains the TensorFlow session state (and similarly returns some session ID). The latter is more performant. Both will require some sort of garbage collection/session timeout logic."


Question:
How can i use "leaky_relu" as an activation in Tensorflow "tf.layers.dense"? Using Tensorflow 1.5, I am trying to add leaky_relu activation to the output of a dense layer while I am able to change the alpha of leaky_relu (check here). I know I can do it as follows:
```
output = tf.layers.dense(input, n_units)
output = tf.nn.leaky_relu(output, alpha=0.01)
```
I was wondering if there is a way to write this in one line as we can do for relu: ouput = tf.layers.dense(input, n_units, activation=tf.nn.relu) I tried the following but I get an error:
```
output = tf.layers.dense(input, n_units, activation=tf.nn.leaky_relu(alpha=0.01))
TypeError: leaky_relu() missing 1 required positional argument: 'features'
```
Is there a way to do this?

Response:
If you're really adamant about a one liner for this, you could use the partial() method from the functools module, as follow:
```
import tensorflow as tf
from functools import partial

output = tf.layers.dense(input, n_units, activation=partial(tf.nn.leaky_relu, alpha=0.01))
```
It should be noted that partial() does not work for all operations and you might have to try your luck with partialmethod() from the same module. Hope this helps you in your endeavour.


Question:
What does a tensorflow "op" do? Below I have created a tf placeholder named "op testing":
```
self.center_words = tf.placeholder(tf.int32, shape=[self.batch_size], name='op testing')
print("Extracting the op", self.center_words.op)
```
When I print that self.center_words.op it prints out a structure like this:
```
op: "Placeholder"
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "shape"
  value {
    shape {
      dim {
        size: 128
      }
    }
  }
}
```
This works for any TensorFlow variable, function output, etc. What is this .op?

Response: TensorFlow Operations, also known as Ops, are nodes that perform computations on or with Tensor objects. After computation, they return zero or more tensors, which can be used by other Ops later in the graph. To create an Operation, you call its constructor in Python, which takes in whatever Tensor parameters needed for its calculation, known as inputs, as well as any additional information needed to properly create the Op, known as attributes. The Python constructor returns a handle to the Operation’s output (zero or more Tensor objects), and it is this output which can be passed on to other Operations or Session.run


Question: Analysis of the output from tf.nn.dynamic_rnn tensorflow function I am not able to understand the output from tf.nn.dynamic_rnn tensorflow function. The document just tells about the size of the output, but it doesn't tell what does each row/column means. From the documentation: outputs: The RNN output Tensor. If time_major == False (default), this will be a Tensor shaped: [batch_size, max_time, cell.output_size]. If time_major == True, this will be a Tensor shaped: [max_time, batch_size, cell.output_size]. Note, if cell.output_size is a (possibly nested) tuple of integers or TensorShape objects, then outputs will be a tuple having the same structure as cell.output_size, containing Tensors having shapes corresponding to the shape data in cell.output_size. state: The final state. If cell.state_size is an int, this will be shaped [batch_size, cell.state_size]. If it is a TensorShape, this will be shaped [batch_size] + cell.state_size. If it is a (possibly nested) tuple of ints or TensorShape, this will be a tuple having the corresponding shapes. The outputs tensor is a 3-D matrix but what does each row/column represent?

Response: tf.dynamic_rnn provides two outputs, outputs and state. outputs contains the output of the RNN cell at every time instant. Assuming the default time_major == False, let's say you have an input composed of 10 examples with 7 time steps each and a feature vector of size 5 for every time step. Then your input would be 10x7x5 (batch_sizexmax_timexfeatures). Now you give this as an input to a RNN cell with output size 15. Conceptually, each time step of each example is input to the RNN, and you would get a 15-long vector for each of those. So that is what outputs contains, a tensor in this case of size 10x7x15 (batch_sizexmax_timexcell.output_size) with the output of the RNN cell at each time step. If you are only interested in the last output of the cell, you can just slice the time dimension to pick just the last element (e.g. outputs[:, -1, :]). state contains the state of the RNN after processing all the inputs. Note that, unlike outputs, this doesn't contain information about every time step, but only about the last one (that is, the state after the last one). Depending on your case, the state may or may not be useful. For example, if you have very long sequences, you may not want/be able to processes them in a single batch, and you may need to split them into several subsequences. If you ignore the state, then whenever you give a new subsequence it will be as if you are beginning a new one; if you remember the state, however (e.g. outputting it or storing it in a variable), you can feed it back later (through the initial_state parameter of tf.nn.dynamic_rnn) in order to correctly keep track of the state of the RNN, and only reset it to the initial state (generally all zeros) after you have completed the whole sequences. The shape of state can vary depending on the RNN cell that you are using, but, in general, you have some state for each of the examples (one or more tensors with size batch_sizexstate_size, where state_size depends on the cell type and size).


Question: What is tf.bfloat16 "truncated 16-bit floating point"? What is the difference between tf.float16 and tf.bfloat16 as listed in https://www.tensorflow.org/versions/r0.12/api_docs/python/framework/tensor_types ? Also, what do they mean by "quantized integer"?

Response:
bfloat16 is a tensorflow-specific format that is different from IEEE's own float16, hence the new name. The b stands for (Google) Brain. Basically, bfloat16 is a float32 truncated to its first 16 bits. So it has the same 8 bits for exponent, and only 7 bits for mantissa. It is therefore easy to convert from and to float32, and because it has basically the same range as float32, it minimizes the risks of having NaNs or exploding/vanishing gradients when switching from float32. From the sources:
```
// Compact 16-bit encoding of floating point numbers. This representation uses
// 1 bit for the sign, 8 bits for the exponent and 7 bits for the mantissa.  It
// is assumed that floats are in IEEE 754 format so the representation is just
// bits 16-31 of a single precision float.
//
// NOTE: The IEEE floating point standard defines a float16 format that
// is different than this format (it has fewer bits of exponent and more
// bits of mantissa).  We don't use that format here because conversion
// to/from 32-bit floats is more complex for that format, and the
// conversion for this format is very simple.
```
As for quantized integers, they are designed to replace floating points in trained networks to speed up processing. Basically, they are a sort of fixed point encoding of real numbers, albeit with an operating range that is chosen to represent the observed distribution at any given point of the net. More on quantization here.


Question:
Using Training TFRecords that are stored on Google Cloud My goal is to use training data (format: tfrecords) stored on Google Cloud storage when I run my Tensorflow Training App, locally. (Why locally? : I am testing before I turn it into a training package for Cloud ML) Based on this thread I shouldn't have to do anything since the underlying Tensorflow API's should be able to read a gs://(url) However thats not the case and the errors I see are of the format: 2017-06-06 15:38:55.589068: I tensorflow/core/platform/cloud/retrying_utils.cc:77] The operation failed and will be automatically retried in 1.38118 seconds (attempt 1 out of 10), caused by: Unavailable: Error executing an HTTP request (HTTP response code 0, error code 6, error message 'Couldn't resolve host 'metadata'') 2017-06-06 15:38:56.976396: I tensorflow/core/platform/cloud/retrying_utils.cc:77] The operation failed and will be automatically retried in 1.94469 seconds (attempt 2 out of 10), caused by: Unavailable: Error executing an HTTP request (HTTP response code 0, error code 6, error message 'Couldn't resolve host 'metadata'') 2017-06-06 15:38:58.925964: I tensorflow/core/platform/cloud/retrying_utils.cc:77] The operation failed and will be automatically retried in 2.76491 seconds (attempt 3 out of 10), caused by: Unavailable: Error executing an HTTP request (HTTP response code 0, error code 6, error message 'Couldn't resolve host 'metadata'') I'm not able to follow where I have to begin debugging this error. Here is a snippet that reproduced the problem and also shows the tensorflow API's that I am using.
```
def _preprocess_features(features):
        """Function that returns preprocessed images"""

def _parse_single_example_from_tfrecord(value):
    features = (
        tf.parse_single_example(value,
                                features={'image_raw': tf.FixedLenFeature([], tf.string),
                                          'label': tf.FixedLenFeature([model_config.LABEL_SIZE], tf.int64)
                                          })
        )
    return features

def _read_and_decode_tfrecords(filename_queue):
    reader = tf.TFRecordReader()
    # Point it at the filename_queue
    _, value = reader.read(filename_queue)
    features = _parse_single_example_from_tfrecord(value)
    # decode the binary string image data
    image, label = _preprocess_features(features)
    return image, label

def test_tfread(filelist):
  train_filename_queue = (
    tf.train.string_input_producer(filelist,
                                   num_epochs=None,
                                   shuffle=True))
  image, label = (
    _read_and_decode_tfrecords(train_filename_queue))
  return image

images= test_tfread(["gs://test-bucket/t.tfrecords"])
sess = tf.Session(config=tf.ConfigProto(
                allow_soft_placement=True,
                log_device_placement=True))
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(sess=sess, coord=coord)
try:
  for step in range(model_config.MAX_STEPS):
      _ = sess.run([images])
finally:
  # When done, ask the threads to stop.
  coord.request_stop()
# Finally, wait for them to join (i.e. cleanly shut down)
coord.join(threads)
```

Response: Try executing the following command gcloud auth application-default login


Question:
additive Gaussian noise in Tensorflow I'm trying to add Gaussian noise to a layer of my network in the following way.
```
def Gaussian_noise_layer(input_layer, std):
    noise = tf.random_normal(shape = input_layer.get_shape(), mean = 0.0, stddev = std, dtype = tf.float32)
    return input_layer + noise
```
I'm getting the error: ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 2600, 2000, 1) My minibatches need to be of different sizes sometimes, so the size of the input_layer tensor will not be known until the execution time. If I understand correctly, someone answering Cannot convert a partially converted tensor in TensorFlow suggested to set shape to tf.shape(input_layer). However then, when I try to apply a convolutional layer to that noisy layer I get another error: ValueError: dims of shape must be known but is None What is the correct way of achieving my goal of adding Gaussian noise to the input layer of a shape unknown until the execution time?

Response:
To dynamically get the shape of a tensor with unknown dimensions you need to use tf.shape() For instance
```
import tensorflow as tf
import numpy as np

def gaussian_noise_layer(input_layer, std):
    noise = tf.random_normal(shape=tf.shape(input_layer), mean=0.0, stddev=std, dtype=tf.float32)
    return input_layer + noise

inp = tf.placeholder(tf.float32, shape=[None, 8], name='input')
noise = gaussian_noise_layer(inp, .2)
noise.eval(session=tf.Session(), feed_dict={inp: np.zeros((4, 8))})
```


Question: Debugging nans in the backward pass I'm trying to debug a somewhat complicated and non-canonical NN architecture. Computing the forward pass is fine and is giving me the expected results, but when I try to optimize using Adam or any of the standard optimizers, even after one iteration with a very small learning rate I get nans everywhere. I'm trying to localize them and was wondering if there's a way to catch the first occurrence of a nan and detect in which op it arose? I tried tf.add_check_numerics_ops() but it doesn't appear to be doing anything, or perhaps I'm using it incorrectly.

Response:
Debugging NaNs can be tricky, especially if you have a large network. tf.add_check_numerics_ops() adds ops to the graph that assert that each floating point tensor in the graph does not contain any NaN values, but does not run these checks by default. Instead it returns an op that you can run periodically, or on every step, as follows:
```py
train_op = ...
check_op = tf.add_check_numerics_ops()

sess = tf.Session()
sess.run([train_op, check_op])  # Runs training and checks for NaNs
```


Question: Keras VGG16 preprocess_input modes I'm using the Keras VGG16 model. I've seen it there is a preprocess_input method to use in conjunction with the VGG16 model. This method appears to call the preprocess_input method in imagenet_utils.py which (depending on the case) calls _preprocess_numpy_input method in imagenet_utils.py. The preprocess_input has a mode argument which expects "caffe", "tf", or "torch". If I'm using the model in Keras with TensorFlow backend, should I absolutely use mode="tf"? If yes, is this because the VGG16 model loaded by Keras was trained with images which underwent the same preprocessing (i.e. changed input image's range from [0,255] to input range [-1,1])? Also, should the input images for testing mode also undergo this preprocessing? I'm confident the answer to the last question is yes, but I would like some reassurance. I would expect Francois Chollet to have done it correctly, but looking at https://github.com/fchollet/deep-learning-models/blob/master/vgg16.py either he is or I am wrong about using mode="tf". Updated info @FalconUA directed me to the VGG at Oxford which has a Models section with links for the 16-layer model. The information about the preprocessing_input mode argument tf scaling to -1 to 1 and caffe subtracting some mean values is found by following the link in the Models 16-layer model: information page. In the Description section it says: "In the paper, the model is denoted as the configuration D trained with scale jittering. The input images should be zero-centered by mean pixel (rather than mean image) subtraction. Namely, the following BGR values should be subtracted: [103.939, 116.779, 123.68]."

Response: The mode here is not about the backend, but rather about on what framework the model was trained on and ported from. In the keras link to VGG16, it is stated that: These weights are ported from the ones released by VGG at Oxford So the VGG16 and VGG19 models were trained in Caffe and ported to TensorFlow, hence mode == 'caffe' here (range from 0 to 255 and then extract the mean [103.939, 116.779, 123.68]). Newer networks, like MobileNet and ShuffleNet were trained on TensorFlow, so mode is 'tf' for them and the inputs are zero-centered in the range from -1 to 1.


Question: How do I use distributed DNN training in TensorFlow? Google released TensorFlow today. I have been poking around in the code, and I don't see anything in the code or API about training across a cluster of GPU servers. Does it have distributed training functionality yet?

Response: Updated: Distributed TensorFlow Documentation Distributed TensorFlow Source The release occurred on 2/26/2016 and was announced by coauthor Derek Murray in the original issue here and uses gRPC for inter-process communication. Previous: Before the update above, a distributed implementation of TensorFlow had not been released yet. Support for a distributed implementation was the topic of this issue where coauthor Vijay Vasudevan wrote: we are working on making a distributed implementation available, it's currently not in the initial release and Jeff Dean later provided an update: Our current internal distributed extensions are somewhat entangled with Google internal infrastructure, which is why we released the single-machine version first. The code is not yet in GitHub, because it has dependencies on other parts of the Google code base at the moment, most of which have been trimmed, but there are some remaining ones. We realize that distributed support is really important, and it's one of the top features we're prioritizing at the moment.


Question:
tf.gradients is not supported when eager execution is enabled. Use tf.GradientTape instead
```
from tensorflow.keras.applications import VGG16
from tensorflow.keras import backend as K

model = VGG16(weights='imagenet',
              include_top=False)

layer_name = 'block3_conv1'
filter_index = 0

layer_output = model.get_layer(layer_name).output
loss = K.mean(layer_output[:, :, :, filter_index])

grads = K.gradients(loss, model.input)[0]
```
I am unable to execute grads = K.gradients(loss, model.input)[0], it generates an error : tf.gradients is not supported when eager execution is enabled. Use tf.GradientTape instead

Response:
You have two options to resolve this error: .gradients is deprecated in TF2 - Replace gradients with GradientTape as suggested here https://github.com/tensorflow/tensorflow/issues/33135 Simply disable the eager-execution constrain form tf2 with the compat mode for tf1 Example running code for solution 2:
```
from tensorflow.keras.applications import VGG16
from tensorflow.keras import backend as K
import tensorflow as tf
tf.compat.v1.disable_eager_execution()

model = VGG16(weights='imagenet',
              include_top=False)

layer_name = 'block3_conv1'
filter_index = 0

layer_output = model.get_layer(layer_name).output
loss = K.mean(layer_output[:, :, :, filter_index])

grads = K.gradients(loss, model.input)[0]
```


Question:
AttributeError: module 'tensorflow_core._api.v2.config' has no attribute 'list_physical_devices' Am using Tensorflow 2.0 on Ubuntu 18.04. On running
```
tf.config.list_physical_devices('GPU')
```
I get the above error. What is the workaround for this?

Response: Run tf.config.experimental.list_physical_devices('GPU') to see all the GPUs or run tf.config.experimental.list_physical_devices(device_type=None) to see all the devices You can also use tf.test.is_gpu_available(). It says True if it detects the available gpu.


Question:
tf.data.Dataset.padded_batch pad differently each feature I have a tf.data.Dataset instance which holds 3 different features label which is a scalar sequence_feature which is a sequence of scalars seq_of_seqs_feature which is a sequence of sequences feature I am trying to use tf.data.Dataset.padded_batch() to genereate padded data as input to my model - and I want to pad every feature differently. Example batch:
```
[{'label': 24,
  'sequence_feature': [1, 2],
  'seq_of_seqs_feature': [[11.1, 22.2],
                          [33.3, 44.4]]},
 {'label': 32,
  'sequence_feature': [3, 4, 5],
  'seq_of_seqs_feature': [[55.55, 66.66]]}]
```
Expected output:
```
[{'label': 24,
  'sequence_feature': [1, 2, 0],
  'seq_of_seqs_feature': [[11.1, 22.2],
                          [33.3, 44.4]]},
 {'label': 32,
  'sequence_feature': [3, 4, 5],
  'seq_of_seqs_feature': [[55.55, 66.66],
                           0.0, 0.0    ]}]
```
As you can see the label feature should not be padded, and the sequence_feature and seq_of_seqs_feature should be padded by the corresponding longest entry in the given batch.

Response:
The tf.data.Dataset.padded_batch() method allows you to specify padded_shapes for each component (feature) of the resulting batch. For example, if your input dataset is called ds:
```
padded_ds = ds.padded_batch(
    BATCH_SIZE,
    padded_shapes={
        'label': [],                          # Scalar elements, no padding.
        'sequence_feature': [None],           # Vector elements, padded to longest.
        'seq_of_seqs_feature': [None, None],  # Matrix elements, padded to longest
    })                                        # in each dimension.
```
Notice that the padded_shapes argument has the same structure as your input dataset's elements, so in this case it takes a dictionary with keys that match your feature names.


Question:
TensorFlow getting elements of every row for specific columns If A is a TensorFlow variable like so
```
A = tf.Variable([[1, 2], [3, 4]])
```
and index is another variable
```
index = tf.Variable([0, 1])
```
I want to use this index to select columns in each row. In this case, item 0 from first row and item 1 from second row. If A was a Numpy array then to get the columns of corresponding rows mentioned in index we can do
```
x = A[np.arange(A.shape[0]), index]
```
and the result would be
```
[1, 4]
```
What is the TensorFlow equivalent operation/operations for this? I know TensorFlow doesn't support many indexing operations. What would be the work around if it cannot be done directly?

Response:
You can extend your column indices with row indices and then use gather_nd:
```
import tensorflow as tf

A = tf.constant([[1, 2], [3, 4]])
indices = tf.constant([1, 0])

# prepare row indices
row_indices = tf.range(tf.shape(indices)[0])

# zip row indices with column indices
full_indices = tf.stack([row_indices, indices], axis=1)

# retrieve values by indices
S = tf.gather_nd(A, full_indices)

session = tf.InteractiveSession()
session.run(S)
```


Question: How can I reuse a Dense layer? I have a network in Tensorflow, and I want to define a function that passes it's input through a tf.layers.dense layer (obviously, the same one). I see the reuse argument, but in order to use it properly it seems I need to keep a global variable just to remember if my function was called already. Is there a cleaner way?

Response:
I find tf.layers.Dense cleaner than the above answers. All you need is a Dense object defined beforehand. Then you can reuse it any number of times.
```
import tensorflow as tf

# Define Dense object which is reusable
my_dense = tf.layers.Dense(3, name="optional_name")

# Define some inputs
x1 = tf.constant([[1,2,3], [4,5,6]], dtype=tf.float32)
x2 = tf.constant([[4,5,6], [7,8,9]], dtype=tf.float32)

# Use the Dense layer
y1 = my_dense(x1)
y2 = my_dense(x2)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    y1 = sess.run(y1)
    y2 = sess.run(y2)
    print(y1)
    print(y2)
```
In fact tf.layers.dense function internally constructs a Dense object and pass your input to that object. For more details, check the code.