Question:
How does shuffling work with ImageDataGenerator in Machine Learning? I'm creating an image classification model with Inception V3 and have two classes. I've split my dataset and labels into two numpy arrays.The data is split with trainX and testY as the images and trainY and testY as the corresponding labels.
```
data = np.array(data, dtype="float")/255.0
labels = np.array(labels,dtype ="uint8")

(trainX, testX, trainY, testY) = train_test_split(
                                data,labels,
                                test_size=0.2,
                                random_state=42)

train_datagen = keras.preprocessing.image.ImageDataGenerator(
          zoom_range = 0.1,
          width_shift_range = 0.2,
          height_shift_range = 0.2,
          horizontal_flip = True,
          fill_mode ='nearest')

val_datagen = keras.preprocessing.image.ImageDataGenerator()

train_generator = train_datagen.flow(
        trainX,
        trainY,
        batch_size=batch_size,
        shuffle=True)

validation_generator = val_datagen.flow(
                testX,
                testY,
                batch_size=batch_size)
```
When I shuffle train_generator with ImageDataGenerator, will the images still match the corresponding labels? Also should the validation dataset be shuffled as well?

Response:
Yes, the images will still match the corresponding labels so you can safely set shuffle to True. Under the hood it works as follows. Calling .flow() on the ImageDataGenerator will return you a NumpyArrayIterator object, which implements the following logic for shuffling the indices:
```
def _set_index_array(self):
    self.index_array = np.arange(self.n)
    if self.shuffle: # if shuffle==True, shuffle the indices
        self.index_array = np.random.permutation(self.n)
```
self.index_array is then used to yield both the images (x) and the labels (y) (code truncated for readability):
```
def _get_batches_of_transformed_samples(self, index_array):
    batch_x = np.zeros(tuple([len(index_array)] + list(self.x.shape)[1:]),
                       dtype=self.dtype)
    # use index_array to get the x's
    for i, j in enumerate(index_array):
        x = self.x[j]
        ... # data augmentation is done here
        batch_x[i] = x
     ...
     # use the same index_array to fetch the labels
     output += (self.y[index_array],)

    return output
```
Check out the source code yourself, it might be easier to understand than you think. Shuffling the validation data shouldn't matter too much. The main point of shuffling is to introduce some extra stochasticity in the training process.


Question:
Why is the Mean Average Percentage Error(mape) extremely high? I have obtained code from machinelearningmastery I modified the model.compile() function to add mape metrics to find out the Mean Absolute Percentage Error. After running the code, the mape at every epoch comes so huge, considering it as a percentage metric. Am I missing something obvious or is the output right? The output looks like:
```
Epoch 91/100
0s - loss: 0.0103 - mean_absolute_percentage_error: 1764997.4502
Epoch 92/100
0s - loss: 0.0103 - mean_absolute_percentage_error: 1765653.4924
Epoch 93/100
0s - loss: 0.0102 - mean_absolute_percentage_error: 1766505.5107
Epoch 94/100
0s - loss: 0.0102 - mean_absolute_percentage_error: 1766814.5450
Epoch 95/100
0s - loss: 0.0102 - mean_absolute_percentage_error: 1767510.8146
Epoch 96/100
0s - loss: 0.0101 - mean_absolute_percentage_error: 1767686.9054
Epoch 97/100
0s - loss: 0.0101 - mean_absolute_percentage_error: 1767076.2169
Epoch 98/100
0s - loss: 0.0100 - mean_absolute_percentage_error: 1767014.8481
Epoch 99/100
0s - loss: 0.0100 - mean_absolute_percentage_error: 1766592.8125
Epoch 100/100
0s - loss: 0.0100 - mean_absolute_percentage_error: 1766348.6332
```
My code that I ran (which omits the prediction part) goes as follows:
```
import numpy
from numpy import array
import matplotlib.pyplot as plt
from pandas import read_csv
import math
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error

# convert an array of values into a dataset matrix
def create_dataset(dataset, look_back=1):
        dataX, dataY = [], []
        for i in range(len(dataset)-look_back-1):
                a = dataset[i:(i+look_back), 0]
                dataX.append(a)
                dataY.append(dataset[i + look_back, 0])
        return numpy.array(dataX), numpy.array(dataY)
# fix random seed for reproducibility
numpy.random.seed(7)
# load the dataset
dataframe = read_csv('airlinepassdata.csv', usecols=[1], engine='python', skipfooter=3)
dataset = dataframe.values

#dataset = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0])
dataset = dataset.astype('float32')
# normalize the dataset
scaler = MinMaxScaler(feature_range=(0, 1))
dataset = scaler.fit_transform(dataset)
# split into train and test sets
train_size = int(len(dataset) * 0.67)
test_size = len(dataset) - train_size
train, test = dataset[0:train_size,:], dataset[train_size:len(dataset),:]
# reshape into X=t and Y=t+1
look_back = 1
trainX, trainY = create_dataset(train, look_back)
testX, testY = create_dataset(test, look_back)
# reshape input to be [samples, time steps, features]
trainX = numpy.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
testX = numpy.reshape(testX, (testX.shape[0], 1, testX.shape[1]))
# create and fit the LSTM network
model = Sequential()
model.add(LSTM(4, input_shape=(1, look_back)))
model.add(Dense(1))
model.compile(loss='mse', optimizer='adam', metrics=['mape'])
model.fit(trainX, trainY, nb_epoch=100, batch_size=50, verbose=2)
```

Response:
I solved this by setting the fuzz factor epsilon to one with keras.backend.set_epsilon(1) before calling the compile. The hint was in the source code
```
def mean_absolute_percentage_error(y_true, y_pred):
diff = K.abs((y_true - y_pred) / K.clip(K.abs(y_true),
                                        K.epsilon(),
                                        None))
return 100. * K.mean(diff, axis=-1)
```
Meaning that, for some unknown reason, the K.abs(y_true) term in the MAPE calculation on the training set is lower than the fuzz default (1e-7), so it uses that default value instead, thus the huge numbers.


Question: TensorFlow equivalent of numpy.all() As stated in the title, is there a TensorFlow equivalent of the numpy.all() function to check if all the values in a bool tensor are True? What is the best way to implement such a check?

Response:
Use tf.reduce_all, as follows:
```
import tensorflow as tf
a=tf.constant([True,False,True,True],dtype=tf.bool)
res=tf.reduce_all(a)
sess=tf.InteractiveSession()
res.eval()
```
This returns False. On the other hand, this returns True:
```
import tensorflow as tf
a=tf.constant([True,True,True,True],dtype=tf.bool)
res=tf.reduce_all(a)
sess=tf.InteractiveSession()
res.eval()
```


Question:
TypeError: argument of type 'float' is not iterable I am new to python and TensorFlow. I recently started understanding and executing TensorFlow examples, and came across this one: https://www.tensorflow.org/versions/r0.10/tutorials/wide_and_deep/index.html I got the error, TypeError: argument of type 'float' is not iterable, and I believe that the problem is with the following line of code:
```
df_train[LABEL_COLUMN] = (df_train['income_bracket'].apply(lambda x: '>50K' in x)).astype(int)
```
(income_bracket is the label column of the census dataset, with '>50K' being one of the possible label values, and the other label is '=<50K'. The dataset is read into df_train. The explanation provided in the documentation for the reason to do the above is, "Since the task is a binary classification problem, we'll construct a label column named "label" whose value is 1 if the income is over 50K, and 0 otherwise.") If anyone could explain me what is exactly happening and how should I fix it, that'll be great. I tried using Python2.7 and Python3.4, and I don't think that the problem is with the version of the language. Also, if anyone is aware of great tutorials for someone who is new to TensorFlow and pandas, please share the links. Complete program:
```
import pandas as pd
import urllib
import tempfile
import tensorflow as tf

gender = tf.contrib.layers.sparse_column_with_keys(column_name="gender", keys=["female", "male"])
race = tf.contrib.layers.sparse_column_with_keys(column_name="race", keys=["Amer-Indian-Eskimo", "Asian-Pac-Islander", "Black", "Other", "White"])
education = tf.contrib.layers.sparse_column_with_hash_bucket("education", hash_bucket_size=1000)
marital_status = tf.contrib.layers.sparse_column_with_hash_bucket("marital_status", hash_bucket_size=100)
relationship = tf.contrib.layers.sparse_column_with_hash_bucket("relationship", hash_bucket_size=100)
workclass = tf.contrib.layers.sparse_column_with_hash_bucket("workclass", hash_bucket_size=100)
occupation = tf.contrib.layers.sparse_column_with_hash_bucket("occupation", hash_bucket_size=1000)
native_country = tf.contrib.layers.sparse_column_with_hash_bucket("native_country", hash_bucket_size=1000)

age = tf.contrib.layers.real_valued_column("age")
age_buckets = tf.contrib.layers.bucketized_column(age, boundaries=[18, 25, 30, 35, 40, 45, 50, 55, 60, 65])
education_num = tf.contrib.layers.real_valued_column("education_num")
capital_gain = tf.contrib.layers.real_valued_column("capital_gain")
capital_loss = tf.contrib.layers.real_valued_column("capital_loss")
hours_per_week = tf.contrib.layers.real_valued_column("hours_per_week")

wide_columns = [gender, native_country, education, occupation, workclass, marital_status, relationship, age_buckets, tf.contrib.layers.crossed_column([education, occupation], hash_bucket_size=int(1e4)), tf.contrib.layers.crossed_column([native_country, occupation], hash_bucket_size=int(1e4)), tf.contrib.layers.crossed_column([age_buckets, race, occupation], hash_bucket_size=int(1e6))]

deep_columns = [
  tf.contrib.layers.embedding_column(workclass, dimension=8),
  tf.contrib.layers.embedding_column(education, dimension=8),
  tf.contrib.layers.embedding_column(marital_status, dimension=8),
  tf.contrib.layers.embedding_column(gender, dimension=8),
  tf.contrib.layers.embedding_column(relationship, dimension=8),
  tf.contrib.layers.embedding_column(race, dimension=8),
  tf.contrib.layers.embedding_column(native_country, dimension=8),
  tf.contrib.layers.embedding_column(occupation, dimension=8),
  age, education_num, capital_gain, capital_loss, hours_per_week]

model_dir = tempfile.mkdtemp()
m = tf.contrib.learn.DNNLinearCombinedClassifier(
    model_dir=model_dir,
    linear_feature_columns=wide_columns,
    dnn_feature_columns=deep_columns,
    dnn_hidden_units=[100, 50])

COLUMNS = ["age", "workclass", "fnlwgt", "education", "education_num",
  "marital_status", "occupation", "relationship", "race", "gender",
  "capital_gain", "capital_loss", "hours_per_week", "native_country", "income_bracket"]
LABEL_COLUMN = 'label'
CATEGORICAL_COLUMNS = ["workclass", "education", "marital_status", "occupation", "relationship", "race", "gender", "native_country"]
CONTINUOUS_COLUMNS = ["age", "education_num", "capital_gain", "capital_loss", "hours_per_week"]

train_file = tempfile.NamedTemporaryFile()
test_file = tempfile.NamedTemporaryFile()
urllib.urlretrieve("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data", train_file.name)
urllib.urlretrieve("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test", test_file.name)

df_train = pd.read_csv(train_file, names=COLUMNS, skipinitialspace=True)
df_test = pd.read_csv(test_file, names=COLUMNS, skipinitialspace=True, skiprows=1)
df_train[LABEL_COLUMN] = (df_train['income_bracket'].apply(lambda x: '>50K' in x)).astype(int)
df_test[LABEL_COLUMN] = (df_test['income_bracket'].apply(lambda x: '>50K' in x)).astype(int)

def input_fn(df):

  continuous_cols = {k: tf.constant(df[k].values)
                     for k in CONTINUOUS_COLUMNS}

  categorical_cols = {k: tf.SparseTensor(
      indices=[[i, 0] for i in range(df[k].size)],
      values=df[k].values,
      shape=[df[k].size, 1])
                      for k in CATEGORICAL_COLUMNS}

  feature_cols = dict(continuous_cols.items() + categorical_cols.items())
  label = tf.constant(df[LABEL_COLUMN].values)
  return feature_cols, label

def train_input_fn():
    return input_fn(df_train)

def eval_input_fn():
    return input_fn(df_test)

m.fit(input_fn=train_input_fn, steps=200)
results = m.evaluate(input_fn=eval_input_fn, steps=1)
for key in sorted(results):
    print("%s: %s" % (key, results[key]))
```
Thank you PS: Full stack trace for the error
```
Traceback (most recent call last):

File "/home/jaspreet/PycharmProjects/TicTacTensorFlow/census.py", line 73, in <module>
    df_train[LABEL_COLUMN] = (df_train['income_bracket'].apply(lambda x: '>50K' in x)).astype(int)

File "/usr/lib/python2.7/dist-packages/pandas/core/series.py", line 2023, in apply
    mapped = lib.map_infer(values, f, convert=convert_dtype)

File "inference.pyx", line 920, in pandas.lib.map_infer (pandas/lib.c:44780)

File "/home/jaspreet/PycharmProjects/TicTacTensorFlow/census.py", line 73, in <lambda>
    df_train[LABEL_COLUMN] = (df_train['income_bracket'].apply(lambda x: '>50K' in x)).astype(int)

TypeError: argument of type 'float' is not iterable
```

Response:
As you can see, when you inspect the test.data, you will obviously see that the first line of data has "NAN" in income_bracket field. I have further inspected that this is the only line contains "NAN" by doing:
```
ib = df_test ["income_bracket"]
t = type('12')
for idx,i in enumerate(ib):
    if(type(i) != t):
        print idx,type(i)
```
RESULT: 0  So you may just skip this row by: df_test = pd.read_csv(file_test , names=COLUMNS, skipinitialspace=True, skiprows=1)


Question:
Properly set up exponential decay of learning rate in tensorflow I need to apply an exponential decay of learning rate every 10 epochs. Initial learning rate is 0.000001, and decay factor is 0.95 is this the proper way to set it up?
```
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
        initial_learning_rate=0.000001,
        decay_steps=(my_steps_per_epoch*10),
        decay_rate=0.05)
opt = tf.keras.optimizers.SGD(learning_rate=lr_schedule, momentum=0.9)
```
The formula of exponential decay is current_lr = initial_lr * (1 - decay_factor)^t Except that in the code it is implemented as :
```
decayed_learning_rate = learning_rate *
                      decay_rate ^ (global_step / decay_steps)
```
To my knowledge, decay_rate should be 1 - decay_factor and decay_steps should mean how many steps are performed before applying the decay, in my case my_steps_per_epoch*10. Is that correct? EDIT: If I pause and save my model (using callbacks) after the 10th epoch, and then resume by loading the model and calling model.fit with initial_epoch=10 and epochs=11, will it start in the 11th epoch and apply the exponential decay?

Response:
decay_steps can be used to state after how many steps (processed batches) you will decay the learning rate. I find it quite useful to just specify the initial and the final learning rate and calculate the decay_factor automatically via the following:
```
initial_learning_rate = 0.1
final_learning_rate = 0.0001
learning_rate_decay_factor = (final_learning_rate / initial_learning_rate)**(1/epochs)
steps_per_epoch = int(train_size/batch_size)

lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
                initial_learning_rate=initial_learning_rate,
                decay_steps=steps_per_epoch,
                decay_rate=learning_rate_decay_factor,
                staircase=True)
```


Question:
Why is my tf_gradients returning None?
```
# Defining the tf ops
prob_placeholder = tf.placeholder(tf.float32, shape=(2))
log_placeholder = tf.log(prob_placeholder)
grads_placeholder = tf.gradients(ys=tf.log(prob_placeholder), xs=model.weights)

# t is some index into the holders (which are lists)
# s is some state || p_a is some list of [p_1, 1 - p_1] || a_ is either 0 or 1 ||  r is 1

prob_ = tf_sess.run(prob_placeholder, {prob_placeholder: p_a})
log_ = tf_sess.run(log_placeholder, {prob_placeholder: prob_})
print(prob_, log_)
grads_ = tf_sess.run(grads_placeholder, {prob_placeholder: prob_})
```
Basically I'm not sure why it's returning None.
```
TypeError: Fetch argument None has invalid type <type 'NoneType'>
```
I've tried adding print statements and I can see prob_ and log_ come out just fine but I'm not sure what's happening in tf.gradients that is causing the issue above. model.weights are basically the weights of the model that I'm using.

Response: prob_placeholder does not have any explicit dependence on model.weights, i.e. it is not functionally dependent on model.weights the way you have defined them. Hence, though technically the gradient should be zero, it is computed as None due to technical reasons in TensorFlow.


Question:
Constraining a neural network's output to be within an arbitrary range I have a custom neural network that I am training on data and seek to constrain the network's output values to always be between two arbitrary constants: [lower_bound,upper_bound]. Are there any best practices for encoding this constraint in a loss function? Below I have written a minimal working example whereby I construct and train a neural network on generated data. Additionally, I place the arbitrary constraint that the output should be between [lower_bound,upper_bound] = [-0.5,0.75] in the loss function being optimized. But I attempt it using a relatively crude method of finding all instances where predicted values exceed the bounds, and then simply making the loss function for these terms a large value (and otherwise zero if the predicted value is within the given bounds):
```
lower_bound = -0.5 #a guessed a priori lower bound on the output
upper_bound = 0.75 #a guessed a priori upper bound on the output
cond_v1_1 = tf.greater(self.v1_pred[:,0], upper_bound*tf.ones(tf.shape(self.v1_pred[:,0])))
cond_v1_2 = tf.greater(-1.0*self.v1_pred[:,0], lower_bound*tf.ones(tf.shape(self.v1_pred[:,0])))
self.red_v1 = tf.where(cond_v1_1, 100000.0*tf.ones(tf.shape(self.v1_pred[:,0])), 0.0*tf.zeros(tf.shape(self.v1_pred[:,0])))
self.red_v1 = tf.where(cond_v1_2, 100000.0*tf.ones(tf.shape(self.v1_pred[:,0])), self.red_v1)
self.loss_cond = tf.reduce_sum(1.0*tf.square(self.red_v1))
```
But are there any methods or loss functions to better encode this constraint when training the neural network? Perhaps a smoother loss function easier for the optimizers to handle and/or revisions to my code itself? Any comments and further thoughts on best practices for penalizing/training the neural network in the code below given a bound on outputs would be greatly appreciated.
```
import numpy as np
import tensorflow as tf

end_it = 1000 #number of iterations
frac_train = 1.0 #randomly sampled fraction of data to create training set
frac_sample_train = 0.01 #randomly sampled fraction of data from training set to train in batches
layers = [2, 20, 20, 20, 1]

#Generate training data
len_data = 10000
x_x = np.array([np.linspace(0.,1.,len_data)])
x_y = np.array([np.linspace(0.,1.,len_data)])
y_true = np.array([np.linspace(-0.2,0.2,len_data)])

N_train = int(frac_train*len_data)
idx = np.random.choice(len_data, N_train, replace=False)

x_train = x_x.T[idx,:]
y_train = x_y.T[idx,:]
v1_train = y_true.T[idx,:]

sample_batch_size = int(frac_sample_train*N_train)

np.random.seed(1234)
tf.set_random_seed(1234)
import logging
logging.getLogger('tensorflow').setLevel(logging.ERROR)
tf.logging.set_verbosity(tf.logging.ERROR)

class NeuralNet:
    def __init__(self, x, y, v1, layers):
        X = np.concatenate([x, y], 1)
        self.lb = X.min(0)
        self.ub = X.max(0)
        self.X = X
        self.x = X[:,0:1]
        self.y = X[:,1:2]
        self.v1 = v1
        self.layers = layers
        self.weights_v1, self.biases_v1 = self.initialize_NN(layers)
        self.sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=False,
                                                     log_device_placement=False))
        self.x_tf = tf.placeholder(tf.float32, shape=[None, self.x.shape[1]])
        self.y_tf = tf.placeholder(tf.float32, shape=[None, self.y.shape[1]])
        self.v1_tf = tf.placeholder(tf.float32, shape=[None, self.v1.shape[1]])
        self.v1_pred = self.net(self.x_tf, self.y_tf)
        lower_bound = -0.5 #a guessed a priori lower bound on the output
        upper_bound = 0.75 #a guessed a priori upper bound on the output
        cond_v1_1 = tf.greater(self.v1_pred[:,0], upper_bound*tf.ones(tf.shape(self.v1_pred[:,0])))
        cond_v1_2 = tf.greater(-1.0*self.v1_pred[:,0], lower_bound*tf.ones(tf.shape(self.v1_pred[:,0])))
        self.red_v1 = tf.where(cond_v1_1, 100000.0*tf.ones(tf.shape(self.v1_pred[:,0])), 0.0*tf.zeros(tf.shape(self.v1_pred[:,0])))
        self.red_v1 = tf.where(cond_v1_2, 100000.0*tf.ones(tf.shape(self.v1_pred[:,0])), self.red_v1)
        self.loss_cond = tf.reduce_sum(1.0*tf.square(self.red_v1))
        self.loss_data = tf.reduce_mean(tf.square(self.v1_tf - self.v1_pred))
        self.loss = self.loss_cond + self.loss_data
        self.optimizer = tf.contrib.opt.ScipyOptimizerInterface(self.loss,
                                                                var_list=self.weights_v1+self.biases_v1,
                                                                method = 'L-BFGS-B',
                                                                options = {'maxiter': 50,
                                                                           'maxfun': 50000,
                                                                           'maxcor': 50,
                                                                           'maxls': 50,
                                                                           'ftol' : 1.0 * np.finfo(float).eps})
        self.optimizer_Adam = tf.train.AdamOptimizer()
        self.train_op_Adam_v1 = self.optimizer_Adam.minimize(self.loss, var_list=self.weights_v1+self.biases_v1)
        init = tf.global_variables_initializer()
        self.sess.run(init)
    def initialize_NN(self, layers):
        weights = []
        biases = []
        num_layers = len(layers)
        for l in range(0,num_layers-1):
            W = self.xavier_init(size=[layers[l], layers[l+1]])
            b = tf.Variable(tf.zeros([1,layers[l+1]], dtype=tf.float32), dtype=tf.float32)
            weights.append(W)
            biases.append(b)
        return weights, biases
    def xavier_init(self, size):
        in_dim = size[0]
        out_dim = size[1]
        xavier_stddev = np.sqrt(2/(in_dim + out_dim))
        return tf.Variable(tf.truncated_normal([in_dim, out_dim], stddev=xavier_stddev), dtype=tf.float32)
    def neural_net(self, X, weights, biases):
        num_layers = len(weights) + 1
        H = 2.0*(X - self.lb)/(self.ub - self.lb) - 1.0
        for l in range(0,num_layers-2):
            W = weights[l]
            b = biases[l]
            H = tf.tanh(tf.add(tf.matmul(H, W), b))
        W = weights[-1]
        b = biases[-1]
        Y = tf.add(tf.matmul(H, W), b)
        return Y
    def net(self, x, y):
        v1_out = self.neural_net(tf.concat([x,y], 1), self.weights_v1, self.biases_v1)
        v1 = v1_out[:,0:1]
        return v1
    def callback(self, loss):
        global Nfeval
        print(str(Nfeval)+' - Loss in loop: %.3e' % (loss))
        Nfeval += 1
    def fetch_minibatch(self, x_in, y_in, v1_in, N_train_sample):
        idx_batch = np.random.choice(len(x_in), N_train_sample, replace=False)
        x_batch = x_in[idx_batch,:]
        y_batch = y_in[idx_batch,:]
        v1_batch = v1_in[idx_batch,:]
        return x_batch, y_batch, v1_batch
    def train(self, end_it):
        it = 0
        while it < end_it:
            x_res_batch, y_res_batch, v1_res_batch = self.fetch_minibatch(self.x, self.y, self.v1, sample_batch_size) # Fetch residual mini-batch
            tf_dict = {self.x_tf: x_res_batch, self.y_tf: y_res_batch,
                       self.v1_tf: v1_res_batch}
            self.sess.run(self.train_op_Adam_v1, tf_dict)
            self.optimizer.minimize(self.sess,
                                    feed_dict = tf_dict,
                                    fetches = [self.loss],
                                    loss_callback = self.callback)
            it = it + 1
    def predict(self, x_star, y_star):
        tf_dict = {self.x_tf: x_star, self.y_tf: y_star}
        v1_star = self.sess.run(self.v1_pred, tf_dict)
        return v1_star

model = NeuralNet(x_train, y_train, v1_train, layers)

Nfeval = 1
model.train(end_it)
```

Response:
The best way (IMHO) to do such a thing is to enforce it via the output activation function. We can use a tf.nn.sigmoid as a basis, which is bounded between [0, 1], and slightly shift and scale it.
```
def bounded_output(x, lower, upper):
    scale = upper - lower
    return scale * tf.nn.sigmoid(x) + lower
```
In your case, call it with lower=-0.5 and upper=0.75. This will shift the sigmoid so that the lowest output is -0.5, and the range is 0.75 + 0.5 = 1.25, which puts the upper limit at 0.75. Adding this as an output activation at the final layer of your net means outputs cannot fall outside the range. One problem: This can lead to bad gradients because the function saturates as it approaches the limits. So if your network is producing outputs close to those limits, gradients will be small and learning can be slow.


Question:
Object is enumerable but not indexable? Problem summary and question I'm trying to look at some of the data inside an object that can be enumerated over but not indexed. I'm still newish to python, but I don't understand how this is possible. If you can enumerate it, why can't you access the index through the same way enumerate does? And if not, is there a way to access the items individually? The actual example
```
import tensorflow_datasets as tfds

train_validation_split = tfds.Split.TRAIN.subsplit([6, 4])

(train_data, validation_data), test_data = tfds.load(
    name="imdb_reviews",
    split=(train_validation_split, tfds.Split.TEST),
    as_supervised=True)
```
Take a select subset of the dataset
```
foo = train_data.take(5)
```
I can iterate over foo with enumerate:
```
[In] for i, x in enumerate(foo):
    print(i)
```
which generates the expected output:
```
0
1
2
3
4
```
But then, when I try to index into it foo[0] I get this error:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-44-2acbea6d9862> in <module>
----> 1 foo[0]

TypeError: 'TakeDataset' object does not support indexing
```

Response: Python only allows these things if the class has methods for them: __getitem__ is required for the [] syntax. __iter__ and __next__1 are required to iterate. Any class can define one without defining the other. __getattr__ is usually not defined if it would be inefficient. 1 __next__ is required on the class returned by __iter__.


Question:
Adding a variable into Keras/TensorFlow CNN dense layer I was wondering if it is possible to add a variable into a convolutional neural network's dense layer (so as well as the connections from the previous convolutional layers, there would be an additional feature set that could be used for discriminatory purposes)? If this is possible, can anyone point me to an example/documentation explaining how to do so? I am hoping to use Keras, but am happy to use TensorFlow if Keras is too restrictive. EDIT: In this case, the way that I would think that this should work is that I provide a list containing images and associated feature sets to the neural network (and during training the associated classifications). EDIT2: The architecture that I want looks something like:
```
___________      _________      _________      _________     ________    ______
              | Conv    |     | Max    |     | Conv    |     | Max    |    |       |   |     |
    Image --> | Layer 1 | --> | Pool 1 | --> | Layer 2 | --> | Pool 2 | -->|       |   |     |
              |_________|     |________|     |_________|     |________|    | Dense |   | Out |
                                                                           | Layer |-->|_____|
   Other      ------------------------------------------------------------>|       |
   Data                                                                    |       |
                                                                           |_______|
```

Response:
Indeed, as @Marcin said, you can use a merge layer. I advise you to use the Functionnal API for this. If you're not familiar with it, read some doc here. Here is your doodled network model using the keras API :
```
from keras.layers.core import *
from keras.models import Model

# this is your image input definition. You have to specify a shape.
image_input = Input(shape=(32,32,3))
# Some more data input with 10 features (eg.)
other_data_input = Input(shape=(10,))

# First convolution filled with random parameters for the example
conv1 = Convolution2D(nb_filter = nb_filter1, nb_row = nb_row1, nb_col=_nb_col1, padding = "same", activation = "tanh")(image_input)
# MaxPool it
conv1 = MaxPooling2D(pool_size=(pool_1,pool_2))(conv1)
# Second Convolution
conv2 = Convolution2D(nb_filter = nb_filter2, nb_row = nb_row2, nb_col=_nb_col2, padding = "same", activation = "tanh")(conv1)
# MaxPool it
conv2  = MaxPooling2D(pool_size=(pool_1,pool_2))(conv2)
# Flatten the output to enable the merge to happen with the other input
first_part_output = Flatten()(conv2)

# Merge the output of the convNet with your added features by concatenation
merged_model = keras.layers.concatenate([first_part_output, other_data_input])

# Predict on the output (say you want a binary classification)
predictions = Dense(1, activation ='sigmoid')(merged_model)

# Now create the model
model = Model(inputs=[image_input, other_data_input], outputs=predictions)
# see your model
model.summary()

# compile it
model.compile(optimizer='adamax', loss='binary_crossentropy')
```
There you go :) It is quite easy in the end, define how many inputs and outputs you want, just specify them in a list when you create the Model object. When you fit it, also feed them separately, in a list.


Question:
How to load a trained TF1 protobuf model into TF2? Update: This is a bug in tensorflow. Track progress here. I have created and trained a model using stable-baselines, which uses Tensorflow 1. Now I need to use this trained model in an environment where I only have access to Tensorflow 2 or PyTorch. I figured I would go with Tensorflow 2 as the documentation says I should be able to load models created with Tensorflow 1. I can load the pb file without a problem in Tensorflow 1:
```
global_session = tf.Session()

with global_session.as_default():
    model_loaded = tf.saved_model.load_v2('tensorflow_model')
    model_loaded = model_loaded.signatures['serving_default']

init = tf.global_variables_initializer()
global_session.run(init)
```
However in Tensorflow 2 I get the following error:
```
can_be_imported = tf.saved_model.contains_saved_model('tensorflow_model')
assert(can_be_imported)
model_loaded = tf.saved_model.load('tensorflow_model/')

ValueError: Node 'loss/gradients/model/batch_normalization_3/FusedBatchNormV3_1_grad/FusedBatchNormGradV3' has an _output_shapes attribute inconsistent with the GraphDef for output #3: Dimension 0 in both shapes must be equal, but are 0 and 64. Shapes are [0] and [64].
```
Model definition:
```
NUM_CHANNELS = 64

BN1 = BatchNormalization()
BN2 = BatchNormalization()
BN3 = BatchNormalization()
BN4 = BatchNormalization()
BN5 = BatchNormalization()
BN6 = BatchNormalization()
CONV1 = Conv2D(NUM_CHANNELS, kernel_size=3, strides=1, padding='same')
CONV2 = Conv2D(NUM_CHANNELS, kernel_size=3, strides=1, padding='same')
CONV3 = Conv2D(NUM_CHANNELS, kernel_size=3, strides=1)
CONV4 = Conv2D(NUM_CHANNELS, kernel_size=3, strides=1)
FC1 = Dense(128)
FC2 = Dense(64)
FC3 = Dense(7)

def modified_cnn(inputs, **kwargs):
    relu = tf.nn.relu
    log_softmax = tf.nn.log_softmax

    layer_1_out = relu(BN1(CONV1(inputs)))
    layer_2_out = relu(BN2(CONV2(layer_1_out)))
    layer_3_out = relu(BN3(CONV3(layer_2_out)))
    layer_4_out = relu(BN4(CONV4(layer_3_out)))

    flattened = tf.reshape(layer_4_out, [-1, NUM_CHANNELS * 3 * 2])

    layer_5_out = relu(BN5(FC1(flattened)))
    layer_6_out = relu(BN6(FC2(layer_5_out)))

    return log_softmax(FC3(layer_6_out))

class CustomCnnPolicy(CnnPolicy):
    def __init__(self, *args, **kwargs):
        super(CustomCnnPolicy, self).__init__(*args, **kwargs, cnn_extractor=modified_cnn)

model = PPO2(CustomCnnPolicy, env, verbose=1)
```
Model saving in TF1:
```
with model.graph.as_default():
    tf.saved_model.simple_save(model.sess, 'tensorflow_model', inputs={"obs": model.act_model.obs_ph},
                                   outputs={"action": model.act_model._policy_proba})
```
Fully reproducible code can be found in the following 2 google colab notebooks: Tensorflow 1 saving and loading Tensorflow 2 loading Direct link to the saved model: model

Response:
You can use compatibility layer of TensorFlow. All v1 functionality is available under tf.compat.v1 namespace. I managed to load your model in TF 2.1 (nothing special about that version, I just have it locally):
```py
import tensorflow as tf

tf.__version__
Out[2]: '2.1.0'

model = tf.compat.v1.saved_model.load_v2('~/tmp/tensorflow_model')

model.signatures
Out[3]: _SignatureMap({'serving_default': <tensorflow.python.eager.wrap_function.WrappedFunction object at 0x7ff9244a6908>})
```


Question:
Comparing Conv2D with padding between Tensorflow and PyTorch I am trying to import weights saved from a Tensorflow model to PyTorch. So far the results have been very similar. I ran into a snag when the model calls for conv2d with stride=2. To verify the mismatch, I set up a very simple comparison between TF and PyTorch. First, I compare conv2d with stride=1.
```
import tensorflow as tf
import numpy as np
import torch
import torch.nn.functional as F

np.random.seed(0)
sess = tf.Session()

# Create random weights and input
weights = torch.empty(3, 3, 3, 8)
torch.nn.init.constant_(weights, 5e-2)
x = np.random.randn(1, 3, 10, 10)

weights_tf = tf.convert_to_tensor(weights.numpy(), dtype=tf.float32)
# PyTorch adopts [outputC, inputC, kH, kW]
weights_torch = torch.Tensor(weights.permute((3, 2, 0, 1)))

# Tensorflow defaults to NHWC
x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_torch = torch.Tensor(x)

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, 1, 1, 1],
                         padding="SAME")

# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=1, stride=1)

sess.run(tf.global_variables_initializer())
tf_result = sess.run(tf_conv2d)

diff = np.mean(np.abs(tf_result.transpose((0, 3, 1, 2)) - torch_conv2d.detach().numpy()))
print('Mean of Abs Diff: {0}'.format(diff))
```
The result of this execution is:
```
Mean of Abs Diff: 2.0443112092038973e-08
```
When I change stride to 2, the results start to vary.
```
# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, 2, 2, 1],
                         padding="SAME")

# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=1, stride=2)
```
The result of this execution is:
```
Mean of Abs Diff: 0.2104552686214447
```
According to PyTorch documentation, conv2d uses zero-padding defined by the padding argument. Thus, zeros are added to the left, top, right, and bottom of the input in my example. If PyTorch simply adds padding on both sides based on the input parameter, it should be easy to replicate in Tensorflow.
```
# Manually add padding - consistent with PyTorch
paddings = tf.constant([[0, 0], [1, 1], [1, 1], [0, 0]])
x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_tf = tf.pad(x_tf, paddings, "CONSTANT")

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, 2, 2, 1],
                         padding="VALID")
```
The result of this comparison is:
```
Mean of Abs Diff: 1.6035047067930464e-08
```
What this tells me is that if I am somehow able to replicate the default padding behavior from Tensorflow into PyTorch, then my results will be similar. This question inspected the behavior of padding in Tensorflow. TF documentation explains how padding is added for "SAME" convolutions. I discovered these links while writing this question. Now that I know the padding strategy of Tensorflow, I can implement it in PyTorch.

Response:
To replicate the behavior, padding sizes are calculated as described in the Tensorflow documentation. Here, I test the padding behavior by setting stride=2 and padding the PyTorch input.
```
import tensorflow as tf
import numpy as np
import torch
import torch.nn.functional as F

np.random.seed(0)
sess = tf.Session()

# Create random weights and input
weights = torch.empty(3, 3, 3, 8)
torch.nn.init.constant_(weights, 5e-2)
x = np.random.randn(1, 3, 10, 10)

weights_tf = tf.convert_to_tensor(weights.numpy(), dtype=tf.float32)
weights_torch = torch.Tensor(weights.permute((3, 2, 0, 1)))

# Tensorflow padding behavior. Assuming that kH == kW to keep this simple.
stride = 2
if x.shape[2] % stride == 0:
    pad = max(weights.shape[0] - stride, 0)
else:
    pad = max(weights.shape[0] - (x.shape[2] % stride), 0)

if pad % 2 == 0:
    pad_val = pad // 2
    padding = (pad_val, pad_val, pad_val, pad_val)
else:
    pad_val_start = pad // 2
    pad_val_end = pad - pad_val_start
    padding = (pad_val_start, pad_val_end, pad_val_start, pad_val_end)

x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_torch = torch.Tensor(x)
x_torch = F.pad(x_torch, padding, "constant", 0)

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, stride, stride, 1],
                         padding="SAME")

# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=0, stride=stride)

sess.run(tf.global_variables_initializer())
tf_result = sess.run(tf_conv2d)

diff = np.mean(np.abs(tf_result.transpose((0, 3, 1, 2)) - torch_conv2d.detach().numpy()))
print('Mean of Abs Diff: {0}'.format(diff))
```
The output is:
```
Mean of Abs Diff: 2.2477470551507395e-08
```
I wasn't quite sure why this was happening when I started writing this question, but a bit of reading clarified this very quickly. I hope this example can help others.


Question:
Q-values exploding when training DQN I'm training a DQN to play OpenAI's Atari environment, but the Q-values of my network quickly explode far above what is realistic. Here's the relevant portion of the code:
```
for state, action, reward, next_state, done in minibatch:
        if not done:
            # To save on memory, next_state is just one frame
            # So we have to add it to the current state to get the actual input for the network
            next_4_states = np.array(state)
            next_4_states = np.roll(next_4_states, 1, axis=3)
            next_4_states[:, :, :, 0] = next_state
            target = reward + self.gamma * \
                np.amax(self.target_model.predict(next_4_states))
        else:
            target = reward
        target_f = self.target_model.predict(state)
        target_f[0][action] = target

        self.target_model.fit(state, target_f, epochs=1, verbose=0)
```
The discount factor is 0.99 (it doesn't happen with discount factor 0.9, but also doesn't converge because it can't think far enough ahead). Stepping through the code, the reason it's happening is all the Q values that aren't meant to be updated (the ones for actions we didn't take) increase slightly. It's my understanding that passing the networks own output to the network during training should keep the output the same, not increase or decrease it. Is there something wrong with my model? Is there some way I can mask the update so it only updates the relevant Q value? EDIT: My model creation code is here:
```
def create_model(self, input_shape, num_actions, learning_rate):
        model = Sequential()
        model.add(Convolution2D(32, 8, strides=(4, 4),
                                activation='relu', input_shape=(input_shape)))
        model.add(Convolution2D(64, 4, strides=(2, 2), activation='relu'))
        model.add(Convolution2D(64, 3, strides=(1, 1), activation='relu'))
        model.add(Flatten())
        model.add(Dense(512, activation='relu'))
        model.add(Dense(num_actions))

        model.compile(loss='mse', optimizer=Adam(lr=learning_rate))

        return model
```
I create two of these. One for the online network and one for the target.

Response:
Which predictions get updated? Stepping through the code, the reason it's happening is all the Q values that aren't meant to be updated (the ones for actions we didn't take) increase slightly. It's my understanding that passing the networks own output to the network during training should keep the output the same, not increase or decrease it. Below I have drawn a very simple neural network with 3 input nodes, 3 hidden nodes, and 3 output nodes. Suppose that you have only set a new target for the first action, and simply use the existing predictions as targets again for the other actions. This results in only a non-zero (for simplicity I'll just assume greater than zero) error (denoted by delta in the image) for the first action/output, and errors of 0 for the others. I have drawn the connections through which this error will be propagated from output to hidden layer in bold. Note how each of the nodes in the hidden layer still gets an error. When these nodes then propagate their errors back to the input layer, they'll do this through all of the connections between input and hidden layer, so all of those weights can be modified. So, imagine all those weights got updated, and now imagine doing a new forwards pass with the original inputs. Do you expect output nodes 2 and 3 to have exactly the same outputs as before? No, probably not; the connections from hidden nodes to the last two outputs may still have the same weights, but all three hidden nodes will have different activation levels. So no, the other outputs are not guaranteed to remain the same. Is there some way I can mask the update so it only updates the relevant Q value? Not easily no, if at all. The problem is that the connections between pairs of layers other than the connections between the final pair are not action-specific, and I don't think you want them to be either. Target Network Is there something wrong with my model? One thing I'm seeing is that you seem to be updating the same network that is used to generate targets:
```
target_f = self.target_model.predict(state)
```
and
```
self.target_model.fit(state, target_f, epochs=1, verbose=0)
```
both use self.target_model. You should use separate copies of the network for those two lines, and only after longer periods of time copy the updated network's weights into the network used to compute targets. For a bit more on this, see Addition 3 in this post. Double DQN Apart from that, it is well known that DQN can still have a tendency to overestimate Q values (though it generally shouldn't completely explode). This can be addressed by using Double DQN (note: this is an improvement that was added later on top of DQN).


Question: Keras: binary_crossentropy & categorical_crossentropy confusion After using TensorFlow for quite a while I have read some Keras tutorials and implemented some examples. I have found several tutorials for convolutional autoencoders that use keras.losses.binary_crossentropy as the loss function. I thought binary_crossentropy should not be a multi-class loss function and would most likely use binary labels, but in fact Keras (TF Python backend) calls tf.nn.sigmoid_cross_entropy_with_logits, which actually is intended for classification tasks with multiple, independent classes that are not mutually exclusive. On the other hand, my expectation for categorical_crossentropy was to be intended for multi-class classifications where target classes have a dependency on each other, but are not necessarily one-hot encoded. However, the Keras documentation states: (...) when using the categorical_crossentropy loss, your targets should be in categorical format (e.g. if you have 10 classes, the target for each sample should be a 10-dimensional vector that is all-zeros expect for a 1 at the index corresponding to the class of the sample). If I am not mistaken, this is just the special case of one-hot encoded classification tasks, but the underlying cross-entropy loss also works with probability distributions ("multi-class", dependent labels)? Additionally, Keras uses tf.nn.softmax_cross_entropy_with_logits (TF python backend) for the implementation, which itself states: NOTE: While the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of labels is a valid probability distribution. If they are not, the computation of the gradient will be incorrect. Please correct me if I am wrong, but it looks to me that the Keras documentation is - at least - not very "detailed"?! So, what is the idea behind Keras' naming of the loss functions? Is the documentation correct? If the binary cross entropy would really rely on binary labels, it should not work for autoencoders, right?! Likewise the categorical crossentropy: should only work for one-hot encoded labels if the documentation is correct?!

Response: You are right by defining areas where each of these losses are applicable: binary_crossentropy (and tf.nn.sigmoid_cross_entropy_with_logits under the hood) is for binary multi-label classification (labels are independent). categorical_crossentropy (and tf.nn.softmax_cross_entropy_with_logits under the hood) is for multi-class classification (classes are exclusive). See also the detailed analysis in this question. I'm not sure what tutorials you mean, so can't comment whether binary_crossentropy is a good or bad choice for autoencoders. As for the naming, it is absolutely correct and reasonable. Or do you think sigmoid and softmax names sound better? So the only confusion left in your question is the categorical_crossentropy documentation. Note that everything that has been stated is correct: the loss supports one-hot representation. This function indeed works with any probability distribution for labels (in addition to one-hot vectors) in case of tensorflow backend and it could be included into the doc, but this doesn't look critical to me. Moreover, need to check if soft classes are supported in other backends, theano and CNTK. Remember that keras tries to be minimalistic and targets for most popular use cases, so I can understand the logic here.


Question: Image classification with Keras on Tensorflow: how to find which images are misclassified during training? I use Keras 2.0 (TensorFlow backend) on Ubuntu 17.04 to do binary image classification. Everything works great except I'd like to see which images are misclassified. How do I do that? Also, unsure if it'd answer my problem, but in TensorBoard I can't get the image tab to work, so don't know if that'd help. I've done a lot of googling, of course, but I just can't find an answer.

Response:
Simply predict the classifications and compare with your true values...
```
predicted = model.predict(trainingImages)
```
Subtracting and removing the sign should result in near zero results for the right ones and high results for the wrong ones:
```
result = numpy.absolute(trainingClasses-predicted)
```


Question:
How to read data into Tensorflow? I'm trying to read data from CSV files to tensorflow, https://www.tensorflow.org/versions/r0.7/how_tos/reading_data/index.html#filenames-shuffling-and-epoch-limits The sample code in official document is like this:
```
col1, col2, col3, col4, col5 = tf.decode_csv(value, record_defaults=record_defaults)
```
To read the file, I need to know how many columns and lines in the file beforehand, and if there are 1000 columns, I need to define 1000 variables like col1, col2, col3, col4, col5,..., col1000 , this doesn't look like an efficient way to read data. My questions What is the best way to read CSV files into Tensorflow ? Is there any way to read Database (such as mongoDB) in Tensorflow ?

Response:
You definitely don't need to define col1, col2, to col1000... generally, you might do things like this:
```
columns = tf.decode_csv(value, record_defaults=record_defaults)
features = tf.pack(columns)
do_whatever_you_want_to_play_with_features(features)
```
I do not know any off-the-shelf way to directly read data from MongoDB. Maybe you can just write a short script to convert data from MongoDB in a format that Tensorflow supports, I would recommend binary form TFRecord, which is much faster to read than csv record. This is a good blog post about this topic. Or you can choose to implement a customized data reader by yourself, see the official doc here.


Question:
How to mark one function not a test for pytest? I'm using pytest to test some code based on TensorFlow. A TestCase is defined for simplicity like:
```
class TestCase(tf.test.TestCase):
    # ...
```
The problem is tf.test.TestCase provide a useful function self.test_session(), which was treated as a test method in pytest since its name starts with test_. The result pytest report more succeed tests than test methods I defined due to test_session() methods. I use the following code to skip test_session:
```
class TestCase(tf.test.TestCase):
    @pytest.mark.skip
    @contextmanager
    def test_session(self):
        with super().test_session() as sess:
            yield sess
```
However there would be some "s" in test report indicating there are some skip tests. Is there anyway I can mark one exact method not a test method without changing pytest test discovery rules globally?

Response:
You can set __test__ = False, either directly, or by writing a simple decorator. The latter should behave similarly to Nose's nottest decorator.
```py
def nottest(obj):
    obj.__test__ = False
    return obj

class TestMyTest:

    def test_should_not_collect_1(self):
        assert False
    test_should_not_collect_1.__test__ = False

    @nottest
    def test_should_not_collect_2(self):
        assert False

    def test_should_collect(self):
        assert True

def test_should_not_collect_1():
    assert False
test_should_not_collect_1.__test__ = False

@nottest
def test_should_not_collect_2():
    assert False

def test_should_collect():
    assert True
```
When running pytest, this only runs the methods which are not marked:
```
$ pytest test.py -v
====================================== test session starts ======================================
platform darwin -- Python 3.9.1, pytest-7.0.1, pluggy-1.0.0 -- /Users/lucaswiman/.pyenv/versions/3.9.1/bin/python3.9
cachedir: .pytest_cache
rootdir: /private/tmp
plugins: anyio-2.2.0
collected 2 items

test.py::TestMyTest::test_should_collect PASSED                                           [ 50%]
test.py::test_should_collect PASSED                                                       [100%]

======================================= 2 passed in 0.04s =======================================
```
This behavior is documented here: Since Pytest 2.6, users can prevent pytest from discovering classes that start with Test by setting a boolean __test__ attribute to False.


Question:
Tensorflow hashtable lookup with arrays I'm trying to get HashMap type of functionality to work with tensorflow. I got it to work when keys and values are of int type. But when they are arrays it gives error - ValueError: Shapes (2,) and () are not compatible on line default_value)
```
import numpy as np
import tensorflow as tf

input_tensor = tf.constant([1, 1], dtype=tf.int64)
keys = tf.constant(np.array([[1, 1],[2, 2],[3, 3]]),  dtype=tf.int64)
values = tf.constant(np.array([[4, 1],[5, 1],[6, 1]]),  dtype=tf.int64)
default_value = tf.constant(np.array([1, 1]),  dtype=tf.int64)

table = tf.contrib.lookup.HashTable(
        tf.contrib.lookup.KeyValueTensorInitializer(keys, values),
        default_value)

out = table.lookup(input_tensor)
with tf.Session() as sess:
    table.init.run()
    print(out.eval())
```

Response:
Unfortunately, tf.contrib.lookup.HashTable only works with one dimensional tensors. Here's an implementation with tf.SparseTensors, which of course only works if your keys are integer (int32 or int64) tensors. For the values I'm storing the two columns in two separate tensors, but if you have many columns, you might want to just store them in a large tensor, and store the indices as values in one tf.SparseTensor. This code (tested):
```py
import tensorflow as tf

lookup = tf.placeholder( shape = ( 2, ), dtype = tf.int64 )
default_value = tf.constant( [ 1, 1 ], dtype = tf.int64 )
input_tensor = tf.constant( [ 1, 1 ], dtype=tf.int64)
keys = tf.constant( [ [ 1, 2 ], [ 3, 4 ], [ 5, 6 ] ],  dtype=tf.int64 )
values = tf.constant( [ [ 4, 1 ], [ 5, 1 ], [ 6, 1 ] ],  dtype=tf.int64 )
val0 = values[ :, 0 ]
val1 = values[ :, 1 ]

st0 = tf.SparseTensor( keys, val0, dense_shape = ( 7, 7 ) )
st1 = tf.SparseTensor( keys, val1, dense_shape = ( 7, 7 ) )

x0 = tf.sparse_slice( st0, lookup, [ 1, 1 ] )
y0 = tf.reshape( tf.sparse_tensor_to_dense( x0, default_value = default_value[ 0 ] ), () )
x1 = tf.sparse_slice( st1, lookup, [ 1, 1 ] )
y1 = tf.reshape( tf.sparse_tensor_to_dense( x1, default_value = default_value[ 1 ] ), () )

y = tf.stack( [ y0, y1 ], axis = 0 )

with tf.Session() as sess:
    print( sess.run( y, feed_dict = { lookup : [ 1, 2 ] } ) )
    print( sess.run( y, feed_dict = { lookup : [ 1, 1 ] } ) )
```
will output: [4 1] [1 1] as desired (looks up the value [ 4, 1 ] for the key [ 1, 2 ] and the default value [ 1, 1 ] for [ 1, 1 ], which points to a non-existent entry.)


Question:
Can tensorflow sess.run() really release GIL (global interpreter look) of python? I want to run multiple train_op in parallel in a tensorflow session. The answer here says that tensorflow sess.run() can release the GIL of python. I try the example in that anwser, but it seems that we still have a GIL. I have 8 GPUs available. When num_threads is 4, it takes 24 seconds. When num_threads is 8, it takes 54 seconds. Here is the code:
```
from threading import Thread
import tensorflow as tf
import time

num_threads = 8

a = []
for i in range(num_threads):
    with tf.device('/cpu:0'):
        a.append(tf.get_variable(name='a_%d'%i, shape=[5000, 50, 5, 5, 5, 5], initializer=tf.truncated_normal_initializer()))

b = []
for i in range(num_threads):
    with tf.device('/cpu:0'):
        b.append(tf.get_variable(name='b_%d'%i, shape=[5000, 50, 5, 5, 5, 5], initializer=tf.truncated_normal_initializer()))

train_ops = []
for i in range(num_threads):
    with tf.device('gpu:%d'%i):
        loss = tf.multiply(a[i], b[i], name='loss_%d'%i)
        train_ops.append(tf.train.GradientDescentOptimizer(0.01).minimize(loss))

sess = tf.Session()
sess.run(tf.initialize_all_variables())

def train_function(train_op):
    for i in range(20):
        sess.run(train_op)

train_threads = []
for train_op in train_ops:
    train_threads.append(Thread(target=train_function, args=(train_op,)))

start = time.time()
for t in train_threads:
    t.start()
for t in train_threads:
    t.join()
end = time.time()

print('elapsed time is:', end-start)
```
my question is whether it is because I did not implement the method correctly. If this way can not release the GIL, then how to release the GIL? I know distributed tensorflow via gRPC can release the GIL, but gRPC is expensive comparing to multithreading(like pthread in C). I want each thread communicating with each other, and I want to reduce the communication overhead as much as possible. Any answer or hint would be really appreciated! If there is no way to release GIL, is it possible to write a c++ extension to do multithreading. If not, is it possible to use other language which does not have GIL other than python. Thanks!

Response:
Tensorflow does release the GIL only when sess.run is called (see this comment). You are calling sess.run from within code that is restricted by the GIL; Therefore sess.run is called on each training op sequentially. I believe that the GIL releasing is intended for interactions with tf.py_func. What you are trying to accomplish is already implemented by tensorflow with hardly any extra code. Tensorflow already launches kernels on different devices concurrently. Your code also has a huge inefficiency, that you store the weights on the CPU. This is a huge bottleneck. Every iteration the weights are being copied to each GPU and the gradients copied back to the CPU where they are updated (i.e. update happens on the CPU!). When you increase the number of GPUs involved you multiply the number of copies and the CPU update time grows linearly. I fixed your code to follow the best practices:
```
import tensorflow as tf
import time

num_threads = 1

n = 5000

a = []
for i in range(num_threads):
    #store each variable one the device that it will be used on
    with tf.device('gpu:%d'%i):
        a.append(tf.get_variable(name='a_%d'%i, shape=[n, 50, 5, 5, 5, 5], initializer=tf.truncated_normal_initializer()))

b = []
for i in range(num_threads):
    with tf.device('gpu:%d'%i):
        b.append(tf.get_variable(name='b_%d'%i, shape=[n, 50, 5, 5, 5, 5], initializer=tf.truncated_normal_initializer()))

train_ops = []
for i in range(num_threads):
    #now when a and b are accessed when the graph is executed
    #the variables will already be in VRAM
    with tf.device('gpu:%d'%i):
        loss = tf.multiply(a[i], b[i], name='loss_%d'%i)
        train_ops.append(tf.train.GradientDescentOptimizer(0.01).minimize(loss))

sess = tf.Session()

sess.run(tf.initialize_all_variables())

#dry run
sess.run(train_ops)

start = time.time()
for i in range(200):
    sess.run(train_ops)
end = time.time()

print('elapsed time is:', end-start)
```
The runtimes I now get are 3.67962 and 3.64852 for 1 and 2 GPU runs with 200 iterations instead of 20. I only have access to 2 GPUs so I couldn't test on 4, but the result should be the same. You can read more about how to use tensorflow with multiple GPUs on their website. Notice that I also included a dry run. This is required in tensorflow becuase the first call to sess.run allocates memory on each GPU. This means that the more GPUs you have the more time the first call is, so it should be ignored.


Question: How to implement a matrix multiplication in Keras? I just want to implement a function that given a matrix X returns the covariance matrix of X (X^T*X), which is just a simple matrix multiplication. In Tensorflow it's gonna be easy: tf.matmul(X, tf.transpose(X)) But I didn't expect that it's a nightmare with Keras. The APIs in Keras like multiply and dot don't fit my request. I also tried different ways (Lambda layer and mixed with TF operations) but still failed, occurred lots of errors. Hope someone may help. Thanks.

Response:
Actually you do have the analogous in Keras. Try dot(x, transpose(x)). A working example comparing the two platforms follows.
```py
import keras.backend as K
import numpy as np
import tensorflow as tf

def cov_tf(x_val):
    x = tf.constant(x_val)
    cov = tf.matmul(x, tf.transpose(x))
    return cov.eval(session=tf.Session())

def cov_keras(x_val):
    x = K.constant(x_val)
    cov = K.dot(x, K.transpose(x))
    return cov.eval(session=tf.Session())

if __name__ == '__main__':
    x = np.random.rand(4, 5)
    delta = np.abs(cov_tf(x) - cov_keras(x)).max()
    print('Maximum absolute difference:', delta)
```
The maximum absolute difference is printed and gives me something around 1e-7.


Question:
Tensorflow: simultaneous prediction on GPU and CPU I’m working with tensorflow and I want to speed up the prediction phase of a pre-trained Keras model (I'm not interested in the training phase) by using simultaneously the CPU and one GPU. I tried to create 2 different threads that feed two different tensorflow sessions (one that runs on CPU and the other that runs on GPU). Each thread feeds a fixed number of batches (e.g. if we have an overall of 100 batches, I want to assign 20 batches for CPU and 80 on GPU, or any possible combination of the two) in a loop and combine the result. It would be better if the split was done automatically. However even in this scenario, it seems that the batches are being fed in a synchronous way, because even sending few batches to the CPU and computing all the others in the GPU (with the GPU as bottleneck) I observed that the overall prediction time is always higher with respect to the test made only using the GPU. I would expect it to be faster because when only the GPU is working the CPU usage is about 20-30%, thus there is some CPU available to speed up the computation. I read a lot of discussions but they all deal with parallelism with multiple GPUs and not between GPU and CPU. Here is a sample of the code I have written: the tensor_cpu and tensor_gpu objects are loaded from the same Keras model in this way:
```
with tf.device('/gpu:0'):
    model_gpu = load_model('model1.h5')
    tensor_gpu = model_gpu(x)

with tf.device('/cpu:0'):
    model_cpu = load_model('model1.h5')
    tensor_cpu = model_cpu(x)
```
Then the prediction is done as following:
```
def predict_on_device(session, predict_tensor, batches):
    for batch in batches:
        session.run(predict_tensor, feed_dict={x: batch})

def split_cpu_gpu(batches, num_batches_cpu, tensor_cpu, tensor_gpu):
    session1 = tf.Session(config=tf.ConfigProto(log_device_placement=True))
    session1.run(tf.global_variables_initializer())
    session2 = tf.Session(config=tf.ConfigProto(log_device_placement=True))
    session2.run(tf.global_variables_initializer())

    coord = tf.train.Coordinator()

    t_cpu = Thread(target=predict_on_device, args=(session1, tensor_cpu, batches[:num_batches_cpu]))
    t_gpu = Thread(target=predict_on_device, args=(session2, tensor_gpu, batches[num_batches_cpu:]))

    t_cpu.start()
    t_gpu.start()

    coord.join([t_cpu, t_gpu])

    session1.close()
    session2.close()
```
How can I achieve this CPU/GPU parallelization? I think I'm missing something. Any kind of help would be very appreciated!

Response:
Here's my code that demonstrates how CPU and GPU execution can be done in parallel:
```
import tensorflow as tf
import numpy as np
from time import time
from threading import Thread

n = 1024 * 8

data_cpu = np.random.uniform(size=[n//16, n]).astype(np.float32)
data_gpu = np.random.uniform(size=[n    , n]).astype(np.float32)

with tf.device('/cpu:0'):
    x = tf.placeholder(name='x', dtype=tf.float32)

def get_var(name):
    return tf.get_variable(name, shape=[n, n])

def op(name):
    w = get_var(name)
    y = x
    for _ in range(8):
        y = tf.matmul(y, w)
    return y

with tf.device('/cpu:0'):
    cpu = op('w_cpu')

with tf.device('/gpu:0'):
    gpu = op('w_gpu')

def f(session, y, data):
    return session.run(y, feed_dict={x : data})

with tf.Session(config=tf.ConfigProto(log_device_placement=True, intra_op_parallelism_threads=8)) as sess:
    sess.run(tf.global_variables_initializer())

    coord = tf.train.Coordinator()

    threads = []

    # comment out 0 or 1 of the following 2 lines:
    threads += [Thread(target=f, args=(sess, cpu, data_cpu))]
    threads += [Thread(target=f, args=(sess, gpu, data_gpu))]

    t0 = time()

    for t in threads:
        t.start()

    coord.join(threads)

    t1 = time()

print t1 - t0
```
The timing results are: CPU thread: 4-5s (will vary by machine, of course). GPU thread: 5s (It does 16x as much work). Both at the same time: 5s Note that there was no need to have 2 sessions (but that worked for me too). The reasons you might be seeing different results could be some contention for system resources (GPU execution does consume some host system resources, and if running the CPU thread crowds it, that could worsen the performance) incorrect timing part of your model can only run on GPU/CPU bottleneck elsewhere some other problem


Question:
What does the property losses of the Bayesian layers of TensorFlow Probability represent? I am running the example code on Bayesian Neural Network implemented using Tensorflow Probability. My question is about the implementation of the ELBO loss used for variational inference. The ELBO equals the summation of two terms, namely neg_log_likelihood and kl implemented in the code. I have difficulty understanding the implementation of the kl term. Here is how the model is defined:
```
with tf.name_scope("bayesian_neural_net", values=[images]):
  neural_net = tf.keras.Sequential()
  for units in FLAGS.layer_sizes:
    layer = tfp.layers.DenseFlipout(units, activation=FLAGS.activation)
    neural_net.add(layer)
  neural_net.add(tfp.layers.DenseFlipout(10))
  logits = neural_net(images)
  labels_distribution = tfd.Categorical(logits=logits)
```
Here is how the 'kl' term defined:
```
kl = sum(neural_net.losses) / mnist_data.train.num_examples
```
I am not sure what neural_net.losses is returning here, since there is no loss function defined for neural_net. Clearly, there will be some values returned by neural_net.losses, but I don't know what is the meaning of returned value. Any comments on this? My guess is the L2 norm, but I am not sure. If that is the case, we are still missing something. According to the VAE paper, appendix B, the authors derived the KL term when the prior is standard normal. It turns out to be pretty close to an L2 norm of the variational parameters except there are additional log variance terms and a constant term. Any comments on this?

Response:
The losses attribute of a TensorFlow Keras Layer represents side-effect computation such as regularizer penalties. Unlike regularizer penalties on specific TensorFlow variables, here, the losses represent the KL divergence computation. Check out the implementation here as well as the docstring's example: We illustrate a Bayesian neural network with variational inference, assuming a dataset of features and labels.
```
import tensorflow_probability as tfp
  model = tf.keras.Sequential([
      tfp.layers.DenseFlipout(512, activation=tf.nn.relu),
      tfp.layers.DenseFlipout(10),
  ])
  logits = model(features)
  neg_log_likelihood = tf.nn.softmax_cross_entropy_with_logits(
      labels=labels, logits=logits)
  kl = sum(model.losses)
  loss = neg_log_likelihood + kl
  train_op = tf.train.AdamOptimizer().minimize(loss)
```
It uses the Flipout gradient estimator to minimize the Kullback-Leibler divergence up to a constant, also known as the negative Evidence Lower Bound. It consists of the sum of two terms: the expected negative log-likelihood, which we approximate via Monte Carlo; and the KL divergence, which is added via regularizer terms which are arguments to the layer.


Question:
Why is AdamOptimizer duplicated in my graph? I am fairly new to the internals of TensorFlow. Towards trying to understand TensorFlow's implementation of AdamOptimizer, I checked the corresponding subgraph in TensorBoard. There seems to be a duplicate subgraph named name + '_1', where name='Adam' by default. The following MWE produces the graph below. (Note that I have expanded the x node!)
```
import tensorflow as tf

tf.reset_default_graph()
x = tf.Variable(1.0, name='x')
train_step = tf.train.AdamOptimizer(1e-1, name='MyAdam').minimize(x)

init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    with tf.summary.FileWriter('./logs/mwe') as writer:
        writer.add_graph(sess.graph)
```
I am confused because I would expect the above code to produce just a single namespace inside the graph. Even after examining the relevant source files (namely adam.py, optimizer.py and training_ops.cc), it's not clear to me how/why/where the duplicate is created. Question: What is the source of the duplicate AdamOptimizer subgraph? I can think of the following possibilities: A bug in my code Some sort of artifact generated in TensorBoard This is expected behavior (if so, then why?) A bug in TensorFlow Edit: Cleanup and clarification Due to some initial confusion, I cluttered my original question with detailed instructions for how to set up a reproducible environment with TensorFlow/TensorBoard which reproduces this graph. I have now replaced all that with the clarification about expanding the x node.

Response:
This is not a bug, just a perhaps questionable way of leaking outside of your own scope. First, not a bug: The Adam optimizer is not duplicated. As can be seen in your graph, there is a single /MyAdam scope, not two. No problem here. However, there are two MyAdam and MyAdam_1 subscopes added to your variable scope. They correspond respectively to the m and v variables (and their initialization operations) of the Adam optimizer for this variable. This is where choices made by the optimizer are debatable. You could indeed reasonably expect the Adam optimizer operations and variables to be strictly defined within its assigned scope. Instead, they choose to creep in the optimized variables' scope to locate the statistics variables. So, debatable choice to say the least, but not a bug, in the sense that the Adam optimizer is indeed not duplicated. EDIT Note that this way of locating variables is common across optimizers -- you can observe the same effect with a MomentumOptimizer for example. Indeed, this is the standard way of creating slots for optimizers -- see here:
```
# Scope the slot name in the namespace of the primary variable.
# Set "primary.op.name + '/' + name" as default name, so the scope name of
# optimizer can be shared when reuse is True. Meanwhile when reuse is False
# and the same name has been previously used, the scope name will add '_N'
# as suffix for unique identifications.
```
So as I understand it, they chose to locate the statistics of a variable within a subscope of the scope of the variable itself, so that if the variable is shared/reused, then its statistics are also shared/reused and do not need to be recomputed. This is indeed a reasonable thing to do, even if again, creeping outside of your scope is somewhat unsettling.


Question: Understanding tensorflow profiling results This example shows how to profile tensorflow programs. I have used this tool to profile my program, a simple LSTM. And the results is shown as: /gpu:0/stream:all Compute(pid 5) /job:localhost/replica:0/task:0/gpu:0 Compute(pid 3) My question : a)what is the meaning of each row. b)Especially what is the difference between /gpu:0/stream:all Compute(pid 5) and /job:localhost/replica:0/task:0/gpu:0 Compute(pid 3). c)Why their execution time are different, namely 0.072ms and 0.094ms.

Response: Here's an update from one of the engineers: The '/gpu:0/stream:*' timelsines are hardware tracing of CUDA kernel execution times. The '/gpu:0' lines are the TF software device enqueueing the ops on the CUDA stream (usually takes almost zero time)


Question:
Keras / tensorflow - limit number of cores (intra_op_parallelism_threads not working) I've been trying to run keras on a CPU cluster, and for this I need to limit the number of cores used (it's a shared system). So to limit the number of cores, I landed on this answer. However, this simply doesn't work. I tried running with this basic code:
```python
from keras.applications.vgg16 import VGG16
from keras import backend as K
import numpy as np

conf = K.tf.ConfigProto(device_count={'CPU': 1},
                        intra_op_parallelism_threads=2,
                        inter_op_parallelism_threads=2)
K.set_session(K.tf.Session(config=conf))
model = VGG16(weights='imagenet', include_top=False)
x = np.random.randn(1000, 224, 224, 3)
features = model.predict(x)
```
When I run this and check htop, it uses all (128) logical cores. Is this a bug in keras? Or am I doing something wrong? Keras says that my CPU supports SSE4.1 and SSE4.2, which are not used because I didn't compile from binary. Will compiling from binary also fix the original question? EDIT: I've found a workaround when launching the keras script from a unix machine:
```python
taskset -c 0-23 python keras_script.py
```
This will run the script on the first 24 cores of the machine. It works, but it would still be nice if this was available from within keras/tensorflow.

Response:
I found this snippet of code that works for me, hope it helps:
```
from keras import backend as K
import tensorflow as tf
jobs = 2 # it means number of cores
config = tf.ConfigProto(intra_op_parallelism_threads=jobs,
                         inter_op_parallelism_threads=jobs,
                         allow_soft_placement=True,
                         device_count={'CPU': jobs})
session = tf.Session(config=config)
K.set_session(session)
```


Question:
Keras ValueError: Input 0 is incompatible with layer conv2d_1: expected ndim=4, found ndim=5 I have checked all the solutions, but still, I am facing the same error. My training images shape is (26721, 32, 32, 1), which I believe it is 4 dimension, but I don't know why error shows it is 5 dimension.
```
model = Sequential()

 model.add(Convolution2D(16, 5, 5, border_mode='same', input_shape= input_shape ))
```
So this is how I am defining model.fit_generator
```
model.fit_generator(train_dataset, train_labels, nb_epoch=epochs, verbose=1,validation_data=(valid_dataset, valid_labels), nb_val_samples=valid_dataset.shape[0],callbacks=model_callbacks)
```

Response: The problem is input_shape. It should actually contain 3 dimensions only. And internally keras will add the batch dimension making it 4. Since you probably used input_shape with 4 dimensions (batch included), keras is adding the 5th. You should use input_shape=(32,32,1).


Question:
TensorFlow 2.0 dataset.__iter__() is only supported when eager execution is enabled I'm using the following custom training code in TensorFlow 2:
```
def parse_function(filename, filename2):
    image = read_image(fn)
    def ret1(): return image, read_image(fn2), 0
    def ret2(): return image, preprocess(image), 1
    return tf.case({tf.less(tf.random.uniform([1])[0], tf.constant(0.5)): ret2}, default=ret1)

dataset = tf.data.Dataset.from_tensor_slices((train,shuffled_train))
dataset = dataset.shuffle(len(train))
dataset = dataset.map(parse_function, num_parallel_calls=4)
dataset = dataset.batch(1)
dataset = dataset.prefetch(buffer_size=4)

@tf.function
def train(model, dataset, optimizer):
    for x1, x2, y in enumerate(dataset):
        with tf.GradientTape() as tape:
            left, right = model([x1, x2])
            loss = contrastive_loss(left, right, tf.cast(y, tf.float32))
        gradients = tape.gradient(loss, model.trainable_variables)
        optimizer.apply_gradients(zip(gradients, model.trainable_variables))

siamese_net.compile(optimizer=tf.keras.optimizers.RMSprop(learning_rate=1e-3))
train(siamese_net, dataset, tf.keras.optimizers.RMSprop(learning_rate=1e-3))
```
This code gives me the error:
```
dataset.__iter__() is only supported when eager execution is enabled.
```
However, it's in TensorFlow 2.0 so eager is enabled by default. tf.executing_eagerly() also returns 'True'.

Response:
I fixed it by enabling eager execution after importing tensorflow:
```
import tensorflow as tf

tf.enable_eager_execution()
```
Reference: Tensorflow


Question:
Why the negative reshape (-1) in MNIST tutorial? Reading the Tensorflow MNIST tutorial, I stumbled over the line
```
x_image = tf.reshape(x, [-1,28,28,1])
```
28, 28 comes from width, height, 1 comes from the number of channels. But why -1? I guess this is related to mini-batch training, but I wondered why -1 and not 1 (which seems to give the same result in numpy). (Probably related: Why does the reshape of numpy give the same results for -1,-2 and 1)?

Response: -1 means that the length in that dimension is inferred. This is done based on the constraint that the number of elements in an ndarray or Tensor when reshaped must remain the same. In the tutorial, each image is a row vector (784 elements) and there are lots of such rows (let it be n, so there are 784n elements). So, when you write x_image = tf.reshape(x, [-1, 28, 28, 1]) TensorFlow can infer that -1 is n.


Question: What's the difference between sparse_softmax_cross_entropy_with_logits and softmax_cross_entropy_with_logits? I recently came across tf.nn.sparse_softmax_cross_entropy_with_logits and I can not figure out what the difference is compared to tf.nn.softmax_cross_entropy_with_logits. Is the only difference that training vectors y have to be one-hot encoded when using sparse_softmax_cross_entropy_with_logits? Reading the API, I was unable to find any other difference compared to softmax_cross_entropy_with_logits. But why do we need the extra function then? Shouldn't softmax_cross_entropy_with_logits produce the same results as sparse_softmax_cross_entropy_with_logits, if it is supplied with one-hot encoded training data/vectors?

Response: Having two different functions is a convenience, as they produce the same result. The difference is simple: For sparse_softmax_cross_entropy_with_logits, labels must have the shape [batch_size] and the dtype int32 or int64. Each label is an int in range [0, num_classes-1]. For softmax_cross_entropy_with_logits, labels must have the shape [batch_size, num_classes] and dtype float32 or float64. Labels used in softmax_cross_entropy_with_logits are the one hot version of labels used in sparse_softmax_cross_entropy_with_logits. Another tiny difference is that with sparse_softmax_cross_entropy_with_logits, you can give -1 as a label to have loss 0 on this label.


Question: List of tensor names in graph in Tensorflow The graph object in Tensorflow has a method called "get_tensor_by_name(name)". Is there anyway to get a list of valid tensor names? If not, does anyone know the valid names for the pretrained model inception-v3 from here? From their example, pool_3, is one valid tensor but a list of all of them would be nice. I looked at the paper referred to and some of the layers seems to correspond to the sizes in table 1 but not all of them.

Response: The paper is not accurately reflecting the model. If you download the source from arxiv it has an accurate model description as model.txt, and the names in there correlate strongly with the names in the released model. To answer your first question, sess.graph.get_operations() gives you a list of operations. For an op, op.name gives you the name and op.values() gives you a list of tensors it produces (in the inception-v3 model, all tensor names are the op name with a ":0" appended to it, so pool_3:0 is the tensor produced by the final pooling op.)


Question:
Tensorflow Enqueue operation was cancelled I had built a convolutional neural network in tensorflow. It is trained and now I am unpacking it and performing evaluations.
```
import main
import Process
import Input

eval_dir = "/Users/Zanhuang/Desktop/NNP/model.ckpt-250"
checkpoint_dir = "/Users/Zanhuang/Desktop/NNP/checkpoint"

def evaluate():
  with tf.Graph().as_default() as g:
    images, labels = Process.eval_inputs()
    forward_propgation_results = Process.forward_propagation(images)
    init_op = tf.initialize_all_variables()
    saver = tf.train.Saver()
    top_k_op = tf.nn.in_top_k(forward_propgation_results, labels, 1)

  with tf.Session(graph=g) as sess:
    tf.train.start_queue_runners(sess=sess)
    sess.run(init_op)
    saver.restore(sess, eval_dir)
    print(sess.run(top_k_op))

def main(argv=None):
    evaluate()

if __name__ == '__main__':
  tf.app.run()
```
Unfortunately a strange error has popped up and I have no clue why.
```
W tensorflow/core/kernels/queue_base.cc:2
W tensorflow/core/kernels/queue_base.cc:294] _0_input_producer: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_batch/fifo_queue: Skipping cancelled enqueue attempt with queue not closed
E tensorflow/core/client/tensor_c_api.cc:485] Enqueue operation was cancelled
     [[Node: batch/fifo_queue_enqueue = QueueEnqueue[Tcomponents=[DT_FLOAT, DT_INT32], _class=["loc:@batch/fifo_queue"], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batch/fifo_queue, Cast_1, Cast)]]
E tensorflow/core/client/tensor_c_api.cc:485] Enqueue operation was cancelled
     [[Node: batch/fifo_queue_enqueue = QueueEnqueue[Tcomponents=[DT_FLOAT, DT_INT32], _class=["loc:@batch/fifo_queue"], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batch/fifo_queue, Cast_1, Cast)]]
E tensorflow/core/client/tensor_c_api.cc:485] Enqueue operation was cancelled
     ....
     [[Node: batch/fifo_queue_enqueue = QueueEnqueue[Tcomponents=[DT_FLOAT, DT_INT32], _class=["loc:@batch/fifo_queue"], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batch/fifo_queue, Cast_1, Cast)]]
E tensorflow/core/client/tensor_c_api.cc:485] Enqueue operation was cancelled
     [[Node: batch/fifo_queue_enqueue = QueueEnqueue[Tcomponents=[DT_FLOAT, DT_INT32], _class=["loc:@batch/fifo_queue"], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batch/fifo_queue, Cast_1, Cast)]]
W tensorflow/core/kernels/queue_base.cc:294] _1_batch/fifo_queue: Skipping cancelled enqueue attempt with queue not closed
...
W tensorflow/core/kernels/queue_base.cc:294] _1_batch/fifo_queue: Skipping cancelled enqueue attempt with queue not closed
E tensorflow/core/client/tensor_c_api.cc:485] Enqueue operation was cancelled
```
This is only a part of it.

Response:
Update from chat -- the program runs successfully, and the messages that are printed are due to Python killing threads while they are running as the process exits. The messages are harmless but it's possible to avoid them by stopping threads manually using pattern below.
```
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(sess=sess, coord=coord)
<do stuff>
coord.request_stop()
coord.join(threads)
```


Question:
Making a list and appending to it in TensorFlow I am new to TensorFlow. I'm not able to understand how to create a dynamic "pythonic" list in TensorFlow. Basically, I perform some computation on a tensor object (train_data[i]) and append it to a "list" X, which I want to be a tensor with shape (100,) I want to do something like this:
```
X = []
for i in range(100):
    q = tf.log(train_data[i])
    print(q)    #Tensor("Log:0", shape=(), dtype=float32)
    X.append(q)
```
I want X to be a Tensor with shape (100,), basically a column vector which is a tensor object. If I run the code above, I instead get a python list of TensorObjects.

Response:
If you want to convert X to a (100,) tensor you could add X = tf.stack(X) after your for loop:
```
X = []
for i in range(100):
  q = tf.log(train_data[i])
  print(q)    #Tensor("Log:0", shape=(), dtype=float32)
  X.append(q)
X = tf.stack(X)
```
This is a useful construct where you may want to tf.unstack some tensor, loop over the resulting list, and then use tf.stack to get back to a single tensor.


Question: Download pretrained ImageNet model of ResNet, VGG, etc. (.PB file) I have downloaded a pre-trained model on ImageNet of Inception v3 from http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz (found this link while following one of the tutorials on codelabs). This zip file contains .pb file that I can directly import as GraphDef in TensorFlow. I would like to know if there are similar .pb files for other architectures as well, such as ResNet, VGG16, MobileNet, etc. If yes, could you provide the link of those? Thanks in advance. Kind Regards, Ajay

Response: You can find many pretrained models here: https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models And the corresponding codes are here: https://github.com/tensorflow/models/tree/master/research/slim/nets


Question:
Keras: How to get layer shapes in a Sequential model I would like to access the layer size of all the layers in a Sequential Keras model. My code:
```
model = Sequential()
model.add(Conv2D(filters=32,
               kernel_size=(3,3),
               input_shape=(64,64,3)
        ))
model.add(MaxPooling2D(pool_size=(3,3), strides=(2,2)))
```
Then I would like some code like the following to work
```
for layer in model.layers:
    print(layer.get_shape())
```
.. but it doesn't. I get the error: AttributeError: 'Conv2D' object has no attribute 'get_shape'

Response:
According to official doc for Keras Layer, one can access layer output/input shape via layer.output_shape or layer.input_shape.
```py
from keras.models import Sequential
from keras.layers import Conv2D, MaxPool2D

model = Sequential(layers=[
    Conv2D(32, (3, 3), input_shape=(64, 64, 3)),
    MaxPool2D(pool_size=(3, 3), strides=(2, 2))
])

for layer in model.layers:
    print(layer.output_shape)

# Output
# (None, 62, 62, 32)
# (None, 30, 30, 32)
```


Question:
Unknown initializer: GlorotUniform when loading Keras model I trained my CNN (VGG) through google colab and generated .h5 file. Now problem is, I can predict my output successfully through google colab but when i download that .h5 trained model file and try to predict output on my laptop, I am getting error when loading the model. Here is the code:
```
import tensorflow as tf
from tensorflow import keras
import h5py

# Initialization

loaded_model = keras.models.load_model('./train_personCount_model.h5')
```
And the error:
```
ValueError: Unknown initializer: GlorotUniform
```

Response:
I fixed the problem: Before:
```py
from keras.models import load_model
classifierLoad = load_model('model/modeltest.h5')
```
Works for me
```py
import tensorflow as tf
classifierLoad = tf.keras.models.load_model('model/modeltest.h5')
```


Question: How do I get the gradient of the loss at a TensorFlow variable? The feature I'm after is to be able to tell what the gradient of a given variable is with respect to my error function given some data. One way to do this would be to see how much the variable has changed after a call to train, but obviously that can vary massively based on the learning algorithm (for example it would be almost impossible to tell with something like RProp) and just isn't very clean. Thanks in advance.

Response:
The tf.gradients() function allows you to compute the symbolic gradient of one tensor with respect to one or more other tensors—including variables. Consider the following simple example:
```
data = tf.placeholder(tf.float32)
var = tf.Variable(...)              # Must be a tf.float32 or tf.float64 variable.
loss = some_function_of(var, data)  # some_function_of() returns a `Tensor`.

var_grad = tf.gradients(loss, [var])[0]
```
You can then use this symbolic gradient to evaluate the gradient in some specific point (data):
```
sess = tf.Session()

var_grad_val = sess.run(var_grad, feed_dict={data: ...})
```


Question:
How to understand tf.get_collection() in TensorFlow I am confused by tf.get_collection() form the docs, it says that Returns a list of values in the collection with the given name. And an example from the Internet is here
```
from_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, from_scope)
```
Is it means that it collects variables from tf.GraphKeys.TRAINABLE_VARIABLES to from_scope? However, how can I use this function if I want to get variables from another scope? Thank you!

Response: A collection is nothing but a named set of values. Every value is a node of the computational graph. Every node has its name and the name is composed by the concatenation of scopes, / and values, like: preceding/scopes/in/that/way/value get_collection, without scope allow fetching every value in the collection without applying any filter operation. When the scope parameter is present, every element of the collection is filtered and its returned only if the name of the node starts with the specified scope.


Question:
Accessing filename from file queue in Tensor Flow I have a directory of images, and a separate file matching image filenames to labels. So the directory of images has files like 'train/001.jpg' and the labeling file looks like:
```
train/001.jpg 1
train/002.jpg 2
...
```
I can easily load images from the image directory in Tensor Flow by creating a filequeue from the filenames:
```
filequeue = tf.train.string_input_producer(filenames)
reader = tf.WholeFileReader()
img = reader.read(filequeue)
```
But I'm at a loss for how to couple these files with the labels from the labeling file. It seems I need access to the filenames inside the queue at each step. Is there a way to get them? Furthermore, once I have the filename, I need to be able to look up the label keyed by the filename. It seems like a standard Python dictionary wouldn't work because these computations need to happen at each step in the graph.

Response:
Given that your data is not too large for you to supply the list of filenames as a python array, I'd suggest just doing the preprocessing in Python. Create two lists (same order) of the filenames and the labels, and insert those into either a randomshufflequeue or a queue, and dequeue from that. If you want the "loops infinitely" behavior of the string_input_producer, you could re-run the 'enqueue' at the start of every epoch. A very toy example:
```
import tensorflow as tf

f = ["f1", "f2", "f3", "f4", "f5", "f6", "f7", "f8"]
l = ["l1", "l2", "l3", "l4", "l5", "l6", "l7", "l8"]

fv = tf.constant(f)
lv = tf.constant(l)

rsq = tf.RandomShuffleQueue(10, 0, [tf.string, tf.string], shapes=[[],[]])
do_enqueues = rsq.enqueue_many([fv, lv])

gotf, gotl = rsq.dequeue()

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    tf.train.start_queue_runners(sess=sess)
    sess.run(do_enqueues)
    for i in xrange(2):
        one_f, one_l = sess.run([gotf, gotl])
        print "F: ", one_f, "L: ", one_l
```
The key is that you're effectively enqueueing pairs of filenames/labels when you do the enqueue, and those pairs are returned by the dequeue.


Question:
How can I separate runs of my TensorFlow code in TensorBoard? My TensorBoard plots treat successive runs of my TensorFlow code as if they are all part of the same run. For example if I first run my code (below) with FLAGS.epochs == 10 and then rerun it with FLAGS.epochs == 40 I get which "loops back" at the end of the first run to begin the second. Is there a way to treat multiple runs of my code as distinct logs, that, for example, could be compared or individually viewed?
```
from __future__ import (absolute_import, print_function, division, unicode_literals)

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

# Basic model parameters as external flags.
flags = tf.app.flags
FLAGS = flags.FLAGS
flags.DEFINE_float('epochs', 40, 'Epochs to run')
flags.DEFINE_integer('mb_size', 40, 'Mini-batch size. Must divide evenly into the dataset sizes.')
lags.DEFINE_float('learning_rate', 0.15, 'Initial learning rate.')
flags.DEFINE_float('regularization_weight', 0.1 / 1000, 'Regularization lambda.')
flags.DEFINE_string('data_dir', './data', 'Directory to hold training and test data.')
flags.DEFINE_string('train_dir', './_tmp/train', 'Directory to log training (and the network def).')
flags.DEFINE_string('test_dir', './_tmp/test', 'Directory to log testing.')

def variable_summaries(var, name):
    with tf.name_scope("summaries"):
        mean = tf.reduce_mean(var)
        tf.scalar_summary('mean/' + name, mean)
        with tf.name_scope('stddev'):
            stddev = tf.sqrt(tf.reduce_sum(tf.square(var - mean)))
            tf.scalar_summary('sttdev/' + name, stddev)
    tf.scalar_summary('max/' + name, tf.reduce_max(var))
    tf.scalar_summary('min/' + name, tf.reduce_min(var))
    tf.histogram_summary(name, var)

def nn_layer(input_tensor, input_dim, output_dim, neuron_fn, layer_name):
    with tf.name_scope(layer_name):
        # This Variable will hold the state of the weights for the layer
        with tf.name_scope("weights"):
            weights = tf.Variable(tf.truncated_normal([input_dim, output_dim], stddev=0.1))
            variable_summaries(weights, layer_name + '/weights')
        with tf.name_scope("biases"):
            biases = tf.Variable(tf.constant(0.1, shape=[output_dim]))
            variable_summaries(biases, layer_name + '/biases')
        with tf.name_scope('activations'):
            with tf.name_scope('weighted_inputs'):
                weighted_inputs = tf.matmul(input_tensor, weights) + biases
                tf.histogram_summary(layer_name + '/weighted_inputs', weighted_inputs)
            output = neuron_fn(weighted_inputs)
            tf.histogram_summary(layer_name + '/output', output)
    return output, weights

# Collect data
mnist = input_data.read_data_sets(FLAGS.data_dir, one_hot=True)

# Inputs and outputs
x = tf.placeholder(tf.float32, shape=[None, 784])
y_ = tf.placeholder(tf.float32, shape=[None, 10])

# Network structure
o1, W1 = nn_layer(x, 784, 30, tf.nn.sigmoid, 'hidden_layer')
y, W2 = nn_layer(o1, 30, 10, tf.nn.softmax, 'output_layer')

with tf.name_scope('accuracy'):
    with tf.name_scope('loss'):
        cost = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))
        loss = cost + FLAGS.regularization_weight * (tf.nn.l2_loss(W1) + tf.nn.l2_loss(W2))
    with tf.name_scope('correct_prediction'):
        correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
    with tf.name_scope('accuracy'):
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
    tf.scalar_summary('accuracy', accuracy)
    tf.scalar_summary('loss', loss)

train_step = tf.train.GradientDescentOptimizer(FLAGS.learning_rate).minimize(loss)

# Logging
train_writer = tf.train.SummaryWriter(FLAGS.train_dir, tf.get_default_graph())
test_writer = tf.train.SummaryWriter(FLAGS.test_dir)
merged = tf.merge_all_summaries()

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())

    for ep in range(FLAGS.epochs):
        for mb in range(int(len(mnist.train.images)/FLAGS.mb_size)):
            batch_xs, batch_ys = mnist.train.next_batch(FLAGS.mb_size)
            sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})

        summary = sess.run(merged, feed_dict={x: mnist.test.images, y_: mnist.test.labels})
        test_writer.add_summary(summary, ep+1)
```

Response:
You can put your runs into separate subdirectories, like:
```
./logdir/2016111301/
./logdir/2016111302/
./logdir/2016111401/
```
than you call your tensorboard function on the root dir:
```
tensorboard --logdir=logdir
```
Than you will have the separate logfiles like:


Question:
Keras: UnboundLocalError: local variable 'logs' referenced before assignment I am relatively new to python, and while attempting to train a chatbot I received the error: ‘UnboundLocalError: local variable 'logs' referenced before assignment‘. I used model.fit to train:
```
model.fit(x_train, y_train, epochs=7)
```
And I received the error:
```
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-10-847c83704a3f> in <module>()
      2           x_train,
      3           y_train,
----> 4           epochs=7
      5           )

1 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
     64   def _method_wrapper(self, *args, **kwargs):
     65     if not self._in_multi_worker_mode():  # pylint: disable=protected-access
---> 66       return method(self, *args, **kwargs)
     67
     68     # Running inside `run_distribute_coordinator` already.

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
    854               logs = tmp_logs  # No error, now safe to assign to logs.
    855               callbacks.on_train_batch_end(step, logs)
--> 856         epoch_logs = copy.copy(logs)
    857
    858         # Run validation.

UnboundLocalError: local variable 'logs' referenced before assignment
```
I ran this in google colab, with the link here: https://colab.research.google.com/drive/18uTvvKYDrd8CQi31kg6vX2Dbxg1gD20X?usp=sharing I used the chatterbot/english dataset on kaggle: https://www.kaggle.com/kausr25/chatterbotenglish

Response:
This issue looks similar to the problem I had while working with small datasets and it is covered in this thread: #38064. I solved my particular issue setting a smaller batch_size, in my case:
```
batch_size = 2
```


Question:
How to get PI in tensorflow? I could not find any information about mathemtical constants in the Tensorflow API neither in Basic Math functions nor in Math Ops. I was able to get it by
```
import math as m
pi = tf.constant(m.pi)
```
However, this would mean to include another library, - so I wonder if there is this functionallity for mathemtical constants like pi, or euler already provided inside tensorflow?

Response: math is in the Python standard library, not a third-party module. I don't see why using it is a problem whatsoever.