Question:
How to save TextVectorization to disk in tensorflow? I have trained a TextVectorization layer (see below), and I want to save it to disk, so that I can reload it next time? I have tried pickle and joblib.dump(). It does not work.
```py
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization

text_dataset = tf.data.Dataset.from_tensor_slices(text_clean)

vectorizer = TextVectorization(max_tokens=100000, output_mode='tf-idf',ngrams=None)

vectorizer.adapt(text_dataset.batch(1024))
```
The generated error is the following:
```py
InvalidArgumentError: Cannot convert a Tensor of dtype resource to a NumPy array
```
How can I save it?

Response:
Instead of pickling the object, pickle the configuration and weights. Later unpickle it and use configuration to create the object and load the saved weights. Official docs here. Code
```py
text_dataset = tf.data.Dataset.from_tensor_slices([
                                                   "this is some clean text",
                                                   "some more text",
                                                   "even some more text"])
# Fit a TextVectorization layer
vectorizer = TextVectorization(max_tokens=10, output_mode='tf-idf',ngrams=None)
vectorizer.adapt(text_dataset.batch(1024))

# Vector for word "this"
print (vectorizer("this"))

# Pickle the config and weights
pickle.dump({'config': vectorizer.get_config(),
             'weights': vectorizer.get_weights()}
            , open("tv_layer.pkl", "wb"))

print ("*"*10)
# Later you can unpickle and use
# `config` to create object and
# `weights` to load the trained weights.

from_disk = pickle.load(open("tv_layer.pkl", "rb"))
new_v = TextVectorization.from_config(from_disk['config'])
# You have to call `adapt` with some dummy data (BUG in Keras)
new_v.adapt(tf.data.Dataset.from_tensor_slices(["xyz"]))
new_v.set_weights(from_disk['weights'])

# Lets see the Vector for word "this"
print (new_v("this"))
```
Output:
```py
tf.Tensor(
[[0.         0.         0.         0.         0.91629076 0.
  0.         0.         0.         0.        ]], shape=(1, 10), dtype=float32)
**********
tf.Tensor(
[[0.         0.         0.         0.         0.91629076 0.
  0.         0.         0.         0.        ]], shape=(1, 10), dtype=float32)
```


Question:
ValueError: Feature not in features dictionary I am attempting to write a simple deep machine learning model using TensorFlow. I'm using a toy dataset I made up in Excel just to get the model working and accepting data. My code is as follows:
```
import pandas as pd
import numpy as np
import tensorflow as tf

raw_data = np.genfromtxt('ai/mock-data.csv', delimiter=',', dtype=str)
my_data = np.delete(raw_data, (0), axis=0) #deletes the first row, axis=0 indicates row, axis=1 indicates column
my_data = np.delete(my_data, (0), axis=1) #deletes the first column

policy_state = tf.feature_column.categorical_column_with_vocabulary_list('policy_state', [
    'AL', 'CA', 'MI'
])

modern_classic_ind = tf.feature_column.categorical_column_with_vocabulary_list('modern_classic_ind', [
    '0', '1'
])

h_plus_ind = tf.feature_column.categorical_column_with_vocabulary_list('h_plus_ind', [
    '0', '1'
])

retention_ind = tf.feature_column.categorical_column_with_vocabulary_list('retention_ind', [
    '0', '1'
])

feature_columns = [
    tf.feature_column.indicator_column(policy_state),
    tf.feature_column.indicator_column(modern_classic_ind),
    tf.feature_column.indicator_column(h_plus_ind)
]
classifier = tf.estimator.DNNClassifier(feature_columns=feature_columns,
                                      hidden_units=[10, 20, 10],
                                      n_classes=3,
                                      model_dir="/tmp/ret_model")

train_input_fn = tf.estimator.inputs.numpy_input_fn(
x={"x": np.array(my_data[:, 0:3], dtype=str)},
y=np.array(np.array(my_data[:, 3], dtype=str)),
num_epochs=None,
shuffle=True)

classifier.train(input_fn=train_input_fn, steps=2000)
```
Unfortunately, I am getting the following error. I have tried trimming the labels off the csv file versus leaving them, naming the feature columns different things, and changing the type of the numpy array. The error persists. ValueError: Feature h_plus_ind is not in features dictionary. If I remove h_plus_ind, it simply throws the error on a different column.

Response:
When using tf.feature_columns, the data you feed in your input_fn should have the same keys as the feature columns previously created. So, the x of your train_input_fn should be a dictionary, with keys named after the feature_columns. A mock example :
```
x = {"policy_state": np.array(['AL','AL','AL','AL','AL']),
     "modern_classic_ind": np.array(['0','0','0','0','0']),
     "h_plus_ind": np.array(['0','0','0','0','0']),}
```


Question:
Tensorflow while_loop for training In my problem I need run GD with 1 example from data on each training step. It's known problem that session.run() has overhead and therefore it is too long to train model. In attempt to avoid overhead I tried to use while_loop and train model on all data with one run() call. But it approach don't work and train_op don't execute even ones. Below simple example of what I'm doing:
```
data = [k*1. for k in range(10)]
tf.reset_default_graph()

i = tf.Variable(0, name='loop_i')
q_x = tf.FIFOQueue(100000, tf.float32)
q_y = tf.FIFOQueue(100000, tf.float32)

x = q_x.dequeue()
y = q_y.dequeue()
w = tf.Variable(0.)
b = tf.Variable(0.)
loss = (tf.add(tf.mul(x, w), b) - y)**2

gs = tf.Variable(0)

train_op = tf.train.GradientDescentOptimizer(0.05).minimize(loss, global_step=gs)

s = tf.Session()
s.run(tf.initialize_all_variables())

def cond(i):
    return i < 10

def body(i):
    return tf.tuple([tf.add(i, 1)], control_inputs=[train_op])

loop = tf.while_loop(cond, body, [i])

for _ in range(1):
    s.run(q_x.enqueue_many((data, )))
    s.run(q_y.enqueue_many((data, )))

s.run(loop)
s.close()
```
What I'm doing wrong? Or there is another solution of this problem with too expensive overhead? Thanks!

Response:
The reason the model does not appear to train is because the input reading, gradient calculation, and the minimize() call are all defined outside (and hence, in dataflow terms, before) the body of the tf.while_loop(). This means that all of these parts of the model run only once, before the loop executes, and the loop itself has no effect. A slight refactoring—to move the dequeue() operations, gradient calculation, and minimize() call inside the loop—fixes the problem and allows your program to train:
```python
optimizer = tf.train.GradientDescentOptimizer(0.05)

def cond(i):
    return i < 10

def body(i):
    # Dequeue a new example each iteration.
    x = q_x.dequeue()
    y = q_y.dequeue()

    # Compute the loss and gradient update based on the current example.
    loss = (tf.add(tf.mul(x, w), b) - y)**2
    train_op = optimizer.minimize(loss, global_step=gs)

    # Ensure that the update is applied before continuing.
    return tf.tuple([tf.add(i, 1)], control_inputs=[train_op])

loop = tf.while_loop(cond, body, [i])
```
UPDATE: Here's a complete program the executes the while loop, based on the code in your question:
```python
import tensorflow as tf

# Define a single queue with two components to store the input data.
q_data = tf.FIFOQueue(100000, [tf.float32, tf.float32])

# We will use these placeholders to enqueue input data.
placeholder_x = tf.placeholder(tf.float32, shape=[None])
placeholder_y = tf.placeholder(tf.float32, shape=[None])
enqueue_data_op = q_data.enqueue_many([placeholder_x, placeholder_y])

gs = tf.Variable(0)
w = tf.Variable(0.)
b = tf.Variable(0.)
optimizer = tf.train.GradientDescentOptimizer(0.05)

# Construct the while loop.
def cond(i):
    return i < 10

def body(i):
    # Dequeue a single new example each iteration.
    x, y = q_data.dequeue()
    # Compute the loss and gradient update based on the current example.
    loss = (tf.add(tf.multiply(x, w), b) - y) ** 2
    train_op = optimizer.minimize(loss, global_step=gs)
    # Ensure that the update is applied before continuing.
    with tf.control_dependencies([train_op]):
        return i + 1

loop = tf.while_loop(cond, body, [tf.constant(0)])

data = [k * 1. for k in range(10)]

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for _ in range(1):
        # NOTE: Constructing the enqueue op ahead of time avoids adding
        # (potentially many) copies of `data` to the graph.
        sess.run(enqueue_data_op,
                 feed_dict={placeholder_x: data, placeholder_y: data})
    print (sess.run([gs, w, b]))  # Prints before-loop values.
    sess.run(loop)
    print (sess.run([gs, w, b]))  # Prints after-loop values.
```


Question:
How use TPU in google colab Google colab brings TPUs in the Runtime Accelerator. I found an example, How to use TPU in Official Tensorflow github. But the example not worked on google-colaboratory. It stuck on following line:
```
tf.contrib.tpu.keras_to_tpu_model(model, strategy=strategy)
```
When I print available devices on colab it return [] for TPU accelerator. Does anyone knows how to use TPU on colab?

Response:
Here's a Colab-specific TPU example: https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/shakespeare_with_tpu_and_keras.ipynb The key lines are those to connect to the TPU itself:
```
# This address identifies the TPU we'll use when configuring TensorFlow.
TPU_WORKER = 'grpc://' + os.environ['COLAB_TPU_ADDR']

...

tpu_model = tf.contrib.tpu.keras_to_tpu_model(
training_model,
strategy=tf.contrib.tpu.TPUDistributionStrategy(
    tf.contrib.cluster_resolver.TPUClusterResolver(TPU_WORKER)))
```
(Unlike a GPU, use of the TPU requires an explicit connection to the TPU worker. So, you'll need to tweak your training and inference definition in order to observe a speedup.)


Question:
How to disable GPU in keras with tensorflow? I want to compare processing time of my code with and without gpu. My backend of keras is Tensorflow. So it uses a GPU automatically. I use a model of keras/examples/mnist_mlp.py for comparing. I checked the processing time like below. Then, how do I disable my GPU? Should ~/.keras/keras.json be modified?
```
$ time python mnist_mlp.py
Test loss: 0.109761892007
Test accuracy: 0.9832
python mnist_mlp.py  38.22s user 3.18s system 162% cpu 25.543 total
```

Response:
Have you tried something like this? :
```
$ CUDA_VISIBLE_DEVICES='' time python mnist_mlp.py
```
CUDA_VISIBLE_DEVICES is usually used to hide some GPU's to cuda. Here you hide them all as you don't put any visible device.


Question:
Could validation data be a generator in tensorflow.keras 2.0? In official documents of tensorflow.keras, validation_data could be: tuple (x_val, y_val) of Numpy arrays or tensors tuple (x_val, y_val, val_sample_weights) of Numpy arrays dataset For the first two cases, batch_size must be provided. For the last case, validation_steps could be provided. It does not mention if generator could act as validation_data. So I want to know if validation_data could be a datagenerator? like the following codes:
```
net.fit_generator(train_it.generator(), epoch_iterations * batch_size, nb_epoch=nb_epoch, verbose=1,
                  validation_data=val_it.generator(), nb_val_samples=3,
                  callbacks=[checker, tb, stopper, saver])
```
Update: In the official documents of keras, the same contents, but another sentense is added: dataset or a dataset iterator Considering that dataset For the first two cases, batch_size must be provided. For the last case, validation_steps could be provided. I think there should be 3 cases. Keras' documents are correct. So I will post an issue in tensorflow.keras to update the documents.

Response:
Yes it can, that's strange that it is not in the doc but is it working exactly like the x argument, you can also use a keras.Sequence or a generator. In my project I often use keras.Sequence that acts like a generator Minimum working example that shows that it works :
```py
import numpy as np
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, Flatten

def generator(batch_size): # Create empty arrays to contain batch of features and labels
    batch_features = np.zeros((batch_size, 1000))
    batch_labels = np.zeros((batch_size,1))
    while True:
        for i in range(batch_size):
            yield batch_features, batch_labels

model = Sequential()
model.add(Dense(125, input_shape=(1000,), activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

train_generator = generator(64)
validation_generator = generator(64)

model.fit(train_generator, validation_data=validation_generator, validation_steps=100, epochs=100, steps_per_epoch=100)
```
100/100 [==============================] - 1s 13ms/step - loss: 0.6689 - accuracy: 1.0000 - val_loss: 0.6448 - val_accuracy: 1.0000 Epoch 2/100 100/100 [==============================] - 0s 4ms/step - loss: 0.6223 - accuracy: 1.0000 - val_loss: 0.6000 - val_accuracy: 1.0000 Epoch 3/100 100/100 [==============================] - 0s 4ms/step - loss: 0.5792 - accuracy: 1.0000 - val_loss: 0.5586 - val_accuracy: 1.0000 Epoch 4/100 100/100 [==============================] - 0s 4ms/step - loss: 0.5393 - accuracy: 1.0000 - val_loss: 0.5203 - val_accuracy: 1.0000


Question:
Why Pearson correlation is different between Tensorflow and Scipy I compute the Pearson correlation in 2 ways : In Tensorflow, I use the following metric :
```
tf.contrib.metrics.streaming_pearson_correlation(y_pred, y_true)
```
When I evaluate my network on test data, I got following results : loss = 0.5289223349094391 pearson = 0.3701728057861328 (Loss is mean_squared_error) Then I predict the test data and compute the same metrics with Scipy :
```
import scipy.stats as measures
per_coef = measures.pearsonr(y_pred, y_true)[0]
mse_coef = np.mean(np.square(np.array(y_pred) - np.array(y_true)))
```
And I get following results : Pearson = 0.5715300096509959 MSE = 0.5289223312665985 Is it a known issue ? Is it normal ? Minimal, complete and verifiable example
```
import tensorflow as tf
import scipy.stats as measures

y_pred = [2, 2, 3, 4, 5, 5, 4, 2]
y_true = [1, 2, 3, 4, 5, 6, 7, 8]

## Scipy
val2 = measures.pearsonr(y_pred, y_true)[0]
print("Scipy's Pearson = {}".format(val2))

## Tensorflow
logits = tf.placeholder(tf.float32, [8])
labels = tf.to_float(tf.Variable(y_true))

acc, acc_op = tf.contrib.metrics.streaming_pearson_correlation(logits,labels)

sess = tf.Session()
sess.run(tf.local_variables_initializer())
sess.run(tf.global_variables_initializer())
sess.run(acc, {logits:y_pred})
sess.run(acc_op, {logits:y_pred})

print("Tensorflow's Pearson:{}".format(sess.run(acc,{logits:y_pred})))
```

Response:
In the minimal verifiable example you gave, y_pred and y_true are lists of integers. In the first line of the scipy.stats.measures.pearsonr source, you will see that the inputs are converted to numpy arrays with x = np.asarray(x). We can see the resulting data types of these arrays via:
```
print(np.asarray(y_pred).dtype)  # Prints 'int64'
```
When dividing two int64 numbers, SciPy uses float64 precision, while TensorFlow will use float32 precision in the example above. The difference can be quite large, even for a single division:
```

```python
'%.15f' % (8.5 / 7)
#Output
#'1.214285714285714'
```

```python
'%.15f' % (np.array(8.5, dtype=np.float32) / np.array(7, dtype=np.float32))
#Output
#'1.214285731315613'
```

```python
'%.15f' % (np.array(8.5, dtype=np.float32) / np.array(7, dtype=np.float32) - 8.5 / 7)
#Output
#'0.000000017029899'
#```
#You can get the same results for SciPy and TensorFlow by using float32 precision for y_pred and y_true:
#```
#import numpy as np
#import tensorflow as tf
#import scipy.stats as measures
```

y_pred = np.array([2, 2, 3, 4, 5, 5, 4, 2], dtype=np.float32)
y_true = np.array([1, 2, 3, 4, 5, 6, 7, 8], dtype=np.float32)

## Scipy
val2 = measures.pearsonr(y_pred, y_true)[0]
print("Scipy's Pearson: \t\t{}".format(val2))

## Tensorflow
logits = tf.placeholder(tf.float32, [8])
labels = tf.to_float(tf.Variable(y_true))

acc, acc_op = tf.contrib.metrics.streaming_pearson_correlation(logits,labels)

sess = tf.Session()
sess.run(tf.local_variables_initializer())
sess.run(tf.global_variables_initializer())
sess.run(acc, {logits:y_pred})
sess.run(acc_op, {logits:y_pred})

print("Tensorflow's Pearson: \t{}".format(sess.run(acc,{logits:y_pred})))
```
prints
```none
Scipy's Pearson:        0.38060760498046875
Tensorflow's Pearson:   0.38060760498046875
```
Differences between SciPy's and TensorFlow's computation In the test scores you report, the difference is quite high. I took a look at the source and found the following differences: 1. Update ops The result of tf.contrib.metrics.streaming_pearson_correlation is not stateless. It returns the correlation coefficient op, together with an update_op for new incoming data. If you call the update op with different data before calling the coefficient op with the actual y_pred, it will give a completely different result:
```
sess.run(tf.global_variables_initializer())

for _ in range(20):
    sess.run(acc_op, {logits: np.random.randn(*y_pred.shape)})

print("Tensorflow's Pearson: \t{}".format(sess.run(acc,{logits:y_pred})))
```
prints
```none
Scipy's Pearson:        0.38060760498046875
Tensorflow's Pearson:   -0.0678008571267128
```
2. Different formulae SciPy: TensorFlow: While mathematically the same, the computation of the correlation coefficient is different in TensorFlow. It uses the sample covariance for (x, x), (x, y) and (y, y) to compute the correlation coefficient, which can introduce different rounding errors.


Question:
What am I missing from this csv reader for TensorFlow? It is mostly a copy paste from the tutorial, on the website. I am getting an error: Invalid argument: ConcatOp : Expected concatenating dimensions in the range [0, 0), but got 0 [[Node: concat = Concat[N=4, T=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](concat/concat_dim, DecodeCSV, DecodeCSV:1, DecodeCSV:2, DecodeCSV:3)]] the contents of my csv file is: 3,4,1,8,4
```
import tensorflow as tf

filename_queue = tf.train.string_input_producer(["test2.csv"])

reader = tf.TextLineReader()
key, value = reader.read(filename_queue)

# Default values, in case of empty columns. Also specifies the type of the
# decoded result.
record_defaults = [[1], [1], [1], [1], [1]]
col1, col2, col3, col4, col5 = tf.decode_csv(
    value, record_defaults=record_defaults)
# print tf.shape(col1)

features = tf.concat(0, [col1, col2, col3, col4])
with tf.Session() as sess:
  # Start populating the filename queue.
  coord = tf.train.Coordinator()
  threads = tf.train.start_queue_runners(coord=coord)

  for i in range(1200):
    # Retrieve a single instance:
    example, label = sess.run([features, col5])

  coord.request_stop()
  coord.join(threads)
```

Response:
The issue arises due to the shape of the tensors in your program. TL;DR Instead of tf.concat() you should use tf.pack(), which will transform the four scalar col tensors into a 1-D tensor of length 4. Before we start, note that you can use the get_shape() method on any Tensor object to get static shape information about that tensor. For example, the commented-out line in your code could be:
```
print col1.get_shape()
# ==> 'TensorShape([])' - i.e. `col1` is a scalar.
```
The value tensor returned by reader.read() is a scalar string. tf.decode_csv(value, record_defaults=[...]) produces, for each element of record_defaults, a tensor of the same shape as value, i.e. a scalar in this case. A scalar is a 0-dimensional tensor with a single element. tf.concat(i, xs) is not defined on scalars: it concatenates a list of N-dimensional tensors (xs) into a new N-dimensional tensor, along dimension i, where 0 <= i < N, and there is no valid i if N = 0. The tf.pack(xs) operator is designed to solve this problem simply. It takes a list of k N-dimensional tensors (with the same shape) and packs them into an N+1-dimensional tensor with size k in the 0th dimension. If you replace the tf.concat() with tf.pack(), your program will work:
```
# features = tf.concat(0, [col1, col2, col3, col4])
features = tf.pack([col1, col2, col3, col4])

with tf.Session() as sess:
  # Start populating the filename queue.
  # ...
```


Question:
Early stopping with multiple conditions I am doing multi-class classification for a recommender system (item recommendations), and I'm currently training my network using sparse_categorical_crossentropy loss. Therefore, it is reasonable to perform EarlyStopping by monitoring my validation loss, val_loss as such:
```
tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=10)
```
which works as expected. However, the performance of the network (recommender system) is measured by Average-Precision-at-10, and is tracked as a metric during training, as average_precision_at_k10. Because of this, I could also perform early stopping with this metric as such:
```
tf.keras.callbacks.EarlyStopping(monitor='average_precision_at_k10', patience=10)
```
which also works as expected. My problem: Sometimes the validation loss increases, whilst the Average-Precision-at-10 is improving and vice-versa. Because of this, I would need to monitor both, and perform early stopping, if and only if both are deteriorating. What I would like to do:
```
tf.keras.callbacks.EarlyStopping(monitor=['val_loss', 'average_precision_at_k10'], patience=10)
```
which obviously does not work. Any ideas how this could be done?

Response:
With guidance from Gerry P above I managed to create my own custom EarlyStopping callback, and thought I post it here in case anyone else are looking to implement something similar. If both the validation loss and the mean average precision at 10 does not improve for patience number of epochs, early stopping is performed.
```
class CustomEarlyStopping(keras.callbacks.Callback):
    def __init__(self, patience=0):
        super(CustomEarlyStopping, self).__init__()
        self.patience = patience
        self.best_weights = None

    def on_train_begin(self, logs=None):
        # The number of epoch it has waited when loss is no longer minimum.
        self.wait = 0
        # The epoch the training stops at.
        self.stopped_epoch = 0
        # Initialize the best as infinity.
        self.best_v_loss = np.Inf
        self.best_map10 = 0

    def on_epoch_end(self, epoch, logs=None):
        v_loss=logs.get('val_loss')
        map10=logs.get('val_average_precision_at_k10')

        # If BOTH the validation loss AND map10 does not improve for 'patience' epochs, stop training early.
        if np.less(v_loss, self.best_v_loss) and np.greater(map10, self.best_map10):
            self.best_v_loss = v_loss
            self.best_map10 = map10
            self.wait = 0
            # Record the best weights if current results is better (less).
            self.best_weights = self.model.get_weights()
        else:
            self.wait += 1
            if self.wait >= self.patience:
                self.stopped_epoch = epoch
                self.model.stop_training = True
                print("Restoring model weights from the end of the best epoch.")
                self.model.set_weights(self.best_weights)

    def on_train_end(self, logs=None):
        if self.stopped_epoch > 0:
            print("Epoch %05d: early stopping" % (self.stopped_epoch + 1))
```
It is then used as:
```
model.fit(
    x_train,
    y_train,
    batch_size=64,
    steps_per_epoch=5,
    epochs=30,
    verbose=0,
    callbacks=[CustomEarlyStopping(patience=10)],
)
```


Question:
InvalidArgumentError indices[i,0] = x is not in [0, x) in keras I have the code using keras 1.2 and tensorflow 1.1. I have run it but with error
```
import numpy as np
import keras
from keras import backend as K
from keras import initializers
from keras.models import Sequential, Model, load_model, save_model
from keras.layers.core import Dense, Lambda, Activation
from keras.layers import Embedding, Input, Dense, Multiply, Reshape, Flatten
from keras.optimizers import Adagrad, Adam, SGD, RMSprop
from keras.regularizers import l2

from sklearn.metrics import average_precision_score
from sklearn.metrics import  auc

def init_normal(shape, name=None):

    return initializers.lecun_uniform(seed=None)

def get_model(num_a, num_b, num_c, dim, regs=[0,0,0]):

    a = Input(shape=(1,), dtype='int32', name = 'a')
    b = Input(shape=(1,), dtype='int32', name = 'b')
    c = Input(shape=(1,), dtype='int32', name = 'c')

    Embedding_a = Embedding(input_dim = num_a, output_dim = dim,
                              embeddings_initializer='uniform', W_regularizer = l2(regs[0]), input_length=1)
    Embedding_b = Embedding(input_dim = num_b, output_dim = dim,
                              embeddings_initializer='uniform', W_regularizer = l2(regs[1]), input_length=1)
    Embedding_c = Embedding(input_dim = num_c, output_dim = dim,
                              embeddings_initializer='uniform', W_regularizer = l2(regs[2]), input_length=1)

    a_latent = Flatten()(Embedding_a(a))
    b_latent = Flatten()(Embedding_b(b))
    c_latent = Flatten()(Embedding_c(c))

    predict_vector = Multiply()([a_latent, b_latent, b_latent])
    prediction = Dense(1, activation='sigmoid', init='lecun_uniform', name = 'prediction')(predict_vector)

    model = Model(input=[a, b, c], output=prediction)

    return model

def evaluate_model(model, test_pos, test_neg):

    global _model
    global _test_pos
    global _test_neg
    _model = model
    _test_pos = test_pos
    _test_neg = test_neg
    print(_test_neg)

    a, b, c, labels = [],[],[],[]
    for item in _test_pos:

        a.append(item[0])
        b.append(item[1])
        c.append(item[2])
        labels.append(1)

    for item in _test_neg:

        a.append(item[0])
        b.append(item[1])
        c.append(item[2])
        labels.append(0)

    a = np.array(a)
    b = np.array(b)
    c = np.array(c)

    predictions = _model.predict([a, b, c],
                             batch_size=100, verbose=0)
    return average_precision_score(labels, predictions), auc(labels, predictions)

model = get_model(4, 8, 12, 2, [0,0,0])
model.compile(optimizer=Adam(lr=0.001), loss='binary_crossentropy')

pos_test = [[0, 0, 2], [4, 8, 8], [2, 5, 4], [0, 0, 0]]
neg_test = [[3, 3, 2], [2, 1, 8], [1, 4, 1], [3, 3, 12]]

aupr, auc = evaluate_model(model, pos_test, neg_test)
print(aupr, auc)
```
However, It give me error:any way to fix it?
```
InvalidArgumentError (see above for traceback): indices[1,0] = 4 is not in [0, 4)
     [[Node: embedding_4/embedding_lookup = Gather[Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@embedding_4/embeddings"], validate_indices=true, _device="/job:localhost/replica:0/task:0/cpu:0"](embedding_4/embeddings/read, _recv_a_1_0)]]
```

Response:
The problem is, you defined embedding input_dim as 4, 8 and 12 while it should be is 5, 9, 13. Because input_dim in embedding should be max_index + 1. It is also clearly mentioned in Keras docs: Size of the vocabulary, i.e. maximum integer index + 1. How to fix the issue? Change get_model method to:
```
model = get_model(5, 9, 13, 2, [0, 0, 0])
```
Or alternatively change index of data to:
```
pos_test = [[0, 0, 2], [3, 7, 7], [2, 5, 4], [0, 0, 0]]
neg_test = [[3, 3, 2], [2, 1, 7], [1, 4, 1], [3, 3, 11]]
```


Question: How does Tensorflow support Cuda streams? Does Tensorflow utilize Cuda streams automatically for concurrent execution of the computation graph on a single GPU or should streams be assigned manually to ops/tensors ?

Response: For now, TensorFlow only uses one compute stream, and multiple copy streams. Some kernels may choose to use multiple streams for computation, while maintaining a single-stream semantics. Our experiment showed that enabling multi-stream automatically does not bring much performance gains, since most of our kernels are large enough to utilize all processors in GPU. But enabling multi-stream would disable our current design to recycle GPU memory aggressively. This is a decision we might revisit in the future. If that happens, it is likely for TensorFlow to automatically assign ops/kernels to different Cuda streams, without exposing them to users.


Question: How to run tensorflow inference for multiple models on GPU in parallel? Do you know any elegant way to do inference on 2 python processes with 1 GPU tensorflow? Suppose I have 2 processes, first one is classifying cats/dogs, 2nd one is classifying birds/planes, each process is running different tensorflow model and run on GPU. These 2 models will be given images from different cameras continuously. Usually, tensorflow will occupy all memory of the entire GPU. So when you start another process, it will crash saying OUT OF MEMORY or failed convolution CUDA or something along that line. Is there a tutorial/article/sample code that shows how to load 2 models in different processes and both run in parallel? This is very useful also in case you are running a model inference while you are doing some heavy graphics e.g. playing games. I also want to know how running the model affects the game. I've tried using python Thread and it works but each model predicts 2 times slower (and you know that python thread is not utilizing multiple CPU cores). I want to use python Process but it's not working. If you have sample few lines of code that work I would appreciate that very much. I've attached current Thread code also:

Response:
As summarized here, you can specify the proportion of GPU memory allocated per process.
```
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.333)

sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))
```
Using Keras, it may be simpler to allow 'memory growth' which will expand the allocated memory on demand as described here.
```
import tensorflow as tf
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    try:
        for gpu in gpus:
            tf.config.experimental.set_memory_growth(gpu, True)
    except RuntimeError as e:
        print(e)
```
The following should work for Tensorflow 2.0:
```
from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession

config = ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.2
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)
```


Question: Can Tensorflow be used for global minimization of multivariate functions? I've been curious if TF can be used for global optimization of a function. For example, could it be used to efficiently find the ground state of a Lennard-Jones potential? Would it be any better or worse than existing optimization methods, like Basin-Hopping? Part of my research involves searching for the ground state of large, multi-component molecules. Traditional methods (BH, ect.) are good for this, but also quite slow. I've looked into TF and there are parts that seem robust enough to apply to this problem, although my limited web search doesn't appear to show any use of TF to this problem.

Response: The gradient descent performed to train neural networks is considering only a local region of the function. There is thus no guarantee that it will converge toward a global minimum (which is actually fine for most machine-learning algorithms ; given the really high dimensionality of the considered spaces, one is usually happy to find a good local minimum without having to explore around too much). That being said, one could certainly use Tensorflow (or any such frameworks) to implement a local optimizer for the global basin-hopping scheme, e.g. as follows (simplified algorithm): Choose a starting point; Use your local optimizer to get the local minimum; Apply some perturbation to the coordinates of this minimum; From this new position, re-use your local optimizer to get the next local minimum; Keep the best minimum, repeat from 3. Actually, some people are currently trying to implement this exact scheme, interfacing TF with scipy.optimize.basinhopping(). Current development and discussions can be found in this Github issue.


Question: Using deep learning models from TensorFlow in other language environments [closed] Closed. This question needs to be more focused. It is not currently accepting answers. Want to improve this question? Guide the asker to update the question so it focuses on a single, specific problem. Narrowing the question will help others answer the question concisely. You may edit the question if you feel you can improve it yourself. If edited, the question will be reviewed and might be reopened. Closed 7 years ago. Improve this question I have a decent amount of experience with TensorFlow, and I am about to embark on a project which will ultimately culminate in using a TensorFlow trained model in a C# production environment. Essentially, I will have live data which will come into the C# environment, and I will ultimately need to output decisions / take certain actions based on the output of my model in TensorFlow. This is basically just a constraint of the existing infrastructure. I can think of a couple of potentially bad ways to implement this, such as writing the data to disk and then calling the Python part of the application and then finally reading the result output by the Python application and taking some action based on it. This is slow, however. Are there faster ways to accomplish this same integrated relationship between C# and the Python-based Tensorflow. I see that there appear to be some ways to do this with C++ and TensorFlow, but what about C#?

Response: This is a prime use case for TensorFlow Serving, which lets you create a C++ process that can run inference on a trained TensorFlow model, and serves inference requests over gRPC. You can write client code in any language that gRPC supports. Take a look at the MNIST tutorial: C++ server and Python client components.


Question:
Is it still necessary to implement `compute_output_shape()` when defining a custom tf.keras Layer? I have implemented a custom Layer in tf.keras, using TensorFlow 2.1.0. In the past, when using the stand-alone Keras, it was important to define the compute_output_shape(input_shape) method in any custom layer so that the computational graph could be created. Now, having moved to TF2, I found out that even if I remove that method from my custom implementation the layer still works as expected. Apparently, it works both in eager and graph mode. This is an example of what I mean:
```py
from tensorflow.keras.layers import Layer, Input
from tensorflow.keras.models import Sequential
import numpy as np

class MyLayer(Layer):
    def call(self, inputs):
        return inputs[:, :-1]  # Do something that changes the shape

m = Sequential([MyLayer(), MyLayer()])
m.predict(np.ones((10, 3)))  # This would not have worked in the past
```
Is it safe to say that compute_output_shape() is not necessary anymore? Am I missing something important? In the documentation there's no explicit mention of removing compute_output_shape(), although none of the examples implements it explicitly. Thanks

Response: It is not mentioned in the Tensorflow Documentation but in Chapter 12, Custom Models and Training with TensorFlow of the book, Hands-on Machine Learning using Scikit-Learn and Tensorflow (2nd Edition Updated for Tensorflow 2) of O'REILLY Publications, written by Aurelien Geron, it is mentioned as shown in the screenshot below: To answer your question, yes, it is safe to say compute_output_shape is not needed unless the Layer is Dynamic. This is evident from this Tensorflow Tutorial on Custom Layer where compute_output_shape is not used. Hope this helps. Happy Learning!


Question: Most scalable way for using generators with tf.data ? tf.data guide says `from_generator` has limited scalability tf.data has a from_generator initializer, it doesn't seem like it's scalable. From the official guide Caution: While this is a convienient approach it has limited portability and scalibility. It must run in the same python process that created the generator, and is still subject to the Python GIL. https://www.tensorflow.org/guide/data#consuming_python_generators And in the official documentation NOTE: The current implementation of Dataset.from_generator() uses tf.numpy_function and inherits the same constraints. In particular, it requires the Dataset- and Iterator-related operations to be placed on a device in the same process as the Python program that called Dataset.from_generator(). The body of generator will not be serialized in a GraphDef, and you should not use this method if you need to serialize your model and restore it in a different environment. NOTE: If generator depends on mutable global variables or other external state, be aware that the runtime may invoke generator multiple times (in order to support repeating the Dataset) and at any time between the call to Dataset.from_generator() and the production of the first element from the generator. Mutating global variables or external state can cause undefined behavior, and we recommend that you explicitly cache any external state in generator before calling Dataset.from_generator(). https://www.tensorflow.org/api_docs/python/tf/data/Dataset#from_generator However, generators are the a fairly common method in training over very large amounts of data. So there must be some alternative best practice for this, but the official Tensorflow data guide doesn't not give any information on this.

Response: Iterate through your generator and write the data to a TFRecord. Then use TFRecordDataset. This is the guide. https://www.tensorflow.org/tutorials/load_data/tfrecord TF is built to use these types of Datasets effectively with multi-gpu. Sharding the data to disk also improves shuffling.


Question: What does "relu" stand for in tf.nn.relu? In its API documentation, it says "Computes rectified linear". Is it Re(ctified) L(inear)... what is U then?

Response: Re(ctified) L(inear) (U)nit Usually a layer in a neural network has some input, say a vector, and multiplies that by a weight matrix, resulting i.e. again in a vector. Each value in the result (usually a float) is then considered an output. However, most layers in neural networks nowadays involve nonlinearities, hence an add-on function that, you might say, adds complexity to these output values. For long these have been sigmoids and tanhs. But more recently people use a function that results in 0 if the input is negative, and the input itself if that input is 0 or positive. This specific add-on function (or better "activation function") is called a relu.


Question:
How can I view Tensor as an image? I learned some data with tensorflow. For the test, I saw the shape of the final result. It was tensor of (1, 80, 80, 1). I use matplotlib or PIL to do this, I wanted to see the image after changing to a pie array. But I could not change the tensor to numpy. I could not do anything because of the session even if I used eval (). There is no way to convert tensor to numpy. Can I see the tensor as an image?
```
(mytensor1) # mytensor

arr = np.ndarray(mytensor1)
arr_ = np.squeeze(arr)
plt.imshow(arr_)
plt.show()
```
but there is error message: TypeError: expected sequence object with len >= 0 or a single integer

Response:
You can use squeeze function from numpy. For example
```
arr = np.ndarray((1,80,80,1))#This is your tensor
arr_ = np.squeeze(arr) # you can give axis attribute if you wanna squeeze in specific dimension
plt.imshow(arr_)
plt.show()
```
Now, you can easily display this image (e.g. above code, assuming you are using matplotlib.pyplot as plt).


Question:
Attempting to reset tensorflow graph when using keras, failing I'm spinning up a Python 3 API w/gunicorn that uses keras to calculate vectors for an image, pretty straightforward. How can I reset the data stored in memory for each request? Slowly over time the requests increase in the time it takes to respond. I've run a profiler and it's specifically this line in tensorflow (also memory usage goes up slowly over time per process):
```
#tensorflow/python/framework/ops.py:2317:_as_graph_def
graph.node.extend([op.node_def])
```
It takes longer as more data is in the node. Here is the code I execute:
```
# We have 11439MiB of GPU memory, lets only use 2GB of it:
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.22
sess = tf.Session(config=config)
set_session(sess)
sess.graph.as_default()

# Get the vector for the image
img_size = (224,224)
vgg = VGG16(include_top=False, weights='imagenet')
img = kimage.load_img(tmpfile.name, target_size=img_size)
x = kimage.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)
pred = vgg.predict(x)
vectors = pred.ravel().tolist()
```
I thought as_default() would help, but it doesn't. I also tried closing the session after I get the list of vectors, and that fails.

Response:
```
from keras import backend as K
K.clear_session()
```


Question: Asynchronous computation in TensorFlow Recently I've been toying with TensorFlow and I mentioned that the framework is not able to use all my available computational resources. In Convolutional Neural Networks tutorial they mention that Naively employing asynchronous updates of model parameters leads to sub-optimal training performance because an individual model replica might be trained on a stale copy of the model parameters. Conversely, employing fully synchronous updates will be as slow as the slowest model replica. Although they mention it in both in the tutorial and in a whitepaper I did not really find a way to do the asynchronous parallel computation on a local machine. Is it even possible? Or is it part of the distributed to-be-released version of TensorFlow. If it is, then how?

Response:
Asynchronous gradient descent is supported in the open-source release of TensorFlow, without even modifying your graph. The easiest way to do it is to execute multiple concurrent steps in parallel:
```py
loss = ...

# Any of the optimizer classes can be used here.
train_op = tf.train.GradientDescentOptimizer(0.01).minimize(loss)

sess = tf.Session()
sess.run(tf.initialize_all_variables())

def train_function():
  # TODO: Better termination condition, e.g. using a `max_steps` counter.
  while True:
    sess.run(train_op)

# Create multiple threads to run `train_function()` in parallel
train_threads = []
for _ in range(NUM_CONCURRENT_STEPS):
  train_threads.append(threading.Thread(target=train_function))

# Start the threads, and block on their completion.
for t in train_threads:
  t.start()
for t in train_threads:
  t.join()
```
This example sets up NUM_CONCURRENT_STEPS calls to sess.run(train_op). Since there is no coordination between these threads, they proceed asynchronously. It's actually more challenging to achieve synchronous parallel training (at present), because this requires additional coordination to ensure that all replicas read the same version of the parameters, and that all of their updates become visible at the same time. The multi-GPU example for CIFAR-10 training performs synchronous updates by making multiple copies of the "tower" in the training graph with shared parameters, and explicitly averaging the gradients across the towers before applying the update. N.B. The code in this answer places all computation on the same device, which will not be optimal if you have multiple GPUs in your machine. If you want to use all of your GPUs, follow the example of the multi-GPU CIFAR-10 model, and create multiple "towers" with their operations pinned to each GPU. The code would look roughly as follows:
```py
train_ops = []

for i in range(NUM_GPUS):
  with tf.device("/gpu:%d" % i):
    # Define a tower on GPU `i`.
    loss = ...

    train_ops.append(tf.train.GradientDescentOptimizer(0.01).minimize(loss))

def train_function(train_op):
  # TODO: Better termination condition, e.g. using a `max_steps` counter.
  while True:
    sess.run(train_op)

# Create multiple threads to run `train_function()` in parallel
train_threads = []
for train_op in train_ops:
  train_threads.append(threading.Thread(target=train_function, args=(train_op,))

# Start the threads, and block on their completion.
for t in train_threads:
  t.start()
for t in train_threads:
  t.join()
```
Note that you might find it convenient to use a "variable scope" to facilitate variable sharing between the towers.


Question:
How can I multiply a vector and a matrix in tensorflow without reshaping? This:
```
import numpy as np
a = np.array([1, 2, 1])
w = np.array([[.5, .6], [.7, .8], [.7, .8]])

print(np.dot(a, w))
# [ 2.6  3. ] # plain nice old matrix multiplication n x (n, m) -> m

import tensorflow as tf

a = tf.constant(a, dtype=tf.float64)
w = tf.constant(w)

with tf.Session() as sess:
    print(tf.matmul(a, w).eval())
```
results in:
```none
C:\_\Python35\python.exe C:/Users/MrD/.PyCharm2017.1/config/scratches/scratch_31.py
[ 2.6  3. ]
# bunch of errors in windows...
Traceback (most recent call last):
  File "C:\_\Python35\lib\site-packages\tensorflow\python\framework\common_shapes.py", line 671, in _call_cpp_shape_fn_impl
    input_tensors_as_shapes, status)
  File "C:\_\Python35\lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "C:\_\Python35\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape must be rank 2 but is rank 1 for 'MatMul' (op: 'MatMul') with input shapes: [3], [3,2].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/MrD/.PyCharm2017.1/config/scratches/scratch_31.py", line 14, in <module>
    print(tf.matmul(a, w).eval())
  File "C:\_\Python35\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1765, in matmul
    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
  File "C:\_\Python35\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 1454, in _mat_mul
    transpose_b=transpose_b, name=name)
  File "C:\_\Python35\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 763, in apply_op
    op_def=op_def)
  File "C:\_\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 2329, in create_op
    set_shapes_for_outputs(ret)
  File "C:\_\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 1717, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "C:\_\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 1667, in call_with_requiring
    return call_cpp_shape_fn(op, require_shape_fn=True)
  File "C:\_\Python35\lib\site-packages\tensorflow\python\framework\common_shapes.py", line 610, in call_cpp_shape_fn
    debug_python_shape_fn, require_shape_fn)
  File "C:\_\Python35\lib\site-packages\tensorflow\python\framework\common_shapes.py", line 676, in _call_cpp_shape_fn_impl
    raise ValueError(err.message)
ValueError: Shape must be rank 2 but is rank 1 for 'MatMul' (op: 'MatMul') with input shapes: [3], [3,2].

Process finished with exit code 1
```
(not sure why the same exception is raised inside its handling) The solution suggested in Tensorflow exception with matmul is reshaping the vector to a matrix but this leads to needlessly complicated code - is there still no other way to multiply a vector with a matrix? Incidentally using expand_dims (as suggested in the link above) with default arguments raises a ValueError - that's not mentioned in the docs and defeats the purpose of having a default argument.

Response:
tf.einsum gives you the ability to do exactly what you need in concise and intuitive form:
```
with tf.Session() as sess:
    print(tf.einsum('n,nm->m', a, w).eval())
    # [ 2.6  3. ]
```
You even get to write your comment explicitly n x (n, m) -> m. It is more readable and intuitive in my opinion. My favorite use case is when you want to multiply a batch of matrices with a weight vector:
```
n_in = 10
n_step = 6
input = tf.placeholder(dtype=tf.float32, shape=(None, n_step, n_in))
weights = tf.Variable(tf.truncated_normal((n_in, 1), stddev=1.0/np.sqrt(n_in)))
Y_predict = tf.einsum('ijk,kl->ijl', input, weights)
print(Y_predict.get_shape())
# (?, 6, 1)
```
So you can easily multiply weights over all batches with no transformations or duplication. This you can not do by expanding dimensions like in other answer. So you avoid the tf.matmul requirement to have matching dimensions for batch and other outer dimensions: The inputs must, following any transpositions, be tensors of rank >= 2 where the inner 2 dimensions specify valid matrix multiplication arguments, and any further outer dimensions match.


Question:
How to define weight decay for individual layers in TensorFlow? In CUDA ConvNet, we can write something like this (source) for each layer:
```
[conv32]
epsW=0.001
epsB=0.002
momW=0.9
momB=0.9
wc=0
```
where wc=0 refers to the L2 weight decay. How can the same be achieved in TensorFlow?

Response:
You can add all the variables you want to add weight decay to, to a collection name 'variables' and then you calculate the L2 norm weight decay for the whole collection.
```
# Create your variables
  weights = tf.get_variable('weights', collections=['variables'])

  with tf.variable_scope('weights_norm') as scope:
    weights_norm = tf.reduce_sum(
      input_tensor = WEIGHT_DECAY_FACTOR*tf.pack(
          [tf.nn.l2_loss(i) for i in tf.get_collection('weights')]
      ),
      name='weights_norm'
  )

  # Add the weight decay loss to another collection called losses
  tf.add_to_collection('losses', weights_norm)

  # Add the other loss components to the collection losses
  # ...

  # To calculate your total loss
  tf.add_n(tf.get_collection('losses'), name='total_loss')
```


Question:
All runs are not visible on TensorBoard I am using tensorboard to visualize three runs. I have a folder, logs, which contains three files:
```
2016-03-18_22-11-12
2016-03-18_22-11-27
2016-03-18_22-23-46
```
when I run tensorboard --logdir . (from logs), only 2016-03-18_22-23-46 is visible: And if I delete 2016-03-18_22-23-46 from logs and restart tensorboard, then only 2016-03-18_22-11-27 is visible. Any idea of what's happening here? Edit: the log files are (to my surprise), quite big: here is the result of du -h:
```
1,1G    ./2016-03-18_22-23-46
925M    ./2016-03-18_22-11-12
934M    ./2016-03-18_22-11-27
2,9G    .
```
EDIT: The above run structure can be obtained via logging and checkpointing into subdirectories of the tensorboard log-dir:
```
run_time = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
FLAGS.train_dir = '/datalab/tf_runs/' + run_time #Directory to put the training data.
summary_writer = tf.train.SummaryWriter(FLAGS.train_dir, sess.graph_def)
```

Response: Tensorboard takes a while to parse the log files. If you refresh the graph, you can see it showing more and more iterations. When it finished parsing one, the next run will pop up. It is a bit annoying that there is no visual indication for this, maybe this is worth a feature request at https://github.com/tensorflow.


Question:
How to display Runtime Statistics in Tensorboard using Estimator API in a distributed environment This article illustrates how to add Runtime statistics to Tensorboard:
```
run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
    run_metadata = tf.RunMetadata()
    summary, _ = sess.run([merged, train_step],
                          feed_dict=feed_dict(True),
                          options=run_options,
                          run_metadata=run_metadata)
    train_writer.add_run_metadata(run_metadata, 'step%d' % i)
    train_writer.add_summary(summary, i)
    print('Adding run metadata for', i)
```
which creates the following details in Tensorboard: This is fairly straightforward on a single machine. How could one do this in a distributed environment using Estimators?

Response:
I use the following hook, based on ProfilerHook, to have the estimator output the run metadata into the model directory and inspect it later with Tensorboard.
```
import tensorflow as tf
from tensorflow.python.training.session_run_hook import SessionRunHook, SessionRunArgs
from tensorflow.python.training import training_util
from tensorflow.python.training.basic_session_run_hooks import SecondOrStepTimer

class MetadataHook(SessionRunHook):
    def __init__ (self,
                  save_steps=None,
                  save_secs=None,
                  output_dir=""):
        self._output_tag = "step-{}"
        self._output_dir = output_dir
        self._timer = SecondOrStepTimer(
            every_secs=save_secs, every_steps=save_steps)

    def begin(self):
        self._next_step = None
        self._global_step_tensor = training_util.get_global_step()
        self._writer = tf.summary.FileWriter (self._output_dir, tf.get_default_graph())

        if self._global_step_tensor is None:
            raise RuntimeError("Global step should be created to use ProfilerHook.")

    def before_run(self, run_context):
        self._request_summary = (
            self._next_step is None or
            self._timer.should_trigger_for_step(self._next_step)
        )
        requests = {"global_step": self._global_step_tensor}
        opts = (tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
            if self._request_summary else None)
        return SessionRunArgs(requests, options=opts)

    def after_run(self, run_context, run_values):
        stale_global_step = run_values.results["global_step"]
        global_step = stale_global_step + 1
        if self._request_summary:
            global_step = run_context.session.run(self._global_step_tensor)
            self._writer.add_run_metadata(
                run_values.run_metadata, self._output_tag.format(global_step))
            self._writer.flush()
        self._next_step = global_step + 1

    def end(self, session):
        self._writer.close()
```
To use it, one creates the estimator instance (my_estimator) as usual, whether it is pre-made one or a custom estimator. The desired operation is called passing an instance of the class above as a hook. For example:
```
hook = MetadataHook(save_steps=1, output_dir=<model dir>)
my_estimator.train( train_input_fn, hooks=[hook] )
```
The run metadata will be placed in the model dir and can be inspected by TensorBoard.


Question: How to import pre-downloaded MNIST dataset from a specific directory or folder? I have downloaded the MNIST dataset from LeCun site. What I want is to write the Python code in order to extract the gzip and read the dataset directly from the directory, meaning that I don't have to download or access to the MNIST site anymore. Desire process: Access folder/directory --> extract gzip --> read dataset (one hot encoding) How to do it? Since almost all tutorials have to access to the either the LeCun or Tensoflow site to download and read the dataset. Thanks in advance!

Response:
If you have the MNIST data extracted, then you can load it low-level with NumPy directly:
```
def loadMNIST( prefix, folder ):
    intType = np.dtype( 'int32' ).newbyteorder( '>' )
    nMetaDataBytes = 4 * intType.itemsize

    data = np.fromfile( folder + "/" + prefix + '-images-idx3-ubyte', dtype = 'ubyte' )
    magicBytes, nImages, width, height = np.frombuffer( data[:nMetaDataBytes].tobytes(), intType )
    data = data[nMetaDataBytes:].astype( dtype = 'float32' ).reshape( [ nImages, width, height ] )

    labels = np.fromfile( folder + "/" + prefix + '-labels-idx1-ubyte',
                          dtype = 'ubyte' )[2 * intType.itemsize:]

    return data, labels

trainingImages, trainingLabels = loadMNIST( "train", "../datasets/mnist/" )
testImages, testLabels = loadMNIST( "t10k", "../datasets/mnist/" )
```
And to convert to hot-encoding:
```
def toHotEncoding( classification ):
    # emulates the functionality of tf.keras.utils.to_categorical( y )
    hotEncoding = np.zeros( [ len( classification ),
                              np.max( classification ) + 1 ] )
    hotEncoding[ np.arange( len( hotEncoding ) ), classification ] = 1
    return hotEncoding

trainingLabels = toHotEncoding( trainingLabels )
testLabels = toHotEncoding( testLabels )
```


Question:
how to create confusion matrix for classification in tensorflow I have CNN model which has 4 output nodes, and I am trying to compute the confusion matrix so that i can know the individual class accuracy. I am able to compute the overall accuracy. In the link here, Igor Valantic gave a function which can compute the confusion matrix variables. it gives me an error at correct_prediction = tf.nn.in_top_k(logits, labels, 1, name="correct_answers") and the error is TypeError: DataType float32 for attr 'T' not in list of allowed values: int32, int64 I have tried typecasting logits to int32 inside function mentioned def evaluation(logits, labels), it gives another error at computing correct_prediction = ... as TypeError:Input 'predictions' of 'InTopK' Op has type int32 that does not match expected type of float32 how to calculate this confusion matrix ?
```
sess = tf.Session()
model = dimensions() # CNN input weights are calculated
data_train, data_test, label_train, label_test =  load_data(files_test2,folder)
data_train, data_test, = reshapedata(data_train, data_test, model)
# input output placeholders
x  = tf.placeholder(tf.float32, [model.BATCH_SIZE, model.input_width,model.input_height,model.input_depth]) # last column = 1
y_ = tf.placeholder(tf.float32, [model.BATCH_SIZE, model.No_Classes])
p_keep_conv = tf.placeholder("float")
#
y  = mycnn(x,model, p_keep_conv)
# loss
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y, y_))
# train step
train_step = tf.train.AdamOptimizer(1e-3).minimize(cost)
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
true_positives, false_positives, true_negatives, false_negatives = evaluation(y,y_)
lossfun = np.zeros(STEPS)
sess.run(tf.global_variables_initializer())

for i in range(STEPS):
    image_batch, label_batch = batchdata(data_train, label_train, model.BATCH_SIZE)
    epoch_loss = 0
    for j in range(model.BATCH_SIZE):
        sess.run(train_step, feed_dict={x: image_batch, y_: label_batch, p_keep_conv:1.0})
        c = sess.run( cost, feed_dict={x: image_batch, y_: label_batch, p_keep_conv: 1.0})
        epoch_loss += c
    lossfun[i] = epoch_loss
    print('Epoch',i,'completed out of',STEPS,'loss:',epoch_loss )
 TP,FP,TN,FN = sess.run([true_positives, false_positives, true_negatives,  false_negatives], feed_dict={x: image_batch, y_: label_batch, p_keep_conv:1.0})
```
this is my code snippet

Response:
You can simply use Tensorflow's confusion matrix. I assume y are your predictions, and you may or may not have num_classes (which is optional)
```
y_ = placeholder_for_labels # for eg: [1, 2, 4]
y = mycnn(...) # for eg: [2, 2, 4]

confusion = tf.confusion_matrix(labels=y_, predictions=y, num_classes=num_classes)
```
If you print(confusion), you get
```
[[0 0 0 0 0]
   [0 0 1 0 0]
   [0 0 1 0 0]
   [0 0 0 0 0]
   [0 0 0 0 1]]
```
If print(confusion) is not printing the confusion matrix, then use print(confusion.eval(session=sess)). Here sess is the name of your TensorFlow session.


Question:
How to fill a tensor in C++ I'm creating a tensor like this:
```
tensorflow::Tensor a(tensorflow::DT_FLOAT, tensorflow::TensorShape());
```
I know how to fill a scalar value:
```
a.scalar<float>()() = 8.0;
```
But I don't know how to fill a tensor like [1, 4, 2].

Response:
There are a few options. If the tensor is really a small vector, like in your case, you can do the following :
```
tensorflow::Tensor a(tensorflow::DT_FLOAT, tensorflow::TensorShape(3));
a.vec<float>()(0) = 1.0f;
a.vec<float>()(1) = 4.0f;
a.vec<float>()(2) = 2.0f;
```
If you want to construct a slightly larger and/or multi-dimensional tensor, then tensorflow::ops::Input::Initializer declared in tensorflow/cc/framework/ops.h has many constructors that lets you construct a Tensor from various kinds of C++ constants such as simple primitive constants and nested initializer lists representing a multi-dimensional array. For example, if you want to construct a 2x2 matrix, you can do the following :
```
#include "tensorflow/cc/framework/cc/ops.h"

tensorflow::ops::Input::Initializer a({{1, 2}, {3, 4}});
// a.tensor will be a Tensor with type DT_INT32 and shape {2, 2}.
```


Question:
Tensorflow error: Using a `tf.Tensor` as a Python `bool` is not allowed I am struggling to implement an activation function in tensorflow in Python. The code is the following:
```python
def myfunc(x):
    if (x > 0):
        return 1
    return 0
```
But I am always getting the error: Using a tf.Tensor as a Python bool is not allowed. Use if t is not None:

Response:
Use tf.cond:
```
tf.cond(tf.greater(x, 0), lambda: 1, lambda: 0)
```
Another solution, which in addition supports multi-dimensional tensors:
```
tf.sign(tf.maximum(x, 0))
```
Note, however, that the gradient of this activation is zero everywhere, so the neural network won't learn anything with it.


Question: How to create dataset in the same format as the FSNS dataset? I'm working on this project based on TensorFlow. I just want to train an OCR model by attention_ocr based on my own datasets, but I don't know how to store my images and ground truth in the same format as FSNS datasets. Is there anybody also work on this project or know how to solve this problem?

Response:
The data format for storing training/test is defined in the FSNS paper https://arxiv.org/pdf/1702.03970.pdf (Table 4). To store tfrecord files with tf.Example protos you can use tf.python_io.TFRecordWriter. There is a nice tutorial, an existing answer on the stackoverflow and a short gist. Assume you have an numpy ndarray img which has num_of_views images stored side-by-side (see Fig. 3 in the paper): and a corresponding text in a variable text. You will need to define some function to convert a unicode string into a list of character ids padded to a fixed length and unpadded as well. For example:
```py
char_ids_padded, char_ids_unpadded = encode_utf8_string(
   text='abc',
   charset={'a':0, 'b':1, 'c':2},
   length=5,
   null_char_id=3)
```
the result should be:
```py
char_ids_padded = [0,1,2,3,3]
char_ids_unpadded = [0,1,2]
```
If you use functions _int64_feature and _bytes_feature defined in the gist you can create a FSNS compatible tf.Example proto using a following snippet:
```py
char_ids_padded, char_ids_unpadded = encode_utf8_string(
   text, charset, length, null_char_id)
example = tf.train.Example(features=tf.train.Features(
  feature={
    'image/format': _bytes_feature("PNG"),
    'image/encoded': _bytes_feature(img.tostring()),
    'image/class': _int64_feature(char_ids_padded),
    'image/unpadded_class': _int64_feature(char_ids_unpadded),
    'height': _int64_feature(img.shape[0]),
    'width': _int64_feature(img.shape[1]),
    'orig_width': _int64_feature(img.shape[1]/num_of_views),
    'image/text': _bytes_feature(text)
  }
))
```


Question:
Why the accuracy and binary_accuracy in keras have same result? I created a simple model for binary classification with Keras. The code is:
```py
# create model
    model = Sequential()
    model.add(Dense(250, input_dim=1, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    # Compile model
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy', 'binary_accuracy'])
```
My purpose was check the result of accuracy and binary_accuracy is understand difference between them. As Keras says binary_accuracy accuracy have threshold that default is .5, that `accuracy' haven't. When I test them with sample data the result is difference but in the train of model thy have same results in each epoch. for this true and predicted sample I tested accuracy and binary_accuracy:
```py
y_true = [[1], [1], [0], [0]]
   y_pred = [[0.51], [1], [0], [0.4]]
```
For binary_accuracy is:
```py
m = tf.keras.metrics.BinaryAccuracy()
m.update_state(y_true, y_pred)
m.result().numpy()
```
that result is: 1 For accuracy is:
```py
m = tf.keras.metrics.Accuracy()
m.update_state(y_true, y_pred)
m.result().numpy()
```
and the result is: '.5' But in the above model it is same for each of them in each epoch. Edit By changing the compile to this the result changed:
```py
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy', tf.keras.metrics.BinaryAccuracy(threshold=.7)])
```
Why accuracy work like binary_accuracy with threshold=0.5 in model but not in out of model?

Response: According to tf.keras.Model.compile() documentation: When you pass the strings 'accuracy' or 'acc', we convert this to one of tf.keras.metrics.BinaryAccuracy, tf.keras.metrics.CategoricalAccuracy, tf.keras.metrics.SparseCategoricalAccuracy based on the loss function used and the model output shape. We do a similar conversion for the strings 'crossentropy' and 'ce' as well. In your case it was transformed to BinaryAccuracy and hence result is the same. However tf.keras.metrics.Accuracy is something completely different. If you read the documentation: Calculates how often predictions equal labels. which means it looks at unique values of y_pred and y_true and treats every unique value as a distinct label. In your case 0.51 and 0.4 are treated as a separate labels and because they are not equal to 1 and 0, respectively, you get 0.5 Apologies for marking this question as a duplicate at first, the behaviour is different in tf.keras than in keras package


Question:
How to use multiple inputs in Tensorflow 2.x Keras Custom Layer? I'm trying to use multiple inputs in custom layers in Tensorflow-Keras. Usage can be anything, right now it is defined as multiplying the mask with the image. I've search SO and the only answer I could find was for TF 1.x so it didn't do any good.
```
class mul(layers.Layer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        # I've added pass because this is the simplest form I can come up with.
        pass

    def call(self, inputs):
        # magic happens here and multiplications occur
        return(Z)
```

Response:
EDIT: Since TensorFlow v2.3/2.4, the contract is to use a list of inputs to the call method. For keras (not tf.keras) I think the answer below still applies. Implementing multiple inputs is done in the call method of your class, there are two alternatives: List input, here the inputs parameter is expected to be a list containing all the inputs, the advantage here is that it can be variable size. You can index the list, or unpack arguments using the = operator:
```
def call(self, inputs):
      Z = inputs[0] * inputs[1]

      #Alternate
      input1, input2 = inputs
      Z = input1 * input2

      return Z
```
Multiple input parameters in the call method, works but then the number of parameters is fixed when the layer is defined:
```
def call(self, input1, input2):
      Z = input1 * input2

      return Z
```
Whatever method you choose to implement this depends if you need fixed size or variable sized number of arguments. Of course each method changes how the layer has to be called, either by passing a list of arguments, or by passing arguments one by one in the function call. You can also use *args in the first method to allow for a call method with a variable number of arguments, but overall keras' own layers that take multiple inputs (like Concatenate and Add) are implemented using lists.


Question:
How to save a list of numpy arrays into a single file and load file back to original form [duplicate] This question already has answers here: NumPy save some arrays at once (2 answers) Closed 5 years ago. I am currently trying to save a list of numpy arrays into a single file, an example of such a list can be of the form below
```py
import numpy as np
np_list = []
for i in range(10):
    if i % 2 == 0:
        np_list.append(np.random.randn(64))
    else:
        np_list.append(np.random.randn(32, 64))
```
I can combine all of them using into a single file using savez by iterating through list but is there any other way? I am trying to save weights returned by the function model.get_weights(), which is a list of ndarray and after retrieving the weights from the saved file I intend to load those weights into another model using model.set_weights(np_list). Therefore the format of the list must remain the same. Let me know if anyone has an elegant way of doing this.

Response:
I would go with np.save and np.load because it's platform-independent, faster than savetxt and works with lists of arrays, for example:
```py
import numpy as np

a = [
    np.arange(100),
    np.arange(200)
]
np.save('a.npy', np.array(a, dtype=object), allow_pickle=True)
b = np.load('a.npy', allow_pickle=True)
```
This is the documentation for np.save and np.load. And in this answer you can find a better discussion How to save and load numpy.array() data properly? Edit Like @AlexP mentioned numpy >= v1.24.2 does not support arrays with different sizes and types, so that's why the casting is necessary.


Question: How can I compute element-wise conditionals on batches in TensorFlow? I basically have a batch of neuron activations of a layer in a tensor A of shape [batch_size, layer_size]. Let B = tf.square(A). Now I want to compute the following conditional on each element in each vector in this batch: if abs(e) < 1: e ← 0 else e ← B(e) where e is the element in B that is at the same position as e. Can I somehow vectorize the entire operation with a single tf.cond operation?

Response:
You may want to look at tf.where(condition, x, y) For your issue:
```py
A = tf.placeholder(tf.float32, [batch_size, layer_size])
B = tf.square(A)

condition = tf.less(tf.abs(A), 1.)

res = tf.where(condition, tf.zeros_like(B), B)
```


Question:
What does the filter parameter mean in Conv2d layer? I am getting confused with the filter paramater, which is the first parameter in the Conv2D() layer function in keras. As I understand the filters are supposed to do things like edge detection or sharpening the image or blurring the image, but when I am defining the model as
```
input_shape = (32, 32, 3)
model = Sequential()
model.add( Conv2D(64, kernel_size=(5, 5), activation='relu', input_shape=input_shape, strides=(1,1), padding='same') )
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(Conv2D(64, kernel_size=(5, 5), activation='relu', input_shape=input_shape, strides=(1,1), padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(Conv2D(128, kernel_size=(5, 5), activation='relu', input_shape=input_shape, strides=(1,1), padding='same'))
model.add(Flatten())
model.add(Dense(3072, activation='relu'))
model.add(Dense(2048, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))
```
I am not mentioning the the edge detection or blurring or sharpening anywhere in the Conv2D function. The input images are 32 by 32 RGB images. So my question is, when I define the Convolution layer as Conv2D(64, ...), does this 64 means 64 different types of filters, such as vertical edge, horizontal edge, etc, which are chosen by keras at random? if so then is the output of the convolution layer (with 64 filters and 5x5 kernel and 1x1 stride) on a 32x32 1-channel image is 64 images of 28x28 size each. How are these 64 images combined to form a single image for further layers?

Response: The filters argument sets the number of convolutional filters in that layer. These filters are initialized to small, random values, using the method specified by the kernel_initializer argument. During network training, the filters are updated in a way that minimizes the loss. So over the course of training, the filters will learn to detect certain features, like edges and textures, and they might become something like the image below (from here). It is very important to realize that one does not hand-craft filters. These are learned automatically during training -- that's the beauty of deep learning. I would highly recommend going through some deep learning resources, particularly https://cs231n.github.io/convolutional-networks/ and https://www.youtube.com/watch?v=r5nXYc2wYvI&list=PLypiXJdtIca5sxV7aE3-PS9fYX3vUdIOX&index=3&t=3122s.


Question:
TF Keras how to get expected input shape when loading a model? Is it possbible to get the expected input shape from a 'model.h5' file? I have two models for the same dataset but with different options and shapes. The first one expects a dim of (None, 64, 48, 1) and the seconds model need input shape (None, 128, 96, 3). (Note: The width or the height are not fixed and can change when I train again). The channels problem was easy to "fix" (or bypass rather) by just using try: and except because there are only two options (1 for grayscale image and 3 for rgb image):
```
channels = self.df["channels"][0]
        file = ""
        try:
            images, src_images, data = self.get_images()
            images = self.preprocess_data(images, channels)
            predictions, file = self.load_model(images, file)
            self.predict_data(src_images, predictions, data)
        except:
            if channels == 1:
                print("Except channels =", channels)
                channels = 3
                images, src_images, data = self.get_images()
                images = self.preprocess_data(images, channels)
                predictions = self.load_model(images, file)
                self.predict_data(src_images, predictions, data)
            else:
                channels = 1
                print("Except channels =", channels)
                images, src_images, data = self.get_images()
                images = self.preprocess_data(images, channels)
                predictions = self.load_model(images, file)
                self.predict_data(src_images, predictions, data)
```
This workaround however cannot be used for the width and height of an image because there basically unlimited amount of options. Besides that it is rather slow because I read all the data twice and preprocess it twice for no reason. Is there a way to load the model.h5 file and print the expected input shape in a form like this?:
```
[None, 128, 96, 3]
```

Response:
I finally found the answer myself.
```
config = model.get_config() # Returns pretty much every information about your model
print(config["layers"][0]["config"]["batch_input_shape"]) # returns a tuple of width, height and channels
```
This will output the following:
```
(None, 128, 96, 3)
```


Question:
What do the functions tf.squeeze and tf.nn.rnn do? What do the functions tf.squeeze and tf.nn.rnn do? I searched these API, but I can't find argument, examples etc. Also, what is the shape of p_inputs formed by the following code using tf.squeeze, and what is the meaning and case of using tf.nn.rnn?
```
batch_num = 10
step_num = 2000
elem_num = 26

p_input = tf.placeholder(tf.float32, [batch_num, step_num, elem_num])
p_inputs = [tf.squeeze(t, [1]) for t in tf.split(1, step_num, p_input)]
```

Response: The best source of answers to questions like these is the TensorFlow API documentation. The two functions you mentioned create operations and symbolic tensors in a dataflow graph. In particular: The tf.squeeze() function returns a tensor with the same value as its first argument, but a different shape. It removes dimensions whose size is one. For example, if t is a tensor with shape [batch_num, 1, elem_num] (as in your question), tf.squeeze(t, [1]) will return a tensor with the same contents but size [batch_num, elem_num]. The tf.nn.rnn() function returns a pair of results, where the first element represents the outputs of a recurrent neural network for some given input, and the second element represents the final state of that network for that input. The TensorFlow website has a tutorial on recurrent neural networks with more details.


Question:
Tensorflow dynamic_rnn parameters meaning I'm struggling to understand the cryptic RNN docs. Any help with the following will be greatly appreciated.
```
tf.nn.dynamic_rnn(cell, inputs, sequence_length=None, initial_state=None, dtype=None, parallel_iterations=None, swap_memory=False, time_major=False, scope=None)
```
I'm struggling to understand how these parameters relate to the mathematical LSTM equations and RNN definition. Where is the cell unroll size? Is it defined by the 'max_time' dimension of the inputs? Is the batch_size only a convenience for splitting long data or it's related to minibatch SGD? Is the output state passed across batches?

Response:
tf.nn.dynamic_rnn takes in a batch (with the minibatch meaning) of unrelated sequences. cell is the actual cell that you want to use (LSTM, GRU,...) inputs has a shape of batch_size x max_time x input_size in which max_time is the number of steps in the longest sequence (but all sequences could be of the same length) sequence_length is a vector of size batch_size in which each element gives the length of each sequence in the batch (leave it as default if all your sequences are of the same size. This parameter is the one that defines the cell unroll size. Hidden state handling The usual way of handling hidden state is to define an initial state tensor before the dynamic_rnn, like this for instance :
```
hidden_state_in = cell.zero_state(batch_size, tf.float32)
output, hidden_state_out = tf.nn.dynamic_rnn(cell,
                                             inputs,
                                             initial_state=hidden_state_in,
                                             ...)
```
In the above snippet, both hidden_state_in and hidden_state_out have the same shape [batch_size, ...] (the actual shape depends on the type of cell you use but the important thing is that the first dimension is the batch size). This way, dynamic_rnn has an initial hidden state for each sequence. It will pass on the hidden state from time step to time step for each sequence in the inputs parameter on its own, and hidden_state_out will contain the final output state for each sequence in the batch. No hidden state is passed between sequences of the same batch, but only between time steps of the same sequence. When do I need to feed back the hidden state manually? Usually, when you're training, every batch is unrelated so you don't have to feed back the hidden state when doing a session.run(output). However, if you're testing, and you need the output at each time step, (i.e. you have to do a session.run() at every time step) you'll want to evaluate and feed back the output hidden state using something like this :
```
output, hidden_state = sess.run([output, hidden_state_out],
                                feed_dict={hidden_state_in:hidden_state})
```
otherwise tensorflow will just use the default cell.zero_state(batch_size, tf.float32) at each time step which equates to reinitialising the hidden state at each time step.


Question:
Tensorflow Relu Misunderstanding I've recently been doing a Udacity Deep Learning course which is based around TensorFlow. I have a simple MNIST program which is about 92% accurate:
```
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))

y = tf.nn.softmax(tf.matmul(x, W) + b)

y_ = tf.placeholder(tf.float32, [None, 10])
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))

train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

init = tf.initialize_all_variables()

sess = tf.Session()
sess.run(init)

for i in range(1000):
    batch_xs, batch_ys = mnist.train.next_batch(100)
    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})

correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels}))
```
My next assignment it to Turn the logistic regression example with SGD into a 1-hidden layer neural network with rectified linear units nn.relu() and 1024 hidden nodes I am having a mental block about this. Currently I have a 784 x 10 Matrix of weights, and a 10 element long bias vector. I don't understand how I connect the resulting 10 element vector from WX + Bias to 1024 Relus. If anyone could explain this to me I'd be very grateful.

Response: Right now you have something like this and you need something like this (this diagram is missing ReLU layer which goes after +b1)


Question:
Input shape in keras (This loss expects targets to have the same shape as the output) this is my first time using keras, I'm trying to follow a tutorial I've found online and fit my own data to it. I have a matrix and binary labels.
```
> str(d_train)
 num [1:1062, 1:180] -0.04748 0.04607 -0.05429 -0.0126 -0.00219 ...
> str(trainlabels)
 num [1:1062, 1:2] 0 0 0 0 0 0 1 0 0 0 ...
```
my code:
```
model = keras_model_sequential()
model %>%
  layer_dense(units = 8, activation = 'relu', input_shape = c(180)) %>%
  layer_dense(units = 3, activation = "softmax")
summary(model)
## Compile
model %>%
  compile(loss = "binary_crossentropy",
          optimizer = "adam",
          metrics = "accuracy")
## Fit model
history = model %>%
  fit(d_train,
      trainlabels,
      epoch=200,
      batch_size=32,
      validation_split=0.2)
```
I can't seem to fit the model, I'm getting this error message:
```
Error in py_call_impl(callable, dots$args, dots$keywords) :
  ValueError: A target array with shape (1062, 2) was passed for an output of shape (None, 3) while using as loss `binary_crossentropy`. This loss expects targets to have the same shape as the output.
```
Based on the error message, asking for different shape of my input array, I tried to change the dimensions around with no luck.

Response:
I am not an R expert, but here:
```
layer_dense(units = 3, activation = "softmax")
```
You are telling Keras that the output of your network has three classes. Your labels have shape (1062, 2) which suggest it has two classes, hence there is an inconsistency. You could just change units = 2 in your last dense and it should work. Also note that you are using the softmax activation, and in that case you should prefer to use the categorical_crossentropy loss. To use binary_crossentropy for binary classification, you should have units = 1, sigmoid activation, and labels should be (1062, 1) or (1062,), which means they are 0-1 encoded.


Question:
Can I delete events.out.tfevents.XXXXXXXXXX.computer_name files from training folder I am training faster_rcnn_inception module for object detection on custom dataset. In training directory,we found folder called eval_0 and tensorflow generated events.out.tfevents.xxxxxx files. Training Directory structure as follows
```
+training_dir
    +eval_0
     -events.out.tfevents.1542309785.instance-1  1.2GB
     -events.out.tfevents.1542367255.instance-1  5.3GB
     -events.out.tfevents.1542369886.instance-1  3.6GB
     -events.out.tfevents.1542624154.instance-1  31MB
     -events.out.tfevents.1543060258.instance-1  19MB
     -events.out.tfevents.1543066775.instance-2  1.6GB
 -events.out.tfevents.1542308099.instance-1  17MB
 -events.out.tfevents.1542308928.instance-1  17MB
 -events.out.tfevents.1542366369.instance-1  17MB
 -events.out.tfevents.1542369000.instance-1  17MB
 -events.out.tfevents.1542623262.instance-1  17MB
 -events.out.tfevents.1543064936.instance-2  17MB
 -events.out.tfevents.1543065796.instance-2  17MB
 -events.out.tfevents.1543065880.instance-2  17MB
 -model.ckpt-96004.data-00000-of-00001
 -model.ckpt-96004.data-00000-of-00001
 -model.ckpt-96004.index
 -model.ckpt-96004.meta
 -model.ckpt-96108.data-00000-of-00001
 -model.ckpt-96108.index
 -model.ckpt-96108.meta
```
As per my understanding, tfevents files in eval_0 folder are summery files of evaluation and tfevents files in training_dir are summery files of training. I have interrupted training process several times and continued from recent checkpoint. I also understand restarting training process generates new tfevents files. My Questions as follows: Why training tfevents_files have same size, but in case if eval_0/tfevents_files size varies ? Why interrupting training generates new tfevents_file in training folder, but same not observed in case of eval_0? Can I delete all tfevents files in eval_0 except latest one? Does it affect on training or evolution history?

Response: tfevents files are not essential for training and can be safely removed. In Tensorflow tfevents are created by FileWriters and are generally used to store summary output. Here are some common examples of how tf.summaries are used: storing a description of the tensorflow graph before training starts writing a value of the loss function for every training step storing a histogram of activations or weights for a layer once per epoch storing an example of output image of the network once on every validation storing average precision (or any other metric) for the whole validation set This information is not essential for training and can therefore be deleted. Yet, it might come in handy for debugging or studying behavior of the model. TensorBoard is the most common tool to read and visualize data stored in tfevent files. Anyone can read and interpret TFRecord files manually using protobuf protocol and it's implementation for Python, C++ and other. tfevents are written in TFRecord format. TFRecord is a simple format for storing a sequence of binary records. Tensorflow always appends new events/summaries to the end of the file if file already exists. This explains file grows. Due to details of implementation of optimization routine provided with tensorflow/models/reserach/object_detection training and evaluation event files have different behaviour. Namely, evaluation event file is created using a FileWriter directly, which will reuse latest existing event file in the log_dir whenever one exists. Implementation also has large number of summaries that are collected regularly, which increases event file during training. For the training routine, on the other hand, developers explicitly specify an empty list of summaries when training is done on TPU. Which means that event file is created once and is never used afterwards. This behaviour can be different when training is performed on non-TPU hardware or summarize_gradients option is enabled for training.


Question:
Tensorflow error "shape Tensorshape() must have rank 1"
```
import tensorflow as tf
import numpy as np
import os
from PIL import Image
cur_dir = os.getcwd()

def modify_image(image):
  #resized = tf.image.resize_images(image, 180, 180, 3)
   image.set_shape([32,32,3])
   flipped_images = tf.image.flip_up_down(image)
   return flipped_images

def read_image(filename_queue):
  reader = tf.WholeFileReader()
  key,value = reader.read(filename_queue)
  image = tf.image.decode_jpeg(value)
  return key,image

def inputs():
 filenames = ['standard_1.jpg', 'standard_2.jpg' ]
 filename_queue = tf.train.string_input_producer(filenames)
 filename,read_input = read_image(filename_queue)
 reshaped_image = modify_image(read_input)
 reshaped_image = tf.cast(reshaped_image, tf.float32)
 label=tf.constant([1])
 return reshaped_image,label

def weight_variable(shape):
 initial = tf.truncated_normal(shape, stddev=0.1)
 return tf.Variable(initial)

def bias_variable(shape):
 initial = tf.constant(0.1, shape=shape)
 return tf.Variable(initial)

def conv2d(x, W):
 return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')

def max_pool_2x2(x):
 return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],
                    strides=[1, 2, 2, 1], padding='SAME')

x = tf.placeholder(tf.float32, shape=[None,32,32,3])
y_ = tf.placeholder(tf.float32, shape=[None, 1])
image,label=inputs()
image=tf.reshape(image,[-1,32,32,3])
label=tf.reshape(label,[-1,1])
image_batch=tf.train.batch([image],batch_size=2)
label_batch=tf.train.batch([label],batch_size=2)

W_conv1 = weight_variable([5, 5, 3, 32])
b_conv1 = bias_variable([32])

image_4d=x_image = tf.reshape(image, [-1,32,32,3])

h_conv1 = tf.nn.relu(conv2d(image_4d, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)

W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])

h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
h_pool2 = max_pool_2x2(h_conv2)

W_fc1 = weight_variable([8 * 8 * 64, 1024])
b_fc1 = bias_variable([1024])
h_pool2_flat = tf.reshape(h_pool2, [-1, 8*8*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)
keep_prob = tf.placeholder(tf.float32)
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)

W_fc2 = weight_variable([1024, 2])
b_fc2 = bias_variable([2])

y_conv=tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)
cross_entropy= -tf.reduce_sum(tf.cast(image_batch[1],tf.float32)*tf.log(y_conv))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

for i in range(20000):
 sess.run(train_step,feed_dict={x:image_batch[0:1],y_:label_batch[0:1]})
```
I am trying to run the tensorflow convolutional model over my own images of dimension of[32x32x3] images. The images are being read properly and are being assigned to the placeholder, during training. The problem arises during running the train_step op. When I execute the graph I get the following error.
```
TensorShape([Dimension(2), Dimension(1), Dimension(32), Dimension(32), Dimension(3)]) must have rank 1
```
But when I see the example here, the images are in the form of [batch_size,height,width,depth] tensors only. And this example works fine. Am I missing anything?

Response:
I think the error is coming from this line:
```
cross_entropy= -tf.reduce_sum(tf.cast(image_batch[1],tf.float32)*tf.log(y_conv))
```
image_batch is a 5-D tensor, with shape [2, 1, 32, 32, 3], where 2 is the batch_size argument to tf.train.batch(), and the 1 was added by the earlier image = tf.reshape(image, [-1, 32, 32, 3]). (N.B. This reshape is unnecessary, since tf.train.batch() already adds a batch dimension, and you end up having to undo the effect of the reshape when you later construct image_4d). In TensorFlow, the slicing operation (i.e. image_batch[1]) is slightly less flexible than in NumPy. The number of dimensions specified in the slice must be equal to the rank of the tensor: i.e. you must specify all five dimensions for this to work. You could specify image_batch[1, :, :, :, :] to get a 4-D slice of image_batch. I noticed a few other issues in your program, however: The cross_entropy calculation seems strange. Typically this uses a predicted label and compares it to the known label, as opposed to the image data. The feed on the training step seems to have no effect, because placeholders x and y_ are unused in your program. Furthermore, you appear to be feeding a tf.Tensor (indeed, an illegal slice of image_batch), so this will fail when you execute that statement. If you intend to use feeding, you should feed in NumPy arrays holding the input data. If you're not using feeding—i.e. using the tf.WholeFileReader as shown in your program—you will need to call tf.train.start_queue_runners() to get started. Otherwise your program will hang, waiting for input.


Question: TensorFlow - tf.layers vs tf.contrib.layers In TensorFlow, tf.layers and tf.contrib.layers share a lot of functionality (standard 2D convolutional layers, batch normalization layers, etc). Is the difference between these two just that the contrib.layers package is still experimental where the layers package is considered stable? Or is one being replaced by the other? Other differences? Why are these two separate?

Response: You've answered your own question. The description on the official documentation for the tf.contrib namespace is: contrib module containing volatile or experimental code. So tf.contrib is reserved for experimental features. APIs in this namespace are allowed to change rapidly between versions, whereas the others usually can't without a new major version. In particular, the functions in tf.contrib.layers are not identical to those found in tf.layers, although some of them might be replicated with different names. As for whether you should use them, that depends on whether you are willing to handle sudden breaking changes. Code that doesn't rely on tf.contrib may be easier to migrate to future versions of TensorFlow.


Question: In Tensorflow, what is the difference between a Variable and a Tensor? The Tensorflow documentation states that a Variable can be used any place a Tensor can be used, and they seem to be fairly interchangeable. For example, if v is a Variable, then x = 1.0 + v becomes a Tensor. What is the difference between the two, and when would I use one over the other?

Response: It's true that a Variable can be used any place a Tensor can, but the key differences between the two are that a Variable maintains its state across multiple calls to run() and a variable's value can be updated by backpropagation (it can also be saved, restored etc as per the documentation). These differences mean that you should think of a variable as representing your model's trainable parameters (for example, the weights and biases of a neural network), while you can think of a Tensor as representing the data being fed into your model and the intermediate representations of that data as it passes through your model.


Question:
How can I add an optional input to a graph in TensorFlow? I basically want to have the option to feed input to the middle of the graph and compute the output going from there. One idea I had is to use tf.placeholder_with_default that defaults to a zero tensor. Then I could mix the optional inputs using addition, however addition on a large shape this seems to be a lot of unnecessary computation. Are there better ways of accomplishing that?
```py
input_enabled = tf.placeholder_with_default(tf.constant(1.), [1])

input_shape = [None, in_size]
input = tf.placeholder_with_default(tf.zeros(input_shape), input_shape)
// ...
bottleneck_shape = [None, bottleneck_size]
bottleneck = input_enabled * f(prev_layer) + tf.placeholder_with_default(tf.zeros(bottleneck_shape), bottleneck_shape)
// ...

// Using graph with input at first layer:
sess.run([output], feed_dict={input: x})

// Using graph with input at bottleneck layer:
sess.run([output], feed_dict={bottleneck: b, input_enabled: 0.})
```

Response:
I understand better thanks to your code. Basically the schema is:
```
input       <- you can feed here
         |
     (encoder)
         |
     bottleneck    <- you can also feed here instead
         |
     (decoder)
         |
       output
```
You want two use cases: train: feed an image into input, compute the output test: feed a code into the bottleneck, compute the output You don't need to create a placeholder for bottleneck, because sess.run() allows you to feed values to non placeholders in the Graph:
```py
input_shape = [None, in_size]
input = tf.placeholder(tf.float32, input_shape)
# ...

bottleneck = f(prev_layer)  # of shape [None, bottleneck_size]
# ...

# Using graph with input at first layer:
sess.run([output], feed_dict={input: x})

# Using graph with input at bottleneck layer:
sess.run([output], feed_dict={bottleneck: b})
```
From the documentation of sess.run(): The optional feed_dict argument allows the caller to override the value of tensors in the graph. Each key in feed_dict can be one of the following types: If the key is a Tensor, the value may be a Python scalar, string, list, or numpy ndarray that can be converted to the same dtype as that tensor. Additionally, if the key is a placeholder, the shape of the value will be checked for compatibility with the placeholder.


Question:
Simple Keras Network in GradientTape: LookupError: No gradient defined for operation 'IteratorGetNext' (op type: IteratorGetNext) I've built a very simple TensorFlow Keras model with a single dense layer. It works perfectly fine outside a GradientTape block, but inside a GradientTape block it raises LookupError: No gradient defined for operation 'IteratorGetNext' (op type: IteratorGetNext) Code to reproduce:
```py
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
import tensorflow as tf
import numpy as np

print(tf.__version__)

model = Sequential()
model.add(Dense(1, input_shape=(16,)))

fake_data = np.random.random((1, 16))

print(model.predict(fake_data).shape) # works
with tf.GradientTape() as tape:
    print(model.predict(fake_data).shape) # LookupError: No gradient defined for operation 'IteratorGetNext' (op type: IteratorGetNext)
```
This appears to work in TensorFlow 2.0.0, however it fails in TensorFlow 2.1.0 and 2.2.0 Here is a notebook that replicates the issue.

Response:
try to redefine the predict operation in GradientTape in this way
```
with tf.GradientTape() as tape:
    print(model(fake_data).shape)
```


Question: Tensorflow dense_to_sparse [duplicate] This question already has answers here: Sparse Tensor (matrix) from a dense Tensor Tensorflow (5 answers) Closed 5 years ago. I am trying to convert a uncompressed sparse array into a format accepted by tf.SparseTensor. There is an inbuilt function tf.sparse_to_dense that does exactly the opposite I am trying to do. So my question is there any inbuilt function in Tensorflow or Python to do this conversion?

Response:
according to this question: you can do it with this: You can use tf.where and tf.gather_nd to do that:
```
a = np.reshape(np.arange(24), (3, 4, 2))
with tf.Session() as sess:
    a_t = tf.constant(a)
    idx = tf.where(tf.not_equal(a_t, 0))
    # Use tf.shape(a_t, out_type=tf.int64) instead of a_t.get_shape() if tensor shape is dynamic
    sparse = tf.SparseTensor(idx, tf.gather_nd(a_t, idx), a_t.get_shape())
    dense = tf.sparse_tensor_to_dense(sparse)
    b = sess.run(dense)
np.all(a == b)

```python
True
#Output
#```
```


Question:
In Tensorflow, how can I rename a certain operation name? I think this question may sound weird. Let's say, there is a tensor a.
```
x = tf.placeholder(tf.float32, [None, 2400], name="x")
y = tf.placeholder(tf.float32, [None, 2400], name="y")
a = tf.add(x, y, name="a")
```
Is there an efficient way to refer a with a different name, like out? I am thinking a dummy operation like
```
b = tf.add(a, 0, name="out")
```
I am asking this because I am trying different network architectures, and regardless of architectures, I would like to access the output tensor with a consistent name like
```
tf.get_default_graph().get_tensor_by_name("out:0")`
```
For now, output tensors are depending on architectures like fc1/fc1wb:0 or fc2/fc2wb:0. How can I wrap the final op with a certain name?

Response:
This answer suggests that tf.Graph is append only, you cannot modify it. That being said, a slightly better way to rename a tensor is to use tf.identity like this:
```
tf.identity(a, name="out")
```
EDIT: After figuring out answer to one of my own questions, I can see a slightly better way to do this:
```
graph_def.node[<index of the op you want to change>].name = "out"
```
the index is usually -1 because final output tensors are in the end of the graph_def