RWKV Integrated Bidirectional Backbone with Output Network
RIBBON1
Synopsis
RIBBON1 is a multi-label text classification architecture designed to identify 28 different emotion categories in text. Unlike single label classification, multiple categories can be predicted for a given input.
Several further processing layers are built on top of RWKV-7 (which forms the basis of the architecture). These consist of a dual pooling, a bidirectional GRU and Multi-Head Attention. Each of these processes the representations in a different way prior to them reaching the final classification layer.
The model maintains a lot of the characteristics from across the input sequence and is able to capture the context via a combination of recurrent processing and attention.
RIBBON1 uses RWKV/RWKV7-Goose-Pile-168M-HF as its backbone. It takes the initial processing of the input sequence and uses the output of the last_hidden_state to produce the token level representations. The hidden_size of the model is used to represent each token. The rest of the processing layers (which make up the rest of the architecture) are then applied to those representations. It's built on RWKV-7 with all the other layers operating on top of its output rather than in place of it.
A fairly standard PyTorch MultiheadAttention layer takes the output from the RWKV backbone. It provides an alternative way of representing the relationships between tokens across the sequence, even though the RWKV itself is already doing the contextual processing.
The configuration used by the attention layer is as follows:
- Dimension embedding:
hidden_size - There are eight heads.
- Dropout rate:
0.1
There's also a residual connection. The attention output is added to the initial hidden states prior to applying layer normalisation and dropout.
The normalised representations go into a two layer bidirectional GRU following the attention layer. That layer is able to take information from tokens both before and after a given position and incorporate that into the representation of that token by processing the sequence in both directions.
There are two levels in its architecture and in each direction there's a hidden size of hidden_size // 2. An output of the original hidden_size dimension is produced by concatenating both directions. Dropout is fixed at 0.1 between the GRU layers. The pooling then acts on the final representations.
Additive attention pooling (v_attn) is a two layer linear projection with a Tanh activation to compute an attention score for each token.
The scoring function is:
scores = W_2(tanh(W_1 * h_t))
The resulting scores are passed through a Softmax to produce attention weights. The padding positions are masked out with negative infinity prior to the Softmax. This ensures that the padding tokens don't contribute to the final weighted vector. The resulting vector is the sequence represented in terms of the relative importance of its individual tokens.
The classification head takes the combined pooling representation and maps it to the 28 emotion categories.
Its structure consists of:
Linear(2 * hidden_size, hidden_size)
GELU()
Dropout(0.1)
Linear(hidden_size, 28)
Wow, amazing, I know.
Then we have the first linear layer reducing the dimensionality of the combined representation. GELU is the activation function, followed by dropout. The final linear layer gives us 28 logits, one for each of the individual emotion categories.
Training uses BCEWithLogitsLoss (Binary Cross-Entropy with Logits). This is the appropriate loss function for multi-label classification as it's possible for multiple emotion classes to be active simultaneously.
Rather than picking one category out of the full set, each emotion is treated as an independent binary prediction. That way the model can assign multiple labels to the same input if the text contains more than one emotion that it can detect.
Now, you probably want some sort of summery, this wont be the same for ALL models but it is the base on RIBBON1:
| Component | Configuration |
|---|---|
| Backbone | RWKV-7 (168M) |
| Attention | 8-head Multi-Head Attention |
| Recurrent Layer | 2-layer Bidirectional GRU |
| GRU Dropout | 0.1 |
| Attention Pooling | Additive Attention |
| Additional Pooling | Masked Max Pooling |
| Classification Head | Linear → GELU → Dropout → Linear |
| Output Categories | 28 |
| Task | Multi-label Emotion Classification |
| Loss Function | BCEWithLogitsLoss |