ComputerAssignments/CA10_knowledge_graphs/docs/GNN_NOTEBOOK_GUIDE.md

Graph Neural Networks Notebook Guide

📚 Notebook: 02_Graph_Neural_Networks.ipynb

Overview

This comprehensive notebook explores Graph Neural Networks (GNNs) applied to Knowledge Graphs, demonstrating cutting-edge architectures for multi-relational reasoning.

What's Included

1. Introduction and Setup ✅

• Import all necessary libraries (PyTorch, PyTorch Geometric, etc.)
• Configure device (CPU/GPU)
• Set random seeds for reproducibility
• Import custom KG modules

2. Knowledge Graph Creation ✅

• Create a sample knowledge graph with entities and relations
• Convert to PyTorch Geometric Data format
• Display graph statistics
• Visualize graph structure

3. GCN: Graph Convolutional Networks ✅

Theory: GCN aggregates information from neighbors using layer-wise propagation

Formula:

h_v^{(l+1)} = σ(Σ_{u ∈ N(v)} 1/√(|N(u)||N(v)|) W^{(l)} h_u^{(l)})

Implementation:

• Initialize GCN with 2 layers
• Forward pass through the network
• Output entity embeddings
• Analyze embedding statistics

Strengths:

• ✅ Simple and efficient
• ✅ Captures multi-hop neighborhood
• ✅ Fast training

Use Cases: Homogeneous graphs, citation networks, social networks

---

4. GAT: Graph Attention Networks ✅

Theory: GAT learns attention weights to determine neighbor importance

Formula:

α_vu = exp(LeakyReLU(a^T [W h_v || W h_u])) / Σ_k exp(...)
h_v^{(l+1)} = σ(Σ_{u ∈ N(v)} α_vu W^{(l)} h_u^{(l)})

Implementation:

• Multi-head attention mechanism (4 heads)
• Attention weight visualization
• Distribution analysis

Features:

• ✅ Learns edge importance
• ✅ Multi-head attention for robustness
• ✅ Interpretable through attention weights
• ✅ Better handles varying neighbor importance

Visualization:

• Histogram of attention weights
• Top-k important edges
• Attention distribution per layer

Use Cases: Heterogeneous graphs, recommendation systems, molecular graphs

---

5. GraphSAGE ✅

Theory: Sample and aggregate features from node neighborhoods

Formula:

h_v^{(l+1)} = σ(W^{(l)} · CONCAT(h_v^{(l)}, AGG({h_u^{(l)}, ∀u ∈ N(v)})))

Aggregation Functions:

• Mean: Average neighbor features
• Max: Element-wise maximum
• LSTM: Sequential aggregation

Implementation:

• Mean aggregator
• 2-layer architecture
• Batch normalization

Strengths:

• ✅ Scalable to large graphs through sampling
• ✅ Inductive learning (generalizes to unseen nodes)
• ✅ Multiple aggregation strategies
• ✅ Efficient mini-batch training

Use Cases: Large-scale graphs, Pinterest recommendations, protein interactions

---

6. R-GCN: Relational Graph Convolutional Networks ✅

Theory: Specifically designed for multi-relational graphs like Knowledge Graphs

Formula:

h_v^{(l+1)} = σ(Σ_{r∈R} Σ_{u∈N_v^r} 1/|N_v^r| W_r^{(l)} h_u^{(l)} + W_0^{(l)} h_v^{(l)})

Basis Decomposition:

W_r = Σ_{b=1}^B a_rb V_b

Reduces parameters from O(num_relations × d²) to O(num_bases × d²)

Implementation:

• Relation-specific weight matrices
• Basis decomposition for parameter efficiency
• Handles edge types explicitly

Key Features:

• ✅ Handles multiple relation types
• ✅ Basis decomposition for scalability
• ✅ Perfect for Knowledge Graphs
• ✅ Captures relational patterns

Analysis:

• Relation-specific statistics
• Edge type distribution
• Embedding quality per relation

Use Cases: Knowledge graphs, biomedical networks, semantic web

---

7. Training Pipeline ✅

Data Preparation:

• Create positive triples from KG
• Generate negative samples (corrupted head/tail)
• 80/20 train/validation split

Training Process:

• Binary classification: real vs. fake triples
• Loss: BCEWithLogitsLoss
• Optimizer: Adam with weight decay
• Metrics: Loss, Accuracy

Hyperparameters:

• Learning rate: 0.01
• Epochs: 50
• Embedding dim: 64
• Hidden dim: 128
• Dropout: 0.3

---

8. Model Comparison ✅

Comprehensive comparison of all 4 GNN architectures:

Model	Parameters	Embedding Dim	Best For	Complexity
GCN	~50K	64	Simple graphs	O(E × d²)
GAT	~75K	64	Varying importance	O(E × d² × heads)
GraphSAGE	~55K	64	Large graphs	O(S × d²)
R-GCN	~60K	64	Multi-relational	O(R × d²) or O(B × d²)

Performance Metrics:

• Training loss curves
• Validation accuracy
• Convergence speed
• Memory usage

---

9. Embedding Visualization ✅

t-SNE Visualization:

• Reduce embeddings from 64D → 2D
• Color by entity type
• Compare all 4 models side-by-side

Visualization Grid:

┌─────────────┬─────────────┐
│  GCN        │  GAT        │
│  Embeddings │  Embeddings │
├─────────────┼─────────────┤
│  GraphSAGE  │  R-GCN      │
│  Embeddings │  Embeddings │
└─────────────┴─────────────┘

Analysis:

• Cluster quality
• Separation of entity types
• Embedding space structure

---

10. Summary and Conclusions ✅

Key Takeaways:

1. GCN: Best for simple, fast prototyping
2. GAT: Best when interpretability matters
3. GraphSAGE: Best for large-scale graphs
4. R-GCN: Best for multi-relational KGs

Decision Matrix:

Requirement	Recommended Model
Speed	GCN
Interpretability	GAT
Scale	GraphSAGE
Multi-relational	R-GCN
Temporal data	Temporal GNN
Inductive learning	GraphSAGE

Next Steps:

• 🔧 Hyperparameter tuning
• 📊 More advanced metrics (Hits@K, MRR)
• 🌐 Real-world datasets (FB15k-237, WN18RR)
• ⏰ Temporal extensions
• 🔗 Integration with LLMs

---

📖 Theory Deep Dive

Message Passing Framework

All GNNs follow the message passing framework:

1. Message: Compute messages from neighbors

   m_v = Σ_{u∈N(v)} MSG(h_u, h_v, e_uv)
   

2. Aggregate: Combine messages

   m_v_agg = AGG({m_u : u∈N(v)})
   

3. Update: Update node representation

   h_v_new = UPDATE(h_v, m_v_agg)
   

Over-smoothing Problem

Issue: Deep GNNs can make all node embeddings similar

Solutions:

• Skip connections (ResGNN)
• Layer normalization
• Dropout
• Limit number of layers (typically 2-3)

Relation-Aware Models

Challenge: Traditional GNNs treat all edges equally

R-GCN Solution:

• Separate weights per relation type
• Basis decomposition for efficiency
• Self-loops for self-information

---

🚀 Running the Notebook

Prerequisites

pip install torch torch-geometric scikit-learn matplotlib seaborn tqdm numpy

Execution

cd /path/to/CA10_knowledge_graphs/notebooks
jupyter notebook 02_Graph_Neural_Networks.ipynb

Or execute all cells:

jupyter nbconvert --to notebook --execute 02_Graph_Neural_Networks.ipynb \\
  --output 02_Graph_Neural_Networks_executed.ipynb

---

📊 Expected Outputs

Cell Outputs Include:

1. ✅ Device configuration (CPU/GPU)
2. ✅ Knowledge graph statistics
3. ✅ Model architectures
4. ✅ Parameter counts
5. ✅ Forward pass results
6. ✅ Attention weight analysis (GAT)
7. ✅ Training progress
8. ✅ Loss curves
9. ✅ Embedding visualizations
10. ✅ Performance comparison

Visualizations:

• 📈 Training/validation loss curves
• 🎨 t-SNE embedding plots
• 📊 Attention weight distributions
• 🔗 Knowledge graph structure
• 📉 Model comparison charts

---

🔬 Research Applications

1. Drug Discovery

• Task: Predict drug-disease interactions
• Model: R-GCN for multi-relational biomedical KG
• Relations: "treats", "causes", "interacts_with"

2. Recommendation Systems

• Task: Recommend products based on user behavior
• Model: GraphSAGE for large-scale graphs
• Nodes: Users, Products, Categories

3. Question Answering

• Task: Multi-hop reasoning over knowledge
• Model: R-GCN + Attention
• Example: "Who discovered the theory developed by Einstein?"

4. Link Prediction

• Task: Predict missing links in KG
• Model: All GNN models + scoring function
• Metrics: Hits@K, MRR, Mean Rank

---

📚 References

Key Papers:

1. GCN: Kipf & Welling, "Semi-Supervised Classification with GCNs" (ICLR 2017)
2. GAT: Veličković et al., "Graph Attention Networks" (ICLR 2018)
3. GraphSAGE: Hamilton et al., "Inductive Representation Learning" (NIPS 2017)
4. R-GCN: Schlichtkrull et al., "Modeling Relational Data with GCNs" (ESWC 2018)

Additional Resources:

• PyTorch Geometric Documentation
• Stanford CS224W: Machine Learning with Graphs
• Distill.pub: "A Gentle Introduction to GNNs"

---

🎓 Learning Outcomes

After completing this notebook, you will:

• ✅ Understand GNN fundamentals
• ✅ Implement 4 different GNN architectures
• ✅ Apply GNNs to Knowledge Graphs
• ✅ Compare model performance
• ✅ Visualize learned embeddings
• ✅ Choose appropriate models for tasks

---

🛠️ Troubleshooting

Common Issues:

1. Import Errors

pip install torch-scatter torch-sparse torch-geometric -f https://data.pyg.org/whl/torch-<version>.html

2. CUDA Out of Memory

• Reduce batch size
• Reduce embedding dimension
• Use CPU instead

3. Slow Training

• Use GPU if available
• Reduce number of layers
• Use GraphSAGE with sampling

---

📝 License

Part of the System2_in_AI course materials (CA10)

---

Happy Learning! 🚀

For questions or issues, refer to the main CA10 README or project documentation.