Add private RenderFormer V2 component bundle

README.md

@@ -0,0 +1,106 @@
+---
+license: mit
+library_name: renderformer-studio
+tags:
+- neural-rendering
+- rendering
+- transformers
+- materials
+---
+
+# RenderFormer V2
+
+> **Status:** This is a private pre-release bundle. Do not redistribute it or
+> depend on its layout until the release is approved.
+
+This repository bundles the two RenderFormer V2 renderer checkpoints and the
+material-latent models used by the data pipeline. Source code, installation
+instructions, scene conversion, and inference examples live in
+[RenderFormer Studio](https://github.com/iamNCJ/RenderFormer-Studio).
+
+## Components
+
+| Path | Class | Resolution | Purpose |
+| --- | --- | ---: | --- |
+| `transformer_512` | `RenderFormerModel` | 512 | Native-512 V2 renderer |
+| `transformer_2048` | `RenderFormerModel` | 2048 | Native-2048 V2 renderer |
+| `material_autoencoder` | `MaterialAutoencoder` | 256 | Material image to/from 9-D latent |
+| `diffspec_mapper` | `DiffuseSpecularToLatent` | — | Diffuse/specular BRDF to 9-D latent |
+| `metallic_mapper` | `PrincipledBRDFToLatent` | — | Metallic/roughness BRDF to 9-D latent |
+| `metallic_transmission_mapper` | `PrincipledBRDFToLatentWithTransmission` | — | Metallic/transmission BRDF to 9-D latent |
+
+The renderer pipeline loads the Qwen-Image VAE configured by each transformer
+for environment and raw-material encoding. Those auxiliary weights are an
+external dependency and are not duplicated in this bundle.
+
+## Load a renderer
+
+The current package accepts a local component directory. Download only the
+resolution you need, then construct the shared V2 pipeline:
+
+```python
+from pathlib import Path
+
+import torch
+from huggingface_hub import snapshot_download
+from renderformer import RenderFormerV2Pipeline
+
+snapshot = Path(snapshot_download(
+    "RenderFormer/renderformer-v2",
+    allow_patterns=["transformer_512/*"],
+))
+pipeline = RenderFormerV2Pipeline.from_pretrained(
+    snapshot / "transformer_512",
+    device="cuda",
+    torch_dtype=torch.float16,
+)
+```
+
+Use `transformer_2048` in both places for native-2048 inference. The CLI and
+pipeline documentation in RenderFormer Studio describe H5 preparation,
+precision, view batching, and tone mapping.
+
+## Load material components
+
+```python
+from pathlib import Path
+
+from huggingface_hub import snapshot_download
+from renderformer.models.material import (
+    DiffuseSpecularToLatent,
+    MaterialAutoencoder,
+    PrincipledBRDFToLatent,
+    PrincipledBRDFToLatentWithTransmission,
+)
+
+snapshot = Path(snapshot_download(
+    "RenderFormer/renderformer-v2",
+    allow_patterns=[
+        "material_autoencoder/*",
+        "diffspec_mapper/*",
+        "metallic_mapper/*",
+        "metallic_transmission_mapper/*",
+    ],
+))
+material_autoencoder = MaterialAutoencoder.from_pretrained(
+    snapshot / "material_autoencoder", strict=True
+).eval()
+diffspec_mapper = DiffuseSpecularToLatent.from_pretrained(
+    snapshot / "diffspec_mapper", strict=True
+).eval()
+metallic_mapper = PrincipledBRDFToLatent.from_pretrained(
+    snapshot / "metallic_mapper", strict=True
+).eval()
+transmission_mapper = PrincipledBRDFToLatentWithTransmission.from_pretrained(
+    snapshot / "metallic_transmission_mapper", strict=True
+).eval()
+```
+
+The autoencoder's `preprocessor_config.json` records the physical-input
+contract. Prefer `encode_hdr` and `decode_hdr` for finite, nonnegative,
+alpha-premultiplied linear RGB.
+
+## Citation
+
+Please cite both RenderFormer papers when using this bundle. Full BibTeX
+entries are provided in the RenderFormer Studio README.

diffspec_mapper/config.json

@@ -0,0 +1,5 @@
+{
+  "hidden_dim": 128,
+  "latent_dim": 9,
+  "num_layers": 3
+}

diffspec_mapper/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a5f05ce15a1af97769ccb830df70584156e7215f48a3feba1454e9b445fc5a5
+size 75252

material_autoencoder/config.json

@@ -0,0 +1,5 @@
+{
+  "image_channels": 3,
+  "image_size": 256,
+  "latent_dim": 9
+}

material_autoencoder/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:859fd025f02c7a30d430fb19551f8f1c4fed550f37553b175e8205ef9cd09e2d
+size 144083432

material_autoencoder/preprocessor_config.json

@@ -0,0 +1,3 @@
+{
+  "mode": "log10_1p"
+}

metallic_mapper/config.json

@@ -0,0 +1,5 @@
+{
+  "hidden_dim": 128,
+  "latent_dim": 9,
+  "num_layers": 3
+}

metallic_mapper/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:52a3146256d3255657e53682ab9f8e2acd1f9f300432f08f920570af5b4bf006
+size 74228

metallic_transmission_mapper/config.json

@@ -0,0 +1,5 @@
+{
+  "hidden_dim": 256,
+  "latent_dim": 9,
+  "num_layers": 5
+}

metallic_transmission_mapper/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fa4f5f54ad1d00f5109e558ea489d89841a4f0c3181088346b6dec2e668e6853
+size 807740

model_index.json

@@ -0,0 +1,43 @@
+{
+  "schema_version": 1,
+  "components": [
+    {
+      "name": "transformer_512",
+      "class": "RenderFormerModel",
+      "pipeline_class": "RenderFormerV2Pipeline",
+      "path": "transformer_512",
+      "resolution": 512
+    },
+    {
+      "name": "transformer_2048",
+      "class": "RenderFormerModel",
+      "pipeline_class": "RenderFormerV2Pipeline",
+      "path": "transformer_2048",
+      "resolution": 2048
+    },
+    {
+      "name": "material_autoencoder",
+      "class": "MaterialAutoencoder",
+      "path": "material_autoencoder",
+      "resolution": 256
+    },
+    {
+      "name": "diffspec_mapper",
+      "class": "DiffuseSpecularToLatent",
+      "path": "diffspec_mapper",
+      "resolution": null
+    },
+    {
+      "name": "metallic_mapper",
+      "class": "PrincipledBRDFToLatent",
+      "path": "metallic_mapper",
+      "resolution": null
+    },
+    {
+      "name": "metallic_transmission_mapper",
+      "class": "PrincipledBRDFToLatentWithTransmission",
+      "path": "metallic_transmission_mapper",
+      "resolution": null
+    }
+  ]
+}

transformer_2048/config.yaml

@@ -0,0 +1,96 @@
+model_config:
+  latent_dim: 768
+  num_layers: 12
+  num_heads: 6
+  num_kv_heads: null
+  dim_feedforward: 3072
+  num_register_tokens: 16
+  register_token_pe: weighted_mean
+  use_view_token: false
+  dropout: 0.0
+  activation: swiglu
+  norm_type: rms_norm
+  norm_first: true
+  view_indep_qk_norm: true
+  qk_norm: true
+  bias: false
+  pe_type: rope_only_center
+  rope_type: triangle_center
+  rope_double_max_freq: false
+  vertex_pe_num_freqs: 12
+  intra_rope_pe_num_freqs: 40
+  use_vn_encoder: true
+  vn_pe_num_freqs: 6
+  vn_encoder_norm_type: rms_norm
+  texture_encode_patch_size: 4
+  texture_channels: 80
+  texture_encoder_norm_type: rms_norm
+  log_roughness_encoding: false
+  separate_light_strength: true
+  auto_interpolate_texture: true
+  use_env_lighting: true
+  envmap_vae_model_id: Qwen/Qwen-Image
+  envmap_latent_patch_size: 8
+  envmap_latent_channels: 16
+  envmap_ray_dir_map_patch_size: 64
+  envmap_encoder_norm_type: rms_norm
+  use_triangle_ordering: true
+  triangle_ordering_order: hilbert
+  view_transformer_use_triangle_ordering: false
+  view_transformer_triangle_ordering_order: z
+  use_local_attention: false
+  local_attention_window_size_half: 256
+  use_flex_attention: true
+  flex_attention_block_size: 128
+  put_light_tokens_first: true
+  put_light_tokens_in_sink: true
+  put_env_tokens_in_sink: true
+  add_summary_tokens: true
+  summary_block_size: 64
+  use_volumes: true
+  volume_density_input_dim: 64
+  volume_use_density_mlp: true
+  use_perceiver: false
+  num_perceiver_tokens: 64
+  decoding_method: view_transformer
+  texture_decode_patch_size: 32
+  view_transformer_latent_dim: 768
+  view_transformer_ffn_hidden_dim: 3072
+  view_transformer_n_heads: 6
+  view_transformer_num_kv_heads: null
+  view_transformer_n_layers: 6
+  view_transformer_include_self_attn: true
+  view_transformer_self_attn_before_cross: false
+  view_transformer_use_swin_attn: true
+  vdir_pe_type: nerf
+  vdir_num_freqs: 0
+  patch_size: 8
+  output_patch_size: null
+  output_channel: null
+  latent_vae_model_id: null
+  include_alpha: false
+  freeze_radiosity_transformer_if_initialized: false
+  use_dpt_decoder: true
+  dpt_features: 128
+  dpt_out_channels:
+  - 96
+  - 192
+  - 384
+  - 768
+  dpt_out_layers: null
+  view_transformer_use_layers: null
+  custom_init: true
+  deepstack_injection_map: null
+  deepstack_dual_input_size: 64
+  deepstack_decoder_injection_map: null
+dataset_config:
+  img_key: img
+  volume_padding_length: 1024
+trainer_config:
+  img_resolution: 2048
+  turn_to_cam_coord: false
+  learn_ldr: false
+  emission_area_encoding: false
+  clamped_hdr: false
+  clamped_hdr_max: 9.0
+  override_view_tf32_mode: true

transformer_2048/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de25aa4d3df4dc09881c67446046221d3c0a9a5c942295953d140dbec9a7fd72
+size 830351052

transformer_512/config.yaml

@@ -0,0 +1,96 @@
+model_config:
+  latent_dim: 768
+  num_layers: 12
+  num_heads: 6
+  num_kv_heads: null
+  dim_feedforward: 3072
+  num_register_tokens: 16
+  register_token_pe: weighted_mean
+  use_view_token: false
+  dropout: 0.0
+  activation: swiglu
+  norm_type: rms_norm
+  norm_first: true
+  view_indep_qk_norm: true
+  qk_norm: true
+  bias: false
+  pe_type: rope_only_center
+  rope_type: triangle_center
+  rope_double_max_freq: false
+  vertex_pe_num_freqs: 12
+  intra_rope_pe_num_freqs: 40
+  use_vn_encoder: true
+  vn_pe_num_freqs: 6
+  vn_encoder_norm_type: rms_norm
+  texture_encode_patch_size: 4
+  texture_channels: 80
+  texture_encoder_norm_type: rms_norm
+  log_roughness_encoding: false
+  separate_light_strength: true
+  auto_interpolate_texture: true
+  use_env_lighting: true
+  envmap_vae_model_id: Qwen/Qwen-Image
+  envmap_latent_patch_size: 8
+  envmap_latent_channels: 16
+  envmap_ray_dir_map_patch_size: 64
+  envmap_encoder_norm_type: rms_norm
+  use_triangle_ordering: true
+  triangle_ordering_order: hilbert
+  view_transformer_use_triangle_ordering: false
+  view_transformer_triangle_ordering_order: z
+  use_local_attention: false
+  local_attention_window_size_half: 256
+  use_flex_attention: true
+  flex_attention_block_size: 128
+  put_light_tokens_first: true
+  put_light_tokens_in_sink: true
+  put_env_tokens_in_sink: true
+  add_summary_tokens: true
+  summary_block_size: 64
+  use_volumes: true
+  volume_density_input_dim: 64
+  volume_use_density_mlp: true
+  use_perceiver: false
+  num_perceiver_tokens: 64
+  decoding_method: view_transformer
+  texture_decode_patch_size: 32
+  view_transformer_latent_dim: 768
+  view_transformer_ffn_hidden_dim: 3072
+  view_transformer_n_heads: 6
+  view_transformer_num_kv_heads: null
+  view_transformer_n_layers: 6
+  view_transformer_include_self_attn: true
+  view_transformer_self_attn_before_cross: false
+  view_transformer_use_swin_attn: true
+  vdir_pe_type: nerf
+  vdir_num_freqs: 0
+  patch_size: 8
+  output_patch_size: null
+  output_channel: null
+  latent_vae_model_id: null
+  include_alpha: false
+  freeze_radiosity_transformer_if_initialized: false
+  use_dpt_decoder: true
+  dpt_features: 128
+  dpt_out_channels:
+  - 96
+  - 192
+  - 384
+  - 768
+  dpt_out_layers: null
+  view_transformer_use_layers: null
+  custom_init: true
+  deepstack_injection_map: null
+  deepstack_dual_input_size: 64
+  deepstack_decoder_injection_map: null
+dataset_config:
+  img_key: img
+  volume_padding_length: 1024
+trainer_config:
+  img_resolution: 512
+  turn_to_cam_coord: false
+  learn_ldr: false
+  emission_area_encoding: false
+  clamped_hdr: false
+  clamped_hdr_max: 9.0
+  override_view_tf32_mode: true

transformer_512/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1300bb526e7b1c5ec45081a1d48f274132d32a0189348242c3f0a920ece7f3f9
+size 830351052