Oumuamua-7b-base

This is a merge of pre-trained language models created using mergekit.

Output example

Input text

ๆ—ฅๆœฌใงๆœ€ใ‚‚้ซ˜ใ„ๅฑฑใฎๅๅ‰ใฏ

Output text

ๆ—ฅๆœฌใงๆœ€ใ‚‚้ซ˜ใ„ๅฑฑใฎๅๅ‰ใฏใ€ๅฏŒๅฃซๅฑฑใ€‚
ใใฎๆจ™้ซ˜ใฏ3776ใƒกใƒผใƒˆใƒซใ€‚
ไธ–็•Œใงใ‚‚20ไฝไปฅๅ†…ใซๅ…ฅใ‚‹้ซ˜ใ•ใ‚’่ช‡ใ‚‹ใ€‚
ใใฎๅฏŒๅฃซๅฑฑใฎ้บ“ใซใ‚ใ‚‹ใฎใŒใ€้™ๅฒก็œŒๅฏŒๅฃซๅธ‚ใ€‚
ๅฏŒๅฃซๅธ‚ใฏใ€ๅฏŒๅฃซๅฑฑใฎ้บ“ใซใ‚ใ‚‹ใŸใ‚ใ€่ฆณๅ…‰ๅœฐใจใ—ใฆใ‚‚ๆœ‰ๅใงใ‚ใ‚‹ใ€‚
ๅฏŒๅฃซๅฑฑใฎ้บ“ใซใ‚ใ‚‹ใ“ใจใ‹ใ‚‰ใ€ๅฏŒๅฃซๅธ‚ใฏ่ฆณๅ…‰ๅœฐใจใ—ใฆใ‚‚ๆœ‰ๅใงใ‚ใ‚‹ใ€‚
ๅฏŒๅฃซๅฑฑใ‚’็œบใ‚ใ‚‹ใ“ใจใŒใงใใ‚‹ใ‚นใƒใƒƒใƒˆใŒๅคšใใ€็‰นใซๅฏŒๅฃซๅธ‚ใฎไธญๅฟƒ้ƒจใ‹ใ‚‰่ฆ‹ใ‚‹ๅฏŒๅฃซๅฑฑใฏใ€ใใฎ็พŽใ—ใ•ใ‹ใ‚‰ใ€Œๆ—ฅๆœฌไธ€ใฎ็œบๆœ›ใ€ใจ่จ€ใ‚ใ‚Œใฆใ„ใ‚‹ใ€‚

Test environment

This model was tested using text-generation-webui. I use preset min_p and Null preset with temperature=0.3 for Generation.

Usage

Use the base model

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "nitky/Oumuamua-7b-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
prompt = "ๆ—ฅๆœฌใงๆœ€ใ‚‚้ซ˜ใ„ๅฑฑใฎๅๅ‰ใฏ"
input_ids = tokenizer.encode(
    prompt,
    add_special_tokens=False,
    return_tensors="pt"
)
tokens = model.generate(
    input_ids.to(device=model.device),
    max_new_tokens=256,
    do_sample=True,
    temperature=0.3
)

out = tokenizer.decode(tokens[0], skip_special_tokens=True)
print(out)

Merge Details

Merge Method

This model was merged using the Model Stock merge method using tokyotech-llm/Swallow-MS-7b-v0.1 as a base.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

merge_method: task_arithmetic
base_model: mistralai/Mistral-7B-v0.1
models:
  - model: tokyotech-llm/Swallow-MS-7b-v0.1
    parameters:
      weight:
      - filter: embed_tokens
        value: 1.0
      - value: 0
dtype: bfloat16
tokenizer_source: model:tokyotech-llm/Swallow-MS-7b-v0.1
name: Mistral-7B-v0.1-VE-Swallow-MS
---
merge_method: task_arithmetic
base_model: nitky/Flavor-7b # private model
models:
  - model: tokyotech-llm/Swallow-MS-7b-v0.1
    parameters:
      weight:
      - filter: embed_tokens
        value: 1.0
      - value: 0
dtype: bfloat16
tokenizer_source: model:tokyotech-llm/Swallow-MS-7b-v0.1
name: Flavor-7b-VE-Swallow-MS
---
merge_method: task_arithmetic
base_model: stabilityai/japanese-stablelm-base-gamma-7b
models:
  - model: tokyotech-llm/Swallow-MS-7b-v0.1
    parameters:
      weight:
      - filter: embed_tokens
        value: 1.0
      - value: 0
dtype: bfloat16
tokenizer_source: model:tokyotech-llm/Swallow-MS-7b-v0.1
name: japanese-stablelm-base-gamma-7b-VE-Swallow-MS
---
merge_method: task_arithmetic
base_model: Mistral-7B-v0.1-VE-Swallow-MS
models:
  - model: tokyotech-llm/Swallow-MS-7b-v0.1
    parameters:
      weight: 1.0
  - model: Flavor-7b-VE-Swallow-MS
    parameters:
      weight: 0.5
  - model: japanese-stablelm-base-gamma-7b-VE-Swallow-MS
    parameters:
      weight: -0.5
dtype: bfloat16
name: Oumuamua-7b-base-preset
---
merge_method: model_stock
base_model: Mistral-7B-v0.1-VE-Swallow-MS
models:
  - model: tokyotech-llm/Swallow-MS-7b-v0.1
  - model: Oumuamua-7b-base-preset
dtype: bfloat16
name: Oumuamua-7b-base
Downloads last month
13
Safetensors
Model size
7B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for nitky/Oumuamua-7b-base

Paper for nitky/Oumuamua-7b-base