scdynomics.FT_Masked_Tuner
- class scdynomics.FT_Masked_Tuner(*args: Any, **kwargs: Any)
Bases:
Masked_PretrainerFine-tuning module utilizing masked reconstruction with Parameter-Efficient Fine-Tuning (PEFT).
Replicates the masked reconstruction objectives of the Masked_Pretrainer, but applies LoRA or Adapter parameters instead of altering the full base model. Useful for adapting representation foundational knowledge to a specific target dataset.
- __init__(input_drop_prob: float = 0.0, base_model=None, lora_type: str = None, shortcut_type: str = 'input', pre_block_num: int = None, lora_latent_dim: int = 512, lora_dropout: float = 0.0, lora_bias: bool = False, lora_activation: str = 'relu', pool_method: str = None, pool_attn_hidden: int = 32, pool_out_size: int = 1, loss_reduction: str = 'mean', weight_decay: float = 0.001, betas=(0.9, 0.999), learning_rate: float = 0.0001, sch_warmpup_epochs: int = None, sch_warmpup_factor: float = 1, sch_T_0: int = 20, sch_T_mult: int = 2, sch_eta_min: float = 2e-06, **kwargs)
- Parameters:
- input_drop_prob:
float(default:0.0) Probability of applying the input dropout mask.
- base_model:
Masked_Pretrainer(default:None) The pretrained model serving as the foundation.
- lora_type:
str(default:None) Method used to fine-tune (e.g., “lora”, “adapter”).
- shortcut_type:
str(default:"input") Architecture connection setup.
- pre_block_num:
int(default:None) Number of blocks to skip or process before PEFT.
- lora_latent_dim:
int(default:512) The rank dimension configured for the chosen PEFT adapter.
- lora_dropout:
float(default:0.0) Dropout probability for the LoRA layers.
- lora_bias:
bool(default:False) Whether to train biases in the LoRA adapter.
- lora_activation:
str(default:'relu') The activation function passed to the PEFT modules.
- pool_method:
str(default:None) Pooling technique applied (“avg”, “max”, “attn”).
- pool_attn_hidden:
int(default:32) Hidden dimension size for attention pooling.
- pool_out_size:
int(default:1) Output size for the pooling layer.
- loss_reduction:
str(default:"mean") Type of loss reduction (“mean”, “sum”).
- weight_decay:
float(default:0.001) Weight decay (L2 penalty) for the optimizer.
- betas:
tuple(default:(0.9, 0.999)) Beta coefficients for AdamW.
- learning_rate:
float(default:1e-4) Optimizer learning rate.
- sch_warmpup_epochs:
int(default:None) Number of epochs to linearly warm up the learning rate.
- sch_warmpup_factor:
float(default:1.0) Multiplier for the learning rate warmup.
- sch_T_0:
int(default:20) Number of iterations for the first restart in CosineAnnealingWarmRestarts.
- sch_T_mult:
int(default:2) A factor increases Ti after a restart in CosineAnnealingWarmRestarts.
- sch_eta_min:
float(default:2e-6) Minimum learning rate boundary for the scheduler.
- kwargs:
dict Additional scheduling and optimization arguments.
- input_drop_prob: