{ "cells": [ { "cell_type": "markdown", "id": "08bfb77f", "metadata": {}, "source": [ "# Fine-Tuning the Foundation Model\n", "\n", "In this tutorial, we will take the pretrained model from the previous tutorial and fine-tune it for a downstream classification task.\n", "This process demonstrates how to adapt the general model to specialized classification type problems.\n", "\n", "The associated automated script for this process is available in the scDynOmics repository under `scripts/clf.ft.py`.\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": null, "id": "51392790", "metadata": {}, "outputs": [], "source": [ "import torch\n", "import scanpy as sc\n", "import anndata as ad, hdf5plugin\n", "from torch.utils.data import DataLoader\n", "from pytorch_lightning import Trainer, seed_everything\n", "import scdynomics.utils.json as JSON\n", "from scdynomics.utils.trainer import (\n", " set_accelerator,\n", " cal_accum_grad_batches,\n", " make_callbacks,\n", ")\n", "from scdynomics import FT_MLP_Classifier\n", "from scdynomics.utils.data import Multimodal_Corpus, random_split\n", "from scdynomics.ft.pipeline import set_pretrainer, set_layers_and_embed_mod\n", "from scdynomics import ft_kfold_classification, ft_clf_predict, ft_clf_explain" ] }, { "cell_type": "markdown", "id": "0c99d064", "metadata": {}, "source": [ "## 0. Configuration\n", "\n", "First, define the paths for the downstream query dataset, the pretrained foundation model checkpoint, and hyperparameters.\n", "For this tutorial, we default to using the checkpoint generated during the pretraining tutorial.\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": null, "id": "d95b3c64", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Seed set to 42\n" ] } ], "source": [ "# File paths\n", "clf_adata_path = \"../../data/sample/Tbx6WT.h5ad\" # example data included in the scDynOmics github repository relative to this notebook\n", "config_path = \"../../data/config/tiny.adapter.clf.json\" # for production, also create a modified config file, especially with higher value than 1 for trainer_hparams->max_epochs \n", "token_dict_path = \"../../data/gene_dict/token_dict.json.gz\"\n", "default_root_dir = \"logs/\"\n", "\n", "# If the PT ckpt is saved from the pretraining tutorial\n", "base_model_ckpt = \"logs/lightning_logs/version_0/checkpoints/last.ckpt\"\n", "# # For randomly initialized model, set to None\n", "# base_model_ckpt = None\n", "\n", "# Just using the GEX modality for classification\n", "mod1_layer = \"X\"\n", "mod2_layer = None\n", "obs_label = \"cell_state\"\n", "\n", "# HVG selection parameters\n", "nHVGs = 100\n", "hvg_flavor = \"seurat\"\n", "\n", "# Training parameters\n", "valid_fraction = 0.1\n", "test_fraction = 0.2\n", "stratify_test = True\n", "stratify_valid = True\n", "dataloader_worker = 10\n", "seed = 42\n", "accelerator = \"cpu\"\n", "\n", "# Apply seed\n", "seed_everything(seed)\n", "torch.set_float32_matmul_precision(\"medium\")" ] }, { "cell_type": "markdown", "id": "1d18458a", "metadata": {}, "source": [ "## 1. Summarized Pipeline Execution\n", "\n", "The entire fine-tuning process, including data splitting, base model loading, and classification, can be executed using the high-level :func:`scdynomics.ft_kfold_classification` pipeline function.\n", "This provides a unified interface for downstream adaptation.\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": null, "id": "5098fc76", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Seed set to 42\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Loaded pretrainer model\n", "Keep modalities: [0]\n", "Keep genes: 101; with 12 TFs\n", "Performing single train-test split\n", "Fold 1/1\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/usr/lib/python3.11/site-packages/lightning_fabric/plugins/environments/slurm.py:204: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python /usr/lib/python3 ...\n", "GPU available: True (cuda), used: False\n", "TPU available: False, using: 0 TPU cores\n", "HPU available: False, using: 0 HPUs\n", "/usr/lib/python3.11/site-packages/pytorch_lightning/trainer/setup.py:177: GPU available but not used. You can set it by doing `Trainer(accelerator='gpu')`.\n", "\n", " | Name | Type | Params | Mode \n", "----------------------------------------------------------\n", "0 | ft_encoder | FT_Encoder | 5.4 M | train\n", "1 | decision | Sequential | 13.6 K | train\n", "2 | criterion | CrossEntropyLoss | 0 | train\n", "3 | accuracy | MulticlassAccuracy | 0 | train\n", "4 | f1 | MulticlassF1Score | 0 | train\n", "5 | auroc | MulticlassAUROC | 0 | train\n", "----------------------------------------------------------\n", "2.2 M Trainable params\n", "3.2 M Non-trainable params\n", "5.4 M Total params\n", "21.762 Total estimated model params size (MB)\n", "294 Modules in train mode\n", "0 Modules in eval mode\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Initializing new FT model\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "`Trainer.fit` stopped: `max_epochs=1` reached.\n", "Restoring states from the checkpoint path at logs/fold_1/lightning_logs/version_0/checkpoints/epoch=0-step=34.ckpt\n", "Loaded model weights from the checkpoint at logs/fold_1/lightning_logs/version_0/checkpoints/epoch=0-step=34.ckpt\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Testing the model\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/usr/lib/python3.11/site-packages/torchmetrics/utilities/prints.py:43: UserWarning: No negative samples in targets, false positive value should be meaningless. Returning zero tensor in false positive score\n", " warnings.warn(*args, **kwargs) # noqa: B028\n", "/usr/lib/python3.11/site-packages/torchmetrics/utilities/prints.py:43: UserWarning: No positive samples in targets, true positive value should be meaningless. Returning zero tensor in true positive score\n", " warnings.warn(*args, **kwargs) # noqa: B028\n" ] }, { "data": { "text/html": [ "
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃        Test metric        ┃       DataLoader 0        ┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│       test.accuracy       │    0.5526315569877625     │\n",
       "│        test.auroc         │    0.4736842215061188     │\n",
       "│          test.f1          │    0.38847118616104126    │\n",
       "│         test.loss         │    0.7271869778633118     │\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "
\n" ], "text/plain": [ "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n", "┃\u001b[1m \u001b[0m\u001b[1m Test metric \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m DataLoader 0 \u001b[0m\u001b[1m \u001b[0m┃\n", "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n", "│\u001b[36m \u001b[0m\u001b[36m test.accuracy \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.5526315569877625 \u001b[0m\u001b[35m \u001b[0m│\n", "│\u001b[36m \u001b[0m\u001b[36m test.auroc \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.4736842215061188 \u001b[0m\u001b[35m \u001b[0m│\n", "│\u001b[36m \u001b[0m\u001b[36m test.f1 \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.38847118616104126 \u001b[0m\u001b[35m \u001b[0m│\n", "│\u001b[36m \u001b[0m\u001b[36m test.loss \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.7271869778633118 \u001b[0m\u001b[35m \u001b[0m│\n", "└───────────────────────────┴───────────────────────────┘\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Fold 1/1 done\n", "Fold 1 metrics: {'train.loss': 0.5477855205535889, 'train.grad_norm': 1.4536314010620117, 'vali.loss': 0.7176894545555115, 'vali.accuracy': 0.5625, 'vali.f1': 0.34761905670166016, 'vali.auroc': 0.6041666269302368, 'test.loss': 0.7271869778633118, 'test.accuracy': 0.5526315569877625, 'test.f1': 0.38847118616104126, 'test.auroc': 0.4736842215061188}\n" ] } ], "source": [ "# Run k-fold classification\n", "trainer, pretrainer, classifier, clf_data = ft_kfold_classification(\n", " config_path=config_path,\n", " clf_adata_path=clf_adata_path,\n", " hvg_selection_mod=None, # that automatically uses X. Specify it if not using X\n", " nHVGs=nHVGs,\n", " hvg_flavor=hvg_flavor,\n", " mod1_layer=mod1_layer,\n", " mod2_layer=mod2_layer,\n", " obs_label=obs_label,\n", " n_folds=1, # Just one train/valid/test split, set to >1 for k-fold CV\n", " valid_fraction=valid_fraction,\n", " test_fraction=test_fraction,\n", " stratify_test=stratify_test,\n", " stratify_valid=stratify_valid,\n", " dataloader_worker=dataloader_worker,\n", " base_model_ckpt=base_model_ckpt,\n", " tuned_model_ckpt=None,\n", " test_ckpt=\"best\",\n", " token_dict_path=token_dict_path,\n", " default_root_dir=default_root_dir,\n", " save_callback_metrics=True,\n", " save_label_dict=True,\n", " seed=seed,\n", " accelerator=accelerator,\n", " ndevices=1\n", ")" ] }, { "cell_type": "markdown", "id": "b5aa9a84", "metadata": {}, "source": [ "## 2. Detailed Step-by-Step Breakdown\n", "\n", "Below, we deconstruct the :func:`scdynomics.ft_kfold_classification` pipeline to explore the individual steps involved in data preprocessing, model initialization, and fine-tuning execution.\n", "\n", "---" ] }, { "cell_type": "markdown", "id": "40434e01", "metadata": {}, "source": [ "### 2.1 Data Preprocessing\n", "\n", "Load the query dataset meant for fine-tuning.\n", "We expect the target labels for the classification task (e.g., cell types or states) to be stored within the `.obs` attribute of the AnnData object." ] }, { "cell_type": "markdown", "id": "811d3d5c", "metadata": {}, "source": [ "#### 2.1.1 Highly Variable Gene (HVG) Selection\n", "\n", "*(Optional)* We can subset the dataset to highly variable genes to reduce dimensionality and focus the classifier on the most informative features." ] }, { "cell_type": "code", "execution_count": 4, "id": "4b991425", "metadata": {}, "outputs": [], "source": [ "adata = ad.read_h5ad(clf_adata_path)\n", "sc.pp.highly_variable_genes(\n", " adata,\n", " n_top_genes=nHVGs,\n", " subset=True,\n", " flavor= hvg_flavor,\n", ")" ] }, { "cell_type": "markdown", "id": "3911a762", "metadata": {}, "source": [ "#### 2.1.2 Load Pretrained Model and Align Feature Space\n", "\n", "Here, we load the :class:`scdynomics.Masked_Pretrainer` base model and automatically align its feature space (token embeddings) with the query dataset using the :func:`scdynomics.ft.pipeline.set_pretrainer` utility." ] }, { "cell_type": "code", "execution_count": 5, "id": "c97688a1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loaded pretrainer model\n", "Keep modalities: [0]\n", "Keep genes: 101; with 12 TFs\n" ] } ], "source": [ "# Set the layers\n", "layers, embed_mod = set_layers_and_embed_mod(mod1_layer, mod2_layer)\n", "\n", "# Load the pretrained base model and the corresponding data corpus for classification\n", "pretrainer, clf_data = set_pretrainer(\n", " clf_data = Multimodal_Corpus(\n", " adata = adata,\n", " label_key = obs_label,\n", " layers = layers,\n", " backed = False,\n", " ),\n", " embed_mod = embed_mod,\n", " base_model_ckpt = base_model_ckpt,\n", " token_dict_path = token_dict_path,\n", ")" ] }, { "cell_type": "markdown", "id": "50ad300f", "metadata": {}, "source": [ "#### 2.1.3 Dataset Splitting and Dataloader Initialization\n", "\n", "We partition the aligned corpus into training, validation, and test sets.\n", "These are then wrapped in PyTorch :class:`torch.utils.data.DataLoader` objects for efficient batch processing.\n" ] }, { "cell_type": "code", "execution_count": 6, "id": "9299652e", "metadata": {}, "outputs": [], "source": [ "# Load the config file\n", "config = JSON.decode(config_path)\n", "dl_hparams = config[\"dataloader_hparams\"]\n", "\n", "# Split train and test data\n", "train_list, valid_list, test_list = random_split(\n", " clf_data,\n", " test_fraction=test_fraction,\n", " valid_fraction=valid_fraction,\n", " stratified_test=stratify_test,\n", " stratified_valid=stratify_valid,\n", " random_seed=seed,\n", ")\n", "\n", "# Create dataloaders\n", "train_data = DataLoader(\n", " train_list[0],\n", " num_workers = dataloader_worker,\n", " batch_size = dl_hparams[\"batch_per_device\"],\n", " shuffle = True\n", ")\n", "val_data = DataLoader(\n", " valid_list[0],\n", " num_workers = dataloader_worker,\n", " batch_size = dl_hparams[\"batch_per_device\"],\n", " shuffle = False\n", ") if valid_list[0] is not None else None\n", "test_data = DataLoader(\n", " test_list[0],\n", " num_workers = dataloader_worker,\n", " batch_size = dl_hparams[\"batch_per_device\"],\n", " shuffle = False\n", ") if test_list[0] is not None else None" ] }, { "cell_type": "markdown", "id": "525af966", "metadata": {}, "source": [ "### 2.2 Model Initialization\n", "\n", "We initialize the :class:`scdynomics.FT_MLP_Classifier`.\n", "This downstream module encapsulates our pretrained base model,\n", " freezing its core weights while injecting Adapter modules for parameter efficient fine-tuning and appending an adaptable Multi-Layer Perceptron (MLP) head for classification.\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": 7, "id": "5f4c2dc7", "metadata": {}, "outputs": [], "source": [ "classifier = FT_MLP_Classifier(\n", " base_model = pretrainer,\n", " n_classes = len(clf_data.label_dict),\n", " steps_per_epoch = len(train_data),\n", " **config[\"classifier_hparams\"],\n", ")" ] }, { "cell_type": "markdown", "id": "e61bc813", "metadata": {}, "source": [ "### 2.3 Trainer Setup & Execution\n", "\n", "We configure the PyTorch Lightning :class:`lightning.pytorch.Trainer` and invoke the `.fit()` method to begin the supervised fine-tuning process on the downstream task.\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": null, "id": "eedc48f9", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/lib/python3.11/site-packages/lightning_fabric/plugins/environments/slurm.py:204: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python /usr/lib/python3 ...\n", "GPU available: True (cuda), used: False\n", "TPU available: False, using: 0 TPU cores\n", "HPU available: False, using: 0 HPUs\n", "/usr/lib/python3.11/site-packages/pytorch_lightning/trainer/setup.py:177: GPU available but not used. You can set it by doing `Trainer(accelerator='gpu')`.\n", "\n", " | Name | Type | Params | Mode \n", "----------------------------------------------------------\n", "0 | ft_encoder | FT_Encoder | 5.4 M | train\n", "1 | decision | Sequential | 13.6 K | train\n", "2 | criterion | CrossEntropyLoss | 0 | train\n", "3 | accuracy | MulticlassAccuracy | 0 | train\n", "4 | f1 | MulticlassF1Score | 0 | train\n", "5 | auroc | MulticlassAUROC | 0 | train\n", "----------------------------------------------------------\n", "2.2 M Trainable params\n", "3.2 M Non-trainable params\n", "5.4 M Total params\n", "21.762 Total estimated model params size (MB)\n", "294 Modules in train mode\n", "0 Modules in eval mode\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "`Trainer.fit` stopped: `max_epochs=1` reached.\n" ] } ], "source": [ "# Init the Trainer object\n", "trainer = Trainer(\n", " devices = 1,\n", " accelerator = accelerator,\n", " default_root_dir = default_root_dir + f\"fold_1/\",\n", " accumulate_grad_batches = cal_accum_grad_batches(config, 1),\n", " callbacks = make_callbacks(config),\n", " **config[\"trainer_hparams\"]\n", ")\n", "\n", "# Train the model\n", "trainer.fit(\n", " model=classifier,\n", " train_dataloaders=train_data,\n", " val_dataloaders=val_data,\n", ")\n", "\n", "# Get the callback metrics\n", "callback_metrics = {\n", " k: float(v)\n", " for k,v in trainer.callback_metrics.items()\n", "}" ] }, { "cell_type": "markdown", "id": "b91878b6", "metadata": {}, "source": [ "### 2.4 Test Set Evaluation\n", "\n", "After training, we evaluate the best model checkpoint against the held-out test set to calculate the final accuracy, F1 score, and AUROC metrics.\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": 9, "id": "61ba4793", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Restoring states from the checkpoint path at logs/fold_1/lightning_logs/version_0/checkpoints/epoch=0-step=34.ckpt\n", "Loaded model weights from the checkpoint at logs/fold_1/lightning_logs/version_0/checkpoints/epoch=0-step=34.ckpt\n", "/usr/lib/python3.11/site-packages/torchmetrics/utilities/prints.py:43: UserWarning: No negative samples in targets, false positive value should be meaningless. Returning zero tensor in false positive score\n", " warnings.warn(*args, **kwargs) # noqa: B028\n", "/usr/lib/python3.11/site-packages/torchmetrics/utilities/prints.py:43: UserWarning: No positive samples in targets, true positive value should be meaningless. Returning zero tensor in true positive score\n", " warnings.warn(*args, **kwargs) # noqa: B028\n" ] }, { "data": { "text/html": [ "
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃        Test metric        ┃       DataLoader 0        ┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│       test.accuracy       │    0.5526315569877625     │\n",
       "│        test.auroc         │    0.3815789520740509     │\n",
       "│          test.f1          │    0.38847118616104126    │\n",
       "│         test.loss         │    0.6895833015441895     │\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "
\n" ], "text/plain": [ "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n", "┃\u001b[1m \u001b[0m\u001b[1m Test metric \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m DataLoader 0 \u001b[0m\u001b[1m \u001b[0m┃\n", "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n", "│\u001b[36m \u001b[0m\u001b[36m test.accuracy \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.5526315569877625 \u001b[0m\u001b[35m \u001b[0m│\n", "│\u001b[36m \u001b[0m\u001b[36m test.auroc \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.3815789520740509 \u001b[0m\u001b[35m \u001b[0m│\n", "│\u001b[36m \u001b[0m\u001b[36m test.f1 \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.38847118616104126 \u001b[0m\u001b[35m \u001b[0m│\n", "│\u001b[36m \u001b[0m\u001b[36m test.loss \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.6895833015441895 \u001b[0m\u001b[35m \u001b[0m│\n", "└───────────────────────────┴───────────────────────────┘\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "callback_metrics: {'train.loss': 0.9454113245010376, 'train.grad_norm': 1.589394211769104, 'vali.loss': 0.6885690093040466, 'vali.accuracy': 0.5625, 'vali.f1': 0.34761905670166016, 'vali.auroc': 0.375, 'test.loss': 0.6895833015441895, 'test.accuracy': 0.5526315569877625, 'test.f1': 0.38847118616104126, 'test.auroc': 0.3815789520740509}\n" ] } ], "source": [ "# test the model\n", "trainer.test(\n", " model=classifier,\n", " dataloaders=test_data,\n", " ckpt_path=\"best\"\n", ")\n", "\n", "# Get the callback metrics\n", "for k,v in trainer.callback_metrics.items():\n", " callback_metrics[k] = float(v)\n", "print(f\"callback_metrics: {callback_metrics}\")" ] }, { "cell_type": "markdown", "id": "5f5202b2", "metadata": {}, "source": [ "## 3. Apply Fine-Tuned Classifier for Prediction\n", "\n", "Once fine-tuned, the classifier can be used to predict labels for new cells.\n", "The :func:`scdynomics.ft_clf_predict` function automates the inference process and generates a classification report if true labels are provided.\n", "\n", "The associated automated script is available in the scDynOmics repository under `scripts/clf.predict.py`.\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": null, "id": "4d4475a2", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Seed set to 42\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Loaded pretrainer model\n", "Keep modalities: [0]\n", "Keep genes: 101; with 12 TFs\n", "Predictions saved to logs/preds.json\n", "Classification report saved to logs/preds.report.json\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/usr/lib/python3.11/site-packages/sklearn/metrics/_classification.py:1833: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n", " _warn_prf(average, modifier, f\"{metric.capitalize()} is\", result.shape[0])\n", "/usr/lib/python3.11/site-packages/sklearn/metrics/_classification.py:1833: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n", " _warn_prf(average, modifier, f\"{metric.capitalize()} is\", result.shape[0])\n", "/usr/lib/python3.11/site-packages/sklearn/metrics/_classification.py:1833: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n", " _warn_prf(average, modifier, f\"{metric.capitalize()} is\", result.shape[0])\n" ] } ], "source": [ "# Set the paths\n", "label_dict_path = default_root_dir + f\"label_dict.json\"\n", "tuned_model_ckpt = default_root_dir + f\"fold_1/lightning_logs/version_0/checkpoints/epoch=0-step=34.ckpt\"\n", "\n", "# Run prediction\n", "all_preds, clf_report = ft_clf_predict(\n", " clf_adata=clf_data.adata, # Directly use the adata since it's already preprocessed and aligned\n", " mod1_layer=mod1_layer,\n", " mod2_layer=mod2_layer,\n", " obs_label=obs_label,\n", " label_dict_path=label_dict_path,\n", " token_dict_path=token_dict_path,\n", " base_model_ckpt=base_model_ckpt,\n", " tuned_model_ckpt=tuned_model_ckpt,\n", " outpath= default_root_dir + f\"preds.json\",\n", " accelerator = accelerator,\n", " seed=seed,\n", ")" ] }, { "cell_type": "markdown", "id": "65f0550c", "metadata": {}, "source": [ "## 4. Explain Fine-Tuned Classifier\n", "\n", "To ensure interpretability, scDynOmics includes an explanation module utilizing Integrated Gradients.\n", "This allows us to quantify the contribution of individual features (genes) to the model's decisions.\n", "\n", "The associated automated script is available in the scDynOmics repository under `scripts/clf.explain.py`.\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": null, "id": "818749a4", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Seed set to 42\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Loaded pretrainer model\n", "Keep modalities: [0]\n", "Keep genes: 101; with 12 TFs\n", "Launching Class 0 Explanation...\n", "Explainer: 0 samples predicted as class 0.\n", "Class 0 Explanation Done: 0.09 minutes\n", "Launching Class 1 Explanation...\n", "Explainer: 190 samples predicted as class 1.\n", "\tProcessing 1th batch...\n", "\t\tFinished batch 1: 0.11 minutes\n", "\tProcessing 2th batch...\n", "\t\tFinished batch 2: 0.10 minutes\n", "Class 1 Explanation Done: 0.30 minutes\n", "Explanation saved to logs/explain.txt\n" ] } ], "source": [ "# Explanation settings\n", "explain_sample_limit = 2\n", "sample_method = 'top'\n", "explain_batch_size = 1\n", "explain_step = 100\n", "\n", "# Run the explanation\n", "explain_result = ft_clf_explain(\n", " clf_adata = clf_data.adata, # Directly use the adata since it's already preprocessed and aligned\n", " mod1_layer = mod1_layer,\n", " mod2_layer = mod2_layer,\n", " dataloader_worker = dataloader_worker,\n", " token_dict_path = token_dict_path,\n", " base_model_ckpt = base_model_ckpt,\n", " tuned_model_ckpt = tuned_model_ckpt,\n", " explain_sample_limit = explain_sample_limit,\n", " sample_method= sample_method,\n", " explain_batch_size = explain_batch_size,\n", " explain_step = explain_step,\n", " outpath = default_root_dir + f\"explain.txt\",\n", " accelerator = accelerator,\n", " seed = seed,\n", ")" ] }, { "cell_type": "markdown", "id": "20d163e0", "metadata": {}, "source": [ "## 4. TensorBoard Visualization\n", "\n", "The training progress and metrics can be monitored using TensorBoard by running the following command in the terminal:\n", "\n", "---\n", "```bash\n", "tensorboard --logdir logs/fold_1\n", "```" ] }, { "cell_type": "markdown", "id": "a8c89adb", "metadata": {}, "source": [ "## 5. Cleanup\n", "\n", "Remove the generated logs and checkpoints to free up space.\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": null, "id": "70aed14f", "metadata": {}, "outputs": [], "source": [ "import shutil\n", "shutil.rmtree(default_root_dir)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.6" } }, "nbformat": 4, "nbformat_minor": 5 }