# Developing-Heart-Local-Improved - Visium Analysis

**Pages:** 39

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/Visium/cell2location/wholeheart_with_DA/no2_mapping_OCT_DA.html

**Contents:**
  - Loading packages¶
- Functions¶
- Read in metadata¶
- Data, parameters¶
- Loading visium data¶
- Add metadata and annotation¶
- Remove spots based on manual annotation¶
- Filter out low quality spots¶
- Remove mitochondrial genes¶
- Run cell2location mapping¶

**Examples:**

Example 1 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl

import cell2location
import scvi

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text
import seaborn as sns
```

Example 2 (unknown):
```unknown
Global seed set to 0
```

Example 3 (python):
```python
# import gc
# import torch
```

Example 4 (python):
```python
def read_and_qc(sample_name, path):
    r""" This function reads the data for one 10X spatial experiment into the anndata object.
    It also calculates QC metrics. Modify this function if required by your workflow.

    :param sample_name: Name of the sample
    :param path: path to data
    """

    adata = sc.read_visium(path + str(sample_name),
                           count_file='filtered_feature_bc_matrix.h5', load_images=True)
    adata.var_names_make_unique(join='-')
    
    adata.obs['sample'] = sample_name
    
    # rename genes to ENSEMBL
    adata.var['SYMBOL'] = adata.var_names
    adata.var_names = adata.var['gene_ids']
    adata.var_names.name = None

    # Calculate QC metrics
    sc.pp.calculate_qc_metrics(adata, inplace=True)
    adata.var['mt'] = [gene.startswith('MT-') for gene in adata.var['SYMBOL']]
    adata.obs['mt_frac'] = adata[:, adata.var['mt'].tolist()].X.sum(1).A.squeeze()/adata.obs['total_counts']

    # add sample name to obs names
    adata.obs["sample"] = [str(i) for i in adata.obs['sample']]
    adata.obs_names = adata.obs["sample"] \
                          + '_' + adata.obs_names
    adata.obs.index.name = 'spot_id'

    return adata

def select_slide(adata, s, s_col='sample'):
    r""" This function selects the data for one slide from the spatial anndata object.

    :param adata: Anndata object with multiple spatial experiments
    :param s: name of selected experiment
    :param s_col: column in adata.obs listing experiment name for each location
    """

    slide = adata[adata.obs[s_col].isin([s]), :]
    s_keys = list(slide.uns['spatial'].keys())
    s_spatial = np.array(s_keys)[[s in k for k in s_keys]][0]

    slide.uns['spatial'] = {s_spatial: slide.uns['spatial'][s_spatial]}

    return slide
```

Example 5 (python):
```python
meta=pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/metadata/HeartTeamSamples_Mappeddata_20230208.csv',
                index_col=0)
meta=meta[['Publication','sangerID', 'combinedID',
       'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender',
       'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status']]

# select slides for foetal publication
meta=meta[meta['Publication'].isin(['Foetal'])]
# select only OCT slides
meta=meta[meta['modality'].isin(['Visium'])]
# select only Healthy slides
meta=meta[meta['status'].isin(['Healthy'])]
```

Example 6 (unknown):
```unknown
# select slides which contain DuctusArterious
sampleIDs = ['HCAHeartST10298118','HCAHeartST10298119','HCAHeartST10298121','HCAHeartST10317187','HCAHeartST10317188']
meta = meta[meta['sangerID'].isin(sampleIDs)]
meta
```

Example 7 (unknown):
```unknown
results_folder = '/nfs/team205/heart/cell2location/foetal'

# create paths and names to results folders for reference regression and cell2location models
ref_run_name = f'{results_folder}/reference_signatures/ver28Feb2023/whole_woPeriCell'
run_name = f'{results_folder}/cell2location_map/ver28Feb2023'
```

Example 8 (unknown):
```unknown
visium_path='/nfs/team205/heart/mapped/spaceranger110/'

celltype_key='fine_grain'
```

Example 9 (python):
```python
# Read the data of the interested region into anndata objects
slides = []
for i in sampleIDs:
    slides.append(read_and_qc(i, path=visium_path))

# Combine anndata objects together
adata_vis = slides[0].concatenate(
    slides[1:],
    batch_key="sample",
    uns_merge="unique",
    batch_categories=sampleIDs,
    index_unique=None
)
```

Example 10 (unknown):
```unknown
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
```

Example 11 (python):
```python
print(adata_vis.shape)
adata_vis.obs['sample'].value_counts()
```

Example 12 (unknown):
```unknown
(5552, 33538)
```

Example 13 (unknown):
```unknown
HCAHeartST10298121    1180
HCAHeartST10298119    1176
HCAHeartST10298118    1106
HCAHeartST10317188    1083
HCAHeartST10317187    1007
Name: sample, dtype: int64
```

Example 14 (python):
```python
# PLOT QC FOR EACH SAMPLE
fig, axs = plt.subplots(len(slides), 4, figsize=(15, 4*len(slides)-4))
for i, s in enumerate(adata_vis.obs['sample'].unique()):
    #fig.suptitle('Covariates for filtering')

    slide = select_slide(adata_vis, s)
    sns.distplot(slide.obs['total_counts'],
                 kde=False, ax = axs[i, 0])
    axs[i, 0].set_xlim(0, adata_vis.obs['total_counts'].max())
    axs[i, 0].set_xlabel(f'total_counts | {s}')

    sns.distplot(slide.obs['total_counts']\
                 [slide.obs['total_counts']<2000],
                 kde=False, bins=40, ax = axs[i, 1])
    axs[i, 1].set_xlim(0, 2000)
    axs[i, 1].set_xlabel(f'total_counts | {s}')

    sns.distplot(slide.obs['n_genes_by_counts'],
                 kde=False, bins=60, ax = axs[i, 2])
    axs[i, 2].set_xlim(0, adata_vis.obs['n_genes_by_counts'].max())
    axs[i, 2].set_xlabel(f'n_genes_by_counts | {s}')

    sns.distplot(slide.obs['n_genes_by_counts']\
                 [slide.obs['n_genes_by_counts']<1000],
                 kde=False, bins=60, ax = axs[i, 3])
    axs[i, 3].set_xlim(0, 1000)
    axs[i, 3].set_xlabel(f'n_genes_by_counts | {s}')

plt.tight_layout()
```

Example 15 (python):
```python
Trying to set attribute `._uns` of view, copying.
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
Trying to set attribute `._uns` of view, copying.
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/seaborn/distributions.py:2657: RuntimeWarning: Mean of empty slice.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
```

Example 16 (python):
```python
# metadata
adata_vis.obs = adata_vis.obs.reset_index().merge(meta, how='left', left_on='sample', right_on='sangerID').set_index('spot_id')
adata_vis
```

Example 17 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5552 × 33538
    obs: 'in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'Publication', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status'
    var: 'gene_ids', 'feature_types', 'genome', 'SYMBOL', 'mt', 'n_cells_by_counts-HCAHeartST10298118', 'mean_counts-HCAHeartST10298118', 'log1p_mean_counts-HCAHeartST10298118', 'pct_dropout_by_counts-HCAHeartST10298118', 'total_counts-HCAHeartST10298118', 'log1p_total_counts-HCAHeartST10298118', 'n_cells_by_counts-HCAHeartST10298119', 'mean_counts-HCAHeartST10298119', 'log1p_mean_counts-HCAHeartST10298119', 'pct_dropout_by_counts-HCAHeartST10298119', 'total_counts-HCAHeartST10298119', 'log1p_total_counts-HCAHeartST10298119', 'n_cells_by_counts-HCAHeartST10298121', 'mean_counts-HCAHeartST10298121', 'log1p_mean_counts-HCAHeartST10298121', 'pct_dropout_by_counts-HCAHeartST10298121', 'total_counts-HCAHeartST10298121', 'log1p_total_counts-HCAHeartST10298121', 'n_cells_by_counts-HCAHeartST10317187', 'mean_counts-HCAHeartST10317187', 'log1p_mean_counts-HCAHeartST10317187', 'pct_dropout_by_counts-HCAHeartST10317187', 'total_counts-HCAHeartST10317187', 'log1p_total_counts-HCAHeartST10317187', 'n_cells_by_counts-HCAHeartST10317188', 'mean_counts-HCAHeartST10317188', 'log1p_mean_counts-HCAHeartST10317188', 'pct_dropout_by_counts-HCAHeartST10317188', 'total_counts-HCAHeartST10317188', 'log1p_total_counts-HCAHeartST10317188'
    uns: 'spatial'
    obsm: 'spatial'
```

Example 18 (python):
```python
adata_vis.obs.head()
```

Example 19 (python):
```python
# annotation
path_to_annot='/nfs/team205/heart/visium_annotations/foetal'
category_name='JC_edit'

# read in annotation csv file and concatenate
for i, ID in enumerate(sampleIDs):
    if f'{ID}_JC.csv' in os.listdir(path_to_annot):
    
        print(ID)
        annot_df = pd.read_csv(f'{path_to_annot}/{ID}_JC.csv')
        annot_df['spot_id']=ID+'_'+annot_df['Barcode']

        if i==0:
            annot_df_all=annot_df.copy()
        else:
            annot_df_all=pd.concat([annot_df_all,annot_df])
    else:
        print(f'{ID} is not annotated')

# add annotations to adata_vis
adata_vis.obs=pd.concat([adata_vis.obs,
                     annot_df_all.set_index('spot_id')[[category_name]].reindex(adata_vis.obs.index)],
                     axis=1)
# replace nan to 'none'
adata_vis.obs[category_name].fillna('none', inplace=True)

# minor modification
annotation_edit_dict = {
     'AV Ring':'AV ring',
    'Aorta':'Great vessels',
    'Coronary Artery':'Coronary artery',
    'Mitral Valve':'AV valves',
    'Mitral valve':'AV valves',
    'Myocardium - Ventricular - Trabecular':'Myocardium - Ventricular - Trabeculated',
    'Myocardim - Ventricular - Trabeculated':'Myocardium - Ventricular - Trabeculated',
    'Myocardim - Ventricular - Compact':'Myocardium - Ventricular - Compact',
    'unknown':'Unannotated',
    'Unknown':'Unannotated',
}
adata_vis.obs.replace({category_name:annotation_edit_dict}, inplace=True)
set(adata_vis.obs[category_name])
```

Example 20 (unknown):
```unknown
HCAHeartST10298118
HCAHeartST10298119
HCAHeartST10298121
HCAHeartST10317187
HCAHeartST10317188
```

Example 21 (unknown):
```unknown
{'Cavity',
 'Epicardium',
 'Great vessels',
 'Myocardium - Atrial',
 'Myocardium - Ventricular - Compact',
 'Myocardium - Ventricular - Trabeculated',
 'none'}
```

Example 22 (python):
```python
# remove 'none' and 'no_tissue'
adata_vis = adata_vis[adata_vis.obs[category_name].isin(['none','Out_of_tissue'])==False]

# remove 'Cavity'
adata_vis = adata_vis[adata_vis.obs[category_name].isin(['Cavity'])==False]

# remove 'Unannotated'
adata_vis = adata_vis[adata_vis.obs[category_name].isin(['Unannotated'])==False]
```

Example 23 (python):
```python
# order annotations
annotations_order = ['Myocardium - Atrial',
                     'Myocardium - Ventricular - Compact',
                     'Myocardium - Ventricular - Trabeculated',
                    # 'Myocardium - Papillary Muscle',
                    'Epicardium',
                     # 'AV valves',
                     # 'Aortic valve',
                    # 'Valve apparatus',
                   # 'AV ring',
                    # 'Connective tissue',
                     'Great vessels',
                     # 'Coronary artery',
                     # 'Nerve',
                     ]
adata_vis.obs[category_name] = adata_vis.obs[category_name].astype('category')
adata_vis.obs[category_name].cat.reorder_categories(annotations_order,inplace=True)
```

Example 24 (unknown):
```unknown
Trying to set attribute `.obs` of view, copying.
/tmp/ipykernel_26774/3723571678.py:17: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
```

Example 25 (python):
```python
pd.crosstab(adata_vis.obs['age'],adata_vis.obs[category_name])
```

Example 26 (python):
```python
# filter out low quality spots
sc.pp.filter_cells(adata_vis, min_counts = 500)
sc.pp.filter_cells(adata_vis, min_genes = 300)

# Spot number per slide
adata_vis.obs['sample'].value_counts()
```

Example 27 (unknown):
```unknown
HCAHeartST10298121    1151
HCAHeartST10298119    1139
HCAHeartST10298118    1082
HCAHeartST10317188    1070
HCAHeartST10317187     991
Name: sample, dtype: int64
```

Example 28 (python):
```python
# find mitochondria-encoded (MT) genes
adata_vis.var['MT_gene'] = [gene.startswith('MT-') for gene in adata_vis.var['SYMBOL']]
print(sum(adata_vis.var['MT_gene']))
    
# remove MT genes for spatial mapping (keeping their counts in the object)
adata_vis.obsm['MT'] = adata_vis[:, adata_vis.var['MT_gene'].values].X.toarray()
adata_vis = adata_vis[:, ~adata_vis.var['MT_gene'].values]
print(sum(adata_vis.var['MT_gene']))
```

Example 29 (python):
```python
# convert adata_vis.var_names to symbol
adata_vis.var.set_index('SYMBOL',inplace=True)
adata_vis.var.head()
```

Example 30 (python):
```python
%%time

# read in reference signatures
inf_aver = pd.read_csv(f"{ref_run_name}/inf_aver.csv", index_col=0)

# find shared genes and subset both anndata and reference signatures
intersect = np.intersect1d(adata_vis.var_names, inf_aver.index)
adata_vis = adata_vis[:, intersect].copy()
inf_aver = inf_aver.loc[intersect, :].copy()
print(f'shape of inf_aver: {inf_aver.shape}')
print('reference cell types')
print(inf_aver.columns)

# subset adata
adata_vis=adata_vis.copy()
# prepare anndata for cell2location model
scvi.data.setup_anndata(adata=adata_vis, batch_key="sample")
scvi.data.view_anndata_setup(adata_vis)
```

Example 31 (python):
```python
shape of inf_aver: (16306, 58)
reference cell types
Index(['AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight',
       'VentricularCardiomyocytesLeftCompact',
       'VentricularCardiomyocytesRightCompact',
       'VentricularCardiomyocytesLeftTrabeculated',
       'VentricularCardiomyocytesRightTrabeculated',
       'SinoatrialNodeCardiomyocytes', 'AtrioventricularNodeCardiomyocytes',
       'VentricularConductionSystemProximal',
       'VentricularConductionSystemDistal',
       'GreatVesselAdventitialFibroblasts',
       'CoronaryVesselAdventitialFibroblasts',
       'MyocardialInterstitialFibroblasts1',
       'MyocardialInterstitialFibroblasts2', 'Myofibroblasts',
       'LymphNodeFibroblasticReticularCells', 'ValveInterstitialCells',
       'GreatVesselSmoothMuscleCells', 'CoronarySmoothMuscleCells',
       'DuctusArteriosusSmoothMuscleCells', 'CoronaryPericytes',
       'GreatVesselArterialEndothelialCells',
       'GreatVesselVenousEndothelialCells', 'CoronaryArterialEndothelialCells',
       'CoronaryVenousEndothelialCells', 'CoronaryCapillaryEndothelialCells',
       'EndocardialCells', 'EndocardialCushionCells', 'ValveEndothelialCells',
       'LymphaticEndothelialCells', 'MesothelialEpicardialCells',
       'EpicardiumDerivedCells', 'NeuronPrecursors', 'ChromaffinCells',
       'SympatheticNeurons', 'ParaympatheticNeurons', 'SchwannCellPrecursors',
       'SchwannCells', 'MonocytesMPOpos', 'Monocytes', 'MonocyteDerivedCells',
       'MacrophagesCX3CR1pos', 'MacrophagesTIMD4pos', 'MacrophagesLYVE1pos',
       'MacrophagesATF3pos', 'DendriticCellsType1', 'DendriticCellsMature',
       'PlasmacytoidDendriticCells', 'MastCells', 'Megakaryocytes',
       'TCellsCD4pos', 'TCellsCD8pos', 'TregsCD4pos', 'ProBCells', 'BCells',
       'BCellsMS4A1pos', 'NaturalKillerCells', 'InnateLymphoidCells'],
      dtype='object')
INFO     Using batches from adata.obs["sample"]                                              
INFO     No label_key inputted, assuming all cells have same label                           
INFO     Using data from adata.X
```

Example 32 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function setup_anndata is deprecated; Please use the model-specific setup_anndata methods instead. The global method will be removed in version 0.15.0.
```

Example 33 (python):
```python
INFO     Successfully registered anndata object containing 5433 cells, 16306 vars, 5 batches,
         1 labels, and 0 proteins. Also registered 0 extra categorical covariates and 0 extra
         continuous covariates.                                                              
INFO     Please do not further modify adata until model is trained.
```

Example 34 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function view_anndata_setup is deprecated; This method will be removed in 0.15.0. Please avoid building any new dependencies on it.
```

Example 35 (unknown):
```unknown
Anndata setup with scvi-tools version 0.14.4.
```

Example 36 (unknown):
```unknown
Data Summary              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃             Data             ┃ Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│            Cells             │ 5433  │
│             Vars             │ 16306 │
│            Labels            │   1   │
│           Batches            │   5   │
│           Proteins           │   0   │
│ Extra Categorical Covariates │   0   │
│ Extra Continuous Covariates  │   0   │
└──────────────────────────────┴───────┘
```

Example 37 (python):
```python
SCVI Data Registry              
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃     Data      ┃    scvi-tools Location    ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│       X       │          adata.X          │
│ batch_indices │ adata.obs['_scvi_batch']  │
│    labels     │ adata.obs['_scvi_labels'] │
└───────────────┴───────────────────────────┘
```

Example 38 (python):
```python
Label Categories                        
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃      Source Location      ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['_scvi_labels'] │     0      │          0          │
└───────────────────────────┴────────────┴─────────────────────┘
```

Example 39 (python):
```python
Batch Categories                         
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃   Source Location   ┃     Categories     ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['sample'] │ HCAHeartST10298118 │          0          │
│                     │ HCAHeartST10298119 │          1          │
│                     │ HCAHeartST10298121 │          2          │
│                     │ HCAHeartST10317187 │          3          │
│                     │ HCAHeartST10317188 │          4          │
└─────────────────────┴────────────────────┴─────────────────────┘
```

Example 40 (unknown):
```unknown
CPU times: user 1.15 s, sys: 283 ms, total: 1.43 s
Wall time: 1.43 s
```

Example 41 (python):
```python
# create and train the model
mod = cell2location.models.Cell2location(
    adata_vis, cell_state_df=inf_aver, 
    # the expected average cell abundance: tissue-dependent 
    # hyper-prior which can be estimated from paired histology:
    N_cells_per_location=20,
    # hyperparameter controlling normalisation of
    # within-experiment variation in RNA detection (using default here):
    detection_alpha=20
) 
adata_vis.var.head()
```

Example 42 (python):
```python
mod.train(max_epochs=20000, 
          # train using full data (batch_size=None)
          batch_size=None, 
          # use all data points in training because 
          # we need to estimate cell abundance at all locations
          train_size=1,
          use_gpu=True)

# plot ELBO loss history during training, removing first 100 epochs from the plot
mod.plot_history(1000)
plt.legend(labels=['full data training']);

# In this section, we export the estimated cell abundance (summary of the posterior distribution).
adata_vis = mod.export_posterior(
    adata_vis, sample_kwargs={'num_samples': 1000, 'batch_size': mod.adata.n_obs, 'use_gpu': True}
)

# Save model
mod.save(f"{run_name}/DuctusArteriosus", overwrite=True)

# Save anndata object with results
adata_vis.write(f"{run_name}/DuctusArteriosus/sp.h5ad")

# Examine reconstruction accuracy to assess if there are any issues with mapping
# the plot should be roughly diagonal, strong deviations will signal problems
mod.plot_QC()
```

Example 43 (unknown):
```unknown
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/configuration_validator.py:99: UserWarning: you passed in a val_dataloader but have no validation_step. Skipping val loop
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
```

Example 44 (unknown):
```unknown
Epoch 20000/20000: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20000/20000 [44:49<00:00,  7.44it/s, v_num=1, elbo_train=5.14e+7]
Sampling local variables, batch: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:36<00:00, 36.09s/it]
Sampling global variables, sample: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 999/999 [00:34<00:00, 29.37it/s]
```

Example 45 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sample' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'Publication' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sangerID' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'combinedID' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'donor' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'donor_type' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'region' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'region_finest' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'age' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'gender' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'facility' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'cell_or_nuclei' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'modality' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'kit_10x' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'flushed' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'status' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'feature_types' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'genome' as categorical
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/Visium/Others/plotting_multiple_MP-vessel.html

**Contents:**
- Loading packages¶
- Read in adata¶
- Plotting¶

**Examples:**

Example 1 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import seaborn as sns

from cell2location.plt import plot_spatial

import os
```

Example 2 (python):
```python
# import a module given the full path?
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)

spec = importlib.util.spec_from_file_location("module.name", "/home/jovyan/projects/P46_Adult-heart_Visium/notebooks/ver8regions_torepo/_utils.py")
_utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = _utils
spec.loader.exec_module(_utils)
```

Example 3 (python):
```python
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 4 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk'
```

Example 5 (python):
```python
adata_vis = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-FFPE_foetal_raw.h5ad')
adata_vis
```

Example 6 (unknown):
```unknown
AnnData object with n_obs × n_vars = 19973 × 18085
    obs: 'in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'JC_edit'
    var: 'gene_ids', 'feature_types', 'genome', 'SYMBOL', 'mt'
    uns: 'spatial'
    obsm: 'MT', 'means_cell_abundance_w_sf', 'q05_cell_abundance_w_sf', 'q95_cell_abundance_w_sf', 'spatial', 'stds_cell_abundance_w_sf'
```

Example 7 (python):
```python
# add cell2loc output to .obs
# get the cell2loc output
cell2loc_out = adata_vis.obsm['q05_cell_abundance_w_sf'].copy()
## read in cell type name update dictionary
celltype_rename = utils.read_pkl('/nfs/team205/heart/anndata_objects/Foetal/fine_grain_rename_dictionary.pkl')
## update cell type name
cellnames = adata_vis.obsm['q05_cell_abundance_w_sf'].columns
cellnames = [x.split('q05cell_abundance_w_sf_')[1] for x in cellnames]
cellnames = [celltype_rename[x] for x in cellnames]
## add
adata_vis.obs[cellnames] = cell2loc_out
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 19973 × 18085
    obs: 'in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'JC_edit', 'AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight', 'VentricularCardiomyocytesLeftCompact', 'VentricularCardiomyocytesRightCompact', 'VentricularCardiomyocytesLeftTrabeculated', 'VentricularCardiomyocytesRightTrabeculated', 'SinoatrialNodeCardiomyocytes', 'AtrioventricularNodeCardiomyocytes', 'VentricularConductionSystemProximal', 'VentricularConductionSystemDistal', 'GreatVesselAdventitialFibroblasts', 'CoronaryVesselAdventitialFibroblasts', 'MyocardialInterstitialFibroblasts1', 'MyocardialInterstitialFibroblasts2', 'Myofibroblasts', 'LymphNodeFibroblasticReticularCells', 'ValveInterstitialCells', 'GreatVesselSmoothMuscleCells', 'CoronarySmoothMuscleCells', 'CoronaryPericytes', 'GreatVesselArterialEndothelialCells', 'GreatVesselVenousEndothelialCells', 'CoronaryArterialEndothelialCells', 'CoronaryVenousEndothelialCells', 'CoronaryCapillaryEndothelialCells', 'EndocardialCells', 'EndocardialCushionCells', 'ValveEndothelialCells', 'LymphaticEndothelialCells', 'MesothelialEpicardialCells', 'EpicardiumDerivedCells', 'NeuronPrecursors', 'ChromaffinCells', 'SympatheticNeurons', 'ParaympatheticNeurons', 'SchwannCellPrecursors', 'SchwannCells', 'MonocytesMPOpos', 'Monocytes', 'MonocyteDerivedCells', 'MacrophagesCX3CR1pos', 'MacrophagesTIMD4pos', 'MacrophagesLYVE1pos', 'MacrophagesATF3pos', 'DendriticCellsType1', 'DendriticCellsMature', 'PlasmacytoidDendriticCells', 'MastCells', 'Megakaryocytes', 'TCellsCD4pos', 'TCellsCD8pos', 'TregsCD4pos', 'ProBCells', 'BCells', 'BCellsMS4A1pos', 'NaturalKillerCells', 'InnateLymphoidCells', 'SinoatrialNodePacemakerCells', 'AtrioventricularNodePacemakerCells', 'MyocardialInterstitialFibroblasts', 'SubEpicardialFibroblasts', 'ParasympatheticNeurons'
    var: 'gene_ids', 'feature_types', 'genome', 'SYMBOL', 'mt'
    uns: 'spatial'
    obsm: 'MT', 'means_cell_abundance_w_sf', 'q05_cell_abundance_w_sf', 'q95_cell_abundance_w_sf', 'spatial', 'stds_cell_abundance_w_sf'
```

Example 9 (python):
```python
set(adata_vis.obs['sample'])
```

Example 10 (unknown):
```unknown
{'BHF_F_Hea13295830', 'BHF_F_Hea13295831'}
```

Example 11 (python):
```python
slide = _utils.select_multislide(adata_vis, ['BHF_F_Hea13295830'])
sc.pl.spatial(slide,cmap='RdPu',color='JC_edit', ncols=5, wspace=0,
              size=0,img_key='hires',alpha_img=1,vmin=0, vmax='p99.4',
             title='H&E', legend_loc=None)
```

Example 12 (unknown):
```unknown
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
```

Example 13 (unknown):
```unknown
?plot_spatial
```

Example 14 (python):
```python
Signature: plot_spatial(adata, color, img_key='hires', show_img=True, **kwargs)
Docstring:
Plot spatial abundance of cell types (regulatory programmes) with colour gradient
and interpolation (from Visium anndata).

This method supports only 7 cell types with these colours (in order, which can be changed using reorder_cmap).
'yellow' 'orange' 'blue' 'green' 'purple' 'grey' 'white'

:param adata: adata object with spatial coordinates in adata.obsm['spatial']
:param color: list of adata.obs column names to be plotted
:param kwargs: arguments to plot_spatial_general
:return: matplotlib figure
File:      ~/my-conda-envs/cellpymc/lib/python3.7/site-packages/cell2location/plt/plot_spatial.py
Type:      function
```

Example 15 (python):
```python
# select up to 6 clusters
# https://github.com/BayraktarLab/cell2location/blob/bd2aae833f81d8ab68684aa44f863b36e3af0aed/cell2location/plt/plot_spatial.py#L108

slide = _utils.select_multislide(adata_vis, ['BHF_F_Hea13295830'])

clust_labels = ['MacrophagesLYVE1pos',
                'CoronaryVenousEndothelialCells',
                'CoronarySmoothMuscleCells'
                ]
clust_col = ['' + str(i) for i in clust_labels] # in case column names differ from labels

plt.rcParams['axes.grid'] = False
with mpl.rc_context({'figure.figsize': (15, 15)}):
    fig = plot_spatial(
        adata=slide,
        # labels to show on a plot
        color=clust_col, labels=clust_labels,
        show_img=True, img_alpha=0.5,
        # 'fast' (white background) or 'dark_background'
        style='fast',
        # limit color scale at 99.2% quantile of cell abundance
        max_color_quantile=0.997,
        # size of locations (adjust depending on figure size)
        circle_diameter=3,
        colorbar_position='right',
        colorbar_shape = {'vertical_gaps': 0.5, 'horizontal_gaps': 0.3,
                                    'width': 0.2, 'height': 0.1},
        # crop_x=(650,820),
        # crop_y=(1200,1420),
        reorder_cmap = [3,4,1], # 'yellow' 'orange' 'blue' 'green' 'purple' 'grey' 'white'
    )
    plt.savefig(f"{figdir}/Visium_MP-Vessels_BHF_F_Hea13295830.pdf", bbox_inches="tight",dpi=300)
```

Example 16 (unknown):
```unknown
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
```

Example 17 (python):
```python
# select up to 6 clusters
# https://github.com/BayraktarLab/cell2location/blob/bd2aae833f81d8ab68684aa44f863b36e3af0aed/cell2location/plt/plot_spatial.py#L108

slide = _utils.select_multislide(adata_vis, ['BHF_F_Hea13295830'])

clust_labels = ['MacrophagesLYVE1pos',
                'CoronaryVenousEndothelialCells',
                'CoronarySmoothMuscleCells'
                ]
clust_col = ['' + str(i) for i in clust_labels] # in case column names differ from labels

plt.rcParams['axes.grid'] = False
with mpl.rc_context({'figure.figsize': (12, 10)}):
    fig = plot_spatial(
        adata=slide,
        # labels to show on a plot
        color=clust_col, labels=clust_labels,
        show_img=True, img_alpha=0.5,
        # 'fast' (white background) or 'dark_background'
        style='fast',
        # limit color scale at 99.2% quantile of cell abundance
        max_color_quantile=0.997,
        # size of locations (adjust depending on figure size)
        circle_diameter=30,
        colorbar_position='right',
        colorbar_shape = {'vertical_gaps': 0.5, 'horizontal_gaps': 0.4,
                                    'width': 0.1, 'height': 0.1},
        crop_x=(1500,1680),
        crop_y=(1280,1510),
        reorder_cmap = [3,4,1] # 'yellow' 'orange' 'blue' 'green' 'purple' 'grey' 'white'
    )
    plt.savefig(f"{figdir}/Visium_MP-Vessels_BHF_F_Hea13295830_zoomin1.pdf", bbox_inches="tight",dpi=300)
```

Example 18 (unknown):
```unknown
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
```

Example 19 (python):
```python
# select up to 6 clusters
# https://github.com/BayraktarLab/cell2location/blob/bd2aae833f81d8ab68684aa44f863b36e3af0aed/cell2location/plt/plot_spatial.py#L108

slide = _utils.select_multislide(adata_vis, ['BHF_F_Hea13295830'])

clust_labels = ['MacrophagesLYVE1pos',
                'CoronaryVenousEndothelialCells',
                'CoronarySmoothMuscleCells'
                ]
clust_col = ['' + str(i) for i in clust_labels] # in case column names differ from labels

plt.rcParams['axes.grid'] = False
with mpl.rc_context({'figure.figsize': (12, 10)}):
    fig = plot_spatial(
        adata=slide,
        # labels to show on a plot
        color=clust_col, labels=clust_labels,
        show_img=True, img_alpha=1,
        # 'fast' (white background) or 'dark_background'
        style='fast',
        # limit color scale at 99.2% quantile of cell abundance
        max_color_quantile=0.997,
        # size of locations (adjust depending on figure size)
        circle_diameter=0,
        colorbar_position=None,
        colorbar_shape = {'vertical_gaps': 0.5, 'horizontal_gaps': 0.4,
                                    'width': 0.1, 'height': 0.1},
        crop_x=(1500,1680),
        crop_y=(1280,1510),
        reorder_cmap = [3,4,1] # 'yellow' 'orange' 'blue' 'green' 'purple' 'grey' 'white'
    )
    plt.savefig(f"{figdir}/Visium_MP-Vessels_BHF_F_Hea13295830_zoomin1_HE.pdf", bbox_inches="tight",dpi=300)
```

Example 20 (unknown):
```unknown
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
```

Example 21 (python):
```python
# select up to 6 clusters
# https://github.com/BayraktarLab/cell2location/blob/bd2aae833f81d8ab68684aa44f863b36e3af0aed/cell2location/plt/plot_spatial.py#L108

slide = _utils.select_multislide(adata_vis, ['BHF_F_Hea13295830'])

clust_labels = ['MacrophagesLYVE1pos',
                'CoronaryVenousEndothelialCells',
                'CoronarySmoothMuscleCells'
                ]
clust_col = ['' + str(i) for i in clust_labels] # in case column names differ from labels

plt.rcParams['axes.grid'] = False
with mpl.rc_context({'figure.figsize': (12, 10)}):
    fig = plot_spatial(
        adata=slide,
        # labels to show on a plot
        color=clust_col, labels=clust_labels,
        show_img=True, img_alpha=0.5,
        # 'fast' (white background) or 'dark_background'
        style='fast',
        # limit color scale at 99.2% quantile of cell abundance
        max_color_quantile=0.997,
        # size of locations (adjust depending on figure size)
        circle_diameter=20,
        colorbar_position='right',
        colorbar_shape = {'vertical_gaps': 0.5, 'horizontal_gaps': 0.6,
                                    'width': 0.1, 'height': 0.1},
        crop_x=(90,380),
        crop_y=(1640,1790),
        reorder_cmap = [3,4,1] # 'yellow' 'orange' 'blue' 'green' 'purple' 'grey' 'white'
    )
    plt.savefig(f"{figdir}/Visium_MP-Vessels_BHF_F_Hea13295830_zoomin2.pdf", bbox_inches="tight",dpi=300)
```

Example 22 (unknown):
```unknown
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
```

Example 23 (python):
```python
# select up to 6 clusters
# https://github.com/BayraktarLab/cell2location/blob/bd2aae833f81d8ab68684aa44f863b36e3af0aed/cell2location/plt/plot_spatial.py#L108

slide = _utils.select_multislide(adata_vis, ['BHF_F_Hea13295830'])

clust_labels = ['MacrophagesLYVE1pos',
                'CoronaryVenousEndothelialCells',
                'CoronarySmoothMuscleCells'
                ]
clust_col = ['' + str(i) for i in clust_labels] # in case column names differ from labels

plt.rcParams['axes.grid'] = False
with mpl.rc_context({'figure.figsize': (12, 10)}):
    fig = plot_spatial(
        adata=slide,
        # labels to show on a plot
        color=clust_col, labels=clust_labels,
        show_img=True, img_alpha=1,
        # 'fast' (white background) or 'dark_background'
        style='fast',
        # limit color scale at 99.2% quantile of cell abundance
        max_color_quantile=0.997,
        # size of locations (adjust depending on figure size)
        circle_diameter=0,
        colorbar_position=None,
        colorbar_shape = {'vertical_gaps': 0.5, 'horizontal_gaps': 0.6,
                                    'width': 0.1, 'height': 0.1},
        crop_x=(90,380),
        crop_y=(1640,1790),
        reorder_cmap = [3,4,1] # 'yellow' 'orange' 'blue' 'green' 'purple' 'grey' 'white'
    )
    plt.savefig(f"{figdir}/Visium_MP-Vessels_BHF_F_Hea13295830_zoomin2_HE.pdf", bbox_inches="tight",dpi=300)
```

Example 24 (unknown):
```unknown
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig3_SAN_niche/cellphonedb-VisiumHD_no2_run.html

**Contents:**
- Submid job¶
- Check error files¶
- Check output files¶

Job <97409> is submitted to queue .

**Examples:**

Example 1 (unknown):
```unknown
!cat /nfs/team205/heart/cellphonedb/run_script/run_cellphone_v5_foetal_visiumhd_HEA_FOET14880396_statistical_SAnode.py
```

Example 2 (python):
```python
import pandas as pd
import sys
import os
from cellphonedb.src.core.methods import cpdb_statistical_analysis_method

# Input files
cpdb_file_path = '/nfs/team205/heart/cellphonedb/db/v5.0.0/cellphonedb.zip'
meta_file_path = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/inputs/meta_SAnode.txt'
counts_file_path = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/inputs/log_norm_counts_SAnode.h5ad'
out_path = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/SAnode'
if not os.path.exists(out_path):
    os.makedirs(out_path)
min_percent = sys.argv[1]

print(f'##### All cells, min %:{min_percent} #####')

# run
cpdb_results = cpdb_statistical_analysis_method.call(
    cpdb_file_path = cpdb_file_path,                 # mandatory: CellphoneDB database zip file.
    meta_file_path = meta_file_path,                 # mandatory: tsv file defining barcodes to cell label.
    counts_file_path = counts_file_path,             # mandatory: normalized count matrix - a path to the counts file, or an in-memory AnnData object
    counts_data = 'hgnc_symbol',                     # defines the gene annotation in counts matrix.
    active_tfs_file_path = None,           # optional: defines cell types and their active TFs.
    microenvs_file_path = None,       # optional (default: None): defines cells per microenvironment.
    score_interactions = True,                       # optional: whether to score interactions or not. 
    iterations = 1000,                               # denotes the number of shufflings performed in the analysis.
    threshold = float(min_percent),                                 # defines the min % of cells expressing a gene for this to be employed in the analysis.
    threads = 5,                                     # number of threads to use in the analysis.
    debug_seed = 42,                                 # debug randome seed. To disable >=0.
    result_precision = 3,                            # Sets the rounding for the mean values in significan_means.
    pvalue = 0.1,                                   # P-value threshold to employ for significance.
    subsampling = False,                             # To enable subsampling the data (geometri sketching).
    subsampling_log = False,                         # (mandatory) enable subsampling log1p for non log-transformed data inputs.
    subsampling_num_pc = 100,                        # Number of componets to subsample via geometric skectching (dafault: 100).
    subsampling_num_cells = 1000,                    # Number of cells to subsample (integer) (default: 1/3 of the dataset).
    separator = '|',                                 # Sets the string to employ to separate cells in the results dataframes "cellA|CellB".
    debug = False,                                   # Saves all intermediate tables employed during the analysis in pkl format.
    output_path = out_path,                          # Path to save results.
    output_suffix = None                             # Replaces the timestamp in the output files by a user defined string in the  (default: None).
    )
```

Example 3 (unknown):
```unknown
module load cellgen/conda
conda activate cpdb
cd /nfs/team205/heart/cellphonedb

MIN_PERCENT=0.05
bsub \
-G team205 \
-q normal -M20000 -R"select[mem>20000] rusage[mem=20000] span[hosts=1]" \
-n 4 \
-o ./log/foetal_visiumhd_HEA_FOET14880396_output.%J \
-e ./log/foetal_visiumhd_HEA_FOET14880396_errorfile.%J \
python ./run_script/run_cellphone_v5_foetal_visiumhd_HEA_FOET14880396_statistical_SAnode.py $MIN_PERCENT
```

Example 4 (unknown):
```unknown
%%bash
FILE_path=/nfs/team205/heart/cellphonedb/log/foetal_visiumhd_HEA_FOET14880396_errorfile.97409
echo ${FILE_path}
cat ${FILE_path}
```

Example 5 (unknown):
```unknown
/nfs/team205/heart/cellphonedb/log/foetal_visiumhd_HEA_FOET14880396_errorfile.97409
100%|██████████| 1000/1000 [00:14<00:00, 71.33it/s]
100%|██████████| 11/11 [00:00<00:00, 278.56it/s]
100%|██████████| 11/11 [00:00<00:00, 806.77it/s]
100%|██████████| 121/121 [00:29<00:00,  4.05it/s]
```

Example 6 (unknown):
```unknown
%%bash
PATH_out=/nfs/team205/heart/cellphonedb/log/foetal_visiumhd_HEA_FOET14880396_output.97409
cat ${PATH_out}
```

Example 7 (python):
```python
##### All cells, min %:0.05 #####
Reading user files...
The following user files were loaded successfully:
/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/inputs/log_norm_counts_SAnode.h5ad
/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/inputs/meta_SAnode.txt
[ ][CORE][04/05/25-10:49:38][INFO] [Cluster Statistical Analysis] Threshold:0.05 Iterations:1000 Debug-seed:42 Threads:5 Precision:3
[ ][CORE][04/05/25-10:49:38][WARNING] Debug random seed enabled. Set to 42
[ ][CORE][04/05/25-10:49:39][INFO] Running Real Analysis
[ ][CORE][04/05/25-10:49:39][INFO] Running Statistical Analysis
[ ][CORE][04/05/25-10:49:54][INFO] Building Pvalues result
[ ][CORE][04/05/25-10:49:54][INFO] Building results
[ ][CORE][04/05/25-10:49:54][INFO] Scoring interactions: Filtering genes per cell type..
[ ][CORE][04/05/25-10:49:54][INFO] Scoring interactions: Calculating mean expression of each gene per group/cell type..
[ ][CORE][04/05/25-10:49:54][INFO] Scoring interactions: Calculating scores for all interactions and cell types..
Saved deconvoluted to /lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/SAnode/statistical_analysis_deconvoluted_05_04_2025_105025.txt
Saved deconvoluted_percents to /lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/SAnode/statistical_analysis_deconvoluted_percents_05_04_2025_105025.txt
Saved means to /lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/SAnode/statistical_analysis_means_05_04_2025_105025.txt
Saved pvalues to /lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/SAnode/statistical_analysis_pvalues_05_04_2025_105025.txt
Saved significant_means to /lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/SAnode/statistical_analysis_significant_means_05_04_2025_105025.txt
Saved interaction_scores to /lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/SAnode/statistical_analysis_interaction_scores_05_04_2025_105025.txt

------------------------------------------------------------
Sender: LSF System <lsfadmin@node-14-11>
Subject: Job 97409: <python ./run_script/run_cellphone_v5_foetal_visiumhd_HEA_FOET14880396_statistical_SAnode.py 0.05> in cluster <farm22> Done

Job <python ./run_script/run_cellphone_v5_foetal_visiumhd_HEA_FOET14880396_statistical_SAnode.py 0.05> was submitted from host <farm22-head1> by user <kk18> in cluster <farm22> at Sun May  4 10:47:34 2025
Job was executed on host(s) <4*node-14-11>, in queue <normal>, as user <kk18> in cluster <farm22> at Sun May  4 10:47:36 2025
</nfs/users/nfs_k/kk18> was used as the home directory.
</nfs/team205/heart/cellphonedb> was used as the working directory.
Started at Sun May  4 10:47:36 2025
Terminated at Sun May  4 10:50:39 2025
Results reported at Sun May  4 10:50:39 2025

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
python ./run_script/run_cellphone_v5_foetal_visiumhd_HEA_FOET14880396_statistical_SAnode.py 0.05
------------------------------------------------------------

Successfully completed.

Resource usage summary:

    CPU time :                                   179.27 sec.
    Max Memory :                                 2015 MB
    Average Memory :                             513.89 MB
    Total Requested Memory :                     20000.00 MB
    Delta Memory :                               17985.00 MB
    Max Swap :                                   -
    Max Processes :                              8
    Max Threads :                                13
    Run time :                                   166 sec.
    Turnaround time :                            185 sec.

The output (if any) is above this job summary.



PS:

Read file <./log/foetal_visiumhd_HEA_FOET14880396_errorfile.97409> for stderr output of this job.
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig3_SAN_niche/VisiumHD_CCSaxis_no2_calculate_axis.html

**Contents:**
- Variables¶
- Read in data¶
- Calculate axis¶
- Save¶
- Plot for figure¶

bin2cell output which has celltypist output as well

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import tissue_tag as tt
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 4 (unknown):
```unknown
os.getcwd()
```

Example 5 (unknown):
```unknown
'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3'
```

Example 6 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects'
sample_id = 'HEA_FOET14880396'
```

Example 7 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 8 (python):
```python
# read in 
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 9 (unknown):
```unknown
AnnData object with n_obs × n_vars = 129803 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_4Jan2025', 'HistologicalAnnotation', 'SANaxis_landmarks'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'HistologicalAnnotation_colors', 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'fine_grain_4Jan2025_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 10 (python):
```python
# select cells to calculate axis
cells = [
     'SinoatrialNodePacemakerCellsHorn',
    'SinoatrialNodePacemakerCellsHead',
    'SinoatrialNodePacemakerCellsTail',
]
bdata = adata[adata.obs['fine_grain_4Jan2025'].isin(cells)]
obs_df_sel = bdata.obs.copy()
obs_df_sel.rename(columns={'array_row':'y','array_col':'x'},inplace=True)
```

Example 11 (python):
```python
# calculate distance
print('calculating distances')
dist2clusterall = tt.dist2cluster_fast(obs_df_sel, annotation='SANaxis_landmarks', KNN=1, logscale=False)
dist2clusterall
```

Example 12 (unknown):
```unknown
calculating distances
calculating distance matrix with cKDTree
end
nan
start
```

Example 13 (unknown):
```unknown
{'end': array([1251.38301643, 1505.08749736, 1600.76852645, ...,  223.18975964,
         355.30243964,  435.07815142]),
 'nan': array([0., 0., 0., ..., 0., 0., 0.]),
 'start': array([1113.68595365, 1453.53120161, 1529.98277576, ...,  183.28824696,
         717.60829512,  805.5865061 ])}
```

Example 14 (unknown):
```unknown
obs_df_sel = tt.calculate_axis_2p(obs_df_sel, 'SANaxis_landmarks', ['start','end'], output_col='out',prefix='L2_dist_')
```

Example 15 (python):
```python
bdata.obs['SAN_axis'] = np.nan
bdata.obs.loc[obs_df_sel.index,'SAN_axis'] = obs_df_sel['out']
```

Example 16 (python):
```python
/tmp/ipykernel_1233989/1043464966.py:1: ImplicitModificationWarning: Trying to modify attribute `.obs` of view, initializing view as actual.
  bdata.obs['SAN_axis'] = np.nan
```

Example 17 (python):
```python
# add to adata
adata.obs['SAN_axis'] = np.nan
adata.obs.loc[bdata.obs_names,'SAN_axis'] = bdata.obs['SAN_axis'].copy()
```

Example 18 (python):
```python
mask = ((adata.obs['array_row'] >= 1800) & 
        (adata.obs['array_row'] <= 2300) & 
        (adata.obs['array_col'] >= 600) & 
        (adata.obs['array_col'] <= 1000)
       )
bdata = adata[mask]

sc.pl.spatial(bdata, color='SAN_axis',cmap='gist_rainbow',
              img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
              size=7,legend_fontsize=30, spot_size=10,
                  frameon=False) # save=f'_{c}_b2c',
```

Example 19 (python):
```python
adata.write(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 20 (unknown):
```unknown
AnnData object with n_obs × n_vars = 129803 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_4Jan2025', 'HistologicalAnnotation', 'SANaxis_landmarks', 'SAN_axis'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'HistologicalAnnotation_colors', 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'fine_grain_4Jan2025_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 21 (python):
```python
# read in 
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad')

mask = ((adata.obs['array_row'] >= 1800) & 
        (adata.obs['array_row'] <= 2300) & 
        (adata.obs['array_col'] >= 600) & 
        (adata.obs['array_col'] <= 1000)
       )
bdata = adata[mask]

sc.pl.spatial(bdata, color='SAN_axis',cmap='gist_rainbow',
              img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
              size=7,legend_fontsize=30, spot_size=10,
                  frameon=False,show=False)
plt.savefig(f'{figdir}/VisiumHD_SANaxis.pdf', bbox_inches="tight",dpi=300)
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/Visium/Others/Parasymp-vs-Symp_aggr.html

**Contents:**
- Import required modules¶
- Function¶
- Read in anndata¶
- Merge categories: tissue annotation and week¶
- Preapare cell2location output anndata¶
- Select regions and cells to plot¶
- Get spots which includes PSNS or SNS and mean abundance¶
- How about in single-cell data¶

**Examples:**

Example 1 (python):
```python
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import seaborn as sns
import os 
from mpl_toolkits.axes_grid1 import make_axes_locatable

import scipy
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
-----
anndata             0.9.2
matplotlib          3.7.2
mpl_toolkits        NA
numpy               1.24.4
pandas              1.5.3
scanpy              1.9.3
scipy               1.10.1
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                         10.0.0
anyio                       NA
arrow                       1.2.3
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.12.1
backcall                    0.2.0
brotli                      1.0.9
certifi                     2023.11.17
cffi                        1.15.1
charset_normalizer          3.2.0
colorama                    0.4.6
comm                        0.1.4
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.6.8
decorator                   5.1.1
defusedxml                  0.7.1
executing                   1.2.0
fastjsonschema              NA
fqdn                        NA
h5py                        3.9.0
idna                        3.4
igraph                      0.10.6
importlib_metadata          NA
importlib_resources         NA
ipykernel                   6.25.1
isoduration                 NA
jedi                        0.19.0
jinja2                      3.1.2
joblib                      1.3.2
json5                       NA
jsonpointer                 2.0
jsonschema                  4.19.0
jsonschema_specifications   NA
jupyter_events              0.7.0
jupyter_server              2.7.1
jupyterlab_server           2.24.0
kiwisolver                  1.4.5
leidenalg                   0.10.1
llvmlite                    0.40.1
markupsafe                  2.1.3
natsort                     8.4.0
nbformat                    5.9.2
numba                       0.57.1
overrides                   NA
packaging                   23.1
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
plotly                      5.16.1
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.5
ptyprocess                  0.7.0
pure_eval                   0.2.2
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.16.1
pyparsing                   3.0.9
pythonjsonlogger            NA
pytz                        2023.3
referencing                 NA
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rpds                        NA
ruamel                      NA
send2trash                  NA
six                         1.16.0
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
sphinxcontrib               NA
stack_data                  0.6.2
statsmodels                 0.14.0
texttable                   1.6.7
threadpoolctl               3.2.0
tornado                     6.3.3
traitlets                   5.9.0
typing_extensions           NA
uri_template                NA
urllib3                     2.0.4
wcwidth                     0.2.6
webcolors                   1.13
websocket                   1.6.2
yaml                        6.0.1
zipp                        NA
zmq                         25.1.1
zstandard                   0.19.0
```

Example 5 (unknown):
```unknown
-----
IPython             8.12.0
jupyter_client      8.3.0
jupyter_core        4.12.0
jupyterlab          4.0.5
notebook            7.0.2
-----
Python 3.8.17 (default, Jul  5 2023, 21:04:15) [GCC 11.2.0]
Linux-5.4.0-171-generic-x86_64-with-glibc2.17
-----
Session information updated at 2024-03-13 13:18
```

Example 6 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 7 (python):
```python
sc.settings.verbosity = 1          # verbosity: errors (0), warnings (1), info (2), hints (3)
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 8 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk'
```

Example 9 (python):
```python
import matplotlib.colors as mcolors
from matplotlib.cm import ScalarMappable
from matplotlib.lines import Line2D

def plot_cell2loc_out(
    adata_c2l,
    celltype,
    regions_sel,
    xaxis,
    yaxis,
    abundance_thresh = 0.05,
    mean_abundance_max = 0.3,
    count_log_max=7,
    nspots_for_legend = [0, 50, 100],
    savefig = None
):
    # select spots which have the celltype of interest
    # define based on an abundance threshold
    mask = adata_c2l.to_df()[celltype]>abundance_thresh
    ad = adata_c2l[mask]
    print(f'n_spot which contains {celltype}: {ad.shape[0]}')

    # preparing dataframe
    plot_df = ad.to_df()[[celltype]]
    plot_df[[yaxis,xaxis]] = ad.obs[[yaxis,xaxis]].copy()
    plot_df['count'] = 1
    plot_df = plot_df.groupby([yaxis,xaxis]).sum().loc[regions_sel]
    plot_df['mean'] = plot_df[celltype]/plot_df['count']
    plot_df['count_log'] = np.log1p(plot_df['count'])

    # plotting
    plt.rcParams['axes.grid'] = False
    plt.rcParams['figure.figsize'] = 1.2,4
    norm = mcolors.Normalize(vmin=0, vmax=mean_abundance_max)
    sns.scatterplot(data=plot_df, x=xaxis,y=yaxis,
                    hue='mean', hue_norm=norm,
                    size='count_log', size_norm=(0,count_log_max),sizes=(0, 200),
                   palette='Reds',edgecolor='black')

    # Remove the "mean" legend
    plt.legend([],[], frameon=False)

    # Custom legend for sizes - example
    # Define a few representative sizes and their corresponding original count values
    # sizes = nspots_for_legend
    log_sizes = np.log1p(nspots_for_legend)  # Calculate log sizes for positioning in the plot (optional if you want exact matches)
    # Calculate markersizes based on your specific transformation and plotting parameters
    # This step involves converting log_sizes into the scale used in your plot
    markersizes = np.interp(log_sizes, [0, count_log_max], [0, 200])  # Adjust the range [0, 200] as per your plot
    # Create custom legend handles for sizes
    size_handles = [Line2D([0], [0], marker='o', color='w', markerfacecolor='gray', 
                       markersize=np.sqrt(markersize), label=f'{size}') for size, markersize in zip(nspots_for_legend, markersizes)]

    # Add the custom size legend to the plot
    plt.legend(handles=size_handles, title="n_spot", bbox_to_anchor=(2.5, 1), loc=2, borderaxespad=0.)

    # Show the colorbar with the normalized hue scale
    cbar = plt.colorbar(ScalarMappable(norm=norm, cmap="Reds"))
    cbar.set_label('Mean abundance', rotation=270, labelpad=15)
    plt.xticks(rotation=90)
    plt.margins(x=0.3,y=0.07)
    plt.title(celltype)
    if savefig!=None:
        plt.savefig(savefig, bbox_inches="tight",dpi=300)
    plt.show()
```

Example 10 (python):
```python
adata_vis = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium_foetal_healthy-all_raw.h5ad')
adata_vis.var_names_make_unique()

# log-transform
sc.pp.normalize_total(adata_vis, target_sum=1e4)
sc.pp.log1p(adata_vis)

adata_vis
```

Example 11 (unknown):
```unknown
/nfs/team205/kk18/miniconda3/envs/vitro/lib/python3.8/site-packages/anndata/_core/anndata.py:1840: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
```

Example 12 (unknown):
```unknown
AnnData object with n_obs × n_vars = 39223 × 33562
    obs: 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'cell2loc-map_group', 'JC_edit', 'sample_for_cell2loc', 'week', 'leiden', 'region_cluster', 'AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight', 'VentricularCardiomyocytesLeftCompact', 'VentricularCardiomyocytesRightCompact', 'VentricularCardiomyocytesLeftTrabeculated', 'VentricularCardiomyocytesRightTrabeculated', 'SinoatrialNodeCardiomyocytes', 'AtrioventricularNodeCardiomyocytes', 'VentricularConductionSystemProximal', 'VentricularConductionSystemDistal', 'GreatVesselAdventitialFibroblasts', 'CoronaryVesselAdventitialFibroblasts', 'MyocardialInterstitialFibroblasts1', 'MyocardialInterstitialFibroblasts2', 'Myofibroblasts', 'LymphNodeFibroblasticReticularCells', 'ValveInterstitialCells', 'GreatVesselSmoothMuscleCells', 'CoronarySmoothMuscleCells', 'CoronaryPericytes', 'GreatVesselArterialEndothelialCells', 'GreatVesselVenousEndothelialCells', 'CoronaryArterialEndothelialCells', 'CoronaryVenousEndothelialCells', 'CoronaryCapillaryEndothelialCells', 'EndocardialCells', 'EndocardialCushionCells', 'ValveEndothelialCells', 'LymphaticEndothelialCells', 'MesothelialEpicardialCells', 'EpicardiumDerivedCells', 'NeuronPrecursors', 'ChromaffinCells', 'SympatheticNeurons', 'ParaympatheticNeurons', 'SchwannCellPrecursors', 'SchwannCells', 'MonocytesMPOpos', 'Monocytes', 'MonocyteDerivedCells', 'MacrophagesCX3CR1pos', 'MacrophagesTIMD4pos', 'MacrophagesLYVE1pos', 'MacrophagesATF3pos', 'DendriticCellsType1', 'DendriticCellsMature', 'PlasmacytoidDendriticCells', 'MastCells', 'Megakaryocytes', 'TCellsCD4pos', 'TCellsCD8pos', 'TregsCD4pos', 'ProBCells', 'BCells', 'BCellsMS4A1pos', 'NaturalKillerCells', 'InnateLymphoidCells', 'DuctusArteriosusSmoothMuscleCells', 'AtrialCardiomyocytesLeft_minmax', 'AtrialCardiomyocytesRight_minmax', 'VentricularCardiomyocytesLeftCompact_minmax', 'VentricularCardiomyocytesRightCompact_minmax', 'VentricularCardiomyocytesLeftTrabeculated_minmax', 'VentricularCardiomyocytesRightTrabeculated_minmax', 'SinoatrialNodeCardiomyocytes_minmax', 'AtrioventricularNodeCardiomyocytes_minmax', 'VentricularConductionSystemProximal_minmax', 'VentricularConductionSystemDistal_minmax', 'GreatVesselAdventitialFibroblasts_minmax', 'CoronaryVesselAdventitialFibroblasts_minmax', 'MyocardialInterstitialFibroblasts1_minmax', 'MyocardialInterstitialFibroblasts2_minmax', 'Myofibroblasts_minmax', 'LymphNodeFibroblasticReticularCells_minmax', 'ValveInterstitialCells_minmax', 'GreatVesselSmoothMuscleCells_minmax', 'CoronarySmoothMuscleCells_minmax', 'CoronaryPericytes_minmax', 'GreatVesselArterialEndothelialCells_minmax', 'GreatVesselVenousEndothelialCells_minmax', 'CoronaryArterialEndothelialCells_minmax', 'CoronaryVenousEndothelialCells_minmax', 'CoronaryCapillaryEndothelialCells_minmax', 'EndocardialCells_minmax', 'EndocardialCushionCells_minmax', 'ValveEndothelialCells_minmax', 'LymphaticEndothelialCells_minmax', 'MesothelialEpicardialCells_minmax', 'EpicardiumDerivedCells_minmax', 'NeuronPrecursors_minmax', 'ChromaffinCells_minmax', 'SympatheticNeurons_minmax', 'ParaympatheticNeurons_minmax', 'SchwannCellPrecursors_minmax', 'SchwannCells_minmax', 'MonocytesMPOpos_minmax', 'Monocytes_minmax', 'MonocyteDerivedCells_minmax', 'MacrophagesCX3CR1pos_minmax', 'MacrophagesTIMD4pos_minmax', 'MacrophagesLYVE1pos_minmax', 'MacrophagesATF3pos_minmax', 'DendriticCellsType1_minmax', 'DendriticCellsMature_minmax', 'PlasmacytoidDendriticCells_minmax', 'MastCells_minmax', 'Megakaryocytes_minmax', 'TCellsCD4pos_minmax', 'TCellsCD8pos_minmax', 'TregsCD4pos_minmax', 'ProBCells_minmax', 'BCells_minmax', 'BCellsMS4A1pos_minmax', 'NaturalKillerCells_minmax', 'InnateLymphoidCells_minmax', 'DuctusArteriosusSmoothMuscleCells_minmax', 'annotation', 'annotation_final', 'annotation_final_coarse'
    var: 'gene_ids'
    uns: 'JC_edit_colors', 'annotation_colors', 'annotation_final_colors', 'cell2loc-map_group_colors', 'leiden', 'neighbors', 'region_cluster_colors', 'sample_for_cell2loc_colors', 'umap', 'log1p'
    obsm: 'MT', 'X_umap', 'means_cell_abundance_w_sf', 'q05_cell_abundance_w_sf', 'q05_cell_abundance_w_sf_ndarray', 'q95_cell_abundance_w_sf', 'spatial', 'stds_cell_abundance_w_sf'
    obsp: 'connectivities', 'distances'
```

Example 13 (python):
```python
# modify annotations for matching with factor analysis
annotation_edit_dict = {
   'AV valves':'Valves',
    'Aortic valve':'Valves',
    'Valve apparatus':'Valves',
    'Atrium - Left':'Atrium',
     'Atrium - Right':'Atrium',
    'Ventricle - Compact - Left':'Ventricle - Compact',
    'Ventricle - Compact - Right':'Ventricle - Compact',
    'Ventricle - Trabeculated - Left':'Ventricle - Trabeculated',
    'Ventricle - Trabeculated - Right':'Ventricle - Trabeculated',
}
adata_vis.obs['annotation_final_mod']=adata_vis.obs['annotation_final'].astype('str').copy()
adata_vis.obs.replace({'annotation_final_mod':annotation_edit_dict}, 
                      inplace=True)
adata_vis.obs['annotation_final_mod'] = adata_vis.obs['annotation_final_mod'].astype('category')

#reorder annotations
annotations_order = [
   'Atrium',
        'Ventricle - Compact',
     'Ventricle - Trabeculated',
        'SAnode',
        'AVnode',
    'VCS - Proximal',
        'VCS - Distal',
        'Great vessel',
        'Coronary vessel',
    'Ductus arteriosus',
        'Adventitia - Coronary',
     'Adventitia - GV',
    'Epicardium',
        'Endocardium - Atrial', 
    'Endocardium - Ventricular',
        'Endocardial cusion',
        'AV ring',
    'Valves',
    'LN',
            ]
adata_vis.obs['annotation_final_mod'].cat.reorder_categories(annotations_order, inplace=True)
```

Example 14 (python):
```python
pd.crosstab(adata_vis.obs['week'],adata_vis.obs['modality'])
```

Example 15 (python):
```python
pd.crosstab(adata_vis.obs['annotation_final_mod'],adata_vis.obs['modality'])
```

Example 16 (python):
```python
set(adata_vis.obs['week'])
```

Example 17 (unknown):
```unknown
{'13W', '16W', '4W', '5W', '7W'}
```

Example 18 (python):
```python
tissue_merge = {
    # 'AV ring',
    'AVnode':'Node',
    'Adventitia - Coronary':'Coronary tissues',
    'Adventitia - GV':'GV tissues',
    # 'Atrium',
    'Coronary vessel':'Coronary tissues',
    # 'Ductus arteriosus',
     # 'Endocardial cusion',
     'Endocardium - Atrial': 'Endocardium',
     'Endocardium - Ventricular': 'Endocardium',
     # 'Epicardium',
    'Great vessel':'GV tissues',
     # 'LN',
     'SAnode':'Node',
     'VCS - Distal':'VCS',
     'VCS - Proximal':'VCS',
     # 'Valves',
     'Ventricle - Compact': 'Ventricle',
     'Ventricle - Trabeculated': 'Ventricle'
}

week_merge = {
    '4W': '4-5W',
    '5W': '4-5W',
    '7W': '7W',
    '13W': '13-16W',
    '16W': '13-16W', 
}

# merge
adata_vis.obs['annotation_merged'] = adata_vis.obs['annotation_final_mod'].astype('str').copy()
adata_vis.obs.replace({'annotation_merged':tissue_merge},inplace=True)
adata_vis.obs['week_merged'] = adata_vis.obs['week'].astype('str').copy()
adata_vis.obs.replace({'week_merged':week_merge},inplace=True)
adata_vis.obs['week_merged'] = adata_vis.obs['week_merged'].astype('category')
adata_vis.obs['week_merged'].cat.reorder_categories(['4-5W','7W','13-16W'],inplace=True)
```

Example 19 (python):
```python
# get the cell2loc output
cell2loc_out = adata_vis.obsm['q05_cell_abundance_w_sf'].copy()
## read in cell type name update dictionary
celltype_rename = utils.read_pkl('/nfs/team205/heart/anndata_objects/Foetal/fine_grain_rename_dictionary.pkl')
## update cell type name
cellnames = adata_vis.obsm['q05_cell_abundance_w_sf'].columns
cellnames = [x.split('q05cell_abundance_w_sf_')[1] for x in cellnames]
cellnames = [celltype_rename[x] for x in cellnames]

# select obs columns
obs_columns = ['array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes',
               'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 
               'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei',
               'modality', 'kit_10x', 'flushed', 'status', 'cell2loc-map_group', 'JC_edit', 'sample_for_cell2loc', 'week', 'leiden', 'annotation_final','annotation_final_mod',
              'annotation_merged','week_merged'
              ]

# make anndata
adata_c2l = anndata.AnnData(
    X=adata_vis.obsm['q05_cell_abundance_w_sf_ndarray'].copy(),
    obs=adata_vis.obs[obs_columns],
    var=pd.DataFrame(cellnames).set_index(0),
    obsm=adata_vis.obsm.copy(),
    obsp=adata_vis.obsp.copy(),
    uns=adata_vis.uns.copy()
)
adata_c2l
```

Example 20 (unknown):
```unknown
AnnData object with n_obs × n_vars = 39223 × 58
    obs: 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'cell2loc-map_group', 'JC_edit', 'sample_for_cell2loc', 'week', 'leiden', 'annotation_final', 'annotation_final_mod', 'annotation_merged', 'week_merged'
    uns: 'JC_edit_colors', 'annotation_colors', 'annotation_final_colors', 'cell2loc-map_group_colors', 'leiden', 'neighbors', 'region_cluster_colors', 'sample_for_cell2loc_colors', 'umap', 'log1p'
    obsm: 'MT', 'X_umap', 'means_cell_abundance_w_sf', 'q05_cell_abundance_w_sf', 'q05_cell_abundance_w_sf_ndarray', 'q95_cell_abundance_w_sf', 'spatial', 'stds_cell_abundance_w_sf'
    obsp: 'connectivities', 'distances'
```

Example 21 (python):
```python
# select tissues
df = pd.crosstab(adata_c2l.obs['annotation_merged'],adata_c2l.obs['week_merged'])
regions_sel = list(df.index[(df>0).sum(axis=1)==3])
regions_sel
```

Example 22 (unknown):
```unknown
['Atrium',
 'Coronary tissues',
 'Endocardial cusion',
 'Endocardium',
 'Epicardium',
 'GV tissues',
 'Node',
 'VCS',
 'Ventricle']
```

Example 23 (python):
```python
adata_c2l.to_df()['ParasympatheticNeurons'].hist(bins=1000)
plt.yscale('log')
plt.xlim(0,0.5)
```

Example 24 (python):
```python
adata_c2l.to_df()['SympatheticNeurons'].hist(bins=1000)
plt.yscale('log')
plt.xlim(0,0.5)
```

Example 25 (python):
```python
celltype = 'ParasympatheticNeurons'
plot_cell2loc_out(
    adata_c2l,
    celltype=celltype,
    regions_sel=regions_sel,
    xaxis='week_merged',
    yaxis='annotation_merged',
    abundance_thresh = 0.05,
    mean_abundance_max = 0.3,
    count_log_max=7,
    nspots_for_legend = [0, 10, 100],
    savefig = f"{figdir}/visium_dotplot_{celltype}.pdf"
)
```

Example 26 (unknown):
```unknown
n_spot which contains ParasympatheticNeurons: 2850
```

Example 27 (python):
```python
/tmp/ipykernel_152303/533952583.py:58: MatplotlibDeprecationWarning: Unable to determine Axes to steal space for Colorbar. Using gca(), but will raise in the future. Either provide the *cax* argument to use as the Axes for the Colorbar, provide the *ax* argument to steal space from it, or add *mappable* to an Axes.
  cbar = plt.colorbar(ScalarMappable(norm=norm, cmap="Reds"))
```

Example 28 (python):
```python
celltype = 'SympatheticNeurons'
plot_cell2loc_out(
    adata_c2l,
    celltype=celltype,
    regions_sel=regions_sel,
    xaxis='week_merged',
    yaxis='annotation_merged',
    abundance_thresh = 0.05,
    mean_abundance_max = 0.3,
    count_log_max=7,
    nspots_for_legend = [0, 10, 100],
    savefig = f"{figdir}/visium_dotplot_{celltype}.pdf"
)
```

Example 29 (unknown):
```unknown
n_spot which contains SympatheticNeurons: 532
```

Example 30 (python):
```python
/tmp/ipykernel_152303/533952583.py:58: MatplotlibDeprecationWarning: Unable to determine Axes to steal space for Colorbar. Using gca(), but will raise in the future. Either provide the *cax* argument to use as the Axes for the Colorbar, provide the *ax* argument to steal space from it, or add *mappable* to an Axes.
  cbar = plt.colorbar(ScalarMappable(norm=norm, cmap="Reds"))
```

Example 31 (python):
```python
# read in sigle-cell data
adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad')

# subset neural compartment
adata = adata[adata.obs['coarse_grain']=='Neural']

# filtering genes
sc.pp.filter_genes(adata, min_cells=3)

# log-transform
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

adata
```

Example 32 (python):
```python
/nfs/team205/kk18/miniconda3/envs/vitro/lib/python3.8/site-packages/scanpy/preprocessing/_simple.py:251: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var['n_cells'] = number
```

Example 33 (unknown):
```unknown
AnnData object with n_obs × n_vars = 6149 × 26660
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'region', 'age', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'donor_by_library-prep', 'multiplexed', 'SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet', 'batch_key', '_scvi_batch', 'FACSgate', 'fine_grain', 'mid_grain', 'coarse_grain', 'sex', 'week', 'trimester', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', '_scvi_labels', 'stress_score', 'hb1_score'
    var: 'gene_name_scRNA-0', 'gene_id', 'n_cells'
    uns: 'FACSgate_colors', '_scvi_manager_uuid', '_scvi_uuid', 'age_colors', 'cell_or_nuclei_colors', 'coarse_grain_colors', 'cycling_colors', 'donor_colors', 'fine_grain_colors', 'heart_or_greatvessels_colors', 'hvg', 'kit_10x_colors', 'modality_colors', 'neighbors', 'phase_colors', 'region_colors', 'sex_colors', 'trimester_colors', 'umap', 'week_colors', 'log1p'
    obsm: 'X_scVI', 'X_umap', '_scvi_extra_categorical_covs', '_scvi_extra_continuous', '_scvi_extra_continuous_covs', 'latent_gene_encoding'
    obsp: 'connectivities', 'distances'
```

Example 34 (python):
```python
set(adata.obs.region)
```

Example 35 (unknown):
```unknown
{'aorta',
 'apex',
 'atria',
 'atrial septum',
 'base',
 'ductus arteriosus',
 'heart without node',
 'left ventricle',
 'node',
 'outflow tract',
 'pericardium',
 'pulmonary arches and branches',
 'right ventricle',
 'right ventricle and papillaries',
 'whole sample'}
```

Example 36 (python):
```python
# merge regions
region_merge = {
    # 'aorta',
 # 'apex',
 'atria': 'atrium',
 'atrial septum': 'atrium',
 # 'base',
 # 'ductus arteriosus',
 # 'heart without node',
 # 'left ventricle',
 # 'node',
 # 'outflow tract',
 # 'pericardium',
 # 'pulmonary arches and branches',
 'right ventricle': 'right ventricle',
 'right ventricle and papillaries': 'right ventricle',
 # 'whole sample'
}

# merge
adata.obs['region_merged'] = adata.obs['region'].astype('str').copy()
adata.obs.replace({'region_merged':region_merge},inplace=True)
```

Example 37 (python):
```python
# subset
# nuclei
adata_sub = adata[adata.obs['cell_or_nuclei']=='nuclei']
# non whole heart data
adata_sub = adata_sub[adata_sub.obs['region']!='whole sample']
adata_sub = adata_sub[adata_sub.obs['region']!='heart without node']
display(pd.crosstab(adata_sub.obs['fine_grain'],adata_sub.obs['week']))
display(pd.crosstab(adata_sub.obs['fine_grain'],adata_sub.obs['region']))
```

Example 38 (python):
```python
# remove 4W since it's tiny number of nuclei
adata_sub = adata_sub[adata_sub.obs['week']!='4W']
display(pd.crosstab(adata_sub.obs['fine_grain'],adata_sub.obs['week']))
```

Example 39 (python):
```python
# select by celltype
ad = adata_sub[adata_sub.obs['fine_grain'].isin(['ParasympatheticNeurons','SympatheticNeurons'])]
# ad = adata_sub.copy()
print(ad.shape)

# preparing dataframe
plot_df = ad.obs[['region_merged','fine_grain']].copy()
plot_df['count'] = 1
plot_df = plot_df.groupby(['region_merged','fine_grain']).sum()
# add missing regions as count = 0
missing_regions = set(adata_sub.obs.region_merged) - set(ad.obs.region_merged)
for reg in missing_regions:
    plot_df.loc[(reg,'SympatheticNeurons'),'count']=0
    plot_df.loc[(reg,'ParasympatheticNeurons'),'count']=0
# plot_df['mean'] = plot_df[celltype]/plot_df['count']
plot_df['count_log'] = np.log1p(plot_df['count'])

# plotting
plt.rcParams['axes.grid'] = False
plt.rcParams['figure.figsize'] = 1,3
vmax = 50
norm = mcolors.Normalize(vmin=0, vmax=vmax)
sns.scatterplot(data=plot_df,x='fine_grain',y='region_merged',edgecolor='black',
                hue='count', size='count', palette='Blues',size_norm=(0,vmax),sizes=(0, 200),hue_norm=norm,
                legend=False
               )
# Show the colorbar with the normalized hue scale
plt.colorbar(ScalarMappable(norm=norm, cmap="Blues"))
# plt.legend(loc=None, bbox_to_anchor=(4, 0.5))
plt.xticks(rotation=90)
plt.margins(x=0.6,y=0.1)
plt.title('Nuclei number')
plt.savefig(f"{figdir}/snRNAseq_dotplot_PSNS-SNS.pdf", bbox_inches="tight",dpi=300)
plt.show()
```

Example 40 (unknown):
```unknown
(181, 26660)
```

Example 41 (python):
```python
/tmp/ipykernel_152303/2641053010.py:28: MatplotlibDeprecationWarning: Unable to determine Axes to steal space for Colorbar. Using gca(), but will raise in the future. Either provide the *cax* argument to use as the Axes for the Colorbar, provide the *ax* argument to steal space from it, or add *mappable* to an Axes.
  plt.colorbar(ScalarMappable(norm=norm, cmap="Blues"))
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no7_manual-annotation_fibroblasts.html

**Contents:**
- Variables¶
- Read in data¶
- Remove mid-grain markers from the feature¶
- PCA, Clustering, UMAP¶
- Annotation¶
- Save¶

2543 rows × 7 columns

--> cluster 1 has low total_counts and bin_counts. and no specific marker expression --> will annotated as "unassigned"

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import celltypist
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 4 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 5 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets'
sample_id = 'HEA_FOET14880396'
celltype = 'Fibroblasts'
```

Example 6 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 7 (python):
```python
# read in
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 20318 × 17926
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
# read in
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0
                              )
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group']!=celltype]
midgrain_markers
```

Example 10 (python):
```python
# remove
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]
bdata
```

Example 11 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 20318 × 16642
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 12 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata,min_dist=0.5)
sc.pl.embedding(adata,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 13 (unknown):
```unknown
number of hvgs: 3277
```

Example 14 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=['celltypist_midmod2fine', 'conf_score_midmod2fine'],
                wspace=0.7)
```

Example 15 (python):
```python
'SLC18A3' in adata.var_names
```

Example 16 (python):
```python
# markers for fine-grained cell types
sc_markers = pd.read_csv(f'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_{celltype}_finegrain_markers.csv',
                         index_col=0)
markers = {}
n_top = 10
for c in set(sc_markers['group']):
    markers_sub = sc_markers[sc_markers['group']==c]
    markers_sub.sort_values('logfoldchanges',ascending=False)
    markers[c]=list(markers_sub[:n_top]['names'])

genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in adata.var_names)&(x not in genes)]
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in adata.var_names]
genes = list(set(genes))
len(genes)
```

Example 17 (python):
```python
sc.pl.embedding(adata,basis='X_umap',color=genes,
               ncols=6, wspace=0.2,cmap='RdPu',vmax='p99.9')
```

Example 18 (unknown):
```unknown
resolution_sel = 0.6
```

Example 19 (python):
```python
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in adata.var_names]

sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              markers, 
              groupby=f'leiden_{str(resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 20 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: Myofibroblasts, MyocardialInterstitialFibroblasts, GreatVesselAdventitialFibroblasts, etc.
```

Example 21 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 22 (unknown):
```unknown
[1.59681474 1.90298047 2.95755701 1.65405184 3.1673657 ]
```

Example 23 (python):
```python
# DEG analysis without other cell type markers
# with removing mid_grain markers
# remove mid_grain markers
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]

print(bdata.X.data[:5])
bdata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(bdata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(bdata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(bdata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 24 (unknown):
```unknown
[1.59681474 2.95755701 1.65405184 3.1673657  1.84266849]
```

Example 25 (python):
```python
sc.pl.dotplot(adata,
              ['COL1A1','COL1A2','COL3A1','IL11'], 
              groupby=f'leiden_{str(resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 26 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=f'leiden_{str(resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 27 (python):
```python
sc.pl.violin(adata,keys=['total_counts','bin_count'],groupby=f'leiden_{str(resolution_sel)}')
```

Example 28 (unknown):
```unknown
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 29 (python):
```python
# annotate
adata.obs['fine_grain_manual'] = adata.obs[f'leiden_{str(resolution_sel)}'].copy()
adata.obs.replace({'fine_grain_manual':{
    '0':'Fibroblasts',
    '1':'unassigned',
    '2':'ValveInterstitialCells',
    '3':'Fibroblasts',
    '4':'FibroblastsCOL2A1pos',
     '5':'Fibroblasts_DoubletsWithCMs',
}},inplace=True)
adata.obs['fine_grain_manual'].value_counts()
```

Example 30 (unknown):
```unknown
fine_grain_manual
Fibroblasts                    7849
unassigned                     5380
ValveInterstitialCells         5186
FibroblastsCOL2A1pos           1648
Fibroblasts_DoubletsWithCMs     255
Name: count, dtype: int64
```

Example 31 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 32 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby='fine_grain_manual')
sc.tl.dendrogram(adata, groupby='fine_grain_manual')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 33 (unknown):
```unknown
[1.59681474 1.90298047 2.95755701 1.65405184 3.1673657 ]
```

Example 34 (python):
```python
# save
adata.write(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
print(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
```

Example 35 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets/HEA_FOET14880396_Fibroblasts_b2c_cells_filtered_lognorm.h5ad
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/Visium/Others/plotting_multiple_LRaCMs.html

**Contents:**
- Loading packages¶
- Function¶
- Read in adata¶
- Plotting¶

**Examples:**

Example 1 (python):
```python
# this is for the jupyter notebook at JupyterLab
# to make sys.path between the terminal and Jupyter same
import sys

print(sys.path) # especially the path to site-packages are different... so thats why couldn't find correct package...

# remove weird paths...
sys.path = sys.path[:4]+[sys.path[6]]
print(sys.path)
```

Example 2 (unknown):
```unknown
['/nfs/team205/kk18/notebooks/P48_Foetal-heart_Visium/notebooks/notebooks_for_manuscript', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python39.zip', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/lib-dynload', '', '/nfs/users/nfs_k/kk18/.local/lib/python3.9/site-packages', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages']
['/nfs/team205/kk18/notebooks/P48_Foetal-heart_Visium/notebooks/notebooks_for_manuscript', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python39.zip', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/lib-dynload', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages']
```

Example 3 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import seaborn as sns

from cell2location.plt import plot_spatial

import os
```

Example 4 (python):
```python
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Global seed set to 0
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
```

Example 5 (python):
```python
import session_info
session_info.show()
```

Example 6 (python):
```python
-----
anndata             0.9.1
cell2location       0.1.3
matplotlib          3.7.1
numpy               1.24.3
pandas              2.0.1
scanpy              1.9.3
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 7 (python):
```python
PIL                 9.5.0
absl                NA
asttokens           NA
attr                23.1.0
backcall            0.2.0
certifi             2023.05.07
cffi                1.15.1
charset_normalizer  3.1.0
chex                0.1.7
comm                0.1.3
contextlib2         NA
cycler              0.10.0
cython_runtime      NA
dateutil            2.8.2
debugpy             1.6.7
decorator           5.1.1
defusedxml          0.7.1
docrep              0.3.2
etils               1.2.0
executing           1.2.0
flax                0.6.10
fsspec              2023.5.0
google              NA
h5py                3.8.0
idna                3.4
igraph              0.10.4
importlib_metadata  NA
importlib_resources NA
ipykernel           6.23.0
ipython_genutils    0.2.0
jax                 0.4.9
jaxlib              0.4.9
jedi                0.18.2
joblib              1.2.0
kiwisolver          1.4.4
leidenalg           0.9.1
lightning_fabric    1.9.5
lightning_utilities 0.8.0
llvmlite            0.40.0
ml_collections      NA
ml_dtypes           0.1.0
mpl_toolkits        NA
mpmath              1.3.0
msgpack             1.0.5
mudata              0.2.2
multipledispatch    0.6.0
natsort             8.3.1
numba               0.57.0
numpyro             0.11.0
nvfuser             NA
opt_einsum          v3.3.0
optax               0.1.5
packaging           23.1
parso               0.8.3
patsy               0.5.3
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
platformdirs        3.5.0
prompt_toolkit      3.0.38
psutil              5.9.5
ptyprocess          0.7.0
pure_eval           0.2.2
pycparser           2.21
pydev_ipython       NA
pydevconsole        NA
pydevd              2.9.5
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.15.1
pynndescent         0.5.10
pyparsing           3.0.9
pyro                1.8.4
pytorch_lightning   1.9.5
pytz                2023.3
requests            2.30.0
rich                NA
scipy               1.10.1
scvi                0.20.3
setuptools          66.0.0
six                 1.16.0
sklearn             1.2.2
stack_data          0.6.2
statsmodels         0.14.0
sympy               1.12
texttable           1.6.7
threadpoolctl       3.1.0
toolz               0.12.0
torch               2.0.1+cu117
torchmetrics        0.11.4
tornado             6.3.1
tqdm                4.65.0
traitlets           5.9.0
tree                0.1.8
typing_extensions   NA
umap                0.5.3
urllib3             2.0.2
wcwidth             0.2.6
yaml                6.0
zipp                NA
zmq                 25.0.2
zoneinfo            NA
```

Example 8 (unknown):
```unknown
-----
IPython             8.13.2
jupyter_client      8.2.0
jupyter_core        5.3.0
notebook            6.5.4
-----
Python 3.9.16 (main, Mar  8 2023, 14:00:05) [GCC 11.2.0]
Linux-5.4.0-169-generic-x86_64-with-glibc2.27
-----
Session information updated at 2024-02-12 20:45
```

Example 9 (python):
```python
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 10 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots'
```

Example 11 (python):
```python
# This function selects the data for multiple slides from the spatial anndata object.
# Modified `select_slide` in cell2loc for multiple slides

# @param adata: anndata object with multiple spatial experiments
# @param s: list of selected slides
# @param s_col: column in adata.obs listing experiment name for each location

def select_multislide(adata, s, s_col='sample'):

    slides = adata[adata.obs[s_col].isin(s), :]
    s_keys = list(slides.uns['spatial'].keys())
    
    uns_dict = {}
    for ss in s:
        s_spatial=np.array(s_keys)[[ss in k for k in s_keys]][0]
        uns_dict[s_spatial]=slides.uns['spatial'][s_spatial]
    
    slides.uns['spatial'] = uns_dict

    return slides
```

Example 12 (python):
```python
adata_vis = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-FFPE_foetal_raw.h5ad')

# add abundance to .obs
df = adata_vis.obsm['q05_cell_abundance_w_sf'].copy()
## rename columns
df.columns = [x.replace('q05cell_abundance_w_sf_','') for x in df.columns]
adata_vis.obs[df.columns]=df.copy()

# set symboßl as var_names
adata_vis.var.set_index('SYMBOL',inplace=True)

adata_vis
```

Example 13 (unknown):
```unknown
AnnData object with n_obs × n_vars = 19973 × 18085
    obs: 'in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'JC_edit', 'AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight', 'VentricularCardiomyocytesLeftCompact', 'VentricularCardiomyocytesRightCompact', 'VentricularCardiomyocytesLeftTrabeculated', 'VentricularCardiomyocytesRightTrabeculated', 'SinoatrialNodeCardiomyocytes', 'AtrioventricularNodeCardiomyocytes', 'VentricularConductionSystemProximal', 'VentricularConductionSystemDistal', 'GreatVesselAdventitialFibroblasts', 'CoronaryVesselAdventitialFibroblasts', 'MyocardialInterstitialFibroblasts1', 'MyocardialInterstitialFibroblasts2', 'Myofibroblasts', 'LymphNodeFibroblasticReticularCells', 'ValveInterstitialCells', 'GreatVesselSmoothMuscleCells', 'CoronarySmoothMuscleCells', 'CoronaryPericytes', 'GreatVesselArterialEndothelialCells', 'GreatVesselVenousEndothelialCells', 'CoronaryArterialEndothelialCells', 'CoronaryVenousEndothelialCells', 'CoronaryCapillaryEndothelialCells', 'EndocardialCells', 'EndocardialCushionCells', 'ValveEndothelialCells', 'LymphaticEndothelialCells', 'MesothelialEpicardialCells', 'EpicardiumDerivedCells', 'NeuronPrecursors', 'ChromaffinCells', 'SympatheticNeurons', 'ParaympatheticNeurons', 'SchwannCellPrecursors', 'SchwannCells', 'MonocytesMPOpos', 'Monocytes', 'MonocyteDerivedCells', 'MacrophagesCX3CR1pos', 'MacrophagesTIMD4pos', 'MacrophagesLYVE1pos', 'MacrophagesATF3pos', 'DendriticCellsType1', 'DendriticCellsMature', 'PlasmacytoidDendriticCells', 'MastCells', 'Megakaryocytes', 'TCellsCD4pos', 'TCellsCD8pos', 'TregsCD4pos', 'ProBCells', 'BCells', 'BCellsMS4A1pos', 'NaturalKillerCells', 'InnateLymphoidCells'
    var: 'gene_ids', 'feature_types', 'genome', 'mt'
    uns: 'spatial'
    obsm: 'MT', 'means_cell_abundance_w_sf', 'q05_cell_abundance_w_sf', 'q95_cell_abundance_w_sf', 'spatial', 'stds_cell_abundance_w_sf'
```

Example 14 (python):
```python
# select up to 6 clusters
# https://github.com/BayraktarLab/cell2location/blob/bd2aae833f81d8ab68684aa44f863b36e3af0aed/cell2location/plt/plot_spatial.py#L108

slide_id = 'BHF_F_Hea13295830'

slide = select_multislide(adata_vis, [slide_id])

clust_labels = ['AtrialCardiomyocytesLeft',
                'AtrialCardiomyocytesRight'
                ]
clust_col = ['' + str(i) for i in clust_labels] # in case column names differ from labels

plt.rcParams['axes.grid'] = False
with mpl.rc_context({'figure.figsize': (15, 15)}):
    fig = plot_spatial(
        adata=slide,
        # labels to show on a plot
        color=clust_col, labels=clust_labels,
        show_img=True, img_alpha=0.5,
        # 'fast' (white background) or 'dark_background'
        style='fast',
        # limit color scale at 99.2% quantile of cell abundance
        max_color_quantile=0.997,
        # size of locations (adjust depending on figure size)
        circle_diameter=3,
        colorbar_position='bottom',
        colorbar_shape = {'vertical_gaps': 0.1, 'horizontal_gaps': 5,
                                    'width': 1, 'height': 0.02},
        # crop_x=(650,820),
        # crop_y=(1200,1420),
        reorder_cmap = [1,4] # 'yellow' 'orange' 'blue' 'green' 'purple' 'grey' 'white'
    )
    plt.savefig(f"{figdir}/visium_cell2loc_LRaCMs_{slide_id}.pdf", bbox_inches="tight",dpi=300)
```

Example 15 (unknown):
```unknown
/tmp/ipykernel_42878/2805736760.py:18: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  slides.uns['spatial'] = uns_dict
```

Example 16 (unknown):
```unknown
f"{figdir}/visium_cell2loc_LRaCMs_{slide_id}.pdf"
```

Example 17 (unknown):
```unknown
'/nfs/team205/heart/fetal_ms_figures/raw_plots/visium_cell2loc_LRaCMs_BHF_F_Hea13295830.pdf'
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no7_manual-annotation_EpicardialCells.html

**Contents:**
- Variables¶
- Read in data¶
- Remove mid-grain markers from the feature¶
- PCA, Clustering, UMAP¶
- Markers¶
- Save¶

2392 rows × 7 columns

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import celltypist
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 4 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 5 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets'
sample_id = 'HEA_FOET14880396'
celltype = 'EpicardialCells'
```

Example 6 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 7 (python):
```python
# read in
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 603 × 13050
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
# read in
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0
                              )
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group']!=celltype]
midgrain_markers
```

Example 10 (python):
```python
# remove
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]
bdata
```

Example 11 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 603 × 11843
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 12 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata,min_dist=0.5)
sc.pl.embedding(adata,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 13 (unknown):
```unknown
number of hvgs: 3273
```

Example 14 (python):
```python
# read in markers identified using single-nucleus data
markers_df = pd.read_csv(f'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_{celltype}_finegrain_markers.csv')

mask1 = markers_df['group_mean']>1
mask2 = markers_df['logfoldchanges']>1
markers_df = markers_df[mask1&mask2]

markers_df
```

Example 15 (python):
```python
markers = {}
n_top = 10
for c in set(markers_df['group']):
    markers_df_sub = markers_df[markers_df['group']==c]
    markers_df_sub.sort_values('logfoldchanges',ascending=False)
    markers[c]=list(markers_df_sub[:n_top]['names'])

# manually add genes
markers['Cycling'] = ['MKI67','TOP2A'] 

genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in list(adata.var_names))&(x not in genes)]
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in list(adata.var_names)]
len(genes)
```

Example 16 (python):
```python
sc.pl.embedding(adata,basis='X_umap',color=genes, # size=30,
               ncols=6, wspace=0.2,cmap='RdPu',vmax='p100')
```

Example 17 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=['celltypist_midmod2fine', 'conf_score_midmod2fine'],
                wspace=0.7)
```

Example 18 (python):
```python
# umap per celltype to highlight
group_col = 'celltypist_midmod2fine'
groups_to_plot = list(set(adata.obs[group_col]))

plt.rcParams['axes.grid'] = False
fig,axs=plt.subplots(nrows=2,ncols=4,figsize=(15,7))
# Flatten the axs array for easier iteration in case of multiple rows and columns
axs_flat = axs.flatten()

for i,grp in enumerate(groups_to_plot):
    sc.pl.umap(adata, color=[group_col], groups=grp, 
               size=80, frameon=False,
               show=False, ax=axs_flat[i], legend_loc=None, legend_fontsize=5,
               title=grp
              )
# Turn off any remaining subplots
for j in range(len(groups_to_plot), len(axs_flat)):
    fig.delaxes(axs_flat[j])

# fig.suptitle('Main Title for the Figure')
fig.tight_layout()
```

Example 19 (unknown):
```unknown
resolution_sel = 1
```

Example 20 (python):
```python
markers_others = {
    'CM': ['TTN', 'TNNT2','MYH6',
        'MYH7', 'MYL2', 'FHL2', # vCN
            'NPPA', 'MYL7', 'MYL4'], #aCM
    'FB': ['DCN', 'GCN', 'PDGFRA','COL1A1','COL1A2'],
    'EC': ['VWF', 'PECAM1', 'CDH5','RGCC', 'FABP5',],
    'Peri': ['RGS5', 'ABCC9', 'KCNJ8'],
    'SMC': ['MYH11', 'TAGLN', 'ACTA2'],
    'Meso': ['MSLN', 'WT1', 'BNC1'],
    'Neuro': ['PLP1', 'NRXN1', 'NRXN3','PRPH', 'NEFL', 'NEFM', 'NEFH', 'STMN2', 'SST', 'SLC10A4', 'SLC18A3', 'TH', 'NPY'],
    'Adipo': ['GPAM', 'FASN', 'LEP'],
    'Myelo': ['CD14', 'C1QA', 'CD68','LYVE1','TIMD4'],
    'Lymph': ['CD8A', 'IL7R', 'CD40LG'],
    'B': ['CD19','SDC1','IGHG1','IGHA1'],
    'Mast': ['KIT', 'CPA3'],
}

for key in markers_others.keys():
    markers_others[key]=[x for x in markers_others[key] if x in adata.var_names]

sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              markers_others, 
              groupby=f'leiden_{str(resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 21 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CM, FB, EC, etc.
```

Example 22 (python):
```python
sc.tl.dendrogram(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby=f'leiden_{str(resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 23 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: EpicardiumDerivedCells, MesothelialEpicardialCells, Cycling
```

Example 24 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 25 (unknown):
```unknown
[2.04256052 2.77787835 1.93707821 2.4739898  3.19327673]
```

Example 26 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True, n_genes=5)
dp.add_totals().show()
```

Example 27 (unknown):
```unknown
[2.04256052 2.77787835 1.93707821 2.4739898  3.19327673]
```

Example 28 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=f'leiden_{str(resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 29 (python):
```python
sc.pl.violin(adata,keys=['total_counts','bin_count'],groupby=f'leiden_{str(resolution_sel)}')
```

Example 30 (unknown):
```unknown
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 31 (python):
```python
# annotate
adata.obs['fine_grain_manual'] = adata.obs[f'leiden_{str(resolution_sel)}'].copy()
adata.obs.replace({'fine_grain_manual':{
    '0':'MesothelialEpicardialCells', # contamination from CMs
    '1':'MesothelialEpicardialCells',
    '2':'EpicardiumDerivedCells',
    '3':'MesothelialEpicardialCells',
    '4':'MesothelialEpicardialCells', 
    '5':'MesothelialEpicardialCells',
}},inplace=True)
adata.obs['fine_grain_manual'].value_counts()
```

Example 32 (unknown):
```unknown
fine_grain_manual
MesothelialEpicardialCells    491
EpicardiumDerivedCells        112
Name: count, dtype: int64
```

Example 33 (python):
```python
ad = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad',backed='r')
ad = ad[ad.obs['mid_grain']==celltype]
set(ad.obs['fine_grain'])
```

Example 34 (unknown):
```unknown
{'EpicardiumDerivedCells', 'MesothelialEpicardialCells'}
```

Example 35 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 36 (python):
```python
markers['MesothelialEpicardialCells']=markers['MesothelialEpicardialCells']+['WT1','MSLN']

sc.tl.dendrogram(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 37 (unknown):
```unknown
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 38 (python):
```python
# save
adata.write(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
print(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
```

Example 39 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets/HEA_FOET14880396_EpicardialCells_b2c_cells_filtered_lognorm.h5ad
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no7_manual-annotation_lymphoid.html

**Contents:**
- Variables¶
- Read in data¶
- Remove mid-grain markers from the feature¶
- PCA, Clustering, UMAP¶
- Save¶

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import celltypist
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 4 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 5 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets'
sample_id = 'HEA_FOET14880396'
celltype = 'LymphoidCells'
```

Example 6 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 7 (python):
```python
# read in
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 87 × 10288
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
# read in
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0
                              )
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group']!=celltype]
midgrain_markers.head()
```

Example 10 (unknown):
```unknown
'GATA2' in list(midgrain_markers['names'])
```

Example 11 (python):
```python
# remove
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]
bdata
```

Example 12 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 87 × 9290
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 13 (python):
```python
# remove cell cycle genes
cell_cycle_genes = ['MCM5','PCNA','TYMS','FEN1','MCM2','MCM4','RRM1','UNG','GINS2','MCM6','CDCA7','DTL','PRIM1','UHRF1','MLF1IP','HELLS','RFC2','RPA2','NASP',
                'RAD51AP1','GMNN','WDR76','SLBP','CCNE2','UBR7','POLD3','MSH2','ATAD2','RAD51','RRM2','CDC45','CDC6','EXO1','TIPIN','DSCC1','BLM','CASP8AP2',
                'USP1','CLSPN','POLA1','CHAF1B','BRIP1','E2F8','HMGB2','CDK1','NUSAP1','UBE2C','BIRC5','TPX2','TOP2A','NDC80','CKS2','NUF2','CKS1B','MKI67',
                'TMPO','CENPF','TACC3','FAM64A','SMC4','CCNB2','CKAP2L','CKAP2','AURKB','BUB1','KIF11','ANP32E','TUBB4B','GTSE1','KIF20B','HJURP','CDCA3','HN1',
                'CDC20','TTK','CDC25C','KIF2C','RANGAP1','NCAPD2','DLGAP5','CDCA2','CDCA8','ECT2','KIF23','HMMR','AURKA','PSRC1','ANLN','LBR','CKAP5','CENPE','CTCF',
                'NEK2','G2E3','GAS2L3','CBX5','CENPA']
mask = [x not in cell_cycle_genes for x in bdata.var_names]
bdata = bdata[:,mask]
bdata.shape
```

Example 14 (unknown):
```unknown
'MYH11' in list(bdata.var_names)
```

Example 15 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata,min_dist=0.5)
sc.pl.embedding(adata,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
# leiden clustering
resolutions_list = [0.2,0.4,0.5,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 16 (unknown):
```unknown
number of hvgs: 2398
```

Example 17 (python):
```python
markers={
     'T cell': ['CD3E', 'CD3D','CD4', 'CD8A', 'CD8B','TRAC','TRBC1','TRDC','TRGC1'],
    'naive': ['TCF7', 'LEF1', 'SELL'],
    'CD4T_reg': ['FOXP3', 'CTLA4', 'IL2RA', 'IKZF2'],
    'cytotoxic': ['PRF1', 'GZMB', 'GZMH', 'NKG7'],
    'NK': ['KLRF1', 'KLRD1', 'NCAM1','FCGR3A','IFNG'],
    'ILC3': ['KIT', 'IL18R1','RORC','IL17A'],
    
    'B': ['CD19','CD20','CD22','CD79A', 'CD79B','MS4A1','IL7R',],
    '':['CCR7','TNFRSF13B',],
    'ProB':['KIT','SPN','CD34'],
    'B1':['CD27','CCR10','MS4A1','SPN'],
    'Plasma': ['CD27','CCR10','CD38','SDC1','JCAIN'],
    'Abs':['IGHM','IGHD', 'IGHG1','IGHG2', 'IGHG3','IGHG4', 'IGHA1','IGHE','IGLC1','IGKC'],
    'Gene editing': ['AICDA', 'RAG1','RAG2'],
    
    
    'Cycling':['MKI67','TOP2A'],

    'TFs': ['TBX21', 'GATA3', 'RORC', 'BCL6',],
}
genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in adata.var_names)&(x not in genes)]
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in adata.var_names]
sc.pl.embedding(adata,basis='X_umap',color=genes,
               ncols=6, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 18 (unknown):
```unknown
resolution_sel = 1
```

Example 19 (python):
```python
markers_others = {
    'CM': ['TTN', 'TNNT2','MYH6',
        'MYH7', 'MYL2', 'FHL2', # vCN
            'NPPA', 'MYL7', 'MYL4'], #aCM
    'FB': ['DCN', 'GCN', 'PDGFRA','COL1A1','COL1A2'],
    'EC': ['VWF', 'PECAM1', 'CDH5','RGCC', 'FABP5',],
    'Peri': ['RGS5', 'ABCC9', 'KCNJ8'],
    'SMC': ['MYH11', 'TAGLN', 'ACTA2'],
    'Meso': ['MSLN', 'WT1', 'BNC1'],
    'Neuro': ['PLP1', 'NRXN1', 'NRXN3','PRPH', 'NEFL', 'NEFM', 'NEFH', 'STMN2', 'SST', 'SLC10A4', 'SLC18A3', 'TH', 'NPY'],
    'Adipo': ['GPAM', 'FASN', 'LEP'],
    'Myelo': ['CD14', 'C1QA', 'CD68','LYVE1','TIMD4'],
    'Lymph': ['CD8A', 'IL7R', 'CD40LG'],
    'B': ['CD19','SDC1','IGHG1','IGHA1'],
    'Mast': ['KIT', 'CPA3'],
}

for key in markers_others.keys():
    markers_others[key]=[x for x in markers_others[key] if x in adata.var_names]

sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              markers_others, 
              groupby=f'leiden_{str(resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 20 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CM, FB, EC, etc.
```

Example 21 (python):
```python
sc.tl.dendrogram(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby=f'leiden_{str(resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 22 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: T cell, naive, CD4T_reg, etc.
```

Example 23 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 24 (unknown):
```unknown
[1.27187559 1.95641568 1.75023687 1.96760567 2.7810601 ]
```

Example 25 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True, n_genes=5)
dp.add_totals().show()
```

Example 26 (unknown):
```unknown
[1.27187559 1.95641568 1.75023687 1.96760567 2.7810601 ]
```

Example 27 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=f'leiden_{str(resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 28 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=['celltypist_midmod2fine', 'conf_score_midmod2fine'],
                wspace=0.7)
```

Example 29 (python):
```python
sc.pl.violin(adata,keys=['total_counts','bin_count'],groupby=f'leiden_{str(resolution_sel)}')
```

Example 30 (unknown):
```unknown
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 31 (python):
```python
# annotate
adata.obs['fine_grain_manual'] = adata.obs[f'leiden_{str(resolution_sel)}'].copy()
adata.obs.replace({'fine_grain_manual':{
    '0':'InnateLymphoidCells',
    '1':'InnateLymphoidCells',
    '2':'InnateLymphoidCells',
    '3':'BCells',
    '4':'InnateLymphoidCells',
}},inplace=True)
adata.obs['fine_grain_manual'].value_counts()
```

Example 32 (unknown):
```unknown
fine_grain_manual
InnateLymphoidCells    69
BCells                 18
Name: count, dtype: int64
```

Example 33 (python):
```python
ad = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad',backed='r')
ad[ad.obs['mid_grain']==celltype].obs['fine_grain'].value_counts()
```

Example 34 (unknown):
```unknown
fine_grain
TCellsCD4pos           5660
BCellsMS4A1pos         3470
NaturalKillerCells     2483
TCellsCD8pos           2297
TregsCD4pos            2096
BCells                 2058
InnateLymphoidCells    1719
ProBCells              1301
Name: count, dtype: int64
```

Example 35 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 36 (python):
```python
sc.tl.dendrogram(adata,groupby='fine_grain_manual')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 37 (unknown):
```unknown
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 38 (python):
```python
# save
adata.write(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
print(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig2_Overview_of_cellular_niches_and_TissueTypist/tissuetypist_assess_VisiumHD.html

**Contents:**
- Import modules¶
- Read in qeury adata¶
- Read in reference adata¶
- Harmonise cell type names¶
- Calculate cell type proportions per niche¶
  - Query¶
  - Reference¶
- Subset shared cell types¶

**Examples:**

Example 1 (python):
```python
import numpy as np
import pandas as pd
import scanpy as sc
import anndata
import os
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

sc.settings.verbosity = 1  # verbosity: errors (0), warnings (1), info (2), hints (3)
```

Example 2 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/rfs/project/rfs-iCNyzSAaucw/kk837/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 3 (unknown):
```unknown
figdir = '/home/kk837/rfs/rfs-teichlab-nfs-iCNyzSAaucw/kk837/notebooks/Foetal/fetal_ms_figures'
```

Example 4 (unknown):
```unknown
coord_columns = ('window_col', 'window_row')
list(coord_columns)
```

Example 5 (unknown):
```unknown
['window_col', 'window_row']
```

Example 6 (python):
```python
# read in 
sample_id = 'HEA_FOET14880396'
celltype_col = 'fine_grain_4Jan2025'
adata = sc.read_h5ad(f'/rfs/project/rfs-iCNyzSAaucw/kk837/data_objects/Foetal/VisiumHD/{sample_id}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 7 (unknown):
```unknown
AnnData object with n_obs × n_vars = 129803 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_4Jan2025', 'HistologicalAnnotation', 'SANaxis_landmarks', 'SAN_axis', 'SAN_axis_bin_annot', 'tt_prediction'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'HistologicalAnnotation_colors', 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'fine_grain_4Jan2025_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'spatial', 'tt_prediction_colors', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 8 (python):
```python
adata.X.data[:5]
```

Example 9 (unknown):
```unknown
array([1.27187559, 1.95641568, 1.75023687, 1.96760567, 2.7810601 ])
```

Example 10 (python):
```python
# update cell type name for plotting
df = pd.read_csv('/rfs/project/rfs-iCNyzSAaucw/kk837/data_objects/Foetal/finegrain_name_mapping.csv')
celltype_mapping = df.set_index('Full_name')['Short_name_finalised'].to_dict()
adata.obs.replace({celltype_col:{
    'SinoatrialNodePacemakerCellsHorn':'SinusHornPacemakerCells'
}},inplace=True)
adata.obs.replace({celltype_col:celltype_mapping},inplace=True)
```

Example 11 (python):
```python
/tmp/ipykernel_240910/4045961491.py:4: FutureWarning: The behavior of Series.replace (and DataFrame.replace) with CategoricalDtype is deprecated. In a future version, replace will only be used for cases that preserve the categories. To change the categories, use ser.cat.rename_categories instead.
  adata.obs.replace({celltype_col:{
/tmp/ipykernel_240910/4045961491.py:7: FutureWarning: The behavior of Series.replace (and DataFrame.replace) with CategoricalDtype is deprecated. In a future version, replace will only be used for cases that preserve the categories. To change the categories, use ser.cat.rename_categories instead.
  adata.obs.replace({celltype_col:celltype_mapping},inplace=True)
```

Example 12 (python):
```python
adata_ref = sc.read_h5ad('/rfs/project/rfs-iCNyzSAaucw/kk837/data_objects/Foetal/Visium/visium_foetal_healthy-all_cell2loc-adata.h5ad')
adata_ref
```

Example 13 (unknown):
```unknown
AnnData object with n_obs × n_vars = 39223 × 58
    obs: 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'cell2loc-map_group', 'JC_edit', 'sample_for_cell2loc', 'week', 'leiden', 'annotation_final', 'annotation_final_mod'
    uns: 'JC_edit_colors', 'annotation_colors', 'annotation_final_colors', 'cell2loc-map_group_colors', 'leiden', 'neighbors', 'region_cluster_colors', 'sample_for_cell2loc_colors', 'umap'
    obsm: 'MT', 'X_umap', 'means_cell_abundance_w_sf', 'q05_cell_abundance_w_sf', 'q05_cell_abundance_w_sf_ndarray', 'q95_cell_abundance_w_sf', 'spatial', 'stds_cell_abundance_w_sf'
    obsp: 'connectivities', 'distances'
```

Example 14 (python):
```python
# update cell type name
df = pd.read_csv('/rfs/project/rfs-iCNyzSAaucw/kk837/data_objects/Foetal/finegrain_name_mapping.csv')
celltype_mapping = df.set_index('Full_name')['Short_name_finalised'].to_dict()
adata_ref.var_names = [celltype_mapping[x] for x in adata_ref.var_names]
```

Example 15 (python):
```python
celltype_mapping_2 = {
    'BMS4':'B',
    'CorAdvFB':'FB',
    'GVAdvFB':'FB',
    'MonoMPO':'Mono',
    'MyoFB':'FB',
    'MyoFb':'FB',
     'ProB':'B',
    'SubEpiFb':'FB',
    'vCMCL':'vCMC',
    'vCMCR':'vCMC',
    'vCMTL':'vCMT',
    'vCMTR':'vCMT',
    'FBCOL2A1':'FB',
    'GVSMCP':'GVSMC',
    'GVSMCT':'GVSMC',
    'SANPCHd':'SANPC',
     'SHPC':'SANPC',
     'SANPCTl':'SANPC',
}
adata.obs[celltype_col] = [celltype_mapping_2[x] if x in celltype_mapping_2.keys() else x for x in adata.obs[celltype_col]]
adata_ref.var_names = [celltype_mapping_2[x] if x in celltype_mapping_2.keys() else x for x in adata_ref.var_names]
```

Example 16 (python):
```python
set(adata_ref.var_names) - set(adata.obs[celltype_col])
```

Example 17 (unknown):
```unknown
{'DCMat',
 'DCT1',
 'LNFRC',
 'MPATF3',
 'MPTIMD4',
 'MastC',
 'NK',
 'TCD4',
 'TCD8',
 'Treg',
 'pDC'}
```

Example 18 (python):
```python
set(adata.obs[celltype_col]) - set(adata_ref.var_names)
```

Example 19 (unknown):
```unknown
{'Eryth',
 'Fibroblasts_DoubletsWithCMs',
 'PericFib',
 'PericInt',
 'PericardialCells_DoubletsWithCMs',
 'SchwNeuro',
 'VentricularCardiomyocytes_Doublets',
 'VentricularCardiomyocytes_DoubletsWithErythrocytes',
 'unassigned'}
```

Example 20 (python):
```python
ctab = pd.crosstab(adata.obs[celltype_col],adata.obs['tt_prediction'])
prop_df = (ctab/ctab.sum(axis=0))
prop_df.sum(axis=0)
```

Example 21 (unknown):
```unknown
tt_prediction
AV ring                      1.0
AVnode                       1.0
Adventitia - GV              1.0
Atrium                       1.0
Coronary vessel              1.0
Ductus arteriosus            1.0
Endocardial cusion           1.0
Endocardium - Atrial         1.0
Endocardium - Ventricular    1.0
Epicardium                   1.0
Great vessel                 1.0
SAnode                       1.0
VCS - Distal                 1.0
VCS - Proximal               1.0
Valves                       1.0
Ventricle - Compact          1.0
Ventricle - Trabeculated     1.0
dtype: float64
```

Example 22 (unknown):
```unknown
prop_df.head()
```

Example 23 (python):
```python
df = adata_ref.to_df()
df['annotation_final_mod'] = adata_ref.obs['annotation_final_mod'].copy()
df = df.groupby('annotation_final_mod').sum()
df = df.groupby(df.columns, axis=1).sum()

prop_df_ref = (df.T/df.sum(axis=1))
prop_df_ref.sum(axis=0)
```

Example 24 (unknown):
```unknown
/tmp/ipykernel_240910/4260203183.py:3: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  df = df.groupby('annotation_final_mod').sum()
/tmp/ipykernel_240910/4260203183.py:4: FutureWarning: DataFrame.groupby with axis=1 is deprecated. Do `frame.T.groupby(...)` without axis instead.
  df = df.groupby(df.columns, axis=1).sum()
```

Example 25 (unknown):
```unknown
annotation_final_mod
Atrium                       1.0
Ventricle - Compact          1.0
Ventricle - Trabeculated     1.0
SAnode                       1.0
AVnode                       1.0
VCS - Proximal               1.0
VCS - Distal                 1.0
Great vessel                 1.0
Coronary vessel              1.0
Ductus arteriosus            1.0
Adventitia - Coronary        1.0
Adventitia - GV              1.0
Epicardium                   1.0
Endocardium - Atrial         1.0
Endocardium - Ventricular    1.0
Endocardial cusion           1.0
AV ring                      1.0
Valves                       1.0
LN                           1.0
dtype: float64
```

Example 26 (unknown):
```unknown
prop_df_ref.head()
```

Example 27 (python):
```python
# shared cell types
shared = [x for x in prop_df.index if x in prop_df_ref.index]
print(len(shared))
prop_df = prop_df.loc[shared]
prop_df_ref = prop_df_ref.loc[shared]
```

Example 28 (python):
```python
# shared niches
shared = [x for x in prop_df.columns if x in prop_df_ref.columns]
print(len(shared))
prop_df = prop_df[shared]
prop_df_ref = prop_df_ref[shared]
```

Example 29 (python):
```python
print(prop_df.shape)
print(prop_df_ref.shape)
```

Example 30 (unknown):
```unknown
(38, 17)
(38, 17)
```

Example 31 (python):
```python
%%time
corr, pval = utils.df_pearson_corr_with_p_vectorized(prop_df, prop_df_ref, p_adjust='fdr_bh')

# draw the initial clustermap (for ordering only)
cg = sns.clustermap(
    corr,
    cmap='vlag',
    center=0,
    metric='euclidean',
    method='average',
    figsize=(6, 6)
)

# 4) extract the leaf order and reindex both matrices
order = cg.dendrogram_row.reordered_ind
labels = list(corr.index[order])

corr_ord = corr.loc[labels, labels]
pval_ord = pval.loc[labels, labels]

# 5) replot as a heatmap and annotate significance
fig, ax = plt.subplots(figsize=(6, 6))
sns.heatmap(
    corr_ord,
    cmap='vlag',
    center=0,
    square=True,
    cbar_kws={'label': 'Pearson r'},
    ax=ax
)

# add asterisks for p < 0.01
n = corr_ord.shape[0]
for i in range(n):
    for j in range(n):
        if pval_ord.iat[i, j] < 0.01:
            ax.text(j + 0.5, i + 0.5, '*',
                    ha='center', va='center',
                    fontsize=10)

ax.set_xticklabels(labels, rotation=90)
ax.set_xlabel('Reference')
ax.set_yticklabels(labels, rotation=0)
ax.set_ylabel('Visium HD, TissueTypist prediction')
plt.tight_layout()
plt.savefig(f'{figdir}/TissueTypist_celltype-prop-corr_VisiumHD.pdf', bbox_inches="tight",dpi=300)
plt.show()
```

Example 32 (unknown):
```unknown
CPU times: user 932 ms, sys: 236 ms, total: 1.17 s
Wall time: 7.21 s
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no7_manual-annotation_neurons.html

**Contents:**
- Variables¶
- Read in data¶
- Remove mid-grain markers from the feature¶
- PCA, Clustering, UMAP¶
- Annotation¶
- Save¶

2445 rows × 7 columns

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import celltypist
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
-----
celltypist          1.6.3
matplotlib          3.9.2
numpy               2.0.2
pandas              2.2.3
scanpy              1.10.3
seaborn             0.13.2
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                 11.0.0
anndata             0.10.9
asttokens           NA
certifi             2024.08.30
charset_normalizer  3.4.0
comm                0.2.2
cycler              0.12.1
cython_runtime      NA
dateutil            2.9.0.post0
debugpy             1.8.9
decorator           5.1.1
exceptiongroup      1.2.2
executing           2.1.0
google              NA
h5py                3.12.1
idna                3.10
igraph              0.10.2
importlib_metadata  NA
importlib_resources NA
ipykernel           6.29.5
jedi                0.19.2
joblib              1.4.2
kiwisolver          1.4.7
legacy_api_wrap     NA
leidenalg           0.9.0
llvmlite            0.43.0
mpl_toolkits        NA
natsort             8.4.0
numba               0.60.0
packaging           24.2
parso               0.8.4
patsy               1.0.1
pexpect             4.9.0
platformdirs        4.3.6
prompt_toolkit      3.0.48
psutil              6.1.0
ptyprocess          0.7.0
pure_eval           0.2.3
pydev_ipython       NA
pydevconsole        NA
pydevd              3.2.3
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.18.0
pyparsing           3.2.0
pytz                2024.2
requests            2.32.3
scipy               1.13.1
six                 1.16.0
sklearn             1.5.2
stack_data          0.6.3
statsmodels         0.14.4
texttable           1.6.7
threadpoolctl       3.5.0
tornado             6.4.2
traitlets           5.14.3
typing_extensions   NA
urllib3             2.2.3
wcwidth             0.2.13
yaml                6.0.2
zipp                NA
zmq                 26.2.0
zoneinfo            NA
```

Example 5 (unknown):
```unknown
-----
IPython             8.18.1
jupyter_client      8.6.3
jupyter_core        5.7.2
-----
Python 3.9.20 | packaged by conda-forge | (main, Sep 30 2024, 17:49:10) [GCC 13.3.0]
Linux-5.15.0-126-generic-x86_64-with-glibc2.35
-----
Session information updated at 2024-12-07 22:32
```

Example 6 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 7 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 8 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets'
sample_id = 'HEA_FOET14880396'
celltype = 'Neurons'
```

Example 9 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 10 (python):
```python
# read in
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 11 (unknown):
```unknown
AnnData object with n_obs × n_vars = 589 × 14494
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 12 (python):
```python
# read in
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0
                              )
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group']!=celltype]
midgrain_markers
```

Example 13 (python):
```python
# remove
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]
bdata
```

Example 14 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 589 × 13304
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 15 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata,min_dist=0.5)
sc.pl.embedding(adata,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 16 (unknown):
```unknown
number of hvgs: 3519
```

Example 17 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
```

Example 18 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=['celltypist_midmod2fine', 'conf_score_midmod2fine'],
                wspace=0.7)
```

Example 19 (python):
```python
'SLC18A3' in adata.var_names
```

Example 20 (python):
```python
# markers for fine-grained cell types
markers = {
    "ChromaffinCells": [
        'DLK1',
        'EPAS1',
        'KCNJ6',
        "TH",    # Tyrosine hydroxylase
        "PNMT",  # Phenylethanolamine N-methyltransferase
        "CHGA",  # Chromogranin A
        "CHGB",  # Chromogranin B
        "PHOX2B"
    ],
    "NeuronPrecursors": [
        "NES",    # Nestin
        "PAX6",   # Paired box 6
        "TUBB3",  # Tubulin Beta 3 Class III
        "DCX",     # Doublecortin
        'ASCL1','NELL2','BNC2'
    ],
    "ParasympatheticNeurons": [
        "PHOX2B",
        "CHAT",    # Choline O-Acetyltransferase
        "SLC18A3", # Solute Carrier Family 18 Member A3 (Vesicular Acetylcholine Transporter)
        "GATA3",
        'NEFL','SST'
    ],
    "SympatheticNeurons": [
        "TH",
        "PHOX2B",
        "GATA3",
        "CHGA",
        "CHGB",
        'DBH','VSTM2L'
    ]
}

genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in adata.var_names)&(x not in genes)]
sc.pl.embedding(adata,basis='X_umap',color=genes,
               ncols=5, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 21 (unknown):
```unknown
resolution_sel = 1
```

Example 22 (python):
```python
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in adata.var_names]

sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              markers, 
              groupby=f'leiden_{str(resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 23 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: ChromaffinCells, NeuronPrecursors, ParasympatheticNeurons, etc.
```

Example 24 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 25 (unknown):
```unknown
[1.1620675  1.88821923 2.69821449 1.59010602 1.98053876]
```

Example 26 (python):
```python
# DEG analysis without other cell type markers
# with removing mid_grain markers
# remove mid_grain markers
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]

print(bdata.X.data[:5])
bdata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(bdata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(bdata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(bdata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 27 (unknown):
```unknown
[1.1620675  1.88821923 2.69821449 1.59010602 2.22351621]
```

Example 28 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=f'leiden_{str(resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 29 (python):
```python
sc.pl.violin(adata,keys=['total_counts','bin_count'],groupby=f'leiden_{str(resolution_sel)}')
```

Example 30 (unknown):
```unknown
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 31 (python):
```python
# annotate
adata.obs['fine_grain_manual'] = adata.obs[f'leiden_{str(resolution_sel)}'].copy()
adata.obs.replace({'fine_grain_manual':{
    '0':'ChromaffinCells',
    '1':'SympatheticNeurons',
    '2':'ChromaffinCells',
    '3':'ParasympatheticNeurons',
    '4':'NeuronPrecursors',
     '5':'ParasympatheticNeurons',
}},inplace=True)
adata.obs['fine_grain_manual'].value_counts()
```

Example 32 (unknown):
```unknown
fine_grain_manual
ChromaffinCells           265
ParasympatheticNeurons    153
SympatheticNeurons        110
NeuronPrecursors           61
Name: count, dtype: int64
```

Example 33 (python):
```python
ad = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad',backed='r')
set(ad.obs['fine_grain'])
```

Example 34 (unknown):
```unknown
{'AtrialCardiomyocytesCycling',
 'AtrialCardiomyocytesLeft',
 'AtrialCardiomyocytesRight',
 'AtrioventricularNodePacemakerCells',
 'BCells',
 'BCellsMS4A1pos',
 'ChromaffinCells',
 'CoronaryArterialEndothelialCells',
 'CoronaryCapillaryEndothelialCells',
 'CoronaryPericytes',
 'CoronarySmoothMuscleCells',
 'CoronaryVenousEndothelialCells',
 'CoronaryVesselAdventitialFibroblasts',
 'DendriticCellsMature',
 'DendriticCellsType1',
 'DuctusArteriosusSmoothMuscleCells',
 'EndocardialCells',
 'EndocardialCushionCells',
 'EpicardiumDerivedCells',
 'GreatVesselAdventitialFibroblasts',
 'GreatVesselArterialEndothelialCells',
 'GreatVesselSmoothMuscleCells',
 'GreatVesselVenousEndothelialCells',
 'InnateLymphoidCells',
 'LymphNodeFibroblasticReticularCells',
 'LymphaticEndothelialCells',
 'MacrophagesATF3pos',
 'MacrophagesCX3CR1pos',
 'MacrophagesLYVE1pos',
 'MacrophagesTIMD4pos',
 'MastCells',
 'Megakaryocytes',
 'MesothelialEpicardialCells',
 'MonocyteDerivedCells',
 'Monocytes',
 'MonocytesMPOpos',
 'MyocardialInterstitialFibroblasts',
 'Myofibroblasts',
 'NaturalKillerCells',
 'NeuronPrecursors',
 'ParasympatheticNeurons',
 'PericardialCellsFibrous',
 'PericardialCellsIntermediate',
 'PericardialCellsParietal',
 'PlasmacytoidDendriticCells',
 'ProBCells',
 'SchwannCellPrecursors',
 'SchwannCells',
 'SinoatrialNodePacemakerCells',
 'SubEpicardialFibroblasts',
 'SympatheticNeurons',
 'TCellsCD4pos',
 'TCellsCD8pos',
 'TregsCD4pos',
 'ValveEndothelialCells',
 'ValveInterstitialCells',
 'VentricularCardiomyocytesCycling',
 'VentricularCardiomyocytesLeftCompact',
 'VentricularCardiomyocytesLeftTrabeculated',
 'VentricularCardiomyocytesRightCompact',
 'VentricularCardiomyocytesRightTrabeculated',
 'VentricularConductionSystemDistal',
 'VentricularConductionSystemProximal'}
```

Example 35 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 36 (python):
```python
sc.tl.dendrogram(adata,groupby='fine_grain_manual')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 37 (python):
```python
# save
adata.write(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
print(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
```

Example 38 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets/HEA_FOET14880396_Neurons_b2c_cells_filtered_lognorm.h5ad
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/Visium/cell2location/wholeheart_with_DA/no1_reference-signatures_woPericardial.html

**Contents:**
- Loading packages¶
- Data, parameters¶
- Loading scRNA-seq reference data¶
- Remove cells¶
- Add Atria Ventricle key
- Estimation of reference cell type signatures (NB regression)¶

17277 rows × 58 columns

**Examples:**

Example 1 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl

import cell2location
import scvi

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text
import seaborn as sns
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
-----
anndata             0.7.8
cell2location       NA
matplotlib          3.5.0
numpy               1.20.3
pandas              1.3.4
scanpy              1.9.3
scvi                0.14.4
seaborn             0.11.2
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                         8.4.0
absl                        NA
adjustText                  NA
arviz                       0.10.0
attr                        21.2.0
backcall                    0.2.0
beta_ufunc                  NA
binom_ufunc                 NA
cffi                        1.15.0
cftime                      1.5.1.1
colorama                    0.4.4
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.5.1
decorator                   5.1.0
defusedxml                  0.7.1
deprecate                   0.3.0
descartes                   NA
dill                        0.3.4
docrep                      0.3.2
entrypoints                 0.3
fastprogress                0.2.7
fontTools                   4.28.1
fsspec                      2021.11.0
google                      NA
h5py                        3.6.0
igraph                      0.9.8
ipykernel                   6.5.1
ipython_genutils            0.2.0
ipywidgets                  7.6.5
jedi                        0.18.1
joblib                      1.1.0
kiwisolver                  1.3.2
lazylinker_ext              NA
leidenalg                   0.8.8
llvmlite                    0.37.0
matplotlib_inline           NA
mizani                      0.7.3
mpl_toolkits                NA
natsort                     8.0.0
nbinom_ufunc                NA
netCDF4                     1.5.8
numba                       0.54.1
numexpr                     2.7.3
opt_einsum                  v3.3.0
packaging                   21.3
palettable                  3.3.0
parso                       0.8.2
patsy                       0.5.2
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
plotnine                    0.8.0
prompt_toolkit              3.0.22
ptyprocess                  0.7.0
pycparser                   2.21
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.6.0
pydevd_concurrency_analyser NA
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pyexpat                     NA
pygments                    2.10.0
pymc3                       3.10.0
pynndescent                 0.5.5
pyparsing                   3.0.6
pyro                        1.7.0
pytorch_lightning           1.3.8
pytz                        2021.3
rich                        NA
scipy                       1.7.2
setuptools                  58.0.4
setuptools_scm              NA
six                         1.16.0
sklearn                     1.0.1
statsmodels                 0.13.1
storemagic                  NA
tensorboard                 2.7.0
texttable                   1.6.4
theano                      1.0.11
threadpoolctl               3.0.0
torch                       1.10.0+cu102
torchmetrics                0.6.0
tornado                     6.1
tqdm                        4.62.3
traitlets                   5.1.1
typing_extensions           NA
umap                        0.5.2
wcwidth                     0.2.5
xarray                      0.20.1
yaml                        5.4.1
zmq                         22.3.0
```

Example 5 (unknown):
```unknown
-----
IPython             7.29.0
jupyter_client      7.0.6
jupyter_core        4.9.1
notebook            6.4.6
-----
Python 3.9.7 (default, Sep 16 2021, 13:09:58) [GCC 7.5.0]
Linux-4.15.0-202-generic-x86_64-with-glibc2.27
-----
Session information updated at 2023-03-22 07:40
```

Example 6 (unknown):
```unknown
results_folder = '/nfs/team205/heart/cell2location/foetal'

# create paths and names to results folders for reference regression and cell2location models
ref_run_name = f'{results_folder}/reference_signatures/ver28Feb2023'
run_name = f'{results_folder}/cell2location_map/ver28Feb2023'
```

Example 7 (unknown):
```unknown
scRNA_path='/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw.h5ad'
celltype_key='fine_grain'
```

Example 8 (python):
```python
# Read scRNA data
adata = sc.read(scRNA_path)
adata
```

Example 9 (unknown):
```unknown
AnnData object with n_obs × n_vars = 297473 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'region', 'age', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'donor_by_library-prep', 'multiplexed', 'SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet', 'batch_key', '_scvi_batch', 'FACSgate', 'fine_grain', 'mid_grain', 'coarse_grain', 'sex', 'week', 'trimester', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', '_scvi_labels', 'stress_score', 'hb1_score'
    var: 'gene_name_scRNA-0', 'gene_id'
    uns: 'FACSgate_colors', '_scvi_manager_uuid', '_scvi_uuid', 'age_colors', 'cell_or_nuclei_colors', 'coarse_grain_colors', 'cycling_colors', 'donor_colors', 'fine_grain_colors', 'heart_or_greatvessels_colors', 'hvg', 'kit_10x_colors', 'mid_grain_colors', 'modality_colors', 'neighbors', 'phase_colors', 'region_colors', 'sex_colors', 'trimester_colors', 'umap', 'week_colors'
    obsm: 'X_scVI', 'X_umap', '_scvi_extra_categorical_covs', '_scvi_extra_continuous', '_scvi_extra_continuous_covs', 'latent_gene_encoding'
    obsp: 'connectivities', 'distances'
```

Example 10 (python):
```python
adata.X.data[:10]
```

Example 11 (unknown):
```unknown
array([ 1.,  1.,  5.,  5.,  8.,  4., 13.,  4.,  3.,  1.], dtype=float32)
```

Example 12 (python):
```python
set(adata.obs[celltype_key])
```

Example 13 (unknown):
```unknown
{'AtrialCardiomyocytesCycling',
 'AtrialCardiomyocytesLeft',
 'AtrialCardiomyocytesRight',
 'AtrioventricularNodeCardiomyocytes',
 'BCells',
 'BCellsMS4A1pos',
 'ChromaffinCells',
 'CoronaryArterialEndothelialCells',
 'CoronaryCapillaryEndothelialCells',
 'CoronaryPericytes',
 'CoronarySmoothMuscleCells',
 'CoronaryVenousEndothelialCells',
 'CoronaryVesselAdventitialFibroblasts',
 'DendriticCellsMature',
 'DendriticCellsType1',
 'DuctusArteriosusSmoothMuscleCells',
 'EndocardialCells',
 'EndocardialCushionCells',
 'EpicardiumDerivedCells',
 'GreatVesselAdventitialFibroblasts',
 'GreatVesselArterialEndothelialCells',
 'GreatVesselSmoothMuscleCells',
 'GreatVesselVenousEndothelialCells',
 'InnateLymphoidCells',
 'LymphNodeFibroblasticReticularCells',
 'LymphaticEndothelialCells',
 'MacrophagesATF3pos',
 'MacrophagesCX3CR1pos',
 'MacrophagesLYVE1pos',
 'MacrophagesTIMD4pos',
 'MastCells',
 'Megakaryocytes',
 'MesothelialEpicardialCells',
 'MonocyteDerivedCells',
 'Monocytes',
 'MonocytesMPOpos',
 'MyocardialInterstitialFibroblasts1',
 'MyocardialInterstitialFibroblasts2',
 'Myofibroblasts',
 'NaturalKillerCells',
 'NeuronPrecursors',
 'ParaympatheticNeurons',
 'PericardialCells1',
 'PericardialCells2',
 'PericardialCells3',
 'PlasmacytoidDendriticCells',
 'ProBCells',
 'SchwannCellPrecursors',
 'SchwannCells',
 'SinoatrialNodeCardiomyocytes',
 'SympatheticNeurons',
 'TCellsCD4pos',
 'TCellsCD8pos',
 'TregsCD4pos',
 'ValveEndothelialCells',
 'ValveInterstitialCells',
 'VentricularCardiomyocytesCycling',
 'VentricularCardiomyocytesLeftCompact',
 'VentricularCardiomyocytesLeftTrabeculated',
 'VentricularCardiomyocytesRightCompact',
 'VentricularCardiomyocytesRightTrabeculated',
 'VentricularConductionSystemDistal',
 'VentricularConductionSystemProximal'}
```

Example 14 (python):
```python
# Make sure .var_names are ENSEMBL IDs
adata.var.head()
```

Example 15 (python):
```python
adata.obs['batch_key'].value_counts()
```

Example 16 (unknown):
```unknown
Hst41_Nuclei_Multiome-v1    104878
Hst33_Nuclei_Multiome-v1     38234
Hst39_Nuclei_Multiome-v1     25386
C99_Cell_5prime-v2           17547
C92_Nuclei_Multiome-v1       13770
C94_Cell_5prime-v2           12985
Hst40_Nuclei_Multiome-v1      9708
C97_Cell_5prime-v2            9702
BRC2251_Cell_3prime-v3        9121
BRC2256_Cell_3prime-v3        7948
Hst36_Nuclei_Multiome-v1      7268
BRC2252_Cell_3prime-v3        6854
C86_Cell_5prime-v2            6376
BRC2260_Cell_3prime-v3        6004
BRC2263_Cell_3prime-v3        5662
BRC2262_Cell_3prime-v3        4129
C87_Nuclei_Multiome-v1        3907
C98_Nuclei_Multiome-v1        2660
C104_Nuclei_Multiome-v1       2490
C85_Nuclei_Multiome-v1        2436
C82_Nuclei_Multiome-v1         408
Name: batch_key, dtype: int64
```

Example 17 (python):
```python
set(adata.obs['batch_key'])
```

Example 18 (unknown):
```unknown
{'BRC2251_Cell_3prime-v3',
 'BRC2252_Cell_3prime-v3',
 'BRC2256_Cell_3prime-v3',
 'BRC2260_Cell_3prime-v3',
 'BRC2262_Cell_3prime-v3',
 'BRC2263_Cell_3prime-v3',
 'C104_Nuclei_Multiome-v1',
 'C82_Nuclei_Multiome-v1',
 'C85_Nuclei_Multiome-v1',
 'C86_Cell_5prime-v2',
 'C87_Nuclei_Multiome-v1',
 'C92_Nuclei_Multiome-v1',
 'C94_Cell_5prime-v2',
 'C97_Cell_5prime-v2',
 'C98_Nuclei_Multiome-v1',
 'C99_Cell_5prime-v2',
 'Hst33_Nuclei_Multiome-v1',
 'Hst36_Nuclei_Multiome-v1',
 'Hst39_Nuclei_Multiome-v1',
 'Hst40_Nuclei_Multiome-v1',
 'Hst41_Nuclei_Multiome-v1'}
```

Example 19 (python):
```python
# remove pericardial cells
adata = adata[adata.obs['fine_grain'].isin(['PericardialCells1','PericardialCells2','PericardialCells3'])==False]

# remove cycling cells
adata = adata[adata.obs['cycling']=='no']

adata.shape
```

Example 20 (unknown):
```unknown
(186909, 36601)
```

Example 21 (python):
```python
'''
atrial_or_ventricular = {
    # 'AtrialCardiomyocytesCycling':'atrial',
 'AtrialCardiomyocytesLeft':'atrial',
 'AtrialCardiomyocytesRight':'atrial',
 'AtrioventricularNodeCardiomyocytes':'either',
 'BCells':'either',
 'BCellsMS4A1pos':'either',
 'ChromaffinCells':'either',
 'CoronaryArterialEndothelialCells':'either',
 'CoronaryCapillaryEndothelialCells':'either',
 'CoronaryPericytes':'either',
 'CoronarySmoothMuscleCells':'either',
 'CoronaryVenousEndothelialCells':'either',
 'CoronaryVesselAdventitialFibroblasts':'either',
 'DendriticCellsMature':'either',
 'DendriticCellsType1':'either',
 'DuctusArteriosusSmoothMuscleCells':'atrial',
 'EndocardialCells':'either',
 'EndocardialCushionCells':'either',
 'EpicardiumDerivedCells':'either',
 'GreatVesselAdventitialFibroblasts':'atrial',
 'GreatVesselArterialEndothelialCells':'atrial',
 'GreatVesselSmoothMuscleCells':'atrial',
 'GreatVesselVenousEndothelialCells':'atrial',
 'InnateLymphoidCells':'either',
 'LymphNodeFibroblasticReticularCells':'either',
 'LymphaticEndothelialCells':'either',
 'MacrophagesATF3pos':'either',
 'MacrophagesCX3CR1pos':'either',
 'MacrophagesLYVE1pos':'either',
 'MacrophagesTIMD4pos':'either',
 'MastCells':'either',
 'Megakaryocytes':'either',
 'MesothelialEpicardialCells':'either',
 'MonocyteDerivedCells':'either',
 'Monocytes':'either',
 'MonocytesMPOpos':'either',
 'MyocardialInterstitialFibroblasts1':'either',
 'MyocardialInterstitialFibroblasts2':'either',
 'Myofibroblasts':'either',
 'NaturalKillerCells':'either',
 'NeuronPrecursors':'either',
 'ParaympatheticNeurons':'either',
 # 'PericardialCells1':'either',
 # 'PericardialCells2':'either',
 # 'PericardialCells3':'either',
 'PlasmacytoidDendriticCells':'either',
 'ProBCells':'either',
 'SchwannCellPrecursors':'either',
 'SchwannCells':'either',
 'SinoatrialNodeCardiomyocytes':'atrial',
 'SympatheticNeurons':'either',
 'TCellsCD4pos':'either',
 'TCellsCD8pos':'either',
 'TregsCD4pos':'either',
 'ValveEndothelialCells':'either',
 'ValveInterstitialCells':'either',
 # 'VentricularCardiomyocytesCycling':'ventricular',
 'VentricularCardiomyocytesLeftCompact':'ventricular',
 'VentricularCardiomyocytesLeftTrabeculated':'ventricular',
 'VentricularCardiomyocytesRightCompact':'ventricular',
 'VentricularCardiomyocytesRightTrabeculated':'ventricular',
 'VentricularConductionSystemDistal':'ventricular',
 'VentricularConductionSystemProximal':'ventricular'}

if set(adata.obs['fine_grain']) == set(atrial_or_ventricular.keys()):
    adata.obs['atrialorventricular']=adata.obs['fine_grain'].astype('str').copy()
    adata.obs.replace({'atrialorventricular':atrial_or_ventricular},inplace=True)
else:
    print('ERROR: the cell type names are not matching')

pd.set_option("max_rows", 100)
pd.crosstab(adata.obs['fine_grain'],adata.obs['atrialorventricular'])
'''
```

Example 22 (unknown):
```unknown
regions = ['whole_woPeriCell'] #,'atrial','ventricular'
```

Example 23 (python):
```python
%%time

from cell2location.utils.filtering import filter_genes
from cell2location.models import RegressionModel

for reg in regions:
    print(f'##### {reg} #####')
    
    '''
    # subset adata
    if reg == 'whole_woPeriCell':
        adata_ref = adata.copy()
    else:
        adata_ref = adata[adata.obs['atrialorventricular'].isin(['either',reg])]
    '''
    
    adata_ref = adata.copy()
    
    # show cell types
    adata_ref.obs['fine_grain'].value_counts()
    
    # before we estimate the reference cell type signature we recommend to perform very permissive genes selection
    # in this 2D histogram orange rectangle lays over excluded genes.
    # In this case, the downloaded dataset was already filtered using this method,
    # hence no density under the orange rectangle
    selected = filter_genes(adata_ref, cell_count_cutoff=5, cell_percentage_cutoff2=0.03, nonz_mean_cutoff=1.12)
    # filter the object
    adata_ref = adata_ref[:, selected].copy()

    # remove cell_states, which have only 2 cells or less
    counts = adata_ref.obs[celltype_key].value_counts()
    adata_ref = adata_ref[adata_ref.obs[celltype_key].isin(list(counts.keys()[counts>2]))]

    # prepare anndata for the regression model
    adata_ref=adata_ref.copy()
    scvi.data.setup_anndata(adata=adata_ref, 
                            # 10X reaction / sample / batch
                            batch_key='batch_key', 
                            # cell type, covariate used for constructing signatures
                            labels_key=celltype_key, 
                            # multiplicative technical effects (platform, 3' vs 5', donor effect)
                            # categorical_covariate_keys=['Method']
                           )
    scvi.data.view_anndata_setup(adata_ref)

    # create and train the regression model
    mod = RegressionModel(adata_ref) 

    # Use all data for training (validation not implemented yet, train_size=1)
    mod.train(max_epochs=1000, batch_size=2500, train_size=1, lr=0.002, use_gpu=True)

    # plot ELBO loss history during training, removing first 20 epochs from the plot
    mod.plot_history(20)

    # In this section, we export the estimated cell abundance (summary of the posterior distribution).
    adata_ref = mod.export_posterior(
        adata_ref, sample_kwargs={'num_samples': 1000, 'batch_size': 2500, 'use_gpu': True}
    )

    # Save model
    mod.save(f"{ref_run_name}/{reg}/", overwrite=True)

    # Save anndata object with results
    # adata_file = f"{ref_run_name}/{reg}/sc.h5ad"
    adata_ref.write(f"{ref_run_name}/{reg}/sc.h5ad")

    mod.plot_QC()

    # export estimated expression in each cluster
    if 'means_per_cluster_mu_fg' in adata_ref.varm.keys():
        inf_aver = adata_ref.varm['means_per_cluster_mu_fg'][[f'means_per_cluster_mu_fg_{i}' 
                                        for i in adata_ref.uns['mod']['factor_names']]].copy()
    else:
        inf_aver = adata_ref.var[[f'means_per_cluster_mu_fg_{i}' 
                                        for i in adata_ref.uns['mod']['factor_names']]].copy()
    inf_aver.columns = adata_ref.uns['mod']['factor_names']
    inf_aver.iloc[0:5, 0:5]

    # save inf_aver
    inf_aver.to_csv(f"{ref_run_name}/{reg}/inf_aver.csv")
    
    del adata_ref,mod,inf_aver,selected,counts
```

Example 24 (unknown):
```unknown
##### whole_woPeriCell #####
```

Example 25 (python):
```python
INFO     Using batches from adata.obs["batch_key"]                                           
INFO     Using labels from adata.obs["fine_grain"]                                           
INFO     Using data from adata.X
```

Example 26 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function setup_anndata is deprecated; Please use the model-specific setup_anndata methods instead. The global method will be removed in version 0.15.0.
```

Example 27 (python):
```python
INFO     Successfully registered anndata object containing 186909 cells, 17277 vars, 21      
         batches, 58 labels, and 0 proteins. Also registered 0 extra categorical covariates  
         and 0 extra continuous covariates.                                                  
INFO     Please do not further modify adata until model is trained.
```

Example 28 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function view_anndata_setup is deprecated; This method will be removed in 0.15.0. Please avoid building any new dependencies on it.
```

Example 29 (unknown):
```unknown
Anndata setup with scvi-tools version 0.14.4.
```

Example 30 (unknown):
```unknown
Data Summary               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃             Data             ┃ Count  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│            Cells             │ 186909 │
│             Vars             │ 17277  │
│            Labels            │   58   │
│           Batches            │   21   │
│           Proteins           │   0    │
│ Extra Categorical Covariates │   0    │
│ Extra Continuous Covariates  │   0    │
└──────────────────────────────┴────────┘
```

Example 31 (python):
```python
SCVI Data Registry              
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃     Data      ┃    scvi-tools Location    ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│       X       │          adata.X          │
│ batch_indices │ adata.obs['_scvi_batch']  │
│    labels     │ adata.obs['_scvi_labels'] │
└───────────────┴───────────────────────────┘
```

Example 32 (python):
```python
Label Categories                                       
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃     Source Location     ┃                 Categories                 ┃ scvi-tools Encodin ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['fine_grain'] │          AtrialCardiomyocytesLeft          │         0          │
│                         │         AtrialCardiomyocytesRight          │         1          │
│                         │    VentricularCardiomyocytesLeftCompact    │         2          │
│                         │   VentricularCardiomyocytesRightCompact    │         3          │
│                         │ VentricularCardiomyocytesLeftTrabeculated  │         4          │
│                         │ VentricularCardiomyocytesRightTrabeculated │         5          │
│                         │        SinoatrialNodeCardiomyocytes        │         6          │
│                         │     AtrioventricularNodeCardiomyocytes     │         7          │
│                         │    VentricularConductionSystemProximal     │         8          │
│                         │     VentricularConductionSystemDistal      │         9          │
│                         │     GreatVesselAdventitialFibroblasts      │         10         │
│                         │    CoronaryVesselAdventitialFibroblasts    │         11         │
│                         │     MyocardialInterstitialFibroblasts1     │         12         │
│                         │     MyocardialInterstitialFibroblasts2     │         13         │
│                         │               Myofibroblasts               │         14         │
│                         │    LymphNodeFibroblasticReticularCells     │         15         │
│                         │           ValveInterstitialCells           │         16         │
│                         │        GreatVesselSmoothMuscleCells        │         17         │
│                         │         CoronarySmoothMuscleCells          │         18         │
│                         │     DuctusArteriosusSmoothMuscleCells      │         19         │
│                         │             CoronaryPericytes              │         20         │
│                         │    GreatVesselArterialEndothelialCells     │         21         │
│                         │     GreatVesselVenousEndothelialCells      │         22         │
│                         │      CoronaryArterialEndothelialCells      │         23         │
│                         │       CoronaryVenousEndothelialCells       │         24         │
│                         │     CoronaryCapillaryEndothelialCells      │         25         │
│                         │              EndocardialCells              │         26         │
│                         │          EndocardialCushionCells           │         27         │
│                         │           ValveEndothelialCells            │         28         │
│                         │         LymphaticEndothelialCells          │         29         │
│                         │         MesothelialEpicardialCells         │         30         │
│                         │           EpicardiumDerivedCells           │         31         │
│                         │              NeuronPrecursors              │         32         │
│                         │              ChromaffinCells               │         33         │
│                         │             SympatheticNeurons             │         34         │
│                         │           ParaympatheticNeurons            │         35         │
│                         │           SchwannCellPrecursors            │         36         │
│                         │                SchwannCells                │         37         │
│                         │              MonocytesMPOpos               │         38         │
│                         │                 Monocytes                  │         39         │
│                         │            MonocyteDerivedCells            │         40         │
│                         │            MacrophagesCX3CR1pos            │         41         │
│                         │            MacrophagesTIMD4pos             │         42         │
│                         │            MacrophagesLYVE1pos             │         43         │
│                         │             MacrophagesATF3pos             │         44         │
│                         │            DendriticCellsType1             │         45         │
│                         │            DendriticCellsMature            │         46         │
│                         │         PlasmacytoidDendriticCells         │         47         │
│                         │                 MastCells                  │         48         │
│                         │               Megakaryocytes               │         49         │
│                         │                TCellsCD4pos                │         50         │
│                         │                TCellsCD8pos                │         51         │
│                         │                TregsCD4pos                 │         52         │
│                         │                 ProBCells                  │         53         │
│                         │                   BCells                   │         54         │
│                         │               BCellsMS4A1pos               │         55         │
│                         │             NaturalKillerCells             │         56         │
│                         │            InnateLymphoidCells             │         57         │
└─────────────────────────┴────────────────────────────────────────────┴────────────────────┘
```

Example 33 (python):
```python
Batch Categories                              
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃    Source Location     ┃        Categories        ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['batch_key'] │  BRC2251_Cell_3prime-v3  │          0          │
│                        │  BRC2252_Cell_3prime-v3  │          1          │
│                        │  BRC2256_Cell_3prime-v3  │          2          │
│                        │  BRC2260_Cell_3prime-v3  │          3          │
│                        │  BRC2262_Cell_3prime-v3  │          4          │
│                        │  BRC2263_Cell_3prime-v3  │          5          │
│                        │  C82_Nuclei_Multiome-v1  │          6          │
│                        │  C85_Nuclei_Multiome-v1  │          7          │
│                        │    C86_Cell_5prime-v2    │          8          │
│                        │  C87_Nuclei_Multiome-v1  │          9          │
│                        │  C92_Nuclei_Multiome-v1  │         10          │
│                        │    C94_Cell_5prime-v2    │         11          │
│                        │    C97_Cell_5prime-v2    │         12          │
│                        │  C98_Nuclei_Multiome-v1  │         13          │
│                        │    C99_Cell_5prime-v2    │         14          │
│                        │ C104_Nuclei_Multiome-v1  │         15          │
│                        │ Hst33_Nuclei_Multiome-v1 │         16          │
│                        │ Hst36_Nuclei_Multiome-v1 │         17          │
│                        │ Hst39_Nuclei_Multiome-v1 │         18          │
│                        │ Hst40_Nuclei_Multiome-v1 │         19          │
│                        │ Hst41_Nuclei_Multiome-v1 │         20          │
└────────────────────────┴──────────────────────────┴─────────────────────┘
```

Example 34 (unknown):
```unknown
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/configuration_validator.py:99: UserWarning: you passed in a val_dataloader but have no validation_step. Skipping val loop
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
```

Example 35 (unknown):
```unknown
Epoch 1000/1000: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [6:10:43<00:00, 22.24s/it, v_num=1, elbo_train=1.19e+9]
Sampling local variables, batch:   0%|                                                                                                                                                                                                  | 0/75 [00:02<?, ?it/s]
Sampling global variables, sample: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 999/999 [00:17<00:00, 58.03it/s]
```

Example 36 (unknown):
```unknown
CPU times: user 4h 39min, sys: 2h 58min 41s, total: 7h 37min 42s
Wall time: 6h 13min 10s
```

Example 37 (python):
```python
pd.read_csv(f"{ref_run_name}/{reg}/inf_aver.csv", index_col=0)
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no7_manual-annotation_glia.html

**Contents:**
- Variables¶
- Read in data¶
- Remove mid-grain markers from the feature¶
- PCA, Clustering, UMAP¶
- Annotation¶
- Save¶

2540 rows × 7 columns

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import celltypist
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 4 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 5 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets'
sample_id = 'HEA_FOET14880396'
celltype = 'Glia'
```

Example 6 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 7 (python):
```python
# read in
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 588 × 13981
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
# read in
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0
                              )
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group']!=celltype]
midgrain_markers
```

Example 10 (python):
```python
# remove
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]
bdata
```

Example 11 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 588 × 12746
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap', 'dendrogram_leiden_0.6', 'rank_genes_groups'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 12 (python):
```python
# remove cell cycle genes
cell_cycle_genes = ['MCM5','PCNA','TYMS','FEN1','MCM2','MCM4','RRM1','UNG','GINS2','MCM6','CDCA7','DTL','PRIM1','UHRF1','MLF1IP','HELLS','RFC2','RPA2','NASP',
                'RAD51AP1','GMNN','WDR76','SLBP','CCNE2','UBR7','POLD3','MSH2','ATAD2','RAD51','RRM2','CDC45','CDC6','EXO1','TIPIN','DSCC1','BLM','CASP8AP2',
                'USP1','CLSPN','POLA1','CHAF1B','BRIP1','E2F8','HMGB2','CDK1','NUSAP1','UBE2C','BIRC5','TPX2','TOP2A','NDC80','CKS2','NUF2','CKS1B','MKI67',
                'TMPO','CENPF','TACC3','FAM64A','SMC4','CCNB2','CKAP2L','CKAP2','AURKB','BUB1','KIF11','ANP32E','TUBB4B','GTSE1','KIF20B','HJURP','CDCA3','HN1',
                'CDC20','TTK','CDC25C','KIF2C','RANGAP1','NCAPD2','DLGAP5','CDCA2','CDCA8','ECT2','KIF23','HMMR','AURKA','PSRC1','ANLN','LBR','CKAP5','CENPE','CTCF',
                'NEK2','G2E3','GAS2L3','CBX5','CENPA']
mask = [x not in cell_cycle_genes for x in bdata.var_names]
bdata = bdata[:,mask]
bdata.shape
```

Example 13 (unknown):
```unknown
(588, 12652)
```

Example 14 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata,min_dist=0.5)
sc.pl.embedding(adata,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 15 (unknown):
```unknown
number of hvgs: 3367
```

Example 16 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=['celltypist_midmod2fine', 'conf_score_midmod2fine'],
                wspace=0.7)
```

Example 17 (python):
```python
'SLC18A3' in adata.var_names
```

Example 18 (python):
```python
# markers for fine-grained cell types
markers = {
    "astrocytes": ["GFAP", "S100B", "ALDH1L1"],
    "oligodendrocytes": ["OLIG2", "MBP", "PDGFRA"],
    "microglia": ["AIF1"],  # IBA1 protein is encoded by the AIF1 gene
    "schwann_cells": ["SOX10", "MPZ", "S100B"],
    "schwann_cell_precursors": [
                                "PLS3",'SLITRK6','IL1RAPL2'
                               ],
    'Cycling':['MKI67','TOP2A'],
    'Others':['NGF'],
    'Neuronal':['SYT1','PCSK1N']
}

genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in adata.var_names)&(x not in genes)]
sc.pl.embedding(adata,basis='X_umap',color=genes,
               ncols=5, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 19 (unknown):
```unknown
resolution_sel = 0.6
```

Example 20 (python):
```python
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in adata.var_names]

sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              markers, 
              groupby=f'leiden_{str(resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 21 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: astrocytes, oligodendrocytes, microglia, etc.
```

Example 22 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', n_genes=15, return_fig=True)
dp.add_totals().show()
```

Example 23 (unknown):
```unknown
[2.1933994  2.3171509  1.76055317 2.69164533 1.61733065]
```

Example 24 (python):
```python
# DEG analysis without other cell type markers
# with removing mid_grain markers
# remove mid_grain markers
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]

print(bdata.X.data[:5])
bdata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(bdata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(bdata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(bdata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 25 (unknown):
```unknown
[2.1933994  2.3171509  1.76055317 2.69164533 1.61733065]
```

Example 26 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=f'leiden_{str(resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 27 (python):
```python
sc.pl.violin(adata,keys=['total_counts','bin_count'],groupby=f'leiden_{str(resolution_sel)}')
```

Example 28 (unknown):
```unknown
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 29 (python):
```python
# annotate
adata.obs['fine_grain_manual'] = adata.obs[f'leiden_{str(resolution_sel)}'].copy()
adata.obs.replace({'fine_grain_manual':{
    '0':'SchwannCells',
    '1':'SchwannCellPrecursors',
    '2':'SchwannCells',
    '3':'SchwannCells_and_Neurons', # likely doublets with neurons
}},inplace=True)
adata.obs['fine_grain_manual'].value_counts()
```

Example 30 (unknown):
```unknown
fine_grain_manual
SchwannCells                354
SchwannCellPrecursors       163
SchwannCells_and_Neurons     71
Name: count, dtype: int64
```

Example 31 (python):
```python
ad = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad',backed='r')
ad[ad.obs['mid_grain']==celltype].obs['fine_grain'].value_counts()
```

Example 32 (unknown):
```unknown
fine_grain
SchwannCells             2263
SchwannCellPrecursors    1278
Name: count, dtype: int64
```

Example 33 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 34 (python):
```python
# markers for fine-grained cell types
markers = {
    "astrocytes": ["GFAP", "S100B", "ALDH1L1"],
    "oligodendrocytes": ["OLIG2", "MBP", "PDGFRA"],
    "microglia": ["AIF1"],  # IBA1 protein is encoded by the AIF1 gene
    "schwann_cells": ["SOX10", "MPZ", "S100B"],
    "schwann_cell_precursors": [
                                "PLS3",'SLITRK6','IL1RAPL2'
                               ],
    'Cycling':['MKI67','TOP2A'],
    'Others':['NGF'],
    'Neuronal':['SYT1','PCSK1N']
}
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in adata.var_names]

sc.tl.dendrogram(adata,groupby='fine_grain_manual')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 35 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: SchwannCells, SchwannCellPrecursors, SchwannCells_and_Neurons
var_group_labels: astrocytes, oligodendrocytes, microglia, etc.
```

Example 36 (python):
```python
# save
adata.write(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
print(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
```

Example 37 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets/HEA_FOET14880396_Glia_b2c_cells_filtered_lognorm.h5ad
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig2_Overview_of_cellular_niches_and_TissueTypist/tissuetypist_VisiumHD_map_tt-prediction-to-b2c.html

**Contents:**
- Read the objects¶
- Import annotations from loupe browser¶
- Plot¶
- Save cdata¶

311232 rows × 1 columns

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
/home/kk837/.conda/envs/generic_env/lib/python3.10/site-packages/session_info/main.py:213: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  mod_version = _find_version(mod.__version__)
```

Example 4 (python):
```python
-----
matplotlib          3.10.1
numpy               2.1.3
pandas              2.2.3
scanpy              1.11.1
seaborn             0.13.2
session_info        v1.0.1
-----
```

Example 5 (python):
```python
Cython              3.1.1
PIL                 11.1.0
anndata             0.11.4
asciitree           NA
asttokens           NA
charset_normalizer  3.4.1
cloudpickle         3.1.1
comm                0.2.2
cycler              0.12.1
cython              3.1.1
cython_runtime      NA
dask                2024.11.2
dateutil            2.9.0.post0
debugpy             1.8.13
decorator           5.2.1
exceptiongroup      1.2.2
executing           2.2.0
h5py                3.13.0
igraph              0.11.8
importlib_metadata  NA
ipykernel           6.29.5
jedi                0.19.2
jinja2              3.1.6
joblib              1.4.2
kiwisolver          1.4.8
legacy_api_wrap     NA
leidenalg           0.10.2
llvmlite            0.44.0
markupsafe          3.0.2
mpl_toolkits        NA
msgpack             1.1.0
natsort             8.4.0
numba               0.61.0
numcodecs           0.13.1
packaging           24.2
parso               0.8.4
patsy               1.0.1
platformdirs        4.3.7
prompt_toolkit      3.0.50
psutil              7.0.0
pure_eval           0.2.3
pyarrow             19.0.1
pydev_ipython       NA
pydevconsole        NA
pydevd              3.2.3
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.19.1
pyparsing           3.2.3
pytz                2025.2
scipy               1.15.2
six                 1.17.0
sklearn             1.5.2
stack_data          0.6.3
statsmodels         0.14.4
tblib               3.1.0
texttable           1.7.0
threadpoolctl       3.6.0
tlz                 1.0.0
toolz               1.0.0
tornado             6.4.2
traitlets           5.14.3
typing_extensions   NA
wcwidth             0.2.13
yaml                6.0.2
zarr                2.18.3
zipp                NA
zmq                 26.3.0
zoneinfo            NA
```

Example 6 (unknown):
```unknown
-----
IPython             8.34.0
jupyter_client      8.6.3
jupyter_core        5.7.2
-----
Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0]
Linux-4.18.0-553.53.1.el8_10.x86_64-x86_64-with-glibc2.28
-----
Session information updated at 2025-07-06 10:18
```

Example 7 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 8 (unknown):
```unknown
os.getcwd()
```

Example 9 (unknown):
```unknown
'/rfs/project/rfs-iCNyzSAaucw/kk837/notebooks/Foetal/TissueTypist/CamU/tissuetypist_package'
```

Example 10 (python):
```python
adata_dir = '/rfs/project/rfs-iCNyzSAaucw/kk837/data_objects/Foetal/VisiumHD'
sample_id = 'HEA_FOET14880396'
```

Example 11 (python):
```python
!ls {adata_dir}
```

Example 12 (unknown):
```unknown
archived-Dec2024_HEA_FOET14880396_b2c.h5ad
cellcharter
cellphonedb
for_Theo
HEA_AD14880301_b2c.h5ad
HEA_FOET14880396_8um-bins_filtered.h5ad
HEA_FOET14880396_8um-bins_filtered_lognorm.h5ad
HEA_FOET14880396_b2c_cells_filtered_celltype-selected_lognorm.h5ad
HEA_FOET14880396_b2c_cells_filtered.h5ad
HEA_FOET14880396_b2c_cells_filtered_lognorm.h5ad
HEA_FOET14880396_b2c_cells_filtered_lognorm_morpho_obs.csv
HEA_FOET14880396_b2c_cells.h5ad
HEA_FOET14880396_b2c.h5ad
HEA_FOET14880397_b2c_cells_filtered.h5ad
HEA_FOET14880397_b2c_cells_filtered_lognorm.h5ad
HEA_FOET14880397_b2c_cells.h5ad
HEA_FOET14880397_b2c.h5ad
subsets
```

Example 13 (python):
```python
# read in b2c object
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_b2c.h5ad') # read 2um object which has b2c cell grouping - see tutorial - https://nbviewer.org/github/Teichlab/bin2cell/blob/main/notebooks/demo.ipynb 
cdata = sc.read_h5ad(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad') # read b2c object
```

Example 14 (python):
```python
# read in 8um object to get tt_prediction results
adata_8um = sc.read_h5ad(f'{adata_dir}/{sample_id}_8um-bins_filtered.h5ad',backed='r')
adata_8um
```

Example 15 (unknown):
```unknown
AnnData object with n_obs × n_vars = 311232 × 18085 backed at '/rfs/project/rfs-iCNyzSAaucw/kk837/data_objects/Foetal/VisiumHD/HEA_FOET14880396_8um-bins_filtered.h5ad'
    obs: 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'n_genes', 'binned_8_row', 'binned_8_col', 'section', 'sliding_window_assignment', 'window_col', 'window_row', 'tt_prediction'
    var: 'gene_ids', 'feature_types', 'genome', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts'
```

Example 16 (python):
```python
anno_df = adata_8um.obs[['tt_prediction']].copy()
anno = 'tt_prediction'
cell_grouping = 'labels_joint'
```

Example 17 (python):
```python
def map_loupe_to_b2c(
        adata,
        cdata,
        anno_df,
        anno,
        cell_grouping = 'labels_he_expanded', 
        ):
     

    # Calculate the binned row and column indices in adata
    adata.obs['binned_8_row'] = adata.obs['array_row'] // 4
    adata.obs['binned_8_col'] = adata.obs['array_col'] // 4

    # Create a combined index in adata and set it as the index
    adata.obs['mapping_index_08'] = 's_008um_' + adata.obs['binned_8_row'].astype(str).str.zfill(5) + '_' + adata.obs['binned_8_col'].astype(str).str.zfill(5) + '-1'
    adata.obs['mapping_index_08'].value_counts() # most should have 16 but there are some on the tissue edge that actually have less due to partial overlap 

    # read annotations from df and map to the 2um object 
    print('annotations to map:')
    print(anno_df.head(5))
    mapping_dict = pd.Series(anno_df[anno].values,index=anno_df.index).to_dict()
    adata.obs[anno] = adata.obs['mapping_index_08'].map(mapping_dict) # update 2um object 

    # assign annotation to cells by pandas.Series.mode(). The mode is the value that appears most often which captures the most frequent annotation for that cell from 2um data
    filtered_data = adata.obs[(adata.obs[cell_grouping] != 0) & adata.obs[anno].notna()]
    cell_annotation = (
        filtered_data.groupby(cell_grouping)[anno]
        .apply(lambda x: x.mode().iloc[0] if not x.mode().empty else None)
    )
    cell_annotation.index = cell_annotation.index.astype('str')
    cdata.obs[anno] = cdata.obs.index.map(cell_annotation).astype('category') # label cells 
    print('Annotations in b2c object:')
    print(cdata.obs[anno].value_counts())

map_loupe_to_b2c(adata,
                 cdata,
                 anno_df,
                 anno,
                 cell_grouping)
```

Example 18 (unknown):
```unknown
annotations to map:
                                  tt_prediction
s_008um_00301_00321-1                   AV ring
s_008um_00602_00290-1              Great vessel
s_008um_00377_00750-1  Ventricle - Trabeculated
s_008um_00383_00696-1  Ventricle - Trabeculated
s_008um_00526_00291-1              Great vessel
Annotations in b2c object:
tt_prediction
Ventricle - Compact          43759
Adventitia - GV              24215
Atrium                       11901
Great vessel                 11489
Ventricle - Trabeculated      8742
VCS - Distal                  5460
VCS - Proximal                4430
AV ring                       4371
Endocardium - Atrial          2857
Epicardium                    2108
Ductus arteriosus             2050
Endocardium - Ventricular     1964
SAnode                        1559
Endocardial cusion            1264
AVnode                         674
Coronary vessel                625
Valves                         526
Name: count, dtype: int64
```

Example 19 (unknown):
```unknown
AnnotationPalette={ 
    'Atrium':'#eac075',
        'Ventricle - Compact':'#f1acfa',
     'Ventricle - Trabeculated':'#f09951',
        'SAnode':'#8A2BE2',
        'AVnode':'#062b4a',
    'VCS - Proximal':'#FFFF00',
        'VCS - Distal':'#7FFFD4',
        'Epicardium':'#01ccc8',
        'Endocardium - Atrial':'#DC143C',
        'Endocardium - Ventricular':'#f53527',
        'Endocardial cusion':'#ff7e97',
        'Great vessel':'#ff9b6e',
    'Ductus arteriosus':'#8a6800',
        'Coronary vessel':'#206601',
        'Adventitia - Coronary':'#CAFF70',
     'Adventitia - GV':'#005aa0',
        'AV ring':'#e478ff',
    'Valves':'#3300FF',
    'LN':'#9088ff',
}
```

Example 20 (python):
```python
sc.settings.set_figure_params(dpi=120)
sc.pl.spatial(cdata, 
                  color=anno, 
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=1.5,alpha_img=0.3,
                  frameon=False,palette=AnnotationPalette
                 )
```

Example 21 (python):
```python
/tmp/ipykernel_237158/3235274587.py:2: FutureWarning: Use `squidpy.pl.spatial_scatter` instead.
  sc.pl.spatial(cdata,
```

Example 22 (python):
```python
cdata.write(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad')
cdata
```

Example 23 (unknown):
```unknown
AnnData object with n_obs × n_vars = 129803 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_4Jan2025', 'HistologicalAnnotation', 'SANaxis_landmarks', 'SAN_axis', 'SAN_axis_bin_annot', 'tt_prediction'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'HistologicalAnnotation_colors', 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'fine_grain_4Jan2025_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'spatial', 'umap', 'tt_prediction_colors'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no3_bin2cell_GEX-segmentation_sigma5.html

**Contents:**
- Read in data¶
- Cell detection using gene expression data¶
- Joint HE and GEX labels¶
- ~Remove too small cells~¶
- Group bins into cells!¶
- Save¶

The H&E segmentation is not guaranteed to be perfect. There may be some regions that have expression data but lack a visible nuclei to seed a cell. Alternately, nuclei may take on unusual shapes and not be detected by the model. Performing segmentation on a representation of total expression per bin may be able to detect some of the missed cells. However, this kind of segmentation only performs well on sparse tissue, and struggles to differentiate individual cells in dense regions. As such, we use it as secondary means of object identification, preferring H&E where able.

The input image is going to be a representation of the total counts per bin, with a Gaussian filter with a sigma of 5 (measured in pixels) applied for a little smoothing.

The segmentation is performed via StarDist's fluorescence model, and identifies cells rather than nuclei, meaning subsequent label expasion is not necessary. Just like with the H&E model, lowering prob_thresh to make the calls less stringent is recommended. Increasing nms_thresh requires more of the putative objects to be overlapping for them to be merged into a single label, which may help a bit in denser regions.

The resulting calls are loaded like the H&E ones were, except the input image was based on array rather than spatial coordinates.

We can visualise the cells identified in the GEX space segmentation in our previously defined region.

Armed with both the H&E and gene expression segmentation results, we can now fill in the gaps in the H&E with GEX calls. We only add GEX cells that do not overlap with any H&E labels.

A few extra GEX-derived cells show up in our test region. Note the presence of .obs["labels_joint_source"] which specifies whether any given label came from the primary or secondary labelling.

At this point, the counts have been destriped and bins have been assigned to cells based on both H&E and GEX segmentation. Time to group the bins into cells!

The object features the sum of gene expression of the constituent bins, and means of the array and spatial coordinates to represent the cell centroids. .obs["labels_joint_source"] is taken as well, providing information on which segmentation any given cell stems from. Let's visualise our test fragment of the tissue, now in cell space!

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import os

import matplotlib.pyplot as plt
import seaborn as sns

import bin2cell as b2c

#create directory for stardist input/output files
os.makedirs("stardist", exist_ok=True)
```

Example 2 (python):
```python
adata = sc.read_h5ad('/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/HEA_FOET14880396_b2c.h5ad')
adata
```

Example 3 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5001851 × 18051
    obs: 'in_tissue', 'array_row', 'array_col', 'n_counts', 'destripe_factor', 'n_counts_adjusted', 'labels_he_0p01', 'labels_he_0p03', 'labels_he_0p05', 'labels_he_0p1', 'labels_he_default', 'labels_he_sel', 'labels_he_expanded_1', 'labels_he_expanded_2', 'labels_he_expanded_3', 'labels_he_expanded_4', 'labels_he_expanded_sel'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells'
    uns: 'bin2cell', 'spatial'
    obsm: 'spatial', 'spatial_cropped_150_buffer'
```

Example 4 (python):
```python
# to re-start gex segmentation
# not required if initial attemp
# adata.obs.drop(columns=['labels_gex_0p002', 'labels_gex_0p01', 'labels_gex_0p05', 'labels_gex_sel'],inplace=True)
```

Example 5 (python):
```python
%%time
mpp = 0.3
b2c.grid_image(adata, "n_counts_adjusted", mpp=mpp, sigma=5, save_path="stardist/gex.tiff")
```

Example 6 (unknown):
```unknown
CPU times: user 34.7 s, sys: 14 s, total: 48.7 s
Wall time: 1min 10s
```

Example 7 (python):
```python
%%time
prob_thresh_dict = {
    '0p005':0.005,
    '0p01':0.01,
     '0p03':0.03,
    '0p05':0.05,
}

for key,prob_thresh in prob_thresh_dict.items():
    print(f'### {key} ###')
    b2c.stardist(image_path="stardist/gex.tiff", 
                 labels_npz_path=f"stardist/gex_prob{key}.npz", 
                 stardist_model="2D_versatile_fluo", 
                 prob_thresh=prob_thresh, 
                 nms_thresh=0.5
                )
    print('')
```

Example 8 (unknown):
```unknown
### 0p005 ###
```

Example 9 (unknown):
```unknown
2024-12-05 17:50:10.809546: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-12-05 17:50:11.174332: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-12-05 17:50:11.723242: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1733421011.999613  690313 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1733421012.075924  690313 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-12-05 17:50:12.987381: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
```

Example 10 (unknown):
```unknown
Found model '2D_versatile_fluo' for 'StarDist2D'.
```

Example 11 (unknown):
```unknown
2024-12-05 17:51:24.727765: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
```

Example 12 (python):
```python
Loading network weights from 'weights_best.h5'.
Loading thresholds from 'thresholds.json'.
Using default values: prob_thresh=0.479071, nms_thresh=0.3.
effective: block_size=(4096, 4096), min_overlap=(128, 128), context=(128, 128)
```

Example 13 (unknown):
```unknown
functional.py (238): The structure of `inputs` doesn't match the expected structure.
Expected: ['input']
Received: inputs=Tensor(shape=(1, 256, 256, 1))
  0%|          | 0/36 [00:00<?, ?it/s]functional.py (238): The structure of `inputs` doesn't match the expected structure.
Expected: ['input']
Received: inputs=Tensor(shape=(1, None, None, 1))
100%|██████████| 36/36 [49:28<00:00, 82.46s/it]
```

Example 14 (python):
```python
Found 155283 objects

### 0p01 ###
Found model '2D_versatile_fluo' for 'StarDist2D'.
Loading network weights from 'weights_best.h5'.
Loading thresholds from 'thresholds.json'.
Using default values: prob_thresh=0.479071, nms_thresh=0.3.
effective: block_size=(4096, 4096), min_overlap=(128, 128), context=(128, 128)
```

Example 15 (unknown):
```unknown
100%|██████████| 36/36 [50:34<00:00, 84.30s/it]
```

Example 16 (python):
```python
Found 123972 objects

### 0p03 ###
Found model '2D_versatile_fluo' for 'StarDist2D'.
Loading network weights from 'weights_best.h5'.
Loading thresholds from 'thresholds.json'.
Using default values: prob_thresh=0.479071, nms_thresh=0.3.
effective: block_size=(4096, 4096), min_overlap=(128, 128), context=(128, 128)
```

Example 17 (unknown):
```unknown
100%|██████████| 36/36 [43:22<00:00, 72.29s/it]
```

Example 18 (python):
```python
Found 83831 objects

### 0p05 ###
Found model '2D_versatile_fluo' for 'StarDist2D'.
Loading network weights from 'weights_best.h5'.
Loading thresholds from 'thresholds.json'.
Using default values: prob_thresh=0.479071, nms_thresh=0.3.
effective: block_size=(4096, 4096), min_overlap=(128, 128), context=(128, 128)
```

Example 19 (unknown):
```unknown
100%|██████████| 36/36 [42:00<00:00, 70.02s/it]
```

Example 20 (unknown):
```unknown
Found 69281 objects

CPU times: user 1h 56min 24s, sys: 25min 44s, total: 2h 22min 8s
Wall time: 3h 9min 6s
```

Example 21 (python):
```python
prob_thresh_dict = {
    '0p005':0.005,
    '0p01':0.01,
     '0p03':0.03,
    '0p05':0.05,
}

for key,prob_thresh in prob_thresh_dict.items():
    print(f'### {key} ###')
    b2c.insert_labels(adata, 
                  labels_npz_path=f"stardist/gex_prob{key}.npz", 
                  basis="array", 
                  mpp=mpp, 
                  labels_key=f"labels_gex_{key}"
                 )
```

Example 22 (unknown):
```unknown
### 0p005 ###
### 0p01 ###
### 0p03 ###
### 0p05 ###
```

Example 23 (python):
```python
mask = ((adata.obs['array_row'] >= 1450) & 
        (adata.obs['array_row'] <= 1550) & 
        (adata.obs['array_col'] >= 1400) & 
        (adata.obs['array_col'] <= 1600)
       )
bdata = adata[mask]

for key in prob_thresh_dict.keys():
    #0 means unassigned
    bdata = bdata[bdata.obs[f"labels_gex_{key}"]>0]
    bdata.obs[f"labels_gex_{key}"] = bdata.obs[f"labels_gex_{key}"].astype(str)
    print(len(set(bdata.obs[f"labels_gex_{key}"])))
    sc.pl.spatial(bdata, color=[None, f"labels_gex_{key}"], img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  legend_loc=None,ncols=2
                 )
```

Example 24 (unknown):
```unknown
2949655472.py (11): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 25 (unknown):
```unknown
2949655472.py (11): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 26 (unknown):
```unknown
2949655472.py (11): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 27 (unknown):
```unknown
2949655472.py (11): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 28 (python):
```python
mask = ((adata.obs['array_row'] >= 1450) & 
        (adata.obs['array_row'] <= 1550) & 
        (adata.obs['array_col'] >= 1400) & 
        (adata.obs['array_col'] <= 1600)
       )
bdata = adata[mask]
sc.pl.spatial(bdata, color=[None, 'n_counts_adjusted'], img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  legend_loc=None,ncols=2
                 )
```

Example 29 (python):
```python
#the label viewer wants a crop of the processed image
#get the corresponding coordinates spanning the subset object
crop = b2c.get_crop(bdata, basis="array", mpp=mpp)

#GEX pops better with percentile normalisation performed
rendered = b2c.view_labels(image_path="stardist/gex.tiff", 
                           labels_npz_path="stardist/gex_prob0p005.npz", 
                           crop=crop,
                           stardist_normalize=True
                          )
plt.imshow(rendered)
plt.title('sigma: 5, gex_prob_thresh: 0p005')
```

Example 30 (unknown):
```unknown
Text(0.5, 1.0, 'sigma: 5, gex_prob_thresh: 0p005')
```

Example 31 (python):
```python
#the label viewer wants a crop of the processed image
#get the corresponding coordinates spanning the subset object
crop = b2c.get_crop(bdata, basis="array", mpp=mpp)

#GEX pops better with percentile normalisation performed
rendered = b2c.view_labels(image_path="stardist/gex.tiff", 
                           labels_npz_path="stardist/gex_prob0p05.npz", 
                           crop=crop,
                           stardist_normalize=True
                          )
plt.imshow(rendered)
plt.title('sigma: 5, gex_prob_thresh: 0p05')
```

Example 32 (unknown):
```unknown
Text(0.5, 1.0, 'sigma: 5, gex_prob_thresh: 0p05')
```

Example 33 (python):
```python
gex_prob_thresh_sel = '0p05' # sigma: 5
adata.obs['labels_gex_sel'] = adata.obs[f'labels_gex_{gex_prob_thresh_sel}']
```

Example 34 (python):
```python
# save
adata.write('/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/HEA_FOET14880396_b2c.h5ad')
```

Example 35 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5001851 × 18051
    obs: 'in_tissue', 'array_row', 'array_col', 'n_counts', 'destripe_factor', 'n_counts_adjusted', 'labels_he_0p01', 'labels_he_0p03', 'labels_he_0p05', 'labels_he_0p1', 'labels_he_default', 'labels_he_sel', 'labels_he_expanded_1', 'labels_he_expanded_2', 'labels_he_expanded_3', 'labels_he_expanded_4', 'labels_he_expanded_sel', 'labels_gex_0p005', 'labels_gex_0p01', 'labels_gex_0p03', 'labels_gex_0p05', 'labels_gex_sel'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells'
    uns: 'bin2cell', 'spatial'
    obsm: 'spatial', 'spatial_cropped_150_buffer'
```

Example 36 (python):
```python
# read in
mpp=0.3
adata = sc.read_h5ad('/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/HEA_FOET14880396_b2c.h5ad')
adata
```

Example 37 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5001851 × 18051
    obs: 'in_tissue', 'array_row', 'array_col', 'n_counts', 'destripe_factor', 'n_counts_adjusted', 'labels_he_0p01', 'labels_he_0p03', 'labels_he_0p05', 'labels_he_0p1', 'labels_he_default', 'labels_he_sel', 'labels_he_expanded_1', 'labels_he_expanded_2', 'labels_he_expanded_3', 'labels_he_expanded_4', 'labels_he_expanded_sel', 'labels_gex_0p005', 'labels_gex_0p01', 'labels_gex_0p03', 'labels_gex_0p05', 'labels_gex_sel'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells'
    uns: 'bin2cell', 'spatial'
    obsm: 'spatial', 'spatial_cropped_150_buffer'
```

Example 38 (python):
```python
b2c.salvage_secondary_labels(adata, 
                             primary_label="labels_he_expanded_sel", 
                             secondary_label="labels_gex_sel", 
                             labels_key="labels_joint"
                            )
```

Example 39 (unknown):
```unknown
Salvaged 10198 secondary labels
```

Example 40 (python):
```python
mask = ((adata.obs['array_row'] >= 1450) & 
        (adata.obs['array_row'] <= 1550) & 
        (adata.obs['array_col'] >= 1400) & 
        (adata.obs['array_col'] <= 1600)
       )
bdata = adata[mask]

#0 means unassigned
bdata = bdata[bdata.obs['labels_joint']>0]
bdata.obs['labels_joint'] = bdata.obs['labels_joint'].astype(str)

sc.pl.spatial(bdata, color=[None, "labels_joint_source"],
              img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer")
```

Example 41 (unknown):
```unknown
2034932217.py (10): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 42 (python):
```python
#0 means unassigned
adata_assigned = adata[adata.obs['labels_joint']>0]
counts = adata_assigned.obs['labels_joint'].value_counts()
plt.figure(figsize=(10, 3))
sns.histplot(counts, kde=False, bins=200)
```

Example 43 (unknown):
```unknown
<Axes: xlabel='count', ylabel='Count'>
```

Example 44 (python):
```python
counts_1st = adata_assigned[adata_assigned.obs['labels_joint_source']=='primary'].obs['labels_joint'].value_counts()
plt.figure(figsize=(10, 3))
sns.histplot(counts_1st, kde=False, bins=200)
```

Example 45 (unknown):
```unknown
<Axes: xlabel='count', ylabel='Count'>
```

Example 46 (python):
```python
counts_2nd = adata_assigned[adata_assigned.obs['labels_joint_source']=='secondary'].obs['labels_joint'].value_counts()
plt.figure(figsize=(10, 3))
sns.histplot(counts_2nd, kde=False, bins=200)
```

Example 47 (unknown):
```unknown
<Axes: xlabel='count', ylabel='Count'>
```

Example 48 (python):
```python
'''
# remove
bin_counts_thresh = 5
labels_to_remove = counts.index[counts<bin_counts_thresh]
mask = adata.obs['labels_joint'].isin(labels_to_remove)
bins_to_remove = adata.obs_names[mask]
adata.obs.loc[bins_to_remove,'labels_joint'] = 0

adata_assigned = adata[adata.obs['labels_joint']>0]
counts = adata_assigned.obs['labels_joint'].value_counts()
plt.figure(figsize=(10, 3))
sns.histplot(counts, kde=False, bins=200)
'''
```

Example 49 (python):
```python
'''
mask = ((adata.obs['array_row'] >= 1450) & 
        (adata.obs['array_row'] <= 1550) & 
        (adata.obs['array_col'] >= 1400) & 
        (adata.obs['array_col'] <= 1600)
       )
bdata = adata[mask]

#0 means unassigned
bdata = bdata[bdata.obs['labels_joint']>0]
bdata.obs['labels_joint'] = bdata.obs['labels_joint'].astype(str)

sc.pl.spatial(bdata, color=[None, "labels_joint_source"], img_key="0.3_mpp", basis="spatial_cropped")
'''
```

Example 50 (python):
```python
# save adata
adata.write('/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/HEA_FOET14880396_b2c.h5ad')
adata
```

Example 51 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5001851 × 18051
    obs: 'in_tissue', 'array_row', 'array_col', 'n_counts', 'destripe_factor', 'n_counts_adjusted', 'labels_he_0p01', 'labels_he_0p03', 'labels_he_0p05', 'labels_he_0p1', 'labels_he_default', 'labels_he_sel', 'labels_he_expanded_1', 'labels_he_expanded_2', 'labels_he_expanded_3', 'labels_he_expanded_4', 'labels_he_expanded_sel', 'labels_gex_0p005', 'labels_gex_0p01', 'labels_gex_0p03', 'labels_gex_0p05', 'labels_gex_sel', 'labels_joint', 'labels_joint_source'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells'
    uns: 'bin2cell', 'spatial'
    obsm: 'spatial', 'spatial_cropped_150_buffer'
```

Example 52 (python):
```python
cdata = b2c.bin_to_cell(adata, labels_key="labels_joint", spatial_keys=["spatial", "spatial_cropped_150_buffer"])
cdata
```

Example 53 (unknown):
```unknown
AnnData object with n_obs × n_vars = 134590 × 18051
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells'
    uns: 'spatial'
    obsm: 'spatial', 'spatial_cropped_150_buffer'
```

Example 54 (python):
```python
cell_mask = ((cdata.obs['array_row'] >= 1450) & 
        (cdata.obs['array_row'] <= 1550) & 
        (cdata.obs['array_col'] >= 1400) & 
        (cdata.obs['array_col'] <= 1600)
       )
ddata = cdata[cell_mask]
sc.pl.spatial(ddata, color=["bin_count", "labels_joint_source"], 
              img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer", ncols=2)
```

Example 55 (unknown):
```unknown
anndata.py (1146): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 56 (unknown):
```unknown
cdata.write('/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/HEA_FOET14880396_b2c_cells.h5ad')
```

Example 57 (unknown):
```unknown
cdata.X.data[:5]
```

Example 58 (unknown):
```unknown
array([0.55027094, 1.30175513, 1.01929175, 1.31881525, 3.24395094])
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no7_manual-annotation_PericardialCells.html

**Contents:**
- Variables¶
- Read in data¶
- Remove mid-grain markers from the feature¶
- PCA, Clustering, UMAP¶
- Markers¶
- Save¶

2480 rows × 7 columns

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import celltypist
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 4 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 5 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets'
sample_id = 'HEA_FOET14880396'
celltype = 'PericardialCells'
```

Example 6 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 7 (python):
```python
# read in
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 3061 × 15212
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
# read in
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0
                              )
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group']!=celltype]
midgrain_markers
```

Example 10 (python):
```python
# remove
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]
bdata
```

Example 11 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 3061 × 13976
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 12 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata,min_dist=0.5)
sc.pl.embedding(adata,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 13 (unknown):
```unknown
number of hvgs: 3497
```

Example 14 (python):
```python
# read in markers identified using single-nucleus data
markers_df = pd.read_csv(f'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_{celltype}_finegrain_markers.csv')

mask1 = markers_df['group_mean']>1
mask2 = markers_df['logfoldchanges']>1
markers_df = markers_df[mask1&mask2]

markers_df
```

Example 15 (python):
```python
markers = {}
n_top = 10
for c in set(markers_df['group']):
    markers_df_sub = markers_df[markers_df['group']==c]
    markers_df_sub.sort_values('logfoldchanges',ascending=False)
    markers[c]=list(markers_df_sub[:n_top]['names'])

# manually add genes
markers['Cycling'] = ['MKI67','TOP2A'] 

genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in list(adata.var_names))&(x not in genes)]
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in list(adata.var_names)]
len(genes)
```

Example 16 (python):
```python
sc.pl.embedding(adata,basis='X_umap',color=genes, size=20,
               ncols=6, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 17 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=['celltypist_midmod2fine', 'conf_score_midmod2fine'],
                wspace=0.7)
```

Example 18 (python):
```python
# umap per celltype to highlight
group_col = 'celltypist_midmod2fine'
groups_to_plot = list(set(adata.obs[group_col]))

plt.rcParams['axes.grid'] = False
fig,axs=plt.subplots(nrows=2,ncols=4,figsize=(15,7))
# Flatten the axs array for easier iteration in case of multiple rows and columns
axs_flat = axs.flatten()

for i,grp in enumerate(groups_to_plot):
    sc.pl.umap(adata, color=[group_col], groups=grp, 
               size=30, frameon=False,
               show=False, ax=axs_flat[i], legend_loc=None, legend_fontsize=5,
               title=grp
              )
# Turn off any remaining subplots
for j in range(len(groups_to_plot), len(axs_flat)):
    fig.delaxes(axs_flat[j])

# fig.suptitle('Main Title for the Figure')
fig.tight_layout()
```

Example 19 (unknown):
```unknown
resolution_sel = 1
```

Example 20 (python):
```python
markers_others = {
    'CM': ['TTN', 'TNNT2','MYH6',
        'MYH7', 'MYL2', 'FHL2', # vCN
            'NPPA', 'MYL7', 'MYL4'], #aCM
    'FB': ['DCN', 'GCN', 'PDGFRA','COL1A1','COL1A2'],
    'EC': ['VWF', 'PECAM1', 'CDH5','RGCC', 'FABP5',],
    'Peri': ['RGS5', 'ABCC9', 'KCNJ8'],
    'SMC': ['MYH11', 'TAGLN', 'ACTA2'],
    'Meso': ['MSLN', 'WT1', 'BNC1'],
    'Neuro': ['PLP1', 'NRXN1', 'NRXN3','PRPH', 'NEFL', 'NEFM', 'NEFH', 'STMN2', 'SST', 'SLC10A4', 'SLC18A3', 'TH', 'NPY'],
    'Adipo': ['GPAM', 'FASN', 'LEP'],
    'Myelo': ['CD14', 'C1QA', 'CD68','LYVE1','TIMD4'],
    'Lymph': ['CD8A', 'IL7R', 'CD40LG'],
    'B': ['CD19','SDC1','IGHG1','IGHA1'],
    'Mast': ['KIT', 'CPA3'],
}

for key in markers_others.keys():
    markers_others[key]=[x for x in markers_others[key] if x in adata.var_names]

sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              markers_others, 
              groupby=f'leiden_{str(resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 21 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CM, FB, EC, etc.
```

Example 22 (python):
```python
sc.tl.dendrogram(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby=f'leiden_{str(resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 23 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: PericardialCellsParietal, PericardialCellsIntermediate, PericardialCellsFibrous, etc.
```

Example 24 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 25 (unknown):
```unknown
[2.1685468  2.22569867 2.00533587 2.08507898 2.52518411]
```

Example 26 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True, n_genes=5)
dp.add_totals().show()
```

Example 27 (unknown):
```unknown
[2.1685468  2.22569867 2.00533587 2.08507898 2.52518411]
```

Example 28 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=f'leiden_{str(resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 29 (python):
```python
sc.pl.violin(adata,keys=['total_counts','bin_count'],groupby=f'leiden_{str(resolution_sel)}')
```

Example 30 (unknown):
```unknown
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 31 (unknown):
```unknown
'PericardialCells'
```

Example 32 (python):
```python
# annotate
adata.obs['fine_grain_manual'] = adata.obs[f'leiden_{str(resolution_sel)}'].copy()
adata.obs.replace({'fine_grain_manual':{
    '0':'PericardialCellsFibrous',
    '1':'PericardialCellsFibrous',
    '2':'PericardialCellsIntermediate',
    '3':'PericardialCellsFibrous',
    '4':'PericardialCellsFibrous', # contamination from valveECs
    '5':'PericardialCellsIntermediate',
    '6':'PericardialCells_DoubletsWithCMs',
}},inplace=True)
adata.obs['fine_grain_manual'].value_counts()
```

Example 33 (unknown):
```unknown
fine_grain_manual
PericardialCellsFibrous             2231
PericardialCellsIntermediate         730
PericardialCells_DoubletsWithCMs     100
Name: count, dtype: int64
```

Example 34 (python):
```python
ad = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad',backed='r')
ad = ad[ad.obs['mid_grain']==celltype]
set(ad.obs['fine_grain'])
```

Example 35 (unknown):
```unknown
{'PericardialCellsFibrous',
 'PericardialCellsIntermediate',
 'PericardialCellsParietal'}
```

Example 36 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 37 (python):
```python
# save
adata.write(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
print(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig3_SAN_niche/VisiumHD_CCSaxis_no4_neighbouring-cells.html

**Contents:**
- Variables¶
- Read in data¶
- Detect spatial neighbours¶

bin2cell output which has celltypist output as well

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import squidpy as sq
```

Example 2 (python):
```python
/software/cellgen/team205/kk18/envs/squidpy/lib/python3.10/site-packages/dask/dataframe/__init__.py:31: FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning.
  warnings.warn(
/software/cellgen/team205/kk18/envs/squidpy/lib/python3.10/site-packages/numba/core/decorators.py:246: RuntimeWarning: nopython is set for njit and is ignored
  warnings.warn('nopython is set for njit and is ignored', RuntimeWarning)
/software/cellgen/team205/kk18/envs/squidpy/lib/python3.10/site-packages/anndata/utils.py:429: FutureWarning: Importing read_text from `anndata` is deprecated. Import anndata.io.read_text instead.
  warnings.warn(msg, FutureWarning)
```

Example 3 (python):
```python
# import session_info
# session_info.show()
```

Example 4 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/tissueaxis.py")
tissueaxis = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = tissueaxis
spec.loader.exec_module(tissueaxis)
```

Example 5 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 6 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects'
sample_id = 'HEA_FOET14880396'
```

Example 7 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 8 (unknown):
```unknown
celltype_key = 'fine_grain_4Jan2025'
axis_name = 'SAN_axis'
```

Example 9 (python):
```python
# read in 
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 10 (unknown):
```unknown
AnnData object with n_obs × n_vars = 129803 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_4Jan2025', 'HistologicalAnnotation', 'SANaxis_landmarks', 'SAN_axis', 'SAN_axis_bin_annot'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'HistologicalAnnotation_colors', 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'fine_grain_4Jan2025_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 11 (python):
```python
# celltype name mapping for plotting
df = pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/finegrain_name_mapping.csv')
celltype_mapping = df.set_index('Full_name')['Short_name_finalised'].to_dict()
adata.obs.replace({'fine_grain_4Jan2025':celltype_mapping},inplace=True)
```

Example 12 (python):
```python
/tmp/ipykernel_1580522/3810385248.py:4: FutureWarning: The behavior of Series.replace (and DataFrame.replace) with CategoricalDtype is deprecated. In a future version, replace will only be used for cases that preserve the categories. To change the categories, use ser.cat.rename_categories instead.
  adata.obs.replace({'fine_grain_4Jan2025':celltype_mapping},inplace=True)
```

Example 13 (python):
```python
%%time
sq.gr.spatial_neighbors(adata, coord_type="generic", n_neighs=20, spatial_key="spatial")
```

Example 14 (unknown):
```unknown
CPU times: user 988 ms, sys: 122 ms, total: 1.11 s
Wall time: 1.14 s
```

Example 15 (python):
```python
spatial_connect = adata.obsp['spatial_connectivities'].copy()
```

Example 16 (python):
```python
# mask for cells which has macrophage pseudotime
mask = adata.obs[f'{axis_name}_bin_annot']!='nan'
```

Example 17 (python):
```python
spatial_connect[:,mask].shape
```

Example 18 (unknown):
```unknown
(129803, 1510)
```

Example 19 (unknown):
```unknown
# Step 1: Transpose the sparse matrix
sparse_matrix_T = spatial_connect[:,mask].T
```

Example 20 (python):
```python
from tqdm import tqdm
```

Example 21 (python):
```python
# Step 2: Initialize a dictionary to store neighbor cell type counts for each pseudotime cell
neighbor_cell_types = {i: [] for i in range(sparse_matrix_T.shape[0])}
# Step 3: Identify neighbors for each pseudotime cell and aggregate cell types
cell_types = list(adata.obs[celltype_key])
for i in tqdm(range(sparse_matrix_T.shape[0])):
    neighbors = np.nonzero(sparse_matrix_T[i])[1]  # Get indices of neighbors
    neighbor_types = [cell_types[j] for j in neighbors]
    neighbor_cell_types[i].extend(neighbor_types)
# Step 4: Convert to DataFrame for easier analysis
neighbor_df = pd.DataFrame.from_dict(neighbor_cell_types, orient='index')
# Calculate frequency or proportion of each cell type for each pseudotime
cell_type_counts = neighbor_df.apply(pd.Series.value_counts, axis=1).fillna(0)
# add pseudotime
cell_type_counts.index = adata.obs_names[mask]
```

Example 22 (unknown):
```unknown
100%|██████████| 1510/1510 [00:01<00:00, 1035.96it/s]
```

Example 23 (python):
```python
cell_type_counts[f'{axis_name}_bin_annot']=adata.obs.loc[cell_type_counts.index,f'{axis_name}_bin_annot']
df = cell_type_counts.groupby(f'{axis_name}_bin_annot').mean()
df = df.dropna()
df.max().sort_values(ascending=False)
```

Example 24 (unknown):
```unknown
/tmp/ipykernel_1580522/1380053985.py:2: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  df = cell_type_counts.groupby(f'{axis_name}_bin_annot').mean()
```

Example 25 (unknown):
```unknown
SANPCHd       13.281853
SANPCHn       13.113043
SANPCT        10.009346
aCMR           6.728571
unassigned     5.616822
FB             4.430769
AVNPC          2.859813
GVVenEC        1.127660
Endocard       1.014286
ParaN          0.574468
MesEpiC        0.430769
CorPeri        0.343629
MacCX3         0.324324
aCML           0.288889
CorCapEC       0.287425
SchwC          0.148936
MPLYVE1        0.142857
LymphEC        0.121739
GVSMCT         0.107692
SchwNeuro      0.095745
ValveIC        0.073333
GVSMC          0.046332
EPDC           0.034749
vCMC           0.034749
PericFib       0.023166
ValveEC        0.014286
dtype: float64
```

Example 26 (python):
```python
cell_type_counts[f'{axis_name}_bin_annot']=adata.obs.loc[cell_type_counts.index,f'{axis_name}_bin_annot']
df = cell_type_counts.groupby(f'{axis_name}_bin_annot').mean()
df = df.dropna()


mean_thresh = 0.1
df = df[df.columns[df.max()>mean_thresh]].T

# normalise with max value
max_arr = df.max(axis=1)
for col in df.columns:
    df[col] = df[col]/max_arr

# remove "unclassified"
df = df[df.index!='unassigned']
```

Example 27 (unknown):
```unknown
/tmp/ipykernel_1580522/3071229161.py:2: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  df = cell_type_counts.groupby(f'{axis_name}_bin_annot').mean()
```

Example 28 (python):
```python
plt.rcParams['axes.grid'] = False
sns.clustermap(df,col_cluster=False,figsize=(3.5,6),
               dendrogram_ratio=0.05,cbar_pos=(-0.05, 0.8, 0.02, 0.1))
plt.savefig(f'{figdir}/VisiumHD_SANaxis_celltypes.pdf', bbox_inches="tight",dpi=300)
plt.show()
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig3_SAN_niche/VisiumHD_CCSaxis_no3_genes.html

**Contents:**
- Variables¶
- Read in data¶
- Binning axis¶
- Annotate bins¶
- Axon guidance genes¶

bin2cell output which has celltypist output as well

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42
```

Example 2 (python):
```python
# import deprecated
# import session_info
# session_info.show()
```

Example 3 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/tissueaxis.py")
tissueaxis = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = tissueaxis
spec.loader.exec_module(tissueaxis)
```

Example 4 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 5 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects'
sample_id = 'HEA_FOET14880396'
```

Example 6 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 7 (python):
```python
# read in 
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 129803 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_4Jan2025', 'HistologicalAnnotation', 'SANaxis_landmarks', 'SAN_axis', 'SAN_axis_bin_annot'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'HistologicalAnnotation_colors', 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'fine_grain_4Jan2025_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 9 (unknown):
```unknown
celltype_key = 'fine_grain_4Jan2025'
```

Example 10 (unknown):
```unknown
axis_name = 'SAN_axis'
```

Example 11 (python):
```python
# make bin
adata.obs[f'{axis_name}_bin'] = pd.cut(adata.obs[axis_name],bins=10)
```

Example 12 (python):
```python
set(adata.obs[f'{axis_name}_bin'])
```

Example 13 (unknown):
```unknown
{Interval(-0.2, 0.0, closed='right'),
 Interval(-0.4, -0.2, closed='right'),
 Interval(-0.6, -0.4, closed='right'),
 Interval(-0.8, -0.6, closed='right'),
 Interval(-1.002, -0.8, closed='right'),
 Interval(0.0, 0.2, closed='right'),
 Interval(0.2, 0.4, closed='right'),
 Interval(0.4, 0.6, closed='right'),
 Interval(0.6, 0.8, closed='right'),
 Interval(0.8, 1.0, closed='right'),
 nan}
```

Example 14 (python):
```python
bdata = adata[adata.obs[axis_name].isna()==False]
bdata
```

Example 15 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 1510 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_4Jan2025', 'HistologicalAnnotation', 'SANaxis_landmarks', 'SAN_axis', 'SAN_axis_bin_annot', 'SAN_axis_bin'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'HistologicalAnnotation_colors', 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'fine_grain_4Jan2025_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 16 (python):
```python
sc.pp.filter_genes(bdata, min_cells=3)
bdata.shape
```

Example 17 (unknown):
```unknown
(1510, 14912)
```

Example 18 (python):
```python
# annotate bin
annot_dict = {
    '(-1.002, -0.8]':'SinusHorn_1',
     '(-0.8, -0.6]':'SinusHorn_2',
     '(-0.6, -0.4]':'SinusHorn_3',
     '(-0.4, -0.2]':'SinusHorn_4',
     '(-0.2, 0.0]':'SAN_head_1',
     '(0.0, 0.2]':'SAN_head_2',
     '(0.2, 0.4]':'SAN_head_3',
     '(0.4, 0.6]':'SAN_tail_1',
     '(0.6, 0.8]':'SAN_tail_2',
     '(0.8, 1.0]':'SAN_tail_3',
}
bdata.obs[f'{axis_name}_bin_annot'] = bdata.obs[f'{axis_name}_bin'].astype('str').copy()
bdata.obs.replace({f'{axis_name}_bin_annot':annot_dict}, inplace=True)
bdata.obs[f'{axis_name}_bin_annot'] = bdata.obs[f'{axis_name}_bin_annot'].astype('category')
bdata.obs[f'{axis_name}_bin_annot'] = bdata.obs[f'{axis_name}_bin_annot'].cat.reorder_categories(list(annot_dict.values()))
bdata.obs[f'{axis_name}_bin_annot'].cat.categories
```

Example 19 (unknown):
```unknown
Index(['SinusHorn_1', 'SinusHorn_2', 'SinusHorn_3', 'SinusHorn_4',
       'SAN_head_1', 'SAN_head_2', 'SAN_head_3', 'SAN_tail_1', 'SAN_tail_2',
       'SAN_tail_3'],
      dtype='object')
```

Example 20 (python):
```python
df = bdata.obs[[celltype_key]].copy()
df['count']=1
df.reset_index(inplace=True)
df = pd.pivot_table(df, values='count', index=['index'],
                       columns=[celltype_key])
df.fillna(0,inplace=True)
df[f'{axis_name}_bin']=bdata.obs.loc[df.index,f'{axis_name}_bin_annot']
df = df.groupby(f'{axis_name}_bin').sum()
max_arr = df.max(axis=1)
for col in df.columns:
    df[col] = df[col]/max_arr
plt.rcParams['axes.grid'] = False
plt.rcParams['figure.figsize'] = 3,5
sns.heatmap(df)
```

Example 21 (python):
```python
/tmp/ipykernel_1580563/441468744.py:4: FutureWarning: The default value of observed=False is deprecated and will change to observed=True in a future version of pandas. Specify observed=False to silence this warning and retain the current behavior
  df = pd.pivot_table(df, values='count', index=['index'],
/tmp/ipykernel_1580563/441468744.py:8: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  df = df.groupby(f'{axis_name}_bin').sum()
```

Example 22 (unknown):
```unknown
<Axes: xlabel='fine_grain_4Jan2025', ylabel='SAN_axis_bin'>
```

Example 23 (python):
```python
# add annotation to original adata
adata.obs[f'{axis_name}_bin_annot'] = 'nan'
adata.obs.loc[bdata.obs_names, f'{axis_name}_bin_annot'] = bdata.obs[f'{axis_name}_bin_annot']
adata.obs[f'{axis_name}_bin_annot'] = adata.obs[f'{axis_name}_bin_annot'].astype('category')
adata.obs[f'{axis_name}_bin_annot'] = adata.obs[f'{axis_name}_bin_annot'].cat.reorder_categories(list(annot_dict.values())+['nan'])
adata.obs[f'{axis_name}_bin_annot'].cat.categories
```

Example 24 (unknown):
```unknown
Index(['SinusHorn_1', 'SinusHorn_2', 'SinusHorn_3', 'SinusHorn_4',
       'SAN_head_1', 'SAN_head_2', 'SAN_head_3', 'SAN_tail_1', 'SAN_tail_2',
       'SAN_tail_3', 'nan'],
      dtype='object')
```

Example 25 (python):
```python
# save
adata.obs.drop([f'{axis_name}_bin'],axis=1,inplace=True)
adata.write(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 26 (python):
```python
# from embryonic/foetal vs adult comparison
genes = ['SEMA3A', 'SLIT2', 'SLIT3', 'EFNB2', 'CXCL12', 'FGF10', 'VEGFC', 'NELL1']
genes = [x for x in genes if x in bdata.var_names]
print(len(genes))
```

Example 27 (unknown):
```unknown
tissueaxis.plot_axis(bdata,
              axis_col=axis_name,
              binned_col=f'{axis_name}_bin_annot',
              feature_type='gene', # 'gene' or 'celltype'
              features_to_plot=genes,
              features_to_remove=None,
              threshold_express_prop=0.1, # at quantile=0.9
              threshold_cell_abundance=None, # at quantile=0.9
              correlation=False,
              min_max_scale=True,
              row_cluster=True,
              figsize=(4,4),
              cbar_pos=(-0, 0.65, 0.03, 0.15),
              cmap='viridis',
          xticklabels=True,
          yticklabels=True,
              title=None,
              return_feature = False,
         save=f'{figdir}/VisiumHD_SANaxis_axon-guidance-genes.pdf',
    save_dpi=300,
             )
```

Example 28 (unknown):
```unknown
/nfs/team205/kk18/function/python/tissueaxis.py:57: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  df_gene_exp_prop = df_gene_exp_bi.groupby(f'{axis_col}_binned').mean()
/nfs/team205/kk18/function/python/tissueaxis.py:134: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  df_plot = df_plot.set_index('index').groupby(['variable',f'{axis_col}_binned']).mean().reset_index()
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/Visium/cell2location/wholeheart_without_DA/no2_mapping_OCT.html

**Contents:**
  - Loading packages¶
- Functions¶
- Read in metadata¶
- Data, parameters¶
- Loading visium data¶
- Filter out low quality spots¶
- Remove mitochondrial genes¶
- Add metadata and annotation¶
- Remove spots based on manual annotation¶
- Combine pcw13w-Atrial and pcw13w-Ventricle data to have whole heart in one sample¶
- Run per group: cell2location mapping¶

**Examples:**

Example 1 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl

import cell2location
import scvi

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text
import seaborn as sns
```

Example 2 (unknown):
```unknown
Global seed set to 0
```

Example 3 (python):
```python
import gc
import torch
```

Example 4 (python):
```python
def read_and_qc(sample_name, path):
    r""" This function reads the data for one 10X spatial experiment into the anndata object.
    It also calculates QC metrics. Modify this function if required by your workflow.

    :param sample_name: Name of the sample
    :param path: path to data
    """

    adata = sc.read_visium(path + str(sample_name),
                           count_file='filtered_feature_bc_matrix.h5', load_images=True)
    adata.var_names_make_unique(join='-')
    
    adata.obs['sample'] = sample_name
    
    # rename genes to ENSEMBL
    adata.var['SYMBOL'] = adata.var_names
    adata.var_names = adata.var['gene_ids']
    adata.var_names.name = None

    # Calculate QC metrics
    sc.pp.calculate_qc_metrics(adata, inplace=True)
    adata.var['mt'] = [gene.startswith('MT-') for gene in adata.var['SYMBOL']]
    adata.obs['mt_frac'] = adata[:, adata.var['mt'].tolist()].X.sum(1).A.squeeze()/adata.obs['total_counts']

    # add sample name to obs names
    adata.obs["sample"] = [str(i) for i in adata.obs['sample']]
    adata.obs_names = adata.obs["sample"] \
                          + '_' + adata.obs_names
    adata.obs.index.name = 'spot_id'

    return adata

def select_slide(adata, s, s_col='sample'):
    r""" This function selects the data for one slide from the spatial anndata object.

    :param adata: Anndata object with multiple spatial experiments
    :param s: name of selected experiment
    :param s_col: column in adata.obs listing experiment name for each location
    """

    slide = adata[adata.obs[s_col].isin([s]), :]
    s_keys = list(slide.uns['spatial'].keys())
    s_spatial = np.array(s_keys)[[s in k for k in s_keys]][0]

    slide.uns['spatial'] = {s_spatial: slide.uns['spatial'][s_spatial]}

    return slide
```

Example 5 (python):
```python
meta=pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/metadata/HeartTeamSamples_Mappeddata_20230208_for-cell2loc.csv',
                index_col=0)
meta
```

Example 6 (python):
```python
pd.crosstab(meta['donor'],meta['cell2loc-map_group'])
```

Example 7 (unknown):
```unknown
# select slides
# Healthy, OCT, and without DA
meta = meta[meta['cell2loc-map_group'].isin(['4PCW','5PCW','7PCW','13PCW','20PCW'])]

sampleIDs = list(meta['sangerID'])
len(sampleIDs)
```

Example 8 (unknown):
```unknown
results_folder = '/nfs/team205/heart/cell2location/foetal'

# create paths and names to results folders for reference regression and cell2location models
ref_run_name = f'{results_folder}/reference_signatures/ver28Feb2023/whole_woPeriCell_woDASMC'
run_name = f'{results_folder}/cell2location_map/ver28Feb2023'
```

Example 9 (unknown):
```unknown
visium_path='/nfs/team205/heart/mapped/spaceranger110/'

celltype_key='fine_grain'
```

Example 10 (python):
```python
# Read the data of the interested region into anndata objects
slides = []
for i in sampleIDs:
    slides.append(read_and_qc(i, path=visium_path))

# Combine anndata objects together
adata_vis = slides[0].concatenate(
    slides[1:],
    batch_key="sample",
    uns_merge="unique",
    batch_categories=sampleIDs,
    index_unique=None
)
```

Example 11 (unknown):
```unknown
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
```

Example 12 (python):
```python
# PLOT QC FOR EACH SAMPLE
fig, axs = plt.subplots(len(slides), 4, figsize=(15, 4*len(slides)-4))
for i, s in enumerate(adata_vis.obs['sample'].unique()):
    #fig.suptitle('Covariates for filtering')

    slide = select_slide(adata_vis, s)
    sns.distplot(slide.obs['total_counts'],
                 kde=False, ax = axs[i, 0])
    axs[i, 0].set_xlim(0, adata_vis.obs['total_counts'].max())
    axs[i, 0].set_xlabel(f'total_counts | {s}')

    sns.distplot(slide.obs['total_counts']\
                 [slide.obs['total_counts']<2000],
                 kde=False, bins=40, ax = axs[i, 1])
    axs[i, 1].set_xlim(0, 2000)
    axs[i, 1].set_xlabel(f'total_counts | {s}')

    sns.distplot(slide.obs['n_genes_by_counts'],
                 kde=False, bins=60, ax = axs[i, 2])
    axs[i, 2].set_xlim(0, adata_vis.obs['n_genes_by_counts'].max())
    axs[i, 2].set_xlabel(f'n_genes_by_counts | {s}')

    sns.distplot(slide.obs['n_genes_by_counts']\
                 [slide.obs['n_genes_by_counts']<1000],
                 kde=False, bins=60, ax = axs[i, 3])
    axs[i, 3].set_xlim(0, 1000)
    axs[i, 3].set_xlabel(f'n_genes_by_counts | {s}')

plt.tight_layout()
```

Example 13 (python):
```python
Trying to set attribute `._uns` of view, copying.
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
Trying to set attribute `._uns` of view, copying.
```

Example 14 (python):
```python
# filter out low quality spots
sc.pp.filter_cells(adata_vis, min_counts = 500)
sc.pp.filter_cells(adata_vis, min_genes = 300)

# Spot number per slide
adata_vis.obs['sample'].value_counts()
```

Example 15 (unknown):
```unknown
HCAHeartST13162330    2806
HCAHeartST11605165    2429
HCAHeartST13162331    2338
HCAHeartST13162329    1777
HCAHeartST10317190    1337
HCAHeartST11605166    1321
HCAHeartST10298116    1219
HCAHeartST10298120    1170
HCAHeartST10298117    1126
HCAHeartST11605168    1082
HCAHeartST11605167    1044
HCAHeartST10298115    1025
HCAHeartST10317189    1004
HCAHeartST10317184     897
HCAHeartST10298114     872
HCAHeartST10317186     855
HCAHeartST10317185     824
HCAHeartST13162332     505
Name: sample, dtype: int64
```

Example 16 (python):
```python
# find mitochondria-encoded (MT) genes
adata_vis.var['MT_gene'] = [gene.startswith('MT-') for gene in adata_vis.var['SYMBOL']]
print(sum(adata_vis.var['MT_gene']))
    
# remove MT genes for spatial mapping (keeping their counts in the object)
adata_vis.obsm['MT'] = adata_vis[:, adata_vis.var['MT_gene'].values].X.toarray()
adata_vis = adata_vis[:, ~adata_vis.var['MT_gene'].values]
print(sum(adata_vis.var['MT_gene']))
```

Example 17 (unknown):
```unknown
meta.head()
```

Example 18 (python):
```python
adata_vis.obs = adata_vis.obs.reset_index().merge(meta, how='left', left_on='sample', right_on='sangerID').set_index('spot_id')
adata_vis
```

Example 19 (unknown):
```unknown
AnnData object with n_obs × n_vars = 23631 × 33525
    obs: 'in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'n_counts', 'n_genes', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'cell2loc-map_group'
    var: 'gene_ids', 'feature_types', 'genome', 'SYMBOL', 'mt', 'n_cells_by_counts-HCAHeartST10298114', 'mean_counts-HCAHeartST10298114', 'log1p_mean_counts-HCAHeartST10298114', 'pct_dropout_by_counts-HCAHeartST10298114', 'total_counts-HCAHeartST10298114', 'log1p_total_counts-HCAHeartST10298114', 'n_cells_by_counts-HCAHeartST10298115', 'mean_counts-HCAHeartST10298115', 'log1p_mean_counts-HCAHeartST10298115', 'pct_dropout_by_counts-HCAHeartST10298115', 'total_counts-HCAHeartST10298115', 'log1p_total_counts-HCAHeartST10298115', 'n_cells_by_counts-HCAHeartST10298116', 'mean_counts-HCAHeartST10298116', 'log1p_mean_counts-HCAHeartST10298116', 'pct_dropout_by_counts-HCAHeartST10298116', 'total_counts-HCAHeartST10298116', 'log1p_total_counts-HCAHeartST10298116', 'n_cells_by_counts-HCAHeartST10298117', 'mean_counts-HCAHeartST10298117', 'log1p_mean_counts-HCAHeartST10298117', 'pct_dropout_by_counts-HCAHeartST10298117', 'total_counts-HCAHeartST10298117', 'log1p_total_counts-HCAHeartST10298117', 'n_cells_by_counts-HCAHeartST10298120', 'mean_counts-HCAHeartST10298120', 'log1p_mean_counts-HCAHeartST10298120', 'pct_dropout_by_counts-HCAHeartST10298120', 'total_counts-HCAHeartST10298120', 'log1p_total_counts-HCAHeartST10298120', 'n_cells_by_counts-HCAHeartST10317184', 'mean_counts-HCAHeartST10317184', 'log1p_mean_counts-HCAHeartST10317184', 'pct_dropout_by_counts-HCAHeartST10317184', 'total_counts-HCAHeartST10317184', 'log1p_total_counts-HCAHeartST10317184', 'n_cells_by_counts-HCAHeartST10317185', 'mean_counts-HCAHeartST10317185', 'log1p_mean_counts-HCAHeartST10317185', 'pct_dropout_by_counts-HCAHeartST10317185', 'total_counts-HCAHeartST10317185', 'log1p_total_counts-HCAHeartST10317185', 'n_cells_by_counts-HCAHeartST10317186', 'mean_counts-HCAHeartST10317186', 'log1p_mean_counts-HCAHeartST10317186', 'pct_dropout_by_counts-HCAHeartST10317186', 'total_counts-HCAHeartST10317186', 'log1p_total_counts-HCAHeartST10317186', 'n_cells_by_counts-HCAHeartST10317189', 'mean_counts-HCAHeartST10317189', 'log1p_mean_counts-HCAHeartST10317189', 'pct_dropout_by_counts-HCAHeartST10317189', 'total_counts-HCAHeartST10317189', 'log1p_total_counts-HCAHeartST10317189', 'n_cells_by_counts-HCAHeartST10317190', 'mean_counts-HCAHeartST10317190', 'log1p_mean_counts-HCAHeartST10317190', 'pct_dropout_by_counts-HCAHeartST10317190', 'total_counts-HCAHeartST10317190', 'log1p_total_counts-HCAHeartST10317190', 'n_cells_by_counts-HCAHeartST11605165', 'mean_counts-HCAHeartST11605165', 'log1p_mean_counts-HCAHeartST11605165', 'pct_dropout_by_counts-HCAHeartST11605165', 'total_counts-HCAHeartST11605165', 'log1p_total_counts-HCAHeartST11605165', 'n_cells_by_counts-HCAHeartST11605166', 'mean_counts-HCAHeartST11605166', 'log1p_mean_counts-HCAHeartST11605166', 'pct_dropout_by_counts-HCAHeartST11605166', 'total_counts-HCAHeartST11605166', 'log1p_total_counts-HCAHeartST11605166', 'n_cells_by_counts-HCAHeartST11605167', 'mean_counts-HCAHeartST11605167', 'log1p_mean_counts-HCAHeartST11605167', 'pct_dropout_by_counts-HCAHeartST11605167', 'total_counts-HCAHeartST11605167', 'log1p_total_counts-HCAHeartST11605167', 'n_cells_by_counts-HCAHeartST11605168', 'mean_counts-HCAHeartST11605168', 'log1p_mean_counts-HCAHeartST11605168', 'pct_dropout_by_counts-HCAHeartST11605168', 'total_counts-HCAHeartST11605168', 'log1p_total_counts-HCAHeartST11605168', 'n_cells_by_counts-HCAHeartST13162329', 'mean_counts-HCAHeartST13162329', 'log1p_mean_counts-HCAHeartST13162329', 'pct_dropout_by_counts-HCAHeartST13162329', 'total_counts-HCAHeartST13162329', 'log1p_total_counts-HCAHeartST13162329', 'n_cells_by_counts-HCAHeartST13162330', 'mean_counts-HCAHeartST13162330', 'log1p_mean_counts-HCAHeartST13162330', 'pct_dropout_by_counts-HCAHeartST13162330', 'total_counts-HCAHeartST13162330', 'log1p_total_counts-HCAHeartST13162330', 'n_cells_by_counts-HCAHeartST13162331', 'mean_counts-HCAHeartST13162331', 'log1p_mean_counts-HCAHeartST13162331', 'pct_dropout_by_counts-HCAHeartST13162331', 'total_counts-HCAHeartST13162331', 'log1p_total_counts-HCAHeartST13162331', 'n_cells_by_counts-HCAHeartST13162332', 'mean_counts-HCAHeartST13162332', 'log1p_mean_counts-HCAHeartST13162332', 'pct_dropout_by_counts-HCAHeartST13162332', 'total_counts-HCAHeartST13162332', 'log1p_total_counts-HCAHeartST13162332', 'MT_gene'
    uns: 'spatial'
    obsm: 'spatial', 'MT'
```

Example 20 (python):
```python
adata_vis.obs.head()
```

Example 21 (python):
```python
# annotation
path_to_annot='/nfs/team205/heart/visium_annotations/foetal'
category_name='JC_edit'

# read in annotation csv file and concatenate
for i, ID in enumerate(sampleIDs):
    if f'{ID}_JC.csv' in os.listdir(path_to_annot):
    
        print(ID)
        annot_df = pd.read_csv(f'{path_to_annot}/{ID}_JC.csv')
        annot_df['spot_id']=ID+'_'+annot_df['Barcode']

        if i==0:
            annot_df_all=annot_df.copy()
        else:
            annot_df_all=pd.concat([annot_df_all,annot_df])
    else:
        print(f'{ID} is not annotated')

# add annotations to adata_vis
adata_vis.obs=pd.concat([adata_vis.obs,
                     annot_df_all.set_index('spot_id')[[category_name]].reindex(adata_vis.obs.index)],
                     axis=1)
# replace nan to 'none'
adata_vis.obs[category_name].fillna('none', inplace=True)

# minor modification
annotation_edit_dict = {
     'AV Ring':'AV ring',
    'Aorta':'Great vessels',
    'Coronary Artery':'Coronary artery',
    'Mitral Valve':'AV valves',
    'Mitral valve':'AV valves',
    'Myocardium - Ventricular - Trabecular':'Myocardium - Ventricular - Trabeculated',
    'Myocardim - Ventricular - Trabeculated':'Myocardium - Ventricular - Trabeculated',
    'Myocardim - Ventricular - Compact':'Myocardium - Ventricular - Compact',
    'unknown':'Unannotated',
    'Unknown':'Unannotated',
}
adata_vis.obs.replace({category_name:annotation_edit_dict}, inplace=True)
set(adata_vis.obs[category_name])
```

Example 22 (unknown):
```unknown
HCAHeartST10298114
HCAHeartST10298115
HCAHeartST10298116
HCAHeartST10298117
HCAHeartST10298120
HCAHeartST10317184
HCAHeartST10317185
HCAHeartST10317186
HCAHeartST10317189
HCAHeartST10317190
HCAHeartST11605165
HCAHeartST11605166
HCAHeartST11605167
HCAHeartST11605168
HCAHeartST13162329
HCAHeartST13162330
HCAHeartST13162331
HCAHeartST13162332
```

Example 23 (unknown):
```unknown
{'AV ring',
 'AV valves',
 'Aortic valve',
 'Cavity',
 'Connective tissue',
 'Coronary artery',
 'Epicardium',
 'Great vessels',
 'Myocardium - Atrial',
 'Myocardium - Papillary Muscle',
 'Myocardium - Ventricular - Compact',
 'Myocardium - Ventricular - Trabeculated',
 'Out_of_tissue',
 'Unannotated',
 'Valve apparatus',
 'none'}
```

Example 24 (python):
```python
# remove 'none' and 'no_tissue'
adata_vis = adata_vis[adata_vis.obs[category_name].isin(['none','Out_of_tissue'])==False]

# remove 'Cavity'
adata_vis = adata_vis[adata_vis.obs[category_name].isin(['Cavity'])==False]

# remove 'Unannotated'
adata_vis = adata_vis[adata_vis.obs[category_name].isin(['Unannotated'])==False]
```

Example 25 (python):
```python
# order annotations
annotations_order = ['Myocardium - Atrial',
                     'Myocardium - Ventricular - Compact',
                     'Myocardium - Ventricular - Trabeculated',
                    'Myocardium - Papillary Muscle',
                    'Epicardium',
                     'AV valves',
                     'Aortic valve',
                    'Valve apparatus',
                   'AV ring',
                    'Connective tissue',
                     'Great vessels',
                     'Coronary artery',
                     # 'Nerve',
                     ]
adata_vis.obs[category_name] = adata_vis.obs[category_name].astype('category')
adata_vis.obs[category_name].cat.reorder_categories(annotations_order,inplace=True)
```

Example 26 (unknown):
```unknown
Trying to set attribute `.obs` of view, copying.
/tmp/ipykernel_79813/3572501826.py:17: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
```

Example 27 (python):
```python
pd.crosstab(adata_vis.obs['cell2loc-map_group'],adata_vis.obs[category_name])
```

Example 28 (python):
```python
# check the slides for 'PCW-13w'
mask = adata_vis.obs['age']=='PCW-13w'
adata_vis[mask].obs['sample'].value_counts()
# 'HCAHeartST11605166': Atrium
# 'HCAHeartST11605165': Ventricle
```

Example 29 (unknown):
```unknown
HCAHeartST11605165    2428
HCAHeartST11605166    1120
Name: sample, dtype: int64
```

Example 30 (python):
```python
# combine
adata_vis.obs['sample_for_cell2loc']=adata_vis.obs['sample'].astype('str')
adata_vis.obs.replace({'sample_for_cell2loc':{
    'HCAHeartST11605165':'HCAHeartST11605165_HCAHeartST11605166',
    'HCAHeartST11605166':'HCAHeartST11605165_HCAHeartST11605166'
}},inplace=True)
adata_vis.obs['sample_for_cell2loc'].value_counts()
```

Example 31 (unknown):
```unknown
HCAHeartST11605165_HCAHeartST11605166    3548
HCAHeartST13162330                       2806
HCAHeartST13162331                       2338
HCAHeartST13162329                       1623
HCAHeartST10298116                       1187
HCAHeartST10298120                       1108
HCAHeartST10298117                       1085
HCAHeartST10298115                        991
HCAHeartST10317189                        944
HCAHeartST10317184                        836
HCAHeartST10298114                        812
HCAHeartST10317185                        704
HCAHeartST10317186                        693
HCAHeartST11605167                        692
HCAHeartST11605168                        639
HCAHeartST10317190                        609
HCAHeartST13162332                        413
Name: sample_for_cell2loc, dtype: int64
```

Example 32 (python):
```python
adata_vis.obs.columns
```

Example 33 (unknown):
```unknown
Index(['in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts',
       'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts',
       'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes',
       'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac',
       'n_counts', 'n_genes', 'sangerID', 'combinedID', 'donor', 'donor_type',
       'region', 'region_finest', 'age', 'gender', 'facility',
       'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status',
       'cell2loc-map_group', 'JC_edit', 'sample_for_cell2loc'],
      dtype='object')
```

Example 34 (unknown):
```unknown
set(meta['cell2loc-map_group'])
```

Example 35 (unknown):
```unknown
{'13PCW', '20PCW', '4PCW', '5PCW', '7PCW'}
```

Example 36 (python):
```python
# convert adata_vis.var_names to symbol
adata_vis.var.set_index('SYMBOL',inplace=True)
adata_vis.var.head()
```

Example 37 (python):
```python
%%time

# read in reference signatures
inf_aver = pd.read_csv(f"{ref_run_name}/inf_aver.csv", index_col=0)

# find shared genes and subset both anndata and reference signatures
intersect = np.intersect1d(adata_vis.var_names, inf_aver.index)
adata_vis = adata_vis[:, intersect].copy()
inf_aver = inf_aver.loc[intersect, :].copy()
print(f'shape of inf_aver: {inf_aver.shape}')
print('reference cell types')
print(inf_aver.columns)

for group in set(meta['cell2loc-map_group']):
    print(f'####### {group} #######')
    
    # subset adata
    adata_vis_sub=adata_vis[adata_vis.obs['cell2loc-map_group']==group].copy()
    samples = list(set(adata_vis_sub.obs['sample_for_cell2loc']))
    samples.sort()
    print(f'samples in this run: {samples}')
    adata_vis_sub=adata_vis_sub.copy()
    # prepare anndata for cell2location model
    scvi.data.setup_anndata(adata=adata_vis_sub, batch_key="sample_for_cell2loc")
    scvi.data.view_anndata_setup(adata_vis_sub)

    # create and train the model
    mod = cell2location.models.Cell2location(
        adata_vis_sub, cell_state_df=inf_aver, 
        # the expected average cell abundance: tissue-dependent 
        # hyper-prior which can be estimated from paired histology:
        N_cells_per_location=20,
        # hyperparameter controlling normalisation of
        # within-experiment variation in RNA detection (using default here):
        detection_alpha=20
    ) 

    mod.train(max_epochs=20000, 
              # train using full data (batch_size=None)
              batch_size=None, 
              # use all data points in training because 
              # we need to estimate cell abundance at all locations
              train_size=1,
              use_gpu=True)

    # plot ELBO loss history during training, removing first 100 epochs from the plot
    mod.plot_history(1000)
    plt.legend(labels=['full data training']);

    # In this section, we export the estimated cell abundance (summary of the posterior distribution).
    adata_vis_sub = mod.export_posterior(
        adata_vis_sub, sample_kwargs={'num_samples': 1000, 'batch_size': mod.adata.n_obs, 'use_gpu': True}
    )

    # Save model
    mod.save(f"{run_name}/{group}", overwrite=True)

    # Save anndata object with results
    adata_vis_sub.write(f"{run_name}/{group}/sp.h5ad")

    # Examine reconstruction accuracy to assess if there are any issues with mapping
    # the plot should be roughly diagonal, strong deviations will signal problems
    mod.plot_QC()
    
    del mod,adata_vis_sub
    gc.collect()
    torch.cuda.empty_cache()
```

Example 38 (python):
```python
shape of inf_aver: (16314, 57)
reference cell types
Index(['AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight',
       'VentricularCardiomyocytesLeftCompact',
       'VentricularCardiomyocytesRightCompact',
       'VentricularCardiomyocytesLeftTrabeculated',
       'VentricularCardiomyocytesRightTrabeculated',
       'SinoatrialNodeCardiomyocytes', 'AtrioventricularNodeCardiomyocytes',
       'VentricularConductionSystemProximal',
       'VentricularConductionSystemDistal',
       'GreatVesselAdventitialFibroblasts',
       'CoronaryVesselAdventitialFibroblasts',
       'MyocardialInterstitialFibroblasts1',
       'MyocardialInterstitialFibroblasts2', 'Myofibroblasts',
       'LymphNodeFibroblasticReticularCells', 'ValveInterstitialCells',
       'GreatVesselSmoothMuscleCells', 'CoronarySmoothMuscleCells',
       'CoronaryPericytes', 'GreatVesselArterialEndothelialCells',
       'GreatVesselVenousEndothelialCells', 'CoronaryArterialEndothelialCells',
       'CoronaryVenousEndothelialCells', 'CoronaryCapillaryEndothelialCells',
       'EndocardialCells', 'EndocardialCushionCells', 'ValveEndothelialCells',
       'LymphaticEndothelialCells', 'MesothelialEpicardialCells',
       'EpicardiumDerivedCells', 'NeuronPrecursors', 'ChromaffinCells',
       'SympatheticNeurons', 'ParaympatheticNeurons', 'SchwannCellPrecursors',
       'SchwannCells', 'MonocytesMPOpos', 'Monocytes', 'MonocyteDerivedCells',
       'MacrophagesCX3CR1pos', 'MacrophagesTIMD4pos', 'MacrophagesLYVE1pos',
       'MacrophagesATF3pos', 'DendriticCellsType1', 'DendriticCellsMature',
       'PlasmacytoidDendriticCells', 'MastCells', 'Megakaryocytes',
       'TCellsCD4pos', 'TCellsCD8pos', 'TregsCD4pos', 'ProBCells', 'BCells',
       'BCellsMS4A1pos', 'NaturalKillerCells', 'InnateLymphoidCells'],
      dtype='object')
####### 20PCW #######
samples in this run: ['HCAHeartST13162329', 'HCAHeartST13162330', 'HCAHeartST13162331']
INFO     Using batches from adata.obs["sample_for_cell2loc"]                                 
INFO     No label_key inputted, assuming all cells have same label                           
INFO     Using data from adata.X
```

Example 39 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function setup_anndata is deprecated; Please use the model-specific setup_anndata methods instead. The global method will be removed in version 0.15.0.
```

Example 40 (python):
```python
INFO     Successfully registered anndata object containing 6767 cells, 16314 vars, 3 batches,
         1 labels, and 0 proteins. Also registered 0 extra categorical covariates and 0 extra
         continuous covariates.                                                              
INFO     Please do not further modify adata until model is trained.
```

Example 41 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function view_anndata_setup is deprecated; This method will be removed in 0.15.0. Please avoid building any new dependencies on it.
```

Example 42 (unknown):
```unknown
Anndata setup with scvi-tools version 0.14.4.
```

Example 43 (unknown):
```unknown
Data Summary              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃             Data             ┃ Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│            Cells             │ 6767  │
│             Vars             │ 16314 │
│            Labels            │   1   │
│           Batches            │   3   │
│           Proteins           │   0   │
│ Extra Categorical Covariates │   0   │
│ Extra Continuous Covariates  │   0   │
└──────────────────────────────┴───────┘
```

Example 44 (python):
```python
SCVI Data Registry              
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃     Data      ┃    scvi-tools Location    ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│       X       │          adata.X          │
│ batch_indices │ adata.obs['_scvi_batch']  │
│    labels     │ adata.obs['_scvi_labels'] │
└───────────────┴───────────────────────────┘
```

Example 45 (python):
```python
Label Categories                        
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃      Source Location      ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['_scvi_labels'] │     0      │          0          │
└───────────────────────────┴────────────┴─────────────────────┘
```

Example 46 (python):
```python
Batch Categories                                
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃         Source Location          ┃     Categories     ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['sample_for_cell2loc'] │ HCAHeartST13162329 │          0          │
│                                  │ HCAHeartST13162330 │          1          │
│                                  │ HCAHeartST13162331 │          2          │
└──────────────────────────────────┴────────────────────┴─────────────────────┘
```

Example 47 (unknown):
```unknown
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/configuration_validator.py:99: UserWarning: you passed in a val_dataloader but have no validation_step. Skipping val loop
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
```

Example 48 (unknown):
```unknown
Epoch 20000/20000: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20000/20000 [50:20<00:00,  6.62it/s, v_num=1, elbo_train=5.82e+7]
Sampling local variables, batch: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:44<00:00, 44.80s/it]
Sampling global variables, sample: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 999/999 [00:36<00:00, 27.29it/s]
```

Example 49 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sample' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sangerID' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'combinedID' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'donor' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'donor_type' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'region' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'region_finest' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'age' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'gender' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'facility' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'cell_or_nuclei' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'modality' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'kit_10x' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'flushed' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'status' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'cell2loc-map_group' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sample_for_cell2loc' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'feature_types' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'genome' as categorical
```

Example 50 (python):
```python
####### 5PCW #######
samples in this run: ['HCAHeartST11605167', 'HCAHeartST11605168']
INFO     Using batches from adata.obs["sample_for_cell2loc"]                                 
INFO     No label_key inputted, assuming all cells have same label                           
INFO     Using data from adata.X
```

Example 51 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function setup_anndata is deprecated; Please use the model-specific setup_anndata methods instead. The global method will be removed in version 0.15.0.
```

Example 52 (python):
```python
INFO     Successfully registered anndata object containing 1331 cells, 16314 vars, 2 batches,
         1 labels, and 0 proteins. Also registered 0 extra categorical covariates and 0 extra
         continuous covariates.                                                              
INFO     Please do not further modify adata until model is trained.
```

Example 53 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function view_anndata_setup is deprecated; This method will be removed in 0.15.0. Please avoid building any new dependencies on it.
```

Example 54 (unknown):
```unknown
Anndata setup with scvi-tools version 0.14.4.
```

Example 55 (unknown):
```unknown
Data Summary              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃             Data             ┃ Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│            Cells             │ 1331  │
│             Vars             │ 16314 │
│            Labels            │   1   │
│           Batches            │   2   │
│           Proteins           │   0   │
│ Extra Categorical Covariates │   0   │
│ Extra Continuous Covariates  │   0   │
└──────────────────────────────┴───────┘
```

Example 56 (python):
```python
SCVI Data Registry              
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃     Data      ┃    scvi-tools Location    ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│       X       │          adata.X          │
│ batch_indices │ adata.obs['_scvi_batch']  │
│    labels     │ adata.obs['_scvi_labels'] │
└───────────────┴───────────────────────────┘
```

Example 57 (python):
```python
Label Categories                        
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃      Source Location      ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['_scvi_labels'] │     0      │          0          │
└───────────────────────────┴────────────┴─────────────────────┘
```

Example 58 (python):
```python
Batch Categories                                
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃         Source Location          ┃     Categories     ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['sample_for_cell2loc'] │ HCAHeartST11605167 │          0          │
│                                  │ HCAHeartST11605168 │          1          │
└──────────────────────────────────┴────────────────────┴─────────────────────┘
```

Example 59 (unknown):
```unknown
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/configuration_validator.py:99: UserWarning: you passed in a val_dataloader but have no validation_step. Skipping val loop
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
```

Example 60 (unknown):
```unknown
Epoch 20000/20000: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20000/20000 [32:32<00:00, 10.24it/s, v_num=1, elbo_train=1.7e+7]
Sampling local variables, batch: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:23<00:00, 23.56s/it]
Sampling global variables, sample: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 999/999 [00:22<00:00, 43.63it/s]
```

Example 61 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sample' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sangerID' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'combinedID' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'donor' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'donor_type' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'region' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'region_finest' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'age' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'gender' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'facility' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'cell_or_nuclei' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'modality' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'kit_10x' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'flushed' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'status' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'cell2loc-map_group' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sample_for_cell2loc' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'feature_types' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'genome' as categorical
```

Example 62 (python):
```python
####### 7PCW #######
samples in this run: ['HCAHeartST10298114', 'HCAHeartST10298115', 'HCAHeartST10298116', 'HCAHeartST10298117', 'HCAHeartST10298120', 'HCAHeartST10317184', 'HCAHeartST10317185', 'HCAHeartST10317186', 'HCAHeartST10317189', 'HCAHeartST10317190']
INFO     Using batches from adata.obs["sample_for_cell2loc"]                                 
INFO     No label_key inputted, assuming all cells have same label                           
INFO     Using data from adata.X
```

Example 63 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function setup_anndata is deprecated; Please use the model-specific setup_anndata methods instead. The global method will be removed in version 0.15.0.
```

Example 64 (python):
```python
INFO     Successfully registered anndata object containing 8969 cells, 16314 vars, 10        
         batches, 1 labels, and 0 proteins. Also registered 0 extra categorical covariates   
         and 0 extra continuous covariates.                                                  
INFO     Please do not further modify adata until model is trained.
```

Example 65 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function view_anndata_setup is deprecated; This method will be removed in 0.15.0. Please avoid building any new dependencies on it.
```

Example 66 (unknown):
```unknown
Anndata setup with scvi-tools version 0.14.4.
```

Example 67 (unknown):
```unknown
Data Summary              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃             Data             ┃ Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│            Cells             │ 8969  │
│             Vars             │ 16314 │
│            Labels            │   1   │
│           Batches            │  10   │
│           Proteins           │   0   │
│ Extra Categorical Covariates │   0   │
│ Extra Continuous Covariates  │   0   │
└──────────────────────────────┴───────┘
```

Example 68 (python):
```python
SCVI Data Registry              
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃     Data      ┃    scvi-tools Location    ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│       X       │          adata.X          │
│ batch_indices │ adata.obs['_scvi_batch']  │
│    labels     │ adata.obs['_scvi_labels'] │
└───────────────┴───────────────────────────┘
```

Example 69 (python):
```python
Label Categories                        
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃      Source Location      ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['_scvi_labels'] │     0      │          0          │
└───────────────────────────┴────────────┴─────────────────────┘
```

Example 70 (python):
```python
Batch Categories                                
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃         Source Location          ┃     Categories     ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['sample_for_cell2loc'] │ HCAHeartST10298114 │          0          │
│                                  │ HCAHeartST10298115 │          1          │
│                                  │ HCAHeartST10298116 │          2          │
│                                  │ HCAHeartST10298117 │          3          │
│                                  │ HCAHeartST10298120 │          4          │
│                                  │ HCAHeartST10317184 │          5          │
│                                  │ HCAHeartST10317185 │          6          │
│                                  │ HCAHeartST10317186 │          7          │
│                                  │ HCAHeartST10317189 │          8          │
│                                  │ HCAHeartST10317190 │          9          │
└──────────────────────────────────┴────────────────────┴─────────────────────┘
```

Example 71 (unknown):
```unknown
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/configuration_validator.py:99: UserWarning: you passed in a val_dataloader but have no validation_step. Skipping val loop
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
```

Example 72 (unknown):
```unknown
Epoch 20000/20000: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20000/20000 [1:01:37<00:00,  5.41it/s, v_num=1, elbo_train=7.64e+7]
Sampling local variables, batch: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:46<00:00, 46.24s/it]
Sampling global variables, sample: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 999/999 [00:42<00:00, 23.48it/s]
```

Example 73 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sample' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sangerID' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'combinedID' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'donor' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'donor_type' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'region' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'region_finest' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'age' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'gender' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'facility' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'cell_or_nuclei' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'modality' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'kit_10x' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'flushed' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'status' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'cell2loc-map_group' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sample_for_cell2loc' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'feature_types' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'genome' as categorical
```

Example 74 (python):
```python
####### 13PCW #######
samples in this run: ['HCAHeartST11605165_HCAHeartST11605166']
INFO     Using batches from adata.obs["sample_for_cell2loc"]                                 
INFO     No label_key inputted, assuming all cells have same label                           
INFO     Using data from adata.X
```

Example 75 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function setup_anndata is deprecated; Please use the model-specific setup_anndata methods instead. The global method will be removed in version 0.15.0.
```

Example 76 (python):
```python
INFO     Successfully registered anndata object containing 3548 cells, 16314 vars, 1 batches,
         1 labels, and 0 proteins. Also registered 0 extra categorical covariates and 0 extra
         continuous covariates.                                                              
INFO     Please do not further modify adata until model is trained.
```

Example 77 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function view_anndata_setup is deprecated; This method will be removed in 0.15.0. Please avoid building any new dependencies on it.
```

Example 78 (unknown):
```unknown
Anndata setup with scvi-tools version 0.14.4.
```

Example 79 (unknown):
```unknown
Data Summary              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃             Data             ┃ Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│            Cells             │ 3548  │
│             Vars             │ 16314 │
│            Labels            │   1   │
│           Batches            │   1   │
│           Proteins           │   0   │
│ Extra Categorical Covariates │   0   │
│ Extra Continuous Covariates  │   0   │
└──────────────────────────────┴───────┘
```

Example 80 (python):
```python
SCVI Data Registry              
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃     Data      ┃    scvi-tools Location    ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│       X       │          adata.X          │
│ batch_indices │ adata.obs['_scvi_batch']  │
│    labels     │ adata.obs['_scvi_labels'] │
└───────────────┴───────────────────────────┘
```

Example 81 (python):
```python
Label Categories                        
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃      Source Location      ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['_scvi_labels'] │     0      │          0          │
└───────────────────────────┴────────────┴─────────────────────┘
```

Example 82 (python):
```python
Batch Categories                                       
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃        Source Location         ┃             Categories              ┃ scvi-tools Encodin ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['sample_for_cell2loc │ HCAHeartST11605165_HCAHeartST116051 │         0          │
└────────────────────────────────┴─────────────────────────────────────┴────────────────────┘
```

Example 83 (unknown):
```unknown
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/configuration_validator.py:99: UserWarning: you passed in a val_dataloader but have no validation_step. Skipping val loop
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
```

Example 84 (unknown):
```unknown
Epoch 20000/20000: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20000/20000 [38:29<00:00,  8.66it/s, v_num=1, elbo_train=3.63e+7]
Sampling local variables, batch: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:29<00:00, 29.27s/it]
Sampling global variables, sample: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 999/999 [00:28<00:00, 35.40it/s]
```

Example 85 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sample' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sangerID' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'combinedID' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'donor' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'donor_type' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'region' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'region_finest' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'age' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'gender' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'facility' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'cell_or_nuclei' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'modality' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'kit_10x' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'flushed' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'status' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'cell2loc-map_group' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sample_for_cell2loc' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'feature_types' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'genome' as categorical
```

Example 86 (python):
```python
####### 4PCW #######
samples in this run: ['HCAHeartST13162332']
INFO     Using batches from adata.obs["sample_for_cell2loc"]                                 
INFO     No label_key inputted, assuming all cells have same label                           
INFO     Using data from adata.X                                                             
INFO     Successfully registered anndata object containing 413 cells, 16314 vars, 1 batches, 
         1 labels, and 0 proteins. Also registered 0 extra categorical covariates and 0 extra
         continuous covariates.                                                              
INFO     Please do not further modify adata until model is trained.
```

Example 87 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function setup_anndata is deprecated; Please use the model-specific setup_anndata methods instead. The global method will be removed in version 0.15.0.
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function view_anndata_setup is deprecated; This method will be removed in 0.15.0. Please avoid building any new dependencies on it.
```

Example 88 (unknown):
```unknown
Anndata setup with scvi-tools version 0.14.4.
```

Example 89 (unknown):
```unknown
Data Summary              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃             Data             ┃ Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│            Cells             │  413  │
│             Vars             │ 16314 │
│            Labels            │   1   │
│           Batches            │   1   │
│           Proteins           │   0   │
│ Extra Categorical Covariates │   0   │
│ Extra Continuous Covariates  │   0   │
└──────────────────────────────┴───────┘
```

Example 90 (python):
```python
SCVI Data Registry              
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃     Data      ┃    scvi-tools Location    ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│       X       │          adata.X          │
│ batch_indices │ adata.obs['_scvi_batch']  │
│    labels     │ adata.obs['_scvi_labels'] │
└───────────────┴───────────────────────────┘
```

Example 91 (python):
```python
Label Categories                        
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃      Source Location      ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['_scvi_labels'] │     0      │          0          │
└───────────────────────────┴────────────┴─────────────────────┘
```

Example 92 (python):
```python
Batch Categories                                
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃         Source Location          ┃     Categories     ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['sample_for_cell2loc'] │ HCAHeartST13162332 │          0          │
└──────────────────────────────────┴────────────────────┴─────────────────────┘
```

Example 93 (unknown):
```unknown
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/configuration_validator.py:99: UserWarning: you passed in a val_dataloader but have no validation_step. Skipping val loop
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
```

Example 94 (unknown):
```unknown
Epoch 20000/20000: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20000/20000 [30:21<00:00, 10.98it/s, v_num=1, elbo_train=5.22e+6]
Sampling local variables, batch: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:19<00:00, 19.73s/it]
Sampling global variables, sample: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 999/999 [00:20<00:00, 49.56it/s]
```

Example 95 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sample' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sangerID' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'combinedID' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'donor' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'donor_type' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'region' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'region_finest' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'age' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'gender' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'facility' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'cell_or_nuclei' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'modality' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'kit_10x' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'flushed' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'status' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'cell2loc-map_group' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'sample_for_cell2loc' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'feature_types' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'genome' as categorical
```

Example 96 (unknown):
```unknown
CPU times: user 3h 14min 40s, sys: 28min 21s, total: 3h 43min 2s
Wall time: 3h 43min 2s
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no6_clustering-umap_per-mid.html

**Contents:**
- Variables¶
- Read in data¶
- Save UMAP and leiden clustering results¶

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
-----
matplotlib          3.9.2
numpy               2.0.2
pandas              2.2.3
scanpy              1.10.3
seaborn             0.13.2
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                 11.0.0
anndata             0.10.9
asttokens           NA
charset_normalizer  3.4.0
comm                0.2.2
cycler              0.12.1
cython_runtime      NA
dateutil            2.9.0.post0
debugpy             1.8.9
decorator           5.1.1
exceptiongroup      1.2.2
executing           2.1.0
google              NA
h5py                3.12.1
igraph              0.10.2
importlib_metadata  NA
importlib_resources NA
ipykernel           6.29.5
jedi                0.19.2
joblib              1.4.2
kiwisolver          1.4.7
legacy_api_wrap     NA
leidenalg           0.9.0
llvmlite            0.43.0
mpl_toolkits        NA
natsort             8.4.0
numba               0.60.0
packaging           24.2
parso               0.8.4
patsy               1.0.1
pexpect             4.9.0
platformdirs        4.3.6
prompt_toolkit      3.0.48
psutil              6.1.0
ptyprocess          0.7.0
pure_eval           0.2.3
pydev_ipython       NA
pydevconsole        NA
pydevd              3.2.3
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.18.0
pyparsing           3.2.0
pytz                2024.2
scipy               1.13.1
six                 1.16.0
sklearn             1.5.2
stack_data          0.6.3
statsmodels         0.14.4
texttable           1.6.7
threadpoolctl       3.5.0
tornado             6.4.2
traitlets           5.14.3
typing_extensions   NA
wcwidth             0.2.13
yaml                6.0.2
zipp                NA
zmq                 26.2.0
zoneinfo            NA
```

Example 5 (unknown):
```unknown
-----
IPython             8.18.1
jupyter_client      8.6.3
jupyter_core        5.7.2
-----
Python 3.9.20 | packaged by conda-forge | (main, Sep 30 2024, 17:49:10) [GCC 13.3.0]
Linux-5.15.0-126-generic-x86_64-with-glibc2.35
-----
Session information updated at 2024-12-05 23:49
```

Example 6 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 7 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects'
sample_id = 'HEA_FOET14880396'
```

Example 8 (unknown):
```unknown
os.getcwd()
```

Example 9 (unknown):
```unknown
'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3'
```

Example 10 (python):
```python
adata= sc.read_h5ad(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 11 (unknown):
```unknown
AnnData object with n_obs × n_vars = 129803 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 12 (python):
```python
adata.X.data[:5]
```

Example 13 (unknown):
```unknown
array([1.27187559, 1.95641568, 1.75023687, 1.96760567, 2.7810601 ])
```

Example 14 (python):
```python
adata.obs['celltypist_mid_fin'].unique()
```

Example 15 (unknown):
```unknown
['LymphoidCells', 'MyeloidCells', 'unassigned', 'Fibroblasts', 'PericardialCells', ..., 'Glia', 'EpicardialCells', 'AtrialCardiomyocytes', 'EndocardialCells', 'VentricularCardiomyocytes']
Length: 14
Categories (14, object): ['AtrialCardiomyocytes', 'BloodVesselEndothelialCells', 'EndocardialCells', 'EpicardialCells', ..., 'Neurons', 'PericardialCells', 'VentricularCardiomyocytes', 'unassigned']
```

Example 16 (python):
```python
sc.pl.embedding(adata,
                    basis=f"X_umap",
                    color='celltypist_mid_fin',
                groups='unassigned',
                   ncols=4
                   )
```

Example 17 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets'
```

Example 18 (python):
```python
adata.uns['log1p']['base'] = None
```

Example 19 (python):
```python
%%time
for celltype in adata.obs['celltypist_mid_fin'].unique():
    if celltype == 'unassigned':
        continue
    
    print(f'##### {celltype} #####')
    # subset
    adata_sub = adata[adata.obs['celltypist_mid_fin']==celltype]
    sc.pp.filter_genes(adata_sub, min_cells=3)
    print(adata_sub.shape)
    # HVGs
    sc.pp.highly_variable_genes(adata_sub, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
    print(f'number of hvgs: {sum(adata_sub.var.highly_variable)}')
    # PCA
    sc.tl.pca(adata_sub, use_highly_variable=True)
    # create neighbour graph
    sc.pp.neighbors(adata_sub,n_neighbors=15,use_rep='X_pca')
    # visualisation with umap
    sc.tl.umap(adata_sub,min_dist=0.5)
    sc.pl.embedding(adata_sub,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                    wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
    # leiden clustering
    resolutions_list = [0.2,0.4,0.6,1,1.2,1.5,2]
    for resolution in resolutions_list:
        sc.tl.leiden(adata_sub,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
    sc.pl.embedding(adata_sub,basis='X_umap',
                    color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                    wspace=0.3)
    # save
    adata_sub.write(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
    print('')
```

Example 20 (unknown):
```unknown
##### LymphoidCells #####
```

Example 21 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 22 (unknown):
```unknown
(87, 10288)
number of hvgs: 2647
```

Example 23 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_pca.py:377: FutureWarning: Argument `use_highly_variable` is deprecated, consider using the mask argument. Use_highly_variable=True can be called through mask_var="highly_variable". Use_highly_variable=False can be called through mask_var=None
  warn(msg, FutureWarning)
```

Example 24 (unknown):
```unknown
<timed exec>:24: FutureWarning: In the future, the default backend for leiden will be igraph instead of leidenalg.

 To achieve the future defaults please pass: flavor="igraph" and n_iterations=2.  directed must also be False to work with igraph's implementation.
```

Example 25 (unknown):
```unknown
##### MyeloidCells #####
```

Example 26 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 27 (unknown):
```unknown
(526, 13502)
number of hvgs: 3182
```

Example 28 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_pca.py:377: FutureWarning: Argument `use_highly_variable` is deprecated, consider using the mask argument. Use_highly_variable=True can be called through mask_var="highly_variable". Use_highly_variable=False can be called through mask_var=None
  warn(msg, FutureWarning)
```

Example 29 (unknown):
```unknown
##### Fibroblasts #####
```

Example 30 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 31 (unknown):
```unknown
(20318, 17926)
number of hvgs: 3448
```

Example 32 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_pca.py:377: FutureWarning: Argument `use_highly_variable` is deprecated, consider using the mask argument. Use_highly_variable=True can be called through mask_var="highly_variable". Use_highly_variable=False can be called through mask_var=None
  warn(msg, FutureWarning)
```

Example 33 (unknown):
```unknown
##### PericardialCells #####
```

Example 34 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 35 (unknown):
```unknown
(3061, 15212)
number of hvgs: 3858
```

Example 36 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_pca.py:377: FutureWarning: Argument `use_highly_variable` is deprecated, consider using the mask argument. Use_highly_variable=True can be called through mask_var="highly_variable". Use_highly_variable=False can be called through mask_var=None
  warn(msg, FutureWarning)
```

Example 37 (unknown):
```unknown
##### LymphaticEndothelialCells #####
```

Example 38 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 39 (unknown):
```unknown
(1025, 14030)
number of hvgs: 3656
```

Example 40 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_pca.py:377: FutureWarning: Argument `use_highly_variable` is deprecated, consider using the mask argument. Use_highly_variable=True can be called through mask_var="highly_variable". Use_highly_variable=False can be called through mask_var=None
  warn(msg, FutureWarning)
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/numpy/lib/_nanfunctions_impl.py:1424: RuntimeWarning: All-NaN slice encountered
  return _nanquantile_unchecked(
```

Example 41 (unknown):
```unknown
##### BloodVesselEndothelialCells #####
```

Example 42 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 43 (unknown):
```unknown
(1765, 15377)
number of hvgs: 3724
```

Example 44 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_pca.py:377: FutureWarning: Argument `use_highly_variable` is deprecated, consider using the mask argument. Use_highly_variable=True can be called through mask_var="highly_variable". Use_highly_variable=False can be called through mask_var=None
  warn(msg, FutureWarning)
```

Example 45 (unknown):
```unknown
##### MuralCells #####
```

Example 46 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 47 (unknown):
```unknown
(10785, 17693)
number of hvgs: 3102
```

Example 48 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_pca.py:377: FutureWarning: Argument `use_highly_variable` is deprecated, consider using the mask argument. Use_highly_variable=True can be called through mask_var="highly_variable". Use_highly_variable=False can be called through mask_var=None
  warn(msg, FutureWarning)
```

Example 49 (unknown):
```unknown
##### Neurons #####
```

Example 50 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 51 (unknown):
```unknown
(589, 14494)
number of hvgs: 3715
```

Example 52 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_pca.py:377: FutureWarning: Argument `use_highly_variable` is deprecated, consider using the mask argument. Use_highly_variable=True can be called through mask_var="highly_variable". Use_highly_variable=False can be called through mask_var=None
  warn(msg, FutureWarning)
```

Example 53 (unknown):
```unknown
##### Glia #####
```

Example 54 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 55 (unknown):
```unknown
(588, 13981)
number of hvgs: 3625
```

Example 56 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_pca.py:377: FutureWarning: Argument `use_highly_variable` is deprecated, consider using the mask argument. Use_highly_variable=True can be called through mask_var="highly_variable". Use_highly_variable=False can be called through mask_var=None
  warn(msg, FutureWarning)
```

Example 57 (unknown):
```unknown
##### EpicardialCells #####
```

Example 58 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 59 (unknown):
```unknown
(603, 13050)
number of hvgs: 3578
```

Example 60 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_pca.py:377: FutureWarning: Argument `use_highly_variable` is deprecated, consider using the mask argument. Use_highly_variable=True can be called through mask_var="highly_variable". Use_highly_variable=False can be called through mask_var=None
  warn(msg, FutureWarning)
```

Example 61 (unknown):
```unknown
##### AtrialCardiomyocytes #####
```

Example 62 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 63 (unknown):
```unknown
(13904, 17650)
number of hvgs: 3349
```

Example 64 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_pca.py:377: FutureWarning: Argument `use_highly_variable` is deprecated, consider using the mask argument. Use_highly_variable=True can be called through mask_var="highly_variable". Use_highly_variable=False can be called through mask_var=None
  warn(msg, FutureWarning)
```

Example 65 (unknown):
```unknown
##### EndocardialCells #####
```

Example 66 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 67 (unknown):
```unknown
(1526, 14451)
number of hvgs: 3615
```

Example 68 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_pca.py:377: FutureWarning: Argument `use_highly_variable` is deprecated, consider using the mask argument. Use_highly_variable=True can be called through mask_var="highly_variable". Use_highly_variable=False can be called through mask_var=None
  warn(msg, FutureWarning)
```

Example 69 (unknown):
```unknown
##### VentricularCardiomyocytes #####
```

Example 70 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 71 (unknown):
```unknown
(61075, 17994)
number of hvgs: 2287
```

Example 72 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_pca.py:377: FutureWarning: Argument `use_highly_variable` is deprecated, consider using the mask argument. Use_highly_variable=True can be called through mask_var="highly_variable". Use_highly_variable=False can be called through mask_var=None
  warn(msg, FutureWarning)
```

Example 73 (unknown):
```unknown
CPU times: user 4min 18s, sys: 49.1 s, total: 5min 7s
Wall time: 6min 48s
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/Visium/cell2location/wholeheart_without_DA/no1_reference-signatures_woPericardial_woDASMCs.html

**Contents:**
- Loading packages¶
- Data, parameters¶
- Loading scRNA-seq reference data¶
- Remove cells¶
- Add Atria Ventricle key
- Estimation of reference cell type signatures (NB regression)¶

**Examples:**

Example 1 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl

import cell2location
import scvi

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text
import seaborn as sns
```

Example 2 (unknown):
```unknown
Global seed set to 0
```

Example 3 (python):
```python
import session_info
session_info.show()
```

Example 4 (python):
```python
-----
anndata             0.7.8
cell2location       NA
matplotlib          3.5.0
numpy               1.20.3
pandas              1.3.4
scanpy              1.9.3
scvi                0.14.4
seaborn             0.11.2
session_info        1.0.0
-----
```

Example 5 (python):
```python
PIL                         8.4.0
absl                        NA
adjustText                  NA
arviz                       0.10.0
attr                        21.2.0
backcall                    0.2.0
beta_ufunc                  NA
binom_ufunc                 NA
cffi                        1.15.0
cftime                      1.5.1.1
colorama                    0.4.4
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.5.1
decorator                   5.1.0
defusedxml                  0.7.1
deprecate                   0.3.0
descartes                   NA
dill                        0.3.4
docrep                      0.3.2
entrypoints                 0.3
fastprogress                0.2.7
fontTools                   4.28.1
fsspec                      2021.11.0
google                      NA
h5py                        3.6.0
igraph                      0.9.8
ipykernel                   6.5.1
ipython_genutils            0.2.0
ipywidgets                  7.6.5
jedi                        0.18.1
joblib                      1.1.0
kiwisolver                  1.3.2
lazylinker_ext              NA
leidenalg                   0.8.8
llvmlite                    0.37.0
matplotlib_inline           NA
mizani                      0.7.3
mpl_toolkits                NA
natsort                     8.0.0
nbinom_ufunc                NA
netCDF4                     1.5.8
numba                       0.54.1
numexpr                     2.7.3
opt_einsum                  v3.3.0
packaging                   21.3
palettable                  3.3.0
parso                       0.8.2
patsy                       0.5.2
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
plotnine                    0.8.0
prompt_toolkit              3.0.22
ptyprocess                  0.7.0
pycparser                   2.21
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.6.0
pydevd_concurrency_analyser NA
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pyexpat                     NA
pygments                    2.10.0
pymc3                       3.10.0
pynndescent                 0.5.5
pyparsing                   3.0.6
pyro                        1.7.0
pytorch_lightning           1.3.8
pytz                        2021.3
rich                        NA
scipy                       1.7.2
setuptools                  58.0.4
setuptools_scm              NA
six                         1.16.0
sklearn                     1.0.1
statsmodels                 0.13.1
storemagic                  NA
tensorboard                 2.7.0
texttable                   1.6.4
theano                      1.0.11
threadpoolctl               3.0.0
torch                       1.10.0+cu102
torchmetrics                0.6.0
tornado                     6.1
tqdm                        4.62.3
traitlets                   5.1.1
typing_extensions           NA
umap                        0.5.2
wcwidth                     0.2.5
xarray                      0.20.1
yaml                        5.4.1
zmq                         22.3.0
```

Example 6 (unknown):
```unknown
-----
IPython             7.29.0
jupyter_client      7.0.6
jupyter_core        4.9.1
notebook            6.4.6
-----
Python 3.9.7 (default, Sep 16 2021, 13:09:58) [GCC 7.5.0]
Linux-4.15.0-200-generic-x86_64-with-glibc2.27
-----
Session information updated at 2023-03-14 11:24
```

Example 7 (unknown):
```unknown
results_folder = '/nfs/team205/heart/cell2location/foetal'

# create paths and names to results folders for reference regression and cell2location models
ref_run_name = f'{results_folder}/reference_signatures/ver28Feb2023'
run_name = f'{results_folder}/cell2location_map/ver28Feb2023'
```

Example 8 (unknown):
```unknown
scRNA_path='/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw.h5ad'
celltype_key='fine_grain'
```

Example 9 (python):
```python
# Read scRNA data
adata = sc.read(scRNA_path)
adata
```

Example 10 (unknown):
```unknown
AnnData object with n_obs × n_vars = 297473 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'region', 'age', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'donor_by_library-prep', 'multiplexed', 'SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet', 'batch_key', '_scvi_batch', 'FACSgate', 'fine_grain', 'mid_grain', 'coarse_grain', 'sex', 'week', 'trimester', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', '_scvi_labels', 'stress_score', 'hb1_score'
    var: 'gene_name_scRNA-0', 'gene_id'
    uns: 'FACSgate_colors', '_scvi_manager_uuid', '_scvi_uuid', 'age_colors', 'cell_or_nuclei_colors', 'coarse_grain_colors', 'cycling_colors', 'donor_colors', 'fine_grain_colors', 'heart_or_greatvessels_colors', 'hvg', 'kit_10x_colors', 'mid_grain_colors', 'modality_colors', 'neighbors', 'phase_colors', 'region_colors', 'sex_colors', 'trimester_colors', 'umap', 'week_colors'
    obsm: 'X_scVI', 'X_umap', '_scvi_extra_categorical_covs', '_scvi_extra_continuous', '_scvi_extra_continuous_covs', 'latent_gene_encoding'
    obsp: 'connectivities', 'distances'
```

Example 11 (python):
```python
adata.X.data[:10]
```

Example 12 (unknown):
```unknown
array([ 1.,  1.,  5.,  5.,  8.,  4., 13.,  4.,  3.,  1.], dtype=float32)
```

Example 13 (python):
```python
set(adata.obs[celltype_key])
```

Example 14 (unknown):
```unknown
{'AtrialCardiomyocytesCycling',
 'AtrialCardiomyocytesLeft',
 'AtrialCardiomyocytesRight',
 'AtrioventricularNodeCardiomyocytes',
 'BCells',
 'BCellsMS4A1pos',
 'ChromaffinCells',
 'CoronaryArterialEndothelialCells',
 'CoronaryCapillaryEndothelialCells',
 'CoronaryPericytes',
 'CoronarySmoothMuscleCells',
 'CoronaryVenousEndothelialCells',
 'CoronaryVesselAdventitialFibroblasts',
 'DendriticCellsMature',
 'DendriticCellsType1',
 'DuctusArteriosusSmoothMuscleCells',
 'EndocardialCells',
 'EndocardialCushionCells',
 'EpicardiumDerivedCells',
 'GreatVesselAdventitialFibroblasts',
 'GreatVesselArterialEndothelialCells',
 'GreatVesselSmoothMuscleCells',
 'GreatVesselVenousEndothelialCells',
 'InnateLymphoidCells',
 'LymphNodeFibroblasticReticularCells',
 'LymphaticEndothelialCells',
 'MacrophagesATF3pos',
 'MacrophagesCX3CR1pos',
 'MacrophagesLYVE1pos',
 'MacrophagesTIMD4pos',
 'MastCells',
 'Megakaryocytes',
 'MesothelialEpicardialCells',
 'MonocyteDerivedCells',
 'Monocytes',
 'MonocytesMPOpos',
 'MyocardialInterstitialFibroblasts1',
 'MyocardialInterstitialFibroblasts2',
 'Myofibroblasts',
 'NaturalKillerCells',
 'NeuronPrecursors',
 'ParaympatheticNeurons',
 'PericardialCells1',
 'PericardialCells2',
 'PericardialCells3',
 'PlasmacytoidDendriticCells',
 'ProBCells',
 'SchwannCellPrecursors',
 'SchwannCells',
 'SinoatrialNodeCardiomyocytes',
 'SympatheticNeurons',
 'TCellsCD4pos',
 'TCellsCD8pos',
 'TregsCD4pos',
 'ValveEndothelialCells',
 'ValveInterstitialCells',
 'VentricularCardiomyocytesCycling',
 'VentricularCardiomyocytesLeftCompact',
 'VentricularCardiomyocytesLeftTrabeculated',
 'VentricularCardiomyocytesRightCompact',
 'VentricularCardiomyocytesRightTrabeculated',
 'VentricularConductionSystemDistal',
 'VentricularConductionSystemProximal'}
```

Example 15 (python):
```python
# Make sure .var_names are ENSEMBL IDs
adata.var.head()
```

Example 16 (python):
```python
adata.obs['batch_key'].value_counts()
```

Example 17 (unknown):
```unknown
Hst41_Nuclei_Multiome-v1    104878
Hst33_Nuclei_Multiome-v1     38234
Hst39_Nuclei_Multiome-v1     25386
C99_Cell_5prime-v2           17547
C92_Nuclei_Multiome-v1       13770
C94_Cell_5prime-v2           12985
Hst40_Nuclei_Multiome-v1      9708
C97_Cell_5prime-v2            9702
BRC2251_Cell_3prime-v3        9121
BRC2256_Cell_3prime-v3        7948
Hst36_Nuclei_Multiome-v1      7268
BRC2252_Cell_3prime-v3        6854
C86_Cell_5prime-v2            6376
BRC2260_Cell_3prime-v3        6004
BRC2263_Cell_3prime-v3        5662
BRC2262_Cell_3prime-v3        4129
C87_Nuclei_Multiome-v1        3907
C98_Nuclei_Multiome-v1        2660
C104_Nuclei_Multiome-v1       2490
C85_Nuclei_Multiome-v1        2436
C82_Nuclei_Multiome-v1         408
Name: batch_key, dtype: int64
```

Example 18 (python):
```python
set(adata.obs['batch_key'])
```

Example 19 (unknown):
```unknown
{'BRC2251_Cell_3prime-v3',
 'BRC2252_Cell_3prime-v3',
 'BRC2256_Cell_3prime-v3',
 'BRC2260_Cell_3prime-v3',
 'BRC2262_Cell_3prime-v3',
 'BRC2263_Cell_3prime-v3',
 'C104_Nuclei_Multiome-v1',
 'C82_Nuclei_Multiome-v1',
 'C85_Nuclei_Multiome-v1',
 'C86_Cell_5prime-v2',
 'C87_Nuclei_Multiome-v1',
 'C92_Nuclei_Multiome-v1',
 'C94_Cell_5prime-v2',
 'C97_Cell_5prime-v2',
 'C98_Nuclei_Multiome-v1',
 'C99_Cell_5prime-v2',
 'Hst33_Nuclei_Multiome-v1',
 'Hst36_Nuclei_Multiome-v1',
 'Hst39_Nuclei_Multiome-v1',
 'Hst40_Nuclei_Multiome-v1',
 'Hst41_Nuclei_Multiome-v1'}
```

Example 20 (python):
```python
# remove pericardial cells
adata = adata[adata.obs['fine_grain'].isin(['PericardialCells1','PericardialCells2','PericardialCells3'])==False]

# remove ductus arterial SMCs
adata = adata[adata.obs['fine_grain'].isin(['DuctusArteriosusSmoothMuscleCells'])==False]

# remove cycling cells
adata = adata[adata.obs['cycling']=='no']

adata.shape
```

Example 21 (unknown):
```unknown
(185953, 36601)
```

Example 22 (python):
```python
'''
atrial_or_ventricular = {
    # 'AtrialCardiomyocytesCycling':'atrial',
 'AtrialCardiomyocytesLeft':'atrial',
 'AtrialCardiomyocytesRight':'atrial',
 'AtrioventricularNodeCardiomyocytes':'either',
 'BCells':'either',
 'BCellsMS4A1pos':'either',
 'ChromaffinCells':'either',
 'CoronaryArterialEndothelialCells':'either',
 'CoronaryCapillaryEndothelialCells':'either',
 'CoronaryPericytes':'either',
 'CoronarySmoothMuscleCells':'either',
 'CoronaryVenousEndothelialCells':'either',
 'CoronaryVesselAdventitialFibroblasts':'either',
 'DendriticCellsMature':'either',
 'DendriticCellsType1':'either',
 # 'DuctusArteriosusSmoothMuscleCells':'atrial',
 'EndocardialCells':'either',
 'EndocardialCushionCells':'either',
 'EpicardiumDerivedCells':'either',
 'GreatVesselAdventitialFibroblasts':'atrial',
 'GreatVesselArterialEndothelialCells':'atrial',
 'GreatVesselSmoothMuscleCells':'atrial',
 'GreatVesselVenousEndothelialCells':'atrial',
 'InnateLymphoidCells':'either',
 'LymphNodeFibroblasticReticularCells':'either',
 'LymphaticEndothelialCells':'either',
 'MacrophagesATF3pos':'either',
 'MacrophagesCX3CR1pos':'either',
 'MacrophagesLYVE1pos':'either',
 'MacrophagesTIMD4pos':'either',
 'MastCells':'either',
 'Megakaryocytes':'either',
 'MesothelialEpicardialCells':'either',
 'MonocyteDerivedCells':'either',
 'Monocytes':'either',
 'MonocytesMPOpos':'either',
 'MyocardialInterstitialFibroblasts1':'either',
 'MyocardialInterstitialFibroblasts2':'either',
 'Myofibroblasts':'either',
 'NaturalKillerCells':'either',
 'NeuronPrecursors':'either',
 'ParaympatheticNeurons':'either',
 # 'PericardialCells1':'either',
 # 'PericardialCells2':'either',
 # 'PericardialCells3':'either',
 'PlasmacytoidDendriticCells':'either',
 'ProBCells':'either',
 'SchwannCellPrecursors':'either',
 'SchwannCells':'either',
 'SinoatrialNodeCardiomyocytes':'atrial',
 'SympatheticNeurons':'either',
 'TCellsCD4pos':'either',
 'TCellsCD8pos':'either',
 'TregsCD4pos':'either',
 'ValveEndothelialCells':'either',
 'ValveInterstitialCells':'either',
 # 'VentricularCardiomyocytesCycling':'ventricular',
 'VentricularCardiomyocytesLeftCompact':'ventricular',
 'VentricularCardiomyocytesLeftTrabeculated':'ventricular',
 'VentricularCardiomyocytesRightCompact':'ventricular',
 'VentricularCardiomyocytesRightTrabeculated':'ventricular',
 'VentricularConductionSystemDistal':'ventricular',
 'VentricularConductionSystemProximal':'ventricular'}

if set(adata.obs['fine_grain']) == set(atrial_or_ventricular.keys()):
    adata.obs['atrialorventricular']=adata.obs['fine_grain'].astype('str').copy()
    adata.obs.replace({'atrialorventricular':atrial_or_ventricular},inplace=True)
else:
    print('ERROR: the cell type names are not matching')

pd.set_option("max_rows", 100)
pd.crosstab(adata.obs['fine_grain'],adata.obs['atrialorventricular'])
'''
```

Example 23 (unknown):
```unknown
regions = ['whole_woPeriCell_woDASMC'] #,'atrial','ventricular'
```

Example 24 (python):
```python
%%time

from cell2location.utils.filtering import filter_genes
from cell2location.models import RegressionModel

for reg in regions:
    print(f'##### {reg} #####')
    
    # subset adata
    if reg == 'whole_woPeriCell_woDASMC':
        adata_ref = adata.copy()
    else:
        adata_ref = adata[adata.obs['atrialorventricular'].isin(['either',reg])]
    
    # show cell types
    adata_ref.obs['fine_grain'].value_counts()
    
    # before we estimate the reference cell type signature we recommend to perform very permissive genes selection
    # in this 2D histogram orange rectangle lays over excluded genes.
    # In this case, the downloaded dataset was already filtered using this method,
    # hence no density under the orange rectangle
    selected = filter_genes(adata_ref, cell_count_cutoff=5, cell_percentage_cutoff2=0.03, nonz_mean_cutoff=1.12)
    # filter the object
    adata_ref = adata_ref[:, selected].copy()

    # remove cell_states, which have only 2 cells or less
    counts = adata_ref.obs[celltype_key].value_counts()
    adata_ref = adata_ref[adata_ref.obs[celltype_key].isin(list(counts.keys()[counts>2]))]

    # prepare anndata for the regression model
    adata_ref=adata_ref.copy()
    scvi.data.setup_anndata(adata=adata_ref, 
                            # 10X reaction / sample / batch
                            batch_key='batch_key', 
                            # cell type, covariate used for constructing signatures
                            labels_key=celltype_key, 
                            # multiplicative technical effects (platform, 3' vs 5', donor effect)
                            # categorical_covariate_keys=['Method']
                           )
    scvi.data.view_anndata_setup(adata_ref)

    # create and train the regression model
    mod = RegressionModel(adata_ref) 

    # Use all data for training (validation not implemented yet, train_size=1)
    mod.train(max_epochs=1000, batch_size=2500, train_size=1, lr=0.002, use_gpu=True)

    # plot ELBO loss history during training, removing first 20 epochs from the plot
    mod.plot_history(20)

    # In this section, we export the estimated cell abundance (summary of the posterior distribution).
    adata_ref = mod.export_posterior(
        adata_ref, sample_kwargs={'num_samples': 1000, 'batch_size': 2500, 'use_gpu': True}
    )

    # Save model
    mod.save(f"{ref_run_name}/{reg}/", overwrite=True)

    # Save anndata object with results
    # adata_file = f"{ref_run_name}/{reg}/sc.h5ad"
    adata_ref.write(f"{ref_run_name}/{reg}/sc.h5ad")

    mod.plot_QC()

    # export estimated expression in each cluster
    if 'means_per_cluster_mu_fg' in adata_ref.varm.keys():
        inf_aver = adata_ref.varm['means_per_cluster_mu_fg'][[f'means_per_cluster_mu_fg_{i}' 
                                        for i in adata_ref.uns['mod']['factor_names']]].copy()
    else:
        inf_aver = adata_ref.var[[f'means_per_cluster_mu_fg_{i}' 
                                        for i in adata_ref.uns['mod']['factor_names']]].copy()
    inf_aver.columns = adata_ref.uns['mod']['factor_names']
    inf_aver.iloc[0:5, 0:5]

    # save inf_aver
    inf_aver.to_csv(f"{ref_run_name}/{reg}/inf_aver.csv")
    
    del adata_ref,mod,inf_aver,selected,counts
```

Example 25 (unknown):
```unknown
##### whole_woPeriCell_woDASMC #####
```

Example 26 (python):
```python
INFO     Using batches from adata.obs["batch_key"]                                           
INFO     Using labels from adata.obs["fine_grain"]                                           
INFO     Using data from adata.X
```

Example 27 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function setup_anndata is deprecated; Please use the model-specific setup_anndata methods instead. The global method will be removed in version 0.15.0.
```

Example 28 (python):
```python
INFO     Successfully registered anndata object containing 185953 cells, 17286 vars, 21      
         batches, 57 labels, and 0 proteins. Also registered 0 extra categorical covariates  
         and 0 extra continuous covariates.                                                  
INFO     Please do not further modify adata until model is trained.
```

Example 29 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function view_anndata_setup is deprecated; This method will be removed in 0.15.0. Please avoid building any new dependencies on it.
```

Example 30 (unknown):
```unknown
Anndata setup with scvi-tools version 0.14.4.
```

Example 31 (unknown):
```unknown
Data Summary               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃             Data             ┃ Count  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│            Cells             │ 185953 │
│             Vars             │ 17286  │
│            Labels            │   57   │
│           Batches            │   21   │
│           Proteins           │   0    │
│ Extra Categorical Covariates │   0    │
│ Extra Continuous Covariates  │   0    │
└──────────────────────────────┴────────┘
```

Example 32 (python):
```python
SCVI Data Registry              
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃     Data      ┃    scvi-tools Location    ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│       X       │          adata.X          │
│ batch_indices │ adata.obs['_scvi_batch']  │
│    labels     │ adata.obs['_scvi_labels'] │
└───────────────┴───────────────────────────┘
```

Example 33 (python):
```python
Label Categories                                       
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃     Source Location     ┃                 Categories                 ┃ scvi-tools Encodin ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['fine_grain'] │          AtrialCardiomyocytesLeft          │         0          │
│                         │         AtrialCardiomyocytesRight          │         1          │
│                         │    VentricularCardiomyocytesLeftCompact    │         2          │
│                         │   VentricularCardiomyocytesRightCompact    │         3          │
│                         │ VentricularCardiomyocytesLeftTrabeculated  │         4          │
│                         │ VentricularCardiomyocytesRightTrabeculated │         5          │
│                         │        SinoatrialNodeCardiomyocytes        │         6          │
│                         │     AtrioventricularNodeCardiomyocytes     │         7          │
│                         │    VentricularConductionSystemProximal     │         8          │
│                         │     VentricularConductionSystemDistal      │         9          │
│                         │     GreatVesselAdventitialFibroblasts      │         10         │
│                         │    CoronaryVesselAdventitialFibroblasts    │         11         │
│                         │     MyocardialInterstitialFibroblasts1     │         12         │
│                         │     MyocardialInterstitialFibroblasts2     │         13         │
│                         │               Myofibroblasts               │         14         │
│                         │    LymphNodeFibroblasticReticularCells     │         15         │
│                         │           ValveInterstitialCells           │         16         │
│                         │        GreatVesselSmoothMuscleCells        │         17         │
│                         │         CoronarySmoothMuscleCells          │         18         │
│                         │             CoronaryPericytes              │         19         │
│                         │    GreatVesselArterialEndothelialCells     │         20         │
│                         │     GreatVesselVenousEndothelialCells      │         21         │
│                         │      CoronaryArterialEndothelialCells      │         22         │
│                         │       CoronaryVenousEndothelialCells       │         23         │
│                         │     CoronaryCapillaryEndothelialCells      │         24         │
│                         │              EndocardialCells              │         25         │
│                         │          EndocardialCushionCells           │         26         │
│                         │           ValveEndothelialCells            │         27         │
│                         │         LymphaticEndothelialCells          │         28         │
│                         │         MesothelialEpicardialCells         │         29         │
│                         │           EpicardiumDerivedCells           │         30         │
│                         │              NeuronPrecursors              │         31         │
│                         │              ChromaffinCells               │         32         │
│                         │             SympatheticNeurons             │         33         │
│                         │           ParaympatheticNeurons            │         34         │
│                         │           SchwannCellPrecursors            │         35         │
│                         │                SchwannCells                │         36         │
│                         │              MonocytesMPOpos               │         37         │
│                         │                 Monocytes                  │         38         │
│                         │            MonocyteDerivedCells            │         39         │
│                         │            MacrophagesCX3CR1pos            │         40         │
│                         │            MacrophagesTIMD4pos             │         41         │
│                         │            MacrophagesLYVE1pos             │         42         │
│                         │             MacrophagesATF3pos             │         43         │
│                         │            DendriticCellsType1             │         44         │
│                         │            DendriticCellsMature            │         45         │
│                         │         PlasmacytoidDendriticCells         │         46         │
│                         │                 MastCells                  │         47         │
│                         │               Megakaryocytes               │         48         │
│                         │                TCellsCD4pos                │         49         │
│                         │                TCellsCD8pos                │         50         │
│                         │                TregsCD4pos                 │         51         │
│                         │                 ProBCells                  │         52         │
│                         │                   BCells                   │         53         │
│                         │               BCellsMS4A1pos               │         54         │
│                         │             NaturalKillerCells             │         55         │
│                         │            InnateLymphoidCells             │         56         │
└─────────────────────────┴────────────────────────────────────────────┴────────────────────┘
```

Example 34 (python):
```python
Batch Categories                              
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃    Source Location     ┃        Categories        ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['batch_key'] │  BRC2251_Cell_3prime-v3  │          0          │
│                        │  BRC2252_Cell_3prime-v3  │          1          │
│                        │  BRC2256_Cell_3prime-v3  │          2          │
│                        │  BRC2260_Cell_3prime-v3  │          3          │
│                        │  BRC2262_Cell_3prime-v3  │          4          │
│                        │  BRC2263_Cell_3prime-v3  │          5          │
│                        │  C82_Nuclei_Multiome-v1  │          6          │
│                        │  C85_Nuclei_Multiome-v1  │          7          │
│                        │    C86_Cell_5prime-v2    │          8          │
│                        │  C87_Nuclei_Multiome-v1  │          9          │
│                        │  C92_Nuclei_Multiome-v1  │         10          │
│                        │    C94_Cell_5prime-v2    │         11          │
│                        │    C97_Cell_5prime-v2    │         12          │
│                        │  C98_Nuclei_Multiome-v1  │         13          │
│                        │    C99_Cell_5prime-v2    │         14          │
│                        │ C104_Nuclei_Multiome-v1  │         15          │
│                        │ Hst33_Nuclei_Multiome-v1 │         16          │
│                        │ Hst36_Nuclei_Multiome-v1 │         17          │
│                        │ Hst39_Nuclei_Multiome-v1 │         18          │
│                        │ Hst40_Nuclei_Multiome-v1 │         19          │
│                        │ Hst41_Nuclei_Multiome-v1 │         20          │
└────────────────────────┴──────────────────────────┴─────────────────────┘
```

Example 35 (unknown):
```unknown
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/configuration_validator.py:99: UserWarning: you passed in a val_dataloader but have no validation_step. Skipping val loop
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
```

Example 36 (unknown):
```unknown
Epoch 1000/1000: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [6:02:33<00:00, 21.75s/it, v_num=1, elbo_train=1.19e+9]
Sampling local variables, batch:   0%|                                                                                                                                                                             | 0/75 [00:00<?, ?it/s]
Sampling global variables, sample: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 999/999 [00:16<00:00, 59.54it/s]
```

Example 37 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'atrialorventricular' as categorical
```

Example 38 (unknown):
```unknown
CPU times: user 5h 33min 38s, sys: 2h 59min 2s, total: 8h 32min 41s
Wall time: 6h 5min 40s
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no7_manual-annotation_MuralCells.html

**Contents:**
- Variables¶
- Read in data¶
- Remove mid-grain markers from the feature¶
- PCA, Clustering, UMAP¶
- Markers¶
- Subset "uncertain" cells¶
- Add the annotation to the original adata¶
- Save¶

2545 rows × 7 columns

--> cluster 5 has low total_counts and bin_counts. and no specific marker expression --> will annotated as "unassigned"

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import celltypist
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 4 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 5 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets'
sample_id = 'HEA_FOET14880396'
celltype = 'MuralCells'
```

Example 6 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 7 (python):
```python
# read in
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 10785 × 17693
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
# read in
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0
                              )
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group']!=celltype]
midgrain_markers
```

Example 10 (unknown):
```unknown
'GATA2' in list(midgrain_markers['names'])
```

Example 11 (python):
```python
# remove
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]
bdata
```

Example 12 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 10785 × 16421
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 13 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata,min_dist=0.5)
sc.pl.embedding(adata,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 14 (unknown):
```unknown
number of hvgs: 3008
```

Example 15 (python):
```python
'HEY2' in adata.var_names
```

Example 16 (python):
```python
# read in markers identified using single-nucleus data
markers_df = pd.read_csv(f'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_{celltype}_finegrain_markers.csv')

mask1 = markers_df['group_mean']>1
mask2 = markers_df['logfoldchanges']>1
markers_df = markers_df[mask1&mask2]

markers_df
```

Example 17 (python):
```python
markers = {}
n_top = 10
for c in set(markers_df['group']):
    markers_df_sub = markers_df[markers_df['group']==c]
    markers_df_sub.sort_values('logfoldchanges',ascending=False)
    markers[c]=list(markers_df_sub[:n_top]['names'])

# manually add genes
markers['Cycling'] = ['MKI67','TOP2A'] 

genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in list(adata.var_names))&(x not in genes)]
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in list(adata.var_names)]
len(genes)
```

Example 18 (python):
```python
sc.pl.embedding(adata,basis='X_umap',color=genes, size=10,
               ncols=6, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 19 (unknown):
```unknown
resolution_sel = 1
```

Example 20 (python):
```python
markers_others = {
    'CM': ['TTN', 'TNNT2','MYH6',
        'MYH7', 'MYL2', 'FHL2', # vCN
            'NPPA', 'MYL7', 'MYL4'], #aCM
    'FB': ['DCN', 'GCN', 'PDGFRA','COL1A1','COL1A2'],
    'EC': ['VWF', 'PECAM1', 'CDH5','RGCC', 'FABP5',],
    'Peri': ['RGS5', 'ABCC9', 'KCNJ8'],
    'SMC': ['MYH11', 'TAGLN', 'ACTA2'],
    'Meso': ['MSLN', 'WT1', 'BNC1'],
    'Neuro': ['PLP1', 'NRXN1', 'NRXN3','PRPH', 'NEFL', 'NEFM', 'NEFH', 'STMN2', 'SST', 'SLC10A4', 'SLC18A3', 'TH', 'NPY'],
    'Adipo': ['GPAM', 'FASN', 'LEP'],
    'Myelo': ['CD14', 'C1QA', 'CD68','LYVE1','TIMD4'],
    'Lymph': ['CD8A', 'IL7R', 'CD40LG'],
    'B': ['CD19','SDC1','IGHG1','IGHA1'],
    'Mast': ['KIT', 'CPA3'],
}

for key in markers_others.keys():
    markers_others[key]=[x for x in markers_others[key] if x in adata.var_names]

sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              markers_others, 
              groupby=f'leiden_{str(resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 21 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CM, FB, EC, etc.
```

Example 22 (python):
```python
sc.tl.dendrogram(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby=f'leiden_{str(resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 23 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CoronaryPericytes, DuctusArteriosusSmoothMuscleCells, GreatVesselSmoothMuscleCells, etc.
```

Example 24 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 25 (unknown):
```unknown
[2.35484271 1.76428034 1.58063847 1.74054927 1.3504131 ]
```

Example 26 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True, n_genes=5)
dp.add_totals().show()
```

Example 27 (unknown):
```unknown
[2.35484271 1.76428034 1.58063847 1.74054927 1.3504131 ]
```

Example 28 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=f'leiden_{str(resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 29 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=['celltypist_midmod2fine', 'conf_score_midmod2fine'],
                wspace=0.7)
```

Example 30 (python):
```python
# umap per celltype to highlight
group_col = 'celltypist_midmod2fine'
groups_to_plot = list(set(adata.obs[group_col]))

plt.rcParams['axes.grid'] = False
fig,axs=plt.subplots(nrows=2,ncols=4,figsize=(15,7))
# Flatten the axs array for easier iteration in case of multiple rows and columns
axs_flat = axs.flatten()

for i,grp in enumerate(groups_to_plot):
    sc.pl.umap(adata, color=[group_col], groups=grp, 
               size=30, frameon=False,
               show=False, ax=axs_flat[i], legend_loc=None, legend_fontsize=5,
               title=grp
              )
# Turn off any remaining subplots
for j in range(len(groups_to_plot), len(axs_flat)):
    fig.delaxes(axs_flat[j])

# fig.suptitle('Main Title for the Figure')
fig.tight_layout()
```

Example 31 (python):
```python
sc.pl.violin(adata,keys=['total_counts','bin_count'],groupby=f'leiden_{str(resolution_sel)}')
```

Example 32 (unknown):
```unknown
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 33 (unknown):
```unknown
'MuralCells'
```

Example 34 (python):
```python
# annotate
adata.obs['fine_grain_manual'] = adata.obs[f'leiden_{str(resolution_sel)}'].copy()
adata.obs.replace({'fine_grain_manual':{
    '0':'MuralCells',
    '1':'MuralCells',
    '2':'MuralCells', 
    '3':'MuralCells',
    '4':'MuralCells',
    '5':'unassigned',
    '6':'MuralCells',
    '7':'MuralCells',
    '8':'MuralCells',
}},inplace=True)
adata.obs['fine_grain_manual'].value_counts()
```

Example 35 (unknown):
```unknown
fine_grain_manual
MuralCells    9969
unassigned     816
Name: count, dtype: int64
```

Example 36 (python):
```python
ad = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad',backed='r')
ad = ad[ad.obs['mid_grain']==celltype]
set(ad.obs['fine_grain'])
```

Example 37 (unknown):
```unknown
{'CoronaryPericytes',
 'CoronarySmoothMuscleCells',
 'DuctusArteriosusSmoothMuscleCells',
 'GreatVesselSmoothMuscleCells'}
```

Example 38 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 39 (python):
```python
adata_sub = adata[adata.obs['fine_grain_manual'].isin(['MuralCells'])]

# drop leiden clustering results
resolutions_list = [0.2,0.4,0.6,1, 1.2,1.5,2]
for resolution in resolutions_list:
    adata_sub.obs.drop(f'leiden_{str(resolution)}',axis=1,inplace=True)
adata_sub
```

Example 40 (unknown):
```unknown
AnnData object with n_obs × n_vars = 9969 × 17693
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap', 'dendrogram_leiden_1', 'rank_genes_groups', 'fine_grain_manual_colors'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 41 (python):
```python
# remove mid_grain markers
mask = [x not in list(midgrain_markers['names']) for x in adata_sub.var_names]
bdata = adata_sub[:,mask]
bdata
```

Example 42 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 9969 × 16421
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap', 'dendrogram_leiden_1', 'rank_genes_groups', 'fine_grain_manual_colors'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 43 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata_sub.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata_sub,n_neighbors=15,use_rep='X_pca')
```

Example 44 (unknown):
```unknown
number of hvgs: 2864
```

Example 45 (python):
```python
sc.tl.umap(adata_sub,min_dist=0.1)
```

Example 46 (python):
```python
%%time
# reclustering
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1, 1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata_sub,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata_sub,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 47 (unknown):
```unknown
CPU times: user 4.44 s, sys: 63.9 ms, total: 4.5 s
Wall time: 4.78 s
```

Example 48 (python):
```python
sc.pl.embedding(adata_sub,basis='X_umap',color=genes, size=10,
               ncols=6, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 49 (python):
```python
# umap per celltype to highlight
group_col = 'celltypist_midmod2fine'
groups_to_plot = list(set(adata_sub.obs[group_col]))

plt.rcParams['axes.grid'] = False
fig,axs=plt.subplots(nrows=2,ncols=4,figsize=(15,7))
# Flatten the axs array for easier iteration in case of multiple rows and columns
axs_flat = axs.flatten()

for i,grp in enumerate(groups_to_plot):
    sc.pl.umap(adata_sub, color=[group_col], groups=grp, 
               size=30, frameon=False,
               show=False, ax=axs_flat[i], legend_loc=None, legend_fontsize=5,
               title=grp
              )
# Turn off any remaining subplots
for j in range(len(groups_to_plot), len(axs_flat)):
    fig.delaxes(axs_flat[j])

# fig.suptitle('Main Title for the Figure')
fig.tight_layout()
```

Example 50 (python):
```python
adata_sub_resolution_sel = 1
```

Example 51 (python):
```python
markers_others = {
    'CM': ['TTN', 'TNNT2','MYH6',
        'MYH7', 'MYL2', 'FHL2', # vCN
            'NPPA', 'MYL7', 'MYL4'], #aCM
    'FB': ['DCN', 'GCN', 'PDGFRA','COL1A1','COL1A2'],
    'EC': ['VWF', 'PECAM1', 'CDH5','RGCC', 'FABP5',],
    'Peri': ['RGS5', 'ABCC9', 'KCNJ8'],
    'SMC': ['MYH11', 'TAGLN', 'ACTA2'],
    'Meso': ['MSLN', 'WT1', 'BNC1'],
    'Neuro': ['PLP1', 'NRXN1', 'NRXN3','PRPH', 'NEFL', 'NEFM', 'NEFH', 'STMN2', 'SST', 'SLC10A4', 'SLC18A3', 'TH', 'NPY'],
    'Adipo': ['GPAM', 'FASN', 'LEP'],
    'Myelo': ['CD14', 'C1QA', 'CD68','LYVE1','TIMD4'],
    'Lymph': ['CD8A', 'IL7R', 'CD40LG'],
    'B': ['CD19','SDC1','IGHG1','IGHA1'],
    'Mast': ['KIT', 'CPA3'],
}

for key in markers_others.keys():
    markers_others[key]=[x for x in markers_others[key] if x in adata_sub.var_names]

sc.tl.dendrogram(adata_sub, groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.pl.dotplot(adata_sub,
              markers_others, 
              groupby=f'leiden_{str(adata_sub_resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 52 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CM, FB, EC, etc.
```

Example 53 (python):
```python
sc.tl.dendrogram(adata_sub,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.pl.dotplot(adata_sub,
              var_names=markers,
              groupby=f'leiden_{str(adata_sub_resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 54 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CoronaryPericytes, DuctusArteriosusSmoothMuscleCells, GreatVesselSmoothMuscleCells, etc.
```

Example 55 (python):
```python
# DEG analysis
print(adata_sub.X.data[:5])
adata_sub.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata_sub,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.tl.dendrogram(adata_sub, groupby=f'leiden_{str(adata_sub_resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata_sub, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 56 (unknown):
```unknown
[2.35484271 1.76428034 1.58063847 1.74054927 1.3504131 ]
```

Example 57 (python):
```python
# DEG analysis
print(adata_sub.X.data[:5])
adata_sub.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata_sub,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.tl.dendrogram(adata_sub, groupby=f'leiden_{str(adata_sub_resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata_sub, dendrogram=True, standard_scale='var', return_fig=True, n_genes=5)
dp.add_totals().show()
```

Example 58 (unknown):
```unknown
[2.35484271 1.76428034 1.58063847 1.74054927 1.3504131 ]
```

Example 59 (python):
```python
sc.pl.embedding(adata_sub,basis='X_umap',
                color=f'leiden_{str(adata_sub_resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 60 (unknown):
```unknown
{'CoronaryPericytes',
 'CoronarySmoothMuscleCells',
 'DuctusArteriosusSmoothMuscleCells',
 'GreatVesselSmoothMuscleCells'}
```

Example 61 (unknown):
```unknown
'MuralCells'
```

Example 62 (python):
```python
# annotate
adata_sub.obs['fine_grain_manual'] = adata_sub.obs[f'leiden_{str(adata_sub_resolution_sel)}'].copy()
adata_sub.obs.replace({'fine_grain_manual':{
    '0':'GreatVesselSmoothMuscleCells', 
    '1':'DuctusArteriosusSmoothMuscleCells',
    '2':'GreatVesselSmoothMuscleCells',
    '3':'GreatVesselSmoothMuscleCellsTFPI2pos',
    '4':'GreatVesselSmoothMuscleCells',
    '5':'CoronaryPericytes',
    '6':'CoronarySmoothMuscleCells',
    '7':'DuctusArteriosusSmoothMuscleCells',
    '8':'GreatVesselSmoothMuscleCellsPTGER4pos',
    '9':'GreatVesselSmoothMuscleCells',
}},inplace=True)
adata_sub.obs['fine_grain_manual'].value_counts()
```

Example 63 (unknown):
```unknown
fine_grain_manual
GreatVesselSmoothMuscleCells             4843
DuctusArteriosusSmoothMuscleCells        2145
GreatVesselSmoothMuscleCellsTFPI2pos     1519
CoronaryPericytes                         753
CoronarySmoothMuscleCells                 424
GreatVesselSmoothMuscleCellsPTGER4pos     285
Name: count, dtype: int64
```

Example 64 (python):
```python
sc.pl.embedding(adata_sub,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 65 (python):
```python
markers['Others']=['PTGER4','TFPI2']

sc.tl.dendrogram(adata_sub,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.pl.dotplot(adata_sub,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 66 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: GreatVesselSmoothMuscleCells, DuctusArteriosusSmoothMuscleCells, GreatVesselSmoothMuscleCellsTFPI2pos, etc.
var_group_labels: CoronaryPericytes, DuctusArteriosusSmoothMuscleCells, GreatVesselSmoothMuscleCells, etc.
```

Example 67 (python):
```python
# add the annotation to the original adata
adata.obs['fine_grain_manual'] = adata.obs['fine_grain_manual'].astype('str')
adata.obs.loc[adata_sub.obs_names,'fine_grain_manual'] = adata_sub.obs['fine_grain_manual'].astype('str').copy()
adata.obs['fine_grain_manual'].value_counts()
```

Example 68 (unknown):
```unknown
fine_grain_manual
GreatVesselSmoothMuscleCells             4843
DuctusArteriosusSmoothMuscleCells        2145
GreatVesselSmoothMuscleCellsTFPI2pos     1519
unassigned                                816
CoronaryPericytes                         753
CoronarySmoothMuscleCells                 424
GreatVesselSmoothMuscleCellsPTGER4pos     285
Name: count, dtype: int64
```

Example 69 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 70 (unknown):
```unknown
... storing 'fine_grain_manual' as categorical
```

Example 71 (python):
```python
sc.tl.dendrogram(adata,groupby='fine_grain_manual')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 72 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: CoronaryPericytes, CoronarySmoothMuscleCells, DuctusArteriosusSmoothMuscleCells, etc.
var_group_labels: CoronaryPericytes, DuctusArteriosusSmoothMuscleCells, GreatVesselSmoothMuscleCells, etc.
```

Example 73 (python):
```python
# save
adata.write(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
print(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
```

Example 74 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets/HEA_FOET14880396_MuralCells_b2c_cells_filtered_lognorm.h5ad
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no7_manual-annotation_BloodVesselEC.html

**Contents:**
- Variables¶
- Read in data¶
- Remove mid-grain markers from the feature¶
- PCA, Clustering, UMAP¶
- Markers¶
- Save¶

2512 rows × 7 columns

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import celltypist
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 4 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 5 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets'
sample_id = 'HEA_FOET14880396'
celltype = 'BloodVesselEndothelialCells'
```

Example 6 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 7 (python):
```python
# read in
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 1765 × 15377
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
# read in
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0
                              )
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group']!=celltype]
midgrain_markers
```

Example 10 (unknown):
```unknown
'GATA2' in list(midgrain_markers['names'])
```

Example 11 (python):
```python
# remove
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]
bdata
```

Example 12 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 1765 × 14107
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 13 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata,min_dist=0.5)
sc.pl.embedding(adata,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 14 (unknown):
```unknown
number of hvgs: 3441
```

Example 15 (python):
```python
'HEY2' in adata.var_names
```

Example 16 (python):
```python
# read in markers identified using single-nucleus data
markers_df = pd.read_csv(f'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_{celltype}_finegrain_markers.csv')

mask1 = markers_df['group_mean']>1
mask2 = markers_df['logfoldchanges']>1
markers_df = markers_df[mask1&mask2]

markers_df
```

Example 17 (python):
```python
markers = {}
n_top = 10
for c in set(markers_df['group']):
    markers_df_sub = markers_df[markers_df['group']==c]
    markers_df_sub.sort_values('logfoldchanges',ascending=False)
    markers[c]=list(markers_df_sub[:n_top]['names'])

# manually add genes
markers['Cycling'] = ['MKI67','TOP2A'] 

genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in list(adata.var_names))&(x not in genes)]
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in list(adata.var_names)]
len(genes)
```

Example 18 (python):
```python
sc.pl.embedding(adata,basis='X_umap',color=genes, size=10,
               ncols=6, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 19 (unknown):
```unknown
resolution_sel = 1
```

Example 20 (python):
```python
markers_others = {
    'CM': ['TTN', 'TNNT2','MYH6',
        'MYH7', 'MYL2', 'FHL2', # vCN
            'NPPA', 'MYL7', 'MYL4'], #aCM
    'FB': ['DCN', 'GCN', 'PDGFRA','COL1A1','COL1A2'],
    'EC': ['VWF', 'PECAM1', 'CDH5','RGCC', 'FABP5',],
    'Peri': ['RGS5', 'ABCC9', 'KCNJ8'],
    'SMC': ['MYH11', 'TAGLN', 'ACTA2'],
    'Meso': ['MSLN', 'WT1', 'BNC1'],
    'Neuro': ['PLP1', 'NRXN1', 'NRXN3','PRPH', 'NEFL', 'NEFM', 'NEFH', 'STMN2', 'SST', 'SLC10A4', 'SLC18A3', 'TH', 'NPY'],
    'Adipo': ['GPAM', 'FASN', 'LEP'],
    'Myelo': ['CD14', 'C1QA', 'CD68','LYVE1','TIMD4'],
    'Lymph': ['CD8A', 'IL7R', 'CD40LG'],
    'B': ['CD19','SDC1','IGHG1','IGHA1'],
    'Mast': ['KIT', 'CPA3'],
}

for key in markers_others.keys():
    markers_others[key]=[x for x in markers_others[key] if x in adata.var_names]

sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              markers_others, 
              groupby=f'leiden_{str(resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 21 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CM, FB, EC, etc.
```

Example 22 (python):
```python
sc.tl.dendrogram(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby=f'leiden_{str(resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 23 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CoronaryVenousEndothelialCells, GreatVesselArterialEndothelialCells, CoronaryArterialEndothelialCells, etc.
```

Example 24 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 25 (unknown):
```unknown
[1.27693184 1.26688208 1.1734066  2.12904094 1.26688208]
```

Example 26 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True, n_genes=5)
dp.add_totals().show()
```

Example 27 (unknown):
```unknown
[1.27693184 1.26688208 1.1734066  2.12904094 1.26688208]
```

Example 28 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=f'leiden_{str(resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 29 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=['celltypist_midmod2fine', 'conf_score_midmod2fine'],
                wspace=0.7)
```

Example 30 (python):
```python
# umap per celltype to highlight
group_col = 'celltypist_midmod2fine'
groups_to_plot = list(set(adata.obs[group_col]))

plt.rcParams['axes.grid'] = False
fig,axs=plt.subplots(nrows=2,ncols=4,figsize=(15,7))
# Flatten the axs array for easier iteration in case of multiple rows and columns
axs_flat = axs.flatten()

for i,grp in enumerate(groups_to_plot):
    sc.pl.umap(adata, color=[group_col], groups=grp, 
               size=30, frameon=False,
               show=False, ax=axs_flat[i], legend_loc=None, legend_fontsize=5,
               title=grp
              )
# Turn off any remaining subplots
for j in range(len(groups_to_plot), len(axs_flat)):
    fig.delaxes(axs_flat[j])

# fig.suptitle('Main Title for the Figure')
fig.tight_layout()
```

Example 31 (python):
```python
sc.pl.violin(adata,keys=['total_counts','bin_count'],groupby=f'leiden_{str(resolution_sel)}')
```

Example 32 (unknown):
```unknown
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 33 (python):
```python
# annotate
adata.obs['fine_grain_manual'] = adata.obs[f'leiden_{str(resolution_sel)}'].copy()
adata.obs.replace({'fine_grain_manual':{
    '0':'GreatVesselVenousEndothelialCells',
    '1':'CoronaryCapillaryEndothelialCells',
    '2':'CoronaryCapillaryEndothelialCells', # contamination from CMs
    '3':'CoronaryCapillaryEndothelialCells',
    '4':'GreatVesselArterialEndothelialCells',
    '5':'CoronaryArterialEndothelialCells',
    '6':'GreatVesselArterialEndothelialCells',
    '7':'CoronaryVenousEndothelialCells',
    '8':'CoronaryArterialEndothelialCells',
    '9':'CoronaryCapillaryEndothelialCells', # contamination from erythrocytes
}},inplace=True)
adata.obs['fine_grain_manual'].value_counts()
```

Example 34 (unknown):
```unknown
fine_grain_manual
CoronaryCapillaryEndothelialCells      848
GreatVesselVenousEndothelialCells      313
GreatVesselArterialEndothelialCells    308
CoronaryArterialEndothelialCells       219
CoronaryVenousEndothelialCells          77
Name: count, dtype: int64
```

Example 35 (python):
```python
ad = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad',backed='r')
ad = ad[ad.obs['mid_grain']==celltype]
set(ad.obs['fine_grain'])
```

Example 36 (unknown):
```unknown
{'CoronaryArterialEndothelialCells',
 'CoronaryCapillaryEndothelialCells',
 'CoronaryVenousEndothelialCells',
 'GreatVesselArterialEndothelialCells',
 'GreatVesselVenousEndothelialCells'}
```

Example 37 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 38 (python):
```python
sc.tl.dendrogram(adata,groupby='fine_grain_manual')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 39 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: GreatVesselVenousEndothelialCells, CoronaryCapillaryEndothelialCells, GreatVesselArterialEndothelialCells, etc.
var_group_labels: CoronaryVenousEndothelialCells, GreatVesselArterialEndothelialCells, CoronaryArterialEndothelialCells, etc.
```

Example 40 (python):
```python
# save
adata.write(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
print(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
```

Example 41 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets/HEA_FOET14880396_BloodVesselEndothelialCells_b2c_cells_filtered_lognorm.h5ad
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/Visium/Others/Plotting_separately_SAnode.html

**Contents:**
- Import required modules¶
- Prepare functions¶
- Read in anndata¶
- Plotting¶
- Cells¶
- Genes¶

**Examples:**

Example 1 (python):
```python
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

import scanpy as sc
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 
import seaborn as sns
import os 
from mpl_toolkits.axes_grid1 import make_axes_locatable
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
-----
matplotlib          3.7.2
mpl_toolkits        NA
numpy               1.24.4
pandas              1.5.3
scanpy              1.9.3
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                         10.0.0
anndata                     0.9.2
anyio                       NA
arrow                       1.2.3
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.12.1
backcall                    0.2.0
brotli                      1.0.9
certifi                     2023.11.17
cffi                        1.15.1
charset_normalizer          3.2.0
colorama                    0.4.6
comm                        0.1.4
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.6.8
decorator                   5.1.1
defusedxml                  0.7.1
executing                   1.2.0
fastjsonschema              NA
fqdn                        NA
h5py                        3.9.0
idna                        3.4
igraph                      0.10.6
importlib_metadata          NA
importlib_resources         NA
ipykernel                   6.25.1
isoduration                 NA
jedi                        0.19.0
jinja2                      3.1.2
joblib                      1.3.2
json5                       NA
jsonpointer                 2.0
jsonschema                  4.19.0
jsonschema_specifications   NA
jupyter_events              0.7.0
jupyter_server              2.7.1
jupyterlab_server           2.24.0
kiwisolver                  1.4.5
leidenalg                   0.10.1
llvmlite                    0.40.1
markupsafe                  2.1.3
natsort                     8.4.0
nbformat                    5.9.2
numba                       0.57.1
overrides                   NA
packaging                   23.1
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
plotly                      5.16.1
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.5
ptyprocess                  0.7.0
pure_eval                   0.2.2
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.16.1
pyparsing                   3.0.9
pythonjsonlogger            NA
pytz                        2023.3
referencing                 NA
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rpds                        NA
ruamel                      NA
scipy                       1.10.1
send2trash                  NA
six                         1.16.0
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
sphinxcontrib               NA
stack_data                  0.6.2
statsmodels                 0.14.0
texttable                   1.6.7
threadpoolctl               3.2.0
tornado                     6.3.3
traitlets                   5.9.0
typing_extensions           NA
uri_template                NA
urllib3                     2.0.4
wcwidth                     0.2.6
webcolors                   1.13
websocket                   1.6.2
yaml                        6.0.1
zipp                        NA
zmq                         25.1.1
zstandard                   0.19.0
```

Example 5 (unknown):
```unknown
-----
IPython             8.12.0
jupyter_client      8.3.0
jupyter_core        4.12.0
jupyterlab          4.0.5
notebook            7.0.2
-----
Python 3.8.17 (default, Jul  5 2023, 21:04:15) [GCC 11.2.0]
Linux-5.4.0-169-generic-x86_64-with-glibc2.17
-----
Session information updated at 2024-02-10 23:20
```

Example 6 (python):
```python
# import a module given the full path?
import importlib.util
import sys

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/_utils.py")
_utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = _utils
spec.loader.exec_module(_utils)
```

Example 7 (python):
```python
sc.settings.verbosity = 1          # verbosity: errors (0), warnings (1), info (2), hints (3)
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 8 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk'
```

Example 9 (python):
```python
adata_vis = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT_foetal_raw.h5ad')

# add abundance to .obs
df = adata_vis.obsm['q05_cell_abundance_w_sf'].copy()
## rename columns
df.columns = [x.replace('q05cell_abundance_w_sf_','') for x in df.columns]
adata_vis.obs[df.columns]=df.copy()

# add prop
## convert to proportions # should use np.nansam rather than sum
df = df.apply(lambda x: x/np.nansum(x), axis=1) 
adata_vis.obsm['prop']=df.copy()
adata_vis.obs['prop_'+df.columns] = df.copy()

# set symboßl as var_names
adata_vis.var.set_index('SYMBOL',inplace=True)

# log-normalise
sc.pp.filter_genes(adata_vis, min_cells=3)
sc.pp.normalize_total(adata_vis, target_sum=1e4)
sc.pp.log1p(adata_vis)

adata_vis
```

Example 10 (python):
```python
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
/tmp/ipykernel_131063/2172206593.py:13: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  adata_vis.obs['prop_'+df.columns] = df.copy()
```

Example 11 (unknown):
```unknown
AnnData object with n_obs × n_vars = 14261 × 22248
    obs: 'in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'cell2loc-map_group', 'JC_edit', 'sample_for_cell2loc', 'AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight', 'VentricularCardiomyocytesLeftCompact', 'VentricularCardiomyocytesRightCompact', 'VentricularCardiomyocytesLeftTrabeculated', 'VentricularCardiomyocytesRightTrabeculated', 'SinoatrialNodeCardiomyocytes', 'AtrioventricularNodeCardiomyocytes', 'VentricularConductionSystemProximal', 'VentricularConductionSystemDistal', 'GreatVesselAdventitialFibroblasts', 'CoronaryVesselAdventitialFibroblasts', 'MyocardialInterstitialFibroblasts1', 'MyocardialInterstitialFibroblasts2', 'Myofibroblasts', 'LymphNodeFibroblasticReticularCells', 'ValveInterstitialCells', 'GreatVesselSmoothMuscleCells', 'CoronarySmoothMuscleCells', 'CoronaryPericytes', 'GreatVesselArterialEndothelialCells', 'GreatVesselVenousEndothelialCells', 'CoronaryArterialEndothelialCells', 'CoronaryVenousEndothelialCells', 'CoronaryCapillaryEndothelialCells', 'EndocardialCells', 'EndocardialCushionCells', 'ValveEndothelialCells', 'LymphaticEndothelialCells', 'MesothelialEpicardialCells', 'EpicardiumDerivedCells', 'NeuronPrecursors', 'ChromaffinCells', 'SympatheticNeurons', 'ParaympatheticNeurons', 'SchwannCellPrecursors', 'SchwannCells', 'MonocytesMPOpos', 'Monocytes', 'MonocyteDerivedCells', 'MacrophagesCX3CR1pos', 'MacrophagesTIMD4pos', 'MacrophagesLYVE1pos', 'MacrophagesATF3pos', 'DendriticCellsType1', 'DendriticCellsMature', 'PlasmacytoidDendriticCells', 'MastCells', 'Megakaryocytes', 'TCellsCD4pos', 'TCellsCD8pos', 'TregsCD4pos', 'ProBCells', 'BCells', 'BCellsMS4A1pos', 'NaturalKillerCells', 'InnateLymphoidCells', 'prop_AtrialCardiomyocytesLeft', 'prop_AtrialCardiomyocytesRight', 'prop_VentricularCardiomyocytesLeftCompact', 'prop_VentricularCardiomyocytesRightCompact', 'prop_VentricularCardiomyocytesLeftTrabeculated', 'prop_VentricularCardiomyocytesRightTrabeculated', 'prop_SinoatrialNodeCardiomyocytes', 'prop_AtrioventricularNodeCardiomyocytes', 'prop_VentricularConductionSystemProximal', 'prop_VentricularConductionSystemDistal', 'prop_GreatVesselAdventitialFibroblasts', 'prop_CoronaryVesselAdventitialFibroblasts', 'prop_MyocardialInterstitialFibroblasts1', 'prop_MyocardialInterstitialFibroblasts2', 'prop_Myofibroblasts', 'prop_LymphNodeFibroblasticReticularCells', 'prop_ValveInterstitialCells', 'prop_GreatVesselSmoothMuscleCells', 'prop_CoronarySmoothMuscleCells', 'prop_CoronaryPericytes', 'prop_GreatVesselArterialEndothelialCells', 'prop_GreatVesselVenousEndothelialCells', 'prop_CoronaryArterialEndothelialCells', 'prop_CoronaryVenousEndothelialCells', 'prop_CoronaryCapillaryEndothelialCells', 'prop_EndocardialCells', 'prop_EndocardialCushionCells', 'prop_ValveEndothelialCells', 'prop_LymphaticEndothelialCells', 'prop_MesothelialEpicardialCells', 'prop_EpicardiumDerivedCells', 'prop_NeuronPrecursors', 'prop_ChromaffinCells', 'prop_SympatheticNeurons', 'prop_ParaympatheticNeurons', 'prop_SchwannCellPrecursors', 'prop_SchwannCells', 'prop_MonocytesMPOpos', 'prop_Monocytes', 'prop_MonocyteDerivedCells', 'prop_MacrophagesCX3CR1pos', 'prop_MacrophagesTIMD4pos', 'prop_MacrophagesLYVE1pos', 'prop_MacrophagesATF3pos', 'prop_DendriticCellsType1', 'prop_DendriticCellsMature', 'prop_PlasmacytoidDendriticCells', 'prop_MastCells', 'prop_Megakaryocytes', 'prop_TCellsCD4pos', 'prop_TCellsCD8pos', 'prop_TregsCD4pos', 'prop_ProBCells', 'prop_BCells', 'prop_BCellsMS4A1pos', 'prop_NaturalKillerCells', 'prop_InnateLymphoidCells'
    var: 'gene_ids', 'feature_types', 'genome', 'mt', 'n_cells'
    uns: 'spatial', 'log1p'
    obsm: 'MT', 'means_cell_abundance_w_sf', 'q05_cell_abundance_w_sf', 'q95_cell_abundance_w_sf', 'spatial', 'stds_cell_abundance_w_sf', 'prop'
```

Example 12 (python):
```python
# add "annotation_final"
annot = pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/visium_foetal_healthy-all_spot-annotation.csv',
                    index_col=0
                   )
adata_vis.obs['annotation_final'] = annot[['final']].reindex(adata_vis.obs_names)
adata_vis.obs.head()
```

Example 13 (python):
```python
slide_id = 'HCAHeartST10317184'
slide = _utils.select_multislide(adata_vis, [slide_id], s_col='sample')

with mpl.rc_context({'figure.figsize': [3, 3],
                            'axes.titlesize':16,}):
    # cells
    print(slide_id)
    sc.pl.spatial(slide,cmap='YlGnBu',
                  color=['SinoatrialNodeCardiomyocytes','ParaympatheticNeurons','MacrophagesCX3CR1pos','SympatheticNeurons'], 
                  ncols=5, wspace=0.2,hspace=0.3,
                  size=1.2,img_key='hires',alpha_img=0.3,vmin=0, vmax='p99.4',
                  title=slide_id
                  # palette = palette
                  # save=f'_SAN-{ID}_core_cellstates.pdf'
                 )
```

Example 14 (unknown):
```unknown
/nfs/team205/kk18/function/python/_utils.py:95: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  slides.uns['spatial'] = uns_dict
```

Example 15 (unknown):
```unknown
HCAHeartST10317184
WARNING: The title list is shorter than the number of panels. Using 'color' value instead for some plots.
WARNING: The title list is shorter than the number of panels. Using 'color' value instead for some plots.
WARNING: The title list is shorter than the number of panels. Using 'color' value instead for some plots.
```

Example 16 (python):
```python
slide_id = 'HCAHeartST10317184'
slide = _utils.select_multislide(adata_vis, [slide_id], s_col='sample')

with mpl.rc_context({'figure.figsize': [3, 3],
                            'axes.titlesize':12,}):
    # cells
    print(slide_id)
    sc.pl.spatial(slide,cmap='YlGnBu',
                  color=['SinoatrialNodeCardiomyocytes','ParaympatheticNeurons','MacrophagesCX3CR1pos'], 
                  ncols=5, wspace=0.1,hspace=0.3,
                  size=1.2,img_key='hires',alpha_img=0.3,vmin=0, vmax='p99.4',
                  crop_coord = [5900,7500,10700,12800],
                  # title=slide_id
                  # palette = palette
                  # save=f'_SAN-{ID}_core_cellstates.pdf'
                 )
```

Example 17 (unknown):
```unknown
/nfs/team205/kk18/function/python/_utils.py:95: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  slides.uns['spatial'] = uns_dict
```

Example 18 (unknown):
```unknown
HCAHeartST10317184
```

Example 19 (python):
```python
slide_id = 'HCAHeartST10317184'
slide = _utils.select_multislide(adata_vis, [slide_id], s_col='sample')

with mpl.rc_context({'figure.figsize': [3, 3],
                            'axes.titlesize':16,}):
    # cells
    print(slide_id)
    sc.pl.spatial(slide,cmap='RdPu',
                  color=['SEMA3A', 'SLIT2', 'SLIT3', 'EFNB2', 'CXCL12'], 
                  ncols=5, wspace=0.2,hspace=0.3,
                  size=1.2,img_key='hires',alpha_img=0.3,vmin=0, vmax='p99.4',
                  # title=slide_id
                  # palette = palette
                  # save=f'_SAN-{ID}_core_cellstates.pdf'
                 )
```

Example 20 (unknown):
```unknown
/nfs/team205/kk18/function/python/_utils.py:95: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  slides.uns['spatial'] = uns_dict
```

Example 21 (unknown):
```unknown
HCAHeartST10317184
```

Example 22 (python):
```python
slide_id = 'HCAHeartST10317184'
slide = _utils.select_multislide(adata_vis, [slide_id], s_col='sample')

with mpl.rc_context({'figure.figsize': [3, 3],
                            'axes.titlesize':24,}):
    # cells
    print(slide_id)
    sc.pl.spatial(slide,cmap='RdPu',
                  color=['SEMA3A', 'SLIT2', 'SLIT3', 'EFNB2', 'CXCL12'], 
                  ncols=5, wspace=0.1,hspace=0.3,
                  size=1.2,img_key='hires',alpha_img=0.3,vmin=0, vmax='p99.4',
                  crop_coord = [5900,7500,10700,12800],
                  # title=slide_id
                  # palette = palette
                  save=f'_visium_section_axon-guidance-genes.pdf'
                 )
    # plt.savefig(f"{figdir}/visium_section_axon-guidance-genes.pdf", bbox_inches="tight",dpi=300)
```

Example 23 (unknown):
```unknown
/nfs/team205/kk18/function/python/_utils.py:95: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  slides.uns['spatial'] = uns_dict
```

Example 24 (unknown):
```unknown
HCAHeartST10317184
WARNING: saving figure to file figures/show_visium_section_axon-guidance-genes.pdf
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no7_manual-annotation_myeloid.html

**Contents:**
- Variables¶
- Read in data¶
- Remove mid-grain markers from the feature¶
- PCA, Clustering, UMAP¶
- Subset and reclustering¶
- Transfer annotation to the original adata¶
- Save¶

--> cluster 3 and 8 are MacrophagesCX3CR1pos (cluster 4 is likely to be monocytes)

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import celltypist
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
-----
celltypist          1.6.3
matplotlib          3.9.2
numpy               2.0.2
pandas              2.2.3
scanpy              1.10.3
seaborn             0.13.2
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                 11.0.0
anndata             0.10.9
asttokens           NA
certifi             2024.08.30
charset_normalizer  3.4.0
comm                0.2.2
cycler              0.12.1
cython_runtime      NA
dateutil            2.9.0.post0
debugpy             1.8.9
decorator           5.1.1
exceptiongroup      1.2.2
executing           2.1.0
google              NA
h5py                3.12.1
idna                3.10
igraph              0.10.2
importlib_metadata  NA
importlib_resources NA
ipykernel           6.29.5
jedi                0.19.2
joblib              1.4.2
kiwisolver          1.4.7
legacy_api_wrap     NA
leidenalg           0.9.0
llvmlite            0.43.0
mpl_toolkits        NA
natsort             8.4.0
numba               0.60.0
packaging           24.2
parso               0.8.4
patsy               1.0.1
pexpect             4.9.0
platformdirs        4.3.6
prompt_toolkit      3.0.48
psutil              6.1.0
ptyprocess          0.7.0
pure_eval           0.2.3
pydev_ipython       NA
pydevconsole        NA
pydevd              3.2.3
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.18.0
pyparsing           3.2.0
pytz                2024.2
requests            2.32.3
scipy               1.13.1
six                 1.16.0
sklearn             1.5.2
stack_data          0.6.3
statsmodels         0.14.4
texttable           1.6.7
threadpoolctl       3.5.0
tornado             6.4.2
traitlets           5.14.3
typing_extensions   NA
urllib3             2.2.3
wcwidth             0.2.13
yaml                6.0.2
zipp                NA
zmq                 26.2.0
zoneinfo            NA
```

Example 5 (unknown):
```unknown
-----
IPython             8.18.1
jupyter_client      8.6.3
jupyter_core        5.7.2
-----
Python 3.9.20 | packaged by conda-forge | (main, Sep 30 2024, 17:49:10) [GCC 13.3.0]
Linux-5.15.0-126-generic-x86_64-with-glibc2.35
-----
Session information updated at 2024-12-08 11:26
```

Example 6 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 7 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 8 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets'
sample_id = 'HEA_FOET14880396'
celltype = 'MyeloidCells'
```

Example 9 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 10 (python):
```python
# read in
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 11 (unknown):
```unknown
AnnData object with n_obs × n_vars = 526 × 13502
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2', 'fine_grain_manual', 'leiden_0.5'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'dendrogram_fine_grain_manual', 'dendrogram_leiden_0.4', 'dendrogram_leiden_0.6', 'dendrogram_leiden_1', 'dendrogram_leiden_1.2', 'dendrogram_leiden_1.5', 'fine_grain_manual_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.5', 'leiden_0.5_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'rank_genes_groups', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 12 (python):
```python
# read in
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0
                              )
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group']!=celltype]
midgrain_markers.head()
```

Example 13 (unknown):
```unknown
'GATA2' in list(midgrain_markers['names'])
```

Example 14 (python):
```python
# remove
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]
bdata
```

Example 15 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 526 × 12303
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2', 'fine_grain_manual', 'leiden_0.5'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'dendrogram_fine_grain_manual', 'dendrogram_leiden_0.4', 'dendrogram_leiden_0.6', 'dendrogram_leiden_1', 'dendrogram_leiden_1.2', 'dendrogram_leiden_1.5', 'fine_grain_manual_colors', 'hvg', 'leiden_0.2', 'leiden_0.4', 'leiden_0.5', 'leiden_0.5_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'rank_genes_groups', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 16 (python):
```python
# remove cell cycle genes
cell_cycle_genes = ['MCM5','PCNA','TYMS','FEN1','MCM2','MCM4','RRM1','UNG','GINS2','MCM6','CDCA7','DTL','PRIM1','UHRF1','MLF1IP','HELLS','RFC2','RPA2','NASP',
                'RAD51AP1','GMNN','WDR76','SLBP','CCNE2','UBR7','POLD3','MSH2','ATAD2','RAD51','RRM2','CDC45','CDC6','EXO1','TIPIN','DSCC1','BLM','CASP8AP2',
                'USP1','CLSPN','POLA1','CHAF1B','BRIP1','E2F8','HMGB2','CDK1','NUSAP1','UBE2C','BIRC5','TPX2','TOP2A','NDC80','CKS2','NUF2','CKS1B','MKI67',
                'TMPO','CENPF','TACC3','FAM64A','SMC4','CCNB2','CKAP2L','CKAP2','AURKB','BUB1','KIF11','ANP32E','TUBB4B','GTSE1','KIF20B','HJURP','CDCA3','HN1',
                'CDC20','TTK','CDC25C','KIF2C','RANGAP1','NCAPD2','DLGAP5','CDCA2','CDCA8','ECT2','KIF23','HMMR','AURKA','PSRC1','ANLN','LBR','CKAP5','CENPE','CTCF',
                'NEK2','G2E3','GAS2L3','CBX5','CENPA']
mask = [x not in cell_cycle_genes for x in bdata.var_names]
bdata = bdata[:,mask]
bdata.shape
```

Example 17 (unknown):
```unknown
(526, 12209)
```

Example 18 (unknown):
```unknown
'MYH11' in list(bdata.var_names)
```

Example 19 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata,min_dist=0.5)
sc.pl.embedding(adata,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
# leiden clustering
resolutions_list = [0.2,0.4,0.5,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 20 (unknown):
```unknown
number of hvgs: 2881
```

Example 21 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
```

Example 22 (python):
```python
markers={
    'Mono':['CCR2','FCN1','VCAN','MPO'],
    'MP':['C1QA','C1QB','CD68','MRC1','CD163','MERTK'],
    'CX3CR1+':['CX3CR1','C3','TREM2','P2RY12','CD200R1'],
    'LYVE1+':['LYVE1','FOLR2'],
     'pDC':['IRF8','IRF4','IL3RA'],
     'DC_mature':['FLT3', 'CCR7','LAMP3', 'IL15', 'CD83', 'LY75', ],
    'Megakaryocyte':['ITGA2B','MPL','ITGB3',],
    'Mast': ['KIT', 'CPA3','HDC','GATA2'],
    'Cycling':['MKI67','TOP2A'],
}
genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in adata.var_names)&(x not in genes)]
sc.pl.embedding(adata,basis='X_umap',color=genes,
               ncols=5, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 23 (unknown):
```unknown
resolution_sel = 0.6
```

Example 24 (python):
```python
markers = {
    'CM': ['TTN', 'TNNT2','MYH6',
        'MYH7', 'MYL2', 'FHL2', # vCN
            'NPPA', 'MYL7', 'MYL4'], #aCM
    'FB': ['DCN', 'GCN', 'PDGFRA','COL1A1','COL1A2'],
    'EC': ['VWF', 'PECAM1', 'CDH5','RGCC', 'FABP5',],
    'Peri': ['RGS5', 'ABCC9', 'KCNJ8'],
    'SMC': ['MYH11', 'TAGLN', 'ACTA2'],
    'Meso': ['MSLN', 'WT1', 'BNC1'],
    'Neuro': ['PLP1', 'NRXN1', 'NRXN3','PRPH', 'NEFL', 'NEFM', 'NEFH', 'STMN2', 'SST', 'SLC10A4', 'SLC18A3', 'TH', 'NPY'],
    'Adipo': ['GPAM', 'FASN', 'LEP'],
    'Myelo': ['CD14', 'C1QA', 'CD68','LYVE1','TIMD4'],
    'Lymph': ['CD8A', 'IL7R', 'CD40LG'],
    'B': ['CD19','SDC1','IGHG1','IGHA1'],
    'Mast': ['KIT', 'CPA3'],
}

for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in adata.var_names]

sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              markers, 
              groupby=f'leiden_{str(resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 25 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CM, FB, EC, etc.
```

Example 26 (python):
```python
markers={
    'Mono':['CCR2','FCN1','VCAN','MPO'],
    'MP':['C1QA','C1QB','CD68','MRC1','CD163','MERTK'],
    'CX3CR1+':['CX3CR1','C3','TREM2','P2RY12','CD200R1'],
    'LYVE1+':['LYVE1','FOLR2'],
     'pDC':['IRF8','IRF4','IL3RA'],
     'DC_mature':['FLT3', 'CCR7','LAMP3', 'IL15', 'CD83', 'LY75', ],
    'Megakaryocyte':['ITGA2B','MPL','ITGB3',],
    'Mast': ['KIT', 'CPA3','HDC','GATA2'],
    'Cycling':['MKI67','TOP2A'],
}
genes = []
markers_sel = {}
for k,g in markers.items():
    markers_sel[k] = [x for x in g if x in adata.var_names]
    genes = genes + [x for x in g if (x in adata.var_names)&(x not in genes)]
    
sc.tl.dendrogram(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              var_names=markers_sel,
              groupby=f'leiden_{str(resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 27 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: Mono, MP, CX3CR1+, etc.
```

Example 28 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 29 (unknown):
```unknown
[1.41297585 1.03895088 1.42284238 1.5805253  1.88093143]
```

Example 30 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True, n_genes=5)
dp.add_totals().show()
```

Example 31 (unknown):
```unknown
[1.41297585 1.03895088 1.42284238 1.5805253  1.88093143]
```

Example 32 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=f'leiden_{str(resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 33 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=['celltypist_midmod2fine', 'conf_score_midmod2fine'],
                wspace=0.7)
```

Example 34 (python):
```python
sc.pl.violin(adata,keys=['total_counts','bin_count'],groupby=f'leiden_{str(resolution_sel)}')
```

Example 35 (unknown):
```unknown
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 36 (python):
```python
# annotate
adata.obs['fine_grain_manual'] = adata.obs[f'leiden_{str(resolution_sel)}'].copy()
adata.obs.replace({'fine_grain_manual':{
    '0':'Macrophages',
    '1':'Macrophages',
    '2':'Monocytes',
    '3':'Erythrocytes',
    '4':'Megakaryocytes',
}},inplace=True)
adata.obs['fine_grain_manual'].value_counts()
```

Example 37 (unknown):
```unknown
fine_grain_manual
Macrophages       440
Monocytes          36
Erythrocytes       26
Megakaryocytes     24
Name: count, dtype: int64
```

Example 38 (python):
```python
ad = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad',backed='r')
set(ad.obs['fine_grain'])
```

Example 39 (unknown):
```unknown
{'AtrialCardiomyocytesCycling',
 'AtrialCardiomyocytesLeft',
 'AtrialCardiomyocytesRight',
 'AtrioventricularNodePacemakerCells',
 'BCells',
 'BCellsMS4A1pos',
 'ChromaffinCells',
 'CoronaryArterialEndothelialCells',
 'CoronaryCapillaryEndothelialCells',
 'CoronaryPericytes',
 'CoronarySmoothMuscleCells',
 'CoronaryVenousEndothelialCells',
 'CoronaryVesselAdventitialFibroblasts',
 'DendriticCellsMature',
 'DendriticCellsType1',
 'DuctusArteriosusSmoothMuscleCells',
 'EndocardialCells',
 'EndocardialCushionCells',
 'EpicardiumDerivedCells',
 'GreatVesselAdventitialFibroblasts',
 'GreatVesselArterialEndothelialCells',
 'GreatVesselSmoothMuscleCells',
 'GreatVesselVenousEndothelialCells',
 'InnateLymphoidCells',
 'LymphNodeFibroblasticReticularCells',
 'LymphaticEndothelialCells',
 'MacrophagesATF3pos',
 'MacrophagesCX3CR1pos',
 'MacrophagesLYVE1pos',
 'MacrophagesTIMD4pos',
 'MastCells',
 'Megakaryocytes',
 'MesothelialEpicardialCells',
 'MonocyteDerivedCells',
 'Monocytes',
 'MonocytesMPOpos',
 'MyocardialInterstitialFibroblasts',
 'Myofibroblasts',
 'NaturalKillerCells',
 'NeuronPrecursors',
 'ParasympatheticNeurons',
 'PericardialCellsFibrous',
 'PericardialCellsIntermediate',
 'PericardialCellsParietal',
 'PlasmacytoidDendriticCells',
 'ProBCells',
 'SchwannCellPrecursors',
 'SchwannCells',
 'SinoatrialNodePacemakerCells',
 'SubEpicardialFibroblasts',
 'SympatheticNeurons',
 'TCellsCD4pos',
 'TCellsCD8pos',
 'TregsCD4pos',
 'ValveEndothelialCells',
 'ValveInterstitialCells',
 'VentricularCardiomyocytesCycling',
 'VentricularCardiomyocytesLeftCompact',
 'VentricularCardiomyocytesLeftTrabeculated',
 'VentricularCardiomyocytesRightCompact',
 'VentricularCardiomyocytesRightTrabeculated',
 'VentricularConductionSystemDistal',
 'VentricularConductionSystemProximal'}
```

Example 40 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 41 (python):
```python
markers={
    'Mono':['CCR2','FCN1','VCAN','MPO'],
    'MP':['C1QA','C1QB','CD68','MRC1','CD163','MERTK'],
    'CX3CR1+':['CX3CR1','C3','TREM2','P2RY12','CD200R1'],
    'LYVE1+':['LYVE1','FOLR2'],
     'pDC':['IRF8','IRF4','IL3RA'],
     'DC_mature':['FLT3', 'CCR7','LAMP3', 'IL15', 'CD83', 'LY75', ],
    'Megakaryocyte':['ITGA2B','MPL','ITGB3',],
    'Mast': ['KIT', 'CPA3','HDC','GATA2'],
    'Cycling':['MKI67','TOP2A'],
}
genes = []
markers_sel = {}
for k,g in markers.items():
    markers_sel[k] = [x for x in g if x in adata.var_names]
    genes = genes + [x for x in g if (x in adata.var_names)&(x not in genes)]

sc.tl.dendrogram(adata,groupby='fine_grain_manual')
sc.pl.dotplot(adata,
              var_names=markers_sel,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 42 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: Macrophages, Monocytes, Erythrocytes, etc.
var_group_labels: Mono, MP, CX3CR1+, etc.
```

Example 43 (python):
```python
adata_sub = adata[adata.obs['fine_grain_manual'].isin(['Monocytes','Macrophages'])]
adata_sub.obs['fine_grain_manual'].value_counts()
```

Example 44 (unknown):
```unknown
fine_grain_manual
Macrophages    440
Monocytes       36
Name: count, dtype: int64
```

Example 45 (python):
```python
# drop leiden clustering results
resolutions_list = [0.2,0.4,0.6,1, 1.2,1.5,2]
for resolution in resolutions_list:
    adata_sub.obs.drop(f'leiden_{str(resolution)}',axis=1,inplace=True)
adata_sub
```

Example 46 (unknown):
```unknown
AnnData object with n_obs × n_vars = 476 × 13502
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_manual', 'leiden_0.5'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'dendrogram_fine_grain_manual', 'dendrogram_leiden_0.4', 'dendrogram_leiden_0.6', 'dendrogram_leiden_1', 'dendrogram_leiden_1.2', 'dendrogram_leiden_1.5', 'hvg', 'leiden_0.2', 'leiden_0.4', 'leiden_0.5', 'leiden_0.5_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'rank_genes_groups', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 47 (python):
```python
## Remove mid-grain markers from the feature
# read in markers
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0)
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group']!=celltype]
# remove
mask = [x not in list(midgrain_markers['names']) for x in adata_sub.var_names]
bdata = adata_sub[:,mask]
bdata
```

Example 48 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 476 × 12303
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_manual', 'leiden_0.5'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'dendrogram_fine_grain_manual', 'dendrogram_leiden_0.4', 'dendrogram_leiden_0.6', 'dendrogram_leiden_1', 'dendrogram_leiden_1.2', 'dendrogram_leiden_1.5', 'hvg', 'leiden_0.2', 'leiden_0.4', 'leiden_0.5', 'leiden_0.5_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'rank_genes_groups', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 49 (python):
```python
# remove cell cycle genes
cell_cycle_genes = ['MCM5','PCNA','TYMS','FEN1','MCM2','MCM4','RRM1','UNG','GINS2','MCM6','CDCA7','DTL','PRIM1','UHRF1','MLF1IP','HELLS','RFC2','RPA2','NASP',
                'RAD51AP1','GMNN','WDR76','SLBP','CCNE2','UBR7','POLD3','MSH2','ATAD2','RAD51','RRM2','CDC45','CDC6','EXO1','TIPIN','DSCC1','BLM','CASP8AP2',
                'USP1','CLSPN','POLA1','CHAF1B','BRIP1','E2F8','HMGB2','CDK1','NUSAP1','UBE2C','BIRC5','TPX2','TOP2A','NDC80','CKS2','NUF2','CKS1B','MKI67',
                'TMPO','CENPF','TACC3','FAM64A','SMC4','CCNB2','CKAP2L','CKAP2','AURKB','BUB1','KIF11','ANP32E','TUBB4B','GTSE1','KIF20B','HJURP','CDCA3','HN1',
                'CDC20','TTK','CDC25C','KIF2C','RANGAP1','NCAPD2','DLGAP5','CDCA2','CDCA8','ECT2','KIF23','HMMR','AURKA','PSRC1','ANLN','LBR','CKAP5','CENPE','CTCF',
                'NEK2','G2E3','GAS2L3','CBX5','CENPA']
mask = [x not in cell_cycle_genes for x in bdata.var_names]
bdata = bdata[:,mask]
bdata.shape
```

Example 50 (unknown):
```unknown
(476, 12209)
```

Example 51 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata_sub.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata_sub,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata_sub,min_dist=0.5)
sc.pl.embedding(adata_sub,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
# leiden clustering
resolutions_list = [0.2,0.4,0.5,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata_sub,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata_sub,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 52 (unknown):
```unknown
number of hvgs: 2992
```

Example 53 (python):
```python
markers={
    'Mono':['CCR2','FCN1','VCAN','MPO'],
    'MP':['C1QA','C1QB','CD68','MRC1','CD163','MERTK'],
    'CX3CR1+':['CX3CR1','C3','TREM2','P2RY12','CD200R1'],
    'LYVE1+':['LYVE1','FOLR2'],
     'pDC':['IRF8','IRF4','IL3RA'],
     'DC_mature':['FLT3', 'CCR7','LAMP3', 'IL15', 'CD83', 'LY75', ],
    'Megakaryocyte':['ITGA2B','MPL','ITGB3',],
    'Mast': ['KIT', 'CPA3','HDC','GATA2'],
    'Cycling':['MKI67','TOP2A'],
}
genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in adata_sub.var_names)&(x not in genes)]
sc.pl.embedding(adata_sub,basis='X_umap',color=genes,
               ncols=5, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 54 (unknown):
```unknown
sub_resolution_sel = 1
```

Example 55 (python):
```python
sc.tl.dendrogram(adata_sub,groupby=f'leiden_{str(sub_resolution_sel)}')
sc.pl.dotplot(adata_sub,
              var_names=markers_sel,
              groupby=f'leiden_{str(sub_resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 56 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: Mono, MP, CX3CR1+, etc.
```

Example 57 (python):
```python
# DEG analysis
print(adata_sub.X.data[:5])
adata_sub.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata_sub,groupby=f'leiden_{str(sub_resolution_sel)}')
sc.tl.dendrogram(adata_sub, groupby=f'leiden_{str(sub_resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata_sub, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 58 (unknown):
```unknown
[1.41297585 1.03895088 1.42284238 1.5805253  1.88093143]
```

Example 59 (python):
```python
sc.pl.embedding(adata_sub,basis='X_umap',
                color=f'leiden_{str(sub_resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 60 (python):
```python
# calculate CX3CR1/LYVE1 expression ratio
# to annotate MacrophagesCX3CR1pos
df = adata_sub.to_df()[['CX3CR1','LYVE1']].copy()
df[f'leiden_{str(sub_resolution_sel)}'] = adata_sub.obs[f'leiden_{str(sub_resolution_sel)}']
df = df.groupby(f'leiden_{str(sub_resolution_sel)}').mean()
df['CX3CR1-to-LYVE1'] = df['CX3CR1']/df['LYVE1']
df.sort_values('CX3CR1-to-LYVE1',ascending=False)
```

Example 61 (python):
```python
# annotate
adata_sub.obs['fine_grain_manual'] = adata_sub.obs[f'leiden_{str(sub_resolution_sel)}'].copy()
adata_sub.obs.replace({'fine_grain_manual':{
    '0':'MacrophagesLYVE1pos',
    '1':'MacrophagesLYVE1pos',
    '2':'MacrophagesLYVE1pos',
    '3':'MacrophagesCX3CR1pos',
    '4':'Monocytes',
    '5':'MacrophagesLYVE1pos',
    '6':'MonocyteDerivedCells',
     '7':'MacrophagesLYVE1pos',
     '8':'MacrophagesCX3CR1pos',
    '9':'MacrophagesLYVE1pos',
}},inplace=True)
adata_sub.obs['fine_grain_manual'].value_counts()
```

Example 62 (unknown):
```unknown
fine_grain_manual
MacrophagesLYVE1pos     299
MacrophagesCX3CR1pos     90
Monocytes                47
MonocyteDerivedCells     40
Name: count, dtype: int64
```

Example 63 (python):
```python
ad = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad',backed='r')
set(ad.obs['fine_grain'])
```

Example 64 (unknown):
```unknown
{'AtrialCardiomyocytesCycling',
 'AtrialCardiomyocytesLeft',
 'AtrialCardiomyocytesRight',
 'AtrioventricularNodePacemakerCells',
 'BCells',
 'BCellsMS4A1pos',
 'ChromaffinCells',
 'CoronaryArterialEndothelialCells',
 'CoronaryCapillaryEndothelialCells',
 'CoronaryPericytes',
 'CoronarySmoothMuscleCells',
 'CoronaryVenousEndothelialCells',
 'CoronaryVesselAdventitialFibroblasts',
 'DendriticCellsMature',
 'DendriticCellsType1',
 'DuctusArteriosusSmoothMuscleCells',
 'EndocardialCells',
 'EndocardialCushionCells',
 'EpicardiumDerivedCells',
 'GreatVesselAdventitialFibroblasts',
 'GreatVesselArterialEndothelialCells',
 'GreatVesselSmoothMuscleCells',
 'GreatVesselVenousEndothelialCells',
 'InnateLymphoidCells',
 'LymphNodeFibroblasticReticularCells',
 'LymphaticEndothelialCells',
 'MacrophagesATF3pos',
 'MacrophagesCX3CR1pos',
 'MacrophagesLYVE1pos',
 'MacrophagesTIMD4pos',
 'MastCells',
 'Megakaryocytes',
 'MesothelialEpicardialCells',
 'MonocyteDerivedCells',
 'Monocytes',
 'MonocytesMPOpos',
 'MyocardialInterstitialFibroblasts',
 'Myofibroblasts',
 'NaturalKillerCells',
 'NeuronPrecursors',
 'ParasympatheticNeurons',
 'PericardialCellsFibrous',
 'PericardialCellsIntermediate',
 'PericardialCellsParietal',
 'PlasmacytoidDendriticCells',
 'ProBCells',
 'SchwannCellPrecursors',
 'SchwannCells',
 'SinoatrialNodePacemakerCells',
 'SubEpicardialFibroblasts',
 'SympatheticNeurons',
 'TCellsCD4pos',
 'TCellsCD8pos',
 'TregsCD4pos',
 'ValveEndothelialCells',
 'ValveInterstitialCells',
 'VentricularCardiomyocytesCycling',
 'VentricularCardiomyocytesLeftCompact',
 'VentricularCardiomyocytesLeftTrabeculated',
 'VentricularCardiomyocytesRightCompact',
 'VentricularCardiomyocytesRightTrabeculated',
 'VentricularConductionSystemDistal',
 'VentricularConductionSystemProximal'}
```

Example 65 (python):
```python
sc.pl.embedding(adata_sub, basis=f"X_umap",
                color='fine_grain_manual',
                wspace=0.3)
```

Example 66 (python):
```python
sc.tl.dendrogram(adata_sub,groupby='fine_grain_manual')
sc.pl.dotplot(adata_sub,
              var_names=markers_sel,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 67 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: MacrophagesLYVE1pos, MacrophagesCX3CR1pos, Monocytes, etc.
var_group_labels: Mono, MP, CX3CR1+, etc.
```

Example 68 (python):
```python
adata.obs['fine_grain_manual'] = adata.obs['fine_grain_manual'].astype('str')
adata.obs.loc[adata_sub.obs_names,'fine_grain_manual'] = adata_sub.obs['fine_grain_manual'].astype('str').copy()
```

Example 69 (python):
```python
sc.pl.embedding(adata, basis=f"X_umap",
                color='fine_grain_manual'
                )
```

Example 70 (unknown):
```unknown
... storing 'fine_grain_manual' as categorical
```

Example 71 (python):
```python
# save
adata.write(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
print(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/Visium/Others/Plotting_multiple_Compact-Trabeculated.html

**Contents:**
- Import required modules¶
- Prepare functions¶
- Read in anndata¶
- Plot the uncropped section¶

**Examples:**

Example 1 (python):
```python
# this is for the jupyter notebook at JupyterLab
# to make sys.path between the terminal and Jupyter same
import sys

print(sys.path) # especially the path to site-packages are different... so thats why couldn't find correct package...

# remove weird paths...
sys.path = sys.path[:4]+[sys.path[6]]
print(sys.path)
```

Example 2 (unknown):
```unknown
['/nfs/team205/kk18/notebooks/P48_Foetal-heart_Visium/notebooks/notebooks_for_manuscript', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python39.zip', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/lib-dynload', '', '/nfs/users/nfs_k/kk18/.local/lib/python3.9/site-packages', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages']
['/nfs/team205/kk18/notebooks/P48_Foetal-heart_Visium/notebooks/notebooks_for_manuscript', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python39.zip', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/lib-dynload', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages']
```

Example 3 (python):
```python
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

import scanpy as sc
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 
import seaborn as sns
import os 
from mpl_toolkits.axes_grid1 import make_axes_locatable

from cell2location.plt import plot_spatial
```

Example 4 (python):
```python
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Global seed set to 0
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
```

Example 5 (python):
```python
import session_info
session_info.show()
```

Example 6 (python):
```python
-----
cell2location       0.1.3
matplotlib          3.7.1
mpl_toolkits        NA
numpy               1.24.3
pandas              2.0.1
scanpy              1.9.3
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 7 (python):
```python
PIL                 9.5.0
absl                NA
anndata             0.9.1
asttokens           NA
attr                23.1.0
backcall            0.2.0
certifi             2023.05.07
cffi                1.15.1
charset_normalizer  3.1.0
chex                0.1.7
comm                0.1.3
contextlib2         NA
cycler              0.10.0
cython_runtime      NA
dateutil            2.8.2
debugpy             1.6.7
decorator           5.1.1
defusedxml          0.7.1
docrep              0.3.2
etils               1.2.0
executing           1.2.0
flax                0.6.10
fsspec              2023.5.0
google              NA
h5py                3.8.0
idna                3.4
igraph              0.10.4
importlib_metadata  NA
importlib_resources NA
ipykernel           6.23.0
ipython_genutils    0.2.0
jax                 0.4.9
jaxlib              0.4.9
jedi                0.18.2
joblib              1.2.0
kiwisolver          1.4.4
leidenalg           0.9.1
lightning_fabric    1.9.5
lightning_utilities 0.8.0
llvmlite            0.40.0
ml_collections      NA
ml_dtypes           0.1.0
mpmath              1.3.0
msgpack             1.0.5
mudata              0.2.2
multipledispatch    0.6.0
natsort             8.3.1
numba               0.57.0
numpyro             0.11.0
nvfuser             NA
opt_einsum          v3.3.0
optax               0.1.5
packaging           23.1
parso               0.8.3
patsy               0.5.3
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
platformdirs        3.5.0
prompt_toolkit      3.0.38
psutil              5.9.5
ptyprocess          0.7.0
pure_eval           0.2.2
pycparser           2.21
pydev_ipython       NA
pydevconsole        NA
pydevd              2.9.5
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.15.1
pynndescent         0.5.10
pyparsing           3.0.9
pyro                1.8.4
pytorch_lightning   1.9.5
pytz                2023.3
requests            2.30.0
rich                NA
scipy               1.10.1
scvi                0.20.3
setuptools          66.0.0
six                 1.16.0
sklearn             1.2.2
stack_data          0.6.2
statsmodels         0.14.0
sympy               1.12
texttable           1.6.7
threadpoolctl       3.1.0
toolz               0.12.0
torch               2.0.1+cu117
torchmetrics        0.11.4
tornado             6.3.1
tqdm                4.65.0
traitlets           5.9.0
tree                0.1.8
typing_extensions   NA
umap                0.5.3
urllib3             2.0.2
wcwidth             0.2.6
yaml                6.0
zipp                NA
zmq                 25.0.2
zoneinfo            NA
```

Example 8 (unknown):
```unknown
-----
IPython             8.13.2
jupyter_client      8.2.0
jupyter_core        5.3.0
notebook            6.5.4
-----
Python 3.9.16 (main, Mar  8 2023, 14:00:05) [GCC 11.2.0]
Linux-5.4.0-169-generic-x86_64-with-glibc2.27
-----
Session information updated at 2024-02-12 20:46
```

Example 9 (python):
```python
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 10 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk'
```

Example 11 (python):
```python
# This function selects the data for multiple slides from the spatial anndata object.
# Modified `select_slide` in cell2loc for multiple slides

# @param adata: anndata object with multiple spatial experiments
# @param s: list of selected slides
# @param s_col: column in adata.obs listing experiment name for each location

def select_multislide(adata, s, s_col='sample'):

    slides = adata[adata.obs[s_col].isin(s), :]
    s_keys = list(slides.uns['spatial'].keys())
    
    uns_dict = {}
    for ss in s:
        s_spatial=np.array(s_keys)[[ss in k for k in s_keys]][0]
        uns_dict[s_spatial]=slides.uns['spatial'][s_spatial]
    
    slides.uns['spatial'] = uns_dict

    return slides
```

Example 12 (python):
```python
adata_vis = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-FFPE_foetal_raw.h5ad')

# add abundance to .obs
df = adata_vis.obsm['q05_cell_abundance_w_sf'].copy()
## rename columns
df.columns = [x.replace('q05cell_abundance_w_sf_','') for x in df.columns]
adata_vis.obs[df.columns]=df.copy()

# set symboßl as var_names
adata_vis.var.set_index('SYMBOL',inplace=True)

adata_vis
```

Example 13 (unknown):
```unknown
AnnData object with n_obs × n_vars = 19973 × 18085
    obs: 'in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'JC_edit', 'AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight', 'VentricularCardiomyocytesLeftCompact', 'VentricularCardiomyocytesRightCompact', 'VentricularCardiomyocytesLeftTrabeculated', 'VentricularCardiomyocytesRightTrabeculated', 'SinoatrialNodeCardiomyocytes', 'AtrioventricularNodeCardiomyocytes', 'VentricularConductionSystemProximal', 'VentricularConductionSystemDistal', 'GreatVesselAdventitialFibroblasts', 'CoronaryVesselAdventitialFibroblasts', 'MyocardialInterstitialFibroblasts1', 'MyocardialInterstitialFibroblasts2', 'Myofibroblasts', 'LymphNodeFibroblasticReticularCells', 'ValveInterstitialCells', 'GreatVesselSmoothMuscleCells', 'CoronarySmoothMuscleCells', 'CoronaryPericytes', 'GreatVesselArterialEndothelialCells', 'GreatVesselVenousEndothelialCells', 'CoronaryArterialEndothelialCells', 'CoronaryVenousEndothelialCells', 'CoronaryCapillaryEndothelialCells', 'EndocardialCells', 'EndocardialCushionCells', 'ValveEndothelialCells', 'LymphaticEndothelialCells', 'MesothelialEpicardialCells', 'EpicardiumDerivedCells', 'NeuronPrecursors', 'ChromaffinCells', 'SympatheticNeurons', 'ParaympatheticNeurons', 'SchwannCellPrecursors', 'SchwannCells', 'MonocytesMPOpos', 'Monocytes', 'MonocyteDerivedCells', 'MacrophagesCX3CR1pos', 'MacrophagesTIMD4pos', 'MacrophagesLYVE1pos', 'MacrophagesATF3pos', 'DendriticCellsType1', 'DendriticCellsMature', 'PlasmacytoidDendriticCells', 'MastCells', 'Megakaryocytes', 'TCellsCD4pos', 'TCellsCD8pos', 'TregsCD4pos', 'ProBCells', 'BCells', 'BCellsMS4A1pos', 'NaturalKillerCells', 'InnateLymphoidCells'
    var: 'gene_ids', 'feature_types', 'genome', 'mt'
    uns: 'spatial'
    obsm: 'MT', 'means_cell_abundance_w_sf', 'q05_cell_abundance_w_sf', 'q95_cell_abundance_w_sf', 'spatial', 'stds_cell_abundance_w_sf'
```

Example 14 (unknown):
```unknown
slide_id = 'BHF_F_Hea13295831'
```

Example 15 (python):
```python
slide = select_multislide(adata_vis, [slide_id])
    
# plot in spatial coordinates
with mpl.rc_context({#'axes.facecolor':  'black',
                     'figure.figsize': [4.5, 5]}):

    sc.pl.spatial(slide, cmap='RdPu',
                  color=['JC_edit'], 
                  ncols=5, size=0, 
                  img_key='hires', alpha_img=1,wspace=0.3,
                  vmin=0, vmax='p99.5',# palette=palette
                  legend_loc=None,
                  title='H&E',
                  show=False
                 )
    plt.savefig(f"{figdir}/visium_HE_{slide_id}.pdf", bbox_inches="tight",dpi=300)
```

Example 16 (unknown):
```unknown
/tmp/ipykernel_46793/2805736760.py:18: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  slides.uns['spatial'] = uns_dict
```

Example 17 (unknown):
```unknown
slide_id = 'BHF_F_Hea13295830'
```

Example 18 (python):
```python
slide = select_multislide(adata_vis, [slide_id])
    
# plot in spatial coordinates
with mpl.rc_context({#'axes.facecolor':  'black',
                     'figure.figsize': [4.5, 5]}):

    sc.pl.spatial(slide, cmap='RdPu',
                  color=['JC_edit'], 
                  ncols=5, size=0, 
                  img_key='hires', alpha_img=1,wspace=0.3,
                  vmin=0, vmax='p99.5',# palette=palette
                  legend_loc=None,
                  title='H&E',
                  show=False
                 )
    plt.savefig(f"{figdir}/visium_HE_{slide_id}.pdf", bbox_inches="tight",dpi=300)
```

Example 19 (unknown):
```unknown
/tmp/ipykernel_46793/2805736760.py:18: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  slides.uns['spatial'] = uns_dict
```

Example 20 (python):
```python
slide = select_multislide(adata_vis, [slide_id])
    
# plot in spatial coordinates
with mpl.rc_context({#'axes.facecolor':  'black',
                     'figure.figsize': [4.5, 5]}):

    sc.pl.spatial(slide, cmap='RdPu',
                  color=['JC_edit'], 
                  ncols=5, size=0, 
                  img_key='hires', alpha_img=1,wspace=0.3,
                  vmin=0, vmax='p99.5',# palette=palette
                  crop_coord = [6000,12000,11500,16000],
                  legend_loc=None,
                  title='',
                  show=False,
                 )
    plt.savefig(f"{figdir}/visium_HE_cropped-LV_{slide_id}.pdf", bbox_inches="tight",dpi=300)
```

Example 21 (unknown):
```unknown
/tmp/ipykernel_46793/2805736760.py:18: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  slides.uns['spatial'] = uns_dict
```

Example 22 (python):
```python
clust_show = ['VentricularCardiomyocytesLeftCompact','VentricularCardiomyocytesLeftTrabeculated']
order_cmap = [4,1] # 0:yellow, 1:orange, 2:blue, 3:green, 4:purple, 5:grey, 6:white

plt.rcParams['axes.grid'] = False
with mpl.rc_context({"figure.figsize": (20,20)}):

        fig = plot_spatial(slide, 
                      coords=slide.obsm['spatial'], 
                      show_img=True, img_alpha=1,
                      max_color_quantile=0.99, 
                      circle_diameter=0, alpha_scaling=1, labels=clust_show, 
                      #style='dark_background', # fast dark_background
                         colorbar_position=None,
                         colorbar_label_kw={'size': 20, 'weight': 'bold', 'y': 1.29},
                         colorbar_shape={'vertical_gaps': 2, 'horizontal_gaps': 0, 
                                         'width': 0.2, 'height': 0.35},
                         colorbar_tick_size=20,
                           color = clust_show,
                            crop_x = (500,1350),
                           crop_y = (1250,1825)
                    )
        plt.savefig(f"{figdir}/visium_HE_cropped-LV_{slide_id}.pdf", bbox_inches="tight",dpi=300)
```

Example 23 (python):
```python
clust_show = ['VentricularCardiomyocytesLeftCompact','VentricularCardiomyocytesLeftTrabeculated']
order_cmap = [4,1] # 0:yellow, 1:orange, 2:blue, 3:green, 4:purple, 5:grey, 6:white

plt.rcParams['axes.grid'] = False
with mpl.rc_context({"figure.figsize": (20,20)}):

        fig = plot_spatial(slide, 
                      coords=slide.obsm['spatial'], 
                      show_img=True, img_alpha=0.3,
                      max_color_quantile=0.99, 
                      circle_diameter=8, alpha_scaling=1, labels=clust_show, 
                      #style='dark_background', # fast dark_background
                         colorbar_position='right',
                         colorbar_label_kw={'size': 20, 'weight': 'bold', 'y': 1.29},
                         colorbar_shape={'vertical_gaps': 0.3, 'horizontal_gaps': 0.4, 
                                         'width': 0.2, 'height': 0.1},
                         colorbar_tick_size=20,
                           color = clust_show,
                           reorder_cmap = order_cmap,
                            crop_x = (500,1350),
                           crop_y = (1250,1825)
                    )
        plt.savefig(f"{figdir}/visium_cell2loc-outs_cropped-LV_{slide_id}.pdf", bbox_inches="tight",dpi=300)
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig3_SAN_niche/VisiumHD_CCSaxis_no1_annotate-landmark.html

**Contents:**
- Variables¶
- Read in data¶
- Plot¶
- Annotate¶
- Save¶

bin2cell output which has celltypist output as well

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import celltypist
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 4 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects'
sample_id = 'HEA_FOET14880396'
```

Example 5 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 6 (python):
```python
# read in 
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 7 (unknown):
```unknown
AnnData object with n_obs × n_vars = 129803 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_4Jan2025', 'HistologicalAnnotation'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'HistologicalAnnotation_colors', 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'fine_grain_4Jan2025_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 8 (python):
```python
sc.pl.embedding(adata,basis='X_umap',color=["celltypist_mid_fin"],wspace=0.2,cmap='RdPu')
```

Example 9 (python):
```python
sc.pl.embedding(adata,basis='X_umap',color=['fine_grain_4Jan2025'],wspace=0.2,cmap='RdPu')
```

Example 10 (python):
```python
cells = [
    'SinoatrialNodePacemakerCellsHorn',
    'SinoatrialNodePacemakerCellsHead',
    'SinoatrialNodePacemakerCellsTail',
    # 'AtrioventricularNodePacemakerCells',
    # 'VentricularConductionSystem'
    
    # 'MacrophagesCX3CR1pos',
    # 'ParasympatheticNeurons'
]

celltype_key = 'fine_grain_4Jan2025'
adata.uns[f"{celltype_key}_colors"][list(adata.obs[celltype_key].cat.categories).index('SinoatrialNodePacemakerCellsHorn')]='tab:orange'
adata.uns[f"{celltype_key}_colors"][list(adata.obs[celltype_key].cat.categories).index('SinoatrialNodePacemakerCellsHead')]='tab:blue'
adata.uns[f"{celltype_key}_colors"][list(adata.obs[celltype_key].cat.categories).index('SinoatrialNodePacemakerCellsTail',)]='tab:green'
# adata.uns[f"{celltype_key}_colors"][list(adata.obs[celltype_key].cat.categories).index('AtrioventricularNodePacemakerCells')]='orange'
# adata.uns[f"{celltype_key}_colors"][list(adata.obs[celltype_key].cat.categories).index('VentricularConductionSystem')]='mediumpurple'

sc.pl.spatial(adata, color=celltype_key, groups=cells,
              img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
              size=10,legend_fontsize=30,spot_size=10,
                  frameon=False,) # save=f'_{c}_b2c',
```

Example 11 (python):
```python
cells = [
     'SinoatrialNodePacemakerCellsHorn',
    'SinoatrialNodePacemakerCellsHead',
    'SinoatrialNodePacemakerCellsTail',
    # 'AtrioventricularNodePacemakerCells',
    # 'VentricularConductionSystem'
]

mask = ((adata.obs['array_row'] >= 1700) & 
        (adata.obs['array_row'] <= 2300) & 
        (adata.obs['array_col'] >= 600) & 
        (adata.obs['array_col'] <= 1200)
       )
bdata = adata[mask]
n_celltype = len(bdata.obs[celltype_key].cat.categories)
bdata.uns[f'{celltype_key}_colors'] = list(np.repeat('#ffff00',n_celltype))

# update colors
for c in cells:
    # get color in adata
    ind = list(adata.obs[celltype_key].cat.categories).index(c)
    color = adata.uns[f'{celltype_key}_colors'][ind]
    # update color in bdata
    ind = list(bdata.obs[celltype_key].cat.categories).index(c)
    bdata.uns[f'{celltype_key}_colors'][ind] = color

sc.pl.spatial(bdata, color=celltype_key, groups=cells,
              img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
              size=10,legend_fontsize=30,spot_size=10,
                  frameon=False,) # save=f'_{c}_b2c',
```

Example 12 (python):
```python
# start
cells = [
     'SinoatrialNodePacemakerCellsHorn',
    #'SinoatrialNodePacemakerCellsHead',
    #'SinoatrialNodePacemakerCellsTail',
    # 'AtrioventricularNodePacemakerCells',
    # 'VentricularConductionSystem'
]

mask = ((adata.obs['array_row'] >= 2250) & 
        (adata.obs['array_row'] <= 2300) & 
        (adata.obs['array_col'] >= 700) & 
        (adata.obs['array_col'] <= 820)
       )
bdata = adata[mask]
n_celltype = len(bdata.obs[celltype_key].cat.categories)
bdata.uns[f'{celltype_key}_colors'] = list(np.repeat('#ffff00',n_celltype))

# update colors
for c in cells:
    # get color in adata
    ind = list(adata.obs[celltype_key].cat.categories).index(c)
    color = adata.uns[f'{celltype_key}_colors'][ind]
    # update color in bdata
    ind = list(bdata.obs[celltype_key].cat.categories).index(c)
    bdata.uns[f'{celltype_key}_colors'][ind] = color

sc.pl.spatial(bdata, color=celltype_key, groups=cells,
              img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
              size=10,legend_fontsize=30,spot_size=10,
                  frameon=False,) # save=f'_{c}_b2c',

# assign
start_cells = bdata.obs_names[bdata.obs[celltype_key]=='SinoatrialNodePacemakerCellsHorn']
print(len(start_cells))
```

Example 13 (python):
```python
# end
cells = [
    #  'SinoatrialNodePacemakerCellsHorn',
    # 'SinoatrialNodePacemakerCellsHead',
    'SinoatrialNodePacemakerCellsTail',
    # 'AtrioventricularNodePacemakerCells',
    # 'VentricularConductionSystem'
]

mask = ((adata.obs['array_row'] >= 1800) & 
        (adata.obs['array_row'] <= 2200) & 
        (adata.obs['array_col'] >= 800) & 
        (adata.obs['array_col'] <= 870)
       )
bdata = adata[mask]
n_celltype = len(bdata.obs[celltype_key].cat.categories)
bdata.uns[f'{celltype_key}_colors'] = list(np.repeat('#ffff00',n_celltype))

# update colors
for c in cells:
    # get color in adata
    ind = list(adata.obs[celltype_key].cat.categories).index(c)
    color = adata.uns[f'{celltype_key}_colors'][ind]
    # update color in bdata
    ind = list(bdata.obs[celltype_key].cat.categories).index(c)
    bdata.uns[f'{celltype_key}_colors'][ind] = color

sc.pl.spatial(bdata, color=celltype_key, groups=cells,
              img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
              size=10,legend_fontsize=30,spot_size=10,
                  frameon=False,) # save=f'_{c}_b2c',

# assign
end_cells = bdata.obs_names[bdata.obs[celltype_key]=='SinoatrialNodePacemakerCellsTail']
print(len(end_cells))
```

Example 14 (python):
```python
# plot landmarks
adata.obs['SANaxis_landmarks'] = 'nan'
adata.obs.loc[start_cells,'SANaxis_landmarks'] = 'start'
adata.obs.loc[end_cells,'SANaxis_landmarks'] = 'end'

mask = ((adata.obs['array_row'] >= 1700) & 
        (adata.obs['array_row'] <= 2300) & 
        (adata.obs['array_col'] >= 600) & 
        (adata.obs['array_col'] <= 1200)
       )
bdata = adata[mask]

sc.pl.spatial(bdata, color='SANaxis_landmarks', groups=['start','end'],
              img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
              size=10,legend_fontsize=30, spot_size=10,
                  frameon=False,) # save=f'_{c}_b2c',
```

Example 15 (unknown):
```unknown
... storing 'SANaxis_landmarks' as categorical
```

Example 16 (python):
```python
adata.write(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 17 (unknown):
```unknown
... storing 'SANaxis_landmarks' as categorical
```

Example 18 (unknown):
```unknown
AnnData object with n_obs × n_vars = 129803 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_4Jan2025', 'HistologicalAnnotation', 'SANaxis_landmarks'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'HistologicalAnnotation_colors', 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'fine_grain_4Jan2025_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig3_SAN_niche/VisiumHD_CCSaxis_no5_HD_to_single-cell.html

**Contents:**
- Variables¶
- Read in data¶
- Subset CCS cells¶
- Select gene features¶
- Single-cell data: generate latent space and perform clustering¶
- CellTypist model training¶
- CellTypist - Predict¶
- Manually annotate cycling cells and transitional cells¶
- Clustering-based vs CellTypist-based, which is better?¶
- Annotate¶
- Save¶

bin2cell output which has celltypist output as well

1510 rows × 14912 columns

Gene features to use: - all genes - HVGs - axis genes: with or without midgrain cell type markers

--> CellTypist, All gene is slightly better than clustering --> will go with CellTypist

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42
```

Example 2 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/tissueaxis.py")
tissueaxis = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = tissueaxis
spec.loader.exec_module(tissueaxis)

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 3 (python):
```python
import scanpy.external as sce
```

Example 4 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 5 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects'
sample_id = 'HEA_FOET14880396'
```

Example 6 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 7 (unknown):
```unknown
celltype_key = 'fine_grain_8Dec2024'
axis_name = 'SAN_axis'
```

Example 8 (python):
```python
# read in HD data
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 9 (unknown):
```unknown
AnnData object with n_obs × n_vars = 129803 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_8Dec2024', 'AtrialCCS_landmarks', 'SANaxis_landmarks', 'SAN_axis', 'SAN_axis_bin_annot'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'fine_grain_8Dec2024_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 10 (python):
```python
# read in single-cell data
adata_sc = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad')

# subset multiome data
adata_sc = adata_sc[adata_sc.obs['kit_10x']=='Multiome-v1']
adata_sc
```

Example 11 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 211145 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'region', 'age', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'donor_by_library-prep', 'multiplexed', 'SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet', 'batch_key', '_scvi_batch', 'FACSgate', 'fine_grain', 'mid_grain', 'coarse_grain', 'sex', 'week', 'trimester', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', '_scvi_labels', 'stress_score', 'hb1_score'
    var: 'gene_name_scRNA-0', 'gene_id'
    uns: 'FACSgate_colors', '_scvi_manager_uuid', '_scvi_uuid', 'age_colors', 'cell_or_nuclei_colors', 'coarse_grain_colors', 'cycling_colors', 'donor_colors', 'fine_grain_colors', 'heart_or_greatvessels_colors', 'hvg', 'kit_10x_colors', 'modality_colors', 'neighbors', 'phase_colors', 'region_colors', 'sex_colors', 'trimester_colors', 'umap', 'week_colors'
    obsm: 'X_scVI', 'X_umap', '_scvi_extra_categorical_covs', '_scvi_extra_continuous', '_scvi_extra_continuous_covs', 'latent_gene_encoding'
    obsp: 'connectivities', 'distances'
```

Example 12 (python):
```python
# subset
cells = [
     'SinoatrialNodePacemakerCellsHead',
 'SinoatrialNodePacemakerCellsHorn',
 'SinoatrialNodePacemakerCellsTail',
]

adata = adata[adata.obs[celltype_key].isin(cells)]
adata.obs[celltype_key].value_counts()
```

Example 13 (unknown):
```unknown
fine_grain_8Dec2024
SinoatrialNodePacemakerCellsHead    636
SinoatrialNodePacemakerCellsHorn    501
SinoatrialNodePacemakerCellsTail    373
Name: count, dtype: int64
```

Example 14 (python):
```python
adata.X.data[:5]
```

Example 15 (unknown):
```unknown
array([1.31041406, 1.56012627, 1.65582528, 1.31041406, 1.65582528])
```

Example 16 (python):
```python
cells = [
     'SinoatrialNodePacemakerCells',
]

adata_sc = adata_sc[adata_sc.obs['fine_grain'].isin(cells)]
adata_sc.obs['fine_grain'].value_counts()
```

Example 17 (unknown):
```unknown
fine_grain
SinoatrialNodePacemakerCells    833
Name: count, dtype: int64
```

Example 18 (python):
```python
print(adata_sc.X.data[:5])
sc.pp.normalize_total(adata_sc, target_sum=1e4)
sc.pp.log1p(adata_sc)
print(adata_sc.X.data[:5])
```

Example 19 (unknown):
```unknown
[3. 1. 1. 4. 1.]
```

Example 20 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_normalization.py:207: UserWarning: Received a view of an AnnData. Making a copy.
  view_to_actual(adata)
```

Example 21 (unknown):
```unknown
[2.1097739 1.2283119 1.2283119 2.3666706 1.2283119]
```

Example 22 (python):
```python
# filter genes
sc.pp.filter_genes(adata, min_cells=3)
print(adata.shape)
sc.pp.filter_genes(adata_sc, min_cells=3)
print(adata_sc.shape)
```

Example 23 (python):
```python
/software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:283: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var["n_cells"] = number
```

Example 24 (unknown):
```unknown
(1510, 14912)
(833, 19057)
```

Example 25 (python):
```python
adata.to_df()
```

Example 26 (python):
```python
# prepare input data
# Assume `cell_by_gene_matrix` is your gene expression DataFrame
# and `axis_values` is a pandas Series with axis values
cell_by_gene_matrix = adata.to_df().copy()
axis_values = adata.obs[axis_name].copy()
```

Example 27 (python):
```python
%%time
import numpy as np
import pandas as pd
from sklearn.preprocessing import PolynomialFeatures
from sklearn.metrics import r2_score
import statsmodels.api as sm
from tqdm import tqdm

# Define polynomial degree range
polynomial_degrees = [1, 2, 3, 4]

# Dictionary to store results for each gene
gene_results = {}

# Iterate through each gene
for gene in tqdm(cell_by_gene_matrix.columns):
    # Store the best fit info for the gene
    best_r2 = -np.inf
    best_degree = None
    best_pvalue = None
    
    # Extract gene expression as dependent variable
    y = cell_by_gene_matrix[gene].values.reshape(-1, 1)
    
    # Iterate through polynomial degrees
    for degree in polynomial_degrees:
        # Create polynomial features
        poly = PolynomialFeatures(degree)
        X_poly = poly.fit_transform(axis_values.values.reshape(-1, 1))
        
        # Fit the model using statsmodels for p-values
        model = sm.OLS(y, X_poly).fit()
        
        # Calculate R^2 score and p-values
        r2 = model.rsquared
        pvalues = model.pvalues  # p-values for each term in the polynomial
        
        # Get the minimum p-value as an indicator of significance
        min_pvalue = np.min(pvalues)
        
        # Update best fit if current model is better
        if r2 > best_r2:
            best_r2 = r2
            best_degree = degree
            best_pvalue = min_pvalue
            
    # Store the best result for the gene
    gene_results[gene] = {'best_r2': best_r2, 'best_degree': best_degree, 'best_pvalue': best_pvalue}

# Convert results to a DataFrame
results_df = pd.DataFrame.from_dict(gene_results, orient='index')
```

Example 28 (unknown):
```unknown
100%|██████████| 14912/14912 [01:27<00:00, 171.15it/s]
```

Example 29 (unknown):
```unknown
CPU times: user 1min 8s, sys: 81.1 ms, total: 1min 8s
Wall time: 1min 28s
```

Example 30 (python):
```python
# Filter genes with significant association (e.g., R^2 > 0.3 and p-value < 0.05)
significant_df = results_df[(results_df['best_r2'] > 0.1) & (results_df['best_pvalue'] < 0.05)]
significant_genes = list(significant_df.index)
print(len(significant_genes))
significant_df.index
```

Example 31 (unknown):
```unknown
Index(['NPPA', 'NPPB', 'DHRS3', 'HSPB7', 'FABP3', 'SLC2A1', 'CNN3', 'GJA5',
       'PRRX1', 'CD34',
       ...
       'MMP15', 'TOP2A', 'COL1A1', 'CKM', 'VSTM2L', 'TGM2', 'MB', 'LGALS1',
       'SMS', 'ATP1B4'],
      dtype='object', length=102)
```

Example 32 (python):
```python
# remove mid-grained cell type markers
# read in
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0
                              )
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group'].isin(['AtrialCardiomyocytes','VentricularCardiomyocytes'])==False]

significant_genes_without_midgrain_markers = [x for x in significant_genes if x not in list(midgrain_markers['names'])]
print(len(significant_genes_without_midgrain_markers))
```

Example 33 (python):
```python
tissueaxis.plot_axis(adata,
              axis_col=axis_name,
              binned_col=f'{axis_name}_bin_annot',
              feature_type='gene', # 'gene' or 'celltype'
              features_to_plot=significant_genes,
              features_to_remove=None,
              threshold_express_prop=0.1, # at quantile=0.9
              threshold_cell_abundance=None, # at quantile=0.9
              correlation=False,
              min_max_scale=True,
              row_cluster=True,
              figsize=(4,18),
              cbar_pos=(-0.2, 0.5, 0.05, 0.18),
              cmap='viridis',
          xticklabels=True,
          yticklabels=True,
              title=None,
              return_feature = False,
                     # save=f'{figdir}/tm-axis_OCT_genes_{key}.pdf',
    save_dpi=300,
             )
```

Example 34 (unknown):
```unknown
/nfs/team205/kk18/function/python/tissueaxis.py:57: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  df_gene_exp_prop = df_gene_exp_bi.groupby(f'{axis_col}_binned').mean()
/nfs/team205/kk18/function/python/tissueaxis.py:134: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  df_plot = df_plot.set_index('index').groupby(['variable',f'{axis_col}_binned']).mean().reset_index()
```

Example 35 (python):
```python
tissueaxis.plot_axis(adata,
              axis_col=axis_name,
              binned_col=f'{axis_name}_bin_annot',
              feature_type='gene', # 'gene' or 'celltype'
              features_to_plot=significant_genes_without_midgrain_markers,
              features_to_remove=None,
              threshold_express_prop=0.1, # at quantile=0.9
              threshold_cell_abundance=None, # at quantile=0.9
              correlation=False,
              min_max_scale=True,
              row_cluster=True,
              figsize=(4,12),
              cbar_pos=(-0.2, 0.5, 0.05, 0.18),
              cmap='viridis',
          xticklabels=True,
          yticklabels=True,
              title=None,
              return_feature = False,
                     # save=f'{figdir}/tm-axis_OCT_genes_{key}.pdf',
    save_dpi=300,
             )
```

Example 36 (unknown):
```unknown
/nfs/team205/kk18/function/python/tissueaxis.py:57: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  df_gene_exp_prop = df_gene_exp_bi.groupby(f'{axis_col}_binned').mean()
/nfs/team205/kk18/function/python/tissueaxis.py:134: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  df_plot = df_plot.set_index('index').groupby(['variable',f'{axis_col}_binned']).mean().reset_index()
```

Example 37 (unknown):
```unknown
os.getcwd()
```

Example 38 (unknown):
```unknown
'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3'
```

Example 39 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(adata_sc, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(adata_sc.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(adata_sc, use_highly_variable=True)
# create neighbour graph
sce.pp.harmony_integrate(adata_sc, 'batch_key')
sc.pp.neighbors(adata_sc, use_rep='X_pca_harmony',n_neighbors=15)
sc.tl.umap(adata_sc, min_dist=0.5, spread=1.0)
```

Example 40 (unknown):
```unknown
number of hvgs: 4721
```

Example 41 (unknown):
```unknown
2024-12-12 16:30:53,521 - harmonypy - INFO - Computing initial centroids with sklearn.KMeans...
Computing initial centroids with sklearn.KMeans...
2024-12-12 16:30:53,610 - harmonypy - INFO - sklearn.KMeans initialization complete.
sklearn.KMeans initialization complete.
2024-12-12 16:30:53,613 - harmonypy - INFO - Iteration 1 of 10
Iteration 1 of 10
2024-12-12 16:30:53,706 - harmonypy - INFO - Iteration 2 of 10
Iteration 2 of 10
2024-12-12 16:30:53,799 - harmonypy - INFO - Iteration 3 of 10
Iteration 3 of 10
2024-12-12 16:30:53,887 - harmonypy - INFO - Iteration 4 of 10
Iteration 4 of 10
2024-12-12 16:30:53,971 - harmonypy - INFO - Iteration 5 of 10
Iteration 5 of 10
2024-12-12 16:30:54,029 - harmonypy - INFO - Iteration 6 of 10
Iteration 6 of 10
2024-12-12 16:30:54,081 - harmonypy - INFO - Iteration 7 of 10
Iteration 7 of 10
2024-12-12 16:30:54,172 - harmonypy - INFO - Iteration 8 of 10
Iteration 8 of 10
2024-12-12 16:30:54,325 - harmonypy - INFO - Iteration 9 of 10
Iteration 9 of 10
2024-12-12 16:30:54,429 - harmonypy - INFO - Iteration 10 of 10
Iteration 10 of 10
2024-12-12 16:30:54,529 - harmonypy - INFO - Stopped before convergence
Stopped before convergence
```

Example 42 (python):
```python
sc.pl.embedding(adata_sc, basis='umap',
                    color=['kit_10x','donor','fine_grain'],
               size=20, wspace=0.6, frameon=False,legend_fontsize=14)
```

Example 43 (python):
```python
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata_sc,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata_sc,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 44 (python):
```python
genes = ['SHOX2','HCN4','TBX3','TBX18',
         'GJA1', # CX43
         'GJA5', # CX40
         'NKX2-5',
         'NPPA',
         'TOP2A','MKI67'
        ]

sc.pl.embedding(adata_sc, basis='umap',
                        color=genes,
                   size=30, wspace=0.2, frameon=False,cmap='RdPu')
```

Example 45 (unknown):
```unknown
leiden_res_sel = 1
```

Example 46 (python):
```python
import celltypist
```

Example 47 (python):
```python
sc.pp.highly_variable_genes(adata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(adata.var.highly_variable)}')
hvgs = adata.var_names[adata.var.highly_variable]
hvgs[:5]
```

Example 48 (unknown):
```unknown
number of hvgs: 3600
```

Example 49 (unknown):
```unknown
Index(['HES4', 'ISG15', 'C1orf159', 'ACAP3', 'INTS11'], dtype='object')
```

Example 50 (python):
```python
features = {}
features['all'] = list(adata.var_names)
features['hvg'] = hvgs
features['axis_gene'] = significant_genes
features['axis_gene_wo_midmarkers'] = significant_genes_without_midgrain_markers
```

Example 51 (python):
```python
for key,value in features.items():
    print(key)
    model = celltypist.train(adata[:,value],  # selecting features
                         labels = celltype_key,
                         n_jobs = 10,
                         feature_selection = False, # since manually selected
                         check_expression = False
                        )
    # Save the model
    model.write(f'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/celltypist/model_HD-SANPC_{key}.pkl')
```

Example 52 (unknown):
```unknown
🍳 Preparing data before training
```

Example 53 (unknown):
```unknown
🔬 Input data has 1510 cells and 14912 genes
⚖️ Scaling input data
🏋️ Training data using logistic regression
✅ Model training done!
🍳 Preparing data before training
```

Example 54 (unknown):
```unknown
🔬 Input data has 1510 cells and 3600 genes
⚖️ Scaling input data
🏋️ Training data using logistic regression
✅ Model training done!
🍳 Preparing data before training
```

Example 55 (unknown):
```unknown
🔬 Input data has 1510 cells and 102 genes
⚖️ Scaling input data
🏋️ Training data using logistic regression
✅ Model training done!
🍳 Preparing data before training
```

Example 56 (unknown):
```unknown
axis_gene_wo_midmarkers
```

Example 57 (unknown):
```unknown
🔬 Input data has 1510 cells and 55 genes
⚖️ Scaling input data
🏋️ Training data using logistic regression
✅ Model training done!
```

Example 58 (unknown):
```unknown
os.getcwd()
```

Example 59 (unknown):
```unknown
'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3'
```

Example 60 (python):
```python
adata_sc.X.data[:5]
```

Example 61 (unknown):
```unknown
array([2.1097739, 1.2283119, 1.2283119, 2.3666706, 1.2283119],
      dtype=float32)
```

Example 62 (python):
```python
for key in features.keys():
    print(key)
    predictions = celltypist.annotate(adata_sc, 
                                      model = f'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/celltypist/model_HD-SANPC_{key}.pkl', 
                                      majority_voting = False)
    # plot
    celltypist.dotplot(predictions, use_as_reference = f'leiden_{str(leiden_res_sel)}', use_as_prediction = 'predicted_labels', vmax=1, show=False)
    # add to anndata
    adata_sc = predictions.to_adata(insert_labels = True, insert_conf = True)
    adata_sc.obs.rename(columns={
        'predicted_labels':f'celltypist_prediction_{key}',
        'conf_score':f'celltypist_conf_score_{key}'
    },inplace=True)
```

Example 63 (unknown):
```unknown
⚠️ Warning: invalid expression matrix, expect ALL genes and log1p normalized expression to 10000 counts per cell. The prediction result may not be accurate
🔬 Input data has 833 cells and 19057 genes
🔗 Matching reference genes in the model
```

Example 64 (unknown):
```unknown
🧬 13037 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
⚠️ Warning: invalid expression matrix, expect ALL genes and log1p normalized expression to 10000 counts per cell. The prediction result may not be accurate
🔬 Input data has 833 cells and 19057 genes
🔗 Matching reference genes in the model
```

Example 65 (unknown):
```unknown
🧬 3177 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
⚠️ Warning: invalid expression matrix, expect ALL genes and log1p normalized expression to 10000 counts per cell. The prediction result may not be accurate
🔬 Input data has 833 cells and 19057 genes
🔗 Matching reference genes in the model
```

Example 66 (unknown):
```unknown
🧬 101 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
⚠️ Warning: invalid expression matrix, expect ALL genes and log1p normalized expression to 10000 counts per cell. The prediction result may not be accurate
🔬 Input data has 833 cells and 19057 genes
🔗 Matching reference genes in the model
🧬 54 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 67 (unknown):
```unknown
axis_gene_wo_midmarkers
```

Example 68 (python):
```python
for key in features.keys():
    # plot on umap
    sc.pl.embedding(adata_sc, basis='umap',
                        color=[f'celltypist_conf_score_{key}',f'celltypist_prediction_{key}'],
                   size=30, wspace=0.4, frameon=False,legend_fontsize=14)
```

Example 69 (python):
```python
genes = ['SHOX2',
    'TBX3','TBX18',
         'GJA1', # CX43
         'GJA5', # CX40
         'NKX2-5','NPPA']
genes = [x for x in genes if x in adata_sc.var_names]
sc.pl.dotplot(adata_sc,
              var_names=genes,
              groupby=f'leiden_{str(leiden_res_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 70 (python):
```python
# DEG analysis
print(adata_sc.X.data[:5])
adata_sc.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata_sc,groupby=f'leiden_{str(leiden_res_sel)}')
sc.tl.dendrogram(adata_sc, groupby=f'leiden_{str(leiden_res_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata_sc, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 71 (unknown):
```unknown
[2.1097739 1.2283119 1.2283119 2.3666706 1.2283119]
```

Example 72 (unknown):
```unknown
leiden_res_sel
```

Example 73 (python):
```python
sc.pl.embedding(adata_sc, basis='umap',
                        color=[f'leiden_{str(leiden_res_sel)}'],legend_loc='on data',
                   size=30, wspace=0.4, frameon=False,legend_fontsize=14)
```

Example 74 (python):
```python
# annotate
adata_sc.obs['SANPC_subpopulations'] = adata_sc.obs[f'leiden_{str(leiden_res_sel)}'].copy()
adata_sc.obs.replace({'SANPC_subpopulations':{
    '0':'SinoatrialNodePacemakerCellsHead',
    '1':'SinoatrialNodePacemakerCellsTail',
    '2':'SinoatrialNodeTransitionalCells',
    '3':'SinoatrialNodePacemakerCellsTail',
    '4':'SinoatrialNodeTransitionalCells',
    '5':'SinoatrialNodePacemakerCellsHead',
    '6':'SinoatrialNodePacemakerCellsHorn',
    '7':'SinoatrialNodeTransitionalCells',
    '8':'SinoatrialNodePacemakerCellsTail',
    '9':'SinoatrialNodePacemakerCellsHead',
    '10':'SinoatrialNodePacemakerCellsCycling',
    '11':'SinoatrialNodePacemakerCellsHead'
}},inplace=True)
adata_sc.obs['SANPC_subpopulations'].value_counts()
```

Example 75 (unknown):
```unknown
SANPC_subpopulations
SinoatrialNodePacemakerCellsHead       289
SinoatrialNodePacemakerCellsTail       226
SinoatrialNodeTransitionalCells        209
SinoatrialNodePacemakerCellsHorn        65
SinoatrialNodePacemakerCellsCycling     44
Name: count, dtype: int64
```

Example 76 (python):
```python
sc.pl.embedding(adata_sc, basis='umap',
                        color=['SANPC_subpopulations'],
                   size=30, wspace=0.4, frameon=False,legend_fontsize=14)
```

Example 77 (python):
```python
for key in features.keys():
    print(key)
    predictions = celltypist.annotate(adata_sc, 
                                      model = f'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/celltypist/model_HD-SANPC_{key}.pkl', 
                                      majority_voting = False)
    # plot
    celltypist.dotplot(predictions, use_as_reference = 'SANPC_subpopulations', use_as_prediction = 'predicted_labels', vmax=1, show=False)
```

Example 78 (unknown):
```unknown
⚠️ Warning: invalid expression matrix, expect ALL genes and log1p normalized expression to 10000 counts per cell. The prediction result may not be accurate
🔬 Input data has 833 cells and 19057 genes
🔗 Matching reference genes in the model
```

Example 79 (unknown):
```unknown
🧬 13037 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
⚠️ Warning: invalid expression matrix, expect ALL genes and log1p normalized expression to 10000 counts per cell. The prediction result may not be accurate
🔬 Input data has 833 cells and 19057 genes
🔗 Matching reference genes in the model
```

Example 80 (unknown):
```unknown
🧬 3177 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
⚠️ Warning: invalid expression matrix, expect ALL genes and log1p normalized expression to 10000 counts per cell. The prediction result may not be accurate
🔬 Input data has 833 cells and 19057 genes
🔗 Matching reference genes in the model
🧬 101 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 81 (unknown):
```unknown
axis_gene
axis_gene_wo_midmarkers
```

Example 82 (unknown):
```unknown
⚠️ Warning: invalid expression matrix, expect ALL genes and log1p normalized expression to 10000 counts per cell. The prediction result may not be accurate
🔬 Input data has 833 cells and 19057 genes
🔗 Matching reference genes in the model
🧬 54 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 83 (python):
```python
# remove cycling clusters
adata_sc_sel = adata_sc[adata_sc.obs['SANPC_subpopulations'].isin(['SinoatrialNodePacemakerCellsCycling'])==False]
adata_sc_sel.obs['SANPC_subpopulations'].value_counts()
```

Example 84 (unknown):
```unknown
SANPC_subpopulations
SinoatrialNodePacemakerCellsHead    289
SinoatrialNodePacemakerCellsTail    226
SinoatrialNodeTransitionalCells     209
SinoatrialNodePacemakerCellsHorn     65
Name: count, dtype: int64
```

Example 85 (python):
```python
# markers for assessing
genes = ['SHOX2',
    'TBX3','TBX18',
         'GJA1', # CX43
         'GJA5', # CX40
         'NKX2-5',# 'NPPA'
        ]
genes = [x for x in genes if x in adata_sc.var_names]
```

Example 86 (python):
```python
# clustering based annotation
sc.pl.dotplot(adata_sc_sel,
                  var_names=genes,
                  groupby='SANPC_subpopulations',
                  standard_scale='var',
                  dendrogram=True,
                  title=key
                 )
```

Example 87 (python):
```python
WARNING: dendrogram data not found (using key=dendrogram_SANPC_subpopulations). Running `sc.tl.dendrogram` with default parameters. For fine tuning it is recommended to run `sc.tl.dendrogram` independently.
```

Example 88 (python):
```python
# celltypist based annotation
conf_score_thresh = 0.6

transitional_obsnames = list(adata_sc_sel.obs_names[adata_sc_sel.obs['SANPC_subpopulations']=='SinoatrialNodeTransitionalCells'])
for key in features.keys():
    ad = adata_sc_sel.copy()
    ad.obs['tmp_annotation'] = ad.obs[f'celltypist_prediction_{key}'].astype('str').copy()
    ad.obs.loc[transitional_obsnames, 'tmp_annotation'] = 'SinoatrialNodeTransitionalCells'
    # remove low confidence cells
    ad = ad[ad.obs[f'celltypist_conf_score_{key}']>conf_score_thresh]
    print(key)
    print(ad.obs['tmp_annotation'].value_counts())
    sc.pl.dotplot(ad,
                  var_names=genes,
                  groupby='tmp_annotation',
                  standard_scale='var',
                  dendrogram=True,
                 )
```

Example 89 (unknown):
```unknown
all
tmp_annotation
SinoatrialNodePacemakerCellsHead    405
SinoatrialNodeTransitionalCells     183
SinoatrialNodePacemakerCellsTail     93
SinoatrialNodePacemakerCellsHorn     52
Name: count, dtype: int64
```

Example 90 (unknown):
```unknown
... storing 'tmp_annotation' as categorical
```

Example 91 (unknown):
```unknown
... storing 'tmp_annotation' as categorical
```

Example 92 (unknown):
```unknown
hvg
tmp_annotation
SinoatrialNodePacemakerCellsHead    296
SinoatrialNodeTransitionalCells     130
SinoatrialNodePacemakerCellsTail    117
SinoatrialNodePacemakerCellsHorn     23
Name: count, dtype: int64
```

Example 93 (unknown):
```unknown
... storing 'tmp_annotation' as categorical
```

Example 94 (unknown):
```unknown
axis_gene
tmp_annotation
SinoatrialNodePacemakerCellsHead    302
SinoatrialNodePacemakerCellsHorn    100
SinoatrialNodeTransitionalCells      85
SinoatrialNodePacemakerCellsTail     29
Name: count, dtype: int64
```

Example 95 (unknown):
```unknown
... storing 'tmp_annotation' as categorical
```

Example 96 (unknown):
```unknown
axis_gene_wo_midmarkers
tmp_annotation
SinoatrialNodePacemakerCellsHead    144
SinoatrialNodePacemakerCellsTail    115
SinoatrialNodeTransitionalCells      95
SinoatrialNodePacemakerCellsHorn     80
Name: count, dtype: int64
```

Example 97 (unknown):
```unknown
conf_score_thresh = 0.6
key = 'all'
```

Example 98 (python):
```python
transitional_obsnames = list(adata_sc.obs_names[adata_sc.obs['SANPC_subpopulations']=='SinoatrialNodeTransitionalCells'])
cyclingcell_obsnames = list(adata_sc.obs_names[adata_sc.obs['SANPC_subpopulations']=='SinoatrialNodePacemakerCellsCycling'])
low_conf_obsnames = list(adata_sc.obs_names[adata_sc.obs[f'celltypist_conf_score_{key}']<conf_score_thresh])

adata_sc.obs['SANPC_subpopulations'] = adata_sc.obs[f'celltypist_prediction_{key}'].astype('str').copy()
adata_sc.obs.loc[low_conf_obsnames,'SANPC_subpopulations'] = 'SinoatrialNodePacemakerCells_celltypist-low-confidence'
adata_sc.obs.loc[transitional_obsnames,'SANPC_subpopulations'] = 'SinoatrialNodeTransitionalCells'
adata_sc.obs.loc[cyclingcell_obsnames,'SANPC_subpopulations'] = 'SinoatrialNodePacemakerCellsCycling'
adata_sc.obs['SANPC_subpopulations'].value_counts()
```

Example 99 (unknown):
```unknown
SANPC_subpopulations
SinoatrialNodePacemakerCellsHead                          405
SinoatrialNodeTransitionalCells                           209
SinoatrialNodePacemakerCellsTail                           93
SinoatrialNodePacemakerCellsHorn                           52
SinoatrialNodePacemakerCellsCycling                        44
SinoatrialNodePacemakerCells_celltypist-low-confidence     30
Name: count, dtype: int64
```

Example 100 (python):
```python
sc.pl.embedding(adata_sc, basis='umap',
                        color=['SANPC_subpopulations'],
                   size=30, wspace=0.4, frameon=False,legend_fontsize=14)
```

Example 101 (unknown):
```unknown
... storing 'SANPC_subpopulations' as categorical
```

Example 102 (python):
```python
# markers for assessing
genes = ['SHOX2',
    'TBX3','TBX18',
         'GJA1', # CX43
         'GJA5', # CX40
         'NKX2-5',# 'NPPA'
         'MKI67','TOP2A'
        ]
genes = [x for x in genes if x in adata_sc.var_names]

sc.pl.dotplot(adata_sc,
                  var_names=genes,
                  groupby='SANPC_subpopulations',
                  standard_scale='var',
                  dendrogram=True,
                  title=key
                 )
```

Example 103 (python):
```python
# without cycling and transitional cells
genes = ['SHOX2',
    'TBX3','TBX18',
         'GJA1', # CX43
         'GJA5', # CX40
         'NKX2-5',# 'NPPA'
        ]
genes = [x for x in genes if x in adata_sc.var_names]

sc.pl.dotplot(adata_sc[adata_sc.obs['SANPC_subpopulations'].isin(['SinoatrialNodePacemakerCellsCycling','SinoatrialNodeTransitionalCells','SinoatrialNodePacemakerCells_celltypist-low-confidence'])==False],
                  var_names=genes,
                  groupby='SANPC_subpopulations',
                  standard_scale='var',
                  dendrogram=False,
                  title=key
                 )
```

Example 104 (python):
```python
# read in original raw data again
adata_sc_all = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad')
```

Example 105 (python):
```python
# subset multiome and sanpc data
adata_sc_sanpc_multiome = adata_sc_all[adata_sc_all.obs['kit_10x']=='Multiome-v1']
adata_sc_sanpc_multiome = adata_sc_sanpc_multiome[adata_sc_sanpc_multiome.obs['fine_grain']=='SinoatrialNodePacemakerCells']
adata_sc_sanpc_multiome
```

Example 106 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 833 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'region', 'age', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'donor_by_library-prep', 'multiplexed', 'SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet', 'batch_key', '_scvi_batch', 'FACSgate', 'fine_grain', 'mid_grain', 'coarse_grain', 'sex', 'week', 'trimester', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', '_scvi_labels', 'stress_score', 'hb1_score'
    var: 'gene_name_scRNA-0', 'gene_id'
    uns: 'FACSgate_colors', '_scvi_manager_uuid', '_scvi_uuid', 'age_colors', 'cell_or_nuclei_colors', 'coarse_grain_colors', 'cycling_colors', 'donor_colors', 'fine_grain_colors', 'heart_or_greatvessels_colors', 'hvg', 'kit_10x_colors', 'modality_colors', 'neighbors', 'phase_colors', 'region_colors', 'sex_colors', 'trimester_colors', 'umap', 'week_colors'
    obsm: 'X_scVI', 'X_umap', '_scvi_extra_categorical_covs', '_scvi_extra_continuous', '_scvi_extra_continuous_covs', 'latent_gene_encoding'
    obsp: 'connectivities', 'distances'
```

Example 107 (python):
```python
# copy the labels
adata_sc_sanpc_multiome.obs['SANPC_subpopulations'] = adata_sc.obs['SANPC_subpopulations'].reindex(adata_sc_sanpc_multiome.obs_names)
adata_sc_sanpc_multiome.obs['SANPC_subpopulations'].value_counts()
```

Example 108 (unknown):
```unknown
SANPC_subpopulations
SinoatrialNodePacemakerCellsHead                          405
SinoatrialNodeTransitionalCells                           209
SinoatrialNodePacemakerCellsTail                           93
SinoatrialNodePacemakerCellsHorn                           52
SinoatrialNodePacemakerCellsCycling                        44
SinoatrialNodePacemakerCells_celltypist-low-confidence     30
Name: count, dtype: int64
```

Example 109 (python):
```python
adata_sc_sanpc_multiome.write('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated_SANPC.h5ad')
print(adata_sc_sanpc_multiome.X.data[:5])
```

Example 110 (unknown):
```unknown
[3. 1. 1. 4. 1.]
```

Example 111 (python):
```python
adata_sc_sanpc_multiome
```

Example 112 (unknown):
```unknown
AnnData object with n_obs × n_vars = 833 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'region', 'age', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'donor_by_library-prep', 'multiplexed', 'SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet', 'batch_key', '_scvi_batch', 'FACSgate', 'fine_grain', 'mid_grain', 'coarse_grain', 'sex', 'week', 'trimester', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', '_scvi_labels', 'stress_score', 'hb1_score', 'SANPC_subpopulations'
    var: 'gene_name_scRNA-0', 'gene_id'
    uns: 'FACSgate_colors', '_scvi_manager_uuid', '_scvi_uuid', 'age_colors', 'cell_or_nuclei_colors', 'coarse_grain_colors', 'cycling_colors', 'donor_colors', 'fine_grain_colors', 'heart_or_greatvessels_colors', 'hvg', 'kit_10x_colors', 'modality_colors', 'neighbors', 'phase_colors', 'region_colors', 'sex_colors', 'trimester_colors', 'umap', 'week_colors'
    obsm: 'X_scVI', 'X_umap', '_scvi_extra_categorical_covs', '_scvi_extra_continuous', '_scvi_extra_continuous_covs', 'latent_gene_encoding'
    obsp: 'connectivities', 'distances'
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no7_manual-annotation_EndocardialCells.html

**Contents:**
- Variables¶
- Read in data¶
- Remove mid-grain markers from the feature¶
- PCA, Clustering, UMAP¶
- Markers¶
- Subset "uncertain" cells¶
- Add the annotation to the original adata¶
- Save¶

2385 rows × 7 columns

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import celltypist
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 4 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 5 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets'
sample_id = 'HEA_FOET14880396'
celltype = 'EndocardialCells'
```

Example 6 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 7 (python):
```python
# read in
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 1526 × 14451
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
# read in
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0
                              )
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group']!=celltype]
midgrain_markers
```

Example 10 (python):
```python
# remove
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]
bdata
```

Example 11 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 1526 × 13211
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 12 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata,min_dist=0.5)
sc.pl.embedding(adata,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 13 (unknown):
```unknown
number of hvgs: 3336
```

Example 14 (python):
```python
# read in markers identified using single-nucleus data
markers_df = pd.read_csv(f'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_{celltype}_finegrain_markers.csv')

mask1 = markers_df['group_mean']>1
mask2 = markers_df['logfoldchanges']>1
markers_df = markers_df[mask1&mask2]

markers_df
```

Example 15 (python):
```python
markers = {}
n_top = 10
for c in set(markers_df['group']):
    markers_df_sub = markers_df[markers_df['group']==c]
    markers_df_sub.sort_values('logfoldchanges',ascending=False)
    markers[c]=list(markers_df_sub[:n_top]['names'])

# manually add genes
markers['Cycling'] = ['MKI67','TOP2A'] 

genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in list(adata.var_names))&(x not in genes)]
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in list(adata.var_names)]
len(genes)
```

Example 16 (python):
```python
sc.pl.embedding(adata,basis='X_umap',color=genes, size=20,
               ncols=6, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 17 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=['celltypist_midmod2fine', 'conf_score_midmod2fine'],
                wspace=0.7)
```

Example 18 (python):
```python
# umap per celltype to highlight
group_col = 'celltypist_midmod2fine'
groups_to_plot = list(set(adata.obs[group_col]))

plt.rcParams['axes.grid'] = False
fig,axs=plt.subplots(nrows=2,ncols=4,figsize=(15,7))
# Flatten the axs array for easier iteration in case of multiple rows and columns
axs_flat = axs.flatten()

for i,grp in enumerate(groups_to_plot):
    sc.pl.umap(adata, color=[group_col], groups=grp, 
               size=30, frameon=False,
               show=False, ax=axs_flat[i], legend_loc=None, legend_fontsize=5,
               title=grp
              )
# Turn off any remaining subplots
for j in range(len(groups_to_plot), len(axs_flat)):
    fig.delaxes(axs_flat[j])

# fig.suptitle('Main Title for the Figure')
fig.tight_layout()
```

Example 19 (unknown):
```unknown
resolution_sel = 1
```

Example 20 (python):
```python
markers_others = {
    'CM': ['TTN', 'TNNT2','MYH6',
        'MYH7', 'MYL2', 'FHL2', # vCN
            'NPPA', 'MYL7', 'MYL4'], #aCM
    'FB': ['DCN', 'GCN', 'PDGFRA','COL1A1','COL1A2'],
    'EC': ['VWF', 'PECAM1', 'CDH5','RGCC', 'FABP5',],
    'Peri': ['RGS5', 'ABCC9', 'KCNJ8'],
    'SMC': ['MYH11', 'TAGLN', 'ACTA2'],
    'Meso': ['MSLN', 'WT1', 'BNC1'],
    'Neuro': ['PLP1', 'NRXN1', 'NRXN3','PRPH', 'NEFL', 'NEFM', 'NEFH', 'STMN2', 'SST', 'SLC10A4', 'SLC18A3', 'TH', 'NPY'],
    'Adipo': ['GPAM', 'FASN', 'LEP'],
    'Myelo': ['CD14', 'C1QA', 'CD68','LYVE1','TIMD4'],
    'Lymph': ['CD8A', 'IL7R', 'CD40LG'],
    'B': ['CD19','SDC1','IGHG1','IGHA1'],
    'Mast': ['KIT', 'CPA3'],
}

for key in markers_others.keys():
    markers_others[key]=[x for x in markers_others[key] if x in adata.var_names]

sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              markers_others, 
              groupby=f'leiden_{str(resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 21 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CM, FB, EC, etc.
```

Example 22 (python):
```python
sc.tl.dendrogram(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby=f'leiden_{str(resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 23 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: EndocardialCushionCells, ValveEndothelialCells, EndocardialCells, etc.
```

Example 24 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 25 (unknown):
```unknown
[1.84898172 2.52192218 2.53797117 1.50684599 2.05779654]
```

Example 26 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True, n_genes=5)
dp.add_totals().show()
```

Example 27 (unknown):
```unknown
[1.84898172 2.52192218 2.53797117 1.50684599 2.05779654]
```

Example 28 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=f'leiden_{str(resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 29 (python):
```python
sc.pl.violin(adata,keys=['total_counts','bin_count'],groupby=f'leiden_{str(resolution_sel)}')
```

Example 30 (unknown):
```unknown
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 31 (python):
```python
# annotate
adata.obs['fine_grain_manual'] = adata.obs[f'leiden_{str(resolution_sel)}'].copy()
adata.obs.replace({'fine_grain_manual':{
    '0':'EndocardialCells',
    '1':'ValveEndothelialCells',
    '2':'uncertain',
    '3':'ValveEndothelialCells',
    '4':'EndocardialCushionCells', # contamination from valveECs
    '5':'EndocardialCushionCells',
    '6':'Erythrocytes',
}},inplace=True)
adata.obs['fine_grain_manual'].value_counts()
```

Example 32 (unknown):
```unknown
fine_grain_manual
ValveEndothelialCells      585
EndocardialCells           349
uncertain                  297
EndocardialCushionCells    284
Erythrocytes                11
Name: count, dtype: int64
```

Example 33 (python):
```python
ad = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad',backed='r')
ad = ad[ad.obs['mid_grain']==celltype]
set(ad.obs['fine_grain'])
```

Example 34 (unknown):
```unknown
{'EndocardialCells', 'EndocardialCushionCells', 'ValveEndothelialCells'}
```

Example 35 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 36 (python):
```python
adata_sub = adata[adata.obs['fine_grain_manual'].isin(['uncertain'])]

# drop leiden clustering results
resolutions_list = [0.2,0.4,0.6,1, 1.2,1.5,2]
for resolution in resolutions_list:
    adata_sub.obs.drop(f'leiden_{str(resolution)}',axis=1,inplace=True)
adata_sub
```

Example 37 (unknown):
```unknown
AnnData object with n_obs × n_vars = 297 × 14451
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'log1p', 'neighbors', 'pca', 'spatial', 'umap', 'dendrogram_leiden_1', 'rank_genes_groups', 'dendrogram_leiden_2', 'dendrogram_leiden_1.5', 'fine_grain_manual_colors'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 38 (python):
```python
# remove mid_grain markers
mask = [x not in list(midgrain_markers['names']) for x in adata_sub.var_names]
bdata = adata_sub[:,mask]
bdata
```

Example 39 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 297 × 13211
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'log1p', 'neighbors', 'pca', 'spatial', 'umap', 'dendrogram_leiden_1', 'rank_genes_groups', 'dendrogram_leiden_2', 'dendrogram_leiden_1.5', 'fine_grain_manual_colors'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 40 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata_sub.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata_sub,n_neighbors=15,use_rep='X_pca')
```

Example 41 (unknown):
```unknown
number of hvgs: 3218
```

Example 42 (python):
```python
sc.tl.umap(adata_sub,min_dist=0.1)
```

Example 43 (python):
```python
%%time
# reclustering
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1, 1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata_sub,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata_sub,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 44 (unknown):
```unknown
CPU times: user 1.44 s, sys: 39.8 ms, total: 1.48 s
Wall time: 1.53 s
```

Example 45 (python):
```python
sc.pl.embedding(adata_sub,basis='X_umap',color=genes, size=10,
               ncols=6, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 46 (python):
```python
# umap per celltype to highlight
group_col = 'celltypist_midmod2fine'
groups_to_plot = list(set(adata_sub.obs[group_col]))

plt.rcParams['axes.grid'] = False
fig,axs=plt.subplots(nrows=2,ncols=4,figsize=(15,7))
# Flatten the axs array for easier iteration in case of multiple rows and columns
axs_flat = axs.flatten()

for i,grp in enumerate(groups_to_plot):
    sc.pl.umap(adata_sub, color=[group_col], groups=grp, 
               size=30, frameon=False,
               show=False, ax=axs_flat[i], legend_loc=None, legend_fontsize=5,
               title=grp
              )
# Turn off any remaining subplots
for j in range(len(groups_to_plot), len(axs_flat)):
    fig.delaxes(axs_flat[j])

# fig.suptitle('Main Title for the Figure')
fig.tight_layout()
```

Example 47 (python):
```python
adata_sub_resolution_sel = 0.6
```

Example 48 (python):
```python
markers_sub = {k:v for k,v in markers.items() if k in ['EndocardialCushionCells','EndocardialCells']}

sc.tl.dendrogram(adata_sub,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.pl.dotplot(adata_sub,
              var_names=markers_sub,
              groupby=f'leiden_{str(adata_sub_resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 49 (unknown):
```unknown
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 50 (python):
```python
sc.pl.embedding(adata_sub,basis='X_umap',
                color=f'leiden_{str(adata_sub_resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 51 (python):
```python
# annotate
adata_sub.obs['fine_grain_manual'] = adata_sub.obs[f'leiden_{str(adata_sub_resolution_sel)}'].copy()
adata_sub.obs.replace({'fine_grain_manual':{
    '0':'EndocardialCushionCells', 
    '1':'EndocardialCells',
}},inplace=True)
adata_sub.obs['fine_grain_manual'].value_counts()
```

Example 52 (unknown):
```unknown
fine_grain_manual
EndocardialCushionCells    247
EndocardialCells            50
Name: count, dtype: int64
```

Example 53 (python):
```python
sc.pl.embedding(adata_sub,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 54 (python):
```python
markers_sub = {k:v for k,v in markers.items() if k in ['EndocardialCushionCells','EndocardialCells']}

sc.tl.dendrogram(adata_sub,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.pl.dotplot(adata_sub,
              var_names=markers_sub,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 55 (unknown):
```unknown
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 56 (python):
```python
# add the annotation to the original adata
adata.obs['fine_grain_manual'] = adata.obs['fine_grain_manual'].astype('str')
adata.obs.loc[adata_sub.obs_names,'fine_grain_manual'] = adata_sub.obs['fine_grain_manual'].astype('str').copy()
adata.obs['fine_grain_manual'].value_counts()
```

Example 57 (unknown):
```unknown
fine_grain_manual
ValveEndothelialCells      585
EndocardialCushionCells    531
EndocardialCells           399
Erythrocytes                11
Name: count, dtype: int64
```

Example 58 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 59 (unknown):
```unknown
... storing 'fine_grain_manual' as categorical
```

Example 60 (python):
```python
sc.tl.dendrogram(adata,groupby='fine_grain_manual')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 61 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: EndocardialCells, EndocardialCushionCells, Erythrocytes, etc.
var_group_labels: EndocardialCushionCells, ValveEndothelialCells, EndocardialCells, etc.
```

Example 62 (python):
```python
# save
adata.write(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
print(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
```

Example 63 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets/HEA_FOET14880396_EndocardialCells_b2c_cells_filtered_lognorm.h5ad
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig3_SAN_niche/cellphonedb-VisiumHD_no0_download_DB_v5.html

**Contents:**
- Display database versions¶
- Define the version and the path to download database¶
- Download¶

Ref: https://github.com/ventolab/CellphoneDB/blob/master/notebooks/T00_DownloadDB.ipynb

**Examples:**

Example 1 (python):
```python
import pandas as pd
import glob
import os
```

Example 2 (python):
```python
from IPython.display import HTML, display
from cellphonedb.utils import db_releases_utils

display(HTML(db_releases_utils.get_remote_database_versions_html()['db_releases_html_table']))
```

Example 3 (unknown):
```unknown
# -- Version of the databse
cpdb_version = 'v5.0.0'

# -- Path where the input files to generate the database are located
cpdb_target_dir = os.path.join('/nfs/team205/heart/cellphonedb/db', cpdb_version)
```

Example 4 (python):
```python
from cellphonedb.utils import db_utils

db_utils.download_database(cpdb_target_dir, cpdb_version)
```

Example 5 (unknown):
```unknown
Downloaded cellphonedb.zip into /nfs/team205/heart/cellphonedb/db/v5.0.0
Downloaded complex_input.csv into /nfs/team205/heart/cellphonedb/db/v5.0.0
Downloaded gene_input.csv into /nfs/team205/heart/cellphonedb/db/v5.0.0
Downloaded interaction_input.csv into /nfs/team205/heart/cellphonedb/db/v5.0.0
Downloaded protein_input.csv into /nfs/team205/heart/cellphonedb/db/v5.0.0
Downloaded uniprot_synonyms.tsv into /nfs/team205/heart/cellphonedb/db/v5.0.0/sources
Downloaded transcription_factor_input.csv into /nfs/team205/heart/cellphonedb/db/v5.0.0/sources
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig3_SAN_niche/cellphonedb-VisiumHD_no3_plotting_SAN_to-ParaN.html

**Contents:**
- Import modules¶
- Variables¶
- Read in input adata¶
- Read in cpdb outputs¶
- Plot¶
  - Receptor expressed in PSNS¶
- Slect interactions specific to Sinus Horn and SAN¶

**Examples:**

Example 1 (python):
```python
import scanpy as sc
import pandas as pd
pd.set_option('display.max_rows', 100)
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
import seaborn as sns

import ktplotspy as kpy

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import os 
from matplotlib.patches import Rectangle
import itertools
```

Example 2 (python):
```python
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 3 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 4 (python):
```python
niche_of_interest = 'SAnode'
path_to_cpdb = f'/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/{niche_of_interest}'
path_to_adata = f'/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/inputs/log_norm_counts_{niche_of_interest}.h5ad'
celltype_col = 'fine_grain_4Jan2025'
```

Example 5 (python):
```python
# read in adata
adata = sc.read_h5ad(path_to_adata)
adata
```

Example 6 (unknown):
```unknown
AnnData object with n_obs × n_vars = 1289 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_4Jan2025', 'HistologicalAnnotation', 'cluster_cellcharter_k-8', 'cluster_cellcharter_k-10', 'cluster_cellcharter_k-12', 'cluster_cellcharter_k-14', 'cluster_cellcharter_k-16', 'cluster_cellcharter_k-18', 'cluster_cellcharter_k-20', 'cluster_cellcharter_k-25', 'cluster_cellcharter_k-30', 'cluster_cellcharter_k-35', 'cluster_cellcharter_k-40', 'cluster_cellcharter_k-45', 'cluster_cellcharter_k-50', 'predicted_labels', 'conf_score', 'predicted_labels_cellcharter-cluster-k50', 'conf_score_cellcharter-cluster-k50', 'spatial_leiden_1', 'spatial_leiden_5', 'spatial_leiden_0.5', 'spatial_leiden_2', 'spatial_leiden_10', 'spatial_leiden_100', 'spatial_leiden_1000', 'KMeans_k9216_cluster', 'predicted_labels_spatial_KMeans_k9216', 'conf_score_spatial_KMeans_k9216', 'KMeans_k4608_cluster', 'predicted_labels_spatial_KMeans_k4608', 'conf_score_spatial_KMeans_k4608', 'KMeans_k3072_cluster', 'KMeans_k1843_cluster', 'predicted_labels_spatial_KMeans_k3072', 'conf_score_spatial_KMeans_k3072', 'predicted_labels_spatial_KMeans_k1843', 'conf_score_spatial_KMeans_k1843', 'sliding_window_assignment', 'predicted_labels_sliding_window_assignment', 'conf_score_sliding_window_assignment', 'tt_prediction'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'HistologicalAnnotation_colors', 'KMeans_k1843_centroids', 'KMeans_k3072_centroids', 'KMeans_k4608_centroids', 'KMeans_k9216_centroids', 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'cluster_cellcharter_k-10_colors', 'cluster_cellcharter_k-12_colors', 'cluster_cellcharter_k-14_colors', 'cluster_cellcharter_k-16_colors', 'cluster_cellcharter_k-18_colors', 'cluster_cellcharter_k-20_colors', 'cluster_cellcharter_k-25_colors', 'cluster_cellcharter_k-30_colors', 'cluster_cellcharter_k-35_colors', 'cluster_cellcharter_k-40_colors', 'cluster_cellcharter_k-45_colors', 'cluster_cellcharter_k-50_colors', 'cluster_cellcharter_k-8_colors', 'conf_score_cellcharter-cluster-k50_colors', 'conf_score_sliding_window_assignment_colors', 'fine_grain_4Jan2025_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'predicted_labels_cellcharter-cluster-k50_colors', 'predicted_labels_colors', 'predicted_labels_spatial_KMeans_k1843_colors', 'predicted_labels_spatial_KMeans_k3072_colors', 'predicted_labels_spatial_KMeans_k4608_colors', 'predicted_labels_spatial_KMeans_k9216_colors', 'spatial', 'spatial_leiden_0.5', 'spatial_leiden_1', 'spatial_leiden_10', 'spatial_leiden_100', 'spatial_leiden_1000', 'spatial_leiden_1_colors', 'spatial_leiden_2', 'spatial_leiden_5', 'spatial_leiden_5_colors', 'spatial_neighbors', 'tt_prediction_colors', 'umap'
    obsm: 'X_cellcharter', 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances', 'spatial_connectivities', 'spatial_distances'
```

Example 7 (unknown):
```unknown
timestamp = '05_04_2025_105025'
timestamp
```

Example 8 (unknown):
```unknown
'05_04_2025_105025'
```

Example 9 (python):
```python
# read in
cpdbout_means =  pd.read_table(f'{path_to_cpdb}/statistical_analysis_means_{timestamp}.txt',index_col=0)
cpdbout_pvals =  pd.read_table(f'{path_to_cpdb}/statistical_analysis_pvalues_{timestamp}.txt',index_col=0)
cpdbout_sigmeans =  pd.read_table(f'{path_to_cpdb}/statistical_analysis_significant_means_{timestamp}.txt',index_col=0)
interaction_metadata = pd.read_csv('/nfs/team205/heart/cellphonedb/customDB/interaction_meta/interaction_metadata_v5.0.0_2024-1-26.csv')
```

Example 10 (python):
```python
# add gene names to cpdb outputs
cpdbout_means['gene_name_1'] = list(interaction_metadata.set_index('id_cp_interaction').loc[cpdbout_means.index,'gene_name_1'])
cpdbout_means['gene_name_2'] = list(interaction_metadata.set_index('id_cp_interaction').loc[cpdbout_means.index,'gene_name_2'])
cpdbout_pvals['gene_name_1'] = list(interaction_metadata.set_index('id_cp_interaction').loc[cpdbout_pvals.index,'gene_name_1'])
cpdbout_pvals['gene_name_2'] = list(interaction_metadata.set_index('id_cp_interaction').loc[cpdbout_pvals.index,'gene_name_2'])
cpdbout_sigmeans['gene_name_1'] = list(interaction_metadata.set_index('id_cp_interaction').loc[cpdbout_sigmeans.index,'gene_name_1'])
cpdbout_sigmeans['gene_name_2'] = list(interaction_metadata.set_index('id_cp_interaction').loc[cpdbout_sigmeans.index,'gene_name_2'])
```

Example 11 (python):
```python
# insert gene_name columns to the cpdb output
# to select interactions based on the list
cpdbout_means[['gene_name_1','gene_name_2']] = interaction_metadata.set_index('id_cp_interaction').loc[cpdbout_means.index,['gene_name_1','gene_name_2']]
cpdbout_pvals[['gene_name_1','gene_name_2']] = interaction_metadata.set_index('id_cp_interaction').loc[cpdbout_pvals.index,['gene_name_1','gene_name_2']]
cpdbout_sigmeans[['gene_name_1','gene_name_2']] = interaction_metadata.set_index('id_cp_interaction').loc[cpdbout_sigmeans.index,['gene_name_1','gene_name_2']]
```

Example 12 (python):
```python
# query a gene, to check of the existence of the its interactions in the database
gene = 'FGF10'
mask1 = [gene in x.split(',') for x in interaction_metadata['gene_name_1']]
mask2 = [gene in x.split(',') for x in interaction_metadata['gene_name_2']]
mask = [any(x) for x in zip(mask1,mask2)]
interaction_metadata[mask][['id_cp_interaction','gene_name_1','gene_name_2']]
```

Example 13 (unknown):
```unknown
receptor_genes = [
    # Netrin receptors
    "DCC", "NEO1", "UNC5A", "UNC5B", "UNC5C", "UNC5D",
    # Slit (Robo) receptors
    "ROBO1", "ROBO2", "ROBO3", "ROBO4",
    # Semaphorin receptors
    "NRP1", "NRP2",
    "PLXNA1", "PLXNA2", "PLXNA3", "PLXNA4",
    "PLXNB1", "PLXNB2", "PLXNB3",
    "PLXNC1", "PLXND1",
    # Ephrin (Eph) receptors
    "EPHA1", "EPHA2", "EPHA3", "EPHA4", "EPHA5", "EPHA6", "EPHA7", "EPHA8", "EPHA10",
    "EPHB1", "EPHB2", "EPHB3", "EPHB4", "EPHB6",
    # Neurotrophin / GDNF family receptors
    "NTRK1", "NTRK2", "NTRK3", "NGFR", "RET",
    "GFRA1", "GFRA2", "GFRA3", "GFRA4",
    # Neuregulin (ErbB) receptors
    "ERBB2", "ERBB3", "ERBB4",
    # Additional innervation‑related receptors
    "EDNRB", "LRP8", "VLDLR", "L1CAM", "NCAM1",
    # Others
    "CXCR4","CNTNAP4","FGFR2"
]
receptor_celltype = 'ParaN'

# ligand_genes = ['SEMA3A', 'SLIT2', 'SLIT3', 'EFNB2', 'CXCL12', 'FGF10', 'VEGFC', 'NELL1']
ligand_celltypes = ['SHPC','SANPCHd','SANPCTl','aCMR']
```

Example 14 (python):
```python
def receiving_signal(
    cpdbout_means,
    cpdbout_pvals,
    receptor_celltype,
    receptor_genes,
    ligand_celltypes=None
):
    # Subset results
    
    ### get columns
    cellpair_cols = [x for x in cpdbout_means.columns if '|' in x]
    other_cols = [x for x in cpdbout_means.columns if '|' not in x]
    
    ### get dataframe which has receptor_celltype and the receptor_genes at the "b/2" side (majority will be like this)
    mask_row = (
        cpdbout_means['gene_name_2']
        .str.split(',')
        .apply(lambda names: any(g in receptor_genes for g in names))
    )
    if ligand_celltypes!=None:
        cellpair_cols_sel = [x for x in cellpair_cols if (f'|{receptor_celltype}' in x)&(x.split('|')[0] in ligand_celltypes)]
    else:
        cellpair_cols_sel = [x for x in cellpair_cols if (f'|{receptor_celltype}' in x)]
    col_sel = other_cols + cellpair_cols_sel
    means_2 = cpdbout_means.loc[mask_row, col_sel]
    pvals_2 = cpdbout_pvals.loc[mask_row, col_sel]

    ### get dataframe which has receptor_celltype and the receptor_genes at the "a/1" side
    mask_row = (
        cpdbout_means['gene_name_1']
        .str.split(',')
        .apply(lambda names: any(g in receptor_genes for g in names))
    )
    if ligand_celltypes!=None:
        cellpair_cols_sel = [x for x in cellpair_cols if (f'{receptor_celltype}|' in x)&(x.split('|')[1] in ligand_celltypes)]
    else:
        cellpair_cols_sel = [x for x in cellpair_cols if (f'{receptor_celltype}|' in x)]
    col_sel = other_cols + cellpair_cols_sel
    means_1 = cpdbout_means.loc[mask_row, col_sel]
    pvals_1 = cpdbout_pvals.loc[mask_row, col_sel]
    # remove rows which are already exist in means_2
    index_keep = list(set(means_1.index)-set(means_2.index))
    means_1 = means_1.loc[index_keep]
    pvals_1 = pvals_1.loc[index_keep]
    # reverse gene_a and gene_b
    mapper = {'partner_a':'partner_b',
              'partner_b':'partner_a',
              'gene_a':'gene_b',
              'gene_b':'gene_a',
              'receptor_a':'receptor_b',
              'receptor_b':'receptor_a',
              'directionality':'directionality_reverse'}
    means_1.rename(columns=mapper,inplace=True)
    pvals_1.rename(columns=mapper,inplace=True)

    new_columns = []
    for col in means_1.columns:
        if '|' in col:
            cell_a = col.split('|')[0]
            cell_b = col.split('|')[1]
            new_columns.append(f'{cell_b}|{cell_a}')
        else:
            new_columns.append(col)
    means_1.columns = new_columns
    pvals_1.columns = new_columns

    new_interacting_pair = []
    for ind in means_1.index:
        partner_a = means_1.loc[ind,'partner_a']
        partner_b = means_1.loc[ind,'partner_b']
        if 'complex:' in partner_a:
            partner_a = partner_a.replace('complex:','')
        else:
            partner_a = means_1.loc[ind,'gene_a']
        if 'complex:' in partner_b:
            partner_b = partner_b.replace('complex:','')
        else:
            partner_b = means_1.loc[ind,'gene_b']
        new_interacting_pair.append(f'{partner_a}_{partner_b}')
    means_1['interacting_pair'] = new_interacting_pair
    pvals_1['interacting_pair'] = new_interacting_pair

    ### concatenate
    means = pd.concat([means_2,means_1])
    pvals = pd.concat([pvals_2,pvals_1])

    ### remove NaN
    means = means.dropna(axis=0, how='all')
    means = means.dropna(axis=1, how='all')
    pvals = pvals.dropna(axis=0, how='all')
    pvals = pvals.dropna(axis=1, how='all')

    ### formatting
    means = means.reset_index()
    pvals = pvals.reset_index()
    means.drop(labels=['gene_name_1','gene_name_2','directionality_reverse'],axis=1,inplace=True)
    pvals.drop(labels=['gene_name_1','gene_name_2','directionality_reverse'],axis=1,inplace=True)
    
    return means, pvals
```

Example 15 (python):
```python
means,pvals = receiving_signal(
    cpdbout_means,
    cpdbout_pvals,
    receptor_celltype,
    receptor_genes,
    ligand_celltypes=ligand_celltypes
)

# reorder interaction columns
non_interaction_columns = [x for x in means.columns if '|' not in x]
interaction_columns = [f'{x}|ParaN' for x in ligand_celltypes[::-1]]
means = means[non_interaction_columns+interaction_columns]
pvals = pvals[non_interaction_columns+interaction_columns]
interaction_columns = [f'{i}_{x}|ParaN' for i,x in enumerate(ligand_celltypes[::-1])]
means.columns = non_interaction_columns+interaction_columns
pvals.columns = non_interaction_columns+interaction_columns

# rename cell types of interest for regulating the order
for i,cell in enumerate(ligand_celltypes[::-1]):
    adata.obs.replace({celltype_col:{cell:f'{i}_{cell}'}},inplace=True)

plot = kpy.plot_cpdb(
    adata=adata,
    cell_type1='.',
    cell_type2='.',  # this means all cell-types
    means=means,
    pvals=pvals,
    alpha=0.1,
    celltype_key=celltype_col,
    genes=None,
    figsize=(5.2, 9),
    title='SAN, CMs-to-ParaN,\nall AxonG',
    highlight_size =1.5,
    standard_scale=False
)
plot.save(f"{figdir}/VisiumHD_CellPhoneDB_SAN_CM-AxonG-to-ParaN_all-ligands.pdf", bbox_inches="tight",dpi=300)
plot
```

Example 16 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/cpdb/lib/python3.8/site-packages/plotnine/ggplot.py:587: PlotnineWarning: Saving 5.2 x 9 in image.
/software/cellgen/team205/kk18/envs/cpdb/lib/python3.8/site-packages/plotnine/ggplot.py:588: PlotnineWarning: Filename: /nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024/VisiumHD_CellPhoneDB_SAN_CM-AxonG-to-ParaN_all-ligands.pdf
```

Example 17 (unknown):
```unknown
<Figure Size: (520 x 900)>
```

Example 18 (unknown):
```unknown
# select interactions only significant at SHPC or SANPCs
pval_thresh = 0.1
mask = pvals['0_aCMR|ParaN'] > pval_thresh
means = means.loc[mask]
pvals = pvals.loc[mask]
```

Example 19 (python):
```python
plot = kpy.plot_cpdb(
    adata=adata,
    cell_type1='.',
    cell_type2='.',  # this means all cell-types
    means=means,
    pvals=pvals,
    alpha=0.1,
    celltype_key=celltype_col,
    genes=None,
    figsize=(5.2, 5.5),
    title='SAN, CMs-to-ParaN,\nall AxonG',
    highlight_size =1.5,
    standard_scale=False
)
plot.save(f"{figdir}/VisiumHD_CellPhoneDB_SAN_CM-AxonG-to-ParaN_selected.pdf", bbox_inches="tight",dpi=300)
plot
```

Example 20 (unknown):
```unknown
/software/cellgen/team205/kk18/envs/cpdb/lib/python3.8/site-packages/plotnine/ggplot.py:587: PlotnineWarning: Saving 5.2 x 5.5 in image.
/software/cellgen/team205/kk18/envs/cpdb/lib/python3.8/site-packages/plotnine/ggplot.py:588: PlotnineWarning: Filename: /nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024/VisiumHD_CellPhoneDB_SAN_CM-AxonG-to-ParaN_selected.pdf
```

Example 21 (unknown):
```unknown
<Figure Size: (520 x 550)>
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig2_Overview_of_cellular_niches_and_TissueTypist/tissuetypist_VisiumHD_TissueTypist-prediction_plotting.html

**Contents:**
- Import required modules¶
- Read in adata¶
- Plotting tissue labels¶
- Plotting cropped images - predicted labels¶
  - Atrium¶
  - Ventricle¶
  - Great vessel¶
- Plotting cropped images - cell types¶
  - Atrium¶
- UMAP¶

**Examples:**

Example 1 (python):
```python
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.filterwarnings('ignore')

import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 
import seaborn as sns
import os 
from mpl_toolkits.axes_grid1 import make_axes_locatable

import celltypist
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/tissueaxis.py")
tissueaxis = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = tissueaxis
spec.loader.exec_module(tissueaxis)
```

Example 4 (python):
```python
sc.settings.verbosity = 1          # verbosity: errors (0), warnings (1), info (2), hints (3)
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 5 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 6 (unknown):
```unknown
os.getcwd()
```

Example 7 (unknown):
```unknown
'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3'
```

Example 8 (unknown):
```unknown
# set palette and categorie order
AnnotationPalette={ 
    'Atrium':'#eac075',
        'Ventricle - Compact':'#f1acfa',
     'Ventricle - Trabeculated':'#f09951',
        'SAnode':'#8A2BE2',
        'AVnode':'#062b4a',
    'VCS - Proximal':'#FFFF00',
        'VCS - Distal':'#7FFFD4',
        'Epicardium':'#01ccc8',
        'Endocardium - Atrial':'#DC143C',
        'Endocardium - Ventricular':'#f53527',
        'Endocardial cusion':'#ff7e97',
        'Great vessel':'#ff9b6e',
    'Ductus arteriosus':'#8a6800',
        'Coronary vessel':'#206601',
        'Adventitia - Coronary':'#CAFF70',
     'Adventitia - GV':'#005aa0',
        'AV ring':'#e478ff',
    'Valves':'#3300FF',
    'LN':'#9088ff',
}
```

Example 9 (python):
```python
sample_id = 'HEA_FOET14880396'
path_adata = f'/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/{sample_id}_b2c_cells_filtered_celltype-selected_lognorm.h5ad'
```

Example 10 (python):
```python
adata = sc.read_h5ad(path_adata)
adata
```

Example 11 (unknown):
```unknown
AnnData object with n_obs × n_vars = 92164 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_4Jan2025', 'HistologicalAnnotation', 'cluster_cellcharter_k-8', 'cluster_cellcharter_k-10', 'cluster_cellcharter_k-12', 'cluster_cellcharter_k-14', 'cluster_cellcharter_k-16', 'cluster_cellcharter_k-18', 'cluster_cellcharter_k-20', 'cluster_cellcharter_k-25', 'cluster_cellcharter_k-30', 'cluster_cellcharter_k-35', 'cluster_cellcharter_k-40', 'cluster_cellcharter_k-45', 'cluster_cellcharter_k-50', 'predicted_labels', 'conf_score', 'predicted_labels_cellcharter-cluster-k50', 'conf_score_cellcharter-cluster-k50', 'spatial_leiden_1', 'spatial_leiden_5', 'spatial_leiden_0.5', 'spatial_leiden_2', 'spatial_leiden_10', 'spatial_leiden_100', 'spatial_leiden_1000', 'KMeans_k9216_cluster', 'predicted_labels_spatial_KMeans_k9216', 'conf_score_spatial_KMeans_k9216', 'KMeans_k4608_cluster', 'predicted_labels_spatial_KMeans_k4608', 'conf_score_spatial_KMeans_k4608', 'KMeans_k3072_cluster', 'KMeans_k1843_cluster', 'predicted_labels_spatial_KMeans_k3072', 'conf_score_spatial_KMeans_k3072', 'predicted_labels_spatial_KMeans_k1843', 'conf_score_spatial_KMeans_k1843', 'sliding_window_assignment', 'predicted_labels_sliding_window_assignment', 'conf_score_sliding_window_assignment', 'tt_prediction'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'HistologicalAnnotation_colors', 'KMeans_k1843_centroids', 'KMeans_k3072_centroids', 'KMeans_k4608_centroids', 'KMeans_k9216_centroids', 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'cluster_cellcharter_k-10_colors', 'cluster_cellcharter_k-12_colors', 'cluster_cellcharter_k-14_colors', 'cluster_cellcharter_k-16_colors', 'cluster_cellcharter_k-18_colors', 'cluster_cellcharter_k-20_colors', 'cluster_cellcharter_k-25_colors', 'cluster_cellcharter_k-30_colors', 'cluster_cellcharter_k-35_colors', 'cluster_cellcharter_k-40_colors', 'cluster_cellcharter_k-45_colors', 'cluster_cellcharter_k-50_colors', 'cluster_cellcharter_k-8_colors', 'conf_score_cellcharter-cluster-k50_colors', 'conf_score_sliding_window_assignment_colors', 'fine_grain_4Jan2025_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'predicted_labels_cellcharter-cluster-k50_colors', 'predicted_labels_colors', 'predicted_labels_sliding_window_assignment_colors', 'predicted_labels_spatial_KMeans_k1843_colors', 'predicted_labels_spatial_KMeans_k3072_colors', 'predicted_labels_spatial_KMeans_k4608_colors', 'predicted_labels_spatial_KMeans_k9216_colors', 'sliding_window_assignment_colors', 'spatial', 'spatial_leiden_0.5', 'spatial_leiden_1', 'spatial_leiden_10', 'spatial_leiden_100', 'spatial_leiden_1000', 'spatial_leiden_1_colors', 'spatial_leiden_2', 'spatial_leiden_5', 'spatial_leiden_5_colors', 'spatial_neighbors', 'tt_prediction_colors', 'umap'
    obsm: 'X_cellcharter', 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances', 'spatial_connectivities', 'spatial_distances'
```

Example 12 (python):
```python
adata.X.data[:5]
```

Example 13 (unknown):
```unknown
array([1.27187559, 1.95641568, 1.75023687, 1.96760567, 2.7810601 ])
```

Example 14 (unknown):
```unknown
group_col = 'tt_prediction'
```

Example 15 (python):
```python
# plot
sc.pl.spatial(adata, 
                  color=[group_col], 
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=2,palette=AnnotationPalette,
                  frameon=False,show=False
                 )
plt.savefig(f'{figdir}/VisiumHD_euploid_TissueTypist_predicted-labels.pdf', bbox_inches="tight",dpi=300)
```

Example 16 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 35 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'C', 'D', 'E', 'G', 'N', 'P', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v', 'x', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 38, 39, 40, 42, 49, 51, 54, 55, 57, 66, 68, 69, 70, 71, 72, 74, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89, 91, 92]
Closed glyph list over 'MATH': 35 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'C', 'D', 'E', 'G', 'N', 'P', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v', 'x', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 38, 39, 40, 42, 49, 51, 54, 55, 57, 66, 68, 69, 70, 71, 72, 74, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89, 91, 92]
Closing glyph list over 'GSUB': 35 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'C', 'D', 'E', 'G', 'N', 'P', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v', 'x', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 38, 39, 40, 42, 49, 51, 54, 55, 57, 66, 68, 69, 70, 71, 72, 74, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89, 91, 92]
Closed glyph list over 'GSUB': 35 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'C', 'D', 'E', 'G', 'N', 'P', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v', 'x', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 38, 39, 40, 42, 49, 51, 54, 55, 57, 66, 68, 69, 70, 71, 72, 74, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89, 91, 92]
Closing glyph list over 'glyf': 35 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'C', 'D', 'E', 'G', 'N', 'P', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v', 'x', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 38, 39, 40, 42, 49, 51, 54, 55, 57, 66, 68, 69, 70, 71, 72, 74, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89, 91, 92]
Closed glyph list over 'glyf': 35 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'C', 'D', 'E', 'G', 'N', 'P', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v', 'x', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 38, 39, 40, 42, 49, 51, 54, 55, 57, 66, 68, 69, 70, 71, 72, 74, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89, 91, 92]
Retaining 35 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 17 (python):
```python
labels_to_plot = adata.obs[group_col].cat.categories.copy()
print(len(labels_to_plot))
labels_to_plot
```

Example 18 (unknown):
```unknown
Index(['AV ring', 'AVnode', 'Adventitia - GV', 'Atrium', 'Coronary vessel',
       'Ductus arteriosus', 'Endocardial cusion', 'Endocardium - Atrial',
       'Endocardium - Ventricular', 'Epicardium', 'Great vessel', 'SAnode',
       'VCS - Distal', 'VCS - Proximal', 'Valves', 'Ventricle - Compact',
       'Ventricle - Trabeculated'],
      dtype='object')
```

Example 19 (python):
```python
fig,axs=plt.subplots(nrows=3,ncols=6,figsize=(24,9))
# Flatten the axs array for easier iteration in case of multiple rows and columns
axs_flat = axs.flatten()

for i,label in enumerate(labels_to_plot):
    ad = adata.copy()
    # update colors
    # get color in adata
    ind = list(ad.obs[group_col].cat.categories).index(label)
    ad.uns[f'{group_col}_colors'][ind] = 'greenyellow'
    # plot
    sc.pl.spatial(ad, 
                  color=group_col, groups=label,
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=7,legend_loc=None,
                  frameon=False,title=label,
                  show=False, ax=axs_flat[i],
                 )
# Turn off any remaining subplots
for j in range(len(labels_to_plot), len(axs_flat)):
    fig.delaxes(axs_flat[j])

# fig.suptitle('Main Title for the Figure')
fig.tight_layout()
```

Example 20 (python):
```python
mask = ((adata.obs['array_row'] >= 1500) & 
        (adata.obs['array_row'] <= 2300) & 
        (adata.obs['array_col'] >= 400) & 
        (adata.obs['array_col'] <= 1200)
       )
bdata = adata[mask]

sc.pl.spatial(bdata, 
              color=[
                      'HistologicalAnnotation'
                        ], 
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=0,alpha_img=1,legend_loc=None,
                  frameon=False, show=False
                 )
plt.savefig(f'{figdir}/VisiumHD_euploid_Atrium_HE.pdf', bbox_inches="tight",dpi=300)
```

Example 21 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 15 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closed glyph list over 'MATH': 15 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closing glyph list over 'GSUB': 15 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closed glyph list over 'GSUB': 15 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closing glyph list over 'glyf': 15 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closed glyph list over 'glyf': 15 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Retaining 15 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 22 (python):
```python
n_annotation = len(bdata.obs['HistologicalAnnotation'].cat.categories)
bdata.uns['HistologicalAnnotation'] = list(np.repeat('#ffff00',n_annotation))

# update colors
for c in bdata.obs['HistologicalAnnotation'].cat.categories:
    # get color in adata
    ind = list(adata.obs['HistologicalAnnotation'].cat.categories).index(c)
    color = adata.uns['HistologicalAnnotation_colors'][ind]
    # update color in bdata
    ind = list(bdata.obs['HistologicalAnnotation'].cat.categories).index(c)
    bdata.uns['HistologicalAnnotation_colors'][ind] = color

sc.pl.spatial(bdata, 
                  color=[
                      'HistologicalAnnotation'
                        ], 
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=1.5,alpha_img=0.3,
                  frameon=False, show=False
                 )
plt.savefig(f'{figdir}/VisiumHD_euploid_Atrium_HistologicalAnnotation.pdf', bbox_inches="tight",dpi=300)
```

Example 23 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 24 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'G', 'H', 'N', 'V', 'a', 'c', 'e', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'v']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 42, 43, 49, 57, 68, 70, 72, 74, 76, 79, 80, 81, 82, 85, 86, 87, 88, 89]
Closed glyph list over 'MATH': 24 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'G', 'H', 'N', 'V', 'a', 'c', 'e', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'v']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 42, 43, 49, 57, 68, 70, 72, 74, 76, 79, 80, 81, 82, 85, 86, 87, 88, 89]
Closing glyph list over 'GSUB': 24 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'G', 'H', 'N', 'V', 'a', 'c', 'e', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'v']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 42, 43, 49, 57, 68, 70, 72, 74, 76, 79, 80, 81, 82, 85, 86, 87, 88, 89]
Closed glyph list over 'GSUB': 24 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'G', 'H', 'N', 'V', 'a', 'c', 'e', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'v']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 42, 43, 49, 57, 68, 70, 72, 74, 76, 79, 80, 81, 82, 85, 86, 87, 88, 89]
Closing glyph list over 'glyf': 24 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'G', 'H', 'N', 'V', 'a', 'c', 'e', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'v']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 42, 43, 49, 57, 68, 70, 72, 74, 76, 79, 80, 81, 82, 85, 86, 87, 88, 89]
Closed glyph list over 'glyf': 24 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'G', 'H', 'N', 'V', 'a', 'c', 'e', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'v']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 42, 43, 49, 57, 68, 70, 72, 74, 76, 79, 80, 81, 82, 85, 86, 87, 88, 89]
Retaining 24 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 24 (python):
```python
sc.pl.spatial(bdata, 
                  color=[
                      group_col
                        ], 
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=1.5,alpha_img=0.3,palette=AnnotationPalette,
                  frameon=False, show=False
                 )
plt.savefig(f'{figdir}/VisiumHD_euploid_Atrium_TissueTypist_window_predicted-labels.pdf', bbox_inches="tight",dpi=300)
```

Example 25 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 34 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'C', 'D', 'E', 'G', 'N', 'P', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v', 'x', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 38, 39, 40, 42, 49, 51, 54, 55, 57, 66, 68, 69, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89, 91, 92]
Closed glyph list over 'MATH': 34 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'C', 'D', 'E', 'G', 'N', 'P', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v', 'x', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 38, 39, 40, 42, 49, 51, 54, 55, 57, 66, 68, 69, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89, 91, 92]
Closing glyph list over 'GSUB': 34 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'C', 'D', 'E', 'G', 'N', 'P', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v', 'x', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 38, 39, 40, 42, 49, 51, 54, 55, 57, 66, 68, 69, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89, 91, 92]
Closed glyph list over 'GSUB': 34 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'C', 'D', 'E', 'G', 'N', 'P', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v', 'x', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 38, 39, 40, 42, 49, 51, 54, 55, 57, 66, 68, 69, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89, 91, 92]
Closing glyph list over 'glyf': 34 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'C', 'D', 'E', 'G', 'N', 'P', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v', 'x', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 38, 39, 40, 42, 49, 51, 54, 55, 57, 66, 68, 69, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89, 91, 92]
Closed glyph list over 'glyf': 34 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'C', 'D', 'E', 'G', 'N', 'P', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v', 'x', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 38, 39, 40, 42, 49, 51, 54, 55, 57, 66, 68, 69, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89, 91, 92]
Retaining 34 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 26 (python):
```python
mask = ((adata.obs['array_row'] >= 1300) & 
        (adata.obs['array_row'] <= 2100) & 
        (adata.obs['array_col'] >= 2400) & 
        (adata.obs['array_col'] <= 3400)
       )
bdata = adata[mask]

sc.pl.spatial(bdata, 
              color=[
                      'HistologicalAnnotation'
                        ], 
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=0,alpha_img=1,legend_loc=None,
                  frameon=False, show=False
                 )
plt.savefig(f'{figdir}/VisiumHD_euploid_Ventricle_HE.pdf', bbox_inches="tight",dpi=300)
```

Example 27 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 15 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closed glyph list over 'MATH': 15 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closing glyph list over 'GSUB': 15 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closed glyph list over 'GSUB': 15 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closing glyph list over 'glyf': 15 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closed glyph list over 'glyf': 15 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Retaining 15 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 28 (python):
```python
n_annotation = len(bdata.obs['HistologicalAnnotation'].cat.categories)
bdata.uns['HistologicalAnnotation'] = list(np.repeat('#ffff00',n_annotation))

# update colors
for c in bdata.obs['HistologicalAnnotation'].cat.categories:
    # get color in adata
    ind = list(adata.obs['HistologicalAnnotation'].cat.categories).index(c)
    color = adata.uns['HistologicalAnnotation_colors'][ind]
    # update color in bdata
    ind = list(bdata.obs['HistologicalAnnotation'].cat.categories).index(c)
    bdata.uns['HistologicalAnnotation_colors'][ind] = color

sc.pl.spatial(bdata, 
                  color=[
                      'HistologicalAnnotation'
                        ], 
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=1.5,alpha_img=0.3,
                  frameon=False, show=False
                 )
plt.savefig(f'{figdir}/VisiumHD_euploid_Ventricle_HistologicalAnnotation.pdf', bbox_inches="tight",dpi=300)
```

Example 29 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 18 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'H', 'V', 'a', 'c', 'e', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 57, 68, 70, 72, 74, 76, 79, 81, 82, 85, 86, 87]
Closed glyph list over 'MATH': 18 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'H', 'V', 'a', 'c', 'e', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 57, 68, 70, 72, 74, 76, 79, 81, 82, 85, 86, 87]
Closing glyph list over 'GSUB': 18 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'H', 'V', 'a', 'c', 'e', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 57, 68, 70, 72, 74, 76, 79, 81, 82, 85, 86, 87]
Closed glyph list over 'GSUB': 18 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'H', 'V', 'a', 'c', 'e', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 57, 68, 70, 72, 74, 76, 79, 81, 82, 85, 86, 87]
Closing glyph list over 'glyf': 18 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'H', 'V', 'a', 'c', 'e', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 57, 68, 70, 72, 74, 76, 79, 81, 82, 85, 86, 87]
Closed glyph list over 'glyf': 18 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'H', 'V', 'a', 'c', 'e', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 57, 68, 70, 72, 74, 76, 79, 81, 82, 85, 86, 87]
Retaining 18 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 30 (python):
```python
sc.pl.spatial(bdata, 
                  color=[
                      group_col
                        ], 
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=1.5,alpha_img=0.3,palette=AnnotationPalette,
                  frameon=False, show=False
                 )
plt.savefig(f'{figdir}/VisiumHD_euploid_Ventricle_TissueTypist_window_predicted-labels.pdf', bbox_inches="tight",dpi=300)
```

Example 31 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 27 glyphs before
Glyph names: ['.notdef', '.null', 'C', 'D', 'E', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore']
Glyph IDs:   [0, 1, 2, 3, 16, 38, 39, 40, 54, 55, 57, 66, 68, 69, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88]
Closed glyph list over 'MATH': 27 glyphs after
Glyph names: ['.notdef', '.null', 'C', 'D', 'E', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore']
Glyph IDs:   [0, 1, 2, 3, 16, 38, 39, 40, 54, 55, 57, 66, 68, 69, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88]
Closing glyph list over 'GSUB': 27 glyphs before
Glyph names: ['.notdef', '.null', 'C', 'D', 'E', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore']
Glyph IDs:   [0, 1, 2, 3, 16, 38, 39, 40, 54, 55, 57, 66, 68, 69, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88]
Closed glyph list over 'GSUB': 27 glyphs after
Glyph names: ['.notdef', '.null', 'C', 'D', 'E', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore']
Glyph IDs:   [0, 1, 2, 3, 16, 38, 39, 40, 54, 55, 57, 66, 68, 69, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88]
Closing glyph list over 'glyf': 27 glyphs before
Glyph names: ['.notdef', '.null', 'C', 'D', 'E', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore']
Glyph IDs:   [0, 1, 2, 3, 16, 38, 39, 40, 54, 55, 57, 66, 68, 69, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88]
Closed glyph list over 'glyf': 27 glyphs after
Glyph names: ['.notdef', '.null', 'C', 'D', 'E', 'S', 'T', 'V', 'a', 'b', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore']
Glyph IDs:   [0, 1, 2, 3, 16, 38, 39, 40, 54, 55, 57, 66, 68, 69, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88]
Retaining 27 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 32 (python):
```python
mask = ((adata.obs['array_row'] >= 2600) & 
        (adata.obs['array_row'] <= 3000) & 
        (adata.obs['array_col'] >= 1200) & 
        (adata.obs['array_col'] <= 1650)
       )
bdata = adata[mask]

sc.pl.spatial(bdata, 
              color=[
                      'HistologicalAnnotation'
                        ], 
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=0,alpha_img=1,legend_loc=None,
                  frameon=False, show=False
                 )
plt.savefig(f'{figdir}/VisiumHD_euploid_GreatVessel_HE.pdf', bbox_inches="tight",dpi=300)
```

Example 33 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 15 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closed glyph list over 'MATH': 15 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closing glyph list over 'GSUB': 15 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closed glyph list over 'GSUB': 15 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closing glyph list over 'glyf': 15 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Closed glyph list over 'glyf': 15 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'H', 'a', 'c', 'g', 'i', 'l', 'n', 'nonmarkingreturn', 'o', 's', 'space', 't']
Glyph IDs:   [0, 1, 2, 3, 36, 43, 68, 70, 74, 76, 79, 81, 82, 86, 87]
Retaining 15 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 34 (python):
```python
n_annotation = len(bdata.obs['HistologicalAnnotation'].cat.categories)
bdata.uns['HistologicalAnnotation'] = list(np.repeat('#ffff00',n_annotation))

# update colors
for c in bdata.obs['HistologicalAnnotation'].cat.categories:
    # get color in adata
    ind = list(adata.obs['HistologicalAnnotation'].cat.categories).index(c)
    color = adata.uns['HistologicalAnnotation_colors'][ind]
    # update color in bdata
    ind = list(bdata.obs['HistologicalAnnotation'].cat.categories).index(c)
    bdata.uns['HistologicalAnnotation_colors'][ind] = color

sc.pl.spatial(bdata, 
                  color=[
                      'HistologicalAnnotation'
                        ], 
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=1.5,alpha_img=0.3,
                  frameon=False, show=False
                 )
plt.savefig(f'{figdir}/VisiumHD_euploid_GreatVessel_HistologicalAnnotation.pdf', bbox_inches="tight",dpi=300)
```

Example 35 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 29 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'D', 'G', 'H', 'L', 'N', 'P', 'a', 'c', 'd', 'e', 'f', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'v', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 39, 42, 43, 47, 49, 51, 68, 70, 71, 72, 73, 74, 76, 79, 80, 81, 82, 85, 86, 87, 88, 89, 92]
Closed glyph list over 'MATH': 29 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'D', 'G', 'H', 'L', 'N', 'P', 'a', 'c', 'd', 'e', 'f', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'v', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 39, 42, 43, 47, 49, 51, 68, 70, 71, 72, 73, 74, 76, 79, 80, 81, 82, 85, 86, 87, 88, 89, 92]
Closing glyph list over 'GSUB': 29 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'D', 'G', 'H', 'L', 'N', 'P', 'a', 'c', 'd', 'e', 'f', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'v', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 39, 42, 43, 47, 49, 51, 68, 70, 71, 72, 73, 74, 76, 79, 80, 81, 82, 85, 86, 87, 88, 89, 92]
Closed glyph list over 'GSUB': 34 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'D', 'G', 'H', 'L', 'N', 'P', 'a', 'c', 'd', 'e', 'f', 'fi', 'fl', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'uniFB00', 'uniFB03', 'uniFB04', 'v', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 39, 42, 43, 47, 49, 51, 68, 70, 71, 72, 73, 74, 76, 79, 80, 81, 82, 85, 86, 87, 88, 89, 92, 5038, 5039, 5040, 5041, 5042]
Closing glyph list over 'glyf': 34 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'D', 'G', 'H', 'L', 'N', 'P', 'a', 'c', 'd', 'e', 'f', 'fi', 'fl', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'uniFB00', 'uniFB03', 'uniFB04', 'v', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 39, 42, 43, 47, 49, 51, 68, 70, 71, 72, 73, 74, 76, 79, 80, 81, 82, 85, 86, 87, 88, 89, 92, 5038, 5039, 5040, 5041, 5042]
Closed glyph list over 'glyf': 34 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'D', 'G', 'H', 'L', 'N', 'P', 'a', 'c', 'd', 'e', 'f', 'fi', 'fl', 'g', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'uniFB00', 'uniFB03', 'uniFB04', 'v', 'y']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 39, 42, 43, 47, 49, 51, 68, 70, 71, 72, 73, 74, 76, 79, 80, 81, 82, 85, 86, 87, 88, 89, 92, 5038, 5039, 5040, 5041, 5042]
Retaining 34 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 36 (python):
```python
sc.pl.spatial(bdata, 
                  color=[
                      group_col
                        ], 
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=1.5,alpha_img=0.3,palette=AnnotationPalette,
                  frameon=False, show=False
                 )
plt.savefig(f'{figdir}/VisiumHD_euploid_GreatVessel_TissueTypist_window_predicted-labels.pdf', bbox_inches="tight",dpi=300)
```

Example 37 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 26 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'D', 'E', 'G', 'V', 'a', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 39, 40, 42, 57, 66, 68, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89]
Closed glyph list over 'MATH': 26 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'D', 'E', 'G', 'V', 'a', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 39, 40, 42, 57, 66, 68, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89]
Closing glyph list over 'GSUB': 26 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'D', 'E', 'G', 'V', 'a', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 39, 40, 42, 57, 66, 68, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89]
Closed glyph list over 'GSUB': 26 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'D', 'E', 'G', 'V', 'a', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 39, 40, 42, 57, 66, 68, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89]
Closing glyph list over 'glyf': 26 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'D', 'E', 'G', 'V', 'a', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 39, 40, 42, 57, 66, 68, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89]
Closed glyph list over 'glyf': 26 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'D', 'E', 'G', 'V', 'a', 'c', 'd', 'e', 'hyphen', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 't', 'u', 'underscore', 'v']
Glyph IDs:   [0, 1, 2, 3, 16, 36, 39, 40, 42, 57, 66, 68, 70, 71, 72, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89]
Retaining 26 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 38 (python):
```python
# celltype name mapping for plotting
df = pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/finegrain_name_mapping.csv')
celltype_mapping = df.set_index('Full_name')['Short_name_finalised'].to_dict()
adata.obs.replace({'fine_grain_4Jan2025':celltype_mapping},inplace=True)
```

Example 39 (python):
```python
celltype_key = 'fine_grain_4Jan2025'
set(adata.obs[celltype_key])
```

Example 40 (unknown):
```unknown
{'AVNPC',
 'B',
 'ChromC',
 'CorArtEC',
 'CorCapEC',
 'CorPeri',
 'CorSMC',
 'CorVenEC',
 'DASMC',
 'EPDC',
 'EndoCush',
 'Endocard',
 'Eryth',
 'FB',
 'FBCOL2A1',
 'GVArtEC',
 'GVSMC',
 'GVSMCP',
 'GVSMCT',
 'GVVenEC',
 'ILC',
 'LymphEC',
 'MPLYVE1',
 'MacCX3',
 'MegaK',
 'MesEpiC',
 'MoDC',
 'Mono',
 'NeuroPre',
 'ParaN',
 'PericFib',
 'PericInt',
 'SANPCHd',
 'SANPCHn',
 'SANPCT',
 'SchwC',
 'SchwNeuro',
 'SchwP',
 'SympN',
 'VCSDist',
 'VCSProx',
 'ValveEC',
 'ValveIC',
 'aCML',
 'aCMR',
 'vCMC',
 'vCMT'}
```

Example 41 (python):
```python
# HE
mask = ((adata.obs['array_row'] >= 1700) & 
        (adata.obs['array_row'] <= 2300) & 
        (adata.obs['array_col'] >= 600) & 
        (adata.obs['array_col'] <= 1100)
       )
bdata = adata[mask]

sc.pl.spatial(bdata, 
              color=['fine_grain_4Jan2025'],
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=0,alpha_img=1,
                  legend_loc=None, frameon=False, show=False
                 )
plt.savefig(f'{figdir}/VisiumHD_euploid_SAN_for-Fig1_HE.pdf',bbox_inches="tight",dpi=300)
```

Example 42 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 17 glyphs before
Glyph names: ['.notdef', '.null', 'J', 'a', 'e', 'f', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'r', 'space', 'two', 'underscore', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 23, 24, 45, 66, 68, 72, 73, 74, 76, 81, 85]
Closed glyph list over 'MATH': 17 glyphs after
Glyph names: ['.notdef', '.null', 'J', 'a', 'e', 'f', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'r', 'space', 'two', 'underscore', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 23, 24, 45, 66, 68, 72, 73, 74, 76, 81, 85]
Closing glyph list over 'GSUB': 17 glyphs before
Glyph names: ['.notdef', '.null', 'J', 'a', 'e', 'f', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'r', 'space', 'two', 'underscore', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 23, 24, 45, 66, 68, 72, 73, 74, 76, 81, 85]
Closed glyph list over 'GSUB': 20 glyphs after
Glyph names: ['.notdef', '.null', 'J', 'a', 'e', 'f', 'fi', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'r', 'space', 'two', 'underscore', 'uniFB00', 'uniFB03', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 23, 24, 45, 66, 68, 72, 73, 74, 76, 81, 85, 5038, 5039, 5041]
Closing glyph list over 'glyf': 20 glyphs before
Glyph names: ['.notdef', '.null', 'J', 'a', 'e', 'f', 'fi', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'r', 'space', 'two', 'underscore', 'uniFB00', 'uniFB03', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 23, 24, 45, 66, 68, 72, 73, 74, 76, 81, 85, 5038, 5039, 5041]
Closed glyph list over 'glyf': 20 glyphs after
Glyph names: ['.notdef', '.null', 'J', 'a', 'e', 'f', 'fi', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'r', 'space', 'two', 'underscore', 'uniFB00', 'uniFB03', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 23, 24, 45, 66, 68, 72, 73, 74, 76, 81, 85, 5038, 5039, 5041]
Retaining 20 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 43 (python):
```python
mask = ((adata.obs['array_row'] >= 1700) & 
        (adata.obs['array_row'] <= 2300) & 
        (adata.obs['array_col'] >= 600) & 
        (adata.obs['array_col'] <= 1100)
       )
bdata = adata[mask]

bdata.obs[celltype_key] = bdata.obs[celltype_key].astype('str')
bdata.obs.replace({celltype_key:{
    'SANPCHn':'SANPC',
    'SANPCHd':'SANPC',
    'SANPCT':'SANPC'
}},inplace=True)
bdata.obs[celltype_key] = bdata.obs[celltype_key].astype('category')

celltypes_to_plot = ['SANPC',
                     'CorCapEC',
                     'CorPeri',
                     'GVVenEC',
                     'MesEpiC',
                     'FB',
                     'aCMR',
                     'AVNPC',
                     'ParaN',
                     'SchwP',
                    ]

# modify colors
bdata.uns[f"{celltype_key}_colors"][list(bdata.obs[celltype_key].cat.categories).index('GVVenEC')]='red'
bdata.uns[f"{celltype_key}_colors"][list(bdata.obs[celltype_key].cat.categories).index('ParaN')]='blue'
bdata.uns[f"{celltype_key}_colors"][list(bdata.obs[celltype_key].cat.categories).index('SANPC')]=(0.5, 0.0, 0.5, 0.50) # purple, 30% opacity

sc.pl.spatial(bdata, 
              color=['fine_grain_4Jan2025'], groups=celltypes_to_plot,
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=1.5,alpha_img=0.3,
                  frameon=False, show=False
                 )
plt.savefig(f'{figdir}/VisiumHD_euploid_SAN_for-Fig1.pdf',bbox_inches="tight",dpi=300)
```

Example 44 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 32 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'E', 'F', 'G', 'J', 'M', 'N', 'P', 'R', 'S', 'V', 'a', 'e', 'f', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 'two', 'underscore', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 23, 24, 36, 37, 38, 40, 41, 42, 45, 48, 49, 51, 53, 54, 57, 66, 68, 72, 73, 74, 76, 81, 82, 83, 85, 86]
Closed glyph list over 'MATH': 32 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'E', 'F', 'G', 'J', 'M', 'N', 'P', 'R', 'S', 'V', 'a', 'e', 'f', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 'two', 'underscore', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 23, 24, 36, 37, 38, 40, 41, 42, 45, 48, 49, 51, 53, 54, 57, 66, 68, 72, 73, 74, 76, 81, 82, 83, 85, 86]
Closing glyph list over 'GSUB': 32 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'E', 'F', 'G', 'J', 'M', 'N', 'P', 'R', 'S', 'V', 'a', 'e', 'f', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 'two', 'underscore', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 23, 24, 36, 37, 38, 40, 41, 42, 45, 48, 49, 51, 53, 54, 57, 66, 68, 72, 73, 74, 76, 81, 82, 83, 85, 86]
Closed glyph list over 'GSUB': 35 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'E', 'F', 'G', 'J', 'M', 'N', 'P', 'R', 'S', 'V', 'a', 'e', 'f', 'fi', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 'two', 'underscore', 'uniFB00', 'uniFB03', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 23, 24, 36, 37, 38, 40, 41, 42, 45, 48, 49, 51, 53, 54, 57, 66, 68, 72, 73, 74, 76, 81, 82, 83, 85, 86, 5038, 5039, 5041]
Closing glyph list over 'glyf': 35 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'E', 'F', 'G', 'J', 'M', 'N', 'P', 'R', 'S', 'V', 'a', 'e', 'f', 'fi', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 'two', 'underscore', 'uniFB00', 'uniFB03', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 23, 24, 36, 37, 38, 40, 41, 42, 45, 48, 49, 51, 53, 54, 57, 66, 68, 72, 73, 74, 76, 81, 82, 83, 85, 86, 5038, 5039, 5041]
Closed glyph list over 'glyf': 35 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'E', 'F', 'G', 'J', 'M', 'N', 'P', 'R', 'S', 'V', 'a', 'e', 'f', 'fi', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'o', 'p', 'r', 's', 'space', 'two', 'underscore', 'uniFB00', 'uniFB03', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 23, 24, 36, 37, 38, 40, 41, 42, 45, 48, 49, 51, 53, 54, 57, 66, 68, 72, 73, 74, 76, 81, 82, 83, 85, 86, 5038, 5039, 5041]
Retaining 35 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 45 (python):
```python
print(bdata.obs[celltype_key].cat.categories)
bdata.uns[f"{celltype_key}_colors"]
```

Example 46 (unknown):
```unknown
Index(['AVNPC', 'CorCapEC', 'CorPeri', 'DASMC', 'EPDC', 'EndoCush', 'Endocard',
       'Eryth', 'FB', 'GVSMC', 'GVSMCT', 'GVVenEC', 'LymphEC', 'MPLYVE1',
       'MacCX3', 'MesEpiC', 'MoDC', 'ParaN', 'PericFib', 'SANPC', 'SchwC',
       'SchwNeuro', 'ValveEC', 'ValveIC', 'aCML', 'aCMR'],
      dtype='object')
```

Example 47 (unknown):
```unknown
['#ffff00',
 '#1ce6ff',
 '#ff34ff',
 '#a30059',
 '#ffdbe5',
 '#63ffac',
 '#b79762',
 '#004d43',
 '#8fb0ff',
 '#997d87',
 '#5a0007',
 'red',
 '#4a3b53',
 '#ff2f80',
 '#ba0900',
 '#6b7900',
 '#00c2a0',
 'blue',
 '#b903aa',
 (0.5, 0.0, 0.5, 0.5),
 '#7b4f4b',
 '#013349',
 '#00846f',
 '#372101',
 '#ffb500',
 '#c2ffed',
 '#cc0744',
 '#c0b9b2']
```

Example 48 (python):
```python
celltypes_plot = {
    'SANPC':"#8000801A", # purple, 15% opacity
    'ParaN':'blue',
    'MacCX3':'red'
}

for celltype,color in celltypes_plot.items():
    # update colors
    bdata.uns[f"{celltype_key}_colors"][list(bdata.obs[celltype_key].cat.categories).index(celltype)]=color

sc.pl.spatial(bdata, 
              color=[celltype_key], 
              groups=list(celltypes_plot.keys()),
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer",
                  size=3,alpha_img=0.3,
                  frameon=False, show=False
                 )
plt.savefig(f'{figdir}/VisiumHD_euploid_SAN_{"-".join(list(celltypes_plot.keys()))}.pdf',bbox_inches="tight",dpi=300)
```

Example 49 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 26 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'C', 'J', 'M', 'N', 'P', 'S', 'X', 'a', 'c', 'e', 'f', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'r', 'space', 'three', 'two', 'underscore', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 22, 23, 24, 36, 38, 45, 48, 49, 51, 54, 59, 66, 68, 70, 72, 73, 74, 76, 81, 85]
Closed glyph list over 'MATH': 26 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'C', 'J', 'M', 'N', 'P', 'S', 'X', 'a', 'c', 'e', 'f', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'r', 'space', 'three', 'two', 'underscore', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 22, 23, 24, 36, 38, 45, 48, 49, 51, 54, 59, 66, 68, 70, 72, 73, 74, 76, 81, 85]
Closing glyph list over 'GSUB': 26 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'C', 'J', 'M', 'N', 'P', 'S', 'X', 'a', 'c', 'e', 'f', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'r', 'space', 'three', 'two', 'underscore', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 22, 23, 24, 36, 38, 45, 48, 49, 51, 54, 59, 66, 68, 70, 72, 73, 74, 76, 81, 85]
Closed glyph list over 'GSUB': 29 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'C', 'J', 'M', 'N', 'P', 'S', 'X', 'a', 'c', 'e', 'f', 'fi', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'r', 'space', 'three', 'two', 'underscore', 'uniFB00', 'uniFB03', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 22, 23, 24, 36, 38, 45, 48, 49, 51, 54, 59, 66, 68, 70, 72, 73, 74, 76, 81, 85, 5038, 5039, 5041]
Closing glyph list over 'glyf': 29 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'C', 'J', 'M', 'N', 'P', 'S', 'X', 'a', 'c', 'e', 'f', 'fi', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'r', 'space', 'three', 'two', 'underscore', 'uniFB00', 'uniFB03', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 22, 23, 24, 36, 38, 45, 48, 49, 51, 54, 59, 66, 68, 70, 72, 73, 74, 76, 81, 85, 5038, 5039, 5041]
Closed glyph list over 'glyf': 29 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'C', 'J', 'M', 'N', 'P', 'S', 'X', 'a', 'c', 'e', 'f', 'fi', 'five', 'four', 'g', 'i', 'n', 'nonmarkingreturn', 'r', 'space', 'three', 'two', 'underscore', 'uniFB00', 'uniFB03', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 21, 22, 23, 24, 36, 38, 45, 48, 49, 51, 54, 59, 66, 68, 70, 72, 73, 74, 76, 81, 85, 5038, 5039, 5041]
Retaining 29 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 50 (python):
```python
ADATA_PATH = "/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad"
ad = sc.read_h5ad(ADATA_PATH, backed='r')

# Step 1: Create mapping (already done)
cat = "mid_grain"
mid2colours = dict(zip(ad.obs[cat].cat.categories, ad.uns[f"{cat}_colors"]))

# Step 2: Extract the labels and colors in the correct order
mid_grains = list(mid2colours.keys())
palette = list(mid2colours.values())
num_mid_grains = len(mid_grains)

# Step 3: Plot
plt.figure(figsize=(num_mid_grains * 0.4, 2))  # wide figure
sns.palplot(palette)
plt.xticks(np.arange(num_mid_grains), mid_grains, rotation=90)
plt.title("Mid Grain to Color Mapping")
plt.tight_layout()
plt.show()
```

Example 51 (unknown):
```unknown
<Figure size 448x160 with 0 Axes>
```

Example 52 (python):
```python
adata.obs['celltypist_mid_fin'] = pd.Categorical(
    adata.obs['celltypist_mid_fin'],
    categories=ad.obs[cat].cat.categories,
    ordered=True
)
```

Example 53 (python):
```python
sc.pl.embedding(adata,
                    basis=f"X_umap",
                    color='celltypist_mid_fin',palette=palette,
                show=False
                   )
plt.savefig(f'{figdir}/VisiumHD_UMAP-midgrain_for-Fig1.pdf',bbox_inches="tight",dpi=300)
```

Example 54 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 38 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'E', 'F', 'G', 'L', 'M', 'N', 'P', 'S', 'V', 'X', 'a', 'b', 'c', 'd', 'e', 'f', 'h', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'r', 's', 'space', 't', 'two', 'u', 'underscore', 'y']
Glyph IDs:   [0, 1, 2, 3, 20, 21, 36, 37, 38, 40, 41, 42, 47, 48, 49, 51, 54, 57, 59, 66, 68, 69, 70, 71, 72, 73, 75, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 92]
Closed glyph list over 'MATH': 38 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'E', 'F', 'G', 'L', 'M', 'N', 'P', 'S', 'V', 'X', 'a', 'b', 'c', 'd', 'e', 'f', 'h', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'r', 's', 'space', 't', 'two', 'u', 'underscore', 'y']
Glyph IDs:   [0, 1, 2, 3, 20, 21, 36, 37, 38, 40, 41, 42, 47, 48, 49, 51, 54, 57, 59, 66, 68, 69, 70, 71, 72, 73, 75, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 92]
Closing glyph list over 'GSUB': 38 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'E', 'F', 'G', 'L', 'M', 'N', 'P', 'S', 'V', 'X', 'a', 'b', 'c', 'd', 'e', 'f', 'h', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'r', 's', 'space', 't', 'two', 'u', 'underscore', 'y']
Glyph IDs:   [0, 1, 2, 3, 20, 21, 36, 37, 38, 40, 41, 42, 47, 48, 49, 51, 54, 57, 59, 66, 68, 69, 70, 71, 72, 73, 75, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 92]
Closed glyph list over 'GSUB': 43 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'E', 'F', 'G', 'L', 'M', 'N', 'P', 'S', 'V', 'X', 'a', 'b', 'c', 'd', 'e', 'f', 'fi', 'fl', 'h', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'r', 's', 'space', 't', 'two', 'u', 'underscore', 'uniFB00', 'uniFB03', 'uniFB04', 'y']
Glyph IDs:   [0, 1, 2, 3, 20, 21, 36, 37, 38, 40, 41, 42, 47, 48, 49, 51, 54, 57, 59, 66, 68, 69, 70, 71, 72, 73, 75, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 92, 5038, 5039, 5040, 5041, 5042]
Closing glyph list over 'glyf': 43 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'E', 'F', 'G', 'L', 'M', 'N', 'P', 'S', 'V', 'X', 'a', 'b', 'c', 'd', 'e', 'f', 'fi', 'fl', 'h', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'r', 's', 'space', 't', 'two', 'u', 'underscore', 'uniFB00', 'uniFB03', 'uniFB04', 'y']
Glyph IDs:   [0, 1, 2, 3, 20, 21, 36, 37, 38, 40, 41, 42, 47, 48, 49, 51, 54, 57, 59, 66, 68, 69, 70, 71, 72, 73, 75, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 92, 5038, 5039, 5040, 5041, 5042]
Closed glyph list over 'glyf': 43 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'E', 'F', 'G', 'L', 'M', 'N', 'P', 'S', 'V', 'X', 'a', 'b', 'c', 'd', 'e', 'f', 'fi', 'fl', 'h', 'i', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'r', 's', 'space', 't', 'two', 'u', 'underscore', 'uniFB00', 'uniFB03', 'uniFB04', 'y']
Glyph IDs:   [0, 1, 2, 3, 20, 21, 36, 37, 38, 40, 41, 42, 47, 48, 49, 51, 54, 57, 59, 66, 68, 69, 70, 71, 72, 73, 75, 76, 79, 80, 81, 82, 83, 85, 86, 87, 88, 92, 5038, 5039, 5040, 5041, 5042]
Retaining 43 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig3_SAN_niche/cellphonedb-VisiumHD_no1_prepare-data.html

**Contents:**
- Import modules¶
- Set path¶
- Read in anndata¶
- Subset cellular niche of interest¶
- Save¶

https://github.com/ventolab/CellphoneDB/blob/master/notebooks/0_prepare_your_data_from_anndata.ipynb

**Examples:**

Example 1 (python):
```python
import numpy as np
import pandas as pd
import scanpy as sc
import anndata
import os
import sys
from scipy import sparse
import pickle
import matplotlib.pyplot as plt

sc.settings.verbosity = 1  # verbosity: errors (0), warnings (1), info (2), hints (3)
```

Example 2 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 3 (unknown):
```unknown
sample_id = 'HEA_FOET14880396'
cellphone_input_dir = f'/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/{sample_id}/inputs'
cellphone_input_dir
```

Example 4 (unknown):
```unknown
'/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/inputs'
```

Example 5 (unknown):
```unknown
celltype_col = 'fine_grain_4Jan2025'
```

Example 6 (unknown):
```unknown
niche_label_col = 'tt_prediction'
niche_of_interest = 'SAnode'
```

Example 7 (python):
```python
# read in 
adata = sc.read_h5ad(f'/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/{sample_id}_b2c_cells_filtered_celltype-selected_lognorm.h5ad')
# rename sinus-horn cells
adata.obs.replace({celltype_col:
                   {'SinoatrialNodePacemakerCellsHorn':'SinusHornPacemakerCells'}},
                  inplace=True
                 )
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 92164 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_4Jan2025', 'HistologicalAnnotation', 'cluster_cellcharter_k-8', 'cluster_cellcharter_k-10', 'cluster_cellcharter_k-12', 'cluster_cellcharter_k-14', 'cluster_cellcharter_k-16', 'cluster_cellcharter_k-18', 'cluster_cellcharter_k-20', 'cluster_cellcharter_k-25', 'cluster_cellcharter_k-30', 'cluster_cellcharter_k-35', 'cluster_cellcharter_k-40', 'cluster_cellcharter_k-45', 'cluster_cellcharter_k-50', 'predicted_labels', 'conf_score', 'predicted_labels_cellcharter-cluster-k50', 'conf_score_cellcharter-cluster-k50', 'spatial_leiden_1', 'spatial_leiden_5', 'spatial_leiden_0.5', 'spatial_leiden_2', 'spatial_leiden_10', 'spatial_leiden_100', 'spatial_leiden_1000', 'KMeans_k9216_cluster', 'predicted_labels_spatial_KMeans_k9216', 'conf_score_spatial_KMeans_k9216', 'KMeans_k4608_cluster', 'predicted_labels_spatial_KMeans_k4608', 'conf_score_spatial_KMeans_k4608', 'KMeans_k3072_cluster', 'KMeans_k1843_cluster', 'predicted_labels_spatial_KMeans_k3072', 'conf_score_spatial_KMeans_k3072', 'predicted_labels_spatial_KMeans_k1843', 'conf_score_spatial_KMeans_k1843', 'sliding_window_assignment', 'predicted_labels_sliding_window_assignment', 'conf_score_sliding_window_assignment', 'tt_prediction'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'HistologicalAnnotation_colors', 'KMeans_k1843_centroids', 'KMeans_k3072_centroids', 'KMeans_k4608_centroids', 'KMeans_k9216_centroids', 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'cluster_cellcharter_k-10_colors', 'cluster_cellcharter_k-12_colors', 'cluster_cellcharter_k-14_colors', 'cluster_cellcharter_k-16_colors', 'cluster_cellcharter_k-18_colors', 'cluster_cellcharter_k-20_colors', 'cluster_cellcharter_k-25_colors', 'cluster_cellcharter_k-30_colors', 'cluster_cellcharter_k-35_colors', 'cluster_cellcharter_k-40_colors', 'cluster_cellcharter_k-45_colors', 'cluster_cellcharter_k-50_colors', 'cluster_cellcharter_k-8_colors', 'conf_score_cellcharter-cluster-k50_colors', 'conf_score_sliding_window_assignment_colors', 'fine_grain_4Jan2025_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'predicted_labels_cellcharter-cluster-k50_colors', 'predicted_labels_colors', 'predicted_labels_sliding_window_assignment_colors', 'predicted_labels_spatial_KMeans_k1843_colors', 'predicted_labels_spatial_KMeans_k3072_colors', 'predicted_labels_spatial_KMeans_k4608_colors', 'predicted_labels_spatial_KMeans_k9216_colors', 'sliding_window_assignment_colors', 'spatial', 'spatial_leiden_0.5', 'spatial_leiden_1', 'spatial_leiden_10', 'spatial_leiden_100', 'spatial_leiden_1000', 'spatial_leiden_1_colors', 'spatial_leiden_2', 'spatial_leiden_5', 'spatial_leiden_5_colors', 'spatial_neighbors', 'tt_prediction_colors', 'umap'
    obsm: 'X_cellcharter', 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances', 'spatial_connectivities', 'spatial_distances'
```

Example 9 (python):
```python
adata.X.data[:5]
```

Example 10 (unknown):
```unknown
array([1.27187559, 1.95641568, 1.75023687, 1.96760567, 2.7810601 ])
```

Example 11 (python):
```python
# update cell type name
df = pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/finegrain_name_mapping.csv')
celltype_mapping = df.set_index('Full_name')['Short_name_finalised'].to_dict()
adata.obs.replace({celltype_col:celltype_mapping},inplace=True)
```

Example 12 (python):
```python
adata_sub = adata[adata.obs[niche_label_col]==niche_of_interest]
adata_sub
```

Example 13 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 1313 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_4Jan2025', 'HistologicalAnnotation', 'cluster_cellcharter_k-8', 'cluster_cellcharter_k-10', 'cluster_cellcharter_k-12', 'cluster_cellcharter_k-14', 'cluster_cellcharter_k-16', 'cluster_cellcharter_k-18', 'cluster_cellcharter_k-20', 'cluster_cellcharter_k-25', 'cluster_cellcharter_k-30', 'cluster_cellcharter_k-35', 'cluster_cellcharter_k-40', 'cluster_cellcharter_k-45', 'cluster_cellcharter_k-50', 'predicted_labels', 'conf_score', 'predicted_labels_cellcharter-cluster-k50', 'conf_score_cellcharter-cluster-k50', 'spatial_leiden_1', 'spatial_leiden_5', 'spatial_leiden_0.5', 'spatial_leiden_2', 'spatial_leiden_10', 'spatial_leiden_100', 'spatial_leiden_1000', 'KMeans_k9216_cluster', 'predicted_labels_spatial_KMeans_k9216', 'conf_score_spatial_KMeans_k9216', 'KMeans_k4608_cluster', 'predicted_labels_spatial_KMeans_k4608', 'conf_score_spatial_KMeans_k4608', 'KMeans_k3072_cluster', 'KMeans_k1843_cluster', 'predicted_labels_spatial_KMeans_k3072', 'conf_score_spatial_KMeans_k3072', 'predicted_labels_spatial_KMeans_k1843', 'conf_score_spatial_KMeans_k1843', 'sliding_window_assignment', 'predicted_labels_sliding_window_assignment', 'conf_score_sliding_window_assignment', 'tt_prediction'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'HistologicalAnnotation_colors', 'KMeans_k1843_centroids', 'KMeans_k3072_centroids', 'KMeans_k4608_centroids', 'KMeans_k9216_centroids', 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'cluster_cellcharter_k-10_colors', 'cluster_cellcharter_k-12_colors', 'cluster_cellcharter_k-14_colors', 'cluster_cellcharter_k-16_colors', 'cluster_cellcharter_k-18_colors', 'cluster_cellcharter_k-20_colors', 'cluster_cellcharter_k-25_colors', 'cluster_cellcharter_k-30_colors', 'cluster_cellcharter_k-35_colors', 'cluster_cellcharter_k-40_colors', 'cluster_cellcharter_k-45_colors', 'cluster_cellcharter_k-50_colors', 'cluster_cellcharter_k-8_colors', 'conf_score_cellcharter-cluster-k50_colors', 'conf_score_sliding_window_assignment_colors', 'fine_grain_4Jan2025_colors', 'hvg', 'log1p', 'neighbors', 'pca', 'predicted_labels_cellcharter-cluster-k50_colors', 'predicted_labels_colors', 'predicted_labels_spatial_KMeans_k1843_colors', 'predicted_labels_spatial_KMeans_k3072_colors', 'predicted_labels_spatial_KMeans_k4608_colors', 'predicted_labels_spatial_KMeans_k9216_colors', 'spatial', 'spatial_leiden_0.5', 'spatial_leiden_1', 'spatial_leiden_10', 'spatial_leiden_100', 'spatial_leiden_1000', 'spatial_leiden_1_colors', 'spatial_leiden_2', 'spatial_leiden_5', 'spatial_leiden_5_colors', 'spatial_neighbors', 'tt_prediction_colors', 'umap'
    obsm: 'X_cellcharter', 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances', 'spatial_connectivities', 'spatial_distances'
```

Example 14 (python):
```python
# filter cell types
n_cell_thresh = 10
val_counts = adata_sub.obs[celltype_col].value_counts()
celltype_sel = list(val_counts.index[val_counts>n_cell_thresh])
adata_sub = adata_sub[adata_sub.obs[celltype_col].isin(celltype_sel)]
adata_sub.obs[celltype_col].values.describe()
```

Example 15 (python):
```python
meta = adata_sub.obs.copy()
meta['cell_id'] = meta.index
meta = meta[['cell_id', celltype_col]]
meta.to_csv(f'{cellphone_input_dir}/meta_{niche_of_interest}.txt', index=False, sep = "\t")

adata_sub.write(f'{cellphone_input_dir}/log_norm_counts_{niche_of_interest}.h5ad')
```

Example 16 (unknown):
```unknown
f'{cellphone_input_dir}/meta_{niche_of_interest}.txt'
```

Example 17 (unknown):
```unknown
'/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/inputs/meta_SAnode.txt'
```

Example 18 (unknown):
```unknown
f'{cellphone_input_dir}/log_norm_counts_{niche_of_interest}.h5ad'
```

Example 19 (unknown):
```unknown
'/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/cellphonedb/HEA_FOET14880396/inputs/log_norm_counts_SAnode.h5ad'
```

Example 20 (unknown):
```unknown
!ls -lh {cellphone_input_dir}
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no1_bin2cell_HE-nucleus-segmentation.html

**Contents:**
- Read in data¶
- Pre-processing¶
- H&E segmentation¶

The required inputs are the 2um bin output from the quantification pipeline, along with the high resolution H&E image that was used with it. Download both and extract them into the same directory as the notebook before proceeding.

Let's slightly filter the object - require the genes to show up in three spots, and require the spots to have any information at all (the data is extremely sparse at this stage).

Over the course of the demo, bin2cell will perform a segmentation of both the H&E image and a gene expression representation of the data. When performing segmentation, the resolution of the input images is controlled via the mpp parameter. This stands for microns per pixel and translates to how many micrometers are captured in each pixel of the input. For example, if using the array coordinates (present as .obs["array_row"] and .obs["array_col"]) as an image, each of the pixels would have 2 micrometers in it, so the mpp of that particular representation is 2.

In local testing, using an mpp of 0.5 has worked well with both GEX and H&E segmentation.

Since we're already generating a custom resolution H&E image, b2c.scaled_he_image() stores it within the object so it can be used for visualisation. The function crops the image to an area around the actual spatial grid present in the object, and the new coordinates are captured in .obsm["spatial_cropped"]. The new image can be used for plotting by providing basis="spatial_cropped" and img_key="0.5_mpp" to sc.pl.spatial(). For segmentation purposes, the image needs to be saved to the drive, and the function does so to a user-specified save_path.

Visium HD suffers from variable bin sizing. When printing the chips, the 2um bins can have about 10% variability in their width/height. Inspecting the total counts per spot reveals a characteristic striped appearance, with some rows/columns capturing visibly fewer transcripts than others.

To overcome this, b2c.destripe() identifies a user-specified quantile (by default 0.99) of total counts for each row, then divides the counts of the spots in that row by that value. This procedure is then repeated for the columns. .obs[adjusted_counts_key] (by default "n_counts_adjusted") is obtained by multiplying the resulting per-spot factor by the global quantile of count totals, and the count matrix is by default rescaled to match it.

Prior to the adjustment, the count total has a visible striped appearance. The effect is diminished after the correction.

We will begin by performing H&E segmentation with StarDist's H&E model. This is likely to be the main segmentation used in analysis due to consistently picking out nuclei, getting reliable information on the location of cells in the data. It is recommended to lower the prob_thresh, making the model more lenient with regard to what it calls as nuclei - the default setting is quite stringent, while we want to seed a good number of putative cells in the object.

The segmentation results are turned to a sparse matrix and efficiently stored in an .npz file. If desired, they can be read via scipy.sparse.load_npz(), and the sparse matrix's dimensions will match those of the input image.

However, it is important to ensure that the gene expression bin grid is well aligned with the H&E image, or the mismatch will have bins loading segmentation results for sections of the tissue they are quite distant from. Visualising the total expression across parts of the tissue, similar to the plot made to show the "striped" effect earlier, is likely to be helpful in assessing the situation. In the event of a misalignment, the alignment can be corrected via 10X's loupe browser.

We can now load the resulting cell calls into the object. For any given bin, its corresponding pixel in the segmented image will be located and any potential label transferred over. It's important to inform b2c.insert_labels() of whether the segmented image was based on array (GEX visualisation of the grid) or spatial (rescaled H&E image) coordinates, which exact spatial representation matches the image, and what mpp was used.

Let's visualise the nuclear H&E segmentation on our selected region. The nuclei are identified successfully.

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import os

import bin2cell as b2c

#create directory for stardist input/output files
os.makedirs("stardist", exist_ok=True)
```

Example 2 (unknown):
```unknown
os.chdir('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3')
os.getcwd()
```

Example 3 (unknown):
```unknown
'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3'
```

Example 4 (unknown):
```unknown
!ls /lustre/scratch126/cellgen/team205/sharedData/ny1/24-11-26-visiumhd/HEA_FOET14880396/outs
```

Example 5 (unknown):
```unknown
binned_outputs	  metrics_summary.csv  probe_set.csv  web_summary.html
feature_slice.h5  molecule_info.h5     spatial
```

Example 6 (unknown):
```unknown
path = "/lustre/scratch126/cellgen/team205/sharedData/ny1/24-11-26-visiumhd/HEA_FOET14880396/outs/binned_outputs/square_002um"
spaceranger_image_path = '/lustre/scratch126/cellgen/team205/sharedData/ny1/24-11-26-visiumhd/HEA_FOET14880396/outs/spatial'
source_image_path = "/lustre/scratch126/cellgen/team205/kk18/VisiumHD/source_images/C194_HEA_0_FFPE_1_STAN_1EAF2_s3_2024_04_23_13_40_00.tiff"
```

Example 7 (python):
```python
adata = b2c.read_visium(path, 
                        spaceranger_image_path=spaceranger_image_path, 
                        source_image_path = source_image_path
                       )
adata.var_names_make_unique()
adata
```

Example 8 (unknown):
```unknown
anndata.py (1758): Variable names are not unique. To make them unique, call `.var_names_make_unique`.
anndata.py (1758): Variable names are not unique. To make them unique, call `.var_names_make_unique`.
```

Example 9 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5032713 × 18085
    obs: 'in_tissue', 'array_row', 'array_col'
    var: 'gene_ids', 'feature_types', 'genome'
    uns: 'spatial'
    obsm: 'spatial'
```

Example 10 (python):
```python
sc.pp.filter_genes(adata, min_cells=3)
sc.pp.filter_cells(adata, min_counts=1)
adata
```

Example 11 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5001851 × 18051
    obs: 'in_tissue', 'array_row', 'array_col', 'n_counts'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells'
    uns: 'spatial'
    obsm: 'spatial'
```

Example 12 (python):
```python
mpp = 0.3
b2c.scaled_he_image(adata, mpp=mpp, save_path="stardist/he.tiff")
```

Example 13 (unknown):
```unknown
Cropped spatial coordinates key: spatial_cropped_150_buffer
Image key: 0.3_mpp_150_buffer
```

Example 14 (python):
```python
b2c.destripe(adata,
             quantile=0.99, 
    counts_key='n_counts',
    factor_key='destripe_factor',
    adjusted_counts_key='n_counts_adjusted',
    adjust_counts=True)
```

Example 15 (unknown):
```unknown
_construct.py (149): Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
```

Example 16 (python):
```python
#define a mask to easily pull out this region of the object in the future
mask = ((adata.obs['array_row'] >= 1450) & 
        (adata.obs['array_row'] <= 1550) & 
        (adata.obs['array_col'] >= 1400) & 
        (adata.obs['array_col'] <= 1600)
       )

bdata = adata[mask]
sc.pl.spatial(bdata, color=[None, "n_counts", "n_counts_adjusted"], color_map="OrRd",
              img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer")
```

Example 17 (unknown):
```unknown
anndata.py (1146): Trying to modify attribute `.var` of view, initializing view as actual.
anndata.py (1146): Trying to modify attribute `.var` of view, initializing view as actual.
```

Example 18 (python):
```python
sc.pl.spatial(adata,img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer")
```

Example 19 (unknown):
```unknown
KeyboardInterrupt
```

Example 20 (python):
```python
adata.shape
```

Example 21 (unknown):
```unknown
(5001851, 18051)
```

Example 22 (python):
```python
prob_thresh_dict = {
    '0p01':0.01,
     '0p03':0.03,
    '0p05':0.05,
    '0p1':0.1,
}

for key,prob_thresh in prob_thresh_dict.items():
    print(f'### {key} ###')
    b2c.stardist(image_path="stardist/he.tiff", 
                 labels_npz_path=f"stardist/he_{key}.npz", 
                 stardist_model="2D_versatile_he", 
                 prob_thresh=prob_thresh
                )
    print('')
```

Example 23 (unknown):
```unknown
### 0p01 ###
```

Example 24 (unknown):
```unknown
2024-12-05 10:30:10.228900: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-12-05 10:30:10.908609: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-12-05 10:30:11.205982: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1733394611.683311 2850957 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1733394611.753294 2850957 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-12-05 10:30:12.674898: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
```

Example 25 (unknown):
```unknown
# default setting
b2c.stardist(image_path="stardist/he.tiff", 
                 labels_npz_path=f"stardist/he_default.npz", 
                 stardist_model="2D_versatile_he", 
                 # prob_thresh=prob_thresh
                )
```

Example 26 (python):
```python
prob_thresh_dict = {
    '0p01':0.01,
     '0p03':0.03,
    '0p05':0.05,
    '0p1':0.1,
    'default':'default'
}

for key,prob_thresh in prob_thresh_dict.items():
    print(f'### {key} ###')
    b2c.insert_labels(adata, 
                  labels_npz_path=f"stardist/he_{key}.npz", 
                  basis="spatial", 
                  spatial_key="spatial_cropped",
                  mpp=mpp, 
                  labels_key=f"labels_he_{key}"
                 )

adata
```

Example 27 (python):
```python
mask = ((adata.obs['array_row'] >= 1050) & 
        (adata.obs['array_row'] <= 1100) & 
        (adata.obs['array_col'] >= 1100) & 
        (adata.obs['array_col'] <= 1150)
       )

bdata = adata[mask]

for key in prob_thresh_dict.keys():
    #0 means unassigned
    bdata = bdata[bdata.obs[f'labels_he_{key}']>0]
    bdata.obs[f'labels_he_{key}'] = bdata.obs[f'labels_he_{key}'].astype(str)
    print(len(set(bdata.obs[f'labels_he_{key}'])))
    sc.pl.spatial(bdata, color=[None,f'labels_he_{key}'], 
                  img_key="0.3_mpp", basis="spatial_cropped", legend_loc=None,
                  ncols=2
                 )
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/Visium/cell2location/wholeheart_without_DA/no2_mapping_FFPE.html

**Contents:**
  - Loading packages¶
- Functions¶
- Read in metadata¶
- Data, parameters¶
- Loading visium data¶
- Remove mitochondrial genes¶
- Run cell2location mapping¶

**Examples:**

Example 1 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl

import cell2location
import scvi

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text
import seaborn as sns

import os
```

Example 2 (unknown):
```unknown
Global seed set to 0
```

Example 3 (python):
```python
def read_and_qc(sample_name, path):
    r""" This function reads the data for one 10X spatial experiment into the anndata object.
    It also calculates QC metrics. Modify this function if required by your workflow.

    :param sample_name: Name of the sample
    :param path: path to data
    """

    adata = sc.read_visium(path + str(sample_name),
                           count_file='filtered_feature_bc_matrix.h5', load_images=True)
    adata.var_names_make_unique(join='-')
    
    adata.obs['sample'] = sample_name
    
    # rename genes to ENSEMBL
    adata.var['SYMBOL'] = adata.var_names
    adata.var_names = adata.var['gene_ids']
    adata.var_names.name = None

    # Calculate QC metrics
    sc.pp.calculate_qc_metrics(adata, inplace=True)
    adata.var['mt'] = [gene.startswith('MT-') for gene in adata.var['SYMBOL']]
    adata.obs['mt_frac'] = adata[:, adata.var['mt'].tolist()].X.sum(1).A.squeeze()/adata.obs['total_counts']

    # add sample name to obs names
    adata.obs["sample"] = [str(i) for i in adata.obs['sample']]
    adata.obs_names = adata.obs["sample"] \
                          + '_' + adata.obs_names
    adata.obs.index.name = 'spot_id'

    return adata

def select_slide(adata, s, s_col='sample'):
    r""" This function selects the data for one slide from the spatial anndata object.

    :param adata: Anndata object with multiple spatial experiments
    :param s: name of selected experiment
    :param s_col: column in adata.obs listing experiment name for each location
    """

    slide = adata[adata.obs[s_col].isin([s]), :]
    s_keys = list(slide.uns['spatial'].keys())
    s_spatial = np.array(s_keys)[[s in k for k in s_keys]][0]

    slide.uns['spatial'] = {s_spatial: slide.uns['spatial'][s_spatial]}

    return slide
```

Example 4 (python):
```python
meta=pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/metadata/HeartTeamSamples_Mappeddata_20230208.csv',
                index_col=0)
meta=meta[['Publication','sangerID', 'combinedID',
       'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender',
       'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status']]

# select slides for foetal publication
meta=meta[meta['Publication'].isin(['Foetal'])]
# select only FFPE slides
meta=meta[meta['modality'].isin(['Visium-FFPE'])]
# select only Healthy slides
meta=meta[meta['status'].isin(['Healthy'])]

meta = meta.sort_values(['Publication','sangerID'])
```

Example 5 (unknown):
```unknown
sampleIDs = list(meta['sangerID'])
len(sampleIDs)
```

Example 6 (unknown):
```unknown
results_folder = '/nfs/team205/heart/cell2location/foetal'

# create paths and names to results folders for reference regression and cell2location models
ref_run_name = f'{results_folder}/reference_signatures/ver28Feb2023'
run_name = f'{results_folder}/cell2location_map/ver28Feb2023'
```

Example 7 (unknown):
```unknown
visium_path='/nfs/team205/heart/mapped/spaceranger200/'

celltype_key='fine_grain'
```

Example 8 (python):
```python
# Read the data of the interested region into anndata objects
# rename spatial/tissue_positions.csv as spatial/tissue_positions_list.csv
# Read the data of the interested region into anndata objects
slides = []
for i in sampleIDs:
    slides.append(read_and_qc(i, path=visium_path))

# Combine anndata objects together
adata_vis = slides[0].concatenate(
    slides[1:],
    batch_key="sample",
    uns_merge="unique",
    batch_categories=sampleIDs,
    index_unique=None
)
adata_vis
```

Example 9 (unknown):
```unknown
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
```

Example 10 (unknown):
```unknown
AnnData object with n_obs × n_vars = 23294 × 18085
    obs: 'in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac'
    var: 'gene_ids', 'feature_types', 'genome', 'SYMBOL', 'mt', 'n_cells_by_counts-BHF_F_Hea13295830', 'mean_counts-BHF_F_Hea13295830', 'log1p_mean_counts-BHF_F_Hea13295830', 'pct_dropout_by_counts-BHF_F_Hea13295830', 'total_counts-BHF_F_Hea13295830', 'log1p_total_counts-BHF_F_Hea13295830', 'n_cells_by_counts-BHF_F_Hea13295831', 'mean_counts-BHF_F_Hea13295831', 'log1p_mean_counts-BHF_F_Hea13295831', 'pct_dropout_by_counts-BHF_F_Hea13295831', 'total_counts-BHF_F_Hea13295831', 'log1p_total_counts-BHF_F_Hea13295831'
    uns: 'spatial'
    obsm: 'spatial'
```

Example 11 (python):
```python
# filter out low quality spots
sc.pp.filter_cells(adata_vis, min_counts = 5000)
sc.pp.filter_cells(adata_vis, min_genes = 1000)

# Spot number per slide
adata_vis.obs['sample'].value_counts()
```

Example 12 (unknown):
```unknown
BHF_F_Hea13295830    11111
BHF_F_Hea13295831     8862
Name: sample, dtype: int64
```

Example 13 (python):
```python
# find mitochondria-encoded (MT) genes
adata_vis.var['MT_gene'] = [gene.startswith('MT-') for gene in adata_vis.var['SYMBOL']]
print(sum(adata_vis.var['MT_gene']))
    
# remove MT genes for spatial mapping (keeping their counts in the object)
adata_vis.obsm['MT'] = adata_vis[:, adata_vis.var['MT_gene'].values].X.toarray()
adata_vis = adata_vis[:, ~adata_vis.var['MT_gene'].values]
print(sum(adata_vis.var['MT_gene']))
```

Example 14 (python):
```python
# convert adata_vis.var_names to symbol
adata_vis.var.set_index('SYMBOL',inplace=True)
```

Example 15 (python):
```python
%%time


inf_aver = pd.read_csv(f"{ref_run_name}/whole_woPeriCell_woDASMC/inf_aver.csv", index_col=0)

# find shared genes and subset both anndata and reference signatures
intersect = np.intersect1d(adata_vis.var_names, inf_aver.index)
adata_vis = adata_vis[:, intersect].copy()
inf_aver = inf_aver.loc[intersect, :].copy()
print(f'shape of inf_aver: {inf_aver.shape}')

print('reference cell types')
print(inf_aver.columns)

# subset adata
adata_vis=adata_vis.copy()
# prepare anndata for cell2location model
scvi.data.setup_anndata(adata=adata_vis, batch_key="sample")
scvi.data.view_anndata_setup(adata_vis)

# create and train the model
mod = cell2location.models.Cell2location(
    adata_vis, cell_state_df=inf_aver, 
    # the expected average cell abundance: tissue-dependent 
    # hyper-prior which can be estimated from paired histology:
    N_cells_per_location=20,
    # hyperparameter controlling normalisation of
    # within-experiment variation in RNA detection (using default here):
    detection_alpha=20
) 

mod.train(max_epochs=20000, 
          # train using full data (batch_size=None)
          batch_size=None, 
          # use all data points in training because 
          # we need to estimate cell abundance at all locations
          train_size=1,
          use_gpu=True)

# plot ELBO loss history during training, removing first 100 epochs from the plot
mod.plot_history(1000)
plt.legend(labels=['full data training']);

# In this section, we export the estimated cell abundance (summary of the posterior distribution).
adata_vis = mod.export_posterior(
    adata_vis, sample_kwargs={'num_samples': 1000, 'batch_size': mod.adata.n_obs, 'use_gpu': True}
)

# Save model
mod.save(f"{run_name}/FFPE_16PCW", overwrite=True)

# Save anndata object with results
adata_vis.write(f"{run_name}/FFPE_16PCW/sp.h5ad")

# Examine reconstruction accuracy to assess if there are any issues with mapping
# the plot should be roughly diagonal, strong deviations will signal problems
mod.plot_QC()
```

Example 16 (python):
```python
shape of inf_aver: (13540, 57)
reference cell types
Index(['AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight',
       'VentricularCardiomyocytesLeftCompact',
       'VentricularCardiomyocytesRightCompact',
       'VentricularCardiomyocytesLeftTrabeculated',
       'VentricularCardiomyocytesRightTrabeculated',
       'SinoatrialNodeCardiomyocytes', 'AtrioventricularNodeCardiomyocytes',
       'VentricularConductionSystemProximal',
       'VentricularConductionSystemDistal',
       'GreatVesselAdventitialFibroblasts',
       'CoronaryVesselAdventitialFibroblasts',
       'MyocardialInterstitialFibroblasts1',
       'MyocardialInterstitialFibroblasts2', 'Myofibroblasts',
       'LymphNodeFibroblasticReticularCells', 'ValveInterstitialCells',
       'GreatVesselSmoothMuscleCells', 'CoronarySmoothMuscleCells',
       'CoronaryPericytes', 'GreatVesselArterialEndothelialCells',
       'GreatVesselVenousEndothelialCells', 'CoronaryArterialEndothelialCells',
       'CoronaryVenousEndothelialCells', 'CoronaryCapillaryEndothelialCells',
       'EndocardialCells', 'EndocardialCushionCells', 'ValveEndothelialCells',
       'LymphaticEndothelialCells', 'MesothelialEpicardialCells',
       'EpicardiumDerivedCells', 'NeuronPrecursors', 'ChromaffinCells',
       'SympatheticNeurons', 'ParaympatheticNeurons', 'SchwannCellPrecursors',
       'SchwannCells', 'MonocytesMPOpos', 'Monocytes', 'MonocyteDerivedCells',
       'MacrophagesCX3CR1pos', 'MacrophagesTIMD4pos', 'MacrophagesLYVE1pos',
       'MacrophagesATF3pos', 'DendriticCellsType1', 'DendriticCellsMature',
       'PlasmacytoidDendriticCells', 'MastCells', 'Megakaryocytes',
       'TCellsCD4pos', 'TCellsCD8pos', 'TregsCD4pos', 'ProBCells', 'BCells',
       'BCellsMS4A1pos', 'NaturalKillerCells', 'InnateLymphoidCells'],
      dtype='object')
INFO     Using batches from adata.obs["sample"]                                              
INFO     No label_key inputted, assuming all cells have same label                           
INFO     Using data from adata.X
```

Example 17 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function setup_anndata is deprecated; Please use the model-specific setup_anndata methods instead. The global method will be removed in version 0.15.0.
```

Example 18 (python):
```python
INFO     Successfully registered anndata object containing 19973 cells, 13540 vars, 2        
         batches, 1 labels, and 0 proteins. Also registered 0 extra categorical covariates   
         and 0 extra continuous covariates.                                                  
INFO     Please do not further modify adata until model is trained.
```

Example 19 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function view_anndata_setup is deprecated; This method will be removed in 0.15.0. Please avoid building any new dependencies on it.
```

Example 20 (unknown):
```unknown
Anndata setup with scvi-tools version 0.14.4.
```

Example 21 (unknown):
```unknown
Data Summary              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃             Data             ┃ Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│            Cells             │ 19973 │
│             Vars             │ 13540 │
│            Labels            │   1   │
│           Batches            │   2   │
│           Proteins           │   0   │
│ Extra Categorical Covariates │   0   │
│ Extra Continuous Covariates  │   0   │
└──────────────────────────────┴───────┘
```

Example 22 (python):
```python
SCVI Data Registry              
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃     Data      ┃    scvi-tools Location    ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│       X       │          adata.X          │
│ batch_indices │ adata.obs['_scvi_batch']  │
│    labels     │ adata.obs['_scvi_labels'] │
└───────────────┴───────────────────────────┘
```

Example 23 (python):
```python
Label Categories                        
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃      Source Location      ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['_scvi_labels'] │     0      │          0          │
└───────────────────────────┴────────────┴─────────────────────┘
```

Example 24 (python):
```python
Batch Categories                         
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃   Source Location   ┃    Categories     ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['sample'] │ BHF_F_Hea13295830 │          0          │
│                     │ BHF_F_Hea13295831 │          1          │
└─────────────────────┴───────────────────┴─────────────────────┘
```

Example 25 (unknown):
```unknown
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/configuration_validator.py:99: UserWarning: you passed in a val_dataloader but have no validation_step. Skipping val loop
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
```

Example 26 (unknown):
```unknown
Epoch 20000/20000: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20000/20000 [1:34:26<00:00,  3.53it/s, v_num=1, elbo_train=2.93e+8]
Sampling local variables, batch: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [01:08<00:00, 68.84s/it]
Sampling global variables, sample: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 999/999 [00:57<00:00, 17.35it/s]
```

Example 27 (unknown):
```unknown
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'in_tissue' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'array_row' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'array_col' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'feature_types' as categorical
/lustre/scratch117/cellgen/team205/kk18/miniconda3/envs/cell2loc_env/lib/python3.9/site-packages/anndata/_core/anndata.py:1228: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
... storing 'genome' as categorical
```

Example 28 (unknown):
```unknown
CPU times: user 1h 16min 33s, sys: 22min 38s, total: 1h 39min 12s
Wall time: 1h 39min 37s
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/Visium/Others/Plotting_multiple_SAnode.html

**Contents:**
- Loading packages¶
- Read in adata¶
- Plotting¶

**Examples:**

Example 1 (python):
```python
# this is for the jupyter notebook at JupyterLab
# to make sys.path between the terminal and Jupyter same
import sys

print(sys.path) # especially the path to site-packages are different... so thats why couldn't find correct package...

# sys.path from terminal at scvi0190_env
# obtained by excuting `python -c "import sys; print('\n'.join(sys.path))"`
sys.path = [
    "/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python39.zip",
    "/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9",
    "/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/lib-dynload",
    "/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages"
]

print(sys.path)
```

Example 2 (unknown):
```unknown
['/nfs/team205/kk18/notebooks/P48_Foetal-heart_Visium/notebooks/notebooks_for_manuscript', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python39.zip', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/lib-dynload', '', '/nfs/users/nfs_k/kk18/.local/lib/python3.9/site-packages', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages']
['/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python39.zip', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/lib-dynload', '/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages']
```

Example 3 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text
import seaborn as sns

from cell2location.plt import plot_spatial

import os
```

Example 4 (python):
```python
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Global seed set to 0
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/nfs/team205/kk18/miniconda3/envs/cell2loc_v013_env/lib/python3.9/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
```

Example 5 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/_utils.py")
_utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = _utils
spec.loader.exec_module(_utils)
```

Example 6 (python):
```python
sc.settings.verbosity = 1          # verbosity: errors (0), warnings (1), info (2), hints (3)
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 7 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk'
```

Example 8 (python):
```python
adata_vis = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT_foetal_raw.h5ad')
adata_vis
```

Example 9 (unknown):
```unknown
AnnData object with n_obs × n_vars = 14261 × 33538
    obs: 'in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'cell2loc-map_group', 'JC_edit', 'sample_for_cell2loc'
    var: 'gene_ids', 'feature_types', 'genome', 'SYMBOL', 'mt'
    uns: 'spatial'
    obsm: 'MT', 'means_cell_abundance_w_sf', 'q05_cell_abundance_w_sf', 'q95_cell_abundance_w_sf', 'spatial', 'stds_cell_abundance_w_sf'
```

Example 10 (python):
```python
'''
# filtering genes and log-normalise
sc.pp.filter_genes(adata_vis, min_cells=3)
sc.pp.normalize_total(adata_vis, target_sum=1e4)
sc.pp.log1p(adata_vis)

# var.names=SYMBOL
adata_vis.var.set_index('SYMBOL',inplace=True)
adata_vis
'''

# get the cell2loc output
cell2loc_out = adata_vis.obsm['q05_cell_abundance_w_sf'].copy()
## read in cell type name update dictionary
celltype_rename = utils.read_pkl('/nfs/team205/heart/anndata_objects/Foetal/fine_grain_rename_dictionary.pkl')
## update cell type name
cellnames = adata_vis.obsm['q05_cell_abundance_w_sf'].columns
cellnames = [x.split('q05cell_abundance_w_sf_')[1] for x in cellnames]
cellnames = [celltype_rename[x] for x in cellnames]
# add cell2loc output to .obs
adata_vis.obs[cellnames]=cell2loc_out.copy()
adata_vis
```

Example 11 (unknown):
```unknown
AnnData object with n_obs × n_vars = 14261 × 33538
    obs: 'in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'cell2loc-map_group', 'JC_edit', 'sample_for_cell2loc', 'AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight', 'VentricularCardiomyocytesLeftCompact', 'VentricularCardiomyocytesRightCompact', 'VentricularCardiomyocytesLeftTrabeculated', 'VentricularCardiomyocytesRightTrabeculated', 'SinoatrialNodePacemakerCells', 'AtrioventricularNodePacemakerCells', 'VentricularConductionSystemProximal', 'VentricularConductionSystemDistal', 'GreatVesselAdventitialFibroblasts', 'CoronaryVesselAdventitialFibroblasts', 'MyocardialInterstitialFibroblasts', 'SubEpicardialFibroblasts', 'Myofibroblasts', 'LymphNodeFibroblasticReticularCells', 'ValveInterstitialCells', 'GreatVesselSmoothMuscleCells', 'CoronarySmoothMuscleCells', 'CoronaryPericytes', 'GreatVesselArterialEndothelialCells', 'GreatVesselVenousEndothelialCells', 'CoronaryArterialEndothelialCells', 'CoronaryVenousEndothelialCells', 'CoronaryCapillaryEndothelialCells', 'EndocardialCells', 'EndocardialCushionCells', 'ValveEndothelialCells', 'LymphaticEndothelialCells', 'MesothelialEpicardialCells', 'EpicardiumDerivedCells', 'NeuronPrecursors', 'ChromaffinCells', 'SympatheticNeurons', 'ParasympatheticNeurons', 'SchwannCellPrecursors', 'SchwannCells', 'MonocytesMPOpos', 'Monocytes', 'MonocyteDerivedCells', 'MacrophagesCX3CR1pos', 'MacrophagesTIMD4pos', 'MacrophagesLYVE1pos', 'MacrophagesATF3pos', 'DendriticCellsType1', 'DendriticCellsMature', 'PlasmacytoidDendriticCells', 'MastCells', 'Megakaryocytes', 'TCellsCD4pos', 'TCellsCD8pos', 'TregsCD4pos', 'ProBCells', 'BCells', 'BCellsMS4A1pos', 'NaturalKillerCells', 'InnateLymphoidCells'
    var: 'gene_ids', 'feature_types', 'genome', 'SYMBOL', 'mt'
    uns: 'spatial'
    obsm: 'MT', 'means_cell_abundance_w_sf', 'q05_cell_abundance_w_sf', 'q95_cell_abundance_w_sf', 'spatial', 'stds_cell_abundance_w_sf'
```

Example 12 (python):
```python
slide = _utils.select_multislide(adata_vis, ['HCAHeartST10317184'])

clust_labels = ['SinoatrialNodePacemakerCells',
                'AtrioventricularNodePacemakerCells',
                # 'VentricularConductionSystemProximal',
                'VentricularConductionSystemDistal',
               'ParasympatheticNeurons',
                'MacrophagesCX3CR1pos'
                ]
clust_col = ['' + str(i) for i in clust_labels] # in case column names differ from labels

plt.rcParams['axes.grid'] = False
with mpl.rc_context({'figure.figsize': (15, 15)}):
    fig = plot_spatial(
        adata=slide,
        # labels to show on a plot
        color=clust_col, labels=clust_labels,
        show_img=True, img_alpha=0.5,
        # 'fast' (white background) or 'dark_background'
        style='fast',
        # limit color scale at 99.2% quantile of cell abundance
        max_color_quantile=0.997,
        # size of locations (adjust depending on figure size)
        circle_diameter=5,
        colorbar_position='right',
        colorbar_shape = {'vertical_gaps': 0.8, 'horizontal_gaps': 1,
                                    'width': 0.1, 'height': 0.1},
        # crop_x=(650,820),
        # crop_y=(1200,1420),
        reorder_cmap = [3,0,2,1,4] # 'yellow' 'orange' 'blue' 'green' 'purple' 'grey' 'white'
    )
    plt.savefig(f"{figdir}/Visium_SANP-PSNC-CX3CR1MP_HCAHeartST10317184.pdf", bbox_inches="tight",dpi=300)
```

Example 13 (unknown):
```unknown
/nfs/team205/kk18/function/python/_utils.py:95: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  slides.uns['spatial'] = uns_dict
```

Example 14 (python):
```python
# select up to 6 clusters
# https://github.com/BayraktarLab/cell2location/blob/bd2aae833f81d8ab68684aa44f863b36e3af0aed/cell2location/plt/plot_spatial.py#L108

slide = _utils.select_multislide(adata_vis, ['HCAHeartST10317184'])

clust_labels = ['SinoatrialNodePacemakerCells',
                'AtrioventricularNodePacemakerCells',
                # 'VentricularConductionSystemProximal',
                'VentricularConductionSystemDistal',
               'ParasympatheticNeurons',
                'MacrophagesCX3CR1pos'
                ]
clust_col = ['' + str(i) for i in clust_labels] # in case column names differ from labels

plt.rcParams['axes.grid'] = False
with mpl.rc_context({'figure.figsize': (15, 15)}):
    fig = plot_spatial(
        adata=slide,
        # labels to show on a plot
        color=clust_col, labels=clust_labels,
        show_img=True, img_alpha=0.5,
        # 'fast' (white background) or 'dark_background'
        style='fast',
        # limit color scale at 99.2% quantile of cell abundance
        max_color_quantile=0.997,
        # size of locations (adjust depending on figure size)
        circle_diameter=60,
        colorbar_position='right',
        colorbar_shape = {'vertical_gaps': 0.8, 'horizontal_gaps': 1,
                                    'width': 0.1, 'height': 0.1},
        crop_x=(650,820),
        crop_y=(1200,1420),
        reorder_cmap = [3,0,2,1,4] # 'yellow' 'orange' 'blue' 'green' 'purple' 'grey' 'white'
    )
    plt.savefig(f"{figdir}/Visium_SANP-PSNC-CX3CR1MP_cropped_HCAHeartST10317184.pdf", bbox_inches="tight",dpi=300)
```

Example 15 (unknown):
```unknown
/nfs/team205/kk18/function/python/_utils.py:95: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  slides.uns['spatial'] = uns_dict
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no2_bin2cell_expand-label.html

**Contents:**
- Read in data¶
- Pre-processing¶
- Plot HE cell segmentation¶
- Select prob_thresh¶
- Expand labels¶
- Save the object¶

Let's slightly filter the object - require the genes to show up in three spots, and require the spots to have any information at all (the data is extremely sparse at this stage).

Visium HD suffers from variable bin sizing. When printing the chips, the 2um bins can have about 10% variability in their width/height. Inspecting the total counts per spot reveals a characteristic striped appearance, with some rows/columns capturing visibly fewer transcripts than others.

To overcome this, b2c.destripe() identifies a user-specified quantile (by default 0.99) of total counts for each row, then divides the counts of the spots in that row by that value. This procedure is then repeated for the columns. .obs[adjusted_counts_key] (by default "n_counts_adjusted") is obtained by multiplying the resulting per-spot factor by the global quantile of count totals, and the count matrix is by default rescaled to match it.

Prior to the adjustment, the count total has a visible striped appearance. The effect is diminished after the correction.

We can now load the resulting cell calls into the object. For any given bin, its corresponding pixel in the segmented image will be located and any potential label transferred over. It's important to inform b2c.insert_labels() of whether the segmented image was based on array (GEX visualisation of the grid) or spatial (rescaled H&E image) coordinates, which exact spatial representation matches the image, and what mpp was used.

Let's visualise the nuclear H&E segmentation on our selected region. The nuclei are identified successfully.

The package comes with a utility function to visualise StarDist segmentation on a fragment of the input image. As we can see, the calls are faithfully captured in the bins.

However, StarDist's H&E segmentation merely identifies nuclei, while there's more to a cell than just the nucleus. b2c.expand_labels() finds bins up to max_bin_distance (by default 2) bins away from a labelled nucleus, and joins them into the corresponding cell. In the event of a bin being equidistant from two nuclei, it's assigned to one based on the similarity of their gene expression profiles, as represented in PCA space.

it doesn't change --> go with 2

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import os

import bin2cell as b2c

#create directory for stardist input/output files
os.makedirs("stardist", exist_ok=True)
```

Example 2 (unknown):
```unknown
os.chdir('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3')
os.getcwd()
```

Example 3 (unknown):
```unknown
'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3'
```

Example 4 (unknown):
```unknown
!ls /lustre/scratch126/cellgen/team205/sharedData/ny1/24-11-26-visiumhd/HEA_FOET14880396/outs
```

Example 5 (unknown):
```unknown
binned_outputs	  metrics_summary.csv  probe_set.csv  web_summary.html
feature_slice.h5  molecule_info.h5     spatial
```

Example 6 (unknown):
```unknown
path = "/lustre/scratch126/cellgen/team205/sharedData/ny1/24-11-26-visiumhd/HEA_FOET14880396/outs/binned_outputs/square_002um"
spaceranger_image_path = '/lustre/scratch126/cellgen/team205/sharedData/ny1/24-11-26-visiumhd/HEA_FOET14880396/outs/spatial'
source_image_path = "/lustre/scratch126/cellgen/team205/kk18/VisiumHD/source_images/C194_HEA_0_FFPE_1_STAN_1EAF2_s3_2024_04_23_13_40_00.tiff"
```

Example 7 (python):
```python
adata = b2c.read_visium(path, 
                        spaceranger_image_path=spaceranger_image_path, 
                        source_image_path = source_image_path
                       )
adata.var_names_make_unique()
adata
```

Example 8 (unknown):
```unknown
anndata.py (1758): Variable names are not unique. To make them unique, call `.var_names_make_unique`.
anndata.py (1758): Variable names are not unique. To make them unique, call `.var_names_make_unique`.
```

Example 9 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5032713 × 18085
    obs: 'in_tissue', 'array_row', 'array_col'
    var: 'gene_ids', 'feature_types', 'genome'
    uns: 'spatial'
    obsm: 'spatial'
```

Example 10 (python):
```python
sc.pp.filter_genes(adata, min_cells=3)
sc.pp.filter_cells(adata, min_counts=1)
adata
```

Example 11 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5001851 × 18051
    obs: 'in_tissue', 'array_row', 'array_col', 'n_counts'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells'
    uns: 'spatial'
    obsm: 'spatial'
```

Example 12 (python):
```python
mpp = 0.3
b2c.scaled_he_image(adata, mpp=mpp, save_path="stardist/he.tiff")
```

Example 13 (unknown):
```unknown
Cropped spatial coordinates key: spatial_cropped_150_buffer
Image key: 0.3_mpp_150_buffer
```

Example 14 (python):
```python
b2c.destripe(adata,
             quantile=0.99, 
    counts_key='n_counts',
    factor_key='destripe_factor',
    adjusted_counts_key='n_counts_adjusted',
    adjust_counts=True)
```

Example 15 (unknown):
```unknown
_construct.py (149): Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
```

Example 16 (python):
```python
#define a mask to easily pull out this region of the object in the future
mask = ((adata.obs['array_row'] >= 1450) & 
        (adata.obs['array_row'] <= 1550) & 
        (adata.obs['array_col'] >= 1400) & 
        (adata.obs['array_col'] <= 1600)
       )

bdata = adata[mask]
sc.pl.spatial(bdata, color=[None, "n_counts", "n_counts_adjusted"], color_map="OrRd",
              img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer")
```

Example 17 (unknown):
```unknown
anndata.py (1146): Trying to modify attribute `.var` of view, initializing view as actual.
anndata.py (1146): Trying to modify attribute `.var` of view, initializing view as actual.
```

Example 18 (python):
```python
adata.shape
```

Example 19 (unknown):
```unknown
(5001851, 18051)
```

Example 20 (python):
```python
prob_thresh_dict = {
    '0p01':0.01,
     '0p03':0.03,
    '0p05':0.05,
    '0p1':0.1,
    'default':'default'
}

for key,prob_thresh in prob_thresh_dict.items():
    print(f'### {key} ###')
    b2c.insert_labels(adata, 
                  labels_npz_path=f"stardist/he_{key}.npz", 
                  basis="spatial",
                  spatial_key="spatial_cropped_150_buffer",
                  mpp=mpp, 
                  labels_key=f"labels_he_{key}"
                 )

adata
```

Example 21 (unknown):
```unknown
### 0p01 ###
### 0p03 ###
### 0p05 ###
### 0p1 ###
### default ###
```

Example 22 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5001851 × 18051
    obs: 'in_tissue', 'array_row', 'array_col', 'n_counts', 'destripe_factor', 'n_counts_adjusted', 'labels_he_0p01', 'labels_he_0p03', 'labels_he_0p05', 'labels_he_0p1', 'labels_he_default'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells'
    uns: 'spatial', 'bin2cell'
    obsm: 'spatial', 'spatial_cropped_150_buffer'
```

Example 23 (python):
```python
mask = ((adata.obs['array_row'] >= 1050) & 
        (adata.obs['array_row'] <= 1100) & 
        (adata.obs['array_col'] >= 1100) & 
        (adata.obs['array_col'] <= 1150)
       )

bdata = adata[mask]

for key in prob_thresh_dict.keys():
    #0 means unassigned
    bdata = bdata[bdata.obs[f'labels_he_{key}']>0]
    bdata.obs[f'labels_he_{key}'] = bdata.obs[f'labels_he_{key}'].astype(str)
    print(len(set(bdata.obs[f'labels_he_{key}'])))
    sc.pl.spatial(bdata, color=[None,f'labels_he_{key}'], 
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer", legend_loc=None,
                  ncols=2
                 )
```

Example 24 (unknown):
```unknown
2683271253.py (12): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 25 (unknown):
```unknown
2683271253.py (12): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 26 (unknown):
```unknown
2683271253.py (12): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 27 (unknown):
```unknown
2683271253.py (12): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 28 (unknown):
```unknown
2683271253.py (12): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 29 (python):
```python
prob_thresh_sel = '0p05' 
adata.obs['labels_he_sel'] = adata.obs[f'labels_he_{prob_thresh_sel}']
```

Example 30 (python):
```python
#the label viewer wants a crop of the processed image
#get the corresponding coordinates spanning the subset object
crop = b2c.get_crop(bdata, basis="spatial", spatial_key="spatial_cropped_150_buffer", mpp=mpp)

rendered = b2c.view_labels(image_path="stardist/he.tiff", 
                           labels_npz_path=f"stardist/he_{prob_thresh_sel}.npz", 
                           crop=crop
                          )
plt.imshow(rendered)
```

Example 31 (python):
```python
<matplotlib.image.AxesImage at 0x14c6e3bd1c30>
```

Example 32 (python):
```python
%%time
max_bin_distance_list = [1,2,3,4]
for max_bin_distance in max_bin_distance_list:
    print(max_bin_distance)
    b2c.expand_labels(adata, 
                      labels_key=f"labels_he_sel", 
                      expanded_labels_key=f"labels_he_expanded_{max_bin_distance}",
                      max_bin_distance=max_bin_distance
                     )
```

Example 33 (unknown):
```unknown
1
2
3
4
CPU times: user 5min 23s, sys: 14.2 s, total: 5min 37s
Wall time: 7min 18s
```

Example 34 (python):
```python
bdata = adata[mask]

for max_bin_distance in max_bin_distance_list:
    #0 means unassigned
    bdata = bdata[bdata.obs[f"labels_he_expanded_{max_bin_distance}"]>0]
    bdata.obs[f"labels_he_expanded_{max_bin_distance}"] = bdata.obs[f"labels_he_expanded_{max_bin_distance}"].astype(str)

    sc.pl.spatial(bdata, color=[None, f"labels_he_expanded_{max_bin_distance}"], 
                  img_key="0.3_mpp_150_buffer", basis="spatial_cropped_150_buffer")
```

Example 35 (unknown):
```unknown
788255881.py (6): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 36 (unknown):
```unknown
788255881.py (6): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 37 (unknown):
```unknown
788255881.py (6): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 38 (unknown):
```unknown
788255881.py (6): Trying to modify attribute `.obs` of view, initializing view as actual.
```

Example 39 (python):
```python
max_bin_distance_sel = 2
adata.obs['labels_he_expanded_sel'] = adata.obs[f"labels_he_expanded_{max_bin_distance_sel}"]
```

Example 40 (python):
```python
adata.write('/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/HEA_FOET14880396_b2c.h5ad')
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig2_Overview_of_cellular_niches_and_TissueTypist/Visium_tissue-by-celltype_dotplot.html

**Contents:**
- Import required modules¶
- Read in anndata¶
- Preapare cell2location output anndata¶
- Select regions and cells to plot¶
- Dot plot¶

**Examples:**

Example 1 (python):
```python
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import seaborn as sns
import os 
from mpl_toolkits.axes_grid1 import make_axes_locatable

import scipy
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
/home/kk837/.conda/envs/generic_env/lib/python3.10/site-packages/session_info/main.py:213: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  mod_version = _find_version(mod.__version__)
```

Example 4 (python):
```python
-----
anndata             0.11.4
matplotlib          3.10.1
mpl_toolkits        NA
numpy               2.1.3
pandas              2.2.3
scanpy              1.11.1
scipy               1.15.2
seaborn             0.13.2
session_info        v1.0.1
-----
```

Example 5 (python):
```python
Cython              3.1.1
PIL                 11.1.0
asciitree           NA
asttokens           NA
charset_normalizer  3.4.1
cloudpickle         3.1.1
comm                0.2.2
cycler              0.12.1
cython              3.1.1
cython_runtime      NA
dask                2024.11.2
dateutil            2.9.0.post0
debugpy             1.8.13
decorator           5.2.1
exceptiongroup      1.2.2
executing           2.2.0
h5py                3.13.0
igraph              0.11.8
importlib_metadata  NA
ipykernel           6.29.5
jedi                0.19.2
jinja2              3.1.6
joblib              1.4.2
kiwisolver          1.4.8
legacy_api_wrap     NA
leidenalg           0.10.2
llvmlite            0.44.0
markupsafe          3.0.2
msgpack             1.1.0
natsort             8.4.0
numba               0.61.0
numcodecs           0.13.1
packaging           24.2
parso               0.8.4
patsy               1.0.1
platformdirs        4.3.7
prompt_toolkit      3.0.50
psutil              7.0.0
pure_eval           0.2.3
pyarrow             19.0.1
pydev_ipython       NA
pydevconsole        NA
pydevd              3.2.3
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.19.1
pyparsing           3.2.3
pytz                2025.2
six                 1.17.0
sklearn             1.5.2
stack_data          0.6.3
statsmodels         0.14.4
tblib               3.1.0
texttable           1.7.0
threadpoolctl       3.6.0
tlz                 1.0.0
toolz               1.0.0
tornado             6.4.2
traitlets           5.14.3
typing_extensions   NA
wcwidth             0.2.13
yaml                6.0.2
zarr                2.18.3
zipp                NA
zmq                 26.3.0
zoneinfo            NA
```

Example 6 (unknown):
```unknown
-----
IPython             8.34.0
jupyter_client      8.6.3
jupyter_core        5.7.2
-----
Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0]
Linux-4.18.0-553.53.1.el8_10.x86_64-x86_64-with-glibc2.28
-----
Session information updated at 2025-07-14 22:15
```

Example 7 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/rfs/project/rfs-iCNyzSAaucw/kk837/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 8 (python):
```python
sc.settings.verbosity = 1          # verbosity: errors (0), warnings (1), info (2), hints (3)
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 9 (unknown):
```unknown
figdir = '/rfs/project/rfs-iCNyzSAaucw/kk837/notebooks/Foetal/fetal_ms_figures'
```

Example 10 (python):
```python
adata_vis = sc.read_h5ad('/rfs/project/rfs-iCNyzSAaucw/kk837/data_objects/Foetal/Visium/visium_foetal_healthy-all_raw.h5ad')
adata_vis
```

Example 11 (unknown):
```unknown
/home/kk837/.conda/envs/generic_env/lib/python3.10/site-packages/anndata/_core/anndata.py:1758: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
```

Example 12 (unknown):
```unknown
AnnData object with n_obs × n_vars = 39223 × 33562
    obs: 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'cell2loc-map_group', 'JC_edit', 'sample_for_cell2loc', 'week', 'leiden', 'region_cluster', 'AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight', 'VentricularCardiomyocytesLeftCompact', 'VentricularCardiomyocytesRightCompact', 'VentricularCardiomyocytesLeftTrabeculated', 'VentricularCardiomyocytesRightTrabeculated', 'SinoatrialNodeCardiomyocytes', 'AtrioventricularNodeCardiomyocytes', 'VentricularConductionSystemProximal', 'VentricularConductionSystemDistal', 'GreatVesselAdventitialFibroblasts', 'CoronaryVesselAdventitialFibroblasts', 'MyocardialInterstitialFibroblasts1', 'MyocardialInterstitialFibroblasts2', 'Myofibroblasts', 'LymphNodeFibroblasticReticularCells', 'ValveInterstitialCells', 'GreatVesselSmoothMuscleCells', 'CoronarySmoothMuscleCells', 'CoronaryPericytes', 'GreatVesselArterialEndothelialCells', 'GreatVesselVenousEndothelialCells', 'CoronaryArterialEndothelialCells', 'CoronaryVenousEndothelialCells', 'CoronaryCapillaryEndothelialCells', 'EndocardialCells', 'EndocardialCushionCells', 'ValveEndothelialCells', 'LymphaticEndothelialCells', 'MesothelialEpicardialCells', 'EpicardiumDerivedCells', 'NeuronPrecursors', 'ChromaffinCells', 'SympatheticNeurons', 'ParaympatheticNeurons', 'SchwannCellPrecursors', 'SchwannCells', 'MonocytesMPOpos', 'Monocytes', 'MonocyteDerivedCells', 'MacrophagesCX3CR1pos', 'MacrophagesTIMD4pos', 'MacrophagesLYVE1pos', 'MacrophagesATF3pos', 'DendriticCellsType1', 'DendriticCellsMature', 'PlasmacytoidDendriticCells', 'MastCells', 'Megakaryocytes', 'TCellsCD4pos', 'TCellsCD8pos', 'TregsCD4pos', 'ProBCells', 'BCells', 'BCellsMS4A1pos', 'NaturalKillerCells', 'InnateLymphoidCells', 'DuctusArteriosusSmoothMuscleCells', 'AtrialCardiomyocytesLeft_minmax', 'AtrialCardiomyocytesRight_minmax', 'VentricularCardiomyocytesLeftCompact_minmax', 'VentricularCardiomyocytesRightCompact_minmax', 'VentricularCardiomyocytesLeftTrabeculated_minmax', 'VentricularCardiomyocytesRightTrabeculated_minmax', 'SinoatrialNodeCardiomyocytes_minmax', 'AtrioventricularNodeCardiomyocytes_minmax', 'VentricularConductionSystemProximal_minmax', 'VentricularConductionSystemDistal_minmax', 'GreatVesselAdventitialFibroblasts_minmax', 'CoronaryVesselAdventitialFibroblasts_minmax', 'MyocardialInterstitialFibroblasts1_minmax', 'MyocardialInterstitialFibroblasts2_minmax', 'Myofibroblasts_minmax', 'LymphNodeFibroblasticReticularCells_minmax', 'ValveInterstitialCells_minmax', 'GreatVesselSmoothMuscleCells_minmax', 'CoronarySmoothMuscleCells_minmax', 'CoronaryPericytes_minmax', 'GreatVesselArterialEndothelialCells_minmax', 'GreatVesselVenousEndothelialCells_minmax', 'CoronaryArterialEndothelialCells_minmax', 'CoronaryVenousEndothelialCells_minmax', 'CoronaryCapillaryEndothelialCells_minmax', 'EndocardialCells_minmax', 'EndocardialCushionCells_minmax', 'ValveEndothelialCells_minmax', 'LymphaticEndothelialCells_minmax', 'MesothelialEpicardialCells_minmax', 'EpicardiumDerivedCells_minmax', 'NeuronPrecursors_minmax', 'ChromaffinCells_minmax', 'SympatheticNeurons_minmax', 'ParaympatheticNeurons_minmax', 'SchwannCellPrecursors_minmax', 'SchwannCells_minmax', 'MonocytesMPOpos_minmax', 'Monocytes_minmax', 'MonocyteDerivedCells_minmax', 'MacrophagesCX3CR1pos_minmax', 'MacrophagesTIMD4pos_minmax', 'MacrophagesLYVE1pos_minmax', 'MacrophagesATF3pos_minmax', 'DendriticCellsType1_minmax', 'DendriticCellsMature_minmax', 'PlasmacytoidDendriticCells_minmax', 'MastCells_minmax', 'Megakaryocytes_minmax', 'TCellsCD4pos_minmax', 'TCellsCD8pos_minmax', 'TregsCD4pos_minmax', 'ProBCells_minmax', 'BCells_minmax', 'BCellsMS4A1pos_minmax', 'NaturalKillerCells_minmax', 'InnateLymphoidCells_minmax', 'DuctusArteriosusSmoothMuscleCells_minmax', 'annotation', 'annotation_final', 'annotation_final_coarse', 'annotation_final_mod'
    var: 'gene_ids'
    uns: 'JC_edit_colors', 'annotation_colors', 'annotation_final_colors', 'cell2loc-map_group_colors', 'leiden', 'neighbors', 'region_cluster_colors', 'sample_for_cell2loc_colors', 'umap'
    obsm: 'MT', 'X_umap', 'means_cell_abundance_w_sf', 'q05_cell_abundance_w_sf', 'q05_cell_abundance_w_sf_ndarray', 'q95_cell_abundance_w_sf', 'spatial', 'stds_cell_abundance_w_sf'
    obsp: 'connectivities', 'distances'
```

Example 13 (python):
```python
# modify annotations for matching with factor analysis
annotation_edit_dict = {
   'AV valves':'Valves',
    'Aortic valve':'Valves',
    'Valve apparatus':'Valves',
    'Atrium - Left':'Atrium',
     'Atrium - Right':'Atrium',
    'Ventricle - Compact - Left':'Ventricle - Compact',
    'Ventricle - Compact - Right':'Ventricle - Compact',
    'Ventricle - Trabeculated - Left':'Ventricle - Trabeculated',
    'Ventricle - Trabeculated - Right':'Ventricle - Trabeculated',
}
adata_vis.obs['annotation_final_mod']=adata_vis.obs['annotation_final'].astype('str').copy()
adata_vis.obs.replace({'annotation_final_mod':annotation_edit_dict}, 
                      inplace=True)
adata_vis.obs['annotation_final_mod'] = adata_vis.obs['annotation_final_mod'].astype('category')

#reorder annotations
annotations_order = [
   'Atrium',
        'Ventricle - Compact',
     'Ventricle - Trabeculated',
        'SAnode',
        'AVnode',
    'VCS - Proximal',
        'VCS - Distal',
        'Great vessel',
        'Coronary vessel',
    'Ductus arteriosus',
        'Adventitia - Coronary',
     'Adventitia - GV',
    'Epicardium',
        'Endocardium - Atrial', 
    'Endocardium - Ventricular',
        'Endocardial cusion',
        'AV ring',
    'Valves',
    'LN',
            ]
adata_vis.obs['annotation_final_mod'] = adata_vis.obs['annotation_final_mod'].cat.reorder_categories(annotations_order)
```

Example 14 (python):
```python
# get the cell2loc output
cell2loc_out = adata_vis.obsm['q05_cell_abundance_w_sf'].copy()
## read in cell type name update dictionary
celltype_rename = utils.read_pkl('/rfs/project/rfs-iCNyzSAaucw/kk837/data_objects/Foetal/fine_grain_rename_dictionary.pkl')
## update cell type name
cellnames = adata_vis.obsm['q05_cell_abundance_w_sf'].columns
cellnames = [x.split('q05cell_abundance_w_sf_')[1] for x in cellnames]
cellnames = [celltype_rename[x] for x in cellnames]

# select obs columns
obs_columns = ['array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes',
               'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 
               'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei',
               'modality', 'kit_10x', 'flushed', 'status', 'cell2loc-map_group', 'JC_edit', 'sample_for_cell2loc', 'week', 'leiden', 'annotation_final','annotation_final_mod']

# make anndata
adata_c2l = anndata.AnnData(
    X=adata_vis.obsm['q05_cell_abundance_w_sf_ndarray'].copy(),
    obs=adata_vis.obs[obs_columns],
    var=pd.DataFrame(cellnames).set_index(0),
    obsm=adata_vis.obsm.copy(),
    obsp=adata_vis.obsp.copy(),
    uns=adata_vis.uns.copy()
)
adata_c2l
```

Example 15 (unknown):
```unknown
AnnData object with n_obs × n_vars = 39223 × 58
    obs: 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'sangerID', 'Publication', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'cell2loc-map_group', 'JC_edit', 'sample_for_cell2loc', 'week', 'leiden', 'annotation_final', 'annotation_final_mod'
    uns: 'JC_edit_colors', 'annotation_colors', 'annotation_final_colors', 'cell2loc-map_group_colors', 'leiden', 'neighbors', 'region_cluster_colors', 'sample_for_cell2loc_colors', 'umap'
    obsm: 'MT', 'X_umap', 'means_cell_abundance_w_sf', 'q05_cell_abundance_w_sf', 'q05_cell_abundance_w_sf_ndarray', 'q95_cell_abundance_w_sf', 'spatial', 'stds_cell_abundance_w_sf'
    obsp: 'connectivities', 'distances'
```

Example 16 (python):
```python
df = pd.read_csv('/rfs/project/rfs-iCNyzSAaucw/kk837/notebooks/Foetal/finegrain_name_mapping.csv').set_index('Full_name')
mapping = df['Short_name_finalised'].to_dict()
'AtrialCardiomyocytesLeft' in mapping.keys()
```

Example 17 (python):
```python
# map fine-grained cell types to short version# map cell type name to short versions
df = pd.read_csv('/rfs/project/rfs-iCNyzSAaucw/kk837/notebooks/Foetal/finegrain_name_mapping.csv').set_index('Full_name')
mapping = df['Short_name_finalised'].to_dict()
adata_c2l.var_names = [mapping[x] if x in mapping.keys() else x for x in adata_c2l.var_names]
adata_c2l.var
```

Example 18 (python):
```python
# make the value 0 below an arbitrary threshold
# to calculate proportion
abundance_thresh = 0.05
adata_c2l.raw = adata_c2l
adata_c2l.X[adata_c2l.X<abundance_thresh] = 0
adata_c2l.X = scipy.sparse.csr_matrix(adata_c2l.X)
```

Example 19 (python):
```python
/tmp/ipykernel_295047/1462688745.py:5: SparseEfficiencyWarning: Comparing a sparse matrix with a scalar greater than zero using < is inefficient, try using >= instead.
  adata_c2l.X[adata_c2l.X<abundance_thresh] = 0
/home/kk837/.conda/envs/generic_env/lib/python3.10/site-packages/scipy/sparse/_index.py:210: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil and dok are more efficient.
  self._set_arrayXarray(i, j, x)
```

Example 20 (python):
```python
# select regions
# regions without LN
regions_sel = [ 'Great vessel',
               'Ductus arteriosus',
               'Coronary vessel',
                'Adventitia - GV',
               'Adventitia - Coronary',
               'AV ring',
               'Valves',
               'Atrium',
               'SAnode',
               'AVnode',
               'VCS - Proximal',
               'VCS - Distal',
               'Ventricle - Trabeculated',
               'Ventricle - Compact',
                'Epicardium',
               'Endocardium - Atrial',
               'Endocardium - Ventricular',        
               'Endocardial cusion',
 ]
ad = adata_c2l[adata_c2l.obs['annotation_final_mod'].isin(regions_sel)]
```

Example 21 (python):
```python
# select cells to plot based on expression proportion

## calculate expression proportion
exp_prop = utils.get_exp_proportion(ad,groupby='annotation_final_mod', use_rep='X')

## select cells which are above an arbitrary threshold at any of the niches
prop_thresh = 0.4
cells_sel = list(exp_prop.max(axis=1).index[exp_prop.max(axis=1)>prop_thresh])

# get cell order based on hierarchical clustering
## mean abundance per niche
exp_mean = utils.grouped_obs_mean(ad[:,cells_sel], group_key='annotation_final_mod').T
## min-max normalisation
exp_mean_scaled = (exp_mean-exp_mean.min())/(exp_mean.max()-exp_mean.min())
## clustering
g = sns.clustermap(exp_mean_scaled.T, figsize=(10,15), dendrogram_ratio=0.1)
row_order = g.dendrogram_row.reordered_ind
cell_order = exp_mean_scaled.columns[row_order]
```

Example 22 (python):
```python
ad = adata_c2l[adata_c2l.obs['annotation_final_mod'].isin(regions_sel)]
ad.obs['annotation_final_mod'] = ad.obs['annotation_final_mod'].astype('category')
ad.obs['annotation_final_mod'] = ad.obs['annotation_final_mod'].cat.reorder_categories(regions_sel)

sc.tl.dendrogram(ad, groupby='annotation_final_mod')

sc.pl.dotplot(ad,
              var_names=cell_order,
              standard_scale='var',
              groupby='annotation_final_mod',
                               dendrogram=True,
             cmap='YlGn',
              colorbar_title='Mean abundance\nin group',
              show=False
                  )
plt.savefig(f"{figdir}/visium_tissue-by-celltype_dotplot.pdf", bbox_inches="tight",dpi=300)
```

Example 23 (python):
```python
/tmp/ipykernel_295047/3506546109.py:2: ImplicitModificationWarning: Trying to modify attribute `.obs` of view, initializing view as actual.
  ad.obs['annotation_final_mod'] = ad.obs['annotation_final_mod'].astype('category')
/home/kk837/.conda/envs/generic_env/lib/python3.10/site-packages/scanpy/tools/_utils.py:40: UserWarning: You’re trying to run this on 58 dimensions of `.X`, if you really want this, set `use_rep='X'`.
         Falling back to preprocessing with `sc.pp.pca` and default params.
  warnings.warn(
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no5_celltypist_prediction.html

**Contents:**
- Variables¶
- Read in data¶
- Normalise the counts¶
- UMAP¶
- CellTypist prediction: coarse-grain¶
- CellTypist prediction: mid-grain¶
- celltypist prediction: fine-grain¶
- Save¶

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import celltypist
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 4 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects'
sample_id = 'HEA_FOET14880396'
```

Example 5 (unknown):
```unknown
os.getcwd()
```

Example 6 (unknown):
```unknown
'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3'
```

Example 7 (python):
```python
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_b2c_cells_filtered.h5ad')
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 129803 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts'
    uns: 'spatial'
    obsm: 'spatial', 'spatial_cropped_150_buffer'
```

Example 9 (python):
```python
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
```

Example 10 (python):
```python
adata.X.data[:5]
```

Example 11 (unknown):
```unknown
array([1.27187559, 1.95641568, 1.75023687, 1.96760567, 2.7810601 ])
```

Example 12 (python):
```python
# sc.pp.regress_out(adata, ["total_counts", "pct_counts_mt"])
```

Example 13 (python):
```python
# HVGs
sc.pp.highly_variable_genes(adata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
sc.pl.highly_variable_genes(adata)
print(f'number of hvgs: {sum(adata.var.highly_variable)}')
# PCA
sc.tl.pca(adata, use_highly_variable=True)
# create neighbour graph
sc.pp.neighbors(adata,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata,min_dist=0.2)
sc.pl.embedding(adata,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"pct_counts_mt"],wspace=0.2,cmap='RdPu',vmax='p100')
```

Example 14 (unknown):
```unknown
number of hvgs: 3066
```

Example 15 (python):
```python
# coarse-grained
predictions = celltypist.annotate(adata, 
                                  model = '/nfs/team205/kk18/notebooks/Foetal/VisiumHD/mpp0p3/celltypist_models/from_foetal_portal/foetal_heart_coarse.pkl', 
                                  majority_voting = False)

# add to anndata
adata = predictions.to_adata(insert_labels = True, insert_conf = True)
adata.obs.rename(columns={'predicted_labels':'celltypist_coarse',
                              'conf_score':'conf_score_coarse'
                             },inplace=True)
adata.obs.head()
```

Example 16 (unknown):
```unknown
🔬 Input data has 129803 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 17059 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 17 (python):
```python
sc.pl.embedding(adata,
                    basis=f"X_umap",
                    color=['celltypist_coarse','conf_score_coarse'],
                   ncols=4,wspace=0.5
                   )
```

Example 18 (python):
```python
# celltypist confidence
plt.figure(figsize=(6, 2))
sns.histplot(adata.obs[f'conf_score_coarse'], kde=False, bins=100)
# plt.yscale('log')
plt.title('coarse_grained')
plt.show()
```

Example 19 (python):
```python
# check
conf_thresh = 0.5 # starting from relaxed threshold
pd.crosstab(adata.obs['celltypist_coarse'],adata.obs[f'conf_score_coarse']<conf_thresh)
```

Example 20 (python):
```python
# finalised coarse-grained labels
conf_thresh = 0.5
adata.obs['celltypist_coarse_fin'] = adata.obs['celltypist_coarse'].astype('str').copy()
low_quality_cells = adata.obs_names[adata.obs[f'conf_score_coarse']<conf_thresh]
adata.obs.loc[low_quality_cells,'celltypist_coarse_fin'] = 'unassigned'
adata.obs['celltypist_coarse_fin'].value_counts()
```

Example 21 (unknown):
```unknown
celltypist_coarse_fin
Cardiomyocytes    76088
Mesenchymal       36370
unassigned        10482
Endothelium        4380
Neural             1254
Leukocytes          626
Epicardium          603
Name: count, dtype: int64
```

Example 22 (python):
```python
sc.pl.embedding(adata,
                    basis=f"X_umap",
                    color=['celltypist_coarse_fin'],
                   ncols=4,wspace=0.5
                   )
```

Example 23 (unknown):
```unknown
... storing 'celltypist_coarse_fin' as categorical
```

Example 24 (python):
```python
sc.pl.embedding(adata,
                    basis=f"X_umap",
                    color=['celltypist_coarse_fin'],groups='unassigned',
                   ncols=4,wspace=0.5
                   )
```

Example 25 (python):
```python
adata.obs['celltypist_coarse2midmod'] = np.nan
adata.obs['conf_score_coarse2midmod'] = np.nan

for celltype in adata.obs['celltypist_coarse_fin'].unique():
    if celltype not in ['unassigned', 'Epicardium']: # 'Epicardium' is also 'Epicardium' in mid-grained
        print(celltype)
        # subset
        adata_sub = adata[adata.obs['celltypist_coarse_fin']==celltype]
        # predict
        predictions = celltypist.annotate(adata_sub, 
                                      model = f'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/mpp0p3/celltypist_models/coarse2midmod_{celltype}.pkl', 
                                      majority_voting = False)
        # add to anndata
        adata_sub = predictions.to_adata(insert_labels = True, insert_conf = True)
        adata.obs.loc[adata_sub.obs_names,'celltypist_coarse2midmod'] = adata_sub.obs['predicted_labels']
        adata.obs.loc[adata_sub.obs_names,'conf_score_coarse2midmod'] = adata_sub.obs['conf_score']
        print('')
```

Example 26 (unknown):
```unknown
🔬 Input data has 626 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 16956 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 27 (unknown):
```unknown
Mesenchymal
```

Example 28 (unknown):
```unknown
🔬 Input data has 36370 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 17166 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 29 (unknown):
```unknown
Endothelium
```

Example 30 (unknown):
```unknown
🔬 Input data has 4380 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 16989 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 31 (unknown):
```unknown
🔬 Input data has 1254 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 16779 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 32 (unknown):
```unknown
Cardiomyocytes
```

Example 33 (unknown):
```unknown
🔬 Input data has 76088 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 16534 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 34 (python):
```python
sc.pl.embedding(adata,
                    basis=f"X_umap",
                    color=['celltypist_coarse2midmod','conf_score_coarse2midmod'],
                   ncols=4,wspace=0.5
                   )
```

Example 35 (unknown):
```unknown
... storing 'celltypist_coarse2midmod' as categorical
```

Example 36 (python):
```python
# check
conf_thresh = 0.8
pd.crosstab(adata.obs['celltypist_coarse2midmod'],adata.obs[f'conf_score_coarse2midmod']<conf_thresh)
```

Example 37 (python):
```python
# finalised mid-grained labels
conf_thresh = 0.8
adata.obs['celltypist_mid_fin'] = adata.obs['celltypist_coarse_fin'].astype('str').copy()
adata.obs.replace({'celltypist_mid_fin':{'Epicardium':'EpicardialCells'}},inplace=True)
passed = adata.obs_names[adata.obs['conf_score_coarse2midmod']>=conf_thresh]
low_quality_cells = adata.obs_names[adata.obs[f'conf_score_coarse2midmod']<conf_thresh]

adata.obs.loc[passed,'celltypist_mid_fin'] = adata.obs.loc[passed,'celltypist_coarse2midmod'].copy()
adata.obs.loc[low_quality_cells,'celltypist_mid_fin'] = 'unassigned'
adata.obs['celltypist_mid_fin'].value_counts()
```

Example 38 (unknown):
```unknown
celltypist_mid_fin
VentricularCardiomyocytes      61075
Fibroblasts                    20318
unassigned                     13951
AtrialCardiomyocytes           13904
MuralCells                     10785
PericardialCells                3061
BloodVesselEndothelialCells     1765
EndocardialCells                1526
LymphaticEndothelialCells       1025
EpicardialCells                  603
Neurons                          589
Glia                             588
MyeloidCells                     526
LymphoidCells                     87
Name: count, dtype: int64
```

Example 39 (python):
```python
sc.pl.embedding(adata,
                    basis=f"X_umap",
                    color=['celltypist_mid_fin'],
                   ncols=4,wspace=0.5
                   )
```

Example 40 (unknown):
```unknown
... storing 'celltypist_mid_fin' as categorical
```

Example 41 (python):
```python
adata.obs['celltypist_midmod2fine'] = np.nan
adata.obs['conf_score_midmod2fine'] = np.nan

for celltype in adata.obs['celltypist_mid_fin'].unique():
    if celltype not in ['unassigned', 'LymphaticEndothelialCells']: # 'LymphaticEndothelialCells' is also 'LymphaticEndothelialCells' in fine-grained
        print(celltype)
        # subset
        adata_sub = adata[adata.obs['celltypist_mid_fin']==celltype]
        # predict
        predictions = celltypist.annotate(adata_sub, 
                                      model = f'/nfs/team205/kk18/notebooks/Foetal/VisiumHD/mpp0p3/celltypist_models/midmod2fine_{celltype}.pkl', 
                                      majority_voting = False)
        # add to anndata
        adata_sub = predictions.to_adata(insert_labels = True, insert_conf = True)
        adata.obs.loc[adata_sub.obs_names,'celltypist_midmod2fine'] = adata_sub.obs['predicted_labels']
        adata.obs.loc[adata_sub.obs_names,'conf_score_midmod2fine'] = adata_sub.obs['conf_score']
        print('')
```

Example 42 (unknown):
```unknown
LymphoidCells
```

Example 43 (unknown):
```unknown
🔬 Input data has 87 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 17008 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 44 (unknown):
```unknown
MyeloidCells
```

Example 45 (unknown):
```unknown
🔬 Input data has 526 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 17087 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 46 (unknown):
```unknown
Fibroblasts
```

Example 47 (unknown):
```unknown
🔬 Input data has 20318 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 17022 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 48 (unknown):
```unknown
PericardialCells
```

Example 49 (unknown):
```unknown
🔬 Input data has 3061 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 16563 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 50 (unknown):
```unknown
BloodVesselEndothelialCells
```

Example 51 (unknown):
```unknown
🔬 Input data has 1765 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 16931 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 52 (unknown):
```unknown
🔬 Input data has 10785 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 16683 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 53 (unknown):
```unknown
🔬 Input data has 589 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 16254 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 54 (unknown):
```unknown
🔬 Input data has 588 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 16388 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 55 (unknown):
```unknown
EpicardialCells
```

Example 56 (unknown):
```unknown
🔬 Input data has 603 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 16982 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 57 (unknown):
```unknown
AtrialCardiomyocytes
```

Example 58 (unknown):
```unknown
🔬 Input data has 13904 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 16481 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 59 (unknown):
```unknown
EndocardialCells
```

Example 60 (unknown):
```unknown
🔬 Input data has 1526 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 16303 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 61 (unknown):
```unknown
VentricularCardiomyocytes
```

Example 62 (unknown):
```unknown
🔬 Input data has 61075 cells and 18047 genes
🔗 Matching reference genes in the model
🧬 16613 features used for prediction
⚖️ Scaling input data
🖋️ Predicting labels
✅ Prediction done!
```

Example 63 (python):
```python
sc.pl.embedding(adata,
                    basis=f"X_umap",
                    color=['celltypist_midmod2fine','conf_score_midmod2fine'],
                   ncols=1,wspace=0.5
                   )
```

Example 64 (unknown):
```unknown
... storing 'celltypist_midmod2fine' as categorical
```

Example 65 (python):
```python
adata.write(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 66 (unknown):
```unknown
AnnData object with n_obs × n_vars = 129803 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'spatial', 'log1p', 'hvg', 'pca', 'neighbors', 'umap', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_coarse2midmod_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors'
    obsm: 'spatial', 'spatial_cropped_150_buffer', 'X_pca', 'X_umap'
    varm: 'PCs'
    obsp: 'distances', 'connectivities'
```

Example 67 (python):
```python
print(f'{adata_dir}/{sample_id}_b2c_cells_filtered_lognorm.h5ad')
```

Example 68 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/HEA_FOET14880396_b2c_cells_filtered_lognorm.h5ad
```

---
